Title: DMA: Enhancing Retrieval-Augmented Generation with Adaptive Human Feedback
Venue: NeurIPS 2025
Accepted: no
Keywords: Retrieval-Augmented Generation (RAG)

Online Learning
Abstract: Retrieval-augmented generation (RAG) systems typically rely on static retrieval methods, limiting their adaptability to dynamic environments. In this paper, we propose a novel online learning framework called Dynamic Memory Alignment (DMA), designed specifically to enhance retrieval performance and content generation in RAG through adaptive incorporation of multi-level human feedback. DMA systematically integrates real-time feedback signals at document, list, and response levels, effectively adjusting memory management strategies to optimize relevance and adaptability in online interactive environments. Extensive evaluations demonstrate DMA’s competitive foundational retrieval performance across multiple standard knowledge-intensive benchmarks. Notably, DMA achieves significant advantages on datasets reflecting natural conversational interactions (TriviaQA, HotpotQA), highlighting its particular suitability for online GenAI dialogue applications. Moreover, a multi-month industrial deployment demonstrates that DMA substantially improves user engagement in real-world applications. These results underscore DMA’s ability to maintain robust foundational retrieval capabilities while excelling at dynamic, real-time adaptation in interactive online environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Dynamic Memory Alignment (DMA), a novel online learning framework for enhancing Retrieval-Augmented Generation (RAG) systems. The key innovation lies in integrating multi-level human feedback—spanning document, list, and response levels—into the continuous adaptation of both retrieval and reranking modules. By systematically capturing and modeling real-time user preferences, DMA aims to address the inherent rigidity of static retrieval pipelines. The authors demonstrate the framework’s efficacy through extensive experiments, including a large-scale industrial randomized controlled trial (RCT) and evaluations on multiple public QA benchmarks. Results show significant gains in user satisfaction and competitive performance relative to state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper tackles a critical limitation in existing RAG systems: their inability to adapt to evolving user intent and content drift due to static retrieval mechanisms. DMA's design for real-time, feedback-driven adaptation offers a meaningful advancement.\n2. The paper introduces a well-structured taxonomy for modeling feedback at three levels—document, list, and response—ensuring holistic alignment between retrieval decisions and user intent.\n3. The authors provide rigorous empirical evidence, including industrial-scale deployment results and strong performance on public QA benchmarks, demonstrating both practical viability and generalizability of the proposed method.\n\n### Weaknesses\n1. Several critical components of the DMA framework lack adequate description:\n  - The paper briefly mentions that list-level feedback incorporates both explicit and implicit signals (Line 172) but fails to define what constitutes implicit feedback or how it is transformed into training-ready preference labels.\n  - The knowledge distillation process from upstream rerankers to the final GBDT model (Lines 211-213) remains under-explained. Key questions about soft label generation and GBDT training procedures (loss function, input features, etc.) remain unanswered.\n  - Important implementation details are missing, including hyperparameters for the rerankers and PPO optimization, the exact number of retrieved documents (k), and architectural specifics of the reranking models.\n2. The paper suffers from several notational ambiguities that hinder readability:\n  - The symbol R is overloaded, used for both the retriever (Line 116) and the reward model (Line 203).\n  - The symbol y refers inconsistently to a binary preference label (Line 169), a generated system response (Line 171), and a preference over responses (Line 176). Similarly, symbols a (Line 122) and r (Line 178) both denote responses in different contexts.\n  - Figure 2 introduces terms like “sat” and (*) without clear explanation.\n  - The first feedback level is called “Point Level” in Figure 1 but “Document-level” in the main text (Line 167), causing confusion.\n  - Additionally, all formulas are presented inline, making the presentation of key objectives like the PPO loss difficult to follow. Block-level math formatting would substantially improve clarity.\n3. Some conclusions in the paper would benefit from stronger quantitative backing:\n  - The “Takeaway” section in the ablation study (Lines 316–318) provides only qualitative observations about the relative importance of feedback levels. This discussion would be more convincing if supported by quantitative metrics or a more detailed breakdown of ablation results.\n  - The paper would also benefit from more concrete case studies or qualitative examples showing how DMA adapts to user feedback in specific, realistic scenarios.\n\nQUESTIONS:\n1. Could the authors provide concrete examples of what constitutes \"implicit feedback\" at the list level? Additionally, what heuristics or modeling strategies are used to transform these signals into actionable training labels?\n2. How exactly is the ListNet loss (Line 199) computed? Specifically, how are the ground-truth relevance scores for the target distribution $P_{\\text{true}}$ derived from user feedback? Are they heuristic, probabilistic, or model-predicted?\n3. The paper states that BGE-m3 is used as the retriever backbone (Line 227), but what architectures are used for the pointwise and listwise rerankers? Are they also BGE-based, or do they use different encoder models?\n4. Could the authors formally define the state space, action space, and the reward function used in the PPO-based listwise reranker training (Lines 205–210)? This is critical for understanding the RL framing and reproducibility.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a critical limitation of conventional Retrieval-Augmented Generation (RAG) systems: their reliance on static retrieval strategies, which hinders adaptability to dynamic user intents or evolving content in real-time applications (e.g., chatbots). To solve this, the authors propose Dynamic Memory Alignment (DMA), a novel online learning framework that continuously optimizes RAG retrieval through multi-level human feedback. DMA bridges the gap between static RAG pipelines and dynamic user environments by establishing a closed-loop system where retrieval strategies evolve via real-time feedback, enabling sustained performance in open-ended, user-facing GenAI applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Novel hierarchical modeling (document/list/response/session) with specialized objectives: BCE (document), ListNet (list), PPO+reward modeling (response), and session-guided fusion.\n\n2. Industrial-Scale Validation: 24.57% ↑ user satisfaction in 100k-session RCT and SOTA on TriviaQA/HotpotQA provide strong empirical backing.\n\n3. Propose a novel framework to jointly optimize document-, list-, and response-level signals via PPO+distillation.\n\nWeaknesses:\n1. About the limited generalization analysis:\n- Schema-Bound QA Underperformance: Competitive but not SOTA on NQ/WebQSP (Table 3). Why does DMA lag in entity-centric tasks? Insufficient discussion.\n- Feedback Sparsity Mitigation: \"Adaptive scheduling\" (§5.1) defers updates but lacks low-resource solutions (e.g., synthetic feedback).\n\n2. About deployment Ethics and over-personalization risks: \n- Bias Amplification: Human feedback may reinforce retrieval biases (e.g., prioritizing popular documents). §6 mentions \"echo chambers\" but offers no guardrails.\n- Privacy Tradeoffs: Anonymization protects users but obscures demographic fairness analysis (e.g., satisfaction gaps across user groups)\n\n3. Deployment on proprietary infrastructure (telecom provider) and unreleased code/data limit independent validation.\n\nQUESTIONS:\n1. Why does DMA underperform on schema-bound QA (e.g., NQ, WebQSP in Table 3) despite excelling in conversational tasks? Is this due to feedback bias, architecture limitations, or data distribution?\n\n2. What exact input features (e.g., embeddings, ranking scores) and fusion methods (e.g., weighted logits, multi-loss) are used for GBDT distillation?\n\n3. Can core innovations (e.g., feedback scheduling, distillation) be validated without proprietary infrastructure/data?\n\n4. How do you address feedback-induced biases (e.g., popularity bias → filter bubbles) beyond §6’s theoretical discussion?\n\n5. DMA’s performance tied to massive private LLMs (QWen-72B)? What are the tradeoffs with smaller/OSS models (e.g., Llama-3-70B)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This work aims to enhance the adaptability of Retrieval-Augmented Generation (RAG) in dynamic environments. The author introduces the DMA framework, which adaptively incorporates multi-level human feedback and integrates real-time feedback signals at the document, list, and response levels. Online experiments demonstrate some effectiveness.

STRENGTHS AND WEAKNESSES:
**Strengths:**

The work focuses on improving RAG’s adaptability to dynamic environments and conducts online experiments to validate its effectiveness.


**Weaknesses:**

1. Unclear Methodology:

- How are various forms of human feedback signals identified and categorized? Further details are needed, especially regarding their interaction with the prominent LLMs mentioned in Section 4.2. Additionally, how is the reliability of the identification process ensured?

- Notations and formulas should be clarified, including model definitions and prediction functions.

- Simple description of a lightweight ensemble model should be added.

2. The improvement observed in the online experiments is modest and does not sufficiently demonstrate the effectiveness of the proposed approach.
3. The DMA system is tailored for one specific online setting, but its performance on public datasets is subpar, indicating limited robustness.
4. There is no ablation study on public benchmarks, nor are dataset statistics provided.

QUESTIONS:
Please see Strengths And Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
This paper introduced Dynamic Memory Alignment (DMA), a novel online learning framework for enhancing retrieval-augmented generation (RAG) systems using multi-level human feedback (document-, list-, and response-level). The authors proposed an unified architecture that integrates these feedback signals into an adaptive memory reranking pipeline. DMA is evaluated through both a multi-month industrial deployment and on standard QA benchmarks, where it shows strong gains in user satisfaction and performance.

STRENGTHS AND WEAKNESSES:
**Strengths**  
1. **Novel Framework:**  DMA utilises a novel framework decomposed into taxonomy, reward modeling, and online adaptation. 
2 **Empirical Validation:**  Proposedframework was tested on a number of benchmark datasets.

**Weakness**
1. Some of the experimental details are missing (e.g. The frequency of collecting feedback for pointwise and listwise scenario). 
2. No code is provided for reproducibility
3. There has been a number of work on adaptive feedback in RAG systems. These methods should be used as a baseline.

[1] "Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity"-Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, Jong Park, NAACL-2024
[2] "Adaptive Retrieval-Augmented Generation for Conversational Systems"-Xi Wang, Procheta Sen, Ruizhe Li, Emine Yilmaz,NAACL-2025

QUESTIONS:
Please check the weakness section also.

1. How sensitive is DMA to the noise in implicit feedback?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Dynamic Memory Alignment (DMA), a novel online learning framework that enhances retrieval-augmented generation (RAG) systems by adaptively integrating multi-level human feedback (document-, list-, and response-level) to optimize memory management in dynamic environments. Unlike traditional static RAG methods, DMA continuously refines retrieval strategies through real-time feedback, improving both relevance and adaptability in interactive settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Innovative Online Adaptation: Proposes DMA, a framework enabling real-time retrieval adaptation in RAG systems through multi-level human feedback (document/list/response), overcoming static retrieval limitations.\n2. Practical Feedback Integration: Develops systematic techniques to transform sparse user signals into actionable retrieval optimization, maintaining robustness while enabling personalization.\n3. Strong Empirical Validation: The authors demonstrate DMA’s effectiveness through extensive evaluations on standard knowledge-intensive benchmarks.\n\nWeaknesses:\n1. Ambiguity in Terminology and Scope: The use of the term “memory” in both the title and the body of the paper is ambiguous and underdefined.\n2. Insufficient Experimental Comparison: The evaluation section does not provide sufficient empirical comparisons against a broader range of state-of-the-art reranking baselines.\n3. Limited Theoretical Contribution: the paper lacks substantial theoretical innovation. The approach primarily combines several existing techniques, such as multi-granular feedback modeling and reward-based reranking, without offering new algorithmic insights or principled theoretical foundations.\n\nQUESTIONS:\n1.  The central term “memory” appears throughout the paper, including in the title, yet its concrete meaning remains vague. Could the authors clarify whether memory refers to the retriever's corpus or cached retrieval traces?\n2. The paper claims to enable real-time adaptation via online learning, but it remains unclear whether the models (e.g., reward model or GBDT) are updated incrementally during deployment or periodically retrained offline.\n3. The experimental comparison currently lacks several recent or strong baselines in reranking and retrieval adaptation, such as reinforcement learning-based or contrastive rerankers.\n\nLIMITATIONS:\nCode, data, and the industrial feedback logs are not released “due to privacy,” so the community cannot replicate the central online-learning claims .", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-12"}