reviews: 
Comments: The paper proposes the DDR framework for tuning agentic systems. The key idea is to sample multiple actions from an agent, and score them based on the downstream reward they fetch (computed via unrolling all subsequent agents in the chain). The actions are ordered by this score, and the DPO method is used to increase the likelihood of high scoring actions over low scoring actions.

The framework is applied to optimize a two agent RAG system consisting of a knowledge refinement agent (selecting relevant documents for RAG), and a generation agent that generates the response based on the selected document.

Through experiments and case studies, the authors show that the framework can perform better than several baselines, and can be better balance internal parametric knowledge with external information.
Strengths: The paper addresses an important concern in RAG system of balancing use of external information versus internal parametric knowledge 
of the generation module. This is common problem for production RAG systems where retrieval is never perfect and often suffers from conflicting and/or missing information.

The idea of jointly tuning retrieval and generation modules is interesting and not studied as much in the literature (although some prior art does exist; see next section).

The results in Sections 5.3 and 5.4 are compelling and show a nice advancement over baseline methods in dealing with noisy retrieval in RAG systems.
Weaknesses: The main weakness of the paper is inadequate comparison with related approaches on jointly tuning retriever and generator modules, which make it hard to appreciate the technical contribution of the paper. 

There is prior work on supervising the retrieval module of RAG system based on performance of the generator (e.g., Wang et al. 2023, https://arxiv.org/pdf/2311.08377). In my eyes, the novelty of this paper is the use of DPO instead of a supervised tuning (which would just maximize likelihood of action fetching highest reward). Thus, if I understand correctly, the novelty of this work is the negative gradient induced by the contrastive DPO loss. However, this is not discussed in the paper. It would help to study the impact of the negative gradient in depth, via ablation studies. For instance, is the negative gradient responsible for the claim in Section 5.3: "DDR can help learn more factual knowledge during training while also preventing the loss of previously memorized information".

The paper highlights RAG-DDR's ability to effectively combine parametric knowledge with external knowledge. To assess this, it would be helpful to see a comparison with adaptive approaches that use different retrieval modules (including no retrieval) for different queries. For instance, see: Jeong et al., 2024: https://arxiv.org/pdf/2403.14403

Other helpful ablations and baselines to experiment with:
* Section 3.2 notes that RAG-DDR tunes the generation module first followed by the knowledge retriever (KR) module. It would be helpful to perform ablations where the KR module is tuned before the generation module (as in Wang et al,. 2023).

* When evaluating vanilla LLM and vanilla RAG, it would help to experiment with large LLMs like Gemini-1.5-pro, GPT4o to understand if the inability to balance parametric knowledge vs. external knowledge is an artifact of smaller LLMs.

The paper introduces the framework of differentiable data rewards (DDR). However, the framework is only evaluated for simple two agent systems. Consequently, DDR does not come across as a fully developed contribution. Instead, the key contribution, at least to me, is the use of DPO in jointly trying retrieval and generation modules. I would encourage the authors to build on this aspect.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3