Title: Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Reward Shaping

Knowledge Distillation and Distribution

Knowledge Transfer
Abstract: Reward shaping is effective in addressing the sparse-reward challenge in reinforcement learning (RL) by providing immediate feedback through auxiliary, informative rewards. Based on the reward shaping strategy, we propose a novel multi-task reinforcement learning framework that integrates a centralized reward agent (CRA) and multiple distributed policy agents. The CRA functions as a knowledge pool, aimed at distilling knowledge from various tasks and distributing it to individual policy agents to improve learning efficiency. Specifically, the shaped rewards serve as a straightforward metric for encoding knowledge. This framework not only enhances knowledge sharing across established tasks but also adapts to new tasks by transferring meaningful reward signals. We validate the proposed method on both discrete and continuous domains, including the representative Meta-World benchmark, demonstrating its robustness in multi-task sparse-reward settings and its effective transferability to unseen tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel multi-task reinforcement learning (MTRL) framework called CenRA, which incorporates a Centralized Reward Agent (CRA) to distill and distribute task-specific knowledge across different policy agents through dense shaped rewards. The CRA learns from the experience of all policy agents and returns \"knowledge rewards\" that augment sparse environmental feedback. An information synchronization mechanism balances the CRA’s focus by considering task similarity and learning progress, ensuring fair and effective knowledge distribution. Experiments on discrete and continuous tasks, including Meta-World, 2DMaze, 3DPickup, and MujocoCar, demonstrate significant improvements in learning efficiency, task balance, and knowledge transfer compared to strong baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Clear Motivation and Problem Relevance: Tackles the widely acknowledged issue of sparse rewards in MTRL, providing a practical and novel reward-centric solution.\n\n2. Innovative Framework Design: The separation between reward generation (CRA) and control (policy agents) is conceptually clean and modular, allowing easy integration into existing RL pipelines.\n\n3. Knowledge Distillation via Dense Rewards: Using learned rewards as a medium for inter-task knowledge transfer is both elegant and effective.\n\n4. Information Synchronization Mechanism: The combination of task similarity and learning performance for adaptive sampling ensures balanced system-wide learning.\n\n5. Strong Empirical Results: Demonstrates consistent improvements across multiple environments and outperforms competitive baselines.\n\n6. Knowledge Transfer to Unseen Tasks: Shows clear evidence that CRA retains and transfers useful knowledge, accelerating learning in new tasks.\n\n7. Reproducibility: Detailed algorithm descriptions, code availability, hyperparameter listings, and experimental setups are thoroughly presented.\n\nWeaknesses\n\n1. Assumption of Homogeneous Input/Output Shapes: CRA requires the same state and action shape across tasks, limiting its applicability to more heterogeneous domains.\n\n2. Fixed Trade-off Parameter ($\\alpha$) for Sampling Weights: The performance of the synchronization mechanism relies on a manually tuned α, which may not generalize well across tasks or domains.\n\n3. Computational Overhead: Maintaining and updating a centralized CRA while running multiple policy agents and computing cross-attention-based similarity adds nontrivial complexity.\n\n4. Limited Theoretical Analysis: Although the empirical results are strong, there is no theoretical discussion on the optimality or convergence of the CRA-generated rewards.\n\n5. CRA as a Bottleneck: In large-scale settings, the CRA may become a performance bottleneck due to its central role in reward generation and shared memory access.\n\n6. Underexplored Comparisons: More ablations comparing CenRA to existing multi-task policy distillation or attention-based modular methods would strengthen the position of the proposed architecture.\n\nQUESTIONS:\n1. How does the CRA generalize when tasks differ slightly in state/action space dimensions or in reward scale? Could preprocessing help extend to those cases?\n\n2. Is there any risk that the CRA introduces reward biases that harm long-term task-specific optimality?\n\n3. How does CenRA perform in continual learning or lifelong RL settings, where tasks arrive sequentially?\n\n4. Could the framework benefit from integrating meta-learning techniques to adapt the CRA or policy agents more efficiently?\n\n5. Have you explored ways to automatically adjust the α parameter in the synchronization mechanism instead of fixing it manually?\n\n6. What is the computational cost and training time increase introduced by CRA compared to decentralized baselines like ReLara?\n\nSince this article is outside the scope of my research, I will refer to the opinions of other reviewers in the future.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper follows the idea of knowledge sharing by reward shaping and propose a framework to apply it to multitask RL problems. The proposed framework utilizes $N$ policies to solve $N$ tasks, and a centralized policy that shapes the rewards of these policies. To train the centralized policy, the authors use a weighted sampling approach, which is later justified in ablation experiments. The framework is evaluated in several benchmarks, demonstrating the performance of the proposed framework comparing to various baselines. The generalization ability of the centralized reward agent to unseen tasks is also demonstrated.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The framework is simple and effective, it allows using any backbone RL algorithm, where each task could be in different domain.\n2. Extensive experiments across various domains.\n3. The paper is well-written, Figure 1 and Algorithm 1 helps the reader to understand the method.\n4. Justifies the proposed sampling method with an ablation experiment.\n\n## Weaknesses\n1. The ablation experiment validates the usage of the proposed weighted sampling, although it raises a concern regarding the contribution of the reward-generating agent. Consider adding the following baseline: shared critic that depends on the task info as well, trained with/without the proposed weighted sampling approach. \n2. More MTRL/ contextual RL methods are missing, most of the baselines are not intended for this setting. I am not sure which methods are considered SOTA in this setting, but there are more recent works that fit for this setting.\n3. As I understand, the reported results were generated from a single training of each baseline. It is important to evaluate over a few instances of each baseline, since the training process may vary for different random seeds. This is a major concern since the paper relies on the empirical evidence in these experiments.\n\nQUESTIONS:\n1. Did you experiment in multi domain setting (i.e., tasks can be in different domains)?\n2. Is the reward balanced (same scale) across tasks among all the presented simulations? Do you have a suggestion how to adapt for such cases?\n3. for the DQN/SAC baselines, did you use a single instance for all tasks or one instance per task?\n\nLIMITATIONS:\nLimitations of this work are mostly discussed, I think that a few sentences regarding working with tasks that have different reward scale should be included.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the use of reward shaping for multi-task reinforcement learning via a centralised reward agent. In overly simple words, CenRA is an extension of the reward shaping methodology to multi-task reinforcement learning. However, behind this \"simple\" sentence, there is a novel idea that is worth exploring.\n\nIndeed, while reward shaping via an assistant reward agent is not a new concept, the co-articulation of this paradigm with multi-task reinforcement learning has not been explored before to the best of my knowledge, and could be done in very different ways. The authors propose to have a shared reply buffer (to reduce memory usage and simplify the code) as well as adaptive strategies to weight the various tasks depending on their similarity and the learning performance of the associated task-specific agents.\n\nExperiments are conducted and reported in several environment sets, exhibiting consistent results and validating the intuition behind the paper.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea has not been explored before.\n- The model design is simple and takes into account memory/code factors.\n- The paper is well written and easy to understand.\n\nWeaknesses:\n- The motivation for which only similarity and learning weights are chosen is unclear, as it is the way to combine those weights.\n- The intuition behind how to compute the weights is clear, but this computation is potentially prone to numerical instabilities.\nThese weaknesses are an issue because the weighting strategy is the main methodological contribution of the paper.\n\nQUESTIONS:\nThe core idea of the paper is to adopt the reward assistant agent paradigm for multi-task reinforcement learning. In this regard, the main methodological contribution should be in this \"adoption\". The authors have chosen fairly simple weight models and weight combination strategies (which I see as a positive thing).\n\nHowever, there is not much discussion on other modelling possibilities. Weights and their combinations can be modelled in very different ways, and this can have a severe impact on the experiments. I wonder if the authors carried out an in-depth exploration.\n\nIn addition, regarding the task similarity weights, the task centroid \"c\" could be zero (or arbitrarily close to zero), thus opening the door to strong numerical stabilities. I would like the authors to comment on that point.\n\nLikewise, for the task performance weights, the numerical range of the rewards of the various tasks is potentially very different (there are no constraints whatsoever announced by the authors), and therefore using the average past rewards to re-weight the replay buffer can be very problematic since tasks that naturally have a large reward would be visited less often, but for the wrong reason.\n\nOverall, I believe that the main contribution of the paper (these weights and weight combination strategies) deserve more exploration both in terms of internal comparison performance as well as in terms of numerical instabilities.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a Centralized Reward Agent based Multi-Task RL framework (CenRA), where a centralized reward agent learns the policy to predict knowledge rewards that optimize the accumulated environment rewards it will receive, and policy agents for each task update their own policies by any RL algorithm to optimize the augmented reward (addition of the environment reward and the scaled knowledge reward). CenRA shows empirically improved performance compared to prior work on Meta-World, 2DMaze, 3DPickup, and MujocoCar.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The paper is well structured, easy to follow, and clear.\n* The proposed method, including the core knowledge distillation and distribution, as well as the information synchronization mechanism (similarity weight and performance weight), is reasonable to mitigate potential issues in such a framework.\n* The experiments are conducted on Meta-World benchmark and additional 3 environments, showing consistent improvements.\n* Although I have some questions about completeness (as described in Weaknesses), the paper has included important analyses like an ablation study of the similarity weight and performance weight, knowledge transfer to unseen tasks, and the visualization of learned knowledge rewards.\n\n### Weaknesses\nMy questions are mainly about the experiment design and analysis to validate some components’ effectiveness:\n* Since the 2DMaze involves the least diverse tasks (only the position and colors are different, but the goals are the same; the unseen task even shares the same colors as one of the seen task) among the four benchmarks, the claim of the success of knowledge transfer in * Section 5.2 will be strengthened if the paper can show Meta-World’s visualization results. \n* In addition to Table3 for sampling weight ablation study, can the authors also show the Meta-World’s results? If the number of tasks is a concern to list the table, it will also be beneficial to see the results averaged across tasks.\n\nQUESTIONS:\nAs described in Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}