Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning
Authors: Haozhe Ma
Zhengding Luo
Thanh Vinh Vo
Kuankuan Sima
Tze-Yun Leong
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Reward Shaping
Knowledge Distillation and Distribution
Knowledge Transfer
Primary area: reinforcement_learning
TL;DR: To incorporate reward shaping approach into multi-task reinforcement learning, we propose a Centralized Reward Agent based MTRL framework (CenRA) to share and transfer knowledge across multiple tasks.

Abstract: Reward shaping is effective in addressing the sparse-reward challenge in reinforcement learning (RL) by providing immediate feedback through auxiliary, informative rewards. Based on the reward shaping strategy, we propose a novel multi-task reinforcement learning framework that integrates a centralized reward agent (CRA) and multiple distributed policy agents. The CRA functions as a knowledge pool, aimed at distilling knowledge from various tasks and distributing it to individual policy agents to improve learning efficiency. Specifically, the shaped rewards serve as a straightforward metric for encoding knowledge. This framework not only enhances knowledge sharing across established tasks but also adapts to new tasks by transferring meaningful reward signals. We validate the proposed method on both discrete and continuous domains, including the representative Meta-World benchmark, demonstrating its robustness in multi-task sparse-reward settings and its effective transferability to unseen tasks.

Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning
Haozhe Ma 1.2, Zhengding
Vinh Vo', Kuankuan Sima", Tze-Yun Leong' School of Computing. National University of Singapore
TikTook Pte. Ltd., Singapore
School of Electrical and Electronic Engineering, Nanyang Technological University
*Department of Electrical and Computer Engineering, National University of Singapore (haozhe ma, kuankuan sts_sssmmmmss_ @u. nus .us.edu, Senettyyyuuuuuuu.u {votv, leongty) @nus. edu..eeduuss.. sg. luoz00210e ntu. edu. sg
Abstract
Reward shaping is effective in addressing the sparse-reward challenge in reinforce- ment learning (RL) by providing immediate feedback through auxiliary, informative rewards. Based on the reward shaping strategy, we propose a novel multi-task reinforcement learning framework that integrates centralized reward agent (CRA) and multiple distributed policy agents. The CRA functions as a knowledge pool, aimed at distilling knowledge from various tasks and distributing policy agents to improve learning efficiency. Specifically, the shaped rewards serve as straightforward metric for encoding knowledge. This framework not only enhances knowledge sharing across established tasks but also adapts to new tasks by transferring meaningful reward signals. We validate the proposed method on both discrete and continuous domains, including the representative benchmark, demonstrating its robustness in multi-task sparse-reward settings and its effective transferability to unseen tasks.
individual
Introduction
Reinforcement learning (RL) has made significant progress across various domains, such as robotics [Kober et al. 2013). gaming (Lample and Chaplot 2017]. autonomous vehicles [Aradi] 2020 signal processing Luo et al. 2024), and large language models [Shinn et al. 2023 Ouyang al. 2022 However, environments with sparse and delayed rewards remain significant challenge, as the absence of immediate feedhack hinders the agent from distinguishing the value of states and leads aimless exploration [Ladosz al.) 2022]. Reward Shaping (RS) has been proven to be an effective technique for addressing this challenge by providing additional dense and informative rewards (Sorg et al., 2010b a). Concurrently, multi-task reinforcement learning (MTRL) is becoming increasingly important due to its ability to transfer knowledge across tasks. In this context, the auxil- iary rewards infused with task-specific information in RS offer a straightforward means distribute knowledge among different tasks. Integrating RS techniques into MTRL is a highly promising and intuitive direction enhance the efficacy of multi-task learning systems.
Numerous MTRL algorithms for knowledge transfer have been developed. Policy distillation methods identify and combine commonalities across different policies [Rusu et al., 2016 Teh al. 2017 Parisotto et al. 2016 Xu et al., 2024); representation sharing methods extract and share the common
features or gradients among agents Yang al. 2020 D Eramo al.) 2020 Sodhani al.) 202
and parameter sharing methods design architectural modules to reuse parameters or layers across
networks (Sun al. 2022 Cheng et al. 2023 Despite their potential, these strategies often face slow
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

adaptation to and limited utilization of transferred knowledge. Therefore, leveraging reward shaping, which directly adds a metric to the reward function, offers a compelling alternative to address these limitations.
Regarding reward shaping, not all shaped rewards effectively serve medium for knowledge transfer. Specifically, the Mltrinsi--motiiatioo--asss
rewards are typically designed using heuristics to generate task-agnostic signals. Examples include incorporating exploration bonuses [Bellemare et al.
2016
Ostrovski et al.) 2017 Devidze et al. 2022]. rewarding novel states (Tang et al. |01 Burda
et al. 2018]. and encouraging curiosity- driven behaviors (Pathak et al., 2017] Mavor- et al. Although these approaches encourage broader exploration, they are not directly related to specific tasks and thus lack transferability. Consequently, we focus on another branch of RS methods, contexxtl rewards, which automatically learn and encode task-specific information, such as hidden values, states contributions, or future-oriented insights, that can be effectively shared across
2022).
various tasks Ma et al. 2024a 2025b a Mguni et al. 2023, Memarian et al. 2021]
To share task-related knowledge in MTRL via RS techniques, and inspired by the ReLara frame- work [Ma et al. [024a], which integrates an assistant reward agent to densify sparse environmental rewards, we propose the Centralized Reward Agent based MTRL IRAmework (CenRA P The framework consists of two main components: a centralized reward agent (CRA) and multiple dis- tributed policy agents. Each policy agent individually learns control behaviors within its respective tasks and shares its experiences with the CRA. The CRA extracts common knowledge from these experiences and learns generate dense rewards that are encoded with -ssciipc information. These rewards are then distributed back to the policy agents to augment their original environmental rewards. Additionally, given that different tasks may contribute variably to the MTRL system, we introduce an information synchronization mechanism to further balance knowledge distribution by considering task similarity and agent learning progress, thereby ensuring system-wide optimal performance. The main contributions of this paper are summarized as follows:
(i) We propose the CenRA framework to address MTRL problems. It incorporates a CRA that functions as a knowledge pool, efficiently distilling and distributing valuable information from various tasks to policy agents while adapting to new tasks.
(i) CenRA leverages reward shaping techniques to infuse insights via dense rewards. This approach not only provides a direct signal for policy agents to absorb knowledge but also effectively addresses the sparse-reward challenge.
(ii) We introduce an information synchronization mechanism that considers both task similarity and agent learning progress to balance mulli-taskk learning. This mechanism provides a novel direction for maintaining system equilibrium in MTRL.
(iv) CenRA is validated in both discrete and continuous control MTRL environments with sparse extrinsic rewards. CenRA outperforms baseline models in learning efficiency, knowledge transferability, and wide performance.
2 Related Work
Multi-task reinforcement learning (MTRL) has attracted significant attention recently due to its potential to share knowledge across multiple tasks, thereby improving learning performance (Caruana 1993]. We discuss existing MTRL literature from three main directions:
Knowledge Transfer methods focus on identifying and transferring task-relevant features across diverse tasks [Zeng et al. [021]. Policy distillation [Rusu al., 2016) a well-studied approach to extract and share task-specific behaviors or representations that many works are built on: Teh et al. 2017 introduced Distral, which distills a centroid policy from multiple task losk-policies: Paariotto et al. 2016 developed Actor-Mimic, where a single policy is trained to mimic several expert policies from different tasks; while Yin and Pan [0017] incorporated hierarchical prioritized experience replay buffer to select and learn multi-task experiences; Heageees:eHHeellle Hessel et al. [2019) further proposed an adaptation mechanism to equalize the impact of each task in policy distillation Additionally, Xu et al. (2020) explored the transfer of offline knowledge to train policies, and further leveraged online learning for fine-tuning. Bai et al. [2023) introduced a dual-phase learning approach, optimizing individual policies while correcting them across multiple tasks. Mysore et al. [2022) used separate critics for each task to accompany a single actor to integrate their feedback. These methods mitigate gradient
"The source code is accessible at: https://github http://hhhhh.ccc////a/c///////la

---

interference to an extent, however, balancing the distribution of knowledge across tasks is crucial. Without a careful trade-off, the performance of the entire system could be compromised.
Representation Sharing methods explore architectural solutions of reusing network modules or
representing commonalities the MTRL problem (D'Eramo et al.) 2020 Devin al. 2017 HHon
et al. 2022 Ma et al. 2024b 2023]. Sun et al. [2022] used a parameter compositional approach to learn and share subspace of parameters, allowing policies for various tasks be interpolated within it. Yang et al. 2020] employed soft modularization to learn foundational policies and utilized
a routing network to generate probabilities to combine them. He et al. [2024) introduced the Dynamic Depth Routing framework, which dynamically adjusts the use of network modules in response to task difficulty. (Sodhani et al. [202]] leveraged task-related metadata to create composable representations. Cheng et al.] [202]] and Lan et al.] [2023] both incorporated attention mechanisms: the former employed attention-basee mixture of experts to capture task relationships, while the latter used Temporal Attention for contrastive learning purposes. Although these methods demonstrate efficacy in learning shared representations, they may struggle to fully capture the complexity of highly diverse tasks. Moreover, adapting shared structures to new tasks typically requires extra design efforts.
Single-Policy Generalization methods learn single policy to solve multiple tasks simultaneously or continuously, in the absence of information from prior policies or task-specific details, in which case, the primary goal is to enhance the policy's 's generalization capabilities. Model- -ree meta- meta-learning techniques have been proposed to enhance the multi-task generalization (Finn et al. [017]] Yang et al.)
[2017] designed sharing network structure that allows an agent to learn multiple tasks concurrently,
Vuong et al. (2019) introduced a confidence-sharing agent to detect and define shared regions between tasks to support single policy learning. Waan et al. 22200] proposed transfer learning framework to handle mismatches in state and action spaces. Additionally, several methods focus on overcoming
gradient interference to enhance the generalization in various tasks [Chen et al., 2018 Yu et al.) 2020a], while Ammar et al. [2014) developed consecutive learning policy gradient approach. These methods are efficient in saving computational resources, but the generalization ability of the policy may be constrained when faced with out-of-diitributtin or previously unseen tasks.
3 Preliminaries
Markoy Decision Process (MDP) models sequential decision-making problems under uncertainty. An MDP represents the interaction between an agent and its environment as a tuple (S, P, R, 7). where S is the state space. A is the action space. P S x A $ + [0, 1] is the probability of transiting from one state to another given an action, : S x A + E is the reward function, and E [0, 1] is the discount factor to modulate the importance of future versus immediate reward.
Multi-Task Reinforcement Learning (MTRL) addresses the challenge of learning multiple tasks simultaneously within an integrated model to leverage commonalities and differences across tasks. Typically, MTRL introduces a task space T, assuming all tasks sampled from this space and thus follow a unique distribution. Each task is modeled as an independent MDP, An MTRL agent aims to learn optimal policies म; S - A for each task T: T, to maximize their corresponding expected cumulative rewards, or returns, denoted by G. = E Σ (= 7 ''Ra( R2(*,, a.)].
RL with an Assistant Reward Agent (ReLara) Ma et al. 2024al introduces dual-agent framework designed to tackle the challenge of sparse rewards in RL. Within this framework, the original agent is termed as policy agent, while an assistant reward agent is integrated to enrich the feedback mechanism by generating dense, informative rewards. The reward agent, trained as a self-contained RL agent, autonomously extracts hidden value information from the environmental states and the actions of the policy agent to craft meaningful reward signals. These signals significantly improve learning efficiency by providing immediate and pertinent bonuses.
4 Methodology
We propose the Centralized Reward Agent fRAmework (CenRA) for MTRL, which incorporates a centralized reward agent (CRA) to support multiple reinforcement learning agents across multiple tasks. A high-level illustration of the CenRA framework is shown in Figure [ The CRA is responsible for extracting general task-specific knowledge from various tasks and distributing valuable information to the policy agents by reconstructing their reward models. The detailed methodology
3

---

knowlodge reward y
Contralized roward agent ArW
Knowlalge distilaaioon
sampling
comrol
Knowledge distributiio
Concationatod replay bufler D
replay bufler
ppplay butfer
replay butfer
Policy agent A
Policy agent
Policy A
pedformncc
perfortunce
peyformmnc
Task
Learning
Learning
Learning
www. www.eea
stm
Pew
Similarity weight 501
Performance weight weer
Figure I: A high-level illustration of the CenRA framework. The centralized reward agent functions knowledge repository. distilling information from various tasks and distributing it to individual policy agents to enhance learning efficiency.
Furthermore, to mitigate the potential disparities in the information that each task contributes, which might lead an imbalance in knowledge distribution, we introduce an information synchronization mechanism by considering two main factors: the similarity of the tasks and the online learning performance of the policy agents, details given in Section 4.2 Finally, the overall framework of CenRA is presented in Section
for knowledge extraction and sharing is presented in Section 4.1
4.1 Knowledge Distillation and Distribution
4.1.1 Problem Formulation
We consider an MTRL setting comprising N distinct tasks {Ti, .T,....,Tyy all executed within the same type of environment E. We assume that the shape of state E S and action E A remain uniform across tasks, to ensure the CRA processes consistent inputs. Despite this uniformity, each task may feature different state spaces, action spaces, goals, and transition dynamics. For instance, series of mazes with the same size but varying map configurations would satisfy this condition. For each task T; we denote the transition function as (((''(s, (86[1,,0 a) and the reward function [, a).
The centralized reward agent (CRA) is denoted as Ared and multiple policy agents are denoted as {.APol .. ...., ..... }. Each policy agent _4 operates independently to complete its corre- sponding task T7, utilizing appropriate RL algorithms backbones. For example, implement- ing DQN Mnih et al. 2015 for discrete control tasks, while TD3 [Fujimoto et al., 2018 or SAC Haarnoja al. 2018a for continuous control tasks. Moreover, the policy of CRA Arwd is rwd and the internal policy of policy agent Apol x!!!
4.1.2 Centralized Reward Agent
The CRA And aims to extract environment--elevaan knowledge and distribute it to policy agents by generating additional dense rewards support their original reward functions. Similar the ReLara framework [Ma et al. 2024a]. we model the CRA as self-contained RI agent, yet, as an extension to ReLara, CRA designed concurrently interact with multiple policy agents and their respective tasks. The CRA's policy red generates continuous rewards given both an environmental state and a policy agent's behavior. Specifically, x rww maps the Cartesian product of the state space and action space, S x A, to defined reward space, which constrains the rewards to a range of real numbers, C For simplicity, we denote the observation of the CRA as rwwd (si, a,). where 8; ~ T, and a, ~ -oo (s.). To distinguish from the environmental reward, the generated reward is termed as knowledge reward, denoted as krow
R RR[Rmmn Rmaaz]
We adopt an off off- policy actor-critic algorithm to optimize the CRA (Konda and Tsitsiklis 1999 To aggregate and reuse experiences from all policy agents, concatenated replay buffer D U D, is

---

constructed, where D; represents the replay buffer of each policy agent A Besides, each transition is augmented with the CRA-generated knowledge reward, , Specifically, the transition from policy agent AA stored in the replay buffer is defined as (stwd, , rr (T)))T)). The augmented transition includes all necessary information for optimizing both the CRA and each corresponding policy agent, thus making the concatenated replay buffer shared resource across the entire framework and minimizing storage overhead.
The CRA's update process involves using these stored transitions to optimize the reward -ererrtnnn actor rwd and the value estimation critic. The objective function for the critic module is:
J(Vrud)=
E TwND"
16:1
d,
= =r m --gV VV (st V (s) (
where Tr (stwd, rg"u, (T following objective function:
D. Concurrently, the actor module is updated through the
J (
E
E log x. awd (r (squid) .de]]
TeND knw wondewarddee (-1o
4.1.3 Policy Agents with Knowledge Rewards
Each policy agent Ap stores the experiences in its corresponding replay buffer D;, They receive two types of rewards: the environmental reward r!" from their respective task T; and the knowledge reward wkn from CRA. The augmented reward is given by:
pol env + An kone
x rrdd (8;, (i)
where À E (0, 1] is a scaling weight factor. The optimal policy n. for each agent is derived by maximizing the cumulative augmented reward:
arg max
E (#1,0L) pe pod
pol
worth noting that the environmental reward r5"V retrieved from the replay buffer (if adopting an off- off-policy approach). Conversely, the knowledge reward An is computed in real-time using the most recently updated Arwd, ensuring it reflects the latest learning advancements. Lastly, each policy agent is able to employ any suitable RL algorithm, whether on-policy or off off-policy, to best address its specific task, which enhances the CenRA framework's s generality and flexibility.
4.2 Information Synchronization of Policy Agents
In the CenRA, the information provided by different tasks may exhibit significant disparities, poten- tially leading to an imbalance in knowledge extraction and distribution. We introduce an information synchronization mechanism for CenRA to maintain a balanced manner from the perspective of the entire system. Specifically, we control the quantity of samples that CRA retrieves from each task't s replay buffer D; by sampling weight w, by considering two aspects: the similarity among tasks and the real-time learning performance of the policy agents.
Similarity Weight wii is derived from the similarity among tasks, enabling the CRA focus on relatively outlier tasks. To simplify computation, we use the hidden layers extracted from each policy agent's neural network encoders to represent the tasks' features. To reduce randomness, we average the hidden features of the most recent K steps. We adopt a cross-attention mechanism to calculate the similarity weight [Vaswani et al., 2017]. Specifically, for task T,, let H; denote the averaged hidden feature vector, which serves as the key, and the centroid of all tasks e acts as the query. Then, the similarity s; of task T; to the centroid of the task cluster is calculated as:
C C..H. H. 8, VD
c
1 N
Σ
H K
where D is the dimension of the hidden feature to prevent gradient vanishing exploding. A larger 8; indicates a greater similarity between T; and the centroid. It worth noting that, to avoid the centroid approaching zero due to feature vectors H, pointing opposite directions, all latent representations
5

---

H; in our framework are extracted from ReLU activation layers. This ensures that every element of H, non-negative, effectively preventing feature cancellation and maintaining a well-defined and numerically stable centroid c. Given our assumption is that the tasks farther from the centroid require more attention, the similarity weight is defined as waem Softmax (1//1, 1/8.....,11
Performance Weight is determined by the real-time learning performance of each policy agent, ensure the CRA focuses more on lagging tasks. Similar to the similarity weight, av- erage the environmental rewards ri from the most recent K steps, denoted as Retil to measure the recent learning trends. The performance weight is then defined as wper Softmax ([1/ 1 1/R2 ................... 1/
The final sampling weight w is formulated as w == own + (1 a) where a is a hyperpa- rameter balance the two aspects. The CRA samples from each replay buffer D; according to w. ensuring a balanced and effective knowledge extraction and learning.
4.3
Overall Framework
The overall framework of CenRA is summarized in Algorithm 1 The CRA and policy agents are updated alternately and asynchronously, with the frequency of updating the CRA adjustable according to the actual situation. Sampling weights are calculated in real-time, using the most recently optimized encoders and the current learning performance, ensuring CRA continuously adjusts its focus optimally balance knowledge extraction across multiple tasks.
The learned CRA acts as a robust knowledge pool, which is able to support new tasks by transferring knowledge through auxiliary reward signals. This is particularly beneficial in sparse-reward envi- ronments, as the knowledge rewards can guide the policy agents toward the correct direction and reduce exploration burden. Additionally, the CRA be further optimized alongside new tasks in continuous learning scheme that enhances adaptability and effectiveness in dynamic settings.
Algorithm Centralized Reward Agent based MTRL
Require: Multiple tasks {Ti, Ti..T22... ....... TN}.
Require: Policy agents {Appol Abo ..... Apot) }.
Require: Centralized reward agent Aruid
Require: Concatenated replay buffer D U Dz.
1: for each iteration do
2: for each task T; do
3:
(St, (,t,at, St,,ur,ffme Se-1, 00+11 Interact (Apol, Ti) It Arwd ae) of stud = (st, stndd (st+1, ac+1)
Interact and collect one transition Sample an off-policy knowledge reward
4:
5:
6:
D, DD+D, ((s)
(wd .rt rt onno
knw
8811 (T:)(T:))
Store the transition in corresponding D;
7:
Update policy agent Agdl end for
Update Apdl using backbone RL algorithm
8:
9:
w sim + (1 ae
Calculate sampling weight Draw samples based on the sampling weight
10: [spuddyAem .rgee s
grud [Ti] ~ D Dlw 11: Update centralized reward agent A rod
12:
end for
5 Experiments
We conduct experiments four MTRL domains as shown in Figure the widely used Meta-World benchmark (including ML.0 with 10 tasks and ML.00 with 50 tasks) Yuu et al. 2020b). 2DMaze, 3DPickup (Chevalier-Boisver et al. 2024), and MujocoCar [Ji al. 2023]. All tasks, including those in Meta-World, are crafted to provide sparse environmental rewards, where the agent receives a reward of only upon successful completion of the final objective, and 0 otherwise. The detailed task configurations are provided in Appendix A

---

target areu
dfferet akjpas
(a) Meta- World (ML10-spar and ML50-sparse)
(b) 2DMaze
(c) 3DPickup
(d) MajocoCar
Figure 2: Environments with multiple tasks. (a) Meta-World: two sparse-reward versions are used: ML10-sparse and ML50-sparse, including diverse robotic manipulation tasks. (b) 2DMaze: 2D maze tasks where the agent must pick up a key and then pass through a door to exit. (c) 3DPickup: 3D maze tasks where the agent aims to navigate and pick up different target objects at different locations. (d) CajocoCar: mujoco- basood race car aims to navigate to different specified areas.
20MM
373Pock
Madorsaar
Mare #1
Mocee
Ball
Targe
Toget R2
Mara
Mosset
Tayun
Tiapet
Stape thaamas)
Supeçn Doumanc)
Stepe Disaurds)
Steps ca Mocmmndd
Skepe Extannni)
Stezk thenanada)
CaiRA
Ralan
CM4TA
MCAT
PaCa
Figure 3: Comparison of CenRA with baselines in 2DMaze, 3DPickup, and MujocoCar domains.
5.1 Comparative Evaluation MTRL
We benchmark CenRA against several state-of-the-art baselines; (a) the backbone RL algorithms of the policy agents: DQN (Mnih et al.) 2015] for discrete control tasks and SAC (Haarnoja et al.) [018b] for continuous control tasks: (b) the ReLara algorithm [Ma al., 2024a which can be regarded as decentralized variant of CenRA, where each policy agent is paired with a separate reward agent,
without cross-task -ask information sharing; (c) the TD-MPC2 algorithm (Hansen al.
2024]; (d) the
Contrastive Modules with Temporal Attention (CMTA) algorithm [Lan et al. 2023 (e) the Policy Optimization and Policy Correction (PiCor) algorithm [Bai et al. [023]: (f) the Multi-Critt Actor Learning (MCAL) algorithm [Mysore al.] 2022]; (g) the Parameter-compositionn MTRL (PaCo) algorithm [Sun et al. 2022]; ; (h) the Shared-Critic (SC) algorithm [Zhang al., 202T]: and (i) the MTRL with Soft Modularization (SoftModule) (Yang al. 2020). They implemented by either the CleanRL library [Huang et al. 2022] or official codebases. Each task is trained with 10 different random seeds, and the average results reported.
In the Meta- Meta-World domain, ML.IO-ppars provides 10 training tasks and held-out test tasks, while ML50-sparss includes 45 training tasks and 5 test tasks. For the remaining domains, each consists of 4 training tasks and test task. this section, we evaluate the final returns achieved by the trained agents, averaged over all training tasks in each domain, shown in Table We additionally report episodic returns and their standard errors throughout training the 2DMuze, 3DPickup, and MujocoCar domains in Figure 3 To ensure fair comparison, we adopt consistent hyperparameters (where applicable) and identical network architectures across all experiments; detailed configurations provided in Appendix B
We observe that CenRA consistently outperforms all baselines in three main aspects. First, achieves the highest episodic returns in all tasks, demonstrating superior learning efficiency and faster convergence. Moreover, it demonstrates good stability and robustness, exhibiting fewer fluctuations and oscillations, especially after convergence, compared to other models. Notably. all tasks provide only sparse rewards, CenRA addresses this challenge through the auxiliary dense rewards with meaningful information, effectively guiding learning. This mechanism not only distinguishes CenRA from other structurally shared methods, also provides targeted solution to the sparse-reward problem. Second, while baselines like PiCor and MCAL often show uneven progress across different
7

---

Table Episodic returns (mean standard error) of all trained agents tested over 100 episodes and averaged across all training tasks in each domain (1 higher is better).
Algorithm
Mpao-sparre
2DMaze
3DPickup
MujocoCar
CenRA (ours)
0.875 0.121
0.755 0.034
0.913 + 0.023
0.880 0.060
514.875 0.675
DQNISAC
0.256 - 0.056
0.189
0.012
0.645 = 0.070
0.243 = 0.048
198,000 = 0.453
ReLara
0.674 + 0.105
0.541 + 0.057
0.803 = 0.065
0.565 = 0.088
429,800 = 0.655
TD-MPC2
0.823 + 0.091
0.608 0.032
0.884 + 0.046
0.712 0.051
505.341 = 0.712
CMTA
0.787 + 0.076
0.603 0.026
0.753 + 0.037
0.695 + 0.043
480.187
0.623
PiCor
0.865 + 0.230
0.12
0.818 + 0.053
0.438 + 0.085
437,550
0.663
MCAL
0.842 + 0.067
0.885 1 0.080
0.548 + 0.068
369.200 = 0.595
PaCo
0,854 + 0.045
0.582
0.834 1 0.057
0.557 + 0.072
421.210 = 0.635
SC
0.556 + 0.063
0.354 1
0.798 1 0.052
0.687 + 0.038
400.254 = 0.518
SoftModule
0.630 0.042
0.423 0.057
0.822 + 0.076
0.486 0.055
355. 125 0.594
new 20M
new 3DPickur
new MuiseuCar
trioodicretnns
Stege (hoasanes)
Staps da Thoasadd
Stegs (bousard))
leanting
Ceuldow louning
(1) Before gesting the key. (A)efter gotting the key.
(a) Comparison of the learning performance of CenRA with the base- lines new tasks in 2DMaze, 3DPickup and MujocoCar domains.
(b) Actions yiclding the highest knowl-
edge rewards a new 2DMaze task.
Figure 4: Experimental results for knowledge transfer to new tasks.
tasks within the same domain, CenRA maintains well-balanced performance by showing relatively consistent learning progress and minimal variability across each four-task groups. This ensures that no single task dominates or lags behind, which is crucial in multi-task learning. Third, the CRA effectively enhances knowledge sharing among tasks. This is evident from the comparison with ReLara, which uses independent reward agents and lacks the mechanism for knowledge exchange. By extracting and distributing insights from one task to another, the CRA improves the learning efficiency of individual tasks, highlighting the advantages of integrated knowledge management.
5.2 Knowledge Transfer to New Tasks
In this section, we assess the CRA's ability to transfer previously learned knowledge to unseen tasks. Specifically, we continue using the trained CRA model in Section[5.]] while initializing new policy agents to tackle new tasks from the same domain. These include test tasks for ML1O-sparse and ML50-sparse, and I test task for each of the remaining domains, none of which were encountered during the initial training. For the CenRA, we explore two scenarios: (1) the CRA continues to be optimized in collaboration with the new policy agent (CenRA w/ learning); and (2) only the policy agent updated while the CRA remains fixed, relying only on its previously acquired knowledge (CenRA w/o learning). We compare the two settings against the backbone algorithms and ReLara. In ReLara, the reward agent is trained anew without pre-learned knowledge. The results are presented in Figure 4a and Table
We observe that CenRA with further learning achieved rapid convergence, mainly due to the CRA's ability to retain previously acquired knowledge while continuing to adapt to new tasks through ongoing optimization. Remarkably, even without any additional training, CenRA still outperforms both ReLara, which requires training a new reward agent, and the backbone algorithms, which lack additional information. This advantage stems from the CRA's ability to encode and transfer cettrrommnt-releewee knowledge, which can then be directly reused by new policy agents to guide their learning. Such knowledge transfer is particularly critical in our experiments involving challenging sparse-reward tasks. Without any external knowledge, learning would require extensive exploration. However, the CRA provides meaningful dense rewards that significantly accelerate the learning process, even during the initial phases.
8

---

Table Episodic returns (mean + standard error) of all trained agents in the new tasks, tested over 100 episodes in each domain (1 higher is better).
Algorithm
MLJ0-sparse
MLS0-sparse
2DMaze
3DPickup
MujocoCar
CenRA w/ learning
0.902 0.021
0.824 = 0.012
0.952 = 0.010
0.963 0.002
532.080 + 1.610
CenRA w/o learning
0.887 0.011
0.809 = 0.009
0.894 = 0.032
0.678 + 0.003
524.727 + 0.588
ReLara
0.702 0.086
0.612 = 0.012
0.759 = 0.056
0.263 + 0.002
224.648 + 0.492
DQN/SAC
0.228 0.105
0.210 + 0.034
0.263 = 0.084
0.158 + 0.003
129.055 + 0.296
Table 3: Comparison of CenRA with ablation of different batch sampling control weights.
2DMaze
Var. +
Algo.
(x10
Maze #1
Maze #2
Maze #3
Maze #4
CenRA (a 0.5)
0.893 + 0.033
0.908 + 0.022
0.924 + 0.020
0.932 + 0.020
0.021
CenRA (a 0.25)
0.889 = 0.031
0,901 0.025
0.915 0.023
0,925 + 0.024
0.065
CenRA (a- 0.75)
0.891 1 0.030
0.905 0.024
0.918 0.021
0.928 1 0.022
0.049
w/o w "im (a 0)
0.884 + 0.033
0.922 1 0.021
0.873 1 0.041
0.820 1 0.039
0.172
w/o wer (a 1)
0.758 1 0.062
0.884 1 0.030
0.824 1 0.052
0.867 1 0.020
0.284
both
0.632 = 0.053
0.833 0.041
0.629 0.08
0.802 + 0.054
1.235
3DPickup
Var. +
Algo.
(x10
Ball
Cube
Key
Health kit
CenRA (a 0.5)
0.951 + 0.020
0.683 0.090
0.795 0.062
0.688 + 0.067
1.570
CenRA (a = 0.25)
0.942 + 0.023
0.671 + 0.091
0.782 + 0.066
0.715 + 0.061
1.650
CenRA (a 0.75)
0.938 1 0.025
0.665 1 0.095
0.775 + 0.069
0.702 1 0,065
1.723
w/o w sim = 0)
0.822 1 0.065
0.702 0.093
0.704 0.072
0.887 1 0.038
0.892
w/o wer (a = 1)
0.779 + 0.072
0.404 1 0.093
0.631 0.080
0.438 1 0.102
3.051
w/o both
0.811 + 0.073
0.457 + 0.058
0.483 + 0.079
0.370 1 0.079
3.796
MujocoCar
Var.
Algo.
(x10*)
Target #1
Target #2
Target #3
Target #4
CenRA (a 0.5)
588.221 + 0.732
549.337
0.640
447.743 + 0.672
474.320 + 0.657
4.244
CenRA (a 0.25)
575.153
0.740
538.912 = 0.655
439.850 + 0.680
462.116 + 0.665
4.871
CenRA (a- 0.75)
580.431
0.735
542.765 = 0.648
441.033 + 0.675
468.529
0.660
4.533
w/o un mim (a 0)
319.926 + 0.590
486.767
0.712
332.506 + 0.695
260.921 + 0.544
9.288
w/o wPr (a 1)
320.887
0.891
355.325
0.677
344.145 = 0.872
308.215 = 0.723
0.457
w/o both
57.532 + 0.352
257.010 = 0.677
677.285
= 0.540
77.635 0.255
82.720
To further demonstrate CenRA's transferability, select the 2DMaze environment to visualize the knowledge provided by CRA when facing an unseen task. As shown in Figure|4b) we plot the directions of actions that yield the maximum knowledge reward each position, categorized into two scenarios: before and after obtaining the key. While some guidance in peripheral regions may appear slightly misaligned, most states receive reasonable rewards that align with human understanding. This demonstrates the effectiveness of knowledge transfer and with such dense rewards, the agent's s adaptation to new tasks is able be well supported. In addition, better understand the knowledge learned by the CRA, we provide a case study in Appendix C where we visualize the CRA-provided sask-specific,
rewards. This further verifies that the CRA is capable of capturing and semantically meaningful signals across different tasks.
5.3 Effect of Sampling Weight
We conduct experiments to understand the effects of the information synchronization mechanism
in the CenRA. Specifically, we compare the full CenRA model against five variants: (a) and (b) CenRA with different values the balance factor a, i.e., a = 0.25 and a 0.75, to examine the impact of different weight combinations; (c) CenRA without the similarity weight wis (i.e., a = 0);
(d) CenRA without the performance weight wper (i.e., a 1); and (e) CenRA without the entire sampling weight. To better illustrate the differences among tasks and highlight the role of sampling

---

weights in task coordination and synchronization, we select the four-task domains, i..., 2DMaze, 3DPickup, and MujocoCar. The results are shown in Table
The results indicate that the two weights, which control the allocation of samples drawn from each policy agent's ' experiences, mainly influence the overall learning performance. Specifically, the absence of sampling weight leads to unbalanced learning outcomes, which is observed by episodic returns across four tasks. In contrast, when both weights are
the increased variance incorporated, the learning process becomes notably more stable, indicating that the joint consideration of task similarity and learning progress is essential for coordinated optimization. While the full CenRA model does not always achieve the lowest variance, it consistently outperforms the other three ablation models regarding overall system performance.
Both weights play essential roles in information synchronization, with the performance weight uper having more significant impact. It allows the CRA to focus more on policy agents that are underperforming or progressing slowly, ensuring balanced system-wide learning. Moreover, different choices of the balancing factor emphasize distinet aspects of synchronization: larger a highlights task similarity and promotes uniformity across related tasks, whereas smaller a prioritizes compensating lagging tasks by amplifying the effect of wper, This flexible weighting further enhances stability and adaptability, demonstrating that considering the overall learning performance of the multi-task system is a central objective that CenRA seeks achieve.
6 Discussion and Conclusion
We propose a novel framework CenRA that integrates reward shaping into multi-task reinforcement learning. The framework shares domain knowledge across tasks to improve learning efficiency and effectively addresses the sparse-reward challenge. Specifically, the centralized reward agent (CRA) functions a knowledge pool, responsible for distilling and distributing knowledge across tasks. Furthermore, the information synchronization mechanism mitigates imbalances in knowledge distri- bution, ensuring optimal system-wide performance. Experiments demonstrate that dense knowledge rewards generated by the CRA effectively guide policy learning, leading to faster convergence than baseline methods. CenRA also demonstrates superior and robust transferability new tasks.
CenRA's main limitation is its requirement for consistent state and action dimensions across tasks. Future work could explore preprocessing techniques to adapt the framework to varying task structures, broadening its applicability. Additionally, the fixed trade-off between similarity weight and perfor- mance weight may not be ideal. A more flexible approach, such as adaptive weight regulation, could further enhance the framework. Moreover, the performance weight might favor underperforming tasks to achieve overall balance, but could limit the performance ceiling of high- perforreffoig tasks, indicating the need for a more effective trade-off mechanism.
Acknowledgment
This work was supported by an Academic Research Grant MOE-TI 251RES2408 and a Research Scholarship from the Ministry of Education, Singapore.
References
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Online multi-task learning for policy gradient methods. In International conference on machine learning, pages 1206-1214, PMLR, 2014.
Szilárd Aradi. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 23(2):740-759, 2020.
Fengshuo Bai, Hongming Zhang, Tianyang Tao, Zhiheng Wu, Yanna Wang, and Bo Xu. Picor: Multi- task deep reinforcement learning with policy correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6728-6736,
6736, 2023.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 29, 2016.
10

---

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Caruana. Multitask learning: A knowledge-based source of inductive biasl. Proceedings of the Tenth International Conference on Machine Learning, pages 41-48. Citeseer, 1993.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference machine learning, pages 794 403.. PMLR, 2018.
Guangran Cheng, Lu Dong, Wenzhe Cai, and Changyin Sun. Multi-task reinforcement learning with attention-based mixture of experts. IEEE Robotics and Automation Letters, 8(6):3812-3819 2023.
Maxime Chevalier-Boisvert,, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances in Neural Information Processing Systems, 36, 2024.
Carlo D' Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement learning. In International Conference on Learning Representa- tions, 2020.
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guiis guided reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35:5829-5842, 2022.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine, Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2169-211111111 2176. IEEE, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/ 259055 2018b.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2024.
Jinmin He, Kai Li. Yifan Zang, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Not all tasks are equally difficult: Multi-task deep reinforcement learning with dynamic depth routing. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 12376 12384, 2024.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796-3803, 2019.
Sunghoon Hong, Deunsol Yoon, and Kee-Eung Kim. Structure- transformer policy for inhomo- geneous multi-task reinforcement learning. International Conference on Learning Representa- tions, 2021.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoAGo GM Araãšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
11

---

Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan. Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong. Josef Dai, and Yaodong Yang. Safety gymnasium: unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 2023.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11): 2013.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms, Advances in neural information processing systems, 12, 1999.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Siming Lan, Rui Zhang, Qi Yi, Jiaming Guo, Shaohui Peng, Yunkai Gao, Fan Wu, Ruizhi Chen, Zi- dong Du, Xing Hu, et al. Contrastive modules with temporal attention for multi-task reinforcement learning. Advances Neural Information Processing Systems, 36, 2023.
Zhengding Luo, Haozhe Ma, Dongyuan Shi, and Woon-Seng Gan. Gfanc-rl: Reinforcement learning- based generative fixed-filter active noise control. Available SSRN 4837239, 2024.
Haozhe Ma. Thanh Vinh Vo, and Tze- Yun Leong. Hierarchical reinforcement learning with human-ai collaborative gub-goals optimization. In Proceedings of the 2023 international conference autonomous agents and multiagent systems, pages 2310-2312, 2023.
Haozhe Ma, Kuankuan Sima, Thanh Vinh Vo, Di Fu, and Tze- Yun Leong. Reward shaping for reinforcement learning with an assistant reward agent. In Forty-first International Conference on Machine Learning. PMLR, 2024a.
Haozhe Ma, Thanh Vinh Vo, and Tze Leong. Mixed-initiative bayesian sub-goal optimization in hierarchical reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 1328-1336, 2024b.
Haozhe Ma, Fangling Li, Jing Yu Lim, Zhengding Luo, Thanh Vinh Vo, and Tze-Yun Leong. Catching two birds with one stone: Reward shaping with dual random networks for balancing exploration and exploitation. In Ferty-second International Conference on Machine Learning, 2025a.
Haozhe Ma. Zhengding Luo, Thanh Vinh Vo, Kuankuan Sima, and Tze- Yun Leong. Highly efficient self-adaptive reward shaping for reinforcement learning. In Thirteenth International Conference on Learning Representations, 2025b.
Augustine Mlaoo-Parker. Kimberly Young, Caswell Barry, and Lewis Griffin. How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference on Machine Learning, pages 1.520-112444 15240. PMLR, 2022.
Farzan Memarian, Wonjoon Goo, Rudolf Lioutikov, Scott Niekum, and Ufuk Topeu. Self-supervised online reward shaping in sparse-reward environments. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2369-2375. IEEE, 2021.
David Mguni, Taher Jafferjee, Jianhong Wang, Nicolas Perez-Nieves, Wenbin Song, Feifei Tong, Matthew Taylor, Tianpei Yang, Zipeng Dai, Hui Chen, et al. Learning shape rewards using a game of two partners. In AAAI Conference on Artificial Intelligence, volume 37, pages 11604- 11612, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 51 (55550):559-533 2015.
Siddharth Mysore, George Cheng, Yunqi Zhao, Kate Saenko, and Meng Wu. Multi-critic actor learn- ing: Teaching policies to act with style. International Conference on Learning Representations, 2022.
12

---

Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In Intermational Conference on Machine Learning, pages 2721-2730. PMLR, 2017.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback, Advances in neural information processing systems, 35:27730- 27744, 2022.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In International Conference on Learning Representations, 2016.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pages 2778-2788 PMLR, 2017.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukeuoglu, and Raia Hadsell. Policy distillation. In International Conference on Learning Representations, 2016.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 1186634-6622,00 2023.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. International Conference on Machine Learning, pages 9767-9779. PMLR, 2021.
Jonathan Sorg, Richard Lewis, and Satinder Singh. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23, 2010a.
Jonathan Sorg. Satinder Singh, and Richard I. Lewis. Internal rewards mitigate agent boundedness. In International Conference on Machine Learning, pages 1007-1014, 2010b.
Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-compositiona multi-task reinforcement learning. Advances in Neural Information Processing Systems, 35: 21495 21507, 2022.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Yee Teh, Vietor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Tung-Long Vuong, Do-Van Nguyen, Tai-Long Nguyen, Cong-Minn Minh Bui, Hai-Dang Kieu, Viet-Cuong Ta, Quoc-Long Tran, and Thanh-Ha Le. Sharing experience in multitask reinforcement learning. In International Joint Conference on Artificial Intelligence, pages 3642-3648, 2019.
Michael Wan, Tammay Gangwani, and Jian Peng. Mutual information based knowledge transfer under state-action dimension mismatch. In Conference on Uncertainty in Artificial Intelligence, pages 1218-1227. PMLR, 2020.
Tengye Xu, Zihao Li, and Qinyuan Ren. Meta-reinffrreement learning robust to distributional shift via performing lifelong in-context learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 55112-55125. PMLR, 2024.
13

---

Zhiyuan Xu, Kun Wu, Zhengping Che, Jian Tang, and Jieping Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. Advances in Neural Information Processing Systems, 10:15146-15155 2020.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances Neural Information Processing Systems, 13:4767-4777, 2020.
Zhaoyang Yang, Kathryn E Merrick, Hussein A Abbass, and Lianwen Jin. Multi-task deep rein- forcement learning for continuous action control. In International Joint Conference Artificial Intelligence, volume 17, pages 3301-3307, 2017.
Haiyan Yin and Sinno Pan. Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33: 5824-5836, 2020a.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages 1094-1 1000. PMLR, 2020b.
Sihan Zeng, Malik Aqeel Anwar, Thinh T Doan, Arijit Raychowdhury, and Justin Romberg. A decentralized policy gradient approach to multi-task reinforcement learning. In Uncertainty in Artificial Intelligence, pages 1002-1012. PMLR, 2021.
Gengzhi Zhang, Liang Feng, and Yaqing Hou. task actor-critic with knowledge transfer via shared critic. In Asian Conference on Machine Learning, pages 580-593, PMLR, 2021.
14

---

A Mutli-Task Experimental Configurations
We conduct experiments in four domains with multiple tasks: Meta- World, 2DMaze, 3DPickup, and MujocoCar. The detailed configurations of each task are illustrated in Figure The Meta- World tasks illustration is adapted from Yu et al. 2020b]
aseembly
pull
picking
topdowrn
topdown woll
wall
dal turn
dinossemale
Goor closs
door
drover cicss
drower open
Toucel open
foucet close
hommer
box close
umplug
side
sice
Side
S:00
hole
pick place
picto slice
plate side
plote aide
biete dide
push book
push
push wall
reach
door unlock
Meta- Meta-World training testing lasks.
Maze
Maze
Moze #3
Maze #4
New 2DMe task
D
Ball
Cube
Key
Health kit
New 3DPickup task
Targer #1
Target #2
Target #3
Target #4
New MujocooCar task
Figure 5: Illustration of multiple tasks in different domains experiments.
B
Network Structures and Hyperparameters
B.1
Network Structures
Figure iillustrates the structures of all networks employed in our experiments.
B.2
Hyperparameters
We have observed that CenRA demonstrated high robustness and was not sensitive hyperparameter choices. T shows the hyperparameters used in all the experiments.
15

---

Q-value
Q-valie
actions
layer
layer
sample
ReLU
ReLU
Normal distributions
Q-valao
FC layer (flason 1310
FC layer (fluse $120
PC layer (226.1
ReLU
ReLU
mcams
standard deviations
ReLU
bunb
Convolution 2D (82 64.2)
Convolation 2D 644 128.)
FC layer
PC layer 025 acilen slags)
PC layer 239)
(25 actins-lupe)
ReLU
Rel.U
ReLUU
Relt:
Convolution 0223
Convolution 5S 6433
FC layer (256 2255
FC layer fulorratton
ass 260)
ReLU
ReLU
R+LU
FC layer (obvereatice 201)
(observation, action)
Convolution 2D
Convolution 2D
(upur,cksarol, (63)
observation
observation
observation
Q-network for 2DMaze
(b) Q-network for 3DPickup
( SAC policy network
(d) SAC Q-network
Figure 6: The structures of neural networks in our implementation.
Table 4: The hyperparameters of CenRA used in our experiments.
Module
Hyperparameters
Values
discounted factor ?y batch size
0.99
256
actor module learning rate critic module learning rate
3 x 10
Centralized Reward Agent rnd
10 3
policy networks update frequency (steps) target networks update frequency (steps) target networks soft update weight
2
1 10-3
burn-in steps
5000
knowledge reward weight discounted factor
0.5
0.99
Policy Agent A (DQN Agent)
replay buffer size 'D,]
1 10"
batch size
128
burn-in steps
10000
knowledge reward weight
0.5
discounted factor
0.99
replay buffer size 'D:] batch size
1 x 10
256
Policy Agent A (SAC Agent)
actor module learning rate critic module learning SAC entropy term factor a lcarning rate policy networks update frequeney (steps) target networks update frequency (steps) target networks soft update weight burn-in steps
3 x 10
1 10
1 x 10 2 1
x 10 10000
B.3 Computing Resources
The experiments in this paper were conducted on computing cluster, with the detailed hardware configurations listed in Table 5
Table The computing resources used in the experiments.
Component
Specification
Operating System (OS)
Ubuntu 20.04 Intel Xeon Gold 6326 256GB
Central Processing Unit (CPU)
Random Access Memory (RAM)
Graphics Processing Unit (GPU)
1x NVIDIA A100 20GB
Brand
Supermicro 2022
16

---

C What Has the Centralized Reward Agent Learned?
In this section, we visualize the learned knowledge rewards by the centralized reward agent Arwd in the 2DMaze environment. After training on the four tasks in Section 5.1 of the paper, we let the CRA generate the knowledge rewards for each action every state and visualize the action direction that yields the maximum rewards,
arg max, x "=nd" (s;, a), 8; S, in Figure
The shaded areas in the figures represent regions within the real task that the agent cannot reach, as cannot access the space behind the door without picking up the key. However, forced the agent into these areas for evaluation. Outside the shaded regions, we observe that the CRA successfully learned meaningful knowledge rewards. Before picking up the key, the agent received the highest reward in the corresponding state when moving towards the key. Similarly, after picking the key, the agent received the highest reward when moving towards the door and the final target. This demonstrates that in scenarios where the original environmental rewards are sparse, these detailed knowledge rewards can effectively guide the agent converge more quickly.
Before getting key
After getting key
Before getting key
After getting key
(a) Maze #1
(b) Maze #2
Before getting key
After getting key
Before getting key
After getting key
(c) Maze #3
(d) Maze #4
Figure 7: The actions yielding maximum knowledge rewards in the four 2DMaze tasks.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the main claims of the paper, including the contributions made (check the last paragraph in Section Tp. the theoretical and experimental results, and the scope of generalization other settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are discussed in the last paragraph of Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[Yes]
The paper includes full set of assumptions and complete derivations for the theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Answer: [Yes]
Justification: The paper provides the code with detailed instructions in the supplementary materials. The algorithm flow is detailed in Algorithm L and the experimental details, including network structures and hyperparameters, are provided in Appendix B These disclosed.
ensure that all necessary information for reproducing the main experimental results is fully
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
19

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have attached the complete code as supplementary materials, with detailed instructions how the code. We will make the complete code public after the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ mublic/guidese otte//oommmmmmiiiiooooooiioooiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. http/uullcggguucee htt///CooeSubiiiseeeeee asioonooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary to understand the results. The complete codes attached supplementary materials. Detailed instructions for using the provided code, including scripts to run the codes one step and environment setup instructions, included in the supplementary materials. The experimental details, including network structures and hyperparameters, are provided in Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For all experiments this paper, we run different seeds and use the average value as the result, with the standard error also included in the experimental results.
20

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources for the experiments, including the type of compute workers (CPU or GPU), memory, and execution time, are listed Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
The research conducted in this paper adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
21

---

Answer:
[NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deg.oymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not involve the release of data or models that pose a high risk for misuse. Therefore, specific safeguards necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All references and codes used in the paper are open access. The creators original owners are properly credited, and the licenses and terms of use explicitly mentioned and respected. This includes citing the original papers, stating the versions the assets used, and including URLs where applicable.
Guidelines:
22

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprssithcaaeeo thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The documentation for the new assets introduced in the paper is provided alongside the supplementary material. This includes details about the codes, training procedures, and license information.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, so this question is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
23

---

Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, so IRB approval equivalent is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
24


Reviews:
[{"review_id": "c5Tuu6p8Hi", "reviewer": "Reviewer_V8Sn", "summary": "The paper proposes a Centralized Reward Agent based Multi-Task RL framework (CenRA), where a centralized reward agent learns the policy to predict knowledge rewards that optimize the accumulated environment rewards it will receive, and policy agents for each task update their own policies by any RL algorithm to optimize the augmented reward (addition of the environment reward and the scaled knowledge reward). CenRA shows empirically improved performance compared to prior work on Meta-World, 2DMaze, 3DPickup, and MujocoCar.", "questions": "As described in Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* The paper is well structured, easy to follow, and clear.\n* The proposed method, including the core knowledge distillation and distribution, as well as the information synchronization mechanism (similarity weight and performance weight), is reasonable to mitigate potential issues in such a framework.\n* The experiments are conducted on Meta-World benchmark and additional 3 environments, showing consistent improvements.\n* Although I have some questions about completeness (as described in Weaknesses), the paper has included important analyses like an ablation study of the similarity weight and performance weight, knowledge transfer to unseen tasks, and the visualization of learned knowledge rewards.\n\n### Weaknesses\nMy questions are mainly about the experiment design and analysis to validate some components’ effectiveness:\n* Since the 2DMaze involves the least diverse tasks (only the position and colors are different, but the goals are the same; the unseen task even shares the same colors as one of the seen task) among the four benchmarks, the claim of the success of knowledge transfer in * Section 5.2 will be strengthened if the paper can show Meta-World’s visualization results. \n* In addition to Table3 for sampling weight ablation study, can the authors also show the Meta-World’s results? If the number of tasks is a concern to list the table, it will also be beneficial to see the results averaged across tasks.", "was_revised": true, "final_justification": "As the authors have addressed all of my concerns and most of the other reviewers' questions with evidence, I have raised my score accordingly.", "rebuttal": "Dear reviewer,\n\nThanks for the comments and below we provide detailed responses.\n\n## Weakness 1\n\n> Since the 2DMaze involves the least diverse tasks (only the position and colors are different, but the goals are the same; the unseen task even shares the same colors as one of the seen task) among the four benchmarks, the claim of the success of knowledge transfer in Section 5.2 will be strengthened if the paper can show Meta-World’s visualization results.\n\nRegarding the visualization of the Meta-World tasks in Section 5.2, due to response format constraints, we are unable to upload PDF files or share anonymized image links. Therefore, we present a tabular format to “visualize” the CRA-generated rewards for the *door-unlock* task in Meta-World, which involves multi-step manipulation and spatio-temporal reasoning. The table illustrates how the CRA assigns different rewards at various stages of task completion, such as approaching the door handle, aligning with it, and rotating it to unlock, etc., highlighting its ability to produce meaningful rewards. The real visualizations with the corresponding figures will be included in the revised manuscript.\n\n| Description of Task Stage | CRA Reward |\n| :---: | :---: |\n| Initialization | 0.001 |\n| Agent moves towards the door handle. | 0.122 |\n| Agent is close to the door handle. | 0.213 |\n| Agent's gripper is roughly aligned with the handle. | 0.458 |\n| Agent's gripper has successfully grasped the handle. | 0.463 |\n| Agent starts rotating the handle. | 0.702 |\n| Handle is partially rotated. | 0.785 |\n| Handle is fully rotated to the unlocked position. | 0.833 |\n| Agent pushes the door slightly open. | 0.857 |\n| Door is fully open. | 0.925 |\n\n## Weakness 2\n\n> In addition to Table3 for sampling weight ablation study, can the authors also show the Meta-World’s results? If the number of tasks is a concern to list the table, it will also be beneficial to see the results averaged across tasks.\n\nRegarding the ablation study of sampling weights, due to the large number of tasks in the Meta-World benchmark (45 training tasks), we didn't present the task-wise results in the main paper. Here, we report the averaged results across all tasks within each domain to summarize the impact of different ablation settings. Additionally, we extend the analysis by introducing two more ablation settings with different trade-offs between the similarity weight and the performance weight by setting $\\alpha = 0.25$ and $\\alpha = 0.75$. The detailed results are presented in the table below. These results further confirm the effectiveness and robustness of our weighting strategy.\n\nTable 1: The average performance per domain of CenRA with different $\\alpha$ sampling weights (**higher is better**).\n\n| Algo. | *ML10-sparse* | *ML50-sparse* | *2DMaze* | *3DPickup* | *MujocoCar* |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| CenRA ($\\alpha = 0.5$) | **0.875 ± 0.121** | **0.755 ± 0.034** | **0.913 ± 0.023** | **0.880 ± 0.060** | **514.875 ± 0.675** |\n| **CenRA ($\\alpha = 0.25$)** | 0.852 ± 0.115 | 0.738 ± 0.038 | 0.895 ± 0.028 | 0.865 ± 0.062 | 498.210 ± 0.705 |\n| **CenRA ($\\alpha = 0.75$)** | 0.868 ± 0.125 | 0.749 ± 0.036 | 0.908 ± 0.025 | 0.862 ± 0.061 | 507.450 ± 0.690 |\n| w/o $w^{sim}$ ($\\alpha = 0$) | 0.785 ± 0.095 | 0.670 ± 0.050 | 0.872 ± 0.033 | 0.778 ± 0.068 | 349.025 ± 0.635 |\n| w/o $w^{per}$ ($\\alpha = 1$) | 0.730 ± 0.080 | 0.615 ± 0.045 | 0.832 ± 0.040 | 0.563 ± 0.085 | 332.125 ± 0.790 |\n| w/o both | 0.680 ± 0.070 | 0.550 ± 0.040 | 0.720 ± 0.055 | 0.530 ± 0.078 | 258.850 ± 0.458 |\n\nTable 2: The variances across multiple tasks per domain of CenRA with different $\\alpha$ sampling weights (**lower is better**).\n\n| Algo. | *ML10-sparse* | *ML50-sparse* | *2DMaze* | *3DPickup* | *MujocoCar* |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| CenRA ($\\alpha = 0.5$) | **0.08** | **0.16** | **0.02** | 1.57 | 4.24 |\n| **CenRA ($\\alpha = 0.25$)** | 0.18 | 0.22 | 0.03 | 1.82 | 6.88 |\n| **CenRA ($\\alpha = 0.75$)** | 0.17 | 0.21 | 0.03 | 1.65 | 5.53 |\n| w/o $w^{sim}$ ($\\alpha = 0$) | 0.25 | 0.18 | 0.17 | **0.89** | 9.29 |\n| w/o $w^{per}$ ($\\alpha = 1$) | 0.35 | 0.24 | 0.28 | 3.05 | **0.46** |\n| w/o both | 0.40 | 0.68 | 1.24 | 3.80 | 82.7 |\n\nOnce again, we appreciate your valuable feedback and hope our response has addressed your concerns."}, {"review_id": "VEirUFMQmS", "reviewer": "Reviewer_UzuT", "summary": "This paper explores the use of reward shaping for multi-task reinforcement learning via a centralised reward agent. In overly simple words, CenRA is an extension of the reward shaping methodology to multi-task reinforcement learning. However, behind this \"simple\" sentence, there is a novel idea that is worth exploring.\n\nIndeed, while reward shaping via an assistant reward agent is not a new concept, the co-articulation of this paradigm with multi-task reinforcement learning has not been explored before to the best of my knowledge, and could be done in very different ways. The authors propose to have a shared reply buffer (to reduce memory usage and simplify the code) as well as adaptive strategies to weight the various tasks depending on their similarity and the learning performance of the associated task-specific agents.\n\nExperiments are conducted and reported in several environment sets, exhibiting consistent results and validating the intuition behind the paper.", "questions": "The core idea of the paper is to adopt the reward assistant agent paradigm for multi-task reinforcement learning. In this regard, the main methodological contribution should be in this \"adoption\". The authors have chosen fairly simple weight models and weight combination strategies (which I see as a positive thing).\n\nHowever, there is not much discussion on other modelling possibilities. Weights and their combinations can be modelled in very different ways, and this can have a severe impact on the experiments. I wonder if the authors carried out an in-depth exploration.\n\nIn addition, regarding the task similarity weights, the task centroid \"c\" could be zero (or arbitrarily close to zero), thus opening the door to strong numerical stabilities. I would like the authors to comment on that point.\n\nLikewise, for the task performance weights, the numerical range of the rewards of the various tasks is potentially very different (there are no constraints whatsoever announced by the authors), and therefore using the average past rewards to re-weight the replay buffer can be very problematic since tasks that naturally have a large reward would be visited less often, but for the wrong reason.\n\nOverall, I believe that the main contribution of the paper (these weights and weight combination strategies) deserve more exploration both in terms of internal comparison performance as well as in terms of numerical instabilities.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The idea has not been explored before.\n- The model design is simple and takes into account memory/code factors.\n- The paper is well written and easy to understand.\n\nWeaknesses:\n- The motivation for which only similarity and learning weights are chosen is unclear, as it is the way to combine those weights.\n- The intuition behind how to compute the weights is clear, but this computation is potentially prone to numerical instabilities.\nThese weaknesses are an issue because the weighting strategy is the main methodological contribution of the paper.", "was_revised": true, "final_justification": "I updated my rating from Borderline Reject to Borderline Accept accordingly to the authors rebutal.", "rebuttal": "Dear reviewer,\n\nThank you for your comments. We have addressed your concerns in detail, along with the identified weaknesses and questions.\n\n## Weakness 1 and Question 1\n\n> (Weakness) The motivation for which only similarity and learning weights are chosen is unclear, as it is the way to combine those weights.\n\n> (Question) However, there is not much discussion on other modelling possibilities. Weights and their combinations can be modelled in very different ways, and this can have a severe impact on the experiments. I wonder if the authors carried out an in-depth exploration.\n\nRegarding the motivation for using similarity and performance weights in the information synchronization mechanism:\n\n1. **Motivation**: Our design is motivated by a systematic consideration of two complementary aspects in MTRL. The **similarity weight** captures structural disparities across tasks, allowing the CRA to attend more to outlier tasks and promote effective knowledge sharing. The **performance weight** monitors the real-time learning progress of each agent, encouraging the CRA to allocate more attention to underperforming tasks. The combination of the two enables the system to simultaneously maintain *representation diversity* and *training balance*, achieving both broad knowledge distillation and efficient convergence.\n\n2. **Justification**: These weights are not only intuitive and interpretable, but also computationally efficient, as they rely on existing quantities—latent features and recent reward statistics. Moreover, their individual importance is empirically validated in Section 5.3, where removing either component leads to performance drops and increased variance across tasks (see Table 3).\n\n3. **Exploration of More Formulations**: We intentionally began with these two **representative and controllable factors** to establish a principled and effective baseline. We agree that other formulations are worth exploring. There are two potential directions for future work:\n\n* Learnable or adaptive weighting schemes;\n* Alternative performance proxies, such as TD-error, policy entropy, or gradient norms.\n\n4. **More experimental results on different weights combinations**: Finally, beyond the ablation results in Table 3, we have conducted an additional study (see the followng table) analyzing the effect of different trade-off parameter $\\alpha$. The results indicate that while both weights significantly contribute to performance, the framework remains robust across a range of reasonable $\\alpha$ values, demonstrating the flexibility of the proposed design.\n\nTable 1: The average performance per domain of CenRA with different $\\alpha$ sampling weights (**higher is better**).\n\n| Algo. | *ML10-sparse* | *ML50-sparse* | *2DMaze* | *3DPickup* | *MujocoCar* |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| CenRA ($\\alpha = 0.5$) | **0.875 ± 0.121** | **0.755 ± 0.034** | **0.913 ± 0.023** | **0.880 ± 0.060** | **514.875 ± 0.675** |\n| **CenRA ($\\alpha = 0.25$)** | 0.852 ± 0.115 | 0.738 ± 0.038 | 0.895 ± 0.028 | 0.865 ± 0.062 | 498.210 ± 0.705 |\n| **CenRA ($\\alpha = 0.75$)** | 0.868 ± 0.125 | 0.749 ± 0.036 | 0.908 ± 0.025 | 0.862 ± 0.061 | 507.450 ± 0.690 |\n| w/o $w^{sim}$ ($\\alpha = 0$) | 0.785 ± 0.095 | 0.670 ± 0.050 | 0.872 ± 0.033 | 0.778 ± 0.068 | 349.025 ± 0.635 |\n| w/o $w^{per}$ ($\\alpha = 1$) | 0.730 ± 0.080 | 0.615 ± 0.045 | 0.832 ± 0.040 | 0.563 ± 0.085 | 332.125 ± 0.790 |\n| w/o both | 0.680 ± 0.070 | 0.550 ± 0.040 | 0.720 ± 0.055 | 0.530 ± 0.078 | 258.850 ± 0.458 |\n\n\nTable 2: The variances across multiple tasks per domain of CenRA with different $\\alpha$ sampling weights (**lower is better**).\n\n| Algo. | *ML10-sparse* | *ML50-sparse* | *2DMaze* | *3DPickup* | *MujocoCar* |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| CenRA ($\\alpha = 0.5$) | **0.08** | **0.16** | **0.02** | 1.57 | 4.24 |\n| **CenRA ($\\alpha = 0.25$)** | 0.18 | 0.22 | 0.03 | 1.82 | 6.88 |\n| **CenRA ($\\alpha = 0.75$)** | 0.17 | 0.21 | 0.03 | 1.65 | 5.53 |\n| w/o $w^{sim}$ ($\\alpha = 0$) | 0.25 | 0.18 | 0.17 | **0.89** | 9.29 |\n| w/o $w^{per}$ ($\\alpha = 1$) | 0.35 | 0.24 | 0.28 | 3.05 | **0.46** |\n| w/o both | 0.40 | 0.68 | 1.24 | 3.80 | 82.7 |\n\n\n## Weakness 2 and Question 2\n\n> (Weakness) The intuition behind how to compute the weights is clear, but this computation is potentially prone to numerical instabilities. These weaknesses are an issue because the weighting strategy is the main methodological contribution of the paper.\n\n> (Question) In addition, regarding the task similarity weights, the task centroid \"c\" could be zero (or arbitrarily close to zero), thus opening the door to strong numerical stabilities. I would like the authors to comment on that point.\n\nRegarding the concern that the task centroid $\\mathbf{c}$ could become zero (or arbitrarily close to zero), potentially causing numerical instabilities in the similarity computation, we would like to clarify that this issue has been explicitly considered in our design. To mitigate the risk of feature vectors pointing in opposing directions and canceling each other out, in our approach, all latent features $\\mathbf{H}\\_i$ are extracted from the activation outputs of neural network layers, specifically, ReLU activations in our implementation, ensuring that all elements of $\\mathbf{H}\\_i$ are non-negative. This effectively prevents cancellation effects and ensures that the resulting centroid $\\mathbf{c}$ remains well-defined and numerically stable. The relevant implementation details can be found in the `get_features` function in `CenRA/Networks.py` within our submitted codes. We will further clarify this point in the revised manuscript.\n\n## Question 3\n\n> Likewise, for the task performance weights, the numerical range of the rewards of the various tasks is potentially very different (there are no constraints whatsoever announced by the authors), and therefore using the average past rewards to re-weight the replay buffer can be very problematic since tasks that naturally have a large reward would be visited less often, but for the wrong reason.\n\nRegarding the concern on reward scale differences affecting the performance weights, we would like to clarify that in our current setup, as the multi-task environments are defined within the same domain, thus their reward functions are consistent in numerical scale, e.g, there are **no cases** where some tasks yield rewards in the range $[0, 1]$ while others operate in $[0, 100]$. Therefore, the recent average rewards used in the performance weighting are directly comparable across tasks. However, considering that in more general settings where tasks may have different reward magnitudes, we suggest applying a normalization wrapper to rescale the rewards of all tasks to a common range before computing the performance weights, which can be easily implemented by using `gym.wrappers.TransformReward` for standard OpenAI Gym environments.\n\nThanks again and hope our response has addressed your concerns."}, {"review_id": "41oKjuZlVV", "reviewer": "Reviewer_vuKu", "summary": "The paper follows the idea of knowledge sharing by reward shaping and propose a framework to apply it to multitask RL problems. The proposed framework utilizes $N$ policies to solve $N$ tasks, and a centralized policy that shapes the rewards of these policies. To train the centralized policy, the authors use a weighted sampling approach, which is later justified in ablation experiments. The framework is evaluated in several benchmarks, demonstrating the performance of the proposed framework comparing to various baselines. The generalization ability of the centralized reward agent to unseen tasks is also demonstrated.", "questions": "1. Did you experiment in multi domain setting (i.e., tasks can be in different domains)?\n2. Is the reward balanced (same scale) across tasks among all the presented simulations? Do you have a suggestion how to adapt for such cases?\n3. for the DQN/SAC baselines, did you use a single instance for all tasks or one instance per task?", "limitations": "Limitations of this work are mostly discussed, I think that a few sentences regarding working with tasks that have different reward scale should be included.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The framework is simple and effective, it allows using any backbone RL algorithm, where each task could be in different domain.\n2. Extensive experiments across various domains.\n3. The paper is well-written, Figure 1 and Algorithm 1 helps the reader to understand the method.\n4. Justifies the proposed sampling method with an ablation experiment.\n\n## Weaknesses\n1. The ablation experiment validates the usage of the proposed weighted sampling, although it raises a concern regarding the contribution of the reward-generating agent. Consider adding the following baseline: shared critic that depends on the task info as well, trained with/without the proposed weighted sampling approach. \n2. More MTRL/ contextual RL methods are missing, most of the baselines are not intended for this setting. I am not sure which methods are considered SOTA in this setting, but there are more recent works that fit for this setting.\n3. As I understand, the reported results were generated from a single training of each baseline. It is important to evaluate over a few instances of each baseline, since the training process may vary for different random seeds. This is a major concern since the paper relies on the empirical evidence in these experiments.", "was_revised": true, "final_justification": "The authors addressed my concerns; added a baseline to demonstrate the contribution of the reward sampling method, included more MTRL baselines, and clarified the seeds issue. In light of the other reviews and responses, I raise my score to accept.", "rebuttal": "Dear reviewer,\n\nThanks for your comments, and below we provide detailed responses.\n\n## Weaknesses\n\n**Weakness 1**:\n\n> The ablation experiment validates the usage of the proposed weighted sampling, although it raises a concern regarding the contribution of the reward-generating agent. Consider adding the following baseline: shared critic that depends on the task info as well, trained with/without the proposed weighted sampling approach.\n\nRegarding the contribution of the reward agent and suggested shared-critic baseline:\n\nWe have included the *Shared-Critic* framework for MTRL as an additional baseline [3]. The author-proposed method doesn't have weighted sampling, so we compare with both the original shared-critic method (SC w/o sampling) and a modified version that incorporates our proposed weighted sampling approach (SC w/ sampling). The results are shown as follows:\n\n| Algorithm | *ML10-sparse* | *ML50-sparse* | *2DMaze* | *3DPickup* | *MujocoCar* |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| CenRA (ours)| **0.875 ± 0.121** | **0.755 ± 0.034** | **0.913 ± 0.023** | **0.880 ± 0.060** | **514.875 ± 0.675** |\n| **Shared-Critic (SC w/o sampling)** | 0.556 ± 0.063 | 0.354 ± 0.023 | 0.798 ± 0.052 | 0.687 ± 0.038 | 400.254 ± 0.518 |\n| **Shared-Critic (SC w/ sampling)** | 0.604 ± 0.058 | 0.407 ± 0.031 | 0.725 ± 0.049 | 0.526 ± 0.045 | 430.179 ± 0.567 |\n\nFrom the results, our CenRA method consistently outperforms SC. We highlight two key advantages:\n\n1. **Reward-Based Knowledge Transfer**: CenRA used a centralized reward agent to share knowledge by generating *dense and informative rewards* for each policy agent. This allows CenRA to provide *step-wise guidance*, which is particularly beneficial in challenging sparse-reward environments. In contrast, SC's shared critic is still trained only on sparse rewards, so it requires extensive exploration for the actors to learn effectively.\n2. **Contribution of the Reward Agent**: Even with our proposed weighted sampling, the performance of SC is still lower than CenRA, indicating that the centralized reward agent plays a crucial role in enhancing knowledge transfer. The weighted sampling helps balance the contributions of different tasks, but it does not fully compensate for the lack of a dedicated reward shaping mechanism.\n\nThe new experiments will be added into the revised manuscript.\n\n[3] Zhang, Gengzhi et al., \"Multi-task Actor-Critic with Knowledge Transfer via a Shared Critic\". In Proceedings of the 13th Asian Conference on Machine Learning (Vol. 157, pp. 580–593). PMLR.\n\n**Weakness 2**:\n\n> More MTRL/ contextual RL methods are missing, most of the baselines are not intended for this setting. I am not sure which methods are considered SOTA in this setting, but there are more recent works that fit for this setting.\n\nRegarding the concern that more recent MTRL methods are missing, we would like to emphasize that our paper already includes several representative and competitive baselines. Specifically, we compared against PiCor (AAAI 2023), MCAL (ICLR 2022), PaCo (NeurIPS 2022), and SoftModule (NeurIPS 2022), all of which are recognized as strong MTRL algorithms.\n\nIn addition, we have conducted further comparisons with several more recent and relevant baselines: (1) **TD-MPC2** (ICLR 2024) [1]; (2) **CMTA** (NeurIPS 2023) [2]; (3) **Shared-Critic (SC)** (ACML 2021) [3]; (4) **Distral** (NeurIPS 2017) [4]. In which the TD-MPC2 and CMTA can be considered as SOTA methods in MTRL, while the SC and Distral are also two representative baselines for multi-task RL. The results of these additional baselines are summarized in the following table:\n\n| Algorithm | *ML10-sparse* | *ML50-sparse* | *2DMaze* | *3DPickup* | *MujocoCar* |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| CenRA (ours)| **0.875 ± 0.121** | **0.755 ± 0.034** | **0.913 ± 0.023** | **0.880 ± 0.060** | **514.875 ± 0.675** |\n| DQN/SAC | 0.256 ± 0.056 | 0.189 ± 0.012 | 0.645 ± 0.070 | 0.243 ± 0.048 | 198.000 ± 0.453 |\n| ReLara | 0.674 ± 0.105 | 0.541 ± 0.057 | 0.803 ± 0.065 | 0.565 ± 0.088 | 429.800 ± 0.655 |\n| PiCor | 0.865 ± 0.230 | 0.672 ± 0.123 | 0.818 ± 0.053 | 0.438 ± 0.085 | 437.550 ± 0.663 |\n| MCAL | 0.842 ± 0.067 | 0.605 ± 0.055 | 0.885 ± 0.080 | 0.548 ± 0.068 | 369.200 ± 0.595 |\n| PaCo | 0.854 ± 0.045 | 0.582 ± 0.022 | 0.834 ± 0.057 | 0.557 ± 0.072 | 421.210 ± 0.635 |\n| SoftModule | 0.630 ± 0.042 | 0.423 ± 0.057 | 0.822 ± 0.076 | 0.486 ± 0.055 | 355.125 ± 0.594 |\n| TD-MPC2 | 0.823 ± 0.091 | 0.608 ± 0.032 | 0.884 ± 0.046 | 0.712 ± 0.051 | 505.341 ± 0.712 |\n| CMTA | 0.787 ± 0.076 | 0.603 ± 0.026 | 0.753 ± 0.037 | 0.695 ± 0.043 | 480.187 ± 0.623 |\n| Shared-Critic (SC w/o sampling) | 0.556 ± 0.063 | 0.354 ± 0.023 | 0.798 ± 0.052 | 0.687 ± 0.038 | 400.254 ± 0.518 |\n| Shared-Critic (SC w/ sampling) | 0.604 ± 0.058 | 0.407 ± 0.031 | 0.725 ± 0.049 | 0.526 ± 0.045 | 430.179 ± 0.567 |\n| Distral | 0.509 ± 0.072 | 0.406 ± 0.020 | 0.663 ± 0.062 | 0.655 ± 0.033 | 380.098 ± 0.499 |\n\nThe experimental results from these newly added baselines further validate the effectiveness of CenRA, which consistently achieves strong or superior performance across all evaluated domains. These new results will be included in the revised version of the manuscript.\n\n[1] Hansen, Nicklas et al. \"TD-MPC2: Scalable, Robust World Models for Continuous Control.\" The 12th International Conference on Learning Representations (2024).\n\n[2] Lan, Siming et al., \"Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning\". In Proceedings of the 37th International Conference on Machine Learning (2023). PMLR.\n\n[3] Zhang, Gengzhi et al., \"Multi-task Actor-Critic with Knowledge Transfer via a Shared Critic\". In Proceedings of the 13th Asian Conference on Machine Learning (Vol. 157, pp. 580–593). PMLR.\n\n[4] Yee Whye Teh et al., \"Distral: Robust multitask reinforcement learning\". Advances in Neural Information Processing Systems 30 (2017): 4496-4506.\n\n**Weakness 3**:\n\n> As I understand, the reported results were generated from a single training of each baseline. It is important to evaluate over a few instances of each baseline, since the training process may vary for different random seeds. This is a major concern since the paper relies on the empirical evidence in these experiments.\n\nRegarding the evaluation with different random seeds, in our experiments, we indeed run multiple instances with different random seeds for each setting and baseline. This has been stated in our paper (**Line 247**): *\"Each task is trained with 10 different random seeds, and the average results are reported.\"*.\n\n\n## Questions\n\n> 1. Did you experiment in multi-domain setting (i.e., tasks can be in different domains)?\n\nRegarding the multi-domain setting: our experiments are conducted under the standard MTRL setting, where tasks are generally assumed to share the same domain but differ in their specific configurations, targets, and other factors. For example, in the Meta-World benchmark, all tasks involve robotic manipulation but vary in terms of the objects to be grasped or the types of interactions required. These tasks differ in observations and targets but still reside within a consistent domain (e.g., robotic control).\n\nIn contrast, *multi-domain* settings where tasks span across fundamentally different domains are not usually considered in MTRL, (e.g., learning robotic manipulation and autonomous driving simultaneously). Such tasks involve distinct fundamental knowledge, making knowledge sharing less meaningful or even detrimental. Designing a unified framework for multi-domain MTRL remains an open and challenging problem, which we consider beyond the scope of this work.\n\n> 2. Is the reward balanced (same scale) across tasks among all the presented simulations? Do you have a suggestion how to adapt for such cases?\n\nRegarding the reward scale across tasks, we would like to clarify that currently, we are using the standard MTRL setting, as the tasks are defined within the same domain, their reward functions are consistent in numerical scale, e.g, there are **no cases** where some tasks yield rewards in the range $[0, 1]$ while others operate in $[0, 100]$. However, considering that in more general settings where tasks may have different reward magnitudes, we suggest applying a normalization wrapper to rescale the rewards of all tasks to a common range before computing the performance weights, which can be easily implemented by using `gym.wrappers.TransformReward` for standard OpenAI Gym environments.\n\n> 3. For the DQN/SAC baselines, did you use a single instance for all tasks or one instance per task?\n\nRegarding the DQN/SAC baselines, we assume that they are single-task settings, so we use **one independent instance of the DQN or SAC agent per task**. Each task is trained separately by assuming that they don't share any knowledge across tasks.\n\nThanks again, and we hope our response has addressed your concerns."}, {"review_id": "nKaoMkjTmn", "reviewer": "Reviewer_6NLb", "summary": "This paper proposes a novel multi-task reinforcement learning (MTRL) framework called CenRA, which incorporates a Centralized Reward Agent (CRA) to distill and distribute task-specific knowledge across different policy agents through dense shaped rewards. The CRA learns from the experience of all policy agents and returns \"knowledge rewards\" that augment sparse environmental feedback. An information synchronization mechanism balances the CRA’s focus by considering task similarity and learning progress, ensuring fair and effective knowledge distribution. Experiments on discrete and continuous tasks, including Meta-World, 2DMaze, 3DPickup, and MujocoCar, demonstrate significant improvements in learning efficiency, task balance, and knowledge transfer compared to strong baselines.", "questions": "1. How does the CRA generalize when tasks differ slightly in state/action space dimensions or in reward scale? Could preprocessing help extend to those cases?\n\n2. Is there any risk that the CRA introduces reward biases that harm long-term task-specific optimality?\n\n3. How does CenRA perform in continual learning or lifelong RL settings, where tasks arrive sequentially?\n\n4. Could the framework benefit from integrating meta-learning techniques to adapt the CRA or policy agents more efficiently?\n\n5. Have you explored ways to automatically adjust the α parameter in the synchronization mechanism instead of fixing it manually?\n\n6. What is the computational cost and training time increase introduced by CRA compared to decentralized baselines like ReLara?\n\nSince this article is outside the scope of my research, I will refer to the opinions of other reviewers in the future.", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Clear Motivation and Problem Relevance: Tackles the widely acknowledged issue of sparse rewards in MTRL, providing a practical and novel reward-centric solution.\n\n2. Innovative Framework Design: The separation between reward generation (CRA) and control (policy agents) is conceptually clean and modular, allowing easy integration into existing RL pipelines.\n\n3. Knowledge Distillation via Dense Rewards: Using learned rewards as a medium for inter-task knowledge transfer is both elegant and effective.\n\n4. Information Synchronization Mechanism: The combination of task similarity and learning performance for adaptive sampling ensures balanced system-wide learning.\n\n5. Strong Empirical Results: Demonstrates consistent improvements across multiple environments and outperforms competitive baselines.\n\n6. Knowledge Transfer to Unseen Tasks: Shows clear evidence that CRA retains and transfers useful knowledge, accelerating learning in new tasks.\n\n7. Reproducibility: Detailed algorithm descriptions, code availability, hyperparameter listings, and experimental setups are thoroughly presented.\n\nWeaknesses\n\n1. Assumption of Homogeneous Input/Output Shapes: CRA requires the same state and action shape across tasks, limiting its applicability to more heterogeneous domains.\n\n2. Fixed Trade-off Parameter ($\\alpha$) for Sampling Weights: The performance of the synchronization mechanism relies on a manually tuned α, which may not generalize well across tasks or domains.\n\n3. Computational Overhead: Maintaining and updating a centralized CRA while running multiple policy agents and computing cross-attention-based similarity adds nontrivial complexity.\n\n4. Limited Theoretical Analysis: Although the empirical results are strong, there is no theoretical discussion on the optimality or convergence of the CRA-generated rewards.\n\n5. CRA as a Bottleneck: In large-scale settings, the CRA may become a performance bottleneck due to its central role in reward generation and shared memory access.\n\n6. Underexplored Comparisons: More ablations comparing CenRA to existing multi-task policy distillation or attention-based modular methods would strengthen the position of the proposed architecture.", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your feedback. We respond to your comments as follows:\n\n## Weakness 1 & Question 1\n\nRegarding the assumption of homogeneous input/output shapes, the current design requires all tasks to share the same state and action shapes. This limitation is already discussed in the conclusion.\n\nHowever, we also propose a potential solution to extend CenRA to heterogeneous multi-task settings by **padding the state and action vectors** of tasks with shorter dimensions, aligning them to a unified shape. This idea has been adopted in previous works such as *TD-MPC2* (Hansen, Nicklas et al. ICLR 2024) and *KTM-DRL* (Xu, Zhiyuan, et al. NeurIPS 2020). We believe that this approach can be effectively applied to CenRA, allowing it to handle tasks with varying state/action dimensions.\n\n## Weakness 2 & Questions 5\n\nRegarding the trade-off parameter $\\alpha$ in the sampling weights, we would like to clarify that $\\alpha$ mainly aims to balance the contributions of the two weights. In our experiments, we adopt a fixed $\\alpha=0.5$, which assigns equal importance to both components, and ensures the final sampling weight in the range $[0,1]$: $\\omega=0.5 \\omega^{sim} + 0.5 \\omega^{per}$.\n\nWe want to highlight that, the $\\alpha$ is fixed, but the two weights $\\omega^{sim}$ and $\\omega^{per}$ are dynamically computed along with the training process, where $\\omega^{sim}$ captures structural disparities across tasks, allowing CRA to attend more to outlier tasks, and $\\omega^{per}$ monitors the real-time learning progress of each agent, allowing CRA to attend more to underperforming tasks. Since the two weights are dynamically computed, fixing $\\alpha$ does not limit the flexibility and actually is a more reasonable choice. This is also a straightforward way for users to set the relative importance of the two criteria if needed.\n\nFinally, to provide a more comprehensive analysis, beyond the ablation results in Table 3, we have conducted an additional study analyzing the effect of different trade-off $\\alpha$. The results indicate that while both weights significantly contribute to performance, the framework remains robust across a range of reasonable $\\alpha$ values, demonstrating the flexibility of the proposed design.\n\nTable 1: Average performance per domain (**higher is better**)\n\n|Algo.|*ML10-sparse*|*ML50-sparse*|*2DMaze*|*3DPickup*|*MujocoCar*|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|CenRA($\\alpha=0.5$)|0.88±0.12|0.76±0.03|0.91±0.02|0.88±0.06|514.88±0.68|\n|CenRA($\\alpha=0.25$)|0.85±0.11|0.74±0.04|0.90±0.03|0.87±0.06|498.21±0.71|\n|CenRA($\\alpha=0.75$)|0.87±0.12|0.75±0.04|0.91±0.03|0.86±0.06|507.45±0.69|\n|w/o $w^{sim}$|0.79±0.10|0.67±0.05|0.87±0.03|0.78±0.07|349.03±0.64|\n|w/o $w^{per}$|0.73±0.08|0.62±0.05|0.83±0.04|0.56±0.09|332.13±0.79|\n|w/o both|0.68±0.07|0.55±0.04|0.72±0.06|0.53±0.08|258.85±0.46|\n\nTable 2: The variances across multiple tasks per domain (**lower is better**)\n\n|Algo.|*ML10-sparse*|*ML50-sparse*|*2DMaze*|*3DPickup*|*MujocoCar*|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|CenRA ($\\alpha=0.5$)|**0.08**|**0.16**|**0.02**|1.57|4.24|\n|**CenRA($\\alpha=0.25$)**|0.18|0.22|0.03|1.82|6.88|\n|**CenRA($\\alpha=0.75$)**|0.17|0.21|0.03|1.65|5.53|\n|w/o $w^{sim}$|0.25|0.18|0.17|**0.89**|9.29|\n|w/o $w^{per}$|0.35|0.24|0.28|3.05|**0.46**|\n|w/o both|0.40|0.68|1.24|3.80|82.7|\n\n## Weakness 3 & Question 6\n\nRegarding the computational cost, the training time increase introduced by CRA, and the comparison with decentralized baselines like ReLara, we want to share the following highlights:\n\n1. **Memory cost of agents**: Assume there are $N$ tasks and the reward agent (RA) and policy agent (PA) have roughly equal model sizes. In ReLara, each task maintains its independent RA, thus, the total memory cost is proportional to $2N$ (i.e., $N$ PAs + $N$ RAs). In contrast, CenRA uses only a single centralized RA shared across all tasks, resulting in a total memory cost of $N+1$, which is significantly more efficient as the number of tasks grows. Even compared to the single-task setting, where each task is paired with one RL agent (i.e., $N$ agents in total), CenRA achieves comparable efficiency by requiring only one additional RA.\n2. **Replay buffer memory**: Both CenRA and ReLara train their RAs by sampling from the replay buffers of PAs, so the memory overhead related to experience storage is equivalent.\n3. **Training time cost**: Assuming that training one agent (policy or reward) takes approximately the same amount of time, decentralized ReLara requires training $2N$ agents, while CenRA trains only $N + 1$ agents. Therefore, CenRA achieves a clear advantage in overall training time, especially in large-scale task settings.\n4. **Resources distillation**: Fundamentally, CenRA is a centralized extension of ReLara, designed to distill and share task-relevant reward knowledge. In decentralized ReLara, each RA may redundantly learn overlapping knowledge across tasks. CenRA addresses this redundancy by unifying reward modeling into a single agent, thereby enabling more effective knowledge sharing while reducing both memory and computational burden.\n\nOverall, CenRA is more efficient than decentralized approaches like ReLara.\n\n## Weakness 4 & Question 2\n\nRegarding the theoretical analysis of the CRA-generated rewards on optimality and convergence, as well as the risk of introducing reward biases to the original task-specific rewards, we would like to highlight: CenRA is an extension of ReLara, which has already demonstrated the ability to learn meaningful and task-aligned reward structures. ReLara ensures that the shaped rewards are consistent with the original task-specific objectives and do not interfere with the long-term optimality of the learned policy. Given that CenRA follows a similar principle, we believe the risk of introducing harmful reward bias remains minimal. Furthermore, all experiments in our paper evaluate the **original environmental rewards**, further validating that CenRA improves performance in terms of the original reward structure.\n\n## Weakness 5\n\nRegarding the concern that CRA may become a bottleneck in large-scale settings:\n\n1. **Benefit from large-scale MTRL**: As the number of tasks increases, the CRA gains a greater advantage in capturing task correlations and extracting shared knowledge. The global perspective allows it to better model cross-task commonalities and provide more informative shaped rewards. That said, we acknowledge that in extremely large-scale or highly heterogeneous task settings, the CRA’s capacity could become a limiting factor. Addressing this scalability challenge could involve several strategies: \n- **Modular reward agents**: Dividing the CRA into multiple specialized agents, each focusing on a subset of tasks or task clusters.\n- **Task clustering**: Grouping similar tasks and allowing the CRA to operate at a higher level of abstraction, reducing the complexity of the reward generation process.\n2. **On shared memory and data storage overhead**: CRA samples experiences directly from the replay buffers maintained by individual policy agents, without requiring additional memory, no matter how many tasks. However, we recognize that as the number of tasks grows, the cumulative memory of all replay buffers may become significant. In such cases, distributed storage or memory-efficient experience compression strategies could be employed to mitigate the issue.\n\n## Weakness 6\n\nRegarding more comparisons to existing policy distillation and attention-based modular methods, we have conducted further comparisons: (1) **TD-MPC2** (Hansen, Nicklas et al., ICLR 2024); (2) **CMTA** (Lan, Siming et al., NeurIPS 2023); (3) **Shared-Critic (SC)** (Zhang, Gengzhi et al., ACML 2021);  (4) **Distral** (Yee Whye Teh et al., NeurIPS 2017). In which *Distral* is a representative policy distillation method, while *CMTA* is a recent attention-based method. The results are shown below:\n\n|Algorithm|*ML10-sparse*|*ML50-sparse*|*2DMaze*|*3DPickup*|*MujocoCar*|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|CenRA (ours)|0.88±0.12|0.76±0.03|0.91±0.02|0.88±0.06|514.88±0.68|\n|TD-MPC2|0.82±0.09|0.61±0.03|0.88±0.05|0.71±0.05|505.34±0.71|\n|CMTA|0.79±0.08|0.60±0.03|0.75±0.04|0.70±0.04|480.19±0.62|\n|SC w/o sampling|0.56±0.06|0.35±0.02|0.80±0.05|0.69±0.04|400.25±0.52|\n|SC w/ sampling|0.60±0.06|0.41±0.03|0.73±0.05|0.53±0.05|430.18±0.57|\n|Distral|0.51±0.07|0.41±0.02|0.66±0.06|0.66±0.03|380.10±0.50|\n\nThe additional results further validate the effectiveness of CenRA, which consistently achieves strong or superior performance across all domains. The new results will be included in the revised manuscript.\n\n## Question 3\n\nRegarding CenRA's performance in continual learning or lifelong RL settings, we want to highlight that they can be viewed as a special case of our CenRA framework, where tasks arrive sequentially rather than concurrently. CenRA is well-suited to this setting, which is evidenced in our experiments (Section 5.2). The CRA is capable of transferring learned knowledge from previous tasks to unseen ones, and more importantly, CRA can be continuously updated with new tasks. This enables CenRA to effectively handle sequential task settings.\n\n## Question 4\n\nRegarding the potential benefit of integrating meta-learning techniques, we appreciate the reviewer’s insightful suggestion. We agree that meta-learning holds strong potential for enhancing both the adaptability and efficiency of CenRA, e.g., meta-learning could be integrated by:\n* Learning a meta-reward function that adapts more quickly to new tasks with limited data\n* Employing meta-initialization strategies (e.g., MAML-style) to speed up policy agent learning on new tasks\n* Introducing a task encoder to learn latent task embeddings, which could inform both reward generation and sampling strategies.\n\nWe believe integrating meta-learning is a promising future research direction.\n\nOnce again, thank you for your comments. We hope our responses have addressed your concerns."}]

Author rebuttal:

Meta-review: This paper proposes CenRA, a centralized reward agent framework for multi-task reinforcement learning. By decoupling reward generation from policy optimization, the method enables knowledge transfer through dense shaped rewards, addressing the challenge of sparse rewards and accelerating learning across tasks. Experiments on benchmarks demonstrate improvements over baselines and robustness in transferring to unseen tasks. The reviewers highlighted several strengths: the conceptual clarity of the framework and its empirical validation with extensive baselines. Overall, the consensus is positive. I recommend Accept.
Decision: Accept (poster)