ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Authors: Myungsik Cho
Jongeui Park
Jeonghye Kim
Youngchul Sung
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
multi-task reinforcement learning
reward scaling
Primary area: reinforcement_learning->deep_rl
TL;DR: We propose Adaptive Reward Scaling (ARS) for multi-task RL, which balances rewards across tasks and includes periodic network resets to enhance stability and efficiency, outperforming baselines on Meta-World and excelling on complex tasks.

Abstract: Multi-task reinforcement learning (RL) encounters significant challenges due to varying task complexities and their reward distributions from the environment. To address these issues, in this paper, we propose Adaptive Reward Scaling (ARS), a novel framework that dynamically adjusts reward magnitudes and leverages a periodic network reset mechanism. ARS introduces a history-based reward scaling strategy that ensures balanced reward distributions across tasks, enabling stable and efficient training. The reset mechanism complements this approach by mitigating overfitting and ensuring robust convergence. Empirical evaluations on the Meta-World benchmark demonstrate that ARS significantly outperforms baseline methods, achieving superior performance on challenging tasks while maintaining overall learning efficiency. These results validate ARS's effectiveness in tackling diverse multi-task RL problems, paving the way for scalable solutions in complex real-world applications.

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Myungsik Cho Jongeui Park
Jeonghye Kim
Youngchul Sung'
Abstract
Multi-task reinforcement learning (multi-task RL) (Wilson et al., 2007; Zeng al., 2018; Yang et al., 2020; Sun et al., 2022; Hendawy et al., 2024; Cho et al., 2024) enables a sin- gle policy network to handle multiple tasks with shared pa- rameters (Caruana, 1997), improving data efficiency. How- ever, faces the challenge of negative transfer, where learn- ing one task disrupts others, destabilizing training and limit- ing scalability (Sun et al., 2020).
Multi-task reinforcement learning (RL) encoun- ters significant challenges due to varying task complexities and their reward distributions from the environment. To address these issues, in this paper, we propose Adaptive Reward Scal- ing (ARS), a novel framework that dynamically adjusts reward magnitudes and leverages peri- odic network reset mechanism. ARS introduces history-based reward scaling strategy that en- sures balanced reward distributions across tasks, enabling stable and efficient training. The reset mechanism complements this approach by mit- igating overfitting and ensuring robust conver- gence. Empirical evaluations on the Mela-World benchmark demonstrate that ARS significantly outperforms baseline methods, achieving supe- rior performance on challenging tasks while main- taining overall learning efficiency. These results validate ARS's effectiveness in tackling diverse multi-task RL problems, paving the way for seal- able solutions in complex real-world applications.
In addition to negative transfer, this paper introduces an- other critical issue in multi-task RL: the significant variation in reward scales across tasks, which can adversely affect overall performance. To explore this issue, we focus on the impact of reward scaling approach in multi-task scenarios, studied in single-task settings (Henderson et al., 2018; Wu et al., 2018), which involves applying a constant scaling fac- tor rewards. We observed that the varying magnitudes of rewards across tasks cause biases during training, resulting in overfitting on tasks with highly amplified rewards and a decline in average performance across all tasks.
To address these challenges, we propose Adaptive Reward Scaling (ARS), a novel framework that dynamically adjusts reward scaling factors, ensuring balanced reward magni- tudes across diverse tasks. ARS is built upon two core components: (1) adaptive reward scaling that adjusts scaling factors by analyzing the distribution of rewards within each task's experience replay buffer (Mnih et al., 2015), and (2) periodic resetting of network parameters to prevent over- fitting and improve convergence. This approach ensures that challenging tasks receive adequate emphasis on their rewards, preventing overfitting to simpler tasks with higher rewards and enhancing overall performance. Experiments conducted on the Meta- World benchmark (Yu et al., 2019), which includes 50 robotic manipulation tasks, demonstrate that ARS significantly outperforms baseline methods. Addi- tionally, ARS integrates seamlessly into existing off ooolcy algorithms with minimal modifications, making it practical and effective solution for real-works applications.
1. Introduction
In recent years, the field of deep reinforcement learning (RL) has achieved remarkable success in addressing complex con-
trol problems, such as mastering Atari games (Hafner et al.,
2021; Kapturowski al., 2023; Schwarzer et al., 2023) and advancing locomotion control (Haarnoja et al., 2018b; Fujimoto et al., 2018; Cetin et al., 2022). Despite these achievements, the field remains limited by a task-specific paradigm that demands substantial data and computational resources to train separate policies for each task. General- izing single policy perform effectively across multiple tasks poses significant challenge, particularly in domains such robotic control (Kaufmann et al., 2023; Tang al., 2024), where mastering diverse skills is essential.
The primary contributions of this work are:
'School of Electrical Engineering, Korea Advanced Institute of Science and Technology. Daejcon, Korea. Correspondence to: Youngchul Sung <ycsung@ krr>
The introduction of reward scale variation of a chal- lenge in multi-task RL, demonstrating how fixed re- ward scaling can lead to biased training, overfitting, and overall performance degradation
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
A novel framework, adaptive reward scaling mech- anism with reset strategies, which robustly handles varying reward distributions and enhances the learning process for challenging tasks.
as D, it stores transitions (st, ag Its 8(+1) collected from the agent's i interactions, which are reused during training stabilize updates to the policy or value function.
To extend this concept to multi-task RL, each task T in the task set C. is assigned a distinct replay buffer Dj. This design isolates experiences for each task, enabling the agent to adapt effectively the unique dynamics and reward structures of individual tasks.
Empirical validation on the Seta-World benchmark Empirical, demonstrating superior performance and providing detailed insights into the mechanisms driving ARS's effectiveness.
2.3. Reward Scaling
2. Preliminaries
Reward scaling (Henderson al., 2018; Wu et al., 2018) is a common preprocessing technique in RL that adjusts reward magnitudes to enhance learning stability and convergence. In single-task RL, the reward signal is transformed using scaling factor cPew, helping to mitigate the impact of incon- sistent reward magnitudes and enabling more stable updates to the value function or policy.
2.1. Multi-Task Reinforcement Learning
RL. problem typically represented as Markov decision process (MDP), defined as .M- = (S, Prr,, r. p. 7). Here, S represents the state space, A is the action space, P S x A x S + g+ is the transition probability, r S A - IR is the reward function, p : S + R+ is the distribution over initial states, and 7 E [0, is the discount factor. At each time step t, the agent receives an observation of the state 8 E S and selects an action at E A according to the policy (a; st). The environment returns reward r. rste, ae) and transitions to the next state $1+1 according to the transition probability distribution P(s++1 8t, a.). In single-task RI, the goal is to maximize the expected sum of discounted rewards:
When applied multi-task RL, the need for reward scal- ing becomes even more pronounced due to the variation in reward distributions across tasks. To address these differ- ences, task-specific reward scaling is implemented, where each task's rewards are adjusted using unique scaling fac- tors {et J This approach ensures consistency in reward magnitudes across tasks, supporting stable learning.
H
2.4. Parameter Resetting
((T) ETNOT
In single-task deep RL, early learning often leads to overfit- ting to initial experiences, a phenomenon known as primacy bias, as highlighted by Nikishin et al. (2022). To mitigate this, Nikishin al. (2022) proposed a method that periodi- cally resets the parameters of the RL agent while preserving the replay buffer. This method leverages the replay buffer's stored experiences to enable the agent to recover quickly from the reset, bypassing the primacy bias. Although resets temporarily reduce performance, the agent rapidly regains its capabilities by focusing on high-quality trajectories ac- quired later in the learning process, resulting in significant improvements across various environments.
In multi-task RL, the focus shifts to optimizing a policy that achieves high performance across a broad range of tasks. Specifically, multi-task RL contains a set of tasks C = [ and a distribution p(T) over these tasks, where each task To is represented by an MDP M, = (S,A, Pi, ri, Ps: 7; H). The tasks share common state and action spaces but differ in reward functions, transition dy- namies, and initial state distributions. Assuming p(T) is uni- form, the goal is to learn shared policy that maximizes the average return over all tasks. Formally, the objective is:
max (T,, T)
In multi-task deep RL, resetting deep networks offers addi- tional benefits. Beyond counteracting primacy bias, it also addresses biases toward tasks learned early in training. By redistributing focus across tasks, the reset mechanism pro- motes more balanced learning and enhances performance across the entire task set (Cho al., 2024).
where (TT, T) denotes the expected sum of discounted re- wards for task T under policy и.
In this work, we use a shared policy Ta(als, 2), which in- corporates a task representation alongside the state s. We represent each task using a one-hot encoding. To train the policy, we adopt Soft Actor-Critt (SAC) (Haarnoja et al., 2018b) as the underlying algorithm.
3. Motivation: Reward Scaling in Deep RL
This section introduces a motivating example highlighting the limitations of conventional deep multi-task RL, which assigns an equal reward scale (ceTH = 110) to all target tasks. The challenge becomes particularly evident when the
2.2. Replay Buffer
In policy RL, the replay buffer (Mnih et al., 2015) im- proves sample efficiency and training stability. Denoted
2

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
SAC-MI
SAC-MT
SAC-NT (rs x100 Challening Trisk)
0.8
Success Ratio
0.6
Time Steps
Thmee Stops
(a) push
(b) pick-place
0.4
0.2
0.0
Task ID
Figure 2. Comparison the final success rate per task the MT10 benchmark among SAC-MT, SAC-MT with reward scaling applied to all tasks, and SAC-MT with reward scaling applied only to the challenging tasks: 'push," "pick-place, and 'peg-insert-side.
(c) -hace-
(c) basketball
Figure 1. Leariing curves of four tasks: 'push,' 'pick-place', 'shelf- place' 'basketball'. With reward scaling factor of 100, tasks succeed, whereas without reward scaling, all tasks fail.
identification. Figure 2 (blue) shows the final success ratio per task with SAC-MT. Notably, tasks with IDs 1, 2, and 7 ("push,' pick-place," and 'peg-insert-side failed train successfully.
target tasks have significantly different reward distributions.
3.1. Reward Scaling in Single-Task RL
In order to investigate the failure in these challenging tasks with the SAC-MT algorithm, we focused on the magnitude of the initial reward for each task in the MT10 benchmark. We observed that the reward scale for certain challenging tasks is substantially lower compared to others, differing by factor of 10 100. This imbalance negatively affected the training of the Q-network for these tasks, shown in Figure 3. Specifically, the Q-values for these tasks were neg- ative, even though the rewards from BeradWorld are always positive. This highlights the challenges of training the Q- network when reward scales are not sufficiently high. More learning curves of Q-value are provided in Appendix D.1
Previous works (Henderson al., 2018; Wu al., 2018) demonstrate the effectiveness of reward scaling in single- task RI. To investigate its impact within the Meta- Meta-World benchmark (Yu et al., 2019), a widely used environment for evaluating multi-task RL, we conducted independent ex- periments on four challenging tasks- "asss-'puhh." fück-place" sthelf-place and 'basketball', These experiments utilized the Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2018b), comparing performance across reward seal- ing factors-(1.0, 100.0), where reward scaling factor of 1.0 represents the naive SAC method.
As shown in Figure 1, reward scaling method outperforms the non-scaling approach and ensures successful task com- pletion, demonstrating its effectiveness in single-task RL.
3.2.2. FIXED REWARD SCALING MULTI-TASK RL
To further investigate the effects of reward scaling in multi- task RL, conducted additional two experiments with SAC-MT. In the first, reward scaling factor of 100 was applied uniformly across all tasks, while in the second, the scaling was selectively applied only to the challenging locks-'push? 'pick-place," and "peg-insert-side" -levvnng the other tasks unscaled. Figure shows the final success ratios achieved with SAC-MT when incorporating reward scaling. Orange represents the scenario where scaling was applied to all tasks, and green denotes the selective scaling approach. In the uniform scaling experiment, Task ID I showed progress, but Task IDs 2 and remained unresolved. In contrast, with selective scaling applied only to challeng- ing tasks, Task IDs 1, 2, and 7 were successfully trained.
3.2. Challenges of Reward Scale Multi-Task RL
In the previous subsection, we observed that the reward scal- ing approach effectively supports the training of successful policies in Meta-World. Here, we examine the impact of the reward scaling method multi-task RI. agent.
3.2. 1. FAILURES IN CHALLENGING TASKS WITHOUT THE REWARD SCALING APPROACH
We considered the MT10 benchmark, consisting of 10 ma- nipulation tasks in Meta- Meta-World, and trained the agent with the SAC-MT algorithm, using one-hot encoding for task
3

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Algorithm Adaptive Reward Scaling (ARS)
1: Initialize policy network Te, Q-value network Q2
valuvss
2: Initialize replay buffer Dj for each task Ti C
3: for t 1,2, ...., .., Timit do
0
4: for all T, E do
5: Interact with the environment of T. with a random policy and store data DD end for end for
Time Stops
Time Stops
(a) push
(b) pick-place
Figure Learning curves Q-values for two tasks: 'push," and "pick-place".
8: Initialize the reward scaling factors (ctT))) Ji=1 using (6)
9: for t Timit + 1, Tini + 2,..., do
10: for all Tj E C do
11: Interact with the environment of T5 with my and
However, this approach caused many other tasks to fail during training, reducing the overall average performance.
store data in D;
12:
end for
This observation highlights a key challenge in multi-tassk RL: careful reward scaling is essential for handling tasks with diverse reward distributions. Improper scaling can result in overfitting on tasks with highly amplified rewards, failure to learn others, and overall performance decline. To address this challenge, the remainder of this paper introduces novel approach effectively adjust reward scaling in multi-task RL, aiming to achieve better overall performance.
13: Update 0 and 1/, using the data in (D:) and the
scaling factors {c Jiml
14:
if I % Treset = 0 then
15:
Update {ct using (6)
16:
Randomly reinitialize and e
17:
end
18:
end for
policy distribution To and state-action value function Q Each task T; in the task set C is assigned a separate buffer D;, maintained independently to store task-specific interactions. The goal is to maximize the sum of each task's objective, i.e., ΣTiec J(x0, Tı). The key components of ARS are further detailed in the following subsections.
4. Adaptive Reward Scaling for Multi-Tass Reinforcement Learning
To address the issue raised in the previous section regarding the effective handling of varying reward distributions in multi-task RL, we propose the Adaptive Reward Scaling (ARS) algorithm. This approach is specifically designed to address imbalances in reward scales across tasks in multi- task RL, thereby improving overall learning efficiency. The ARS framework comprises two key components: a reset mechanism and dynamic reward scaling strategy, which work together to optimize training performance.
4.1. History-Based Reward Scaling Strategy
From the observations in the previous section, assigning higher reward scaling factors to complex tasks is necessary. However, using fixed reward scaling approach is inade- quate as shown in Figure 2, as the reward magnitude for tasks with high scaling factor progressively increases dur- ing training, leading to biases toward those tasks. To address this, the reward for each task T, should be adaptively scaled.
History-Basee Based Reward Scaling Strategy: At the core of ARS is reward scaling strategy that adjusts ward scales for tasks with lower reward magnitudes, ensuring uniform reward scaling across tasks. This ap- proach leverages a metric derived from the distribution of rewards within each task's 's experience replay buffer, enabling real-time and adaptive reward scaling.
This raises the question: "How should the scaling factor for each task T; be determined in real-time?" To address this, examine the training objective in standard off off-policy multi- task RL implementations. Given task set C = [ and the corresponding experience replay buffers [D.) the agent is trained using the following objectives:
Reset Mechanism: ARS integrates reset mechanism to mitigate biases toward the tasks with highly ampli- fied rewards. Through periodic reinitialization of net- work parameters while maintaining the replay buffer, this approach enhances adaptability and promotes im- proved performance across diverse tasks.
(0)
E,ND (0=(0;)))
(3)
CC
T.EC
(Q())
Bre.a)DD
[ (
(4)
C
T.EC
The overall structure of the proposed ARS framework is presented in Appendix F. ARS employs two networks: a
where f. (0; 8) and (1); a, , s') are the per-sample actor and critic loss functions, respectively.

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
In this objective, the reward scaling factor only influences critic training. For the SAC algorithm, the per-sample critic loss for (s, E D; is expressed as
are updated, while retaining all historical data in the re- play buffers. This approach maintains stable critic learn- ing throughout training, effectively balancing performance across a diverse set of tasks. The main contribution of our work is the novel reward scaling framework, thus any type of multi-task RL method can be used for practical imple- mentation. The overall process of the proposed method summarized in Algorithm 1.
E (V11,,, (I (1+ ((Q0((',mmss')) - a log (s')) (s,
(5)
Critic training, therefore, can be viewed as regression problem, where the targets vary, and the mean reward in the experience replay buffer D; plays a crucial role in determin- ing the magnitude of the critic values.
4.3. Enhancements
Our ARS framework builds upon the SAC-MT algorithm, simplest baseline for multi-task RL. In the main results, we refer to this combination as ARS. Beyond the default ARS setup, we apply layer normalization (Ba et al., 2016), well-known technique for stabilizing learning in deep RL (Ball et al., 2023), to both the input and hidden layers of the critic network, and denote this variant as ARS-LN, Building on ARS-LN, we further incorporate Low-Rank Adaptation (LoRA) (Hu et al., 2022) introduce task-specific param- eterization during the later stages of Saainiinn--seccccallllly. after 75% and 83.3% of the total training steps for MT10 and MT50, respectively. We denote this variant as ARS-LoRA. For efficient adaptation, we use rank of r = 8 for MT10 and r = 16 for MT50. Further implementation details re- garding the integration of LoRA into ARS are provided in Appendix B.
Building on these observations, we define the reward scaling factor e.ew for each task Ti as the following equalizer rule:
max (fr1..... FNl)
new
(6)
Fi
where F; is the mean reward for task T; within its replay buffer D, for i 1,.... .N. This formulation ensures consistent reward magnitudes across tasks, enabling sta- ble critic learning without bias towards easy tasks. Using this adaptive reward scaling factor c,th, the critic loss for (s, a, s') 5's E Dj is modified as follows:
e (1b; (44Hh r ((Q - a log Tog')())) - -Qp(s,))
5. Experiments
Benchmarks. To evaluate the effectiveness of the pro- posed method on various tasks, we conducted experiments using the Meta- Woold benchmark (Yu et al., 2019), which includes 50 distinct robotic control tasks involving a Sawyer arm in the MuJoCo environment (Todorov et al., 2012). Our experiments used two setups: MT10 and MT50, which con- sist of 10 and 50 manipulation tasks, respectively. A detailed description of the benchmarks are provided in Appendix A.
4.2. Reset Mechanism
A key issue with the naive adaptive reward scaling approach is that the rewards stored in the experience replay buffer fluc- tuate throughout training due to the changing reward scaling factor. This variability can destabilize critic training, as the regression target in Equation (7) changes frequently. Addi- tionally, it may introduce biases toward tasks with highly amplified reward scaling factors, ultimately leading to sub- optimal performance.
Baselines. We compared the proposed method against the following baseline approaches: 1) SAC with Multi-Task (SAC-MT): A shared policy that uses a one-hot task iden- tification encoding along with the current state as input. 2) SAC with Multi-Task Multi-Head (SAC-MT-MH) (Yu et al., 2019): Similar to SAC-MT but incorporates an inde- pendent final layer (multi-head) in the policy network for each task. 3) SAC with Soft Modularization (SAC-Soft- Modular) (Yang et al., 2020): A policy architecture utilizing multiple modules with a soft modularization technique that determines a routing strategy for each task. 4) Gradient Surgery for Multi-Task Learning (PCGrad) (Yu et al., 2020): This method addresses conflicting gradients in multi-task learning by projecting gradients onto a shared subspace. 5) Parameter-Comppsitiona Multi-Task Reinforcement Learn- ing (PaCo)(Sun et al., 2022): multi-task RL approach
To address this issue, we adopt the reset mechanism pro- posed by Nikishin et al. (2022), which involves resetting the policy parameters 0 and Q-value function parameters th while preserving the experience replay buffers. Furthermore, we adjust the reward scaling factor ciTh only prior to resets, ensuring stable critic training throughout the process.
To conclude, our 'History-Based Reward Scaling' strategy adaptively scales rewards across tasks of varying complex- ities, preventing bias toward simpler tasks with larger raw magnitude of rewards. However, dynamically changing reward scales can destabilize training critic network. To address this, we employ a Reset Mechanism, which resets the policy and critic networks whenever the reward scales
5

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Table Comparison of average and per-task success rates (%) on the Meta-World MT10 benchmark. Refer to Appendix A for task names corresponding to each task ID.
Task ID
Algorithm
1
2
3
7
8
Average
SAC-MT
98+22.4
0++0.0
0+:0.0
100+-0.0
1000+0.0
100-00.0
96+-2.0
0=0.0
100+00.0
100++0.0
69.4 + 0.8
SAC-MT-MII
100
28 281.118
0.10.0
100
98.12.5
100
100
46
100.10.0
100.1..0
77.2 11,9
Soft Modular
100.10.0
32.114.9
01:.0
100
100:10.0
1.0
1.0
12.
100.10.0
100.10.0
74.4 10.5
PCGrad
9.7
0+:0.0
0.:0.0
100
100:
1.0
100
54:t: 39.9
100:00.0
1.0+
74.8 13.7
PaCo
100+-0.0
44+-25.2
0+00.0
100+-0.0
100++0.0
100000.0
1.0=0000
80 + 40
100+00.0
100+00.0
82.4 = 14.2
MOORE
100
92.5 LL 7.1
0.1..00
100
100
100
100
67.5 46.5
100.10.0
100.10.0
86.0 T 4.8
SMT
96.13.7
62.117.9
13.8
100:10.0
100:10.0
1.0
100:10.0
76.:1.9
100.1..00
100.10.0
86.8 1 8.6
ARS (Ours)
91.7
3.5
15.
98.8::3.5
100
1.0
1.0
96.3:17744
100:00.0
100:
97.3 +: 2.1
ARS-LN (Ours)
93.8+7.4
98.8+33.5
93.8++9.2
100-+0.0
100-
100-00.0
100
100::0.0
100+00.0
100+00.0
98.6 + 1.5
ARS-LORA (Ours)
100+-0.0
97.5+7.1
98.8+3.5
100=:0.0
100:=0.0
100=00.0
100==0.0
98.8++3.5
100++0.0
100++0.0
99.5 + 0.8
that utilizes parameter compositionality by constructing task-specific parameters from a common pool of shared base parameters. 6) Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts (MOORE) (Hendawy et al., 2024): Neural network architecture utilizing mixture- of-experts with gram-schmidt process 7) Scheduled Multi- Task Training for Multi-Task RL (SMT) (Cho et al., 2024): A framework designed to mitigate simplicity bias using a "Hard Tasks First" scheduling scheme combined with a reset mechanism.
Table 2. Comparison of average bottom-k - success rates on the Mcta Mcta-World MTS0 benchmark.
Average Bottom-& Success Ratio (%)
Algorithm
20
40
SAC-MT
0.0 0.0
0.0 + 0.0
14.1 3.11
3..5
47.6 + 2.0
SAC-MT-ME
0.0 0.0
0,0 0.0
1..1
340020
47.2 = 1.7
Soft Modular
0.000.00
18
23.7 12.3
54.1 42.6.99
7.6
PCGrad
0.0:1:000
0.0 0.0
21.0
12.9
39,9
10.7
51.9
$1.9188.5
PaCo
0.0
4.6
26.1 15.0
44.6
11.2
55.6 9.1
MOORE
0,0
15.9
4..2
35
6.12
SMT
0.0
8.
26.8
13.1
56.0 9.
8.0
Training settings. All methods are trained using 20 million samples in the MT10 benchmark and 100 million samples in the MT50 benchmark. Policy evaluation is based on the success ratio across all tasks, where the success ratio for a specific task is determined by averaging its success rate over 10 episodes with different sampled goals. We report the mean performance along with the standard deviations of policies trained across different random seeds, as summa- rized in Tables and 2. Details of the hyperparameters used in the experiments can be found in Appendix C.
ARS (Ours)
26.3 9.6
6.1
45.3
57.5 3,6
65.9 +: 2.9
A (Ours)
21.0 11.4
49.1 6.5
64.3
4..9
78.3 + 3.7
oRA (Ours)
29.3 7.7
58.4 4.5
7..0++3.0
79.0 +: 2.2
83.2 + 1.8
success rate, successfully solving all tasks with least 97% accuracy. This marks significant advancement over exist- ing methods and a major breakthrough as the first to solve the MT10 benchmark using scratch multi-task RL train- ing. Notably, ARS framework performs exceptionally well on challenging tasks, such as task ID ('push"), ), 2 ('pick- place') ), and 7 effectiveness of ARS's s adaptive reward scaling combined with the reset mechanism, which emphasizes addressing dif- ficult tasks and accelerating progress during the early stages of training.
5.1. Results Meta-World
) These results highlight the
The results for two benchmarks, MT10 and MT50, are pre- sented in Table 1 and Table 2, respectively. These findings highlight the effectiveness of the ARS algorithm's 's inno- vative strategy for multi-task RL, especially its adaptive reward scaling and reset mechanism. Together, these com- ponents drive notable performance improvements across a wide range of tasks in the Stra-World benchmark.
Results on MT50. The MT50 results in Table 2 high- light the consistent effectiveness of ARS, which outper- forms other methods, particularly in the average bottom-k success ratios, suggested by Cho et al. (2024). In partic- ular, in the most challenging subset of tasks (bottom- 10), where all baseline methods fail, ARS variants achieve av- erage success rates of 9.1%, 21.0%, and 29.3%, respec- demoosstttttttinnngggg a clear advantage on difficult tasks. Moreover, the use of Layer Normalization and LoRA fine- tuning remarkably improves performance from 65.9% to
Results MT10. As shown in the MT10 results in Table 1, our ARS framework with three variants achieves outstanding performance, attaining average success rates of 97.3%, 98.6%, and 99.5%, respectively, which are the highest among all compared methods. Furthermore, ARS- LoRA achieves the best overall performance with a 99.5%
6

---

ARS: Adaptive Reward Scaling for Malk Reinforcement Learning
Table Results of incorporating the ARS framework into existing
Effective Solvable Task Ratio (ESTR)
MT
multi task RL methods.
Soft Modular
PCGrad
Multi Task Algorithm
0.8
MOORE
Benchmark
ARS
SAC-MT
PCGrad
Soft Modular
MOORF
ARS
MTI0
W/ ARS
97.3+221
95.9442.6
1,.
08.0+-0.4
0.6
ARSHUN
w/o ARS
69.4110.8
74.81:3.7
74.41:10.5
86.0.14.8
ARSILORA
MTS0
W/ ARS
65.9.12.9
72.4.12.6
75.5.13.6
84.6.1..6
0.4
W/o ARS
47.6+2.0
$1.98855
54.1
64.2+3.11
0.2
78.3% and 83.2%, respectively, in terms of average success rate across all tasks. This marks significant advancement over existing methods and represents a major milestone as the first instance of successfully solving diverse tasks in the MT50 benchmark through scratch multi-task RL training. Note that while the performance of MOORE (Hendawy al., 2024) appears competitive with our basic ARS's per- formance, this is not fair comparison, the MOORE method uses approximately 4x more parameters than ours. To examine the effects of network capacity, we conducted an ablation study on the number of parameters in Section 5.4. With a bigger network, ARS-LN achieves 88.9% % average success rate for the MT50 benchmark. Notably, ARS-LN successfully solves all tasks with a success rate of at least 45%, except for two tasks: assembly and disassemble, re- sulting in average success rate of 92.3% across 48 tasks.
0.5 Threshold (6)
0.3
Figure 4. Comparison the Effective Solvable Task Ratio (ESTR) with respect the threshold ó in the MT50 benchmark.
5.2. Analysis of the Reward Scaling Approach
In this section, assess the effectiveness of the proposed reward scaling method by analyzing changes in the reward scaling factor ct for each task Tú. Figure 5 shows the learning curve of the reward scale factors (cTT)) on a loga- rithmic scale for the MT10 benchmark. Initially, the agent assigns higher scaling factors to three difficult tasks includ- ing 'push,' 'pick-place, and 'peg-insert-sice due their low initial reward magnitudes. Over time, these factors gradually become more uniform, shown in Figure 5. This indicates that as tasks with high scaling factors are suc- cessfully learned, their mean rewards increase, leading to reduction in the scaling factor.
To further investigate the advantages of mastering diverse tasks, we analyze the Effective Solvable Task Ratio (ESTR) on the MT50 benchmark. The ESTR is defined as:
(Te C EvvalFinalSR(T) 2 267 EST
N
5.3. Integration of the ARS Framework with Off-Policy Multi-Task RL Methods
where 5 is the success ratio threshold, and EvalFinalSR represents the final average success ratio for a task T, A higher ESTR indicates that the agent successfully solves more tasks. We compute the ESTR for threshold values 8 e (0.1,0.3 0.5, 0.7). The threshold 8 specifies the criteria for task to be considered solvable, with tasks achieving an average success ratio equal to above ó classified as solvable. As increases, the criteria become stricter, causing the ESTR metric to decrease accordingly.
The proposed ARS framework is a universal solution that can be applied to any off-policy multi-task reinforcement learning (RL) method. To evaluate its applicability, we incorporated the dynamic reward scaling and reset mecha- nisms of ARS into various off-policy multi-task approaches, including PCGrad, Soft Modular, and MOORE.
Table presents the performance of these algorithms com- bined with the ARS framework (training curves are provided in Appendix D.3). The results demonstrate that the ARS framework consistently improves all evaluated multi-task RL algorithms on both the MTI0 and MT50 benchmarks, highlighting its broad compatibility and effectiveness. In particular, the integration of ARS boosts the average success rate by at least 20% across all off-policy MTRL methods on MT50. Notably, MOORE (Hendawy et al., 2024) inte- grated with ARS achieves the highest average success rate of 84.6% on MT50 benchmark, performance level never
As shown in the ESTR results Figure 4, our ARS framework demonstrates exceptional performance across all threshold values 8 E {0.1, 0.3, 0.5, 0.7), indicating that ARS successfully solves more tasks than other baselines. Notably, ARS significantly outperforms competing methods as å decreases, highlighting its ability to address a broader range of tasks, including those with lower success ratios. In contrast, the baseline methods tend to focus on only a subset of tasks, failing to tackle more challenging ones effectively.

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Table Ablation study on the components of ARS.
reach
Algorithm
Reward Scale Factor (log)
2.0
push
plok-ppace
Task set
ARS w/o reset
ARS w/ reward scaling
ARS
doer open
drawer--pen
Causy
85.9:111.
10010.0
99.6:1
1.5
drawen dese
button prezs topdown
Camouli
15.1
0.8
91.7-17.3
peg- insert---d-de
window
C
69.8 -18.1
70.3 - 0.5
97.3:12.11
1.0
wirdow-ccoo
Table S. Ablation study on the reward scaling method.
Reward Scaling Method
Task
Normalization
Stundard Deviation
Meun Reward
0.25
0.50
1.00
1.25
1.75
2.00
time_ Step
1e7
Cus
85.9 26.0
99.11:2.0
99.611
Camica
75.0 = 33.3
82.9+22.0
91.7:77.3
Figure 5. Learning curve of the reward scale factor otth on a loga- rithmic scale for each task Ti within the MT10 benchmark. Com- plex tasks such as "pack-place," 'push,' and 'peg- inserrstt--sde have larger reward scaling factors compared others.
82.4 = 27.9
94.3+7.1
97.3++2,1
respectively. Table 5 presents the average success ratio for each scaling metric. The results show that scaled-up based on the mean rewards consistently outperforms other approaches, highlighting the effectiveness of our method.
before reached by other multi-task RL approaches.
Finally, we conducted an ablation study to investigate the effect of network capacity. While the MOORE (Hendawy et al., 2024) model integrated with ARS achieves the highest performance the MT50 benchmark among multi-task methods, a key contributing factor is its larger network size. Each network in MOORE contains approximately 2.0M parameters, whereas our main result in Table uses only 0.5M. To assess the impact of model size, we increased the hidden unit size in our architecture to 512, 800, and 1024. The configuration with 800 hidden units has parameter count comparable to that of the MOORE model. As shown in Table 6, the performance of ARS improves consistently with increased capacity, whereas SAC-MT (w/o ARS) does not. This highlights ARS as a scalable and effective method for leveraging larger models through reset and dynamic reward scaling. Remarkably, ARS-LN with 1024 hidden units achieves the highest average success rate of 88.9% across all evaluated MTRL methods. Furthermore, despite having similar network capacity, ARS-LN with 800 hidden units outperforms the MOORE model integrated with ARS. This performance gain demonstrates the effectiveness of the layer normalization technique, even though its adoption is significantly simpler compared to the complex model design of MOORE, Additional ablation studies are provided in Appendix E.
5.4. Ablation Studies
First, We conducted ablation studies evaluate the key com- ponents of ARS: (I) the adaptive reward scaling scheme and (2) the reset mechanism. To assess their contributions, we performed experiments where each component was omitted individually, denoted using the notation "w/o." Specifically, the' ARS w/o reward scaling' corresponds to the naive re- setting framework proposed by (Nikishin al., 2022).
Table 4 shows the average success ratio for the task set configurations in the MT10 benchmark (training curves pro- vided in Appendix D.2)). Here, Cenry represents the set of seven tasks with the highest success ratios, while the remain- ing tasks constitute the difficult set, Caitiiutt Removing the reset mechanism or reward scaling results in performance drops of 27.5% and 27.0%, respectively. The absence of the reset mechanism significantly reduces the performance of the easy task set, underscoring its importance for stable train- ing, On the other hand, reward scaling notably enhances the performance of the difficult task set, demonstrating its effectiveness in addressing varying reward distributions and ensuring balanced learning across tasks. These observa- tions highlight the significance of both components in the effectiveness of the ARS framework.
Next, to verify the effectiveness of the proposed reward scaling method, we conducted experiments using alternative scaling approaches including mean reward, standard devia- tion reward, and normalization. The scaled rewards based on standard deviation (f?) and normalization (Gnermal) for each task T. are defined as p. r and pnommal r-F. o(rr)
6. Related Works
Multi-task RL. Multi-task learning has become a key area in machine learning, focusing on algorithms that perform well across various tasks. In the context of RL, it aims
a(r.)*
8

---

ARS: Adaptive Reward Scaling for Malti-Task Reinforcement Learning
Table 6. Ablation study on the network capacity.
Network Capacity: Hidden Layer Dimensions
Algorithm
[400, 400,400000 400, 400]
1512,512,512,512
[800, 800, 800, 800]
[1024, 1024, 1024, 1024]
ARS-LN
78.3
84.2:1:.7
88.0112.9
88.912.66
ARS
65.9
73.6:t4.4
7:.1
78.7::2.9
SAC-MT
47.6++2.0
48.8++2.6
17.2+11.7
45.5+:2.4
to develop models capable of handling diverse tasks (Wil- son et al., 2007; Pinto& & Gupta, 2017; Zeng et al., 2018; Hausman et al., 2018).
ward scaling one of the key factors for stabilizing training: Henderson et al. (2018) demonstrated that SAC and DDPG are sensitive to this choice, while Wu et al. (2018) proposed Automatic Reward Scaling (ANS), which adjusts a global factor based on return statistics. In multi-task RI., PrpArt (Hessel et al., 2019) introduced scale- invariant updates by normalizing critic targets through an adaptive affine trans- formation, enabling training across 57 Atari games with different reward scales. However, PopArt only normalizes the value head and overlooks the broader impact of reward scaling on deep RI. training, which remains critical for solv- ing difficult tasks.
Various approach for Multi-taks RL Addressing neg- ative transfer is crucial for the success of multi-task RL, and various strategies have been developed to tackle this issue. (i) Distillation methods leverage policy distillation to merge knowledge from multiple tasks into a unified model but often require task-specific networks, increasing resource deuands(Rusu et al., 60; Parisotto et al., 2016; Teh et al., 2017). (ii) Modular networks assign distinct modules to tasks with task- besk-specific routing, enabling strategic param- eter sharing to reduce interference and negative transfer (Rusu et al., 2016b; Devin et al., 2017; Andreas et al., 2017: Haarnoja al., 201 Sa; Yang et al., 2020; Sun et al., 2022: Hendawy al., 2024). (iii) Gradient-baid based approaches ana- lyze task gradients to address conflicts but face challenges due to gradient noise and variability (Zhang & Yeung, 2013; Chen et al., 2018; Hu et al., 2019; Yu et al., 2020). (iv) Explicit shaicy-sharing methods share behaviors or policies across tasks to obtain good samples from different tasks, enabling knowledge transfer without sharing parameters (Zhang et al., 2023; He al., 2024). (v) Task scheduling methods use a scheduling framework to prioritize and train more effective tasks earlier in the training process of diverse tasks. (Cho et al., 2024).
Our ARS differs in that it equalizes the mean reward across tasks using a simple, parameter-free rule based on replay buffer statistics (Eq. 6), automatically assigning higher seal- ing factors to harder tasks. As demonstrated in Sections 5. and E.3, this lightweight framework consistently outperforms PopArt-style and other normalization methods on MT10 and MT50.
7. Conclusion
In this work, we introduced Adaptive Reward Scaling (ARS), a novel framework designed to tackle the critical challenges caused by varying reward distributions. By em- ploying history-based reward scaling strategy, ARS dy- namically adjusts reward magnitudes to balance training focus across diverse tasks. The integration of reset mecha- nism further enhances ARS by mitigating biases introduced by early learned tasks, ensuring improved adaptability and convergence. Together, these innovations enable ARS to achieve state-of-the-art performance on challenging bench- marks such as Soold, demonstrating its effectiveness in handling diverse and complex task sets. Future work could extend ARS to other multi-task RL algorithms and more diverse domains, as well as refining its reward seal- ing and reset strategies for greater efficiency. Investigating alternative approaches to task representation and value nor- malization, alongside scaling ARS to multi-agent or hier- archical RL frameworks, presents exciting directions for further research. By addressing key limitations in multi-task RL, ARS contributes significantly to advancing the field, paving the way for more scalable and robust solutions to real-world problems.
In contrast, our method focuses on the reward function. The varying reward distributions make training challenging, so we introduce an adaptive reward scaling scheme enable stable and efficient training in multi-task RL.
Resetting Deep RL Agent In deep RL, primacy bias, or overfitting to carly experiences, has been addressed through periodic resets of agent parameters while retaining the re- play buffer (Nikishin et al., 2022). Extending this concept, D'Oro et al. (2023) Subsequent advancements include reset frequency modulation (D' d'Oro et al., 2023), ensemble-baid based resets (Kim et al., 2023), and refined mechanisms achieving human-level Atari performance (Schwarzer et al., 2023).
Our method extends these ideas to multi-task RL setting, where resets not only address primacy bias but also mitigate biases toward tasks learned earlier training.
Reward Scaling in Deep RL Early studies showed that re-

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Impact Statement
of Machine Learning Research, pp. 2784-2810. PMLR, 2022.
major challenge in reinforcement learning (RL) is gener- alization, where a policy trained for one task often struggles to perform effectively on different tasks. Addressing this issue is crucial for training policies that can handle multi- ple related tasks effectively. In this paper, we introduced novel reward scaling framework for multi-task RL that improves overall performance. Our approach contributes to the advancement of RL in real-world applications requiring the ability to solve multiple similar tasks.
Chen, Z., Badrinarayanan, V., Lee, C., and Rabinovich, A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the 35th International Conference on Machine Learn- ing, ICML 2018, Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 793-802. PMLR, 2018.
Cho, M., Park, J., Lee, S., and Sung, Y. Hard tasks first: Multi-task reinforcement learning through task schedul- ing. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Acknowledgments
This work was supported in part by Institute of Informa- tion & Communications Technology Planning & Evalua- tion (ITPP) grant funded by the Korea government (MSIT) NN..RS-0022-1220111 Development of Artificial Intelli- gence Technology for lelf-Improving Competency-Aware Learning Capabilities, 50%) and in part by the Institute of Information & Communications Technology Planning & Evaluation (ITP) grant funded by the Korea govern- Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones, 50%)
Devin, C., Gupta, A., Darrell, T., Abbeel, P., and Levine, S. Learning modular neural network policies for multi- task and multi-robot transfer. In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29 June 3, 2017, pp. 2169- 2176. IEEE, 2017.
ment (MSIT) Ww..RS2222-112222
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P., Belle- mare, M. G., and Courville, A. C. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
References
Andreas, J., Klein, D., and Levine, S. Modular multitask reinforcement learning with policy sketches. In Proceed- ings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 Au- gust 2017, volume 70 of Proceedings of Machine Learn- ing Research, pp. 166-175. PMLR, 2017.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Ma- chine Learning, ICML 2018, Stockholmsmässan, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Pro- ceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018
Ba, L. Kiros, J. R., and Hinton, G. E. Layer normalization. CoRR, atbs/660..0ml
2016.
Haarnoja, T., Pong. V., Zhou, A., Dalal, M., Abbeel, P., and Levine, S. Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pp. 6244-6251. IEEE, 2018a.
Ball, P. J., Smith, L. M., Kostrikov, I., and Levine, S. Effi cient online reinforcement learning with offline data. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Ma- chine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 1577-1594. PMLR, 2023.
Haamoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learn- ing, ICMI, 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018b.
Caruana, R. Multitask learning. Mach. Learn., 28(1):41-75, 1997.
Cetin, E., Ball, P. J., Roberts, S. J., and Çeliktutan, O. Sta- bilizing off-policy deep reinforcement learning from pix- cls. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C.,, Niu, G., and Sabato. S. (eds.), International Confer- ence on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings
Hafner, D., Lillicrap, T. P., Norouzi, M., and Ba, J. Master- ing atari with discrete world models. In 9th International Conference on Learning Representations, ICLR 2021,
10

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Hawaii, USA, January 27 February 1, 2019, pp. pp. 3812- 3821. AAAT Press, 2019.
Hausman, K., Springenberg, J. T., Wang, Z.,, Heess, N., and Riedmiller, M. A. Learning an embedding space for transferable robot skills. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3. 2018, Conference Track Pro- ceedings. OpenReview.net, 2018.
Kapturowski, S., Campos, V., Jiang, R., Rakicevic, N., van Hasselt, H., Blundell, C., and Badia, A. P. Human-level atari 200x faster. The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., and Scaramuzza, D. Champion- level drone racing using deep reinforcement learn- ing. Nat., 620(7976):982 287822977777 2023. doi: 10.1038/ $41586-023-06411944
He, J., Li, K., Zang, Y., Fu, H., Fu, Q.. Xing, J., and Cheng, J. Efficient multi-task reinforcement learning with cross- task policy guidance. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024.
Hendawy, A., Peters, J., and Eramo, C. Multi-task rein- forcement learning with mixture of orthogonal experts. In The Twelfth International Conference on Learning Repre- sentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Kim, W., Shin, Y., Park, J., and Sung, Y. Sample-efficient and safe deep reinforcement learning via reset deep en- semble agents. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10-16, 2023, 2023.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that mat- ters. Mcllraith, A. and Weinberger, K. Q. (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Ap- plications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3207-3214. AAAI Press, 2018. doi: N1.1609/AAAAV1322111 V3211.11694, 1664.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Confer- ence Track Proceedings, 2015.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fid- jeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H.,, Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nat., (11(5515)4) 2015.
Hessel, M., Soyer, H., Espeholt, L., Czarnecki, W., Schmitt, S., and van Hasselt, H. Multi-task deep reinforcement learning with popart. In The Third AAAI Confer- ence on Artificial Intelligence, AAAI 2019, The Thirty- First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 Febru- ary 2019, pp. 3796-3803, AAAI Press, 2019. doi: 10.1609/AAAII 03300AAAAAL33000..333333777
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P., and Courville, A. C. The primacy bias in deep reinforcement learning. In International Conference on Machine Learn- ing. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. (6828-16844 PMLR, 2022.
Parisotto, E., Ba, L. J., and Salakhutdinov, R. Actor-mimic: Deep multitask and transfer reinforcement learning. In Learning Representa-
Hu, F. J., Shen, Y., Wallis, P., Allen-Zhu, 7., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
4th International Conference tions, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Pinto, L. and Gupta, A. Learning to push by grasping: Using multiple tasks for effective learning, In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29 June 3, 2017, pp. 2161-2168. IEEE, 2017.
Hu, H., Dey, D., Hebert, M.. and Bagnell, J. A. Learn- ing anytime predictions in neural networks via adaptive loss balancing. In The Phirty-Third AAAI Conference Artificial Intelligence, AAAI 2019, The Thirty-Fiis First Inno- vative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intellignnce EAAI 2019, Honolulu,
Rusu, A. A., Colmenarejo. S. G., Gülçehre, Ç., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukeuoglu,

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
K., and Hadsell, R. Policy distillation. In 4th Interna- tional Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016a.
Wu, Y.-H., Sun, F.-Y., Chang, Y.-Y., and Lin, S.-D, Ans: adaptive network scaling for deep rectifier reinforcement learning models. arXiv preprint arXiv:180900211 12, 2018,
Yang, R., Xu, H., Wu, Y., and Wang. X. Multi-task reinforce- ment learning with soft modularization. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. CoRR, abs/1606.04671, 2016b.
Yu, T. Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In 3rd Annual Conference on Robot Learning 2019, Osaka, Japan, October 30 November 1, 2019, Proceed- ings, volume 100 of Proceedings of Machine Learning Research, pp. 1094-1100, PMLR, 2019.
Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp. 30365-30380. PMLR, 2023.
Sun, L.., Zhang, H., Xu, W., and Tomizuka, M. Paco: Parameter-composiiinal multi-task reinforcement learn- ing. Advances in Neural Information Processing Systems, 35:21495-21507, 2022.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
Sun, X., Panda, R., Feris, R., and Saenko, K. Adashare: Learning what to share for efficient deep multi-task learn- ing. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., and Funkhouser, T. A. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1 1-5, 2018, pp. 4238 4245. IEEF, 2018.
Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Martín- Martín, R., and Stone, P. Deep reinforcement learning for robotics: A survey of real-world successes. Annual Review of Control, Robotics, and Autonomous Systems, 8, 2024.
Zhang, G., Jain, A., Hwang, I., Sun, S., and Lim, J. Efficient multi-task reinforcement learning via selective behavior sharing. CoRR, abs/2302.00671, 2023. doi: 10:48500/AXXIV.330000666
Teh, Y. W., Bapst, V., Czarnecki, W. M., Quan, J., Kirk- patrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA. pp. 4496 4506, 2017.
Zhang, Y. and Yeung, D. A regularization approach to learning task relationships in multitask learning. ACM Trans. Knowl. Discov. Data, 8(3):1:1-12:333 2013.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physies engine for model-based control. In 2012 IEEF/RSJ Inter- national Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pp. 5026-5033. IEEE, 2012.
Wilson, A., Fern, A., Ray, S., and Tadepalli, P. Multi-task reinforcement learning: a hierarchical bayesian approach. In Machine Learning, Proceedings of the Twenty-Fourhh International Conference (ICML 2007), Corvallis, Ore- gon, USA, June 20-24, 2007, volume 227 of ACM Inter- national Conference Proceeding Series, pp. 1015-1022.
ACM, 2007.
12

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
A. MT10 and MTS0 Environments
To evaluate multi-task reinforcement learning methods, we adopt the robotic manipulation tasks provided by the Meta- World benchmark suite (Yu al., 2019). Meta-World contains 50 distinct tasks and defines two standard benchmarks for multi task RL: MT10 and MT50, comprising 10 and 50 tasks, respectively. The MTI0 benchmark consists of the following 10 tasks with their corresponding task ID numbers: (1) reach, (2) push, (3) pick-place, (4) door-open, (5) drawer-open, (6) drawer-close, (7) sutton-press-oodoown, (8) peg-insert-side, (9) window-open, and (10) window-close.
Performance is measured using the success rate per task. An episode is considered successful if the agent completes the task at least once during the episode, following common evaluation protocols prior work (Yu et al., 2020; Haarnoja et al., 201 8b; Sun al., 2022; Hendawy al., 2024; Cho et al., 2024). In addition, we adopt modified versions of MT10 and MT50 as our default evaluation setting, where each task is trained with randomly sampled goal positions and evaluated across 10 randomized goal configurations, consistent with prior studies (Yang al., 2020; Sun al., 2022; Hendawy et al., 2024).
B. Low-Rank Adaptation for Multi-Task RL
Low-Rank Adaptation (LoRA) (Hu et al., 2022) injects a learnable, low-rank update into a frozen weight matrix so that only small number of additional parameters are trained while the original backbone remains intact. In our experiments we attach LoRA adapters the actor and critic multilayer pereeptrons (MLPs) used in ARS-LN.
Parameterisation.
For any linear layer with weights Wo E Ind xd. we introduce two trainable matrices A E Br and B E Rr of rank < min(doa, dm). The effective weight becomes
AB
W = Wo +
(9)
where a is scaling constant set to the LoRA rank (8 for MTIO, 16 for MT50). Gradients flow only through A and B; Wo is frozen throughout.
Adapter placement. We wrap the input projection and all hidden linear layers of both the actor and critic with LoRA. The final output heads are left untouched preserve output variance accumulated during pre-LoRA training.
Activation schedule. To avoid destabilizing early training, LoRA adapters are disabled (i.e. A B for the first 0.75 x T training steps on MTI0 and 0.833 x T steps on MT50, where is the total number of gradient updates reported in Table 8. From that point onwards the adapters are enabled and trained jointly with the actor and critic optimizer.
Parameter overhead. The additional learnable parameters introduced by LoRA are r(deu + dm)) per adapted layer. With 8 (MT10) and r= 16 (MT50) this overhead amounts to less than 3% of the original network size, providing efficient task-specific specialization without significant memory or compute cost.
C. Hyperparameters
In this section, we provide the hyperparameters for ARS used in the MT10 and MT50 experiments in Table along with some general hyperparameters used across ARS and the baselines in Table 8. Specifically, for the Soft Modular method, we adopt the deep structure proposed in (Yang et al., 2020), consisting of 4 layers with 4 modules and hidden unit size of 128.
Table 7. ARS specific hyperparameters.
Hyperparameters
MT10
MT50
number of Reset
4
6
13

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Table & General Multi-task RL hyperparammeters..
Hyperparameters
MT10
MT50
training steps
2 107
1 x 10a
number of reset (Nueset)
4
replay buffer size per task
1 x 10
x 10"
episode length
500
optimizer
Adam (Kingma & Ba, 2015)
batch size per task
100
learning rate (all networks)
3e-4
activation for critic
Tanh
activation for actor discount factor (?)
ReLU
0.99
MLP hidden layer size
[400, 400, 400, 400]
target network update period tau(r)
5e-3
D. Additional Results
D.1. Learning Curves of Q-Values with the SAC-MT Algorithm
In addition Figure Section 3.2, we plot additional learning curves of Q-values for all tasks in the MT10 benchmark. As shown in Figure 6, the training of the Q-network struggles with the more challenging tasks "psss"" 'pick-place, and "peg-insert-side." where their Q-values remain negative.
(a) reach
push
pick-place
(d) door-open
(c) drawer-open
(f) drawer-close
(g) button-prees--oopdown
(h) peg-insert-side
window-open
window-close
Figure 6. Learning curves of valulu with SAC-MT for all tasks in MT10 benchmark
14

---

ARS: Adaptive Reward Scaling for Mult Reinforcement Learning
D.2. Learning Curves of Average Success Ratios MTIO Based on Key ARS Components
Figures 7 and show the average success rates of ARS and its two variants (without reward scaling and without reset) in MT10 experiments. "ARS w/o reset' exhibits greater variation and instability due to biases from amplified rewards, whereas ARS w/o reward scaling' is more stable but underperforms challenging tasks, such "push," 'pick-place, and ppeg-insert-sid," due to low reward magnitudes, shown in Figure 7. The full ARS outperforms both variants, achieving improved stability and success rates, with at least 0.93 average success across all random seeds.
Tiee
Tiee
(a) reach
(b) push
(c) pick-place
(d) door-open
(c) drawer-open
Time Steps
Time Shape
Time Shepe
Time Stepa
Time Stops
(f) drawer-close
button- topdstn
(h) peg-insert-side
window-open
(i) window-close
Figure Learning curves of the success ratio averaged over all tasks in the MT10 benchmark, based the key ARS components: (1) the adaptive reward scaling scheme and (2) the reset mechanism.
1.0
Success Ratio
0.8
0.6
0.4
0.2
SAC-MT
ARS w/o reward scaling
ARS w/o reset
ARS
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Time Steps
1e7
Figure 8. Learning curves of average success ratios across tasks the MT10 benchmark, based on the key ARS components: (1) the adaptive reward scaling scheme and (2) the reset mechanism.
15

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
D.3. Learning Curves of Average Success Ratios MT50
Figure 9 shows that all oft-policy multi-task RL methods incorporating our ARS framework surpass the previous state-of-the- art (SOTA) performance (Cho al., 2024) the MT50 benchmark, indicated by the dotted line. Even the simplest method, SAC-MT, achieves exceptional results, emphasizing the remarkable effectiveness of the proposed framework. Furthermore, our approach achieves at least the previous SOTA performance using only half the samples, i.e., 50 million.
Figure 10 show that the performance of ARS improves with capacity, whereas SAC-MT (w/o ARS) does not. This confirms ARS as an effective approach for scaling model size and improving performance via reset and dynamic reward scaling.
SAC-MT (400 X4]
1.0
1.0
SAC-MT (512 X4)
X4)
Success Ratio
0.8
Success Ratio
ARS-LN (512 X41
(400
A
ARS-LN (800 X4)
ARS-LN (1024 X4)
Previous SOTA
0.4
0.4
SAC-MT ARS-LN
SAC-MT ARS
0.2
0.2
PCGrad ARS
SoftModular ARS
0.0
0.0
Previous SOTA
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Time Steps
Time Steps
Figure Learning curves average success ratios across tasks in the MT50 benchmark using off off-policy multi-task RL methods incorporating the ARS framework. The dashed line indicates previous state-of-the-art performance (Cho et al., 2024).
Figure Learning curves average success ratios across tasks the MT50 benchmark based on the network capacity.
16

---

ARS: Adaptive Reward Scaling for Mulk Reinforcement Learning
E. Additional Ablation Studies
E.1. Ablation Study on the Hyperparameter Threset
Incorporating our ARS framework into other off-policy multi-task RL algorithms introduces a unique hyperparameter: the number resets (Rrese)) To evaluate its impact, we conducted ablation studies on Preset- Figure shows the average success ratios for different Preset values. The results show that performance remains robust across various values, though lower Peeset generally achieves better performance. This trend is likely due the reduced frequency of model updates with higher Preset, resulting in insufficient updates.
1.0
ARS with 4
ARS Dreset
ARS with Draint= 9
0.8
Success Ratio
0.6
0.4
1
2
3
7
8
9
All
Task ID
Figure 11. Comparison of the final success rate per task in the MT10 henchmark based on the number of resets (Maeses.
E.2. A blation Study on the Presence of Reset Mechanisms Actor and Critic Networks
In this subsection, we conduct an ablation study to analyze the impact of reset mechanisms in the actor and critic networks.
Table 9 shows the average success ratio based on the reset strategy, showing the advantage of resetting both the actor and critic networks.
Table Ablation study on the Reset Mechansim
Reset Strategy
Benchmark
No reset
Reset Critic Only
Reset Actor & Critic
MT10
69.
96.3+2.3
97.3
17

---

ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
E.3. Comparison with PopArt
Prior work, PopArt (Hessel al., 2019), identifies the issue of varying reward scales and addresses it by introducing scale-invariant updates, which normalize critic targets using adaptive affine layer. To evaluate the effectiveness of our proposed adaptive reward scaling (ARS) method against PopArt's scale-invare invariant updates, integrate the latter into the SAC-MT algorithm and compare it with the ARS framework. We vary the update frequency of the scale-invariant updates over 1, 10, 100, 500. As shown in Table 10, our ARS consistently outperforms PopArt on the MTI0 benchmark across all frequencies, demonstrating the effectiveness the ARS framework.
Table 10. Results of average ratio (%) on Meta- World MTI0 for PopArt Variants
Update Frequency for PopArt
10
100
500
ARS
MTIO
67.1 + 12.8
73.3 + 7.6 75.0 + 6.3
71.5 + 7.8
97.3 + 2.1
E.4. Ablation Study on the Horizon Length
In the default setup of the MetaWorld-v2 (Yu et al., 2019) environments, the horizon length is set to 500, which was used consistently across all our experiments. However, we noticed that the experiments reported in the PaCo and MOORE papers (Sun et al., 2022; Hendawy et al., 2024) were conducted using horizon length 150, making a direct comparison between the reported results and our ARS method inappropriate. To address this, we reported the results of PaCo and MOORE with 500 in the main results (Table & 2 in Section 5.1).
the horizon length
In addition to the results with a horizon length of 500, we compare the proposed method, ARS-LN, with two baselines (PaCo and MOORE) on both the MT10 and MT50 benchmarks using shorter horizon length of 150. As shown in Table 11, ARS-LN sigmificantly outperforms both baselines, achieving the highest average success rates of 98.4% on MT and 91.0% on MT50, respectively. Notably, the scaling law observed with increased network capacity (as described Section 5.4) still holds the horizon-155 setting. Moreover, ARS-LN even achieves slightly higher performance with horizon length of 150 compared the 500-horizon setting.
Table 11. Comparisons average ratio (%) MetadWorld MTIO MT50 benchmark with Horizon Legnth
ARS-IN (Hidden Layer Dimensions)
[400, 400, 400, 400]
[800, 800, 800, 800]
[1024, 1024, 1024, 1024]
PaCo
MOORE
MT10
98.4 + 1.5
85.4
88.7 + 5.6
MT50
83.3:4..3
1.6
91.0+2244
11.
72.9+33.3
18

---

ARS: Adaptive Reward Scaling for Mulk Reinforcement Learning
F. Overview of the ARS framework
This section provides an overview of the proposed ARS framework. The orange box highlights its key components, including iissery-based reward scaling. supported by the reset mechanism (green box). It also illustrates that ARS can be seamlessly integrated into any off-policy multi-task RL algorithm with minimal eftort.
Task??;
Task
Sampling
{a no mm( (s))
(Bee1~ 1(.1ss.))
Multi- Agent
Store (st af ste l
Update the parameters
Te Qu
Replay Buffer D.
Replay Buffer
Multi-Task Off Policy Algorithm
Reset Networks
True
If T % Ireset
Initialize parameters of networks
SAC-MT, SAC-MT-MH, Soft Modular, etc.
Calculate Reward Scaling Factor
False
Update rew J1=1
Apply reward scaling factor
(fit cew X rijt=
N
ARS Framework
Figure 12. Overview of the ARS framework. The orange box illustrates the history based reward scaling, supported by a reset mechanism.
19


Reviews:
[{"review_id": "bX9uFjEVqZ", "reviewer": "Reviewer_SzVu", "summary": "This paper introduces Adaptive Reward Scaling, a novel framework for multi-task reinforcement learning that dynamically adjusts reward magnitudes using a history-based scaling strategy and integrates a periodic network reset mechanism to mitigate overfitting and biases toward simpler tasks. The empirical results on the Meta-World show some improvements in success rates compared to several established baselines.\n\n**Update after rebuttal**--I appreciate the authors’ thorough revision and the additional experiments addressing my concerns (especially the attempt with MOORE). I hope those will be implemented in the revised draft. I have decided to increase my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer SzVus for their thoughtful feedback and valuable suggestions, which have significantly improved our paper. We have carefully addressed each comment, strengthened our experimental results, and clarified our key contributions accordingly. Below, we respond in detail to each point raised by the reviewer\n\n\n**Other Multi-Task RL Baseline (MOORE [1]):**\n\nWe appreciate the reviewer’s suggestion to include an additional multi-task RL baseline, specifically MOORE [1]. In response, we conducted comparative experiments using MOORE on the MT10 and MT50 benchmarks.\n\nIn the default setup of the MetaWorld v2 environments, the horizon length is set to 500, which was used consistently across all our experiments. However, we noticed that the experiments reported in the original MOORE paper [1] were conducted using a horizon length of 150, making a direct comparison between the reported MOORE results and our ARS method inappropriate.\n\nTo address this, we attempted to run MOORE with the horizon length set to 500.  Unfortunately, we encountered significant computational challenges, as the official implementation requires approximately 200 hours to complete on the MT50 benchmark. To ensure fairness, we evaluated ARS using a horizon length of 150. The ARS results are shown in Table 1 at the following anonymous link:\n\nhttps://sites.google.com/view/icml25ars\n\nThe MOORE setup (n_expert = 6) on MT50 uses significantly more parameters than our default ARS (400×4). To ensure a fair comparison, we evaluated MOORE against a larger ARS variant (800×4) with a similar number of parameters. Despite the comparable model size, ARS consistently outperformed MOORE on both benchmarks while also requiring dramatically lower computational costs. Specifically, on MT50, MOORE requires approximately 200 hours of training, while ARS (800×4) achieves superior performance in just 22 hours—an order-of-magnitude reduction in training time.  These results demonstrate that our ARS framework achieves significant performance improvements **without incurring substantial computational overhead**. We will include MOORE results in the final version.\n\n[1] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. \"Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts.\" The Twelfth International Conference on Learning Representations.\n\n**Discussion of Recent Advances in Reward Normalization and Adaptive Scaling across Different RL Domains** .\n\nWe thank the reviewer for recommending additional related work. We will expand our related work section accordingly to enhance the paper.\n\n**Evaluation is Limited to Meta-World Benchmarks** \n\nWe acknowledge the importance of including experiments beyond Meta-World. While the MT10 and MT50 benchmarks are widely recognized and challenging in multi-task RL—with no method fully solving both—they serve as a strong demonstration of our approach's effectiveness. Nevertheless, we welcome suggestions for additional benchmarks and will gladly conduct further experiments to strengthen our claims.\n\n**Lower Performance of Soft Modular with ARS in MT10**\n\nWe attribute the initially lower performance of Soft Modular with ARS on MT10 to suboptimal hyperparameter tuning. By increasing the batch size per task from 100 to 128, we significantly improved performance to $98.8 \\pm 1.3$.\n\n**Computational Overhead and Potential Limitations Introduced by the Reset Mechanism**\n\nThe reset mechanism introduces minimal computational overhead, occurring only during reset periods. With our reset strategy (updating re-initialized networks 1,000 times per reset), the total number of updates increases only slightly from 2,000,000 to 2,006,000—an **increase of just 0.3%**.\n\n**How can ARS be extended or adapted to other multi-task or multi-agent settings?**\n\nAs illustrated in Algorithm 1 and Figure 14, the ARS framework is easily adaptable to any off-policy multi-task RL method by incorporating an adaptive reward scaling factor based on the replay buffer.\n\nIn multi-agent settings, we can consider either individual rewards per agent or a global reward. With a global reward, applying ARS is challenging since it effectively becomes a single task. In contrast, individual rewards allow each agent to be treated as a separate task, making ARS more applicable. However, caution is needed—unlike multi-task learning where tasks are independent, agents in multi-agent systems are interdependent. Thus, careful design of the adaptive reward scaling factor is crucial.\n\n**Limited Theoretical Analysis**\n\nWe agree with the reviewer that the theoretical analysis is limited. However, we believe the extensive experimental validation provides strong support for the effectiveness of ARS. We plan to address a comprehensive theoretical analysis in future work.\n\n**Other Comments Or Suggestions:**\n\nWe thank the reviewer for identifying typos, issues with bold symbols, and citation errors. These will be corrected thoroughly in the revised paper.", "claims_and_evidence": "The paper claims that the adaptive reward scaling combined with resets improves training stability and overall performance in multi-task settings. These claims are supported by comprehensive experimental evidence, including detailed success rate tables and ablation studies isolating the contributions of each component. The experimental results are convincing, though a few recent method is missing in baseline and additional experiments on diverse benchmarks could further validate the generality of the method.", "methods_and_evaluation_criteria": "The proposed approach is well-motivated and clearly explained. The use of the replay buffer to compute task-specific mean rewards for scaling and the integration of resets to counteract biases are both novel and appropriate for the challenges in multi-task RL. The evaluation criteria—success ratios, effective solvable task ratios, and ablation studies—make sense and they demonstrate the performance gains in straightforward way.", "theoretical_claims": "The paper does not emphasize formal proofs or theoretical guarantees but rather focuses on the algorithmic innovation and empirical validation.", "experimental_designs_or_analyses": "The experimental design is robust, comparing ARS against several state-of-the-art baselines using standard benchmarks. The inclusion of ablation studies provides clear insight into the effectiveness of both the reward scaling mechanism and the reset strategy. It could have been better to explore beyond MetaWorld suite, though I understand the current situation; exploring additional environments could further support the claims.", "relation_to_broader_scientific_literature": "ARS builds on existing work in reward scaling, modular networks, and resetting mechanisms in deep RL. The paper successfully situates its contributions within the broader context of multi-task RL research by comparing with methods such as SAC-MT, PCGrad, and Soft Modular. This connection to prior work is well articulated.", "essential_references_not_discussed": "The author should have the paper called Multi-task Reinforcement Learning with Mixture of Orthogonal Experts (MOORE, ICLR 2024) both in Related work section and experiments as a baseline. In addition, it might benefit from a discussion of very recent advances in reward normalization and adaptive scaling across different RL domains to highlight its broader applicability and limitations.", "other_strengths_and_weaknesses": "*Strengths:*\n\n- Clear and innovative formulation of an adaptive reward scaling mechanism.\n- Thorough empirical evaluation with convincing ablation studies.\n- Significant improvements on challenging benchmarks (especially hard problems).\n\n*Weaknesses:*\n\n- Theoretical analysis is somewhat limited.\n- Evaluation is limited to Meta-World benchmarks; broader testing could enhance the claims.", "other_comments_or_suggestions": "- In table 4, you should consider statistical significance when you bold the method that shows the highest performance. For instance, in easy tasks, second column is within range of third column.\n- Citing mistake in line 400 right column.\n- Typo in line 423. Multi-taks → Multi-task", "questions_for_authors": "- Do you have any intuition why SAC-MT works well with ARS in MT10, while Soft Modular works well with ARS in MT 50?\n- Can you provide more insights into the computational overhead and potential limitations introduced by the reset mechanism.\n- How can ARS be extended or adapted to other multi-task or multi-agent settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rokhL9oAEI", "reviewer": "Reviewer_mXD6", "summary": "The paper introduces Adaptive Reward Scaling (ARS), a novel framework designed to tackle the difficulties caused by varying reward distributions in multi-task reinforcement learning.  \n\nARS employs a history-based reward scaling strategy that dynamically adjusts reward magnitudes to ensure balanced training focus across diverse tasks. Additionally, ARS incorporates a reset mechanism that mitigates biases introduced by early learned tasks, enhancing adaptability and convergence. The framework integrates seamlessly into existing off-policy algorithms and has demonstrated state-of-the-art performance on Meta-World benchmark.\n\n## update after rebuttal\nI don't have major concerns regarding this paper. I still recommend acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer mXD6 for their thoughtful feedback and valuable suggestions, which have significantly improved our paper. We have carefully addressed each comment, strengthened our experimental results, and clarified our key contributions accordingly. Below, we respond in detail to each point raised by the reviewer\n\n**Limited Environments** \n\nWe appreciate the reviewer’s suggestion of three benchmarks for multi-task RL.  We believe that the MT10 and MT50 benchmarks from Meta-World are widely recognized and present significant challenges in multi-task RL. Since no existing method has fully solved both, demonstrating strong performance on these benchmarks effectively highlights the strength of our approach.\n\nThat said, we agree that including the suggested benchmarks could further strengthen our claims. Benchmarks [2] and [3] are primarily designed for demonstration-driven settings, such as imitation learning and offline RL. While our current focus is on the online setting, we believe that extending our ARS method to offline multi-task RL would be an interesting direction for future work. We appreciate the reviewer for introducing these valuable benchmarks.\n\n**Reward Scale Issue with Sparse Reward Setting** \n\nEven in sparse reward settings, task difficulty can vary significantly across tasks, leading to large variations in returns during training. This can cause an uneven reward distribution in the replay buffer. Applying the ARS framework can help mitigate this issue.\n\n**Instability caused by varying reward scales**\n\nLarge variations in reward scales during training can indeed destabilize the learning process. To mitigate this, we update the reward scales only during reset periods—four times in MT10 and six times in MT50. This reset mechanism helps maintain stability by preventing frequent changes in reward scaling. Additionally, since the networks are initially trained with the established reward scales, this approach further supports stable training throughout.", "claims_and_evidence": "I think most of the claims in this submission are supported by evidence.", "methods_and_evaluation_criteria": "Yes, they make sense to me.", "theoretical_claims": "This submission does not include proofs.", "experimental_designs_or_analyses": "I went through all the experiments, and most of them make sense to me.", "relation_to_broader_scientific_literature": "Reward scaling has been shown to be effective in prior studies [1, 2]; however, most of these works focus on single-task settings. In contrast, this paper addresses the multi-task setting, where reward scaling poses greater challenges due to varying reward magnitudes across tasks.\n\n[1] Wu, Yueh-Hua, et al. \"ANS: adaptive network scaling for deep rectifier reinforcement learning models.\" arXiv preprint arXiv:1809.02112 (2018).\n\n[2] Henderson, Peter, et al. \"Deep reinforcement learning that matters.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths\n\n- This paper directly addresses the challenge of varying reward distributions across tasks in multi-task RL. Improper reward scales can result in biased training and suboptimal performance. As an RL practiioner, I would say this is a critical yet frequently overlooked issue in the field of RL.\n\n- The ARS framework proposed in this paper includes a history-based reward scaling strategy and a reset mechanism. It's simple but very intuitive.\n\n- The ARS framework demonstrates strong empirical results on the Meta-World benchmark,  solving the MT10 benchmark from scratch.\n   \n- The proposed ARS framework seems to be applicable to any off-policy multi-task RL method. The authors demonstrate its applicability by integrating it into various off-policy multi-agent approaches.\n   \n\n### Weaknesses\n\n- The proposed framework is evaluated solely on Meta-World, a relatively simple benchmark. The conclusions would be more compelling if more challenging tasks were included, such as those in [1, 2, 3].\n\n[1] Zhu, Yuke, et al. \"robosuite: A modular simulation framework and benchmark for robot learning.\" arXiv preprint arXiv:2009.12293 (2020).\n\n[2] Mu, Tongzhou, et al. \"Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.\" arXiv preprint arXiv:2107.14483 (2021).\n\n[3] Chernyadev, Nikita, et al. \"Bigym: A demo-driven mobile bi-manual manipulation benchmark.\" arXiv preprint arXiv:2407.07788 (2024).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- I assume the rewards used in this paper are dense. Would varying reward scales still pose an issue if binary sparse rewards were used instead?\n- The reward scales change dynamically during training. Could this variation in reward scales affect training stability? If not, what mechanisms ensure stability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGhtXMCwXG", "reviewer": "Reviewer_Cp7C", "summary": "Multi-task reinforcement Learning (MTRL) algorithms face challenges when tackling tasks with varying complexities and reward distribution. In this work, the authors propose a method for tackling the varying reward magnitude across tasks by adaptively scaling the reward of each task using a history-based reward scaling strategy. Furthermore, to prevent early overfitting to a few tasks, the authors adopt a resetting mechanism from the single-task deep RL literature to mitigate that. The proposed approach, named ARS, is benchmarked on Metaworld compared to related baselines while showing promising results in handling large scale MTRL setting with varying reward magnitude.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Cp7C for their thoughtful feedback and valuable suggestions, which have significantly improved our paper. We have carefully addressed each comment, strengthened our experimental results, and clarified our key contributions accordingly. Below, we respond in detail to each point raised by the reviewer.\n\n**Other Multi-Task RL Baseline (PopArt and MOORE):**\n\nWe appreciate the reviewer’s suggestion regarding the inclusion of an additional multi-task RL baseline, specifically  PopArt[1] and MOORE [1]. In response, we conducted comparative experiments using MOORE on the MT10 and MT50 benchmarks.\n\nSince the Meta-World v2 environments default to a horizon of 500, we used this setting across all our experiments. However, the original MOORE paper used a horizon of 150, making a direct comparison with our ARS results challenging. To address this, we attempted to run MOORE with a horizon of 500, but the official code required roughly 200 hours of computation on the MT50 benchmark alone. Consequently, we evaluated ARS using a horizon length of 150 for fairness. The ARS results are shown in Table 1 at the following anonymous link:\n\nhttps://sites.google.com/view/icml25ars\n\nNotably, the MOORE setup on MT50 (n_expert=6) includes significantly more parameters than our default ARS (400×4). To ensure a fairer comparison, we tested MOORE against an enlarged ARS variant (800×4), which has a comparable parameter count. Despite the comparable model size, ARS consistently outperformed MOORE on both benchmarks requiring significantly less computational time. For instance, on MT50, MOORE requires about 200 hours to train, whereas ARS (800×4) surpasses its performance in just 22 hours—an order-of-magnitude faster.  These results demonstrate that our ARS framework achieves significant performance improvements **without incurring substantial computational overhead**. We will include MOORE results in the revised version.\n\nWe also tested PopArt on MT10. Because the official implementation is unavailable, we reimplemented PopArt’s scale-invariant updates within SAC-MT, using SAC target values for critic learning. We varied the update frequency over {1, 10, 100, 500}. The outcomes are shown in Table 2 (available anonymously at https://sites.google.com/view/icml25ars). Across all frequencies, our ARS method outperformed PopArt on MT10, and we plan to include the complete results for both MT10 and MT50 in our revised paper.\n\nhttps://sites.google.com/view/icml25ars\n\nOur ARS method consistently outperformed all PopArt variants on the MT10 benchmark. We will include comprehensive PopArt variant results for both the MT10 and MT50 benchmarks in the revised paper.\n\n [1] Hessel, Matteo, et al. \"Multi-task deep reinforcement learning with popart.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019. \n\n[2] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. \"Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts.\" The Twelfth International Conference on Learning Representations.\n\n**Effectiveness of Reset Mechansim**\n\nWe thank the reviewer for suggesting a more thorough analysis of the **Reset Mechanism**.Table 4 in our paper already highlights its importance in stabilizing performance. To further investigate, we analyzed Q-values per task during training on MT10, comparing ARS with and without the reset mechanism. Results are shown in Figure 1 at the following anonymous link:\n\nhttps://sites.google.com/view/icml25ars\n\nAs shown in Figure 1, training without the reset mechanism leads to significantly lower Q-values across tasks, with negative values appearing in tasks like 'push' and 'pick-place'—despite rewards being always positive. Q-values also show greater variance without reset, emphasizing the reset mechanism’s role in ensuring training stability.\n\n**Random Goal Positions**\n\nSorry for the confusion. Random goal positions are used in our setup.\n\n**Lower Performance of PaCo in MT10**\n\nWe use the same hyperparameters for PaCo as in the original paper. The performance gap likely stems from the difference in horizon lengths: the original PaCo uses 150, while Meta-World v2 defaults to 500, which we follow. To isolate this effect, we also run ARS with a horizon of 150. Results are shown in Table 1 at the anonymous link below:\n\nhttps://sites.google.com/view/icml25ars\n\n**ESTR Results**\n\nWe sincerely apologize for the confusion regarding the ESTR results. The reported ESTR value of PaCo at $\\delta = 0.7$ was incorrect and should be revised from 0.6 to 0.5.\n\n**Definition of H**\n\nYes, H is the horizon length.\n\n**Frequency (n_reset) for  ARS w/o reset**\n\nWe investigated the performance of the ARS w/o reset baseline with various values of n_reset . We then selected the best-performing setting for the paper. The value of n_reset  used in the paper for the ARS w/o reset baseline is 40.", "claims_and_evidence": "In this work, two main claims were presented, \n1. The introduction of reward scale variation of a challenge in multi-task RL.\n2. The importance of adaptively scaling the reward of different tasks because of the varying reward magnitude among them.\n3. The role of resetting is integrated to alleviate overfitting to early learned tasks, in addition to stabilizing the critic training since the proposed adaptive scaling can destabilize the critic training since the Q target changes frequently. \n\nIn my opinion, \n1. The problem of the varying reward distribution or reward scale variation is a known issue that has been discussed in the literature [1]. I don't think that introducing this issue should be a contribution to this work. Nevertheless, the fixed reward scaling is an interesting view for this problem, in addition to the connection to the reward scaling in single-task deep RL. In addition, the presented example in Figure 2 shows another dimension of the problem, hence motivating adaptive scaling.\n\n2. The method introduced in this work to adaptively scale the reward magnitude is novel.\n\n3. The resetting mechanism, in general, can help the MTRL training since it has been adopted in prior works [2,3]. In this work, resetting the networks is motivated by more than one reason, as stated above. Since it is an essential component in the algorithm and crucial to enhance the performance, it is important to strongly support the claim, behind adding this component to ARS, by ablation studies. For example, showing how the critic is suffering without resetting the network, given the change in the reward magnitude caused by the adaptive scaling. Otherwise, the resetting mechanism is not really a contribution to this work; it is just an adoption of an existing tool in the literature [3].\n\n[1] Hessel, Matteo, et al. \"Multi-task deep reinforcement learning with popart.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n[2] Sun, Lingfeng, et al. \"Paco: Parameter-compositional multi-task reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 21495-21507.\n[3] Cho, Myungsik, et al. \"Hard tasks first: multi-task reinforcement learning through task scheduling.\" The 41st International Conference on Machine Learning. 2024.", "methods_and_evaluation_criteria": "- I believe the authors proposed a *new method* for adapting the scale of the reward magnitude between tasks, which is effective when looking at the empirical results.\n- For the evaluation criteria, I believe Metaworld is a good benchmark to study the effectiveness of MTRL algorithms, especially on the MT50 scenario, which is large scale.", "theoretical_claims": "- No theoretical claims were provided or discussed in this work.", "experimental_designs_or_analyses": "- In general, all experiments highly suit this work in showing the effectiveness of the proposed method in the MTRL setting.\n- I appreciate the teaser experiment added in Figure 2.\n- I have concerns regarding the baselines. \n     1. As stated before, the problem of varying reward distribution is known in the MTRL literature. One important baseline is PopArt [1],\nwhich is, as far as I know, the first work to discuss this issue in the MTRL setting. This approach is similar to the normalization baseline in the ablation in Table 5, yet not exactly. Popart hasn't been used as a baseline nor cited in this work. \n    2. In addition, MOORE [2] is a recent MTRL approach that reported SOTA results on Metaworld, in particular, on MT50. Moreover, MOORE has never been benchmarked against nor mentioned in the related work section. \n- For the Metaworld MT10 and MT50 scenarios, it is not clear to me if this experiment considers random goal positions (MT10-rand and MT50-rad) [3] or fixed goal positions.\n- The performance of PACO is lower in this work than in the original paper. It could be because of the different network architecture and hyperparameter, but why not follow the original hyperparameter of each method?\n- I have a concern regarding the experiment done in Figure 4. I don't understand how some baselines can have higher ESTR value as the threshold $\\delta$ increases. In particular, PACO has a higher value at $\\delta = 0.7$ than $0.5$.\n\n[1] Hessel, Matteo, et al. \"Multi-task deep reinforcement learning with popart.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n[2] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. \"Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts.\" The Twelfth International Conference on Learning Representations.", "relation_to_broader_scientific_literature": "- I believe this work highlights the varying reward distribution issue, which has been studied previously in the literature [1]. This indicates the importance of looking into the reward magnitude as a cause for the instability of the MTRL training.\n- In addition, the reset strategy proposed shows effectiveness empirically, hence supporting previous claims discussed in the literature [2,3].\n\n[1] Hessel, Matteo, et al. \"Multi-task deep reinforcement learning with popart.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n[2] Sun, Lingfeng, et al. \"Paco: Parameter-compositional multi-task reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 21495-21507.\n[3] Cho, Myungsik, et al. \"Hard tasks first: multi-task reinforcement learning through task scheduling.\" The 41st International Conference on Machine Learning. 2024.", "essential_references_not_discussed": "- I believe this work should cite the following papers for the aforementioned reasons:\n[1] Hessel, Matteo, et al. \"Multi-task deep reinforcement learning with popart.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n[2] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. \"Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts.\" The Twelfth International Conference on Learning Representations.", "other_strengths_and_weaknesses": "- All points have been mentioned in Claims And Evidence & Experimental Designs Or Analyses.", "other_comments_or_suggestions": "- In the Preliminaries Section, the Multi-Task Reinforcement Learning subsection, $H$ was not defined. I believe it is the horizon.", "questions_for_authors": "- For the ablation study in Figure 4, does the ARS w/o reset baseline consider the same frequency ($n_{reset}$) in updating the reward scaling factors? the same frequency when performing the factors update + resetting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a mechanism to scale the rewards of each environment in a multitask rl setting, coupled with parameter resets. They show that compared with various rewards scaling mechanisms, the proposed method performs very well.

In particular the authors have reacted to the comments they've received from the reviewers adding two additional baselines and providing well thought-out responses. While the work restricts itself to metaworld the experiments are done carefully and convincingly.

NOTE: Please take care of duplicating text from Cho et al (2024) with respect to the appendix, but additionally look at the other section of the paper with the same issue in mind. It would be good not to have any issue on this front.

Therefore I'm happy to recommend accept.
Decision: Accept (poster)