Value Function Decomposition for Iterative Design of Reinforcement Learning Agents
Authors: James MacGlashan
Evan Archer
Alisa Devlic
Takuma Seno
Craig Sherstan
Peter R. Wurman
Peter Stone
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
explainable AI
machine learning
decision making
deep learning

Abstract: Designing reinforcement learning (RL) agents is typically a difficult process that requires numerous design iterations. Learning can fail for a multitude of reasons and standard RL methods provide too few tools to provide insight into the exact cause. In this paper, we show how to integrate \textit{value decomposition} into a broad class of actor-critic algorithms and use it to assist in the iterative agent-design process. Value decomposition separates a reward function into distinct components and learns value estimates for each. These value estimates provide insight into an agent's learning and decision-making process and enable new training methods to mitigate common problems. As a demonstration, we introduce SAC-D, a variant of soft actor-critic (SAC) adapted for value decomposition. SAC-D maintains similar performance to SAC, while learning a larger set of value predictions. We also introduce decomposition-based tools that exploit this information, including a new reward \textit{influence} metric, which measures each reward component's effect on agent decision-making. Using these tools, we provide several demonstrations of decomposition's use in identifying and addressing problems in the design of both environments and agents. Value decomposition is broadly applicable and easy to incorporate into existing algorithms and workflows, making it a powerful tool in an RL practitioner's toolbox.

Value Function Decomposition for Iterative Design of Reinforcement Learning Agents
James MacGlashan' Jamss mawglassannsooy.....mmmmmm. com
Evan Archer evan. archer@sony com
Alisa Devlic
alisa. www.ddesownn.c.ooooomomomooommme com
Takuma Seno
Craig Sherstan +
takuma. www.Ssony.ccoooooooo com
craig
Peter R. Wurman
Peter Stone pstoneêcs www.useetecccom edu
peter. www.anwsony.ccooommoo com
Abstract
Designing reinforcement learning (RL) agents is typically difficult process that requires numerous design iterations. Learning can fail for multitude reasons, and standard RL methods provide too few tools provide insight into the exact cause. this paper, we show how to integrate value decomposition into a broad class of actor-critic algorithms and use it to assist in the iterative agent-design process. Value decomposition separates a reward function into distinct components and learns value estimates for each. These value estimates provide insight into agent's learning and decision-making process and enable new training methods to mitigate common problems. As demonstration, of soft actor-critic (SAC) adapted for value decomposition. SAC-D maintains similar performance SAC, while learning larger of value predictions. We also introduce decomposition-base based tools that exploit this information, including a new reward influence metric, which measures each reward component's ss effect on agent deciisonnmmainng Using these tools, we provide several demonstrations of decomposition's use in identifying and addressing problems the design of both environments and agents. Value decomposition is broadly applicable and easy to incorporate into existing algorithms and workflows, making it powerful tool an RL practitioner's 's toolbox.
introduce SAC-D, variant
Introduction
Deep caafooreemen-leeamiinn (RL) approaches have achieved successes in range of application areas such as gaming (55) 32 3d 38),, robotics ([22)), and the natural sciences (125) 351 Despite these successes, applying RI. techniques
complex control problems remains daunting undertaking, where initial attempts often result in underwhelming performance. Unfortunately. there are many reasons why agent may fail learn a good policy, making difficult diagnose which reason(s) caused a particular agent to fail. For example: an agent may fail because the state features were insufficient to make accurate predictions, different task objectives defining the reward function may be imbalanced, the agent may fail to sufficiently explore the state- acte-action space, values may not accurately propagate more distant states, the neural network may have sufficient capacity to
1 Sony Al "Equal contribution
*The University of Texas Austin
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

approximate the policy or value function(s), or, there may be subtle differences between training and evaluation environments. Without way to diagnose the causes of poor performance or to recognize when a problem has been remedied, practitioners typically engage in a long trial-and-error design process until an agent reaches a desired level of performance. Frustrations with this trial-and-error process have been expressed in other work [16].
We describe how value decompasition, simple, broadly-applicable technique, can address these application challenges. In RL, the agent receives a reward that is often a sum of many reward components, each designed to encode some aspect of the desired agent behavior. From this composite reward, it learns single composite value function. Using value decomposition, an agent learns a component value function for each reward component. To perform policy optimization, the composite value function is recovered by taking a weighted sum of the component value functions. While prior work has proposed value decomposition methods for discrete-action Q-learning 19 29 [1], we show how value decomposition can be incorporated into a broad class of actor-critic (AC) methods. In addition, we introduce SAC-D, a version of soft actor-critic (SAC) L3 [4] with value decomposition, and explore its use in multi-dimensional continuous introduce the influence metric, which measures how much an agent's 's decisions are affected by each reward component.
environments. We also
While earlier work focuses on its use in reward design 16 value decomposition can facilitate diagnosis of wide range of issues and enable new training methodologies. To demonstrate its utility, in Sec. g we show how use it to: (1) diagnose insufficient state features; (2) diagnose value prediction errors and exploit the decomposed structure to inject werggroundknowweeeee and (3) identify reward components that are inhibiting exploration and mitigate the effect by gradually incorporating component predictions into policy optimization.
Value decomposition's additional diagnostic and training capabilities come at the cost of more- challenging prediction problem: instead of learning a single value function, many must be learned. To investigate if this difficulty negatively impacts agent performance, we compare the average performance of SAC-D to SAC on benchmark environments. We find that a naive implementation of SAC-D underperforms SAC and then show how to improve SAC-D so that it matches and sometimes exceeds SAC's performance. These improvements may also be applied to value decomposition for other AC algorithms.
While variations of value decomposition has been explored extensively in past work (see Sec. i in this paper, we make the following contributions. (1) We show how to integrate value decomposition into a broad class of actor-critic algorithms. (2) We analyze the performance of different implementations of value decomposition for SAC on a range of benchmark continuous-action environments. (3) We introduce the influence metric: a novel value decomposition metric for measuring how much each reward component affects fecision-making. (4) We provide a set of illustrative examples of how value decomposition and influence can be used to diagnose various kinds of learning challenges. (5) We describe new training methods that exploit the value decomposition structure and can be used to mitigate different learning challenges.
2 Background
2.1 MDPs and Q-functions
In RI, an agent's interaction with the environment is modeled Markov Decision Process (MDP): (S, P, R, 7), where S is a set of states, A is a set of actions, P P:S. x A x S - R is a state transition probability function P(s, a, s') Pr(S.+1 = S' s, = a). R S A + R is a reward function A(s, a) = E RS, = s, A, a). and 7 E [0, 1] discounts future rewards The goal of agent is to learn a policy (a[s) that maps states to an action probability distribution that maximizes the sum of future rewards. The agent is trained to maximize the discounted return E E 7 a.)). The f-funcionn maps state-action pairs to the expected cumulative discounted reward when starting in state taking action a, and then following policy π thereafter:
Qa (8, a)
E
Σ r't R(st. TT) so 8,00 =
(1)
1=
'For continuing tasks, 7 must be < 1, and we only consider algorithms for 1.

---

2.2 Soft actor-critic
Soft actor-critic (SAC) IE IE is an off aff-po actor-critic algorithm parameterized with five neural networks: a stochastic policy network т with parameters 6, and two pairs of Q-functions and target Q-functions with parameters (01, 82) and (81, f2). respectively. As with other actor-critic algorithms, SAC has two main steps: policy evaluation (in which it estimates the Q-function for policy т), and policy improvement (in which it optimize the policy to maximize its Q-function estimates). Unlike other actor-critic algorithms, SAC optimizes a maximum entropy formulation of the MDP, which rewards are augmented with policy entropy bonuses that prevent premature policy collapse. To perform policy evaluation and improvement, SAC minimizes the following loss functions simultaneously:
((ss,a;0)) u)² for {1,2],
Lo. E
(2a)
L EE log (ogg((u)s;)) min Q(s, u; u;
(2b)
jE(1.2)
where (s, transitions are drawn from experience replay buffer. y (minje(1.2) a's',a'; a'; (optionally learned) entropy regularization parameter. The min of Q-function pairs addresses overestimation bias in value function estimation [5I. The parameters 0, and 02 are updated toward 0, and O2 via an exponentially moving average each step.
a log <<a'ls::)) a' x(1)''0), ~ T()s;;), and is an
2.3 Environments
Throughout the paper, we italicize descriptive names for the components of the continuous-action Lunar Lander (LL.), Bipedal Walker (BW) and Bipedal Walker Hardcore (BWH) environments. In LL., an agent must land a spacecraft in the center of a landing zone using as little fuel as possible. The reward components include: a reward for successful landing; penalties for crashing (crash) and engine usage (main, side); shaping rewards used to encourage the agent to stay upright (angle), move towards the center of the landing pad (position) with low velocity (velocity) and land with both legs (right leg, left leg). In BW, an agent learns to make a 2-legged robot walk. The reward components include: reward for forward progress, penalty for falling (failure). cost for actions (control), and a shaping reward to discourage head movement (head). BWH is identical to BW, but adds additional obstacles for the agent navigate.
3 Value decomposition for actor-critic methods
Most RL algorithms estimate the value function and use it to improve its policy. Unfortunately, value functions and policies provide little insight into the agent's decision- making. However, reward functions are often composite functions of multiple component state-action signals. By learning a value function estimate for the current policy for each component, practitioners gain insight into what the agent expects to happen and how these reward components interact. Naturally, policy improvement still requires the composite value function. In Sec. 3.1 show how the composite Q- function can be recovered from the component Q-functions. From this property, a range of actor-critic algorithms can be adapted to use value decomposition by following the below template f
1. Alter Q-function networks have m outputs instead of 1, where m is the number of reward components.
2. Use the base algorithm's Q-function update for each of the m components, replacing the composite reward term with the respective component reward term.
3. Apply the base algorithm's policy improvement step by first recovering the composite Q-function.
For example, this template can be applied to algorithms that use TD(0) 33 Retrace 27 It works with algorithms that improve the policy by differentiating through the Q- function
that use
23 and ones that fit it toward non- parametric target action distributions M.
*Composite state value functions can similarly be recovered; however, Q-funtioons allow for deeper introspection, we focus on that setting in this work.
3

---

Algorithm SAC-D and SAC-D-CAGrad Update
Require:
Experience replay buffer B; twin Q-function parameters 81, (with e 8, U 02) and target parameters 01, 02; policy parameters 6; discount factor 7; entropy parameter a; reward weights w E R=+1, learning rates Ag, Ar; target network step size ny; Boolean use_ eee_caardd for SAC-D-CAGRAD or SAC-D.
I: Sample transition (minihatch) (s, r, B
vector of reward components b I
2: Sample policy actions a' ~ I (-|ss" s's': and u (T|ss; 6)
3: Fm++1 + qaa log [('lss':
b Extend reward vector to include entropy reward Find target network by minimum composite Q-value
4: j + arg min 2m++1 w:Qi(s' (''((s''';;)))
3E(1,2)
5: M < r: + O,,)
6: TQ: + E (Qi(s, 03) y1)²
7: L ao (uls; 6) (1.2) Σ, Em+ u;;,(s;u;0g))
cagrad then
9: O θ L.
10: else
11:
θ θ
Vo m+ Em LQ.
12: end if
13: - 6 An LLLT
14: Update target networks θ + (1 n) n/0
Although this template is conceptually simple, learning component Q-functions poses a more difficult prediction problem: multiple predictions must be learned instead of one composite prediction. Ideally, this increased difficulty would not negatively impact agent performance. In Sec. 3.2 we introduce SAC-D, adaptation of SAC to use value decomposition, and describe additions the above template to maintain performance with conventional SAC. Although these additions are contextualised to SAC, they are general and can be used when adapting other actor-critic algorithms.
made to
3.1 Recovering the composite Q-function
We assume the environment's 's reward function is a linear combination of m components: R(s, a) A D," ws, a). where wy E IR is scalar component weight for the ith component and [, a) IR is the reward function of the ith component for state-action pair (s, a). Applying the linearity of expectation, we find the Q-function inherits the linear structure from the reward
m
Q" a) = Σ w; QT
(3)
where we define the i'th component Q-function as QJ (s, a) 2 ETE. T' 0e) 30 = Unless otherwise specified, we assume w;; = 1 for all i. Because the component weights are factored out of the component Q-functions, they may be varied without changing the component prediction target, allowing the policy to be evaluated for any weight combination. Although the assumption of linearity may seem restrictive, note that each reward component may be non-linear function of state variables, allowing for very expressive environment rewards. Furthermore, many environments, including all the environments we investigate in this paper, are naturally structured as sum of (non-linear) reward components.
a).
3.2 SAC with value decomposition
Here we introduce SAC-D (Alg- I an adaptation of SAC to use value decomposition. Adapting SAC only requires one additional consideration from our template: the entropy bonus reward term SAC adds is treated this approach, which we refer to as SAC-D-Naive, underperforms SAC in many settings. We found two additional modifications essential to match the performance of SAC. The first concerns how we apply the twin-network minimum of Eq, 2 in the context of value decomposition. The second is to use Conflict-A verse Gradient descent (CAGrad) 24 to address optimization problems that arise when training multi-headed neural networks. We refer to SAC with value decomposition and the
m 1 'th reward component: Rm+1(()) (s') A aa log क < (a')s'; Φ) (line 3 However,
"See Theorem A. for the proof. This linear decomposition property of value functions has been explored elsewhere 3 99. but in different contexts and with different motivations. See Sec. for more information.

---

Normalzzdd MermaltzeEpppoe Eptode Revard
1.0
Bipedal Walker Hardcore
Ant
0.96
Kid
100
403
SAC-D-CAGrad
0.75
Average Return
0.90
SAC
SAC-D
0.5
2+3
0.84
SAC-D-Naive
-50
0.78
0.8
1.0
1.5 200 255
1.5
Normalized Episode
Gradient Steps
Steps
(a)
(b)
(c)
(d)
Figure Benchmark results (a)-(b) Comparison of algorithms over environments. For each algorithm, scores are collected from the 10 final checkpoints (10 episodes x 10 final policies), and normalized on individual environments. The 95% CIs are estimated using stratified bootstrap sampling. (a) The interquartile mean (IQM), which gives the mean over the central 50% of values. (b) For each normalized score I, % runs achieving a score at least æ. Grey region, E > 1.0, corresponds to runs outperforming the mean SAC baseline score. (c) Training curves continuous- control benchmarks for BWH. Solid lines represent mean episode return over trials. Shaded areas are confidence intervals. Lines are uniformly smoothed for clarity. (d) Same as (c), but for Ant.
twin-network correction as SAC-D, and the variant with twin-network correction and CAGrad as SAC-D-CAGrad.
Twin-network minimums value decomposition: In SAC, the Q-value target the minimum of two Q-function networks (Eq. 2 Using the same Q-value update rule for each component, as described in our template, suggests using a minimum for each component target: == minje(1.2 Qi(s, 8,). However, this is not good choice in practice. The purpose of the twin- network minimum is to mitigate overestimation bias from the feedback loop of the policy optimizing the Q-function. Because the policy optimizes the composite Q-function, better approach is to all the predictions from the network with the minimum composite Q-function (Alg. [ lines 5 This approach reduces underestimation bias and improves performance compared to element-wiss minimum (see Sec.
Mediating the difficulty of multi-objective optimization: Even though the scalar values of the composite Q-function are identical to those used in SAC, simultancous optimization of all QT components may introduce training problems common in multi objlti-ojeective optimization: conflicting gradients, high curvature and large differences in gradient magnitudes [9] The CAGrad method, designed for the multi-task RL setting, addresses these issues by replacing the gradient of a multi-task objective with a weighted sum of per-task loss gradients. This updated gradient step maximizes the improvement of the worst-perforning task on each optimization step, and still converges minimum of the unmodified loss. We incorporate CAGrad into SAC-D by treating each component as "task", and update the gradient vector accordingly (Alg. line
4 Robustness experiments
We benchmark SAC-D, SAC-D-CAGrad and SAC-D-Naive against SAC on selection of continuous- action Gym environments. For each environment, we exposed existing additive reward components without altering the behavior of the environments or their composite rewards. That is, these environments already implemented their reward functions as a linear combination of separate reward components and we simply exposed that information to the algorithm (for details, see App. B). As outlined App. C we used hyperparameters previously published for use with SAC [4]]]]] for all experiments. We tied SAC-D's hyperparameters to SAC's because our goal is for value decomposition to be a drop-in addition without significant loss in agent-perforee seentt-prffrmancc.. However, it is possible better performance could be reached with tuning.
Figure I(a) shows the performance of each algorithm aggregated across all environments and all experimental runs. Figure [(b) shows the same information, but highlights the distribution of scores across experimental runs. In aggregate, SAC-D-CAGrad slightly outperforms SAC, although it has broader range of performance scores. SAC-D-Naiv significantly underperforms SAC.
headed prediction also improve representation learning, in work auxiliary tasks
18
26
5

---

Q Q2
(4)
Q-1 Q2 1 Q3
Q-2 Q + Qs (f)
Q-3 Q1
(g)
Figure 2: Influence calculation for three-component environment with 2D action space. In this illustrative example, component 2 has high influence, while components 1 and 3 have low influence, but for different reasons. Each figure shows a contour-map view the Q-function over the 2D action space for the same state. (a) The composite Q-function the policy optimizes. The black point is a (near-optimal) policy selection, and the black arrow is the policy gradient. (b)-(d) Each component Q-function. (e)-(g) Component-ablltee composite Q-function surfaces Q-, used to compute influence for each component. Influence is defined by the length of the red line, which is the difference of the policy gradients computed for the composite Q-function (black arrow) and for Q- (orange arrow). (e) Component has low influence because (f) Conversely, component has high influence; without it, the policy would move toward the peak of component (g) While component 3 contributes a large value, it has low influence because the value nearly uniform; much of its value is received regardless of the action selected.
out aut-weighed by component 2.
We provide training curves for the 8 environments investigated in App. D but highlight the atypical training curves for BWH (Fig. [(c) and Ant (Fig. I[d). In the case of BWH, SAC-D-CAGrad significantly outperforms SAC. It was not the goal this work for SAC-D to improve on SAC. Rather, sought to provide more insights into the learning process. As such, make strong claims about when SAC-D can be expected to outperform SAC. Nevertheless, this result does suggest that SAC-D may sometimes benefit from auxiliary task learning. We leave this question subject for future investigation.
In Ant, SAC-D (without CAGrad) underperforms all other methods. We found that infrequent environment termination causes large Q-function errors and leads to catastrophic gradient conflicts. Further analysis of termination issues and CAGrad behavior is described in Appendices and respectively.
Reward component influence
It can be difficult to understand how agent's predictions interact to affect decision decision-making. We now introduce the reward influence metric, which indicates how much each component contributes agent's decision. Intuitively, low influence means that removing component would not alter decision-making: high influence means that removing would significantly alter decision-makinn making..
For multi-dimentee dimensional continuous actions, we define the optimal influence of component in state by how much the optimal policy x" in state differs from the optimal policy when component
removed: I (s) A (s)* ( where n², & arg max, E
Σ 00 7 Eizi Wy Ry(Se, oz 4x)
In practice, we apply two approximations to I, (s) for computational efficiency. First, rather than compare the difference of optimal policies, compare the difference between one step of policy improvement: (s) A || arg max, Q" (s a) arg max, 1* a)lle. where Q7 a) 2 EjØt wy (1]Q7(s (s, a). Second, since arg max is computationally demanding and sensitive to statistical noise, we replace the arg max, (ss, a) policy improvement operator with policy gradient-step
DDsccrete acternn spaces could use probability distance measure, but we focus continuous-action spaces.

---

(a)
side
rightt leg
Flactional al Influence
left leg
Faational factionalInfuumm Infuence
0.8
1.0
forward failure
0.6
entropy
main
0.6
0.4
crash
position
0.4
02
angle
0.2
velocity
0.0
landing
Step
Gradient Steps (1e6)
Figure 3: Fractional influence (a) Fractional influence for evaluation trajectory BWH. For this agent and trajectory, the forward component dominates the failure component in decision-making, except at two points (red arrows). At the first point, the agent was risk of falling into a pit. After the agent stabilizes, the forward component returns to being dominant. After the second point, the agent becomes unable move forward. Failure's s dominance indicates that if the agent cannot move forward, it prefers standing to falling. If it is possible to surmount this obstacle, more exploration may be required. (b) Mean stacked fractional influence across 10 trials of Lunar Lander. We observe: (I) the sparse landing reward doesn't 't meaningfully contribute to decision-making in the earliest part of training: (2) crashing influence starts high and then decreases, suggesting the agent initially learns to avoid crashing; (3) the position, angle, and velocity reward components each shaping reward - maintain large influence, suggesting possible overreliance on shaping rewards; (4) numerous reward components never have much influence; they might be unnecessary require more attention.
operator (typical RL algorithms like SAC): à AV a), where ā is a deterministic policy action selection (such the mode) and À E (0, 1) is step size.
When taking the difference of the gradient-step operator applied to the Q" and Q surfaces, the terms cancel, and Ả can be factored out of the norm. The result is the inffuence metric (Fig. 2
(s; 0) AIV 0) - (sQ a; 0)ll2.
(4)
The raw magnitudes of component influence can informative by themselves; for example, sharper Q-function surface leads to larger influence values. However, to compare influence values across components, typically compute the fractional influence by normalizing the (always non-negative) (s:0)
influence: (s; Σn (s:0))
We several techniques visualize the fractional influence. For trajectories, plotting influences over timesteps may help identify and explain key decision points (Fig. 3(a). When studying an agent's behavior across training, we maintain summary statistics of fractional influence. We visualize mean fractional influence across all components as a stack plot, sorted so that the component with the maximum influence at the end of training is at the bottom. Figure 3(b) shows such diagram for the Lunar Lander environment; we provide figures for environments in App. G
Value decomposition: strategies for agent design
Agent design is often brute-force process of trial and error: when agent doesn't 't perform as expected, we choose some aspect of the agent's 's design to vary, and then we train again. Although this approach succeed, can be expensive in both time and computation.
In this section, we illustrate a different approach, showing how a decomposed reward helps break from trial-and-error by encouraging the designer to consider agent's point of view. In three examples, each using either the LL or BWH environments (Sec. 2.3). we draw on value decomposition tools to: (I) identify learning problems by comparing components' empirical returns to their predictions; (2) constrain component value estimation; (3) identify adverse reward interactions with the influence metrie; (4) dynamically re- reighttht reward components.
These examples are not just meant to demonstrate these specific techniques, nor to demonstrate significant performance improvements these well-tested benchmark environments (in fact, SAC-D eventually produces good policies both environments without additional tuning). Rather, the goal is to showcase general approach agent design for novel applications in which iteration and

---

(a)
100
20
Value flanding)
80
return
Markov
Value (crosh)
10
60
baseline
40
-110
20
-20
constrained
baseline
-30
50
100
150
200
250
0.0
0.2
0.4
0.6
0.8
1.0
Step
Gradient Steps (1e6)
Figure 4: Remedies for incorrect value predictions. (a) For the landing component empirical return (dashed blue), the value prediction (green) for a value function trained with baseline features matches the return well except for a final plateau in the shaded region. Training value function with the added Vg feature (red) improves predictions in this region. The dashed pink line indicates the point after which compute error metrics in App. H (b) Value predictions for the crash component with sign constraints (blue lines; using both target and prediction constraints) and without (orange lines). Each line represents independent trial. Unconstrained estimates can be positive; constrained value estimates keep predictions negative.
failure is costly. Combined with statistical tools that allow us to reason with small-sample statisties, value decomposition provides a vocabulary to describe an agent's behavior and interpretable tools for targeted interventions. Even these small examples are more intuitive and less computationally demanding than they would be with trial-and-error approach.
For all examples, training parameters are identical to the robustness results of Sec.
We analyze the behavior of an agent trained on Lunar Lander by comparing each component's empirical return to the agent's value predictions, Q:- The agent is trained to land successfully, and generally the component value predictions match their empirical returns well. Curiously, all component predictions are flat near the end of the episode. For most components, these flat predictions are a good match for their returns, but not for landing. Investigating the landing dynamics, we found the simulator waits many steps after touchdown before producing a landing reward. During this period, the observations are constant, suggesting the features are inadequate represent landing's s return. To make the observations Markov, we introduce new feature to the agent's 's observations that indicates the duration since the agent's 's velocity (horizontal, vertical and angular) went to zero: Vi (t) = Vol (t)/c, where Vollep (t) is the number of time steps since all the velocities dropped below threshold and is fixed normalizing constant. With this feature, post-landing predictions show a marked improvement (see Figure 4(a) and Appendices
Diagnosing and improving insufficient features:
Diagnosing and mitigating value errors using domain knowledge: Lunar Lander's design makes clear that certain reward components are always non- non-positive (crash, main, side) while others are always non-negative (landing). However, we observe that the agent's 's decomposed value predictions do not always match these bounds. In particular, value predictions of crash have a tendency to oscillate about 0 after the agent learns to land. Value decomposition allows to explicitly enforce sign constraint on crash (Figure 4bb see App. details). In this particular example, constraints do not alter policy learning performance, but the resulting predictions easier interpret, and the same technique may improve performance in more complex environments.
Mitigating an adverse reward with component weight scheduling: Under the BWH reward function, a random policy is far more likely to experience an unsuccessful outcome (falling over) than successful one (walking forward). This bias inhibit agent exploration early in training, causing an agent's policy to fall into a local minimum (the agent stands still), Here, we diagnose this dynamic using component predictions and influence metrics, and remedy it by varying single component weight during training.
We find that the failure component's 's fractional influence dominates the forward component's fractional influence early in training, and that this relationship reverses as agent performance improves (Fig. 5(a) The forward component's near-zero value predictions (Fig. 5(b) early in learning indicate
that in many episodes, the agent neither moves nor expects to move (Fig ((c) Early dominance by the casy-to-observe failure suggests that it is inhibiting exploration.
8

---

Fractional Influence
0,7
forward follure
forward follure
(c)
1.0
scheduled weight
constant weight
0.6
forward raturn
score x
0.75
0.5
Value
10
0.4
0.5
83
0
0.25
0.0
0.2
0.0 0.2
0.4
20
40
60
80
Gradient Steps (1e6)
Gradient Steps (1e6)
Forward Progress Per Episode Score bc)
Figure 5: Remedy for adverse reward (a) The initially low fractional influence of forward (blue curve) indicates it may be initially inhibited by failure penalties with higher influence. (b) The forward value predictions and returns (blue eurves) near zero early in training, indicating the agent does not expect to move forward. (c) An agent trained with its failure weight scheduled from a value close to zero up to one (orange curve) improves early forward progress compared to an agent with a constant failure weight (blue curve). Forward progress is measured as the average forward progress return from all evaluation episodes of the first 200k gradient steps. Shaded areas represent 95% bootstrap confidence intervals from 10 trials.
To mitigate this problem, we vary the failure component weight, Winilure, from 0.01 to 1 over the course of learning according to the schedule described in App. This schedule significantly increases the agents' forward progress (Fig. ((c)), and accelerates learning.
7 Related work
Our work builds upon earlier studies of value decomposition in the explainable reinforcement learning (XRL) literature: DrQ [9)) DrSARSA p2], and HRA [3 Like our approach, these methods learn separate value function estimates for each term of a linear reward function. DrQ and HRA are off off-policy Q-learning- like methods, while DrSARSA is on-policy method. HRA does not converge to globally optimal policy, as each value function is only locally-optimal for the reward component it measures. The RDX and MSX metries proposed by DrQ could be adopted in our setting, but the influence metric is easier to use with continuous-actions and aggregate with summary statistics. Our approach improves upon these prior contributions by: (1) working in continuous-action and discrete-action environments; (2) allowing for and demonstrating dynamic re-weighting of reward components during training; and (3) being applicable family of actor-critic methods. While these approaches and ours explore using value decomposition for explainability, these works focus on describing why an agent took certain actions users, whereas we focus on how to use value decomposition to diagnose and remedy learning challenges.
The Horde architecture 34 and UVFA [30], methods for multi-goal learning, also employ multiple value functions (one for each goal). UVFAs use a parameterized continuous space of goals, while Horde makes multiple discrete value function predictions. The value functions in our work conditioned on a policy that optimizes the global reward, whereas in Horde and UVFAs the value functions are conditioned on independent policies that greedily optimize local goals (similar to HRA). Additionally, in our approach, the composite value function can be recovered from the components. Other value decomposition work includes Empathic Q-learning 22 and Orchestrated Value Mapping 10. The primary difference between our work these other approaches regards the motivation and application of value decomposition. These works focus on how value decomposition can improve sample efficiency, generalization, or other aspects of the core learning problem, rather than how to diagnose and remedy problems.
Mathematically, value decomposition bears resemblance to work on successor features, which has focused primarily state representation and transfer learning + 4
6 12 Methods for multi- objective RL 16 2x learn sets of policies, each with a distinct linear weight over multiple reward objectives. Our work also recovers the value function for different linear reward preferences. but uses this capability to diagnose behavior and learning problems rather than to learn multiple policies.
For broad overview of XRL, we direct the reader to Heuillet et al. 77

---

8 Concluding remarks
We have argued that the iterative design of reinforcement learning agents can be improved through the use of value decomposition, in which we keep individual reward components separate and learn value estimates of each. We provided a simple prescription for deriving value decomposition algorithms from actor-critic methods, and applied it to SAC to derive the SAC-D algorithm. Combined with the CAGrad method, SAC-D meets or exceeds SAC's performance in all environments we tested. We introduced the influence metric, and demonstrated its use in measuring each reward component's effect on an agent's s decisions. Finally, we provided several examples of how value decomposition diagnose and remedy agent learning problems.
Although value decomposition is a simple and broadly applicable tool, we note the following limitations. (1) Our method requires a composite reward function of multiple components. (2) We only study linear reward decomposition. (3) Component predictions only tell you the agent expectations under the single learned policy; changing the weights doesn't 't tell you what to expect after re- re-optimizing the policy for them. (4) Our approach benefits most with methods that learn Q- functions. Methods that optimize the policy with empirical returns have weaker link between agent expectations (component Q-functions) and policy decisions. The influence metric also requires Q-function model.
Our method presents the same societal benefits and risks as other RL, methods. However, we believe this technology has net beneficial impact, because making agent decisions more introspectable enables developers to catch problematic behavior before deploying the technology. One particular concern, however, is that value estimates represent agent's 's beliefs, not ground truth; this should be kept in mind when such predictions are used real-world decision-making, as they may reinforce biases lead to incorrect conclusions.
References
[1] A. Abdolmaleki, J. T. Springenberg, Degrave, Bohez, Y. Tassa, Belov, N. Heess, and M. Riedmiller. Relative entropy regularized policy iteration. arXiv, 1812.02256, 2018.
[2] R. Agarwal, M. Schwarzer, S. Castro, A. Courville, and M. G. Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[3] A. Barreto, W. Dabney, R. Munos, J. Hunt, T. Schaul, H. van Hasselt, and D. Silver. Successor Features for transfer in reinforcement learning. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
[4] A. Barreto, D. Borsa, S. Hou, G. Comanici, E. Aygün, P. Hamel, D. Toyama, J. Hunt, S. Mourad, D. Silver, and D. Precup. The Option Keyboard: Combining skills in reinforcement learning. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
[5] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, Schneider, S. Sidor, I. Sutskever, J. T.Tang F. Wolski, and S. Zhang. Dota 2 with large scale deep reinforcement learning, arXiv, 1912.06680, 2019.
[6] D. Borsa, A. Barreto, J. Quan, D. J. Mankowitz, R. Munos, H. V. Hasselt, D. Silver, and Schaul. Universal Successor Features approximators. arXiv, 1812.07626, 2019.
[7] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAl Gym. arXiv, 1606.01540, 2016.
[8] A. Cassirer, G. Barth-Maron, E. Brevdo. S. Ramos. T. Boyd, T. Sottiaux, and M. Kroiss, Reverb: A framework for experience replay. arXiv, 2102.04436, 2021.
[9] Dayan. Improving generalization for temporal difference learning: The Successor Representation. Neural Computation, 5(4):613 3244, 1993.
10

---

[10] M. Fatemi and A. Tavakoli. Orchestrated value mapping for reinforcement learning, arXiv preprint arXiv:2203.07171, 2022.
[11] Fujimoto, H. van Hoof, and Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning (ICML), 2018.
[12] C. Grimm and S. Singh. Learning independenty--obtiibbbe reward functions. arXiv, 2019.
[13] Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (ICMI.) 2018.
[14] T. Haarnoja, A. Zhou, Hartikainen, G. Tucker, S. Ha, Tan, Kumar, H. Zhu, Gupta. P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. arXiv, 1812.05905, 2018.
[15] H. Hasselt. Double Q-learning. In Conference on Neural Information Processing Systems (NeurIPS), 2010.
[16] C. F. Hayes, R. Rādulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, Mannion, A. Nowé, G. Ramos, M. Restelli, P. Vamplew, and D. M. Roijers. A practical guide to multi- objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1 (661):26,440) Apr. 2022.
[17] A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez. Explainability in deep reinforcement learning. Knowledge-Bass Systems, 214:106685 2021.
[18] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv, 161 1611.05397, 2016.
[19] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and E. Doshi-Velez. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on Explainable Artificial Intelligence, 2019.
[20] D. P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. arXiv, 1412.6980, 2014.
[21] R. Laroche, M. Fatemi, J. Romoff, and H. van Seijen. Multi-advisor reinforcement learning. arXiv preprint arXiv: 770.000756 2017.
[22] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-137, 2016
[23] T. P. Lillierap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
[24] B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu. Conflict-averse gradient descent for multi-task learning. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[25] K. Mills, P. Ronagh, and I. Tamblyn. Finding the ground state of spin hamiltonians with reinforcement learning. Nature Machine Intelligence, 2:509-517, 2020.
[26] Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, D. Kumaran, and R. Hadsell. Learning to navigate in complex environments. arXiv, 1611,03673, 2016.
[27] R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare. Safe and efficient off-policy reinforcement learning. In Conference on Neural Information Processing Systems (NeurIPS), 2016.
[28] D. M. Roijers, Vamplew, Whiteson, and R. Dazeley. survey of multi-objective sequential decision decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.

---

[29] S. J. Russell and A. Zimdars. Q-decomposition for reinforcement learning agents. In International Conference on Machine Learning (ICML), 2003.
[30] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International Conference Machine Learning ICML, 2015.
[31] H. V. Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, and J. Tsang. Hybrid reward architecture for reinforcement learning. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
[32] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, Huang, A. Guez, T. Hubert, Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap. F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game of Go without human knowledge. Nature, 550:354-359, 2017.
[33] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[34] R. S. Sutton, J. Modayil, M. Delp, T. Degris, P. M. Pilarski, A. White, and D. Precup. Horde: scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Conference Autonomous Agents and Multiagent Systems (AAMAS), 2011.
[35] K. Tunyasuvunakool, J. Adler, Z. Wu, T. Green, M. Zielinski, A. Zidek, A. Bridgland, A. Cowie, C. Meyer,, A. Laydon. al. Highly accurate protein structure prediction for the human proteome. Nature, 596:590-596, 2021.
[36] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Sulsky, J. Molloy, T. Paine, C. Gulcehre, Z. Wang. T. Pfaff, Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu,
D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575:350-354, 2019.
[37] P. Virtanen, R. Gommers, T. F. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, Peterson, Weckesser, J. Bright,
van der Walt, M. Brett, J. Wilson, K. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa. P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-274 2020.
[38] P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, Eckert, F. Fuchs, L. Gilpin, P. Khandelwal, V. Kompella, H. Lin, MacAlpine, D. Oller, T. Seno, C. Sherstan, M. D. Thomure, H. Aghabozorgi, L. Barrett, R. Douglas, D. Whitehead, P. Dürr, P. Stone, M. Spranger, and H. Kitano. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602:223-223 2022.
[39] Yu, S. Kumar, Gupta, S. Levine, K. Hausman, and Finn. Gradient surgery for multi-task learning. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We outline three limitations of
our approach in the Concluding Remarks.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We devote paragraph of Sec. X to these issues.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We assume linearity of the reward function and other standard MDP requirements.
(b) Did you include complete proofs of all theoretical results? [Yes] Our paper is largely empirical; however, we provide a proof for the Q-function identity used in the introduction of the composite Q-function.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [No] At present, we are unable to release our source code or data.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [Yes] We report error bars in all experimental results. For results within individual environments, we report standard deviation error bars. To report aggregate results across all environments, bars computed using the rliable framework 2
report bootstrap-resampe error
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the creators of Gym and the eliable toolkit.
(b) Did you mention the license of the assets? [N/A] We have no new assets that require
licenses.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] All empirical results are based on simultation using standard benchmark environments.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "moVcdOC6PO9", "reviewer": "Reviewer_Ssv2", "summary": "This paper provides a framework to integrate value decomposition into a broad class of actor-critic algorithms, and applied it to SAC to derive the SAC-D algorithm. Lots of experiments are conduct to verify SAC-D's performance over SAC. This paper introduces the influence metric, and demonstrated its use in measuring each reward component's effect on an agent's decisions. Finally, they provides several examples of how value decomposition can diagnose and remedy agent learning problems.", "questions": "(1) Since SAC-D is developed by incremental, heuristic refinement  to SAC, it is not clear whether the performance gain obtained by this refinement is limited to a few experiments in this paper. \n(2) Value decomposition in this paper is artificially divided. How does value decomposition with different divisions affect the effect of the algorithm?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nThe operation of integrating value decomposition into reinforcement learning algorithms seems to be common sense, which has been proposed by many existing literature. And the performance of value decomposition has been explored in a large body of literature. The work of this paper seems to be just another confirmation of previous findings. Therefore, this paper is less original.\n\nQuality:\nThis paper is well written and easy to understand. But, I have two concerns as follows. (1) Since SAC-D is developed by incremental, heuristic refinement  to SAC, it is not clear whether the performance gain obtained by this refinement is limited to a few experiments in this paper. (2) The iterative design of RL agents shown in Section 6 is quite interesting. However, it still seems to require strong task-specific priors from experts.\n\nClarity:\nThis paper is well written and easy to understand.\n\nSignificance:\nThe work is a bit significant, but does not give new insights into value decomposition.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hkA5J8gVays", "reviewer": "Reviewer_cBJt", "summary": "This paper proposes a tool to diagnose and design the reward function for training the reinforcement learning (RL) agent. In practice, the reward function in RL consists of multiple terms (e.g., collision penalty and velocity bonus). The proposed method assumes the reward is the linear combination of these terms and learn a Q-function for each reward term. Such a decomposition enables the reward function designer to inspect the influence of each reward on the agent's decision, thus reducing the efforts of tuning the reward function without guidance. Several diagnosis case studies showcase how RL practitioners use this tool to design the reward function.", "questions": "- The current formulation needs the reward function to be a linear combination of multiple rewards. Can the author comment possibility of extending the proposed method to non-linear rewards?\n- In section 6, the results of some diagnosis (e.g., weight scheduling, markov features) can be applied to SAC without value decomposition. Have the authors tried applying weight scheduling and markov features to SAC without value decomposition? This might help answer whether the diagnosis results can be applied back to non-decomposed version. ", "limitations": "The author acknowledged the limitations in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nThough value decomposition is not a new idea, using value decomposition to guide the reward function design and the proposed influence metric are an original contribution, to my best knowledge. \n\nQuality and clarity:\n\nThis paper is easy-to-follow. The experiments are well-designed.\n\nSignificance:\n\nThe proposed reward design scheme is important in RL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dtygV19NmvM", "reviewer": "Reviewer_2638", "summary": "The paper proposes to use value decomposition for the critic of generic actor-critic (AC) methods, and then applies it to soft-actor-critic (SAC) as a particular case. The idea of value decomposition is not new and has been studied in various prior work over past 20 years or so. Thus, the actual contribution of the paper is simply to use value decomposition in the context of AC methods. ", "questions": "- The paper lacks any thorough analysis of possible issues and meaningful solutions with formal ground. This is essential. \n\n- L127-128: \"Because the component weights are factored out of the component Q-functions, they may be varied without changing the component prediction target.\" \n\nThis is not correct. First, changing the weights will change Q, hence, it may change j at line 4 of Algorithm 1. Second, changing Q induces a change of policy. Notice that your value learning is on-policy and changing Q can beaks that because the policy that collects the transition becomes different from the learning policy. At the very least this can make the learning process unstable/slow depending on how much and how fast the weights are changing. This is not a trivial issue. Note also that if Q-learning is used instead of SARSA to make it off-policy, then the entire algorithm will NOT converge to optimality due to Jensen’s inequality. See reference [3] bellow for a detailed analysis. \n\n- L186: Not clear why norm is used (also not clear what norm it is). Is your action-space multi dimensional? If yes, you may need to describe it properly in your section 2.1.\n\nSome references:\n\n[1] van Seijen, H., Fatemi, M., Romoff, J. & Laroche, R. Separation of Concerns in Reinforcement Learning. arXiv [cs.LG] (2016)\n\n[2] Fatemi, M. & Tavakoli, A. Orchestrated Value Mapping for Reinforcement Learning. in International Conference on Learning Representations (2022).\n\n[3] Laroche, R., Fatemi, M., Romoff, J. & van Seijen, H. Multi-Advisor Reinforcement Learning. arXiv [cs.LG] (2017)\n", "limitations": "The authors counted four limitations, which I agree with in general. However, my rejection of the paper is not grounded on these limitations. ", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nNothing in particular.\n\nWeaknesses:\n- As the main idea of the paper is quite simple, I expected a neat mathematical analysis. For example, what is the impact of value decomposition on AC methods and why they should be used? When and how they change bias/variance of the base AC method? Does using value decomposition influence convergence? etc...\n\n- Re-weighting the reward components may make the algorithm completely unstable and even non-convergent. A clear analysis is required, beyond the few presented experimental results. \n\n- The presentation may also be improved. This paper mostly looks like writing a report around some experimental results. \n\n- Minor: some missing key references for value decomposition.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O-BTBKWWus", "reviewer": "Reviewer_yA8d", "summary": "This paper proposes *value decomposition* for diagnosing and iteratively designing actor-critic algorithms. The key idea of value decomposition is to treat the reward signal as constituted by a number of distinct components that together yield value. However, rather than just take some simple operation of these distinct signals, value decomposition proposes to learn a $Q$ function for each separate component of the reward: a Q-network, then, outputs $m$ predictions (for $m$ reward components) rather than just $1$. This relatively simple idea is explored as a mechanism for diagnosing a variety of characteristics of an RL algorithm, and ultimately give rise to the design and improvement of new algorithms. As the paper points out, the core idea of value decomposition is not new, and has been applied to Q-learning variants in the past. Indeed, value decomposition is conceptually similar to typical linear assumptions on the reward or value, but encourages algorithm design to explicitly learn these distinct value components. Moreover, to my knowledge, this is the first extension to actor critic algorithms: the paper proposes SAC-D, a version of Soft Actor-Critic with value decomposition in the mix. Section 4 presents findings from experiments contrasting various forms of SAC-D in different continuous Gym environments, such as Bipedal walker or Ant. \n\nArguably the most significant aspect of value decomposition is that it unlocks new perspectives for diagnosing agent failures, and for iteratively improving the design of these agents in light of these failures. Section 5 presents a series of experiments and insights showcasing how dropping out one of the reward components can impact aspects of the agent. Figure 2, for example, illustrates the the influence each reward component exerts on the Q function through contour maps over different mixes of the reward components $Q_1, Q_2$ and $Q_3$, while Figure 3 explores the _fractional influence_ of each component. I found Figure 3b to be particularly illuminating. These insights culminate in Section 6 that provides a pragmatic view on how to incrementally improve agent design using the diagnostic tool of value decomposition. For instance, in Lunar Lander, certain reward components should be negative, while others should be non-negative. Observing that an agent is incorrectly predicting certain values to be outside their possible bound allows for incremental adjustment of the agent. ", "questions": "Q1: At first I found the extension to SAC quite natural, but reading it through a few times, I find it less clear why SAC is needed for the rest of the story. To me the main claim of the work is that value decomposition can be a powerful diagnostic tool for understanding agent limitations, and fixing them. This perspective applies just as easily to the existing Q-learning approaches to value decomposition. So, I am left wondering: why is SAC the focus, rather than Q-learning? Another way to put it: do you anticipate that the pragmatic suggestions from Section 5 and 6 extend to Q-learning based value decomposition as well?\n\nQ2: Do you anticipate the linearity of the reward composition to be problematic or overly limiting?\n\nQ3: One reaction I have to the work is that linear functions (and approaches focused on linearity) in general are praised for their interpretability. In this way, the idea of value decomposition is very nearly just making a linearity assumption. How, in general, should we think about this approach as different from linear RL, or just assuming that the reward admits a linear decomposition with some unknown weights as in Jin et al. (2020)?\n\nJin, Chi, et al. \"Provably efficient reinforcement learning with linear function approximation.\" Conference on Learning Theory. PMLR, 2020.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[STRENGTHS]\n\nOverall, this paper proposes a novel perspective for diagnosing agent failures and iteratively improving agents as a result. As the introduction of the paper points out, this new viewpoint directly engages with the practical difficulties of deploying RL agents: they often fail for unknown reasons, and staring at a monolithic neural network and chaotic learning curves can often be daunting to debug and react to. The method of value decomposition is simple, easy to understand, and can be broadly applicable. The paper is well written and takes care in motivating many of its claims. It also simply lays out its core contributions, and limitations. The experiments are broad, interesting, and reveal the considerable potential behind the proposed method.\n\nI believe this paper possesses many virtues, and is sure to be of interest to the community. For this reason, pending any issues uncovered by other reviewers, I recommend accepting this paper.\n\n[WEAKNESSES]\n\nThere are several small things that could improve the paper, but none are major. First, the scope of the paper is quite ambitious, but I believe ultimately that sections 5 and 6 and the perspectives therein really constitute the core contributions. In this sense, it is not clear the new algorithm SAC-D or the experiments in Figure 1 are strictly necessary to the core of the work. A mild suggestion, but perhaps they can be moved to the appendix. Otherwise, I have a few minor writing suggestions and typos to fix (see below).\n\nWriting Suggestions:\n- In Algorithm 1, consider using the \\texttt{} wrapper for the boolean \"use_cagrad\"\n- I believe in Equation 2b $Q_i$ should be changed to $Q_j$.\n- The pseudocode for Algorithm 1 might be cleaned up. For instance, I find the added text to detract from the readability of the algorithm. There are a few too many comments as well.\n- Another minor point, but I found the y-axis of Figure 1b to be counter intuitive. The first few times I read this plot, I assumed the \"Normalized Expected Reward\" in Figure 1a was applying its y-axis to both plots. I wonder if there is some visual adjustment to be made that can prevent this collision.\n- Small point of consistency: the subfigure labels differ throughout the paper in style.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept