Solving Continuous Control via Q-learning
Authors: Tim Seyde
Peter Werner
Wilko Schwarting
Igor Gilitschenski
Martin Riedmiller
Daniela Rus
Markus Wulfmeier
Venue: ICLR 2023 poster (2023)
Keywords: reinforcement learning
continuous control
learning efficiency

Abstract: While there has been substantial success for solving continuous control with actor-critic methods, simpler critic-only methods such as Q-learning find limited application in the associated high-dimensional action spaces. However, most actor-critic methods come at the cost of added complexity: heuristics for stabilisation, compute requirements and wider hyperparameter search spaces. We show that a simple modification of deep Q-learning largely alleviates these issues. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods when learning from features or pixels. We extend classical bandit examples from cooperative MARL to provide intuition for how decoupled critics leverage state information to coordinate joint optimization, and demonstrate surprisingly strong performance across a variety of continuous control tasks.

Published as a conference paper at ICLR 2023
SOLVING CONTINUOUS CONTROL VIA --LEARNING
Tim Seyde" MIT CSAIL
Peter Werner MIT CSAIL
Wilko Schwarting MIT CSAIL
Igor Gilitschenski
University of Toronto
Martin Riedmiller DeepMind
Daniela Rus!
Markus Wulfmeier! DeepMind
MIT CSAIL
ABSTRACT
While there has been substantial success for solving continuous control with actor- critic methods, simpler critic-only methods such as Q-learning find limited appli- cation the associated high-dimensional action spaces. However, most actor- critic methods come at the cost of added complexity: heuristics for stabilisation, compute requirements and wider hyperparameter search spaces. We show that a simple modification of deep Q-learning largely alleviates these issues. By com- bining bang-bang action discretization with value decomposition, framing single- agent control cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods when learning from features pixels. We extend classical bandit examples from cooperative MARL to provide intuition for how decoupled critics leverage state information coordinate joint optimization, and demonstrate surprisingly strong performance across variety of continuous control tasks.
Walker Walk
Humanoid Walk
Dog Run
Scaling
3393
605
Dog
[Dol SARGOLRRRA andtnn andnn
Epioode Return
750
Hummanood
250
00
02
64
06
18
1,0
1.0
2.0
48
6.0
180
1,0
200
300
4,
50
60
It. Action Dimensions
20
Timestep (M]
Timestep [M]
Timestep [M]
D4PG
DMPO
Figure 1: Q-learning yields ofate-of-the-ar performance on various continuous control benchmarks. Simply combining bang-bang action discretization with full value decomposition scales to high- dimensional control tasks and recovers performance competitive with recent actor-critic methods. Our Decoupled Q-Networks (DecQN) thereby constitute concise baseline agent highlight the power of simplicity and to help put recent advances in learning continuous control into perspective.
INTRODUCTION
Reinforcement learning provides a powerful framework for autonomous systems to acquire complex behaviors through interaction. Learning efficiency remains central aspect of algorithm design, with a broad spectrum spanning sample-efficient model--ased off off-policy approaches (Ha & Schmidhuberi 2018 Hafner et al. (019) at one extreme and cime-efficient on on-policy approaches leveraging parallel simulation at the other extreme (Rudin et al. 2022 Xu et al. [022)) Particularly in high-dimensional domains with complicated environment dynamics and task objectives, complex trade-off's between representational capacity, exploration capabilities, and optimization accuracy commonly arise.
Continuous state and action spaces yield particularly challenging exploration problems due to the vast set of potential trajectories they induce. Significant research effort has focused improving
"Correspondence to t.seddemii..comm edu. 'Equal advising.
Projecc wehsite: https: aistt..tes...... wtt.aaeeeessgloseoooooooeoo www.ge...o//vieeeoooooooeooo. http://vvwwwweeooopop// edd -et.eedtoooorsessse.ommmm

---

Published as a conference paper at ICLR 2023
efficiency through representation learning in the context of model-frce abstraction model-based
planning (Ha & Schmidhuber 2018 Srinivas et al. 2020; Wulfmeier et al.] 2021), I guided explo-
ration via auxiliary rewards (Osband et al. 2016 Pathak et al. [017]|Sekar et al. 2020 Seyde et al.)
2022b), or constrained optimization particularly to stabilize learning with actor-critic approaches (Schulman al. 2013 |Haaroja et al. 2018 Abdolmaleki al.] 2018). However, recent results have shown that competitive performance can be achieved with strongly reduced, discretized ver- sions of the original action space (Tavkooii al. 2018 Tang & Agrawal 2020; Seyde al. 2021).
This opens the question whether tasks with complex high-dimensional action spaces can be solved using simpler critic-only, discrete action-space algorithms instead. A potential candidate is Q- learning which only requires learning a critic with the policy commonly following via e-greedy or Boltzmann exploration (Watkins Dayan 1992, Mnih et al. 2013). While naive Q-learning strug- gles in high-dimensional action spaces due to exponential scaling of possible action combinations, the multi-agent RL. literature has shown that factored value function representations in combina- tion with centralized training can alleviate some of these challenges (Sunchag et al. [0177; Rashid itt al. 2018), further inspiring transfer to single-agent control settings (Sharma et al. 2017 Tavakoli 2021 J. Other methods have been shown to enable application of critic-only agents continuous ac- tion spaces but require additional, costly, sampling- based optimization (Kalashnikov et al.) 2018).
We build insights at the intersection of these methods to show that surprisingly straightforward variation of deep Q-learning (Mnih et al. 2013), within the framework of Hypergraph Q- Q-Networks (HGQN) (Tavakoli et al. 2021), can indeed solve various state- and pixel-based single-agent con- tinuous control problems at performance levels competitive with state-of-the-ut continuous control algorithms. This is achieved by combination of extreme action space discretization and full value decomposition with extensive parameter-sharing, requiring only small modifications of DQN.
To summarize, this work focuses on the following key contributions:
The DecQN agent simple, decoupled version of DQN combining value decomposition with bang-bang action space discretization to achieve performance competitive with state- of-the-art continuous control actor-critic algorithms on state- and pixel- pixel-based benchmarks.
The related discussion of which aspects are truly required for competitive performance in continuous control as bang-hang control paired with actuator decoupling in the critic and without an explicit policy representation appears sufficient to solve common benchmarks.
An investigation of time-extended collaborative multi-agent bandit settings to determine how decoupled critics leverage implicit communication and the observed state distribution to resolve optimisation challenges resulting from correlations between action dimensions.
2 RELATED WORKS
Discretized Control Continuous control problems are commonly solved with continuous poli- cies (Schulman et al. 2017: Abdolmaleki et al.) 2018 Haarnoja et al. [018 Hafner et al. 2019] Yarats et al. [202)) Recently, it has been shown that even discretized policies can yield compet-
itive performance and favorable exploration in continuous domains with acceleration-level control
TTvakooi et al. 2018 Farquhar et al. 2020 Neunert et al., 2020 Tang & Agrawal 2020 Seyde
ct al. 2021 2022a). When considering discretized policies, discrete action-space algorithms are a
natural choice (Metz et al., al.] 2017 Sharma et al. 2017 Tavakoli (021). Particularly Q-learning based approaches promise reduced model complexity by avoiding explicit policy representations (Watkins) & Dayan 1992). although implicit policies in the form of proposal distributions may be required for scalability (Van de Wiele et al., 2020). We build on perspectives from cooperative multi-agent RL to tackle complex aaglee continuous control tasks with a decoupled extension of Deep Q- Networks (Mnih et al., (2013) over discretized action spaces to reduce agent complexity and to better dissect which components are required for competitive agents in continuous control applications.
Cooperative MARL Conventional Q-learning requires both representation and maximisation over an action space which exponentially grows with the number dimensions and does not scale well to the high-dimensional problems commonly encountered in continuous control of robotic systems. Significant research in multi-agent reinforcement learning (MARL) has focused on improving scal- ability of Q-learning based approaches (Watkins & Dayan 1992). Early works considered indepen-
2

---

Published as a conference paper at ICLR 2023
dent learners (Tan (993). which can face coordination challenges particularly when operating under partial observability (Claus & Boutilier, 1998 Matignon et al. 2012). Some methods approach these challenges through different variations of optumistic updates (Lauer & Riedmiller 2000; Matignon eet al. [007]] |aanait et al., 2006). while importance sampling can assist with off-environment learn- ing (Foerster et al. (017). Furthermore, different degrees of centralization during both training and exccution can address non-stationarity with multiple actors in the environment (Busoniu et al., 2006 Lowe al. 2017; Böhhmer al. 2019). We take a cooperative multi-agent perspective, where individual actuators within single robot aim to jointly optimize a given objective.
Value Decomposition One technique to avoid exponential coupling is value function factoriza-
tion. For example, interactions may be limited to subsets of the state-action space represented by
coordination graph (Guestrin et al.] 2002; Kok & Vlassis 2006). [Schneider et al. (1999) consider
a distributed optimization by sharing return information locally, while Russell & Zimdars (2003)
share value estimates of independent learners with a centralized critic. Similarly, Yang et al. (018)
factorize value functions across interactions with neighbors to determine best response behaviors.
Sunchag et al. (017) introduced Seltue-Decompsttioonn Networks (VDNs) to approximate global value functions as linear combinations of local utility functions under partial observability. Rashid et al. (2018) extend this to non-linear combinations in QMIX, where centralized critic Turther leverages global state information. Son et al. (2019) propose QTRAN to factorizes a transformation of the original value function based on consistency constraint over joint latent representation. These insights can further be transferred to the multi-agent actor-critic setting (Wang et al. 20200] Su al. 2021 Peng al. 1 Our approach scales complex morphologies by leveraging linear factorization over action dimensions within centralized critic conditioned on the global robot state.
Parameter Sharing Efficiency of learning a factored critic can improve through parameter- sharing (Gupta et al. 2017 Böhmer et al. 2020 Christianos et al., 2021) and non-uniform priori- tization of updates (Bargiacchi et al. 2022] l). In particular, the Hybrid Reward Architecture (HRA) proposed by Van Seijen et al. (2017) provides compact representation by sharing a network torso between individual critics (see also (Chu & Ye 2017)). We achieve efficient learning at reduced model complexity through extensive parameter-sharine
within a joint critic that splits into local util- ity functions only at the output neurons, while biasing sampling towards informative transitions through prioritized experience replay (PER) on the temporal-differrnce error (Schaul et al. 2015).
Decoupled Policies Stochastic policies for continuous control are commonly represented by Gaus- sians with diagonal covariance matrix, e.g. (Schulman et al. 2017 Haarnoja et al. 2018 Abdol- maleki et al. 2018), while we leverage decoupled Categoricals as a discrete counterpart. Fully de- coupled discrete control via Q-learning was originally proposed by Sharma et al. (2017) to separate dimensions in Atari. Tavakoli et al. (022I) extend this concept to hypergraph nentwwrkks (HGQN) to leverage mixing across higher-order action subspaces (see also Appendix P.). Here, we demonstrate that learning bang-bank bang controllers, leveraging only extremal actions at the action space boundaries and inclusion of the zero action for bang-off-ban control (Bellman al.) 1956 LaSalle 1960), with full decoupling as a minimal modification to the original DQN agent yields state-of-the-at performance across feature- and pixel-based continuous control benchmarks. Our investigation pro- vides additional motivation for existing work and proposes a straightforward if surprising baseline to calibrate continuous control benchmark performance by pushing simple concepts the extreme.
3 BACKGROUND
We describe the reinforcement learning problem as a Markov Decision Process (MDP) defined by the tuple (S, .A, T, R, 7, where S C N and A c R. denote the state and action space, respec- tively, T: S + S the transition distribution, R S A + I the reward mapping, and y E [0, 1) the discount factor. We define s, and a, to be the state and action at time t and represent the policy by [(a((s))) The discounted infinite horizon return is given by G, Ee 7- R((s, a7). where S141 ~ T[|s,, a.) and a, ~ T( (-)ss). Our objective is to learn the optimal policy maximizing the expected infinite horizon return E.G.) under unknown transition dynamics and reward mappings.
3

---

Published as a conference paper at ICLR 2023
Environment
Critic
Outputs
Prediction
Log Parameter Count
01 -1)
Caliaaa +1)
DecON
Dog
Ciles. +1)
x M
Volen
Q-Network
N-M outputs
Velker
(5, -1)
Figure 2: Decoupled Q-Network (DecQN) with bang-bang actions (2 bins per action dimension). The network predicts one state-action value for each decoupled action based on the observed state. During training, selection proceeds either via indexing for the observed actions at the current timestep (green) or via decoupled maximization along each action dimension at the next timestep. simple e-greedy policy is recovered based on the decoupled argmax over the Q-function.
3.1 POLICY REPRESENTATION
Current state-of-f ofttt-----h--rr algorithms for continuous control applications commonly consider the actor- critic setting with the policy modelled as a continuous distribution TT(aa) (ss) independently from the value estimator (0(st,a;) or Vo(*)). both represented by separate neural networks with parame- ters Φ and 0. Recent results have shown that comparable performance can be achieved with these approaches when replacing the continuous policy distributions by discretized versions (Seyde al. 2021), at the potential cost of principled ways to represent stochastic policies. One may then ask whether we require the actor-critic setting or if simpler discrete control algorithms are sufficient. Qeearnnigg presents itself as a lightweight alternative by only learning the value function and recov- ering the policy by c-greedy or Boltzmann exploration, side-stepping explicit policy gradients.
3.2 DEEP Q-NETWORKS
We consider the framework of Deep Q-Networks (DQN) (Mnih et al. 2013). where the state action value function Qo(st, 0z) is represented by a neural network with parameters 0. The parameters updated in accordance with the Bellman equation by minimizing the temporal-difference (TD) error. We leverage several modifications that accelerate learning and improve stability based on the Rainbow agent (Hessel et al. 2018) implemented in Acme Hoffman al. (2020).
In particular, we leverage target networks for improved stability in combination with double Q- learning to mitigate overestimation (Mnih et al.) 2015) Van Hasselt et al. 2016). We further employ prioritized experience replay (PER) based on the temporal difference error to bias sampling towards more informative transitions and thereby accelerate learning (Schaul et al. 2015). Additionally, combine rewards over several consecutive transitions into multi-step return to improve stability of Bellman backups (Sutton & Barto 2018). A more detailed discussion of these aspects is included in Appendix G while we provide ablation studies on individual components in Appendix E
Combining these modification of the original DQN agent, optimize the following loss function
[( Σ Lo(ue (0)s
where action evaluation employs the target yr = Σ5= (5(( ac+s) + To (st+ns ditr), action selection uses a;+1 arg maxa Qo(s++1,a), Ls(.) is the Huber loss and the batch size is B.
4 DECOUPLED Q-NETWORKS
The Rainbow agent (Hessel al. 2018) from the previous section provides framework for learning high-quality discrete control policies. However, conventional Q-learning methods enumerate the action space and can therefore become inefficient when attempting scale to high dimensions.
4

---

Published as a conference paper at ICLR 2023
4.1 MULTI-AGENT PERSPECTIVE
Scaling Q-eaarnin to high-dimensional action spaces has been studied extensively in MARL set- tings. Particularly in cooperative MARL it is common to consider factorized value functions in combination with centralized training and decentralized execution (CTDE). Factorization reduces the effective dimensionality from the perspective of individual agents by learning utility functions conditioned on local observations. Centralized training considers global information by composing individual utilities and optimizing for consistent behavior among agents. Decentralized execution then assumes that the local utility functions are sufficient for optimal distributed action selection.
4.2 FACTORIZED Q-FUNCTION
Inspired by the MARL literature, we consider our agent to be a team of individual actuators that aim to cooperatively solve the given objective. We further assume that the overall state-action value function Q( a₂) can be locally decomposed along action dimensions into a linear combination of M single action utility functions (8,, af). This simply applies the principle of value decomposi- tion (Sunchag al., (2017) and fits within the general concept of HGQN-type algorithms (Tavakoli
itt al. (2021) as simplified variation of the base case without a hypergraph as was also leveraged by Sharma et al. 2017 for Atari. The overall state-action value function is then represented as
M Q6((),a)) M
Q0(8,, an) =
(2)
where each utility function conditioned on the global robot state to facilitate actuator coordination via implicit communication as we observe in our experiments. This is further assisted by a high- degree of parameter sharing within unified critic (Van Seijen et al., (017)) that only splits to predict decoupled state-action utilities at the outputs. The linear combination of univariate utility functions then allows for efficient decomposition of the argmax operator
arg max Qa(s;, a.)) = -(arx arg max Q7 at).... arg max Q (Be, of d,
o!
of
where each actuator only needs to consider its own utility function. Global optimization over a, then simplifies into parallel local optimizations over af. Training still considers the entire robot state and all joint actions within the centralized value function, while online action selection is decoupled. Inserting this decomposition into the Bellman equation yields decoupled target representation
(86411,44441) M
yt r(st, az) +7
max
We can then insert the factorized value function of Eq, 2 and the decoupled target of Eq. 4 into Eq. We also considered composition based on learned weights inspired by QMIX (Rashid a (018)) Our findings suggest that access to the robot state throughout local utility functions outweighs the potential gains of more complex combinations, see also (Tavakoli et al., 202T) for a more detailed discussion monotonic mixing functions. The following section further discusses the importance of shared observations and state-space coordination based on illustrative matrix games.
EXPERIMENTS
The DecQN agent solves various continuous control benchmark problems at levels competitive with state-of-the-arr continuous control algorithms. First, we provide intuition for how DecQN reacts to coordination challenges based on illustrative matrix games. Then, we show results for learning state-based control on tasks from the DeepMind Control Suite (Tunyasuvunakool et al.) 2020) and
MetaWorld YYu et al.) 2020 including tasks with action dimension (A) = 38 and compare to the state-of-the-ar DMPO and D4PG agents (Abdolmaleki et al.
2018 Bellemare et al. 2017
Barth-Maron et al. 2018). We further provide results for pixel- based control on Control Suite tasks
and compare to the of-te-o-thhear DrQ-v2 and Dreamer-v2 agents Yaarat et al. 2021 Hafner al. 2020). Lastly, we discuss qualitative results for learning a velocity -trcctt-tracingg controller on simulated
Mini Cheetah quadruped in Isaac Gym (Makoviychuk al. 2021) highlighting DecQN's versatility.
5

---

Published as a conference paper at ICLR 2023
Two-step Game
Penalty Game
Rewarr
PQN
DeaQN
DecQN
Decolu
Reward
DecQN
Docone DCN
3
C- 2.0
1.0
0.5
1K-stop
State Stnle
2G
Ageent
Agonc
0
-1
3
-6
1
3
A 5
3
-1
0
-50
-23 -20
-10
Figure 3: DecQN and DQN cooperative matrix games. Left: two-step game where agent selects in step I which payoff matrix is used in step (top vs. bottom). Learned Q-values of DecQN indicate that accurate values around the optimal policy are sufficient (e 0.5) even when the full value distribution cannot be represented well (e = 1.0). Right: matrix game with actions as acceler- ation input to pointmass (x vs. y). DecQN struggles to solve the 1-step game (no dynamics). In the multi-step case, DecQN leverages velocity information to coordinate action selection (middle).
5.1 ILLUSTRATIVE EXAMPLES: COORDINATION IN MATRIX GAMES
Representing the critic as a linear decomposition across agents enables scaling to high-dimensional tasks while limiting the ability to encode the full value distribution (Sunehag et al. 2017 Rashid et al. 2018). This can induce coordination challenges (Lauer & Riedmiller [2000] Matignon et al., 2012. Here, we consider cooperative matrix games proposed by Rashid et al. (2018) and Claus & Boutilier 1998) to illustrate how DecQN still achieves coordination in a variety of settings.
Two-step game: We look at 2-player game with three states proposed by Rashid et al. (2018) (0.8) In state I, agent playing action A/B transitions both agents to state 2/3 without generating reward. In states 2 or 3, the actions of both agents result in the cooperative payout described in Figure (left). Rashid et al. (2018) showed that a VDN-type decomposition is unable to learn the correct value distribution under uniform policy, which our results confirm for DecQN with However, DecQN solves the task for non-uniform sampling with c = 0.5 by learning an accurate value distribution around the optimal policy (right). In online RL settings, agents actively shape the data distribution and learning to differentiate behavior quality is often sufficient for solving a task.
1.0 (middle).
In the following we consider cooperative 2-player matrix games based on Claus & Boutilier (1988). We further augment the tasks by point-mass dynamics, where each agent selects acceleration input along the x or y-axis. We evaluate three variations: "I-step" resets after 1 step (no dynamies); "1 "1K-step" resets after 1000 steps (dynamics); "Optimism" runs "I "IK-step" with optimistic updating inspired by Lauer & Riedmiller (2000). Each agent selects from three actions and we set c 0.5.
Climbing Game
Penalty game: The reward structure defined in action-
Agent 2 (v)
DecON:
-1Kstep
space and provided in Figure 3 (right). The two maxima at (21,0)) = ((-1,-1), ++1,+I) and minima at (a1, az) (+1,-1)) induce a Pareto-selection problem that re- quires coordination. DecQN is unable to consistently solve the 1-step game for high costs i while it does solve the multi-step variation. In the latter, we find that DecQN coordinates actions (acceleration) based on state information (velocity). Computing action outputs over varying velocity inputs indicates a clear cor- relation in directionality across the unambiguous velocity pair- ings (middle). This underlines the importance of shared obser- vations to achieve coordination with decoupled agents.
Reward
Oppdmism
--stoop
-1steep
Agent (6)
0
Action Space
10
Return
State Space
Retur
-10
Climbing game: The reward structure defined in Figure 4 with action (top) and state space (bottom) rewards. In action space, Nash equilibria (a), 02) = ((-1,-111) -1), (0, 0)} are shad- owed by action (a1, a2) = 1)], which has higher min- imum gain should one agent deviate (Matignon et al. 2012). DecQN often settles at the sub-optimal action (2) (0,0) to avoid potential cost of unilateral deviation, which can be re- solved by optimistic updating. However, this can impede learn-
Cost k
Figure 4: Climbing game with action (top) or state space (bot- tom) reward. Optimism helps DecQN in action space, while DecQN matches DQN perfor- mance under state space reward.

---

Published as a conference paper at ICLR 2023
1900
Cartpote Seinqup
3003
Carppole Switquu Spars
3000
Seacher Hard
3000
Finger Sp
Finger Turn
255
1w0
Cheeraah Run
V400
Visiser Mowte
Wolllee Rum
Quedruped Run
190
Iuumanot Stand
250
250
254
253
Itumanssd Maat
Hlumanord
Waa
Irot
Hun
30C
300
400
4000
180
100
300
2006
2004
00
65
13
15
16
66
10 (M]
25.
(M)
25
Timestop [M]
15
29
10
14
[M]
15
20
DecON Bin)
D4PG
DHPO
Figure 5: Performance for state-based control on DeepMind Control Suite and MetaWorld tasks. We compare DecQN with discrete bang-off-ban policy the continuous D4PG and DMPO agents. Mean and standard deviation are computed over 10 seeds with single set of hyperparamneters. DecQN yields performance competitive with state-of-the-art continuous control agents, scaling to high-dimensional Dog tasks via a simple decoupled critic representation and an é-greedy policy.
ing in domains that require state-space exploration as highlighted for state-space reward (bottom). We there find DecQN and DQN to achieve similar performance. In fact, the tasks considered in this paper do not require coordination at the action level but rather in higher-level integrated spaces.
Generally, our objective is learning control of articulated agents, which requires temporarily- extended coordination in state space. We find that while a decoupled critic may not be able to represent the full action-value value distribution, it is often sufficient to differentiate action quality around the optimal policy. In particular, centralized learning combined with shared observations enables actuator coordination in complex, sith-dmmnnsional domains we observe in the next section.
5.2 BENCHMARKING, CONTINUOUS CONTROL FROM STATE INPUTS
We evaluate performance of the DecQN agent on several continuous control environments from the DeepMind Control Suite as well as manipulation tasks for the Sawyer robot from MetaWorld. We consider a Bang-Off-Baae policy encoded via 3-bin Categorical distribution and use the same set of hyperparameters across all tasks. Performance mean and standard deviation across 10 seeds are provided in Figure Jadd we compare to the recent DMPO and D4PG agents, with additional train- ing details provided in Appendix [A] We note that DecQN exhibits very strong performance with respect to the continuous baselines and sometimes even improves upon them, despite operating over a heavily-constrainee
action subspace. This extends even to the high-dimensional Humanoid and Dog domains, with action dimensions dim(A) 21 and dim(A) 38, respectively. Significant research effort has produced highly capable continuous control algorithms through advances in rep- resentation learning, constrained optimization and targeted exploration. It is then surprising that simple decoupled critic operating over a discretized action space and deployed with only an e-greedy policy is able provide highly competitive performance.
5.2.1 SCALABILITY OF THE DECOUPLED REPRESENTATION
To put the benchmarking results into perspective, compare DecQN to the non-decoupled DQN agent for a 3-bin discretization each. DQN conventionally enumerates the action space and scales exponentially in the action dimensionality, whereas DecQN only scales linearly (see also Figure] 2). According to Figure 66 performance of DecQN and DQN remains comparable on low-dimensional tasks, however, for high-dimensional tasks the DQN agent quickly exceeds memory limits due the
7

---

Published as a conference paper at ICLR 2023
1000
Cartpole Swinquu
Rincer Spin
Viaker Wale
Quadruped Ran
Humannlc Walk
Welt
750
800
Timestep [M]
Timestep (M]
Tiresteo [M]
Timestep [M]
Timestep [M)
Timestep [N]
DergN (lm)
Dheogg (21 Bing
DON E3 limn
Figure 6: Comparison between DecQN and regular DQN on a selection of Control Suite tasks. De- coupling enables scaling to complex environments where DQN fails (dotted blue = memory error). DecQN's linear scaling further allows for efficient learning of fine- fine-grained control (21 bins, green).
Cartpote Seingup
Carppore Swingup Sparse
Ball In Casch
Roacher Hard
Finger Spi
1900
1000
150
350
214
203
90
56 68
10
0.
02
04
E.2
1.
0.2
0.4
8.5
96
13
28
49
04
283
00
58
300 350
X3
150
X0
Timestep
Timestep
LM]
[M]
DecON Bino
DrQ V2
Dresmer-2
Figure/ 7: Performance pixel-based control tasks from the DeepMind Control Suite, comparing DecQN with bang-bann bang policy to the continuous DrQ-v2 and Dreamer-v2 agents. We note that DecQN successfully accommodates the additional representation learning challenges. The best per- forming runs on Humanoid indicate that DecQN can efficiently solve complex tasks from vision, po- tentially requiring environment-speccin hyperparameter settings or more sophisticated exploration.
large number of output parameters required. This contrast is amplified when significantly increasing granularity of DecQN's discretization to 21 bins. Generally, increasing granularity increases the action space and in turn yields more options to explore. While exploration over finer discretizations is likely to delay convergence, we observe highly competitive performance with both 3 and 21 bins.
5.3 BENCHMARKING CONTINUOUS CONTROL FROM PIXEL INPUTS
We further evaluate on a selection of visual control tasks to see whether DecQN's decoupled discrete approach is compatible with
Table I: Per-learning iteration
runtime for Quadruped Run
learning latent representation from visual input. To this end, we combine DecQN with convolutional encoder trained under shift- augmentations as proposed by Yarats et al. (202T). Figure 7 com- pares DecQN with a bang-bang parameterization (2-bin) to the state-of-thh-art model-free DrQ-v2 and model- based Dreamer-v2 agents. DecQN and Dreamer-v2 employ a fixed of hyperpa- rameters across all tasks, while DrQ-v2 leverages a few task-specific variations. We note that DecQN highly competitive across several environments while relinquishing some performance on the complex Humanoid task. Scaling DecQN to efficient learning of latent representation for high-dimensional systems under action penalties will be an interesting challenge. As the highest performing runs of DecQN on Humanoid Walk reach competitive performance, we are optimistic that this will be viable under potential parameter adjustments or more sophisticated exploration strategies. As both DrQ-v2 and Dreamer-v2 provide limited data Quadruped Run (3M and 2M timesteps, respectively), we further re-ran the authors' code on our hardware and record average tim- ings for one cycle of data collection and training. We observe favorable runtime results for DecQN in Table T and estimate that code optimization could further reduce timings as observed for DrQ-v2.
Agent DecQN DrQ-v2
Runtime [s]
10.0 = 0.1 10.7 + 0.2
Dreamer-v2
29.0 + 0.8
5.4 COMMAND-CONDITIONE LOCOMOTION CONTROL
We additionally provide qualitative results for learning a command-conditioned locomotion con- troller simulated Mini Cheetah quadruped to demonstrate the broad applicability of decoupled
8

---

Published as a conference paper at ICLR 2023
Chuatal in Isanc Gym
Beference Tracking Ciicle
X emeor
Beference Tracking Stap
e
eeeor [m)
e [m]
DECON
[m]
Time [s]
[m]]
Time (s)
Figure 8: Qualitative results for learning command-conditioed locomotion controller on simu-
lated Mini Cheetah. The agent learns to predict position targets for joint-level PD controllers with
the objective of tracking commanded base velocities. We observe good tracking performance for
following references in state-space at the level of PPO, which a common choice in these settings.
discretized control. We consider a minimal implementation of DecQN in PyTorch simulated with Isaac Gym. The agent aims to track velocity commands by predicting position targets for the leg joints, which are then provided to a low-level proportional-derivayy (PD) controller. As targets are generated in joint angle space, we consider a 31-bin discretization. We evaluate the resulting command-conditionee policy based on tracking performance when following a reference position trajectory. To this end, velocity commands are generated by simple proportional controller on the position error between the reference and the robot base. Figure X showcases tracking for two refer- ences. The circular trajectory has diameter of 5 m and is tracked at velocity of 1 m/s (middle), while the step trajectory introduces lateral jumps of 0.5 m and is tracked at a velocity of 0.7 m/s (right). In both scenarios we observe good tracking performance at levels competitive with the com- monly used PPO agent (Rudin et al. 2022). The system reacts well even to strong jumps in the reference with quick recovery in response to slight overshooting. This underlines the versatility of simple decoupled discrete control, learning policies for various control types including application of acceleration or velocity control and generating position targets for downstream control systems.
CONCLUSION
Recent successes in data-efficient learning for continuous control have been largely driven by actor- critic algorithms leveraging advances in representation learning, constrained optimization, and tar- geted exploration. In this paper, we take step back and investigate how simple variation of Q-lernning with bang- bang action discretization and e-greedy exploration compares to current state- of-the-art algorithms on a variety of continuous control tasks. Decoupled Q-Networks (DecQN) discretizes the action space, fully decouples individual action dimensions, and frames the single- agent control problem as a cooperative multi-agent optimization. This is then solved via centralized training with decentralized execution, where the decoupled critic can be interpreted as discrete analog to the commonly used continuous diagonal Gaussian policies. We observe highly compet- itive performance of DecQN on state-based control tasks while comparing to the state-of-the-an D4PG and DMPO agents, extend the approach to pixel- based control while comparing to the recent DrQ-v2 and Dreamer-v2 agents, and demonstrate that the method solves more complex command- conditioned tracking control on simulated Mini Cheetah quadruped.
DecQN is highly versatile, learning capable policies across variety of tasks, input and output modalities. This is intriguing, as DecQN does not rely on many of the ingredients associated with state-of-the-art performance in continuous control and completely decouples optimisation and ex- ploration across the agent's action dimensions. We provide intuition for how the decoupled critic handles coordination challenges by studying variations of matrix games proposed in prior work. We observe that shared access to the global observations facilitates coordination among decoupled ac- tuators. Furthermore, is often sufficient for critic to accurately reflect relative decision quality under the current state distribution to find the optimal policy. Promising future work includes the interplay with guided exploration and the investigation of these concepts on physical systems.
There is great value in contrasting complex ofate-of-the-art algorithms with simpler ones to dissect the importance of individual algorithmic contributions. Identifying the key ingredients necessary to formulate competitive methods aides in improving the efficiency of existing approaches, while providing better understanding and foundations for future algorithm development. To this end, the DecQN agent may serve straightforward and easy to implement baseline that constitutes a strong lower bound for more sophisticated methods, while raising interesting questions regarding how to assess and compare continuous control benchmark performance.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, and Daniela Rus were supported in part by the Office of Naval Research (ONR) Grant N00014-18-1-2830 and Qualcomm. This article solely reflects the opinions and conclusions of its authors and not any other entity. We thank them for their support. The authors further would like to thank Danijar Hafner for providing benchmark results the Dreamer-v2 agent, and acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posterioni policy optimisation. arXiv preprint arXiv:1806.069: 2018.
Eugenio Bargiacchi, Timothy Verstraeten, and Diederik M Roijers. Cooperative prioritized sweep- ing. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 160-168, 2021.
Gabriel Barth-Maron, Matthew W Hoffmman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint 1rXiv
2018.
Marc G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement
learning. International Conference on Machine Learning, pp. 449-458. PMLR, 2017.
Richard Bellman, Irving Glicksberg, and Oliver Gross. On the "bang-bang" control problem. Quar-
terly of Applied Mathmmatis, 14(1):11-18, 1956.
Wendelin Böhmer, Tabish Rashid, and Shimon Whiteson. Exploration with unreliable intrinsic reward in multi-agent reinforcement learning. arXiv preprint arXiv:1906.02/18,, 2019.
Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In International
Conference on Machine Learning, pp. 980-991. PMLR, 2020.
Lucian Busoniu, Bart De Schutter, and Robert Babuska. Decentralized reinforcement learning con- trol of a robotic manipulator. In 2006 9th International Conference Control, Automation, Robotics and Vision, pp. 1-6. IEEE, 2006.
Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano Albrecht. Scal- ing multi reinforcement learning with selective parameter sharing. In International Confer- ence on Muchine Learning, pp. 1989-1998. PMLR, 2021.
Xiangxiang Chu and Hangjun Ye. Parameter sharing deep deterministic policy gradient for cooper- ative multi-agent reinforcement learning. arXiv preprint arXiv: 2017.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI/IAAI, 1998:2, 1998.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning, In International conference on machine learning, pp. 1096- 1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 32, 2018b.
Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnaeve. Growing action spaces. In International Conference on Machine Learning, pp. 3040- 3051. PMLR, 2020.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International conference on machine learning, pp. 1146-1 155. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2023
Carlos Guestrin, Michail Lagoudakis, and Ronald Parr. Coordinated reinforcement learning. In ICML, volume 2. pp. 227-234. Citeseer, 2002.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agen control using deep reinforcement learning. In International conference on autonomous agents and multiagent systems. pp. 66-83. Springer, 2017.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint
2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint 1rXXvv:8/2.00000 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1912v.1912.01600 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02 2020.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In ICML, 2022.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine. Andrew Cowie. Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006..mm 00979, 2020. URL htlLhttps://arxe org/
abs/2006. 00979
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scal- able deep reinforcement learning for vision-based robotic manipulation. ArXiv, 0bss/1806.002/ 2018.
Benjamin Katz, Jared Di Carlo, and Sangbae Kim. Mini cheetah: A platform for pushing the limits of dynamic quadruped control. In 2019 International Conference Robotics and Automation (ICRA), pp. 2295-6301. IEEE, 2019.
Jelle R Kok and Nikos Vlassis. Collaborative multiagent reinforcement learning by payoff propaga- tion. Journal of Machine Learning Research, 7:1789-1828, 2006.
Joseph P LaSalle. The "bang-bang" principle. IFAC Proceedings Volumes, 1(1):503-507, 1960.
Martin Lauer and Martin Riedmiller. An algorithm for distributed reinforcement learning in coop- erative multi-agent systems. In In Proceedings of the Seventeenth International Conference Machine Learning. Citeseer, 2000.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compettivee environments. Advances in neural informa- fion processing systems, 30, 2017.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint Ttt::2008..14777 2021.
Laëtitia Matignon. Guillaume Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: al- gorithm for decentralized reinforcement learning cooperative multi-agent teams. In 2007 IEFF/RS International Conference Intelligent Robots and Systems, pp. 64-69. IEFF, 2007.

---

Published as a conference paper at ICLR 2023
Lactitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforcement learn- ers in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review, 27:1-31, 2012.
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint / 705.05035, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13Xiv::312.5000 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 518:529 533, 2015.
Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeeier Thomas Lampe, Tobias Springen- berg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller. Continuous-discrtt reinforcement learning for hybrid control in robotics. In Conference on Robot Learning, pp. 735-751. PMLR, 2020.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, Advances in neural information processing systems, 29, 2016.
Liviu Panait, Keith Sullivan, and Sean Luke. Lenient learners in cooperative multiagent systems. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pp. 801-803, 2006.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervvsed prediction. In International conference on machine learning, pp. 2778-2787, PMLR, 2017.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gra- dients. Advances in Neural Information Processing Systems, 34, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp. 4295-4304. PMLR, 2018.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp. 91-100. PMLR, 2022.
Stuart J Russell and Andrew Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 656-663, 2003.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:/5/ 2015.
Jeff G Schneider, Weng- Keen Wong, Andrew W Moore, and Martin A Riedmiller. Distributed value functions. In ICML, 1999.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Ramanan Sekar. Olch Rybkin, Kostas Danilidiis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via culf-super world models. In International Conference on Machine Learning, pp. 9583-8592 PMLR, 2020.
12

---

Published as a conference paper at ICLR 2023
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 34, 2021.
Tim Seyde, Wilko Schwarting, Igor Gilitschenski, Markus Wulfmeier, and Daniela Rus. Strength through diversity: Robust behavior learning via mixture policies. Conference on Robot Learn- ing. pp. 1 1144-1155, PMLR, 2022a.
Tim Seyde, Wilko Schwarting, Sertac Karaman, and Daniela Rus. Learning to plan optimistically: pnncertaintygguidee deep exploration via latent model ensembles. In Conference on Robot Learn- ing. pp. 1156-1167. PMLR, 2022b.
Sahil Sharma, Aravind Suresh, Rahul Ramesh, and Balaraman Ravindran. Learning to factor poli- cies and action-value functions: Factored action space representations for deep reinforcement 2017.
learning. arXiv preprint
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. Interna- lional Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.041.66 2020.
Jianyu Su, Stephen Adams, and Peter Beling. Value-decomposition multi-agent actor-critics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1 11352-11360, 2021.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decompositionn networks for cooperative multi-agent learning. arXiv preprint arXiv: 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In Proceedings of the tenth international conference on machine learning. pp. 330-337, 1993.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5981-5988 2020.
Arash Tavakoli. On structural and temporal credit assigument in reinforcement learning. PhD thesis, Imperial College London, 2021.
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Arash Tavakoli, Mehdi Fatemi, and Petar Kormushev. Learning to represent action values as hypergraph on the action vertices. In International Conference on Learning Representattonss, 2021.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 100022 2020.
Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enor- mous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08116, 2020.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Hann Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Bames,, and Jeffrey Tsang. Hybrid reward architecture for reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2023
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off Off-policy multi-agent decomposed policy gradients. International Conference on Learning Representa- fions, 2020.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Markus Wulfmeier, Arunkumar Byravan, Tim Hertweck, Irina Higgins, Ankush Gupta, Tejas D. Kulkarni, Malcolm Reynolds, Denis Teplyashin, Roland Hafner, Thomas Lampe, and Martin A. Riedmiller. Representation matters: Improving perception and exploration for roboties. 202/ IEEE International Conference on Robotics and Automation (ICRA), pp. 6512-6519, 2021.
Jie Xu, Viktor Makoviychuk, Yashraj Narang. Fabio Ramos, Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. Interna- tional Conference on Learning Representations, 2021.
Yaodong Yang. Rui Luo, Minne Li. Ming Zhou, Weinan Zhang, and Jun Wang. Mcan field multi- agent reinforcement learning, In International Conference on Machine Learning, pp. 5571-5580. PMLR, 2018.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:// github www.c////mniic .ct.///eeaassyartttppyr.oo..cc.a sac 2020.
Denis Yarats, Rob Fergus, Alessandro Lazarie, and Lerrel Pinto. Mastering visual continuous trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 2021.
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning. pp. 1094-1100 1100 PMLR, 2020.
14

---

Published as a conference paper at ICLR 2023
A TRAINING DETAILS
Experiments Control Suite and Matrix Game tasks were conducted on single NVIDIA V100 GPU with 4 CPU cores (state-based) or 20 CPU cores (pixel-based), Experiments MetaWorld and single NVIDIA 2080Ti with CPU cores. Benchmark performance of DecQN is reported in terms of the mean and one standard deviation around the mean on 10 seeds. We implemented DecQN within the Acme framework (Hoffman al., 2020) in TensorFlow. For the Mini Cheetah (Katz, et al. 2019) task we implement a version of DecQN in PyTocch.
Isaac Gym were conducted
Hyperparameters We provide hyperparameter values of DecQN used for benchmarking in Ta- ble 2] A constant set of hyperparameters used throughout all experiments. with modifications to the network architecture for baseen-based tasks. For the matrix games, we further set the n-step parameter to account for the underlying timescale and direct impact that actions have on both the state transition and reward. Results for DQN were obtained with the same parameters without decoupling. The baseline algorithms used the default parameter settings provided in the official implementations by the original authors, within the Acme library in the case of D4PG and DMPO.
Table 2: DecQN hyperparameters for state- and pixel-based control.
Parameter
Value [State]
Value [Pixel]
Optimizer
Adam
Adam
Learning rate
1 x 10-4
x 10-4
Replay size
1 x 106
10
n-step returns
3
3
Action repeat
1
Discount y
0.99
0.99
Batch size
256
256
Hidden size
500
1024
Bottleneck size
100
Gradient clipping
40
40
Target update period
100
100
Imp. sampling exponent
0.2
0.2
Priority exponent Exploration
0.6
0.6
0.1
0.1
Architecture For the state-based experiments, we leverage fully-connected architecture that in- cludes single residual block followed by a layer norm. For the vision-based experiments, we leverage the convolutional encoder with bottleneck layer from Yarats et al. (202T) followed by two fully-comnected layers. In both cases, fully-connected output layer predicts the decoupled state- action values so that the torso up to the final layer remains shared among the decoupled critics.
Action discretization The continuous action space is discretized along each action dimension into evenly spaced discrete actions including the boundary values. Assuming symmetric action bounds, for axis this yields bang-bang control in the case of 2 bins with Anaum and bang-off- bann control in the case of 3 bins with An = Opound,i, i, 0, + abound,5 } and so on.
ib is Cbound, ,
Decoupling The decoupling based on value decomposition (Sunchag et al.) 2017) only requires small modifications of the original DQN structure (see also Sharma al. (2017)) The output layer size is adapted to predict all state-action utilities for action dimensions n. and discrete bins n, to yield output size [..., ng). where [...] indicates the batch dimensions. This reshaped into [..., Pa, r2pl, where the combined state-action value is computed either by indexing with the observed
bin at the current state or maximizing over bins at the next state, followed by a mean over action dimensions to recover state-action values.
The Aeme library is distributed under the Apache 2.0 license and available GitHub The D4PG and DMPO agents are part of the Aeme library. while both DecQN and DQN for con- tinuous control were implemented within the Acme framework. Both Dreamer-v2 and DrQ-v2
Licenses
15

---

Published as a conference paper at ICLR 2023
Episode 50
Episode 500
Episode 1000
State 2
State 3
State 2
State 3
State 2
State 3
0.3
=
Agent 2
A-
1.2
-1.3
-0.3
0.2
1.9
-0.5
-0.5
0.5
6.9
7.4
-2.8
3.0
0.1
-2.4
0.5
1.0
0.6
-1.8
0.4
1.4
6.8
7.3
1.8
7.6
0.2
E
Agent
A
-0.2
-0.3
-0.3
-0.3
0.7
0.5
-0.2
-0.1
6.6
6.1
-2.5
2.4
Occuuancy
0.3
0.1
0.9
0.8
1.3
1.0
1.4
1.6
7.4
6.9
2.6
7.5
0.1
=
Agent
A
0.4
-1.2
-0.3
-2.4
1.6
0.1
0.0
-1.3
6.8
6.6
-1.6
2.4
2.7
1.1
-0.6
3.7
1.8
7.0
6.8
2.0
6.0
0.0
A
B
A
A
A
A
Agent 1
Agent 1
Agent
Agent
Agent
Agent
Figure 9: State distribution and action-values for DecQN on the two-step game from Section 5.1 We consider 3 training stages pre-convergence across 5 seeds. Color indicates cumulative state-action occurrence, numbers represent predicted mean action values. While the decoupled agent struggles to represent action values under a uniform distribution (c = 1.0, bottom), it accurately represents values under the current policy when directly influencing the state distribution (e 0.5, top).
distributed under the MIT license and are available with their benchmarking results on GitHub here and here respectively.
B STATE-ACTION DISTRIBUTION" TWO-STEP GAME
Expanding on the discussion of the two-step matrix game in Section we provide state-action distribution data for the DecQN agent in Figure We evaluate three stages of early training before convergence for seeds. Color of state-action pairings indicates cumulative frequency within replay memory, while numbers represent mean predicted state-action values. We consider different degrees of randomness in the policy by varying the parameter. It can be noted that the agent fails to accurately represent the optimal state-action value state when learning based on a uniform state- action distribution (bottom row). However, enabling the agent to directly influence the underlying distribution through its policy results in accurate learning of state-action values around this policy (top and middle rows). While the decoupled agent may not be able to accurately reflect the full state-action value function, it can be sufficient to do so over a subspace relevant for solving the task.
C ENVIRONMENTS
brief overview of the environments evaluated throughout this paper is provide in Table 3 grouped based their distributors. In particular, highlight the dimensionality of the state space S and action space A as well as the total number timesteps used for training on each environment. For the Mini Cheetah task, note that the number of steps marked with denotes the approximate number of steps per environment, where we use 1024 parallel environments within Isaac Gym.
D ADDITIONAL. BASELINES
We provide additional baselines on a selection Control Suite environments Figure 10 The baselines consist of continuous-contre algorithm with Categorical head based on MPO from Seyde ett al. (202I), the contnuuous-oontr SAC agent from Yarats & Kostrikov (2020), as well as the critic-only methods QT--Opt and AQL-Seq based on data from Van de Wiele al. (2020). DecQN is
16

---

Published as a conference paper at ICLR 2023
Table 3: Description of benchmark environments used throughout the paper.
Suite
Task
dim(S)
dim(AA
Steps [State]
Steps [Pixel]
Control Suite
Ball in Cup Catch
2
1 x 106
Cartpole Swingup
1 106
1 x 106
Cartpole Swingup Sparse
4
1
2 X 106
1 x 106
Cheetah Run
18
6
3 X 106
1 x 106
Dog Run
158
38
5 106
Dog Trot
158
38
5 X 106
Dog Walk
158
38
5 x 106
Finger Spin
2 106
1 X 106 1 X 106
Finger Turn Hard
2
2 x 106
Humanoid Run
54
21
30 X 106
Humanoid Stand
10 10
Humanoid Walk
54
21
20 10
30 10c
Quadruped Run
56
12
5 x 106
10 X 106
Reacher Hard
2
2 106
1 X 106
Walker Run
18
6
3 106
Walker Walk
18
6
1 1 106
x 106
Isaac Gym
Mini Cheetah Tracking
48
12
3 x 106+
Matrix Games
Two Step
6 104
Penalty
9
2 105
Climbing
9
2 X 10
Meta World
Assembly
39
4
2 x 106
Door Open
39
4
2 2 106
Drawer Open
2 106
Hammer
39
4
2 106
Pick Place
39
4
2 106
17

---

Published as a conference paper at ICLR 2023
Seprre Swinguu
Prnger Spmn
Walkrr Walk
Qusoruped Run
HIrmaancc Walk
Walk
644
64
50
18
14
5.6
10
L8
23
20
30
45
60
130
68
60
[M]
Timestep (M]
Dectg
MPD SAC (Comt.)
QT-dipi
Alll-Sww [2wi]
Figure 10: Comparison of DecQN to MPO with Categorical policy head, continuous SAC, as well critic-only Qp-Opt and AQL-Seq. DecQN displays state-of-the-art performance without relying on actor-critic methods or continuous control, while remaining sample-efficient in high high-dimensional action spaces by avoiding sampling-basee methods.
Cartpole Sainnup
Finger Spin
Walker Walk
Quadruped
Humanaid Wark
Walk
64
00
0.5
15
63
54
36
18
13
20
15
65
68
20
60
30
20
50
Timestep
Timestep
Timescep (M]
Timestop
Timestep (M)
DecON Bint
DerON
Bm)
1-step
Humanoid
044
0.5
155
63
10 Timestep
68
40
Timestep [M]
Timestep [M]
Timastap [M]
Timestep [M]
TImmstep (M]
Bint Q-leaming
DexON
D-learing
Swinguo
Ringer
Timestop [M]
Timestes [M]
Timestoo [M]
Timestipp [N]
Timestop [M]
Timestop (M]
Figure 11: Ablations of the DecQN agent on multi-step returns, double Q-learning, and prioritized experience replay (top to bottom). The agent robust to changes individual components while
profiting from PER to select useful interactions complex multi-phase tasks such the Humanoid.
competitive with discretized MPO, further underlining that actor-critic methods are not required to obtain strong benchmark performance. The additional SAC baseline shows improved performance on Humanoid at the cost of being unable to learn on the Dog task, mirroring results of Hansen et al.) believe that most current state-of-the-arr continuous control algorithms capable of achieving comparable benchmark performance. However, these results do not appear to be conditional on using continuous control or actor-critic methods, and be achieved with much simpler Q-learning over discretized bang- bang action spaces with constant e-greedy exploration. We further provide converged performance of QT-Opt and AQL-Seq. While these methods remain competitive on low-dimensional tasks, their performance quickly deteriorates for high-dimensional action spaces due to their reliance on sampling this space.
(2022). Generally,
E ABLATIONS ON RAINBOW COMPONENTS
We provide ablations on three components of the underlying Rainbow agent, namely multi-step turns, double Q-learning and prioritized experience replay in Figure (rows, respectively). Learn- ing generally robust to removal of individual components in light of cumulative reward vergence. improving learning speed primarily on the more complex tasks. In particular, double Q-learning and PER provide boost the Humanoid task enable state-of-the-art performance.
18

---

Published as a conference paper at ICLR 2023
Cartporl Swingus
Finger Spin
Walkrr Wolk
Quadruped Run
Humanoid Walk
Dog Wark
64
63
60
0.5
18
15
26
66
62
36
L4
13
20
36
44
68
40
308
30
20
20
38
10
[M]
TImentep [M]
DecON Bin) DecQN loss
DocON (5 Bin) 8DQ loss
Swinguo
Spin
Wei Volk
Quadruped
Humanoid
Walk
644 Timostep [M]
13 Timestop [M]
30
20
68
40
60
35 Timestep (M]
Timestep [M]
Timestop
Timestop (M]
DecON (21
DeCON
DecON
BDQ
Figure 12: Comparison of DecQN with the original and BDQ loss for 3 (top) and 21 bins (bottom). Aggregating per-dimension utilities transforms independent into joint learners, yielding significant improvements in final performance and learning stability particularly for high-dimensional tasks.
Soin
Walker Wolk
Quadruped
Humanooid
Walk
044
0.5
18
26
43
34
18
20 Timassap
49
3. Timestup (M]
Temscatop
Timsssopp [M]
Timisupp [M]
Tirnstop [M]
DecON Bin
800 Bin)
Timestes [M]
Timestes [M]
Timestep [M]
Timesteo (N)
Timestop [M]
TImeetop (M]
Figure 13: Comparison of DecQN and BDQ for 3 (top) and 21 bins (bottom). DecQN provides sig- nificant advantage on the more complex domains, underlining the importance of design choices such strong architectural centralization and value decomposition via single-action utility functions.
F DISCUSSION OF BDQ AND HGQN
In the following, we provide more detailed discussion of the relation to Branching Dueling Q- Networks (BDQ) Tavakoli al. (0188) as well as Hypergraph Q-Networks (HGQN) Tavakoli et al. (2021) together with associated ablations. Generally, our motivation is to highlight that minimal changes to the original DQN agent enable statc-of-the-arr performance on continuous control bench- marks solely based bang-bang Q-learning with constant exploration.
The BDQ agent from Tavakoli et al. (2018) considers independent learning of per-dimension state action values, employing a dueling architecture with separate branches for each action dimension and exploration based on Gaussian with scheduled noise in combination with fine-grained dis- cretizations. DecQN forces higher degree of centralization by predicting per-dimension state- action utilities without intermediate branching or dueling heads for joint learning within a value decomposition, while using constant e-greedy exploration with focus on only coarse bang-bang control. The most important difference is independent learning in comparison to joint learning based on value decomposition. We provide an ablation of our approach that does not aggregate per- action dimension values and thereby mimics BDQ's independent learning in Figure 12 Particularly for nigh-dimensional tasks learning a value decomposition can significantly improve performance, an effect that is amplified when selecting more fine-grained discretizations. We furthermore eval- uated the original BDQ agent after modifying the default parameters (i.e. batch size, target update frequency, learning frequency, gradient clipping, exploration strategy. multi-step returns). The sults provided in Figure [3 were obtained based on the best configuration we found by increasing the batch size and treating each episode as infinite-horizn. While we believe that the results in
19

---

Published as a conference paper at ICLR 2023
Sppre Swinguo
Pinger Spm
Viaker Walk
Qusoruped Ran
Humanaic Walk
Walk
644
64
50
25
15
15
000
5.6
38
10
18
23
30
50
00
45
60
130
68
10 60
[M]
Timesee
Timestep [M]
DeCON (3 Bini moon
DecON (3 Bin) sum
Figure 14: Comparison of DecQN with original and HGQN aggregation. Aggregating via the mean induces more favorable scaling of the reward in the TD-error, which we found enable graceful scaling to complex task with high-dimensional action spaces to achieve state-of-the-an performance.
Figure 12 offer more informative comparison, we observe a similar trend regarding scaling to high-dimensional tasks in both cases.
The hypergraph Q-networks (HGQN) framework from Tavakoli et al. 2021) considers value decom- position across subsets of action dimensions and introduces higher-ordes order hyperedges between action groupings. DecQN can therefore be interpreted as an instance of the conceptual HGQN (r=1) for- mulation that leverages single-action decomposition without higher-order edges as had previously been investigated with the Atari-based FARAQL agent (Sharma et al., 2017). Our primary focus on simplicity and showing how far can push basic concepts that constitute capable alterna- tives to more sophisticated recent algorithms. There are further several differences in per-dimension utility aggregation, architectural choices regarding the use of branching. loss function, exploration scheduling, as well as in the usage of PER and double Q-learning for continuous control. We pro- vide a brief ablation on using the mean compared to sum aggregation when computing Bellman targets in Figure [4] While this appears as a subtle difference, we found that leveraging the mean yields graceful scaling to complex tasks with high- nigh-dimensional action spaces without any parameter adjustments. This enables state-of-the-arr performance across wide range of environments and input-output modalities with a single set of hyppe--pramettrs.
Generally, motivation is not to advocate for novel algorithm that should be the go-to method for solving continuous control problems. Our core objective is to highlight that current state-of-the-ar art continuous control benchmark performance is at the level of decoupled Q-learning over bang-bang -ang parameterized action spaces with constant exploration. This requires only minor well-directed modification to the original DQN algorithm and yields strong performance for both feature- and pixel-based observation spaces as well as acceleration- velocity-, and position-based action spaces. Our investigation provides additional motivation for existing work while establishing closer connec- tions to classical MARL coordination challenges, as well as extensive experimental evaluation in comparison current state-of-the- -hr---rrttt-ttttt algorithms.
G RAINBOW DQN AGENT
We leverage several modifications of vanilla DQN that accelerate learning and improve stability based on the Rainbow implementation provided by Acme (Hessel et al. 2018 Hoffman et al.) 2020
Target Network Bootstrapping directly from the learned value function can lead to instabilities. Instead, evaluating actions based on target network Qo- ar) improves learning MMnih al. 2015). The target network's weights 0- are updated periodically to match the online weights O.
Double Q-learning Direct maximization based on the value target can yield overestimation error. Decoupling action selection from action evaluation improves stability (Van Hasselt et al., 2016). Action selection then queries the online network, while action evaluation queries the target network. Our implementation further leverages two sets of critics, where we bootstrap based on their average during learning and take their maximum during action selection.
Prioritized Experience Replay Uniform sampling from replay memory limits learning efficiency. Biasing sampling towards more informative transitions can accelerate learning (Schaul et al. 2015). The observed temporal difference error serve as proxy for expected future information content.
20

---

Published as a conference paper at ICLR 2023
Seprr Swinguo
Pinger Spmn
Vikker Walk
Qusoruped Run
Humanaic Walk
Walk
644
64
00
04
5.6
18
23
20
00
25
45
60
130
68
13
20
60
[M]
Timestep
DeeQuN ET
DerigN CST Binn
Figure 15: Comparison of DecQN, DecQN with a distributional CS1 critic, and the distributional IQN agent. The distributional version of DecQN generally yields slightly improved performance convergence, while decoupling improves performance over the non-decoupled IQN baseline.
Timrstop (M]
Temessup (M]
110 aote0 [M]
Timestep [P]]
mmsssp (N)
Timissap [M]
DON
Figure 16: Comparison of DQN, DQN with distributional C51 critic, and the distributional IQN agent. Without decoupling, we do not observe benefits of distributional critics in these domains. Slower convergence Walker Walk with distributional representations could indicate that the sociated increased parameter count translates to a more difficult optimization problem.
Multi-step Returns Instead of directly bootstrapping from the value function at the next state, evaluating the reward explicitly over several transitions as part of multi-step return, such that Gettn R4 + .... +...+""" +2" Re4n-1 +y" (St4n) can improve learning (Sutton & Barto 2018).
H CONTROL-AFFING DYNAMICS AND LINEAR REWARDS
Under deterministic environment dynamics and policy we can simplify the Bellman equation as (st 04) r(se; 04) + (8++)))
Consider a reward structure that is linear in the action dimensions with r(s;; o4) = ンØ r (s,, af). Under given transition tuples and fixed target values, the TD(0) objective can then be formulated as
M
Q" ue) Σ (r((
Y V" (B(+1))) + M
(6)
which can be solved exactly based on a linear value decomposition Q(+:, ax) = Ey-o (s,, d)). Particularly for robotics applications, many common reward structures depend (approximately) linearly on the next state observation while the system dynamics are control-affine such that 8141 = [(+) + g(s))at, with (*;) and g(st) only depending on the current state st. While these strong assumptions, it may provide intuition for why the problem structures considered here may be amenable to decoupled local optimization and for the observed highly competitive performance.
I DISTRIBUTIONAL CRITICS
We briefly investigate replacing our deterministic critic with a distributional C51 head (Bellemare ct al. 2017 The decomposition now proceeds at the probability level via logits = Σド= L during the C51 distribution matching. We do not make any parameter adjustments and compare performance with and without distributional critic for both DecQN and DQN in Figures 1. and 16 respectively, and provide the IQN agent as an extension of the QR- DR-DQN agent for reference (Dabney et al. 2018 Sa b). Our empirical evaluation suggests that distributional critic can slightly increase performance at convergence and sample-efficient performance average with some environment specific variations. Without decoupling, DQN
Both DecQN and DecQN + C51 yield similar
21

---

Published as a conference paper at ICLR 2023
Teprre Swinguo
Finger Spin
Viaterr Walk
Qusoruped Ran
Humanaic Walk
Walk
644
64
00
02
04
5.6
10
18
23
30
50
00
25
45
60
130
68
13
30 40
Timestep
Timesteo [M]
Timesees
Timestep [M]
Deeeij Binn
DO36
Bie)
KIN Bie)
Figure 17: Comparison of the distributional versions of DecQN, DQN, and IQN. Decoupling of the value representation in conjunction with bang-bang action representations plays key role in scaling these approaches to high-dimensional continuous control tasks.
Features
Features
Pixels
Quadruped
Humaneld Walk
Humanoid
Quadruped
Humanoid
Epioodo Rebuun
Return
300
22
363
Timestop (M]
Timestep [M]
Timestop [M]
Timestop [M]
Timestep [M]
Timestep [M]
DocON
(3 Bin)
DecGN (21 BF
DecON
Bin
DecON CS1 (3 Bin)
DecON CS (21 Binl
DecON CSI Bin)
D4PS
DocON (3
Dreamer-v2
Figure 18: DecQN + C51 tasks where regular DecQN did not perform at least as good the best baseline. Adding a distributional critic can further boost performance, underlining the strength and versatility of this very simple approach (vertical line current training status due time constraints).
yields slightly faster convergence than DQN + C51 on Walker Walk which could indicate that the increased parameter count of distributional critics translates to a more challenging optimization problem (see also the immediate memory error of non-decoupled distributional agents on Quadruped Run in Figure [6) Both DecQN and DQN with or without distributional critic improve performance over the IQN baseline. We further provide performance of only the distributional agents in Figure I7 for ease of comparison. It is likely that given sufficient tuning the performance of our distributional variations could be increased even further, while note that the deterministic DecQN agent already matches performance of the distributional D4PG and DMPO actor-critic agents. A more extensive investigation into decoupled distributional Q-learning provides promising avenues for future work.
We note that without any parameter adjustments or tuning, replacing the deterministic critic with the distributional C51 critic in DecQN significantly boosts performance the few environments where DecQN did not perform at least as good as the best baseline. This applies to large bin sizes, feature inputs and pixel inputs as exemplified in Figure 18 (vertical line in Humanoid Walk from pixels denotes current training status, subject to time constraints). These results further underline the versatility and strength of this very simple approach.
J STOCHASTIC ENVIRONMENTS
We extend our study to stochastic versions of a selection of Control Suite tasks. The results in Section [5.]] indicate that coordination among decoupled actors becomes more difficult if the action selection of other actors is less stationary from the perspective of each individual actor. To increase stochasticity, we consider both observation and reward noise represented by additive Gaussian white noise with standard deviation Moise 0.1. Figures[T9Jan proviide results for DecQN, D4PG, and the distributional DecQN + C51 under observation and reward noise, respectively. While we observe similar performance of DecQN and D4PG on most tasks, performance of DecQN is visibly reduced on Humanoid Walk. Humanoid Walk combines several aspects that can hinder exploration, includ- ing implicit staged reward (first get up, then walk) and action penalties, which likely exacerbate coordination challenges when combined with stochasticity. We therefore believe that special care should be taken when applying regular DecQN in environments that have the potential to amplify the coordination challenges observed in the simple matrix game domains of Section 5.1 Adding distributional critic to DecQN allows for better modelling of stochasticity and visibly improves
22

---

Published as a conference paper at ICLR 2023
Tepore Swiingu
Pinger Spm
Vatker Walk
Qusoruped Ran
Humaaaic Walk
Walk
64
50
04
5.
10
18
23
20
00
45
20 60 60
130
DeeONN B4PG
DecON CSI [3
Figure 19: Stochastic observation tasks adding Gaussian white noise to the observations (Anoise 0.1). DecQN appears less robust to observation noise than D4PG, mirroring the findings regarding coordination challenges under high stochasticity in matrix games of Section Adding a distribu- tional critie improves performance and yields faster convergence than D4PG on Quadruped Run.
Cartzorc Swinggu
Ringer Spin
Villeer Walk
Quadrupod Run
Humannncc Walk
Dog Walk
30
64
63
50
02
84
85
00
62
56
10
18
23
30
50
00
48
10
130
68
10
40
54
DeeKUN D4PU
DecUN C&1 [1
Figure 20: Stochastic reward tasks adding Gaussian white noise to the rewards (onoiee = 0.1).
DecQN is less robust reward noise than the distributional D4PG, mirroring the findings regarding coordination challenges under high stochasticity in matrix games of Section 5.1 With a distribu- tional critie, DecQN can directly account for stochastic returns and matches or outperforms D4PG.
performance in stochastic environments, where DecQN + C51 even improves on the distributional D4PG agent in some environments.
23


Reviews:
[{"review_id": "kAUViSI5hZY", "reviewer": "Reviewer_LrJN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of evaluating the performance of simple a simple variation of DQN (Deep Q- Networks), derived by discretizing actions across different dimensions independently, on standard continuous control tasks from the continuous control RL literature. By combining different ideas like value function decomposition and cooperative multi agent RL, the author are able to achieve performance comparable to algorithms designed for continuous control accross a variety of continuous control tasks.", "summary_of_the_review": "Overall I am positive towards the paper. The claim made is simple and thoroughly investigated empirically. The paper does present limited novelty as the ideas discussed have been discussed before in the literature but In my opinion the thorough empirical evaluation adds to the papers contributions and ICLR is a perfect venue for this kind of results. Nevertheless, I would have also expected experiments in environments that explicitly break the 2 assumptions made to define DecDQN.", "strength_and_weaknesses": "Strengths:\n- The paper is clear and easy to follow for the most part\n- The authors perform a thorough empirical evaluation and empirically validate their claim that simple independent action discretization might perform as well as complicated actor-critic architectures\nWeaknesses:\n- Some terms used throughout the paper need to be introduced before being used, e.g. the authors continuously use the term bang-bang discretization without defining it\n- The empirical evaluation is thorough but could be improved. I would have expected to see some domains where the 2 assumptions made by the authors are clearly invalid (independent action dimension and linearity of the value decomposition) and how this affects the performance of the DecDQN compared to the continuous control agents. I suspect that continuous control agents would behave also worse when action dimensions are not independent (since generally they apply diagonal covariance matrices like the authors underline) but we might have a different result when linear value decomposition is not valid, since this is an assumption heavily used by DecDQN. T", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V1oZVZBaBKo", "reviewer": "Reviewer_zBqq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on evaluating the power of a critic-only approach to solving continuous control problems, by combining Q-learning with value decomposition (an approach from multi-agent RL under the *centralized training but decentralized execution* paradigm). They test this approach across a large set of environments from continuous control benchmarks and show that this simple model-free approach performs on par with SOTA actor-critic methods, both for model-free and model-based methods, and based on state features and pixel observations.   ", "summary_of_the_review": "Great empirical evaluation of a simple discrete-action critic-only model-free method on a large set of continuous control benchmarks (including some complex ones) against much more involved SOTA model-free and model-free actor-critic methods for continuous control; showing competitive performance. The method is not novel, but the study is interesting and insightful and the paper should be accepted in my view due to the potential impact it can have: showing strong evidence that simple Q-learning is sufficient for solving high-dimensional continuous control problems, via discretization and combining insights from multi-agent RL to achieve scalability.\n\nMy only concern is overclaiming contributions in the abstract and intro. The authors appropriately cite all related works, to the best of my knowledge, but should tone down the claims particularly in the abstract. ", "strength_and_weaknesses": "**Strengths:** \n- Exploring critic-only methods (e.g. Q-Learning) for continuous control problems is interesting because: (1) they are much simpler; (2) they allow us to have a unified approach for tackling discrete and continuous action problems; ... \n\n- Using discretization for tackling continuous control problems is interesting because: (1) because the agent can potentially capture non-parametric multimodal policy distributions in actor-critic methods; (2) many advances for discrete-action Q-learning (for exploration and credit assignment) can be immediately adopted; (3) curriculum learning can be achieved by growing the action space gradually; ...   \n\n- The experiments are done on a vast and appropriate set of continuous control benchmarks. \n\n- The baselines capture a good range of SOTA model-free and model-based methods for the chosen benchmarks.\n\n- Exposition of the connection of the method to multi-agent RL is appropriate and useful.\n\n\n**Weaknesses:**\n\n- The approach (i.e. Decoupled Q-Learning) is not novel: it is similar to HGQN (r=1) from (Tavakoli et al., 2021) (which the authors have referenced), but where the authors replace the **sum** mixing function with a **mean** mixer to achieve less sensitivity to learning rates in domains with varying action dimensionality. This is an appropriate choice but nonetheless one that is supported under the Action Hypergraph Networks framework.  \n\n- Due to the above reason, I'm not a fan of the completely new name for the method in this paper - but it is not a major issue.\n\n- The abstract makes it seem like this paper is proposing or achieving the following for the first time:\n\n   - This paper is not the first to solve and achieve competitive performance wrt. continuous-action actor-critic methods in continuous control benchmarks via discretization and a scalable version of Q-learning (e.g. Metz et al. (2017) and Tavakoli et al. (2018) are early approaches that have done so).  \n\n   - This paper is not the first to draw a connection between discrete multi-dimensional-action-space RL and multi-agent RL to achieve scalability wrt. increasing action dimensionality (e.g. Tavakoli et al. (2018) is an early instance in the context of Deep RL). \n\n   - This paper is not the first to exploit value-decomposition within Q-learning in single-agent RL (Sharma et al. (2017) and Tavakoli et al. (2021) are two instances).   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H5es9NtAptq", "reviewer": "Reviewer_wU2o", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\n\n\n\nThis paper extended Q-Learning to continuous action space by the action discretization with value decomposition in MARL. The experiment results show that the proposed method is competitive with the state-of-art continuous actor-critic methods.\n\n", "summary_of_the_review": "\n\nThe proposed method in this paper is very simple and effective. Applying the decentralized method in MARL is very intuitive. I'm surprised that this method is never proposed before.\n\nHowever, as the aggregation method is widely discussed in MARL literature and there still remains some performance gap with the state-of-the-art actor-critic methods (e.g. Humanoid Stand), I think the paper should consider using a more complex aggregation function in eq. 7. \n\n", "strength_and_weaknesses": "\n\nStrength:\n\nThe method of the paper is very clear. The method of applying the decentralized technique to the continuous action space problem is very intuitive. The experiment results are also sufficient.\n\n\n\nWeakness:\n\nThe proposed method is quite simple. The author only considered the average aggregation function for the decoupled Q values. As the aggregation method is widely discussed in MARL literature and there still remains some performance gap with the state-of-the-art actor-critic methods (e.g. Humanoid Stand) , I think the paper should consider using a more complex aggregation function. \n\nMinor typos:\n\n- $|\\mathcal{A}|$ should be $\\mathop{dim}(\\mathcal{A})$.\n\n\n\n\n\nSeveral Questions:\n\n- I don't get the necessity of comparing Dreamer in Section 5.3.\n\n- Do you consider using distribution to model the discretized action, as what is done in bang-bang policy?\n\n- In MARL, some works (e.g. QMIX) used a monotonic function to aggregate the Q values for each decentralized action. However, it's interesting to see that the author only uses the average function (eq. 4) and performs well. Do you ever try some other monotonic aggregation function?\n- Can you provide the range of the values for each dimension on some of the tasks? Dies the range vary from different dimensions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0bzu5Bu4Xx", "reviewer": "Reviewer_FMD4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper attempts to solve the problem of higher dimensional actions spaces in the context of deep reinforcement learning. The authors propose DecDQN, which consists of multiple Q-networks over different sets of the action space, resulting in reduced network parameters and potentially better performance.", "summary_of_the_review": "Overall, the paper is well written and clear. The results are logical and structured. However, there are concerns regarding novelty of the idea. I am willing to improve the score as long as the authors better address methods in distributional RL (such as QR-DQN) and add it to the results.", "strength_and_weaknesses": "Strengths:\n- The authors intelligently framed the solution as cooperative MARL.\n- Evaluation in Section 5.1 is strong and shows that  DecDQN can coordinate across Q-networks.\n- 10 seeds per agent\n\nWeaknesses:\n- The baselines are not up to date with other RL agents, such as SAC and DDPG. Would there be similar observed benefits with different algorithms and replacing the Q-networks with the paper's proposed formulation?\n- Figure 6, why does 21 bin Dec DQN generally do worse than 3 bin?\n- Is Sec 5.3 necessary? Dreamer-v2 would be slower due to being a model-based approach which looks ahead.\n-  (Nit) want to cite the library used to train other baselines (assuming it is RLlib)\n- Is there a failure case where DecDQN would fail when compared to other baselines?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster