REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes
Authors: David Ireland
Giovanni Montana
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
learning efficiency
ensembles
value-decomposition
Primary area: reinforcement learning
TL;DR: Analysing Q-value decomposition and addressing credit assignment in high dimensional discrete action spaces.

Abstract: Discrete-action reinforcement learning algorithms often falter in tasks with high-dimensional discrete action spaces due to the vast number of possible actions. A recent advancement leverages value-decomposition, a concept from multi-agent reinforcement learning, to tackle this challenge. This study delves deep into the effects of this value-decomposition, revealing that whilst it curtails the over-estimation bias inherent to Q-learning algorithms, it amplifies target variance. To counteract this, we present an ensemble of critics to mitigate target variance. Moreover, we introduce a regularisation loss that helps to mitigate the effects that exploratory actions in one dimension can have on the value of optimal actions in other dimensions. Our novel algorithm, REValueD, tested on discretised versions of the DeepMind Control Suite tasks, showcases superior performance, especially in the challenging humanoid and dog tasks. We further dissect the factors influencing REValueD's performance, evaluating the significance of the regularisation loss and the scalability of REValueD with increasing sub-actions per dimension.

Published as a conference paper at ICLR 2024
REVALUED REGULARISED ENSEMBLE DECOMPOSITION FOR FACTORISABLE MARKOV DE- CISION PROCESSES
VALUE-
David Ireland; Giovanni Montana" *1 ireland, g montana Soanttanaawwariicck wwwwwick.c..cc......o........kk.
ABSTRACT
Discrete- action reinforcement learning algorithms often falter in tasks with high- dimensional discrete action spaces due to the vast number of possible actions. A recent advancement leverages value-decompositionn, a concept from multi-agent reinforcement learning, to tackle this challenge. This study delves deep into the effects of this value-decomposition, revealing that whilst it curtails the over- estimation bias inherent to Q-learning algorithms, it amplifies target variance. To counteract this, we present an ensemble of critics to mitigate target variance. Moreover, we introduce regularisation loss that helps mitigate the effects that exploratory actions in one dimension can have on the value of optimal actions in other dimensions. Our novel algorithm, REValueD, tested on discretised versions of the DeepMind Control Suite tasks, showcases superior performance, especially in the challenging humanoid and dog tasks. We further dissect the factors influ- encing REValueD's performance, evaluating the significance of the regularisation loss and the scalability of REValueD with increasing sub-actions per dimension.
INTRODUCTION
Deep reinforcement learning (DRL) has emerged as powerful framework that combines the strengths of deep learning and reinforcement learning (RL) to tackle complex decision-making problems in wide range of domains. By leveraging deep neural networks to approximate value functions and policies, DRL has driven significant breakthroughs in numerous areas, from roboties (Gu et al., 2017; Kalashnikov et al., 2018; Andrychowicz et al., 2020) to game playing (Mnih al., 2013; Silver et al., 2016; Vinyals et al., 2017) and autonomous systems (Dosovitskiy et al., 2017; Chen al., 2017b; Kiran et al., 2021). In particular, the use of deep neural networks function approximators has allowed existing reinforcement learning algorithms to scale to tasks with contin- uous states and/or action spaces. Nonetheless, problems featuring high-dimensional, discrete action spaces remains relatively unexplored. In these problems, the action space can be thought of as a Cartesian product of discrete sets, i.e. = A₁ x ... Ay- In this context, A, represents the ith sub- action space, containing discrete isub-)actions For convenience, we henceforth refer to a Markov Decision Process (Bellman, 1957, MDP) with such a factorisable action space factorisable MDP (FMDP).
Traditional DRL algorithms can quickly become ineffective in high dimensional FMDPs, as these algorithms only recognise atomic actions. In this context, an atomic action is defined as any unique combination of sub-actions, each treated singular entity; in an FMDP there are II n, atomic actions. Due to the combinatorial explosion of atomic actions that must be accounted for, standard algorithms such Q-learning (Watkins and Dayan, 1992; Mnih al., 2013) fail to learn in these settings as a result of computational impracticalities.
To address these issues, recent approaches have been proposed that emphasise learning about each sub-action space individually (Tavakoli et al., 2018; Seyde et al., 2022). In particular, the DecQN algorithm proposed by Seyde et al. (2022) utilises strategy known as valueeddccompositon to learn
University of Warwick Aann Turing Institute

---

Published as a conference paper at ICLR 2024
utility values for sub-actions. In their methodology, the utility of each selected sub-action is com- puted independently of others but learnt in such a way that their mean estimates the Q-value for the global action. This approach is inspired by the centralised training with decentralised execu- tion paradigm in multi-agent reinforcement learning (MARL) (Kraemer and Banerjee, 2016), where learning the utility of sub-actions is analogous to learning the value of actions from distinct actors. Using this value-decomposition strategy, the ith utility function only needs to learn values for the ac- tions in A,. Consequenty, the total number of actions for which utility value needs be learned is just ΣN flg This makes the task significantly more manageable, enabling traditional value-based methods like Deep Q-lcarning (Mnih et al., 2013; Hessel et al., 2018) to solve FMDPs. Section 3 provides further details on DecQN.
In this paper, present two primary methodological contributions. First, we build upon DecQN through a theoretical analysis the value-decomposition when coupled with function approxi- mation. It is well established that Q-larming with function approximation suffers from an over- estimation bias in the target (Thrun and Schwartz, 1993; Hasselt, 2010). Consequently, we explore how the celeeddecomposition impacts this bias. We demonstrate that, whilst the DecQN decompo- sition reduces the over-estimation bias in the target Q-values, it inadvertently increases the variance. We further establish that the use of an ensemble of critics can effectively mitigate this increase in variance, resulting in significantly improved performance.
Second, we introduce a regularisation loss that we aim to minimise alongside the DecQN loss. The loss is motivated by the credit assignment issue common in MARL (Weiß, 1995; Wolpert and Tumer, 1999: Zhou et al., 2020: Gronauer and Diepold, 2022), where an exploratory action of one agent can have a negative influence on the value of the optimal action of another agent. Given the similari- ties with MARI,, this credit assignment issue is also an issue when using valoe-decompoositon in FMDPs. By minimising the regularisation loss we help mitigate the impact that exploratory sub- actions can have on the utility values of optimal sub-actions post-update by discouraging substantial changes in individual utility estimates. We achieve this by minimising the Huber loss between the selected sub-action utilities and their corresponding values under the target network.
Our work culminates in approach Decomposition. We benchmark REValueD against DecQN and Branching Ducling Q-Networks (BDQ) (Tavakoli et al., 2018), utilising the discretised variants of DeepMind control suite tasks (Tunyasuvunakool et al., 2020) used by Seyde et al. (2022) for comparison. The experimental out- comes show that REValueD consistently surpasses DecQN and BDQ across majority of tasks. Of significant note is the marked outperformance of REValueD in the humanoid and dog tasks, where the number of sub-action spaces exceedingly high (N 21 and 38, respectively). Further, we perform several ablations on the distinet components of REValueD to evaluate their individual contributions. These include analysing how performance evolves with increasing n, (i.e. the size of [A:)) and examining the impact of the regularisation loss in enhancing the overall performance of REValue.. These extensive experiments underscore the effectiveness and robustness of our ap- proach in handling high-dimensional, discrete action spaces.
call REValueD: Regularised Ensemble Value
RELATED WORK
Single-agent approaches to FMDPs: Several research endeavors have been devoted to exploring the application of reinforcement learning algorithms in environments characterised by large, dis- crete action spaces (Dulac-Arnold et al., 2015; Van de Wiele et al., 2020). However, these strategies are primarily designed to handle large action spaces consisting of numerous atomic actions, and they do not directly address the challenges associated with FMDPs. More recently. efforts specif- ically aimed at addressing the challenges posed by FMDPs have been proposed. These works aim to reason about each individual sub-action independently, leveraging either value-based (Sharma et al., 2017; Tavakoli et al., 2018; 2020; Seyde et al., 2022) or policy-gradient methods (Tang and Agrawal, 2020; Seyde et al., 2021). Some researchers have endeavored to decompose the selection of a global action into sequence prediction problem, where the global action is viewed as a quence of sub-actions (Metz et al., 2017: Pierrot et al., 2021). However, these methods necessitate defining the sequence ahead of time, which can be challenging without prior information. Tang et al. (2022) analyse a similar value-decomposition. taking the sum of utilities as opposed to the mean. Their analysis looks at fundamental properties of the decomposition of the Q-value, whereas
2

---

Published as a conference paper at ICLR 2024
in this work we analyse the bias/variance of the learning target when using function aapetiio-approximatiio in conjunction with delop-ddcooooitiionn In Appendix we analyse the sum value-decomposition theoretically and experimentally as supplement the analysis of the DecQN decomposition.
Multi-agent value-decomposition: A strong connection exists between FMDPs and MARL, es- pecially with respect single-agent methods utillsing celop-decomoosition (Tavakoli et al., 2018; 2020; Seyde et al., 2022). Owing to the paradigm of centralised training with decentralised ex- ecution (Kraemer and Banerjee, 2016), vallu-decoompoiition has gained significant popularity in MARL. This paradigm permits agents to learn in a centralised fashion whilst operating in a de- centralised manner, resulting in a lot of success in MARL (Sunehag al., 2017; Rashid et al., 2018: 2020: Du et al., 2022). Our proposed method, REValueD, acts as a regulariser for value- decomposition rather than standalone algorithm.
Multi-agent value regularisation: Some researchers have sought to regularise Q-values in MARL through hysteresis (Matignon et al., 2007; Omidshafiei et al., 2017) and leniency (Panait al., 2006; Palmer et al., 2017). However, these techniques are primarily designed for independent learners (Tan, 1993; Claus and Boutilier, 1998), where each agent is treated as an individual entity and the impact of other agents is considered as environmental uncertainty. Conversely, REValueD offers more flexible approach that can be applied to various value-decooposition methods, extending its applicabillity beyond independent learners.
Ensembles: The use of function approximators in Q-learning- based reinforcement learning algo- rithms is well known to result in problems due the maximisation bias introduced by the max operator used in the target (Thrun and Schwartz, 1993). To mitigate the effects of this maximisation bias, various studies have proposed the use of ensemble methods (Van Hasselt al., 2016; Lan al., 2020; Wang et al., 2021). Moreover, ensembles have been suggested as a means to enhance explo- ration (Osband et al., 2016: Chen et al., 2017a; Lee et al., 2021: Schäfer et al.. 2023) or reduce variance (Anschel et al., 2017; Chen et al., 2021; Liang et al., 2022). In this work we demonstrate that using an ensemble with the DecQN calue-dccmppoition provably reduces the target variance whilst leaving the over-estimation bias unaffected.
3 BACKGROUND
MARKOV DECISION PROCESSES AND FACTORISABLE MARKOV DECISION PROCESSES
We consider a Markov Decision Process (MDP) as a tuple (S, A.T. Po) where S and A are the state and action spaces, respectively. T reward function, the discount factor and Po the initial state distribution. The objective is to find a policy æ S + [o. 1). a state-conditioned distribution over actions, that maximises the expected (discounted) returns, (po,()..))) 12==0 r'r(st, where r = (So. ai,..., -T- ST) is trajectory generated by the initial state distribution Po. the transition function T and the policy n.
A - S the transition function. r S x A + E the
We define factorisable MDP (FMDP) as MDP where the action space can be factorised as A Ai X ... x AN, where A; is a set of discrete bubb))actions. We refer to a = (a1, (..; .N) ) as the global action, and individual at's 's as sub-actions. If the policy of an FMDP selects sub-actions independently, it is convenient consider the policy as (als)) II n. (a:]ss)) where m, is the policy of the vthh sub-action space.
DECOUPLED Q-NETWORKS
Decoupled Q-Networks (DecQN) were introduced by Seyde et al. (2022) to scale up Deep Q- Networks (Mnih et al., 2013) to FMDPs. Rather than learning a Q-value directly, DecQN learns utility values for each sub-action space. we let Už, (s, az) be the ith utility function, parameterised by 0g, then for a global action a (a,,...,N)) uN) the Q-value is defined as
1 N
(0(s,a)= =
Σ U (s. az)
(3.1)
N
1=1
3

---

Published as a conference paper at ICLR 2024
where 0 {0,)
This decomposition allows for efficient computation of the arg operator.
arg ((s. arg max Ug, (s, 01), arg max Uby (s, (s. E,A,
axEAN
To learn the network parameters 0, the following loss is minimised:
(())
Σ
L(y (s
(3.2)
B
(SaN)))
where the Huber loss, B batch sampled from the replay buffer, y + & Ei=1 ma Ug (d', al) is the Q-learning target, and {0:) correspond to the parame-
ters of the target network.
4 METHODOLOGGY
DECQN TARGET ESTIMATION BIAS AND VARIANCE
Considering the well-known issue positive estimation bias associated with Q-learning with func- tion approximation (Thrun and Schwartz, 1993; Hasselt, 2010), our interest lies in understanding how the DecQN decomposition of the global Q-value impacts this over-estimation bias and the variance in the target. Following the assumptions outlined by Thrun and Schwartz (1993), propose that the approximated Q-function carries some uniformly distributed noise, defined as (s) = Q" (s, + 6saa Here, Q" represents the true Q-function corresponding to policy m, and 6saa are independent, identically distributed (i.i.d) Uniform (bb, b) random variables. We then define the target difference as
Zdqa - r max Qo(s', a) m
max Qo(s'aa) a) max Q" (*')
(4.1)
We refer to Zdan as the target difference when using a DQN, i.e. when no value-decomposition is used.
In terms of the DecQN decomposition of the Q-value, it can be assumed that the uniform approxi- mation error stems from the utilities. Hence, we can define U2 (s, as) U/T (s, a.) + + c,,,, Anal- ogous to the previous scenario, U. the true ith utility function for policy m. and ct.,, are i.i.d Uniform( -b, b) random variables. Using the DecQN decomposition from Equation (3.1) Equa- tion (4.1) we can write the DecQN target difference as
N max U. (s', a,E.:
Zdee = ?
(s', Σ max Uj, (s',a,)
Σ
(4.2)
a,EA,
N
This leads us to our first result:
Theorem 1. Given the definitions of 7dan and 7,ee in Equations (4.1) and (4.2), respectively, we have that:
1. d < [[
2. Var(Z.em) < Var (Z.te)
The detailed proof can be found in Appendix A. The key takeway from this result is that, whilst the expected value of the target difference reduced when using the DecQN decomposition - benefi- cial factor for reducing the over-estimation bias typically associated with Q-learning it also results in an increased variance the target. This trade-off requires further investigation, particularly with respect to its impact the stability and robustness of learning. The increased variance may lead to more fluctuation the utility estimates and cause potential instability in learning, which is notable challenge.

---

Published as a conference paper at ICLR 2024
The proposed methodology for the REValueD approach is characterised by the of an ensemble of critics to address the higher variance observed under the DecQN decomposition. The ensemble approach uses K crities such that each critic Q* (s. a) is the mean of the utilities U² (s, az), where O,, denotes the parameters of the ith utility in the kth critie. Each critic in the ensemble trained with a target
N Σ max afe.A
- N
where as) = K Ex U f.. (s, represents the mean utility across all critics, with 8,, being the parameters of the target utility networks. By applying this ensemble-baset approach in REValuel), we aim to effectively reduce the variance introduced by the DecQN decomposition, thus facilitating stable and efficient learning.
To show that the ensemble-base based approach adopted by REValueD brings the variance down, we define the new target difference that incorporates the ensemble as
1 N
Z =
Σ 1=1
max ('(5',a)) - N max (S*(('s a.EA:
(4.3)
a,EA;
It is important to note that the utility function for each critic in the ensemble is now considered to have some uniformly distributed noise, such that U3,,, (s, 0z) = U. (8, a:) + 8,04 where ei, are assumed to be i..d random variables following Uniform(-be difference, we present our second result:
distribution. Using this target
Theorem 2. Given the definitions of 7:dec and Zg from Equations (4.2) and (4.3), respectively, we have that
1. E [[[z=e]]= :
2. Var(Zens)
Var( Zdec).
The proof is given in Appendix A. Leveraging ensemble of critics, REValueD provides an ef- fective means to counteract the increased variance inherent in the DecQN decomposition. whilst the ensemble framework leaves the expected value of the target difference unaltered, it reduces its variance. This is an essential property, as it underpins stability throughout the learning process. A further advantageous feature that the variance reduction is directly proportional to the ensemble size. denoted by K. This grants us direct means of control over the variance, thus allowing for more controlled learning process. A detailed analysis of how the ensemble size influences the performance of REValueD is presented in Section 5 and in Appendix G we investigate how the ensemble approach affects the training stability.
REGULARISED VALUE- DALUUPDDCOOOOIITONN
FMDPs involve multiple sub-actions being executed simultancously, with the system offering a sin- gle scalar reward as feedback. This creates situation where an optimal sub-action one dimension might be detrimentally impacted by exploratory sub-actions taken in other dimensions. Under such circumstances, if we minimise the DecQN loss as per Equation (3.2), the utility for the optimal sub-action could be undervalued due to extrinsic influences beyond its control. This insight paves the way for the introduction of the regularised component REValueD. Given that feedback is exclusively available for the global action, credit assignment issue often arises. A global action can see optimal sub-actions from one dimension coupled with exploratory sub-actions from other dimensions. These exploratory sub-actions could potentially yield low reward or navigate low- value subsequent state. Both consequences adversely affect the utility values of optimal sub-actions within the global action by resulting in an underestimated TD target for the optimal sub-action.
To counteract this impact, we propose the introduction of a regularisation term in the model's update equation, thereby minimising the following loss:
N
La (0)
1
Σ
Σ
L. (Ua, - (s,
(4.4)
B
(s...r,"))
1=1
5

---

Published as a conference paper at ICLR 2024
where w, 1 exp (-16.1) and 6; = y- -U, (s, (s,))), with y being defined as in Equation (3.2).
The functional form for the weights is chosen such that as |o:| grows larger, the weights tend to one. The rationale being that, for large |o,], the reward/next state is likely influenced by the effect of other sub-actions. As a result, we want to regularise the update to prevent individual utilities from being too under or overvalued. This is achieved by keeping them in close proximity the existing values - process managed by weights that increase as [6;] increases. We offer more insight into the functional form of the weights in Appendix I.
By introducing this regularisation loss, we gain finer control over the impact of an update at the individual utility level. Unlike the DecQN loss, which updates the mean of the utility estimates, our approach facilitates direct regulation of the impact of an update on specific utilities. Consequently, the total loss that REValu aims to minimise is then defined by
Lia( (0) L()) (0)
(4.5)
where & acts as isation loss. Throughout experiments, we keep B at consistent value of 0.5, and we perform an ablation on the sensitivity to 3 in Appendix D. It is worth noting that although we introduce the regularisation loss for single critic (i.e. K 1) for ease of notation, its extension an ensemble of critics is straightforward.
determining the extent to which we aim to minimise the regular-
5 EXPERIMENTS
In this Section benchmark REValueD the discretised versions of the DeepMind Control Suite tasks (Tunyasuvunakool et al., 2020), as utilised by Seyde et al. (2022). These tasks represent challenging control problems, and when discretised, they can incorporate up to 38 distinct sub-action spaces. We also provide results for a selection of discretised MetaWorld tasks (Yu et al., 2020) in Appendix J. Finally, as an additional baseline, we compare REValueD version of DecQN with a distributional critic in Appendix L.
Unless stated otherwise, we employ the same Bang-Off-Bant discretisation as Seyde et al. (2022), in which each dimension of the continuous action is discretised into three bins. For comparative analysis, we evaluate REValueD against the vanilla DecQN (Seyde et al., 2022) and Branching Dueling N-Network (BDQ) (Tavakoli et al., 2018). For more like-to-like comparison, we compare REValueD an ensembled variant of BDQ in Appendix K in select DMC tasks. To measure the performance, after every 1000 updates we plot the returns of test episode where actions are selected according to a greedy policy. Implementation details are given in Appendix C.
DEEPMIND CONTROL SUITE
The mean performance, together with accompanying 95% confidence interval, is shown in Figure 1. This comparison includes the results achieved by REValueD, DecQN, and BDQ on the DM Control Suite tasks. It immediately clear that BDQ the least successful, with a complete failure to learn in the humanoid environments. We also observe that REValueD demonstrates considerable improvement over DecQN in most tasks, especially the more challenging tasks where the number of sub-action spaces is larger. This advantage is particularly noticeable in the humanoid and dog tasks, which have an exceptionally large number of sub-action spaces. These findings provide strong evidence for the increased learning efficiency conferred by REValueD in FMDPs.
The performance of these algorithms as the number of sub-actions per dimension increases is another important factor to consider. We investigate this by varying the number of bins used to discretise the continuous actions. In Figure we see the results in the demanding dog-walk task. For n = 30, REValueD maintains a strong performance, whereas the performance of DecQN has already started to falter. We can see that even for n = 75 or larger, REValueD still demonstrates an acceptable level of performance, whereas DecQN is only just showing signs of learning. Interestingly, BDQ performs consistently in the dog-walk task, despite the large n, values, managing to learn where DecQN falls short. We present further results in Figure 5 in Appendix F.

---

Published as a conference paper at ICLR 2024
boll_in_oup
reacher-hard
walkerwwa
Text eptonde
cargpll swingup_spare
Fis Swift
choctah-run
quadruped walk
25t6
62.
130%
200
1016
230h
100
0.40
humannd--taadd
humensic-oo
humonols-num
dog-walk
dog-trot
CD0 -Tun
100
20th
cDi Eny Interactions
Ew Interncctonns
Env Interactions
Env Interrations
DerQN R.ValueD
BDQ
Figure 1: Performance for Discretised DeepMind Control Suite tasks. We compare REValueD with DecQN and BDQ. The solid line corresponds to the mean of 10 seeds, with the shaded area corresponding to a 95% confidence interval.
dog-wwikk,
dogoweti, 75
100
200
200
Text
0.21
3.40
1.7
0.20
3.00
2.74 Fry Interactions
Col
Interactions
Irheractions
Inbernet
BEvalaaD
Figure 2: Here assess how the performances of DecQN, BDQ and REValueD are affected by increasing the size of each sub-action space. We conduct experiments on the fish-swim, cheetah-run and dog-walk tasks. corresponds to the size of the sub-action spaces, i.e. |A,] n for all The solid line corresponds to the mean of 10 seeds, with the shaded area corresponding to a 95% confidence interval. Further results are given in Figure 5 in Appendix E.
ABLATION STUDIES
To demonstrate the effect that the regularisation loss (Equation (4.4) has on the performance of REValueD we analyse its contribution in the most challenging DM Control Suite tasks. We also provide a further ablation in Appendix H that demon- strates how the loss can delay the negative effects of exploratory updates optimal sub-action
Relative contribution of the regularisation loss:

---

Published as a conference paper at ICLR 2024
This Table demonstrates the impact of the regularisation loss by contrasting the perfor- mance of REValueD with DecQN equipped only with an ensemble. Our comparison leverages the humanoid and dog tasks, deemed to be the most demanding tasks, thus necessitating careful credit assignment for sub-actions. We report the mean = standard error of 10 seeds.
Table 1:
Algorithm
Task
DecQN
De<eNnEnsemble
REValueD
Humanoid-Sune -tannd
604.82 + 85.1
832.99 + 11.3
915.80 + 6.12
Humanoid- Walk
670.62 34.1
817.63 + 7.66
874.33 + 3.63
Humanoid- -unn
416.81 + 8.77
478.11 + 4.59
534.81 + 10.8
Dog- -alk
641.13 28.8
819.95 = 22.0
862.31 + 12.0
Dog-Trot
856.48 + 12.2
878.47 6.33
902.01 + 7.65
Dog-Run
625.66 12.5
750.73 + 11.2
821.17 + 8.10
dog walk
dog trot
dog
33.
2051
3.2
183
100
132
700%
0.35
1.00
3.50
375
dog walk
dog trot
dog run
7004
Irteractions
Pru Internationss
DecQN
Figure 3:
Stochastic environment tasks. In the top row we added Gaussian white noise (o 0.1) to the rewards, whilst in the bottom row we added Gaussian white noise to the state. Further results are given in Figure in Appendix F.
utility values a Tabular FMDP. Table compares the performance of REValueD both with and without the regularisation loss; the latter refer DecQN+Ensemble. As demonstrated in the Table, the inclusion of an ensemble of critics noticeably enhances the performance of DecQN. Fur- ther, the incorporation of the regularisation loss further augments the performance of the ensemble, clearly illustrating the merits of the regularisation loss.
Stochastic Environments: Here we extend our analysis stochastic variants of a selection of the DM Control Suite tasks. Stochastic environments exacerbate the credit assignment problem outlined Section 4 as there is now an extra source of uncertainty that each decoupled actor must contend with. We consider two types of stochasticity, adding Gaussian white noise to the reward and states, respectively. Figure shows the results in the three variants of the dog task. We observe that all three algorithms largely the same in the three dog tasks. When stochasticity is added to the states, the performance of all three algorithms drops, though REValueD outperforms DecQN whilst both outperform BDQ. In Appendix we show show similar results in more stochastic DM Control Suite environments.
robust stochasticity being added to the rewards with the performance being
that the same hierarchy of performance is maintained, i.e.
Assessing the impact the ensemble size: Of particular interest is how the size of the ensemble used in REValue)) affects performance. In Table 2 we provide the final asymptotic results. In gen-
8

---

Published as a conference paper at ICLR 2024
Table 2: Asymptotic results for REValueD with varying ensemble size across various DM Control
Suite tasks. We report the mean + standard error over 10 seeds.
Ensemble Size
Task
3
10
20
Walker-Run
763.97 +: 4.75
779.40 + 1.29
769.25 + 2.59
761.74 2.30
762.32 + 4.95
755.86 +: 7.35
Cheetah-Run
759.01 + 12.5
843.46 + 16.3
781.30 + 14.3
828.60 + 25.9
883.55 + 2.99
807.75
Quadruped-Rum
911.01 + 13.4
927.32 + 8.17
924.52 + 3.92
926.79 4.27
922.33 7.56
Dog-Wald Dog-Walk
838.39 + 8.40
904.47
886.73 7.38
899.77
878.40
Dog-Trot
843.61 +: 19.8
915.67 4.14
912.97 5.09
910.24 10.4
897.34
Dog-Run
675.51 = 9.10
771.25 11.6
815.29 + 15.9
821.17
834.77 + 4.16
823.20 7.29
eral, the performance is reasonably robust to the ensemble size for the walkerrcheeabb/quuuuucccm tasks. particular, we note that an ensemble size of in walker-run has the best asymptotic per- formance. It has been noted that some target variance can be useful for exploration purposes (Chen al., 2021), and so is possible that for an easier task like walker using a higher ensemble reduces the variance too much. However, as the complexity of the task increases we start to see the benefi- cial impact that a larger ensemble size has. This is evident in the demanding dog-run tasks, where performance peaks with an ensemble size of 15. To summarise these findings, it may be beneficial to use a smaller ensemble size for tasks which are easier nature. larger ensemble size improves performance for more complex tasks.
CONCLUSION
Recent works have successfully taken the idea of value-decomposition from MARL and applied it to single agent FMDPs. Motivated by the known issues surrounding maximisation bias Q-learning, analyse how this is affected when using the DecQN value-decomposition We show theoretically that whilst the estimation bias is lowered, the target variance increases. Having a high target variance can cause instabilities learning. To counteract this, introduce ensemble of critics control the target variance, whilst showing that this leaves the estimation bias unaffected.
Drawing on the parallels between single agent FMDPs and MARL, we also address the credit assign- ment issue. Exploratory sub-actions from one dimension can effect the value of optimal sub-actions from another dimension, giving rise to a credit assignment problem. If the global action contained some exploratory sub-actions then the TD-ttrget can be undervalued with respect to optimal sub- actions in the global action. counteract this, we introduce a regularisation loss that we aim to minimise alongside the DecQN loss. The regularisation works at the individual utility level, mitigat- ing the effect exploratory sub-actions by enforcing that utility estimates do not stray too far from their current values.
These two contributions lead to our proposed algorithm: REValueD. We compare the performance of REValueD to DecQN and BDQ tasks from the DM Control Suite. We find that, in most tasks, REValueD greatly outperforms DecQN and BDQ, notably in the challenging humanoid and dog tasks which have N 21 and 38 sub-action spaces, respectively. We also provide various ablation studies, including the relative contribution that the regularisation loss has on the overall performance of REValueD and how the performance changes the number of sub-actions per dimension increases.
Potential avenues for future work include to better understand the benefits of distributional reinforcement learning (Bellemare et al., 2023) in FMDPs, as distributional perspective to learning can help deal with the uncertainty induced by ex- ploratory sub-actions. Further, using uninformative e-greedy exploration strategy can be wasteful FMDPs due to the many possible combinations of sub-actions, and more advanced exploration techniques could be developed to effectively choose optimal sub-actions in one dimension whilst continuing to explore in other dimensions.
rigorous exploration of the work in Appendix L

---

Published as a conference paper at ICLR 2024
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning, In International conference on machine learning, pages 176- 185. PMLR, 2017.
Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty- based offline reinforcement learning. Machine Learning, 113(1):443-44) 488,, 2024. 18
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement PMLR, 2017. 26
learning. International conference on machine learning, pages
Mare G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT
Press, 2023. 9
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama.
Measuring the reliability of reinforcement learning algorithms. arXiv preprint arXiv: 2019. 20
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q- ensembles. arXiv preprint arXiv:/706.01 2017a.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2 2021. 3,9
Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P How. Socially aware motion planning with deep reinforcement learning. In 2017 IEEF/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1343-1350. IEEE, 2017b.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI/IAA,, 1998(746-752::2, 1998. 3
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
Wei Du, Shifei Ding, Lili Guo, Jian Zhang, Chenglong Zhang, and Ling Ding. Value function factorization with dynamic weighting for deep multi-agent reinforcement learning. Information Sciences, 615:191-208, 2022. 3
Gabriel Dulac-Arnold Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep rein- forcement learning in large discrete action spaces. arXiv preprint arXiv: 2015.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey, Artificial Intelligence Review, pages 1-49, 2022. 2
Shixiang Gu, Ethan Holly, Timothy Lillicrap. and Sergey Levine.
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- on robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 2010. 2,
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 2
10

---

Published as a conference paper at ICLR 2024
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.0099 2018. 19
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018. 1
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4992) 2021.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning a rehearsal for decentralized planning. Neurocomputing, 190:82-94 2016. 2,
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. arXiv preprint arXiv:20002.06487, 2020
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. In International Conference on Ma- chine Learning, pages 6131 6131-6141. PMLR, 2021.
Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, and Roy Fox. Reducing variance in temporal-difference value estimation via ensemble of deep networks. In International Conference on Machine Learning, pages 13285-13301. PMLR, 2022. 3
Laëtitia Matignon, Guillaume Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: an al- gorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 64 69. IEEE, 2007. 3
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint 7oXiv:7705.05035, 2017.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 112.5:13 2013. 1,
Shayegan Omidshafici, Jason Pazis, Christopher Amato, Jonathan How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Inter- national Conference on Machine Learning, pages 2681-2690. PMLR, 2017. 3
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, Advances in neural information processing systems, 29, 2016. 3, 18
Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep rein- 2017.
forcement learning. arXiv preprint
Liviu Panait, Keith Sullivan, and Sean Luke. Lenient learners in cooperative multiagent systems. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pages 801-803, 2006. 3
Thomas Pierrot, Valentin Macé, Jean-Baptiste Sevestre, Louis Monier, Alexandre Laterre, Nicolas Perrin, Karim Beguir, and Olivier Sigaud. Factored action spaces in deep reinforcement learning- 2021. 2
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 02:10099-102100 2020.

---

Published as a conference paper at ICLR 2024
Lukas Schäfer, Oliver Slumbers, Stephen McAleer, Yali Du, Stefano V Albrecht, and David Mguni. Ensemble value functions for efficient exploration in multi-agent reinforcement learning. arXiv preprint 2XX1:22.002.33399, 2023. 3
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang- bang-bang& control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 34:27209-27221 2021. 2
Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via q-learning. arXiv preprint arXiv:2210. 12.666 2022. 1, 2, 6, 16, 26
Sahil Sharma, Aravind Suresh, Rahul Ramesh, and Balaraman Ravindran. Learning to factor poli- cies and action-value functions: Factored action space representations for deep reinforcement learning. arXiv preprint arXiv: 175.:2775,00269, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint
2017. 3
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pages 330-337, 1993. 3
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 35:34272-34222 2022. 2, 16, 17
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the aaai conference on artificial intelligence, volume 34, pages 5981-5988, 2020. 2
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offliee RL Work- shop: Offline RL as a "Launchpad", 2022. URL //tp:://////pe/eieew////// net forum? ?id=
SyAS49bBov. 18
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep rein- forcement learning. In Proceedings of the aaai conference artificial intelligence, volume 32, 2018. 1, 2, 3,6 6
Arash Tavakoli, Mehdi Fatemi, and Petar Kormushev. Learning to represent action values hypergraph on the action vertices. arXiv preprini arXiv:2010. 14680, 2020. 2, 3
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionisl Models Summer Schwol, volume 255, page 263. Hillsdale, NJ, 1993. 2, 3, 4, 15
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6: 60002, 2020. 2,
Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enor- mous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08/ 16, 2020. 2
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016. 3
12

---

Published as a conference paper at ICLR 2024
Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets,
Michelle Yco, Alireza Makhzani. Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 4rX1v:1700.004782,2001 2017.
Hang Wang, Sen Lin, and Junshan Zhang. Adaptive ensemble q-learning: Minimizing estimation bias via error feedback. Advances Neural Information Processing Systems, 34:24778-24777 2021. 3
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Gerhard WeiB. Distributed reinforcement learning. In The Biology and technology of intelligent autonomous agents, pages +15-428. Springer, 1995. 2
David H Wolpert and Kagan Tumer. An introduction to collective intelligence. arXiv preprint
cs/9908014, 1999. 2
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1 1000 PMLR, 2020. 6, 24
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in neural information processing systems, 33: 1853-1 1864, 2020.
13

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREMS AND 2
Lemma 1. Let {Xi) be i.i.d Uniform(-Be (ba b) random variables, and let Z = max; (Xi) be their maximum. Then, [
b==1 and Var ( (n+1)((++2)
4
m+1
Proof. Let (x(x)= m-th 25
be the CDF of a random variable with distribution Uniform(-b, b). Thus,
have that
Fz(2)
P(Z S 2)
P(max (V 2)
[Fx(=)]
2 b n
2b
d
++b n-1
Now, the pdf of Z given by fz(2)
F2(2)
From this, we can deduce
dz
2b
that
E [2]
11
+
b
d:
1 11
-
2b
2b
+
+
b
n-1
4b
Var(Z)
d2
E
[2]
26
26
(n + 1) +2)
Note that the first integral can be solved by using the substitution solved by parts followed by similar substitution.
= 24h and the second can be 2b
(TL n n.
Lemma 2. For Th E N, ny 2 we have + En NII
for N 1.
Proof.
is concave and increasing on R4. Thus, using Jensen's 's inequality we #-1 Note that f(x) = a+1
have
1
((m)
Ss
N
N
1=1
Since is increasing we also have
N
N
n4i
74
We have thus shown that
I 11:
1
-
Tli
[[ Mi +1
N
F (1+II m. (2+ m.
Lemma 3. For n, E N, n4 2 2, we have
Σ
(nct for
No
1
Proof. Note that f(x) (x+1) is convex and decreasing {a using Jensen's inequality we have
R4, 2 2). Thus,
1 N
f
Σ ni
S
Σ f(n.)
N
N
N
1=1
i=
14

---

Published as a conference paper at ICLR 2024
Now we need to show that f (II 12)
S $ [ m.): that is, we want to show that
I n2
N
Eimn N
11 1+
m4)
2 [1 m1:)
E
(2+ Φ Σ mi
or, equivalently, that
n4 (1+
2
2 Lim T4
N +
N Zim!
N
Let (ax) = (1 + x)² (1+ ) is straight-forward to verify that g is increasing for x 2 2 and
satisfies g(2x) > 2z(x) for x Z 2. We have that I n4 2 N Ein n4 since all factors
are greater than or equal to 2, and at least one factor is greater than or equal to N Σi=l ni It then follows that
N g
N
N T4e 2
N
II
Th
29 1
2N
Σ
24-1
9
Ng
74
i=1
i=1
where in the last step have used Bernoulli's inequality:
2N-1 = (1 1)N-1 2 1 + (N-
We have thus shown that g (1I 2 Ng + ( and, subsequently, that (II mi) s 1 No Em f(ni).
Theorem 1. Given the definitions of 0 and Zdde in Equations (4.1) and (4.2), respectively, we huve that:
1. d. E[ZAy"):
2. Var (ur(Z < Var Zdec).
Proof of Theorem 1: First note that in Lemmas and 3 we have assumed that n, E N, n, 2 This because n, corresponds to the size of discrete (sub)action space, we know it must be least size 2 (otherwise there would be no decision to make).
1. As shown in Thrun and Schwartz (1993), E[Zdgn]
76
A
For an FMDP, |A|
A
(II nz) from which it follows that E|Z = yb (II n.
For the DecQN decom-
(IT.- ny
position, the expectation is
N
1
doe
E
max 62,0)
R.EAA
7
14:
N
+
Here, we have used the expectation from Lemma 1, as the errors are i.i.d Uniform(
(-b,
(IT m:)
random variables. We have shown in Lemma 2 that $ MN
m--1
(TT +1)+1'
therefore E|Z < E[Z
15

---

Published as a conference paper at ICLR 2024
2. First, since the enrors i.d Uniform(-b, b) random variables the variance from Lemma to get
Var ( = 7 Var (max 6s.a) = =2
462
I1;-
(
11 n4
(2 11: n4
72 N
2
N
4852
Var( (2
Var max fz a,EA,
N2
N
(no + 1)2 22
I We have shown in Lemma 3 that (1+I1-) (2+HE m.) therefore Var(Z (ur( S Var(Zgeo))
I 1 Ei= (m,+)))(m++2)*
Theorem 2. Given the definitions Zdec and Zg from Equations (4.2) and (4.3), respectively, we have that
1. E[ = 1
2. Var( (Zgm))= 1. Var( Zdec)
Proof of Theorem 2
1. Since for fixed the errors are i..d. across k, and in fact have the same distribution as the error terms in the DecQN decomposition, we have that
K
E
max
ikk eaa
N
K
as
1=1
k=
7
KE
max et a.EA: Ea
N
N
?
N
max a.EA:.
S,RL
E
2. Using similar argument to 1., we have that
N
K
Var Z
Var
max i.k a.EA;
N
K
2
i=1
km
22
KVar
max Ca,a; o.EA:
N²
K2
Var max
K
GEE:
1
Var (Zlee)
K
B ANALYSIS OF THE SUM VALUE-DECOMPOSITON
Tang et al. (2022) propose value-decomposition similar to the decomposition proposed by Seyde et al. (2022) (Equation (3.1)). Their value-decomposition approximates the global Q-value as the sum of the utilities, whereas DecQN define their decomposition the mean of the utilities. Using
16

---

Published as a conference paper at ICLR 2024
notation, the value-decomposition proposed by Tang et al. (2022) is defined as
Qo(s.
Σ (s,
i=1
Using this decomposition in Equation (4.1) we can write the sum target difference as
N Σ
2
U0 (6',a)) az) - Σ max (6', a.EA,
max
(B.1)
i=1 a.E.
Whilst the decompositions are similar, we will now show that under this value-decomposition the expected value and the variance of this target difference are higher than that of the DQN target difference.
Theorem 3. Given the definitions of Zd and Z in Equations (4.1) and (B.1), respectively, we have that:
1. E[ & E[
2. Var(Zarm)= < Var Zgum).
Proof of Theorem
1. We can see that E [g
NEE[Zee = 7b Ei= n--1 at so it remains to be shown that
Ea # I )1
m.
First, let f(x) = =-1 and note that lim f(x) 1 and
that is increasing for x 2 2. Since f is increasing, and recalling that n; 2 2, for N we have
Σ t=1
f(ni) > Σ f(2):
i=1
N
Σ
Now, of (II mx) want to show that
N 3
for 2 3 and the case for is trivial. For N = 2 we
7
71
n22
72 + 1
n1 +1
112 +
Re-arranging this term, get
(nin₂ - 1) - 1) - 1)
2
20;
(n) + 1)(m2 + 1)( +1
which is clearly true for n1, n2 2 2, and so the inequality also holds for N
2. We can see that Var(Zsum) 2 (Z,ee) 72 EN (m.+1) As we have shown
46
that Varl (dde) > Var( (dqm) it follows immediately that Var( (25mm > Var (dd)).
Theorem tells us that, when using function approximators, decomposing the Q-function using the sum of utilities leads to higher bias in the target value compared to no decomposition. This is contrast to the DecQN value-decomposition which reduces the bias. However, both decompo- sitions come at the cost of increased variance though show part of the proof that the sum decomposition also has a higher variance than the DecQN decomposition.
Experimentally, compare REValueD and DecQN with the sum value-decompossiion (which name DecQN-Sun (umm)))Smmmaiu in Figure 4. We can see from the Figure that using the sum value-decomposition generally results in poorer performance than the mean. The poor performance of DecQN-Sum compared to DecQN and REValueD can likely be attributed to our findings in Theorem 3, that the sum valme-eccomposition increases the overestimation bias compared to using the mean in
17

---

Published as a conference paper at ICLR 2024
DecQN. In Appendix K we offer further comparison of REValueD with DecQN-Sum equipped with an ensemble for a fairer comparison.
corppole-seee taingutnnpppppp sparso
fidh-owi
chretah-run
qunddupes-wak
206
20th
1804
605
2006
130
2064
304
405
300
1236
2004
40%
500
6A6
1.4
0.M
Test
hummnnsi--run
dag--walk
dog-trot
Cl0g -1un
24
13/1
20.h
<04
636
0.24
2.1%
355
134
Pro Iréeractions
Enu Interactions
Fit Interactions
Fho Instereet 1076
DeoqN REValucD
DecQN-Sum
Figure 4: Further results comparing REValueD and DecQN to DecQN using the sum value- decomposition (DecQN-Sum). The solid line corresponds to the mean of 10 seeds, with the shaded area corresponding to a 95% confidence interval.
C EXPERIMENT DETAILS
Hyperparameters: We largely employ the same hyperparameters the original DecQN study, as detailed in Table 4, along with those specific to REValueD. Exceptions include the decay of the exploration parameter (c) to minimum value instead of keeping it constant, and the use of Polyak- averaging for updating the target network parameters, as opposed to a hard reset after every specified number of updates. We maintain the action selection in REValueD, we follow deep exploration technique similar to that proposed by Osband al. (2016), where we sample single critic from the ensemble at each time-step during training and follow an e-greedy policy based on that critic's s utility estimates. For test-time action selection, we average over the ensemble and then act greedily according to the mean utility values.
hyperparameters across all experiments. During
Architecture and implementation details: As in the original DecQN study, our architecture con- sists of fully connected network featuring residual block followed by layer normalisation. The output from this sequence is fed through fully connected linear layer predicting value estimates for each sub-action space. implementing the ensemble for REValueD, we follow the approach given by Tarasov et al. (2022), which has been demonstrated to have minor slowdowns for relatively small ensemble sizes (Beeson and Montana, 2024, Table 8).
In Table 3 report the mean time taken for DecQN and REValueD, in minutes, when run on the same machine. Each task was run for 500k environment interactions (100k updates). We can from the table REValueD takes around 25% longer than DecQN, though it is important to note that REValueD is more sample efficient than DecQN so less environment interactions are needed to achieve superior performance.
18

---

Published as a conference paper at ICLR 2024
Table Wall time comparison for DecQN and REValueD. Each task was ran for 500k environment interactions (100k updates), and the mean (# variance) is reported in time (minutes) over 3 seeds.
Task
DecQN
REValueD
walker-walk
30.47 0.00
37.05 0.32
dog-walk 55.46 + 0.31
69.55 + 0.32
To expedite data collection, adopt Ape-X-like framework (Horgan al., 2018), with multiple distributed workers each interacting with individual instances of the environment. Their interactions populate centralised replay buffer from which the learner samples transition data learn from. It is crucial to note that all the algorithms we compare (RE (REVal DecQN, BDQ) use the same implementation, ensuring consistent number of environment interactions used for each algorithm for fair comparison.
Environment details: In Table we show the state and sub-action space size of the DM Control Suite tasks we use in the experiments in Section 5. In particular, we can see that the humanoid and dog environments have large state and action spaces, each having 21 and sub-action spaces, respectively. With 3 bins per sub-action space, this would correspond to 3 and 3 atomic actions. is particularly challenging to solve these tasks given the interdependencies between sub-actions.
Table 4: Hyperparameters used for the experiments presented in Section 5,
Algorithm
Parameters
Value
Optimizer
Adam
Learning rate
x 10
Replay size
10
n-step returns
3
Discount, 7
0.99
Batch size
256
General
Hidden size
512
Gradient clipping
40
Target network update parameter, c
0.005
Imp. sampling exponent
0.2
Priority exponent
0.6
Minimum exploration,
0.05
decay rate
0.99995
Regularisation loss coefficient (
0.5
REValueD
Ensemble size K
10
Table Details of the DM Control Suite environments used the experiments presented in Section 5. Note that N corresponds to the number of sub-action spaces in the FMDP.
Task
S
N
Cartpole Swingup Sparse
Reacher Hard
Ball in Cup Catch
Finger Spin
Fish Swim
Cheetah Run
17
Walker Walk/Run
Quadruped Walk/Run
78
12
Humanoid Stand/Walk/Run
Dog Walk/Trot/Run
223
38
19

---

Published as a conference paper at ICLR 2024
Table 6: Asymptotic results for REValueD with varying across various DM Control Suite tasks. We report the mean + standard error over 10 seeds.
Task
0.25
0.5
0.75
Finger-Spin
947.80 + 10.6
847.15 + 20.3
905.86 + 16.1
849.26 + 20.3
927.39 25.9
Walker-Run
755.72 + 3.25
750.91 + 2.95
761.74 + 2.30
768.60 + 1.62
758.87 3.66
Cheetah-Run
756.71 + 20.1
802.80 18.3
828.60 + 25.9
814.62 + 19.5
792.52 + 16.9
Quadruped-Run
885.91 + 5.88
910.16
9
922.57
927.21 + 5.35
Dog-Wer Walk
838.81 + 14.3
886.73 7.38
895.54 9.94
880.50 + 9.96
D SENSITIVITY TO 8
Whilst we did not tune 3 for the results presented in Section 5, here analyse how sensitive REValueD is to the value of 3. Recall that in Equation (4.5) 8 controls how much the regularisation loss contributes the overall loss. In Table we present asymptotic results for various DM Control Suite tasks using varying 3 value. Note that we performed the same number of updates per task as Figure
We can see that REValueD is reasonably robust to 8, typically with the best performing / value lying somewhere between 0.5 and 0.75. In the finger-spin task, we see that B value of offers the best performance. Intuitively this would make sense, since there only two sub-action spaces it is less likely that credit assignment would an issue. Similarly in walker-run we see that there not much difference in performance for the tested 3 values. Much like finger-spin, this can be attributed to the fact that walker-run is a reasonably easy task and has only 6 sub-action spaces. Consequently, credit assignment may not have major impact that a 3 value of 0 performs significantly worse, with the best performing 3 values being 0.5 and 0.75. This further demonstrates the benefits of the regularisation loss, particularly in tasks with large N, whilst also showing the robustness to the value of B.
the performance. In the dog-walk task
E FURTHER RESULTS FOR VARYING 74;
In this Section present further results for varying n, in more DM control suite tasks. The results can be found in Figure 5. We observe that as n. increases. REValueD continues to exhibit better performance compared to DecQN and BDQ However, it's worth noting that in the cheetah-run task, as n, surpasses 75, the performance difference becomes less pronounced.
F FURTHER RESULTS IN STOCHASTIC ENVIRONMENTS
In Figure 6 we present more comparisons for stochastic environments. We that the results remain similar to those observed in Section 5, with the exception of the stochastic state variant of Quadruped-RR Run, where there is not significant difference between REValueD and DecQN.
G TRAINING STABILITY
To investigate how this variance reduction influences the stability of the training process, we present in Figure the Conditional Value Risk (CVaR) for the detrended gradient norms, suggested by Chan et al. (2019). The CVaR provides information about the risk in the tail of distribution and is expressed as
CV)R(g) [glg > VaRys%(())
Here, stands for the detrended gradient norm and VaR (Value at Risk) represents the value at the 95th percentile of all detrended gradient norm values. We perform detrending the gradient norm by calculating the difference in the gradient norm between two consecutive time steps, that
20

---

Published as a conference paper at ICLR 2024
Engarr sp 10
fingor 30
firger spin,
Anngrr 100
0.24
0.24
fadh-owim, 10
Fadh-wwim, 30
fadh-esimm, n 75
Balheestm, 100
793
2002
2.00
3008
400c
5006
1000
2008
3008
5306
100%
2016
3008
5306
100c
2006
3008
530
cheatathruun,
Chestal rutt,
chestah-run,
nun,
100
Inberactions
Enu
Tobered
Inberections
R.ValueD
Figure 5: Further results assessing how the performance of DecQN, BDQ and REValueD are af- fected by increasing the size of each sub-actions space. n corresponds to the size of the sub-action space, i.e. |A,] n for all The solid line corresponds the mean of 10 seeds, with the shaded area corresponding to a 95% confidence interval.
Tun
quadruped
Sinoo Ipporsppn
Test
38
Holiiorrun
quadruped
Sinoo
1004
203c
6:28
EXM
0.20
E.44
3.00
6.24
0.40
3.50
Interactions
Intecactions
Fin Interactions
Internec
DecQN
REValueD - IDQ
Figure 6: Stochastic environment tasks. In the top row we added Gaussian white noise (a = 0.1) to the rewards, whilst in the bottom row we added Gaussian white noise the state.
21

---

Published as a conference paper at ICLR 2024
is, 9++1 V Vel where V denotes the gradient. As shown in the Figure, employing the average the ensemble for the learning tärget in REValueD considerably lowers the CVaR for the tasks under consideration.
Algorithm
REValueD
DecQN
Average CVaR WarrGGrdieent Norm)
3.5
3
2.5
1.5
walker-walk
walker-run
fish-swim
cheetah-run quadruped walk quadruped-run
Task
Figure 7: Conditional Value at Risk (CVaR) for the gradient norms (which have been detrended) for both DecQN and REValueD, evaluated six tasks from the DeepMind Control Suite. The CVaR values presented here are the average across ten separate runs, and the figure includes both the mean values well error bars representing the standard error for each.
H CONTRIBUTION OF THE REGULARISATION LOSS IN A TABULAR FMDP
In this Section we design simple FMDP encompassing sub-action spaces, each containing n sub-actions. The FMDP includes one non-terminal state, with all actions resulting in transition to the same terminal state. We have assumed that each sub-action space has a single optimal sub- action. A reward of +1 is obtained when all optimal sub-actions selected simultaneously, whilst a reward of is given otherwise.
For these experiments, randomly initialise the utility values for every sub-action across all sub- action spaces using a Uniform(-o0 -00.1, 0.1) distribution, with the exception of the optimal sub-action the Nith sub-action space, which is initialised with value of +1. Actions then chosen follow- ing an e-greedy policy (e 0.1). After this, an update is performed on the transition data and we record the frequency which the optimal sub-action in the final sub-action space has been taken. Since the utility values have been randomly initialised, the probability of jointly selecting the first N optimal sub-actions is approximately - which means that most of the transitions are likely to result in reward of -1. Under the DecQN loss, performing an update on transition with a reward of will result in the value of the optimal sub-action the Nth sub-action space being decreased, despite it being initialised at optimal value. We aim to show that using the regularisa- tion loss from Equation (4.4) mitigates the effect that the transition has on the value the optimal sub-action.
Figure 8 shows the mean frequency of optimal Nth sub-action selection for both REValueD and DecQN. The mean is averaged over 1000 runs. and the shaded region represents a 95% confidence interval. A clear observation from this Figure that, as both and n increase, REValueD starts outperform DecQN. This result underscores the efficacy of the additional regularisation loss in
22

---

Published as a conference paper at ICLR 2024
reducing the impact of the sub-optimal sub-actioon actions from the first N dimensions the value of the optimal sub-action in the Nth dimension.
N S, 3
= n
N m 5, 10
N: m 5, 20
Fercertage of time opriaal (1 (3-b0- Lt- (7100)
688
200
400
25
20
400
206
20
400
200
X0
10
20
Pertertige thrm aptinal 13 0np ten cocee
N 20, n
N 20, n 5
20,
10
N- 20, n 20
Pecambogt thee axtinal
200
100
225
202
205
200
XG
50,
10
20
Pacortage ordmm extmaal 140 odtor choia
DecQN
REValueD
Figure& 8: This figure presents the outcomes from the tabular FMDP experiment (see Section 5). The x-axis represents the number of model updates performed, whilst the y-axis signifies the frequency with which the optimal Nth sub-action selected, expressed a percentage. The solid line corresponds to the mean performance over 1000 trials, and the shaded region corresponds to a 95% confidence interval.
1 FUNCTIONAL FORM OF THE REGULARISATION WEIGHTS
In Equation (4.4) we define the weighting w, = exp(-1o.1). The motivation for this form was that, for large differences, we want the weight to tend to 1, and for small differences be close O. In this Section we look to assess how the performance of REValueD is affected when using a different function to define the weights that also satisfies this property. In Figure 9 we compare to a quadratic function of the form w; = min 1). The results can found in Figure 10. We can see that generally speaking, the exponential weighting function offers better performance in terms of sample efficiency, although both reach roughly the same asymptotic performance. The explanation for this can likely be attributed to the functional forms, demonstrated in Figure 9, where the quadratic function will return a weight of for any | 2 1. This leads to regularising the utility functions too harshly. It is possible that temperature parameter could be incorporated to scale 6,, such that we now have 8/ ó as the input to the weighting function, but an analysis of this is beyond the scope of this paper.
23

---

Published as a conference paper at ICLR 2024
0.8
Value
0.6
0.4
0.2
Exponential
Quadratic
-3
-2
1
Delta
Figure 9:
Here we show the plots of the exponential weight function and the quadratic weight function.
quadruped
dog
0.24
0.61
CJM
2.8%
628
3.40
EAM
120
2004
302c
43%
500
0.24
0.49
0.6M
3.80
Ero Tréeractionss
Eno Interactions
Fmn Interactions
Fro Tnstereet
exponential quadratic
Figure 10: This Figure compares the performance of REValueD using the definition the weights from Equation (4.4) (exponential) vs. quadratic function. The solid line corresponds to the mean performance over 10 seeds, and the shaded region corresponds to a 95% confidence interval.
J METAWORLD
In this Section we offer further comparisons between DecQN and REValueD in some manipulation tasks from MetaWorld (Yu et al., 2020). The results can be found in Figure 11. We can that in these tasks REValueD achieves stronger asymptotic performance than DecQN, whilst also generally having smaller variance. This demonstrates the efficacy in domains other than locomotive tasks.
K FURTHER COMPARISONS WITH ENSEMBLED BASELINES
For more like-to-like comparison between REValueD and BDQ/De:QN-Sume have compared RE REValueD with variants these baselines equipped with an ensemble note that we have compared REValueD with DeseNeEnsemble in Table 1. The results for select environments be found in
24

---

Published as a conference paper at ICLR 2024
W1
Text
2006
K00
130m
Cula 3.4%
0.251
EW Insoractions
Eny Intoractions
Ent Enttrr octions
Eiv Diitoraccion
DecQN
REValueD
Figure 11: Here compare the performance of DecQN and REValuel) for discretised variants of manipulation tasks from MetaWorld. The solid line corresponds to the mean of 10 seeds, with the shaded area corresponding 95% confidence interval.
Figure 12. We can see that the results for BDQ remain largely unchanged, with the exception of finger-spin with bin size of 10, where BDQ+Ensemble now attains similar performance to that of REValueD. For DeeQN-Sun-Ennemble we note that the performance finger-spin is improved by equipping ensemble, with the performance achieving similar levels to REValueD) for 30, 75, 100. However, for the other tasks the performance is still largely unchanged.
figer-spin,
Tinger-sphn.o
Engeensphh, n 30
finger- prngerspin..g
fnnger spin, 100
3.20
0.6
0.01
026 2AM
232
E35
233 6AM 1.87
0.0
5MM 3.01
foh-swilm.
6sn-owim,
289
cheotal-run, =
cheztah-run,
Chectal-rur, n 30
cheotah-run, n 75
choctah-run n 100
150
2000
408
1304
1008
20k
30.6
4308
500
100c
2036
2308
400
206
1036
2008
200
48k
206
1308
2004
300
48%
530
dogusall,,
dog
dogonali,
dagewwll,
dogg-wwlke
Test
Interaations
REValueD DesQi--ummEEnemmbl BDQ+Ensembll
Figure 12: Here compare the performance of REValueD with BDQ+Ensemble and Sum- DeseN+En on discretised variants of the DM control suite tasks with varying bin sizes. The solid line corresponds the mean of 10 seeds, with the shaded corresponding 95% confidence interval.
25

---

Published as a conference paper at ICLR 2024
Text
780
1.24
14
0:44
786
332M
C.AM
0.01
Ern Interactions
Brw Interactions
Env Interactions
DesQN-CS1 BLPValueD
Figure 13: Comparison of DecQN-C51 and REValueD. The top row are non-stochastic DM control suite tasks; whilst the middle and bottom rows correspond to the same environments with Gaussian white noise (a = 0.1) added to the rewards and states, respectively. The solid line corresponds to the mean of 10 seeds, with the shaded area corresponding to 95% confidence interval.
L DISTRIBUTIONAL CRITICS
As noted in Section 6, a distributional perspective to learning can help deal with uncertainty induced by exploratory sub-actions. To that end, similar to the work in Appendix I of Seyde et al. (2022), compare our method, REValueD, to variant of DecQN which uses a distributional critic based on C51, introduced by Bellemare al. (2017). Rather than learning expected utility value for each sub-action, the critic now learns a distribution over values for each sub-action. The decomposition now proceeds at the probability level by averaging over logit values L = EM L/N.
We observe the results of the comparison between DecQN-C51 and REValueD in Figure 13, with comparisons being conducted in selected DM Control Suite environments and their stochastic vari- ants. We observe that, even when equipped with a distributional critic, REValueD maintains superior performance in all of the tasks.
26


Reviews:
[{"review_id": "frtKoVXrkS", "reviewer": "Reviewer_fnTY", "summary": "This paper studies RL in large discrete action space with factorization structure. Building upon the value decomposition idea proposed in past work, this paper presents some theoretical analysis which motivates the use of an ensemble of critics to control the variance as well as a new regularization term for better coordination over sub-actions. Experiments on discretized version of DeepMind control suite shows competitive performance.", "questions": "- Bias and Variance\n  - In Tang et al. 2022, the authors showed that using Q decomposition may **increase bias** and **decrease variance**. In theorem 1 of this paper, the authors claim using Q decomposition may **decrease bias** and **increase variance**. These results seem to contradict each other. Could you clarify any difference in the analyzed settings? \n  - Eqn (3.1) is valid to write since we are defining the parameterization of the Q-function $Q_{\\theta}$, however, in Eqn (4.2) for the term related to $U_i^{\\pi_i}$, the decomposition of the true Q-function $Q^{\\pi}$ might not exist, as discussed in Tang et al. 2022. Could you elaborate what assumptions is the current analysis operating under? \n\n- Fairer comparisons with baselines\n  - The proposed method appears to be modifying DecQN to incorporate (1) ensemble and (2) regularization. In Fig 1, REValueD should be compared to ensembled DecQN and ensembled BDQ for a more \"apples-to-apples\" comparison. \n\n- Other questions: \n  - In Fig 2, why does BDQ perform well on this task but not the tasks in Fig 1? Would BDQ with ensemble perform even better? \n  - Minor presentation suggestion: in Table 2, can you bold best results for each task to make it easier to read? \n\n- References:\n  - Tang et al. Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare. NeurIPS 2022. \nhttps://openreview.net/forum?id=wl_o_hilncS", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- Writing is clear. \n- Strong empirical results showing better performance than the baselines considered. \n- Ablation studies show the effect of regularization term and ensemble size, which is helpful for future readers who may want to adopt the proposed method.", "weaknesses": "- The theoretical analysis on \"DecQN target estimation bias and variance\" should be made more rigorous, and some of the conclusions in this paper seems to contradicts previous results in an uncited related work. \n- The justification for the \"Regularized value-decomposition\" seems more of a heuristic rather than a formal justification. \n- The authors should consider adding a few missing baselines to the main experiment in Figure 1 for a fairer comparison\n\nThese are elaborated in Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yJkBbHulg5", "reviewer": "Reviewer_bJ1N", "summary": "The paper proposes REValueD to enable Q-learning algorithms on tasks with high-dimensional discrete action space. With the same concept of value decomposition, REValueD improves DecQN by mitigating the target variance with an ensemble of critics and mitigating the effects of exploratory actions with a regularization loss. Experiments on the DeepMind Control Suite tasks show that REValueD consistently outperforms DecQN and another baseline.", "questions": "1.\tIn Equation (3.1), the value-decomposition form is the direct sum operator and REValueD follows this form, why do the authors not use other value-decomposition forms such as weighted sum?\n2.\tCould the authors explain more about the design of |δi|? Why “for large |δi|, the reward/next state is likely influenced by the effect of other sub-actions.”?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well-written and well-structured. Although the ideas of the ensemble and regularization is incremental, they are straightforward and make sense.\n2.\tThe theoretical analysis shows that REValueD could reduce the target variance of DecQN while maintaining the expectation of the target difference of DecQN unchanged with the help of the ensemble technique.\n3.\tThe experiments demonstrate the effectiveness of the proposed method and ablation studies are given to further validate each component of REValueD.", "weaknesses": "1.\tIn the experiments, only two baselines (DecQN and BDQ) are compared although the authors list several works in the related works.\n2.\tAn introduction and analysis of the action pace and sub-action space in humanoid and dog tasks may help the readers understand the setting and motivation more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OSzf3qCUHO", "reviewer": "Reviewer_Athy", "summary": "This paper studies a factorized Q-learning approach to solve continuous control problems. The approach builds on the recent Decoupled Q-Networks agent and makes two extensions: (1) critic ensembling to mitigate value variance, (2) a regularization objective to mitigate credit assignment issues stemming from exploratory sub-actions. The resulting REValueD agent improves learning efficiency as measured by gradient step count on common DeepMind Control Suite tasks.", "questions": "-\tHow would REValueD ensembling compare to distributional DecQN?\n-\tHow was the ensembling set up to ensure efficiency / what is the computational overhead?\n-\tWhat was the benefit of altering certain hyperparameters (exploration decay, Polyak averaging) and are these sufficient to explain the performance mismatch between DecQN in Figure 2 & 4 (REValueD) vs Figure 6 (DecQN)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe ensembling and regularization objective are natural extensions to the DecQN agent \n-\tThe resulting REValueD agent yields strong performance on the tasks being studied\n-\tAblations on individual components highlight the benefit of each addition in isolation\n-\tThe tabular FMDP experiment in Appendix G is a nice addition\n-\tThe paper is well-motivated, structured and written", "weaknesses": "-\tThe results focus on DeepMind Control Suite tasks from **proprioceptive inputs**. The primary baseline additionally evaluated on MetaWorld, Control Suite from vision, and an Isaac Gym-based locomotion task with a single set of hyperparameters (some minor variations). The results presented here would benefit from additional evaluations on more diverse environments.\n-\t**Control Suite performance** (Figure 1): the results would benefit from displaying reference performance of a strong recent continuous actor-critic agent. The benchmark results (throughout) are provided with “Gradient steps” on the x-axis, which deviates from the conventional “Time steps” metric. Readability would be improved by adding Figure 1 with “Time steps” to the Appendix and providing the necessary conversion values between “Gradient steps\" and \"Time steps” (e.g. gradient steps per time steps ratio).\n-\t**Increasing discretization** (Figure 2 & 4): the robustness of REValueD over increasing discretizations is impressive. Figure 6 of the DecQN paper showed approximately the same performance on both Dog-Walk and Finger-Spin when using 3 vs 21 bins – do you have an intuition for where this difference is coming from (Figure 6 of DecQN yields ~950 on Finger-Spin and ~850 on Dog-Walk for 21 bins). Is this in part due to the hyperparameter modifications over the original DecQN agent mentioned in Appendix B?\n-\t**Stochastic environments** (Figure 3 & 5): the DecQN paper also introduced a distributional DecQN agent based on the C51 critic in Appendix I with experiments in stochastic environments in Appendix J. Distributional critics can be viewed as an alternative method to ensembling in accounting for variability resulting from exploratory sub-actions, potentially side-stepping the computational overhead of explicit ensembles. Discussion and/or experimental comparison between DecQN+C51 or a distributional version of REValueD and ensembled REValueD would further strengthen the paper (this was briefly mentioned in the conclusion).\n-\tA potential downside of ensembles is the **computational overhead** required for training. It would be interesting to see how the plots compare when plotting time on the x-axis. While Appendix B notes that there is “minimal slowdown” for relatively small ensemble sizes, it would be helpful to quantify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)