An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
Authors: Alex Beeson
David Ireland
Giovanni Montana
Venue: Accepted by TMLR ()

Abstract: Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poses substantial challenges or risks.  Pivotal to the success of transferring RL offline is mitigating overestimation bias in value estimates for state-action pairs absent from data. Whilst numerous approaches have been proposed in recent years, these tend to focus primarily on continuous or small-scale discrete action spaces.  Factorised discrete action spaces, on the other hand, have received relatively little attention, despite many real-world problems naturally having factorisable actions.  In this work, we undertake a formative investigation into offline reinforcement learning in factorisable action spaces.  Using value-decomposition as formulated in DecQN as a foundation, we present the case for a factorised approach and conduct an extensive empirical evaluation of several offline techniques adapted to the factorised setting.  In the absence of established benchmarks, we introduce a suite of our own comprising datasets of varying quality and task complexity.  Advocating for reproducible research and innovation, we make all datasets available for public use alongside our code base: https://github.com/AlexBeesonWarwick/OfflineRLFactorisableActionSpaces

Published in Transactions on Machine Learning Research (11/2024)
An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
Alex Beeson 1:2, David Ireland", Giovanni Montanal.s,
Worwick Manufacturing Group, University of Worwick, Coventry, UK
Worwick Medical School, University of Warwick, Coventry, UK Department of Statistics, University of Warwick, Coventry, UK Alan Turing Institute, London, UK
alex. beeson @warwick.cc ac. ac.uk david. ireland wwwwwwck.cooooooo.o.......ooww.w ac. uk g. g.montana @ @warwick. uk
Reviewed on OpenReview: https: // openreview. net/ forum? id= STuzyUffNNV
Abstract
Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poscs substantial challenges or risks. Pivotal to the success of transferring RL oflline mitigaling overestimation bias value estimates for state-action pairs absent from data. Whilst numerous approaches have been proposed in recent years, these tend to focus primarily on continuous or small-scale discrete action spaces. Factorised discrete action spaces, on the other hand, have received relatively little attention, despite many real-world problems naturally having factorisable actions. In this work, we undertake a formative investigation into offline reinforcement learning in factorisable action spaces. Using calop-dccomooiitionn as formulated in DecQN as a foundation, we present the case for a factorised approach and conduct an extensive empirical evaluation of several offline techniques adapted to the factorised setting. In the absence of established benchmarks, we introduce suite of our own comprising datasets of varying quality and task complexity. Advocating for reproducible research and innovation, make all datasets available for public use alongside our code base
1 Introduction
The idea of transferring the successes of reinforcement learning (RL) to the offline setting is enticing one. The opportunity for agents to learn optimal behaviour from sub-optimal data prior to environment interaction extends RL's applicability to domains where data collection is costly, time-consuming or dangerous (Lange et al., 2012). This includes not only those domains where RL has traditionally found favour, such as games and robotics (Mnih et al., 2013; Hessel al., 2018; Kalashnikov et al., 2018; Mahmood et al., 2018), but also areas which online learning presents significant practical and/or othical challenges, such as autonomous driving (Kiran al., 2022) and healthcare (Yu et al., 2021a).
Unfortunately, taking RL offline is not as simple as naively applying standard off- off-policy algorithms to pre- existing datascts. A substantial challenge arises from the compounding and propagation of overestimation bias in value estimates for actions absent from data (Fujimoto el. al., 2019b). This bias stems from the underlying bootstrapping procedure used to derive such estimates and subsequent maximisation to obtain policies, whether implicit such as in Q-learning or explicit as per actor-critic methods (Levine al., 2020). Fundamentally, agents find it difficult accurately determine the value of actions not previously encountered, and thus any attempt to determine optimal behaviour based on these values destined to fail. Online, such

---

Published in Transactions on Machine Learning Research (11/2024)
inaccuracies can be compensated for through continual assimilation of environmental feedback, but offinee such a corrective mechanism is no longer available.
In response to these challenges, there has been a plethora of approaches put forward that aim to both curb the detrimental effects of overestimation bias as well as let agents discover policies that improve over those that collected the data to begin with (Tevine et al., 2020). The last few years particular have seen a wide variety of approaches proposed, making use of policy constraints (Fujimoto et al., 2019b; Zhou et al., 2021; Wu et al., 2019; Kumar et al., 2019; Kostrikov et al., 2021b; Fujimoto & Gu, 2021), conservative value estimation (Kostrikov et al., 2021a; Kumar et al., 2020), uncertainty estimation (An et al., 2021; Chasemipour et al., 2022; Bai et al., 2022; Yang et al., 2022; Nikulin et al., 2023; Beeson & Montana, 2024) and environment modelling (Kidambi et al., 2020; Yu et al., 2021b; Argenson & Dulac-Arnold, 2020; Janner et al., 2022; Yu et al., 2020; Swazinna et al., 2021), to name just a few. Each approach comes with its own strengths and weaknesses in terms of performance, computational efficiency, case of implementation and hyperparameter tuning.
To date, most published research in offline RL has focused either continuous small-scale discrete action spaces. However, many complex real-world problems can be naturally expressed in terms of factorised action spaces, where global actions consist of multiple distinct sub-actions, each representing key aspect of the decision process. Examples include ride-sharing (Lin et al., 2018), recommender systemss (Zhao et al., 2018), robotic assembly (Driess et al., 2020) and healthcare (Liu et al., 2020).
Formally, the factorised action space is considered the Cartesian product of finite number of independent discrete action sets, i.e. A A x ... AN, where A, contains n, (sub-)actions and N corresponds to the dimensionality of the action space. The total number of actions, often referred to as atomic actions, is thus I[; Mi, which can undergo combinatorial explosion if N and/or n, grow large. For example, in robolics a set of actions could correspond to moving a joint up, down or not at all, i.e. = 3. For machines with multiple joints the global action is the set of individual actions for each joint. The total number of possible global actions is thus 3 which grows exponentially as the number of joints increases.
In recognition of this possibility, various strategies in the online setting have been devised to preserve the effectiveness of commonly used discrete RL algorithms (Tavakoli et al., 2018; Tang & Agrawal, 2020). The concept of value- decomposccoooiiiooon (Seyde et al., 2022) is particularly prominent, wherein value estimates for each sub-action space are computed independently, yet are trained to ensure that their aggregate mean converges towards universal value. The overall effect is to reduce the total number of actions for which a value needs to be learnt from a product to a sum, making problems with factorisable actions spaces much more tractable for approaches such as Q-learning. Referring back to the robotics example, by using a decomposed approach the number of actions requiring value estimates is reduced from 3N to 3.N.
In this work, undertake an initial investigation into offinne RL in factorisable action spaces. Using value-decomposition, we show how factorised approach provides several benefits over standard atomic action representation. We conduct an extensive empirical evaluation of a number of offline approaches adapted to the factorised setting using a newly created benchmark designed to test an agent's 's ability to learn complex behaviours from data of varying quality. In the spirit of advancing research in this area, we provide open access to these datasets as well as our full code base: https: //::::/gitttnt //gtttu......./ com/ AlexBeesonWarvvick/
To the best of our knowledge, this investigation represents the first formative analysis of offline RL in factorisable action spaces. We believe our work helps pave the way for developments in this important domain, whilst also contributing to the growing field of offline RL more generally.
2
Preliminaries
2.1 Offline reinforcement learning
Following standard convention, we begin by defining a Markov Decision Process (MDP) with state space S, action space A, environment dynamics T(s' s, s,),, reward function R(s.a) a) and discount factor y E [0,1] (Sutton& & Barto, 2018). An agent interacts with this MDP by following state-dependent policy a(a 8).
2

---

Published in Transactions on Machine Learning Research (11/2024)
with the primary objective of discovering optimal policy x² s) that maximises the expected discounted sum of rewards, Ex ('r(s,a))
A popular approach for achieving this objective is through the use of Q-functions, (" a), which estimate the value of taking action a in state s and following policy x thereafter. In discrete action spaces, optimal Q-values can be obtained by repeated application of the Bellman optimality equation:
Q" a) = r(s, a) + TB mmx (s('saa')
a'
These Q-values can then be used to define an implicit policy such that =(s) arg max, Q(s,a) i.e. the action that maximises the optimal Q-value each state.
Given the scale and complexity of real-world tasks, Q-functions are often parameterised (Mnih et al., 2013), with learnable parameters that are updated so as to minimise the following loss:
L(Ø)=
1
Σ (Qo(s. (Qo(s.a)-y(rs5))) v(r,s')², (s.a.),"))
(1)
B
where y(r, = - y max. Qo(s', a²) is referred to as the target value, and denotes a batch of transitions sampled uniformly at random from replay buffer B. To promote stability during training, when calculating Q-values in the target it is common to use a separate target network Qa(s', a') (Mnih et al., 2013; Hessel et al., 2018) with parameters i updated towards 0 either via a hard reset every specified mumber of steps, or gradually using Polyak-averaging.
In the offline setting, an agent is prohibited from interacting with the environment and must instead learn solely from a pre-existing dataset of interactions B = (st. as, ro. s)). collected from some unknown behaviour policy (or policies) лs (Lange et al., 2012). This lack of interaction allows errors in Q-value estimates to compound and propagate during training, often resulting in a complete collapse of the learning process (Fuji- moto et al., 2019b). Specifically, Q-values for out-of-distribution actions (i.e. those absent from the dataset) suffer from overestimation bias as a result of the maximisation carried out when determining target values (Thrun & Schwartz, 1993; Gaskett, 2002). The outcome spocious Q-values estimates, and policics derived from those estimates consequently being highly sub-optimal. In order to compensate for this overestimation bias, -alauuss must be regularised by staying "close" to the source data (Levine et al., 2020)-
2.2 Decoupled Q-Network
By default, standard Q-learning approaches are based on atomic representations of action spaces (Sutton & Barto, 2018). This means that, in factorisable action space, Q-values must be determined for every possible combination of sub-actions. This potentially renders such approaches highly ineffective due to combinatorial explosion in the number of atomic actions. Recalling that the action space can be thought of as a Cartesian product, then for each A; we have that |AA = nıs and so the total number of atomic actions is I nij This quickly grows unwieldly as the number of sub-action spaces N and/or number of actions within each sub-action space n, increase.
To address this issuc, the Branching Ducling Q-Network (BDQ) proposed by Tavakoli et al. (2018) Icarns value estimates for each sub-action space independently and can be viewed as single-agent analogue to independent Q-learning from multi-agent reinforcement learning (MARL) (Claus & Boutilier, 1998). Seyde et al. (2022) expand on this work with the introduction of the Decoupled Q-Network (DecQN), which computes value estimates in each sub-action space independently, but learns said estimates such that their mean estimates the Q-value for the combined (or global) action. Such an approach highly reminiscent of the notion of value-decomposition used in cooperative MARL (Sunehag et al., 2017; Rashid al., 2020b;a; Du et al., 2022), with sub-action spaces resembling individual agents.
In terms of specifics, DecQN introduces utility function Uj, (8,0.) 0.) for each sub-action space and redefines the Q-value to bc:
1 N
Qo(s,a)= =
M Ub (s,
(2)
N
1=1
3

---

Published in Transactions on Machine Learning Research (11/2024)
where a (a1:... aN) is the global action, 8 are the parameters for the ith utility function and 0 {0,),- are the global set of parameters. The loss in Equation (1) is updated to incorporate this utility function structure:
I(0)
Σ
(Ql(s - v(r,8))
B
(s.a.ri")~
where
N
s(r,s)==1 =
7
Σ
max Uj, a()
N
As each utility function only needs to learn about actions within its sub-action space, this reduces the total number of actions for which a value must be learnt to Σ R4, thus preserving the functionality of established Q-learning algorithms. Whilst there are other valid decomposition methods, in this work we focus primarily on the decomposition proposed in DecQN. In Appendix H we provide a small ablation justifying our choice.
3
Related Work
3.1 Offline RL
Numerous approaches have been proposed to help mitigate Q-value overestimation bias in offline RL. In BCQ (Fujimoto et al., 2019b), this is achieved by cloning a behaviour policy and using generated actions to form the basis a policy which is then optimally perturbed by a separate network. BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019) and Fisher-BRC (Kostrikov et al., 2021a) also make use of cloned behaviour policies, but instead use them minimise divergence metrics between the learned and cloned policy. One-step RL (Brandfonbrener et al., 2021; Gulechre al., 2020a) explores the idca of combining fitted Q-ovaluation and various policy improvement methods to learn policies without having to query actions outside the data. This is expanded upon in Implicit Q-learning (IQL) (Kostrikov et al., 2021b), which substitutes fitted Q- evaluation with expectile regression. TD3-BC (Fujimoto & Gu, 2021) adapts TD3 (Fujimoto et al., 2018) to the offline setting by directly incorporating behavioural cloning into policy updates, with TD3-BCNN/SAC. BC-N (Beeson & Montana, 2024) employing ensembles of Q-functions for uncertainty estiimation to alleviate issues relating to overly restrictive constraints as well as computational burden present in other ensembles based approaches such as SAC-N & EDAC (An et al., 2021), MSG (Ghascmipour et al., 2022), PBRL (Bai et al., 2022) and RORL (Yang et al., 2022).
In the majority of cases the focus is on continuous action spaces, and whilst there have been adaptations and implementations in discrete action spaces (Fujimoto et al., 2019a; Gu et al., 2022), these tend to only consider a small number of (atomic) actions. This is also reflected in benchmark datasets such as D4RL (Fu et al., 2020) and RL Unplugged (Gulcehre et al., 2020b). Our focus is on the relatively unexplored area of factorisable discrete action spaces.
3.2 Action decomposition
Reinforcement learning algorithms have been extensively studied in scenarios involving large, discrete action spaces. In order to overcome the challenges inherent in such scenarios, numerous approaches have been put forward basod on action sub-sampling (Van de Wiele al., 2020; Hubert et al., 2021), action embedding (Dulac-Arnold al., 2015; Gu et al., 2022) and curriculum learning (Farquhar et al., 2020). However, such approaches tailored to handle action spaces comprising numerous atomic actions, and do not inherently tackle the complexities nor utilise the structure posed by factorisable actions.
For factorisable action spaces various methods have been proposed, such as learning about sub-actions independently via value-based (Sharma et al., 2017; Tavakoli et al., 2018) or policy gradient methods (Tang & Agrawal, 2020; Scyde et al., 2021). ).. Others have also framed the problem of action selection in factorisable action spaces as sequence prediction problem, where the sequence consists of the individual sub-actions (Metz et al., 2017; Pierrot et al., 2021; Chebotar et al., 2023).
4

---

Published in Transactions on Machine Learning Research (11/2024)
There exists strong connection between factorisable action spaces and MARL, where the selection of sub-action can be thought of as an individual agent choosing its action in multi-agent setting. Value- decomposition has been shown to be an effective approach in MARL (Sunehag et al., 2017: Rashid et al., 2020b;a; Du et al.; 2022), utilising the centralised training with decentralised execution paradigm (Kracmer & Banerjee, 2016), which allows agents to act independently but learn collectively. DecQN (Seyde el al.; 2022) and REValueD (Treland Montana, 2023) have subsequently shown such ideas can be used with factorised action spaces in single-agent reinforcement learning, demonstrating strong performance on range of tasks that vary in complexity. Theoretical analysis of errors in Q-value estimates has also been conducted in efforts to stabilise training (Treland & Montana, 2023; Thrun & Schwartz, 1993).
In this work, we focus on adapting DecQN to the offline setting by incorporating existing offline techniques. Whilst prior work has explored offline RL with value decomposition (Tang et al., 2022), this was limited specific low-dimensional healthcare applications using only BCQ. Furthermore, accurately evaluating performance in such domains is notorious challenging (Gottesman al., 2018). In contrast, we systematically study multiple offline methods using low and high-dimensional factorised action spaces across a suite of benchmark tasks.
4 A case for factorisation and decomposition in offline RL
As mentioned in Section 2.2, value-deccompositoon provides mechanism for overcoming challenges in stan- dard Q-learning arising from exponential growth in the number of atomic actions. This can be seen most clearly from a practical standpoint, in which the number of actions that require value estimation is sig- nificantly reduced when moving from atomic to factorised action reprosentation (sce Table for example). However, even in cases where atomic representation remains feasible, there are still several benefits to a factorised and decomposed approach which are particularly salient in the offline case.
To see this, we begin by making the assumption that Q-value estimates from function approximation Qo(s, a) carry some noise E(s, a) (Thrun & Schwartz, 1993). This noise can be modelled a random variable, such that @o(s,a)= = (s. a) + <(s,a), where Q" (s. a) is the true but unknown Q-value for policy л. For clarity, we emphasise this noise stems from the fact the function approximator cannot represent the true Q-function prociscly, as opposed to other factors such as sampling variation. As noted by Thrun & Schwartz (1993), the expected overestimation will be maximal if all actions for a particular state share the same target Q-value. Incorporating all these ideas into the Q-learning framework we can define the target difference
zdm =7( max 20('',a') max Q" (''('a)))
(4)
= = (ma c(5' (s))
To incorporate decomposition as defined in Equation 2, the above can be extended to utility value estimates (Ircland & Montana, 2023). Now we assume that utility value estimates from function approximation Uj, (s,0s) carry some noise e'(s, d:), such that (s, (s, (s), where U (s, a;) is the true but unknown utility value for policy Ty. Under the additional assumption that true Q-values decompose in the same way as Equation 2 (Ireland & Montana, 2023), we can define the target difference under DecQN as
1 N
N
doc
max Uū, (j,(5', (5,
Σ
max U,"(('' (6))))) d
N
N
(5)
7 max é (s', (c.)
of
We can expand on each of these by considering the composition of global actions in terms in-distribution and out-of-distribbutin sub-actions. When actions are represented atomically, a particular global action is either in-distribution or out-of-distribution, depending on its presence (or absence) in the dataset. However, under factorised representation global action is composed of sub-actions (a1, 02 -. aN), and
5

---

Published in Transactions on Machine Learning Research (11/2024)
hence it is the individual sub-actions that are either in-distribution out-of-distribution This means that global actions which are out-of-cistribution can contain individual sub-actions that are in-distribution when factorised, as illustrated in Figure 1.
T T
T
-
T v r
1 v v
[ T + ]
v r p
[ V |
v t v
v
Dataset
Atomic
Factorised
Figure In this simple example there are N 3 sub-action dimensions, each with two actions ft, 4}. In-distribution and out-of-distribution actions/sub-actias For particular state, the dataset contains two global actions. Under atomic representation only actions which match those in the dataset are in-distribution. Under factorised representation, individual sub-actions which match those in the dataset are in-distribution. Atomic actions that are out-of fitt-f--disributton can contain sub-actions that are in-distribution when factorised.
highlighted in green and red, respectively.
In recognition of this property, we now differentiate between errors value estimates for in-distribution and out-of-distribution actions. Using the precept that out-o--distr errors are inherently larger than in- distribution errors, let etx (s, a²",, e'"(s, at*) and (s, a""),, end a."ht denote the errors in Q-value/utilit value estimates for in-distribution at" at" and out-of-distribution aut ag global/sub-actions, respectively. The target difference for DQN and DecQN under this framework become
Zdan =1 max e a ( max out (s, a m) arm
(6)
aoue
N
Zdec
max max " (ss, aa") max in (s, a aqtn)) )))
(7)
N
ag"
nout
where we have dropped the prime symbol to avoid overloading notation. Now the maximum is taken pooled sample of errors from two different distributions opposed to single distribution in Equations 4 and 5.
The important point to note here is that both cases the target difference is now dependent on the rela- tive number of in-distribution and out-of-distribution actions/sub-actions Since errors from in-distribution actions/sub-actions are smaller than for out-of-distribution, the larger the number of in-distribution actions/sub-acttios the smaller the expected overestinaation. In the offline setting, we cannot change the status of an action/sub-action from out-of-distribution to in-distribution through environment interaction. However, as noted above, we can improve the coverage of sub-actions through factorisation. Hence, for the same dataset we can potentially reduce overestimation bias moving from an atomic action representation to a factorised one. This is particularly beneficial in the offline setting where issues relating to overestimation bias primarily stem from Q-value estimates for out-of-distribution actions.
In Appendix A we provide support for the ideas presented in this Section by empirically assessing the properties of target differences under DQN and DecQN for uniformly distributed noise.

---

Published in Transactions on Machine Learning Research (11/2024)
4.1 Appropriateness of decomposition in the offline setting
The previous analysis assumes both the approximate and true Q-function decompose per Equation 2. In general, such decomposition is unlikely to perfectly capture intricate aspects of an environment. Nonethe- less, such a decomposition has been shown to be effective in the online setting (Ireland & Montana, 2023), and we note that under the following conditions it can act as reasonable approximation, allowing us to retain the aforementioned benefits of improved action//ub-action coverage.
Weak inter-action dependence
If the effect of each sub-action the reward the next state is relatively independent of other sub-actions, the decomposition is more likely to hold. This condition aligns with the notion of action independence in factored MDPs, where the transition dynamics can be decomposed across sub-actions with minimal cross- interaction. In such cases, the global Q-function can be closely approximaled by a sum of local utility functions. This can also be tied to work on factored MDPs (Kearns & Koller, 1999), where state and action space factorisations allow efficient policy computation under cortain independence assumptions. Of course, true independence of sub-actions in an MDP might be rare in practice, especially when complex dependencies exist between sub-actions (e.g., coordination between multiple control signals). Still, if dependencies are weak, factorisation can be good approximation. It's also important to emphasise this only holds under certain independence assumptions that often arise in factored MDPs but may not be present in more general MDPs.
Approximately factorisable reward structures
The decomposition is also reasonable when the reward function can be roughly expressed as a sum over individual sub-actions, which is again related to the literature reward decomposition in factored MDPs (Guestrin al., 2003). In scenarios where the global reward is combination of local rewards for each sub- action, factorisation provides a natural approximation for value function decomposition. The caveat is that, in many real-world scenarios, rewards are not perfectly factorisable, and there could be interactions between sub-actions that affect the total reward in a non-linear way. Thus, such an approximation may break down in complex environments with highly entangled reward structures. However for certain structured problems, especially those with weak interactions, this approximation can still yield practical benefits.
4.2 Limitations of decomposition in the offline setting
It is important to highlight that while the benefits of a decomposed approach in the offline setting can offset shortcomings in modelling inter-action dependence, there are limitations. For settings where sub-actions are strongly dependent, the value of a global action may differ siguificantly to the average value of its constituent utilities, introducing errors that are no longer outweighed by the reduction in overestimation bias from factorisation/ decomppoiinn..ooooiitttio..oo
As example, consider the phenomena of pharmacodynamic and pharmacokinetic drug interactions. In healthcare setting, it is common for patients to receive multiple drugs as part of a treatment regime, with dosages and routes adjusted based on how the patient responds. If our goal is to train an offline RL to optimise treatment regimes, the level of interaction (positive or negative) between would dictate whether factorised/decomme /deeomosedd approach would be appropriate. Tf there little to no interaction then a factorisod/ /atssd/d/dcoppoeed approach could be effective. If there are strong interactions a dettorised/dccompeee approach would likely fail to capture these important characteristics and hence be much less effective.
5 Algorithms
In this Section we introduce several approaches for adapting DecQN to the offline setting based on existing techniques. We focus on methods that offer distinct takes on combatting overestination bias, namely, policy constraints, conservative value estimation, implicit Q-learning and one-step RL. In each case, attention shifts
7

---

Published in Transactions on Machine Learning Research (11/2024)
from Q-values to utility values, with regularisation performed at the sub-action level. The full procedure for each algorithm can be found in Appendix B
5.1 DecQN-BCQ
Batch Constrained Q-learning (BCQ) (Fujimoto et al., 2019b;a) is a policy constraint approach to offline RL. To compensate for overestimation bias for out-of-distribttio actions, a cloned behaviour policy T is used to restrict the actions available for larget Q-values estimates, such that their probability under the behaviour policy meets a relative threshold т. This can be adapted and integrated into DecQN by cloning separate behaviour policies Tye for each sub-action dimension and restricting respective sub-actions available for corresponding target utility value-estimates. The target value from Equation (3) becomes:
v(r,8)= = r
max (8'.0 .4)
where p¹(a)) T2, = 8 maxa; o (d) 8') is the relative probability of sub-action de under policy T Each cloned behaviour policy is trained via supervised learning with Φ = fo] The full procodure can be found in Algorithm 1.
5.2 DecQN-CQL
Conservative Q-learning (CQL) (Kumar et al., 2020) attempts to combat overestimation bias by targeting Q- values directly. The loss Equation (1) augmented with term that "pushes-up" on Q-value estimates for actions present in the dataset and "pushes-down" for all others. Under one particular variant this additional loss lakes the form
(0)
[log Σ exp(Qo(s, - QQ)
B
8,a~B
a;EA
which equates to performing behavioural cloning using log-likelihood when the policy is a softmax over Q-values (Luo et al., 2023) since
exp(Qo(s,a))
log Σ
exp(Qo(s,a:) - -Q((s,)) =
Σ
log
B
[
Eaee exp(o(s.a)))
8
EA
8,a~B
log Ta s).
B
8
This variant can be adapted and incorporated into DecQN by "pushing-up" on utility value estimates for sub-actions present in data and "pushing-dowwe down" for all others. The additional loss under DecQN becomes:
(cQ
log Σ (x, - Ub, (s,a:))
(8)
s
1=1
ayEA,
where aj denotes the jth sub-action within the ith sub-action space, and a a hyperparameter that controls the overall level of conservatism. The full procedure can be found in Algorithm 2.
Note this particular implementation docs not directly substitute the decomposition per Equation into the original CQL loss. Instead, we apply CQL directly to the utility values and then take the mean across sub-action dimensions. This way avoid having to estimate Q-values for all atomic actions (which a direct substitution would require) and we retain the equivalence to behavioural cloning at the sub-action level as
log exp(U (s, a:)) Ea,ea, cxp(U (,))
log Σ 0, EA
exp(UU,(s, (8, (s,
N
B
B
s
log no (a: s))
B
N 1=1
8
8

---

Published in Transactions on Machine Learning Research (11/2024)
5.3 DecQN-IQL
Implicit Q-learning (TQL) (Kostrikov et al., 2021b) addresses the challenge of overestimation bias by learning policy without having to query actions absent from data. A state and state-action value function are trained on the data and then used to extract a policy via hagaat/gwwwiiiihcoooooooooaaaalo
The state value function (8) is trained via expectile regression, minimising the following loss:
I(W) =
Σ 8,0~
[L5(Q0(8,)) a) (+)];
B
where if we denote Q0(s, a) (s(s then Lī(u) - 1(u 0) j²2 is the asymmetric least squares for
the T E (0, 1) expectile. The statc-action value function Qo a) is trained using the same loss as Equation (1), with the target value now u(r, 8') = (s')
The policy follows that of discrete-action advantage- weighteddeeegedde---ee wera adaptod to the factorised sctting and integrated with DecQN leads the following:
(Luo et al., 2023), which
1 A A(ss,a) + log log (a: 8)
= arg Max di
is the (facvorised) advantage function, Té. (a: 8) is the cloned behaviour policy for sub-action space i trained via supervised learning and is a hyperparameter controlling the balance between reinforcement learning and behavioural cloning. The full procedure can be found in Algorithm 3.
where A(s, =
5.4 DecQN-OneStep
We can derive an alternative approach to IQL which removes the requirement for a separate state value function allogether. Noting that "(x)=E. = Taa [s) we can instead use the cloned behaviour policy To s') and state-action value function Qa a') to calculate the state value function V(s') instead. This can be adapted and incorporated into DecQN by replacing Q(s, a) with its decomposed form per Equation (2) and adjusting the policy to reflect a sub-action structure. We denote this approach DecQN-OneStep it mirrors one-step RL approaches that train state value functions using fitted Q-evaluation (Brandfonbrener et al., 2021). The full procedure can be found in Algorithm 4.
6 Environments and datasets
At present, there are relatively few established auviromments/talle specifically designed for factorised action spaces. As such, there is an absence of benchmark datasets akin to those available for continuous or small- scale discrete action spaces such as D4RL (Fu et al., 2020) and RT Unplugged (Gulcehre et al., 2020b). Tn light of this, we introduce our own benchmarking suite constituting the following environments and tasks.
Maze: A simple mase-based environment first introduced by Chandak al. (2019) in which an agent is tasked with reaching target goal location. The state space is continuous and comprises the agent's 's current location in the maze. The agent's movement is controlled by a series of N actuators which can be turned on or off, with each actuator corresponding to a unit of movement in a single direction, with actuator applying force at an angle of 2wi/N when activated, as illustrated Figure 2. The net outcome of actuator selection is the vectoral summation of the movements associated with each actuator. The agent's 's action space comprises the unique combination of the set of binary decision with respect each actuator, leading to to action set that is exponential in the number of actuators such that |A| = N
DecpMind control suite: A discretised variant of the DeepMind control suite (Tunyasuvunakool et al., 2020), as previously adopted by Seyde et al. (2022); Ireland & Montana (2023). This suite contains a variely of environments and tasks, which although originally designed for continuous control can easily be repurposed for a discrete factorisable sctting by discrotising individual sub-action spaces, thus creating scenarios that can vary in size and complexity. This discretisation process involves selecting a subset of actions from the original continuous space which then become the discrete actions available to the agent. For example,
9

---

Published in Transactions on Machine Learning Research (11/2024)
N
N
N
Examples maze environment. with different numbers of actuators. The star represents the largel goal location, the red dot the agent and the arrows the actuators. Adapted from original Figure in Chandak al. (2019).
Figure 2:
continuous action in the range [-1, can be discretised into three discrete actions corresponding to the subset. 0, l]. We emphasise this discretisation procedure happens prior to data collection, i.e. we are not discretising an existing continuous action dataset.
The choice of environments and tasks refiects our desire to provide a benchmark that is amenable factori- sation and decomposition to varying degrees, allowing us to investigale the limits of the assumptions and conditions outlined in Section 4. For the maze task, the action set is inherently factorisable for each actuator, yet reaching the target goal requires at least some coordination across actuators... For the DeepMind control suite, each sub-action corresponds to the adjustment of specific robotic component (e.g. joint), but to successfully complete the task may require alignment across some or all components. We emphasise that our benchmark is not intended to cover large scale discrete action settings for which iittorisationddeeoooosstttt is not feasible.
For the datasets themselves, we follow similar procedure to D4RL. Using BReQN/REValueD, we train agents to "expert" and "medium" levels of performance and then collect transitions from the rosulting policies. Here, we define "expert" to be the peak performance achieved by DecQN/REValuu and "medium" to be approximately 1/3rd the performance of the "expert". We create a third dataset "medium-expers by combining transitions from these two sources and a fourth "random-mediuu-ccper" containing transitions constituting 45% random and medium transitions and 10% expert. Each these datasets presents a specific challenge to agents, namely the ability learn from optimal or sub-optimal data ("expert" and "medium", respectively) well as data that contains a mixture ("medium-expert" and More details on this training and data collection procedure provided in Appendix C.
7 Experimental evaluation
We train agents using DecQN, DecQN-BCQ, DecQN-CQL, DecQN-TQL, and DecQN-OneStop our bench- mark datasets and evaluate their performance in the simulated environment. We also train and evaluate agents using a factorised equivalent of behavioural cloning to provide a supervised learning baseline. Per- formance is measured in terms of normalised score, where score 100 acorexspert -BCOTerandom representing a random policy and 100 the "expert" policy from the fully trained agent. We repeal exper- iments across five random seeds, reporting results as mean normalised scores + one standard error across soods. For cach set of experiments we provide visual summarics with tabulated results available in Ap- pendix G for completeness. Full implementation details, including network architectures, hyperparameters and training procedures provided in Appendix D
score- scorerandom with
10

---

Published in Transactions on Machine Learning Research (11/2024)
7.1 Case study: DQN-CQL DecQN-CQL
Before considering the full benchmark, we conduct study directly comparing DQN-CQL (i.e. CQL under atomic action representation) and DecQN-CQL. Following the procedure outlined in Section 6 we construct "medium-oxpert" dataset for the "chectab-run" task for number of bins n; E {3, 4, 4,5,6, 5, and compare the performance of resulting policies and overall computation time. In addition we construct a "random-mmddum-xxppr"" dataset for the Maze task with 15 actuators for varying dataset sizes and compare performance and computational resources.
In Figure we see that as the number of sub-actions n, increases for the "cheetah-run" task, DQN-CQL exhibits notable decline in performance, whereas DecQN-CQL, performance declines only marginally. We also see dramatic increase in computation time for DQN-CQL whereas DecQN-CQL remains roughly constant. For DQN-CQL, these outcomes are symptomatic of the combinatorial explosion in the number of actions requiring value-estimation and the associated number of out-of-distribbtiio global actions. These issues are less provalent in DecQN-CQL due to its factorised and decomposed formulation. To provide further insights, in Appendix F we also examine the evolution of Q-values errors during training, finding these errors are consistently lower for DecQN-COL than DQN-CQL, aligning with each algorithm's respective performance.
100
200
Training Time (mins)
Mean Score
60
20
3
6
4
5
Number of bins
Number of bins
DQN-CQL DecQN-CQL
Figure 3: Comparisons performance (left) and computation time (right) for DQN-CQL and DecQN-CQL the
datasot for varying numbers of bins. As the number of bins increases, DQN-CQL suffers notable drops in performance and increases in computation time, whereas DecQN-CQL is relatively resilient. in both areas.
In Figure 4 we see that as the size of the dataset for the Maze task decreases, DQN-CQL exhibits a more notable decline in performance than DecQN-CQL. This is particularly the case when the size of the dataset is very small, i.c. 250 transitions. These outcomes support notion of better action coverage under factorised representation than atomic, where value estimates for observed sub-actions can be combined to provide better estimates of global actions that haven't been observed. Furthermore, this is achieved through much more computationally efficient mcans, with training time for DecQN-CQL being over times faster than DQN-CQL (4mins vs 34mins, respectively) and GPU usage times less (246MBs vs 1728MBs, respectively).
7.2 Maze
In Figure summarise each algorithm's performance for varying numbers of actuators and dataset com- position. Across the board, that DecQN sans offline modification results in policies that perform
11

---

Published in Transactions on Machine Learning Research (11/2024)
100
Mean Score
40
20
5000
2500
1000
500
250
100
Number of transitions in dataset
DQN-CQL DecQN-CQL
Figure 4: Comparison of performance for DQN-CQL and DecQN-CQL on the Maze task with N 15 actuators with wrandom-mddimm-ee datasets of varying size. As the number of transitions in the dataset decreases, DQN-CQL, suffers more notable drops in performance in comparison to DecQN-CQL.
better than random, a direct consequence of aforementioned issues relating to overestimation bias. With the exception of DecQN-BCQ for mondoom-medumg crddium-expert" datasets, all offline methods match or outperform be- havioural cloning, particularly for datasets constituting high levels of sub-optimal trajectories ("medium" and wrandom-medium-exxpee
For datasets containing proportion of expert trajectories Trmedium-expert all offline methods (again with the exception of DecQN-BCQ) are able ex-
and "randomm-mediiu-exxeer tract expert or near-expert level policies. Comparing just the offiine RL methods, we see there is little to separate them for "expert" and "medium-cxpert" regardless of the number of actuators. For "medium" and mednddommmddiuu--xppprrt""" DecQN-CQL exhibits an edge when the number of actuators is between and 12, but for "medium" this lost to PecQN-IQL/Oneste as the number of actuators increases 15.
7.3 DeepMind Control Suite
In Figure 6 we summarise each algorithm's performance across the full range of tasks, setting the number of sub-actions n, = 3. This necessitates the use of value-decompositioon for all but the most simple tasks, as highlighted in Table In general we see that all offline RL methods outperform behavioural cloning across all environments ttasks and datasets, with the exception of DecQN-BCO for "random "random-medium medium-expert" datasots which performs quite poorly. In terms of offline methods specifically, in general DeeQN-CQL. has slight edge over others for lower dimensional lasks such "finger-spin", "fish-swim" and "cheetah-run" whilst DecQNN-QQL/Onettep have the edge for higher-dimensione, tasks such "humanoid-stand" and "dog- trot". For "medium-expert" datasets we see in most cases all offline methods are able to learn expert near-expert level policies. Extracting optimal behaviour from "random- ceriddo--mdiuu-xxpeett datasets proves significantly more challenging than in the Maze environment, likely result. of these environments/ tasks being much more complex coupled with datasets being both highly variable and constituting relatively few expert trajectorics in relation trajectory length.
7.3.1 Increasing the number of sub-actions
To help provide insights into the ability of our chosen offline methods to scale larger and larger action spaces, increase the number of sub-actions within each sub-action space and repeat our experiments. We
For "mediiu--xxpert" datascts note that BC able to achieve expert-level performance par with offline methods. We attribute this the relative simplicity of the environment coupled with reasonable level of expert-level trajectories within the datasets. Such outcomes are replicated in DMC tasks which more complex and have much longer trajectories
12

---

Published in Transactions on Machine Learning Research (11/2024)
Maze 3 actuators
Maze 5 actuators
Maze actuators
200
Mean Score
60
-00
20
20
20
Exp
MedFipp
Med
PansMedEpp
Exp
MedFapp
RaodMerFFp
Exp
MedPip
RondMMedEap
Maze 10 actuators
Maze 12 actuators
Maze 15 actuators
100
100
80
83
8C
Mean Score
-00
Exp
ModEao
Me0
RandMedExp
Exp
MoSE4S
Mca
RandMedEap
Ex0
MedExc
Mos
RandMerExp
Dataset
Dataset
Dataset
BC DecQN
BCQ
CQL
IQL OneStep
Figure 5: Performance comparison on maze task for varying numbers of actuators. For presentation purposes the prefix "DecQN-" " has been omitted for each offine method. In general, all offline RL methods improve over behavioural cloning, with the exception of DecQN-BCQ for "random-medium-expa" without any offiine modification performs poorly across the board.
datasets. DecQN
focus in particularly on the dog-trot environment since this is by far the largest in terms of actions. We collect datasets following the same procedure outlined in Section 6 for number of bins n, E {10, 30, 50, 75, 100).
that our chosen offline methods robust to increases We summarise results in Figure 7. In general, behavioural cloning (with the same exception for DecQN- in the number of bins, continuing to outperform and extract near-expert policies from "medium-expert" datasets, with
BCQ on "random-medd" medimm-expertteeee DeeQN--IQT/-Oneetta maintaining their edge over DecQN-CQL. For arandon-medium-exper datasets we start to notice a decline in performance as we approach the upper end of our number of bins, most noticcably when n = 100. This is likely a consequence of more and more actions exacerbating the difficulties in oblaining good policies from highly variable and largely sub-optimal data.
8 Discussion and conclusion
In this work, we have conducted a formative investigation of offline reinforcement in factorisable action spaces. Through empirical evaluation, have shown how factorised and decomposed approach offers mumerous benefits over standard/atomie atomic approaches. Using bespoke benchmark we have undertaken extensive empirical evaluation of several offline RL approaches adapted to this factorised and decomposed framework, providing insights into each approach's 's ability to learn tasks of varying complexity from datasets of differing size and quality.
In general, our empirical evaluation demonstrates our chosen offline methods adapt well to the factorised setting when combined with valne-decomposition in the form of DecQN. With one exception, all approaches are consistently able to outperform behavioural cloning regardless of data quality, and where datasets contain
13

---

Published in Transactions on Machine Learning Research (11/2024)
FingerSpin
FishSwim
CheetahRun
85
D0
Mean Score
43
Exp
MedFap
Med
RandhledEsp
Exp
MedFapp
RandbledFapp
MedPipp
RondtdedEap
QuadrupedWalk
HumanoidStand
Doat
Mean Score
80
100
60
00
20
20
Exp
MccErp
Ned
RancheedExp
Exp
WccE44
Med
RandMedExp
Exp
MedEe
Mos
RandMedExp
Dataset
Dataset
Dataset
BC DecQN
BCQ
CQL IQL OneStep
Figure 6: Performance comparison across benchmark for ni; = 3. In general, all offline RL methods improve over behavioural cloning, with the exception of DecQN-BCQ for celtom--eiiu-meee datasets. DecQN without any offline modification performs poorly across all environments/tase ttases..st/
sufficient levels of high-quality trajectories (i.e. "expert" and "medium "medium-experrt)) obtain policies. There is however notable room for improvement for datasets with scarcity of high-quality trajec- tories for more complex tasks (i.e. "medium"" and "randoon-medduu--expee" for DMC enviromments/ten tasks).
expert
Our initial investigation opens up numerous other possibilities for future research. One of these is the de- velopment techniques for automatically tuning hyperparametors during training, which at present are not environment/last agnostic (only data quality). In addition, as with continuous counterparts, performance can be enhanced by allowing hyperparameters to vary for each dataset (see Appendix D.1 for examples). Off-policy evaluation could also prove beneficial here (Rebello al., 2023), providing assurances on the quality of policy prior to deployment.
alternative approaches to modelling the behaviour policy Tे may help improve performance for more challenging datasets. Incorporating other methods outlined in Section 3 may also prove beneficial. For example, the use ensembles for capturing uncertainty in value estimates has been shown perform well in combination with behavioural cloning continuous action spaces (Beeson Montana, 2024), and relatively straightforward extension the approaches we consider here.
For SOQ.-/-BCO/--QT//-Oneeeeee
Whilst DecQN offers simple, effective and computationally efficient foundation for offinne RL in factorisable action spaces, we note there some inherent assumptions and limitations to the value decomposition approach that warrant further investigation, as noted in Sections 4.1 - 4.2. In particular, the efficacy of DecQN relies on individual sub-action optimisation leading to globally optimal joint policies. However, for tasks with sparse rewards or complex sub-action dependencies, individually learned sub-policies may fail properly compose into a coherent overall policy. For example, assembly tasks, separately learned pick, place, and connect skills could lead to conflicting behaviors when combined. Additional research into modeling sub-action interactions during decomposition could help overcome this limitation.
14

---

Published in Transactions on Machine Learning Research (11/2024)
DogTrot bins
DogTrot 10 bins
DogTrot 30 bins
60
Mean Score
00
-00
40
20
Exp
MedFipp
Medi
PansMeddEp
Exp
MedFapp
FavoMMelFap
Exp
MedPipp
RondMdedEa
DogTrot- bins
Dog Trat- 75 bins
ToaT 100 bins
100
80
03
Mean Score
-00
Exp
MetEoo
Med
RanaMedExp
Exp
MosE4s
Med
RandMedEap
Exc
MedEx
Mol
RandMedExp
Dataset
Dataset
Dataset
BC DecQN
BCQ
CQL
IQL OneStep
Figure 7: Performance comparison for dog-trot for n, E (3, 10, 30, 50, 75, 100). Each approach is reasonably resilient increases in the number of bins, although for ceailoo--mdium--eee datasets extracting a good policy appears to become more challenging gets very large.
Our technical analysis can also potentially be expanded upon by considering the degree to which a global ootpr/ttoooootttooddsirrutttttnt Implicitly we have assumed errors from actions/sub-actions
action is are the same regardless of how many times global action appears in the dataset. Relaxing this assump- tion could help refine the analysis, particularly for stochastic environments where multiple interactions can provide additional information. Further refinement may also be possible by addressing assumptions Q- function decomposition (as highlighted in Sections 4.1 - 4.2), particularly the true Q-function which may not necessarily decompose in the same manner as DecQN.
Beyond algorithmic considerations, there are also several avenues for exploration in relation to the environ- ments and tasks themselves. While our benchmark provides a broad range of diverse tasks, any particular one is deficient in either complexity (i.e. maze) or being naturally factorisable (i.c. DMC). Thus, the creation of bespoke environments that help bridge this gap would be valuable contribution, providing more realistic scenarios to evaluate against. In addition, future work could investigate more nuanced aspects relating to action space discretisation, such as variable numbers of bins, non-even spacing between actions, clustering of actions and maskod actions.
We hope our work underscores the unique setting and challenges of conducting offline RL in factorisable action spaces and paves the way for future research by providing an accessible and solid foundation from which to build upon.
Acknowledgments AB acknowledges support from University of Warwick and University of Birmingham NHS Foundation Trust. G.M acknowledges support from UKRI Al Turing Acceleration Fellowship (EPSRC OP//V24468/11)
15

---

Published in Transactions on Machine Learning Research (11/2024)
References
Gaon An, Seungyong Moon, Mang-Hyum Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement lcarning with diversified Q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021. 2,
Arthur Argenson and Gabriel Learning Representations, 2020. 2
Model-based oflline planning. In International Conference on
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Confer- ence on Learning Representations, 2022. 2,
Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning. Machine Learning, 113(1):443-455 2024. 2, 4, 14
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. Advances in neural information processing systems, $4:4933-49M4 2021. 4, 9
Yash Chandak, Georgios Theocharous, James Kostas, Scolt Jordan, and Philip Thomas. Learning action representations for reinforcement learning. In International conference on machine learning, pp. 941 950. PMLR, 2019. 10, 23
Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autore- gressive Q-functions. In Conference on Robot Learning, pp. 3909 3928. PMLR, 2023. 4
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AAAT/IAAT, (998(776-752)2 1998. 3
Driess, Ha, and Toussaint. Deep visual reasoning- learning to predict action sequences for assembly tasks. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4645 4651. IEEE, 2020. 2
Wei Du, Shifci Ding, Lili Guo, Jian Zhang, Chenglong Zhang, and Ling Ding. Value function factorization with dynamic weighting for deep multi-agent reinforcement learning. Information Sciences, 615:191 208, 2022. 3,
Gabriel Dulac-Arnola Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXie preprint vxXiv:1512.07676 2015. 4
Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnaeve. Growing action spaces. In International Conference on Machine Learning, pp. 3040-3051. PMLR, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.07211, 2020. 4, 9
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34:20132-20145, 2021. 2,
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587 1596. PMLR, 2018. 4
Scott Fujimoto, Edoardo Conti, Mohammad Ghavanzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01 01700 2019a. 4, 8
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019b. 1, 4, 8
16

---

Published in Transactions on Machine Learning Research (11/2024)
Chris Gaskett. Q-learning for robot control. PhD thesis, Australian National University, 2002. 3
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncortainties for oflline RI. through ensembles, and why their independence matters. Advances in Neurul Information Processing Systems, 35:18267-18228 2022. 2, 4
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv:1805. 12298, 2018. 5
Pengjie Gu, Mengchen Zhao, Chen Chen, Dong Li, Jianye Hao, and Bo An. Learning pseudometric-base action representation for oflline reinforcement learning. In International Conference Machine Learning, pp. 2002-9918, PMLR, 2022. 4
Carlos Guestrin, Daphne Koller, Ronald Parr, and Shobha Venkataraman. Efficient solution algorithms for factored MDPs. Journal of Artificial Intelligence Research, 19:399 468, 2003. 7
Caglar Gulcehre, Sergio Gómez Colmenarejo, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascamu, Nando de Freitas, et al. Addressing extrapolation error in deep offline reinforcement learning. Offline Reinforcement Learning Workshop Neural Information Processing Systems, 2020, 2020a. 4
Caglar Gulcchre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zoina, Rishabh Agarwal, Josh S Merel, Daniel Mankowitz, Cosmin Paduraru, et al. Rl unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:7248-7255 2020b. 4, 9
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirlg-second AAAI conference on ortificial intelligence, 2018. 1, 3
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference Learning Representations, 2018. 24
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning complex action spaces. In International Conference on Machine Learning, pp. 4476 4486. PMLR, 2021. 4
David Ireland and Giovanni Montana. Revalued: Regularised ensemble value-decoompositon for factorisable Markov decision processes. In The Twelfth International Conference on Learning Representations, 2023. 5, 7, 9, 21, 24, 25, 28, 29
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible bchavior synthesis. In International Conference on Machine Learning, pp. 9902-9913. PMLR, 2022. 2
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on rohot learning, pp. 651-673. PMLR, 2018.
Michael Kearns and Daphne Koller. Efficientt reinforcement. learning in factored MDPs. In IJCAT, volume 16, pp. 740-744 1999.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21823, 2020.
2
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X. 2014. 24, 25
17

---

Published in Transactions on Machine Learning Research (11/2024)
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. A1 Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving; A survey. IEEE Transactions Intelligent Transportation Systems, 23(6):4909 4926, 2022. 1
Nya Kostrikov, Rob Forgus, Jonathan Tompson, and Ofrr Nachum. Offline reinforeement learning with Fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774 5783. PMLR, 2021a. 2,
Tla Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-Learningg Tn International Conference on Learning Representations, 2021b. 2, 4, 9
Landon Kraemer and Bikramjit Banerjec. Multi-agent reinforcement lcarning as a rehearsal for decentralized planning. Newrocompuling, 190:82 94, 2016. 5
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019. 2,
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline rein- forcement learning. Advances in Newral Information Processing Systems, 33:1179 1191, 2020. 2,
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Springer, pp. 45-73, 2012. 3
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01653
2020. 1, 2, 3
Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. Efficient large-scale fleet management via multi- agent deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1774-1783, 2018. 2
Siqi Liu, Kay Choong See, Kee Yuan Ngiam, Leo Anthony Celi, Xingzhi Sun, and Mengling Feng. Rein- forcement learning for clinical decision support in critical care: comprchensive review. Journal of medical Internet research, 22(7)=e18477 84777 2020. 2
Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action-quantized offline reinforcement learning for robotic skill learning. In 7th Annual Conference on Robot Learning, 2023. 8,
A Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmark- ing reinforcement learning algorithms real-world robots. In Conference robot learning, pp. 561-591 PMLR, 2018. 1
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep RL. arXiv preprint arxiv: 701.10050000 05035, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXic preprint orXiv:1312.5602 2013.
1, 3
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In Internalional Conference Machine Learning, pp. 21928-26244.. PMLR, 2023. 2
Thomas Pierrot, Valentin Macé, Jean-Baptiste Sevestre, Louis Monier, Alexandre Laterre, Nicolas Perrin, Karim Beguir, and Olivier Sigand. Factored action spaces in deep reinforcement learning. 2021.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33:10199-10221 2020a. 3,
18

---

Published in Transactions on Machine Learning Research (11/2024)
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1 51, 2020b. 3,
Aaman Peter Rebello, Shengpu Tang, Jenna Wiens, and Sonali Parbhoo. Leveraging factored action spaces for off-policy evaluation. In ICML Workshop on New Prontiers in Learning, Control, and Dynamicol Systems, 2023. 14
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeeer, and Daniela Rus. Is bang- bang control all you need? solving continuous control with Bernoulli policics. Advances in Neural Information Processing Systems, 01122200-77222 2021.
Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via Q-learning. In The Eleventh International Conference on Learning Representations, 2022. 2, 3, 5, 9, 24, 25, 29
Sahil Sharma, Aravind Suresh, Rahul Ramesh, and Balaraman Ravindran. Learning to factor policies and action-value functions: Factored action space representations for deep reinforcement learning. arXiv preprint ar XrXi::1705.0 07269,, 2017.
Peter Sunehag, Guy Lever, Audrumas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jader- berg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposttin networks for cooperative multi-agent learning. arXiv preprint arXiv: 1006. 1706.05296, 2017. 3, 5
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. 2,
Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104:104366, 2021.
Shengpu Tang, Maggic Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Newral Information Processing Systems, 1134272-34286 2022. 5
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In
Proceedings of the anai conference on artificial intelligence, volume 34, pp. 5981 5988, 2020. 2, 4
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep reinforcement learning. In Proceedings of the vasi conference on artificial intelligence, volume 32, 2018. 2, 3, 4, 28
Scbastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist. Models Summer School, volume 255, pp. 263. Hillsdale, NJ, 1993. 3, 5, 21
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Frez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Muih. Q-learning in enormous action
spaces via amortized approximate maximization. ai preprint arXiu:2001.00 008100111111., 2020. 4
Yilan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXio
preprint Xix::111111 1111911111611.. 11361, 2019. 2, 4
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline
reinforcement lcarning via conscrvative smoothing. In Advances in Neural Information Processing Systems. 2022. 2, 4
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1 (1):113136 2021a.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129 14142, 2020. 2
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline modcl-based policy optimization. Advances in neural information processing systems,
2021b. 2
Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep reinforcement learning for page-wise recommendations. In Proceedings of the 12th ACM conference recommender systems, pp. 95-103, 2018.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pp. 1719-1735. PMLR, 2021. 2
20

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
A DQN vs DecQN for uniformly distributed errors
To provide support for the ideas presented in Section 4, we investigate properties of the target difference under DQN and DecQN based on uniformly distribution noise.
As demonstrated by Thrun & Schwartz (1993) and Treland & Montana (2023), if c(s, a) are modelled independent identically distributed (i.id.)) uniform random variables Ui-b, 6), then for |A| actions the expectation and variance of the target difference are, respectively,
|
E ZsZ = 7b
(9)
|A| + 1'
|A|
Var(Z,) 74b²
(10)
(IA) + +))((AA] +
For |A] > i.e. more than one action, B[ is positive and increasing w.r.t. |A| whereas Var(Z.) is positive and decreasing. Hence there is a bias/variance trade-off for overestimation in target Q-values.
This trade-off becomes apparent when comparing overostimation for DQN and DocQN, where |A|= - " for the former and |A| = En n4 for the latter. As shown by Ireland & Montana (2023), if e' a,) are also modelled as i.idd uniform random variables U(-b,b). then the expected target difference under DecQN is lower than DQN (R[zgre) < E[ZZ(")), but the variance is higher (V[zdee) 2 V(Z,7")).
We now consider this bias/variance variance tradc-off in the context of in-distribution and out-of-distribution actions/sub-aacions. This time, let U(-b, b) and U(-kb, kb) be the distribution of errors for in-distribution and out-of-listribution either in-distribution or dut-of-disttibuuionn the above conclusions hold, with the additional observation that both bias and variance are lower when all actions are in-distribution compared to all distr--ditt (as b < kb).
sitiins/uub-cctions respectively, where > 1. When all actions//ubbaactions are
The picture becomes more complex when there is mixture of in-distribution and out-of-distribution This stems from the fact there is no longer a closed form for the expectation/ / vaaiiance of the target difference since errors arise from two different distributions. This is exacerbated by the relative coverage of sub-actions and atomic actions. In terms of expectation, since this is an inereasing function of both b and |A] we can deduce the property E|Z < E|z holds regardless of the value of or the number of in-distribution sitions/subbactions. The variance is more nuanced since this is an increasing function for but decreasing for [.]. The value of k and number of in-distribution actions/sub-actions will determine whether the property V[Z > V(Z still holds.
To illustrate this, we conduct series of simulations based on different action/sub-actin configurations and compare how the expectation and variance of the target difference under DQN and DecQN change based on the coverage of atomic actions. The details are as follows.
DQN simulations: Let | and |A be the number of in-distribution and out-of-distribution atomic actions for a given state, respectively, such that |A| = |A'n| + |A For |Λ'*| E {0, (0.,,..,,|A sample | values from a U(-b,b) b) distribution and |A| -|A'1| values from a U(-kb, kb) distribution and store the maximum value for the pooled sample. Repeat this 10000 times and calculate the mean and variance of these stored maximurss. These are the estimates of [[ and V[Zdem).
DecQN simulations Let |A|"| and |49"*| be the number of in-distribution and out-of-distribution sub- actions in dimension i for given state, respectively, such that |A,| ++/qurr) For | E (0.,,..,,|4] |4[} sample | atomic action from all available actions without replacement. and calculate the number of factorised actions [Ai"| in each sub-action dimension i. In each sub-action dimension i sample | values from a U--, b) distribution and |A,] - l|Ai"| values from a U(-kb, kb) distribution and store the maximum value for the pooled sample for each dimension i. Calculate and record the average maximum value across all dimensions N (as per the DecQN decomposition). Repeat this 10000 times and calculate the mean and
21

---

Published in Transactions on Machine Learning Research (11/2024)
variance for these stored maximums. Since sub-action coverage is dependent which atomic actions are sampled, we repeat this entire procedure 100 times to sample broad range of atomic actions and calculate the overall mean and variance across these 100 simulations. These are the estimates of E[Z and V[zgec.
In Figure we summarise these simulations for three configurations: N = each with two sub-actions, N 4 each with two sub-actions and =3 each with three sub-actions. We set = and 2. For the expectation, we see that in all cases [[ - B[ for all values of |A For the variance, we see that in all cases V(Zgrc 2 "[ for values of | close to 0 and |A| but between V[zgrc < V[ This aligns with our intuition, namely that the expectation of the target difference under DecQN will always be lower than DQN, but the variance will depend on the relative proportions of in-distribution actions/sub- actions under atomic and factorised representations. Note that if all atomic actions in-distribution out-of-distribution (i.c. |Aih| or | [Al) the expectation and variance revert back to Equations 9 and 10.
B Algorithms - full procedures
This section contains the full procedures for algorithms detailed in Section 5.
Algorithm 1 DecQN-BCQ
Require: Threshold т, discounter factor y, target network update rate μt, number sub-action spaces N and dataset B.
Initialise utility function parameters 0 {0:) corresponding target parameters 0 = and policy parameters Ø = [d
for t= 0 to T do Sample minibatch of transitions (s,a,r,6) 8') from Φ + arg min, N N -Es,a, log (a; 8)
0 <- arg ming Es,ar (Qo(s, a) y)
where:
Q0(8, a) = 1/N Em
i=1 Uü (8, at),
y - Ei=1 maxo 2() Uj (5', a(),
(a())) = n/. (a') s/)/ maxa; &é, (0/ 80)
0 1 + (1 =²)
end for
Algorithm 2 DecQN-CQL
Require: Conservative coefficient a, discounter factor 7, target network update rate μ, number sub-action
spaces N and dataset B.
Initialise utility function parameters 0 == =0,) and corresponding larget parameters 0 ==0
for == 0 to T do Sample minibatch of transitions (,a,r,s) from B
0 <- arg ming E.aar. (Qa(s, 3) a E., TV ΣN [log Ea,es, exp(Ut) (8, - Ub, (s. a:]]
where:
Qa(8, a) = 11/N == U2 (8, az),
y -=r+ 1-N+1/N max Ui 6. a.))
i 8 1 + (1 - 0
end for
22

---

Published in Transactions on Machine Learning Research (11/2024)
E(2): DQN vs DesQN (N=33 n=2)
Vartzy VSN'VS Vs DecON (N-3, n-
1.5
DON
Doog4
E0]
Vanzi
0.29
24
0.1
0.0
Clohil actions in decntoutioon
Glubo actions inciisoribuion
B(Z) DQN DecQN
Varzz: DQN vs DecON (N=4, n=
1.5
DQN
Deogy
1.5
R21
Var
0.
Glcbal actins Indisntaution
cloba actions Insossstbuuton
E(2): DON DecQN
VartZ) DQN DecQN (N-3, n-3)
ngN
0.5
003
Gicbal actonsin cisribution
Globol mndisttutton
Figure 8: Comparing the expectation and variance of target differences under DQN and DecQN for various configurations and coverages of coomic/sub-actionss
C
Data collection procedures
C.1
Maze
For the Maze environment, we used the default settings as per (Chandak et al., 2019) with the exception that all actions are available from the beginning. To collect the datasets, we first trained agents for varying numbers of actuators using DecQN, parameterising utility functions as 2-layer MI.Ps each with 512 nodes
23

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 3 DecQN-IQL
Require: Expectile 7, discounter factor Y; target network update rate μe, number sub-action spaces N and
datasct B.
Initialise utility function parameters 0 = {0:);=1 and corresponding targel parameters 0 = 0. Initialise state value function parameters v and policy parameters & = [ for / t= 0 to T do
Sample minibatch of transitions (s, r,))) from B
@ < arg m
1 Ei= -E,,a log 0. (a: 8)
N
0 < arg ming s. (Q0 a) y)²
& arg m 5.a (Qg Vo((s)]
where:
Qo(s, a) = 1/N Σ == U (s)
@5(o,a) a) = 1/N Ex=
U (;))
b,
F= (s')
i - 110 + (1 - -1)0
end for
Algorithm DecQN-OneStep
Require: Discounter factor 7, target network update rate μ4, number sub-action spaces N and dataset B. Initialise utility function parameters = {0:) and corresponding target parameters Ô = Initialise policy parameters Φ { for = 0 to T do
Sample minibatch of transitions (s, (s, r') from B
1 @ < arg min, & N Di=1 Es.a, log n. (a: 8)
0 < arg ming E.,,,, (Qo(s, a) y)
where:
Qa(s, a) = 1/N EN U (s)
y == 1/N Di=1 Ea, o (as 88) (f)
Ei
i 8 μ1 + (1 =1)
end for
that take in state and output sub-action utility values. We maintained a fixed exploration parameter with 0.005. We used the Adam optimiser (Kingma& & Ba, 2014) with learning rate 3e - 4 and a batch size of 256.
= 0.1 and updated target network parameters using
Once we have trained the DecQNs we create the benchmark datasets by collecting data using greedy policy derived from the learned utility values. Each dataset contains 10k transitions. For the expert policy we trained the DecQNs until their test performance approached the maximum return possible (approximately 100). For the medium policy we aimed for test performance of approximately 1/3rd of the expert.
C.2 DeepMind Control Suite
To collect the DMC suite datascts we followed the training procedures laid out by (Scyde et al., 2022: Ireland & Montana, 2023) to train the Decoupled Q-networks. To expedite the data collection process during training we used distributed setup using multiple workers to collect data in parallel (Horgan et al., 2018). We parameterised the utility functions using a (shared) single ResNet layer followed by layer norm, followed by a linear head for each of the sub-action spaces which predicts sub-action utility values. Full details regarding network architecture and hyperparameters can be found in Table 1.
Once we have trained the DecQNs we create the benchmark datasets by collecting data using greody policy derived from the learned utility values. For the case study in Section 7.1 each dataset contains 100k transitions and for the main experiments in Section 7.3 each dataset contains 1M transitions. As each episode
24

---

Published in Transactions on Machine Learning Research (11/2024)
is truncated at 1,000 time steps in the DM control suite, this corresponds to collecting 100 episodes for the case study and 1,000 episodes for the main experiments. For the expert policy we trained the DecQNs until their test performance corresponded to the performance given in (Seyde et al., 2022; Ireland & Montana, 2023). For the medium policy we aimed for a test performance of approximately 1/3rd of the reported expert score.
We largely employ the same hyperparameters as the original DecQN study, as detailed in Table 1. Exceptions include the decay of the exploration parameter (c) to minimum value instead of keeping it constant, and the use of -oeyaik-v for updating the target network parameters, as opposed to hard reset after every specified number of updates. Finally, we sample from the replay buffer uniformly at random, as opposed to using a priority. We maintain the same hyperparameters across all our experiments.
For n 3 we use DecQN to train networks and collect datasets. For n > we use REValueD to train networks and collect datasets due to better scaling to higher numbers of bins (Ireland & Montana, 2023).
Table 1: Hyperparamelers used in DecQN and REValueD training.
Parameters
Value
Optimizer
Adam
Learning rate
10-4
Replay size
5 x 105
n-step returns
3
Discount,
0.99
Batch size
256
Hidden size
512
Gradient clipping
40
Target network update parameter,
0.005
Imp. sampling exponent
0.2
Priority exponent
0.6
Minimum exploration, c docay rate
0.05
0.99995
Regularisation loss coefficient 3 Ensemble size K
0.5
10
D Full implementation details
For both Maze and DMC environments//assk,, utility functions are parameterised by neural networks, com- prising a 2-layer MLP with ReLU activation functions and 512 nodes, taking in normalised state as input and outputting utility values for each sub-action space. We use the same architecture for policies, with the output layer a softmax across actions within each sub-actions-space. State value functions mirror this architecture except in the final layer which outputs a single value. We train networks via stochastic gradient descent using the Adam optimiser (Kingma & Ba, 2014) with learning rate 3e-4 and a batch size of 256. For state and state-action value functions we use the Huber loss as opposed to MSE loss. We set the discount factor ? 0.99 and the target network update rate μ = 0.005. We utilise a dual-critic approach, taking the mean across two utility estimates for target Q-values. For the maze task, agents are trained for 100k gradient updates. For the DMC tasks, agents are trained for 1M gradient updates.
The only hyperparameters we tune are the threshold T in BCQ, conservative coefficent a in CQL, expectile T and balance coefficient X in IQL and balance coefficient À in OneStep. We allow these to vary across environment/last but to better reflect real-world scenarios where the quality of data may be unknown, we forbid variation within auvironments/tasle.
Table 2 provides hyperparameters for all environments/tasks. For DecQN-BCQ we searched over T (0.025, 0.05, 0.1, 0.25, 0.5, 0.75). For DecQN-CQL. searched over {0.25, 0.5, 1,2). For DecQN-IQT.
25

---

Published in Transactions on Machine Learning Research (11/2024)
searched over {0.5, 0.6, 0.7,0.8), 10, 20, For DeeQN-OneStep searched over À = 2,, 20,50).
Table 2: Hyperparameters for experiments in Section 7
Environment taaiiinttttakkka task
Number of bins (n)
BCQ r
CQL a
IQL B,
OncStop
Maze
3
0.5
1
0.5, 20
50
Maze
5
0.5
0.25
0.5, 20
20
Maze
7
0.5
0.5
0.5, 50
20
Maze
10
0.5
0.5
0.5, 50
20
Maze
12
0.5
0.5
0.5, 50
50
Maze
15
0.5
0.5
0.5, 20
20
FingerSpin
0.5
0.25
0.5,
FishSwim
0.25
0.25
0.5,
CheetahRun
3
0.05
0.25
0.5, 1
1
QuadrupedWall Walk
3
0.25
0.25
0.5, 2
1
HumanoidStand
3
0.5
0.25
0.5, 2
2
DogTrot Trot
3
0.25
1
0.5, 5
5
DogTro Trot
10
0.25
0.25
0.5, 5
2
Trot
30
0.75
1
0.5, 2
2
DogTrot
50
0.5
0.5
0.5, 2
2
DogTrot
75
0.7
0,5
0.5, 2
2
Dog Trot
100
0.5
2
0.5, 5
5
D.1 Allowing hyperparameter variation within environment/task
As per Section 8, in Table we provide examples of performance improvement after permitting hyperpa- rameter variation within the same environment/task. In general, we lower quality datasets benefit from smaller lyperparameters (i.e. those that weight more towards RL and less towards BC) and higher quality datasets benefit. from larger hyperparameters (i.e. those the weight more towards BC and less towards RL). This mirrors findings from previous papers outlined in Section 3.
E Number of actions under atomic and factorised representations
Table summarises the number of actions requiring value estimation under atomic and factorised represen- tations.
F
Case study Q-value errors
To provide further insights into Q-value errors for the case study in Section 7.1, we examine the evolution of Q-value errors during training. Every gradient updates we obtain a MC estimate of true Q-values using discounted rewards from environmental rollouts. We then compare these MC estimates with Q-values predicted by both DQN-CQL and DecQN-CQL networks for the respective actions taken. To make better use of rollouts (which can be expensive), caleulate MC estimates and DQN- MQN-/DecQN-CQL Q-values for using a discount factor of 7 0.99 discounts rewards by over 99%
the first 500 states in the trajectory, for time-steps beyond 500 (and all tasks considered have trajectory length 1000). In total we perform 10 rollouts, giving 5000 estimates of the error between true and DQN-CQL/ CEQQL/DccQNCQ!! Q-values. In Figure 9 we plot the mean absolute error over the course of training, with the solid line representing the mean across five random soeds and shaded area the standard error. For all values n, we observe the mean absolute error is less for DecQN-CQL than DQN-CQL, particularly for n, > 3, aligning with each algorithm's 's respective performance in Figure 3.
26

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Individual performance allowing hyperparameters to vary within environment/taske "dog-trot', 3. Figures are mean normalisod scores, with 0 and 100 representing random and expert policios, respectively. Highest score highlighted in bold
Environment -dataset
DogTrot (BCQ)
T
0.025
T
0.05
T
0.1
T
0.25
-expert
57.8
85
93.6
94.4
-medium-expert
3.9
10.1
42.7
74.9
-medium
39.8
38.8
34.2
49.1
-random-medhum-oppert
5
5.6
9
0.1
DogTrot (CQL)
0.25
a 0.5
a
a 2
-expert
90.8
95.7
99.5
100.2
-medium-expert
76.6
81.7
84.8
75.1
-medium
50.6
48.3
46.5
45.2
manddm--medum-cxpee
41.2
40.8
43.4
38.6
DogTrot (IQL 0.5)
X
==5 5
À 10
-expert
37.9
82.5
98.9
99.5
-medium-expert
33
64
89.3
98.6
-medium
58.8
56.5
52
47.3
-random- mesiiomm-mdiuu--xppeet
10.6
28.6
44.1
44.7
DogTrot (OneStep)
x=
À x=
À 5
À = 10
-expert
53.3
91.7
101.2
102
-eddim--xpertt
44.5
79.4
93.9
96.6
medium
59.3
57.5
50.2
48
readomm-meddun--exper
23.8
43.9
44.9
45.1
Table 4:
Environment details for DeepMind Control Suite. |S] represents the size of the state space and
N the number of sub-action spaces. II. n4 is the total number of actions under atomic representation and Σ n4 under factorised representation when = 3.
Environment
[S]
N
II,
Σ
Finger Spin
9
2
Fish Swim
24
5
243
15
Cheetah Run
17
6
729
18
Quadruped Walk
78
12
530k
36
Humanoid Stand
67
21
R 10
63
Dog Trot
223
38
~ 10
114
G
Tabulated results
Tabulated results for Figure are presented in Table 5. Tabulated results for Figure are presented in Table 6. Tabulated results for Figure 5 are presented in Table 7 Tabulated results for Figure 6 are presented in Table 8. Tabulated results for Figure 7 are presented in Table
Number seeds
task for ten seeds and compare performance in Tables 10 and 11, respectively. We see consistent results across different seed counts. We expect similar consistency across all tasks as the same underlying methodology is applied throughout our experiments.
To check five seeds is sufficient for evaluation purposes
27

---

Published in Transactions on Machine Learning Research (11/2024)
Number
Number bins
Mean Absolute Error
14
20
12
45
10
Number of bins
Number of bins =
Mean Absolute Error
16
14.
12
10
10
100k
150k
200k
250k
300k
350k
400k
453k
5o0k
50k
100k
150k
200k
250k
300k
350k
400K
45Ck
500k
Updates
Updates
-DeQNN-CD
Figure 9: Comparison of estimated errors Q-values for cetiiia--ru-mmdiuu-o-ppeerrr datasot for varying numbers of bins. Errors are lower for DecQN-CQL for all numbers of bins, most notably for ny >> mirroring the deviation in performance levels between the two approaches.
Table 5: DQN-CQL vs DecQN-CQL performance and computation comparison for cheetah-run task. Performance figures are mean normalised scores one standard error, with 0 and 100 representing random and expert policies, respectively. Computation figures are training time and GPU usage. Actions figures are total number of actions requiring value estimation based on atomic
atomic/facttorised factorised representation.
Method
n2
Actions
Score
Training time (mins)
GPU usage (MB)
DQN-CQL
3
729
79.7 + 6.0
19
266
DQN-CQL
4
4096
30.2 + 1.7
30
412
DQN-CQL
5
15625
32.2 + 1.4
85
958
DQN-CQL
6
46656
32.3 + 4.0
227
2388
DecQN-CQL
3
18
92.7 + 3.7
19
244
DecQN-CQL
4
24
85.4 3.8
19
246
DecQN-CQL
5
30
79.5 + 1.8
20
246
DecQN-CQT.
6
36
84.3 + 2.5
20
246
H Decomposition comparisons
In this Scction we compare the DeeQN dccomposition two alternative methods that can be used for factorisable discrete action spaces. The first based on the Branching Dueling Q-Network (BDQ) proposed by Tavakoli et al. (2018). Using our notation, each utility function is considered its own independent Q- function, i.e.
(s, = U2 (s.
(11)
Each Q-function is trained by bootstrapping from its own larget, and decomposition is used. That is, the target for Qh (s, is given by y ma 0 (s', at). The findings of Treland & Montana (2023) demonstrate that, the online setting, BDQ is unable match the performance of DecQN. This is likely caused by the fact that, each sub-action space now learnt independently. the effects of other sub-actions
28

---

Published in Transactions on Machine Learning Research (11/2024)
Table 6: DQN-CQL vs DecQN-CQL- performance and computation comparison for Maze task with N 15 actuators. Performance figures mean normalised scores + one standard error, with 0 and 100 representing random and expert policies, respectively. Computation figures are training time and GPU usage. Actions figures are total number of actions requiring value estimation based on fomiic/fctooiiee representation.
Method
Number of transitions
Actions
Score
Training time (mins)
GPU usage (MB)
DQN-CQL.
100
32768
0.0 + 0.0
34
1728
DQN-CQL
250
32768
48.8 + 4.9
34
1728
DQN-CQL
500
32768
68.5 + 2.7
34
1728
DQN-CQL.
1000
32768
73.7 + 2.7
34
1728
DQN-CQL
2500
32768
90.7 + 2.5
34
1728
DQN-CQL
5000
32768
96.0 + 1.4
34
1728
DecQN-CQL
100
30
31.9 + 10.6
4
246
DecQN-CQL.
250
30
69.2 + 3.2
246
DecQN-CQL
500
30
84.9 2.3
4
246
DecQN-CQL
1000
30
84.8 + 1.5
4
246
DecQN-CQL
2500
30
96.4 + 1.7
4
246
DecQN-CQL
5000
30
99.7 + 0.2
4
246
are treated effects of the environment dynamics. Due to the fact that cach agent is contimually updating its own policy, this leads to non-stationary environment dynamics, making the learning problem much more challenging.
We also consider an alternative value-decomposition technique to the mcan, namely the sum. That is, we replace the mean operator in Equation with the sum operator:
Qo(s,a)= = Σ (x,
(12)
Whilst this may seem a subtle change, Ireland Montana (2023) proved that the mean and variance of the learning target under this decomposition are both higher than DecQN. Empirical experiments by Seyde et al. (2022); Ireland Montana (2023) also confirm the inferior performance of the sum dccomposition compared to the mean.
see that the sum decomposition is less performant in each of the tasks and datasets than the mean. For BDQ, we see that whilst in some cases performance is better than using the sum decomposition, is generally still less performant than using the mean decomposition. Owing to these results, we focus on the mean decomposition in our main work.
In Table 12
29

---

Published in Transactions on Machine Learning Research (11/2024)
Table 7: Individual performance comparison for Maze for varying numbers actuators. Figures are mean normalised scores + one standard error, with and 100 representing random and expert policies, respectively, At the bottom of the table we also provide totals, split by data quality and the entire set. We see that DecQN-BCQ the least. performant of the oflline methods and that BeQN--QQI/TQL//Oneee similarly overall and expert, medium-exert and and medium datasets and DecQN-CQL. the best
perform
datasets.
Environment -dataset.
BC
DecQN
DecQN-BCQ
DecQN-CQL
DecQN-IQL.
DeoQN-OneStep
Maze (Aclaalors= = 3)
expert
100.1 +
-5.6 +
100.1
100.2
100.1 +
100.1 +
mediium-xxpert
98.7 1
-5.6
100.1 +
99.9 0
-modium
55.3 + 2.5
-5.6 1
54.7 i 1.7
62.8 1 5.3
54.6 1
57.1 1 6.5
raaddo--edium--ere
67.4 + 11.9
15.5 + 46.1
34 + 5.1
95.3 + 3.9
92.2 = 2.9
95.6 + 1.9
Maze (Actuators = 5)
-expert
100.1 1011:0
-1.7
100.1 =
100.1 +
100.1 +
100.1 = 0
mediium-xxpert
99.6 + 0.6
-1.7 4.
99.4 = 0.9
100.1 +
99 + 2.3
100 1 0.1
-medium
32.3 + 4.3
-1.7
32.7 + 2.8
44.8 + 8.4
44.8 + 6.4
44.5 + 3.6
-randoom gaaddon-medium-exper
62.5 12.1
-1.7 1.7
21.4 +
99.4 + 0.5
99.7 0.4
Maze (Actuators =
expert
100.1 1:00000
-0.6 0
100.1 1
100.1 111:00
100.1 +
100.1 +
-modium-oxpert
99.6
100.1
100.1
-modium
68.1 1
-0.6 1
70.9 1
76.8 1 4.2
74 1 4
73 1 4.5
Maudlo--medium-ooo
83.3 1 8.2
-0.6 1
17 + 12.5
99.2 1 1.1
98.1 + 1.5
99.2 1: 0.6
Maze (Actuators 10)
-expert
100.1 +
+
100.1 + 0
100.1 + 0
100.1 +
100.1 =
medeium-expert
98.6 + 1.2
+ 0
99.3 = 0.8
100.1 +
100 +
100 = 0
-medium
40.1 + 5.8
040
6.4
53.4 5.8
43.8 =
42.2 + 6.5
-random- medium-experr
84.5 +
50.9 = 4.7
99.9 +
97.8
99.4 0.7
Maze (Actuators 12)
+
+ 100.1
+ 100.1
+ 100.1 +
+ 100.1 =
-expert
100.1
0
-medium-expert
97.8 + 1.6
0
98.2 =
100.1 =
99 + 0.7
99.8 0.5
-medium
20.7 + 3.2
0+0
25.3 + 6
38.1 + 4.8
26.4 = 4.2
30.1 + 5.8
-random- coreloo--mciiummeppper
79.7 1 6.7
0 1
65.3 1 29.3
98.9 + 1.1
95.7 1 3.9
98.1 + 1.9
Maze (Actuators = 15)
-expett
100.1 +
0 0
100.1 0
100.1 +
100.1 +
100.1 +
mediium-xppert
100 + 0
041
99.7 = 0.6
100.1 0
100.1 - 0
100.1 =
-medium
53.2 + 6.2
040
50.8 4.6
69.1 = 3.4
91.3 = 6.8
94.9 2
-random- medium-
93.1 + 3.2
0
12.1 + 17.9
96.7 4.9
85.1 12.5
Sum
-expert
600.6
-7.9 +
600.6
600.7 0
600.6 +
600.6 -
-eedium-expert
594.3
-7.9 0
594. 6.5
600.6
598.3 +
599.9
-medium
269.7 + 25
-7.9 +
274.4 = 25.5
345 + 31.9
334.9 = 33.4
341.8 + 28.9
-random-mmedummexppee
470.5 + 49.1
13.2 + 46.1
200.7 + 87.5
591.9 8.9
579.9 + 16.7
577.1 + 18
-all
1935.1 79.1
-10.5 46.1
1670.3 119.5
2138.2 40.8
2113.7 + 53.1
2119.4 + 47.5
30

---

Published in Transactions on Machine Learning Research (11/2024)
Table 8: Individual performance comparison for DMC n = 3. Figures are mean normalised scores + one standard error, with and 100 representing random and expert policies, respectively. At the bottom of the table we also provide totals, split by data quality and the entire set. We see that DecQN-BCQ is the least performant of the offline methods and that DecQN- /QL/tIQ//nneteep OneStep perform similarly overall and expert, medium-exert and and medium datasets and DecQN-CQL the best random-meding www--mmeemimmppeeeeee experrt
datasets.
Environment ddaaset
BC
DecQN
DecQN-BCQ
DecQN-CQL
DecQN-IQL
DecQN-OneStep
FingerSpin
-expert
99.5 +
-0.2 + 0.1
100.5 0.8
107.1 0.3
102.9 0.2
102.5 +
-eedim-exppett
77.9 + 6.5
-0.5
86.5 + 9
106.8 + 0.2
102.8 + 0.3
102.7 + 0.2
-medium
38.3 +1
-0.5 +
40.7 + 0.4
49.4 + 0.3
44 + 0.8
45.2 + 0.6
-random--une -randmm-mediune mediimm-expertt
8.2 + 2.3
56.2 + 2.9
62.7 9.5
100 0.5
77.1 5.1
78.3 3.5
FishSwim
expert
82.9 + 12.2
-1.8 1.9
112 + 2.7
120.8 + 14.7
123.8 + 11.7
105.3 + 8.2
-medium-expert
+
0.7 6.5
97.5
127.2
11.1
13.3
112.2
-modium
42.8 + 9.6
4 1.5
56.1 1 9.8
71.5 + 6.3
63.4 1 7.2
76.7 6
-tandom-menn cendoo--mellum-oxperr
23.6 + 9.2
-0.4 1 3.
17.8 1 4.8
52.1 1
37.1 1 9
59.9 + 11
CheetahRun
-expert
99.9 + 1.7
2.7 2.1
105.5 + 0.4
105.6 0.9
104.6 1.1
106.3 + 0.3
mediiun-exprrt.
61.6 + 12.5
0.9 + 0.8
104.2 + 1.6
103.2 + 0.7
102.5 + 1.2
104.8 + 0.7
-medium
40.4 +
+ 0.7
47.1 0.4
48.3 +
47.7 0.5
47.9 0.3
-random--edium-eeee medium-expert
41.5 + 0.6
0.5 + 1.2
22.2 4.4
79.6 + 5.6
61.4 3.3
53.2 1.9
QuadrupedWalk
expert
+
+ 4.6
114.9 1.2
118.2
122.3 1.1
115.3
-medium-expert
63.4 11
8.5 + 155
110.4 + 1.9
115.4 +:
121.2 2
109.4 + 2.1
-medium
39.2 + 8.5
11.3 + 6.3
47.6 + 8.4
48.6 + 7.9
46.3 + 5.7
46.8 = 9.2
Mandom--edium-ee
28 + 6.4
-5.2
5.2::44 4.7
-12.3 + 1.5
76.7 + 5
65.8 1 4.4
69.4 5.7
HumanoidStand
exxert
102.2 + 1.3
-0.1
103.6 + 1.7
109 1.9
116.6 + 0.6
117.2 + 0.8
medium-expprt
63.1 +
0.1
92 2.7
104.7 1.9
113.3 0.4
116.7
-medium
+ 0.5
0
47.2 1.5
51.4 + 0.3
53.8 0.4
53.8 0.4
trandom-medium-exee
31.4 + 2.6
0 + 0.1
17.7 + 5.9
42.7 + 0.9
46 +
47.3 = 0.7
DogTrot
-expert
98 + 0.7
0.1
94.4 0.5
99.5 + 0.7
98.9 + 2.2
101.2 + 1.6
-medium-expertt
62 + 3.7
0 0.3
74.9 + 3.5
84.8 + 3.7
89.3 + 1.4
93.9 = 1.4
-medium
43.8 + 0.5
0.1
49.1 0.6
46.5 + 0.5
52 0.3
50.2 0.2
-random-medium medium-experrt
37.2 +
+ 0.2
0.1 = 0.1
43.4 +
44.1
44.9 0.7
Sum
-expert
580.2 + 19.5
+ 8.8
630.9 13.2
660.2 + 19.9
669.1 16.9
647.8 13.7
-medium-expert
368.6 + 44.5
9.7 9.1
565.5 36.9
642.1 21.6
620.2 17.6
639.7 13.6
-medium
248.9 + 20.5
6.9 + 8.6
287.8 + 41.8
315.7 + 15.6
307.2
320.6 = 16.7
-random-mediin Fzandon-mediu-eepeer
172.9 + 24.7
51.2 + 12.4
108.2 + 39.4
394.5 21.5
331.5 +
353.2 23.5
-all
1370.6 + 109.2
71.8 + 38.9
1592.4 134.3
2012.5 78.6
1928 73.4
1961.3 + 67.5
31

---

Published in Transactions on Machine Learning Research (11/2024)
Table 9: Individual performance comparison for dog-trot {3, 30, 50, 75, 100). Figures are mean normalised scores + one standard error, with and 100 representing random and expert policies, respectively. At the bottom of the table we also provide totals, split by data quality and the entire set. We see across all datasets that DeeeNttoL/Oneette perform best, followed by DecQN-CQL and DecQN-BCQ.
Environment dataset
BC
DecQN De<QN-BCQ DecQN-CQL. DecQN-IQU DecQN-OneStep
DogTrot (n = eexeert
98 = 0.7
0.1 + 0.1
94.4 + 0.5
99.5 + 0.7
98.9 2.2
101.2 = 1.6
-medium-expert
=
0.3
=
84.8
89.3
93.9 = 144
-medium
43.8 + 0.5
0.1 0.1
49.1 = 0.6
46.5 + 0.5
52 + 0.3
50.2 = 0.2
vandom-medium-expee
37.2 + 3.6
0.1 0.2
0.1 + 0.1
43.4 + 0.5
44.1 + 1.2
44.9 + 0.7
DogTrot (n (m)10)
-expert
97.3 + 1.7
+
96.8 = 1.8
99.2 + 1.3
106.5 + 1.1
113.4 = 0.9
-modium-cxpert
+ 5.5
0.5 0.1
+ 4.6
105.1 2.1
109.8
-medium
34.7 + 0.4
0.5 0.2
36.9 + 0.3
39.1 0.6
41 0.8
47.2 = 0.4
vandon--mediu--sppet
33.6 + 4.4
0.1 + 0.1
6.1 + 2.8
33.4 + 1.5
52.1 + 2.3
45.4 = 4.1
DogTrot (n = 30)
-expert
99.7 + 0.4
+3
100.3 + 2.3
99.8 + 0.8
102.8 + 0.4
106.4 4 = 0.8
medium-expeet
70.3 + 4.9
0.6 0.2
74.1 8
90.7 3.7
98.3 1.6
100.7 +
-medium
33.1 + 0.2
0.7 0.2
33.6 - 0.4
33.4 + 0.4
39.9 0.1
40.5 = 0.5
-random- modillo--mdiium-xppeett
22.4 + 0.9
0.2 + 0.1
5.2 + 2.2
22.9 + 2.5
31.6 + 1.3
30.4 = 1.8
DogTrot (n = 50)
expeet
98.2 + 0.5
0.5 0.1
98.6 + 0.6
97.7 + 0.6
99.2 + 0.9
100.4 = 1.4
-medium-expert
67.7 +
-0.3 +
68.7 + 4.2
88.9
95 2.1
97.1 1.4
-medium
34.9 + 0.4
-0.3 +
36.5 + 0.7
36.7 0.2
45.1
0.6
43.1 + 0.7
21 + 2
0.7 1 0.1
9.3 + 1.5
27.6 + 1.3
27.9 + 2.4
36.4 + 1.8
DogTrot (n = 75)
eepeett
98.4 + 0.8
0.6 + 0.2
99.6 + 1.1
98 +1
101.4 + 1.4
102.9 = 0.9
-medium-expert
75.3 + 2.6
0
74.9 + 16.7
85 2
89.9 0.7
97.9 = 2.5
-medium
41.4 + 0.2
0.5 0.1
41.3 + 0.8
42.5 0.1
48.5 0.4
47.8 + 0.5
-tandom-mediune medtom--mdiu--eperttt
27.7 + 0.7
0.7 + 0.2
0 + 0.1
33.4 +
45.3 + 2.9
43.6 + 2.6
DogTrot (n = 100)
-expert
96.4 + 1.1
0.1
95 + 1.8
100.4 + 0.7
103.2 + 0.4
105.5 + 0.8
-medium-expert
70.9 + 6.5
0.1
+.8
74.4 5.1
93.9 1
101.5 2.5
-medium
31.7 + 0.8
0.1
0.5
34.8 0.7
40.9 0.2
41.5 = 0.5
-random- cecttta-mmiiiimxpeeeee
18.9 + 2.7
0.3 + 0.1
12.7 1 2.4
24 + 2.8
26.2 +
Sum
-expert
588 + 3.2
0.7 0.4
584.7 + 14
594.6 + 5.1
612 + 6.4
629.8 + 6.4
-medium-xxpert
404.3 + 28.2
1.2 0.7
449.5 + 64.9
507.2 18.7
571.5 1 8.9
600.9 1 12
-modium
222.6
1.6 0.7
231.9 5.8
233 2.5
267.4 1 2.4
270.3
-random--medim-exppet
160.8 + 14.3
2.1 0.8
30.2 + 17.6
173.4 + 10.2
225 + 12.9
226.9 + 12
-all
1375.7 + 50.2
5.6 + 2.6
1296.3 + 102.3
1508.2 + 36.5
1675.9 + 30.6
1727.9 33.2
Table 10: Performance across 5 seeds.
Environment -dataset
DecQN-BCQ DecQN-CQL DecQN-IQL DeeQN-OneSter
HumanoidStand
-expert
103.6 + 1.7
109 + 1.9
116.6 + 0.6
117.2 + 0.8
-mediuu-expert
92 + 2.7
104.7 + 1.9
113.3 + 0.4
116.7 + 0.7
-medium
47.2 + 1.5
51.4 + 0.3
53.8 + 0.4
53.8 + 0.4
raaddom-mediuu--sqpert
17.7 + 5.9
42.7 + 0.9
46 + 1
47.3 + 0.7
32

---

Published in Transactions on Machine Learning Research (11/2024)
Table 11: Performance across 10 seeds
Environment -dataset
DecQN-BCQ DecQN-CQL DecQN-IQL
HumanoidStand
-expert
103.6 + 1.3
109.6 + 1.5
117 + 0.6
118.1 + 0.7
-medium-expert
90.3 + 3.3
103.4 + 1.8
112.9 + 1.1
116.2 + 0.6
-medium
46.7 = 0.6
51.9 0.4
53.7 + 0.4
53.8 + 0.4
-eandom-medduu-eexpert
19.6 + 2.5
41.4 1.1
45.5 + 1.3
46.9 + 0.5
Table 12: Individual performance comparison for DeeQN-CQL, using mean and sum decompositions and BDQ-CQL for 3. Figures are mean normalised scores one standard error, with and 100 representing random and expert policies, respectively.
Enviromment -dataset
DecQN-CQL, (Mean)
DecQN-CQL, (Sum)
BDQ-CQL
HumanoidStand
-expert
109 + 1.9
95.1 + 1.8
105.5 + 0.7
-medium-expert.
104.7 + 1.9
86.1 + 2.8
94.0 + 5.8
-medium
51.4 + 0.3
44.6 + 0.8
48.2 - 0.5
random--edhnm-sspert
42.7 + 0.9
36.3 + 1.6
43.2 + 0.5
DogTrot
-expert
99.5 + 0.7
93.1 + 1.3
94.9 + 1.2
-medium-oxpert
84.8 + 3.7
81.4 + 2.8
75.7 + 3.6
-medium
46.5 + 0.5
44.3 + 0.3
48.5
- 0.5
43.4 + 0.5
39.9 + 0.9
37.9 + 2.3
33


Reviews:
[{"review_id": "dmgmKJs03r", "reviewer": "Reviewer_fz2Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Factorisable action spaces approach is interesting and results look promising\n- The paper adresses multiple weaknesses of its previous submission, particularly technical clarity and extending the experimental domain\n- Detailed derivation of the factorisable space counterpart of several offline RL algorithms\n- In-depth analysis and insights of Q-values errors\n\nWeaknesses:\n- My main issue about this paper is that the additive decomposition concerns raised in the previous round of reviews were not fully addressed. Points about existence, validity as well as more extensive discussions about the limitations of such assumptions and error analysis in the cases of it being an approximation could have been added.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper benchmarks various offline RL methods after reformulating them in factorisable action spaces - where \"global\" actions are actually decomposed in a series of subactions. After making a case of the usefulness of the factorisable action space approach, the paper provides a theoretical study of offline RL algorithms and their factorisable reformulation, before presenting experimental results on various datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses section", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aRomBXA9fA", "reviewer": "Reviewer_rKiz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nI thought the authors did a really nice job of explaining why action / value\nfactorization is of particular interest in the *offline* RL setting. Dealing with\nlarge action spaces in offline RL is indeed a very interesting problem that I\ndon&rsquo;t think I appreciated enough before having read this paper.\n\nAside from some notational issues mentioned below, I found the paper to be\napproachable and easy to follow.\n\n\n## Weaknesses\n\nI have a few major concerns with the paper, listed below:\n\n1.  **Assumptions**. I believe a critical piece to the story is missing: when does it\n    make sense to factorize the value function / action space this way? The paper\n    cites cooperative multi-agent RL as an inspiration for action/value\n    factorization, but as I see it, the factorization is much more appropriate\n    there: in particular, if the utility of the joint policy is the sum of\n    utilities of all agents&rsquo; policies (which is a reasonable assumption for a\n    fully cooperative game), then the factorized value function can\n    exactly model the atomic one. In the single agent large action space setting,\n    no such reasoning is given. Even in the benchmark domains discussed in the\n    paper, it is not at all clear to me that the factorization *should* accurately\n    model the actual Q-function (or actually agree with it on its ranking of\n    actions; see requested changes). Since this paper claims to take the initial\n    investigation into this approach for offline RL, I believe it is necessary to\n    at least discuss this point, and provide conceptual evidence to suggest that\n    factorizing the value function is appropriate.\n    \n    I did do some more digging to find if any existing works have justified this\n    form of factorization. While the work of Ireland & Montana uses a similar\n    factorization, I have not found any strong justification, or any discussion\n    of which decision problems admit such a factorization. On the other hand, the\n    work of Tang et al. (*Leveraging factored action spaces for efficient offline\n    reinforcement learning in healthcare*) *does* show that generally such a\n    factorization *does not* hold. Their work studied benchmarks where they could\n    justify the decomposition, unlike this work, to my understanding.\n2.  **Benchmarks**. Perhaps in a similar vein to the first point, it is not clear to\n    me why the proposed benchmarks are necessarily appropriate / sufficient here.\n    Firstly, the Maze benchmark is not sufficiently described (I could not\n    understand the function of the $N$ actuators without reading the Chandak et\n    al. paper). Moreover, all benchmarks\n    appear to have action spaces that are simply discretizations of a structured,\n    continuous action space. This is also true of the Maze environment. Furthermore,\n    basically all of the benchmark tasks\n    are locomotion problems, which encompasses a specific structure of\n    reward/value; it would be particularly interesting to understand how the\n    action space factorization influences policy optimization with a broader\n    spectrum of value functions.\n3.  **Algorithms**. The factorized variants of the offline RL algorithms that are\n    presented are fairly straightforward, which isn&rsquo;t a problem in its own right,\n    but in some cases I believe the straightforward approach actually leads to\n    the wrong algorithm (see requested changes). In particular, the DecQN-CQL\n    loss at the very least looks incorrect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the prospects of a form of action space / value function\ndecomposition for offline RL with large action spaces. The authors argue that\noffline RL is particularly challenging with large action spaces, since less of\nthe action space is covered in the dataset, making it more difficult to mitigate\noverestimation bias using common approaches. By factoring the value function\nover a set of small *sub*-action spaces, the authors argue that standard\napproaches to offline RL approaches are amenable to the large action space\nsetting. To validate their claim, the authors propose extensions to common\noffline RL algorithms with factorized value functions as well as a set of large\naction space benchmarks, and evaluate the performance of their methods relative\nto &ldquo;atomic&rdquo; (un-factorized) counterparts. Overall, the authors find that the\nfactorized algorithms scale more gracefully with the size of the action space.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  The definition of factorized action space on page 2 seems too weak. In\n    particular, this definition does not actually restrict the action space in\n    any way (for any action space $\\overline{\\mathcal{A}}$, we can write\n    $\\overline{\\mathcal{A}} \\cong \\overline{\\mathcal{A}}\\times\\{1\\}\\times\\{1\\}\\times\\dots$). Beyond that, even if every\n    factor $\\mathcal{A}_i$ has small cardinality, the factorization does not reduce the\n    total number of actions: really, the focus should be on the property that the\n    individual factors can be modeled independently in some sense, and this is\n    not discussed / motivated enough in section 1.\n2.  In equation (1), the notation for $y$ is incorrect. As written, $y$ is the\n    same for every summand. It should be either changed to $y(s, a, r, s')$, or\n    have $(s, a, r, s', y)\\in\\mathcal{B}$ in the limits of the sum.\n3.  There is another flaw in notation in equation (1). While the sum is over all\n    transitions in the buffer, you are dividing by the cardinality of a\n    minibatch. I suspect the $\\mathcal{B}$ in the subscript of the sum should be a $B$.\n4.  Section 2.2 alludes to various ways of decomposing the $Q$-function, but I\n    think it is worth discussing the implications of these decompositions here.\n    Do the different decomposition strategies imply different structures in the\n    MDP? Can something be said about which MDPs permit equation (2) as a\n    reasonable approximation to the $Q$-function?\n5.  With regard to the &ldquo;noise&rdquo; $\\epsilon(s, \\mathbf{a})$ &#x2013; why is this a random\n    variable? Clearly $Q^\\pi(s, \\mathbf{a})$ is deterministic, so if $\\epsilon(s,\n       \\mathbf{a})$ is a random variable, then $Q_\\theta(s, \\mathbf{a})$ must be as well.\n    Are you modeling approximation error here (in the sense that the estimate\n    $Q_\\theta$ will depend on the random minibatches) or bias (in the sense that the\n    function approximator cannot represent $Q^\\pi$ exactly)? In the former case, I\n    believe some assumptions are missing on the statistical properties of $\\epsilon(s,\n       \\mathbf{a})$ in order for the claim to be true.\n6.  The &ldquo;dropped prime symbol&rdquo; in equations (6-7) is wrong. Particularly there is\n    still a subscripted $s'$ in the LHS that does not appear in the RHS.\n7.  I am almost certain equation (8) is incorrect, since log-sum-exp is\n    nonlinear. Particularly, I do not see how you moved the $1/N$ term from the\n    exponential out of the log/sum.\n8.  The Maze environment should be described in more detail. Without consulting\n    the Chandak et al. paper, I genuinely did not understand what the action\n    space was. At the very least, a quick description should be given, at the\n    very least in an appendix. I also think this section should\n    discuss *why* action factorization is interesting/useful in these environments.\n    Imagine a 2D maze where there are 4 actuators that push the agent north,\n    east, south, west, respectively. Suppose there is an obstacle to the\n    northeast of the agent. Then it might be perfectly fine for the agent to go\n    north or east (separately), but going northeast would be bad. The\n    factorization presented here would represent the action-value for the\n    northeast action as the average between those of the north and east actions,\n    so it would falsely indicate that moving northeast is good.\n9.  A similar argument holds for the dm-control benchmark. I would like to see a\n    discussion about why it is sensible to factorize the $Q$ function here.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uwUIEMy6l6", "reviewer": "Reviewer_mtAx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper is very well written\n* It makes clear what its own contribution is\n* It does not make exaggerated claims\n* The empirical evaluation is extensive\n* The proposed techniques show good results\n* The results are given with uncertainty (here the standard error)\n\n**Weaknesses**\n\nno significant weakness", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "As the title clearly states, the paper deals with “Offline Reinforcement Learning in Factorizable Action Spaces”.\nThe existing technique “Decoupled Q-Network (DecQN)”, specially developed for factorizable action spaces, is tested for offline RL and improved with a variety of techniques known from offline RL. The individual new methods are compared with the standard version of DecQN and with each other.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From my point of view, the paper can be accepted as it is. I think it is a very good paper.\n\nHere are a few comments and suggestions that I think would further improve the quality:\n\n* At “... suffer from overestimation bias as a result of the maximization carried out when determining target values” I think (Thrun & Schwartz, 1993) should be cited, where the “overestimation phenomenon” was presented for the first time as far as I know and possibly also [1], where the term “rising Q” was introduced.\n\n* At “The last few years in particular have seen a wide variety of approaches proposed, making use of ... and environment modeling (Kidambi et al., 2020”, I think it would appear better to mention not only (Kidambi et al, 2020), but also [2] and [3]--in these three papers, developed almost simultaneously, dedicated model-based offline RL methods were presented for the first time.\n\n* At “One-step RL (Brandfonbrener et al., 2021)” I think [4] should also be mentioned, where, as far as I know, one-step RL was presented for the first time.\n\n* In the text, “i.e.” is written in italics, but “et al.” is not. To be consistent, either both should be italicized or both should be non-italicized.\n* In the text, mostly“ offline-RL” is used, but also “offline RL” is used twice. This should be handled consistently. As long as the spelling is consistent, I think it's up to the authors'  liking which one they choose, but I would like to point out that “offline RL” is much more common in the literature.\n* It would be desirable to use more than just 5 repetitions (seeds), e.g. 10 or more if affordable. It is not absolutely necessary, because the statements are rather qualitative and the remaining uncertainties do not influence the core statement, but it would further strengthen the empirical part.\n\n* In the bibliography there are still some unintentional (or incorrect) lower case letters, such as “markov” and “bernoulli”. So please use curly brackets here, such as {Markov}.\n\n[1] C. Gaskett, Q-Learning for robot control, 2002\\\n[2] T. Yu et al, MOPO: Model-based offline policy optimization, 2020\\\n[3] P. Swazinna et al, Overcoming model bias for robust offline deep reinforcement learning, 2021\\\n[4] C. Gulcehre et al, Addressing Extrapolation Error in Deep Offline Reinforcement Learning, 2020", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper does an analysis of a specific type of action decomposition, put forward originally by DecQN in Seyde et al. specifically for the case of offline RL. While there were many technical points that needed clarifying, the authors responded well and many of the specific points have been addressed by further revisions.  

After significant discussion, the reviewers agreed that the paper is well written and makes contributions worthy of publication under TMLR's acceptance criteria. 

However, two of the three reviewers have commented that a major weakness of the paper is the concerns and lack of discussion around the applicability of the action decomposition. One was serious enough to initially state "No" to Claims & Evidence. And, based on the feedback below, this has been brought up about this paper before. Hence, it must be fixed.

Reviewer f2zY states: "My main issue about this paper is that the additive decomposition concerns raised in the previous round of reviews were not fully addressed. Points about existence, validity as well as more extensive discussions about the limitations of such assumptions and error analysis in the cases of it being an approximation could have been added."

Reviewer rKiz states: "Assumptions. I believe a critical piece to the story is missing: when does it make sense to factorize the value function / action space this way?"

In response, the authors have added Section 4.1 relating the action decomposition to ones that have been used in past work, which is good for context, but not sufficient to address the reviewers' criticisms.

The authors should properly address these outstanding criticisms in the camera-ready copy (note: this is a necessary condition of acceptance). The following points have not yet been addressed:

- While the authors acknowledge that the action decomposition does not always work, there is no concrete example or illustration of when it would not work. Please add a Limitations section and present such an example accompanied with an explanation of why this action decomposition fails in this case

- Especially since the benchmarks and data sets are listed as contributions: please relate the choice of benchmarks to the action decomposition, including a discussion of why this action decomposition is appropriate for these benchmarks. These benchmarks are not intended to cover the space of large-action-space MDPs, but specifically those that are amenable to this type of action decomposition. This should be made very clear so the scope of any results using these benchmarks is clear. Are there any other large-action-space RL benchmarks where this decomposition would *not* be suitable? 

- (review by rKiz) The action decomposition in CQL implementation is not the same as the one described in the paper or the one used by the other methods. Please mention and any other discrepancies in the Limitations section; conclusions drawn from results depend on these discrepancies and hence they should be evident to the readers.
Decision: Accept (poster)