Robust and Data-efficient Q-learning by Composite Value-estimation
Authors: Gabriel Kalweit
Maria Kalweit
Joschka Boedecker
Venue: Accepted by TMLR ()

Abstract: In the past few years, off-policy reinforcement learning methods have shown promising results in their application to robot control. Q-learning based methods, however, still suffer from poor data-efficiency and are susceptible to stochasticity or noise in the immediate reward, which is limiting with regard to real-world applications. We alleviate this problem by proposing two novel off-policy Temporal-Difference formulations: (1) Truncated Q-functions which represent the return for the first $n$ steps of a target-policy rollout with respect to the full action-value and (2) Shifted Q-functions, acting as the farsighted return after this truncated rollout. This decomposition allows us to optimize both parts with their individual learning rates, achieving significant learning speedup and robustness to variance in the reward signal, leading to the Composite Q-learning algorithm. We show the efficacy of Composite Q-learning in the tabular case and furthermore employ Composite Q-learning within TD3. We compare Composite TD3 with TD3 and TD3($\Delta$), which we introduce as an off-policy variant of TD($\Delta$). Moreover, we show that Composite TD3 outperforms TD3 as well as TD3($\Delta$) significantly in terms of data-efficiency in multiple simulated robot tasks and that Composite Q-learning is robust to stochastic immediate rewards.

Published in Transactions on Machine Learning Research (07/2022)
Robust and Data-efficient Q-learning by Composite Value-estimation
Gabriel Kalweit
kakweitg Bcs. uni-freiburg de kalweitm Bes. &es.uni- uni- de jboedeck &cs. uni uni-freibury..o de
Maria Kalweit
Joschka Boedecker
Newrorobotics Lab and BrainLinks-Bren Brain Tools
University of Freihurg
Germany
Reviewed on OpenReview: https: // pttp://eeeervvie...mmmm net/ formm? id= ak6Bds2Del
Abstract
In the past few years, off-policy reinforcement learning methods have shown promising results in their application to robot control. Q-learning based methods, however, still suffer from poor data-efficiency and are susceptible to stochasticity or noise in the immediate reward, which limiting with regard to real-world applications. We alleviate this problem by proposing two novel off- off-policy Cemporal--iifereau formulations: (1) Truncated Q-functions which represent the return for the first n steps of target-policy rollout with respect to the full action-value and (2) Shifted Q-functions, acting as the farsighled return after this truncated rollout. This decomposition allows us to optimize both parts with their individual learning rates, achieving significant learning speedup and robustness to variance in the reward signal, leading to the Composite Q-learning algorithm. We show the efficacy of Composite Q-learning in the tabular case and furthermore employ Composite Q-leariing within TD3. We compare Composite TD3 with TD3 and (D3(A), which we introduce as show that Composite TD3 outperforms TD3
an off- off-policy variant of TD(A). Morcover, as well as TD3(A) significantly in terms of data-efficiency in multiple simulated robot tasks and that Composite Q-lcarning is robust to stochastic immediate rewards.
1 Introduction
In recent years, Q-learning Watkinn & Dayan, 1992) has achieved major successes broad range of areas by employing deep neural networks (Mnih et al. 2015 Silver al.) 2018 Lillicrap al. 2016)), including environments of high complexity (Riedmiller et al.) 2018)) and even in first real-world applications (Haarnoja et al. 2019). Deep Q-learning methods, however, are still far from being easily applicable to solving complex real-world tasks due to their high demand for data samples and their lack in learning stability partially caused by the stochastic nature of transitions in real-world settings, such as in robotics, intelligent machine- braainne-briin interfaces or health applications. Especially stochasticity in the immediate reward signal itself may cause policies to converge to suboptimal performance. This stochasticity can be induced by multiple sources of randomness, for example through stochastic next state transitions due to sensor or actuator noise, occlusions,
due to data-driven reward estimation (Mees et al. 2020 Ni et al. 2020 Fu et al. 2018). This increases the complexity of value-estimation tremendously due the long temporal horizon the reward signal has to propagate through, possibly depending on many decision steps and transitions. To overcome the complexity of growing task horizon, in this work, we propose to break down the long-term return into a composition of several short-term predictions over a fired temporal horizon. We approach this via a new formulation of consccutive bootstrapping from value functions corresponding different horizons of the target-polics associated the full return. For its formulation, we define Truncated Q-functions, representing the return for the first steps of a target-policy rollout with respect to the full action-value. Tn

---

Published in Transactions on Machine Learning Research (07/2022)
addition, we introduce Shifted Q-functions which represent the farsighted return after this truncated rollout. Both are then combined in a recursive definition of the Q-function for the final algorithm. Starting from a reformulation of the vanilla Q-learning formalism, our approach offers the flexibility to separately optimize hyperparameters for different parts of valuo-cstimation, showing that Shifted Q-functions suffer less from variance compared to the full return thus allowing for higher learning rates. Most similar to our approach is (D) (Romoff et al., 2019), composing the full return of value functions with increasing discount. Increasing discount factors translate to growing task horizons, leading to a decomposition of the long-term prediction task into multiple subtasks of smaller complexity. In their work, Romoff et al. (2019) argue that values for smaller discount values can potentially stabilize more quickly, leading to faster and more stable learning. Since the values corresponding to larger discount only represent the residual to the values of smaller discount, however, TD(A) does not alleviale the issues arising with stochastic rewards. Other techniques for variance reduction in TD-eearnng are reward estimation (Romoff et al.) 2018) or averaging value functions (Anschel et a 2017). Both tochniques are orthogonal to our work and can be combined with Composite -learning to possibly further enhance learning stability. However, as we show in this work, the increase in performance and data-efficiency comes from the possibility of higher learning rates in the long-term Shifted Q-functions and not the short-sighted Truncated Q-functions. It can thus be argued that reward estimation and averaging come at a cost of slower convergence. Asss et al. (2200) propose to Icarn action-vall values for a fixed-horizon via consecutive bootstrapping. In their formulation, however, the consecutive value functions are estimated over different target-policies with respect to the different horizons and ignore the residual of the full return. Thus, they can be non-optimal with respect to the task at hand.
Our contributions are fourfold. First, we introduce the Composite Q-learning algorithm. Second, we show that the targets of Shifted Q-functions suffer less from variance compared to the full action-value, thus allowing for higher learning rales. Third, we introduce TD3(A, an off-policy extension of TD(A) to deep Q-learning, And fourth, we employ Composite Q-learning within TD3 and evaluate Composite TD3 in various simulated robot tasks under normal and noisy reward functions.
2
Background
We consider tasks modelled as Markov decision processes (MDP), where an agent executes action a E A in some state 8, E S following its stochastic policy : S x A + [0, According to the dynamics model M SxAx8 + [0, 1] of the enviromment, the agent transitions into some state $141 E S and receives scalar reward Tr. The agent aims at maximizing the expected long-term return (st(s) = Eaggerr,aassr ME)
T- yp'rr,les [su] where T is the (possibly infinite) temporal horizon of the MDP and y E 1J the discount factor. It there- fore tries to find n², s.t. V=" 2 V= for all x. If the model of the environment is unknown, model- free methods based on the Bellman Optimality Equation over the so-called action-value Q" (8+, oz) = Buppre..pproMM T-1 (o-rrryss at), can be used, Q"((e, ar) = Tr + y max, Estti~M (Q*( a)]- In the following, we abbreviate Eagg Deggern.eessemel (l by Er..l-.. One of the most popular model-free -reee reinforce- ment learning methods is the sampling-baser off-policy Q-learning algorithm (Watkins & Dayan, 1992). A representative of continuous model-free reinforcement learning with function approximation is the Deep Deterministic Policy Gradient (DDPG) actor-critic method (Lillicrap et al., 2016). In DDPG, actor μ is deterministic mapping from states to actions, μ : S + A, representing the actions that maximize the critic Q i.c. ju(x))= = arg max, Q(st, a). Q and μe are estimated by function approximators Q(:-109) and n(-10*), parameterized by 0 and OF, The critic is optimized on the mean squared error between predic- tions Q(s,,, a,)8 and targets Yj = r + (1(45 (((___(())) (s)-1l0")j80)) where Q' and μ' are target networks, parameterized by g and 0²²_ The parameters of jı are optimized following the deterministic policy gradient thcorem (Silvrr et al.) 2014), V + E, voQ(s, ((ss,a/0))1 )ame), 6n) Von (s) and the parameters of the target networks are updated via Polyak averaging, i.e. 00' t (1-) 1100'+000 +00 and 9M +- (1-T)0"' +70", with r E [o, 1]. The statc-of-the-art actor-critic method TD3 (Fujimoto et al. 2018) then adds three adjust- ments to vanilla DDPG. First, the minimum prediction of two distinct critics is taken for target calculation to alleviate overestimation bias, an approach belonging to the family of Double Q-learning algorithms (yan) Hasselt et al. 2016). Socond, Gaussian smoothing is applied to the farget-policy addressing the variance in updates. Third, actor and target networks are updated every d-th gradient step of the critic, to account for the problem of moving targets.
2

---

Published in Transactions on Machine Learning Research (07 (0722)
Composite Q
Intermediate Predictions
arg m Q
Q
arg max, Q
Q2
max Q
Q1
ang max, Q
Quic
arg max, Q
Q2:00
max, Q
Q1:
arg MAX,
Figure 1: Structure of Composite Q-learning. Qi denote the Truncated and Qine the Shifted Q-functions at step i. Q is the complete Composite Q-function. Directed incoming edges yield the targets for the corresponding value function. Edges denoted by are discounted.
3 Combining Long- Long-term and Short-term Predictions in Q-learning
In the following, we introduce the Composite Q-learning algorithm, along with Off-policy TD(A) as an additional baseline. Both approaches aim at decomposing the long-term value into values of smaller time scales. Composite Q-learning approaches this by dividing the bootstrapping into short-term and a long- term component. Off-policy TD(A) the other hand formalizes delta function to estimate the remainder of an action-value corresponding to smaller discount factor.
3.1 Composite Q-learning
We estimate the return n-step rollouts of the target-policy via Truncated Q-functions which we then com- bine to the full return with modcl-free Shifted Q-functions, approach call Composite Q-learning, while remaining purely off-policy. Since these quantilies cannot be estimated directly from single-step transitions, we introduce a consecut ive bootstrapping scheme based on intermediate predictions. Put differently, Com- posite Q-eearning does not learn from n-step returns, but rather learns a decomposition of the full Q-value on the basis of single-step transitions alonc. For an overview, see Figure of TD3 can be found in the supplementary?
Code based on the implementation
3.1.1 Truncated Q-functions
In order to decompose the action-value into multiple truncated returns, assuine that n << (T - 1) and that (T - -1-t) mod = 0 for task horizon T. We make use of the following observation:
Q" (81, Ex,M [ra + gre+1 + +2 22r42 + rr
+ 7 T- -
++2n
Ex.M
x-113 1 +7"
Σ
2 (-
jmt4n
(1)
Σ j=
(T-(T-))r (
That is, can define the action-value combination of partial sums of length n. We can then define the
Truncated Q-function as (T, ax) = E.. - >5try) which we plug into Equation
(= a+) = E.,M (8, + "" ( + + + T (ST @T uT-
(2)
htttp:://gttuub..cm//ttuutt
3

---

Published in Transactions on Machine Learning Research (07/2022)
Proposition 1. Let as) T be the one-step Truncated Q-function and (st, ra (i_1 a++1)] the i-step Truncated Q-function. Then Q (8))0) represents the truncoted return (v), = Ei, [2 ( (ry)
See the proof of Proposition 1 the appendix. Please note, that the assumption of (T 1 mod a only leads to an casier notation and is not a general restriction. If (T-1 1-t) mod n * 0, then the last partial sum simply has shorter horizon. On the basis of Proposition 1 in the following we estimate Q% az) off- aff-policy via consecutive bootstrapping and stochastic approximation. In order to limit the prediction to horizon we estimate different truncated value functions belonging to increasing horizons, with the first one estimating the immediate reward function. All consecutive value functions then bootstrap from the prediction of the preceding value function evaluating the larget-poliey of full return Q for a fixed horizon. Analogously to vanilla Q-learning, the update procedure given by:
Qi(s,, ar) & (1-an a (1 ae) + onee and Qsx1(s(, ai a;) (r + 7Qi arg max (amaxQ((s:aae ((s++1,
(3)
with learning rate orr- Please note that in order to estimate Equation (2, the dynamics model would be needed to get Steen of a rollout starting in 8t. In the following, we describe an approach to achieve an estimation of Equation model-free.
3.1.2 Shifted Q-functions
To get estimation for the remainder of the rollout Q = E.r,l>"((sse (st+n at4n]] after steps, use consecutive bootstrapping formulation of the Q-prediction as a means to skip the first n rewards of a target-policy rollout.
be the one-step Shifted Q-function and 1101:000 (st, ae) = Et, E....Ml ($1+1, a(41)l the i-step Shifted Q-function. Then Vioc (st,as) represents the shifted return ( at) = E,, (* a(4)).
Proposition 2. Let (5t, az) = E (1x,,)
The proof of Proposition can be found in the appendix. Analogously to the consecutive bootstrapping formulation of Truncated f-uncctoos, the updates for the Shiftod Q-functions are:
(1:00 az) (1-ash) (ss)Qune (st, az) ash max Q(+++1,(++)) a)) and
(4)
(8) az) <- (1 - (ggh) ( (8t, 0.) (ss((QQ(i-)))))))) (#t+1 arg max Q(s++1;a)),
with learning rate ash. The variance of the remainder of the rollout decreases with later initial point in time:
Q" a++:)l = var(y' (reti + 7Q"
varl7'r<4 varl7 ++] Q" (s142)
(5)
+ 1220ol7/ret 7 +1 (14441)
If co (845-11 2 0, holds that:
varly"Q" (stti, ar-))) 2 varl (sseat (s44411)
(6)
Please note that * a++4) corresponds to Shifted Q-functions at different stages shifting, with 0 for the full Q-functio.. hence follows:
varlQ" a.)] varlQT... (st, az)] 2 var(Q2 a:) 2 ... var[Qu (st, az)]-
(7)
The covariance reward and future return likely be positive the immodiate reward depends on the next state (Romoff et al. 2018). Otherwise the covariance zero. Hence, can be argued that this inequality
4

---

Published in Transactions on Machine Learning Research (07 (07/2022)
is strict if the variance of the reward is larger than zero. This also holds for the case of zero variance for the immediate reward, if the immediate reward depends a stochastic next state. We therefore argue that in these settings, as very common in real-world experiments, Shifted Q-functions allow for higher learning rates compared to the full Q-function, since this translates to lower variance in their gradients. We empirically validate this linding in our experiments Section
3.1.3 Composition
Following the definitions of Truncated and Shifted Q-functions, we can compose the full return.
Proposition 3. Let (T(s,, a.) B, Sittin1-p - "3-'r3] be the truncated return and ( at) Ei,r,M(y* (st++n astn)l the shifted return. Then Q" (s;, 0s) = Q7) 0z) + Qh: ar) represents the full -
return, i.e. (8) ( =
The proof of Proposition is in the appendix. Thus, can formalize the update of the full Q-function by:
(s) (1 aq)Q(s;, a,) + ag(ri + r( (8+41, arg max ((+++1, a))
(8)
Qm:oo (86+1, arg max Q(s4+1,))))-
The incorporation of truncated returns breaks down the time scale of the long-term prediction by the Shifted Q-function. We call this algorithm Composite Q-leurning (ef. Algorithm [ Please note that, in the tabular setting, Composite Q-learning is equivalent to Q-learning with learning rate aq when setting an agh aq. However, notable advantage is that Composite Q-learning offers independent optimization for the different learning rates corrosponding to different temporal horizons.
Algorithm 1: Composite Q-learning
initialize Truncated Q-functions Q
initialize Shifted Q-functions Qi.0
initialize Q-function Q for episode E..E' do
get initial state 81
for = 1..7 do
apply e-greedy action a:
observe 8141 and
update Truncated Q-functions by:
(s) + (1 - am ) ( an) + amri
Qisi(ss, ax) < (1 ar, (r.))i(st.a ac) + anr(r. 77::+++004111110 max, ((s++1:)))
update Shifted Q-functions by:
(st, at) +- (1 asa 1 (t) + asu(7 max. Q(s411,a)
(st + ossh (1. osaa>QQu (s1+1, (s+1, max, Q(5+11 a]))
update Q Q-function by:
Q(si, + (1 ag)Q(s.,a)) + ag(ri + (QQu max, ((5 a))
mmoo (8--1, arg max, Q(+++1,a)))

---

Published in Transactions on Machine Learning Research (07 (2222)
3.2
Deep Composite Q-learning
In order to cope with infinite or continuous state and aclion-spaces, we exlend Composite Q-learning to the function approximation setting.
3.2.1 Target Formulation
Let Q"'(.-10)) denote a function approximator with parameters 0 and n outputs, subsequently called heads, estimating Truncated Q-functions Q Each output Q, bootstraps from the prediction of the pre- coding head, with the first approximating the immediate reward function. The targets are therefore given by:
(9) where 2' corresponds to the target-actor maximizing the full Q-value as defined in Section 2 and Q the output of the respective Q-target-network. That is, Q represents evaluations of l at different stages of truncation and Ir serve as intermediate predictions to get V We then only use Q which
こ = r and wnis1 = + + (s (1)+110"))000))
implements the full n-step return, as the first part of the composition of the Q-target, but not the Q values.
The second part of the composition is represented by the Shifted Q-function. Let (,, (10h)) denote the function approximator estimating the Shifted Q-functions Qh by n different outputs, parameterized by gSh We can shift the Q-prediction by bootstrapping without taking the immediate reward into account, to skip the first rewards of target-policy rollout. The Shifted Q-targets for heads Q therefore become:
Sh v QQ'(s)+1 ($5+1 100)) and ya = 1 100)))))))))
(10)
That is, each Q implicitly estimates the x multiplier. In the function approximation setting, we can thus define the Composite Q-target as:
リラ =r + rQQ, TH ((0))
+ ($)+1, H ($)+1 (10+))))))5h))))
(11)
approximated by 109) with parameters 89, We jointly estimate QT, Qsh and Q with function approx- imator Q (G--000)
3.2.2 Entropy Regularization
Each pair Q,T + h lisien is a complete approximation of the true Q-value. Note however, that it is also bootstrapped with the full Q-value. The circular dependency can lead stability issues, due to the amplification of propagated errors. Additionally, higher learning rates for the Shifted Valuo-functions, as motivated in Section 3.1.2 may lead to overfitting. In particular, errors in the truncated predictions are propagated quickly by the Shifted Q-functions, especially when trained with high learning rate. Therefore, it is important to keep the truncated predictions close across the different time steps while also preventing the Shifted Q-functions from running into suboptimal local minima. As means to alleviate these issues, we add a regularization based on the entropy of a Gaussian distribution formed by mean μ(Q,T² + Q." and variance (Q, + Q5" over all n complete predictions of Q. We can then formalize an incentive of the Truncated Q-functions to stay close between predictions whilst forcing the Shifted Q-functions keep the distribution as broad possible. We define the Gaussian distribution No over all Q-predictions therefore by:
No (u(Q." + Q lisisn), o2 + Q lisis)))
(12)
with corresponding entropy:
No = log (Q, + 2 (ssss))
(13)
In order to enhance stability of the learning process a whole while preventing the Shifted Q-functions from overfitting, we not only minimize the mean squared error between targets of and predictions (s),))
6

---

Published in Transactions on Machine Learning Research (07 (072202))
but apply gradient descent on the entropy for the parameters of QTh and gradient ascent on the parameters of Q More specifically, we define the squared error for some sample as:
5j = f Q (5,, (s),,,109))
(14)
and the gradient with regard to all parameters of Qe including the different learning rates as:
= aqD V Q (s), a,10%) + o Ve Q9 aj (0) + ash yen Q a, )
(15)
The regularization then adds to the gradient:
H(ss, (ashh 8sh gas aT Von.).
(16)
Hence, the parameter update in its simplest form becomes:
0c Oc + 1 m E -n)
(17)
However, & more sophisticated optimizer such as Adam (Kingma Ba 2015) can be applied analogously. A detailed description of Deep Deterministic Continuous Composite Q-learning is given in Algorithm 2 For exemplary application within TD3 (as in our experiments in the following), Gaussian policy smoothing has to be added to all largets in Line g well as taking the minimum prediction of two distinct critics for each target. Furthermore, actor and target networks have to be updated with delay.
Algorithm 2: Deep Deterministic Continuous Composite Q-learning
initialize critic Qf, actor μ and targets Q j
initialize replay buffer for episode 1..E do get initial state 81
for = 1..7' do
apply action a, ((se(0")) + where & ~ (((,
observe 841 and r. and save transition (st, acs S1)1, calculate targets:
v ry
Vi = + 1QTH (1 (1s++)00)00
/ 7Q' (15,
15 2hhh (8t+1 10"0 08hr)
0 = r, + +7( Tyl (sy+1, (s0+1)0"d) @hh + (8)41,1 (s/42 (0t)) (0Sh))))
ff [1J w. V vi [
update d on minibatch of size from R according to Equation 17
10
update on Q based on the sampled deterministic policy gradient
adjust parameters of a and p

---

Published in Transactions on Machine Learning Research (07 (0222)
3.3 Off-policy (D(A)
Another way to divide the value function into multiple time scales TD(A) (Romoft al. 2019): To
this point, it has only been applied in on-policy setting. In favor of comparability, extend (D(A) to Q-lcarning, yiclding TD3(A). The main idea of TD(A) is the combination of different value functions corresponding to increasing discount values. Let A denote fixed ordered sequence of increasing discount values:
م
(2 72...., ?k) hustser
(18)
We can then define delta functions W; as:
W1 = 2n and W1>1 = Qy - Q
(19)
Let (1-1)) denote the function approximator estimating Quever with parameters 0 Based on the derivations in (Romoff et al., 2019), the targets for Q-learning can be formalized as:
v/71 = + Or (8j+1, (s)- 102)) and
(20)
MR = 7 (8)+1, pl ($5+1 (0"))110000000000 100)) + 76
0
The authors suggest the use of n-step targets within TD(A) which not casily applicable in an off-policy setting. Furthermore, Composite Q-learning is based single-step transitions. To keep the comparison fair detailed
and clear, we thus compare our approach to single-step Off-policy TD(A) in our experiments. overview of Deep Deterministic Continuous Off- oif-policy TD(A) can be found in Algorithm
3 To transform Off-policy TD(A) to TD3(A), the adjustments as described in Section 3.2.2 have to be applied analogously.
Algorithm 3: Deep Deterministic Continuous Off-policy TD(A)
initialize critic Q actor a and targets Q*,, nl
initialize replay buffer R set discount values م = (10, 7....:
for episode do
get initial state $1
for = 1.T do
apply action a; (==e(00)) + where & (00,o)
observe 8141 and and save transition (st, ar, S441,7)) in R
calculate targets:
v = 710%, (s/41, pf (#)41 (1)+1104))
V1S21 = -(7 - (85+1. / (15+1 10"))
+X (0))
update Q on minibatch b of size m from R
10
update Qn based the sampled deterministic policy gradient
adjust parameters a and
12
8

---

Published in Transactions on Machine Learning Research (07 (0722)
4 Experimental Results
In this section, evaluate Tabular and Deep Composite Q-learning. In the tabular case, we apply Composite Q-learning on deterministic and stochastic chain MDPs of varying horizon to illustrate and analyze the characteristics of the interplay between short-term and long-term predictions. We then discuss advantages of Deep Composite Q-learning under a noise-free reward for the two robot simulation tasks Walker2d-v2 and Hopper-v2 and evaluate Deep Composite Q-learning for walker2d-v2, Hopper-v2 and Humanoid-v2 under a noisy reward function, comparing it to TD3 and TD3(A).
4.1 Tabular Composite Q-learning
4.1.1 Deterministic Chain
We evaluate the effect of composing the long-term return of multiple short-term predictions in the tabular setting. We apply Composite Q-learning in the tabular case to the MDP of horizon K given in Figure 2a We compare Composite Q-learning to vanilla Q-learning, as well as multi-step Q-learning based on subtrajectorie of the exploratory policy and imaginary rollouts of the target-policy with the true model of the MDP as a hypothetical lower bound the required updates.
-2
Composite Qeloariing
Multisstep Q-learning wy
On-golicy Mulli-step Q-haarning
Sit
40
Qulonrning
T:
50xg
Trnasted and Slaited Quviies for (xe-0) o)
Q-vine for (8oo)
-10
-15
-25
Updaates
x10
Updates
XL0
(a)
(b)
Figure 2: (a) In this deterministic-chain MDP of horizon K, the agent ought to arrive at terminal state 8K using actions {a, b, c}. The initial state is so and the optimal policy is given in red. (b) Mean results and two standard deviations over 10 runs on the MDP with a horizon of of so and action a estimated by the different approaches over time. Dashed lines indicate convergence to the optimal policy. The predicted Truncated Q-values for state so and action with horizons 1 to 4, denoted by Th..... Tr,, and predicted Shifted Q-values for state so and action a, denoted by Sh,...,, Sh., are to the right. Dotted lines indicate the true optimal respective Q-values.
20. The left plot depicts the value
Results for K 20 are depicted in Figure 2b where mean performances of the approaches are shown the left and the intermodiate predictions of Composite Q-lcarning on the right. All approaches update the Q-function with a learning rate of 10-3 on the same fixed batch of 10x episodes with a percentage of 10% non-optimal transitions. We set rollout length n = for all respective approaches. Please note that this value can be set arbitrarily, however in order to be beneficial should be set a value smaller than the temporal horizon of the task. In this example, a rollout length of corresponds to the integer square root of the horizon of the MDP, meaning that Truncated and Shifted Q Q-functions have similar temporal horizon

---

Published in Transactions on Machine Learning Research (07/2022)
Table 1: Comparison of convergence speed between Tabular Q-learning and Tabular Composite Q-learning for exemplary runs on the MDP given in Figure 2a with n
Horizon
20
50
100
Speed up over Q-learning 11%
44%
57%
66%
due to the partitioning of the long-term value. An evaluation of the performance for different rollout lengths is shown in Figure B.la in the appendix.
For the experiments in Figure 2b we update the Shifted Q-function with a learning rate of 10 and the Truncated Q-functions with a learning rate of 10-3 Composite Q-learning converges much faster to the true action-value than vanilla Q-learning and is close to the multi-step approach based on model rollouts (lypothetical lower bound). The erroneous updales of on-policy multi-step Q-learning lead to convergence to wrong action-value which is underlining the importance of truly off-policy learning.
Composite Q-learning estimates the short-term rollouts off-policy by design and shifts the long-term value in time so as to have a consistent temporal chaining. All intermediate predictions therefore converge to the true intermediate action-values, which is shown on the right side of Figure 2b] The difference in convergence speed between Q-learning and Composite Q-learning grows with increasing horizon, as shown in Table for the described hyperparameter setting.
OQL (1O-7; (10-9) (ggr (107)
CQL(100)
SQL (10r1)
CIL (10-")
CQL (107))
Q leorning
SQL (10-2)
(10P)
CQL (10-9)
CQL (20")
sigl. (1075)
Q-loariig
Qunaue for (Avo)
Qualue for (Awo)
Quehu fie (ea,n)
Gueche fie
-10-
-00
-15
-1
-00-
-25
2
Updatoe
Updates
Updates
xII
Updatee x.0
Quoalee (su.))
Quaine for (rga)
Quaiin (bc.o)
Quaiue for (sxo)
-10
-10-
-10-
-10-
-164
-15-
-20
-20
Updates
x20
Updates
xur
Uplates
Updato
X00
(b)
(a)
Figure 3: Results of 4 individual runs on the deterministic chain MDP with a horizon of K = 20 for Composite and Shifted Q-learning, (a) with different learning rates for the Shifted Q-function and (b) with different learning rates for the Truncated Q-function (denoted by the numbers in parentheses). The learning rates for the full Q-function and for the (a) Truncated Q- d-functions and the (b) Shifted Q-functions are set to 10-3 in all experiments.
Next, we present an evaluation of different learning rales for the Shifted Q-functions, depicted in Figure 3a Here we compare Composite Q-learning (CQL) and vanilla Q-learning to Shifted Q-learning (SQL), which corresponds to Q-learning with a onc-stop shifted target (replacing the bootstrap of the future return with an evaluation of a one-step Shifted Q-function and without approximate n-step returns from Truncated Q-function). The results show that shifting the value in time alone is slowing down convergence. Precisely,
10

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Shifted Q-learning with a learning rate of 1.0 is equivalent to vanilla Q-learning: the same holds for Composite Q-learning with a learning rate of 10-3 for the Shifted Q-function.
Please note that although the learning rate of Q-learning could be set to a higher value for this very MDP, this example shows the benefit Composite Q-learning even though higher learning rates for Shifted Q-functions alone do not bring any speedup in convergence. Only the combination of Shifted Q-functions and Truncated Q-functions lead an increase of convergence speed, even in the case when Truncated and full Q-function are estimated with the same (slow) learning rate of 10-3. The results underline that the higher learning rates for the Shifted Q-functions only have beneficial effect in combination with truncated predictions.
The counterpart with different learning rates for the Truncated Q-functions, keeping the learning rates for the full Q-estimate and Shifted Q-functions fixed, can be seen Figure 3b While there is improvement convergence using larger learning rate for the Truncated Q-functions, the results show higher variance and less benefit than the Shifted Q-functions in Figure 3a
4.1.2 Stochastic Chain
We further investigate the effect of stochastic immediate rewards Composite Q-learning the chain MDP shown in Figure | All actions lead further into the chain, however they yield different expected rewards. Action has an expected reward of -0.8, corresponding to immediate reward of -1 with 80% chance and 0 with 20%. Action b, on the other hand, has an expected reward of -1011, however it is +1 with 99% chance and -20 with 1%. Composite Q-lcarning with learning rates of 10-2 for the full Q-function, 10-1 for the Shifted Q-functions and 10-3 for the Truncated Q-lunctions compared to Q-learning with the same respective learning rates.
088
0.2,
0.2, 0
0.2,
AK-1
0.99, +1
0.90 +1
0.98,
0.01
6001
0.01,
Figure 4: In this stochastic chain MDP of horizon K, the agent ought to arrive at terminal state SK-1 using actions and b. The initial state 8g- Transitions are stochastic. The optimal policy is given in red.
Results for different horizons and no discount are depicted in Figure 5 can be seen that Q-learning with a high learning rate is converging to wrong action-value and the deviation becomes more significant the longer the horizon of the task. The same holds for Q-learning with learning rate of 10-2 however the deviation becomes smaller. Moreover, Q-learning with the same learning rate as the full Q-function in Composite Q-lcarning is converging much slower for growing horizons, yet the lcarning rate for the Truncated Q-functions is 10 times smaller than for Q-learning. This emphasizes the mutual interplay of Truncated and Shifted Q-functions. Furthermore, Composite Q-learning is converging to the true optimal action-value. Q-learning with a small learning rate of 10-3 is far from achieving a satisfying level of convergence within the time frame of 2 10" updates.
11

---

Published in Transactions on Machine Learning Research (07/2022)
Qukcrning (10 ')
Quloacning (30
Qulnarning (13 ?)
Couaponiie Quloaraing (30 1)
Q-valne For (rs oj and Hotioon 200
Q-ratee ol aux Betton 300
100
-50
for (ee (e.ue) Hocitoo
200
-90
-300
-100
-400
-100
-154
-500
-120
-600
-200
-140
-7000
-100
-800
1.5
2.0
0.5
1.0
0.0
Upeletes
Upelsls
Updntes
Figure 5: Results of Composite Q-lcarning and Q-learning with different learning rates over 5 training runs for horizons (left) 200, (middle) 300 and (right) 1000 and discount. The true action-value is indicated by the dashed line. Composite Q-learning uses learning rate of 10-2 for the full Q-value, 10-1 10-1 for the shifted Q-values and 10 3 for the truncated Q-values within the same experimental evaluation.
Shifted Q-functions thus allow for higher learning rates which in combination with more cautious fitting of short-term predictions leads to increased data efficiency while achieving smaller deviation from the true action-value. Most importantly, the difference in performance between Q-lcarning and Composite Q-lcarning (better value estimation with respect to to high learning rate and better convergence properties comparing to small learning rate) becomes larger for longer horizons. The performance for different rollout lengths are shown in Figure B.1b in the appendix.
4.2 Composite Q-learning with Function Approximation
We apply Composite Q-learning within TD3 and compare against TD3 and TD3(A) on three robot simulation
tasks of OpenAl Gym [Brocknaan et al. 2016) based on MuJoCo (Toorov el. al., 2012): Walker2d-v2, Hopper-v2 and Humanoid-v2. hyperparameter optimization and hyperparameter settings Appendix C We analyze the properties of Composite Q-learning in terms of data-efficiencc and stability on the true reward for Walker2d-v2 and the potential increase of updates per sample for Walker2d-v2 and Hopper-v2 in Section 4.2. before we evaluate Composite TD3 Scction 4.2.2 As main evaluation metric for performance and stability, we provide the mean area under the learning curve normalized by the mean area of Composite Q-learning, in order to set the mean performance of Composite Q-learning as the origin for comparison.
visualization of the enviromments depicted in Figure 6 We discuss
very noisy reward signal for Walker2d-v2, Hopper-v2 and Humanoid-v2 in
Figure Visualization of Walker2d-v2 (left), Hopper-v2 (middle) and Humanoid-v2 (right)..
4.2.1 True Reward Function
In first experiment, compare Composite TD3, TD3(A) and conventional TD3 with both the optimized default learning rate and the highest learning rate used in Composite TD3 (TD3 HL) on the true reward for the Walker2d-v2 environment in Figure 7a over 8 training runs. Composite TD3 and TD3(A) yield a better and more stable performance than TD3 with either learning rate, with Composite TD3 being
3 An additional comparison
multi-step basclines
Appendix F
An additional evaluation of Composite Q Q-learning within DQN Mmih
2015 the LanarLander-v2 environment.
with noisy rewards be found in Appendix F
12

---

Published in Transactions on Machine Learning Research (07 /022)
Weller2d-v2
Walke:2d-22
Hopper-v2
Composite TI3
TD3 IIL.
Camposite TD3
FDD3 ILL
TD3
TD3(A)
TD8
TD3(A)
5000
3500
3000
4000
Eniiatiion Rerurn
Evabuatinn Return
Emnlation Retars
2500-
3000
2000
2000
2000
1500
1000-
1000
500-
0
01
Transitions
x10
Transitions
X10
Transitions
X10
(a)
(b)
Figure 7: (a) Mean performance and half a SD (standard deviation) over training runs for the Walker2d-v2 environment with the default reward. (b) Mean performance and half a SD over 8 training runs of Composite Q-learning on the vanilla reward function and multiple update steps per collected sample for the Walker2d-v2 and Hopper-v2 environment.
the best performing of all approaches. The differences between conventional TD3 and Composite TD3 are significant, shown in Table D.I in the appendix. TD3 is not able to gain performance with higher learning rate and even degenerates. Composite TD3, the other hand, can take advantage from the higher learning rate. We hence argue that the extension of TD3 to Composite TD3 allows for higher lcarning rates compared to the vanilla version, at least for some of the individual parts of value-estimation. In addition, Composite TD3 and (D3(A) show less variance.
In a further experiment using the true reward, we investigate the potential increase in data-efficiency when applying multiple gradient steps per collected sample. Results over 8 training runs be seen in Figure 7b with corresponding significance tests provided in Table D.I in the appendix. While there seems to be limit in benefit for TD3 with respect to multiple gradient steps, Composite TD3 and TD3(A) yield a faster learning speed. In addition, Composite Q-learning has no increase in variance opposed to all other approaches. The high learning rate in combination with multiple gradient steps is harmful with respect to learning stability.
The influence of the horizon of Truncated and Shifted Q-functions, well as the respective regularization weights maximizing and minimizing the entropy, is exemplified for the Walker2d-v2 environment, in Figure B It can be seen that the regularization of the Shifted Q-functions has higher influence on the performance. If the weight is set too high, the variance increases significantly. The same holds for a temporal horizon that is too long. The regularization of the Truncated f-functions has less influence on convergence and variance, however, please note the strong connection of these hyperparameters. Therefore, the best triad setting needs to be found for given problem and yields an interesting direction for meta-reinforcement learning, Prior experiments with other regularization terms only minimizing the entropy also showed enhanced stability, however, it is important to prevent the Shifted Q-functions from overfitting when applying higher learning rates in the TD3 setting. It can be seen that there is tradc-off between complexity of truncated value-estimattion and the faster composition of the value by the Shifted Q-functions. If the rewards are very noisy, it appears to be of advantage to set n to a smaller value in order to simplify the learning problem of truncated valuo-ostimation (as in our experiments in Scction 4.2.2 On the other hand, in order make use of the temporal chaining of the Shifted Q- Q-functions, the horizon should cover multiple time steps.
13

---

Published in Transactions on Machine Learning Research (07/202) 2022)
10
0.8
Noraalasd Agea
Lewrning
0.7
0.5
0.3
10
15 20
0.001
0,002
0.001
0,0005
0.001
0,002
Pb
8%
Figure 8: Normalized area under the learning curve for Composite TD3 in the Walker2d-v2 environment with different truncation horizons n (left) and different regularization weights I (middle) and , (right).. The plots show median and interquartile ranges over training runs, each representing mean evaluation performance over 100 initial states.
4.2.2 Noisy Reward Function
We evaluate Composite TD3 under noisy reward function and compare TD3, TD3 with high learning rate and TD3(A). The immediate reward is replaced by uniform sample u ~ 2[-1, with 40% chance, corresponding to the strongest. noise level described in (Romolf et al. 2019). In order to account for the high variance, we provide mean and standard deviation (SD) over 8 runs. Results can be seen in Figure 9 corresponding significance tests are provided in Table D.2 in the appendix. Composite TD3 proves to be very robust, even for the very complex Humanoid-v2 environment. All other approaches suffer from slower learning speed and high variance. This holds especially for TD3 with high learning rate and (A). We believe this to be caused by overfitting to the noisy reward function due to the high learning rate in TD3 HLL and due to the low gamma of the first value functions in TD)(A). Since all other predictions only add to the prediction of the preceding head, there is nothing to prevent the overestimation bias from being propagated to the other outputs of higher discount value. In Composite Q-lcarning however, Shifted and Truncated Q-functions estimate distinct parts of the temporal chain. This is also highlighted in Figure [0] where the TD-errors Shifted and Truncated Q-functions can be seen along with learning curves of Humanoid-v2, Whilst in the default setting with vanilla reward the Truncated Q-functions converging much faster than the Shifted Q-functions due to the simpler problem induced by the smaller horizon, the opposite holds in
Walker2d-12
Hopper-v2
Humanoid-v2
Composite TD3
TD3
TD3 HT.
T33A
3000
5000
2500
4000
Evalustion Rettur
3000
Evalastion Retura
3000
Exalustion
2000
2000
1000
1000
500
1000
0-
0-
Transitions
x110
Transitions
x10
Transitions
x10
Figure 9: Mean performance and half SD training for (left) 2al (middle) Hopper-v2 and (right) Humanoid-v2 with uniform noise the reward function in (Romoff et al. 2019).
14

---

Published in Transactions on Machine Learning Research (07/2022)
Table 2: Mean normalized area under the learning curve and SD over 8 training runs of Composite TD3 the noisy reward experiments.
Method
Walker2d-v2
Hopper-v2
Humanoid-v2
TD3
68%
22%
76%
28%
68% 1: 30%
TD3 (HL)
56%
18%
72% 1 37%
38% + 29%
TD3(A)
52%
23%
49% + 31%
56% + 26%
Composite TD3
100% + 20%
100% + 25%
100% + 16%
case of the noisy reward. In these experiments, the Truncated Q-functions are not capable of an accurate prediction within the given time frame, yet the Shifted Q-functions can construct an accurate chain given those inaccurate predictions reliably leading to a well- well-performing policy. This backs our hypothesis that Shifted Q-functions can benefit from higher learning rales also in noisy reward settings whereas Truncated Q-functions have to account for variance in the reward as well, which adds the complexity of the problem significantly. This translates to faster learning, underlined by the results in Table It can furthermore be observed in Figure 10 that Truncated Q-functions corresponding to shorter horizons yield smaller TD- error than Truncated Q-functions corresponding to longer horizons. The same holds for Shifted Q-functions where estimations at a later point in time (that is, with stronger shifting) have lower TD-error than Shifted Q-functions estimating from an carlier point in time. This also highlights the importance hyperparameter n, since it directly influences the point in time al which Shifled and Truncated Q-functions are combined. Please note that due the recursive dependencies of the composite action-value and the possibly very long horizon of task, also small diffcrences can propagate quite readily.
True Reward
Sby
Slay
Sh
Evaluation Return
TD-Eirro 3
4000
(a)
2000
0-
2
10
Transitions
Noisy Reward
T
Sla
Shs
Slap
Sb.
Evaluation Retarn
4000
(b)
TD-Erro
2000
0-
2 x 10
1
Transitions
Figure 10: Performance and TD-errors on Humanoid-v2 of Truncated and Shifted Q-functions at 1,....,
with respect to the (a) vanilla and (b) noisy reward. Please note that TD-error here means the deviation from the associated target.
15

---

Published in Transactions on Machine Learning Research (07/2022)
Table 3: Mean maximum return and SD over 8 training runs for the noisy reward experiments.
Method
Walker2d-v2
Hopper-v2
Humanoid-ve v2
TD3
3063 + 813
2386 + 729
4453 = 1070
TD3 (HL)
2950 866
2124 1054
2506 = 2004
TD3(A)
2772 + 1245
1242 + 954
3251 = 1687
Composite TD3
4041 + 476
2931 + 457
5019 = 404
Table shows the mean maximum return achieved by Composite TD3, TD3 with default and high learning rate and (D3(A). Composite TD3 reaches significantly higher returns with lowest variance compared to vanilla TD3 when applied to a noisy reward function, especially for the most complex environment Humanoid-v2.
Our experimental results show that Composite Q-learning offers significant improvement over traditional Q-learning methods and great potential for real-world applications such learning directly real robots, where the extraction of the immediate reward signal can be noisy due to occlusions, complex training setups, scnsor and actuator noise or fitted reward functions.
5 Conclusion
We introduced Composite Q-learning, an off-policy reinforcement learning method that divides learning of the full long-term value into a series of well-defined shorter prediction horizon estimates. It combines Truncated Q-functions acting on a short horizon with Shifted Q-funetions for the remainder of the rollout. As a baseline, we further introduced and evaluated TD3(A) an off-policy variant TD(A). We showed on three simulated robot learning tasks that compositional Q-lcarning mothods can be of advantage with respect to data-efficiency compared to vanilla. Q-learning methods and that Composite TD3 outperforms vanilla TD3 by 24% 32% in terms of area under the learning curve. In addition, Composite Q-learning proved to be very robust to noisy reward signals which is very important for real-world applications where the immediate reward may depend on a stochastic next state or where the reward function is estimated from data.
Going forward, using ensemble uncertainty estimates based on the variance of all predictions from the Composite Q-function could be of benefit in update calculation, transition sampling and exploration. We further believe method to be better fit. for non-stationary reward functions than traditional Q-learning methods due to the flexibility provided by the divided long-term return. Representing the farsighted return after steps of policy rollout in the future, Shifted Q-functions could serve as an implicit dynamics model to further build the bridge between model-based and model-free methods. Lastly, the evaluation of further decomposition is a promising direction for future work and could bring even more ficxibility to the learning process. It could be accomplished by shifting a truncated mnlue-estimation in time. However, this would probably come the cost of more hyperparameters, which could make the employment of more sophisticated hyperparameter optimization and meta-learning methods necessary.
References
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2017.
Kristopher De Asis, Alan Chan, Silviu Pitis, Richard S. Sutton, and Daniel Graves. difference methods for stable reinforcement learning. Tn AAAT Conference on Artificial Intelligence, 2020.
temporal
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl gym. CoRR, 2016.
16

---

Published in Transactions on Machine Learning Research (07/2022)
Nicolai Dorka, Joschka Boedecker, and Wolfram Burgard. Dynamically balanced value estimates for actor- critic methods, 2020.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value expansion for efficient model-free reinforcement learning. CoRR, 2018.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. In Neural Information Processing Systems (NeurIPS), 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018.
Tuomas Haarnoja, Schoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Robolies: Science and Systems (RSS), 2019.
Diederik P. Kingma and Jimmny Ba. Adam: A method for stochastic optimization. In International Confer- ence on Learning Representations (ICLR), 2015.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Sil- ver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
Oier Mees, Markus Merklinger, Gabriel Kalweit, and Wolfram Burgard. Adversarial skill networks: Unsu- pervised robot skill learning from video. In IEEE International Conference on Robotics and Aulomation (ICRA), 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Tianwei Ni, Harshit S. Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Benjamin Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. CoRR, 2020.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost. Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning (ICML), 2018.
Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Trançois-Laved and Joelle Pineau. Reward estimation for variance reduction in deep reinforcement learning. In Conference on Robot Learning (CoRL), 2018.
Joshua RomolT, Peter Henderson, Ahmed Touati, Yann Ollivier, Emma Brunskill, and Joelle Pineau. Sepa- rating value functions across time-scales. CoRR, 2019.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Riedmiller. Deter- ministic policy gradient algorithms. In International Conference on Machine Learning (ICML), 2014.
David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Gracpel, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In
AAAI Conference on Artificial Intelligence, 2016.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.
17

---

Published in Transactions on Machine Learning Research (07 (0222)
A Theory of Composite Q-learning
A.1 Proof of Proposition
First, provide theoretical foundation for Composite Qllearning
Proposition A.1. Let (8, ar) = Tt be the one-step Truncaled Q-function and ( = Tt + ( a(+1)] the i-step Truncated Q-function. Then Q (St, represents the truncated return (s) Ei,x,M(E - y 'ry.
Proof. Proof by induction. (8,, ax) = Tr by definition. The theorem follows from induction step: Q (1 ax) + Er.M
[Q_ (81+1.
(++))((-1)-
1)-1
= + 12 E..M
Σ
7 7
j=(++1
141-1
7
Σ T
-=((+1)
= E
Σ x grrrr
j=(++1
iti-1
Ex,M
Σ p
j=t
A.2 Proof of Proposition 2
Proposition A.2. Let Q (s, at) E (st+1, a+41)l be the one-step Shifted Q-function and Q 1:00 ae) = E 1o0 (1441, the i-step Shifted Q-function. Then Q (s1, represents the shifled return Q (8e, ue) E (xr+i,
Proof. Proof by induction. Q7. (st, ar) Bx.Ml)Q"(S441 a++1) by definition. The theorem follows from induction step:
Q (8), a,)
Er.M 10
Er,M ( Q1
Er,M (F((((((()
E..M Q 0++:]
A.3 Proof of Proposition 3
Proposition A.3. Let Q,,(s, a+) E - p be the truncated return and Q Q ar) "..., a[+n)] the shifted return. Then (st. 0z) Q (st, 0t) (St, at) represents the full relurn, i.c. Q Q" 04) (-
18

---

Published in Transactions on Machine Learning Research (07 (0722)
Proof.
(T(ss, (n ar) +
Q Qn:co (st, at)
b+n-1
E
/ (" attn)
(sttn,
j=
M
Σ
p + 7"
(setn axnn) + Qu:00 (8t4no aetin)
h+n-1
1+2n-1
=E.M
Σ
p- + 7
Σ
g + + ($t+2n ( act2m))
j=tt
tta-1 Σ
++2n-1
En,M
7 1-1
Σ -5 + 7 Q (8t+2m ac+2n)
jut
jmttn
2+2n-1
EE,M
, + -2 (82+2n) a(+2n)
By induction, it follows (st, E,,,M Ej= 70 (ry-
Additional Evaluation of Tabular Composite Q-learning
In this section, we provide an additional evaluation of tabular Composite Q-learning applied to the MDPs
in Figure 2a and Figure
16
16
20-
Q-viltic for state So and action
-5-
Q-vaho state d6 and actioon
10
60
880
-15
120
-20
-100
160
0
2
3
5 x10PP
0.0
0.2
0.4
0.6
0.8
1.0
Updates
Updates
(b)
(a)
Figure B.1: (a) Performance of Composite Q-learning applied to the MDP in Figure
2a with different rollout
lengths (b) Performance of Composite Q-learning applied to the MDP in Figure with different rollout lengths n.
19

---

Published in Transactions on Machine Learning Research (07 (0222)
C Hyperparameter Setting
For all approaches, we use Gaussian noise with o 0.15 for exploration and the optimized learning rate of 10-3 for the full Q-function. Target update (5 10-3) and actor setting (two hidden layers with 400 and 300 neurons and ReLU activation) are set as (Fujimoto et al. 2018). For Humanoid-v2, we use a slightly changed parameter setting with a learning rate of 10 for both actor and critic as suggested (Dorka et al. 2020). For Composite Q-learning, we calculate the full Q-values, well Truncated and Shifted-Q values in one, combined, architecture for improved efficacy. Additionally, we found that it is beneficial to estimate the Truncated and Shifted Q-values in different layers, as depicted in Figure C.1 For all parameters in the layers prior to the full Q-output, we use the paramcter sctting as described above.
8 filly connected layers (200) losky Rel.U)
filly oonncted 500, loak Rof.1)
fully conneted 200, laky ReL.U)
Figure C.1: Optimized architect of the Composite-Q network used in our experiments.
Hyperparameters are optimized for all approaches, including the baselines TD3 and TD3(A), for the given subset of MuJoCo tasks via Random Search over 10 training runs with the configuration space shown in Table C.1 Learning rates for the corresponding output layers of the Truncated and Shifted Q-values are optimized individually. We use learning rale of 6 10-3 for the Truncated Q-functions and 5 10-3 for the Shifted Q-functions. For the noisy experiments, as well as for the experiments with multiple gradient steps per collected sample, we set = 4. For the Walker2d-22 experiment on the true reward function, we set n = 10. In terms of regularization, we set 81. = 0.002 and Asa- = 0.001. For Humanoid-v2, we set the optimized learning rates one magnitude lower. In the experiments with multiple gradient steps per collected sample, we execute five consecutive updates per collected sample and keep the learning rate of the Shifted Q-functions at 10-3 We further evaluate TD3 with the same maximum learning rate. We optimize the number of layers and the number of neurons per layer for the critic in Composite Q-learning (4 layers with 500 neurons and Icaky ReLU activation) and the architecture for the critic in all other approaches (two layers with 500 neurons and leaky ReLU activation). For an overview of the architecture of the Composite-Q network see Figure C.1. For TD3(A), we use the s-schedule as suggested by Romoff et al. 20099 ie. ?1 = 0 and 7 2- with an upper limit of 0.99.
Table C.1: Configuration space of the hyperparameter optimization. For the full Q-function in all approaches,
we used the optimized learning rate of 10-3 as in (Fujimoto et al. 2018). Hyperparameters denoted by were optimized individually for all approaches.
Hyperparameter number of layers* number of neurons*
Values
32,3.4
(300, 400, 500}
activation"
( (ReLU, leaky ReLU)
{6 10 10-110-6,10 10 11006,101..6.11111
6. 10- 5, 10-33
aTr
210-3.2.10-3. 10-3, 10
osS
Br
210-3,2.10----1 10-33
Ash
15-100-,10-3, 10-3
n
(3,4,10,15, 15,
20

---

Published in Transactions on Machine Learning Research (07 (07/2022)
D Significance Tests
We provide significance tests on the mean area under the learning curve (to account for learning speed and stability) between Composite TD3 and TD3, TD3 (HL) and TD3(A). Significance tests for the vanilla reward experiments can be found in Table D.1
Table D.1: p-values for Welch's t-test on the mean area under the learning curve for different runs of Composite TD3 and all other approaches in the vanilla reward experiments. Subparts (a) and (b) correspond to those in Figure 7
TD3
TD3 (HL)
TD3(A)
Walker2d-v2 (a)
8 10
7 7-10-*
6 10
Walker2d-v2 (b)
3- 10-1
2-10
5. 10-10
Hopper-v2 (b)
3 10-5
10
5-101
Significance tests for the noisy reward experiments can be found in Table D.2
Table D.2: p-values for Welch's t-test on the mean area under the learning curve for different runs of Composite TD3 and all other approaches in the noisy reward experiments.
TD3
TD3 (HT.)
TD3(A)
Walker2d-v2
4 10-b
2 10-8
6 10
Hopper-v2
8 10-3
8 10-3
2. 10-6
Hu
3. 10
4- 10-
50-
E Evaluation of Composite DQN on LunarLander-v2
LunarLander-v2
Composite DQN
DQN HI
DQN
200
150
100
Eviluation Rosumn
50-
0-
-100
-150-
200
2000
Episodes
(a)
(b)
Figure F.1: (a) Visualization of the LanarTander-v2 environment. (b) Mean performance and half SD over training runs for LumarLander-v2 with uniform noise on the reward function.
In addition to the continuous actor-critic experiments in Section 4 we evaluate Composite Qelearning within DQN on the LunarLander-v2 environment (shown Figure E. The immediate reward again replacod by uniform sample u 21-1,11 1] with 40% chance. We use a learning rate of 10-2 for the Shifted Q- functions and 10-5 for the Truncated and full Q-functions in Composite Q-learning. We compare to DQN
21

---

Published in Transactions on Machine Learning Research (07 (07/2022)
with learning rate of 10 (DQN) and with learning rate of 10 (DQN HL) in Figure E.1b As can be seen, Composite Q-learning shows the same tendencies in the value-based method DQN as in the actor-critic method TD3.
F Comparison to Multi-step Baselines
Data-cfficiency of on-policy Temporal-Difforenn methods can be enhanced by the use of n-step returns, where a Monte Carlo rollout of length n is combined with a bootstrap of the value function. Reinforcement learning methods of this kind are called multi-step. Since our method takes up on a similar rationale, we lastly add comparison to two multi-step methods.
To employ n-step returns in off-policy setting, subtrajectories of the exploratory policy have to be stored. These stored multi-step returns, however, will differ from the true value of the target-policy. We evaluate this setting as On-policy Multi-step TD3 in the following. To avoid the problem of the multi-step return being off the target-policy, learned dynamics model can be used for imaginary rollouts, the so-called Model-bass based Value Expansion (MVE) (Feinberg et al. (018)). We hence employ Model-based Value Expansion within TD3 as additional baseline, subsequently called MVE-TD3.
We add Gaussian policy smoothing to the rollout of the model. In contrast to Feinberg al. (2018). however, do not assume to have knowledge about the reward function. Our model therefore approximates both. dynamics and reward. For MVE-TD3, for the considered benchmarks. For On-policy Mulli-slep TD3, we use the same truncated horizons for Composite TD3. Code for both multi-step baselines can be found in the supplementary
rollout length of three, as described in Feiinerrg al. (2018)
Results for Walker2d-v2 with vanilla reward and a single update per collected sample, as well as results for Walker2d-v2 and Hopper-v2 with vanilla reward and five updates per sample are depicted in Figure F.la and Figure F.Ib respectively. MVE-TD3 is highly sensitive with regard accumulating errors in reward and state prediction, even when applying the TD-k trick. The erroncous updates of On-policy Multi-step TD3 are also causing harm to value-estination, especially for the more stochastic Hopper-v2 environment. The results emphasize the importance of truly off-policy and model-free truncated value-ostimation.
Walker2d-v2
Walker2d-v2
Hoppe-v22
Compasite TDS
TD3 IIL
Compasice TD3
TDS IIL.
TD3
TD3A)
T13
TD3(A)
On-poliey Multisstop TD3
MVE-TDD TD3
Ourpolicy Multi-step TD3
MVE-TU TDS
5000
3500
1000
3000
4000
Exalastion Rearn.
antion Rearn
2500
3000
3000
2000
Inaalaation
2000
1500
1000
1000
500
2
Transitious
x10
Transitions
x10
Transitions
x10
(b)
Figure F.1: (a) Mean performance and half SD (standard deviation) over 8 training runs for the Walker2d- v2 environment. with the default reward. (b) Mean performance and half SD over 8 training runs of Composite Q-learning on the vanilla reward function and multiple update steps collected sample for the Walker2d-v2 and Hopper-v2 environment.
22


Reviews:
[{"review_id": "bp8Vl0AkB3", "reviewer": "Reviewer_ZC7n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths : \n\n\t1. The paper is very clearly written and the methodology, including the figure explaining the approach was very helpful. It has a very clear explanation of the idea, and is brutally honest about what is novel and not. I like the fact that authors even include an off-policy baseline TD3 (Delta) based on modification of their approach and an existing work. \n\n\t2. The theoretical claims seem justified; although some analysis of the bias variance trade off would probably be useful. The theoretical results seem to justify the use of achieving lower variance returns too, which is helpful but perhaps require more clarity. The n-step returns that composite Q depends on clearly has a bias variance trade-off depending on the choice of n. \n\n\t3. Experiment results are well demonstrated and clearly shows significance in standard control tasks, and more so when additional noise is introduced to rewards. The benefits seem obvious, and I will not complain much about the use of baselines, except the fact that future iterations of the paper probably needs some fair comparisons with baselines that also uses n-step returns (I am ok with if that is not demonstrated in a control task, but in some other domain - see comments below).\n\n\t4. The clarity in the paper, and clearly showing what the algorithm does, and the illustration of the approach is appreciable. \n\n\t5. Algorithmically I think the paper has novelty, and if demonstrated well in some other stochastic environments, then this can be a useful approach. Although it is easy to complain that this paper clearly builds from existing works of Romoff et al., - I think to make this kind of approach work in practice is clearly a good contribution of the paper. I am just not convinced whether the claims are enough experimentally supported - maybe the approach has other benefits, and as such a careful re-writing of the claims would perhaps be helpful. \n\t\n\t\n\nWeaknesses: \n\n\t1. I am worried about the comparisons of baselines somewhat - mainly due to the fact that the baselines don't all seem to use n-step returns, and rather uses 1-step return for the off-policy counterpart? Do the authors have a good sense of how this might affect results? I understand that the proposed method needs to be evaluated for n-step returns, while the baseline being off-policy needs to be done for 1-step return. Why not include other experiment comparions which uses some version of V-traces or other TD(lambda) based algorithms that are applicable for control? There used to be an ACER algorithm for control too, which uses a n-step return version for the critic in control task - which might be helpful.\n\n\t2. The proposed claim of the paper is to improve robustness in presence of stochastic environments or sparse reward signals. I am not fully convinced that the proposed claims are justified through the series of experiments? The control environments are all deterministic; simply adding noise to reward signals does not seem like a very well justified approach? \n\n\n3. I am not sure whether the limitation of applying composite Q learning has to be based on control experiments only. Clearly the method can be general and can be applied widely in any existing off-line setup. It might be helpful to think about a different domain where the claim of \"robustness to stochasticity\" can be better demonstrated?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a composite Q learning algorithm where the temporal difference (TD) update is decomposed into multiple truncated Q functions, the returns of which are based on a fixed temporal horizon. The authors argue that fixing the temporal horizon, and using a decomposition technique to then combine the multiple Q functions together, is a useful approach to handle robustness, in presence of noise in reward signals or if the environment is stochastic. As a bi-product, different hyperparameters can be used since the composition is based on two different Q estimations(based on truncation and shifting of Q values). \n\nThe authors try to demonstrate these claims through a series of experiments, both for continuous control and toy discrete action domains; while the theoretical results argue that even under this compositionality, the full return as in a standard Q learning algorithm is still maintained (thereby, existing theoretical convergence results, or divergence counterexamples would still hold). The main benefit of the compositionality is the use-case in presence of noise reward signals (which demands study in the field to enhance practicality of existing RL algorithms) or when environment is stochastic (where robustness is of importance). The resulting algorithm is mainly integrated into a well known, state of the art TD3 algorithm and experiments demonstrate that this plug-in addition of composite Q values, in the critic estimation of TD3 can be a useful technique. \n\nFrom an algorithm and methodology perspective, the paper does propose a new approach which can be useful, although it comes with its own drawbacks of being able to compute, possibly with different function approximators too, for the different Q functions. The ability to decompose n-step returns into multiple Q functions has been studied in the past, but the paper proposes a different technique of doing this. The practical usefulness of it, and whether it sufficiently demonstrates the claims for which this technique is useful, is perhaps questionable and would require further study. Please see comments below. \n\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\t1. The experiment comparisons with a 1-step version seems concerning to me; it is not clear what to expect if we could compare with a n-step return baseline too, since composite TD3 clearly uses n-step returns.\n\nOverall, my major comment would be that the proposed claim of being able to handle stochasticity is perhaps not well established. It is quite clearly shown in the toy tasks, but not on some of the main tasks of the paper. I am ok with if the toy-ish tasks are included as main draft in the paper - because clearly it does demonstrate the usefulness of the approach there. \n\nI would suggest a careful re-write of the paper in terms of how the proposed claims are demonstrated would indeed be helpful for the wider audience.  Otherwise, the paper is well written and easy to follow. \n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sZeB5FaaPE", "reviewer": "Reviewer_qbp7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.** There are many things to praise about this paper. Succinctly:\n1. _Algorithmic Simplicity_: The main idea giving rise to Composite Q-learning is intuitive, well-explained, and ultimately simple enough that I believe the community can iterate and experiment with it quite readily.\n2. _Clarity_: The paper and appendix are well presented, taking its time to expose the key insights that constitute Composite Q-learning. I have some small suggestions for improving clarity which I discuss below.\n3. _Experimentation_: The experiments, by my reading, are very sensibly-motivated and well-conducted. Results tell a clear story and the claims made in the paper are supported by the empirical evidence. In particular I found Figure 4 and Figure 6 to be very interesting plots.\n\nTo summarize: I believe the contributions in the paper are both of interest to members of the ML community, and the main claims are well supported.\n\n**Weaknesses.**\n1. _Notation (Pseudocode and Superscripts)_: As I detail below in the writing suggestions, I found some of the notation to be overly complicated at the detriment of clarity in some passages. In particular, the use of subscripts and superscripts in some of the Q notation, and especially Algorithm 1, can be much improved.\n2. _Discussion_: I believe some of the experimental section could include further discussion drawing out take aways and insights from the results (specifically Figure 4 and 6---further comments given below).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Composite Q-learning, a conceptually clean new variant of Q-learning that combines the estimation of two kinds of Q-functions: (1) The \"Truncated\" Q function, and (2) the \"Shifted\" Q function. The Truncated Q function is an n-step roll out return, while the Shifted Q function is an estimate of the return _after_ this n step rollout. Composite Q-learning, abstractly, is a generalization of Q-learning in which different learning rates (and hence optimization schedules) can be applied to each of these constituent Q-functions. The paper suggests exposing this extra degree of control over the optimization process, variance can be controlled to a greater degree, thus allowing for the use of higher learning rates (particularly in environments with stochastic rewards, or rewards that depend on a stochastic future state). These insights are combined into a few algorithmic variations of the central idea, including TD3($\\Delta$), that extends TD$(\\Delta)$ to off-policy learning. Thorough experiments are conducted in simulated control tasks such as Walker and Hopper in two varieties: the first uses a clean reward signal, while the second uses a noisy reward signal (following work by Romoff et al. 2019). Data are examined and presented carefully, with the primary evidence supporting the claim that Composite Q-learning can be particularly data-efficient under reward noise.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Reactions**\n\nI was left with one question. I do not believe the authors need to address this question in the paper by any means, but I thought it might be a useful question to think about:\n- Does further temporal decomposition help? That is, Composite Q-learning decomposes Q into the first n steps, then the future value. This extra granularity allows for greater control over the learning process and parameter schedules. But, Equation 2 is suggestive that there might be _arbitrarily_ many such distinctions; is it valuable to continue going beyond a single split? That is, might we consider a generalization in which we maintain 3 Q estimates: $Q_{[1:n]}, Q_{[n:n+k]}, Q_{[n+k:\\infty]}$?  We might generalize this further to arbitrarily many such distinctions. I was curious if the authors had thought about any potential advantages from such an approach.\n\nI have three primary recommendations for improving the paper:\n- Recommendation 1: I believe more discussion of both Figure 4 and Figure 6 could strengthen the work, as it seems there could be many insights we might draw here. Can you expand on what you take to be the important findings from these data? The discussion of Figure 4 in particular is quite brief at the moment.\n- Recommendation 2: I was slightly surprised to see Theorems in the appendix my first read through. Are any of these results central enough to include in the main text?\n- Recommendation 3: I found many of the experiments in the appendix to be quite insightful. If the authors find space, I believe they could have a more natural home in the main text of the paper (perhaps just C.1, if space is really tight).\n\n\n**Writing Suggestions.** I have a few low-level writing suggestions:\n\n1. Introduction:\n- \"environments of higher complexity\": Higher complexity than what? Perhaps \"of high complexity\" would be more suitable here.\n\n2. Background\n- The $\\mapsto$ symbol should be changed to $\\rightarrow$ in the introduction of the MDP components. \"$\\mathcal{X} \\rightarrow \\mathcal{Y}$\" is used to express that a function takes as input elements $x \\in \\mathcal{Y}$ and outputs elements $y \\in \\mathcal{Y}$, whereas \"$\\mapsto$\" actually describes the map: $f : x \\mapsto x^2$, for instance, shows that $f(x) = x^2$.\n- I find the notation $\\mathcal{R}$ for the expected return to be non-standard compared with calling this quantity the \"value\" function, expressed with either $v$ or $V$.\n\n3. Combining...\n\n- The superscript and subscript notation became hard to parse quite quickly, starting around Equation 4, and especially Equation 5 on. I very quickly lost track of the precise meaning of what $\\theta^{Q'}$, $y_j^Q$, $Q^{Tr}$, and found myself regularly going back to double check their definitions. In particular, I found parsing Algorithm 1 to be extremely time consuming. I have two recommendations: (1) Add vertical spacing between the lines of the pseudocode to prevent the superscripts/subscripts from overlapping (under the \"calculate targets\" section), and (2) Consider alternate forms of notation apart from superscripts, such as hats/overlines or changes in letter/font.\n\n4. Experiments:\n- Just to be really precise, it would be useful to define the acronym \"SD\" when it is first used.\n- A personal preference, but I find the use of bolding in tables when the confidence intervals overlap to a large degree to be slightly misleading, and would encourage removing the bold (or just explaining its use in the caption).\n\nBibliography:\n- A few titles should be capitalized: \"Averaged-dqn\" --> \"Averaged-DQN\", \"Openai gym\" --> \"OpenAI gym\", and \"...and go\" --> \"...and Go\".\n- Just a personal preference, but the citation format for conferences is slightly inconsistent throughout. For instance, \"in 4th International Conference on Learning Representations, ICLR, 2016.\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r49DgLAVg8", "reviewer": "Reviewer_uBm3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The paper is pretty well written, it was fairly easy to follow. There are a few things I would suggest improving, which I list below.\n2. The experimental results are clearly presented, with confidence bounds, and evaluated over a number of different settings (in particular, adding stochasticity in the rewards).\n3. The algorithm is clearly presented and discussed.\n\n---\n\n# Weaknesses\n\nMy main criticism of this paper is that, while the results seem compelling, it is not clear to me whether this is a general-purpose method that can help other type of algorithms and/or environments.\n\nOne thing that stood out to me is that the proposed method seems to be aimed at better _value estimation_, yet it is only evaluated on an actor-critic method where value estimation only affects the policy indirectly (e.g. the critic). It would be far more convincing if the authors could evaluate their method on value-based methods (such as DQN and offshoots), as here improper value estimation _directly_ affects the quality of the policy.\n\nA few more specific concerns below.\n\n## Theory\n\nAlthough I appreciate the authors wanting to provide a theoretical backing for their proposal, the theoretical results presented in the appendix are quite weak. In fact, it seems they are just (re)deriving the definitions of truncated and shifted Q-values. In particular, Theorem A.3 is really just saying that the standard definition of Q-values (which is an infinite sum) can be split into two sums: a finite one, and what comes after. This is quite clear (it's just a property of sums) and doesn't really require \"Theorems\".\n\nOne related nit on this point: t the top of page 4, the authors say \"Following Theorem A.1 in the appendix, we approximate $Q^*_n(s_t, a_t)$ off-policy via consecutive bootstrapping.\" Theorem A.1 is not about approximation, it's really just defining truncated Q-functions, so it's not clear what is meant here.\n\n## Implementation details\n\nThere are some implementation details that are introduced but not completely justified. It's not clear how critical they are to the algorithm's performance, but I feel they merit further empirical analysis to properly understand.\n\n1. It seems the authors are using a combination of multiple heads for estimating the truncated Q-functions (perhaps this is what the authors meant by \"consecutive bootstrapping\" above?). This is somewhat unorthodox and it's not clear how necessary this is. Without the multiple heads this becomes quite similar to n-step backups.\nIn page 5 the authors say \"Composite Q-learning is equivalent to Q-learning if the learning rates for Truncated, Shifted and full Q-function are set to the same values.\" I don't think this is correct, precisely because of the use of multiple heads. It would be nice to evaluate the effect of the multiple heads on regular Q-learning (e.g. with the same learning rate values).\n2. In Algorithm 1, the authors are adding $\\xi$ to the action selection. Is this necessary to the algorithm? It seems orthogonal to the idea proposed. Perhaps this is something that is coming from TD3, in which case it would be good to highlight the differences between Algorithm 1 and TD3 (using colors can be effective for this).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Composite Q-learning, a new reinforcement learning (RL) algorithm that combines two estimates of Q-values:\n1) A \"truncated\" Q-function (esentially a discounted, episodic Q-value)\n2) a \"shifted\" Q-function (standard Q-value, but starting from a state further in time).\n\nThe authors claim their method is able to deal with stochasticity in the environment better than prior methods, and is more robust to shifting learning rates, and argue for this by comparing against a strong existing baseline (TD3) on a few MuJoCo tasks.\n\nThe authors provide their code with their submission, which is always appreciated.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "# Main suggestions\nThe suggestions marked as critical are those I feel are needed for me to recommend acceptance. \n1. **Evaluate on value-based methods (critical)** Evaluate incorporating the algorithm on value-based methods such as DQN. (See \"Weaknesses\" above for details).\n2. **Higher learning rates (critical)** Some of the analyses in the toy example (currently in the appendix) could be moved to the main paper, or at least discussed there. In particular, in a number of places in the main paper the authors claim their method is more robust to higher learning rates, but this is only really evaluated in the toy experiment (which is in the appendix). In addition, perhaps the authors could run CompTD3 with a higher learning rate for the MuJoCo tasks. In the figures in the main paper it appears they only tried a higher learning rate with TD3.\n3. Equation (5) seems to imply that $Q^{Sh\\prime}_{i-1}$s is learning the $\\gamma^{i-1}$ multiplier, is this correct? If so, this should be made more explicit.\n4. In section 4 the authors say \"for Soft Actor-Critic the same results can be assumed to hold.\" No. Just because a method works well when added to one algorithm does _not_ mean it is guaranteed to hold on other algorithms. This would require actually running the experiments to verify this. I think running with value-based methods would be more critical, but running on SAC would strengthen the paper a lot more. At a minimum, I'd remove this sentence as there's no real justification for it.\n5. **AUC values (critical)** The authors specify a percentage for AUC, but it's not clear what this percentage represents (e.g. Composite TD3 is reported to obtain 100% AUC). The authors say it is \"mean normalized area under the learning curve\", but it's not clear _what_ it's normalized against.\n\n# Questions\n1. In Figure 6 it seems Tr4 has more variance in the True Reward setting compared to the Noisy reward, which is counterintuitive. Why do we see this?\n2. In Figure 6 why is there such a big difference in TD errors for Tr4 between true and noisy rewards?\n3. In Figure C.5 why does Composite Q-learning have multiple learning rates in parentheses? Did they all have _identical_ learning curves? This seems improbable...\n\n# Minor issues\nSome minor issues that can (hopefully) help improve the paper:\n1. Abstract line to: s/\"in their application for robot control\"/\"in their application to robot control\"/\n2. In the introduction, first paragraph: s/\"are still far away from being\"/\"are still far from being\"\n3. In the introduction, first paragraph: \"their lack in learning stability caused by the stochastic nature of transitions in real-world settings\". Stochasticity affects things, for sure, but it's not _the_ cause, just one of many.\n4. In the introduction, first paragraph: s/\"are then combined in a mutual recursive definition\"/\"are then combined in a recursive definition\"/\n5. Top of page 2: It's not clear what is meant by: \"Since the values corresponding to larger discount values only represent the residual to the values of smaller discount.\"\n6. Top of page 2: Better to not use acronyms (\"w.r.t\") and just write it out explicitly.\n7. I would suggest moving Figure 3 to the top of the page, as there is currently a single (partial) sentence above it that is easy to miss.", "broader_impact_concerns": "No real concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes two TD formulations: truncated Q-functions and shifted Q-functions, which together allow composite Q-learning for jointly learning Q-values across different temporal horizons from only single-step transitions. The method shows improved sample efficiency and more robustness to stochastic rewards, on simulated toy control environments.

Pros:
- All reviewers approved the clarity and simplicity of the presented methods. This could be a good educational paper for RL researchers looking into novel formulations of TD learning, especially bridging single-step and n-step approaches.
- The authors provided sufficient revisions, particularly re-organizing sections of the paper for clarity, and toning down claims originally made about stochastic environments. This helped better align claims and results included in the paper.

Cons:
- I agree with ZC7n that this paper as it stands does not have significant empirical impact. Not likely that many (deep) RL practitioners will switch to use this algorithm.
- Some n-step baseline is highly recommended to be added, e.g. TD3 + n-step without importance correction.
- We did not reach consensus (two accepts, one lean to reject)

The final recommendation of acceptance is based on the promising educational value of this paper, for its clear and simple derivations and analyses; however, we also note that empirical value of this paper is limited due to lack of comparisons (e.g. with n-step baselines) and evaluations on harder/more realistic domains. We encourage the authors to further improve the work, such that it can benefit wider audience of TMLR. 
Decision: Accept (poster)