Extreme Q-Learning: MaxEnt RL without Entropy
Authors: Divyansh Garg
Joey Hejna
Matthieu Geist
Stefano Ermon
Venue: ICLR 2023 notable top 5% (2023)
Keywords: reinforcement learning
offline reinforcement learning
statistical learning
extreme value analysis
maximum entropy rl
gumbel

Abstract: Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by \emph{10+ points} on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks. Visualizations and code can be found on our website.

Published as a conference paper at ICLR 2023
EXTREME Q-LEARNING: MAXENT RL WITHOUT EN- TROPY
Divyansh Garg" Stanford University di divgarg@stanford..ed
Joey Hejna" Stanford University
Matthieu Geist Google Brain mf mfgeist st.gsogttoooolo.o com
jhe stttnnadttaforrr..coo edu
Stefano Ermon
Stanford University smmonsstanford.edd
ABSTRACT
Modern Deep Reinforcement Learning (RI) algorithms require estimates of the maximal Q-value, which are difficult compute in continuous domains with an infinite number of possible actions. In this work, we introduce new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, avoid computing Q-values using out-of -ifiiistinntoinn actions which is often a substantial source of error. Our key insight is introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL. setting without needing to sample from a policy. Using EVT, derive our Extreme Q-Learning framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by 10+ points on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 online DM Control tasks. Visualizations and code can be found on our website
INTRODUCTION
Modern Deep Reinforcement Learning (RL) algorithms have shown broad success in challenging control (Haarnoja et al., 2018; Schulman et al., 2015) and game-playing domains (Mnih et al., 2013). While tabular Q-iteration or value-iteration methods are well understood, state of the art RL algorithms often make theoretical compromises in order to deal with deep networks, high dimensional state spaces, and continuous action spaces. In particular, standard Q-learning algorithms require computing the max or soft-max over the Q-function in order to fit the Bellman equations. Yet, almost all current off-policy RI. algorithms for continuous control only indirectly estimate the Q-value of the next state with separate policy networks. Consequently, these methods only estimate the Q-function of the current policy, instead of the optimal Q", and rely on policy improvement via an actor. Moreover, actor-critic approaches
their own have shown to be catastrophic in the offline settings where actions sampled from a policy are consistently out-of-distribution (Kumar et al., 2020; Fujimoto et al., 2018). As such, computing max Q for Bellman targets remains
issue in deep RL.
One popular approach is to train Maximum Entropy (MaxEnt) policies, in hopes that they are more robust modeling and estimation errors (Ziebart, 2010). However, the Bellman backup 5 used in MaxEnt RL algorithms still requires computing the log-partition function over Q-values, which is usually intractable in digh-dimensional action spaces. Instead, current methods like SAC (Haarnoja et al., 2018) rely on auxiliary policy networks, and as a result do not estimate B", the optimal Bellman backup. Our key insight is to apply extreme value analysis used in branches of Finance and Economics to Reinforcement Learning. Ultimately, this will allow us to directly model the LogSumExp over fuuuctions in the MaxEnt Framework.
"Equal Contribution "https://di github. 1o/x////

---

is Published as a conference paper at ICLR 2023
Intuitively, reward or utility agents will consider the maximum of the set of possible future returns. The Extreme Value Theorem (EVT) tells us that maximal values drawn from any exponential tailed distribution follows the Generalized Extreme Value (GEV) Type-1 distribution, also referred to as the Gumbel Distribution G. The Gumbel distribution is thus a prime candidate for modeling errors in Q-functions. In fact, McFadden's 2000 Nohel-prize winning work in Economics discrete choice models (McFadden, 1972) showed that soft-optimal utility functions with logit (or softmax) choice probabilities naturally arise when utilities are assumed have Gumbel-distributed errors. This was subsequently generalized to stochastic MDPs by Rust (1986). Nevertheless, these results have remained largely unknown in the RL community. By introducing a novel loss optimization framework, we bring them into the world of modern deep RL.
Empirically, we find that even modern deep RL approaches, for which errors are typically assumed to be Gaussian, exhibit errors that better approximate the Gumbel Distribution, see Figure 1. By assuming errors to be Gumbel distributed, we obtain Gumbel Regression, a consistent estimator over log- ong-pariiioon functions even in continuous spaces. Furthermore, making this assumption about Q- values lets us derive a new Bellman loss objective that directly solves for the optimal MaxEnt Bellman operator B instead of the operator under the current policy 8 As soft optimality emerges from framework, we can run MaxEnt RL independently of the policy. In the online setting, we avoid using a policy network explicitly compute entropies. In the offline setting, we completely avoid sampling from learned policy networks, minimizing the aforementioned extrapolation error. Our resulting algo- rithms surpass or consistently match sate--o-the-aas (SOTA) methods while being practically simpler.
In this paper we outline the theoretical motivation for using Gumbel distributions in reinforcement learning, and show how it can be used to derive practical online and offline MaxEnt RL algorithms. Concretely, our contributions are as follows:
We motivate Gumbel Regression and show it allows calculation of the log- lig-partition function (LogSumExpp) in continuous spaces. We apply it to MDPs to present a novel loss objective for RL using makimmum-likelhoodd estimation.
Our formulation extends soft-Q learning offline RL as well as continuous action spaces without the need of policy entropies. It allows us to compute optimal soft-values V and soft- fift-Bellman updates 5* using SGD, which are usually intractable in continuous settings.
We provide the missing theoretical link between soft and conservative Q-learning, showing how these formulations can be made equivalent. We also show how Max-Ent RL emerges naturally from vanilla RL as a conservatism our framework.
Finally, we empirically demonstrate strong results in Offline RL, improving over prior meth- ods by a large margin the D4RL Franka Kitchen tasks, and performing moderately better than SAC and TD3 in Online RI,, while theoretically avoiding actor-critic formulations.
2 PRELIMINARIES
In this section introduce Maximium Entropy (MaxEnt) RL and Extreme Value Theory (EVT), which we use to motivate our framework to estimate extremal values in RL.
We consider an infinite-horizon Markov decision process (MDP), defined by the tuple (S,A,77 P, 7). where S, A represent state and action spaces, P(s'|s, |s, a) represents the environment dynamics, r(s, a) represents the reward function, and 7 (0, 1) represents the discount factor. In the offline RL setting, we are given dataset D = (s, a, r, s') ) of tuples sampled from trajectories under a behavior policy TD without any additional environment interactions. We use P- (s) to denote the distribution of states that a policy π (als) generates. In the MaxEnt framework, an MDP with
referred as a soft-MDP (Bloem & Bambos, 20 and we often use this notation.
2.1 MAXIMUM ENTROPY RL
Standard RL seeks to learn a policy that maximizes the expected sum of (discounted) rewards E, 12 7/r(st, a,),, for (s,, drawn at timestep t from the trajectory distribution that generates. We consider a generalized version of Maximum Entropy RL that augments the standard reward objective with the KL-divergence between the policy and reference distribution Jc:
2

---

Published as a conference paper at ICLR 2023
E. 1E:=0 ("((se, au) Blg [(ay))) (x) )], where 3 is the regularization strength. When j is uniform u, this becomes the standard MaxEnt objective used in online RL up constant. In the offline RL setting, we choose je to be the behavior policy TD that generated the fixed dataset D. Consequently, this objective enforces a conservative KL-constraint on the learned policy, keeping close to the behavior policy (Neu et al., 2017; Haarnoja et al., 2018).
In MaxEnt RL, the soft- Bellmanl operator B* R-5AA + IR defined as (B*Q) a) r(s, a) y PP(-aa)())).aa (s') where Q is the soft-Q function and V* is the optimal soft-value satisfying:
(s) = 3 log Σ 1 u(aas)) exp (Q(s, a) () == L2 - [Q(ssa)]:
where we denote the log-sum-exp (LSE) using an operator & for succinctness". The soft-Bellman operator has a unique contraction Q (Haarnoja et al., 2018) given by the soft-Bellman equation: Q' = =B and the optimal policy satisfies (Haarnoja et al., 2017):
π (a|s) = m(als) exp ((Q" (s, a) - (3)))/B).
(2)
Instead of estimating saluevalues for a policy V=(s) = En Q(s. a) - -Blog ((ass)
[(a|s)] our
approach will seek directly fit the optimal soft-values V*, i.e. the log-sum-exp (LSE) of Q values.
2.2 EXTREME VALUE THEOREM
The Fisher- Fisher-Tippett or Extreme Value Theorem tells us that the maximum of i.i.d. samples from exponentially tailed distributions will asymptotically converge to the Gumbel distribution (n,, (8,, which has PDF p(x)= exp(-(= +)=*)) where z = (x - -w/3 3 with location parameter je and scale parameter
Theorem 1 (Extreme Value Theorem (EVT) (Mood, 1950; Fisher & Tippett, 1928)). For i.i.d. random variables Xn fx, with exponential tails, limm-xx max, (X.) follows the Gumbel (GEV-I) distribution. Furthermore, G is max-stable, iee if X, ~ G. then max; (X,) ~ G holds.
This result is similar to the Central Limit Theorem (CLT), which states that means of i.i.d. errors approach the normal distribution. Thus, under a chain of max operations. any i..d. exponential tailed errors will tend to become Gumbel distributed and stay as such. EVT will ultimately suggest us to characterize nested errors Q-learning as following a Gumbel distribution. In particular, the Gumbel distribution G exhibits unique properties we will exploit.
One intriguing consequence of the Gumbel's max-stability is its ability to convert the maximum over a discrete set into softmax. This is known the Gumbel-Max Trick (Papandreou & Yuille, 2010; Hazan & Jaakkola, 2012). Concretely for i.idd. e, ~ GO, 3) added to a set {..,.... max, (I, +e,) ~ G( log E. exp (z,/B), B). and argmax(z; + e ~ softmax( /). Furthermore, the Max-trick is unique to the Gumbel (Luce, 1977). These properties lead into the McFadden-Reel Rust state below.
e R,
model (McFadden, 1972; Rust, 1986) of MDPs
McFadden Rust model: An MDP following the standard Bellman equations with stochasticity in the rewards due to unobserved state variables will satisfy the noft-Bellman equations over the observed state with actual rewards F(s, a). given two conditions:
1. Additive separability (AS): observed rewards have additive i.i.d. Gumbel noise, i.e. r(s, a) = F(s, a) - c(s, a), with actual rewards F(s, a) and i.i.d. noise e(s, a) ~ G(, 3).
2. Conditional Independence (CI): the noise e(s, a) in given state-action pair is conditionally independent of that in any other state-action pair.
Moreover, the converse also holds: Any MDP satisfying the Bellman equations and following a softmax policy, necessarily has any i.i.d. noise in the rewards with AS + CI conditions be Gumbel distributed. These results were first shown to hold discrete choice theory by McFadden (1972), with the AS + CT conditions derived by Rust (1 (1986) for discrete MDPs. We formalize these results in Appendix A and give succinct proofs using the developed properties of the Gumbel distribution. These results enable the view of soft-MDP rewards. Notably, this result gives a different interpretation of a soft-MDP than entropy regularization to allow us recover the Beft-Bellman equations.
an MDP with hidden i..dd. Gumbel noise in the
IIn continuous action spaces, the sum over actions is replaced with an integral over the distribution j. "oundddd random variables sub-Gaussian (Young, 2020) which have exponential tails.
3

---

Published as a conference paper at ICLR 2023
3 EXTREME Q-LEARNING
In this section, we motivate our Extreme Qlearnning framework, which directly models the soft- optimal values V*, and show naturally extends soft-Q learning. Notably, the Gumbel distribution to derive new optimization framework for RL via maximum-likelihen estimation and apply it both online and offline settings.
3.1 GUMBEL ERROR MODEL
4.0
Gaussian
Although assuming Gumbel errors in MDPs leads to intriguing properties, it is not obvious why the errors might be distributed as such. First, we empirically investigate the distribution of Bellman errors by computing them over the course of training. Specifically, we compute r(s, a) rQ(s', m(s)) Q(s, a) for samples (s, a, s') from the replay-buffer using a single Q-function from SAC (Haarnoja et al., 2018) (See Appendix for more details). In Figure I, we find the errors to be skewed and better fit by Gumbel distribution. We explain this using EVT.
Normalized Frequency
3.6
Gumbel
3.0
255
2.0
15
1.0
0.
0.0
Bellman Error
Figure 1: Bellman errors from SAC on Cheetah Run (Tassa al., 2018). The Gum- distribution better captures the skew versus the Gaussian. Plots for TD3 and more environ- ments can be found in Appendix D.
Consider fitting Q-functions by learning unbiased function approximator Q solve the Bellman equation. We will assume access to M such function approxima- tors, each of which are assumed be independent e.g. parallel runs of model over an experiment. We can see approximate Q-iteration as performing:
(s. as (s,a).
where E[Q] = Q, is the expected value of our prediction Q, for an intended target Q over our estimators, and E, is the (zero-centered) error in our estimate. Here, we assume the error é, comes from the same underlying distribution for each of our estimators, and thus are i.i.d. random variables with a zero-mean. Now, consider the bootstrapped estimate using one of our M estimators chosen randomly:
B (s, = r(s, + max ('(s',a') = r-ss a) + max (Q(('', a') + ee((s'aa)) a')).
(4)
We now examine what happens after subsequent update. At time i + 1, suppose that we fit fresh set of M independent functional approximators Q(+1 with the target 1 Q,, introducing new unbiased error E++1. Then, for Q++1 [[Q++1] 1 it holds that
(s+1(s,)) = r(s. a) + 7B |s.a [.,, I [max(
a()])
(5)
As Q++1 is an expectation over both the dynamics and the functional errors, accounts for all uncertainty (here E[E[+1] 0). But, the i.i.d. error e, remains and will be propagated through the Bellman equations and its chain of max operations. Due to Theorem 1, 6z will become Gumbel distributed in the limit of f, and remain so due the Gumbel distribution's max-stability."
This highlights fundamental issue with approximation-oann based RL algorithms that minimize the Mean- Squared Error (MSE) in the Bellman Equation: they implicitly assume, via maximum likelihood estimation, that errors are Gaussian. In Appendix A, we further study the propagation of errors using the McFadden- -ust MDP model, and use develop a simplified Gumbel Error Model (GEM) for er- rors under functional approximation. In practice, the Gumbel nature of the errors may be weakened as estimators between timesteps share parameters and errors will correlated across states and actions.
3.2 GUMBEL REGRESSION
The goal of our work is directly model the log- sere-tatiionn function (LogSumExp) over Q a) to avoid all of the aforementioned issues with taking a max in the function approximation domain.
*The same holds for hDtt log-sum-exp he expanded over i.i.d. Gumbel random
4

---

Published as a conference paper at ICLR 2023
Gumbel PDF
Gumbel Loss
Gumbel Regression
beta-0.75 beta-1
bela=0,75
0.4
beta-1
b.ta=1.5
beta=1.5
0.3
beta-2
beta-2
0.2
beta-0.1
0.1
beta=0.25
beta-2
-15 -1.0 -3.
0000
0.5
1.0
-1.
X
Residual
Figure 2: Left: The pdf of the Gumbel distribution with μ = 0 and different values of 3. Center: Our Gumbel loss for different values of B. Right: Gumbel regression applied to wwo-dimensional random variable for different values
The smaller the value of B. the more the regression fits the extrema.
In this section we derive an objective function that models the LxpSumExp by simply assuming errors follow a Gumbel distribution. Consider estimating a parameter h for random variable X using samples z; from a dataset D, which have Gumbel distributed noise, i.c. I; = h + e, where & ~ G(,, (3). Then, the average log-likelihood of the dataset 'D as a function of h given as:
Ex [log pl (.:) = EI [-el(ss h)/9) + (Jr - h)/3
(6)
Maximizing the log-likelihoo yields the following convex minimization objective in h,
[(h) BaiDD (c(e--)/8 (#: 1)/3-1
which forms our objective function C(.), which resembles the Linex loss from econometrics (Parsian & Kirmani, 2002) P. ,8 is fixed as a pryper-parameter, and we show its affect on the loss in Figure 2. Critically, the minima of this objective under a fixed ,8 is given by h = 3 log E which resembles the LogSumExp with the summation replaced with an (empirical) expectation. In fact, this solution is the the same as the operator (X) defined for MaxEnt in Section 2.1 with x, sampled from J.. Figure 2, we show plots of Gumbel Regression on a simple dataset with different values of B. As this objective recovers L*(X), we next use it to model soft-values in Max-Ent RL.
3.2.1 THEORY
Here we show that Gumbel regression is well behaved, considering the previously defined operator L for random variables L L*(X)= == log E xx88 First, we show models the extremum.
Lemma 3.1. For any 3, > 32. we have (X) < <X). And (X) = E [X], (X)
sup(X). Thus. for any 8 E (0, oo), the operator L (X) is measure that interpolates between the expectation and the max of X.
The operator (X) known as the cumulant-generain function or the log-Laplace transform, and is a measure of the tail-risk closely linked to the entropic value at risk (EVaR) (Ahmadi- Javid, 2012).
Lemma 3.2. The risk measure L has unique minima at ,9 log E [e x/#] And an empirical risk is an unbiased estimate of the true risk. Furthermore, for B > 1, (()) x 2 EDll, 0)3), thus behaving as the MSE loss with errors NO. 3).
In particular, the empirical loss å over a dataset of N samples can be minimized using stochastic desseet (SGD) methods to give an unbiased estimate of the LogSumFxpp over the N samples.
Lemma 3.3. (X) over a finite N samples is consistent estimator of the log partition function (X). Similarly, exp(L((X)/3) is an unbiased estimator for the partition function Z E X10
We provide PAC learning bounds for Lemma 3.3, and further theoretical discussion on Gumbel Regression in Appendix B.
3.3 MAXENT RL WITHOUT ENTROPY
Given Gumbel Regression can be used to directly model the LogSumExp, we apply it to Q-learning. First, we connect our framework to conservative Q-learning (Kumar et al., 2020).
We add 1 make the loss 0 for a perfect fit, as e" 0 with equality 0.
5

---

Published as a conference paper at ICLR 2023
Lemma 3.4. Consider the loss objective over Q-functions:
[(Q) = Pemp,,,aullll (1.) a am (T Q(s, -1
(8)
where T r(s, a) + y (s Ba (Q a')] is the vanilla Bellman operator under the policy (als). Then minimizing L gives the update rule:
π a 8) ((a s)
Vs, a. k = - BBoo
1 (s,
The above lemma transforms the regular Bellman backup into the soft-Bellman backup without the need for entropies, letting convert standard RL into MaxEnt RL. Here, L(-) does conservative Q-update similar to CQL (Kumar et al., 2020) with the nice property that the implied conservative term is just the KL--constraint between and Jr." This enforces entropy-reggllarizatoo with respect to the behavior policy without the need of entropy. Thus, soft-Q learning naturally emerges conservative update regular Q-learning under objective. Here, Equation 8 is the dual of the KL-divergence between and (Garg et al., 2021), and we motivate this objective for RL and establish formal equivalence with conservative Q-learning in Appendix C.
our policy
In our framework, we use the MaxEnt Bellman operator 3 which gives our ExtremeQ loss, which is the same as our Gumbel loss from the previous section:
(()) Egga ((.))) Q(x.a))88 ((s.))))))))))) Egg (B Q(s.
This gives an update rule: Q++1 as) = B (s. a). (()) here requires estimation of 5 which is very hard continuous action spaces. Under deterministic dynamics, can be obtained without B* as shown in Appendix C. However, in general we still need to estimate B". Next, we motivate how we can solve this issue. Consider the soft-Bellman equation from Section 2.1 (Equation I),
Q = r(s, a) + ((s. [s)(')
(10)
where V*((s = LL L an (Q a)]. Then V* can be directly estimated using Gumbel regression by setting the temperature B to the regularization strength in the MaxEnt framework. This gives the following Extreme loss objective:
J(V) = EE (s.0 - Eg (( (s,* ((s))/
(11)
Lemma 3.5. Minimizing I over values gives the update rule: (*((s) = La
a 12t a)].
Then we can obtain V from Qss, a) using Gumbel regression and substitute in Equation 10 to estimate the optimal bellman backup Q. Thus, Lemma 3.4 and 3.5 give scheme to solve the Max-Ent RL problem without the need of entropy.
3.4 LEARNING POLICIES
In the above section we derived a Q-learning strategy that does not require explicit use of a policy However, in continuous settings we still often want recover a policy that can be in the environment. Per Eq. 2 (Section 2.2), the optimal MaxEnt policy л r*(ass) (QQ.)))) By minimizing the forward KL-divergence between я and the optimal x induced by Q and we obtain the following training objective:
argmax E (w.a))aaae/aaae. (cl log m].
(12)
If we take Pμ to be dataset D generated from behavior policy TD. we exactly recover the AWR objective used by prior works in Offline RL (Peng et al., 2019; Nair et al., 2020), which can easily be computed using the offline dataset. This objective does not require sampling actions, which may
'In fact, theorems of COL (Kumar al., 2020) hold for objective by replacing Deoz with DIZ-

---

Published as a conference paper at ICLR 2023
potentially take Q(s,a) out of distribution. Alternatiely,, if want to sample from the policy instead of the reference distribution 14, we can minimize the Reverse- KL divergence which gives us the SAC-like actor update:
π argmax E (s))ma(a)))) [Q(s. a) - (og(m(alss/uu(ls))
(13)
a
Interestingly, we note this doesn't depend on Vss). If is chosen to be the last policy The the second term becomes the KL-divergence between the current policy and The performing a trust region update on IT (Schulman et al., 2015; Vieillard et al., 2020).' While estimating the log ratio log(1 | (a)(s())))/ ((ass)(((sss)) can be difficult depending on choice of μ. our Gumbel Loss I removes the need for μ during Q learning by estimating soft-Q values of the form Q a) 3 log( ( (ag(((a)s)/u((as))
3.5 PRACTICAL ALGORITIMMS
In this section we develop a practical approach to Extreme Q-learning (X'-QL) for both online and of- fline RL. We consider parameterized functions (s),, Qo a), and Tt (ais) and let D be the training data distribution. A core issue with directly optimizing Eq. 10 is over-optimism about dynamies (Levine, 2018) when using simple-sample estimates for the Bellman backup. To overcome this issue in stochastic settings, we separate out the optimization of Va from that of Qo following Section 3.3. We learn Vo using Eq. 11 to directly fit the optimal soft-values V*(s) based on Gumbel regression. Using (s'))) we can get angle-sample estimates of 5 as r(s, a) + VV')s.. Now we can learn unbiased expectation over the dynamics, Qo x [ (ss, a) + ~V((s') by minimizing the Mean-squarred-error (MSE)
Algorithm Extreme Q-learning (X-QL)
(Under Stochastic Dynamics)
1: Init Qs. Vo, and Tig
2: Let D = {(ss, a.r,s') be data from (of-
fline) or replay buffer (online)
3: for step [I..NN] do
4: Train Qe using L(d) from Eq. 14
5: Train Va using [O) from Eq. 11 (with a ~ 'D (offline) or To (online))
6: Update के via Eq. 12 (offline) Eq. 13
(online)
7: end for
loss between the single-sample targets and QØ:
(()) = (_,.''')> [[Qo(s, a) r(s, a) - (ss)))
(14)
In deterministic dynamics, our approach largely simplified and we directly learn single Qa using Eq. 9 without needing to learn B* or V*. Similarly, we learn soft-optimal policies using Eq. 12 (offline) or Eq. 13 (online) settings.
Offline RL. In the offline setting. D is specified as an offline dataset assumed to be collected with the behavior policy TD. Here, learning values with Eq. 11 has a number of practical benefits. First, we are able to fit the optimal soft-values V without sampling from a policy network, which has been shown to cause large out-of-distribution errors the offline setting where mistakes cannot be corrected by collecting additional data. Second, we inherently enforce KL-constraint on the optimal policy x¹ and the behavior policy TD. This provides tunable conservatism via the temperature B. After offline training of Q₂ and Ve, we can recover the policy post-training using the AWR objective (Eq. 12). Our practical implementation follows the training style of Kostrikov et al. (2021), but we train value network using using our ExtremeQ loss.
Online RL. In the online setting, D is usually given as a replay buffer of previously sampled states and actions. In practice, however, obtaining good estimate of V (s') requires that we sample actions with high Q-values instead of uniform sampling from D. As online learning allows agents to correct over-optimism by collecting additional data, we use a previous version of the policy network T. to sample actions for the Bellman backup, amounting to the trust-region policy updates detailed at the end of Section 3.4. In practice, we modify SAC and TD3 with our formulation. To embue SAC (Haarnoja et al., 2018) with the benefits of Extreme Q-learning, we simply train Vo using Eq. 11 with s D, a Tap (als). This means that we do not use action probabilities when updating the value networks, unlike other MaxEnt RI. approaches. The policy is learned via the objective max [F[2as (s, (s))] with added entropy regularization, as SAC does not use a fixed noise schedule. TD3 by default does not
value network, and thus deterministic dynamics by changing the loss to train Q in TD3 to directly follow Eq. 9. The policy is leamedd as in SAC, except without entropy regularization TD3 uses fixed noise schedule.
our algorithm for
Choosing j to he uniform 24 gives the regular SAC update.
7

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
We compare our Extreme Q-Learning (X-QL) approach to state-of-the-art algorithms across a wide set of continuous control tasks in both online and offline settings. In practice, the exponential nature of the Gumbel regression poses difficult optimization challenges. We provide Offline results on Androit, details of loss implementation, ablations, and hyperparameters in Appendix D.
4.1 OFFLINE RL
Table 1:
Averaged normalized scores on MuJoCo locomotion and Ant Maze tasks, X'-QL-C gives results with the same consistent hyper- parameterssss in each domain, and X- X-QL-T gives results with per-environment and hyper- ppprr-parmetter tuning.
Dataset
BC
10%BC
DT
AWAC
Onestep RL.
TD
CQr.
IQL.
X-QL
X-QQI
malicheetalmmeduun-22
42.6
42.5
42.6
43.5
48.4
48.3
44.0
47.4
47.7
48.3
hupper medium v2
52.9
56.9
67.6
$7.0
59.6
59.3
58.5
66.3
71.1
74.2
75.3
75.0
74.0
72.4
81.8
83.7
72.5
78.3
81.5
84.2
Gyu
balfcheetah- maduueelaa-mmmdin--eel------
36.6
40.6
36.6
40.5
38.1
44.6
45.5
44.2
44,8
45.2
hoyper-mcdium--eploee
18.1
75,9
82.7
97.5
60.9
95.0
94.7
97.3
100.7
vallerr2--mddiun--ppaaa---
26.0
62.5
66.6
49.5
81.8
75.9
82.2
baltchaetulmmcddun--cprrerr
55.2
92.9
86,8
42.8
93,4
9U.7
91.6
86.7
89.8
94.2
52.5
110,9
107.6
55.8
103.3
98.0
105.4
91.5
107.1
111.2
107.5
109.0
108.1
74.5
113.0
110.1
108.8
109.6
110.1
112.7
untmaze umaze-vo .v
54.6
62.8
59.2
56.7
64.3
78.6
74.0
87.5
87.2
93.8
AntMaze
antmaze- sataazze-uzz----e diverse-vO
45.6
50.2
53.0
49.3
60.7
71.4
84.0
62.2
69.17
82.0
antanaze medium play-vD
0.0
73.5
antmazze medium diverse-vi vD
0.0
0.7
0.0
3.0
53.7
70.0
67.8
73.6
antmaze-large aarge-pla--vO
0.0
0.0
0.0
0.0
0.2
15.8
39.6
41
46.5
untmazee large diverse-vO
0.0
6.0
0.0
1.0
0.0
0.0
14.9
47.5
47.3
49.0
Frunka
kitchen complete yo
65.0
43.8
62.5
72.5
82.4
kitchen parrial-vO
38.0
49.8
46.3
73.8
73.7
kitchen-mixed-xx)
51.5
54.6
runtime
10m
10m
9600m
20m
20m
20m
80m
20m
10:20m
10:20m
We very fast convergenice for method
tasks, and saturate performance half the iterations IQL.
Our offline results with fixed hyperparameters for each domain outperform prior methods (Chen et al., 2021; Kumar et al., 2019; 2020; Kostrikov et al., 2021; Fujimoto & Gu, 2021) ) in several environments, reaching state-of-the-arl on the Franka Kitchen tasks, shown in Table 1. We find performance on the Gym locomotion tasks to be already largely saturated without introducing ensembles An al. (2021). but our method achieves consistently high performance across environments. While we attain good performance using fixed hyper-paramene parameters per domain, X'-QL achieves even higher absolute performance and faster convergence than IQL's reported results when hyper-parameters are turned per environment. With additional tuning, we also see particularly large improvements on the AntMaze tasks, which require a significant amount of "stitching" between trajectories (Kostrikoy et al., 2021). Full learning curves are in the Appendix. Like IQL, X-QL can be easily fine-tuned using online data to attain even higher performance as shown in Table 2.
4.2 ONLINE RL.
We compare ExtremeQ variants of SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018), denoted X-SAC and X-TD3, to their vanilla versions on tasks in the DM Control, shown in Figure 3. Across all tasks an ExtremeQ variant matches or surpasses the performance of basclines. We see particularly large gains in the Hopper environment, and more significant gains in comparison to TD3 overall. Consistent with SAC (Haarnoja et al., 201 we find the temperature 3 needs to be tuned for different environments with different reward scales and sparsity. core component of TD3 introduced by Fujimoto et al. (2018) is Double Q-Learning, which takes the minimum of two Q functions to remove overestimate bias in the Q-target. As we assume errors to be Gumbel distributed, we expect our --variants to be more robust to such errors. In all environments except Cheetah Run, our XDTD3 without the Double-Q trick, denoted X-QL DQ. performs better than standard TD3. While the gains from Extreme- learning are modest in online settings, none of our methods require access the policy distribution to learn the Q-values.
Table 2: Finetuning results on the AntMaze environments
Dataset
CQL
IQL
X-QLT
umaze v0
70.1
+ 99.4
86.7
- 96.0
93.8 +; 99.6
mmaze-diverse-vl
31.1
99.4
75.0
84.0
82.0 99.0
modium-play-vo
23.0
- 0.0
72.0
+ 95.0
76.0
- 99.00
madiuum-diverse-e
23.0
97.1
large play v0
1.0
- 0.0
25.5
4 46.0
45.1 59,3
divee-diverse-00
0.0
82.1
8

---

Published as a conference paper at ICLR 2023
Walker- Run
Hopper-Hop
Quadruped-RBe Run
Cheetah-
000
402
891
Rowaan
692
620
202
SAC
180
200
X-SAC
560
403
000
Rewazt
TD3
402
X-TD3
202
TD3
160
X-TD3- DQ
3.0
6.5
1.3
15
2.
3.0
0.5
1.0
1.3
2.0
0.0
3.5
1.2
1.5
2.3
0.0
0.5
1.2
1.5
Env Steps
166
Env Steps
Env Steps
1e6
Env Steps
106
Figure 3: Results on the DM Control for SAC and TD3 based versions of Extreme Q Learning.
RELATED WORK
Our approach builds on works online and offline RL. Here we review the most salient ones. Inspiration for our framework comes from econometrics (Rust, 1986; McFadden, 1972), and our Gumbel loss is motivated by IQ-Leerrn (Garg et al., 2021).
Online RL. Our work bridges the theoretical gap between RL and Max-Ent RL by introducing our Gumbel loss function. Unlike past work in MaxEnt RL (Haarnoja al., 2018; Eysenbach & Levine, 2020), our method does not require explicit entropy estimation and instead addresses the problem of obtaining soft-value estimates (LogSumExp) in high-dimensional or continuous spaces (Vieillard et al., 2021) by directly modeling them via our proposed Gumbel loss, which to our knowledge has not previously been used in RL. Our loss objective is intrinsically linked to the KL divergence, and similar objectives have been used for mutual information estimation (Poole et al., 2019) and statistical learning Parsian & Kirmani (2002); Atiyah et al. (2020). IQ-Learn (Garg et al., 2021) proposes learning Q-functions to solve imitation introduced the same loss in IL to obtain an unbiased dual form for the reverse KL-divergence between an expert and policy distribution. Other works have also used forward KL-divergence to derive policy objectives (Peng et al., 2019) or for regularization (Schulman et al., 2015; A Abdolmaleki et al., 2018). Prior work in RL has also examined using other types of loss functions "Bas-Serrano ease optimization (Asadi & Littman, 2017). Distinct from most off-Policy RL Methods (Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja al., 2018), we directly model B like Haarnoja et al. (2017); Heess et al. (2015) but attain significantly more stable results.
al., 2021) or other formulations of the argmax in order to
Offline RL. Prior works in offline RL can largely be categorized as relying on constrained or regularized Q-learning (Wu et al., 2019; Fujimoto & Gu, 2021; Fujimoto et al., 2019; Kumar et al., 2019; 2020; Nair et al., 2020), or extracting a greedy policy from the known behavior policy (Peng et al., 2019; Brandfonbrener et al., 2021; Chen et al., 2021). Most similar to work, IQL (Kostrikov et al., 2021) fits expectiles of the Q-function of the behavior policy, but is not motivated to solve particular problem or remain conservative. On the other hand, conservatism in CQL (Kumar et al., 2020) is motivated by lower-bounding the Q-function. Our method shares the best of both worlds like IQL we do not evaluate the Q-function on out of distribution actions and like CQL enjoy the benefits of conservatism. Compared to CQL, our approach uses KL constraint with the behavior policy, and for the first time extends soft-Q learning to offline RL without needing a policy or explicit entropy values. Our choice of using the reverse KL divergence for offline RL follows closely with BRAC (Wu et al., 2019) but avoids learning a policy during training.
CONCLUSION
We propose Extreme Q-Learning, new framework for MaxEnt RL that directly estimates the optimal Bellman backup 3 without relying on explicit access to a policy. Theoretically, we bridge the gap between the regular, soft, and conservative Q-learning formulations. Empirically, we show that our framework can be used to develop simple, performant RL algorithms. A number of future directions remain such as improving stability with training with the exponential Gumbel Loss function and inte- grating automatic tuning methods for temperature 3 like SAC (Haarnoja et al., 2018). Finally, we hope that our framework can find general use in Machine Learning for estimating ang-patttionn functions.

---

Published as a conference paper at ICLR 2023
Acknowledgements
Div derived the theory for Extreme Q-lcarning and Gumbel regression framework and ran the tuned offline RL experiments. Jocy the consistent offline experiments and online experiments. Both authors contributed equally to paper writing.
We thank John Schulman and Bo Dai for helpful discussions. Our research supported by NSF(1651565), AFOSR (FA95501910022) ARO (W911NF-21-1-012) ONR, CZ Biohub, and a Sloan Fellowship. Joey supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate (NDSEG) Fellowship Program.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. 9
A. Ahmadi-Javid. Entropic value-at-risk: A new coherent risk measure. Journal of Optimization The- ory and Applications, 155(3):1105-11223 2012. URL 100.111..000...0000 1007 (100000sss110010111111011111
org/
RePEc:spr: wotp::oopppww:::.y2222..
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified q-ensemble. In Neural Information Processing Systems, 2021. 8
Kavosh Asadi and Michael L Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, pp. 243 243-252. PMLR, 2017. 9
Israa Abdzaid Atiyah, Adel Mohammadpour, Narges Ahmadzadehgoli, and S Mahmoud Taheri. Fuzzy c-means clustering using asymmetric loss function. Journal of Statistical Theory and Applications, 19(1):91-101, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 160%.06600006550 2016. 21
Joan Bas-Serrano. Sebastian Curi, Andreas Krause, and Gergely Neu. Logistic q-learning. In International Conference on Artificial Intelligence and Statistics, pp. 3610-3618. PMLR, 2021.
M. Bloem and N. Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. 53rd IEEE Conference on Decision and Control, pp. 4911 1-196, 2014.
David Brandfonbrener,, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off- off-po evaluation. Advances in Neural Information Processing Systems, 11:4933-49444 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021. 8,
Benjamin Eysenbach and Sergey Levine. If maxent (rl} is the answer, what is the question?, 2020. URL https: / /ttp//peeeeveeeeeeeeeeeeeeeeeeee net/ forum? i 1d=SkxxZcNKDS.
R. A. Fisher and L. H. C. Tippett. Limiting forms of the frequency distribution of the largest or smallest member of sample. Mathematical Proceedings of the Cambridge Philosophical Society, 2402):180-190, 1928. doi: 10. 00.00177/033000001000000666
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforeement learning. Advances in Neural Information Processing Systems, 34, 2021. 8, 9
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ArXiv, abs/1802.094777. 2018. 9, 14, 20, 21
Scott Fujimoto, David Meger, and Doina Precup. Off Off-po deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062 PMLR, 2019. 9
10

---

Published as a conference paper at ICLR 2023
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Thirty- Fifth Conference on Neural Information Processing Systems, 2021. URL https: http::::///peeevieeeee.. wttp://eneeeetettn.ecc...m
xqtb5p. 6, 9, 18
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. 2017. 3, 9,
Tuomas Haarojj,, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870 PMLR, 2018. 1, 4, 7, 8,
Tamir Hazan and Tommi Jaakkola. On the partition function and random maximum a-posteriori perturbations. arXiv preprint arXiv:1206.644 6400, 2012.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap. Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. Advances in neural information processing systems, 28, 2015.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021. 8, 9,
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019, 8,9
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020. 1, 6, 8, 17, 18
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 1805500000,00009 2018.
Qing Li. Continuous control benchmark of deepmind control suite and mujoco. https: Com/LQNew LLQNe//Cottti http://ooooooooooooooooooeee Bearrrolleeehhaatiieeeee rk, 2021. 22
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,
David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint Xrv:v: 15t.:09999.99 2015. 9
R.Duncan Luce. The choice axiom after twenty years. Journal of Mathematical Psychol- ogy, 15(3):215-233, 1977. ISSN 0022-2496. doi: https://doi http:/bb..rr/111000000022---9779 90032-3, URL https: /www//wwwwooo..o//////.o/.../. sciencedireett.....ccc...ce com/ cctm///ciencee//ttiille///aaaa/e 0022249677900322 3, 14
Daniel McFadden. Conditional logit analysis of qualitative choice behavior. 1972. 2, 3, 13, 14
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Alexander McFarlane Mood. Introduction to the theory of statistics. 1950.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint urXiv:2006.09359 2020. 6, 6.9
Gergely Neu, Anders Jonsson, and V. Gómez. A unified view of entropy-rreularizee markov decision 2017.
processes. ArXiv,
George Papandreou and Alan L Yuille. Gaussian sampling by local perturbations. Advances in Neural Information Processing Systems, 23, 2010.
Ahmad Parsian and SNUA Kirmani. Estimation under linex loss function. In Handbook of applied econometrics and statistical inference, pp. 75-98. CRC Press, 2002. 5, 5.9
11

---

Published as a conference paper at ICLR 2023
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. errantage-weigheee regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 1rXiv:1910000117,22119 2019. 6,
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A. Alemi, and G. Tucker. On variational bounds of mutual information. In /CML, 2019. 9
John Rust. Structural estimation of markov decision processes. In R. F. Engle and D. McFadden (eds.), Handbook of Econometrics, volume 4, chapter 51, pp. 3081-3143, Elsevier, 1 edition, com/ jrust /truut/papera..te haah/hokkoooooe httpa///aaapesaaacooo.____e__
1986. URL ht https:/ /eeeeeitrrrleeppeeeeeeee rust. pdf. 2, 3,
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1897. PMLR, 2015. 1, 7,
Slavko Simić. On a new converse of jensen's inequality. Publications De L' L'institut Mathematique, 85:107-110, 01 2009. doi: 10.2298/PM009990077. 15
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. 4
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. 1999. 14
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer. Olivier Pietquin. Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in rl. 34th Conference on Neural Information Processing Systems, 2020.
Nino Vieillard, Marcin Andrychowicz, Anton Raichuk, Olivier Pietquin, and Matthieu Geist, Implic- itly regularized rl with implicit q values. arXiv preprint arXiv:2108.07041, 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv:191.111361, 2019. 9
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:// 2020. 20, 21,
gi github.e.. com wtt.a///we...a.attt/te.te..ooeee
G. Alastair Young. High-dimensional statistics: A non-asymptotic viewpoint, martin j. wainwright,
cambridge university press, 2019, xvii 552 pages. f57.99, hardback isbn: 978- 978-1-1084-9802-9 International Statistical Review, 88(1):258-22) 2613, 2020. doi: https://doi.or//10.111 wttp://...oo/....1////i/o... 12370. URL https::tpp.... htw//onnnnnnelllleeoeeeeeeeeenee librarr wiwiilllll wiley.ya. com/ doi /ab/ /10. 1et.l111ininsnii 12370.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal
entropy. Carnegie Mellon University, 2010. 1
12

---

Published as a conference paper at ICLR 2023
A THE GUMBEL ERROR MODEL FOR MDPs
In this section, we functionally analyze Q-learning using our framework and further develop the Gumbel Error Model (GEM) for MDPs.
A.1 RUST-MCFADDEN MODEL OF MDPs
For an MDP following the Bellman equations, we assume the observed rewards to be stochastic due to an unobserved component of the state. Let s be the observed state, and (s, z) be the actual state with hidden component Then,
((s, 2, a) = R(s, a) + + (-le.)) 1e (Vss,
(15)
((s, z) = max Q(s, z. a).
(16)
Lemma A.1. Given, 1) conditional independence (CI) assumption that z' depends only on s', i.e. p(s', //|s a) = R(s, 2) rss a) + (z.
a) and 2) additive separablity (AS) assumption on the hidden noise:
Then for i..dd c(z, a) 3), we recover the soft-Bellman equations for Q(s, z, a) = g(s, a) e(z, a) and v(s) [V (s, (s.z)). with rewards rss a) and entropy regularization 3.
Hence, a soft-MDP in MaxEmiRL is equivalent to an MDP with an extra hidden variable the state that introduces i..dd Gumbel noise in the rewards and follows the AS+ CI conditions.
Proof. We have,
((s, a) r(s, a) + + ( (V(',,
(17)
u(s) [V (s, z)] [a[[mxxq(ss.a + c())]
(18)
From this, we can get fixed-point equations for q and n,
a(s, a) r(s, a) + 7 [ [ a') + e a'))l, T (s)
(19)
Ez Jargmax(q(s a) c(z,a))) AA.
(20)
where A is the set of all policies.
Now, let e(z, a) (), and assumed independent for each (z, a) (or equivalently (s, a) due to the CI condition). Then we can use the Gumbel-Max trick to recover the soft-Bellman equations for q(s, a) and uss) with rewards r(s, a):
q(s, a) = r(s, a) + 7ll a [a(s', a(I]],
(21)
T ()) = softmax(q(ss a)).
(22)
Thus, we have that the soft-Bellman optimality equation and related optimal policy can arise either from the entropic regularization viewpoint from the Gumbel error viewpoint for an MDP.
Corollary A.1.1. Converse: An MDP following the Bellman optimality equation and having policy that is softmax distributed, necessarily has any ii.d. noise the rewards due to hidden state variables be Gumbel distributed, given the AS+CI conditions hold.
Proof. McFadden (McFadden, 1972) proved this converse in his seminal work on discrete choice theory, that for i.i.d. satisfyying Equation 19 with choice policy softmax has be Gumbel distributed. And we show a proof here similar to the original for MDPs.
Considering Equation 20, want T(a|s) s) be softmax distributed. Let have an unknown CDF F and we consider there to be N possible actions. Then,
P( P(argmax(g(s,, a) + e(z, a)) = a.|s, z) P(g(s, a.) + + a.) 2 q((ss az) + E(2, ay) Vi + j |s, z)
P(((2,a)) ((2,a;) {(,,a))) (s, Vi + j |s.
13

---

Published as a conference paper at ICLR 2023
Simplifying the notation, we write c(z. a.) and g(s, 9:.
Then 61, .., CN has joint CDF
G:
N II
G(EI = II
P(e S + - -9))
F(e; + qi - -4)
j=1
j=1
and we can get the required probability t(i) as:
(1)
II j=1.j
F(s + 9: ())
(23)
For softmax(g). McFadden (McFadden, 1972) proved the uniqueness of F to be the Gumbel CDF, assuming translation completeness property to hold for F. Later this uniqueness was shown to hold in general for any N (Luce, 1977).
A.2 GUMBEL ERROR MODEL (GEM) FOR MDPs
To develop our Gumbel Error Model (GEM) for MDPs under functional approximation in Sec- tion 3.1, we follow our simplified scheme of M independent estimators Q, which results in the following equation over Q E[Q]:
(s+ r(s, a) + 1, [maxx(
(')ss,a)))]].
(24)
Here, the maximum of random variables will generally be greater than the true max, i.c. E. (Q(s'' a²) + <(s', a())] 2 max,' Q(s', a') (Thrun & Schwartz, 1999). As result, even initially mean eгror can cause Q updates to propagate consistent overestimation bias through the Bellman equation. This known issue with function approximation in RL. (Fujimoto et al., 2018).
Now, we can use the Rust-McFadden model from before. To account for the stochasticity, we consider extra unobserved state variables = the MDP to the model parameters used in the functional approximation. The errors from functional approximation 6, can thus be considered as noise added in the reward. Here, CI condition holds as is separate from the dynamics and becomes conditionally independent for each state-action aatee-actinn pair and AS condition is implied. Then for Q satisfying Equation 24, we can apply the McFadden- Rust model, which implies that for the policy to be soft-optimal i.e. a softmax over Q, will be Gumbel distributed.
Conversely, for the i.i.d. e ~ g, Q(s, follows the soft-Bellman equations and <(als) = softmax(Q(ss. a)).
This indicates an optimality condition on the MDP for us to eventually attain the optimal softmax policy in the presence of functional boostrapping (Equation 24), the errors should follow the Gumbel distribution.
A.2.1 TIME EVOLUTION OF ERRORS IN MDPs UNDER DETERMINISTIC DYNAMICS
In this section, we characterize the time evolution errors an MDP using GEM. We assume deterministic dynamics to simplify our analysis.
suppose that we know the distribution of Q-values at time / and model the evolution of this distribution through the Bellman equations. Let (s, a) be random variable sampled from the distribution Q-values at time t, then the following Bellman equation holds:
Z,+1 (s, a) = r(s, + max Zz(s',a))
(25)
Here, (s+1(s, a) = max (r,,, a) + (S', , a')] is a maximal distribution and based on EVT should eventually converge to extreme value distribution, which
model as Gumbel.
Concretely, let's assume that we fix Zs a) ~ ((Q a), () for some Qs a) E RR and / > 0. Furthermore, assume that the Q-value distribution is jointly independent over different state- actions i.c. Z(s, a) is independent from (s'', a') for (s, a) + (s', a'). Then max, Z,(s', a') ~ ((V(s'), with (ss) LL[((ss a)] using the Gumbel-max trick.
14

---

Published as a conference paper at ICLR 2023
Then substituting in Equation 25 and rescaling Z with y. get:
1141 a) (((s. a) + ( 2/).
(26)
So very interestingly the Q-distribution becomes a Gumbel process, where the location parameter equation. Similarly, the temperature scales as 7/8 and the
Q(s, a) follows the optimal distribution becomes sharper after every timestep.
After number of timesteps, the unique contraction Q' (s. a). Here, controls the rate of decay of the Gumbel distribution into the collapsed Delta distribution. Thus we get the expected result in deterministic dynamics that the optimal Q-function will be deterministic and its distribution will be peaked.
that (ss, a) eventually collapses to the Delta distibution
So if a Gumbel error enters into the MDP through functional error some other source timestep t in some state s, it will trigger off an wave that propagates the Gumbel error into its child states following Equation 26. Thus, this Gumbel error process will decay at a y rate every timestep and eventually settle down with Q-values reaching the the steady solution Q" The variance of this Gumbel process given x² 32 will decay as x2, similarly the bias will decay y-contraction in the _ norm.
Hence, GEM gives us analytic characterization of error propogation in MDPs under deterministic dynamics.
Nevertheless under stochastic dynamics, characterization of errors using GEM becomes non-trivial as Gumbel is not mean-stable unlike the Gaussian distribution. We hypothesise that the errors will follow some mix of Gumbel-Gausee distributions, and leave this characterization as a future open direction.
B GUMBEL REGRESSION
We characterize the concentration bounds for Gumbel Regression in this section. First, we bound the bias applying L to inputs containing errors. Second, bound the PAC learning error due empirical f over finite N samples.
B. OVERESTIMATION BIAS
Let Q(s, a) be random variable representing Q-value estimate for state and action pair (s, a). We assume that it is unbiased estimate of the true Q-value Q(s, a) with E(Q(s, a)] Q(s, a). Let Q(s, a) E -mQazz Qmaz]
Then, (s) = T (s, a) is the true value function, and (s)) = T (. a) is its estimate.
Lemma B.1. We have (s) < [s) < Ee=j.l(ss, a)] + log cosh(Qmaa))
Proof. The lower bound (s) < [[(((s) (s) is easy to show using Jensen's Inequality as log_sum_erp a convex function.
For the upper bound, we can use a reverse Jensen's inequality (Simić, 2009) that for any convex mapping on the interval [a, holds that:
Σ Das (a:) S
Σ pitss + +(a)) +())
a (a+1) -
2
Setting / - log(.) and - @(s.a)18 we get:
(emual88 +e -maas!
Ea-al- log lar-l-log(e Q(s.a)/8))) < - llog(Baa ( (la 180xxe=0maz
-Qaaz
2
On simplifying.
(()) log( (og(a/n < Eur,)(((s,a) + log cosh(Q
15

---

Published as a conference paper at ICLR 2023
Taking expectations on both sides, [s) ((s,)
a)] + log maz This gives an estimate of how much the LogSumExp overestimates compared to taking the expectation over actions for random variables Q. This bias monotonically decreases with /3, with / = 0 having a max bias of Qma and for large /3 decaying J
B.2 PAC LEARNING BOUNDS FOR GUMBEL REGRESSION
Lemma B.2. ex)(L((X)/B) over finite N samples is an unbiased estimator for the partition
function 7 = [1x] and with a probability at least - 5 holds that:
2 log (1/6)
S 2 sinh((Xoor man/ 1B)
N
Similarly, (X) over a finite N samples consistent estimator of (X) and with a probability
at least - 5 it holds that:
(x) < (X) 8 8
2 log (1/6)
Z
Proof. To prove these concentration bounds, we consider random variables e X1/8 ".., xn/8 with B 0, such that a; - X; S by almost surely, i.. ea:/B s x:/8 s e/18
We consider the sum Sn = Ein x:/9 and use Hoeffding's inequality, so that for all <>> 0:
-212
PS, - ES, 2 t) S exp
(27)
Σ (ebi10 e0s/8)
To simplify, we let Oj = Xmma and b; = Xmax for all i. We also rescale as t = Ns, for >> 0.
Then
-Ns²
(S. ES, 2 No) S exp
(28)
2sinh²(X (Xmar/3)
We can notice that L.H.S. is same as (X)))( cxp((L((X/3)) 2 s), which is the required probability we want. Letting the R.H.S. have value ō, we get
2 log (1 /8)
sinh sinh((Xaa//)))
N
Thus, with probability 1 1 8, it holds that:
2 log (1/8)
< (xp (in(X) sinh(Xmaz /oxmm///))))))
(29)
N
Thus, we get concentration bound on exp((*(X)/3) which is an unbiased estimator of the partition function - exp(L (x)) This bound becomes tighter with increasing 3, and asymptotically (1/8)
behaves as
Similarly, to prove the bound on the log-partition function L (X). we can further take log(-) -) on both sides and use the inequality log(1 x) & to get direct concentration bound on (X)).
+ sinh (onh( -2 (X)/8 2 2og (1/5)
(x) [L(( + Bog
(30)
N
log (1/6) = =x)((x)= + Bsinh (X /aaaaa// 13)e ( (X>/8
(31)
N
-L*(X) siinh( Xmaz/33
log (1/5)
(32)
z3 N
16

---

Published as a conference paper at ICLR 2023
This bound also becomes tighter with increasing B, and asymptotically behaves Xnya / /2
C EXTREME Q-LEARNING
In this section provide additional theoretical details of our algorithm, A'-QI, and its connection conservatism CQL (Kumar et al., 2020).
C.I XLQL
For the noft-Bellman equation given as:
(s a) r(s, a) t(B V(s).
(33)
V(s) = 1 a));
(34)
we have the fixed-point characterization, that can be found with recurrence:
((s) ((-1s)) (r(s, a) P(ssa
(35)
In the main paper we discuss the case of X-QL under stochastic dynamics which requires the estimation of B". Under deterministic dynamic, however, this can be avoided as do not need account for an expectation over the next states, This simplifies the bellman equations. We develop two simple algorithms for this case without needing B'.
Value Iteration. We can write the value-iteration objective as:
Q(s, a) + r(s, a) + VV(ss))
(36)
((0) = (x-px.arp((1))
-
(37)
Here, we learn single model of the values (8(() to directly solve Equation 35. For the current value estimate (s(s), calculate targets r(s, a) + (s) and find estimate (i(ss by fitting L with our objective J. Using our Gumbel Regression framework, we can guarantee that as J finds a consistent estimate of the L and Ve(s) will converge the optimal ((() upto some sampling error.
Q-Iteration. Alternatively, we can develop Q-iteration objective solving the recurrence:
Qs+1(s,a) = r(s, a) + > [Q((',,a))
(38)
r(s. a) + + (7((ss'a))
(39)
2' +
(40)
where rescale 3 to 73 move L out.
This gives the objective:
Q'(s, a) r(s, a) + Qooos',a'),
(41)
a) I(Qs) Ex(x,,,'") [(0((ea) (
(42)
Thus, this gives method directly estimate Qo without learning values, and forms X-TD3 method in the main paper. Note, that 0 a hyperparameter, so we can use an alternative hyperparam- eter 3 = 7 to simplify the above.
We can formalize this as Lemma in the deterministic case:
Lemma C.1. Let
I (T,QQ Q') = =Esas' -'
[ (TT.Q(eaa)-Q((aa)//1 - (T, ('(s,a))/78-1 -1
17

---

Published as a conference paper at ICLR 2023
where Tu is linear operator that maps Q from current (s. a) the next (s', TaQ(s, r(s, a) + +
Then have B Q" argmin JTT - Q"), where Qt is the space of Q-functions.
QeERt
Proof. We use that in deterministic dynamics,
L a mup [T,Q(s, a)] = r(s, a) + " [()] 1" Q(sss
Then solving for the unique minima for J establishes the above results. Thus, optimizing I with a fixed-point is equivalent to Q-iteration with the Bellman operator.
C.2 BRIDGING SOFT AND CONSERVATIVE Q Q-LEARNING
Inherent Convervatism in -L Our method is inherently conservative similar to CQL (Ku- mar et al., 2020) in that it underestimates the value function (in vanilla Q-lcarning) (*) by -B (alx log salss) <D(a(s)
whereas CQL understimates values by factor
a(als) -1] where Tp is the behavior policy. Notice that the underestimation -B (ala())ss) (p(aas
factor transforms V/8 in vanilla Q-learnng into V used in the soft-Q learning formulation. Thus, observe that KL-regularized Q-learning is inherently conservative, and this conservatism is built into our method.
Furthermore, it can be noted that CQL conservatism can be derived as adding x² regularization to an MDP and although not shown by the original work (Kumar al., 2020) or any follow-ups to our awareness, the last term of Eq. 14 in CQL's Appendix B (Kumar et al., 2020), is simply x²(T] TD) and what the original work refers to as DeoL is actually the x² divergence. Thus, possible to show that all the results for CQL hold for our method by simply replacing DeQL with DKz i.e. the x² divergence with the KL divergence everywhere.
We show simple proof below that Deor is the x2 divergence:
(a
DEQL (T,, TD) (s)
Σ a(a s)
TD
Σ (a(a) (s) TD(a +DD(a s)) aaa S)
TD(a |s)
T(a s) (a (s)
aTa s) TT 8)
Σ (I(a)) |s) (a s))
Σ T s)
Tp(a ())
a(a 8) (a
Σ (a
+ 0 since,
xa Σ TD(a s)
=1
( [s)
TD(- |s)), using the definition of chi-square divergence
Why X'-LL is better than CQL for offline RL In light the above results, we know that CQL adds x" regularization the policy æ with respect the behavior policy np, whereas our method does the same using the reverse-KL. divergence.
Now, the Ke divergence has mode-seeking behavior, and thus method will find policy that better fits the mode of the behavior policy and is more robust to random actions in the offline dataset. CQL does not have such property and can be easily affected by noisy actions in the dataset.
Connection to Dual KL representation For given distributions and æ,, we can write their div-eieeeee using the dual representation proposed by IQ-Learn (Garg et al., 2021):
(KL(T p))
1u Ex [x]
1,
#ER
18

---

Published as a conference paper at ICLR 2023
which is maximized for log(T//L))
We make clever substitution to exploit the above relationship. Let (Q -0Q-77 - for
variable Q E R and fixed constant T=Q², then on variable substitution we get the equation:
Ba [D (T((ss) pl s)) min L(Q), with
Q
[(Q) Par a))((1))
[e (mT)0((aa)-(xx1))))
- ((T (s. Q(s,
This gives Equation 8 in Section 3.3 of the main paper, and is minimized for Q = T (og(sr/1)) as we desire. Thus, this lets us transform the regular Bellman update into the soft-Bellman update.
D EXPERIMENTS
In this section we provide additional results and more details on all experimental procedures.
D.1 A ToY EXAMPLE
MSE Loss
XQL Beta =055
XQL Loss, Beta =0.1=0.1
Max
Figure 4: Here show the effect of using different ways of fitting the value function on toy grid world, where the agents goal is to navigate from the beginning the maze on the bottom left to the end of the maze the top left. The color of cach square shows the learned value. As the environment is discrete, we can investigate how well Gumbel Regression fits the maximum of the Q-values. As seen, when MSE loss is used instead of Gumbel regression, the resulting policy is poor at the beginning and the learned values fail to propagate. As we increase the value of beta, we see that the learned values begin better approximate the optimal max policy shown the very right.
D.2 BELLMAN ERROR PLOTS
Cheetah Run
Walker Run
Hopper Hop
Gaussian Gumbel
Guussian
Gaussian Gumbel
Noriallted Prequency
Gambel
2.8
2.5
0.
1.5
-8.2
0.5
1.2
6:1-4.76
-0.04
8.00
0.02
0.55
Bellman Error
Bellman Error
Bellman Error
Figure 5: Additional plots of the error distributions of SAC for different environments. We find that the Gumbel distribution strongly fit emors first two environments, Cheetah and Walker, but provides worse fit in the Hopper environment. Nonetheless, we see performance improvements in Hopper using our approach.
19

---

Published as a conference paper at ICLR 2023
Cheetah Run
Walker Run
Hopper Hop
Normalizet Frequeary
Gaussian Gumbel
160
Geussian
Gumbel
Gumbel
140
130
2.3
1.4
-12
0.5
8-2
6.5
0.5
-22
0.0
9.2
0.4
2.6
2.5
00806 -00% -0,00 0.500 60Z 2006
asre
Bellman Error
Bellman Error
Bellman Error
Figure 6: Plots of the error distributions of TD3 for different environments.
Additional plots of the enror distributions for SAC and TD3 can be found in Figure 5 and Figure 6, respectively. Figure 1 and the aforementioned plots were generated by running RI. algorithms for 100,000 timesteps and logging the bellman errors every 5,000 steps. In particular, the Bellman errors were computed as:
r(s, a) + 70, (s', TT Qe, (s.a) a)
In the above equation Qo. represents the first of the two networks used in the Double Q trick. We do not use target networks to compute the bellman error, and instead compute the fully online quantity. (s') represents the mean deterministic output of the current policy distribution. We used an implementation of SAC based on Yarats & Kostrikov (2020) and an implementation of TD3 based on Fujimoto al. (2018). For SAC we did the entropy term was not added when computing the error as we seek to characterize the standard bellman error and not the soft- bellman error. Before generating plots the errors were clipped the ranges shown. This tended prevented over-fitting to large outliers. The Gumbel and Gaussian curves we fit using MLE via Scipy.
D.3 NUMERIC STABILITY
In practice, naive implementation of the Gumbel loss function I from Equation 11 suffers from stability issues due to the exponential term. We found that stabilizing the loss objective was essential for training. Practically, we follow the common normaalaalziiionn trick used in softmax computation. This amounts to factoring out e emnax, from the loss and consequently scaling the gradients. This adds a per-batch adaptive normalization the learning rate. We additionally clip loss inputs that are too large to prevent outliers. An example code snippet in Pytorch is included below:
def gumbel loss (pred, label, beta, clip):
(label pred) /beta
z torch. torch.clamp (z, -clip, clip)
max_2 = torch.max (2)
max_2 = torch. where (max_2 < -1.0, torch.tensod tensor (-1.0), max_2)
torch.where
max_z max_z. detach)) Detach the gradients
loss terch, (z max_z) zstorch. exp (-max_z) torch.exp (-nax_z) return loss. mean
In some experiments we additionally clip the value of the gradients for stability.
D.4 OFFLINE EXPERIMENTS
In this subsection, we provide additional results in the offline setting and hyper-parameter and implementation details.
Table 3 shows results for the Androit benchmark in D4RL. Again, see strong results for X-QL, where X XL-L--C with the same hyperparameters as used in the Franka Kitchen environments surpasses prior works five of the eight tasks. Figure 7 shows learning curves which include baseline methods. We see that XLL exhibits extremely fast convergence, particularly when tuned. One issue however, is numerical stability. The untuned version of X --Q exhibits divergence the Antmaze environment.
We base our implementation of X-QL off the official implementation of IQL from Kostrikov et al. (2021). We use the same network architecture and also apply the Double-Q trick. We also apply the
20

---

Published as a conference paper at ICLR 2023
Table Evaluation on Adroit tasks from D4RL. X' Y-QL-C gives results same hyper parameters used
the Franka Kitchen as IQL, and X' X-QL-T gives results with per-environment 3 and
tuning.
Datnset
BC
BRAC-p
BEAR
Onestep RL
CQL
IQL
X-QL C
X-QL
pen-human-vo
63.9
8.1
-1.0
37.3
71.3
85.5
hammer- buman-vO
1.2
0.3
0.3
4.4
1.4
2.2
8.2
door human-v0 v0
00.3
-0.3
9.9
4.3
11.5
11.5
Blocate-human-vd
-0.3
-0.3
0.2
0.1
0.17
0.24
pen-eloned-vO
37
1.6
26.5
60.0
39.2
37.3
38.6
53.9
hammer-cloond-x))
2.1
doer cloned vt
0.0
-0.1
-0.1
0.4
0.4
1.6
5,9
5.9
relocat-cloned-vo)
-0.3
-0.3
-0.3
-0.1
-0.1
-0.2
-0.2
-0.2
hopper-mediune expeett
aotmaze-umazee
kitchen-partial
120
100
80
100
60
Reward
80
Reward
60
Reward
60
40
40
IQL
20
20
XQL
20
XQL Consistent
0.0
0.2
0.4
0.6
0.8
L.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
Steps
1e6
Steps
Figure 7: Offline RL Results. We show the returns number of training iterations for the D4RL benchmark, averaged over 6 seeds. For fair comparison, we hatch size of 1024 for each method. XQL Tuned tunes the temperature for each environment, whereas XQL consistent uses default temperature.
same data preprocessing which is described in their appendix. We additionally take their baseline results and use them in Table 1, Table 2, and Table 3 for accurate comparison.
We keep our general algorithm hypee--assmeters and evaluation procedure the same but tune 3 and the gradient clipping value for each environment. Tuning values of /3 was done via typer-parameter sweeps over fixed set of values [0.6, 0.8, 1, 2| 5] for offline save for a few environments where larger values were clearly better. Increasing the batch size tended also help with stability, since our rescaled loss does per-batch normalization. AWAC parameters were left identical to those in IQL. For MuJoCo locomotion tasks we average mean returns over 10 evaluation trajectories and random seeds. For the AntMaze tasks, we average over 1000 evaluation trajectories. We don't 't see stability issues the mujoco locomotion environments, but found that offline runs for the AntMaze environments could occasionally exhibit divergence in training for a small 3 < 1. In order to help mitigate this, we found adding Layer Normalization (Ba et al., 2016) to the Value networks to work well. Full hyper-parametert we used for experiments are given in Table 4.
D.5 OFFLINE ABLATIONS
In this section show hyper-parameter ablations for the offline experiments. In particular, ablate the temperature parameter, 3, and the batch size. The temperature controls the strength of KL penalization between the learned policy and the dataset behavior policy, and small 3 is beneficial for datasets with lots of random noisy actions, whereas a high 3 favors more expert-like datasets.
Because our implementation of the Gumbel regression loss normalizes gradients the batch level, larger batches tended to be more stable and in some environments lead to higher final performance. To show that our tuned X'QL method is not simply better than IQL due to bigger batch sizes, we show comparison with fixed batch size of 1024 in Fig. 7.
D.6 ONLINE EXPERIMENTS
We base our implementation of SAC off pytorch_sav (Yarats & Kostrikov, 2020) but modify it to use a Value function as described in Haarnoja et al. (2017). Empirically we see similar performance with and without using the value function, but leave it in for fair comparison against our X-SAC variant. We base our implementation of TD3 on the original author's 's code from Fujimoto et al. (2018). Like in offline experiments. hyper- paremeerssstesss were left as default except for B, which we tuned for each environment. For online experiments we swept over [1, 2, 5] for -'C and TD3. We found that these values did not work well for TD3 DQ, and swept values [3, 4, 10, 20). In online
21

---

Published as a conference paper at ICLR 2023
walker/d 1ateer22---dddu--epppaaa
aoltmaze-large-diveee
kitcheo-main miieed
60
00
60
50
Rewart
an
4.0
20
60
20
Beta 0.4
Beta
Bota
Beta 0.6
20
Beta 2
10
Deta
Beta 1.0
Beta 8
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.6
0.0
0.2
0.4
0.5
0.8
1.0
Steps
Steps
Steps
Figure 8: Ablation. large temperature and performance drops. When / is too small, the loss becomes sensitive to noisy outliers, and training can diverge. Some environments are more sensitive to than others.
balfcheetah medium
halfcheelah medium expert
animaze umazz divere-
46
40
Reward
Reward
00
25
80
Batch 1024
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Strps
Steps
Steps
Figure 9; Batch Size Ablation. Larger batch sizes can make Gumbel regression more stable,
experiments used an exponential clip value of For SAC ran three seeds in each environment as it tended to be more stable. For TD3 we ran four. Occasionally, our X variants would experience instability due outliers in collected online policy rollouts causing exploding loss terms. We see this primarily in the Hopper and Quadruped environments, and rarely for Cheetah or Walker. For Hopper and Quadruped, we found that approximately one six runs became unstable after about 100k gradient steps. This sort of instability is also common in other online RI. algorithms like PPO due to noisy online policy collection. We restarted runs that become unstable during training. We verified our SAC results by comparing to Yarats & Kostrikov (2020) and our TD3 results by comparing Li (2021). We found that our TD3 implementation performed marginally better overall.
22

---

Published as a conference paper at ICLR 2023
Table Offline RL Hyperparameters used for X QL. The first values given are non per-environment tuned version of X QL, and the values in parenthesis are for the tuned offline results, A' QL-T. V-updates gives the number of value updates per Q update, and increasing it reduces the variance of value updates using Gumbel loss on some hard environments.
Beta
Grad Clip
Batch Size
uuudaates
hallcheetah medium v2
(1)
7(7)
256 (256)
(1)
hopper-mediuns walker2d-medium--
7
(256)
2 210)
(())
256 (256)
(I)
halfcheetahh medium-reeplay--22
(11)
7(5)
256 (256)
(I)
hopper medium replay-v2
2(2)
7(7))
256 (256)
walker22-medium-epppe medium--repla---
2())
7
256 (256)
(1)
hellcheetale medium expert
2(1)
7(5)
(1024)
hopper medium expert- v2
2
7(7)
256 (1024)
(1)
walker2d-medium www-mmdiun-eeper--eee
2(2)
7
256 (1024)
(1)
antmaze- -umazzz-maz--v
0.6
7(7)
256 (256)
(1)
antmaze umaze umazz-diverse--o diverse-vO mediumm play
(.6
7(7)
256 (256)
(1)
(08)
7(7)
256 (1024)
(0.6)
7(7)
256 (256)
(4)
animaze lurge play vO
(0.6)
7
(1024)
0.6 (0.6)
(
256 (1024)
(1)
kitchan-copplote-0
(2)
(())
256 (1024)
(1)
kitchen partial-e0
(
((7)
256 (1024)
kitchen mixed vO
5 (8)
((7)
256 (1024)
(1)
human-v0
(
7(7)
256 (256)
hammer human-v0
5 (0.5)
7(3)
256 (1024)
(4)
doer human-uha
(())
7
(256)
(1)
human--
500.8)
7
256 (1024)
(2)
pea-cloned-vO
(0.88)
7(5)
256 (1024)
(2)
hammer- cloned-v0
5
7(7)
256 (256)
1()
door human relocste-human-v0
(()
(
256 (256)
1())
(()
7())
256 (256)
1())
Table 5: Hyperparameters for online RL Algorithms
Parameter
SAC
TD3
Batch Size
1024
256
Leamning Rate
0.0001
0.001
Critic Froq
1
Freq
Actor and Critic Arch
1024. 1024
256. 256
Buffer Size
1,000,000
1,000,000
Actor Noise
Auto-tuned
0.1, 0.05 (Hopper)
Target Noise
0.2
Table 6: Values of temperature 3 used for online experiments
X'SAC
R-TD33
-DT
Cheetah Run
Walker Run
Hopper Hop
Qundruped Run
5
20
23

---

Published as a conference paper at ICLR 2023
moalichentah-mediiee
balfchentah- w-w.ahh-eeedmm-moomm replay
halfchnetah hud expert
60
80
Roward
20
60
20
60
20
XQL Tuned
10
20
XQL Consiatent
1.0
0.8
1.0
Steps
le6
Steps
1e6
Steps
166
hopper-mediue
hopper-medium-rege
hopper medium expert
100
70
100
Roward
60
Reward
Reward
80
60
0.0
40
20
20
0.8
1.0
Steps
166
Steps
106
Steps
166
walker2d medium
walkerrd modium replay
walker2d medium expert
86
100
60
80
Reward
60
Roward
Reward
80
80
40
20
20
1.0
Steps
166
Steps
166
Steps
166
Figure 10: Offline Mujoco Results. We show the returns number of training iterations for the mujoco benchmarks in D4RL (Averaged over 6 seeds). X-QL Tuned gives results after hyper- parameter tuning to reduce run variance for each environment, and X-QL consistent uses the same hyper- srpea-pprameters for every environment.
antmaze-mednne modium play
80
80
Roward
Roward
8
Reward
40
20
XOL Tuxed
20
XOL Consistent
Steps
lc6
Steps
1et
Steps
1e6
medium diverse
antmaze-large
antmazee large-diverse
80
40
Roward
30
Rewward
4.0
40
20
0
Steps
1c0
Steps
1eB
Steps
1e6
Figure 11: Offline AntMaze Results. We show the returns number of training iterations for the antmaze benchmarks in D4RL (Averaged 6 seeds). X--Q Tuned gives results after hyper- parameter tuning to reduce run variance for each environment, and XQI consistent uses the same hyper-parameters for every environment.
24

---

Published as a conference paper at ICLR 2023
Kilchen-compiete
kitchen-partal
kitchen-miixed
80
60
80
60
Rewnrd
Rewartt
Reward
80
60
20
19
XQL Consistent
0.2
0.4
0.5
0.0
6.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Steps
Steps
Steps
Figure 12: Offline Franka Results. We show the returns vs number of training iterations for the Franka Kitchen benchmarks D4RL (Averaged over sceds). X-QQ Tuned gives results after nyper-parameter tuning to reduce run variance for cach environment, and X-QL consistent uses the same hyper-parameter for every environment.
relscate
XOL Tuand
XOL
Roward
50
9.2
3.0
3.8
0.2
0.6
1.0
6.0
0.2
3.5
1.3
3.0
3.
Steps
pen-clozed
-0.25
-2.2
3.0
2.5
6.2
0.2
3.0
1.5
Offline Androit Results. We show the returns numher of training iterations for the Androit benchmark in D4RI. (Averaged over seeds). X-Q Tuned gives results after hyper-parameter tuning to reduce run variance for each environment, and X'-QI. consistent uses the same hypar-parameters for every environment. On some environments the "consistent" hyperparameters did best.
Figure 13:
25


Reviews:
[{"review_id": "sajIgF32PQ", "reviewer": "Reviewer_pDUA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new Q-learning framework by formulating the TD-error as a Gumbel distribution rather than a Gaussian. This new formulation leads to MaxEnt-RL style RL algorithms, but without the need to sample from out-of-distribution examples. The proposed method shows fairly well performance on standard D4RL benchmarks.", "summary_of_the_review": "This paper provides a new RL framework by modeling the TD-error as a Gumbel distribution, which allows avoiding sampling OOD datapoints while enjoying the advantages of MaxEnt RL. However, the experiments section seems to be misleading in several aspects.", "strength_and_weaknesses": "Strengths\n- Incorporates the MaxEnt RL framework while avoiding the major problem of offline RL (extrapolation error from referring to OOD examples).\n- Modeling TD-error as a Gumbel distribution seems to be more appropriate compared to modeling it as a Gaussian.\n\nWeaknesses\n- On offline RL, the proposed method still requires double-Q learning technique even though the method is expected to be more robust to Q-value overestimation.\n- The experiments table for offline RL (Table 1) is misleading in several aspects:\n\nFirst, the hyperparameters for the proposed method are tuned dataset-wise (refer to Table 4). However, previous works such as CQL or IQL keep their hyperparameters the same at least for each environment. For example, IQL fixes its hyperparameters the same for all MuJoCo locomotion tasks. Thus, the table results are not a fair comparison. Since the paper does not provide any hyperparameter sensitivity results, it is hard to conclude that the proposed method is actually the new \"state-of-the-art\" as the authors argue. Also, even if dataset-wise hyperparameter tuning is freely allowed, recent works seem to show higher performance on several datasets [1, 2].\n\nSecond, the runtime comparison also seems to be misleading. The authors note that the proposed method converges faster than IQL (Figure 6) and runs half the epochs compared IQL. However, based on the source code in the supplementary material, it seems that the proposed method uses a much larger batch size (1024) compared to IQL (256) [3]. Are the authors increasing the batch size and claiming the proposed method converges with much fewer iterations?\n\nQuestions\n- On Lemma 3.4, why is the first expectation over \\mu while the second expectation is over \\pi?\n- On Figure 3, what does -DQ on TD3 mean?\n\n[1] An et al., Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble, NeurIPS 2021.\n\n[2] Cheng at al., Adversarially Trained Actor Critic for Offline Reinforcement Learning, ICML 2022.\n\n[3] https://github.com/ikostrikov/implicit_q_learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZACI593L9u", "reviewer": "Reviewer_PnpD", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Gumbel-regression as an alternative to mean squared error regression for value functions in reinforcement learning. The use of Gumbel distribution is motivated from established theory and empirical observations. The resulting algorithm outperforms state-of-the-art in both online and offline RL benchmarks.", "summary_of_the_review": "No major weaknesses as such, enjoyed reading this paper. It will help to clarify why tuning Beta on each environment does not lead to over-fitting. ", "strength_and_weaknesses": "Strengths:\n- The paper motivates their algorithm based on theoretical foundations and provides Lemmas to support their final loss functions.\n- The paper is well-written and easy to follow.\n- Empirical results against strong baselines show significant improvement\n\nWeaknesses:\n- As far as I can tell, hyper-parameter tuning is done on the test set, there is no separate validation set. \n- It will be great to have this algorithm available as open source. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SZQVC9gzx4T", "reviewer": "Reviewer_HuwQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes an interesting observation that the error after the bellman optimality operation, instead of following gaussian noise, should in theory follow Gumbel distribution. This motivates the usage of Gumbel regression instead of least square to learn the Q functions, and the paper shows its connection to CQL in the policy evaluation regime. The paper also shows in practice one can introduce another V function and policy to deal with continuous action space, and the experiments show that the practical algorithm achieves promising results in both online and offline benchmarks.", "summary_of_the_review": "Overall, this paper makes a very interesting observation and provides a new perspective on how we should perform Q learning. The method is well motivated, the paper is well-written, and the experiment results seem solid. Although there is still some remaining confusion, I still would like to recommend an acceptance for the paper. ", "strength_and_weaknesses": "# Strength\n* The paper is very well organized, with sufficient background introduction so that readers without much previous knowledge could also understand the context. \n* The usage of Gumbel regression is well organized, with good theoretical support.\n* The paper also makes a good connection and shows how the minimizer of the Gumbel regression objective recovers previous update rules that require explicitly policy action distribution knowledge such as SAC or CQL, while XQL does not require the access to the policy during the value function update. \n* The practical algorithm shows very promising improvement over previous methods, especially in the offline benchmarks. The practical algorithm seems also easy to be built on previous online methods. \n# Weakness\n* The motivation for using Gumbel regression seems not very obvious in the continuous action region because it's not obvious how to take the max operator. I am also confused about the presentation of the practical motivation (such as Fig.1) for two reasons: \n1. The bellman error is recorded during the training. However, some value functions we learned during the training may not even represent a valid value function for any policy? Then what does this bellman error represent?\n2. The bellman error is actually not calculated with max, but with the action that the corresponding policy takes. Again I understand this is due to the difficulty from the continuous action space, but this seems to deviate from the theoretical motivation.\n* Although the paper claims that not requiring the access to the policy during value function update is a merit of the proposed algorithm, I could not see why this is significant in practice: in practice, we still need to train a policy anyway so we could assume we always have the access to the policy information? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FIJJ9sm73S", "reviewer": "Reviewer_XF2J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a new class of Q-learning-based algorithm for both online and offline RL for continuous control tasks based on extreme value theory. By using the insight that the maximum of i.i.d. random variables with exponential tails has a Gumbel distribution, the authors derived a new set of update rules which are equivalent to using soft Bellman backups but do not involve the use of entropies. The effectiveness of the algorithm is demonstrated in both online and offline settings using a set of standard benchmarks.", "summary_of_the_review": "Overall, I think the community has a lot to gain from the ideas introduced in this paper, which is why I am recommending this paper for acceptance. However having said that, I do think the authors could have done a better job in justifying why \"without entropy\" is better and I hope the authors can offer some additional insight in the discussion period.", "strength_and_weaknesses": "I think this paper brings forth some very interesting ideas to the table. Using Gumbel regression to model Q updates is to the best of my knowledge quite novel and in my opinion definitely something worth further exploration. The paper overall was quite easy to read and the authors did a very good job of justifying most claims and the technical choices involved in the algorithm. Some of my comments on some specific points in the paper:\n- Intuitively I agree with the authors in that I think this EVT-based approach would work quite well in an offline setting (and experiment results do support this) since it naturally introduces conservatism into the algorithm. However I would definitely like to understand better where the source of improvement comes from and especially some additional comparisons with other algorithms in this regard, this could come in the form of a toy example that gives easier visualization. \n- For the online setting, I'm not fully convinced why this approach would be better than an algorithm like SAC. Performance for online RL do not seem to offer much improvements. The environments used in the paper are generally considered to be fairly easy environments and the number of random seeds used is very small (3 or 4). Also unlike PPO, numerical instability isn't usually a huge issue for vanilla TD3/SAC for these particular environments.\n- Adding to the above point, it does seem that numerical instability is quite a major issue from a practical perspective, though the authors have introduced ways to mitigate this, I feel that this could still be a huge obstacle to wide practical adoption of this approach.\n- One thing you mentioned in the conclusion section about potential future directions is to integrate automatic parameter tuning for the temperature. Could you elaborate on some of the challenges for doing this and why using the mechanism for this introduced in [1] would be difficult?\n\n[1] Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" arXiv preprint arXiv:1812.05905 (2018).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%