Robust Losses for Learning Value Functions
Authors: Andrew Patterson
Victor Liao
Martha White
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning

Abstract: Most value function learning algorithms in reinforcement learning are based on the mean squared (projected) Bellman error. However, squared errors are known to be sensitive to outliers, both skewing the solution of the objective and resulting in high-magnitude and high-variance gradients. Typical strategies to control these high-magnitude updates in RL involve clipping gradients, clipping rewards, rescaling rewards, and clipping errors. Clipping errors is related to using robust losses, like the Huber loss, but as yet no work explicitly formalizes and derives value learning algorithms with robust losses. In this work, we build on recent insights reformulating squared Bellman errors as a saddlepoint optimization problem, and propose a saddlepoint reformulation for a Huber Bellman error and Absolute Bellman error. We show that the resulting solutions have significantly lower error for certain problems and are otherwise comparable, in terms of both absolute and squared value error. We show that the resulting gradient-based algorithms are more robust, for both prediction and control, with less stepsize sensitivity.

Under review as a conference paper at ICLR 2022
ROBUST LOSSES FOR LEARNING VALUE FUNCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Most value function learning algorithms in reinforcement learning are based on the mean squared (projected) Bellman enror. However, squared errors known to be sensitive to outliers, both skewing the solution of the objective and resulting in high- magnitude and high-variance gradients. Typical strategies to control these high- magnitude updates in RL involve clipping gradients, clipping rewards, rescaling rewards, and clipping errors. Clipping errors is related to using robust losses, like the Huber loss, but as yet no work explicitly formalizes and derives value learning algorithms with robust losses. In this work, build on recent insights reformulating squared Bellman errors as saddlepoint optimization problem, and propose saddlepoint reformulation for Huber Bellman error and Absolute Bellman error. We show that the resulting solutions have significantly lower error for certain problems and are otherwise comparable, in terms of both absolute and squared value error. We show that the resulting gradient- paadeett-based algorithms are more robust, for both prediction and control, with less stepsize sensitivity.
INTRODUCTION
Learning value functions from off-policy data remains an open challenge due to high-variance samples and the inability to optimize the objective of interest, Progress towards this goal has been made over years of algorithm development, by reducing the variance for temporal difference (TD) algorithms (Precup et al. 2000, Munos et al., 20166] Mahmood al. 2017): following approximate gradients of
proxy objective the mean squared Bellman error (MSBE) which upper bounds our objective of interest (Dai et al.) 2017 2018 Feng et al.
2019: ; and following an approximate gradient of
2009; Maei et al., 2009 Mahadevan et al.) 2014] Liu et al.) 2016 Ghiassian et al. 2020)). For many years, however, there was only limited set of choices mainly vanilla TD algorithms. Because it unclear preciscly what objective TD was optimizing (Baird 1995 Antos et al. 2008), was difficult to extend the algorithm. The known alternative- optimizing the MSBE suffers from an issue of double sampling in the absence of simulator [Baird] 1995 Scherrer 2010).
projected version of the MSBE (Sutton et al.
Driving the recent innovations are two key advances for objectives in RL. The first is the formalization of the objective underlying TD, called the mean squared projected Bellman error (MSPBE) (Antos) al. 2008 Sutton et al. 2009),, which projects the Bellman error into the space spanned by the function approximator. Many algorithms are built on the originally introduced variants, GTD2 and TD with gradient corrections (TDC) (Sutton et al. 2009). These algorithms, however, are limited to linear function approximation because the MSPBE is defined only for the linear case.
The second advance is the introduction of conjugate form for the MSBE handle the double sampling problem (Dai et al., (017)). By transforming the MSBE using biconjugate functions, the double sampling problem instead became better understood saddlepoint optimization problem. The SBEED algorithm (Dai et al. 2018) later extended the conjugate MSBE to control using smoothed Bellman optimality operator and parameterizing both the policy and value function estimates. By transforming the MSBE into objective for which we can obtain unbiased sample gradients, the conjugate MSBE allowed a natural extension to nonlinear function approximation.
A natural next step is to use these advances to revisit defining and optimizing statistically robust losses for value functions. The MSBE and MSPBE are built on squared errors, which are known to be sensitive to outliers. In the RL setting, this translates into overemphasizing states for which the Bellman residual is high at the cost of obtaining accurate estimates in other states. For example, consider the CliffWorld domain (Sutton and Barto 2018) where the agent must navigate alongside

---

Under review as a conference paper at ICLR 2022
cliff to reach rewarding goal state. Should the agent step into the cliff, the agent is sent back to the start state with a large negative reward. In order to represent the optimal policy, the agent must learn that actions which lead to the cliff yield negative return, while actions that lead towards the goal yield positive return (or at least less negative). Perfectly representing the exact magnitude of negative returns for stepping in the cliff is not useful, as it is sufficient only know that these actions are more negative than actions leading towards the goal. Further, because the agent bootstraps off these value estimates in the states near the goal, squared errors tend magnify these inaccuracies across all states visited in the episode. In general, the ability to replace squared errors with alternatives, like absolute errors or the Huber loss (Huber) 1964), provides another dimension to improve our algorithms and, to the best of our knowledge, has yet to be explored.
Issues with squared errors have been noted in the RL literature and addressed heuristically in control using clipping. DQN uses clipped TD errors by default to avoid large magnitude updates (Mnih) al. 2.15). likewise, Dueling DQN (Wang et af.] (0066) directly clips its gradients. There is close relationship between clipping TD errors and using Huber loss in the linear function approximation setting. However, even for linear Q-learning or TD-leamnngg clipping TD errors does not correspond to minimizing a Huber loss for the Bellman error. Rather, it takes non non-gradient update and applies clipping to that update to mimic the Huber loss which results in a bias similar to residual TD methods (Baird. 1995 /Sttonn and Barto, 2018).
In this work, we develop practical algorithms that can use absolute and Huber errors instead of squared errors for the BE. To do so, we develop unified perspective of the mean absolute Bellman Error (MABE), mean squared Bellman Error (MSBE), and mean Huber Bellman Error (MHBE) which smoothly interpolates between these other two. We rely on the insight that the MSBE can be reformulated into a saddlepoint problem with the introduction of an auxiliary learned variable; using the same strategy, we derive biconjugate form for the MHBE and MABE amenable to simple gradient-descen techniques. The resulting approach is a simple modification of existing gradient TD algorithms, using a different update for the auxiliary variable, making it straightforward to use either of these losses. We show that the MHBE and MABE can significantly change the solution quality, improving accuracy in terms of the mean squared value error, as well as the mean absolute value error. We then show that gradient algorithms for the MHBE tend to perform similar to those for the MSBE, but provide significant robustness improvements in certain cases, particularly under bad state aliasing. Finally, we show that in control with nonlinear function approximation, gradient- based algorithms minimizing the MHBE often outperform those using squared losses or those using non non-grade -on-girdient updates.
2 PROBLEM FORMULATION
We model the agent's interactions with its environment as a Markov Decision Process (MDP), (S,A, P, Rt 7). At each time-step f, the agent observes states S₁ S, selects an action A, A according to policy æ : S - A(A), transitions to the next state S E S according to transition function P: S x A + ASS), and receives a scalar reward signal R441 and discount 7++ E [0. The discount depends on the state, and encodes termination when 7++1 = 0 (White 2017).
For the prediction setting, the agent's goal is to estimate the value function U, for given policy. The value function can be defined recursively, using the Bellman operator
(T) del E [R+t + 7441(((+11) SS s]
where the expectation is taken with respect to the policy and transition dynamics P. The true values Ux are the fixed point for this operator: Tu, = vz Our goal is to approximate Uz, with Vp E F for some (parameterized) function space F. Typically, the quality of this approximation is evaluated under the value error, either mean squared value error (MSVE) or mean absolute value error (MAVE)
MSVE(ve)
d(s) (va(s) Un (s))
MAVE(ua) def d(s))V(ss) 2x (s)
SES
SES
where d: S + j0, 1] is typically the visitation distribution under a behavior policy!
Forr exposition, assume discrete states and actions throughout this paper. The connection to continuous state-actions is straightforward and will explicitly call where this connection is less obvious.
2

---

Under review as a conference paper at ICLR 2022
One objective used to learn approximation by is the mean squared Bellman error (MSBE)
()
def
Σ
d(s)
- vo(s))² -olss))))o(ss)>
Σ d(s))(((0) S s|²
(I)
SES
SES
where 5(0) R&+1 + (4410o(S441) - If E F, then there exists 9 such that Up and O) O. Otherwise, U/ $ F. then this objective trades-off Bellman error across states.
The trade-off in errors across states is impacted by both the weighting d as well as the fact that squared error used. The function approximator focuses on states with high weighting d. which is sensible. However, by using squared error, it heavily emphasizes states with higher error which may not be desirable. In the next section, we develop an approach use robust losses the absolute error and the Huber error-in place of this squared error.
The same approach as above can also be used control to approximate the optimal action-values v² These values can similarly be defined using a Bellman optimality operator
a'EAA Re+1 + 7+41 max q( g(S.+1, a') S: - 8, Ar (T')) def
with =²q"=q" The corresponding mean squared Bellman error for learning approximate qo is
Σ d(s,a) [(T (0) a))
#ES,aEA
where we overload d to be state-action weighting, which typically corresponds the state-action visitation frequency under a policy.
ROBUST BELLMAN ERRORS
In this section, we provide reformulations of the MABE and the MHBE which avoid the double sampling issue by using their biconjugates. For some intuition on how these objectives differ from the MSVE and MSBE, we visualize them in Figure
The absolute Bellman error is straightforward to specify for state, [0)) SS = s] as is the Huber Bellman error, p. (1) S s)), where the Huber function
MSVE
ENPr
a2
if |a| S 2r - otherwise
MSBE
(a)
MHBE
for some T 2 0. A common choice in RL is r = 1.0, which corresponds to using squared error when the magnitude of the error is below 1, and absolute error otherwise.
Objective values and fixed-points on the HardAlias-2 MDP (defined in Section 5). The fixed-points of the absolute and Hu- ber objectives are much better prox- ies for the squared value error than the fixed-pone point the squared Bell- man error. The dotted vertical lines indicate minima each objective.
Figure 1:
The difficulty, however, obtaining sample of the gradient of this objective for the same reason as the MSBE: the double sampling problem. see why, us examine the gradient of the MSBE
VMSBE(0) = Σ ((s)E(((()) S= (E(((0) | S=
SES
To sample this gradient requires sample of 5(0) for the first expectation and an independent sample of ((0) for the second expectation. Otherwise, using the same sample, we estimate the gradient of (80()) S = s) instead of E (8()) = s!². Due to the chain rule, both the MABE and MHBE will suffer from the same issue as the MSBE.
The strategy to reformulate the objectives using conjugates, introducing an auxiliary variable to R, the conjugate is (h)
estimate part of the gradient. For a real-valued function f supzer xh fx). This function f also has conjugate, f"" which is called the biconjugate of f. Further, for any function f that is proper, convex, and lower semi-continuous, the biconjugate (x) = fx) for all by the Fenchel-Moreau theorem (Fenchel 1949; Moreau 1970). Because
the three functions want to reformulate the absolute, huber, and square functions are all proper,
3

---

Under review as a conference paper at ICLR 2022
convex, and lower semi-continuous, this equivalence allows to reformulate these losses using biconjugates to avoid the double sampling issue without changing the solutions to these losses.
To rewrite existing results in notation and provide some intuition, first write the reformulation for the MSBE. The conjugate the squared error f(x) = 1=²² is f*hh = maX hr - _ which is in fact again the squared erro: f (h) _ (this result is well known, but for completeness we include the proof in Appendix A). The biconjugate is (**(x) maxneR ah h2² and f(x) =
f (x)). We can use this get that, x [6))) S s).
E (
max 2E ()0) S = s) h h² hER
we have function space Fall the set of all possible functions h SS + R then we get that
MSBE(() Σ SES
d(s) max S = s) h- -h²) --h²
HER
max
(2(((())) = = h(s)) - h(s)²)
hEJ
#ES
where the maximization comes out of sum using the interchangeability property (Shapiro et al. 2014 Dai et al., 2017) and hss) is function that allows to independently pick maximizer for every state in the summation.
If we have the maximizing function h (s), it is straightforward to sample the gradient. Because h" (s) itself is not directly a function of 0, then the gradient is
V (2s)(2)((()) S, = h" hs) (s) = Σ (s) #ES
Vav(s)) s.=s].
SES
Drawing samples S ~ d(-), A ~ T-1SS), and S P(-IS, A), we can easily compute a stochastic sample of the gradient. practice, we simply optimize the resulting saddlepoint problem with minimization over 0 and maximization over h. Note the optimal h" (s) = E ((0) S, s).
We can use the same procedure for the Huber error and the absolute error. We derive the biconjugate form for the Huber error in the following proposition. Though it is relatively straightforward result new and so worth providing formally.
to obtain, to the best of our knowledge,
Proposition 3.1 The biconjugate of the Huber function is
xh 1
D
x:
h12 2
bE(-7,T)
Due to space restrictions, we provide a complete proof in Appendix A
The absolute value has biconjugate max4e[[1,1] zh. As in the squared error case, this is well known result but we include the proof for completeness in Appendix A Notice again the constrained optimization problem for this biconjugate, as in the case of the Huber biconjugate function.
We can now provide the forms for MABE and MHBE:
(A
max dir
((s)(())((1(0}{{- =
(H
max F. S
d(s))2((s) = h(()²))
Faign is the of all functions hagan :S (--1,1 and Folip. the set of all functions halpp, :S- :8-+1-7,7]
For all of these reformulations, in practice we will have parameterized functions h, and so only approximate the objective. For example, for the MSBE, we may use linear functions Fr (h] h(s) = 0, 22s), 0n E Rdd) where S + Dd is d-dimensional feature generat- ing function. For the MHBE, we might use F, = {h h(s) = clip, (T(Tss)) O, E Rd).
For the conjugate reformulation of the MSBE, limiting Up and hss) to both be linear functions of the same features results in the mean squared Projected Bellman error, as noted by Dai al. (017)).
4

---

Under review as a conference paper at ICLR 2022
There no equivalent existing projected Bellman emrors for these new absolute and Huber variants. We note, however, that when the function space is constrained to only functions of our features x x(s), these conjugate reformulations do not suffer from the same identifiability issue raised for the MSBE (Sutton and Barto 2018 Scherrer
2010 as we show in Appendix B
The final form of the MHBE highlights a connection to recently proposed gradient TD algorithm that seemed to significantly improve stability over vanilla gradient TD methods. The algorithm, called TD with Regularized Corrections (TDRC), adds regularizer the parameters for h (Ghiassian et al. 2020). This regularizer has the effect of constraining (s(s) ) to be closer to zero, and so could be seen as providing some of the same robustness benefits as a Huber loss. Of course, the correspondence by no means exact. Regularizing with Eg on the parameters is different than restricting between - 1 and Further, as shown in that work, TDRC does not alter the fixed-point of its mean squared objective. Using the Huber error in place of the squared error, however, will likely alter the fixed-point.
Finally, we show that the MHBE bounds the MAVE. It is well-known that absolute errors are hard to optimize and so the Huber error acts as smooth approximation the absolute error. Theorem 3.2 first shows that minimizing the MABE acts as a proxy for minimizing the MAVE. Then, by bounding the absolute value error with the Huber error, we get that the MHBE acts as an approximation of the MABE implying that minimizing the MHBE likewise minimizes the MAVE. Because of this approximation, the MHBE has an irreducible error term which is controlled by the Huber parameter т. We provide a complete proof of Theorem 3.2 in Appendix A.1
Theorem 3.2 Let Teap min(T, 1), then arbitrary e Rd and <E<
< Teup we have
d Σ 8=U
Ve
2e Pr (Tu, 10,) + Vc
110 - vl11 s " -
2
4 OPTIMIZING THE OBJECTIVES
The gradient of all the reformulated objectives in terms of 8, for a given h, is actually the same: Exes ((s))(s)) [Vā(0] s). This means that the job of selecting between the absolute, squared, and Huber Bellman errors rests solely on how approximate the secondary variable, h(s). There are many ways to estimate the for the MHBE and MABE. A natural starting point would be to use the same estimate, he, (s) = E [5| | S = s), as in the mean squared case, then apply the corresponding non-linear function to h sisi or clipping. This gives the following updates for the objectives.
MABE
> MHBE
D MSBE
h(s²)) sign (s:))
h(s:) clip, (hea.,
h(st) hen. (5t)
0a = On,t + oh 6 han,, (8t) Von. I. (8t)
(3)
oc+1 0, a()))))) (Vo, v(st) - 7++1 1 Ve, to (8(+1))
Notice if we specifically parameterize ho, (s) 0n (s)) and v(s) (T((s), then we recover the GTD2 algorithm with linear function approximation (Sutton et al. 2009). Because the update for the primary weights is exactly the same as GTD2 and because the clip function encodes box-constraints
on the secondary weights (and so closed and convex), convergence of the GTD2-like algorithm for the MHBE follows directly from Nemirovski et al. (2009).
However, this parameterization is likely not ideal for either MABE or MHBE, because the func- tion approximation does not find the best h under the constraints. Another strategy is to pick a parameterized function class that encodes the constraints. For instance, in the case of the MABE the function, h(s) is effectively 2-class classifier for the sign of E S = s), which can be easily parameterized using a logistic regressor for example. Likewise, for the MHBE, the clipping function can be approximated using rescaled logistic function clip, (x)
tanh(=2).. which we will use for the nonlinear control algorithm Section 6 Despite being approximations, directly parameterizing the sign and clip functions may provide some advantages such as being a smooth approximation to the sign function or a twice differentiable approximation to the clip function; improving their optimization surface.
5

---

Under review as a conference paper at ICLR 2022
Several past works have reported a large performance difference between the GTD2 saddlepoint algorithm and the TDC gradient- coorrcttooo algorithm (White and White 2016 Ghiassian et al. 2018). The primary parameter vector in cradient-correctin error, ó, thus benefits from a direct unbiased error signal. Assuming linear function approximation, the update is (at(st) + A (())))(((+1)))) The saddlepoint algorithms, on the other hand, rely fully on h-as in the updates bebove--poovice
methods depends directly
sample of the TD
low-variance but possibly highly biased update.
Manipulating GTD2 update shows the relationship between the saddlepoint and gradient-correction
-Vob(0)h(s:) = ((ss)Voo(s)) "/o(:)Voo((Se-)))
(h(s;) 5(0) + 6(0) -h(s))) Vo -(0))o(ss))>((())) 7 (h(ss)))))))))))))))))))))(()())
-
TDC update
cxtra term
where we end up with a gradient that looks similar to the update for TDC, but with an extra term that accounts for the deviation between ō and our estimate, h(s). In fact, in the case of linear function approximation, and when h is the best linear approximation, the expected value of this additional term
over all states zero, making TDC an unbiased estimate of the true gradient shown in Appendix C
Similarly, we can construct corresponding gradient correction update to optimize the robust objectives, but at the cost of some additional bias in the approximate gradient because the extra term does not go away in expectation for the MHBE or MABE. As in the original TDC algorithm, the update for the secondary parameters 8₁ remains the same.
0 = 0. + a, (6(st) v(s)) - h41h(s Ve, (S++1))
We report the bias of the gradient for this update, for both the MABE and MHBE, in Appendix C We empirically investigate both updates for all three objectives in Section
PREDICTION EXPERIMENTS
We first investigate the quality of the fixed-points of each objective on several linear prediction tasks. The goal is to demonstrate the large advantage of the robust objectives in some settings, while maintaining reasonable performance across settings. We then investigate two families of online stochastic algorithms to optimize these objectives. We demonstrate that the Huber objective can often improve-ai and never harms- the optimization procedure for online learning.
Environments
We investigate six different problem settings, each chosen to highlight particular challenges for each objective. The first two use challenging state representation with features that aggressively alias across states. The robust objectives perform favorably on these problem settings, while the MSBE finds solutions that tradeoff accuracy in the aliased states poorly. HardAlias-1 is an 8-state random walk where the first, third, and final states share a common feature, and the remaining five states share three features. Alias- is the 2-state problem from Tsitsiklis and Van Roy (1977). which was originally designed to highlight the insufficiency of minimizing the squared Bellman Residual. We lightly modify the reward function of the MDP the optimal value function cannot be perfectly represented allowing each objective to have different minima.
The next investigated problem setting (Outlier) designed highlight the advantages of the MHBE by creating a single outlier state with large magnitude return among large set of states with approximately normally distributed returns. To emulate more realistic learning scenario, we use randomly initialized frozen neural network to generate five features to describe 50 state. The agent starts in a state that has an e = 0.01 chance of terminating immediately with -1000 reward, or a chance of entering the middle state of 49-state random walk.
The next pair investigated problems are chosen to highlight a scenario where the MSBE finds favorable solutions compared to the robust objectives. In these problem settings, the returns are distributed approximately normally across states and states are lightly aliased. We use two random walks, the first with N 5 states (SmallChhin) and the other with N = 19 states (BigChain), with Z The agent receives a reward of -1 or
a randomly initialized neural network representation of size N
+1 on the left and right-most most states respectively.

---

Under review as a conference paper at ICLR 2022
Evaluating the quality of the fixed-points of each objective function according to the MSVE and MAVE across several prediction problems. Error is plotted relative to the best representable value function. The robust losses are better in the hard aliasing domains, the MHBE is slightly better in Outlier, and the MSBE is better on the classic random walks. Note: the error for the MSBE is clipped in 2ard-lias-2 (approx. 25 MAVE).
Figure 2:
min MSBE
MHBE
HardAlas-1
HardAlas2
outier
Bigönain
LiteeChann
SmallChain
HardAlias-1
HardAlias-2
Baird
Hobre
stepsize
stepsize
stepsize
stopsize
Figure 3: MSVE averaged over 100 independent trials, for each stepsize in prediction domains. The mean squared algorithms generally performed well across environments even the adversarially chosen environments -uggesting the difficulty in minimizing the MABE. The Huber algorithms performed best across many environments, often displaying less sensitivity to the choice of stepsize.
The final investigated problem setting (Baird) is the well-known star MDP from Baird 1999). We use this domain to investigate optimization performance in high- variance off- off-policy setting where TD diverges. Because the true values are representable, the fixed- seed-points of each objective are equivalent.
Investigating the fixed-points
In this section, we seek to understand the quality of the solution of each proposed objective function according to the MSVE and MAVE metrics. We first assume access perfect model of each problem setting, then analytically compute the batch gradient for each objective and perform gradient descent using the ADAM optimizer (Kingma and Bal 2015) and iterate averaging. We additionally assume access to the true underlying state for each MDP and that partial observability due to state aliasing occurs from poor feature selection. This setting allows us to understand the properties of our feature generating functions, our MDPs, and our policies for which certain such as the MSBE-may suffer. For problem settings with randomly initialized feature representations, we perform this procedure for 1000 randomly generated representations and report the average error.
Figure 2 shows the relative MSVE and MAVE for each fixed-point across problems. For MSVE, the robust objective fixed-points are consistently either the best or only off by a comparatively small margin from the squared objective fixed-points. The squared objective fixed seddppinnss, however, can have catastrophically bad performance, even on small problems like HardAlias-2.
Investigating the optimization algorithms
We investigate the effectiveness of the optimization algorithms proposed in Section 4 on subset of the linear prediction problem settings. We choose one representative problem setting that favored the MABE and one that favored the MSBE when evaluating the fixed-points. We perform ten thousand update steps for every algorithm on each problem, except on HardAlias-I where we use one hundred thousand to ensure learning has slowed for every algorithm.
We look at performance across a wide range of stepsizes, a E 12-1,,115,,_ the mean and standard error over 100 independent trials for each stepsize, algorithm, and problem setting. To choose the secondary stepsize for each algorithm, we sweep over ratios 7) = a E (2 2--,2-3 3.-3.... ..., 23, 24
1-1), 2..,,, and report
and report performance using the best secondary stepsize for every point.
In Figure 3 we show stepsize sensitivities for each problem and algorithm measured using the MSVE. Due to the similarity in conclusions, we relegate the MAVE sensitivities to Appendix D The mean absolute TDC variant appears to suffer as a result of biased gradient estimates and generally performs worse across stepsizes than its GTD2-based counterpart. Generally the robust GTD2 algorithms and the Huber TDC algorithm show wider stepsize sensitivities and often the MHBE algorithms show marginally better performance for their best choice of stepsize.
7

---

Under review as a conference paper at ICLR 2022
6 NONLINEAR CONTROL EXPERIMENTS
Our experiments so far focused on prediction with linear function approximation. However, one of the primary motivating factors of using conjugate Bellman errors is the natural and theoretically sound extension to nonlinear function approximation and control. In this section, we empirically investigate a Huber algorithm for nonlinear control, where we estimate go and h using neural networks.
The QRC-Huber Algorithm
To estimate the secondary parameters of the MHBE for control, we use two-headed neural network where each head has one output for every action. The first head estimates qo a) and the second head estimates hiss, a). We block gradients from being passed back from the second head of the network, allowing the network's full function approximation resources to be used for predicting (s(s, a) accurately possible. This parameterization extension of the TDRC algorithm to control (Ghiassian et al. 2020). As discussed in Ghiassian et al.) (2020)aam reconfirmed in our experiments in Appendix using the saddlepoint update rule leads to poor performance in control, so we choose to use the gradient correction update.
used for an algorithm called QRC, an
This results in the following update rules
8 ++1 th,t + a (o ( h(s.a)) Von, h(s,a) a,3Rn
0++1 0, (VVoo((s,a)) 7/(8, a) Vo max ((S",
where 0 refers all of the parameters of the neural network, except the parameters for the secondary head, and 3 is the regularization parameter from Ghiassian et al. (2020). Unlike in the prediction setting, choose to twice differentiable approximation of the clipping function to allow easier optimization with pseudo-second order methods like ADAM (Kingma and Ba 2015). We accomplish this using the tanh function Mss a) = tanh( h)ssa)).
Experiments in Classic Control Domains
For the nonlinear control experiments, we investigate three classic control problems problems--Muunttie -Mountain Car, Cart-pole, and Acrobot-from the Gym suite (Brockman al. [016], larger domain with heavily shaped reward-Lunar Lander-and one additional domain designed to be particularly challenging for squared error algorithms, Cliff World. For all domains, discount factor y 0.99 and 0.1 for the e-greedy policy. The episode is cutoff if the agent fails to reach a terminal state in pre-specified number of steps. When cut off, the agent is teleported back to the start state and does not update its value function, thus preventing the agent from bootstrapping over the teleportation transition.
For all environments and algorithms, we sweep broad range of stepsizes and report results for every swept stepsize in Appendix DJ. For HuCeruber, we fix the Huber threshold parameter T = 1 for all domains except Mountain Car, where we use 2. We further ablate the impact of this decision in Appendix [ For the QRC methods, we chose not to use target networks-a frozen, infrequently updated set weights for the bootstrapping target-so that we can highlight the stability provided by using true gradient-base based methods with robust losses. DQN uses targets networks and sweeps over multiple refresh rates and additionally sweeps over its clipping parameter. In total, QRC and tune over 6 meta- pett-parrmeter combinations while DQN tunes over 120.
To demonstrate the stability of each algorithm, we report the full distribution of the performance metric over 100 independent trials for the best stepsize on each domain. We use the average return achieved over the last 25% of steps as our performance metric. We expect algorithms which exhibit stable performance to have narrow, approximately normal distribution centered around higher return, while algorithms which are unstable we expect to have wide performance distributions or even multi-modal distributions. We also report standard learning curves, in Figure
Acrebot
Cartpole
CliffWorld
Mountain Car
Lunar Lander
D8C
000
-160
300
155000 110000
-1000
-250
Average Return
Figure 4: Subplots show the distribution over 100 random seeds. The performance measure is the average return over the last 25% of steps for the best stepsize meta-parameter chosen per-domain. QRC-Huber consistently has approximately normal and narrow distributions around nigh-performance returns. DQN has highly inconsistent behavior over random seeds, with bimodal performance on Mountain Car and Lunar Lander, and very long-tailed performance on Acrobot and Cartpole.
8

---

Under review as a conference paper at ICLR 2022
Acrobot
Carl Pole
Clit World
Mountain Car
Lunar Lander
gfic Puber
GBC QNG Hubbe
QAC
OnC
targed
DON Target Nets
Learning Steas
Leaming Stepe
Leaming Stepe
Leaming Steae
Leaminng Steps
Figure 5: Learning curves for the best meta meta-parameter configuration for each domain, averaged over 100 random seeds. Shaded regions indicate one standard error. In Acrobot and Cart Pole, QRC-Huber and QRC have similar performance. In Acrobot and Cliff World, DQN and QRC-Huber have similar performance. However, in Mountain Car and Lunar Lander, QRC-Huber has significantly better performance than both competitors. QRC-Huber is the only algorithm reliably solve all domains.
Figure 4 shows the performance distributions of each tested algorithm. QRC-Huber exhibits narrow and approximately normal performance distributions for every domain, suggesting the stability of the algorithm over random seeds. The QRC algorithm performs reasonably on the Acrobot and Cart-polt domains, but performs quite poorly on the Cliff World domain. Because QRC is based on the mean squared Bellman error, the poor performance on Cliff World is exactly as expected, since this domain was chosen adversarially to highlight challenges with mean squared errors. While DQN is based on a clipped loss function that appears similar to the mean Huber Bellman error, it does not seem to enjoy the same stability as QRC-Huber, with average performance far worse than QRC-Huber on four of five domains due to high bimodality long-tailed performance distributions. The learning curves in Figure further highlight that QRC-Huber is the most robust of the three, across all five problem setting, either having comparable notably better performance.
Experiments on Minatar
Finally, we demonstrate that QRC-Huber can scale to larger domains using more complex neural network architectures. We use the Minatar suite of five miniaturized Atari games which retain much of the complexity of the full Atari games, while considerably reducing the computational requirements and cost (Young and Tian, 2019). We allow all three control algorithms to sweep over a small range of stepsizes and allow only DQN to sweep over target network refresh rates, as both QRC and QRC-Hube do not require target networks, We set the discount factor 0.99 for all domains and use the same neural network architecture and default parameters as Young and Tian 2019))
To avoid domain overfitting and reduce the cost of meta parameee tuming, we treat the entire Minatar suite as a single problem setting. As such, each algorithm must pick one meta-paramettr setting to use across all five games; favoring algorithms which are stable and insensitive to parameter choices. We scale the expected returns from each domain using probabilistic performance profiles (Jordan) et al. 22020] |Barreto et al.) 2010), then report the average scaled performance across the entire suite with 95% confidence intervals. We run each algorithm with its best meta- peeaa-parreete setting for 30 runs on each domain for a total of 150 runs. Additional procedural details can be found in Appendix L
Table we report the average scaled return across games in the Minatar suite. QRC-Huber outperforms both QRC and DQN on average across domains. Despite having four times the number of meta-parameter combinations and the ability to use target networks, DQN performs considerably worse than either gradient- based algorithm. That QRC and QRC-Huber perform similarly is unsurprising as the largest possible reward in any Minatar game +1, a design decision made in part because many algorithms-such as DQN-are unstable when learning from large rewards. Additional results on the Minatar suite are included in Appendix D
Table 1: Average return on Minatar
QRC-Huber
0.53 0.03
QRC
0.47 + 0.02
DQN
0.36 0.06
7 CONCLUSION
In this work, we extended the saddlepoint reformulation of the mean squared Bellman error, intro- ducing a novel pair of robust losses, the mean absolute Bellman error and the mean Huber Bellman error. We demonstrated that the solutions these robust objectives are comparable to the MSBE, and in some scenarios are significantly better according to the value error. The resultant gradient- based algorithms are less sensitive choice of stepsize in prediction and have more stable performance distributions in control.

---

Under review as a conference paper at ICLR 2022
REFERENCES
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with Bellman- residual minimization based fitted policy iteration and a single sample path. Machine Learning. 2008.
Leemon Baird. Residual Algorithms: Reinforcement Learning with Function Approximation. Ma- chine Learning Proceedings, 1995.
Leemon C. Baird. Reinforcement Learning through Gradient Descent. PhD thesis, Brown University, 1999.
André M.S. Barreto, Heder S. Bernardino, and Helio J.C. Barbosa. Probabilistic performance profiles for the experimental evaluation of stochastic algorithms. Conference on Genetic and Evolutionary Computation, 2010.
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 1983.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint orXiv:/606.015400 2016.
Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from Conditional Distributions via Dual Embeddings. International Conference on Artificial Intelligence and Statistics, 2017.
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation. International Conference on Machine Learning, 2018.
Werner Fenchel. On conjugate convex functions. Canadian Journal of Mathematics, 1949.
Yihao Feng, Lihong Li, and Qiang Liu, A Kernel Loss for Solving the Bellman Equation, Advances in Neural Information Processing Systems 32, 2019.
Sina Ghiassian, Andrew Patterson, Martha White, Richard S. Sutton, and Adam White. Online Off-policy Prediction. arXiv:1811.02597, 2018
Sina Ghiassian, Andrew Patterson, Shivam Garg, Dhawal Gupta, Adam White, and Martha White. Gradient Tefferel- Learning with Regularized Corrections. International Conference on Machine Learning, 2020.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, and Nathaniel Smith. Array programming with NumPy. Nature, 2020.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task Deep Reinforeement Learning with PopArt. Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Peter J. Huber. Robust estimation of a location parameter. In Breakthroughs in Statistics. Springer, 1964.
Scott M. Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip S. Thomas. Evaluating the Performance of Reinforcement Learning Algorithms. International Conference on Machine Learning, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations, 2015.
Bo Liu, Ji Liu, Mohammad Ghavamzadeh, Sridhar Mahadevan, and Marek Petrik. Proximal Gradient Temporal Difference Learning Algorithms. International Joint Conference on Artificial Intelligence, 2016.
10

---

Under review as a conference paper at ICLR 2022
Hamid Reza Maei, Csaba Szepesvári. Shalabh Bhatnagar, Doina Precup, David Silver, and Richard S Sutton. Convergent Cemporal-Differenne Learning with Arbitrary Smooth Function Approximation. Advances in Neural Information Processing Systems, 2009.
Sridhar Mahadevan, Bo Liu, Philip Thomas, Will Dabney, Steve Giguere, Nicholas Jacek, Ian Gemp, and Ji Liu. Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Duce Spaces. arXiv:1405.6757, 2014.
Ashique Rupam Mahmood, Huizhen Yu, and Richard S. Sutton. Multi-step Off off-po Learning Without Importance Sampling Ratios. arXiv: met::072200000 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, and Georg Ostrovski. Human-level control through deep reinforcement learning, Nature, 2015.
Andrew William Moore. Efficient Memory-Ban Basoed Learning for Robot Control. PhD thesis, University of Cambridge, 1990.
Jean Jacques Moreau. Inf-convolution, sous-additivité, convexité des fonctions numériques. Journal de Mathématiques Pures et Appliquées, 1970.
Remi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and Efficient Off-Policy Reinforcement Learning. Advances in Neural Information Processing Systems, 2016.
Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 2009.
Johan S. Obando-Cerrt and Pablo Samuel Castro. Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research. International Conference on Machine Learning, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. Pytorch: An imperative style, high- performance deep learning library. arXiv preprint 1rXiv:1912.01703 2019.
Doina Precup, Richard Sutton, and Satinder Singh. Eligibility Traces for Off-Policy Policy Evaluation. International Conference Machine Learning, 2000.
Bruno Scherrer. Should one compute the Temporal Difference fix point or minimize the Bellman Residual? the unified oblique projection view. International Conference on Machine Learning. 2010.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyúski. Lectures on Stochastic Program- ming: Modeling and Theory. SIAM, 2014.
Richard S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. Advances in Neural Information Processing Systems, 1996.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast desseett methods for temporal-difference learning with linear function approximation. International Conference Machine Learning, 2009.
J.N. Tsitsiklis and Van Roy. An analysis of cemporal-different learning with function approxima- tion. IEEE Transactions Automatic Control, 1997.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando de Freitas. Dueling Network Architectures for Deep Reinforcement Learning. International Conference on Machine Learning, 2016.
Adam White and Martha White. Investigating practical linear temporal difference learning. Interna- fional Conference on Autonomous Agents and Multi-Agent Systems, 2016.
11

---

Under review as a conference paper at ICLR 2022
Martha White. Unifying Task Specification in Reinforcement Learning. International Conference on Machine Learning, 2017.
Kenny Young and Tian Tian. MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments. arXiv:1933.0317, 2019.
12

---

Under review as a conference paper at ICLR 2022
A
BICONJUGATE FORMS
Proposition A.1 The biconjugate of the square function f(x) 1 is fm)()) = maxse hx h²²
Proof: Recall the definition of the convex conjugate and correspondingly the biconjugate:
(*)) sup f(h)}
hel
(*) sup fhr f (h))
heB
Then the conjugate of the square function with dual parameter a,
(x)
sup za
a²
eE
Applying the convex conjugate again and we obtain
! a²
(**(x=)
sup EER
xh sup ae
ha
Clearly, the inner supremum is achieved at a" = h, so plugging in the maximizing value of a, we obtain
(x)
sup HER
xh h22 + h22
2
max heR
xh
h
Finally multiplying by two. (x) = x² and 2 (I) = maX (2xh h²),. arriving at the
biconjugate of the square function used in Dai et al. (2017) and Dai et al. (2018).
Proposition A.2 The biconjugate of the absolute value function f(x) = |x| is ('(()) =
ma. zh.
Proof: The proof follows the same format as the proof of the biconjugate for the square function. Defining the conjugate and biconjugate respectively,
0
when |x|| S
f (x) sup |a| = aekt
00 otherwise.
(x) sup zh f*))..
heR
Simplifying the biconjugate form, we get
when h < 1 otherwise.
(*)
sup
heR
zh
sup xh
D -00 is not feasible
sup kE[-1,1]
rh.
Finally, considering the maximizing values of ht we get h = -1 when < 0 and h = with > 0 so the biconjugate simplifies to
- = sign = f(x)
thus completing the proof.
Proposition A.3 The biconjugate of the huber function is (x) maxke[-r.r) sh 112
13

---

Under review as a conference paper at ICLR 2022
Proof: Let the huber function be defined as
22a² [T|a] 7
if S 7 otherwise.
Pr (a)
Then the convex conjugate is
M (2) = sup
Pr (a)
aE
za
yn²
if if|a| T
sup
va
T|al
+ 7
otherwise.
Resolving the supremum, consider first the case that |x|| S T, then
sup za
= a
2
ra
2 11 22
D a" = x
x² T
for the first case of the piecewise, and for the second case, first consider 0 S x S 7, then
sup
Da - T|a|
2 T
|a|sr
sup(1
T)a
art
-00
because - - 0, and when - < x < O, we have
20 - r|al - |a|sr
sup
1
sup
T |a|
2
as-
1
-T 72
2 72
D a"
IT 3
Finally, because - we have -T
2 122 so the maximum is 1
Next consider the second case that |x|| > then we need not consider all sub-cases because when T, then
sup
7/a| 7
|a|>T
=sup(r T)a-
a>t
and when -7 then
1 T|u|
sup za
-
|e|ST
2
1
sup a<-T
T|a|
2
14

---

Under review as a conference paper at ICLR 2022
and the supremum will be when |a| >
Thus the conjugate function is
if | S otherwise.
(())=
0
Using this to compute the biconjugate, we obtain
f (x)
sup ha (h) KEB
h 21
||| S T
sup AER
hr OC
otherwise
max hac l1? AE(-T,T]
2
where again the constraints the maximization come from encoding the infeasibility of [h] >
A.1 PROOF OF THEOREM 3.2
We start by showing that the mean absolute Bellman error bounds the mean absolute value error. We then show in Lemma A.5 that the Huber function is close approximation of the absolute function, and so emits an upper-bound with some approximation error controlled by the Huber parameter т. Putting these together, then, we show that the mean Huber Bellman error upper bounds the mean absolute value eπror with a small non nonvvanishing approximation term.
Lemma A.4 For any vector Rdd, then
1| w/1 S (I(T Pr, - o/l1.-
Proof:
First notice that Un to policy æ. Then we have
(T Pr,x)) where is the expected reward function with respect
To Tu-w v va PR, -
b. by definition of To
Ta (1 P
b rearrange terms group
= (1 -Pn7)us - (1 - Px,y)u
b Tx = -1- Px,)), rearrange terms
(I (U --16-11) v)
Bringing the (I - Pa,) term to the left side, then
(1- P&,,) (Tv v) Us v. Finally, because the matrix norm induced by n-norm compatible, we get that
[I(1 [ Pa,))
T e|l 2 ||ua - =1l1
thus completing the proof.
Lemma A.5 Let Teap min(r, 1) and r > 0, then for any vector E Idd and c Pap
d
Ve
Ve
|all - Σ
Pr (a,) + 2
2e
1=0
Proof: Our goal is to show Jal <. Cp, (a) for any R. First notice that if aa Tepp then Pr (a) 2 Teap |a| by definition the Huber function and so we are done with this case. However, [a] < Teaps then pr (a) a². Should we try to find some constant C such that (a| s Caa then we easily find that C' 2 lal which goes to infinity as a goes to zero. Instead, we can find |a| Σ C(a² +e)
for arbitrary > 0, which yields C 2 a=
which is bounded. To find C, we have
la
C
max d² + |o|s Top
a DSaSTop Top d² +
max
Ve
2
15

---

Under review as a conference paper at ICLR 2022
because the maximum is obtained Ve. We now have that, for |a| Teap and C Le then
22
|a| I Cpr (a). Choosing C' 2 1 to satisfy the |a] 2 Teap case, and C' y otherwise, we thus obtain
our restriction on Teap min(T, 1) completing the proof.
B PROJECTED BELLMAN ERRORS
In Section 3 we described conjugate form of the Bellman error. This conjugate Bellman error
depends finding the maximizing function h S R from the set of all functions Fall As is highlighted in [Sutton and Barto (2018) Chapter 8), the MSBE is not identifiable; an issue inherited by the conjugate form of the Bellman error. This identifiability issue stems from the fact that the mean Bellman error is defined with respect to non-observable quantity, the states. When states are aliased together due to partial observability, then the agent can see only part of the statespace while optimizing 0 while seeing the entire statespace for optimizing / E Fall. Clearly this is not realistic setting. Modifying modifying this example such that the optimization procedure for h has the same partial observability leads towards an identifiable form of the conjugate Bellman error
Identifiable BE(()= del
max E 2E (8()) S] h(S) - h(S)²] REFA
where Fn def fo x such that h S are all functions defined with respect to features S + Rdd Because Fi. C Fall then the solution to the Identifiable BE may be different from the solution to the conjugate BE due more constrained optimization on h.
As shown in Section 3 the optimal function h² (s) = [[)(S) S = s]. In the finite state setting, we can represent this function as a vector u E IR [S composed of entries (()(S) S = =s]; thus the vector T 0. We can define a projection operator on u as
II a arg min ||u hlla hEF₂
where d S 10 is weighting states Then assuming that Fr, is convex set, we decompose u = =1, 2 in the weighted space h TDū 0 for D del diag(d). From this projection operator, we can define the mean squared Projected Bellman Error
+ where is the component in u that is orthogonal to = =
PBE
def max hEFn SES
Σ d(s) 8) hs)) h(s)²)
Σ d(s)(2u(s))(()) - 1(6)3)
max hEFn
sES
Σ ((s)(2(()))())) - -(s)())))))))))))))))))))))>))}) a h 11
#ES
Σ (())((((s)++ - -()))))
#ES
Σ [(())(2((s)= (M)))) +2 Σ d((s)(())(s))
#ES
SES
Σ d(s))(s)² +2 > (s))(((s)M(s)
#ES
SES
Σ d(s)
SES
[ 6
where the second to last step is because h is orthogonal to under weighting d.
This connection to projected Bellman errors provides some insight for the role of approximating the function h, as well as for the robust objectives discussed in Section Each choice of function space F, results in different linear projection the vector describing the Bellman error in every state, with each projection producing its own orthogonal component which is ignored in the resulting projected objective function. For instance, consider low-rank projection operator r,
16

---

Under review as a conference paper at ICLR 2022
The resulting projected Bellman error could project potentially high-error states error, thus allowing no function approximation resources to be used to represent the value function in that state. When projection is used, equivalently when Il are projected and the values are learned to directly minimize the Bellman error. And when the space F F with Fh and é E F, then recover the original mean squared projected Bellman error of Sutton et al. 2009).
s) the trivial projection, then errors
C BIAS OF TDC GRADIENT
In this section, we discuss the biased gradient estimate used by gradient-correction methods such as TDC or QRC-Huber. We show that in the case of linear function approximation, when / is the best linear approximation of E s], then the gradient is unbiased for the MSBE. However, this is no longer the case when considering the robust objectives nor is it the case when h must be approximated online. As has been seen past results (White and White 2016 Ghiassian al., 2018 2020), this biased gradient does not seem to harm TDC's performance and in fact the lower variace gradient estimate seems to improve empirical performance however, our own experiments Section suggest that for the MABE the biased gradient estimates can often prevent the gradient-correction algorithm from learning.
Manipulating the gradient of the conjugate Bellman error, get -vo(0))(() = h(s) - vo(s)Vo (()))))(s) + (l(+) - Vor)) (h(ss)
extra term
Let h- = E [xxx]] E [a:8] be the optimal linear regression solution for h and let both h and be parameterized with linear function approximation. Then because Vou(s) = x(s), we have
IE [[(h" * 8) (a
IE [ hh" 6)
xxx ht- E |s))
xx 'h- =E
and in expectation across all states
E [(N(S)) - V)
E [aoe
h-" IE [x=8]
E [xaer]
[x>]
B[xd]]
E[rd]
[[2ō]
E [x:6]]
where x = =()).
Unfortunately, consider the case of the robust objectives. Instead of the (h(s) 8) term, we apply nonlinear transformation to only hss). Intuitively, in the case of the MABE the difference between sign(h( (f(()) can arbitrarily large. In the case of the MHBE, the bias due to ignoring this additional term is a function of the clipping parameter T. Clearly as co, then clip, (h(s)) + h((s) and the same argument applies as the MSBE case.
The bias of the gradient estimate used by gradient-correction methods for the MHBE in the case of linear function approximation is
E (clipp,,, (s)(s) 6(0))=llo] Elcclip, (h² (s)) - -(0)]|==11])
when || [s)] <
Σ d(s)) (Isiig(("*(s) |00)||= otherwise
SES
Σ
d(s)) (s)) 8(0)1l==]lo-
17

---

Under review as a conference paper at ICLR 2022
Because when [s) < T, then |clip- (h)(s)) [s*(s) and are again in the case of the gradient of the MSBE. However, when [s)(s) 2 T, then clipp (h- (s)) = r and we accumulate some bias based on how far å is from т. When [ O, then likewise h" (s) 0 because the zero vector is always representable by a linear function approximator (by definition of linearity). Because O, then clip, (h* (s)) 0 and the bias is zero, the fixed point of the algorithm remains unchanged.
D ADDITIONAL RESULTS
In this section we include supplementary results to the main body of the paper. We first investigate the relative ordering of the proposed optimization algorithms when measuring the MAVE instead of the MSVE. We then motivate empirically why we built our nonlinear control algorithm based on a gradient-correctinn
method instead of saddlepoint method by investigating the performance of nonlinear GQ on our benchmark control domains. Finally, we end with two ablation studies investigating the impact of the Huber threshold parameter on each domain as well as a second ablation investigating the impact of the choice to exclude target networks in the main body of the paper, especially for DQN.
SmallChain
HardAllas-2
HardAlias-1
Baird
14.00
Figure 6: MAVE averaged over 100 independent trials, for each swept stepsize in key prediction domains. The mean squared algorithms generally performed well across environments adversarially chosen environments-sug suggessinn the difficulty in minimizing the MABE. The Huber algorithms performed best across many environments, often displaying less sensitivity to the choice of stepsize.
the
In Figure 6 we investigate the stepsize sensitivity of each algorithm on four representative linear prediction problems. The robust algorithms generally have similar best performance as the MSBE algorithms. but with less sensitivity to choice of stepsize. The 4DCTDC algorithm often fails to learn meaningful value function estimate in the given number of training steps. We hypothesize that this is due to the bias in the gradient estimate for gradient-correctton case of the MABE but not in the case of the other objectives. The results in Figure 6 are averaged over 100 independent trials for every choice of stepsize, algorithm, and domain. The shaded regions correspond to standard errors, though error bars are excluded from the TDC algorithms for readability and because the standard errors are negligible. The performance measure for each algorithm is the average error over the last 25% of steps in each domain.
methods, which is pronounced in the
Acrobot
Cart Pole
World
Mountain Car
2:12
2:12
2:1
2-11
2
-500 400
-300 -200
-100
200 400
-20000
-10000-5500 5000
-3000 -750
5500
-20
Rezum
Setur
Return
Return
Figure 7: Comparing the mean return over the last 25% of steps across several saddlepoint methods against QRC-Huber. The saddlepoint methods generally perform very poorly, frequently finding a policy only slightly better than the random policy. These results are consistent with the findings of Ghiassian et al. (2020) and motivate building on gradient-correctoon methods for nonlinear control. Like QRC-ruber, GQ-Huber uses twice differentiable estimate of the clip function and all algorithms use the ADAM optimizer.
18

---

Under review as a conference paper at ICLR 2022
Acrobot
Cart
Cliff World
Mountain Car
2.0
2.0
2.0
2.0
Threshld
Thrested
Threstold
Threchold
0.5
0.5
0.5
0.5
0.25
0.25
-1000
-1000
-500
Return
Figure 8: Ablating the impact of the threshold parameter for the Huber loss function for the QRC- Huber algorithm across the benchmark domains. For three of the domains, QRC-Huber is robust to the choice of threshold parameter with a default value of r = being a good choice. However, the Mountain Car domain shows high-bimodality in performance distribution across multiple random initializations of the neural network for smaller values of the threshold parameter.
Cart Pole
Acrobot
Mountain Car
500
500
500
Target Refresh
Target Refresh
Target Refresh
50
50
DQN
-500 -400
-300
200
-100
200
400
-1000
-750
-500
-250
Return
Return
Figure 9: Ablating the impact of target networks on the performance of the nonlinear control
algorithms on benchmark domains. The gradient- methods receive much less benefit from using target networks than DQN, which requires target networks to achieve above random performance on Cart-pole and to reduce the bimodality of its performance on Mountain Car. Even with target networks, DQN still exhibits large skew and bimodality in its performance distributions, indicating instability.
Acrobot
Cart Pole
Cliff World
Mountain Car
2.42
QRC-1lube
2.9
QRC
2:10
DQN
Stepstoe
Retur
Raturn
Figure 10: Comparing algorithms on benchmark control domains with the area under the learning curve as the performance metric. Unlike Figure early learning is included in the performance metric, giving a sense of the sample complexity of each algorithm. QRC-Huber tends to perform favorably across all four domains compared to QRC and DQN, exhibiting much more narrow performance distributions that are often centered around higher rewards than the competitor algorithms.
Figure 7 investigates the saddlepoint optimization algorithm for nonlinear control across our four benchmark domains. Generally, the Greedy-GQ algorithm performs considerably worse than gradient- correction algorithms; motivating factor for building gradient-correction methods (Dai et al.) 2018 Ghiassian et al., (2020). A possible explanation for this poor performance is the dependency of the representation learning process having accurate estimate of h(s)) which itself depends having a well-learned representation. This circular dependency is less obviously present in gradient- correction methods, which depend on sample of the error signal instead of an estimate of the emro
19

---

Under review as a conference paper at ICLR 2022
Minatar
Figure 11: Evaluating the performance of each control algorithm on the Minatar suite of games. The learning curves show the scaled performance metric averaged across domains with 95% bootstrapped confidence inter- vals about the mean. Because each point in the learning curve has less underlying structure than the aggregate performance metric, the confidence intervals are signifi- cantly more pessimistic than reported in [ As such, the sample mean performance of QRC-Huber is slightly higher than QRC during early learning, but not statisti- cally significantly so in this result. Both gradient-based algorithms considerably outperform DQN with statistical significance, indicating both less domain-sensitivity to meta- parameerssssers as well as better absolute performance.
0.8
QRC-Huber
DQN
QRC
Learning Steps
signal for the primary learning process. Improving the performance of saddlepoint optimization methods for nonlinear control is important future work.
D.1 ABLATING DESIGN DECISIONS
One of the proposed robust objectives depends on a new meta-parameter the threshold for the Huber uunction-hiid was not present previous extensions of the conjugate Bellman error to nonlinear control. While for most of our domains we could reasonably pick a default value of r = and avoid allowing our proposed algorithm more opportunities to tune peeta-parameters, this choice impacted the claims made in the main body of the paper. The choice of 7 depends on the magnitude of the TD errors experienced by the algorithm during optimization, which driven in-part by the magnitude of the rewards, and thus is domain- dependent. To decrease this dependency, we could consider scaling the magnitude of rewards in a fomain-indepeneee way, for instance using PopArt (Hessel et al. 2018).
In Figure g we ablate over several choices of threshold parameter for the QRC-Huber algorithm. We likewise investigated the sensitivity of DQN to choice of threshold parameter, but due to the general instability of DQN it was less clear which values were generally best. Allowing DQN to select its best for each domain does not change the conclusions in Figure 4 so for simplicity we choose to maintain consistency between DQN and QRC-Huber. We see long-tail performance distributions as the threshold parameter is made smaller, likely due to the optimization process spending more time in the mean absolute region of the Huber loss. On the Mountain Car domain, both QRC-Huber and DQN were significantly impacted by 2 and saw strongly bimodal performance distributions.
The poor performance of DQN in Figure 4 -sspecillly on the Cart-pole domain-iss surprising; however recent work has also shown that DQN has shockingly poor performance on a wide variety of domains (Obando-Ceron and Castro 1021). A potential explanatory factor for the poor performance could be the choice exclude target networks from our investigation, which could adversely affect DQN disproportionately compared to the gradient- based algorithms. To understand the impact of our choice to not use target networks, we ablate over the number of steps taken between synchronizing the weights of the target network.
Figure 9 shows the number of steps between target network synchronization for QRC-Huber, QRC, and DQN, where one step of synchronization refers to not using target networks at all. For Acrobot and Cartpole, increasing the number of steps between synchronization appears to harm the performance of the gradient-based methods, likely due to artificially reducing the speed that the bootstrapping targets receive new information. DQN, on the other hand, benefits from the reduced variance in the bootstrapping targets and tends to perform better across all three domains as the target networks updated less frequently; though even with 500 steps between synchronization, DQN performs poorly on Mountain Car.
20

---

Under review as a conference paper at ICLR 2022
E
EXPERIMENTAL DETAILS
E. ENVIRONMENTS
In this section we provide further details for the environments and problem settings used in Sections 5 6 6
HardAlias-1 is an 8-state random walk where the agent starts in the far left state and moves right with 90% probability. The episode terminates on taking the move right action in the far right state. The agent receives -1 reward per step with discount factor of y = 0.99. The first, third, and final states all share a common feature, and the remaining five states use the dependent feature representation from Sutton et al. (2009), resulting in a feature vector of size d = 4 for each state.
The second hard aliasing problem. HardAlias-2, is the 2-state problem from Tsitsiklis and Van Roy (1997). We lightly modify the reward function of the MDP so the optimal value function cannot be perfectly represented, thus allowing each objective to have different minima. The agent receives reward of +I after transitioning from the first state to the second, and reward of O for all other transitions.
Outlier is large 49-state random walk with an additional "entry" state, where the agent has an e c 0.01 chance of terminating immediately with -1000, or a 1 1 chance of entering the middle state of the random walk. To emulate more realistic learning scenario, frozen neural network with ReLU activations and two hidden layers of sizes ten and five respectively to generate five features to describe 50 states. Taking the left action in the far left state of the random walk results in termination and a reward of 1 and correspondingly the right action in the right state results in termination with a reward of +1. The discount factor is set to 7 0.99 and the left action was chosen with probability e in every state.
use randomly initialized
For the two random walks, the first with N = 5 states (SmallChain) and the other with N = 19 states (BigChain), we use a randomly initialized neural network representation with ReLU activations and three hidden layers of sizes h1 4N, h2 NN, and hg 2 units respectively, resulting in a feature representation of size d = h3. These problems are off- off-policy with the target policy taking the left action with 90% probability and the behavior policy taking both actions with equal probability. The discount is y = 0.99 for both problems.
&
Finally, Baird is the well-known star MDP from Baird (1999). is used investigate optimization performance in a high-variance off off-po setting where TD diverges. We do not use it to evaluate the quality of fixed points, because the linear function approximation can represent the true values, and so the fxed-points of each objective are equal in quality.
In the nonlinear experiments, for all domains we use a discount factor of y = 0.99 and an e-greedy policy with e = 0.1. In Mountain Car, Acrobot, and Cliff World, the agent receives reward of 1 per step until termination and in Cart-pole the agent receives a reward of +1 per step. Every environment has an episode cutoff if the agent fails to reach terminal state with a pre-specified number of steps. When cut off, the agent is teleported back to the start state and does not update its value function, thus preventing the agent from bootstrapping over the teleportation transition. All algorithms run for one hundred thousand steps in total across all episodes, except in Lunar Lander where algorithms are run for 250k steps.
In the Mountain Car environment (Moore 1990; Sutton 1996), the goal is to drive an underpowered car the top of a hill. The agent receives as state the position and velocity of the car, and can choose to accelerate forward, backward, or to do nothing on each timestep. The episode terminates when the agent reaches the top of the hill, or is cut off when the agent reaches a maximum 1000 steps. In the Cart-pole pole domain (Barto et al. 1983), the agent balances a pole attached to a cart which can move along single axis. The agent receives as state the position and velocity of the cart, as well as the angle and angular velocity of the pole. The episode ends when the pole falls if the agent reaches the maximum of 500 steps. Finally, the Acrobot domain (Sutton) 1996) has the agent swing a double-jointed arm above threshold by moving only the inner joint. The agent receives state the current angle and velocity of the joints and can take as action, swing left or swing right. The episode terminates when the specified height is achieved, is cut off after 500 steps.
The Cliff World environment- lightly adapted from Sutton and Barto (2018) is a discrete gridworld with 20 states. The agent starts in the bottom left state and seeks to reach the goal state in the bottom
21

---

Under review as a conference paper at ICLR 2022
right. Along the bottom of the grid lies cliff, where the agent receives large penalty of -1000 reward for stepping into the pit and is teleported back to the initial state without terminating the episode. The episode terminates only when the agent reaches the goal state, or is cut off when the agent reaches a maximum of 500 steps.
For all environments, we fix meta- meta-parameters other than the stepsize to their default values. For QRC-Huber, we fix regularizer parameter 3 = and secondary stepsize ratio 1) = 1. For both mean for all domains except Mountain Car,
Huber algorithms, we fix the Huber threshold parameter where we use 2. We further ablate the impact of this decision in Section D We sweep over the stepsize parameter for all algorithms and environments and report results for every swept stepsize.
E.2 FINDING FIXED-POINTS
To find the fixed-points of the objectives in Section [ that assumed access to the underlying dynamics of each MDP to compute exact expected gradients for each update to the primary variable. We use first order stopping conditions to ensure that the optimization procedure has reached fixed-point; i.e. when the norm of the gradient is near zero (specifically less than 10-3). We use ADAM parameters of 3 = 0.99 and A3 = 0.999 along with a moving iterate average with exponential moving average parameter 3 0.9 to reduce oscillation of the gradients and iterates around the fixed-point (especially for the mean absolute objective, where 1 where is the vi
we used an iterative optimization procedure
we performed subgradient descent). We decay the global stepsize according to a
number of update steps taken far.
E.3 LINEAR PREDICTION
For the prediction problems comparing optimization methods in Section 5 we swept over the primary and secondary stepsize for all algorithms allowing each to be tuned independently. We swept values 10] and the ratio between the primary and secondary 20 ) All algorithms have the same number of meta- parameter combinations, so comparison between each algorithm remains fair. Reported results use SGD with a constant stepsize, though results using RMSProp yielded similar conclusions and thus were omitted. All algorithms were evaluated after 10k updates for each domain except the random walk which required 100k updates to reasonably converge.
of the primary stepsize a E (2 stepsize 7)
aa (2-6,2 2- or
E.4 NONLINEAR CONTROL
For all of the nonlinear control algorithms and domains, we used neural network function approxima- tion with two hidden layers and ReLU activation units. For Acrobot, Mountain Car, and Cliff World we used 32 hidden units in both layers, and in Cart Pole we noticed significantly better performance for all algorithms when using 64 hidden units (consistent with the findings obanndo and Castro 2021) which suggested Cart Pole needs considerably more parameters for good performance), finally for Lunar Lander we used 128 hidden units in both layers. We use experience replay buffers to store the 4000 (10k for Lunar Lander) most recent transitions and draw 32 samples to compute mini-batch updates on every timestep. We use the ADAM optimizer with default parameters for all algorithms (3, 0.9 and 32 0.999), but notice little difference in conclusions when using SGD or RMSProp optimizers. All agents are trained using e-greedy behavior policies, with e = 0.1 for every domain. Agents are trained for fixed 100k steps for Acrobot, Mountain Car, and Cart Pole. Cliff World only required 50k steps to achieve good policies, and Lunar Lander required 250k steps.
For the Minatar games, used the same function approximation architecture and meta- meta- parameter settings as in Young and Tian (2019). Specifically the neural network uses a single convolutional layer with 16 channels, a stride-width of 1, and kernel-width of followed by a ReLU activation, the output of the convolutional layer is then flattened and sent to a single fully-connected layer with 128 hidden units and ReLU activation. We use the ADAM optimizer (Kingma and Ba 2015) with default parameters and sweep over stepsizes in a E {2 13 12
2 81 For DQN only, we additionally
We note that the MSBE fixed-point can easily be computed analytically using least-squares solver. For consistency, we use the iterative solver even for the MSBE. Reported results and conclusions are unchanged when using the analytical solutions.
22

---

Under review as a conference paper at ICLR 2022
sweep target network refresh rates in {1, 8, 32, 64] steps. Experiments run for 5M steps for each domain and a replay buffer of size 100k is used.
E.5 MINATAR EXPERIMENTAL PROCEDURE
For the Minatar demonstration, we treat the Minatar domain suite as single problem setting. In doing so, we can take advantage of lower variance performance metrics by averaging performance over each of the domains, allowing us report statistically significant claims using far fewer computational resources. The procedure is as follows.
1. We first swept over several choices of stepsize using only five runs for each game.
2. We then scaled the AUC for each individual run using probabilistic performance profiles (Barreto al. 2010 to value between [0. 1].
3. We picked the best performing stepsize for each algorithm by averaging the scaled AUC across runs and across games.
4. Finally best stepsize for a total of 150 runs and report the average scaled performance in Table [ as well the average scaled performance over time in Figure IT
additional 30 runs for each algorithm each game using that algorithm's
F.6 COMPUTATIONAL RESOURCES
For this paper, used approximately eight CPU years of compute general purpose CPU cluster with modern hardware. We did not use GPUs for any experiment, nor other specialized hardware for training our models. We used the Torch library (Paszke al., 2019) for defining neural networks and autodifferentiation for the nonlinear control experiments, and used the numpy library (Harris et al. 2020) for the linear prediction experiments.
23


Reviews:
[{"review_id": "hoIgxdYFL1W", "reviewer": "Reviewer_VsHs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* A nicely executed paper that starts from a simple premise and connects several prior ideas together -- namely the TDC/GTD2 algorithms with the conjugate formulation of Bellman error minimization and uses this to derive a more robust versions of the aforementioned algorithms.\n* Empirical results investigate both a simpler setup with linear prediction experiments as well as a more end to end non-linear control experiment to validate the practical benefits for policy optimization. \n* One of the novel conceptual contributions in this paper, appears to be making a connection between the auxiliary model $h$ used in the conjugate formulations of the Bellman error with the very differently motivated \"secondary weights\" model in GTD2 and TDC algorithms, where the additional model $h$ in both cases can be viewed as attempting to predict a residual error corresponding to the \"primary weights\" of the value function. \n\n\nRequest for clarifications / other comments for improvement:\n* For the empirical results on non-linear control problems (Figures 4, 5 and Table 1), it is somewhat unclear how much of a role is played by the auxiliary variable learning mechanism versus simply changing the loss from squared error to huber loss. For example, it would be interesting to see how a more naive baseline like semi-gradient update corresponding to the DQN + with Huber loss metric instead of DQN + squared error performs alongside the other three curves in each of the environments. \n* The main algorithmic proposal is based on the closely related prior works TDC, GTD2 and QRC, yet this aspect is somewhat buried until fairly late in the paper. \n* For the last term in Theorem 3.2, $p_\\tau$ is applied to the vector $\\mathcal{T} v - v$, but without specifying any reduction operation -- looking at the proof, it seems to me that there might be a need to add a max operation over the states to this bound. Could you please confirm? The combination of the multiplicative term involving a matrix inverse norm and the max operation over the states above makes this a potentially fairly weak bound in practice.\n* Another case of using an auxiliary model that learns to predict the TD errors has also been proposed for improving learning in actor-critic models, e.g. \"Characterizing the gap between AC and Policy gradient\", Wen et. al., ICML 2021.\n\n\nMinor comments\n* In Equation (4), the notation for state seems inconsistent between t and t+1 across $s, S'$.\n* In the equation for QRC-Huber updates for the auxiliary variable, there is some inconsistent notation across the references $\\theta_h, \\theta_{ht}, \\theta_{h, t+1}$. Also, the variable $\\beta$ in the secondary variable update needs a definition or reference of some sort.\n", "summary_of_the_paper": "This paper starts with the premise that squared error minimization, despite its wide use, might not be the most effective option for learning value functions. The authors hypothesize that this might be because of squared error's emphasis on outlier states where the bellman error is large at the expense of accuracy on other states. To address this, they consider the absolute value and huber errors alongside the squared error objective and propose a saddle-point reformulation for these objectives that requires learning an auxiliary learned state function which, essentially, attempts to predict the residual error at each state.\n\nBased on this motivation, starting with Section 4 the authors then make a connection of the proposed robust loss framework involving a learned auxiliary state function with prior work on algorithms for improving TD learning, namely, GTD2 and TDC.  Empirical analysis covers both prediction tasks as well as control tasks. In the prediction experiments, evaluations are conducted over carefully designed synthetic problems with linear function approximation in order to highlight particular challenges for each objective. These demonstrate that the Huber objective can often improve the prediction accuracy over squared error over a wide range of step sizes (learning rates).\n\nFor the empirical investigation on non-linear control problems, the authors consider a Huber-extension of the previously proposed QRC algorithm (which itself extends the TDC update to be considered in conjunction with the DQN target definition) and demonstrate that QRC-huber can improve on baselines in certain environments while being competitive elsewhere. These algorithms are also validated on a mini version of the Atari domain called Minatar. ", "summary_of_the_review": "This is a nice paper that makes a novel connection between the secondary variable update in GTD2/TDC with the conjugate formulation of Bellman errors involving an auxiliary state function (both of which involve predicting the residual error with a separate model). While the technical contributions in Section 3 (e.g. Theorem 3.2) are not particularly significant, I believe the main value in the paper is the conceptual linking of two different lines of work to derive an improved algorithm over well motivated baselines. The empirical evaluation is well motivated and quite thorough, even if only for a limited set of benchmarks.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "e6WdELwe5VS", "reviewer": "Reviewer_xg5t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this paper is interesting and is well motivated. My main concern is about its novelty. The authors highlighted that their main contribution is the introduction of two novel robust losses which are reformulated based on the least absolute loss and the Huber loss. This leads to the problem: why specifically the two losses? It seems to me that the comments and the analysis of this paper may also apply to other convex Lipschitz losses. It seems to me that it is the Lipschitz constant of the loss that \"change the solution quality\". I would expect more comments in this regard in the paper. \n\nIn addition, as a minor comment, the presentation of the paper could be further improved. For instance, the abbreviations, MHBE and MABE, were used without being defined beforehand.    ", "summary_of_the_paper": "In this paper, the authors studied robust losses for learning value functions in reinforcement learning. The main contribution of this paper lies in the development of two novel robust loss functions for reinforcement learning and a saddle point reformulation based on the Huber and the absolute Belleman error and the biconjugates. ", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z_ONYlv7N68", "reviewer": "Reviewer_78BK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The robustness is important in RL learning. The MHBE defined in this paper is inituitive. And the authors also develop reformulation to solve it in practive.", "summary_of_the_paper": "The paper proposes a mean Huber value error in the TD-learning. And the paper demonstrates the robustness under such loss.", "summary_of_the_review": "The major contribution is to develop a new type of loss, which is not sufficient according to the ICLR standards. \n\nMy main concern is that the contribution is not sufficient. I think the main contribution of this paper is the introduction of a robust loss. The conjugate reformulation is standard. And the bound developed in Theorem 3.2 is not surprising.\n\nThe writing is also not clear. For example, MHBE and MABE are never defined.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9BIEM2jutmr", "reviewer": "Reviewer_9FeJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.  The main contribution of this paper is using Huber Bellman error instead of mean squared Bellman error, since using the conjugate function to solve double sampling has been proposed in [1].\nWeakness. In the last paragraph of page 8, it said that it applied QRC without a target network. Would the performance of QRC become better and outperform QRC-Huber when combined with the target network?", "summary_of_the_paper": "This paper proposed using Huber Bellman error to robustify the loss function in learning value function and proposed using conjugate function to avoid double sampling. It also conducted experiments to justify its algorithm.", "summary_of_the_review": "This paper is novel in the sense that it cooperates Huber loss with value evaluation to robustify value evaluation. It also conducts experiments to justify this improvement. Therefore, I think this paper is marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use the Huber and absolute loss for value function estimation in reinforcement learning, and optimizes it by leveraging a recent primal-dual formulation by Dai et al. 

This is a controversial paper. On one hand, it is a well motivated idea to apply robust loss on RL; the paper implemented the idea well by leveraging the saddle point formulation, and empirically demonstrate its advantages in practice. 

On the other hand, the technical novelty of this paper is limited. The idea of Huber and standard conjugate formulation are straightforward application of existing techniques (despite being well motivated). 

The authors seem to think that there has been no application of Huber loss on RL. But existing implementations of RL already uses Huber loss. For example, in the openAI baselines (https://openai.com/blog/openai-baselines-dqn/), they said the following: 

"Double check your interpretations of papers: In the DQN Nature paper the authors write: “We also found it helpful to clip the error term from the update [...] to be between -1 and 1.”. There are two ways to interpret this statement — clip the objective, or clip the multiplicative term when computing gradient. The former seems more natural, but it causes the gradient to be zero on transitions with high error, which leads to suboptimal performance, as found in one DQN implementation. The latter is correct and has a simple mathematical interpretation — Huber Loss. You can spot bugs like these by checking that the gradients appear as you expect — this can be easily done within TensorFlow by using compute_gradients." 

The authors discussed the first approach above on in the rebuttal, but I am not sure if the authors have considered the second method. If not, it would be worthwhile to discuss and compare with it.  

See also "Agarwal et al. An Optimistic Perspective on Offline Reinforcement Learning" and "Dabney et al. Distributional Reinforcement Learning with Quantile Regression."

On the other hand, I have not seen the application of saddle point approach by primal-dual method of Dai on Huber specially. 

It seems that the proposed algorithm is in the end equivalent to MSBE+primal-dual+ (h with softmax output). If it is that simple, I think it would help the readers to explicitly point this out upfront in the beginning (which is an interesting conceptual connection).  Because the primal-dual approach need to be approximate h with a neural network, the difference of the two methods is vague in the primal-dual space. 

A side mark:  when we say "an objective for which we can obtain *unbiased* sample gradients", i think that the gradient estimator of the augmented Lagrange is unbiased; the gradient estimates of MHBE and MABE are still biased.  

Overall, it is a paper with a well motivated and valuable contribution, but limited in terms of technical depth and novelty.
Decision: Reject