Raisin: Residual Algorithms for Versatile Offline Reinforcement Learning
Authors: Braham Snyder
Yuke Zhu
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
offline RL
residual algorithms
residual gradient

Abstract: The residual gradient algorithm (RG), gradient descent of the Mean Squared Bellman Error, brings robust convergence guarantees to bootstrapped value estimation. Meanwhile, the far more common semi-gradient algorithm (SG) suffers from well-known instabilities and divergence. Unfortunately, RG often converges slowly in practice. Baird (1995) proposed residual algorithms (RA), weighted averaging of RG and SG, to combine RG's robust convergence and SG's speed. RA works moderately well in the online setting. We find, however, that RA works disproportionately well in the offline setting. Concretely, we find that merely adding a variable residual component to SAC increases its score on D4RL gym tasks by a median factor of 54. We further show that using the minimum of ten critics lets our algorithm match SAC-$N$'s state-of-the-art returns using 50$\times$ less compute and no additional hyperparameters. In contrast, TD3+BC with the same minimum-of-ten-critics trick does not match SAC-$N$'s returns on a handful of environments.

Under review as a conference paper at ICLR 2023
RAISIN: RESIDUAL ALGORITHMS FOR VERSATILE OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The residual gradient algorithm (RG), gradient descent of the Mean Squared Bell- man Error, brings robust convergence guarantees bootstrapped value estimation. Meanwhile, the far more common semi-gradient algorithm (SG) suffers from well- known instabilities and divergence. Unfortunately. RG often converges slowly practice. [Bird (1995) proposed residual algorithms (RA), weighted averaging of RG and SG, combine RG's robust convergence and SG's speed. RA works moderately well in the online setting. We find, however, that RA works dispro- portionately well the offline setting. Concretely, we find that merely adding a variable residual component SAC gives state-of-the-en scores for about half of the D4RL gym tasks. We further show that using the minimum ten critics lets algorithm approximately match SAC-N's state-of- oate-ooftthearr art returns using 50x less compute. contrast, TD3+BC with the same minimum-of woa-m------e---titiiee trick does not match SAC-N's returns on many environments. The only hyperparameter we tune is our residual weight - we leave all other hyperparameters unchanged from SAC-N.
INTRODUCTION
Strong data scaling has given us baffling success in supervised learning. Offline reinforcement learning (offline RL) holds promise for RL to scale with that same success, among other benefits. Despite all the compelling motivations of offline RL (Levine et al.) 2020), we still lack simple, versatile, and computationally efficient solution. By versatile, we mean algorithms that attain high returns when trained on any of diverse range of datasets, such as data collected by greatly differing policies.
Arguably the simplest and most versatile approach thus far is SAC-N (n 2021), which uses the minimum of N critics instead of SAC's usual two critics. SAC-N achieves state-of the scores but, unfortunately, requires up to 500 critics for sufficient pessimism benchmark problems. Hu et al. (202)) illustrates that stronger pessimism, specifically a smaller discount factor, enables SAC-N to solve harder tasks (Rajeswaran al. 2017). A smaller discount factor simple and computationally efficient pessimism but not versatile it increases bias (Zhang et al. 2020).
In this paper, we identify residual algorithms (RA) (Barrd 1995) as simple, versatile, and computa- tionally efficient source of pessimism for SAC-N. As we explained, RA saw moderate success in its goal of fusing RG's convergence with SG's speed. Recently, Zhang et al. (2019) found similar success when extending RA to deep learning. But we find RA truly excels in the offline setting. Prior works in both the online and offline settings (Geist et al.) 2016 Fujimoto et al., 2022 Salch & Jiang 2019) show that, while RG performs well with data near the optimal policy, RG consistently fails when the data is far from the optimal policy. Our key insight is that RA allows for the adjustable exploitation of RG's natural pessimism. In other words, weighted RG component may serve superior alternative to the widespread use of a weighted behavior cloning component for offline RL (Fujimoto & Gu 2021 Buckman al. 2020). Critically, however, for the RG component universally works well: you must tune it per dataset, similar to SAC-N (An
also find that no single weight
et al. 202T We discuss potential routes for automatic tuning in Section 5
We propose Raisin, roughly RA for SAC-N, giving D4RL (Fu et al. 2020 gym scores roughly matching SAC-N the state-of-the-art with one-fiftieth of the critics. EDAC (An et al. 2021

---

Under review as a conference paper at ICLR 2023
matches those scores well but requires both five times compute than Raisin and adjustment of two hyperparameters per dataset. Raisin keeps the number of critics small and fixed and solely requires adjusting one hyperparameter for pessimism (the residual weight).
Raisin casily runs at the same speed with one critic as it does with its standard ten critics (on one GPU) thanks to embarrassing parallelization, similar SAC-N (An et al. 2021). We plan release a clean and efficient implementation of Raisin (and SAC-N) upon acceptance of this manuscript.
Meanwhile, TD3+BC (Fujimoto & Gul 2021) equipped with the same minimum-of-- Suaf-N-eritics tool (TD3+BC-N) does not appear versatile. It still does not match the scores of SAC-N on a few datasets (especially the random datasets) despite tuning the pessimism per dataset. Outside of fundamental offline RL research, such simple combinations of behavior cloning and reinforcement learning are debatably the most common approach in the literature (Humphreys et al. 2022 Baker al. 2022
Nakano et al. 2021
IQL computationally efficient and potentially versatile but needs more testing on suboptimal data (e.g., the random datasets of D4RL gym tasks). Maybe even more importantly, IQL is not simple. For example, IQL. uses a learning rate decay for its actor, rescales its rewards by 1000, clips advantages to 100, and adds a third learning rate. Each one of these components may be overfitting. Conversely, Raisin makes a single change to SAC-N, and SAC-N itself makes one change to SAC (Haarnoja et al. 2018), an extensively well-tested algorithm.
Upside-down reinforcement learning (UDRL) (Schmidhuber, 2019 Kumar et al. 2019) has shown some potential in recent work (Chen et al. 2021a Lee et al.) (202)) but scores poorly on suboptimal data (Brandionbrener et al., (022). Not to mention that it wastes capacity learning poor behaviors (Emmons et al. [202T) and that it kicks the maximization can down the road UDRL requires desired-return hyperparameter, a goal specification, or a complicated equivalent.
2 PRELIMINARIES
We consider the standard offline RI. setting of a fixed dataset 'D of transitions (s, r, 6'), where s is state, is action taken in that state, is the reward received for that action, and s is the new state. We denote our policy (-(s), whose goal is to learn actions that maximize the discounted sum of rewards E. 7"r, where 7 is the discount factor. (s,s, a) is our state-action value function, whose goal is to estimate the future discounted sum of rewards given state-action pair.
For comprehensive overview of residual algorithms (RA), see Zhang al. (2019). We will give brief primer here in the context of SAC. Ignoring discounting for simplicity, SAC's ' loss is:
[
min -
L(O, = E
where 0; refers to the parameters 0 of the ith of two Q networks; and y. the next-state target before the minimum operation, is:
g r, r') r (Q0, (d',a') log T (ā'ls'))
ā' ~ T-(-ss).
Q denotes the target network (Mnih et al. 2015), and a is SAC's entropy coefficient.
This critic loss derives from the Mean Squared Bellman Error (MSBE). natural error function for bootstrapped value estimation. But SAC's critic does not quite optimize the MSBE, partly because SAC, like all common value-based RI. algorithms using gradient descent, ignores the gradient contribution from the next-state term. In other words, gradient descent of SAC's critic loss treats the value of the next state (plus the reward) as a fixed target towards which the current state's value is stepped. This is known as the semi-gradient (SG) algorithm. In contrast, performing true gradient descent of the MSBE is called the residual gradient (RG) algorithm. As a true gradient algorithm, RG brings robust convergence guarantees. But, thus far, RG has empirically obtained poor returns and at slow speeds. It also suffers from a few theoretical concerns (Baird 1995) Sutton & Barto 2018): double sampling bias (for which we discuss workarounds in Section
convergence unsatisfactory
2

---

Under review as a conference paper at ICLR 2023
values in certain stochastic settings with function approximation; and non-identifiability (though one contribution of Patterson et al. (2021) is an identifiable MSBE). SG typically obtains higher returns (when it converges) despite its lesser convergence guarantees and instabilities. For these reasons, SG used in practice.
In hopes of the best of both worlds, [Birdd 1995)) proposed RA, weighted averaging of RG and SG. Zhang et al. (2019) extended RA to DDPG, introducing bidirectional target networks as an intuitive analog to the target networks commonly used for SG: since RG uses the current state and next state symmetrically for the gradient, symmetry of the current state and next state for RG's target network likely makes sense as well. Zhang et al. (2019)'s 's critic loss for residual DDPG with bidirectional target networks is:
(90sss0)- vlr.s)) +n
" -0(00ss))- - v(r 27
L(O,D)=
E (aa
where 1/ E [0, 1] controls the weight for backwards bootstrapping: and y. unlike V. uses the main Q
network instead of Q. Both and ÿ otherwise follow DDPG (Lillicrap et al. 2015).
RAISIN
SAC-N's impressive versatility comes at the cost of massive ensembles in some environments. And as discussed in the introduction, it appears likely that even more pessimism may give SAC-N yet more versatility.
Aiming to solve both challenges, introduce Raisin, RA adapted to SAC-N. We incorporate Zhang ct al. (2019'ss 's bidirectional target networks, taking the minimum of critics in the target network terms, creating the critic loss (for the i-th of the N critics):
(amo)-mmnnnie min 10000mmmmmn..... v...))))) (00_
+ ( min (s,a) (0)) v(i,r, (')
27
I(O,, =
E )
where j E I,...,, N: n [0, 1] again controls the weight for backwards bootstrapping: and again y.
unlike D. uses the main Q networks instead of the target networks Q:
Mi,r,s') = r (Qo.('',') - log (ā''s'))) ä' ~ (o(lss)))
Referring Raisin RA SAC-N is only rough explanation we Raisin is not the only possible way to combine RA and SAC-N. For example, we only briefly tested the alternative minimum placement (Q- min O² (Q min y)2. (Where step min in the second term towards the average of Q.) There are certainly reasons to think that formulation could be worse than Raisin (for example, might reduce the possible range pessimism), and Raisin indeed outscored that algorithm in preliminary experiments (not shown). That said, we don't t yet consider alternative RA SAC-N approaches such that one fully ruled out.
We have not yet considered whether ä' ought to be sampled twice, independently for and ğ (or independently for each critic, for that matter). In our experiments, we only sample it once, which empirically works well.
Raisin's policy update otherwise follows SAC (and SAC-N), maximizing only the minimum critic estimate in the policy loss. [Cheng et al. (2022) observed that, with their method, such an SAC-style policy loss resulted potential for this issue in Raisin or SAC-N, but we did test TD3-style policy optimization (maximizing a fixed, arbitrary critic in the policy loss rather than the minimum critic) as they use, finding not to help. We additionally tested maximizing the median, the maximum, the total average, and order statistics of the critic estimates the policy loss, in various combinations with similar approaches to the critic loss. None worked significantly better than Raisin. This aligns with Bai al. (2022)'s
limit cycle between their crities. We not thoroughly investigate the
3

---

Under review as a conference paper at ICLR 2023
Table Score on D4RL v2 environments, mean over four seeds. For readability, scores are rounded to whole numbers, and standard deviations to two digits. Scores 2 90% of the best score are highlighted, BC scores are taken from the SAC-N paper. IQL scores were taken from its paper. SAC-N and SAC-10 scores taken from the SAC-N paper where applicable. SAC-N uses up to N 500 critics. We use 10.
SAC-10
TD3+BC-10 (N 10)
SAC-N (N S 500)
Raisin
Task Name
BC
IQL
(N
10)
(N 10)
hallcheetab-rrndom
0.0
14
28 0.9
30 2.1
28
= 0.9
28 0.9
hopper-random
0.6
= 0.7
17 +
= 0.0
#.1
walker2d-random
+ 1.0
0.0
18 9.2
halfcheetah-meedum
43 + 0.6
47
68
= 1.2
49 = 2.8
68
= 1.2
68 1.2
hopper-medium
54 +
66
# 1.0
87 * 10
100
= 0.3
21
walker2d-medium
11
78
95
1.4
83 6.7
95 1.4
halfcheetah- seachh--eiiimm--pee replay
38 2.1
44
64
0.8
44 + 1.4
64
0.8
64
0.8
Supper-medium-rella
17 +
95
102
= 0.3
102
= 0.5
102
walker2d-mediun--rpla
20
9.8
74
83
1.1
76
secchhetthh-mediu---ppeee
44 1.6
87
107
- 2.0
91 3.7
107
- 2.0
107
- 2.0
topper-mediuu--eppet
4.7
92
8.0
= 0.2
110
111
walker2d-medium expert
88
+ 58
+ 2.0
117
103
hallcheetah-expprt
92 1.5
94
105
2.6
91 - 0.8
105
- 2.6
105
2.6
hopper-expert
108
9.7
= 0.0
112
= 0.2
110
0.3
110
0.4
walker2d-expert
109
0.2
108
2.8
110 = 0.3
107
2.4
109
0.3
comparable conclusions regarding their policy loss. From the other direction, we ran preliminary tests of TD3+BC with an SAC-style policy loss (maximizing the minimum critic) that did not appear to improve TD3+BC.
With fixed 7 = 0, and our default 10 critics, Raisin would be exactly SAC-10 (SAC-N with N 10). When 7) 0 and N Raisin equates SAC.
With a fixed = 1, Raisin could roughly be interpreted as RG-SAC-N. We find RG-SAC-N underperforms certain datasets, such saffcheetah-randdom This supports the conclusion of some works we discuss in Section namely that RG may perform poorly when the data is far from the optimal policy. Pure RG's poor performance persisted with extensive, combinatorial tuning, including various approaches attempting to increase optimism. We give more details in Appendix
We give the complete pseudocode for Raisin Algorithm
4 EXPERIMENTS
In all of our experiments with Raisin, do not attempt to tune any hyperparameters other than pessimism. That is, we use the default hyperparameters of SAC-N, which uses the default hyperpa- rameters of SAC aside from increasing the number of layers (for both the actor and critic) from 2 to 3, following e.g. COL.
Like SAC-N and ATAC (Cheng al. 2022 per dataset crucial for performance, so we study that setting, tuning for both Raisin and the new baseline introduce below. SAC-N's experiments tuned the same way.
find that tuning the pessimism (in our case 7))
4.1 RAISIN- 10 vs. TD3+BC-10
Fixing N = 10, and giving TD3+BC the minimum-of off-oof-iiiieee N-critics trick as well, we find that Raisin generally outscores TD3+BC-10 (even when tune each method's respective pessimism each of
4

---

Under review as a conference paper at ICLR 2023
Table .V 2 scores. SAC scores from An al. (2021). TD3+BC scores (for environments) are taken from the TD3+BC appendix. Unlike our main experiments with TD3+BC-IB 10 above, this experiment is an unfair comparison Raisin-2's favor in that TD3+BC scores here use only 1M gradient steps and single pessimism setting. But the TD3+BC paper's (Fujimoto & Gu 2021 Figures and might imply neither difference would help TD3+BC as much as one might think.
Task Name
TD3+BC
Raisin-2
SAC
halfcheetah-raaddom
11
# 1.1
30
1.4
30 + 1.4
hopper-random
9 0.6
31
+
10 + 1.5
walker2d-random
2 # 1.7
7
+ 7.8
# 0.8
halfcheetab-medium
48 0.3
63
0.4
55 28
hopper-medium
59 4.2
71 29
0.0
walker2d malker2d-medium
84
2.1
89
1.0
0.2
halfchhetta--meddun--ellp
45
+ 0.5
54
+ 2.0
= 1.0
hopper-medium-rellyy
61
19
102
1.2
0.5
malker2d-medium-eee
82
5.5
90 4 5.9
0.3
hallichettb--mediuu--nneeer
4.3
50 38
28 19
hopper--mediun-expeett
98
9.4
80 + 57
0.0
walker2d meeprdd-mmmiiieepeeer
110
0.5
115 1.1
3.9
halfeheetah-expen
97
1.1
43 + 24
-1 + 1.8
hopper-expert walker2d-expert
108
7.0
31 45
+.0
110
0.3
45 + 50
1.3
the datasets). We show this in Table T. For example, Raisin scores 90% or higher than the best score twelve tasks, whereas TD3+BC-IB only achieves this for seven tasks.
An et al. 2021 echoes this finding, where CQL does not match SAC-N's performance even when CQL is given the N critics trick, and the pessimism (N) is tuned per dataset. Their results with CQL-N and our results with TD3+BC-10 show that the N critics trick is no silver bullet for all offline RL algorithms.
As An et al. (2021) found with SAC-N, we find Raisin requires longer to converge on certain environments, so, like them, we run our experiments (Raisin and TD3+BC-10) for 3M gradient steps.
We give the pessimism settings for each method in Appendix
4.2 ENSEMBLE ABLATION: RAISIN-2 (N =
We find Raisin with N 2, which we call Raisin-2 (approximately SAC with residual algorithms and bidirectional target networks), surprisingly improves SAC's D4RL gym scores by a median factor of 54. We show those scores in Table Raisin-2 gives a competitive offline RL algorithm its own, with negligible additional computation but Raisin shines best with scaling (as shown in Table I
Along with the results of the N 10 experiments, this experiment shows that the whole of Raisin greater than the sum of its parts, and that both the RA and SAC-N aspects are necessary. Raisin without SAC-N performs poorly at both -expert and medium-expee exppert datasets, and SAC-10 catastroph- ically fails at e.g. hopper-expert, but their careful combination matches the state of the art. The subpar scores of TD3+BC-10 10 show that such combination does not always work well.
While we again use 3M gradient steps for this experiment, Raisin-2 looks like it would benefit from more gradient steps. We include learning curves in Appendices and D
5

---

Under review as a conference paper at ICLR 2023
10"
70
0%
10
25%
60
50%
10 10
50
75%
Mean Targets
Score
40
100%
10
0%
25%
50%
16
30
75%
100%
10
10"
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Gradient stop
Gradient step
Figure 1: Left: Raisin-1 (N scores hopper-expent with 1M gradient steps, varying 7, the residual weight. = 100%'s massive improvement over 0% suggests our second minimum operation is not the only driver of Raisin's pessimism. Right: Average targets for Raisin-l on hopper-expert, showing that higher avoids unreasonably large Q values. The convergence of r7 0% might also imply that instability alone does not account for the failure of SG.
4.3 WIIY ARE RAS ROBUSTLY PESSIMISTIC HERE? (N 1)
To knowledge, are the first to show that RA can induce robust pessimism, in the sense that RA can empirically turn an online algorithm into strong offline algorithm. In these two experiments, we test two hypotheses for this robust pessimism.
Is it the second minimum? We first test the hypothesis that RAs are robustly pessimistic here primarily due to the second minimum operation we introduced, the current state in the backwards bootstrapping term: min, Qo, (s, a). To do so, we run Raisin with N = 1, effectively removing both new minimum operation and SAC-N's original minimum operation. Even with N increasing the backwards bootstrapping weight drastically increases scores, as we show in the left plot of Figure
Figure (right) additionally shows that larger values decrease the Q targets. Notably, even v = 0% does converge (albeit to massive values) despite poor scores, suggesting RAs indeed play important role beyond improved convergence.
These results rule out the second minimum operation as being Raisin's primary source of increased pessimism.
Is it value function initialization? Wang Ueda 2021 argue that RG (recall that RG equivalent to RA with 100%) has tendency to maintain the average prediction. They point out that, in the tabular case, the changes in RG's value prediction the current state and next state nearly cancel (dependent on the discount factor) when viewing the value predictions aggregated over all states and actions. They further argue that when Q" lies above Q's initialization (as it does our experiments), RG's performance hinges the exploration strategy chosen. This relates the earlier observations that residual algorithms can slow (Baird) 1995).
Accordingly, test the effects of varying the value initialization. With N = 1, increasing the value initialization indeed quickly gives worse scores, shown in the left Figure targets indeed remain nearly constant over gradient steps with larger initializations, shown in the right of Figure
2 And the average Q
Thus, first sight, negative rewards, along with standard near-zero initialization (Andrychowicz et al. 2020), might pose an issue for Raisin. However, returning to our standard N 10, Raisin show in the left of Figure
performs far more robustly,
Consequently, one might expect the corresponding Q targets to look like the N = 1 case but to and more of peculiar rebound effect from
just decrease faster towards Q". Instead,

---

Under review as a conference paper at ICLR 2023
50
Q-Value Initialization
400
350
40
100
200
300
300
Mean Q Targets
30
400
250
Score
200
Q-Value Initialization
150
100
100
10
200
50
300
400
0
0.0 0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Gradient step
Gradient step
Figure 2: Left: Raisin-1 with 7) 100%, scores hopper-expert when varying the value initializa- tion. Scores deteriorate quickly as the initialization increases, For reference, Raisin-10, 10, TD3+BC-10, and SAC-N all achieve average returns of roughly 3,500 on hopper-expert (normalized scores of 110). N = 1 is not robust to large initializations. Right: Raisin-1 with 100%, average Q targets on exppet when varying the value initialization. The slow rates of change align with Wang Ueda 202 's speculation that their argument extends to the function approximation case.
excessive optimism towards excessive pessimism as we increase Q's initialization, all despite the decent returns. We show this in the right of Figure
Granted, both SAC-N and Raisin sometimes exhibit similar behavior high scores with excessively low Q targets even without modifying Q's initialization. For example, we reliably see this with the standard SAC-500 to solve hopper-expert (not shown).
ran limited tests at large value initializations with re-tuning of 7,, further improving performance. Moreover, removing the need to tune 7) may be within close reach through recent work related to RG, some of which nearly eliminated the issue already, and there are clear routes for additional improvements to Raisin's computational efficiency, such as DroQ (Hiraoka al. 2021) and BatchEnsemble (Wen et al. 2020).
In any
discuss in the following section. Not to mention, larger N
These results suggest value function initialization may play a key role in Raisin's pessimism. However, future work in this direction should start by examining in-depth the strange Q-target overshooting effect of SAC-N on its own. We speculate that using independent targets might be important here, following Ghasemipour et (2022).
Is it fixed points? Xiao et al. (2021)'s 's findings suggest that RG and SG may converge to different fixed points in the overparameterized setting. We hypothesize this may also partly explain RG's apparent pessimism. To investigate, we train RG (n 100%) and SG in new runs starting from high-scoring model weights. When N weatchhheta--ranndoo respectively. We then restart training from those high-scoring weights using both RG and SG on each environment. On hopper-expert, SG trained from RG's high-scoring weights reverts to low scores, whereas new RG runs trained from those same weights keep their high scores. Similarly, on halfcheetah-randonn whereas new SG runs starting from those same weights keep their high scores. Our results provide evidence in favor of fixed points playing an important role. We give more details in Appendix
RG and SG attain high scores on hopper-expert and
RG trained from SG's high-scoring weights falls to low scores,
RELATED WORK
ONLINE RESIDUAL APPROACHES
Bi-Res-DDPG (Zhang et al. 2019) reviews RG, SG, RA, and their respective fixed points. They introduce bidirectional target networks for residual algorithms in the context DDPG. Bi-Res- DDPG shows up to 3x improvement in AUC (area under the curve of returns) over DDPG on the
7

---

Under review as a conference paper at ICLR 2023
10
100
10
Mean Q Targets
-10
10
0
Q-Value Initialization
Score
60
2000
-10
4000
40
Q-Value Initialization
6000
-10
0
20
2000
-10"
4000
8000
0
-10"
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5 Gradient step
2.0
2.5
3.0
Gradientt step
1e8
1e6
Figure 3: Left: Raisin-10 with 77 20%, scores hopper-expert when varying the value initializa- tion. Right: Raisin-10 with rj 20%, average Q targets on hopper-expert when varying the value initialization. Note the strange overshooting effect higher value initializations, even though such agents often achieve high scores. We do sometimes observe this even with the standard initialization of 0, for example, SAC-N on hopper-expert with 500 crities, as suggested by An al. (2021)
DeepMind Control Suite TTssa et al., 2018). Whereas we identify RA as natural fit for offline RL, propose what can be seen as Bi-Res-SAC, and show a median 50x improvement in final scores over SAC. We further show that the minimum-of-F -ffiimmumoo-----oooo -oof---critte N-critics trick synergizes with our algorithm particularly well, which lets us match SAC-N with a tiny fraction of the ensemble size and outscore TD3+BC given the same N crities trick. Lastly, we also delve into the sources of our robust pessimism.
(Wang & Ueda 2021) minimizes the maximum of RG's loss and SG's loss. Similar to RA, they claim this captures the best of both worlds, convergence and efficiency. They do not compare against RA, and they study only the online setting. A C-DQN-like approach might offer a way to avoid the need tune 77 or some equivalent. They additionally analyze potential failure modes of RG, as we discussed earlier.
C-DQN
5.2 OFFLINE RESIDUAL APPROACHES
ATAC (Cheng et al. 2022) introduces an adversarially trained algorithm with strong theoretical backing. They also incorporate R.A fixed, equal weighting of RG and SG, using it for stability on top of their offline RL method. In contrast, we use a variable RA to control pessimism, showing among other things that RA integrated into SAC dramatically improves its returns in offline RL when v) is set appropriately.
DBRM (Saleh & Jiang 2019) finds that RG fails at simple offline RL benchmarks when given the tajectories of a Q-learning agent trained online. This holds even with a fully deterministic environment (i.e., no double sampling bias, explained below), and despite the fact that Q-learning (a new agent trained offline on the same dataset) performs fairly well. They propose and test higher norms of the Bellman error to penalize larger residuals more, which requires more optimization steps but gets slightly better performance. It does not match SG's performance. This experimental setting similar to D4RL's -eeiimm replay and full-replay datasets, where we find intermediate values of v) to significantly outperform 1/ E (0. 13.
Xiao et al. (2021) analyzes the fixed points of RG and SG in the overparameterized linear setting, finding they converge to different fixed points, potentially calling for increased caution around the common discussion point that overparameterized neural networks are capable of minimizing the MSBE perfectly by minimizing its projection (Fujimoto al. 2022). the Mean Squared Projected Bellman Error (MSPBE). They further introduce two regularizers for SG and RG, again in the hopes of obtaining the best of both worlds. This is another potential avenue for replacing the need to tune for Raisin, or for ppgrading/removiing Raisin's 's SG component (which, as with the most commonly used RL algorithms, may cause divergence).
8

---

Under review as a conference paper at ICLR 2023
5.3 DOUBLE SAMPLING BIAS
In stochastic environments, the algorithm we have presented looks more like the naive residual gradient (naive RG) algorithm (Sutton & Barto 2018) than the true RG. Naive RG arises when applying RG without a second, independent sample for all stochastic transitions. Naive RG, a biased estimate of true RG, no longer minimizes the MSBE. As such, naive RG can lead to poor solutions in certain stochastic environments.
Our understanding is that MuloCo (Todorov et al. 2012) (at least in the context of the environ- ments of gym (Brockman et al. (2066), which we use in this paper) is stochastic (OpenAl 2022] DeepMind] 2022). Although Fujimoto et al. (2022) claims otherwise, perhaps because they assume the stochasticity minor enough to ignore. That being said, we follow Zhang et al. (2019) in ignoring this stochasticity, which empirically is not a blocker for strong performance. Granted, this suggests properly handling the double sampling bias could lead to yet stronger performance for Raisin. Many prior works, such as K-loss (Feng et al. 2019) and SBEED (Da et al. 2017), have already tackled this issue.
5.4 MISCELLANEOUS RELATED WORK
Estimation Bias and The Update-to-Datee Ratio. Offline RL and SAC-N, in particular, have clear ties to the literature on estimation bias in bootstrapped value learning, such as the many works making trick, and/or high update-to-dan (UTD) ratios (of which offline RL the extreme point). This includes methods such as Maxmin Q-Leamnng (Lan al. 2020). REDQ (Chen al. 2021bb, TQC (Kuznetsov et al. 2020), and more recently DroQ (Hiraoka et al. 2021), which have led impressive results in real-world reinforcement
use of ensembles in ways similar to the Mfnimum-off-N-rriino
learning (Smith al.) 2022).
Theoretically Sound Algorithms. Works such as TDC (uttonn et al. 2009). SBEED (Dai et al. 2017), and QRC (Patterson et al. (2221) gradient corrections build convergent online RL algorithms. Gradient correction algorithms can be seen as approximations to gradient descent of the MSBE (Patterson et al. 202I), which Raisin approximates believe our empirical results may help further develop such sound algorithms, especially in the offline setting. Additionally, the adaptivity of these algorithms might point similar adaptive approach that obviates Raisin's 's need to tune
well (in much different manner). We
Recent Criticism of The Bellman Error. Fujimoto et al. (002)) uses both theory and experiments to point out drawbacks of the Bellman error, primarily its deficiencies concerning the value error. However, they conclude that the Bellman error can be a usable objective for on-policy evaluation. Geist et al. (2016) proposes method for minimizing the Bellman residual through policy search (tan- gentially circumventing double sampling bias in the process) to more fairly compare the minimization of the Bellman residual versus direct maximization of the estimated value. They theoretically and empirically analyze the two approaches, similarly finding that their minimization of the Bellman residual performs poorly when the optimal policy is far from the data.
Pessimism. Buckman et al. (2020) argues that offline RL algorithms that induce pessimism by penalizing deviations from the policy that collected the data may be strietly worse than uncertainty- based approaches. This may further support the notion that Raisin and Raisin-like approaches are a more promising approach than TD3+BC and TD3+BC-like approaches.
6 CONCLUSION
Prior offline RL algorithms are either not versatile, not simple, or computationally inefficient. Raisin works datasets collected by diverse behavior policies, consists of two simple changes to SAC, and requires only two percent of the compute SAC-N despite state-of-the scores.
Important future work includes investigating automatic alternatives to tuning 77. evaluating more challenging benchmarks, integrating a sound alternative to SG (though nearly all other RL algorithms used in practice also based on SG), and further investigations into why Raisin works well.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- Seaettiitty-Baeed Offline Reinforcement Learning with Diversified Q-Ensemble. arXiv, October 2021. doi: 10.48550/arXiv. 2110.01548.
Marcin Andrychowicz. Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthicu Geist, Olivier Pictquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study. arXiv, June 2020. doi: 00-48500/mXxvv220060599999
Chenjia Bai, Lingxiao Wang. Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic Bootstrapping for Uncertainty-Drivee Offline Reinforcement Learning. arXiv, February 2022. doi: 10.48550Warxiv.2222..5555 1566.
L. Baird. Residual Algorithms: Reinforcement Learning with Function Approximation. 1995. URL https: www. www.seeaanniccccceee http:schco..aaooooooaa/////ea/aa www.eiuaaaaalliiaaaaeeee -Algrrithms
Bowen Baker, lgge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video PreTraining (VPT); Learning Act by Watching Unlabeled Online Vidcos. arXiv, June 2022. doi: 10448550/rXx..2200..799
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conme centut-cond supervised learning work for offline reinforcement learning? arXiv, June 2022. doi: 10.48550/arXiv.22066
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Jacob Buckman, Carles Gelada, and Marc G. Bellemare. The Importance of Pessimism in Fixed- Dataset Policy Optimization. arXiv, September 2020. doi: 10.485500axxx.220000079999
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. arXiv, June 2021a. doi: 10. 00-485500/rXii222060014444
Xinyue Chen, Che Wang. Zijian Zhou, and Keith Ross. Randomized Ensembled Double Q-Learning Learning Fast Without a Model. arXiv, January 2021b. doi: 10. 00.485500arxx1..101.05599
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially Trained Actor Critic for Offline Reinforcement Learning. arXiv, February 2022, doi: 10.48550WarXx::222222444
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation. arXiv, December 2017. doi: 1tt.500/aaXXxx.11.112222...
DeepMind. dm control, September 2022. URL control/issues//
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. RvS: What Essential for Offline RL via Supervised Learning? arXiv, December 2021. doi: 10448500arXix:2112.0055
Yihao Feng, Lihong Li, and Qiang Liu. A Kernel Loss for Solving the Bellman Equation. arXiv, May 2019, doi: 10.48550/arxiv. 10506.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep DataeDriven Reinforcement Learning, arXiv, April 2020. doi: 10-48550karxxv.200447771
Scott Fujimoto and Shixiang Shane Gu. Minimalist Approach to Offline Reinforcement Learning. arXiv, June 2021. doi: 00448500/axxiv22166.666
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error. arXiv, January 2022. doi: 2..85550arXxi.2201..2222
10

---

Under review as a conference paper at ICLR 2023
Matthieu Geist, Bilal Piot, and Olivier Pietquin. Is the Bellman residual a bad proxy? arXiv, June 2016. doi: 10.48550/arXiv. 0tw.://.........0.6......3 1606.07636.
Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters. OpenReview, October 2022. URL https: /tttss://openreveew net fortm/foumm?m
264kNih1-rR
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critice Off-Policy Maximum Entropy Deep Reinforeement Learning with Stochastic Actor. arXiv, January 2018. doi: 10.48550/arXiv. 1801 .012900.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout Q-Functions for Doubly Efficient Reinforcement Learning. arXiv, October 2021. doi: 10.485500arXiv.11 10..2034.
Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. On the Role of Discount Factor in Offline Reinforcement Learning. arXiv, June 2022. doi: 10448550/a2X1.2220003338
Peter C. Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap. data-driven approach for learning to control computers. arXiv, February 2022. doi: 10.48550/ arXiv.2202.081327
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-Conditioned Policies. arXiv, December 2019, doi: 111155000rXXv..112.13......
Arseníi Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling Overesti- mation Bias with Truncated Mixture of Continuous Distributional Quantile Critics. arXiv, May 2020. doi: 10448550/axx..2200..4777
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning: Controlling the Estimation Bias of Q-learning. arXiv, February 2020. doi: 10:4850WaXX..2002.06533
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-Game Decision Transformers. arXiv, May 2022. doi: 10448550/aXX::2225.522.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives Open Problems. arXiv, May 2020. doi: 10.48550/arXiv. 2005.01643.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv, September 2015. doi: 155550/rrXii.100.0290000
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik. Ioannis Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533 533, 2015.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answering with human feedback. arXiv, December 2021. doi: 10. 1S550/arXiv.2112.0999
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep Double Descent: Where Bigger Models and More Data Hurt. arXiv, December 2019. doi:
22.S50ArrXiv.1112.22299
OpenAl. gym, September 2022. /ttpa///////////ttooooo////////a
11

---

Under review as a conference paper at ICLR 2023
Andrew Patterson, Adam White, and Martha White. A Generalized Projected Bellman Error for Off-po Value Estimation in Reinforcement Learning, arXiv, April 2021. doi: 10.48550/arXiv. 2104.13844.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforeement Learning and Demonstrations. arXiv, September 2017. doi: 10.48550/a2xivv1709..000
Ehsan Saleh and Nan Jiang. Deterministic bellman residual minimization. In Proceedings of Optimization Foundations for Reinforcement Learning Workshop NeurIPS, 2019.
Juergen Schmidhuber. Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions, arXiv, December 2019. doi: 104445500arrxiv11112222555
Ralf Schoknecht and Artur Merke. Td (0) converges provably faster than the residual gradient algorithm. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 680-687, 2003.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A Walk in the Park: Learning Walk in 20 Minutes With Model-Free Reinforcement Learning. arXiv, August 2022. doi: 10.485500/rxX..22288..oo 077660.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An Introduction. MIT press, 2018.
Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for lemporal-difffrence learning with linear function approximation. In Proceedings of The 26th Annual International Conference On Machine Learning, pp. 1000, 2009.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite. arXiv, January 2018. doi: 10.48550/arXiv. 801,00690.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Zhikang T. Wang and Masahito Ueda. Convergent and Efficient Deep Q Network Algorithm. arXiv, June 2021, doi: 10.485550/rxii.2106... 15419.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemmle: An Alternative Approach to Efficient Ensemble and Lifelong Learning. arXiv, February 2020. doi: 10.48550arxxiv22022.67155
Chenjun Xiao, Bo Dai, Jincheng Mei, Oscar A Ramirez, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Understanding and leveraging overparameterization in recursive value estimation. In International Conference on Learning Representations, 2021.
Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Deep Residual Reinforcement Learning. arXiv, May 2019, doi: 10448500axXi.1905.00022
Shangtong Zhang, Romain Laroche, Harm van Seijen, Shimon Whiteson, and Remi Tachet des Combes. A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms. arXiv, October 2020. doi: 10.48550/axx..2210000069
12

---

Under review as a conference paper at ICLR 2023
A
PESSIMISM SETTINGS
Table 3: TD3+BC-10's multiplies Q in its policy loss (higher is more optimistic). Raisin's multiplies backwards bootstrapping term (lower is more optimistic).
Raisin: v
Raisin-2: (N = 2)
Task Name
TD3+BC-10:
(N 10)
halfche
150
0%
0%
hopper-random
20
10%
40%
walker2d-random
0.5
1%
20%
halfcheetah-meddimm
0%
20%
hopper-medium
10
10%
60%
walker2d-medium
10
0%%
20%
malffhhcttah-medium--a
2
0%%
20%
hoopper-mediu-rreppe
50
0%
20%
walker2dd-mcdum-rrppl
8
0%
40%
meneehhccta--iiuum--e-- expert
2
0%%
0%
mopper-medium-ee expert
2
20%
40%
walker2d medium- expert
30%
20%
halfchectah-erte expert
0.5
0%
40%
hopper-expe expert
2
20%
80%
walker2d-expert
1
5%
60%
B ALGORITHM PSEUDOCODE
Algorithm Raisin. Differences from SAC highlighted.
repeat
2: Sample a batch B = {(s, a,,, from the offline dataset D
3: Compute the shared, pre-minimization, current-state targets Qo, (8, a) for j E EI....,N! NN.
4: Compute the shared, pre-minimization, next-state targets for j ,,...,,,.. N:
T. s') + (Qa, (s', ā') a log (ā'(ss))
ă' ~ TT(-15')
5: Update each critic Qa, descending its gradient:
Qo (0)(),,)) min ,lir,,s)) JEI....N
Σ
Q0, - ,((r,,s))
Ve.
+ n JEl.....
min
(aa)))))
where the next-state prediction is
y(i, s') (Qa, (s', a') log (ā)ls)))
ā' xT(-ss')
6: Update the policy Th ascending its gradient:
Σ
min Qø, (s, ās(s)) a log (ās(s) [s) $s)
B
j=I....,
SEB
where (s) is sample from o: which is differentiable w.r.t. ø via the reparameteriza-
tion trick.
7: Update target networks with o% pód + (1 p)
13

---

Under review as a conference paper at ICLR 2023
C RAISIN (N 10) LEARNING CURVES
Like An et al. 2021), we see strange, highly non-monotonic learning over gradient steps on some datasets. This occurs in the random. We speculate this might be related to epoch-wise double descent (Nakkiran et al. 2019), but it's unclear. We omit few curves where Raisin's 's optimal setting recovers SAC-N (r) 0), since we only re-ran enough of those experiments to verify we still saw the same behavior. (Our code is built on their SAC-N code.)
consistent manner the hopper datasets, particularly hopper-
popcer-eppet-d
120
120
120
130
52mm
walkorrd mwdli
hoopan- 2
120
WalkerDd med wwp.lup
100
1.5
3.0
randamir
Cirad
160
Girad
165
14

---

Under review as a conference paper at ICLR 2023
D RAISIN-2 (N = 2) LEARNING CURVES
With smaller N, the first peak is less sharp. For a similar depiction of this phenomenon as N varies more, see An al. (202T)'s Figure
walkordd meduu experti22
Ioporeimeiammoooo.oo oxpe---
Ippee-espee-t
930
welleer2 22w.ger--22222222
hosper medium madlayyvi
120
Stote
123
130
from
kandomiv2
mandomv2
Caned
3.0
3.0
Crod enco/1p
1ed
Grod encotap
1e0
15

---

Under review as a conference paper at ICLR 2023
E A RAISIN ANALOG FOR TD3
We also test a Raisin analog for TD3 to examine the generality of our approach and to show the advantage of Raisin over method approximately equivalent to the prior work Bi-Res-DDPG (Zhang et al. 2019) (since TD3 is essentially an improved DDPG). Our results show that RA-TD3-N (like Raisin, this is only an approximate nickname because RAA TD3 is underspecified) scores similarly well, whereas RA-TD3-1 (roughly equivalent to Bi-Res-DDPG) scores poorly. The latter is surely due to insufficient pessimism, like with Raisin-2.
We find that RA-TD3-10 does best at smaller values of 1 than Raisin, with the best settings at 0.5% for hopper-random, 7.5% for walker2d walker2d-random. and 10% for hopper-expert. Additionally, we find using an SAC-style policy loss (maximizing the minimum critic instead of an arbitrary critic like TD3) performs better, so we use that here. (As we noted carlier, however, the SAC-style policy loss did not appear to help TD3+BC-10 much.) For RA-TD3-1, find that 7 100% performs best, so we used that here.
Table 4: RA-TD3 scores, with Raisin scores for reference.
Task Name
RA-TD3-1
RA-TD3-10
Raisin-10
hopper-random walker2d-random
27 + 10
31 = 0.1
12 6.3
18 9.2
hopper-expert
12 15
103 + 16
110 0.4
F FIXED POINT EXPERIMENTS
We use N to study the simplest case. We trained SG-SAC-1 (regular SAC, but N = 1) on halfcheetab-rrndom for 250k steps, using 10 seeds. We then train both SG and RG SAC-1 (where RG-SAC-1 is 7 100%) resuming from those weights that were trained with SG. We start from only 5 of the highest-scoring original seeds, since about half of those seeds
stuck at poor scores (because N = =1) and we're only interested here in how the algorithms behave once they reach good fixed point. For each of the 5 saved SG models, we resume 2 more seeds using both SG and RG, giving 10 seeds total for RG resumes and 10 seeds total for SG resumes. We train for another 250k steps. RG's scores fall dramatically, whereas SG scores do not. We show the scores in Table This experiment provides some evidence in favor of Xiao et al. (202T)'s ' finding that TD and BRM converge to different fixed points the overparameterized setting. Put another way, provides some evidence that slow convergence might not be RG's s issue here (though again, Figure 2 suggests slow convergence can still be an issue in certain setups, as argued in c.g. Wang & Ueda (021)).
Table 5: SG-SAC-1 scores on halfcheetalh-random. and scores when resuming training with new runs for both RG and SG from those trained weights. SG resume scores much higher than SG-SAC-1 scores partly because of the additional training time, and also because as discussed we resumed training only from high-scoring seeds.
Task Name
SG-SAC-1
RG-SAC-1 resume
SG-SAC-1 resume
halfcheetab-raadomm
17 15
-2.0 0.3
32 + 3.5
We also run a similar experiment training RG SAC-1 on hopper-expert for 2M steps. Again, we then train both SG and RG SAC-1 (for 500K steps) resumed from those weights. Similar the other experiment, SG scores fall dramatically, but RG scores do not. We show these scores Table This gives additional evidence in favor of fixed points being important here.
16

---

Under review as a conference paper at ICLR 2023
Table RG-SAC-I scores hopper-expert, and scores when resuming training with new runs for both RG and SG from those trained weights.
Task Name
RG-SAC-1
RG-SAC-1 resume
SG-SAC-1 resume
hopper-expert
39 = 20
63 + 24
0.7 + 0.1
We did not load the saved optimizer state - we could test loading the saved optimizer state as well, but we think this would be fairly unlikely to significantly change the results.
G PURE RG
We find pure RG fails to exceed average score of on halfcheetah-randeee where good score would about 28. These poor scores persisted even with extensive, combinatorial tuning of the actor and critic learning rates, 7 (Schoknecht Merke 2003), the value initialization, and various ensemble aggregation techniques attempting to make RG more optimistic. For example, we simultaneously tuned each learning rate along with whether maximize the minimum, maximum, or average of the critics in the policy loss, while at the same time tuning the order statistic of the critics to replace the minimums the critic loss. We hundreds these experiments, usually for 500k gradient steps each. None of those approaches succeeded in raising pure RG's score above 5.
17


Reviews:
[{"review_id": "dDJcqFYgKBg", "reviewer": "Reviewer_kejP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author integrated the Residual gradient algorithm RG into SAC and conducted extensive experiments to prove this new approach can achieve the same level of SAC-N’s performance and, at the same time, greatly reduce the critic number N. The newly proposed method is more “versatile,” which means it can converge under a diverse range of datasets.\n\nThe main contribution of this paper is it gives a comprehensive study of the RA approach’s effect on SAC in the offline learning framework. And through comparing to state-of-the-art algorithms, it proves that combing RA into SAC can achieve the same performance while significantly reducing the number of critics. \n", "summary_of_the_review": "This paper investigates the RA approach, which does not attract much attention in the RL field. Though the author provides many experiments to support their claim, the comparison is not enough to justify their claims( as described in the weakness and quality). ", "strength_and_weaknesses": "Strength: \n\n- The strength of this paper is that 1. It conducted comprehensive experiments to prove the effectiveness of RA with the SAC algorithm and its generality to different datasets.\n\n- The paper gives a very clear motivation of why they investigated a relatively old and not proven effective algorithm and conducted experiments from different angles, such as the reduction of N compared to SG-based SAC methods and different value initialization. \n\nWeakness:\n\n- Most of this paper’s efforts are on the experimental comparison of SG and RA SAC algorithms. And the difference between them is adding the gradient of the next states into the loss function. Though these two approaches have significant differences, the RA approach has already been investigated by previous researchers, as mentioned in the RA-DDPG work.\n\n- In the baseline comparison, the author should also involve past work that extended RA to DDPG to prove their models' effectiveness and novelty. Since in the introduction, the author talked in detail and claimed RA is in nature beneficial to offline learning reinforcement learning, they should provide more information to prove it, not just compare between SAC methods. \n\n- This paper aims to prove that  RA-SAC is more versatile than the SG counterparts, and compared to traditional SAC, it needs fewer critics, thus significantly reducing the computing cost. However, as introduced in section 1, the RA method is slower to converge. So the author should also provide convergence curve comparison, not just RA-SAC’s. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cuzdLzymf3L", "reviewer": "Reviewer_vwJR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper incorporates the resisual algorithm (RA) with SAC-N used in offline RL setting, and finds that the additional RA objective can not only improve the SOTA performance, but also reduce the number of ensemble functions for a more efficient computation.", "summary_of_the_review": "Although the proposed method can achieve significant performance improvement, there lacks adequate analysis and explaination on why combining the RA and SAC-N is useful.", "strength_and_weaknesses": "Strength\n1. The paper empirically shows that adding RA to SAC-N is helpful in reducing the number of ensemble members. \n\nWeakness:\n1. The proposed RAISIN method can be viewed as a combination of RA and SAC-N, but in some sense lacking adequate motivation: RA method is helpful to the stable policy training, and the main issue of SAC-N is the large number of ensemble functions, but why simply combining the two would be good?\n\n2. The proposed method can be easily extended to most current popular offline rl algorithms, so it's better to add more experiments to verify its scalability.\n\n3. About the pessimism settings in Appendix.A, when N=10, this paper chooses $\\eta=0$ on half of tasks, but this choice means an actual equivalence with SAC-10, so I doubt whether the proposed method will work in this case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5InVEzN-oo", "reviewer": "Reviewer_foJV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of offline reinforcement learning and investigates the performance of the residual gradient algorithm (RG) and an improved residual algorithm (R) on SAC. Empirically, the simple trick increases the score on D4RL gym tasks. By adding a residual component to SAC-N, the algorithm (when $N=10$) matches the state of the art performance on most of the tasks compared with SAC-N (which requires hundreds of ensembles). It reduces the computational cause dramatically.\n\nOverall, this paper provides a clear explanation of the RAISIN algorithm (which is simply RA on top of SAC-N) and provides experimental results comparing the performance improvement and the ablation studies.", "summary_of_the_review": "This paper provide a novel observation that RA + SAC-N has good performance and reduced computational cost. The algorithm becomes more robust on a number of gym tasks. However, the performance of RAISIN does not outperform all SOTA results on all tasks (when compared with TD3+BC), and the experiments are not interpreted sufficiently.", "strength_and_weaknesses": "The methodology is simple, and the only parameter to tune is the pessimistic, as everything else is based on top of SAC. The proposed algorithm achieves comparable performance with much less ensembles, which makes the algorithm promising.\n\nThe weaknesses of this paper are listed as follows:\n\n- It seems that the novelty of this paper is limited, the methodology is a direct application of RA on SAC-N, and there are little theoretical discussion on how the performance get improved. \n- In Section 4.1 comparison with TD3+BC-10, the authors simply claim that Raisin outscores TD3+BC-10. However, in table 1, there are some tasks that RAISIN scores lower than TD3+BC, for example on hopper-expert, walker2d-expert, walker2d-medium-expert, hopper-medium-expert, hopper-medium, halfcheetah-random, etc. The authors do not provide sufficient explanation regarding what are the properties in these tasks that makes the performance of RAISIN worse.\n- Since this paper does not provide new techniques or methodologies that shed lights on future researches in offline RL, but only provides experimental studies on a combination of two existing methods. It is important for the authors to provide more thorough empirical explanations and interpretations. The experimental results are good for general papers, but for a paper without enough technical novelty, higher standard in experiments should be expected.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject