Improving Offline RL by Blending Heuristics
Authors: Sinong Geng
Aldo Pacchiano
Andrey Kolobov
Ching-An Cheng
Venue: ICLR 2024 spotlight (2024)
Keywords: offline RL
heuristic
RL
MDP
sequential decision-making
Primary area: reinforcement learning
TL;DR: A method for improving many existing offline RL algorithms' performance by blending Monte-Carlo-based heuristic state value estimates into these algorithms' Bellman operators.

Abstract: We propose **H**e**u**ristic **Bl**ending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping. HUBL modifies the Bellman operators used in these algorithms, partially replacing the bootstrapped values with heuristic ones that are estimated with Monte-Carlo returns. For trajectories with higher returns, HUBL relies more on the heuristic values and less on bootstrapping; otherwise, it leans more heavily on bootstrapping. HUBL is very easy to combine with many existing offline RL implementations by relabeling the offline datasets with adjusted rewards and discount factors. We derive a theory that explains HUBL's effect on offline RL as reducing offline RL's complexity and thus increasing its finite-sample performance.  Furthermore, we empirically demonstrate that HUBL consistently improves the policy quality of four state-of-the-art bootstrapping-based offline RL algorithms (ATAC, CQL, TD3+BC, and IQL), by 9% on average over 27 datasets of the D4RL and Meta-World benchmarks.

Published as a conference paper at ICLR 2024
IMPROVING OFFLINE RL BY BLENDING HEURISTICS
Sinong Geng
Aldo Pacchiano
Andrey Kolobov Microsoft Research Redmond, WA
Ching-An Cheng Microsoft Research Redmond, WA
Princeton University Princeton, NJ
Boston University Broad Institute of MIT and Harvard Boston, MA
ABSTRACT
We propose Hcuristic Blending (HUBL), simple cerformance-improvinn tech- nique for a broad class of offline RI. algorithms based on value bootstrapping. HUBL modifies the Bellman operators used in these algorithms, partially replacing the bootstrapped values with heuristic ones that are estimated with Monte-Carlo returns. For trajectories with higher returns, HUBL relies more on the heuristic values and less on bootstrapping; otherwise, it leans more heavily on bootstrapping. HUBL is very easy to combine with many existing offline RL implementations by relabeling the offline datasets with adjusted rewards and discount factors. We derive a theory that explains HUBL's 's effect on offline RI. as reducing offline RL's complexity and thus increasing its finite-sample performance. Furthermore, we empirically demonstrate that HUBL consistently improves the policy quality of four state-of-the- bootstrapping- based offline RL algorithms (ATAC, CQL, TD3+BC, and IQL), by 9% on average over 27 datasets of the D4RL and Meta- World benchmarks.
INTRODUCTION
Offline reinforcement learning (RL) aims to learn decision-making strategies from static logged datasets (Lange et al. 2012 Fujimoto et al. 2019). It has attracted increased interest in recent years, because the availability large offline datasets are on the rise and online exploration required by alternative approaches such as online RI. (Sutton and Barto 2018) remains expensive and risky in many real-world applications, such as robotics and healthcare.
Among offline RL algorithms, we focus on model-free approaches using dynamic programming with value boot- strapping. These algorithms, including the commonly used CQL (Kumar et al. 2020), TD3+BC (Fujimoto and Gu 2021), IQL (Kostrikov et al. 2022) and ATAC (Cheng ct al. (022), have demonstrated strong performance in offline RL. benchmarks (Fu al. 2020 Gulcehre al., 2020). 'Teyy follow the actor-critic scheme and adopt the principle of pessimism in the face of uncertainty optimize an agent via performance lower bound that pe- nalizes taking unfamiliar actions. Despite their strengths, existing model-free offline RL methods also have major weakness: they do not perform consistently. An algorithm that does well on one dataset may struggle on another, sometimes even underperforming behavior cloning (see Tarasov et al. (2022) and Appendix D.2). These perfor-
Base offline RL method
1: Input: data ID ((s, a, 8 1,,))
2: for each iteration do
57) 3: Sample (s, a, a, r,,,,,,
4: DP update using (8, (s,,,,,s',,)
HUBL + offline RI.
1: Input: data ID {(s, a, 8', r, 7)}
2: D +- Relabel ID with modified
rewards & discounts 7 using
heuristics h
3: for each iteration do
4: Sample (s, a, F, 7,) E
mance fluctuations stand the way of applying even the strongest offline RL approaches practical problems.
5: DP update using (8, (, i,
In this work, we propose Heuristic Blending (HUBL), an easy-to- implemmeeeeeett technique address offline RL's per- formance inconsistency. HUBL is an "aide" that operates
Figure 1: HUBL and offline RI.
in combination with bootstrapping-basee based offline RI. algorithm by using heuristic state value esti-

---

Published as a conference paper at ICLR 2024
mate:D to modify the rewards and discounts in the dataset that the base offline RL algorithm consumes. Effectively, this modification blends heuristic values into dynamic programming to partially replace bootstrapping. Relying less bootstrapping alleviates potential issues that bootstrapping causes and helps achieve more stable performance.
Combining HUBL with offline RL method is very simple, as summarized in Figure | and amounts to running this method on a version of the original dataset with modified rewards F and discounts fy F + > and 5- 7(1 A), where r is blended with heuristic h, 7 is the reduced discount, and À E [o, 1] is blending factor representing the degree of trust towards the heuristic. We propose to set h as the Monte- Carlo returns of the trajectories in the dataset for offline RL. Such heuristics are efficient and stable to compute, unlike bootstrapped Q Q-value estimates. The blending factor À in Intuitively, we want À to be large (relying more on the heuristic) at trajectories where the behavior policy that collected the dataset performs well, and small (relying more on bootstrapped Q- Q-values) otherwise. We provide three practical designs for X; they use only one hyperparameter, which, empirically, does not need active tuning.
HUBL. can be eejeector-deeenndenn
We analyze HUBL's 's performance both theoretically and empirically. Theoretically, we provide a finite-sample performance bound for a tabular offline RL with HUBL by framing it as solving reshaped Markov decision process (MDP). To our knowledge, this is the first theoretical result for RL with heuristics in the offtine setting. Our analysis shows that HUBL performs a bias-regret trade-off. On the one hand, solving the reshaped MDP with a smaller discount factor requires less bootstrapping and is relatively "easier",
the regret is smaller. On the other hand, HUBL induces bias due to reshaping the original MDP. Nonetheless, we demonstrate that the bias can be controlled by setting the À factor based on the above intuition, allowing HUBL to improve the performance of the base offline RL method.
Empirically. we run HUBL with the four aforementioned offline RL methods CQL, TD3+BC, IQL, and ATAC and show that enhancing these SoTA algorithms with HUBL can improve their performance by 9% on average across 27 datasets D4RL (Fu et al. (2020) and Meta-World (Yu et al.) 2020). Notably, in some datasets where the base offlin RL method shows inconsistent performance, HUBL can achieve more than 50% relative performance improvement.
2 RELATED WORK
Bootstrapping-basee -based offline RL A fundamental challenge of bootstrapping- based offline RL is the deadly triad (Sutton and Barto 2018): a negative interference between Z) off-policy learning from data with limited support, 2) value bootstrapping. and 3) the function approximator. Modern offline RL algorithms such as CQL (Kumar et al. 2020). TD3+BC (Fujimoto and Gu 202T). IQL (Kostrikov) jet al. 2022), ATAC (Cheng et al.) 2022 PEVI (Jin et al. 2021). MOReL (Kidambi et al. [0200] and VI-LCB (Rashidinejad et al. 2021 employ pessimism to discourage the agent from taking actions unsupported by the data, which has proved to be an effective strategy to address the issue of limited support. However, they still suffer from combination of errors in bootstrapping and function approximation. HUBL aims to address this with stable-to- cobbp-e Monte-Carke return heuristics and thus is complementary technique to pessimism.
RL by blending multi-step returns The idea of blending multi-step Monte-Carle returns into the bootstrapping operator to reduce the degree of bootstrapping and thereby increase its performance has a long history in RL. This technique has been widely used in temporal difference methods (Sutton and Barto 2018), where the blending is achieved by modifying gradient updates (Seijen and Sutton| 2014] Sutton et al. 2016 Jiang et al. 202T) and reweighting observations (Imani et al. 2018). It has also been applied to improve Q function estimation (Wright et al. 2013). online exploration Ostrovski et al. 2017 Bellemare et al. 2016) and the sensitivity to model misspecification (Zanette et al. 2021). and is especially effective for reware problems Wilcox al. (2022). In contrast to most of the aforementioned works, which focus on blending Monte- online operation, HUBL is designed for the offline setting and acts as a simple data relabeling step. Recently, Wilcox et al. (2022) have also proposed the idea of data relabeling, but their design takes max of multi-step returns and bootstrapped values and as a result tends to overestimate fuuuctions.. We observed this to be detrimental when data has limited support.
returns as part of an RL algorithm's 's
In this work, heuristic is a mapping from states to IR Mausam and Kolobov 2012 See Section 3.2
2

---

Published as a conference paper at ICLR 2024
RL with heuristics More generally, HUBL relates to the framework of blending heuristics (which might be estimating quantities other than a policy's 's value) into bootstrapping (Cheng et al. 202T Hoeller et al., 2020, Beijani al. 2018 Zhong al. 2013). However, the existing results focus only on the online case and do not conclusively show whether blending heuristics is valid in the offline case. For instance, the theoretical analysis in Cheng al. (202T) breaks when applied to the offline setting as we will demonstrate in Section | The major difference is that online approaches rely heavily on collecting new data with the learned policy, which is impossible in the offline case. In this paper, we employ novel analysis technique to demonstrate that blending heuristics is effective even in offline RL. To the best of our knowledge, ours is the first work to extend heuristic blending to the offline setting with both rigorous theoretical analysis and empirical results demonstrating performance improvement. One key insight, inspired by transition-dependent discount factor from White (0117), is the adoption of trajectory-deeeenennt X blending factor, which is both performance-improving design as well as a novel analysis technicality.
Discount regularization HUBL modifies the reward with a heuristic and reduces the discount. Discount regularization, on the other hand, is complexity reduction technique that reduces only the discount. The idea of simplifying decision-makin problems by reducing discount factors can be traced back to Blackwell optimality in the known MDP setting (Blackwell, 1962). Most existing
results on discount regularization (Petrik and Scherrer 2008 Jiang et al. 2015) study the MDP setting or the online RI. setting (Van Seijen et al.] 2019)) Recently, Hu et al. (2022) has shown that
discount regularization can also reduce the complexity of offline RL and serve as an extra source of pessimism. However, as we will show, simply reducing the discount without the compensation of blending heuristics in the offline setting can be excessively pessimistic and introduce large bias that hurts performance. In Section 6.1 we rigorously analyze this bias and empirically demonstrate the advantages of HUBL over discount regularization.
BACKGROUND
In this section, define the problem setup of offline RL (Section 3.1 review dynamic programming with value bootstrapping, and briefly survey methods using heuristics (Section 3.2).
3.1 OFFLINE RL AND NOTATION
We consider offline RL in a Markov decision process (MDP) M = - A.. where S denotes
the state space, A denotes the action space, P denotes the transition function, r : S x A + [o, 1] denotes the reward function, and 7 E [0, 1) denotes the discount factor. A decision-making policy is a mapping from to A, and its value function is defined as (*) = E 7'r(st, s0 = s, ag (-[s:)]. In addition, we define Q= a) == r(s, a) + - IV (s') its state-action
value function (i.e., Q function). We use V* to denote the value function of the optimal policy x², which is our performance target.
In addition, we introduce several definitions of distributions. First, we define the average state distri- bution of a policy starting from initial state su as d= (s, := (1 -7) Et) 7'd (s, a; so). where di (s, a; 80) is the state-action distribution at time L generated by running policy n from an initial state 8n- We assume that the MDP starts with fixed initial state distribution do- With slight abuse of notation, we define the average state distribution starting from do as d= a) == d= (s, a; do). and d' (s, a, ;= d (s, (s'(ss a).
The objective of offline RL is to leam a well-performing policy Å while using a pre-collected offline dataset ID. The agent has no knowledge of the MDP M except information contained in ID, and cannot perform online interactions with environment to collect more data. We assume ID := {r} contains multiple trajectories collected by behavior policy, where each denotes a trajectory with length T,, Suppose these trajectories contain N transition tuples in total. With abuse of notation, we also write ID == {(s, a, s', ,r,,),, where states and action follow distribution j(ss, a) induced by the behavior policy, s' is the state after each transition, r is the reward at s, a, and y is the discount factor of the MDP. Note that the value of discount is the same in each tuple. We use Ω to denote the support of pls, a). We do not make the full support assumption, in the sense that the dataset ID may not contain a tuple for every 8, a, 8' transition in the MDP
[(st, at rr)) 1
3

---

Published as a conference paper at ICLR 2024
3.2 DYNAMIC PROGRAMMING WITH BOOTSTRAPPING AND HEURISTICS
Bootstrapping Many offline RL methods leverage dynamic programming with value bootstrapping. Given policy R, recall Q" and V satisfy the Bellman equation:
(hontstrapping)
("(s,a)= ( r(s,a) + B.g s.0)
These bootstrapping-bent based methods compute Q" (s, a) using an approximated version of (T): given sampled tuples (s, a, r,,, such methods minimize the difference between the sides of (I) with both Q and V replaced with function approximators 2 With limited offline data, learning the
function approximator using bootstrapping can be challenging and yields inconsistent performance across different datasets (Dulac-Arnole et al. [021 Sutton and Barto 2018 Kumar et al. 2019), as
we will also later see in the experiment section (Section 6).
Heuristics A heuristic is value function h S - R calculated using domain knowledge, Monte- Carlo averages, or pre-training. Heuristics are widely used in online RL, planning, and control to improve the performance of decision- maaiing (Kolobov et al. 2010] [hoog et al.) [013] Beijani et al., 2018 Hoeller al. 2020 Cheng et al. (021). In this paper, we focus on the offline setting, and consider heuristics h that approximate the value function of the behavior policy. They can be estimated from ID via Monte-Care methods.
HEURISTIC BLENDING (HUBL)
In this section describe main contribution Heuristic Blending (HUBL), an algorithm that works in combination with bootstrapping-hasee offline RL methods and improves their performance.
4.1 MOTIVATION
HUBL uses heuristic computed the Monte- return of the behavior policy in the training dataset
Algorithm HUBL + Offline RL.
D improve offiin RL. It reduces offline RL
1: Input: Dataset ID = {(s, o, 8 7)}
algorithm's bootstrapping at trajectories where the behavior policy performs well, i.e., where the value
Compute he for each trajectory ID
3: Compute X for each trajectory in ID
of the behavior policy (i.e. the heuristic value) is high. With less amount of bootstrapping, mitigates bootstrapping-inducee issues convergence stabil- ity and performance. addition, since the extent of blending between the heuristic and bootstrapping is trajectory-dependent HUBL introduces only limited performance bias to the base algorithm, and therefore can improve its performance overall.
Relabel r & 7 by ht and d as 7 and 5
and create {(ss a, 8 (,),,)
< Offline RL on ID
4.2 ALGORITIIM
As summarized in Algorithm [ HUBL is easy to implement: first, relabel base offline RL algorithm's training dataset D {(s, 0, s 7)} with modified rewards 7 and discount factors 7/ creating a new dataset = ((s, a, 8 F, 5)): next, run the base algorithm on DD.
The data relabeling is done in three steps:
Step 0: As a preparation step, we convert the data tuples in ID back to trajectories like т {(t,, oz, T- For the next two steps, we work on data trajectories instead of data tuples to compute heuristics and blending factors.
Step 1: Computing heuristic h We compute heuristics by Monte-Carlo returns. For each 3
{(st, oz, TT E ID, we calculate the heuristics as h -E y 'ry and update the data trajectory as T {(t,, ae, Te, h2
*Q-learnigg- hased offline RI. in Kostrikov al. 2022 uses dynamic programming equation similar with (*) = arg max Q" (s)
In our implementation, also train value function approximator to bootstrap at the end of trajectory if the trajectory ends due timeout as opposed reaching the end of the problem horizon terminal state.
4

---

Published as a conference paper at ICLR 2024
Step 2: Computing blending factor De We append scalar A = X(7) E [0.1] at each time point t of each trajectory the blending factor, leading <- ((8t, Tt, )e)] X indicates the confidence in the heuristics on the trajectory. Intuitively, X decides the contribution of heuristics over bootstrapped values in dynamic programming to update the Q-function. We desire A be closer to when the heuristic value he is higher (i.e., at states where the heuristic is closer to the optimal Q Q-value) to make offline RL rely more on the heuristic, and X closer zero when heuristic lower to make offline RL to use more bootstrapping. We experiment with three different designs of X(T):
Constant: As baseline, we consider X(T) = a E [o, 1] for all 8. We show that, despite forcing the same heuristic weight for every state, this formulation already provides performance improvements.
Sigmoid: As an alternative, we use the sigmoid function to construct trajectory-dependenn blending function A(T) ao E h(sx)/TT>,, where a E [0, 1] is a tunable constant and is the sigmoid function. Thus, X(T) varies with the performance of the behavior policy over data.
Rank: Similar to the Sigmoid labeling function, we provide a rank labeling function X(T) = a ErEDD 1 (x(r)s /n where n is the number of trajectories in DD and h(() = En.Er heg
Step 3: Relabeling r and 7 Finally, we relabel the reward as and the discount factor as each
tuple of DD To this end, we first convert the updated data trajectories {{(se, de, Tt: hy, ) back into data tuples {(s, s', r, T, n, X)). where h'' and X' denote the next-step heuristic and blending factor. Then, for each data tuple, we compute
F 7X''
and 3= - ());
to form new dataset 1 {(s, a, 8 F; 7). Intuitively, one can interpret F as injecting heuristic- dependent quantity 7''h' into the original reward, and 7 as reducing bootstrapping by shrinking the original discount factor by a factor of 1 X. We formally justify the design of f and Fy in Section 5
UNDERSTANDING HUBL
In this section, we take a deeper look into how HUBL works. At high level, our theoretical analysis explains that the modification made by HUBL introduces a trade-off between bias and regret (which similar to the variance of policy learning) into the base offline RL algorithm. While an important part of our analysis is for tabular settings (Section 5.3), we believe it still provides valuable intuitions as to why reshaping of rewards and discounts per (2) gives a performance boost to state-of-the-art offline RL algorithms in the continuous-state-spaae benchmarks in Section 6
5.1 HUBL AS MDP RESHAPING
We analyze HUBL by viewing it as solving reshaped MDP M (S,A,,,i,)) constructed by blending heuristics into the original MDP. To this end, we make a simplification by assuming that both the heuristic and blending factor functions of states. Specifically, let Ω denote the support of
the data distribution; we suppose that some functions h(-)= : + IR and A(.) : Ω - [0, 1] are given HUBL. For analysis, we extend them to out-of-f states as below:
h(s)= = { (s))) for E Ω
and A(s, = {
[(s')) for 8, s' E
(3)
otherwise
otherwise
We note that this extension is only for the purpose of analysis, since HUBL never uses the values h(()) and A(-) outside Ω; the given I on out-of-f2 can have any value and the following theorems still hold.
We define the reshaped MDP M with redefined reward function and discount factor as F(s, a) == r(s, a) + -E.r>P((1s.n)) [A(s, s')](' and "(s, s') := >(1 - A(s, s')) respectively. Note that we blend the original reward function with the expected heuristic and blending factor while adjusting the original discount factor correspondingly. The extent of blending is determined by the function x(-). Notice that this reshaped MDP has Paasition-dependeer discount factor (s, which is compatible with generic unifying task specification of (White 2017). This novel definition of
*A special case of trajectory-depence discounts.
5

---

Published as a conference paper at ICLR 2024
deeendden discount factor is a key analysis technique to show that blending heuristics is effective in the offline setting.
Reshaped Dynamic Programming the Bellman equation changes from accordingly into
When solving this reshaped MDP by dynamic programming.
(s, = F(s,a) + E. Ex (s,,s
bootstrapping
heuristic
=r(s,a) + P A(s, 8') +o R P(s,
Here Q" denotes the Q-function of policy ı in M,, and V denotes T's value function. Compared to the original Bellman equation ( it can be seen that X(.) blends the heuristic with bootstrapping: the bigger A's values, the more bootstrapping is replaced by the heuristic. The effect of solving HUBL's reshaped MDP . is twofold. On the one hand, M is different from the original MDP M: the optimal policy for M may not be optimal for M, so solving for M could potentially lead to performance bias. One the other hand, .M has a smaller discount factor and thus is easier to solve than M, the agent needs to plan for smaller horizon. Therefore,
think of applying HUBL to offline RL problems as performing a bias-variance trade-off, which reduces the learning variance due to bootstrapping at the cost of the bias due using a suboptimal heuristic. We will explain this more concretely next.
5.2 BIAS-REGRET DECOMPOSITION
The insight that HUBL reshapes the Bellman equation that the offline RL algorithm uses allows us to characterize HUBL's effects policy learning. Namely, we use the modified Bellman equation from to decompose the performance of the learned policy into bias and regret terms.
Theorem 1. For any Ω + R. optimal policy, it holds that V (do) - (do) = Bias(TT h, A) + Regret(T, h, A). where
+
[0. 1). and policy æ, with V as the value function of the
Bias(T,, h, A) = 7
1-7 E (s. (A(s)) (5') -
E E
Regret( (T, h,,, v= (do) V (=(())
7 (a, - M]. 1
The performance of т depends on both bias and regret. The bias term describes the discrepancy caused by solving the reshaped MDP with A(-). When X(s') = O, the bias becomes zero. The regret term describes the performance of the learned policy in the reshaped MDP. Intuitively. at states whose successors have bigger A(s') values, the reshaped MDP has a smaller discount factor and thus is easier to solve, which leads to smaller regret (i.e., smaller values for 5" (dn) - V (do)). Therefore, solving HUBL-reshaped MDP induces bias term but may generate smaller regret.
Remark The critical difference and novelty - of Theorem I compared to existing theoretical results for RL with heuristics in the offline setting is that both the bias and regret in Theorem depend only on states in the data distribution support Q. This is crucial, because the offline setting we have no access to observations beyond Ω. In contrast, if in the preceding analysis replace Theorem I by. for example, Lemma A. from Cheng et al. (2021) with a constant X, we will get performance decomposition (do) - V" (do) = -V* - (=)d)) + 2A A (((((( - ((m(d)) (d")). The decomposition, however, suggests that the out-of-f values of A(s) or h(s) are important to the performance of HUBL.
T E.,ard Bay [h(8') - (*')
+ (1 - X) (do) - /x (do)) + I
5.3 FINITE-SAMPLE ANALYSIS FOR BIAS AND REGRET
The finite-sample analysis of policy learning that we provide next illustrates more concretely how HUBL trades off bias and regret. Our analysis uses offline value iteration with lower confidence bound (VI-LCB) (Rashidinejad et al., (2221) as the base offline RL method. Following its original convention, make some technical simplifications
make the presentation cleaner. Specifically, we consider a tabular setting with ((s) a E [0, 1] for s E Ω as a constant value, which can be interpreted the quality of the behavior policy averaged across states. Note that although X(s)

---

Published as a conference paper at ICLR 2024
Relative Improvenee
0.00
0.00
0.85
0.63
0.30
0.30
Relative
0.00
0.13
0.13
Sualf yrann
halle
ATAC
CQL.
Average Relative Improvement: 9%
Áverage Relative Improvement: 9%
Relative Improvene
1.00
0.00
0.30
0.30
0.35
0.15
0.00
0.00
ballerrr haltelene halte
bollle Auneet sellowant haate
balll
TDB- BBC
Average Relative Improvement:
Average Relative Improvement: 97%
Figure 2: Relative improvement of HUBL with rank blending on 9 D4RL datasets.
is a constant on Ω. the reshaped MDP is still defined by X(s, s') in (3) (i.e.. A(s, s') = a if s, s' E Ω and zero otherwise). The details of VI-LCB with HUBL are in Appendix due to space limits.
Theorem 2 Summarizes our finite-sample results. The proof of Theorem 2 can be found in Appendix
B
Theorem 2. Under the setup described above, assume that the heuristic h((-) satisfies hu(s) V for any & E Then the bias and the regret in Theorem] 7 are bounded by
72
Bias( A)
I-7 Bi
[V" VV EΩ).
Vihan (1 N(1 7(1 2))2
d= (8,
7d
1
Ep [Regret f, A) S min
Vinaz V
\maa p(s.a)
max (s.a)ERt
pls,
where Vmax denotes a constant upper bound for the value function.
For the bias bound, the assumption hi(s) = (s) for any E f2 is made for the ease of presentation. If it does not hold, an additive error term can be introduced in the bias bound to capture that. For
the regret bound, max d" (s.azda) when it is bounded assumed by existing works (Rashidinejad al. 2021), our results demonstrate how N, y and À affect the regret bound.
[a(aaa) can be infinite, whereby the best regret bound is just Vmaz. But
The implications of Theorem are threefold. First of all, provides finite-sample performance guarantee for HUBL with VI-LCB under the tabular setting, Compared with the performance bound of
original VI-LCB, min
V Vmaz:
[1-7PN maxs,a
V2
man [S]
di (s.a)
ja(a,))
HUBL shrinks discount factor
by À and thus potentially improves the performance while inducing bias. Second, Theorem hints at the source of HUBL's bias and regret of HUBL. The bias is related to the the performance of the behavior (data-collection) policy, characterized by (*(s) V (s). In the extreme case of data being collected by an expert policy, the bias induced by HUBL is O. The regret is affected by d= (a.a)
p(s.a) which describes the deviation of the optimal policy from the data distribution. Finally, Theorem 2 also provides guidance on how construct a blending factor function X(-). To reduce bias, X(s) should be small at states where (s) V* is small. To reduce regret, X(-) should be generally large but small at states where the learned policy is likely to deviate from the behavior policy. Therefore, an ideal X(-) should be large when the behavior policy is close to optimal but small when it deviates from the optimal policy. This is consistent with our design principle in Section 4.2
Extension to Model-based Offline RL Conceptually, we can implement HUBL with model- based offline RL methods by directly relabeling the data, learning the ceuristic-modifiee reward model,
7

---

Published as a conference paper at ICLR 2024
Relative Improvenu
03
02
Relative
Booat per aereer
20
ATAC Average Relative Improvement: 2%
COL Average Relative Improvement:
Rolativn Improvsmms
Bolatio
boodh bandt
Average Relative Improvement:
Average Relative Improvement:
Figure 3: Relative improvement of HUBL with rank labeling on MW datasets.
and then doing model-basee planning with smaller discount, following ( Theorem still applies to model-based algorithms. For Theorem P we expect that similar analysis can be applied to model-based algorithms like MOReL. We defer this direction future work and focus on improving the stability of wide range of model-free offline RL methods.
EXPERIMENTS
We study 27 benchmark datasets in D4RL and Meta-World. We show that HUBL improves the performance of existing offline RL methods by 9% with easy modifications and simple hyperparameter tuning. Remarkably, in some datasets where the base offline RL performs inconsistently poorly, HUBL can achieve more than 50% performance improvement. When the dataset is simple and the base offline RL is already near-optimal, HUBL does significantly harm the performance.
HUBL variants and base offline RL methods We implement HUBI with four state-of-the-ar offline RL algorithms base methods: CQL (Kumar et al. 2020). TD3+BC (Fujimoto and Gu 2021), IQL (Kostrikov et al.) 2022), and ATAC (Cheng et al. 2022). For each base method, we compare the performance of its original version to performance with HUBL running three different blending strategies discussed in Section 4.2] constant, sigmoid and rank. Thus, we experiment with 16 different methods in total. The implementation details of the base methods are in Appendix D.I]
Metrics We use relative normalized score improvement, abbreviated relative improvement, as measure HUBI.'s performance improvement. Specifically, for given task and base method, we first compute the normalized score Tbase achieved by the hase method, and then the normalized score THUBL of HUBL. The relative normalized score improvement is defined as THERL -Taas, We report [rossel
the relative improvement of HUBL averaged over three seeds {0, 1, 10) in this section, with standard deviations, base method performance, behavior cloning performance and absolute normalized scores provided in Appendix D.2 D.5 and D.7
Hyperparameter Tuning For each dataset, the hyperparameters of the base methods are tuned over six different configurations suggested by the original papers. HUBL has one extra hyperparameter, which is fixed for all the datasets but different for each base method. Specifically, a is selected from (0.001, 0.01, 0.1} according the relative improvement averaged over all the datasets. In practice, we notice that a single choice of a around 0.1 is sufficient for good performance across most base offline RI. methods and datasets as demonstrated by the sensitivity analysis in Appendix D.3
8

---

Published as a conference paper at ICLR 2024
6.1 PERFORMANCE IMPROVEMENT OF HUBL
We study 9 benchmark datasets in D4RL and 18 datasets for tasks in Meta-World, collected fol- lowing the procedure detailed in Appendix D.4 The relative improvement HUBL with the rank labeling method is reported in Figure 2 and Figure First, despite being simple and needing little hyperparameter tuning. HUBL improves the performance of base methods in most settings and only slightly hurts in some expert datasets where base offline RL methods are already performing very well with little space for improvement. Second, there are cases where HUBL achieve very significant relative improvement-monn than 50%. Such big improvement happens in the datasets where the base method shows inconsistent performance and underperforms other offline RL algorithms. HUBL solves this inconsistent performance issue by simply relabeling the data. Further, HUBL improves performance even on data with few expert trajectories like hopper-med-rep- rep-22. aalker2d-med-reyy and hopper-med-v2, because HUBL conducts more bootstrapping and relies less on the heuristic on suboptimal trajectories (see Section 4.2)
6.2 ABLATION STUDIES
Sigmoid
Rank
Constant
Wins
30
46
32
Discount Relabeling HUBL relabels both the reward and the discount factor, per In contrast, existing methods like Hu et al. (2022) suggest that a lower discount factor alone, without blending heuristics into rewards, can in general improve offline RL. To assess the need for mod-
Table I: Blending strategy comparison
ATAC
Sigmoid
Rank
Constant
Ablations
-0.01
-0.02
-0.02
ifying both the discount factor and rewards like (2). we consider ablation methods which shrink only the discount factor without F. The achieved average relative im- provement of these ablations is reported and compared with that of HUBL in Table [ HUBL consistently outper- forms these ablations, which justifies HUBL's coordinated modifications in both the reward and the discount factor. The advantage of HUBL is also consistent with what our theoretical analysis predicts. The considered ablations do not modify the rewards and thus are equivalent to solving
without F
HUBL
0.01
0.02
0.01
CQL Ablations without
Sigmoid
Rank
Constant
0.04
0.04
0.04
HUBL
0.11
0.16
0.13
IQL
Sigmoid
Rank
Constant
Ablations
-0.04
-0.04
-0.07
without F
HUBL
0.09
0.09
0.06
TD3+BC
Sigmoid
Rank
Constant
(*( r(s, a) + +.E.s'p s.o)) [11 A's')) (s')..
Ablations
Comparing it with (T). the solution will be consistently smaller than the true Q-function, inducing a pessimistic bias. Crucially, this bias is much more challenging to tackle than the bias induced by HUBL, because the former is inevitable even when the data is from an expert policy.
without F
-0.01
-0.03
-0.04
HUBL
0.06
0.09
Table 2: Average relative improvement of ablations without F.
Comparison of Blending Strategies We present results for HUBL with rank blending, while the results of other blending strategies (sigmoid and constant) are provided in Appendix We notice that the rank blending outperforms the other two. With 27 datasets and 4 base methods, we report the number of cases where a given blending strategy provides
D.5)
have 108 cases. In Table [ the best performance among the three. We can see that rank is favored on average.
We also experiment HUBL in other different settings, including on datasets collected by expert policies (Appendix (.6.1). with online RL methods (Appendix (D.6.2), and in stochastic environments (Appendix D.6.3). HUBL shows consistent performance improvements in these settings.
7 CONCLUSION
In this work, we propose HUBL, a method for improving the performance of offline RL methods by blending heuristics with bootstrapping. HUBL is easy to implement and is generally applicable to various of offline RL methods. Empirically, demonstrate the performance improvement of HUBL on 27 datasets in D4RL and Keta-World We also provide a theoretic sample performance bound for HUBL., which sheds lights HUBL's bias-regret trade-off and blending factor designs.

---

Published as a conference paper at ICLR 2024
REFERENCES
Melda Alaluf, Giulia Crippa, Sinong Geng, Zijian Jing, Nikhil Krishnan, Sanjeev Kulkarni, Wyatt Navarro, Ronnie Sirear, and Jonathan Tang. Reinforcement learning paycheck optimization for multivariate financial goals. Risk & Decision Analysis, 2022.
Wissam Bejjani, Rafael Papallas, Matteo Leonetti, and Mehmet R Dogar. Planning with receding horizon for manipulation in clutter using a learned value function. In 2018 IEEE-RAS 18th International Conference Humanoid Robots (Humanoids), pages 1-9. IEEE, 2018
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
David Blackwell. Discrete dynamic programming. The Annals of Mathematical Statistics, pages 719-726, 1962.
Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems, 34: 13550-13563, 2021.
Ching-An An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.028 02446, 2022.
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-I1), pages 465-472, 2011.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Delei-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. arXiv preprint arXiv:1605.07127, 2016.
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 1109))2419-2466, 2021.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv: /248, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20144 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
S Geng, Z. Kuang, PL Peissig, D Page, L Maursetter, and KE Hansen. Parathyroid hormone independently predicts fracture, vascular events, and death in patients with stage 3 and 4 chronic kidney disease. Osteoporosis International, 30:2019-2025, 2019a.
Sinong Geng, Zhaobin Kuang, Peggy Peissig, and David Page. Temporal poisson square root graphical models. Proceedings of machine learning research, 80:1714, 2018.
Sinong Geng, Minhao Yan, Mladen Kolar, and Sanmi Koyejo. Partially linear additive gaussian graphical models. In International Conference on Machine Learning, pages 2180-2190. PMLR, 2019b.
Sinong Geng, Houssam Nassif, Carlos Manzanares, Max Reppen, and Ronnie Sirear. Deep PQR: Solving inverse reinforcement learning using anchor actions. In International Conference on Machine Learning, pages 1431-3441. PMLR, 2020.
Sinong Geng, Houssam Nassif, and Carlos A Manzanares. A data-driven state aggregation approach for dynamic discrete choice models. arXiv preprint 16X1v:2304.04916, 2023.
10

---

Published as a conference paper at ICLR 2024
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RL unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:7248-7259, 2020.
David Hoeller, Farbod Farshidian, and Marco Hutter. Deep value model predictive control. In Conference on Robot Learning, pages 990-1004. PMLR, 2020.
Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. On the role of discount factor in offline reinforcement learning. In International Conference on Machine Learning, pages 9072-9098. PMLR, 2022.
Ehsan Imani, Eric Graves, and Martha White. An off-policy policy gradient theorem using emphatic weightings, Advances in Neural Information Processing Systems, 31, 2018.
Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1181-1189, 2015.
Ray Jiang, Tom Zahavy, Zhongwen Xu, Adam White, Matteo Hessel, Charles Blundell, and Hado Van Hasselt. Emphatic algorithms for deep reinforcement learning. In International Conference on Machine Learning, pages 5023-5033, PMLR, 2021.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-211 12822,, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Andrey Kolobov, Mausam, and Daniel S. Weld. Classical planning in MDP heuristics: With a little help from generalization. AAAI, 2010.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In ICLR, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Jearning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191,, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
Mausam and Andrey Kolobov. Planning with Markov decision processes: An Al perspective. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6:1-210, 2012.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference machine learning, pages 2721-2730. PMLR, 2017.
Marek Petrik and Bruno Scherrer. Biasing approximate dynamic programming with lower discount factor. Advances in neural information processing systems, 21, 2008.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:1 1702- 2021.
11

---

Published as a conference paper at ICLR 2024
Anton Maximilian Schaefer, Steffen Udluft, and Hans-Georg Zimmermann. A recurrent control neural network for data efficient reinforcement learning. In 2007 IEEE International Symposium Approximate Dynamic Programming and Reinforcement Learning, pages 151-157. IEEE, 2007.
Harm Seijen and Rich Sutton. True online td (lambda). In International Conference on Machine Learning, pages 692-700. PMLR, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, A Rupam Mahmood, and Martha White. An emphatic approach the problem of off-po 2603-2631 2016.
-dffference learning. The Journal of Machine Learning Research, 17(1):
Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104:104366, 2021.
Phillip Swazinna, Steffen Udluft, Daniel Hein, and Thomas Runkler. Comparing model-free and model-based algorithms for offline reinforcement learning. IFAC-PapersOnlline, 55(15):19-26, 2022.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. arXiv preprint arXiv:2210.07105. 2022.
Harm Van Seijen, Mehdi Fatemi, and Arash Tavakoli. Using logarithmic mapping to enable lower discount factors in reinforcement learning. Advances Neural Information Processing Systems, 32, 2019.
Martha White. Unifying task specification in reinforcement learning. In International Conference Machine Learning, pages 3742-3750. PMLR, 2017.
Albert Wilcox, Ashwin Balakrishna, Jules Dedieu, Wyame Benslimane, Daniel Brown, and Ken Goldberg. Monte Carlo augmented actor-critic for sparse reward deep reinforcement learning from suboptimal demonstrations. arXiv preprint arXiv:2210.07432. 2022.
Robert Wright, Steven Loscalzo, Philip Dexter, and Lei Yu. Exploiting multi-step sample trajectories for approximate value iteration. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part I 13, pages 113-128. Springer, 2013.
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1 100. PMLR, 2020.
Andrea Zanette, Ching-An Cheng, and Alekh Agarwal. Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory, pages PMLR, 2021.
Mingyuan Zhong, Mikala Johnson, Yuval Tassa, Tom Frez, and Emanuel Todorov. Value function approximation and model predictive control. 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pages 100-107. IEEE, 2013.
12

---

Published as a conference paper at ICLR 2024
A EXTENDED RESULTS OF THEOREM 1
Below we prove the statement which is a restatement of theorem I
Theorem 3. For any À: Ω > [0, 1] and any h 92 - R, it holds
(*0) - (so) = (vT (so) - (s0)
(5)
E )s ((s') (5) h(s'))s))ss [?] 1
B(s.a, (ss') (s'))ss 22.
Specifically, the performance of depends on both bias and regret. The bias term describes the dis- crepancy caused by solving the reshaped MDP with A(.). The regret term describes the performance of the learned policy in the reshaped MDP.
To prove Theorem 3 we first conduct a regret decomposition as
V*(so) V (5o) = (V*()) Vx (so) + (V" (so) - -*0))) + (vo)) - (s0))
Then, rewrite both (so) - V/* (so) and / (so) V (sa) using the heuristics h(s).
A.1 TECHNICAL LEMMAS
Before proceeding to the proof details of Theorem we first prove a lemma on V (>0()) 7 (so). For policy under the original MDP M and the reshaped MDP M, we aim to quantify the value difference using the heuristics h(s).
Lemma 4, For any policy n.
(=((o) - (= = 1-7 - ((aas") (d)) (s3))))ss E EQ).
7
Proof. By the definition of 8);
1 (sa) (sa)
1
E (r(, ( [*())]
1
El Ir (T'('' r(s,a) 7>s, s'sss') + 7(1 A(s, ((s.s
2
E E(sa,, (Ms, (V h(s'))
7
(Mssa - M(s'))ss,s'EE
? (1 (8)))s)))))))))) 8 E
A.2 PROOF OF THEOREM 3
To prove the theorem, we decompose the regret into the following terms:
V*(so) - /$ (so) (V*(s)) ~= (so) (V (so) v=( + (ū+ V* (s()s)
We apply Lemma 4 rewrite the first and the last terms
V* (*o)) -=" (so) 1- 7
7 E (s,., (A(s)((V=* (5) - hh(s'))sss E
(*0) - y$ (so) = 1- B(s.aoo) (X(3))(())) - e
7
Combining the two completes the proof.
13

---

Published as a conference paper at ICLR 2024
B EXTENDED RESULTS FOR THEOREM 2
To prove Theorem
2 we separately bound the bias term
Bias( f. A) = 1-7 R (A(5)) (5) -(s'))ss E
and the regret term
Regret(s,, 2) = v= 1 (do) V (do) + 7
1-y (( (A(')(h(s') - (s'(ss)ss s' E e
Specifically the bias is bounded by Lemma and the regret is bounded by Lemma 9
B. TECHNICAL LEMMAS
Now, we provide some technical lemmas for the proof of Theorem 2
B.1.1 BIAS COMPONENT
We first prove the upper bound for the bias component.
Lemma 5. Assume h(s) S V* (s). Vs S. holds that V v** (or all E S.
Proof.
V (s) a(),a) + M(- (6))
+
V8) + + MM ((())))))))))))))))))))) + (1 (1- -() (4)- (s)'
V*) F A(8 8 (1(s)) - + ((s V8)) (s')-
S (s) + E (1 A(s,
where in the inequality we used h(8) V (s). Then by contraction argument, show V (s) V <
Lemma 6 (Bias Upperbound). Under the assumptions of Theorem the bias component can be bounded by:
Xy
Bias(ft (an(ff)))= A)
B - E
Proof. Review the bias component
Bias क., A) =
E Bla. (Xss) (s') h(s'))ss, e e..
Under the assumptions of Theorem we derive
Ay
Bias (t, A) =
E F(.... (10) - E
Next by Lemma 5 we have Av 1-y ((aa (5' - v E s 0, and thus
A B Blsa (V (V' - VM('')8, 5' 2].
Bias(A, A)
1-7
14

---

Published as a conference paper at ICLR 2024
B.1.2 REGRET COMPONENT
Next, we focus on the regret component.
Lemma 7. Under the setup of Theorem VP(s) (*()).
Proof. Since Q is the support of μ. this can be shown by the following: for E 92
(*()) - (s)
= Ba + + s's,s)))(s)))) -(1 M(s, (s) r(s,a) (V"
Ba |r + (s') + M(s, s'))"" r(s,a)) VV'('') E ?]
1001- s(s, -((, (5) a
B X(s, (z)
E ?].
Since y < 1, then by an argument of contraction, we have VV(s) - (*(s) for E Ω.
Lemma 8. The difference between V* (s) and V0 (s) can be derived as
(*(s) - 5" (s) = E A(1 - ( h(s:))]].
Proof. By the dynamic programming equation both the original and shaped MDP.
(*( V" (s) = - A) (ss) - V- (s']]
(6)
+ + 7A (1:9) ( -1(8]]]
V*())
Then, we use 6 recursively:
D
(*()) (7(s)-iT) (= AE
[11 ((V hss:))]]--
Lemma 9 (Regret Upperbound). The expected regret is bounded by
(1 7)| max,, p(s,a)
d= (s.a:do)
Ep Reggee((t, A(-)] S Smin
Vimaz, Vmnw
N(1 7(1 -1)4
72
(1 - 1-)(S| ml) u(5.0)
+
1 min Vmmax Vmax
V
N(1 7(1 -})
Proof. Under the setups in Theorem 2 we have
Regret(#t, X()) " (do) V (do) 72 E ( - (5)))s E
Further by Lemma we can replace V by Vn:
7/
Reg "" (do) ÿ (do) 1
E (a - e EQ. (7)
Then, by Lemma 10
(1 [-)) maxx,0
d= (a.o;do)
En|V" (do) v*(da)) S min (Vmaz Vman
p(s.a)
(8)
N(1 7(11 -})}4
15

---

Published as a conference paper at ICLR 2024
and
(1 - 7)|8) max ((2,0)
(s.ard"
ED (V"(d)) v$ (d))
min (Vmaos Vmaz
N(1 7(1 A))4
Note that, by Lemma 11 d stays in f2
Therefore, we get max,,a d (s.aad") p(s,a)
(s.odd pls,a)
aax
min (n p(s,a)
which leads to
(1 1 aa ((s.0)
ED (d( (d min (Kmann Vmax
(9)
N(1 7(1 A))1
Take 8 and
into 7 we derive
(1 7)| maXx, p(..a)
d (s.ad
ED Reegret(#, A())) Smin Vimazs Vinao
N(1 7(1 X))
7\ min 1 7
(1 - - (2)
+
Vmmao Vmax
V
N(1 7(1 11)4
B.2 PROOF OF THEOREM 2
The proof follows by combining Lemma 6 and
C VI-LCB WITH HUBL
We the offline value iteration with lower confidence bound (VI-LCB) (Rashidinejad al. 2021 as the base algorithm to analyze concretely the effects of HUBL with finite samples for the tabular case.
C.I ALGORITHM
We detail the procedure of HUBL when implemented with VI-LCB for the tabular setting. To start with, we introduce several definitions which will be used in the following algorithm and theoretical analysis. Without loss of generality, we assume that the sates take values
21,2.... ||S}} and that |AI). Then, let be [|] x vector which denotes the heuristic
the actions take values in 2, function. We assume each component satisfies h, (*) for 8 E Ω. Let A be [S] x (S] matrix with A, A if s' E Ω and Ag.'' 0 if or s' E Q. We use @ to denote the component-wis multiplication, and Aa,; to denote row of as |S] x 1 vector. With slight abuse of notation, we use t to denote the index of iteration in this section, with T the total number of iterations.
To conduct VI-LCB with HUBL, we follow Algorithm 3 in Rashidinejad et al. (0021) but with a modified updating rule. The procedure is summarized Algorithm 2 Compared with the original VI-LCB, we highlight the key modification in Line 14 with blue color. Specifically, at the tt iteration, based on (2), we modify updating rule of Algorithm 3 in Rashidinejad et al, (2021) into
as( () (s(s,a) a) + typt @ (I-A Vi1 + TPt @ Ag. h. Note that we introduce heuristics by PPt. O Agg h, while reducing the bootstrapping by yP O
(T - A..) V--1-
C.2 REGRET ANALYSIS
In this section, we study the regret under the reshaped MDP constructed by HUBL. Specifically, we bound the regret of Algorithm in Lemma T0
16

---

Published as a conference paper at ICLR 2024
Algorithm 2 HUBL with VI-LCB
1: Input: Batch dataset ID and discount factor ?
2: Set T := Nog_N
1-7
3: Randomly split D into T + 1 sets DD {(s,, Ta ra, sf)} for f0,1,... T), where Do N N observations and other datasets have 2T observations.
consists of 2
4: Set ms, a) -i=1 1{(s,, a.) (s, a)) based on dataset Dp-
5: For all a E A and 8 E S, initialize Qo(s, a) = O, Vo(s) = 0 and set Tp(s) = arg max, (o(ssa).
6: for (or(=10 1, I.... , do
7: Initialize re(s, a) 0 and set Pt be random probability vector.
8: Set m,,,, a) = De= [(s, a;) = (s, a)} based on dataset DD,-
9: Compute penalty (s a) for L 2000 (2g( + 1)|8||AA(N)
L br a) Vinax m)(s,a) V
10:
for (8, E (S, do
11:
if (s, a) 2 1 then
Set pt s;a to be empirical transitions and re(s, a) be empirical average of rewards. Set (x) f rr(ss a) bx a) yP @ (I A..)) V2-1 + @ Aas
13:
14:
Compute V.mid(s) (s) t max, Q2 a) and ()) E arg max, Qr(8,a). a).
for s do
16:
if V.mid (s) < (s) then
Vi(s) t (s) and (s) (s).
18:
else
19:
Vi(8) + (s) and (s) + (8).
20:
Return मे = nr.
Lemma (Regret of VI-LCB with HUBL). Let the assumptions in Section C.3 be satisfied. Then, for any initial distribution dinst in Ω, the regret of Algorithm iis bounded by
[V (dimit) - & (dimit)) E min (Ymens Vmax
(1 -) max,, a. p(s,a)
d= (s,atdannt))
N(1 7(1 a1)
C.3 NOTATIONS
We first provide some matrix notations for MDPs. We use pa [ to denote the transition matrix induced by policy whose (s, a) (s, clement is equal to P(s' P[s'|s, a))') and dt E R (S|AA| to denote state-action distribution induced by policy π whose (s, a) element is equal to (als). Similarly, we use Aç E R (S||s) to denote a extended matrix version of
Further, focus the policies which stay in the data distribution support Q. Such polices formally defined as {Tldd event:
for any (s, a) & Ω and so 2}. We also define clean
EMPP: = (Vs, (1,,,,))((s,.)) + - -P o (1- - A) Ve-il S a))
C.4 TECHNICAL LEMMAS
With the aforementioned definitions and assumptions, we provide the following lemmas.
Lemma 11. Let n be a policy learned by Algorithm 2 Under the event Econer == { Do) stays in for any initial state in Q. The probability of Ecover is greater than
118|[1 min p(s))
sES1
17

---

Published as a conference paper at ICLR 2024
Proof. Under the event Ecover, first fix E Q and show that given s, the learned policy does not take any action a' $ Ω2 By Algorithm
2 for any (s,a") & Ω and = 1,2, ,T, we can derive
Qi(s. d²) - - -Vmax 2000 + 1)] + y Vinax.
Since N and 7 are larger than 1, we can conclude that
Qe(s,a') for any a' such that d') $ Ω.
Next we consider two cases:
Case I: If there exists such that (s,a) E St and that there exists / 1,2, ... ,7 such that
Qz a) > O,
conclude that (s, a) 2 (s(s.a) Q(s a'), which suggests that the learned policy never conducts action & Ω given state s.
Case II: If for any such that (s. Ω and any t 1,2, T, we have
a)) = 0,
according to Algorithm 2 (s) arg max, mo(s, a). By the definition of Ecovers the policy stays in Ω.
Next, we consider the probability of Ecnver. Given a state Ω define the event & {mo(8, a) = for all such that (s, a) E [Q. The probability of & can be derive as
P(E.) (1 p(s)) $
By union bound, we can derive that
P( P(-Eoooe))
1 |S|(1 min n=()) 8
As result, stays in Ω with the probability greater than
1-88[1 min p(s)) 4
seil
Lemma 12. Let be policy that stays in Ω. Then, with dinit any initial distribution in 2 can derive
d' (s.d(s))d (())
(s)it(s) m(s,
S 7(1 -1
Proof. By definition, have
(s, dinit(()) =
1 1 7(1 X) Σ r AS:
= =
dinit)
Therefore, dinit (s) s 1-x(1 d= (s); dina), which finishes the proof.
Lemma 13. Lei og = d d (7P* 2 - (I-A))) For policy that stays in Ω, the following equality holds: of = d (7(1 Pa
Proof. Since т stays in Q pr only accesses the entries in - Aq whose values equal to (1 - A).
This observation finishes the proof.
0
Lemma 14. Let nt be policy that stays in Ω. Under the event EMDr,, for all t = 1,2,
". ,T,
v(T) - -((s)) < (1 2)* + 2 2 E [s)
i=1
18

---

Published as a conference paper at ICLR 2024
Proof. The proof follows the Lemma of Rashidinejad et al. (2021 combined with Lemma 13 Lemma 15. For any policy æ that stays in Q2 the following inequality is true:
d' (s, dinit) S 1- 1-
1 -
d= (s, a; dinit),
for any (s. Ω
Proof. By definition, we have
d (s,a;dmi)))
(1 P)PP =, a,; dinit))
7(1 A)
1=0
(s a; demit)))
(S A, a, dina).
1=0
Therefore,
1
1-7 dc (s, dinit)
d' a; dinit)= 5 1 7(1 A)
PP,S s, A. a, dinst) 1- 7(1 - ->)
1=0
C.5 PROOF LEMMA I0
By the event Ecover, we can decompose the regret into
EDIV* (dinit) - (dinit)
EVIiV' (dinit) & (dinit) {Econer }] + EDIV=' (dinit) V (dinit) (coeer) }
(10)
[ S
SEp (V=' (dinst) V V* (dinst) {Econer H Vinali |S]
where the second inequality is by Lemma
Next, we focus on Ep [v= (dimst) V* (ddmm))) (Eoover }). Following the analysis in C.5 of Rashidinejad et al. (2021) we can derive EDi(V* (dimst) - 7 (denst)) 1 {Ecover
< Em SE[((V [(V*' (dlini)) - & (do)) (Eower))
EED [Bwndine [(V [(V (dimit) の (dimit)) (3t T, (s, (s)) 03 1 (Ecover H|] Ti + EE (Boddidl [(V (dinst) - V= (dinit))
1 (vt= S T, ms(s, (s)) 2 13 {EMDP} L [Ecover]]] ;= T2
+ ED (Esddma (V (dimit) - (dimit))
1 [VI < T, me(s, n (8)) 2 1) {EMDP) (Ecover) H] = Ts.
By Lemma II under the event Ecover, stays in Q. In other words, Lemma 12 applicable to R.
13 and 14 are all
Next, for the following analysis, we consider the case that xt* stays in Q. We will discuss the case that Ω does not cover x² at the end of the proof.
By Lemma and C.5.1 of Rashidinejad et al.
(021)
8V maz S max,
d= (s.a:dini))
Ti
p(s.a)
9(1 (1 2)7)
By Lemma 14 and C.5.2 of Rashidinejad et al. (202I).
de (s.ardenst)
V Viaa
ST max,,a p(s,a)
T2 < T 2)T + +32
1 71
N
19

---

Published as a conference paper at ICLR 2024
By C.5.2 of Rashidinejad et al. (2021).
Vmaz
T3 I N
Combining Tı, T2 Tg. and 10 with T = log N/(1 - 7(1 - N)),
d= (*.aaddin))))
[ maxx, p(8,0)
ED V (dinit) V (dinst)) )
X min
Vinax Vmnx
N(1 7(1 2
Finally, by Lemma we finish the proof:
(1 - 7)| max
d= (s,ardiase) pla.a)
ED [V' (dimii)) V [dimir]]= E min (Ymaas Vmaa
(1I)
N(1 7(1 X))
Note that (11 is derived under the assumption that n² stays in Q. However, it also holds when x is not covered by Ω2 In that case, max de (s.a;danis) xo and ED (V" (dinse) - 5 (dinat) & Vmaz-
p(s,a]
D EXTENDED RESULTS FOR EXPERIMENTS
We conduct HUBI. with four offline RI. base methods on datasets in D4RL, and Meta- World. By blending heuristics with bootstrapping, HUBL reduces the complexity of decision- and provides smaller regret while generating limited bias. We demonstrate that HUBL able to improve the performance of offline RL methods.
D.I IMPLEMENTATION DETAILS
We implement base offline RI. methods with code sources provided Table 3
Table 3: Code source for base methods.
Base Methods
Code Source
ATAC
CQL
Satpsstttter/mmeeeeeaa .dd
TD3+BC
attpsr//gtthu..com//tttt
Experiments with ATAC, IQL, and TD3+BC are ran on Standard_F45 V2 nodes of Azure, and exper- iments with CQL are ran NC6S V2 nodes of Azure. As suggested by the original implementation of the considered base offline RL methods, we use 3-layer fully connected neural networks for policy critics and value networks, where each hidden layer has 256 neurons and ReLU activation and the output layer is linear. The first-order optimization is implemented by ADAM (Kingma and Ba 2014) with minibatch size as 256. The learning rates are selected following the original implementation and are reported Table
Table 4: Learning rates
Base Methods
Policy Network
Q Network
ATAC
5 x 10-7
5 x 10-1
CQL
3 x 10-4
3 x 10 4
IQL
3 x 10-4
x 10-4
TD3 BC
3 x 10
x 10
For each dataset, the hyperparameters of base methods are tuned from six different configurations suggested the original papers. Such configurations are summarized in Table
20

---

Published as a conference paper at ICLR 2024
Table Learning rates
Base Methods
Hyperparameter
Values
ATAC
degree of pessimism
{1.0, 4.0, 16.0, 0.25, 0.625, 10) {5, 20, 80, 1.25, 0.3125, 10.0]
CQL
min Q weight inverse temperature B expectile parameter 7
{6.5, 3.0, 10.0)
IQI.
{0.7, 0.9}
TD3+BC
strength of the regularizer X
[2.5, 0.625, 10.0, 40.0, 0.15625, 1
Meanwhile, HUBL has one extra hyperparameter, a, which is fixed for all the datasets but different for each base method. Specifically, is selected from (0.001, 0.01, 0.1} according to the relative improvement averaged over all the datasets in one task. Later in the robustness analysis, we show that the performance of HUBL is insensitive to the selection of a. For each configuration and each base method, we repeat experiments three times with seeds {0, 1, 10).
D.2 BASE PERFORMANCE AND STANDARD DEVIATIONS FOR D4RL DATASETS
We provide the performance of base offline RL methods and standarde deviations in Table 6
21

---

Published as a conference paper at ICLR 2024
Table Relative normalized score improvement for HUBL and normalized scores of base methods on D4RL datasets.
Sigmoid
Rank
Constant
ATAC
halfchectah- medium--xpert-e
0.02 + 0.0
0.02 0.01
0.02 + 0.0
92.5 0.73
halfchectab-medimm-te pecdium-rpplay-v
0.01 0.01
0.02 0.01
0.02 0.02
+.29
halfcheetah medium-v2
0.00
0.0
0.02
52.54
hopper-medine peeddium-eppet-v2-
0.0
0.0
0.0
111.04
hopper-mediumm--eplayyy
0.0 = 0.0
0.0
0.0
101.23 0.59
hopper-medium--v2
0.01 = 0.04
0.07
85.59 + 4.76
walkeer2d-meddium--pppettt.
0.0 = 0.0
0.0
0.01
111.99 0.8
walkerz2l-mcdduu--eepaayyz
0.03 0.03
0.08 0.0
0.05 + 0.08
84.22 2.7
walker2d-medium-v2
0.01 = 0.01
0.02 = 0.05
0.02 + 0.0
87.2 + 0.67
Average Relative Improvement
0.01
0.02
0.01
Sigmoid
Rank
Constant
COL
nalfihheral-mednun-xxpeeeerrrr
0.02 - 0.02
0.02
0.03 0.06
0.02 1 0.17
1.996119.93
halfcheetah-
0.01
0.18
0.13
0..1
ailichheeta-mmddume
-.01
-0.02 0.01
-0.03
5..05
0.42
0.00
0.38 0.05
78.39 14.51
0.03 +.00
0.01
0.02
9
hopper-meeiium--2
0.18 0.18
0.15
0.06
70.27 8.77
wallker2d-mediuu---pprrryy
0.01
0.201
0.22 0.0
89.38 1 17.54
walkker2b-medduu--eelaayyzz
0.08 0.27
0.38 0.06
0.37 1 0.04
63.35 6.22
walker22d-edduun-22
0.05 0.05
0.05 0.03
0.03 1 0.15
79.41 + 1.82
Average Relative Improvement
0.11
0.16
0.13
Sigmoid
Rank
Constant
IQI.
halfcheetal-medium- expert-v2
0.0 0.01
0.0 0.0
0.05 + 0.03
92.92 + 0.5
halfchcetah- halfchectah-meddum--rppoe medium-replay-v2
0.03 +.02
0.02
-0.04 + 0.03
44.35 + 0.21
halfcheetah-mediim-
-0.01 0.0
0.0
-0.01 0.01
0.16
hopper-mediim-exxprre.
0..0
-0.03 0.02
-0.09 + 0.12
108.13 +.613
hopper-medinee conper--ediumm-rellayyyy
0.72 0.09
0.25
0.27
53.98 + 4.18
hopper-medium--
0.03 0.03
0.00
-0.01 1.15
61.03 3.27
walker2d-mediune -expertt-2
0.01 + 0.0
0.01
0.01 0.0
109. 1 0.2
faikkerdd-meiiu--eee
0.15 - 0.12
0.21 0.002
0.13 0.04
73.57 1 2.64
walker2d- meieruu2--e
0.03 0.02
0.03 103=00 0.01
0.040.0011
81.68 2.48
Average Relative Improvement
0.09
0.09
0.06
Sigmoid
Rank
Constant
TD3+BC
halfcheetah medium- ex
-0.02 0.03
-0.02
0.03
-0.02 = 0.02
94.82
0.33
0.13
0..88 0.16
-0.0
54.38
1.24
halfcheetah-medium--
0.02
-.00
-0.01 0.01
64.63 2.15
wopper-medium--expeee
0.133
-.216
0.03 0.05
98.79 + 12.57
konper--ediiu--relayy-a
0.08
0.21+0.03
0.24
81.43 + 27.77
hopper- medium-v2
0.51 0.17
0.02
0.6 0.19
59.12 - 3.39 3.39
walker2d-medium- expert-v2
0.0 0.01
0.01 +.001
0.0 0.0
111.87 1 0.17
walker2d-nnn walker2d-medium
0.12 + 0.04
0.09 1.07
0.05 0.02
84.95 1.85
walker2d-mer 2eddddd-dddm2222
0.01 0.03
0.08 0.01
= 0.07
84.09 1.94
Average Relative Improvement
0.06
0.09
0.1
Next, we study how the performance of HUBL vary over different epochs. Specifically, we provide plot of the achieved normalized average returns over different epochs for HUBL with TD3+BC on hopper-medium--2 in Figure
22

---

Published as a conference paper at ICLR 2024
Normalzedd Return
Nurmalied Returu
LTTEC
Normalized
Epoch
Epoch
Epoch
(a) HUBL with constant labeling
(b) HUBL with Sigmoid labeling
(c) HUBL with Rank labeling
Figure 4: Average normalized return of HUBL with TD3+BC on hopper-medium-V
D.3 ROBUSTNESS OF HUBL TO
In this section, we demonstrate the robustness of HUBL to a. Specifically, we provide the average relative improvements of HUBL on D4RL datasets in Table Notice that most average relative improvements are positive, which shows that HUBL can improve the performance with different values.
Table 7: Average relative improvements of HUBL under different a's
Sigmoid
Rank
Constant
a 0.001
0.01
0.01
0
a 0.01
0.01
0.01
0.01
0.1
0
0.02
0.01
(a) ATAC
Sigmoid
Rank
Constant
Q 0.001
0.05
0.05
0.03
a
0.01
0.08
0.05
0.10
Q 0.1
0.11
0.16
0.13
(b) CQL
Sigmoid
Rank
Constant
a 0.001
0.02
0
-0.04
a 0.01
-0.01
-0.02
0.1
0.09
0.09
0.06
(c) IQL
Sigmoid
Rank
Constant
a 0.001
0.01
0.01
0.01
0.01
0.01
0.01
0.1
0,06
0.09
0.06
(d) TD3_BC
The selected a's reported in Table
Table 8: Selected a's
Sigmoid
Rank
Constant
ATAC
0.01
0.01
0.1
CQL
0.1
0.1
0.1
IQL
0.1
0.1
0.1
TD3_BC
0.01
0.1
0.1
23

---

Published as a conference paper at ICLR 2024
D.4 DATA COLLECTION FOR WOORD
We collect data for Meta- World tasks using normalized rewards with goal-oriented stopping. Specifi- cally, given that the original rewards of Meta- World are in [0, 10], we shift them by -10 and divide use the hand scripted policy given in Meta- World with different Gaussian noise levels in {0.1, 0.5, 1] to collect 100 trajectory for each task. In this process, trajectory ends if (i) it reaches the max length of a trajectory (150); (ii) finishes the goal. Note that we follow the same rule when testing the performance of a learned policy,
by 10, so that the normalized rewards take values in [-1,0] 0). Then,
D.5 BASE PERFORMANCE AND STANDARD DEVIATIONS FOR META- WOORD DATASETS
provide the performance of base offline RL methods and standard deviations Table and
10
baseline, and also representative for imitation learning and inverse reinforcement learning methods (Geng et al. 2020 2023 Fu et al. 2017). Since
We also consider behavior cloning (BC)
the behavior policy is the scripted policy with Gaussian noise, BC can effective recover the scripted policy and thus is especially competitive.
Table Relative normalized score improvement for HUBL and normalized scores of base methods on MW datasets (part I)
Sigmoid
Rank
Constant
ATAC
BC
nutlon--press-v-nnoieee
-0.0 0.03
0.01 + 0.00
0.01 0.01
58.12
0.45
-58.79
Outton-prrss--2-noii0...
-0.0 = 0.03
-0.03 0.05
0.01
66.16
0.93
-59.58
button-prees----noiis)
0.05 1 0.07
-0.0 0.04
-0.01
62.72 3.84
-61.24
push-back-v2-nnisen.
0.1 0.03
0.04
0.15
-87.12 13.89
104.85
push-bacc--22-niiecU.
0.11 0.07
0.19 0.05
0.17
133.28
14:24
-14.663
push-backk-v-nniiel
0.04
0.00
13.87
-13.63
reach-v2-noise..1
-0.0 0.000.00
-0.0
0.07
17.19
0.76
-18.31
reach-v2-noisel.5
0.37 + 0.07
0.46 0.06
0.43+0..0
33.91
5.62
-24.57
reach-v2-noisel
0.39 + 0.05
0.35 + 0.24
0.5 0.05
46.58
3.98
-43.79
Average Relative Improvement
0.11
0.11
0.14
Sigmoid
Rank
Constant
COL
BC
bullon-prees-v2-noised
0.0 0.03
0.02 I 0.05
0.03 0.03
59.67
1.69
-58.79
hutton-press-v2-noicel
-0.04 + 0.05
-0.02 0.01
-0.02
-58.48 0.85
-59.58
button-prees-v2nnoisel
00.08 - 0.12
0.02
0.14
-67.18
-61.24
push-bbackxv2-noiee..
-0.07 0.07
0.02
0.02
-81.78
104.85
push-baak--v2-noicc0.5
0.25 0.12
0.11
0.19
2..4.61
-13.663
push-back--2-nooiel
0.06 0.13
0.11
0.1
133.66
143.63
reach-v2-noise0.1
-U.04
00.01 1.01
-.00
-
-18.31
reaise-2-noosee0.5
0.51 + 0.05
0.18 0.53
0.4 0.27
-41.02
18.79
-24,77
reach-v2-noisel
0.8 + 0.02
0.15
0.81 0.00
1007.14 32.75
-43.79
Average Relative Improvement
0.16
0.1
0.17
Sigmoid
Rank
Constant
IQL
button-prees-v2-noiise
102:0.00
0.03
0.03 0.03
59.87
1.32
-58.79
button-press--2-noised.
+0.01 0.05
-0.01
-0.14
-00668
-5.588
button-prees--2nnoiiee
-0.09
0.01
0.01
-73.96
-61.24
push-back--22nnoisc..
0.01 0.04
0.04
0.03
$3.79=5.45
104.85
.ush-buack-2--noise..5
0.02 0.06
0.04
-.00
-124.66
-134.63
push-back-v2-moisl
0.01
0.03 0.00
-..0
143.26
6.31
-14.63
reach-v2-noisc0.1
-0.03 + 0.03 0.03
0.00
-0.0
17.37
0.87
-18.31
reach-v2-noosee..5
0.05 + 0.13
0.02 0.08
0.02 0.24
31.51
9.39
-2.57
reach-v2-nnisel
0.12
0.05 0.09
0.01 0.37
42.23
7.4
-43.79
Average Relative Improvement
-0.02
0.02
0.00
Sigmoid
Rank
Constant
TD3+BC
BC
button-prees-v2-noiiss
0.25 + 0.0
0.23 1.0
0.23 0.02
-77.68
22.14
-5,799
button-prees--2-noiie0..
0.044 + 0.05
0.04 1. 0.06
0.09 1 0.07
1.68 59.54
59.58
button-press--2nnoiell
0.02 - 0.07
0.12 0.12 0.23
0.12
$1.03-5..
-61.24
push-baak--v2noiiee.
0.08 + 0.01
-0.01 0.02
0.05
-85.44 5.05
-00.85
push-buck-v2-noiict..
0.07 + 0.07
0.05
0.05
-127.95 9.88
-134.63
push-back-v2-noise.
0.01
0.01
0.04..0.02
130.72 15.6
-143.63
reach-v2-noiseh.1
-0.06+0.00
-0.02+0.00
0.0
17.11
1.28
-18.31
reash-v2-noise0.5
0.12
0.25 0.14
0.18
9.66 65.48
40.69
-24.77
reach-v2-noisel
0.18
0.04 1 0.12
0.1
-43.79
Average Relative Improvement
0.07
0.07
0.06
24

---

Published as a conference paper at ICLR 2024
Table 10: Relative normalized score improvement for HUBL and normalized scores of base methods on MW datasets (part II)
Sigmoid
Rank
Constant
ATAC
assembly-2-noiiel..
-0.01 + 0.03
0.01 + 0.04
-0.0 0.08
-70.22
4.61
-71.66
aseembly-x2-moieel.
0.01 0.02
0.03
0.08 : 0.0
135.77
1.58
122.533
assembly-v2-moisel
0.02
0.04 -
140.12
2.45
138.63
handle-pres--sdee22nmmiis
0.03 0.01
00.02 0.00
-0,03
-32.88 0.64
-36.28
mamull-press-side-22--ooe
0.03
0.02
-0.01 +: 0.03
-33.04
-34.3
suanll--pres-sid----nnuiss
0.04 0.12
0.02 0.1
0.09 1: 0.06
35.33
3.7
-35.72
plate-slide-bucte
salic--buk-sidde-2--iiees
-0.0 0.04
0,03 0.02
0.02 0.05 0.05
33.007
0.87
-37.75
hate--side-back-eeee --e------iieee-2-mie0...
0.07 + 0.41
0.17 0.29
0.15 0.02 0.02
68.9
10.92
-65.47
plate-slide
bbw.kbicccck2222 sate-sliddebbcc-sideeee noisel
1.7333..
0.44
97.9
25.46
128.28
Average Relative Improvement
0.03
0.07
Sigmoid
Rank
Constant
CQL
BC
assemby-v2-noisee.J
0.03 + 0.06
0.06 + 0.17
0.06 + 0.07
-76.29 + 9.41
-71.76
aseemby-xx-noiie0.
-0.0
0.01 + 0.04
-0.01 0.02
-131.35
2.62
-12..53
assembby-x2-muisel
0.0
0.0 0.01
138.94
0.97
-138.63
handle-press-iieee----ooeee
0.11
+.00
0.02
-34.47
4.54
-36.28
pannlle-rees-side----nooee
0.32
0.38
0.46 - 0.01
-70.96
29.39
-34.3
0.15 1 0.12
0.1310.2
0.4 0.05
55.3
33.42
-35.72
plate-slide-bee
bildd-hack-side----niiie
0.05 + 0.03
0.10
0.03 +
-36.31
2.33
-37.75
slide--acc--sie--22222
0.03 0.09
-0.09 0.47
0.113
36.72
2.56
-65.47
plate-slide wls.--iddebcc--dee----ee..
0.36 0.31
1 0.12
67.63
128.28
Average Relative Improvement
0.12
0.09
Sigmoid
Rank
Constant
IQL
BC
assembly-v2-noise..
0.02 = 0.07
0.01 + 0.04
-0.0 0.04
-72.06 # 2.34
-71.76
assembly-v2-noisel
.0.01 0.01
0.011 + 0.02
0.01 0.04
-122.97 6.12
-129.53
sacmbly-v2-noisel
-0.01 +0.0
0.03
0.0
-129.01
1.62
-138.63
handle-rrss-iidb222-noosdd
-0.05 0.01
-0.05
-0.05
-34.1
-36.28
bandll-pres---idee-2-nnohhhh
0.06 0.06
0.007
0.07 - 0.04
-37.28 +.25
-34.3
handle-pres--idd-----oiss
0.11
0.03
0.02
+3111
-35.72
plate-slide-bace
silid--ackk-idee---nniss-
0.01 0.02
-0.01 0.001
0.01 - 0.03
-34.4 1.23
-37.75
0.1
0.04
0.03 0.08
34.03 + 1.83
-65.47
plate-slide-back wew--bbcccid----oooee
0.05 0.18
0.01 0.29
0.04 0.35
43.95 = 10.29
128.28
Average Relative Improvement
0.01
0.01
Sigmoii
Rank
Constant
TD3+BC
BC
ussembby-v2-noosee
0.01 1 0.02
0.02 1 0.03
0.03
0.01 1 0.03
72.44 1 6.03
-71.76
assembly-x2-moisee..
0.02
0.02 + 0.02
0.02
0.02 - 0.04
131.62 5.25
-129.53
assembly-v2-noisel
0.0
0.01
0.0 0.01
-10
-13.663
handie-pras--sid--2--niiss
-0.07
-0.05
-0.05 - 0.02
-.10
-36.28
handle-pres--siee---nnoeee
0.08 0.04
0.08 0.03
0.07 0.05
1.024
-34.3
bandleepress-side----ooe
0.08 = 0.08
0.06 0.08
0.13 - 0.17
52.22
-35.72
siiid-back-iide----nisse
0.16 + 0.06
0.09 0.05
0.03 - 0.05
1.52 -3.33
-37.75
plate-slide-buck siddd--bucck-side-2-nneeee//l
0.17 + 0.24
0.1 0.33
0.07 1: 0.32
40.17 1.64
-65.47
plate-slide- haat-slide-bacc-sideee seddd-ack-side-2--nne-
0.19 = 0.35
0.35 + 0.28
0.07 0.24
99.74 - 31.24
-128.28
Average Relative Improvement
0.07
0.05
D.6 EXTENDED EXPERIMENTS
In this section, we study the performance of HUBL various of scenarios.
D.6.1 HUBL ON EXPERT DATA
We study the performance of HUBL expert collected data. Specifically, we conduct experiments of HUBL with TD3+BC on expert datasets in D4RL. The results are reported in Table II Note that HUBL is able to provide performance improvements in most of the cases. But the room for improvement on such datasets is small - for any offline learning technique. Therefore, due to finite sample error, HUBL. may occasionally hurt performance empirically
25

---

Published as a conference paper at ICLR 2024
Constant
Rank
Sigmoid
Base Performance
halfcheetta--expert-22
1%
0%
1%
97.09
hopper-expert-v2
3%
3%
3%
108,46
walker2d-expert-v2
0%
1%
0%
110.14
Table 11: Relative improvement HUBL for TD3+BC on expert data
D.6.2 HUBL WITH ONLINE RL METHOOSS
We provide experiments where we combine HUBL with DDPG on D4RL datasets, without any additional designs for the offline RL setting. The results are provided in Table T2
Performance of DDPG
13.43
Relative improvement of HUBL with sigmoid 53%
Relative improvement of HUBL with constant 74%
Relative improvement of HUBL with portion 90%
Table 12: Relative improvement of HUBL for DDPG on D4RL. datasets
Note that HUBL is able significantly improve DDPG's performance in relative terms. However, the absolute performance of DDPG+HuBL is much worse than other methods we considered the main paper. This is expected, as HUBL designed to augment offline RL algorithms, opposed to making a general online RI. algorithm (like DDPG) work also offline.
To this more clearly, consider toy example with three actions d1, 02 dz state 8o- Suppose that and a₁ and a2 are all out-of-support, As result, without additional designs (such as pessimism) for offline RL, the policy may take a and 02 regardless how we the heuristic, since the heuristic here would only affect the value for in-support actions and not the value for out-of-support actions. Thus, HUBL cannot turn an online RL algorithm an offline RL algorithm; can only enhance an algorithm that is designed for offline RL already.
D.6.3 HUBL IN STOCHASTIC ENVIRONMENTS
To test the performance of HUBL in stochastic environments, we construct random version of walker2d-v2 by adding 3e the action passed the environment, where a uniform random variable taking values in [-1, 1]. and 3 is scalar controlling the amount of randomness. Note that we keep the scale of noise in [-1, 1], and the action variable itself is normalized to [-1, 1] in D4RL. With the new environment, we regenerate the data using the original walker26-meidum-12 policy and apply TD3+BC with HUBL. The results are presented in Table 13
Sigmoid
Rank
Constant
Base Performance
8 = 0.0001
6%
6%
4%
66.99
0.001
6%
8%
7%
66.73
8 0.01
4%
4%
6%
64.23
4%
-4%
5%
28.09
Table 13: Relative improvement of HUBL for TD3+BC on stochastic walker2d-meiduu--22
The base performance decreases as the noise increases, as expected. However, HUBL is still able to provide some performance improvement under this stochastic setting. We also notice that the constant labeling function is more robust the noise. This makes as the constant function is not sensitive to the heuristic estimation. But if there is more noise, the performance HUBL will further degenerate, which is limitation of HUBL.
26

---

Published as a conference paper at ICLR 2024
D.7 EXTENDED RESULTS FOR ABSOLUTE IMPROVEMENT
We report the experiment results in absolute improvement of normalized score for D4RL, and Meta- World.
Table 14: Normalized score improvement for HUBL and normalized scores of base methods on D4RL datasets.
Sigmoid
Rank
Constant
ATAC
BC
halfcheetah-mediee -enpeeaa--meduum--ee-e--------
2.13 = 0.35
1.55 1.5.F0 0.96
1.76 0.14
92.5
45.83
halfcheetah- mediun-replay-v2
halicheetal-mmdiu--epppypa
0.43 1 0.38
0.96 0.27
1.04
46.89 1: 0.25
37.73
halfcheetah halfchhctaa-mmciun-2222
-0.05 + 0.71
0.16=0.22
0.85 1855:00 0.35
5
42.92
hopper-mediuue v
0.36 + 0.03
0.24
0.27 0.1
111.04
57.51
hopper-medium-replay-
0.01 + 0.48
0.21 0.77
0.26 0.32
101.23 0.59
0.59
32.22
hopper-medium--2
0.44 +: 3.28
1.57 + 6.09
-235 2 12.85
85.59 + 4.76
53.81
walker22-mcciim--xxerr
0.08 + 0.4
1.72 0.5
0.5
0.26 = 0.6
111.99 0.8
105.3
walker2d
mere-2dd-ediu-e-eee reppuum--epla--222
2.76 2.23
7.15 0.31
4.38 - 6.53
84.22 2.7
22.78
walker2d-medium v2
1.6+0.5
-1.95 4.62
1.35 0.17
87.2 0.67
64.18
Average Improvement
0.75
1.31
0.87
Sigmoid
Rank
Constant
CQL.
RC
halfcheetah-medeee medium-eppee-----tt
1.55 1.75
2.66 4.77
13.59
81.96 1 9.93
45.83
saffihhctta-mmedium-eee
4.14 0.22
7.19
0.73
5.26 0.4
40.98 0.81
37.73
halfcheetah-mediine medium-v2
-5.12
-0.86 0.72
0.68
51.05 0.83
42.92
hopper-mediuee wwdiim-eexpeetteee expert-v2
1.08
5.21
3.76
78.39 44.9514.55
57.51
3.29 0.79
0.69 1.31
2.09
97.21 + 3.15
32.22
hopper-mediumm-v2
12.81 10.16
10.87 11.68
=.51
70.27 = 8.77
53.81
walker2d-medium- med-ime--deeeee- -2diiu-exxpert-22
1..44
19.74 0.82
19.36 - 0.14
89.38 1 17.54
105.3
walker2d-medhe wwwiimmrrepla---eaa....
4.84 184=17.22 17.24
24.07 + 3.58
2.224 4 2.24
63.35 1 6.22
22.78
3.62 + 3.86
4.14 + 2.23
2.01 + 11.8
79.41 + 1.82
64.18
Average Improvement
8.54
11.02
9.64
Sigmoid
Rank
Constant
IQL.
BC
aiffcheetah-mediune -eperrt-v2
0.2
-0.28
0.19
-4.99 = 2.6
92.92 1 0.3
45.83
-1.51
-0.08 0.83
-1.86 1.21
44.35 - 0.21
37.73
halfcheetahh medium-v2
0.128
-0.06
-0.68 0.36
49.61
hopper-medium-exppee expert--v2
-4.52
-2.88
-10.22 12.89
108.13+ 2.69
57.51
hopper-medium-regii
4.799
31.57
33.41
32.22
hopper-medium--v2
1.78 2.05
1.27
-0.68
61.03 3.27
53.81
walker2d-mmedimm-ee
1.53
1.06 1.01
0.99 0.28
109.59 0.2
105.3
walker2d-meeium--ee
10.78+99.11
15.49 1.61
9.2 0.2+33.04
73.57 + 2.64
22.78
walker24-mediunn-22
2.17 1.46
2.54 0.9
0.71
81.68 1 2.48
64.18
Average Improvement
5.42
5.4
3.17
Sigmoid
Rank
Constant
TD34BC
BC
sirchhetta-mediummec expertt-22
-1.52 + 2.92
-2.11 2.51
-1.74 1.744:2 2.29
94.82 1 0.33
45.83
halfchectah-meddee www.itttthhmmciiiim-ppapaa---
-5.48 1.47
-4.41 8.83
-0.24 0.82
54.38 + 1.24
1.24
37.73
halfcheetahh sullchheetahmmedum--22
-4.8
2.85
-0.76 0.94
64633 1 2.15
42,92
hopper-medium-eene -x
+..2
15.45 19.55
2.61 4.79
98.79 12.57
57.51
hopper-medium--rplayyy
6.4
17.23 2.15
19.7
81.43 27.77
32,22
hopper-medium-v22
10.26+10.29
41.58 + 1.11
35.2
59.12 4.33
53.81
walker2d-mediune
medium-exxertt-22 expert-v2
0.36
0.92 0.95
0.33
111.87
maalker2d-meduum-eee
3.1
7.83
3.86 1.91
84.95 1.85
22.78
walker2d--ediumm-22
-0.75 2.86
6.81 + 1.13
0.37 5.76
84.09 -
64.18
Average improvement
4.23
5.76
6.86
27

---

Published as a conference paper at ICLR 2024
Table 15: Normalized score improvement for HUBL and normalized scores of base methods on MW datasets (part I)
Sigmoid
Rank
Constant
ATAC
BC
nutton-press--22noiis)))
-0.09 + 1.76
0.43 0.62 0.62
0.41 +0.82
-58.12+ 0.45
-58.79
button-press--2-nniisdl
-0.21 1.8
-2.22 +.5
0.86 2.75
-64.16
-9.58
button-press-V2-noiis
-3.42
-0.23 2.71
2.29
62.72
-61.24
push-back-v2-nodit
3.50 6.53
5.12
87.12
13.89
100.85
push-hack--22noisee.3
14.47
25.88
2..0
-133.28
14.24
-34.63
pusi-baac-x22nuisell
5.38 7.51
3.28 1 12.83
7.94 9.56
141.29
13.87
-13.63
/cach-v2-noiscO.1
-0.03 = 1.08
-0.04
0.0 1.14
-17.19 0.76
-18.31
reach-v2-noise0.5
12.53 + 2.27
15.6
14.49 + 2.68
33.91 + 5.6.2
224.77
icach-v2-noiscl
18.22 + 2.13
16.14 11.32
23.09 + 2.54
46.58 = 3.98
43.79
Average Improvement
6.18
6.94
8.96
Sigmoid
Rank
CQL
button-prees--2-nnossll
0.23 196
1.32 = 2.71
1.51 1.87
-59.77 1.69
-58.99
buttoo-ppres--22noiisU...
-2.29 2.67
-1.18 0.55
-1.77
58,48
-9.58
-5.37 = 7.82
5.46 1.68
2.7
-67.18
-61.24
pust-backkv2-noisee...
-5,54 5.79
1.54 3.86
1.33 1.89
11.7812.88
-10.855
pust-hack-v2-noisct..
30.82 14.66
27.34
23.15 14.97
-124.64+2.45
-134.63
push-hack-v2-nniset
7.69 16.81
14.41
13.58 7.12
133.66 8.17
-143.63
reach-v2-noisc0.11
-0.77
-0.22 0.98
-0.67 0.41
-18.5
-1.31
icach-v2-noisc0.5
21.05 + 1.89
7.28 21.76
16.49
-1..02
18.79
-24.57
ncash-v2-noisel
86.19 2.49
37.15 15.77
86.65 + 3.53
107.14 32.75
44.799
Average Improvement
14.67
10.34
15,93
Sigmoid
Rank
Constant
IQL
bc
button-prees--2-noiidd.
0.53 = 0.6
1.87 1 0.75
1.83 1.33
599877 = 1.32
-5..79
bullon-prees--2-oooee0..
-1.09 1.25
0.63 1.16
88.45 7.68
60.68 1 0.29
-5,558
button-prees-v2-noisee
1.61
5.17
3.19
73.96
-61.24
p
2.93 3.28
3.11
2.47
5.44
-104.85
0.391:11
4.62 6.24
6.016
7.445
-134.63
push-haack-2-nnisel
1.82
3.71
-0.35 #.1
-143.26
-14.663
reace-v2-nnoseo.11
-0.55 0.2
-1.0
0.68
17.37 - 0.87
-18.31
rcach-v2-noisco..5
2.48 - 4.95
-0.88
0.95
7.66
31.51
9.39
-24.57
reach-v2-noisel
3.69 1 5.46
2.12 112:40 4.0
0.33 1 15.48
42.23
7.4
43.79
Average Relative Improvement
-0.41
2.04
-0.7
Sigmoid
Rank
Constant
TD3+BC
BC
button-prees-y2-nooss.
19.12 0.22
18.19
18.01 1 1.73
77.68
22
-5,799
button-prees--22nniisee
-2.47 + 2.73
-2.11 3.77
-5.36 + 4.18
-59.54
1.68
-59.88
butlon-press.v2-nnoisel
1.71 5.97
9.71 = 1..1718.70 18.76
8.4
-81.03 1: 5.79
-61.24
push-hack--22nnoish..
7.18
-1.06 1.65
4.10
-85.44
-104.85
push-hack-v2-noiset.
9.43 8.7
6.1
6.43 + 12.98
-127.95
-134.63
push-back-v2-nnisel
1.26 6.0
8.05
2.35
-139.72 15.6
-143.63
reach-v2-noise0.1
-0.99
rcach--2-noisee..5
5.94
5.67
6.27
40.69 9.66
-24.57
reach-v2-noseel
8.4
2.34 + 7.98
6.74 7.36
65.48 + 2.85
-43.79
Average Improvement
5.41
4.94
4.34
28

---

Published as a conference paper at ICLR 2024
Table 16: Normalized score improvement for HUBL and normalized scores of base methods on MW datasets (part II)
Sigmoid
Rank
Constant
ATAC.
assemblyv2-noissel
-0.99
1.0
-0.23 5.66
-70.22
4.61
-71.76
mveembly-x-nooieel
1.05 2.43
3.59 3.02
11.13 6.31
135.77
1.58
122.533
assembly-v2-noieel
1.99
5.75 1.07
140.12
2.45
138.63
handle-press--idee--------e
-0.93 0.41
-0.49
-0.88
-32.38
-36.28
peanlle-pees-siie-----ooo
0.08 0.86
0.74 0.94
-0.35 1.06
-33.04
0.96
-34.3
press-side-v2nnoiiel
1.25 4.2
0.73 0.55.
3.07 1.05
35.33
3.7
-35.72
plate-slidc-one
-0.07 + 1.44 1.44
-0.98 0.82
0.77 + 1.96
-37.07
0.87
-37.75
plate-slide-buce back-side-v22moisse
5.15 28.23
11.72 20.14
10.47 1.41
68.9
10.92
-65.47
plate-slide slisee-bcbc-bideeeee--e -2-iie--
20.75
42.45
43.25 10.94
97.9
25.46
-28.28
Average Relative Improvement
2.29
6.64
7.43
Sigmoid
Rank
Constant
CQL
BC
aasembly-v2-noisel.]
2.37 4.62
4.2 12.62
4.36 + 5.55
-76.29
9.41
-71.76
asemmly.-v-mooiee..
-0.57 2.00
1.44 5.55
-1.21 2.87
-131.35
2.62
-22.53
asscmbly-v-nnoieel
-0.01
-0.35
0.56 + 2.56
-138.94
0.97
-138.63
handle-press-sidee-2---oo
3.79
+
0.73 6.56
-34.47 4.54
-36.28
mandllepres--sidee---noooe
23.05 + 7.99
26.8
32.31 + 0.94
-70.96
29.39
-34.3
8.33 6.37
6.94=10.90
22.01 1::01122 2.89
-55.3 = 33.42
-35.72
plate-slidc-ban
1.83 + 1.21
-0.02 3.88
0.95 + 3.95
36.33
2.33
-37.75
plate-slide-back
0.96 + 3.25
33.16
17.37
4.994 18.35
36.72
2.56
-65.47
noieiii
24.46 + 21.13
24.72 + 8.42
23.04 + 12.37
67.63
20.56
-18.28
Average Relative Improvement
7.14
6.67
8.65
Sigmoid
Rank
Constant
IQL
BC
ussembly-v2-noisell
1.18 1 4.78
4.78
0.49 3.07
0.02 1 3.2
72.06 1 2.34
-71.76
assembly-v2-noiscu..
1..99 1.27
0.9 1.89
1.13 + 4.46
122.97 6.12
-129.53
assembly-v2-noisel
0.33
-0.76
0.39 + 3.85
-129.01 1.62
-138.63
fandle-press-iidee---nwoo
-1.56
1.67
34.1 1.55
-36.28
handle-press-side-2--ndk--
2.07
2.44
2.56 1.54
37.28 0.25
0.25
-34.3
5.06
1.33 4 1.52
1.12 3.27
46.31 3.11
-35.72
plate- motee-side--ie
hack-side--22-muisee
0.76
0.18110.4
1.122.12
34.4
1.23
-37.75
2.3 3.47
1.6
0.92 2.7
34.03
1.83
-65.47
plate-slide-back bew-bbbakk-s-----eeeee
7.7
12.54
15.22
43.95 = 10.29
128.28
Average Relative Improvement
0.41
0.14
0.34
Sigmoid
Rank
Constant
TD34BC
BC
ussembblyv2-noisee
1.08 1 1.69
1.16 1 2.35
0.71 1 1,95
72.44 1 6.03
-71.76
assembly-v2-noiseu..
0.56 3.001
2.16
2.6 +: 5.8
131.62 = 5.25
-129.53
assembly-v2-noite
0.48 0.46
0.71 1.39
0.71
0.03 1.92
-138.0
133.63
handlle-ress-side--2----
pres---idde----nniee..
3.25 +: 1.77
-1.91
-1.67
35.0
-36.28
bandle-press-side-----oo-
3.17 1.04
2.97 1.82
40.24
-34.3
ponull--rres-sid------ne
4.31 + 4.29
3.39 + 3.97
6.93 8.89
-52.22 +529
-35.72
plate-shie seee-side-bue
6.11 2.3
3.67 1.77
1.09 2.11
39.31
1.52
-37.75
6.77 9.5
4.01
13.35
2.91 1: 13.02
40.17 1.64
-65.47
plate-slide- sele--aac--iie-----nae
17.06 + 32.1
31.78 + 25.94
6.48 22.36
-91.74
31.24
122.28
Average Relative Improvement
4.14
4.46
1.65
E LIMITATIONS AND FUTURE WORK
In our current framework, HUBL calculates heuristic values as Monte-Carlo returns. This feasible in practical scenarios where data comes in the form of trajectories, but much of offline RL literature experiments with batch datasets that consist of disconnected transition tuples. For such datasets, HUBL requires other heuristic calculation strategies.
Empirically, we have showed HUBL's utility on benchmarks where observations are full iillimfformaton low-dimensional state vectors and state transitions are deterministic. While these benchmarks are standard in offline RI., evaluation in more stochastic domains where observations are partial and nigh-dimensionall, such robotics, will give more complete picture of HUBL's utility. So would evaluating HUBL with model-based offline RL such as MOReL (Kidambi et al. 2020). Also, HUBL not applicable bootstrapping- -freeereeeeeeeeeeeeeeeeeeeeeeeee free offline RL methods (Schaefer et al. 2007 Deisenroth and
Rasmussen 2011 Depeweg et al. 2016 Swazinna et al. 2022)
On the theory side, our analysis focuses state-dependent heuristics. In practice, though. heuristic estimates commonly depend on full trajectories, and our theory needs adjustments to account for this. Same goes for applying our theory model-based offline RL such MOReI..
29

---

Published as a conference paper at ICLR 2024
For future direction, aim to develop better heuristics and blending strategies for batch datasets and sparse-reward problems. We also plan to apply HUBL to other tasks in healtheare and finance (Alaluf
et al. 2022
Geng et al. 2018 2019a b) where collecting data is especially expensive.
30


Reviews:
[{"review_id": "7qlCvqtGnU", "reviewer": "Reviewer_8Sx2", "summary": "The paper introduces the HUBL technique for offline RL algorithms, which partially replaces the bootstrapped values with heuristic ones estimated using Monte-Carlo returns.\nTheoretical analysis has been made to understand the improvements brought by HUBL to the original offline RL algorithm, as well as its associated bias and regret.\nExperimentally, results on the D4RL datasets and Meta-World benchmarks show that HUBL offers an average improvement of 9% over the current four SOTA offline RL algorithms.", "questions": "* Please improve the presentation of figures and tables in the paper as i mentioned above.\n* I've noticed that on tasks where baseline offline RL algorithms already perform well, HUBL might decrease the performance. Is there a way to ensure that its enhancements are consistently non-negative?\n* On D4RL, HUBL shows significant improvements over baseline algorithms for the hopper task several times. I wonder whether there exist any shared or general characteristics in situations where HUBL offers substantial advantages. Could this be discussed, or did I perhaps overlook any mention of this aspect?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed HUBL is a general technique that can be seen as a correction to the offline dataset itself, improving the performance of offline RL algorithms.\n* Through theoretical analysis, the introduction of HUBL is discussed as an MDP reshaping, and the analysis of bias and regret is conducted.\n* Extensive experiments empirically demonstrate that HUBL is indeed an effective enhancement technique.", "weaknesses": "* **Presentation**:\n    * The presentation of the experimental results in the graphs lacks clarity. The absence of a horizontal baseline at 0 makes it unclear whether there's an improvement or decline. I believe a horizontal baseline at 0 should be added, and different colors could be considered to depict increases and decreases.\n    * The experimental tables in the appendix have a similar problem. The best performances should be bolded for easier readability.\n* **Limitations**:\nAs the authors discussed in the limitations section, offline datasets based on disconnected transition tuples are challenging to utilize with the HUBL trick unless heuristic values are computed during the construction of datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zD7Yqjd1Ne", "reviewer": "Reviewer_Gwab", "summary": "This work presents Heuristic Blending (HUBL), a new technique that can be attached to offline RL methods to improve their performance. \n\nHUBL works by modifying the rewards and discounts in the dataset that the offline RL algorithm consumes, blending heuristic values to partially replace bootstrapping. \n\nTheoretical and empirical results show HUBL consistency when improving several offline RL algorithms performance by 9% on average over D4RL and Meta World datasets.", "questions": "I've tried to detail most of the limitations and weaknesses of the proposed model in previous section, with some points that would need to be addressed in a rebuttal.\n\nOverall, I see here a strong manuscript with some ideas that are adequate for an ICLR conference.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* This paper meets very good originality, quality, clarity and significance criteria. Good job!\n\n* Section 2 does analyze the main differences of the proposed approach with respect to cited works. It is clear that no previous model has addressed the data relabeling as it has been proposed in this manuscript for the particular setting of offline RL. The use of both data relabeling and heuristic in combination with RL has been explored before, but not in the offline scenario.\n\n* HUBL is an original method to incorporate into existing offline RL methods. It improves them without any need for a complicated modification within any algorithm; only a relabeling of the dataset is needed. It is indeed done dynamically depending on the return of every trajectory. This makes this method quite significant, especially when working with low-quality data.\n\n* All the claims made by the authors are addressed through a comprehensive theoretical and empirical study. The paper is very well written; the notation is excellent, and all the details and formulas are clear.\n\n* The experimental setup proposed is thorough. The paper analyzes how the modification of up to four state-of-the-art offline RL methods behaves across more than 25 different benchmarks.\n\n* The theoretical analysis developed in Section 5 (and the corresponding annexes) is very robust and truly helps to understand what implementing HUBL entails.", "weaknesses": "* For someone not already familiar with offline RL, it can be challenging to follow the comprehensive theoretical analysis developed in this paper, especially in the appendices. \n\n* This claim should be justified: \"Despite their strengths, existing model-free offline RL methods also have a major weakness: they do not perform consistently.\" It would be fantastic if the authors could provide some evidences regarding this issue, as they do in section 3.2 (second to last paragrpah), but providing more details.\n\n* In the experiments section I miss the learning curves where a reader can compare how the reward evolves with an without HUBL.\n\n* I believe the manuscript needs a brief discussion on how the proposed model would be applied to problems with sparse rewards. For instance, in environments like Antmaze, has it been tested?\n\nMinor comments:\n\n-Section 4.1 HUBL introduce -> HUBL introduceS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cZ9GJxKSx4", "reviewer": "Reviewer_zyzv", "summary": "The paper deals with offline RL. It focuses on Q-function based offline RL methods and presents a heuristic to increase the performance of these methods. In extensive experiments a performance increase is observed on average.", "questions": "No questions, but a few notes and comments:\n* \"methd\" -> \"method\"\n\n* At \"Step 1: Computing heuristic ht\" ht should be set boldmath, analogously in Step 2 and Step 3.\n\n* At `Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep\ndata-driven reinforcement learning, 2020` the indication where it was published is missing.\n\n* In the references there are some unintentional lower case letters, e.g. mdp, Monte carlo.\n\n* Just for completeness, I’d like to point out that there are also purely model-based batch/offline RL methods that do not use Q-function and are thus bootstrapping-free [1-4]. See [5] for a discussion. Since the authors have precisely formulated at the very beginning that this paper deals with the algorithm class with Q-function and bootstrapping, a mention of these bootstrapping-free algorithms is probably not necessary.\n\n[1] Schaefer et al., A recurrent control neural network for data efficient reinforcement learning, 2007\\\n[2] Deisenroth and Rasmussen, PILCO: A Model-Based and Data-Efficient Approach to Policy Search, 2011\\\n[3] Depeweg et al., Learning and policy search in stochastic dynamical systems with Bayesian neural networks, 2017\\\n[4] Swazinna et al., Overcoming model bias for robust offline deep reinforcement learning, 2021\\\n[5] Swazinna et al., Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning, 2022", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The presentation is very good.  I would like to emphasize that the limitations mention both the restriction to trajectories and the lack of stochastic MDPs among the benchmarks used. This is exemplary. Also that already in the first sentence `We propose Heuristic Blending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping` it is clearly stated to which class of algorithms the paper refers.\n* The method is investigated as a modification of not just one, but four state of the art bootstrapping-based offline RL algorithms.", "weaknesses": "none", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xAHiDTihCy", "reviewer": "Reviewer_iqAo", "summary": "The authors propose a technique known as Heuristic Blending (HUBL) to enhance the performance of model-free offline Reinforcement Learning (RL) algorithms that are based on value bootstrapping. The primary aim of this approach is to mitigate the challenges associated with bootstrapping and achieve stable performance. \n\nHUBL essentially adapts the rewards and discount factor within the offline dataset utilized by the base offline RL algorithm. It achieves this by modifying the reward through a blending process, combining it with a state-specific heuristic derived from the Monte Carlo return of the behavior policy. It also reduces the discount factor discount factor. The degree of reduction in the discount factor and the reward blending is determined by a trajectory-dependent blending factor. This factor is designed to be high for trajectories in which the behavior policy performs well and low otherwise. \n\nThe authors offer three distinct methods for selecting this blending factor. They support their algorithm with theoretical analysis and experimental results.", "questions": "See weaknesses, in addition to them, \n\n1. Why is the training of a value function needed during step 2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed algorithm is simple and can be implemented with minimal overhead. \n2. The authors provide a complete theoretical analysis, and also conduct extensive experiments, on both deterministic and stochastic environments (although just one) . \n3. The paper is well written and easy to follow.", "weaknesses": "$\\tilde{r}$ **dependence on the behavior policy may cause problems when data is collected using multiple behavior policies.**\n\nConsider a dataset with data from a mixture of policies (say  the medium replay case in D4RL), and for simplicity suppose we assume  a constant lambda, say 0.5. Now, since the reward $\\tilde{r}$ is conditioned on the behavior policy, doesn’t the effective MDP (reconstructed using the relabeled dataset) become non stationary? Is the performance not affected much due to the deterministic nature of the environments chosen for evaluation?\n\n**Choosing $\\lambda$ may not be straightforward**\n\nThe choice of $\\lambda$ should be such that it is high for trajectories with high returns, and low otherwise. But this is hard to determine if the dataset consists of trajectories that perform equally well. This can be seen in results shown in figure 2, as significant improvement is seen on datasets with with a mixture of behavior policies because there is a way to determine the appropriate choice of $\\lambda$ based on relative performance. Now, suppose you have a dataset with low rewards (the random variant in D4RL), then the ranking method will still assign high ranks to most trajectories, and this might result in poor performance as the Monte Carlo estimates using the behavior policy might cause instability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)