Is Value Learning Really the Main Bottleneck in Offline RL?
Authors: Seohong Park
Kevin Frans
Sergey Levine
Aviral Kumar
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
offline reinforcement learning
Primary area: reinforcement_learning

Abstract: While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance of offline RL. Motivated by this observation, we aim to understand the bottlenecks in current offline RL algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: *is the main bottleneck of offline RL indeed in learning the value function, or something else?* To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, we find that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL, often more so than the value learning objective. For instance, we show that common value-weighted behavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test-time policy improvement methods and show that these methods lead to better performance.

Is Value Learning Really the Main Bottleneck in Offline RL?
Seohong Park Kevin Frans!
Sergey Levine* Aviral K
'University of California, Berkeley sechongdberkeley.oo edu
Carnegie Mellon University
Abstract
While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance offline RI.. Motivated by this observation, we aim to understand the bottlenecks in current offline RI. algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: is the main bottleneck of offline RL indeed in learning the value function, or something else? To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, find that the choice of policy extraction algorithm significantly affects the performance and scalability offline RI, often more than the value learning objective. For instance, show that common value-weis weighted hehavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test- ass-time policy improvement methods and show that these methods lead to better performance.
Introduction
Data-driven approaches that convert offline datasets of past experience into policies a predominant approach for solving control problems in several domains [9, 49, 51]. Primarily, there are two paradigms for learning policies from offline data: imitation learning and offline reinforcement learning (RL.). While imitation requires access to high-quality demonstration data, offline RI. loosens this requirement and can learn effective policies even from suboptimal data, which makes offline RL preferable to imitation learning in theory. However, recent results show that tuning imitation learning by collecting more expert data often outperforms offline RL even when provided with sufficient data in practice [36, 48], and it is often unclear what holds back the performance of offline RL.
The primary difference between offline RL and imitation learning is the use of a value function, which absent in imitation learning. The value function drives the learning progress offline RL methods, enabling them to learn from suboptimal data. Value functions are typically trained via temporal- difference (TD) learning, which presents convergence [40, 55] and representational [27, 29, 56] pathologies. This has led to the conventional wisdom that the gap between offline RI and imitation is a direct consequence of poor value lcarning [26, 33, 36). Following up on this conventional wisdom, recent research in the community has been devoted towards improving the value function quality of offline RL algorithms [1, 11, 14, 19, 25, 26]. While improving value functions will definitely help improve performance, question whether this is indeed the best way to maximally improve
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the performance of offline RL, or if there is still headroom to get offline RL to perform better even with current value learning techniques. More concretely, given an offline RL problem, we ask: is the bottleneck in learning the value function, the policy, or something else? What is the best way to improve performance given the bottleneck?
We answer these questions via an extensive empirical study. There are three potential factors that could bottleneck an offline RL algorithm: (B1) imperfect value function estimation, (B2) imperfect policy extraction guided by the learned value function, and (B3) imperfect policy generalization to states that it will visit during evaluation. While all of these contribute in some way to the performance of offline RL, we wish to identify how each of these factors interact in a given scenario and develop ways to improve them. To understand the effect of these factors, we use data size, quality, and coverage as levers for systematically controlling their impacts, and study the "data-scaling" properties, ie., how data quality, coverage, and quantity affect these three aspects of the offline RI algorithm, for three value learning methods and three policy extraction methods on diverse types of environments. These data-scaling properties reveal how the performance of offline RI is bottlenecked in each scenario, hinting at the most effective way to improve the performance.
Through our analysis, we make two surprising observations, which naturally provide actionable advice for both domain-specifer practitioners and future algorithm development offline RL. First, we find that the choice of a policy extraction algorithm often has larger impact on performance than value learning algorithms, despite the policy being subordinate to the value function in theory. This contrasts with the common practice where policy extraction often tends to be an afterthought in the design of value-based offline RL algorithms. Among policy extraction algorithms, we find that
policy gradient (e.g., DDPG+BC [14)) almost always leads to much better performance and favorable data scaling than other widely used methods like malue-weighted regression (e.g., AWR [46, 47, 58]). We then analyze why constrained policy gradient leads to better performance than weighted behavioral cloning via extensive qualitative and quantitative analyses.
Second, we find that the performance of offline RL is often heavily bottlenecked by how well the policy generalizes to test-time states, rather than its performance analysis suggests that existing offline algorithms are often already great at learning an optimal policy from suboptimal data on in-distribution states, the degree that is saturated, and the performance is often simply bottlenecked by the policy accuracy on novel states that the agent encounters at test time. This provides a new perspective on generalization in offinn RL, which differs from the previous focus on pessimism and behavioral regularization. Based on this observation, we provide two practical solutions to improve the generalization bottleneck: the use of high-coverage datasets and test-time policy extraction techniques. In particular, we propose new on-the-fly fly policy improvement techniques that further distill the information in the value function into the policy on test-time states during evaluation rollouts, and show that these methods lead to better performance.
training states. Namely, our
Our main contribution is an analysis of the bottlenecks in offline RL as evaluated via data- acta-scaling properties of various algorithmic choices. Contrary to the conventional belief that value learning is the bottleneck of offline RL algorithms, we find that the performance is often limited by the choice of a policy extraction objective and the degree to which the policy generalizes at test time. This suggests that, with an appropriate policy extraction procedure (e.g., gradient-based policy extraction) and an appropriate recipe for handling generalization (e.g., test-time training with the value function), collecting more high-coverage data to train value function is universally better recipe for improving offline RL performance, whenever the practitioner has access to collecting some new data for learning. These results also imply that more research should be pursued in developing policy learning and generalization recipes to translate value learning advances into performant policies.
2 Related work
Offline reinforcement learning [31, 33] aims to learn a policy solely from previously collected data. The central challenge in offline RL is to deal with the distributional shift in the state-action distributions of the dataset and the learned policy. This shift could lead to catastrophic value overestimation if not adequately handled [33]. To prevent such a failure mode, prior works in offline RL have proposed diverse techniques to estimate more suitable value functions solely from offline data via conservatism [8, 26], out-oofdistributtun
penalization [14, 53, 59], in-sample maximization [17, 25, 61], uncertainty minimization [1, 19, 60], convex duality [32, 41, 50], or contrastive learning [1]]. Then, these methods train policies to maximize the learned value function with behavior-regularized
policy gradient (e.g., DDPG+BC) [14, 34], weighted behavioral eloning (e.g., AWR) [46, 47], or sampling-bass based action selection (e.g., SIBC) [7, 15, 21J. Depending on the algorithm, these value

---

learning and policy extraction stages can either be interleaved [14, 26, 42] or decoupled [5, 11, 17, 25]. Despite the presence of a substantial number of offline RL algorithms, relatively few works have aimed to analyze and understand the practical challenges in offline RI.. Instead of proposing new algorithm, we mainly aim to understand the current bottlenecks in offline RI. via a comprehensive analysis of existing techniques so that we can inform future methodological development.
Several prior works have analyzed individual components of offline RL or imitation learning algo- rithms: value bootstrapping [14, 15], representation learning [27, 29, 62], data quality [4], differences between RL and behavioral cloning (BC) [28], and empirical performance [10, 23, 35, 36, 54]. Our analysis is distinct from these lines of work: we analyze challenges appearing due to the interaction between these individual components of value function learning, policy extraction, and generaliza- tion, which allows us to understand the bottlenecks in offline RL from a holistic perspective. This can inform how a practitioner could extract the most by improving one more of these components, de- pending upon their problem. Perhaps the closest study to ours is Fu et al. [13], which study whether representations, value accuracy, or policy accuracy can explain the performance of offine RI. While this study makes insightful recommendations about which algorithms to use and reveals the potential relationships between some metrics and performance. the conclusions are only drawn from D4RL locomotion tasks [12], which are known to be relatively simple and saturated [48, 53], and the data- scaling properties of algorithms are not considered. In addition, this prior study does not identify policy generalization, which we find to be one of the most substantial yet overlooked bottlenecks in offline RL. In contrast, we conduct a large-scale analysis on diverse environments (e.g., baseed goal-conditioned, and manipulation tasks) and analyze the bottlenecks in offline RL with the aim of providing actionable takeaways that can enhance the performance and scalability of offline RI.
3 Main hypothesis
Our primary goal is to understand when and how the performance of offline RL can be bottlenecked in practice. As discussed earlier, there exist three potential factors that could bottleneck an offline RL algorithm: (B1) imperfect value function estimation from data, (B2) imperfect policy extraction from the learned value function, and (B3) imperfect generalization on the test-time states that the policy visits in evaluation rollouts. We note that the bottleneck of an offline RI. algorithm under a certain dataset can always be attributed to one or some of these factors, since the policy will attain optimal performance if both value learning and policy extraction are perfect, and perfect generalization to test-time states is possible.
Our main hypothesis in this work is that, somewhat contrary to the prior belief that the accuracy of the value function is the primary factor limiting performance of offline RL methods, policy learning is often the main bottleneck of offline RL. In other words, while value function accuracy is certainly important, how the policy is extracted from the value function (B2) and how well the agent generalizes on states that it visits at the deployment time (B3) are often the main factors that significantly affect both the performance and scalability of offline RI. To verify this hypothesis, we conduct two main analyses in this paper. In Section 4, we compare the effects of value learning and policy extraction on performance under various types of environments, datasets, and algorithms (B1 and B2). In Section 5, we analyze the degree to which the policy generalizes on test-time states affects performance (B3).
Empirical analysis 1: Is it the value or the policy? (B1 and B2)
We first perform controlled experiments to identify whether imperfect value functions (B1) or imperfect policy extraction (B2) contribute more to holding back the performance of offline RL in practice. To systematically compare value learning and policy extraction, we run different algorithms while varying the the amounts of data for value function training and policy extraction, and draw data-scaling matrices to visualize the aggregated results. Increasing the amount of data provides convenient lever to control the effect of each component, enabling us draw conclusions about whether the value or the policy serves as a bigger bottleneck in different regimes when different amounts of training data are available (or can be collected by a practitioner for given problem), and to understand the differences between various algorithms.
To clearly dissect value learning from policy learning, we focus on offline RL methods with decoupled value and policy training phases (e.g., One-step RL [5], IQL [25], CRL [11], etc.), where policy learning does not affect value learning. In other words, we focus on methods that first train a value function without involving policies, and then extract a policy from the learned value function with a separate objective. While this might sound a bit restrictive, we surprisingly find that policy learning
3

---

is often the main bottleneck even in these decoupled methods, which attempt to solve a simple, single- step optimization problem for extracting a policy given a static and stationary value function.
4.1 Analysis setup
We now introduce the value learning objectives, policy extraction objectives, and environments that we study in our analysis (see Appendix B for preliminaries).
Value learning objectives. We consider three decoupled value learning objectives that fit value functions without involving policy learning: (1) implicit Q-learning (IQL.) [25], (2) SARSA [5], and (3) contrastive RL (CRL) [11]. IQL fits an optimal Q function (Q") by approximating the Bellman optimality operator with an expectile loss. SARSA fits behavioral Q function (Q9) using the Bellman evaluation operator. In goal-conditioned tasks, we employ CRL instead of SARSA, which similarly fits behavioral Q function, but with different contrastive learning-based objective that leads better performance. We refer to Appendix D.I for detailed descriptions of these value learning methods.
Policy extraction objectives. Prior works in offline RL typically use one of the following objectives to extract a policy from the value function. All of them are built upon the same principle: maximizing values while being close to the behavioral policy, to avoid the exploitation of erroneous critic values.
(1) Weighted behavioral cloning (e.g., AWR). Weighted behavioral cloning is one of the most widely used offline policy extraction objectives for its simplicity [25, 42, 44, 46, 47, 58]. Among weighted behavioral cloning methods, we consider advantage-weighted regression (AWR [46, 47]) in this work, which maximizes the following objective:
max (T) = Fe..o--ee0((000))
log T(a |s],
(1)
where a is an (inverse) temperature hyperparameter. Intuitively, AWR assigns larger weights to higher-advantage transitions when cloning behaviors, which makes the policy selectively copy only good actions from the dataset.
(2) renstiir-constrained policy gradient (e.g., DDPG+BC), Another popular policy extraction objective is behavior-conee constrained policy gradient, which directly maximizes Q values while not deviating far away from the behavioral policy [1, 14, 19, 26, 59]. In this work, we consider the ob- jective that combines deep deterministic policy gradient and behavioral cloning (DDPG+BC [14]):
(DDPG+BC(T)) Eg,,-rQ(ss, n² (s)) + log |s]]
where (s) = (s-x(1ss/a] and a is a hyperparameter that controls the strength of the BC regularizer.
(3) Sampling-based action selection (e.g., SfBC). Instead of learning an explicit policy, some previous methods implicitly define policy as the action with the highest value among action samples from the behavioral policy [7, 15, 18, 21J. In this work, we consider the following objective that selects the arg max action from behavioral candidates (SfBC (7]):
7 = arg max [Q(s,a)], oE (oI....))
where ai...., aN sampled from the learned BC policy n 8) [7.
Environments and datasets. To understand how different value learning and policy extraction ob- jectives affect performance and data scalability, we consider cight environments (Figure 10) across state- and pixel-based, robotic locomotion and manipulation, and goal-conditioned and single-task settings with varying levels of data suboptimality: (1) aece-anmazz--eenn
(2) antmaze-1arge (3) d4r1-hopper, (4) d4rl-walker2d, (5) exorl-walken (6) exorl-chettal (7) kitchen, and (8) gc-roboverse. We highlight some features of these tasks: storl-(walkes cheetah) are tasks with highly suboptimal, diverse datasets collected by exploratory policies, gc-antmaze- and gc-roboverse are goal-conditioned ('gc-") tasks, and gc-roboverse is a pixel-based robotic manipulation task with a 48 48 x 3-dimensional observation space. For some tasks (e.g.. gc-antmaze-large and kitchen), we additionally collect data to enhance dataset sizes to depict scaling properties clearly. We refer Appendix D.2 for the complete task descriptions.
4.2 Results: Policy extraction mechanisms substantially affect data-scaling trends
Figure shows the data-scaling matrices of three policy extraction algorithms (AWR, DDPG+BC, and SfBC) and three value learning algorithms (IQI. and (SARSA or CRL.)) on eight environments, aggregated from total of 15, 488 runs (8 seeds for each cell, numbers after "4" denote standard

---

SC Battaaze Large
antiaza- large
IQL
14
Phen
We
V0 C.4
CHE CIN
KJN 1
Vi
AWB
DOPG
CBL
SARGA
AWR
CARSA
DOPG
BARSA
SRRC
MARMA
AWR
SARSA
BOPG
PARSA
090
Sa
Valee
vark-valee:2d
exor1 walier
ex0:1 -chellh
Vala Date
Vake Lide
Saka Date
Valuo Batk
Valle Dails
Value Bais
Vil Dec
SARIA - AWTI
SARSA DDPG
SAUSA
SBC
SARSA
AWTL
SAIISA
DSPG
SARSA 5m0
SARSA AWIL
SARSA + BDPG
SABIA + SEIC
kitchen
ge-rohoversa
AWH
DOIN
PQU- DESC
IQF- ARRR
TQU DIPG
EQI. SITC
16
15
17
4122
Vale Batk
Vale Data
Value Iuss
Vaae Bata
Yake Bikk
Take Data
SABSA
AWTL
SABIA
DOPG
SABSA
SID
CHLL AWTI
DDPG
CRLL SIBC
Figure 1: Data-scaling matrices of three policy extraction methods (AWR, DDPG+BC, and SIBC) and three value learning methods (IQL and (SARSA CRL)). To see whether the value or the policy imposes bigger bottleneck, we measure performance with varying amounts of data for the value and the policy. The color gradients (1, R, + of these matrices reveal how the performance of offline RL is bottlenecked in each setting.
Table I: DDPG+BC is often the best policy extraction method. We aggregate the performances over the entire data-scaling matrix and then over 8 random seeds in each setting. Scores at or above 95% of the best score are highlighted in bold. The table shows that DDPG+BC is better than or as good as AWR 15 out of 16 settings. We note that policy extraction hyperparameters are individually tuned for each setting (Figure 11).
Task (Valne Algorithm)
AWR
DDPG+BC
SBB
Task (Value Algorithm)
AWR
DDPG-BC
SBC
ge-antnaze (IQL)
51 =s
23
21
exorl-cheetah (IQL)
71 21
101 =7
77 29
ge-aantmaze- -arenaaze-arree (CRL)
37 ss
58
51 12
exorl-cheetah (SARSA)
78
131
89
antnaze large (IQL)
82
ddrl-hopper (IQL)
53
aatzaze-large (SARSA)
0 =D
0 =b
0 =0
ddrl-hopper (SARSA)
56 :1
61 =5
50 =2
kitchen (IQL)
ddrl-walker2d (IQL)
kitchen (SARSA)
79
83
73
dArl-velker22 (SARSA)
79 20
81 =0
81 =1
exorl-valker (IQL)
99
140
ge-roboverse (IQL)
23
20
14
exarl-walker (SARSA)
94 20
193 25
125 21
ge-rohovarse (CRL.)
13
16 29
15 21
deviations). In each matrix, we individually tune the hyperparameter for policy extraction (a or N) for each entry. These matrices show how performance varies with different amounts of data for the value and the policy. In our analysis, we specifically focus on the color gradients of these matrices, which reveal the main limiting factor behind the performance of offline RL in each setting. Note that the color gradients are mostly either vertical, horizontal, or diagonal. Vertical (f) color gradients indicate that the performance is most strongly affected by the amount of policy data, horizontal gradients indicate it is mostly affected by value data, and diagonal (7) gradients indicate both.
Side-b comparisons of the data-scaling matrices from different policy extraction methods in Figure/ suggest that, perhaps surprisingly, different policy extraction algorithms often lead to significantly different performance and data-scaling behaviors, even though they extract policies from the same value function (recall that the use of decoupled algorithms allows us to train a single value function, but use it for policy extraction in different ways). For example, on exorl-cheetan AWR performs remarkably poorly compared to DDPG+BC or StBC on both value learning algorithms. Such a performance gap between policy extraction algorithms exists even when the value function is far from perfect, as can be seen in the low-data regimes in www-aatmaaelaraae and kitchen. In general, we find that the choice of policy extraction procedure affects performance
and
5

---

cc-antmaze large (IQL)
exorl exorl-walke wwaker (IQL)
AWR.A Jo
6.0
AWR 1.0
AWR
3
10.01
AWRR // 0.8)
ANNT
1.00
AWR
16.00
AWTR
100.0
11P 121
16
134
114
02M
020
1M 24
1I 24 11 194
20 01
14
101
oubrod 144 LAN
14,
104
DDPG ia
DDPG
DDPG
DEPG 0.5:
DDPG
DDPG
DDPG 0,001
DDPG ias
AM 2
LOL
1R
1150E 51
101
101
D 151 180
o.brdd 111 1
11 Date
Videe Dara
1044
004000 1941 Datt
Value Duten
1M
BC
RL
BC
RL
Figure 2: Data-scaling matrices of AWR and DDPG+BC with different BC strengths (cx). In
gc- a-taantmaelaarge AWR is always poluny-bounded (1). but DDPG+BC has both policy-bounded (9) and value-bounded (=+) modes, depending on the value of a. Notably, an in-between value of a 1.0 DDPG+BC leads to the best of both worlds (see the bottom left corner of gc- cc--antaaze-lee -arge with 0.1M datascts)!
often more than the choice of a value learning objective except antmaze -argg, where the value function must be learned from sparse-reward, suboptimal datasets with long-horizon trajectories.
Among policy extraction algorithms, we find that DDPG+BC almost always achieves the best performance and scaling behaviors across the board, followed by SiBC, and the performance of AWR falls significantly behind the other two extraction algorithms in many cases (Table 1). Notably, the data-scaling matrices of AWR always have vertical (1) or diagonal (7) color gradients, implying that it does not fully utilize the value function (see Section 4.3 for clearer evidence). In other words, non-careful choice of the policy extraction algorithm (e.g., weighted behavioral cloning) hinders the use of learned value functions, imposing an unnecessary bottleneck on the performance of offline RL.
4.3 Deep dive 1: How different are the scaling properties of AWR and DDPG+BC?
To gain further insights into the difference between weightee behavioral cloning (e.g., AWR) and behavior-regullrizzed policy gradient (e.g.. DDPG+BC), we draw data-scaling matrices with different values of a (in Equations (1) and (2)), hyperparameter that interpolates between RL and BC. Note that a = 0 corresponds to BC in AWR and a = oo corresponds to BC in DDPG+BC. We recall that the previous results (Figure 1) use the best temperature for each matrix entry (i.e., aggregated by the maximum over temperatures), but here we show the full results with individual hyperparameters.
Figure 2 highlights the results on gc-antmaze-large and exorl (see Appendix E for the full results). The results on gc-antmaze-large show clear difference in scaling matrices between AWR and DDPG+BC. That is. AWR is always policy-bounded regardless of the BC strength a (i.e., vertical (1)) color gradients). whereas DDPG+BC has two "modes": it is policy-bounded (1) when a is large, and value-bounded (+>) and when a is small. Intriguingly, an in-between value of a = 1.0 in DDPG+BC enables having the best of both worlds, significantly boosting performances across the entire matrix (note that it achieves very strong performance even with a 0.1M-sized dataset)! This difference in scaling behaviors suggests that the use of the learned value function in weighted behavioral cloning is limited. This becomes more evident in exorl-walke (Figure 2), where AWR fails to achieve strong performance even with very high temperature value (a = 100).
4.4 Deep dive 2:
Why is DDPG+BC better than AWR?
We have far seen several empirical results that suggest behavior-regularii policy gradient (e.g., DDPG+BC) should be preferred to weighted behavioral cloning (e.g., AWR) in any case. What makes DDPG+BC much better than AWR? There are three potential reasons.
First, AWR only has a mode-covering weighted behavioral cloning term, while DDPG+BC has both mode-seeking first-order value maximization and mode-covering behavioral cloning terms. As result, actions learned by AWR always lie within the con- vex hull of dataset actions, whereas DDPG+BC can "hillelimb" the learned value function, even allowing extrapolation to some degree while not deviating too far away from the mode. This not only enables a better use of the value function but produces a wider range of actions. To illustrate this, we plot test-time action sampled from policies learned by AWR and DDPG+BC on exorl- Figure 3 shows that AWR actions are relatively centered around the origin, while DDPG+BC actions are more spread out, which can sometimes help achieve an even higher degree of optimality.
AWR
DDPG BC
Dimanciin
Dimeriion
Dimonsion
Figure 3: AWR vs. DDPG actions.

---

Second, value- weigh-e behavioral cloning uses a much smaller number of effective samples than belavior-regulatiized policy gradient methods, especially when the temperature (a) is large. This because a small number of high-advantage transitions can potentially dominate learning signals for AWR (e.g., a single transition with a weight of eo can dominate other transitions with smaller weights like e²), As result, AWR effectively uses only a fraction of datapoints for policy learning, being susceptible to overfitting. On the other hand, DDPG+BC is based on first-order maximization of the value function without any weighting, and
Training
Validation
AWR (a 0.0)
AWR (cr 3.0)
Polay Los
500k
b0lk
DDPG (or 3.0.
DDPG ia 0.3)
Polley
thus is free from such issue. Figure 4 illustrates this, where
Stepe
Steps
we compare the training and validation policy losses of AWR and DDPG+BC on gc-antmaze- arge with the smallest 0.1M dataset (8 seeds). The results show that AWR with a large temperature (a: = 3.0) causes severe overfitting. Indeed, Figure 1 shows DDPG+BC often achieves significantly better performance than AWR in low-data regimes.
Figure 4: AWR overfits.
Third, AWR has theoretical pathology in the regime with limited samples: since the coefficient multiplying log x (a s) the AWR objective (Equation (1)) is always positive, AWR can increase the likelihood of all dataset actions, regardless of how optimal they are. If the training dataset covers all possible actions, then the condition for normalization of the probability density function of (a s) would alleviate this issue, but this coverage assumption is rarely achieved in practice. Under limited data coverage, and especially when the policy network highly expressive and dataset states are unique (e.g., continuous control problems), AWR can in theory memorize all state-action pairs in the dataset, potentially reverting to unweighted behavioral cloning.
Takeaway: Current policy extraction can inhibit effective use of the value function.
Do not value-weighted behavior cloning (e.g., AWR); use behavior-constrained policy gradient (e.g.. DDPG+BC), regardless of the value learning objective. This enables better scaling of performance with more data and better use of the value function.
Empirical analysis 2: Policy generalization (B3)
We now turn our focus to the third hypothesis, that the degree to which the agent generalizes to states that it visits at the evaluation time has significant impact on performance. This is a unique bottleneck to the offtine RL problem setting, where the agent encounters new, potentially out-of- distribution states at test time.
5.1 Analysis setup
To understand this bottleneck concretely, we first define three key metrics quantifying a notion of accuracy of given policy in terms of distances against the optimal policy. Specifically, we use the following mean squared error (MSE) metrics quantify policy accuracy:
S
pd
Evaluation
D
Derain
Training
Dea
Validation
Figure 5: Three distributions for the MSE metrics.
(Training MSE) EgwDeram (T(()) x {s))
(4)
(Validation MSE) EEnD.... [[(()) ( (s))
(Evaluation MSE) (xx=pt()) [(T(s) -+)
(6)
where Dirain and Dan respectively denote the training and validation datasets, * denotes an optimal policy, which assume access to for evaluation and visualization purposes only. Validation MSE
7

---

anteaze- -reeazz-eodduu
astaaaz--aas -arge
kiichen-nixed
adroit-pen
adro1t-hamer
adroit-door
Return
250
Eval. MSE
0.7
0.35
0.2
6:20
0.30
0.1
0.3
133
6:15
1M
130
251
1M
2M
IM
21
134
25
Val. MSE
6.20
0.1
0.1
0.3-
6:15
0.23
0.0
13t
251
1M
251
0.0
13
251
13
2M
IM
254
131
254
Troin. MSE
0.7
1.4
0.1
6:15
0.0
0.0
1M
251
0.00
154
2M
Gradiem
Gradient Steps
Grudient Stares
Gradient Steps
Crudient Steps
Grodient Stope
Figure 6: How do offline RL policies improve with additional interaction data? In many environments, teniineto--onlne RL only improves evaluation MSEs, while validation MSEs and training MSEs often remain completely flat (see Section for the definitions of these metries). This suggests that current offline RL algorithms may already be great learning an effective policy on in-distribution states, and the performance of offline RL often mainly determined by how well the policy generalizes on its own state distribution at test time.
measures the policy accuracy on states sampled from the same dataset distribution as the training distribution (i.e., in-distribution MSE, Figure 5), while evaluation MSE measures the policy accuracy on states the agent visits at test time, which can potentially be very different from the dataset distribution (i.e., -istrr-distiiuutionn MSE, Figure 5). We note that, while these metries might not always be perfectly indicative of the performance of policy (see Appendix A), they serve as convenient proxies to estimate policy accuracy many continuous-control domains in practice.
One way to measure the degree to which test-time generalization affects performance to evaluate how much room there is for various policy MSE metrics to improve when further training on additional policy rollouts is allowed. The distribution of states induced by rolling out the policy an ideal distribution to improve performance, as the policy receives direct feedback on its own actions at the states it would visit. Hence, by tracking the extent to which various MSEs improve and how their predictive power towards performance evolves over online interaction, we will be able to understand which is a bigger bottleneck: in-distribution generalization (i.e., improvements towards validation MSE under the offline dataset distribution) out-of distribution generalization (i.e., improvements in evaluation MSE under the on-policy state distribution). To this end, we measure these three types of MSES over the course of online interaction, when learning from a policy trained on offline data only (i.e., the fffline-to-onlie RL setting). Specifically, we train offline-to-onllne IQL. agents on six D4RL [12] tasks (antmaze- (arraazeemmmiinnn {medium, large}, kitchen, and adroit- (pen, hammer, , door)), and measure the MSEs with pre-trained expert policies that approximate n (see Appendix D.4).
5.2
Results: Test-time generalization is often the main bottleneck in offline RI.
Figure shows the results (8 seeds with 95% confidence intervals), where denote online training steps in red. The results show that, perhaps surprisingly, in many environments continued training with online interaction only improves evaluation MSEs, while training and validation MSEs often remain completely flat during online training. Also, we can see that the evaluation MSE the most predictive of the performance of offline RL. among the three metrics. In other words, the results show that, despite the fact that on-policy data provides for an oracle distribution to improve policy accuracy, performance improvement often only reflected in the evaluation MSEs computed under the policy's 's own state distribution.
What does this tell us? This indicates that, current offline RL methods may already be sufficiently great at learning the best possible policy within the distribution of states covered by the offline dataset, and the agent's performance is often mainly determined by how well it generalizes under its own state distribution at test time, as suggested by the fact that evaluation MSE is most predictive of performance. This finding somewhat contradicts prior beliefs: while algorithmic techniques in offline RI. largely attempt to improve policy optimality on in-distribution states (by addressing the issue with aut-of-distributin actions), our results suggest that modern offline RL algorithms may
8

---

actantnaze -argge [IQL)
adroit pen (IQL)
AWWR Lriaa Ouly
AWR
AWR Infaas 0.7,
AWTR (Kae 1.00
AWRR (Pfasa 0.0y
AWRR
0.5)
AWR
1.0
AWTR Odeam
Date
39
110
114
133
121
22
129
61M
31M
61M KTI
M
115
81M
119
X
Vilue
Data
Video Bats
Nakar Data
Vidao
Vakao
Viluo
0.7,
1.0)
(Maa
94
137
124
125
13P
IM
51
114
92M
121
X00
310
Vidue Daca
Valu Date
Video Dati
Vake Daza
Video Stace
Vaku Date
Voluo Datt
Vake Data
Low coverage
High coverage
Low coverage
High coveraaee
High optimality
Low optimality
High optimality
Low optimality
Figure 7: Should we use high-coverage or high-optimality datasets? The data-scaling matrices above show that high-coverage datasets can he much more effective than high-optimality datasets. This is hecause high- coverage datasets can improve test-fime policy accuracy. one of the main bottlenecks of offline RL.
already saturate on this axis. Further performance differences may simply be due to the effects of given offline RL objective on novel states, which very few methods explicitly control!
That said, controlling test-time generalization might also appear impossible: while offline RL methods could hillclimb validation accuracy via combination of techniques that address statistical errors such as regularization (e.g.. Dropout [52], LayerNorm [3], etc.), improving test-time policy accuracy requires generalization a potentially very different distribution (Figure 5), which is theoretically impossible to guarantee without additional coverage or structural assumptions, as the test-time state distribution can be arbitrarily adversarial in the worst case. However, we claim that if we actively utilize the information available at test time or have the freedom to design offline datasets, it is possible to improve test-time policy accuracy in practice, and we discuss such solutions below (see Appendix C for further discussions).
5.3 Solution 1: Improve offline data coverage
If we have the freedom to control the data collection process, perhaps the most straightforward way to improve test-time policy accuracy is to use dataset that has as high coverage as possible so that test-time states can be covered by the dataset distribution. However, at the same time, high-coverage datasets often involve exploratory actions, which may compromise the quality (optimality) of the dataset. This makes us wonder in practice: which more important, high coverage or high optimality?
To answer this question, we revert back to our analysis tool of data Sata-scaling matrices from Section and empirically compare the data-scaling matrices on datasets collected by expert policies with different levels of action noises dddata). Figure shows the results of IQL. agents on an-aeetmaze-e large and adroit prroit-pe -pen (8 seeds each). The results suggest that the performance of offline RL. generally improves as the dataset has better state coverage, despite the increase in suboptimality. This is aligned with our findings in Figure 6, which indicate that the main challenge of offline RL is often not learning an effective policy from suboptimal data, but rather learning policy that generalizes well to test-time states. In addition, (DDPG+BC; see Section 4) in this case as well, where we train a policy from high-coverage data. For instance, in low-data regimes in gc- antmaze-large the value function, whereas DDPG+BC still allows the algorithm to improve performance with better value functions. Based our findings, suggest practitioners prioritize high coverage (particularly around the states that the optimal policy will likely visit) over high optimally when collecting datasets.
note that it is crucial to use value gradient- pradient-based policy extraction method
in Figure 7. AWR fails to fully leverage
5.4 Solution 2:
Test-time policy improvement
If we do not wish to modify offline data collection, another way improve test-time policy accuracy is to on-the-fly train or steer the policy guided by the learned value function on test-time states. Especially given that imperfect policy extraction from the value function is often a significant bottleneck in offline RL (Section 4), we propose two simple techniques to further distill the information in the value function into the policy test-time states.
(1) On-the-fly policy extraction (OPEX). Our first idea is to simply adjust policy actions in the direction of the value gradient at evaluation time. Specifically, after sampling an action from the policy a T(- s) at test time, we further adjust the action based on the frozen learned Q function during evaluation rollouts with the following formula:
+ + B V as)
9

---

64r1 -hopper
34rl- velkerrd
exorl exorl-welker "elker
exprl cheettak
Returnn
80-
290
Iih. 9t1 OPEX T1T
860: OPXX TTT
13
50K OPRX TFT
500 OPEA 1TT
kitchen- partial
kitchea-miise
aataaze un
antnaaee -arge
Detum
IQL SBC OPEX 117
X,L
OPEX 1TT
10 SBCC OPEX TFT
RUL SI5C OPEX 1IT
on-the-flyy the-fly policy improvement techniques (OPEX and TTT) lead to substantial performance improvements on diverse tasks, by mitigating the test-time policy generalization bottleneck.
Figure Test-time policy improvement strategies (OPEX and TTT). Our two on
where 3 is a hyperparameter that corresponds to the test-time "learning rate". Intuitively, Equation (7) adjusts the action in the direction that maximally increases the learned Q function. We call this technique on-the-fly policy extraction (OPEX). Note that OPEX requires only a single line of additional code evaluation and does not change the training procedure at all.
(2) Test-time training (TTT). We also propose another variant that further updates the parameters of the policy by continuously extracting the policy from the fixed value function on test-time states, as more rollouts are performed. Specifically, we update the policy with the following objective:
max JTIT(" (T) Re ( (8Q( 1= (8) - B (xI.(n off || x)]
88)
where π denotes the fixed, learned offline RL policy, D (') denotes the mixture of the dataset and evaluation state distributions, and denotes a hyperparameter that controls the strength of the version of OPEX, where we further update the parameters of the policy π to maximize the learned value function, while not deviating too far away from the learned offline RL policy. We call this scheme test-time training (TTT). Note that TTT only trains x based on test-time interaction data, while Q and zf remain fixed.
regularizer. Intuitively, Equation (8) is a "parameter-updating
Figure compares the performances of vanilla IQL, SIBC (Equation (3), another test-time policy extraction method that does not involve gradients), and our two gradient-based test-time policy improvement strategies on eight tasks (8 seeds each, error bars denote 95% confidence intervals). The results show that OPEX and TTT improve performance over vanilla IQL and SiBC in many tasks, often by significant margins, by mitigating the test-time policy generalization bottleneck.
Takeaway: Improving test-time policy accuracy significantly boosts performance.
Test-time policy generalization is one of the most significant bottlenecks in offline RL. Use high-coverage datasets. Improve policy accuracy on test-time states with on- on-the-fly policy improvement techniques.
6 Conclusion: What does our analysis tell us?
In this work, we empirically demonstrated that, contrary to the prior belief that improving the quality of the value function is the primary bottleneck of offline RL, current offline RL methods are often heavily limited by how faithfully the policy is extracted from the value function and how well this policy generalizes test-time states. For practitioners, for effective offline RI: train a value function on as diverse data as possible, and allow the policy to maximally utilize the value function, with the best policy extraction objective (e.g.. DDPG+BC) and/or potential test-time policy improvement strategies. For future algorithms research, our analysis emphasizes two important open questions in offline RL: (1) What the best way extract a policy from the learned value function? (2) How can we train a policy in way that it generalizes well on test- time states? The second question particularly notable, because it suggests a diametrically opposed viewpoint to the prevailing theme of pessimism in offline RL, where only few works have explicitly aimed to address this generalization aspect offline RL [37, 38, 63). We believe finding effective answers to these questions would lead to significant performance gains in offline RI. substantially enhancing applicability and scalability, and would encourage the community to incorporate holistic picture of offline RL alongside the current prominent research on value function learning.
analysis suggests a clear empirical recipe
10

---

Acknowledgments
We thank Benjamin Eysenbach and Dibya Ghosh for insightful discussions about data-scaling matrices and state representations, respectively, and Oleh Rybkin, Fahim Tajwar, Mitsuhiko Nakamoto, Yingjie Miao, Sandra Faust, and Dale Schuurmans for helpful feedback on earlier drafts of this work. This work was partly supported by the Korea Foundation for Advanced Studies (KFAS), National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2146752, and ONR N00014-21-1-2838, This research used the Savio computational cluster resource provided by the Berkeley Research Computing program UC Berkeley.
References
[1] Gaon An, Scungyong Moon, lang-Hyun Kim, and Hyun Oh Song. Uncertainty- beased oftiine reinforcement learning with diversified q-ensemble. In Neural Information Processing Systems (NeurlPS), 2021.
[2] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abheel, and Wojeiech Zaremba. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS), 2017.
[3] Jimmy Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. ArXiv, abs/1607, ftns//607.06650 2016.
[4] Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation leaming. In Neural Information Processing Systems (NeurlPS), 2023.
[5] David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off- policy evaluation. In Neural Information Processing Systems (NeurIPS), 2021.
[6] Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.
[7] Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu, Offline reinforcement learning via high- fidelity generative behavior modeling. In International Conference on Learning Representations (ICLR), 2023.
[8] Ching-An Cheng, Tengyang Xic, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offiine reinforeement leaming. In International Conference on Machine Learning (ICML), 2022.
[9] Open X-Fmbodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan. Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, Animesh Garg. Aniruddha Kembhavi. Annie Xie, Anthony Brohan, Antonin Raflin, Archit Sharma, Arefch Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limeen Limerick, Bcomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu,, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan. Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia. Feiyu Zhao, Freek Suip, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang. Hao Su, Hao-Shu Shu Fang. Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wa, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna. Jonathan Booher. Jonathan Tompson, Jonathan Yang. Jordi Salvador. Joseph J. Lim, Junhyek Han. Kaiyuan Wang. Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishman. Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang. Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Letrel Pinto, Li Fei- Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lec, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Moes, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart'n in in-Mart'in in. Rohan Baijal, Rosario Scalise, Rose Hendrix.
11

---

Roy Lin, Runjia Qian, Ruohan Zhang. Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davehev, Tony Z. Zhao, Travis Armstrong. Trevor Darrell, Trinity Chung. Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yoonyoung Cho. Youngwoon Lee. Yuchen Cui, Yue Cao. Yuch-Hua Wu, Yujin Tang, Yuke Zhu. Yunchu Zhang. Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang. and Zipeng Lin. Open x-embodiment: Robotic learning datasets and rt-x models. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
[10] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? In International Conference Learning Representations (ICLR), 2022.
[11] Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2022.
[12] Justin Fu, Aviral Kumar, Ofir Nachum, Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv, abs/2004.07219, 2020.
[13] Yuwei Fu, Di Wu, and Benoit Boulet. A closer look at offline rl agents. In Neural Information Processing Systems (NeurIPS), 2022.
[14] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
[15] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. International Conference on Machine Learning (ICMI), 2019.
[16] Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should i trust you, bellman? the beliman error is poor replacement for value error. In International Conference on Machine Learning (ICML), 2022.
[17] Divyansh Garg. Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q- geaarinn:: Maxent rl without entropy. In International Conference on Learning Representations (ICI.),, 2023.
[18] Scyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-man q-lcarning operator for simple yet effective offline and online rl. In International Conference on Machine Learning (ICML), 2021.
[19] Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offfine rl through ensembles, and why their independence matters. In Neural Information Processing Systems (NeurIPS), 2022.
[20] Dibya Ghosh. dibyaghosh/jaxri_m, 2023. URL https: htwp://www...to////ioyyg/h/////a
com
[21] Philippe -ansennEEttuuhh, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit Ilaarnin as an actor-critic method with diffusion policies. ArXiv, abs/2304.1055 105773, 2023.
[22] Leslie Pack Kaelbling, Learning to achieve goals. In International Joint Conference on Artificial Intelligence (ICAI). 1993.
[23] Bingyi Kang, Xiao Ma, Yi-Ren Wang, Yang Yue, and Shuicheng Yan. Improving and benchmarking offline reinforcement learning algorithms. ArXiv, abs/2300.00092. 2023.
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[25] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforeement learning with implicit q- g-leaming. In International Conference on Learning Representations (ICLR), 2022.
[26] Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative qelearning for offline reinforce- ment learning. In Neural Information Processing Systems (NeurIPS), 2020.
12

---

[27] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
[28] Aviral Kumar, Joey Hong. Anikait Singh, and Sergey Levine. Should i rum offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations (ICLR), 2021.
[29] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron C. Courville, G. Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. International Conference on Learning Representations (ICLR), 2022.
[30] Cassidy Laidlaw, Stuart Russell, and Anca D. Dragan. Bridging rl theory and practice with the effective horizon. In Neural Information Processing Systems (NewrIPS), 2023.
[31] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-ars pages 45-73. Springer, 2012.
[32] Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joëlle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning (ICML), 2021.
[33] Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offfine reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020.
[34] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference Learning Representations (ICLR), 2016.
[35] Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research (TMLR), 2023.
[36] Ajay Mandlekar, Danfei Xu, J. Wong, Soroush Nasiriany. Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese. Yuke Zhu, and Roberto Marttin in. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[37] Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Improving zero-shot generalization in offline reinforcement learning using generalized similarity functions. In Neural Information Processing Systems (NeurIPS), 2022.
[38] Ishita Mediratta, Qingfei You, Minqi Jiang, and Roberta Raileanu. The generalization gap in offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
[39] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5600 2013.
[40] Rémi Munos. Error bounds for approximate policy iteration. In International Conference on Machine Learning (ICML), 2003.
[41] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. ArXiv, abs/19 abs/1912.00074, 2019.
[42] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelcrating online reinforcement learning with offline datasets. ArXiv, abs/2006.09359, 2020.
[43] Whitney Newey and James L. Powell. Asymmetric least squares estimation and testing. Econometrica, 55: 819-847, 1987.
[44] Scohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Higl: Offline goal-conditioned with latent states as actions. In Neural Information Processing Systems (NeurIPS), 2023.
[45] Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICML), 2024.
[46] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-wcighted regression: Simple and scalable off poolpy reinforcement learning. ArXiv, abs/1910.00177. 2019.
[47] Jan Peters and Stefan Schaal. Reinforcement learning by reward- weighted regression for operational space control. In International Conference Machine Learning (ICML), 2007.
13

---

[48] Rafael Rafailov, Kyle Beltran Hatch, Anikait Singh, Aviral Kumar. Laura Smith, 1lya Kostrikov, Philippe Hansen-Estruch, Victor Kolev, Philip J Ball, Jiajun Wu, et al. D5rl: Diverse datasets for data-driven deep reinforcement learning. In Reinforcement Learning Conference (RLC), 2024.
[49] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo. Alexander Novikov, Gabriel Barth martn, Mai Giménez. Yury Sulsky, Jackie Kay. Jost Tobias Springenberg, al. A generalist agent. Transactions on Machine Learning Research (TMLR), 2022.
[50] Harshit S. Sikchi, Qinqing Zheng, Amy Zhang, and Scott Nickum. Dual Unification and new methods for reinforcement and imitation learning. In International Conference on Learning Representations (ICLR), 2024.
[51] Jost Tobias Springenberg. Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Manfred Otto Heess, and Martin A. Riedmiller. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning (ICML), 2024.
[52] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 15())11229-195, 2014.
[53] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurlPS), 2023.
[54] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented doep offline reinforcement learning library. In Neural Information Processing Systems (NeurIPS), 2023.
[55] Ruosong Wang, Dean Phillips Foster, and Sham M. Kakade, What are the statistical limits of offline with linear function approximation? In International Conference on Learning Representations (ICIR), 2021.
[56] Ruosong Wang. Yifan Wu, Ruslan Salakhutdinov, and Sham M. Kakade, Instabilities of offline rl with pre-trained neural representation. In International Conference on Machine Learning (ICML), 2021.
[57] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning (ICML), 2023.
[58] Ziyun Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg. Scott E. Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Manfred Otto Heess, and Nando de Freitas. Critic regularized regression. In Neural Information Processing Systems (NeurIPS), 2020.
[59] Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv, 11s/191
[60] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforeement learning. In International Conference on Machine Learning (ICML), 2021.
[61] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang. Zhaoran Wang. Victor Chan, and Xianyuan Zhan. Offline rl with oed actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.
[62] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. International Conference on Machine Learning (ICML), 2021.
[63] Rui Yang, Yong Lin, Xiaoteng Ma, Haotian Hu, Chongjie Zhang, and Zhang. What is essential for unseen goal gencralization of offline goal-conditioned rl? In International Conference on Machine Learning (ICML), 2023.
[64] Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Abbeel, Alessandro I Lazaric, and Lerrel Pinto. Don't change the al algorithm, change the data: Exploratory data for offline reinforcement learning. ArXiv, abs/2201. 13425, 2022.
[65] Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing contrastive rl: Techniques for offline goal reaching. ArXiv, 2ew/23006003446, 2023.
14

---

Appendices
A
Limitations
One limitation of our analysis is that the MSE metrics in Equations (4) to (6) are in some sense "proxies" to measure the accuracy of the policy (somewhat similarly how Bellman errors do not always accurately reflect value errors in the context of value learning [16]). For instance, if there exist multiple optimal actions that are potentially very different from one another, or the expert policy used in practice is not sufficiently optimal, the MSE metrics might not be highly indicative of the performance or accuracy of the policy. Nonetheless, we empirically find that there is a strong correlation between the evaluation MSE metric and performance, and believe our analysis could further be refined with potentially more sophisticated metrics (e.g., by considering B[Q" (s, a)] instead of [[(T(s) nt (s))? which we leave for future work.
Another limitation of our analysis in Section 4 is only consider policy extraction in continuous- action environments. In discrete-action environments, our takeaway might not directly apply in its current form because (I) DDPG+BC not straightforwardly defined with discrete actions and (2) it is possible to directly use the Q function to implicitly define a policy (without having a separate policy network). We leave investigating the effect of policy extraction in discrete-action environments for future work.
B Preliminaries
We consider Markov decision process (MDP) defined by M (8, (S, S denotes the state space, A denotes the action space, r S x A + R denotes the reward function, μ E ASS) denotes the initial state distribution, and p: : S x A - A(S) denotes the transition dynamics, where A(X) denotes the set of probability distributions over a set X. We consider the offline RL problem, whose goal is to find policy S + ((A) (or S - A deterministic) that maximizes the dis- count return J(x) = Ex-pp()) (r) r'r(s at)], where p" (r) = p" (su, ag. 81, a1.0a...... 8T, sT,,a)) = ((s so)p(s1 So. ov) ... T(r ST) and y is a discount factor, solely from a static dataset D=
without online interactions. In some experiments, we consider offline goal- conditioned RL [2. 11, 22. 44, 57] well, where the policy and reward function are also condi- tioned on a goal state g, which is sampled from goal distribution Pg E AS. For goal-conditioned RL., we assume a sparse goal-conditioned reward function, r(s,g) g) = I (s = g), which does not re- quire any prior knowledge about the state space. We also assume that the episode ends upon goal- reaching [44, 57].
C Policy generalization: Rethinking the role of state representations
In this section, we introduce another way to improve test-time policy accuracy from the perspective of state representations. Specifically, we claim that we can improve test-time policy ac- curacy by using a "good" representation that naturally enables out-of-distribution generalization. Since this might sound a bit cryptic, we first show results to illustrate this point.
x(a 4.9)
<ti 4(+).
Return
Evaluation MSE
134
3008
15
Figure 9 shows the performances of goal-conditioned BC! on wwwaamamaeaaaaae with two different homeomorphic represen- tations: one with the original state representation 8, and one with
Validation MSE
Training MSE
different representation (ss) with continuous, invertible ò (specifically, á transforms z-y coordinates with invertible tanh kernels; see Appendix D.6). Hence, these two representations contain the exactly same amount of information and are even topologically homeomorphic (under the standard Euclidean topol- ogy). However, they result in very different performances, and
Z01K 13
50085
1M
Grodient Steps
Gondient Stops
Figure 9: A good state represen- tation naturally enables test-time generalization, leading substan- tially better performance.
Here, we use BC (not RL) to focus solely on state representations, obviating potential confounding factors regarding the value function.
15

---

the MSE plots in Figure 9 indicate that this difference is due to nothing other than the better test-time, evaluation MSE (observe that their training and validation MSEs are nearly identical)!
This result sheds light on an important perspective of state representations: a good state representa- tion should be able to enable test-time generalization naturally. While designing such good state representation might require some knowledge inductive biases about the task, results suggest that using such a representation is nonetheless very important in practice, since it affects the perfor- mance of offline RL significantly by improving test-time policy generalization capability.
D Experimental details
We provide the full experimental details in this section.
D.1 Value learning objectives
One-step RL (SARSA). SARSA [5] is one of the simplest offline value learning algorithms. Instead of fitting Bellman optimal value function Q", SARSA aims to fit behavioral value function Qo with TD-Jearning without querying out-of-distribution actions. Concretely, SARSA minimizes the following loss:
min LSARSA (Q) ("t...tta"'>>D(((ssss + ((( a)s,a))))
where s' and a' denote the next state and action, respectively, and Q denotes the target Q network [39]. Despite its apparent simplicity, extracting policy by maximizing the value function learned by SARSA is known to be a surprisingly strong baseline [5,30].
Implicit Q-learning (IQL). Implicit Q-learning (IQL) [25] aims to fit a Bellman optimal value function Q' by approximating the maximum operator with in-sample expectile regression. IQL minimizes the following losses:
min Log (Q) ((a.,"") -176) Q(s,
(10)
Q
min (V) E( (ooNDll(((ss)))) V()))]
(11)
where =|7 -(x)=|1-- 1(x < 0)(2² is the expectile loss [43] with an expectile parameter т. Intuitively, when 7 > 0.5, the expectile loss in Equation (11) penalizes positive errors more than negative errors, which makes V closer to the maximum value of Q. This way, IQL approximates V* and Q" only with in-distribution dataset actions, without referring to the erroneous values at out-of-distribution actions.
Contrastive RL (CRL). Contrastive RL (CRL) [11] is value learning algorithm for offline goal- conditioned RL based on contrastive learning. CRL maximizes the following objective:
max JCRL (f) = E s. (-ssa)) ~x (-) [log a (f(ss a, g)) + log(1 (f(ss,,g a,9 M)), (12)
where a denotes the sigmoid function and pD( 8, a) denotes the geometric future state distribution of the dataset D. Eysenbach et al. [11] show that the optimal solution of Equation (12) is given as
f (s, a, g) ( s, a) a)/pt())) which gives the behavioral goal-conditioned function as Q8 (s, (. = (g s, a) = (g)e (s...) where (g) a policy-independent constant.
D.2 Environments and datasets
We describe the environments and datasets we employ our analysis.
D.2.1
Data-scaling analysis
For the data-scaling analysis in Section 4, we employ the following environments and datasets (Figure 10).
cestmaze-large and gc-antmaze-large are based on the natmazz--arge-ddvvere ronment from the D4RL, suite [12], where the agent must be able to manipulate quadrupedal robot to reach a given target goal (antmaze-large to reach any goal from any other state
envi-
16

---

large
hopper
valker2d
kitchen
Figure 10: Environments.
aat-aam-aaeelarge in given For the dataset for gc-antmaze- in data- scaling analysis, we collect 10M transitions using a noisy expert policy that navigates through the maze. We the same policy and noise level (Øasta 0.2) as the used to collect antmaze- dettmaee-large-dieere------ in D4RL.
d4r1-hopper and d4r1-walkers are the hopper-medil um-v2 and walker2d-medium- tasks from the D4RL locomotion suite. We use the original IM-sized datasets collected by par- tially trained policies [12].
exorl-walker and exorl-cheetah are the valker-run and cheetah-run tasks from the EXORL benchmark [64]. We use the original 10M-sized datasets collected by RND agents [6]. Since the datasets are collected by purely unsupervised exploratory policies, they feature high suboptimality and high state-action diversity.
kitchen based the kitchen-mixed-ro task from the D4RL suite, where the goal is to complete four manipulation tasks (e.g., opening the microwave, moving the kettle) with a robot arm. Since the original dataset size is relatively small, for our data-scaling analysis, we collect large 1M-sized dataset with a noisy. biased expert policy, where we add noises sampled from zero-mean Gaussian distribution with a standard deviation of 0.2 addition to a randomly initialized policy's 's actions to the expert policy's ' actions.
gc-roboverse is based goal-conditioned robotic task, where the goal is to manipulate robot arm rearrange objects match a target image. The agent must able to perform object manipulation purely from 48 48 x images. We use the 1M-sized dataset used by Park et al. [44], Zheng et al. [65].
D.2.2 Policy generalization analysis
For the policy generalization analysis in Section 5, we use the antmaze-medium-diieeeeee -argzee-large-iii divedie-veeeeeeee Sitchen-partial- hammer-cll hammer-cloned-ee cloned-v1, door-cloned- hopper perpoer-mediumm v2, and walker2d-medium. envi- ronments and datasets from the D4RL suite [12] well as the walker-run and cheetah-run from ExORL suite [64].
sitchen-mixed-e pen-cloned-vl,
D.3 Data-scaling matrices
We train agents for 1M steps (500K steps for gc-roboverse) with each pair of value learning and policy extraction algorithms. We evaluate the performance of the agent every 100K steps with 50 rollouts, and report the performance averaged over the last 3 evaluations and over seeds. In Figures and 7, individually tune the policy extraction hyperparameter (a for AWR and DDPG+BC, and N for SfBC) for each cell, and report the performance with the best hyperparameter. To save computation, extract multiple policies with different hyperparameters from the same value function (note that this is possible because we use decoupled offline RI. algorithms). To generate smaller-sized datasets from the original full dataset, we randomly shuffle trajectories the original dataset using a fixed random seed, and take the first K trajectories such that smaller datasets are fully contained in larger datasets.
D.4 MSE metrics
We randomly split the trajectories in dataset into a training set (95%) and a validation set (5%) our experiments. For expert policies n" in the MSE metrics defined in Equations (4) to (6), we use either the original expert policies from the D4RI. suite (adroit-(pes and gc-antmaze-largee or policies pre-trained with offline-to-online RL until their performance saturates (antmaze- (medium, large} and sitchen-mixed. To train "global" expert policies for
hammer, door}
17

---

uetmaze-(mediuu large), we reset the agent to arbitrary locations in the entire maze. This initial state distribution is only used train an expert policy; we use the original initial state distribution for the other experiments.
D.5 Test-time policy improvement methods
In Figure 8, for IQL, SIBC, and OPEX, we train IQL agents (with original AWR) for 500K (kitchen) or 1M (others) gradient steps. For TTT, we further train the policy up to 2M gradient steps with learning rate of 0.00003. In antmaze, we consider both deterministic evaluation and stochastic evaluation with a fixed standard deviation of 0.4 (which roughly matches the learned standard deviation the BC policy), and report the best performance of them for each method.
D.6 State representation experiments
We describe the state representation é used in Appendix C. An antmaze state consists of 2-D r-y coordinates and 27-D proprioceptive information. We transform x and individually with tanh kernels, ie.,
3; tanh
(13)
9: tanh
(14)
6x
where i E (1.2....,32)) 8, = x2 21, 6g = 1/2 y/, and 21....; T32 and .1.......................... U32 are de- fined as numpy. wery.linssper (insacce(2,, 38, 32) and numpy 1 www.sliccneeec.c (-2, 26, 32), respectively. De- noting the 27-D proprioceptive state as Sproprio- Φ(s) defined follows: (((x,, y; Spropriol) [Z,,..., T32, VI,..., ... 1322 Sproprio]. where ';' denotes concatenation. Intuitively, @ is similar to the discretization of the z-y dimensions with bins, but with a continuous, invertible tanh transforma- tion instead of binary discretization.
D.7 Implementation details
Our implementation is based on (for offline goal-conditioned RI.). We use an internal cluster consisting of A5000 GPUs to run our experiments. Each experiment in our work takes no more than 18 hours.
[20] and the official implementation of HIQL [44]
D.7.1 Data-scaling analysis
Default hyperparameters. We mostly follow the original hyperparameters for IQL [25], goal- conditioned IQL [44], and CRL [11]. Tables 2 and list the common and environment-specific hyperparameters, respectively. For SARSA, standard ( loss instead of an expectile loss. For pixel- environments (i.e., gc-roboverse). the same architecture and image augmentation Park et al. [44]. In goal-conditioned environments as well as antmaze tasks, we subtract from rewards, following previous works [25, 44].
the same implementation IQL, but with the
Policy extraction methods. We use Gaussian distributions (without tanh squashing) to model action distributions. We use a fixed standard deviation of for AWR and DDPG+BC and learnable standard deviation for SIBC. For DDPG+BC, we clip actions be within the range of [-1, in the deterministic policy gradient term in Equation (2). We empirically find that this is better than tanh squashing [14] across the board, and is important achieving strong performance in some environments. We list the policy extraction hyperparameters we consider in our experiments in curly brackets in Table 3.
D.7.2 Policy generalization analysis
Hyperparameters. Table 4 lists the hyperparameters that we use in our offline-to-oniee RL and test-time policy improvement experiments. these experiments, we use Gaussian distributions with learnalee standard deviations for action distributions.
18

---

Table 2: Common hyperparameters for data-scaling matrices.
Hyperparameter
Value
Learning rate
0.0003
Optimizer
Adam [24]
Target smoothing coefficient
0.005
Discount factor
0.99
Table 3: Environment-specctie hyperparameters for data-scaling matrices.
Environment
gc-antmaze large
antmaze- -large
d4r1-hopper
d4r1 walker
# gradient steps
10"
10
10"
10
Minibatch size
1024
256
256
256
MLP dimensions
(512, 512, 512)
(256, 256)
(256, 256)
(256, 256)
IQL expectile
0.9
0.9
0.7
0.7
LayerNorm [3]
True
False
True
True
AWR a (IQL)
{0, 1,3, 10)
10,
{0, 1,3,,,
{0, 1,3, 10)
AWR a (SARSA/CRL)
{0, 10, 30, 100)
10,
10,
{0, 1,3,
DDPG+BC (IQL)
(0.1,
(0.1,0.3,
(0,
11,3,10,30
DDPG+RC (SARSA/CRL)
00.1,
(0.1, 1.3)
{1,3, 10.30 (1,16.61)
{1,3, 10,30)
SIBC N (IQL)
64, 64)
(1. 16, 64)
(1,16,64] 64]
SIBC N (SARSACRLL)
16, 64)
(1. 16,64 64)
(1,16.64)
(6,16,44 64)
Environment
exorl- sxorl-valker
exorl-cheetah
kitchen
gc-roboverse
# gradient steps
10c
10c
10
5 10a
Minibatch size
1024 (512, 512)
1024
1024
256
MI. dimensions
(512, 512, 512)
(512, 512, 512)
(512, 512, 512)
IQL expectile
0.9 True
0.9 True
0.7
0.7 True
LayerNorm [3]
False
AWR a (TQL.)
10 10, 100) 10,
10 10, 100)
{0, 1,3, 10)
1,,
AWR (SARSACCRL)
{0, 100 100)
{0, 1,3, 10)
10, 100]
DDPG+BC (IQL)
0..0
(0.0.01,0.1.1)
(10,30, 100, 300)
(3, 30, 100)
DDPG+BC (SARSACRL)
(0,0.01.
(0.0.01.0.1.1
(10,30 100,, 300)
(0, 100)
SIBC N (IQI.)
16,
66,66)
(1,
16,66,
SIBC N (SARSACCRL)
61,16.64
64,
(1,16,64)
(1,16,61)
E Additional results
We provide the full data-scaling matrices with different policy extraction hyperparameters (a for AWR and DDPG+BC, and N for SfBC) in Figure 11.
19

---

Table 4: Hyperparameters for policy generalization analysis.
Hyperparameter
Value
Learning rate
0.0003 Adam [24]
Optimizer
# offline gradient steps
10 (antmaze), 5 x 10 (kitchen, adroit)
total gradient steps
2 x 10
# gradient steps per environment step
Minibatch size
1024 (kitchen). 256 (antmaze, adroit)
MLP dimensions
(512, 512,512) 512) (kitchen), (256, 256) (antmaze, adroit)
Target smoothing coefficient
0.005
Discount factor 7
0.99
LayerNorm [3]
True (kitchen), False (antmaze, adroit)
IQL expectile
0.9 (antmaze), 0.7 (kitchen, adroit)
Policy extraction method
AWR
AWR
10 (antmaze), 0.5 (kitchen), 3 (adroit)
SIBC N
16
0.3 (antmaze), 0.0003 (kitchen), 0.03 (d4r1-hopper)) 0.1 (d4r1-walker24) (exorl-(valker. cheetah))
OPEX 3
0.3 (antmaze), 5 (kitchen), 0.5 (d4r1-hopper), 0.3 (d4r1-waleer2d) 0.01 (exorl-wvaleer, cheetah))
TTT 8
20

---

la-ant (QL
gc- Sc-antmaze-lare (CRL)
AWW la M.00
AWR la
140
AWR i 3.4
AWI (O 10.00
AWR for 0.0
AWR
10.0
AWR 30.0
AWR
100.01
89
1M0
A1SANA 3M
Patt
NUSONSS 15
Patts
Vilue
IN 1M
Vilco
14 MM
DDPG
DDPG
DDPC
DDPG
DDPG 3.0
DDPC
DDPC
DDPG
0224
1:
EIN
013
RIN-
MIR-
XTM-
TTANAS TM
DO 31
191
194
KLOSO 23 1822
Vilue Duts
Value Data
Vidao
Data
Date
Data
Vidao
Data
Data
SMICI IN
1)
SIIC
SPIK
IV
61
IN
SIBC
SIBC
15
3001
92
Valee Date
Value
Valae Data
Valco Data
Vilke Duta
Vikke Duta
antmaze- large (IQL)
large (SARSA)
AWT.
0000
AWR
AWI
1000
AWI 30.00
AWT
AWR (a 3M0
AWR 1000
AWR (cx 20L0)
27
0
16
0254
16
1
0
1.21
022
0
1.00
0
0
KIM
Vidao Tota
Valee Date
Vilao Data
Valeo Tate
Viduo Duta
Valer Date
Viluo Duta
Vaker Tatt
DEPG
8.0
DDPG
1.0
DDPG
0.55
DDPG
0.1
DDPG
3.0
DDPG
1.0
DDFG J 0.50
DDPG tO O.L:
152-
D3-
130-
350-
130-
9
0
IU Bu 1M
Nu n2M
24 IM
11 noM
PM Bu 1M
24
114
0
0
100
0.94
C.N4
0.00
9
1:31 Vilue Data
Vidaa Data
Valua Data
Vikao Data
Vikue Data
Value Duta
d4r1 -hopper (IQL)
d4r1-hopper
(SARSA)
AWNR
AWR
JML
DL
1.21
31
0252
025
8.00
Vaite Dota
Vahee Dats
Vatce Data
Vahue thite
Voice Dats
Vahue Uats
Voice Data
Valee Data
DDPG ta 30.0:
DDPG (ox
10.01
DDPG ja 3.0
DDPG lor 1.0
DDPG la 30.0
DDPG (o
10.00
DDPG io 3.00
DDPG for
1.01
DX
53
X1
51
4
51
50
35
LING
15
0155
111
035E b. Di
0111
Vilao Duda
SBB
SIBC
SIBC NV
SBC (N
SIBC UN
SIBC
IMC
11
1S4.
1.20
1:54
1:M
Vadue Data
Valua Data
Vilue Data
Vilues Chata
Vadua Data
Valu Data
d4r1- 2eee-2al (IQL)
d4rl serl-waale -wakkaalleeddd (SARSA)
AW-T
AWR
AWB
AWR
10.01
AWVI
AWR
AWT 3.00
AWRI
10.0)
150
DX
6.5 odMs -4
14
AdM
34
14
-4
54
154
DDPG la 30.00
DDPG (a
10.00
DDPG 3.00
DDPG 3.0
DDPG (a
30.0)
DDPG
10.00
DDPG ja
3.00
DDPG fos
81
0224
20
50
22
54
0254
70
SE
LIM
Vidao Duta
Valer Date
Voduo Duta
Valeo Date
Vodoo Duti
Valeo Date
Vilao Duten
Vake- Saate
SMB (M
T)
SEB UN
LG1
SIBC NV
64
SBC IN
SIBC IN
LGI
SIBC
64
100-
134-
131-
50
10
64
56
CON
54
1:34
20
2054
1:54
1:51
21

---

exorl uulker (TQL)
exorl walker (SARSA)
10000
1000
301-
201-
1
Videe Shete
Vikee Datee
Volee thete
Vatee Date
DDRG bo - 1.0
DDFC in
0.33
DDPG lo 0.01:
DDPG io 0.00
DDPG fre 1.00
DDPG
0.83
DDPG (o 0.01
DDPS
0.00
201
1014 (96)
035
MoA 125
155
201
110
201
110
194-
IN4
132-
Vilee Dhan
Dats
Vaaw Dats
Vidte Data
Valce Datt
Vadwo Data
SIBC IN
SenC IN
100
SEBCI IN
641
SeIC UN
SEDCI UN
161
SIDDC (N
19
144
135
124
181
13
14
135
131-
120
155
0.00
12
11 24 151 101
24
51 18M
Su
exorl-cheetah (IQL)
exorl-cheetab (SARSA)
AWWT (rV
AWWT (/V = 1.09
AWR (N
AWR (ov
AWT (re aa [UU)
AWB (VV CL 1.0)
AWR in 10.0)
AWI inv
91
100
45
1M4
Viduo Duta
1M
Vilue Dhat
19
Vidto
14
Vikue Dual
DIPG
1.0
DIPG
DDPG
0.01:
DDPG
DOPG
1.00
TPPG
DDPG
0.01
DDPG
0..00
12
1391-
1301-
71
91
3091
3:31
10C
VM
1944
158 24
22
Videe Shat
Value Untt
Valto Datta
Vadw Data
Vidte Datt
Vidke Dam
Valce Data
SIBCIA UN
Al
SIBC
SBBC
UN
SIBC
JAGL
155
JAC. 12
[81
1196
15E
LALL
1EA
170
J0o 775
1X
1N
131
100
154
H52
Valia Dats
Valke Daca
Vadao Dara
Valta Data
Vidke Daca
Viluo Daca
kitchen (IQL)
kitchen (SARSA)
AWR
AWR
18.0
AWR
AWR
48
0000
niu Data
n2u
Data
Vidao Date
Vidao Date
Vidao Data
Date
Datos
Datts
DDPG 300.0) DDPG
in
100.0) DDPG
In
30.01
DDPG In 10.01
DDPG in 300.0, DIPG in
100.0)
DDPS
(n
36.00
DDPS (v
10.01
53
1050
0336
57
40
XIM
Videe Date
Vileo Dote:
Vilao Todes
Video Tutes
Vilee Tute
Vikue Dute:
Volue Duter
Voduo Dader
SIBC IN
SIBCIN IN
LG
SIBCIN IN
(44
SIBC IN
SEBCIN UN
1G:
SEBCI IN
641
20
A.D rdu 194
B.DE kdu 14
E. kdsu 14
ADD ruu 194
A. vudi 14
R. vull 1N
gc-roboverse (TQL)
gc-roboverse (CRL)
AWR (a. 0.00
AWR (cr - 0.11
AWR is 1.00
AWR (or 10.01
AWm (rx =s Bul)
AWB (v> 1.0)
AWR 1800)
AWI inv
10000)
15
20
14
KIM-
12
3131
301
Viluo Duts
Valuo Data
Vidue
15
IM
14
DOPG 100,0) DDPG
DDPG 1000)
DDPG 3.0;
DDPG 100.0)
DDPO
30.0:
DDPG
10:0;
DDIN 3.0)
100
IM
IM
Das
IM
IM
IM
1P
15
14
13
12
15
17
7
6
6
iM
IM
wist
IM
AM
IM
Valuo Data
Value thite
Valce Data
Vidue thite
Vodee Data
Vosce Data
Viice Data
SEBCC (NV
3)
SIBC
146
SIBC (NV B4.
SMAC LN
X)
SEBCI 146
SEBCC
6-4)
PARO Stit
18
23
14
23
J7
16
3]
E2L-
413
10
Vilue
Vilta
Vilkas Datk
Vilca Data
Viloe Data
Vilue Daca
Figure 11: Full data-scaling matrices of AWR, DDPG+BC, and SIBC with different hyperparameters.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we support the claims made in the abstract and introduction with empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This is an empirical analysis paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the full experimental details as well as the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
See the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/ Suuides/CodeSuuei
wwwsssoolllooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nipps wttpccccppcccceeecccccccc.ccccct htapllic/giieessee /Coeeeeemmmdm sesiissssoooooooo onnooicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report 95% bootstrap confidence intervals standard deviations for all of the plots in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttr//nrrss...// cc http///ccccccccccceeeeelleeeeeee
Answer: [Yes]
Justification: Yes, follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is purely algorithmic research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
26

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This is purely algorithmic research.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly acknowledge the code and datasets we use in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wnwerswwihchddee thcode. com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
27

---

Answer:
[NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "aOIXs8KD0C", "reviewer": "Reviewer_BVrx", "summary": "This paper addresses the question of why offline RL often underperforms imitation learning. They formalize the question they choose to ask, \" is the bottleneck in learning the value function, the policy, or something else? What is the best way to improve performance given the bottleneck?\", and provide three potential explanations: imperfect value function estimation, imperfect, policy extraction from value function, and imperfect policy generalization to novel states during evaluation. They next perform a series of experiments to test each hypothesis, and conclude two main reasons for offline RL's underperformance: policy extraction from value functions, and test-time policy generalization. They use these observations to highlight important takeaways for RL practitioner and RL researchers.", "questions": "From what I can tell, most of the empirical results are in environments with continuous action spaces. Do you expect the results to carry over to environments with discrete action spaces?", "limitations": "The authors discuss and address limitations of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I think that this is an exceptionally well-written paper. They make it very clear what their hypotheses are, how they test for each, and what the reader should take away from each subsection. \n\nI further think that this paper addresses important questions in offline RL, and brings to light important directions for future work, which is a valuable contribution. \n\nThe full experimental details as well as code are provided, which is very useful for the community in reproducing the results and building off of this work.", "weaknesses": "At a high level, there are not many weaknesses I can name in this work. One is perhaps that their contributions are mainly empirical observations, and it would be nice to support these with theoretical results (even in very simple settings full of necessary assumptions), but I believe that even without theory this paper is very strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and constructive feedback about this work! We especially appreciate the question on discrete-action MDPs. Please find our answer to the question below.\n\n---\n\n* **“... most of the empirical results are in environments with continuous action spaces. Do you expect the results to carry over to environments with discrete action spaces?”**\n\nThanks for raising this point. There are two main differences between discrete-action and continuous-action MDPs: (1) discrete-action MDPs do not always require a separate policy extraction procedure, as we can directly enumerate over actions to choose the argmax action, and (2) DDPG is not straightforwardly defined in discrete-action MDPs. Hence, one of our takeaway messages (“use DDPG+BC instead of AWR”) may not be directly applicable to discrete-action MDPs. However, we expect that our main findings — namely (1) policy extraction can inhibit the full use of the learned value function (if there’s a separate policy extraction step) and (2) test-time policy generalization is one of the significant bottlenecks in offline RL — still apply to discrete-action MDPs as well. For example, recent works in RL fine-tuning of LLMs, which use a discrete action space, have observed similar findings to us: Tajwar et al. (2024) [1] showed that sampling actions (i.e., responses) from the policy (akin to sampling actions from the policy for making an update in DDPG+BC) lead to better performance than AWR that does not sample a new action from the policy for training, and Cobbe et al. [2] showed the effectiveness of test-time verifier reranking methods, which is closely related to the test-time generalization argument and TTT/OPEX in our paper. While of course we do not focus on LLMs in this study, these prior works illustrate that similar findings are applicable to discrete actions. We will add a discussion about discrete-action MDPs in the paper.\n\nWe would like to thank the reviewer again for the helpful feedback and please let us know if there are any additional concerns or questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kjaNBKQAfc", "reviewer": "Reviewer_Bu3n", "summary": "This paper empirically analyzes the bottlenecks in offline RL from three aspects: value learning, policy extraction, and policy generalization at evaluation time. Through the empirical evaluation, two observations were made: 1) the policy extraction algorithms affect the performance of offline RL significantly, often more than its underlying value learning objective. 2) the sub-optimal performance of offline RL agents is mainly attributed to a lack of generalization of the unseen state during evaluation instead of accuracy in the training distribution.", "questions": "1. For empirical analysis 2, I am wondering if the online data mainly improves value learning, policy learning, or both of them. The current evaluation only shows the improvement of evaluation MSE in actions but I'm wondering if the Q-value estimation loss would follow the same pattern, maybe the author could provide more insights on this?", "limitations": "Limitations were not discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: Good. This paper tries to analyze the bottleneck of offline RL methods and includes some novel and interesting observations that were not systematically discussed before.\n\n**Clarity**: Good. The paper is well-structured and easy to follow. The takeaway after the empirical analysis helps the reader understand the experiment's results.\n\n**Significance**: This is important. The reason behind offline RL's underperformance is an important topic for the future development of offline RL methods.\n\n**Technical Accuracy**:\n\n1. Throughout evaluation with a decent amount of experiments\n2. The experiment design are motivated and backed by hypothesizes", "weaknesses": "1. There is a lack of variance measures, such as standard deviation (std) or confidence intervals (CI), for the data-scaling matrices in Figures 1, 2, and 6, making the numerical results less plausible.\n\n2. For some experiments in empirical analysis 1, an increase in data leads to a decrease in performance. For instance, in Figure 1 (gc-antmaze-large), both IQL+AWR and IQL+DDPG show that the 10-10 configuration performs worse than the 1-10 and 10-1 configurations, which seems to contradict to the assumption, more data leads to better value/policy approximation. These observations lack an adequate explanation.\n\n3. I am concern that one of the main question (as outlined in the title as well) “is value function learning the main bottleneck of offline RL” is not sufficiently examined in the empirical experiments. While the amount of data used for training can be an implicit indicator of how well the value function is trained, it does not directly tell us the distribution of approximation errors (e.g., overestimation). Overestimation could still be a significant issue in offline reinforcement learning. It would strengthen the augment if an direct comparison between the predict value and true value for can be conducted, just like in [1] and [2].\n\n[1] Van Hasselt, H., Guez, A. and Silver, D., 2016, March. Deep reinforcement learning with double q-learning. In *Proceedings of the AAAI conference on artificial intelligence* (Vol. 30, No. 1).\n\n[2] Fujimoto, S., Hoof, H. and Meger, D., 2018, July. Addressing function approximation error in actor-critic methods. In *International conference on machine learning* (pp. 1587-1596). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback about this work. We especially appreciate the reviewer's feedback about statistical significance as well as the question on our claim about value learning. Following the reviewer’s suggestion, we have added variance metrics as well as four more seeds (8 seeds in total) to improve the statistical significance of our results. We believe these changes have strengthened the paper substantially. Please find our detailed answers below.\n\n---\n\n* **“lack of variance measures”**\n\nThanks for raising this point. Following the suggestion, we have added standard deviations to our data-scaling matrices, and report them in **Figure 1 in the additional 1-page PDF**. We have also updated our paper accordingly.\n\n* **“For some experiments in empirical analysis 1, an increase in data leads to a decrease in performance.”**\n\nAs the reviewer mentioned, there are some cases where an increase in data doesn’t necessarily lead to improved performance. We believe this is mainly due to statistical errors. To address this concern, (1) we’ve added 4 more seeds to our data-scaling matrices by adding **7744 more runs** (now we have **8 seeds** in total for every experiment in the paper), and (2) we have added variance metrics to the table. Please find them in **Figure 1 in the new 1-page PDF**. There are still some rare cases where an increase in data doesn't lead to performance improvement, but the new standard deviation metrics tell us that this is likely because of statistical noise. We believe that these changes have significantly strengthened the statistical significance of our results.\n\n* **About the title / overestimation in value learning**\n\nWe agree that overestimation is an important issue in offline value learning in general, and would like to clarify the scope of our claims. Concretely, we would like to clarify that we’re *not* suggesting that value learning is less or not important compared to policy learning in general. Rather, our claim is that, (due to recent advancements in offline value function training) *current* state-of-the-art offline RL methods often tend to be at the point where improvements in policy extraction and generalization can lead to larger improvements in performance; in this sense, we claim that policy extraction and generalization are often the main “bottlenecks” in current offline RL, hence the title. As we show in our experiments (Figures 1 and 2), we are at the point where improvements in value functions (by adding more data) often do not translate to policy performance if the policy learning algorithm is not chosen appropriately. Hence, we think it is useful to concretely highlight and emphasize policy learning as a bottleneck at this point since the community has focused on value learning substantially. Thanks for asking this question, and we have revised our paper (Intro and Section 3) to make this point clearer.\n\n* **“For empirical analysis 2, I am wondering if the online data mainly improves value learning, policy learning, or both of them.”**\n\nThanks for the interesting question. In this paper, we mainly measure the accuracy of the policy because the policy is what is deployed at test time. We expect additional online data would improve both the value and the policy in general, but it might be a bit challenging to faithfully measure the accuracy of the learned value function (as opposed to policy accuracy), given that (1) the Bellman loss often does not necessarily correlate with the actual performance [3] and (2) offline value functions (especially with techniques for mitigating overestimation like IQL/CQL) often involve pessimism/conservatism, which makes it difficult to directly compare against the actual ground-truth Q value. While we do have a comparison between the value and the policy in terms of data-scaling in the paper, we leave dissecting the effect of each component for *generalization* (with additional online data) for future work.\n\n\nWe would like to thank the reviewer again for raising important points about statistical significance and the score of our claims, and please let us know if there are any additional concerns or questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MyEizUt9UK", "reviewer": "Reviewer_C9VZ", "summary": "This paper attempts to understand the relative importance of policy learning and value learning in offline reinforcement learning. The analysis is broken into two parts: (1) when decoupling the policy and value learning steps the authors test the relative data efficiency of the two steps, and (2) the authors test how well the policy generalizes at test time. Experiments are conducted on a wide variety of domains and the authors argue that they show that policy learning is often the main bottleneck of offline RL.", "questions": "See weaknesses", "limitations": "Yes, but could do more to address how the results are not always clear cut.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The high level experimental methodology of comparing the relative importance of policy learning and value learning by varying the dataset size in decoupled algorithms is an interesting idea. It could be useful to guide future work to know if there is more upside to improving on policy learning or value learning. \n\n2. The result that IQL+DDPG outperforms IQL+AWR (which was used in the original IQL paper) is an interesting standalone result.\n\n3. While somewhat preliminary and cursory in the paper, the idea of OPEX and TTT to update the policy at test time with a fixed value is interesting as a way to show that the value function has more information in it than the policy extracts.", "weaknesses": "1. The results are very heuristic and often unclear. The definitions of \"policy bound\" and \"value bound\" in terms of color gradients is vague and not very informative. Visually, it is hard to get much out of the figures which just throw a ton of data at the reader without a very intuitive way to interpret it. This could maybe be resolved by creating a more clear single metric that indicates the tradeoffs or some summary statistics that show aggregate trends averaging across methods or datasets. Currently, the results are pretty hard to parse and not very convincing as a result. \n\n2. The empirical results are not as conclusive as the analysis/text suggests. For example, the main claim of section 4 that \"policy learning is often the main bottleneck of offline RL\" does not seem to be the right takeaway or emphasis of the results. Instead the results in both figure 1 and 2 indicate that sometimes policy learning is the bottleneck and sometimes value learning is the bottleneck.\n\n3. The methodology in section 5 is a not clear. It seems that the authors run offline-to-online IQL (line 304), but this would be using AWR, which the previous section suggests not to do. Moreover, the online algorithm updates not just the policy, but also the value. The paper does not consider the generalization of the value function, but only the policy. Perhaps a cleaner way to test interesting out of distribution generalization would be to consider the state distribution of the optimal policy? This would of course not test near optimal states (so maybe some noise could be added or something as in later experiments), but could be conceptually cleaner than fully changing the setting to online. \n\n4. It is not clear why there is such focus on the generalization of the policies and not the values. They seem to both matter, especially for things like OPEX or TTT to work. It would be interesting to see how well the value functions are generalizing as well as the policies to go to the papers main claims to be comparing the relative importance of these two steps. \n\n5. In general, the paper tries to cram in too many semi-related results. I would encourage the authors to focus the story a bit more clearly and maybe split some parts into the appendix or into a separate paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback about this work. It appears that we were not entirely clear about our main messages in the initial draft, which we believe may have caused some confusion about our claims. Below we describe how we have revised our paper to prevent potential confusion and have added new results about aggregation metrics and generalization analysis.\n\n---\n* **The results are not as conclusive as the paper suggests; they indicate that sometimes policy learning is the bottleneck and sometimes value learning is the bottleneck.**\n\nThank you for raising this valid point. We believe our initial draft was not entirely clear about our analysis framework. We first would like to clarify that there are *two* types of information we can obtain from the data-scaling matrices (Figure 1). \n- (1) By looking at each **individual matrix**, we can see scaling behaviors with the amounts of value and policy data in that *specific* setting. \n- (2) By comparing different matrices from **different value/policy algorithms**, we can understand the effect of each algorithmic component.\n\nAs the reviewer pointed out, if we look at individual matrices (the first perspective), some matrices are value-bounded (i.e., the performances are more affected by the amount of value data) and others are policy-bounded, and the results appear to be less clear and problem-dependent. **However, if we compare different algorithms (the second perspective), we can observe a much clearer trend in our results:** namely, (1) the choice of a policy extraction algorithm often affects performance more than the choice of a value learning algorithm (except antmaze-large), and (2) DDPG+BC is almost always better than AWR. Please refer to our **global response** for aggregation metrics that quantitatively support these observations.\n\nWe then find the reason behind this difference between AWR and DDPG+BC **by now looking at individual matrices** (Section 4.3). The gc-antmaze result in Figure 2 shows a clear difference between the two algorithms: with AWR, an increase in value data doesn’t necessarily translate to performance improvement, unlike with DDPG+BC. This suggests that AWR, one of the most widely used policy extraction algorithms, imposes a **“bottleneck”** that inhibits the full use of the learned value function (the “Takeaway” box in Section 4). In this sense, we argue that policy extraction is often the main bottleneck in current offline RL algorithms (esp. any method that involves weighted/filtered BC, including the recent scaling work [4]). Please note that this is different from saying that every *individual* data-scaling matrix is policy-bounded (which is indeed not true).\n\nWe believe our initial draft was not very clear about these points, and will revise the manuscript to clearly convey these findings.\n\n* **“the results are pretty hard to parse” / “Lack of clear aggregation metrics”**\n\nThank you for the helpful suggestion. Following the suggestion, we have added two types of quantitative aggregation metrics. Please refer to our **global response** for the details.\n\n* **“It is not clear why there is such focus on the generalization of the policies and not the values.”**\n\nThanks for raising this question. First, we would like to note that our main focus in the second analysis (Section 5) is **not** on comparing value generalization and policy generalization (unlike our first analysis); our main point is rather to show that *test-time generalization* (regardless of policy or value) is one of the important yet overlooked issues in offline RL. We used the phrase “policy generalization” in the paper because the policy is what is deployed at test time, but we didn’t intend to mean that policy generalization is necessarily more important than value generalization. Instead, we wanted to show how current offline RL algorithms are often *already* great at learning near-optimal actions on in-distribution states, and how bad they can be on test-time out-of-distribution states. This observation highlights an important (but often overlooked) open question in offline RL, which is very different from the previous main focus on \"value learning\" about pessimism and conservatism. We apologize for this potential confusion (which we think stems from some phrases in Sections 1, 3, and 5), and will revise the paper to make this point very clear (e.g., we will use just “generalization” not “policy generalization” whenever it’s more appropriate).\n\nThat said, we do have some empirical results that allow us to compare policy and value generalization, and we discuss these aspects in the **global response** above in case it’s of interest to the reviewer.\n\n* **The use of IQL+AWR in our offline-to-online RL experiment / MSE under** $d^{\\pi^*}$\n\nThanks for the suggestion. Following the suggestion, we have repeated the same experiment with IQL+DDPG+BC and additionally measured the MSE under $d^{\\pi^*}$. We obtained similar results and our conclusion remains the same. Please refer to our **global response** for the details.\n\n* **“In general, the paper tries to cram in too many semi-related results.”**\n\nOur primary motivation in this work is to understand the best way to improve the performance of current offline RL. To this end, we believe conducting a **holistic** analysis of different elements that affect the performance of offline RL is necessary, and thus we carried out such analyses (across value learning, policy extraction, and generalization) in this paper. That being said, to provide more coherent insights, we have substantially revised several subsections and paragraphs in the current version, which we hope helps address this concern.\n\nWe would like to thank the reviewer again for raising important points about our claims, which substantially helped strengthen the paper. **Please let us know if we have addressed the reviewer’s concerns and if so, we would be grateful if you are willing to upgrade the score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qjAlZOphNg", "reviewer": "Reviewer_VnDz", "summary": "The paper presents an empirical analysis to determine the main challenge in offline RL for control among value function learning, policy extraction, and policy generalization to test-time states. With various deep learning-based experiments, it reaches the conclusion that policy extraction and policy generalization are the main bottlenecks instead of value function learning.", "questions": "1. There seems like there is a relation between DDPG + BC’s focus on limited exploration (some improvement but staying close to behavior policy) and test-time generalization. More specifically, by encouraging a policy to be close to the behavior policy, the test-time generalization should naturally be better since it will not deviate much from those states. Section 4 does not comment on this relation, what are the authors thoughts on this?\n2. Regarding the limitations mentioned at the end, the authors may be interested in this paper [1] as well which discusses how some value-based objectives may be unreliable for performance. [1] does not give an suggestions, but does report a similar finding for another use-case.\n\n[1] Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error. Fujimoto et al. 2022.", "limitations": "Yes the authors report on the limitations in Appendix A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is systematic on specifically isolating the three components such as by using decoupled RL algorithms (having value function learning and policy learning phase separately). This is important to avoid confounding factors.\n2. The paper covers results across various data dimensions such as coverage, sub-optimality, and amount.\n3. The paper has clear takeaways, which are helpful in determining actionable advice.\n4. The paper provides an alternative direction (e.g. better policy extraction algorithms instead of value function learning algorithms)  for researchers to pursue in trying to improve offline RL.", "weaknesses": "1. It is a bit concerning that the main results in Figures 1, 2, 6 are over only 4 seeds. In my experience, offline RL algorithms can be quite fragile especially with so much variation in coverage, dataset amount etc, that 4 seems quite limited. Given that most of the environments are not image-based, more than 4 seeds seems reasonable.\n2. Related to above, the same figures do not report any information on variance of performance, which makes it difficult to determine if the reported means have been accurately reported. I would suggest reviewing some guidelines [1]\n3. While the takeaways are nice, the claim to “always use” (above Section 5) is too strong, in my opinion.\n4. In section 4.4, it's unclear if the first reason is a good explanation for better performance. While the actions of DDPG + BC are more extreme, I don't think extreme actions are what we should be aiming for. The experiments are presented as though these extreme actions lead to good performance (even if this was not the intention). While it may be true in the evaluated domains, it is unclear if this can be a general finding.\n\nI will be willing to increase the score if the issues above are addressed, especially 1 and 2.\n\n[1] Empirical Design in Reinforcement Learning. Patterson et al. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback about this work. We especially appreciate the reviewer's feedback about statistical significance and the clarity of our claims. Following the reviewer’s suggestion, we have added more seeds and variance metrics to improve the statistical significance of the results. We believe these changes have strengthened the paper substantially. Please find our detailed responses below.\n\n---\n\n* **“It is a bit concerning that the main results in Figures 1, 2, 6 are over only 4 seeds.” / “the same figures do not report any information on variance of performance”**\n\nThanks for raising this point. Following the suggestion, we’ve added 4 more seeds to Figures 1, 2, and 6 by adding **7744 more runs** (now we have **8 seeds** in total for every experiment in the paper) and have reported **standard deviation metrics** in our data-scaling matrices. Please find the improved results in **Figure 1 in the additional 1-page PDF**. Our conclusions remain the same, and we have updated these results in the current version of the paper.\n\n* **About extreme actions**\n\nWe fully agree with the reviewer that extreme actions do not necessarily mean they are more optimal. Our original intention was to show that AWR has *limited expressivity* compared to DDPG+BC, because AWR actions *always* lie in the convex hull of the dataset actions (at least in principle), while DDPG+BC can go outside of it. This resulted in more extreme actions for DDPG+BC (Figure 3), but this is not to say that extreme actions are better. We apologize for the confusion and we will clarify this point in the draft to prevent such potential confusion.\n\n* **“the claim to “always use” (above Section 5) is too strong”**\n\nThanks for the suggestion. We initially used the term “always” because we found that DDPG+BC is better than or as good as AWR in most of the cases (15 out of 16 settings — please refer to **Table 1 in the new 1-page PDF**). However, following the suggestion, we have toned down the takeaway by removing the word “always” in the current draft.\n\n* **“DDPG+BC’s focus on limited exploration” / “by encouraging a policy to be close to the behavior policy, the test-time generalization should naturally be better since it will not deviate much from those states”**\n\nThanks for the great question. We also believe that, in general, the higher the BC coefficient in DDPG+BC is, the better it can prevent encountering out-of-distribution states at test time. (We’d also like to note that this also applies to AWR since it has a temperature hyperparameter of a similar role.) However, in practice, due to function approximation or imperfect policy learning, the agent would almost always visit out-of-distribution states, even when there’s *only* the BC term, and thus the generalizability of the policy can still significantly affect performance even in this case. We provide one example of such evidence in Appendix C. That said, as the reviewer mentioned, we expect that a small BC coefficient would incur even further challenges in generalization.\n\n* **About the Fujimoto et al. paper**\n\nThanks for the pointer! We think this paper indeed points out similar limitations of the Bellman error. We will cite and discuss this paper.\n\nWe would like to thank the reviewer again for raising important points about statistical significance and clarity, and please let us know if there are any additional concerns or questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all four reviewers’ detailed feedback and suggestions. We would like to highlight the additional results we provide in the new 1-page PDF.

* **Adding $\mathbf{4}$ more seeds ($\mathbf{8}$ seeds in total) and standard deviation metrics:** Following the reviewers’ suggestions, we have added $\mathbf{4}$ **more seeds** as well as **standard deviation metrics** to our data-scaling matrices by adding $\mathbf{7744}$ **more runs** (**Figure 1** in the new PDF). Now, every experiment in our paper is aggregated over $\mathbf{8}$ **seeds** in total.
* **Aggregation metrics:** Following the suggestion of Reviewer C9VZ, we have added two new aggregation metrics that clearly highlight our takeaways.
  * First, we aggregate normalized returns over the entire data-scaling matrices for each value and policy algorithm (by marginalizing over every other factor), and report the interquartile mean (IQM) metrics [5] in **Figure 2** in the new PDF. The results support our finding that the difference between policy extraction methods is often larger than that between value learning methods.
  * Second, we aggregate the performances from different policy extraction methods for each environment and value algorithm. **Table 1** in the new PDF shows that DDPG+BC is indeed almost always (15 out of 16 tasks) better than or as good as AWR. We will add these results to the paper.
* **Offline-to-online RL experiment with IQL+DDPG+BC:** To address a concern of Reviewer C9VZ, we have repeated our offline-to-online RL experiment (Figure 5 in the original draft) with IQL+DDPG+BC, instead of IQL+AWR.
  * In the original offline-to-online RL experiment, we used IQL+AWR because (1) we wanted to use the simplest setting to illustrate the generalization issue (note that DDPG+BC requires a separate exploration mechanism in the online phase since it learns a deterministic policy, unlike AWR), and (2) most of the pathologies of AWR (e.g., the mode-covering issue, limited expressivity, etc.) go away if we’re allowed to use *online* rollouts [1].
  * That being said, following the suggestion of Reviewer C9VZ, we repeated the same offline-to-online experiments with IQL+DDPG+BC with Gaussian exploration noises, and report the results in **Figure 3** in the new PDF. The new results show a very similar trend to the original results, suggesting that the conclusion remains the same.
  * In addition, following the suggestion of the reviewer, we also measure the MSE metric under the state-marginal distribution of the oracle optimal policy (“$d^{\pi^*}$ MSE”). **Figure 3** in the new PDF shows the results, which suggests that the $d^{\pi^*}$ MSE is also correlated with performance (especially compared to the training/validation MSEs), but the correlation is slightly weaker than the evaluation MSE metric. We believe this is because the evaluation MSE directly measures the accuracy of the current policy under the *current* policy distribution.

---

Additionally, below we provide additional comments about test-time generalization to **Reviewer C9VZ**. We provide them here due to the character limit of each response.

* **Additional comments to Reviewer C9VZ about the point “It is not clear why there is such focus on the generalization of the policies and not the values.”**

As we discussed in the main response, our main focus in the second analysis is *not* to argue that policy generalization is necessarily more important than value generalization, but to highlight the significance of the effect of test-time generalization (regardless of policy or value) on performance, which is an important but often overlooked bottleneck in offline RL. That being said, in the paper, we do have some empirical results that allow us to compare policy generalization and value generalization, and we discuss them here in case it is of interest to the reviewer.

First, as the reviewer pointed out, our experiments with OPEX/TTT imply that *values often generalize better than policies*. OPEX/TTT only updates the policy (or actions) from the *fixed* offline value function on test-time states, and the fact that this often improves performance indicates that the information in the value function has often not been fully transferred into the policy. Hence, in this case, we would be able to say that *policy* generalization is being the “bottleneck” in performance.

Second, we also found that policy generalization *alone* can affect performance significantly. The results in Appendix C suggest this, where we show that just changing the representation of the BC policy can lead to a very significant difference in performance (this in fact leads to near-SOTA performance on gc-antmaze-large, even without using any RL!) and we show that this is due to nothing other than the difference in policy generalizability (note that this is a BC setting, so there’s no value function).

---

Below are the references that we use throughout our response:

[1] Tajwar et al., Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, ICML 2024. \
[2] Cobbe et al., Training Verifiers to Solve Math Word Problems, 2021. \
[3] Fujimoto et al., ​​Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error, ICML 2022. \
[4] Springenberg et al., Offline Actor-Critic Reinforcement Learning Scales to Large Models, ICML 2024. \
[5] Agarwal et al., Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021.

Meta-review: While numerous issues were raised at the initial review stage, clarifications and additional experimentation during the rebuttal convinced all four reviewers that the paper merits acceptance.  The authors should incorporate clarifications and new experimental results from the rebuttal into the final version of their paper.
Decision: Accept (poster)