A Closer Look at Offline RL Agents
Authors: Yuwei Fu
Di Wu
Benoit Boulet
Venue: NeurIPS 2022 Accept (2022)
Keywords: RL
Offline RL
Representation Learning
MBRL

Abstract: Despite recent advances in the field of Offline Reinforcement Learning (RL), less attention has been paid to understanding the behaviors of learned RL agents. As a result, there remain some gaps in our understandings, i.e., why is one offline RL agent more performant than another? In this work, we first introduce a set of experiments to evaluate offline RL agents, focusing on three fundamental aspects: representations, value functions and policies. Counterintuitively, we show that a more performant offline RL agent can learn relatively low-quality representations and inaccurate value functions. Furthermore, we showcase that the proposed experiment setups can be effectively used to diagnose the bottleneck of offline RL agents. Inspired by the evaluation results, a novel offline RL algorithm is proposed by a simple modification of IQL and achieves SOTA performance. Finally, we investigate when a learned dynamics model is helpful to model-free offline RL agents, and introduce an uncertainty-based sample selection method to mitigate the problem of model noises. Code is available at: https://github.com/fuyw/RIQL.

A Closer Look at Offline RL Agents
Yuwei Fu, Di Wu, Benoit Boulet
McGill University
yuwei.fu@ mail.mail.ca, wdi. benoit.boulet) @ meggill.ca
Abstract
Despite recent advances in the field of Offline Reinforcement Learning (RL), less attention has been paid to understanding the behaviors of learned RL agents. As result, there remain some gaps our understandings, i.e., why one offline RL agent more performant than another? In this work, we first introduce a set of experiments evaluate offline RL agents, focusing on three fundamental aspects: representations, value functions and policies. Counterintuitively, we show that a more performant offline RI. agent can learn relatively low-quality representations and inaccurate value functions. Furthermore, we demonstrate that the proposed experiment setups can be effectively used diagnose the bottleneck of offline RL agents. Inspired by the evaluation results, novel offline RL algorithm is proposed by a simple modification investigate when a learned dynamics model is helpful to model-free offline RL agents, and introduce an uncertainty braseeddty---edd sample selection method to mitigate the problem of model noises. Code is available at: https / / thub. com/ con/fuyw/RIQQ)
IQL and achieves SOTA performance. Finally, we
Introduction
Offline Reinforcement Learning (RL), also known as batch RL, refers to the problem learning effective control policies from a fixed offline dataset [39]. Due to the wide availability of logged- data and the increasing computing power, offline RL holds the promise for successful real-world applications [40]. For example, offline RL suits well for scenarios where collecting online data is time-consuming, dangerous or unethical, i.e., robotics, self-driving cars and medical treatments [21]. While most off off-policy RL algorithms are applicable in the offline setting, they usually suffer from the extrapolation error [18, 35, 11] due to out-of-distribution (OOD) samples. Different solutions have been proposed to mitigate this problem, Le., adding constraints [18, 50], behavior cloning (BC)
[7, 57], learning dynamics model [55, 28, 3), incorporating uncertainties [51], using ensembles [1], learning pessimistic value functions [36, 5, 26].
Recently, offline RL algorithms have shown to be effective to solve various challenging tasks [46, 42]. However, most of these works mainly focus on designing new algorithms and less attention has been paid to understanding the behaviors of the learned offline RL agents. As result, some basic questions are still poorly understood. For example:
Why does one offline RL agent perform better than other baseline agents in benchmark task, as illustrated in Fig 1?
Does the more performant agent learn better representations more accurate value functions?
Which of the existing offline policy evaluation/mmprooemmnt methods is more effective?
When a learned dynamics model helpful to a model-free offline RL agent?
Given learned dynamics model, how can we use the generated samples more efficiently?
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Baseline
Paradigm
Constraint Type Speed
Disadvantage
TD3+BC [16]
Policy-regulariie
Actor
Fail in difficult tasks Low computation efficiency
CQL [36]
Pessimism
Critic
X
COMBO [54]
Model-based
Critic
Sensitive to model noises Need to tune parameter
IQL [30]
Generalized BC
Both
Table 1: A brief summary of four SOTA offline RL baselines from different paradigms.
halfchestah medium v2
In spite of several potential intuitions, there is a lack of clear explanations for these questions. Such incom- plete understanding impedes progress, as researchers fail to identify the full impact of algorithmic changes without extensive tuning [12]. Given the increasing number of newly proposed offline RL algorithms, we argue that it is time to benchmark current SOTA offline RL algorithms [15] and shed some lights on these fundamental questions. To this end, we empir- ically compare four SOTA offline RL baseline algo- rithms, as shown in Table 1, from different paradigms on the standard D4RL dataset [14].
Normalized score
TD38C
COL
COMBO
0.6
0.8
Time steps
Why does the COMBO agent per- form better than other baselines?
Figure 1:
In this work, we design a comprehensive set of experiments to compare the representations, value functions and policies of offline RL agents in order gain deeper understanding of the behaviors of SOTA offline RI. agents. We identify a surprising discovery that a more performant agent sometimes has worse representations and inaccurate value functions. This motivates us to take closer look at the learned policies. Our empirical results show that a performant offline RL agent is usually able to select better sub-optimal actions while avoiding bad ones. Furthermore, we demonstrate that the proposed experiment setups can be used to evaluate the effectiveness of existing policy wvaluation/improvement methods. As case study, we introduce a variant of IQL [30] by relaxing the in-sample constraint for the policy improvement step, which achieves better performance. Moreover, we investigate when a learned dynamics model can help a model-free offline RL agent, and we propose an uncertainty-base sample selection method to mitigate the problem of model noises. Our contributions are as follows:
We conduct extensive experiments, focusing on representations, value functions and policies, to compare different SOTA offline RI. agents and explain some fundamental questions.
We show the effectiveness of the proposed experiment setups in diagnosing the bottleneck of offline RL agents with new variant of the IQL algorithm that achieves SOTA performance.
We investigate when a learned dynamics model helps model-free offline RL agents, and we introduce uncertainty-baat based sample selection method that is more robust to model noises.
2 Background
A Markov Decision Process (MDP) M = AS, [44] is specified by a state space S, an action space A, transition kernel p S x A A(S), a reward function R S x A R, and a discount factor E [0, 1). The goal is to find a policy (aas)) : S + (AA), which maps from state to distribution over actions, that maximizes the expected cumulative discounted reward J(T) = 1> 2o0 7're]. The performance of the policy can be defined by the value functions (=(ssa) = E, 15 >rr(|5p = s, ag = a] and (s) := E, E 7 rr = s), where E.[-]] is the expected result when following the policy m
In deep RI., Q-function is parameterized with a neural network Q7 (s, a). Following prior works [20, 34, 32, 38], we denote the penultimate layer of the neural network as the learned representation. d-dimensional mapping ((s, a): : SxA + dd. Such that the Q-function is linear in the representation Q7 a) = T Φ(s, a), where 9 E Rd a vector of weights. If the state space S and action space A are finite, then the representation corresponds to a feature matrix &b e TR [xx whose rows are the vector (ss,, a;) I for the i-th state-action pair (st, a.).

---

critic network
0
value function
0. dhs.
@ a)
a
0 M.s a)
((s
463.0)
0p
probing target
Figure 2: An illustration of the representation probing experiment.
Inductive Bias
Target
Probing Function
reward next state s' inverse action
fe((s, a)) ((((s,
Transition Dynamics
a = fo (c()),,
optimal action a* optimal (*
a (b((s))
Optimal Policy
(s, = ()(
optimal V (*)
V*(s) fo(i(s))
Table 2: Different probing targets.
In this work, we study the offline RL setting [40], where we aim to learn policy (als) purely from a fixed offline dataset = {(s,, ag, s/,, r.)), generated from behavior policy æ (a]ss. Most of recent offline RL algorithms build on the Approximate Dynamics Programming (ADP) method [4] that learn the Q7 (s, a) by minimizing the temporal difference error:
LTD(D, 8) (aans''> [tr + max (7(5', Q7 as)
(1)
where (s, a) is the target network. A major challenge in offline RL is the issue of distributional shift between the learned policy (als) and the behavior policy na (a]s). Specifically, the OOD actions a' can produce erroneously over-estimated target values for (7('', (s', a') in Eq 1. Therefore, many existing offline RL algorithms are motivated to constrain the learned policy to stay close to the behavior policy [50, 16), or penalize large over-estimated Q-values [36, 11]. We provide more extensive backgrounds in Appendix A.
Representation evaluation experiments
Inspired by the huge success achieved by representation learning in (un)supervised learning [6, 25, 25,45), it is natural to ask does this question, we first leverage the representation probing technique [2. 23, 37] to evaluate the learned representations of each baseline agent. Then we use some latest introduced representation metrics [34, 33, 38, 41] as proxies to evaluate the quality of the learned representations.
performant offline RL agent learn better representations? To answer
3.1 Representation probing experiment
For state-action pair (s, a), we use the output of the penultimate layer of the critic network as the critic representation ø(s, a) E Rd as shown Fig 2. We later train another linear model (-) to predict a probing target, such as reward r, by linear regression ğ(s, a) = (o(ss, a)) = 0] o(s, s, a). We evaluate the actor embedding (s)s IRd in similar way. To validate whether the learned representations contain meaningful inductive biases [48, 53), we select two categories of probing targets (Table 2). We first use the next state, reward, and action to detect if the representations learned any semanties about the transition dynamics. We then directly use the optimal action and optimal value functions to check if the representations learned any information about the optimal policy.

---

reword probing
nod state prodinng
inwars action prddegg Ies
135
13
6
100-
655
080
X00P
x33
11P
M55
SEAC
0/8
400
010
000
Tike Stggo
Tix Stgo
Tea Stose
Figure 3: Representation probing experiment result on the halfcheetah-medduee environment. We
label the most performant agent with (+) mark. Curves are averaged over 5 seeds.
reward
next state inverse action optimal action
optimal Q'
optimal V
# Env
O
5
4
Table 3: Number of environments that the most performant agent has the least probing loss.
In the experiment, an online TD3 agent [17] is used to approximate the optimal policy π². We use five checkpoints of the online agent (with different levels of performance) to collect 100K transitions as the probing dataset Pprobe. For each probing target, we use 5-fold cross-validation on D Drrobe to train linear regression model with Mean Squared Error (MSE) loss. The result of the probing experiment on saffcheeeth-meddunne are summarized in Appendix C.I. We also list the number of environments where the most performant offline RL agent has the least probing loss in Table 3.
environment is shown in Fig 3. Results other environments
We can observe that the most performant agent usually has worse probing results except for the optimal action experiment. These results indicate that the transition satamics-based information is that important for an offline RL agent to perform well in the selected benchmark tasks. Further, many baseline agents suffer from large MSE losses the optimal value functions (" a) and v*(s) experiments, which highlights the difficulty to learn accurate value functions in such offline setting due to the limited data coverage and additional policy/value constraints. In addition, as long as the actor representation (bss) preserves the ability to learn good actions (low optimal action probing loss), then the offline RL agent holds the potential to achieve good performance.
3.2 Representation metric experiment
We further utilize some recently proposed metrics [34, 33] as proxies evaluate the quality of the learned representations. We start with the following definitions and more details are in Appendix A.2. Definition (Feature dot-product). Feature dot-product Φ(x, a) (is',
, a') is the dot- dot-product of two critic representations [34], where s' ~ P(-1s, a) and a' ~ <+(-5") is the next state and next action.
I 1
Definition 2 (Effective rank). Effective rank sranks
-minn {k=
2 21 of
Di
feature matrix + |ISS-AA approximates the rank of &b [33), where (o:($)} are the singular values of tb in the decreasing order (01 ... 0a 2 0) and 5 is a threshold parameter, i.e., 0.01.
Definition (Effective dimension). Effective dimension ( = N maxi=1,..... || P 11 of a feature matrix &b E I xd measures the sparsity of the column space of &b [38], where N [8| |A] and Po is the orthogonal projector onto the column space of
For the feature dot-product metric, we also compute the cosine similarity [[o(s.))]l (s).aa)) to decouple the effect the representation norm. To compute the effective rank, we first compute the critic
o(s.a) d(e(a")

---

(CA.) dot-prodact
6(4.9) cosine sim lariyy
(4%.o) dffective rank
dis,0) 0) effaciee dimension
107
12050
105
5d3bc
150
(ombe(-)
107
8100
kg
100
100
0000
18
100
2000
Time Stepss
Steps
Time Strpps
Staps
Figure 4: The best performing COMBO agent learns low-quality representations with large norms, in which ((s,a) is very close to áis', a'). Moreover, feature space collapses significantly.
representation 2 ((s;, a.) for each sample in the probing dataset D probe. Then approximate the covariance matrix of & by C($) Peaubell
and use SVD on C( to compute the singular values ( [27]. Unlike the original implementation of the effective dimension which used a fixed threshold approximate the rank of & we instead use the effective rank sranks (+)). More details are discussed in Appendix C.2.
Fig 4 shows the experiment results on the haffcheetah-mediums environment. Results on other environments are summarized in Appendix C.2. We also add the result for the online TD3 agent for reference. We can observe that the most performant COMBO agent has most severe feuture co-adaptation [34] (large feature dot-product) and representation collapse [33] (small effective rank) problems, which indicate it learned pretty "low quality" critic representations. In terms of these metries alone, some baselines agents such as TD3+BC and IQI. seem to learn similar or better critic representations than the near-optimal online agent. These results are consistent with our findings in the previous probing experiments, which indicate that a performant offline RL agent sometimes does not necessarily need high quality critic representations.
Observation performant offline RL agent sometimes learn low-quality representations.
Value ranking experiments
As described in the last section, performant offline RL agent can learn low- low-quality representations. We therefore turn our focus to the value functions to investigate that does performant offline RL agent learn more accurate Q-functions? Since the pessimism-basee offline RL algorithms adopt an additional penalty to learn lower-bounded Q-functions [36, 54], it could be inappropriate to measure the accuracy of Q-functions using metries like MSE. Therefore, we design the following value ranking experiments that focus the ability of Q-functions to rank actions. The motivation is simple - a "good" Q-function in offline RL can have large MSE loss, but it should be good at ranking actions, such that Q² (s, a.)) (T a.) if (s(ss,))))))))))) a,) > (* where Q" is the optimal Q-function.
In the experiment, we first use different behavior policies (four baseline agents and a near-optimal online agent) to interact with the environment to collect validation set Dvalue = {81, ... 8x }. At each state 8; we use each baseline agent sample m different actions, Lee, x(-[s;) + where is Gaussian noise. Therefore, we have M 4m different actions A; = {a.1. for each state. Then we use baseline agents to rank the M actions at state 8;, i.e., R (s:) = sort(Q"s (si, a.1), Q²= (si, (;M)) )) where Q" is the learned Q-function of the j-th baseline agent. We use two metrics to evaluate the accuracy of the learned Q-functions: (1) Spearman's 's rank corre- lation coefficient (Rank IC) and (2) TopN accuracy. We approximate the optimal rank information using the online agent. Here, the larger rank IC/topN accuracy is, the more accurate is the (s((saa) at ranking actions. More details are described in Appendix D.
aam)
Experiment results are shown in Table and Table 5, where we report the mean result and standard deviation across 5 random seeds. The most performant baseline agents in each environment are colored brown. We can observe that many best-performing agents have near-zero rank IC and lower TopN accuracy results. This indicates that the learned Q-functions are less capable of distinguishing good actions from the bad ones. Interestingly, such inaccurate Q-functions do not prevent the most performant agent to select sub-optimal actions to achieve good performances. Further, we can observe
5

---

TD3+BC
CQL
COMBO
IQL
hulfcheetah med-v2
0.13 (0.01)
0.01 (0.01)
0.01 (0.02)
(..0
balfcheetah-med- meepr----22-2-22222
0.05 (0.01)
0.02 (0.01)
0.02 (0.01)
0.02 (0.02)
calfchectal-med-oxpo
0.07 (0.04)
0.08 (0.04)
(0.03
0.03 (0.02)
bopper-med-ve
0.07 (0.03)
0.02 (0.01)
0.05 (0.02)
0.11 (0.03)
hopper rep
(0.05)
(0.03)
(0.04)
(0.02)
supper-medeexp-ee
0.09 (0.07)
0.11 (0.03)
0.16 (0.03)
0.04 (0.05)
walkerZd med-
0.02 (0.03)
0.01 (0.01)
.0.01 (0.01)
0.08 (0.03)
walker2d med-rep-v2
0.10
-0.04 (0.03)
-0.06 (0.05)
0.10 (0.06)
walker2d walker2d-mod-exp-22
0.10 (0.04)
0.09 (0.01)
0.05 (0.10)
0.15 (0.04)
Table 4: Rank IC measures the ability of Q-function rank different actions
TD3+BC
CQL
COMBO
Topl Ace
Top3 Acc
Topl Acc
Top3 Acc
Topl Acc
Tops Acc
Top1 Ace
Top3 Ace
halfahectah-medd-2
8.72
27.60
1.81
10.40
3.84
15.61
8.11
25.91
halfchectal-mnc-re-222
6.70
22.11
2,86
13.53
3.67
15.43
5.92
20.20
balicheetab-medeee exe-----222-
12.19
5.84
hopper-med-v2
13.20
32.08
1.52
6.92
4.67
14.48
14,90
36.21
hopper-med-rop-.
9.70
27.39
4.51
15.05
8.33
25.63
11.41
30.76
hupper med exp-v2
12.43
30.57
2.77
10.41
7.11
21.19
12.98
31,62
walker2d med v2
7.10
23.56
1.64
9.99
1.77
10.36
8.32
26.48
filker2d--me-rep---
8,93
28.07
2.51
11.13
3.35
13.47
8.24
26.19
walker266-mad-ox--22
6.50
23.38
1.10
8.72
1.93
13.47
9,70
29.86
Table 5: TopN accuracy measures the ability of Q-function select the best action.
that the TD3+BC agent and IQL agent usually learn more accurate value functions but achieves worse performance. This suggests that the policy evaluation method TD3+BC and IQL might be more effective but the policy improvement method is limited. We provide more discussions in section 6.
Observation more performant offline RL sometimes learn less accurate Q-functions.
What does a performant policy look like?
In previous sections, we show that a performant offline RL agent sometimes learn relative low-quality representations and inaccurate value functions. In this section, we try to directly compare the learned policies in order to get glimpse of "what does performant offline RL policy look like". To this end, we first design a policy ranking experiment to measure how well does performant offline RL policy work in practice. Then check how often do the SOTA offline RI. agents take OOD actions.
5.1 Policy ranking experiment
In this experiment, we first use behavior policy to collect 30K transitions to create a test dataset D policy At each state s;, action, respectively. Again, we use the online agent to approximate Q* (s, a) to rank the selected four actions. We use following two metrics evaluate the policy n, of the j-th baseline agent: (1) Average percentage of policy Ty that ranks the first/last (with largest/smallest Q" value) across states. (2) Average MSE of the selected action w.r.t. the optimal action. More details are in Appendix E.
four baseline agents and a behavior cloning agent select
Experiment results are shown in Table and Table 7. We can observe that the learned policy of COMBO [54] agent is quite extreme, which both selects the most optimal and worst actions at the same time. In simple environments such as halfcheetah and hopper, such behavior helps to attain higher performance. However, in the more complex walker environment, the higher percentage of bad actions would lead to early termination and thus achieving lower scores. Moreover, as we can observe in Table 7 that the most performant offline RL agent usually have similar larger MSE loss w.r.t. the optimal action a*. This indicates that a performant policy is good at selecting better actions in different states,
though these sampled actions still sub-optimal (with high MSE).
Observation 3. A performant offline RL policy needs to strike balance at selecting good actions while avoiding bad ones. Even though the selected actions are usually still sub-optimal.

---

TD3+BC
CQL
COMBO
IQL
BC
Pu
Pu
Par
Pu
P.
Pu
Pu
balfehectah-med-v
21.05
13.02
11.18
16.33
41.10
32.86
13.82
11.96
12.85
25.82
halfehhotal-mao--e---222
21.89
16.72
14.45
17.40
28.10
27.09
19.42
20.16
16.15
18.63
halfcheetah med-eepee med-exp-v2
20.59
23.55
19.65
16.07
26.57
27.50
19.17
14,89
14,02
17.98
hopper med-v2
16.84
19.98
17.23
17.63
34.10
31.29
21.20
16.35
10.63
14.74
hopper-med-ree--1
12.76
19.52
23.25
18.68
35.01
32.28
16.13
16.40
12.86
13.12
hopper med exp-v2
15.23
20.22
15.98
20.55
38.46
24.74
15.44
21.18
14.89
13.32
mallker2d-meddee
21.79
19.81
19.68
19.92
23.60
20.98
22.61
18.08
12.32
21.21
wallor2d-med-rep-1
20.99
16.49
15.9 95.95
20.80 80
26.77
32.55
23.05
15.24
13.24
14.92
vallkrr2d-meddexpp-22
25.58
15.64
12.66
20.70
26.37
31.41
23.74
13.09
11.65
19.16
Table 6: Percentage (%) of each agent that selects optimal action (P.) and worst action (Pw).
TDS+BC
CQL
COMBO
IQL
BC
halfcheetahs med v2
8.31 (0.05)
8.41 (0.04)
8.27 (0.06)
(0.03)
8.45 (0.04)
haffcheetai-medd-e--22
8.36 (0.03)
8.57 (0.03)
8.64 (0:08)
8,40 (0.04)
8.35 (0.04)
balfehosttal-mocc-xp----
6.96 (0.07)
7.03 (0.08)
7.16 (0.16)
6.85 (0.07)
7,11 (0.05)
hopper-med-
2.66 (0.01)
(0.04)
(0.02)
2.61 (0.01)
2.71 (0.01)
hopper-med-repe
2.45 (0.10)
2.65 (0.10)
2.67 (0.05)
2.38 (0.09)
2.56 (0.08)
sopper-med-xp-e
2.22 (0.04)
2.29 (0.06)
2.20 (0.07)
2,20 (0.03)
2.20 (0.04)
walker2d med
5.39 (0.02)
5.55 (0.03)
5.50 (0.02)
5.38 (0.02)
5.48 (0.02)
walker2d-med--eeee
rep-v2
6.10 (0.07)
6.61 (0.06)
6.66 14)
6.14 (0.05)
6.34 (0.06)
walker2d- medd2ddd-edd-xpp
4.65 (0.07)
4.83 (0.08)
5.32 (1.06)
4.61 (0.07)
4.73 (0.06)
Table 7: MSE loss w.r.t. the optimal action.
5.2
OOD action experiment
Since existing offline RL algorithms usually attempt minimize the negative effects caused by OOD samples. Therefore, we try to measure how often would the baseline agents take OOD actions. In the experiment, we utilize a learned dynamics model to predict whether a state-action pair (s, a) is OOD or not. The learned dynamics models is probabilistic ensemble [8] as in COMBO [54], where each model (s ss a) A N(Ho, (s,a), a), Eo, a)) outputs a Gaussian distribution with diagonal covariance parameterized by 0,. We train the probabilistic ensemble using the D4RL, dataset. We use the following metric to estimate how likely a state-action pair (s, a) OOD: the uncertainty [55]
estimated by the probabilistic ensemble a (s, a) max;=1,... .N (s, N Σÿ= 12 a)ll2-
In particular, we first compute (s, a) on the original D4RL offline dataset, and then compute the (s, a) on dataset collected by the baseline agent. More details are described in Appendix E.2. We report the median of the estimated uncertainty in Table 8. As we can observe that the three model-free offiin RL baselines usually learn conservative policies which mostly take high confidence actions with low (s,, a). On the other hand, the model-based baseline COMBO learns policies that sometimes take more risky actions with higher a (s,, a). This confirms that taking risky actions in offline RL is a double-edged sword, which sometimes helps to learn from the OOD samples to avoid being too conservative and sometimes incurs the extrapolation error.
Observation 4. Taking OOD actions in offline RL could be double-edged sword that sometimes helps to avoid being over-conservative but sometimes incurs the extrapolation error.
Oflinee Data
TDD+BC
CQL
COMBO
halfcheelahh med v2
10.09
7.96 (0.06)
8.11 (0.15)
10.25 (0.71)
7.96 (0.04)
malfcheetah-med-rre
23.96
15.78 (0.66)
17:51 (0.88)
21.54
16.82 (0.34)
baffehastal-nns--xxp-22
11.03
10.74 (0.86)
(1.54)
23.30 (12.62)
8.50 (0.16)
hopper-med hupper med rep v2
(0.02)
2,00
(0.08)
(0.03)
5.13
4.48 (1.27)
3.76 (0.34)
3.90 (0. 84)
3.59 (0.28)
hopper-med-exp--
1.46
1.29 (0.05)
1,27 (0.02)
1,52 (0.11)
1.32 (0.08)
walker2d med--y2
14.01
9.14 (0.22)
8.08 (0.19)
8.81 (0.99)
10.73 (0.49)
walker2d med rep-v2
27.59
14.21 (1.35)
13.03 (1.55)
14.80 (2.04)
16.09 (0.96)
walker2d-med-eee mede--------22222222
12.87
8.86
8.17 (0.06)
75.93 (149.39)
9:74 (1.37)
Table 8: The median of the estimated uncertainty (s, a) in different tasks.
7

---

Baseline
Critic loss function
Actor loss functions
TD3+BC [16]
LTp(D, Lr 0) (CooL aCo LTD(P sq 0) (Po
Merpa CBc LEAc (D, 8, o)
CQL [36]
COMBO (54]
Laac yg.
IQL
Lga(D,
LAwa(D;
Table 9: Different policy evaluationn/mmprovemenn objectives.
Case study: relaxed in-sample Q-learning (RIQL)
In this section, we conduct a case study to show that the proposed experiments can be used to compare the effectiveness of existing policy evaluation/mmprovveennt methods. We first briefly recap the four baseline algorithms as summarized in Table 9, Here we consider a parameterized Q-function Q7 (s, a), policy T (a) s) and loss weight parameters a, that we abuse in different baselines.
Policy evaluation. TD3+BC [16] uses the default fitted Q-evaluation as in Eq 1. The only difference that the max operator over next action a' is replaced by the policy (s'):
LTD 0) = "(s,,,,)")= (rr + ( TT(s')) Q7 a))²
(2)
CQL [36] further adds conservative penalty to push down large Q-vauues for OOD actions:
(D, = EaND (ogg) Σ exp(Q7
B(a,,) - (g)
(3)
COMBO [54] extends CQL by using an augmented dataset D 1g D D model for policy evaluation, where D Moddel is generated using the learned dynamics model. Unlike other three baselines, IQL [30] uses expectile regression (ER) for policy evaluation purely from in-sample data:
LER(D,0) 0) = (' -DD (r((rss +s07(s' a²) Q7
(4)
where Lī(u) |T - 100-1(u 1uu < 0)(u² and E (0, is hyperparameter.
Policy improvement. TD3+BC adds a simple behavior cloning loss CBc(Ø) = B a to
the original TD3 [17] actor loss DT (Ø) EE,- (Q] To [s))) CQL and COMBO both use
the default SAC [22] actor loss function (ő) = E)) log(TT (Ta(aas)))))
Q7 (s, na (a s))] On the other hand, IQL uses the advantage-weightedd regression (AWR) [43] to imitate high-quality
in-sample actions (a) = [s A exp((Qg(s.a) Vv( log TH (a(s))) where (s) is
a value function that only depends on state s.
Given these different policy waauuation/improvemee show which one more effective practice. Here, we try to show that the previously introduced experiment setups can be helpful tools to answer such questions. Recall the results in the Table 4 and Table 5, we can observe that IQL [30] usually achieves higher rank IC and larger topN accuracy, which suggests that the expectile regression based policy evaluation method in IQL is more capable to learn accurate value functions. However, IQL only achieves the best performance in one benchmark task. In addition, Table 8 indicates that IQL learns conservative policy that usually takes high confidence in-sample data. These results suggest that the AWR-based
methods, there is a lack of clear comparison to
hilllyy-mpoooemett method is effective to avoid taking OOD actions but sometimes it is ever-conservative which limits the performance. To validate this assumption, we introduce a simple variant of IQL, called Relaxed In-sample Q-Learning (RIQL), that replaces the original AWR- actor loss with:
LRIQL(0) LSAc(Ø) (D (T TT) LsAc(Ø) (E,,,)) To (als)).
(5)
In short, we add a KI--divergence constraint to the original SAC actor loss, and we drop the E(s.a)= [log up
o [a]s)] term which is independent of actor parameter d. The motivation is two- fold: (1) Unlike CQL and COMBO, the learned Qfffucctonss in IQL are less discriminating w.r.t. OOD samples. Thus, we need extra policy constraints avoid taking too many OOD actions. (2) We want to use a less conservative actor loss to enable the policy learning from OOD actions. In addition, Eq is similar to the actor loss in [11]. The difference is that we keep the entropy term which helps to prevent the learned policy from collapsing to single point. Experiment results of RIQL is shown in Table 10 and more details are described in Appendix F. We can observe that RIQL outperforms IQL. in all benchmark tasks, and it achieves higher total evaluation scores than other baseline agents. This result also validates our previous assumption that AWR-baser policy improvement method the bottleneck for IQL in some tasks which makes it over-conservative.
8

---

TD3-BC
CQL
COMBO
IQL
RIQL
halfcheetah med-v2
48.89
46.85 (0.22)
54.61 (1.48)
47.43 (0.07)
55.93 (0.27)
hopper-med-22
(0.19(1999)
61.18 (1.16)
92.20 (5.04)
64.64 (3.27)
91.58 (4.23)
walker valker2d-med-x2
84.37 (0.55)
81.21 (0.40)
81.93 (0.66)
80,28 (2.00)
80.85
halfcheetab med- rep--
45.20
44,95 (0.45)
52.18 (0.43)
44.04 (0.71)
52,39 (0.45)
hopper med rep v2
(7.26)
(4.20)
(1.91)
91.94 (14.94)
(7.46)
walker2d-med med-rep--22
(1.34
71.68 (1.70)
62.05 (11.46)
73.53 (7.28)
81.46 (4.72)
hal halfcheetah med exp-v2
89.29 (5.07)
90.89 (0.44)
61.53 (9.76)
88,92 (1.34)
92.91 (1.18)
sopper-med-expe walker2d-med-coe mx-x--xxp-2
96.53 (4.97)
105.19 (2.96)
77.57 (16.57)
91.57 (27.03)
102.47 (5.49)
110.16 (0.18)
104.61 (6.39)
84,98 (42.58)
107 08 (5.28)
108,34 (0.58)
Total
676.82
700.85
663.58
690.04
759,05
Table 10: RIQL achieves better performance than IQL in all benchmark tasks.
7 sicertainty-baseed sample selection for model-based offline RL
We also try to investigate when does learned dynamics model is helpful to a model-free offline RL agent? In particular, we reuse the learned dynamics model in Section 5.2 to generate fake samples to train model-free agent. For example, MIQL/MTD3+BC, which learns from the augmented dataset 'D x7 = DD U DDmddel as in MOPO [55]. We find that such naive MBRL agent (a combination of MOPÖ and model-free offline RL agents) usually performs worse than its model-free counterpart (Table 11) even though they are the same algorithm except for the training data. This result indicates that it is nontrivial to combine learned dynamics model with model-free offline RL agents due to the model noises. COMBO [54] addresses this problem by adding a conservative penalty on the model-generated samples. Here, we introduce an Uncertainty- baased Sample Selection (USS) method to filter fake samples with large model noises.
introduce an MBRL variant of IQL/TD3+BC, named
adopt the model uncertainty (s,, a) defined in Section 5.2 as a proxy to approximate model noise. In addition, we maintain a dynamic uncertainty threshold 8 as follows 6g f <-no Satch (s, a) + (1 - -7)5, where Baacc (s, is the q-th quantile of the uncertainties in the sampled fake trajectories. We only add -ooeel-generated samples with an uncertainty lower than & to the model buffer which is later used train the agent. More details described in Appendix G. From Table 11, we can observe that the proposed USS trick usually helps to improve the performance of the MIQL/MTD3+BC agent. Moreover, USS is inferior IQL/TD3+BC in some tasks which shows that it is still challenging to completely solve the model noise problem.
Since it is hard to measure model noises accurately,
IQL
MIQL
MIQL-USS
TD3+BC
MTD3+BC
MTD3+BC-USS
balfcchottab-mee--22
47.43 (0.07)
(7.69)
53.85 (0.41)
48.89 (0.15)
(1.61
52.41 (0.39)
seeper--ned-22
64.64 (3.27)
8.89 (8.32)
80.79 (15.21)
60.19 (1.99)
46.07 (7.05)
67.74 (4.95)
walker2tl-mee med-x2
80,28 (2.00)
(9.59)
77.21 (4.21)
84.37 (0.55)
85.33 (0.95)
84.80 (1.09)
aaffcheetah-med-re---g
44.04 (0.71)
44.70
48.49 (0.30)
45,20 (0.30)
47,80 (0.70)
47.05 (0:41)
boppor-med-rep-12
91.94
(3.188
87.63 (6.90)
64.96 (7.26)
37.78 (8.71)
(8.93)
walker2d-med meddd--ee-eeee rep------
73.53 (7.28)
59.41 (30.75)
85.83 (7.59)
77,24 (1.31)
85.02 (2.26)
87.84 (2.57)
halfcheetah med exp-v2
88,92 (1.34)
32.47 (4.81)
82.08 (4.54)
89.29 (5.07)
0.79 (1.36)
(2.77)
seoper--ed-exp-v22
91.57 (27.03)
10.41 (8.66)
75.88 (41.26)
96.53 (4.97)
74.65 (20.27)
96.81 (4.12)
walker2d med v2
107.68 (5.28)
87.96 (9.56)
108.25 (3.27)
110.16 (0.18)
110.37 (0.29)
110.38 (0.41)
Total
690.04
334.16
699.98
676.82
488.99
702.29
Table 11: USS able to effectively mitigate the problem of model noises.
8
Conclusion
Besides the rapid development of novel offline RL algorithms, the behaviors of these offline RL agents have not been well studied. In this work, we take a closer look SOTA offline RL agents. Specifically, we introduce a series of experiments to compare the learned representations, value functions and policies of four baseline agents. Surprisingly, we find that a performant agent sometimes has relatively low quality representations and inaccurate value functions. We also show that the proposed experiment setups can be used to compare the effectiveness of different policy evaluation/improvemenn methods by introducing relaxed version of IQL that achieves SOTA performance. Lastly, we investigate when a learned dynamics model can help model-free offline RI. agents, and we introduce an uncertainty- based sample selection method to mitigate the problem of model noises.

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pages 104-114, PMLR, 2020.
[2] Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Mare- Alee-Alee Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. Advances in Neural Information Processing Systems, 32, 2019.
[3] Arthur Argenson and Gabriel Dulac-Arold. Model-based offline planning. arXiv preprint arXiv:2008. 05556, 2020.
[4] Dimitri Bertsekas. Dynamic programming and optimal control: Volume 1, volume 1. Athena scientific, 2012.
[5] Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.067 00779999,9 2020.
[6] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint (rxii:2003304227, 2020.
[7] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang. Yanqiu Wu, Qing Deng, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. arXiv preprint N.Ki:19900.21799 2019.
[8] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforce- ment learning in handful of trials using probabilistic dynamics models. arXiv preprini urXiv::8003 12 14, 2018.
[9] Huiqi Deng, Qihan Ren, Xu Chen, Hao Zhang, Jie Ren, and Quanshi Zhang. Discovering and explaining the representation bottleneck of dnns. arXiv preprint arXiv:2111.00 2021.
[10] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021.
[11] Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Informtion Processing Systems, 34, 2021.
[12] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Conference on Machine Learning, pages 3061-3071. PMLR, 2020.
[13] Roy Frostig, Matthew James Johnson, and Chris Leary. Compiling machine learning programs via high-level tracing. Systems for Machine Learning, pages 23-24, 2018.
[14] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXhv:2004.072.99 2020.
[15) Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint 191iv:1910.01770 708, 2019.
[16] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[17] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1582-1591, 2018.
[18] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. 2018.
[19] Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline through ensembles, and why their independence matters. 2021.
10

---

[20] Dibya Ghosh and Mare G Bellemare. Representations for stable off- off-po reinforcement learning. In International Conference on Machine Learning, pages 3556-3565. PMLR, 2020.
[21] Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. arXiv e-prints, pages arXiv-2006, 2020.
[22] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: /8/2.09922 2018.
[23] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 100..01000000670, 2018.
[24] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. 2019.
[25] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- laiiunla representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[26] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[27] Li Jing. Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09345 2021.
(28) Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.0595/ 2020.
[29] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.0599) 2020.
[30] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leamnin. In International Conference on Learning Representations, 2022.
[31] Uya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08055 2021.
[32] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. parameterization inhibits data-efficient deep reinforcement learning. arXiv:2010. 14498, 2020.
Implicit under- arXiv preprint
[33] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. parameterization inhibits data-efficient deep reinforcement learning. arXiv:2010. /4498, 2020.
Implicit under- arXiv preprint
[34] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint 2xXiv:2//2.04/ 16, 2021.
[35] Aviral Kumar, Justin Fu, George Tucker, and Sergey Off-policy deep reinforcement learn- ing without exploration. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 1906.00999 2019.
[36] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04773 2020.
[37] Iro Laina, Yuki M Asano, and Andrea Vedaldi. Measuring the interpretability of unsupervised representations via quantized reversed probing. International Conference on Learning Representations, 2022.

---

[38] Charline Le Lan, Stephen Tu, Adam Oberman, Rishabh Agarwal, and Mare G Bellemare. On the generalization of representations reinforcement learning. arXiv preprint arX1v:2203.00543, 2022.
[39] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[40] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint vrXiv:2005.06443 2020.
[41] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In Deep RI. Workshop NeurlPS 2021, 2021.
[42] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[43] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weigghtd regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
[44] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning,
pages 8748-8763. PMLR, 2021.
[46] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pages 1 1154-1168. PMLR, 2021.
[47] Julian Schritwwieser, Thomas Hubert, Amol Mandhane. Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 34, 2021.
[48] Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline in low-rank mdps. arXiv preprint arX1v:2110.04652, 2021.
[49] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg. Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
[50] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11382 2019.
[51] Yue WU, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2 40, 2021.
[52] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. 2021.
[53] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. In International Conference on Machine Learning, pages 11784-11794.
PMLR, 2021.
[54] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
[55] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.
12

---

[56] Chi Zhang, Sanmukh Kuppannagari, and Prasanna Viktor. Brac+: Improved behavior regular- ized actor critic for offline reinforcement learning. In Asian Conference on Machine Learning. pages 204-219. PMLR, 2021.
[57] Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011. 13885, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] (Appendix H)
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "fIb-HvuKPKP", "reviewer": "Reviewer_LgJQ", "summary": "The paper provides a comprehensive analysis of the state-of-the-art offline reinforcement learning algorithms. In particular, the paper evaluates the critics and policies from offline RL algorithms using several metrics, including representation probing, when the learned representations are used to predict various quantities, effective rank, action ranking, etc. The authors observe a mismatch between the quality of critics' predictions and the performance of the policy for some methods. Based on this insight, the authors propose a modification of IQL that achieves significant improvements over the original version.", "questions": "* Does the analysis help to improve the other methods as well?\n* Does the analysis extends to other D4RL datasets? In particular, how have you trying running the metrics on antmaze tasks?", "limitations": "The paper can be improved by considering a wider variety of datasets from D4RL and applying the same analysis to enhance the other methods.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The paper is well written and easy to follow.\n* I believe that the analysis of offline reinforcement learning methods is very valuable to the community and has received only limited attention so far. \n* The analysis covers a variety of metrics for evaluating the critic and policy separately.\n* The metrics introduced in the paper can help tune the components of offline reinforcement learning methods separately. The analysis has been applied in practice to the improvement of IQL.\n\n# Weaknesses\n* The paper focuses only on a subset of tasks considered in CQL and IQL.\n* The paper focuses only on improving IQL. \n\nTo sum up, I believe this paper is relevant to the offline reinforcement learning community, and the pros outweigh the cons. Therefore, I recommend this paper for acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YbkzwXX7Yab", "reviewer": "Reviewer_LmNq", "summary": "Introduces a set of experiments to evaluate & diagnose the bottleneck of offline RL algorithms along three axes:\n(1) Representations\n- Representation probing: Use the second last layer of the critic network to predict the next state, reward, and action via linear regression. Similarly, use the actor embedding to predict the optimal action, value function, and Q-function (which is approximated by a trained TD3 policy).\n- Representation metric: Feature dot-product and effective rank of critic representations.\n\n(2) Value functions\n- Rank actions using the learned Q(s, a)\n\nObservation: TD3BC and IQL learn more accurate value functions (policy evaluation might be more effective), but achieve worse performance (policy improvement might be limited). More generally,  an offline RL algorithm can be higher-performing, but learn poor representations and value functions.\n\n(3) Policies\n- Policy ranking experiment: Average MSE of the selected action vs. optimal action.\n- How often does the policy take OOD actions?\n\nObservations:\n- COMBO selects most optimal & worst actions.\n- Performant policy is good at selecting better actions, even if the actions are sub-optimal.\n\nThe paper also introduces a new offline RL algorithm called RIQL (Relaxed In-Sample Q-learning), based on a simple modification of IQL.\n- The method is motivated by empirical, heuristic-based observations: that AWR-based policy improvement is effective to avoid taking OOD actions, but is sometimes over-conservative. It add extra policy constraints and use a less conservative actor loss to enable learning from OOD actions.\n- Note that there is no theoretical justification or provable guarantees for the method.\n\nThe paper also investigates when a learned dynamics model helps model-free offline RL. Introduces an uncertainty-based sample selection method that is more robust to model noises.\n", "questions": "1. Is it possible to add a more different task that is known to be more challenging than the standard locomotion tasks (halfcheetah, hopper, walker2d)?\n\n2. Description of the policy ranking metric in L89/L637 is unclear: “Average percentage of policy pi_j that ranks the first/last (with largest/smallest Q* value) across states.” Did you instead mean: “Percentage of states where the Q*(pi_j(s), s) is largest/smallest out of the baseline and BC agents”?\n\n\n--Minor typos that did not affect this review in any way--\n\nL35: “When does a learned dynamics model is helpful to a model-free offline RL agent” -> “When is a learned dynamics model helpful to a model-free offline RL agent?”\n\nL162: “rank the M action” -> “rank the M actions” ?\n\nL228: “IQL uses expectile regression (ER) to for policy evaluation” -> “IQL uses expectile regression (ER) for policy evaluation”\n\nThere are many duplicate citations, e.g.:\n- [18] and [19]\n- [33] and [34]\n- [38] and [39]\n- [43] and [44]\n- [54] and [55]\n", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper presents thoughtful experiments analyzing different offline RL algorithms. It is an interesting result that performant offline RL algorithms often exhibit poor representations and inaccurate value functions. (However, I wonder if this is due to the environments/tasks, which are all similar, state-based locomotion tasks.) Overall, I think the empirical experiments & analyses are useful for better understanding offline RL algorithms on simple classic control tasks, although it is unclear if these analyses still hold for more challenging tasks.\n\nWeaknesses:\n1. My main concern is that all evaluations are done on the toy classic control locomotion tasks in the D4RL benchmark, which are limited due to the fact that simple filtered behavioral cloning outperforms SOTA offline RL methods on D4RL; and therefore, good performance on these simple D4RL tasks does not necessarily translate to good offline RL performance. The paper's contribution would be greatly improved if the authors add a more challenging and different task other than toy locomotion, such as D4RL AntMaze navigation, manipulation tasks, or image-based tasks such as Atari.\n\n2. The proposed method (RIQL) performs similarly as other baseline methods on the toy locomotion tasks (Table 10), which shows that RIQL is a reasonable offline RL algorithm to use. On the other hand, RIQL is a heuristic-based modification of existing offline RL objectives, and there is no theoretical justification (e.g. provable guarantees about policy improvement) for the proposed method RIQL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WLVUmOINP0M", "reviewer": "Reviewer_HrJL", "summary": "The paper surveys recent offline RL algorithms and seeks to analyze different factors contributing to their performance. Novel evaluation protocols are designed to analyze their representation and behavior. Based on this analysis, the authors propose a well-motivated modification to IQL which achieves strong results on several D4RL datasets.", "questions": "- It would be nice to disentangle the different contributions of model-based RL algorithms, could the analysis of model-based algorithms be better served by using MOPO, a more minimal algorithm?\n- On line 157, the assumption is that an online learned “optimal” Q* would appropriately assign higher Q values to better true performing actions. Would this be flawed due to approximation? We may expect the Q functions’ values on OOD actions to be inaccurate. Could there be further issues from using a TD3-learnt policies which is different from the SAC policy used in many algorithms and used to generate the data?\n", "limitations": "The paper provides good insights on the datasets it tests, however:\n\n- Limited evaluation on med/exp datasets, this could be expanded to include more D4RL-suite datasets.\n- Incomplete information on seeds used during evaluation.\n- Possible inaccuracy of using online Q-functions as ground-truth. Tabular settings where precise Q-function can be determined could elucidate this better.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Well-motivated range of analysis on the representation learned by an offline RL algorithm.\nProposed algorithms (RIQL, USS variants) show strong performance and are motivated by earlier findings in the paper.\n- Novel and insightful analysis of how to integrate model-free methods into a model-based framework, and how to address the failures of a naive approach.\n\nWeaknesses:\n- On line 208 and 269, the uncertainty estimate used is the max mean-discrepancy, this measure is unusual as we would use the standard ensemble variance in supervised learning [1].\n- Evaluation is solely on med/exp datasets, a good understanding of the strengths of each algorithm on mixed/random data where we may expect a higher extrapolation gap would be useful.\n\nMinor:\n- TD3BC should TD3+BC\n- Typo on line 182: dose -> does\n- Line 204, 262: the learned probabilistic model and model-based training are more accurately attributed to MOPO rather than COMBO.\nIn all tables and results: total random seeds should be included.\n\n[1] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept