Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination
Authors: Jiafei Lyu
Xiu Li
Zongqing Lu
Venue: NeurIPS 2022 Accept (2022)

Abstract: The learned policy of model-free offline reinforcement learning (RL) methods is often constrained to stay within the support of datasets to avoid possible dangerous out-of-distribution actions or states, making it challenging to handle out-of-support region. Model-based RL methods offer a richer dataset and benefit generalization by generating imaginary trajectories with either trained forward or reverse dynamics model. However, the imagined transitions may be inaccurate, thus downgrading the performance of the underlying offline RL method. In this paper, we propose to augment the offline dataset by using trained bidirectional dynamics models and rollout policies with double check. We introduce conservatism by trusting samples that the forward model and backward model agree on. Our method, confidence-aware bidirectional offline model-based imagination, generates reliable samples and can be combined with any model-free offline RL method. Experimental results on the D4RL benchmarks demonstrate that our method significantly boosts the performance of existing model-free offline RL algorithms and achieves competitive or better scores against baseline methods.

Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination
Jiafei Lyu', Xiu Li"; Zongqing Lu
Tsinghua Shenzhen International Graduate School, Tsinghua University
School of Computer Science, Peking University
1vjf20@mails 1esf200mailss.tssnnnuaaa edu.cn,
1i xi wwwwsiuuu.i... tsi.guuu...iigguuaau. edu. en, zongqing lu@pku edu.
Abstract
The learned policy of model-free offline reinforcement learning (RL) methods often constrained to stay within the support of datasets to avoid possible dangerous out-of-distributtin actions or states, making it challenging to handle out-of-support region. Model- based RL methods offer richer dataset and benefit generalization by generating imaginary trajectories with either trained forward or reverse dy dynamics model. However, the imagined transitions may be inaccurate, thus downgrading the performance of the underlying offline RL method. In this paper, we propose augment the offline dataset by using trained bidirectional dynamics models and rollout policies with double check. We introduce conservatism by trusting samples that the forward model and backward model agree on. Our method, confidence- aware bidirectional offline model-bate -baeed imagination, generates reliable samples and can be combined with any model-free offline RL method. Experimental results on the D4RL benchmarks demonstrate that our method significantly boosts the performance of existing model-fre free offline RL algorithms and achieves competitive or better scores against baseline methods.
1 Introduction
Offline reinforcement learning (offline RL.), also known batch RL [34], aims learning from static dataset that was previously gathered by an unknown behavioral policy. Offline RL is deemed be promising [16, 14] as online learning requires the agent to continuously interact with the environment, which however may be costly, time-consuming, or even dangerous. The progress in offline RL will undoubtedly scale RL methods being widely applied real-world applications, considering the impressive success in computer vision or natural language processing by adopting large-scale offline datasets [9, 6].
Prior off-policy online RL methods [17, 21, 48] are known to fail on fixed offline datasets, even on expert demonstrations [14], due to extrapolation errors [16]. In the offline setting, the agent can overgeneralize from the static dataset, resulting in arbitrarily wrong estimates upon out-of-distribution (OOD) state-action pairs and dangerous action execution. To address this issue, recent model-free offline RL algorithms compel the learned policy to stay close to the behavioral policy [16, 31, 72], or incorporate some penalties into the critic [50, 32, 29]. However, such approaches often suffer from loss of generalization capability [73, 69], since they purposely avoid OOD states or actions.
Model-based offline RL methods, instead, enrich the logged dataset by generating synthetic samples with the trained forward or reverse (backward) dynamics model [26, 75, 69, 74]. These methods
Cooresponding Authors
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

benefit from better generalization thanks to richer transition samples. Intuitively, the performance of the agent is largely confined by the quality of the model-generrted data, i... learning on bad states or actions will negatively affect the policy via backpropagation. Unfortunately, there is no guarantee that reliable transitions can be generated by the trained forward or backward dynamics model [2].
In this paper, we aim to generate reliable transitions for offline RL via a double check mechanism. The intuition behind this lies in the fact that humans often do double check when they are uncertain and need to be cautious. Besides the forward model, we train the backward model generate simulated rollouts backward and use one to check whether the synthetic samples the other generated are credi- ble. To be specific, we train bidirectional dynamics models along with bidirectional rollout policies. In- stead of injecting pessimism into value estimation, we introduce conservatism into transition, i.e., only samples that the forward model and reverse model agree on are trusted.
Forward imagination
Reverse imagination
$2
51
S1
Figure 1: confidence-awe aware bidirectional offline model- based imagination. To ensure that the forward imagination $2 from $1 is valid and reasonable, one needs to "look back" to check whether the imagined previous state 81 based on &g simi- lar to 81. We trust &2 if the deviation between $1 and 81 is small.
Illustration of the basic idea of
We use Figure/ 1 to further illustrate our insight, where we take forward transition generation as an example, of which the process is identical to the reverse setting- Starting from 81, the forward model predicts next state 82. However, it is hard for the agent to decide whether 82 is trustworthy. One natural solution, which follows human's way of reasoning [22], is backtracking where it comes from, i.e., looking backward to trace previous state $1. and check whether the imagined state &1 based on 82 is different from the true state $1. We are confident to $2 if &1 is similar to $1 and vice versa.
To this end, we propose Confidence-AWee which is a simple yet effective data augmentation method. CABI generally guarantees the reliability of the generated samples via the double check mechanism, and can be combined with any off-the- model-free offline RL. methods, e.g., BCQ [16] and TD3 BC [15], to enjoy better generalization in conservative manner. Extensive experimental results on the D4RL benchmarks [14] show that
Bidirectional Offline Model-Based Imagination (CABI),
CABI significantly boosts the performance the base model-free offline RL methods, and achieves competitive or even better scores against recent model-free and model-based offline RL methods.
2 Related Work
In this paper, we consider offline reinforcement learning [34, 38], which defines the task of learning from static dataset that was collected by an unknown behavior policy. Applications of offline RL include robotics [45, 57, 54], healthcare [19, 70], recommendation system [59, 64], etc.
Model-free offline RL. Since it is risky to execute out-of support actions, existing offline RL algo- rithms are often designed constrain the policy search within the support of the static offline dataset. They realize it via importance sampling [53, 63, 39, 49, 18], explicit or implicit policy constraints [16, 31, 72, 35, 40, 78], learning conservative crities [32, 44, 30, 43, 29,42) 42], and quantifying estima- tion uncertainty [73, 76, 10]. Recently, sequential modeling is also explored in the offline RI. setting [7, 24,4,,4 47]. Despite these advances, model-free offline RL methods suffer from loss of generalization beyond the dataset [69], and CABI is proposed to mitigate it.
out-of-support
Model-based offfine RL. Model-based offline RL algorithms benefit from better generalization as the static dataset is extended by the synthetic samples generated from the trained forward [56, 13, 1] or reverse dynamics model [69]. These methods heavily rely on uncertainty quantification [52, 75, 26. 11], compelling the policy towards the behavior policy [65, 46], representation learning [36, 54], and penalizing Q-values [74]. However, it is hard to judge whether the transitions generated by the trained dynamics model are reliable, and poor imagined samples will negatively affect the performance of the agent. Recently, [75, 41] explore how inaccurate rollouts that are well penalized can still be useful for model- training. However, they work only for basel offline RL methods.
2

---

11
52
50
33
$2
S-1
$1
5-2
S-3
So
$ S-1
$_2
Forward state
Backward state
Sampled state
Admit
Reject
Figure 2:
Illustration the double check mechanism in bidirectional modeling. We use the trained opposite dynamics model to check whether the synthetic transitions generated by the current dynamics model are reliable. We are confident about the transitions that can traced back to the starting state with small errors (green check) and reject those with large disagreements (red cross).
There are also some studies that focus on trajectory pruning [26, 77], while they involve uncertainty measurement. CABI, instead, ensures reliable imaginations by conducting double check with the forward and backward models, which fully exploits the advantages of bidirectional modeling.
Model-based online RL. Model-based online RL methods achieve superior sample efficiency [61, 25, 4, 23] by learning a dynamics model of the environment and planning with the model [62, 68, 71]. Learning a backward dynamics model that produces traces towards the aimed state is also widely explored [12, 37, 20, 33]. Among them, most similar to our work [33], which leverages bidirectional model rollouts for reduced compounding error in the online setting. However, the main differences are: (1) CABI is proposed
augment the fixed dataset instead of performing policy optimization in a mrodel-based way; (2) CABI interpolates double check mechanism for reliable imaginations; (3) Model predictive control (MPC) [5] is not involved in CABI.
Preliminaries
We study RL under Markov Decision Process (MDP) specified by a tuple (S, Po. r, 7), where S the state space, AA is the action space, Po denotes initial state distribution, p(s'(s, ss a) the stochastic transition dynamics, r(s, a) S x A + R is the reward function, and 7 E [0, 1) is the discount factor. The policy T(a s) : S x over actions. The goal of RL is to obtain policy π such that the expected discounted cumulative rewards can be maximized, max, J (T) := EE ss=ppo.ay<z((1s)) ( (=,((1s....)) [E:= ('r(s,, In online RL, the agent learns from the experience collected from the interactions with the environment. However, the offline setting, both interaction and exploration are infeasible, and the agent can only get access to the logged static dataset Denv {(s, s') s')}. which was gathered in advance by the unknown behavior policy. Since the fixed dataset Donv typically a subset of full space x A, the generalization beyond the raw dataset becomes challenging. Model-based RL mitigates this issue by learning a dynamics model p(s') s, a) and reward function f(s, a). and generating synthetic transitions to augment the dataset. However, there is guarantee that the generated samples are reliable (see Section 4. 1), and we focus on addressing this issue in this paper.
R_ is a mapping from states to a probability distribution
Confidence-Aware Bidirectional Offline Model-Based Imagination
In this section, we first use toy example to illustrate the necessity of training bidirectional models with the double check mechanism. Then, we give the detailed framework of our method, Confidence- Aware Bidircctional Offline Model-Based Imagination (CABI).

---

4.1 You Need to Double Check Your State
Many model-free offline RL algorithms suffer from poor generalization as they are trained on fixed dataset with limited samples. Model-based methods extend the logged dataset by generating synthetic transitions from the trained dynamics model. Despite such an advantage, they lack mechanism for checking the reliability of the generated samples. If the model inaccurate, poor transition samples that lie in the out-of-support region of the dataset can be generated, which may downgrade the performance offline RL algorithms.
Human beings tend to conduct double check when they are uncertain about the outcome, e.g.. clinical medicine research [60], autonomous driving [67], etc. Inspired by this nature, we propose to train bidi- rectional dynamics models and admit the samples where the forward model and backward model have few disagreements instead of roughly trusting all generated samples. In this way, we introduce conservatism into the transition itself instead of the critic or the actor networks. We give the illustration double check mechanism in Figure 2.
largt:
D
0
danger!
=
-3
rodius 0.5
We argue that either forward dynamics model or backward dynam- ics model unreliable, and bidirectional modeling in conjunction with the double check mechanism is critical for trustworthy sample generation. We verify this by designing toy task, 2-dimensional environment with continuous state space and action space, namely RiskWorld, as shown in Figure
D2
(-1
Figure 3: Risk
The central point of the square
(-1.5,155) 1.5). Each episode, the agent randomly starts at the region D1 == {(x: y) 1.5) 1.5) (y+ (15) 11,1 0,y y < 0) and takes actions
region in Risk World is (0, 0), and the state space gives D = [-1.5,
0.5, 0.5]. There is danger zone D2 = {(x, v) + + 0.52²) and the done flag would turn into true if the agent steps into this region, along with a reward of -3. The agent will receive reward of +-1 if lies in D3 := ((),v)((= 1.5) + (y 1.5) < 0.8² and it locates at D\ Dz).
1.5, 1.5).
We run random policy in shows the state distribution (blue cross) the dataset, where there are no transitions D2 (red circle area) as the episode terminates if the agent steps into D2. To compare different ways of imagination, we train a forward model, backward model, and a bidirectional model with the double check mechanism on this dataset. The training epoch is set to be 100, and the rollout horizon is set to be 3 for all of them. The detailed experimental setup is available in Appendix A.
for 104 timesteps to collect offline dataset. Figure 4(a)
13
90
05
05
-1.
-10
-14
-.7
-15
-35
-14
-15 -10
-05 20
65
10 15
-15 -10 -55 02
05
14
15
32-10-22-00 221---0-33333 08
-15 -10 -55
20
65 10
(a) Random policy
(b) Forward
(c) Backward
(d) Bidirectional
Figure 4: Visualizations of the offline dataset collected by a random policy (a), and synthetic transitions generated by a forward model (b), backward model (c), a bidirectional model with the double check mechanism (d), based on the offline dataset in (a).
We use the trained forward model, reverse model, and bidirectional model to generate 10 transition samples, and plot the state distributions of their generated samples respectively. As shown in Figure 4(b), the forward model generates many samples the danger zone D: (red circle area). Figure 4(c) reveals that the reverse model generates a lot of illegal samples that lie out of the state space D, and also many transitions that lie the dangerous area D2. These show evidence that both the forward and backward dynamics model fail to output reliable transitions. However, we observe in Figure 4(d) that bidirectional modeling with the double check mechanism successfully produces reliable and conservative synthetic samples, i... out-of-support dangerous samples are not included, because

---

Algorithm CABI (Model Training)
Input: Dataset Deuv: iterations N1, N2, learning rate agg Ceg, 000, Ow
1: Randomly initialize forward model parameters t and reverse model parameters &
2: for i== 1 to N1 do
3: Compute Clvd and Chrd via (1) and (2)
4: Update model parameters: v/ 4- 1/ arg V Cfwd & x as V bbw
5: end for
6: Randomly initialize forward and backward rollout policy parameters 0, w
7: for i= to N2 do
8: Compute Livae and Livae via (3) and (4)
9: Update forward and reverse rollout policy: 0 + 0 - or VeLivaor w t w aw V Sivae
end for
the forward model and backward model have large disagreements at those states. We hence argue that bidirectional modeling with double check is necessary for reliable data generation in offline RL.
4.2 Bidirectional Models Learning in CABI
Bidirectional dynamics models training. Our bidirectional modeling models transition probability and reward function simultancously. i.e., the forward model Py (s', a) and the reverse model (s, ris', a) parameterized by & and respectively. The forward model ( (1',,,,,, a) represents the probability of the next state and corresponding reward given the current state and action, and the backward model P r|s a) outputs the probability of the current state and reward using the next state and action as input. We assume that the predicted reward function f(s, a) only depends on the current state and action a. then the unified model can be decomposed as p. (s', r|s. a) p(s'|s, a) ()((r,s, a) and (s. r|s', a) p(s|s', (is)ss,a)pin aa)(p()ssssss a). We denote the loss functions for the forward model and backward model as Liwe and Lowdd respectively, and optimize them by maximizing the log-likelihood via (1) and (2), where Donv is the raw static dataset.
Co Bls.a,no"" [- lo (6', r)), Cowd (ss,,,;
[- log (s, r(,rls'.
Following prior work [75, 26]. we train an ensemble of bootstrapped probabilistic dynamics models, which has been widely demonstrated to be effective in model-based RI. [8, 23]. Each model in the ensemble is parameterized by multi-layer neural network, which outputs a Gaussian distribution N(,, E). Detailed hyperparameter setup for dynamics models training deferred to Appendix C.
Bidirectional rollout policies training. We additionally train bidirectional generative models, which serve as rollout policies, and are used to generate actions augment the static dataset. We model the rollout policy with a conditional variational autoencoder (CVAE) [27, 58, 16], which offers diverse actions while staying within the span of the dataset. CVAE is made up of an encoder E that outputs the latent variable z under the Gaussian distribution, and decoder D that maps to the desired space. We denote the forward rollout policy as Gf (s) parameterized by 9 = {E1, 11 where & the parameter of the encoder E Efwd (s, a) and 1/ is the parameter of the decoder DÉwd (s, 2). The forward rollout policy is then trained by maximizing its variational lower bound, which equivalent to minimizing the following loss:
Livae (s.a.r,8 (aarr,,e))DDDv.eee (s.a)
IE
[(a- ffwd (s, (s) + DKL (Foudd (s, (s,a)lI((O [)].
ffwd
(3)
where DKL(-II: (-ll) denotes the KL-divergence, and I is an identity matrix. The first term of RHS of (3) represents the reconstruction loss where we want the decoded action to approximate the real action. Then for action generation, (O I), and then pass it with the current state into the decoder Dfwdd (s, 2) to output the action.
first sample latent vector from the multivariate Gaussian distribution
Similarly, the backward rollout policy Gbwa (s') parameterized by a contains an encoder Ebwed (s', a) and decoder Dowd (s', 2), w (52, 22). The loss function of the backward rollout policy gives:
Lbvae (w) (sar)s))) ) DDDen 20 wow plwd (5',a)
[(a- Dbwd (s', x)) + DKI. (Etre (s', (a))))(()))))) (4)
5

---

Algorithm 2 CABI (Data Generation)
Input: Offline dataset Denv horizon H, iteration N
1: Initialize model replay buffer Dmodel
2: for i 1 to N do
3: Sample state 9; and next state St+1 from Deny
4: for = to H - do
5:
Obtain forward rollout {st+5, Oe4si S from St-j by drawing samples from the forward model Py and forward rollout policy G
6:
Generate backward state ši+s from S141+j. and evaluate the deviation of ši+j from st+f
7:
Get backward rollout {st-j. at- Tt-j, - from St+1-j by drawing samples from the backward model P and backward rollout policy G.wdd
8:
Generate forward state 8141-j from 8t-j. Evaluate the deviation of $t+1-j from S1+1-j
9: Add selected imaginations into D moddll
10: end for
end for
12: Get composite dataset Diotal Denv UD
13: Get the final policy Tout with any model-free offline RL algorithm based on the dataset Drotal
We then draw from the Gaussian distribution (0, I), and draw the action from the action decoder Dowd (s', 2) with the next state s' and latent variable 2 as input.
We present the detailed procedure for the model training part of CABI in Algorithm 1,
4.3 Conservative Data Augmentation with CABI
After the bidirectional dynamics models and bidirectional rollout policies well trained, we utilize them to generate imaginary samples. Each time, we sample state 8, from the raw dataset D Deuv to produce imagined forward trajectory ffwd {s++j ar+j, Pt+j; St+ +=)++=))=00 H-1 with the forward dynamics model Py and forward rollout policy Gffdd and sample the next state 8t+1 from Denv to generate imagined reverse trajectory pbwdd {st-ja at-j, Te-ji -
1-1 with the reverse dynamics model Pag and reverse rollout policy Gbwd, For each step in the rollout horizon H, we do double check and reject those badly imagined synthetic transitions.
To be specific, when performing forward imagination from 8, and generating synthetic next state $141, we trace back from $141 with the reverse model, and get the backward state 5g. We evaluate the deviation of 5; from 8t, and trust & if the deviation is small. Similarly, starting from the state st41. we backtrack its previous state š, with the backward dynamics model, and then look forward from 5t to get š141 with the forward dynamics model. We trust &, if the deviation of 6 from 81t1 small.
We keep those trustworthy rollouts and gather them to get the model buffer Dundell We combine the synthetic model buffer D Dmodel with D Dnnv to obtain the final buffer Dootal, i.e., Drotal D U We then can train any model-free offline RL algorithms based on the composite dataset.
One naïve way for implementing double check mechanism is to set a threshold 6, and admit the transition if || 5z l2 I for forward imagination, or - < for backward imagination. However, such method lacks flexibility, and one may need to carefully tune d per dataset based on the strong prior knowledge about the dataset, which impedes the application of double check mechanism. We resort to sorting the transitions in mini-batch by the state deviation from small to large and keep the top 1%% of them that have the smallest deviation. We keep 20% transitions that have the smallest deviation for all of our experiments in Section 5 (empirical study on k is available in Appendix C).
Our method is confidence-aware and conservative as we only admit the transitions that the forward model and backward model agree on, thus excluding those poor transitions from the model buffer Dmodel- The full procedure for the data generation part of CABI is available in Algorithm 2.

---

Table 1: Normalized average comparison of CABI+BCQ against different baselines the Adroit "-v0" tasks, where score represents the performance of random policy and 100 corresponds to expert policy performance. The results averaged over the final 10 evaluations and 5 different random seeds. The highest mean scores are in bold.
Task Name
CABI+BCQ
BCQ
UWAC
BEAR
BC
AWR
CQL
MOPO
COMBO
pen-cloned
54.711.0
44.0
33.1
26.5
56.9
28.0
39.2
-2.1
-2.4
pen-human
7..1
68.9
21.7
-1.0
34.4
12.3
37.5
9.7
27.7
pen-expert
127.6.12.0
114.9
111.9
105.9
85.1
111.0
107.0
-0,6
11.5
door-cloned
0.5+0..2
0.0
0.0
-0.1
-0.1
0.0
0.4
-0.1
0.0
door-human
1.7.1..1
0.0
2.1
-0.3
0.5
0.4
9.9
-0.2
-0.3
door-expert
105
99.0
104.1
103.4
34.9
102.9
101.5
-0.2
4.9
relocate-cloned
-0.2+0..0
-0.3
-0.3
-0.3
-0.1
-022
-0.1
-0.3
-0.1
relmcate-human
0.1:00.1
0.5
0.5
-0.3
0.0
0.0
0.2
-0,3
-0.3
relecate-expert
105.9=110
41.6
105.6
98.6
101.3
91.5
95.0
-0.2
17.2
hammer-cloned
4.3 1.66
0.4
0.4
0.3
0.8
0.4
2.1
0.2
0.4
hammer-human
3.1+2.2
0.5
1.1
0.3
1.5
1.2
4.4
0.2
0.2
hammer-expert
128.9:1099
107.2
110.6
127.3
125.6
39.0
86.7
0.3
0.3
Total Score
607.0
476.7
490.8
460.3
440.8
386.5
483.8
6,4
59.1
Experiments
In this section, combine CABI with off-the-shelf model-free offline RL algorithms and conduct extensive experiments on the D4RL benchmarks [14]. In Section 5.1, we combine CABI with BCQ [16], and evaluate it the challenging Adroit dataset show the effectiveness of conservative data augmentation with CABI. We present detailed ablation study in Section 5.2, where we aim to answer the following questions: (1) Is the double check mechanism critical component for CABI? (2) How does CABI compare with the forward/reverse imagination? (3) How does CABI compare against other augmentation methods, e.g., random selection? Furthermore, we incorporate CABI with another recent model-free offline RL method, TD3 BC [15], and evaluate it on the MuJoCo datasets, show the generality and advantages of CABI. We additionally combine CABI with IQL [28] and evaluate the performance of CABI+IQL on both Adroit tasks and MuJoCo tasks. Due to the space limit, the results are deferred Appendix G.
5.1 Performance on Challenging Adroit Dataset
We demonstrate the benefits of CABI by combining it with BCQ and evaluating it on the challenging Adroit dataset [55]. Adroit dataset involves controlling 24-DoF simulated robotic hand that aims hammering nail, opening a door, twirling pen, or picking/moving a ball. It contains three types of datasets for each task (human, cloned, and expert), yielding total of 12 datasets. This domain is very challenging for prior methods to learn from because the dataset is made of narrow human demonstrations on a sparse reward, high-dimensional robotic manipulation task.
We summarize the overall results in Table 1, where we compare CABL+BCQ against recent model- free offline RL methods, such UWAC [73], CQL [32], BCQ [16], and model-basse offline RL methods, such as MOPO [75], and COMBO [74]. We run MOReL and COMBO on these datasets with reproduced code. Results of MOPO and UWAC the Adroit domain are acquired by running their official codebases, and the results of the rest baselines are taken directly from [14]. All methods are run over 5 different random seeds and normalized average scores are reported in Table 1. We only report the standard deviation for CABI+BCQ, and the full table is deferred to Appendix H.
As shown, CABI significantly boosts the performance of vanilla BCQ almost all datasets, achieving a total score of 607.0 vs. 476.7 of BCQ. CABI+BCQ also surpasses the baseline model-free and model-base offline RL methods on 7 out of 12 datasets and achieves the highest total score.
worth noting that model-based offline RL methods generally fail on the Adroit tasks, because (1) the dataset distribution is narrow and high-dimensional, making it challenging for the trained forward dynamics model to generate accurate and reliable transitions; (2) the actions in the synthetic transitions are generated by the actor during the training process, thus the error may accumulate if
7

---

Table 2: Normalized average score comparison on the Adroit tasks between CABI+BCQ. forward imaginatio+BBCQ. backward imagination+BCQ and BOMI+BCQ, where captures standard deviation. The results are averaged over the final 10 evaluations and 5 different random seeds. The highest mean scores are in bold.
Task name
BCQ
+Forward
+Backward
+BOMI
+CABI
pen-cloned
44.0
41.2.11.1
36.816.6
43.4.16.1
54.7.1..0
pen-human
68.9
57.8+9.3
60.945.66
49.6+1.4
75.1
pen-expert
114.9
114.415,4
118.5114.7
121.811.2
127.612.00
door-cloned
0.0
0.0+0.00
0.0+0..0
0.0+0.0
0.5+0.22
door-human
0.0
-0.110.11
0.010.11
0.010.1
1.710.1
door-expert
99.0
104.2:10.3
103.710.2
102.511.2
105.3100.5
relocate-cloned
-0.3
-0.3+0.0
-0.3=0.0
-0.2+0.0
-.2
relocate-human
0.5
0.010.0
0.0.0.00
0,010.1
0.1
relocate-expert
41.6
72.9+2.0
76.8+6.8
80.1
105.9+1.0
hammer-cloned
0.4
1.710.1
0.410.1
3.123.8
1.3
0.5
2.0+0.2
2.8+0.5
2.1+0.7
3,1+2.2
hammer-expert
107.2
126.8:11.0
126.911.0
126.811..3
128.910.9
Total score
476.7
520.6
526.5
529.2
607.0
the actor is updated towards wrong direction. CABI, instead, alleviates the underlying issues via adopting the CVAE for action generation and conducting double check on state prediction.
5.2 Ablation Study
Is the double check mechanism critical? To answer this question, exclude the double check mechanism in CABI and admit all generated synthetic samples from bidirectional models, which gives rise Bidirectional Offline Model-based Imagination (BOMI). We evaluate CABI+BCQ and BOMI+BCQ on the Adroit tasks with identical parameter configuration over 5 different random seeds and show the average normalized scores in Table It can be seen that BOMI brings some performance improvement on most of the tasks via data augmentation with bidirectional models and rollout policies. However, the generated data may be unreliable (we observe performance drop in pen-cloned, pen-human), which impedes the benefits of bidirectional data augmentation. Such concern can be alleviated with the aid of the double check mechanism. As illustrated in Table CABI+BCQ outperforms BOMI+BCQ on most tasks and incurs much better total score.
CABI against forward/backward imagination. We incorporate BCQ with the pure forward model, backward model, and CABI, and conduct extensive experiments on the Adroit tasks over 5 different random seeds. The forward model and reverse model are trained with the same configuration as CABI. The results are summarized in Table 2. It can be seen that either the forward or reverse model results in limited improvement, which is consistent with the results of BOMI As previously discussed, the forward model and reverse model may generate unreliable transitions. We see such evidence as the performance of BCQ falls on some of the tasks (e.g., pen-cloned) if trained on mere forward or reverse imagination. BCQ+CABI, instead, outperforms BCQ+Forward and BCQ+Backward on all tasks. Hence, we conclude that CABI guarantees trustworthy transitions for training, and brings improvement on almost all of the tasks.
CABI against other augmentation methods. We further compare CABI against three data aug- mentation methods: (1) CABI-random where we replace the CVAE with the random policy as the rollout policy in CABI: (2) R-20 where we randomly select 20% synthetic transitions for bidirectional imagination; (3) EV-20 where we select 20% transitions with the smallest ensemble variance for bidirectional imagination, i.c., we evaluate the variance of the output the ensemble of the forward and backward dynamics models and reject those with large variance. We use BCQ as the base algorithm and run experiments on four Adroit tasks for these augmentation methods with identical parameter setup as CABI (e.g., real data ratio). The results in Table 4 show that CABI performs consistently better than these methods. Since the data augmentation process of CABI is isolated from the policy optimization, we cannot leverage random rollout policy because the generated actions of random policy may possibly lie out of the span of the dataset, which can negatively affect the performance of the agent. Hence, CVAE is critical to ensure a safe and conservative data
8

---

Table 3: Normalized average comparison of CABI+TDS BC baseline methods the D4RL MuJoCo "-v0" dataset, where score O corresponds to random policy performance and 100 corresponds to expert policy performance. The results averaged over the final 10 evaluations and 5 different random seeds. The highest mean scores are in bold.
Task Name
CABI+TD3 BC
TD3 BC
UWAC
MOPO
BCQ
BC
CQL
FisherBRC
halfcheetah--aadoon
15.1
10.2
2.3
35.4
2.2
2.0
21.7
32.2
hopper-random
11.9+0.11
11.0
9.8
11.7
10.6
9.5
10.7
11.4
walker2d-random
6.4.1..55
1.4
3.8
13.6
4.9
1.2
2.7
0.6
halfchectah- medium-replay
44.4=t.22
43.3
38.9
53.1
38.2
34.7
41.9
43.3
nupper-medium-repe
31.3:10.7
31.4
18.0
67.5
33.1
19.7
28.6
35.6
walker2d-medium--eeee
29.4:11.3
25.2
8.4
39.0
15.0
8.3
15.8
42.6
halfchectah-
45.1=2..1
42.8
37.4
42.3
40.7
36.6
37.2
41.3
hopper-mediun
100.4:0.33
99.5
30.3
28.0
54,5
30.0
44.2
99.4
walker2d-meding
82.0+0.44
79.7
17.4
17.8
53.1
11.4
57.5
79.5
halfcheetah- medium exxpett
105.0:0.2
97.9
40.6
63.3
64.7
67.6
27.1
96.1
hopper-medium-erne
112.7+0.0
112.2
95.4
23.7
110.9
89.6
111.4
90.6
108.4:11.3
101.1
14.8
44.6
57.5
12.0
68.1
halfchectah-expert
107.6+0.99
105.7
104.0
89.9
105.2
82.4
106.8
hopper-expert
112.410.1
112.2
109.1
107.0
111.5
111.2
112.3
walker2d-ce 2x-exeert
1.8
105.7
88.4
102.3
56.0
103.8
79.9
Total Score
1020.7
979.3
618.6
784.6
595.3
764.3
974.6
augmentation. Mcanwhile, relying on the ensemble variance for data selection is not trustworthy as the models the ensemble are trained on the identical data and may incur wrong predictions but small variance.
Table 4: Normalized average score comparison on four Adroit tasks. The results are averaged over the final 10 evaluations and 5 different random seeds. CABI-random denotes the rollout policy in CABI is random policy. R-20 denotes Randomly keep 20% transitions, EV-20 denotes keep 20% samples that have the smallest Ensemble Variance.
Task Name
BCQ
+CABI
+R-20
+EV-20
+CABI-random
pen-cloned
44.0
54.7+2..0
41.213.0
40.4#2.0
37.6:17.8
pen-expert
114.9
127.6=2.0
112.6#5.6
118.8:22.5
106.3+3.7
hammer-cloned
0.4
4.3+:..6
0,940.6
0.410.1
0.3+0.0
hammer-expert
107.2
128.910.9
104.2424.6
125.5
103.8=1.5
5.3 Broad Results on MuJoCo Dataset
To show the generality of our method, we integrate CABI with another recent model-free offline RL method, TD3_BC [15], and conduct experiments on 15 MuJoCo datasets. We widely compare CAB++DD3 BC against other recent model-free offline RL methods, such FisherBRC [29]. UWAC [73], CQL [32], and model-based batch RL method, MOPO [75]. run CABI+TD3 BC over different random seeds. We also UWAC using the official codebase on the MuJoCo datasets over different random seeds. The results of TD3 BC, BC, CQL, FisherBRC are taken directly from [15], and the results of other baseline methods are taken from [73].
The experimental results in Table 3 reveal that our approach exceeds all baseline methods on 10 of 15 datasets, and is the strongest in terms of the total score. On almost all of the tasks, we observe performance improvement with CABI over the base TD3_BC algorithm. Unfortunately, with the existence of behavioral cloning term, the performance improvement upon TD3_ is limited. Still, the experimental results Table and show that CABI is powerful data augmentation method and can boost the performance of the base model-free offline RL methods.

---

Conclusion and Limitations
In this paper, we follow human nature and propose do double check during synthetic transition generation to ensure that the imagined samples are conservative and accurate. We admit samples that the forward model and reverse model agree on. Our method, CABI, involves training bidirectional dynamics models and rollout policies and can be combined with any off-the- model-free offline RL algorithms. Extensive experiments on the D4RL benchmarks show that our method significantly boosts the performance of the base model-free offline RL method, and can achieve competitive or better performance against recent baseline methods. For future work, it is interesting to evaluate CABI in the online setting and investigate whether it can benefit model-based online RL as well.
The major limitation of our proposed method lies in the computation cost as we train bidirectional dynamics models and rollout policies. However, since CABI is isolated from policy optimization, we can enhance the dataset beforehand.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Science and Technology Innovation 2030-Key Project under Grant 2021ZD020140 404, in part by the NSF China under Grant 61 61872009, The authors would like to thank the anonymous reviewers for their valuable comments and advice.
References
[1] A. Argenson and G. Dulac-Arold. Model-Based Offline Planning. ArXiv, abs/2008. 055566 2021.
[2] K. Asadi, E. Cater, D. K. Misra, and M. L. Littman. Towards Simple Approach to Multi-step Model-based Reinforcement Learning. ArXiv, 00011811.00128, 2018.
[3] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. ArXiv, abs/1606.0154) 2016.
J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample- Efficiet Reinforcement Learning with Stochastic Ensemble Value Expansion. In Advances Neural Information Prucessing Systems (NeurIPS), 2018.
[5] E. F. Camacho and C. B. Alba. Model Predictive Control. Springer Science Business Media, 2013.
[6] C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. T. Koehn, and T. Robinson. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. ArXiv, abs/1312.3005, 2014.
[7] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision Transformer: Reinforeement Learning via Sequenee Modeling. ArXiv, abs/2106.01345, 2021.
[8] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep Reinforcement Learning in Handful of Trials using Probabilistic Dynamics Models. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[9] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei. ImageNet: A Large-Scake Hicrarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[10] Z. Deng. Z. Fu, L. Wang. Z. Yang, C. Bai, Z. Wang. and J. Jiang. SCORE: Spurious COrrelation REduction for Offline Reinforcement Learning. ArXiv, abs/21 10. 124468 2021.
[11] C. P. Diehl, T. Sievemich, M. Krüger, F. Hoffmann, and T. Bertram. UMBRELLA: Uncertainty- Aware Model-Basee Based Offline Reinforcement Learning Leveraging Planning. ArXiv, abs/2111.11097 2021.
[12] D. Edwards, L. Downs, and J. Davidson. Forward-Backa Backward Reinforcement Learning. ArXiv. abs//800.100m 10227, 2018.
[13] C. Finn and S. Levine. Deep Visual Foresight for Planning Robot Motion. In International Conference on Robotics and Automation (ICRA), 2017.
[14] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for Deep Data-Driven Reinforce- ment Learning. ArXiv, abs/2004.07219, 2020.
2020.
10

---

[15] S. Fujimoto and S. S. Gu. A Minimalist Approach to Offline Reinforcement Learning, In Advances in Neural Information Processing Systems (NeurIPS), 2021.
2021.
[16] S. Fujimoto. D. Meger, and D. Precup. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference on Machine Learning (ICML), 2019.
[17] S. Fujimoto, H. van Hoof, and D. Meger. Addressing Function Approximation Error in Actor- Crrttic Methods. In International Conference on Machine Learning (ICML), 2018.
[18] C. Gelada and M. G. Bellemare. Off Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift. In AAAI Conference on Artificial Intelligence, volume 33, 2019.
[19] O. Gottesman, F. D. Johansson, J. Meier, J. Dent, D. Lec, S. Srinivasan, L. Zhang, Y. Ding, D. Wihl, X. Peng, J. Yao. I. Lage, C. Mosch, L. wei H. Lehman. M. Komorowski, A. A. Faisal, L. A. Celi, D. A. Sontag, and F. Doshi-Veles Settings, ArXiv, abs/ 122.9 2018.
Evaluating Reinforcement Learning Algorithms in Observational Health
[20] A. Goyal, P. Brakel, W. Fedus, S. Singhal. Lillicrap. S. Levine, H. Larochelle, and Bengio. Recall Traces: Backtracking Models for Efficient Reinforcement Learning. In International Conference on Learning Representaatons (ICIR), 2019.
[21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In International Conference on Machine Learning (ICML), 2018.
[22] K. Holyoak and D. Simon. Bidirectional Reasoning in Decision Making by Constraint Satisfaction. Journal of Experimental Psychology: Generul, 128:3-31, 1999.
[23] M. Janner, Fu, M. Zhang, and S. Levine. When to Trust Your Model: Model-Based Policy Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[24] M. Janner, Q Li, and S. Levine. Offline Reinforcement Learning as One Big Sequence Modeling Problem. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[25] L. P. Kaclbling, M. L. Littman, and A. W. Moore. Reinforcement Learning A Survey, Journal of Artificial Intelligence Research, 4:237-283 285, 1996.
[26] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. MOReL:: Model-Based Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[27] D. Kingma and M. Welling. Auto-Encoding Variational Bayes. ArXiv, abs/1312 abs//332..114, 2013.
[28] I. Kostrikov, A. Nair, and S. Levine. Offline Reinforcement Learning with Implicit Q-Learming. In International Conference on Learning Representations. 2022.
[29] Kostrikov, J. Tompson, R. Fergus, and O. Nachum. Offline Reinforcement Learning with Fisher Divergence Critic Regularization. In International Conference on Machine Learning (ICML), 2021.
[30] A. Kumar, R. Agarwal, T. Ma, A. C, Courville, G. Tucker, and S. Levine, DR3: Value-Bass Deep Reinforcement Learning Requires Explicit Regularization. ArXiv, 12.2211
[31] A. Kumar, J. Fu, G. Tucker, and S. Levine. Stabilizing Off-Policy Q-Learming via Bootstrapping Error Reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[32] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[33] H. Lai, J. Shen, W. Zhang, and Y. Yu. Bidirectional Model-based Policy Optimization. In International Conference on Machine Learning (ICML), 2020.
[34] S. Lange, T. Gabel, and M. A. Riedmiller. Batch Reinforcement Learning. Reinforcement Learning, 2012.
[35] R. Laroche and P. Trichelair. Safe Policy Improvement with Baseline Bootstrapping. In International Conference on Machine Learning (ICML), 2019.
[36] B.-J Lee, J. Lee, and K. Kim. Representation Balancing Offline Model-based Reinforcement Learning. In International Conference Learning Representations (ICLR), 2021.
11

---

[37] K. Lee, Y. Seo. S. Lee, H. Lee, and J. Shin. Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning. In International Conference Machine Learning (ICML), 2020.
[38] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives Open Problems. ArXiv, abs/2005.01144 2020.
[39] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Off-Policy Policy Gradient with State Distribution Correction, ArXiv, abs// 904.08473, 2019.
[40] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Provably Good Batch Reinforcement Learning Without Great Exploration. ArXiv, fass/2007.08202. 2020.
[41] C. Lai, P. Ball, J. Parker-Holder, M. A. Osbore, and S. J. Roberts. Revisiting Design Choices Offline Model Based Reinforcement .earning. In International Conference on Learning Representation, 2022.
[42] Lyu, X. Ma, Li, and Z. Lu. Mildly Conservative Q-learnin for Offline Reinforcement Learning. In Thirty-sixth Conference on Neural Information Processing Systems, 2022.
[43] X. Ma, Y. Yang. H. Hu, Liu, J. Yang, C. Zhang, Q Zhao, and Liang. Offline Reinforcement Learning with Value- based Episodic Memory, ArXiv, abs/21 10.09996, 2021.
[44] Y. J. Ma, D. Jayaraman, and O. Bastami. Conservative Offline Distributional Reinforcement Learning. In Advances in Neurai Information Processing Systems (NeurIPS), volume 34, 2021.
[45] A. Mandlekar, F. Ramos, B. Boots, L. Fei-Fei, A. Garg, and D. Fox. IRIS: Implicit Reinforcement without Interaction Scale for Learning Control from Offline Robot Manipulation Data. In International Conference on Robotics and Automation (ICRA), 2020.
[46] T. Matsushima, H. Furuta, Y. Matsuo, O. Nachum, and S. Gu. Deployment-Efficient Reinforcement Learning via Model- Badee Offline Optimization. International Conference on Learning Representations (ICLR), 2021.
[47] L. Meng, M. Wen, Y. Yang, Le, X. Li, W. Zhang, Y. Wen, H. Zhang, J. Wang, and B. Xu. Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks. ArXiv, abs/2112 12.02445, 2021.
[48] R. Munos, T. Stepleton, Harutyunyan, and M. G. Bellemare. Safe and Efficiennt Off-Policy Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[49] O. Nachum, Y. Chow, B. Dai, and Li. DuaIDICE: Bchavior-Agnoste Estimation of Discounted Stationary Distribution Corrections. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[50] O. Nachum, B. Dai, Kostrikov, Y. Chow, L. Li, and Schuurmans. AlgaeDICE: Policy Gradient from Arbitrary Experience. ArXiv, abs/1912.02074, 2019.
[51] A. Nair, M. Dalal, A. Gupta, and Levine. Accelerating Online Reinforcement Learning with Offline Datasets, ArXiv, abs/2006.09 093599 2020.
[52] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and Snoek. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
[53] D. Precup, R. S. Sutton, and S. Dasgupta. Off-Policy Temporal Difference Learning with Function Approximation. In International Conference on Machine Learning (ICML), 2001.
[54] R. Rafailov, T. Yu, A. Rajeswaram, and C. Finn. Offline Reinforcement learning from Images with Latent Space Models. In Learning for Dynamics and Control (L4DC), 2021.
[55] A. Rajeswaran, V. Kumar, A. Gupta, Schulman, E. Todorov, and S. Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. ArXiv, Abs/1709..0088. 2018.
[56] S. Ross and J. A. Bagnell, Agnostic System Identification for Model-Batd -ased Reinforeement Learning, In International Conference on Machine Learning (ICML), 2012.
[57] A. Singh, A. Yu, J. Yang. J. Zhang, A. Kumar, and S. Levine. COG: Connecting New Skills to Past Experience with Offline Reinforcement Learingg ArXiv, A4502010.14500, 2020.
[58] K. Sohn, H. Lee, and X. Yan. Leamingg Structured Output Representation using Deep Conditional Generative Models. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
12

---

[59] A. L. Strehl, J. Langford. L. Li, and S. M. Kakade. Learning from Logged Implicit Exploration Data. In Advances in Neural Information Processing Systems (NeurlPS), 2010.
[60] R. Subramanyam, M. A. Mahmoud, D. Buck, and A. M. Varughese, Infusion Medication Error Reduction by Two-Person Verification: A Quality Improvement Initiative. Pediatrics, 138, 2016.
[61] R. S. Sutton. Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming. In International Conference Machine Learning (ICML), 1990.
[62] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[63] R. S. Sutton, A. R. Mahmood, and M. White. An Emphatic Approach to the Problem of Off- Temporal- Differrnce Learning. Journal of Machine Learning Rexearch, 17:2603-2631, 2016.
[64] A. Swaminathan and T. Joachims. Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization. Mach. Learn. Res., 16:1731-1755, 2015.
[65] P. Swazinna. S. Udluft, and T. Runkler. Overcoming Model Bias for Robust Offline Deep Reinforcement Learning. Engineering Applications of Artificial Intelligence, 104:104366, 2021.
[66] E. Todorov, T. Erez, and Y. Tassa. MuloCo: A Physics Engine for Model-based Control. IEEE/RSI International Conference on Intelligent Robots and Systems, 2012.
[67] D. N. Van, I.. Kuhnert, Thamke, Schlemper, and K.-D. Kuhnert. A Novel Approach for Double- Check of Passable Vegetation Detection in Autonomous Ground Vehicles. In IEEE Conference on Intelligent Transportation Systems, 2012.
[68] T. J. Walsh, Goschin, and M. L.. Littman. Integrating Sample-Based Planning and Model-Based Reinforcement Learning. In AAAI Conference on Artificial Intelligence, 2010.
[69] J. Wang, W. Li, H. Jiang, G. Zhu, S. Li, and C. Zhang. Offline Reinforcement Learning with Reverse Model-based Imagination. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[70] L. Wang, W. Zhang, X. He, and H. Zha. Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
[71] T. Wang and J. Ba. Exploring Model-based Planning with Policy Networks. In International Conference Learning Representations (ICLR), 2020.
[72] Y. Wu, G. Tucker, and O. Nachum. Behavior Regularized Offline Reinforcement Learning. ArXiv, Abs/1911.11361 2019.
[73] Y. Wu, S. Zhai, N. Srivastava, J. M. Susskind, J. Zhang. R. Salakhutdinov, and H. Goh. Uncertainty Weighted Actor-Critic for Offline Reinforcement Leamingg In International Conference on Machine Learning (ICML), 2021.
[74] T. Yu. A. Kumar, Rafailov, A. Rajeswaran, Levine, and Finn. COMBO: Conservative Offline Model-Basee Policy Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[75] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. MOPO: Model-based Offline Policy Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[76] A. Zanette, M. J. Wainwright, and Brunskill. Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
[77] X. Zhan, X. Zhu, and H. Xu. Model-Based Offline Planning with Trajectory Pruning. ArXiv, abs/2105.07351, 2021. 2021.
[78] W. Zhou, S. Bajracharya, and D. Held. PLAS: Latent Action Space for Offline Reinforcement Learning. In Conference on Robot Learning (CoRL., 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report bars (e.g.. with respect the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, you cite the creators? [N/A]
(h) Did you mention the license of the assets? [N/A]
(c) Did you include any
either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're 're
ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Qr0zvj800yg", "reviewer": "Reviewer_ytiJ", "summary": "The paper proposes a method of augmenting an offline dataset for use with model-free offline methods by generating synthetic rollouts that are “double-checked” - i.e., validated by a bidirectional transition model. The paper powerfully demonstrates the principle on a toy environment, and the proposed enhancement leads to a significant performance gain for BCQ on Adroit but a more minor gain for TD3+BC on MuJoCo.\n", "questions": "- Could standard offline model-based algorithms be used with the improved rollouts? Would there be a performance gain by only filtering out inaccurate forward rollouts in an algorithm like MOPO?\n- Are there some states that are consistently discarded due to model inaccuracies? What proportion of states would be rejected with a 1-step rollout and double check?\n- How was the ratio of real data to imagined trajectories tuned?\n", "limitations": "The authors discuss the limitations of their work well. Two points are:\n\n- The authors address the limited gain on the MuJoCo-based environments, but could demonstrate the method is still useful by selecting a different algorithm to augment.\n- [1, 2] show how inaccurate rollouts that are well penalized can still be useful for model-based training, and this point would be good to discuss.\n\n[1] MOPO: Model-based Offline Policy Optimization Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma\n\n[2] Revisiting Design Choices in Offline Model-Based Reinforcement Learning. Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne, Stephen J. Roberts.\n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clearly articulated method for validating the trajectories generated by model rollouts, strong illustrative toy example\n- Enables model rollouts to be integrated with standard offline model-free RL algorithms with strong performance on the Adroit environments\n- Extensive ablations against different data generation procedures and action-augmentation schemes\n\nWeaknesses:\n- Limited gain on the MuJoCo environments which authors attribute to the BC term in TD3+BC, would it be possible to see a similar strong gain with an alternative such as CQL?\n\nMinor:\n- Line 116: perhaps too strong a claim, certain offline model-free algorithms exhibit strong performance (IQL, TD3+BC) without imagined rollouts\n- Line 146: training for a fixed number of epochs could lead to over/underfitting, is there a validation dataset?\n- The number of seeds should be listed in table captions.\n- Evaluation protocol for determining offline performance should be described (e.g. policy taken at the end of training?)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "60H8TNYibCK", "reviewer": "Reviewer_rJ1S", "summary": "This paper presents a new offline model-based RL method that trains a bidirectional model for model rollouts and filters out model-generated states given by the forward model that cannot recover the previous real state from the backward model. By doing so, the method can effectively get rid of inaccurate model-generated states and mitigate the distributional shift problem in offline model-based RL. Through extensive experiments, the method is able to outperform prior offline RL methods on various tasks in D4RL.", "questions": "See cons in the above section.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros:**\n\n1. The method appears to be novel and addresses an important problem in offline RL, i.e. how to incorporate augmented data without introducing much distributional shift.\n\n2. The results of the method on challenging domains such as Adroit are clearly better than the prior methods listed.\n\n3. The paper is clearly written and easy to understand.\n\n**Cons:**\n\n1. The bidirectional models seem to assume that the action is reversible, which I think might not always hold in all MDPs. While the empirical results seem to suggest that incorporating such rollouts double-checked by the bidirectional models is helpful, there are definitely cases where the previous state is not possible to be inferred from the next state and the action, e.g. robotic manipulation situations that are contact-rich. I wonder if the authors can show the methods can also work in those situations (both empirically and theoretically).\n\n2. The empirical evaluations are not that thorough. I believe one important baseline that is missing is S4RL, which also handles data augmentation in offline RL and achieves impressive results in adroit (even better results than the proposed method after skimming through their results table). The authors also didn't compare to MOReL, which is another important model-based offline RL method. The authors should also include evaluations in other domains such as antmaze and kitchen from D4RL.\n\n3. It is a bit strange that the numbers of the baselines in the paper are different from/worse than ones in the original papers, e.g. CQL etc. I think it is important to show the original numbers to ensure fair comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-wxBIP_CqJ", "reviewer": "Reviewer_iDjK", "summary": "The paper proposes a model-based data augmentation method (CABI) for offline RL, which uses the agreement of forward and backward models to assess the validity of transitions. It proceeds as follows:\n1. Train forward and backward dynamics models, and forward and backward rollout policies based on CVAEs\n2. Use models and rollout policies to generate transitions\n3. Apply the backward model to transitions which were generated from the forward model, and vice-versa. Sort the transitions by the deviation of the predicted state from the original state, and keep only the transitions with relatively low deviation.\n\nAfter generating these transitions, any model-free offline RL algorithm can be applied to the augmented dataset. Experiments on the D4RL benchmark demonstrate that the data augmentation is beneficial, improving over the base offline RL algorithm, in almost every case.", "questions": "* “We resort to sorting the transitions in a mini-batch by the state deviation from small to large and keep the top k% of them.” I’m guessing this is a typo and you meant the bottom k%? (Since you want small deviations)\n* What is the exact formula used to compute ensemble variance? This should be included in the paper or appendix.", "limitations": "Yes, they mention that computational cost is a limitation of the work.\nI agree that there is not significant risk of negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The proposed double check is a novel idea, to my knowledge.\n* The paper is clearly written.\n* Experimental evaluation is thorough and demonstrates the performance benefits of CABI across a variety of datasets. The ablation study also shows the benefit of the double check compared to ensemble variance as an uncertainty quantifier.\n\nWeaknesses\n* Hyperparameter tuning: the appendix states that “The best ratio η strongly depends on the dataset and may need to be tuned manually.” One must also tune the rollout horizon.\n* The improvement over the base offline RL algorithm, while consistent across tasks, is not large in most cases. This may be because the generated samples are very similar to the original samples (since only in-support actions are tried).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept