Weighted model estimation for offline model-based reinforcement learning
Authors: Toru Hishinuma
Kei Senda
Venue: NeurIPS 2021 Poster (2021)
Keywords: model-based reinforcement learning

Abstract: This paper discusses model estimation in offline model-based reinforcement learning (MBRL), which is important for subsequent policy improvement using an estimated model. From the viewpoint of covariate shift, a natural idea is model estimation weighted by the ratio of the state-action distributions of offline data and real future data. However, estimating such a natural weight is one of the main challenges for off-policy evaluation, which is not easy to use. As an artificial alternative, this paper considers weighting with the state-action distribution ratio of offline data and simulated future data, which can be estimated relatively easily by standard density ratio estimation techniques for supervised learning. Based on the artificial weight, this paper defines a loss function for offline MBRL and presents an algorithm to optimize it. Weighting with the artificial weight is justified as evaluating an upper bound of the policy evaluation error. Numerical experiments demonstrate the effectiveness of weighting with the artificial weight.

Weighted model estimation for offline model-based reinforcement learning
Toru Hishinuma
Kei Senda Kyoto University
Kyoto University
hishinuma. toru. 43.0kyoto-u.jp
sendadkuaero. kyoto- jp
Abstract
This paper discusses model estimation in offline model-based reinforcement learn ing (MBRL), which is important for subsequent policy improvement using an estimated model. From the viewpoint of covariate shift, natural idea model estimation weighted by the ratio of the state-action distributions of offline data and real future data. However, estimating such a natural weight is one of the main challenges for off-policy evaluation, which is not easy to use. As an artificial alternative, this paper considers weighting with the state-action distribution ratio of offline data and simulated future data, which can be estimated relatively easily by standard density ratio estimation techniques for supervised learning. Based on the artificial weight, this paper defines loss function for offline MBRL. and presents an algorithm to optimize it. Weighting with the artificial weight is justified as evaluating upper bound of the policy evaluation error. Numerical experiments demonstrate the effectiveness of weighting with the artificial weight.
Introduction
Reinforcement learning (RL) framework to learn a policy in an unknown environment [ Model-basee RL (MBRL) is an approach that explicitly estimates transition model and utilizes it to improve policy. Compared to model-free RL, one major advantage of MBRL is the data efficiency 24 which is important for applications where the data collection is expensive, such robotics or healthcare. This paper focuses on offline MBRL, i.e. MBRL that learns a policy from previously collected datasets.
One of the main challenges offline MBRL distribution shift []. The distribution of offline data is different from the distribution of data obtained in the future when applying improved policy. This is situation called covariate shift, where a model estimated using standard supervised learning techniques such as empirical risk minimization (ERM) may not generalize to test data. A major idea in MBRL is to estimate a model using such a standard method and to improve policy using the estimated model in regions where the predictions are accurate 1648), A point to be improved here is model estimation without considering covariate shift. supervised learning under covariate shift, importance- weighted ERM can obtain good predictive performance [9]. If model in offline MBRL can be estimated similar way, then the estimated model will make more accurate predictions, resulting increased policy improvement. Motivated by this, this paper tackles the issue of weighted model estimation considering covariate shift offline MBRL.
From the viewpoint of covariate shift, natural idea is weighting with the ratio of the state-action distributions of offline data and real future data, as the agent uses model to predict real future data. For convenience, this paper calls the ratio "the natural weight." However, estimating the natural weight is one of the main challenges for off- off-policy evaluation [5] [0). Compared to density ratio estimation for supervised learning [Л] the difficulty of off-policy evaluation is the inaccessibility of samples from the distribution real future data.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

As an alternative, this paper considers weighting with the state-action distribution ratio of offline data and "simulated" future data. This paper calls the ratio "the artificial weight," in contrast to the natural weight. Simulated future data can be generated in offline simulation, unlike real future data. Since samples from both distributions are available, the artificial weight can be relatively easily obtained by standard density ratio estimation techniques for supervised learning. Based on the artificial weight, this paper defines a loss function for offline MBRL Section This paper presents an algorithm to optimize it in Section
The question here the validity of weighting with the artificial weight, as it may not natural for covariate shift. This paper justifies it as evaluating an upper bound of the policy evaluation error in Section 4
This paper demonstrates that it works practice in numerical experiments in Section The above is the contribution of this paper.
2 Related Works
Most of offline MBRL researches 7 12 discuss how to learn a policy using a learned model. In contrast, this paper studies how to estimate model considering covariate shift in MBRL, which depends on policy be applied. In principle, these can be combined as an algorithm that repeats between estimating a model and learning a policy, like Algorithm 3
supporanceesamplinge based methods estimate the natural weight, namely, the distribution ratio of offline data and real future data on a trajectory or state(-action) space [53 I0]. The advantage is unbiasedness, while the disadvantage is high variance. Model-based methods, including this paper, evaluate policies utilizing models trained using data. Model-based methods have the opposite advantage and disadvantage of aepporance-sampllng--ooo importance-sampling based and model-based methods [16 In principle, incorporating the idea of this paper as model--bsed component possible.
methods. Doubly robust methods combine
VAML 19 20 defines a state-action-wer wise loss function based on the expected error of Bellman operator and learns model to optimize weighted sum. Similar ideas are found in 21 The main focus is the state-action-we
loss function, while there are few studies about how to weight is. The few studies, e.g. GAMPS (24), use the natural weight. In contrast, this paper simply uses the negative log likelihood loss function for a state-action pair and discusses the artificial weight, which relatively easy to estimate.
This paper is similar to VMBPO 25 that an EM-style algorithm updates a model and a policy using universal loss function. The differences is that this paper considers covariate shift but does not use the idea of RL as inference [26]. Formulation that explicitly incorporates RL as inference is also an interesting topic.
Another approach learning a model robust against covariate shift by reducing the difference between the distributions of offline data and future data on a feature space (22 27 28] Since covariate shift measured in feature space encouraged to decrease, policy improvement increasing covariate shift is also discouraged. That is, such an idea implicitly limits policy improvement. This paper differs in that it is trying to utilize covariate shift.
Preliminaries
This paper discusses a discounted infinite horizon MDP 29
Let be state. Let be an action. Let p(s) be an initial state distribution. Let ((s'|s, a) be a transition probability function. Let r(s, be a reward function. Let y be a discount factor. Let be a policy. The value function is
(s(
Σ Exr(sttaa) at) sa
The expected return is Tpp = Egonp [V]( The state distribution
(p(s) =
( (1-7)
Σ
7 Pr (s' = =,
1
The state-action distribution is (5) (z (s) Note that nip E( (x(0))
I-

---

Let Pa denote the true transition probability, which is unknown to the agent. Let Pa denote the transition probability model parameterized by 0. Let = {(,,. On s J be offline data, which is obtained from P.. Let dB_ be the distribution of the input part of D, i.c. {(5,, an)Jn=1 Let Dz be simulation data generated given (Pa, T) with state initialization with probability 1-7, which is equal to sampling (s, a) from d...- Let co(s,a) Ex |- In Po(s'|s, a)] be the cross entropy between (-Is,, a) and Pa(-Is, a). Let h(s, a) = B. (1s.a) [- In P. (s' [s a)] be the self-entropy of (-)s,)).
4 Loss Function
This paper discusses the theory for the case where the reward function is known. For the unknown reward function, if the prediction of reward model is bounded, the same loss functions can be derived, see Appendix.
4.1 Policy Evaluation
The goal of policy evaluation is estimating 1// when given w. For model-based setting, the goal reduces to finding 9 decreasing 1np. 1 which is bounded as
(B(a,)) [E (P.(s'ls,a) Po( (s))
mpp - 7 5
1-7
BE (s.a)dF5 (11s_()),,) -Po(-ss,))l
v2
<
B Blssa)aag [co(s, a) h)s
B V (isaaaaB (s, (s) a)] E (s, (wf
W B E (ssa))aB [w] ()laa(oossso a)] Aminn
V
(I)
where B= max [v22 (I-7)²
w6 (s, a)
(s.a)
d., (s,a)' and Amin min.,a M(s, a). The first inequality
(s.a)
the Telescoping lemma
21 The second inequality holds from Holder's inequality and ||V S
max, |r(s.a) The third inequality holds from Pinsker's inequality and Jensen's inequality. When
trying optimize the third inequality, ((.a)=a.. [w] (s, ()) a)] difficult to handle, h(s,a)
is unknown. The fourth inequality [ excludes the difficult part from the optimization.
This paper considers decreasing the policy evaluation error by minimizing Equation be minimized is
T What to
L (² Es (w] a)c((s,)) a)] =
P (-w (s, ls In
The empirical average of (=) is
Σ (s
(Sn, an) In (o an),
is loss function weighted with uy (s, a), which is the artificial weight, namely, the distribution ratio of offline data 'D and simulated future data Dy. Thus, weighting with the artificial weight is justified evaluating an upper bound of the policy evaluation error.
Equation (2)
4.2 Policy optimization
The goal of policy optimization finding æ increasing 1/.-- In offline MBRL, the agent utilizes vif for the optimization. A lower bound of n/p is
nr. np.
-mpe n
Z TPP B BV (0)) hmin-
3

---

The first term evaluates the expected return by model Pa. and the second penalizes the policy evaluation error.
Inspired from increasing vpp by maximizing the lower bound, this paper defines a generalization loss function for offline MBRL by
J( .) = + + (= - *min
(3)
where B3' [0, B] is a tunable coefficient to scale the penalty for the policy evaluation error.
The agent optimizes an estimate of Equation (3 where empirical averages of Vp, and L (0) are weighted with the artificial weight. Similarly to the discussion in Section [.1 weighting with the error.
artificial ratio is justified as evaluating the expected return with the penalty for the policy evaluation
Remark. If using the Telescoping lemma with respect to dzp instead of dip,- the policy evaluation error is similarly bounded as
dp (8, a) co(
Inp.
Tip TP B
E B(sa))af
hmin=
(4)
1D (s,a)
where db. (a.a)
is the natural weight. There are two advantages to consider Equation rather
a5 (5,0)
than Equation 4 The first is that the artificial weight can be more easily ontained, described
in Section D The second is that Equation (T) leads to standard MDP optimization problem, as
described Section [.2] while Equation (4) results in more complicated optimization problem
related to both d'p, and dip..
Algorithm
5.1
Policy Evaluation
5.1.1
Full Version
This paper considers gradient- based optimization algorithm. The gradient of Equation 2 is
N
1
Voi=(
Σ
w (sn, an) (Vo In Po(s,, an) + In Po(s', (8,, an 2n) (sn))
}
N
where =s) V In dip, (s), which can be estimated using extension of LSDG 330 described later. The artificial weight, wā. can be estimated using a density ratio estimation method given (D, D5,). The gradient of Equation can be estimated using them.
In practice, the computational efforts to estimate (wā za) are non-negligible. This paper presents a method that iterates between estimating (wā, 2n) and updating locally. To locally update from ((1), this paper local approximate loss function defined by
N
- (0; (10)
1
Σ
wico (Sn, an) {In Pe(s Sn. an) + (0 (()) (In Pocs (5/ an)z (sm))
N
(5) The gradient of Equation with respect can be obtained immediately, as (wJen = ) is fixed. For 0 ((1), Equation 2 and Equation (5) are equal, and their gradients are also equal. That is, for 0= 000 the first-order necessary conditions for optimality are the same. In theory, if assuming that (w > is accurately estimated, the above-mentioned iterative method converges.
importance-weightee ERM is consistent but can be unstable in practice [9]. even when the weight exactly known. Importance-weigge
ERM weighted with wio be more unstable due to the estimation and approximation emrrr of density ratio estimation of w To stabilize the estimation of
this paper replaces witn (#n, an) Equation 55) with [wi (#n, a,)], where a = 0 corresponds FRM, and a corresponds to importance- spportance-weighted ERM weighted with w (8n, a,). That is, is a hyperparameter controlling the trade-off between consistency and stability.

---

Algorithm Weighted model estimation for policy evaluation (full version).
1: Input: D, n,, a, and Paro
2: for 0, 1, .2....d.
3: Generate pio using simulation given (Paco, x).
4: Estimate W using density ratio estimation given (D, Dr
8(1)
5: Estimate "g(s), using eXtended-LSDG given (Pot), T).
6: Obtain PaG+1), maximizing Equation (5) given (D, w ((9) ,00, a).
7:
end for
Algorithm 2 Weighted model estimation for policy evaluation (simplified version).
1: Input: D, m, a, and Po(o).
2: for = 0, 2. do
3: Generate D using simulation given (Poco, x).
4: Estimate wT., using density ratio estimation given (D, Dr (.o)
5: Obtain Poc+1) maximizing Equation (7) given (D, w (cc) a).
6: end for
uses the gradient of In (s) with respect to policy parameters in model-free RL. This paper extends this idea to the gradient with respect to model parameters in MBRL. In discounted MDP, d,, satisfies
Extending LSDG. LSDG 33
5 = (1 >>ppp')(s))))))))))))))>>))))>
Σ 2 (5.
s,a
The gradient with respect to the j-th parameter, 0z.
c.. (5) (8') Σ di, (a(aa))))))))) (oV,, In Po(d'1s,)) + (00
(6)
where 28, (s) Vo In (s). Equation (6 implies that =ū, is the value function satisfying
"forward" Bellman equation, where the transition probability function is Pa. and the reward function is Va, In Po(s'( 18,8:,000 a). To estimate =Ø,- this paper extends LSDG by replacing the gradient of In ι r(a) 8) with 7 Ve, In (s) The eXtended-LSDG estimates 27, using simulation data generated using (Pe, T) Since some off-po evaluation methods estimate the discounted state distribution ratio as a solution to another forward Bellman equation [14 15 5 their extensions could also be used instead of LSDG.
Algorithm summarizes how to optimize for policy evaluation, described above.
5.1.2 Simplified version
The computationally expensive part in Algorithm iis estimating zg.... This requires estimating the same number of the value functions satisfying Equation (6) as the number of model parameters. This unrealistic for neural networks with many parameters. As practical simplification, this paper also considers replacing Equation (5) with
LTT (0(00 = N Σ woen (Sn, an) In Pa (s an).
Algorithm 2 is the resulting simplified algorithm. The price of the simplification is the loss of convergence in theory, as the first-order necessary conditions are no longer the same. For this reason, the number of iterations should be specified. In practice, Algorithm 2 also works well, see Section 6
5.2 Policy Optimization
This paper considers an EM-style optimization algorithm, where the E-step fixes n and optimizes and the M-step fixes 0 and optimizes Instead of optimizing Equation 3 directly, this paper uses
5

---

Algorithm 3 Model-based offline policy optimization based on weighted model estimation.
Input: D, a. B', PPoco, and (0))
2: for = 2........
3: if then
4:
Estimate PaC+n,, using ERM given D,
5: else
6:
Compute b, given D,, 7 ())).
7:
Obtain Pater1), using E-step given (D, b (1) a).
8:
end if'
9:
Estimate Cgc-+1), using ERM given (D, P((+1)).
Compute b, given (B', D, Po (
11: Update π (f++1), using M-step given (Paun, b, C(G11)))
end for
surrogate function in the majorization minimization framework 31.
Jsur (0, n: T() x())
- + b(l (LL"((0 (0) -hmin) +
r(s.a) a)
b(co(s.a) - )mmim) -mm +C,
- -(s,a)
(8)
B¹ and C b(L* (3) (0(+)) hmin). The advantages of using Equation (
where
2v ا=00) hoti)) are that the E-step reduces similar problem with Section 5.1 and that the M-step comes down to MDP optimization problem, as described later.
Let
be an estimate of b, where Amin min (s.a, |- In Paco (8'18, a)]. 2 2/(z00 (att))-mmin Note that (,(()) (o(t) is computed using Equation ② given (D, Po,Por,, w where wore can (
())
be estimated using density ratio estimation given (D, Dr a(t) ), and Dr(()) can be generated using
simulation given (Pato, x()).. That is, can be computed when given (D, Poco, x())).
E-step. To update while fixing (+t), the E-step estimates Equation 8 by
(8) (sn; an)
Tr(n: an) (1-7) + 6ln Po (s,
(9)
N
==1
where the constant terms are ignored. Equation (9)
can be minimized using Algorithm In Pa (s (sn, an) with In (s (s, an) + r(5m.0n or Algorithm b(1-x)
replacing
M-step. To update while fixing 0(++1), the M-step estimates Equation 8 by
r(s,a) b(Egri (s, -hmin)
(ss
(10)
1-7
where dgcei is a model Co and C is ignored. This paper trains ég using input data {(S ) n== and output data {- In Poccn (s', (s,, ). Note that Equation (10) is the expected return in a MDP, where the transition probability is Poutt), and the reward function is
r(s, a) - (6)
-hmi))
Here, Egcs+ (s, a) hiin can be seen as an estimate of the modeling error, and (1 - 76 can seen the coefficient scale the penalty for the modeling error. Since this MDP known to the agent, Equation (10) can be optimized using any MDP optimization approaches, including RI algorithms.
Algorithm summarizes the above-mentioned procedure.

---

Alg.1 (full version)
Alg.2 (simplified version)
Weighted loss function
Angular velocity
-1
22
-30
10
Angle (a)
Iteration
(b)
2.0
1.5
Angular velocity
Angular velocity
000
1.0
3.0
-05
-10
-10-
-15
-10
Angle
Angle
(d)
Figure 1: Pendulum task. (a) state of offline data and real future data, denoted by black circle and red star. (b) learning curves weighted loss function. (c) simulated future states when using ERM. (d) simulated future states when using Algorithm [ In (c) and (d), black crosses are simulated future states, and colormap shows log w (Sn, an) for offline data.
Remark. If a 0, then the E-step reduces to ERM, and Lines 6-7 in Algorithm 3 are skipped. This case can be seen an instantiation of MOPO (8) where Egcx+1) - Amin and are the admissible error estimator and the penalty coefficient, respectively. This means that the M-step essentially the same as MOPO, while the E-step is different from MOPO. In other words, MOPO can be seen applying only M-step to optimize the penalized return. Thus, Aggorthmm3jimppn optimizes the penalized return with respect to both a model and policy by the E-step and M-step.
MOPO in that
Numerical Experiment
6.1 Pendulum
Firstly, to simply illustrate the effectiveness of weighting with the artificial weight, this paper studies policy evaluation
pendulum environment. State is the pair of angle and angular velocity, and action a is the torque. Offline data D is collected using a random policy. The initial state around controller that wwing-up the pendulum to (0, 0) in the real environment. That is, real future data DE. the swing-up behavior towards (0, 0).
(IT, 0), and the cost-free state is (0, 0). Target policy
Figure shows the state part of D and D
The outline of implementation as follows. The agent uses Pa represented by two-layer neural networks with 8 units with tanh activation. The agent estimates wi using density ratio estimation via logistic regression 332 [T]. The agent estimates zī using an extension of LSDG 30 For more details, Appendix.

---

Unweighted vs weighted ERM. The discounted return of π averaged over 100 episodes in the real environment is 15.7. This paper regards this value as the expected return, namely, the goal of policy evaluation. When using the ERM method, the estimated value is - 80.7. When using Algorithm | the estimated value is 17.8. That is, Algorithm [ esttmmates the expected return more accurately. Below, this paper describes why A Dobains such a better estimate.
Figure 1 (c) shows simulated future data Dj, when using the unweighted maximum likelihood method, a typical ERM method. The pendulum is predicted not to swing-up to (0, 0), unlike D7, This result can be described as follows. Based on the unweighted loss function, Pe tries represent 'D as much as possible. However, its expressiveness is insufficient for global prediction. As result, Po does not predict the swing-up behavior.
Figure 1 (d) shows D3, when using Algorithm with 1. The state part of D is colored based on the logarithm of w/ (8n, an) when Algorithm| converges. 'he pendulum is predicted swing-up to (0, 0). like D.g_ This situation can be described as follows. When Algorithm| I converges, Line in Algorithm predicts the swing-up behavior shown in Figure Using the swing-up behavior as D Line 4 estimates wg (8n, an)) shown in Figure 1 After estimating =i at Line 5, Line 6 estimates Pa to predict more accurately where the weight is larger. Since the expressive capability of Pe is collected near the larger weights. Pe can predict the same swing-up behavior again at Line in the next iteration, despite the lack of global expressiveness as mentioned in the paragraph above. This is what happens when Algorithm converges.
Simplified version. Figure (b) shows the lerning curves of Algorithm ! and Algorithm 2 where 0 is initialized by the learning result of the ERM method, and wp (sn, an) is normalized to satisfy + Σ., w6 (s,, an) = 1. The results are averaged over 5 runs, and the errorbar is the standard deviation. The simplified version also reduces the weighted loss function, compared to the ERM method, i.e. the start of the iteration. This result suggests that, although the simplified version has the limitation of not converging, it can be used as better than ERM. The simplified version can also predict the swing-up up behavior, similarly to the full version, see Appendix.
6.2 D4RL MuJoCo Benchmark
Secondly, to demonstrate weighting with the artificial weight on large-scale environments, this paper
studies policy optimization on the D4RL Benchmark (33 based on the MuJoCo simulator [34). This paper shows the results of Algorithm 3 with a 0 and a 0.2. This paper regards the case of = 0 as MBRL baseline, as it can be seen as an instantiation of MOPO (8). Appendix discusses the result when using different values for a for the 2 expertt dataset.
The outline of the implementation is as follows. The agent iterates the main loop of Algorithm twice. The agent uses the simplified version of the E-step. The agent estimates Pr(s', r|s, a) instead of P(s'ls, a). to handle the unknown reward function, as in [ The agent represents Pr(s', r|s. a) using ensemble of models, where each model is parametrized as 7-layer neural networks with 256 hidden units. The agent estimates wy in the same way as in Section 6. The agent estimates co using L2 norm minimization. For the M-step, the agent improves ' using SAC (35], where samples are drawn from D DUD.. This paper implements SAC by modifying the implementation code by (36). To stabilize the M-step, the agent terminates a simulation episode when a state or action variable diverges. In addition, the agent clips a penalized reward so that the absolute value is not more than 10 times the maximum absolute value of the reward data. For more details, see Appendix.
Table I shows the normalized scores of Algorithm B where the results are averaged over 5 runs, and + means the standard deviations. The major differences are found for malkerzd-mnddiu--xppe and Walker2d-medium datasets. The normalized scores of a 0.2 is better for the former and worse for the latter. Below, this paper describes the reasons.
Walker2ddmmedium-epper dataset. The case of a 0.2 moves forward without falling for all runs, while the case of a = 0 falls down for two runs. Figure 2 (a) shows the learning curves of real and simulated returns in the M-step at the second iteration for the seed for that run. The case of a 0.2 obtains similar curves for real and simulation, while the case of a 0 does not. This difference comes from the weighted model estimation. In the case of a = O, the model for the first move is not sufficiently accurate, and this lead to falling at the first move. In the case of a 0.2.
8

---

Table D4RL MuJoCo Benchmark: normalized scores. The scores of CQL and original MOPO taken from 37 and 8].
dataset
CQL 37
original MOPO
a = 0
a 0.2
HalfCheetah-random
35,4
35.4 + 2.5
48.7 + 2.8
49.1 + 3.2
HalfCheetab--medium
44.4
42.3 + 1.6
75.7 + 1.5
73.1
lallCheetaa--eddiun--epggg
46.2
53.1 :: 2.0
72.1 1.4
65.5
62.4
63.3 : 38.0
73.9
81.7
Hopper-random
10.8
1.
30.2
3.5
Hopper-medium
86.6
28.0 12.4
2.77
1.2
Hopper-mediun-replaa
48.6
67.5 24.7
9
104.0
Hopper-medium
medium-expert expert
111.0
13.6 6.0
1.1
104.9
Walker2d-randon
7.0
1..5
1.64
Walker2d-medium
74.5
17.8 + 19.3
81.7 1.2
60.7 + 29.0
Walker2d-medium-re
32.6
39.0
80.7 3.1
82.7 33
Walker2d-medium-exxpp
98.7
44.6 + 12.9
59.5 + 49.4
108.2 0.5
Undiscounter return
Undiscounted return
4000
3000
3000
2000
1000
450009
500000
200000
402000
500000
Training steps M-step
Training steps M-step
(b)
Figure 2: D4RL MuJoCo Benchmark: learning curves of (a) walker2d meeiddd-mmiiimexeeeeee and (b) walker2d-medium.
simulated future data includes predictions of the first move, and similar parts of offline data are weighted with large weights. Based on the weights, the estimated model more accurately predicts the first move. As result, the case of a 0.2 can learn to move forward without falling, based on the more accurate prediction. That is, Algorithm improves the performance based on weighted model estimation. For the other runs, although the case of a moves forward without falling in a real environment, it obtains real and simulation returns that are relatively different, see Appendix.
Walker2d-medinn dataset. The case of a 0.2 falls down for two runs, while the case of a = moves forward without falling. Figure (b) shows the learning curves fo those runs. In Figure (b), for both a = 0 and a = 0.2, the simulation curves do not seem to capture the real returns. The difference of the score seems due the high variance of the real returns.
7 Limitations
This paper has limitations common to standard offline MBRL approaches, such as Markov assumption and reward design. The limitations specific to this paper are as follows. This paper derives Algorithm based on the theory, but it cannot be applied to large-scale problems due to the large amount of computations. This paper also presents Algorithm|] 2 as practical simplified version, but it has no convergence with respect to Equation see Section 5.1

---

Conclusion and Future Direction
This paper discusses weighted model estimation considering covariate shift in offline MBRL. The main idea of this paper is weighting with the artificial weight, which is relatively easy to estimate. Based on this idea, this paper defines loss function for offline MBRL and presents algorithms to optimize it. This paper justifies this idea as evaluating the upper bound of the policy evaluation error. This paper demonstrates the effectiveness of this idea numerical experiments. The main contribution of this paper is showing the validity of weighting with the artificial weight.
The propose algorithm shows clear improvement in Section 6.1 while it improves only one case in Section 6.2 One reason for this gap might be that Section 6.1 needs interplolation, while Section 6.2 requires extrapolation. Although interpolation and near extrapolation can addressed by importance- weighting, far extrapolation would require additional idea, e.g. using structured prior. Since further discussion is needed, we leave future work.
Another interesting direction is extending to Bayesian MBRL 38 39 In supervised learning under covariate shift, the posterior distribution can be defined using weighted likelihood function (2] In offline MBRL, the likelihood function weighted with the artificial weight is effective, as shown in this paper. Combining these ideas, weighted posterior considering covariate shift will be introduced into Bayesian MBRL.
Acknowledgments and Disclosure of Funding
This work was partly supported by Collaborative Research for Evolutionary Mechanical System Technology with Mitsubishi Electric Corporation and by General Research Grant from SECOM Science and Technology Foundation. We thank all anonymous reviewers for their constructive comments.
References
[1] Richard S Sutton, Andrew G Barto, al. Introduction reinforcement learning, volume 135. MIT press Cambridge, 1998.
[2] Christopher G Atkeson and Juan Carlos Santamaria. A comparison of direct and model- based reinforcement learning. In IEEE International Conference on Robotics and Automation, volume 4, pages 3557-3566 IEEE, 1997.
[3] Mare Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408-423, 2013.
[4] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning handful of trials using probabilistic dynamics models. In Neural Information Processing Systems, pages 4759-47700 2018.
[5] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint crXiv:2005.01643 2020.
[6] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Neural Information Processing Systems, volume 32, 2019.
[7] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Neural Information Processing Systems, vol- ume 33, pages 21810-21823, 2020.
[8] Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems, volume 33, pages
2020.
[9] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the logliike function. Journal of Statistical Planning and Inference, 90(2):227-244, 2000.
10

---

[10] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Bchavior-agnostic estimation of discounted stationary distribution corrections. In Neural Information Processing Systems, volume 32, 2019.
[11] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[12] Arthur Argenson and Gabriel Dulac-Aroold. Model-based offline planning. In International Conference on Learning Representations, 2021.
[13] Doina Precup, Richard Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In International Conference on Machine Learning, pages 759-766, 2000.
[14] Assaf Hallak and Shie Mannor. Consistent on-line off-policy evaluation. In International Conference on Machine Learning, pages 1372-1383, 2017.
[15] Carles Gelada and Marc G Bellemare. Off-policy deep reinforcement learning by bootstrapping the covariate shift. In AAAI Conference on Artificial Intelligence, volume 33, pages 3647-3655, 2019.
[16] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652-661. PMLR, 2016.
[17] Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148, 2016.
[18] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference Machine Learning, pages 1447-1456, 2018.
[19) Mass-Massoud Farahmand, Andre Barreto, and Daniel Nikovski. Value-aware loss function for model-based reinforcement learning. In International Conference Artificial Intelligence and Statistics, pages 1486-14944 2017.
[20] Amir-Massoud Farahmand. Iterative value-aware model learning. In Neural Information Processing Systems, pages 9072-9083, 2018
[21] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations, 2019.
[22] Yao Liu, Omer Gottesman, Aniruddh Raghu, Matthieu Komorowski, Aldo A Faisal, Finale Doshi-Velez, and Emma Brunskill. Representation balaneing MDPs for off-policy policy evaluation. In Neural Information Processing Systems, pages 2644-2653, 2018.
[23] Romina Abachi, Mohammad Ghavamzadeh, and Amir-massoud Farahmand. Policy-aware model learning for policy gradient methods. arXiv preprint arXiv:2003. 00030, 2020.
[24] Pierluca D' D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, and Marcello Restelli. Gradient-aware model-based policy search. In AAAI Conference Artificial Intelligence, pages 3801-3808, 2020.
[25] Yinlam Chow, Brandon Cui, Moonkyung Ryu, and Mohammad Ghavamzadeh. Variational model-based policy optimization. In International Joint Conference on Artificial Intelligence, pages 2292-2299, 2021.
[26] Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprint 1005:00000000000 2018
[27] Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-based policy optimization with unsupervised model adaptation. In Neural Information Processing Systems, 2020.
[28] Byung-Jun Lee, Jongmin Lee, and Kee-Fung Kim. Representation balancing offline model- based reinforcement learning. In International Conference on Learning Representations, 2021.

---

[29] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 1994.
[30] Tetsuro Morimura, Eiji Uchibe, Junichiro Yoshimoto, Jan Peters, and Kenji Doya. Derivatives of logarithmic stationary distributions for policy gradient reinforcement learning. Neural Computation, 22(2):34--376, 2010.
[31] Kenneth Lange, David R Hunter, and Ilsoon Yang. Optimization transfer using surrogate objective functions. Journal of Computational and Graphical Statistics, 9(1):1-20, 2000.
[32] Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika, 85(3):19-6630 1998.
[33] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint wwXiv:20040022/4 2020.
[34] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference Intelligent Robots and Systems, pages 2026-5033,200 2012.
[35] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[36] Pranjal Tandon.
sytorch-oft--ccor--coo syforch-ooft--cttr--ciiti 2018.
[37] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems, pages 1179-1191.
Curran Associates, Inc., 2020.
[38] Michael G'Gordon Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. PhD thesis, University of Massachusetts Amherst, 2002.
[39] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforce- ment learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359
2015.
12


Reviews:
[{"review_id": "w3S_H827T0I", "reviewer": "Reviewer_mD7J", "summary": "The paper presents a model-based policy optimization algorithm for offline reinforcement learning. To account for the mismatch between the available dataset and the experience used during training, the proposed method uses estimation of the density ratio of the state distributions induced by the estimated and the real model. This importance weight, called “artificial”, is in contrast with the one, referred to as “natural”, employed by other off-policy methods involving the densities for two different policies under a same model. The resulting model learning objective is inserted into an expectation-maximization algorithm, which is then evaluated in a standard offline RL benchmark.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ideas presented in the paper are appealing: by leveraging a known bound on the performance difference under two different models, a method for model learning based on density ratio estimation techniques can be derived. Similarly, the same objective can be integrated into a policy optimization algorithm, which is reminiscent of recent model-based offline reinforcement learning approaches. The derivation of the algorithm is generally sound, although the practical approximation used in the experiments looses most of the guarantees of the initial counterpart.\n\n**Originality**: The method is, to be the best of my knowledge, a new use of importance ratio estimation for model learning in offline RL. Nevertheless, some reference to relevant related work is missing.\n\n**Quality**: The general quality of the work is mostly okay for the theoretical grounding, but less convincing for the experimental evaluation.\n\n**Clarity**: The clarity of the writing could be improved.\n\n**Significance:** The presented method is a new approach at the intersection between two very active research areas, decision-aware model-based RL and offline RL.\n\n**Major concerns**:\n\n- For the experiment on the D4RL domain, given only 4 runs and largely overlapping error bars, I do not think there is enough experimental evidence to argue that one method is better than the other on this particular domain. Even the explanation provided by the paper for the *Walker2d-medium dataset* is not very convincing for me: the learning process can cause the agent to fall for many other, confounding, reasons (e.g., the random initialization of the function approximators in use) apart from fundamental differences in the two methods and I do not see enough nuances in the experimental design to discern whether that one is the specific cause or not.\n- Given the nice properties shown for the method in Section 6.1, I would have expected a clear difference in performance with respect to the baseline in Section 6.2. How can the lack of it be explained? Is the difference in performance in the first experiment due to the limited capacity of the model? What happens if more hidden units are used for the model of the dynamics?\n- The exposition is often unclear for a generic reader. To just give two examples, in line 19, it is written that the paper focuses on offline MBRL, without being explicit on the meaning of this expression; in line 187, it is used the expression \"policy evaluation\", in a sense that is different from the one the typical RL researcher would expect (i.e., computing the value function for a policy).\n\n**Minor concerns**:\n\n- As reported in the related works, the method presented in the paper is related to previous work in decision-aware model learning. By contrast with what is written in the section, the use of a weighted log-likelihood objective for learning a model in the context of offline RL was already employed in (D'Oro, Pierluca, et al. \"Gradient-aware model-based policy search.\" Proceedings of the AAAI Conference on Artificial Intelligence. 2020.), which incorporates a trajectory-based  importance weight to handle the mismatch between behavior and target policies.\n- Can you elaborate again on the passage from Equation (3) to Equation (4)? This should not be so important for the paper overall, but is the importance weight (which is unbounded) missing from the second term? Also, in the same derivation, is an absolute value missing from the first inequality?\n\n**Recommendation:** At the current state, I tend to recommend to reject the paper. I do not see enough experimental, nor theoretical, evidence about the advantages of the proposed method over existing baselines. I am open to increase my score if the authors adequately address my points of concern.\n\n——\n\n**After rebuttal:** I thank the authors for their response. After reading it, I tend to recommend to accept the paper. The bug disclosed and fixed by the authors does not change my general opinion, and my updated score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitations are mostly addressed in the related section of the paper. A question the authors should answer (and that could be a possible limitation for the method) is whether the advantages provided by the model learning procedure are only clear in the presence of model misspecification.\n\nGiven the mostly theoretical and algorithmic nature of the paper, I do not foresee any immediate societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "reC7BxsJhJ", "reviewer": "Reviewer_Ty7z", "summary": "This paper presents a model-based algorithm for offline reinforcement learning (RL) that attempts to address the covariate shift issue by re-weighting the model losses for different datapoints. Based on an upper bound of the policy evaluation error, a joint loss function is defined over both the policy and the model parameters, and the two are optimized in an alternating fashion. Several simplifications and tricks are described to make the algorithm computationally tractable.\n\nOn a simple pendulum environment, the proposed algorithm predicts the policy value much more accurately than a model based on empirical risk minimization (ERM). It also obtained good results on the MuJoCo tasks from the D4RL benchmark.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The algorithm in the paper is novel to my knowledge. Despite some similarities to MOPO, it approaches the model estimation differently (using weighted losses and training jointly with the policy), leading to a more complicated algorithm but better results on D4RL tasks. Considering that distribution shift is a major challenge in offline RL, this strikes me as a useful contribution.\n\nGiven that the algorithm requires a number of simplifications and tricks, it would be useful to ablate those design choices where possible. For example, the authors take a majorization-minimization approach “To make each optimization step easier”, but give no explanation as to what goes wrong if no MM surrogate is used. Does it diverge? Or are the results just not as good?\n\nThe algorithm’s claimed performance on the D4RL MuJoCo tasks is strong compared to previous algorithms. However, I had to consult the D4RL paper to determine this. I would recommend putting one or more of the baseline algorithms in the table for ease of comparison.\n\nI found it odd that numerical results on D4RL were not presented for the simplified version of the algorithm. It appears to work similarly to the full version on the pendulum environment as depicted in Figure 1(b), but it would be good to verify that it scales to more challenging environments, especially given that its primary motivation (relative to the full version) is to reduce the computational burden.\n\nThe writing of the paper is understandable, although the grammar could be improved in many places, and there are several typos.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qe-w95SmNG", "reviewer": "Reviewer_PdDh", "summary": "The paper addresses the problem of covariate shift in offline memory-based reinforcement learning by proposing a novel method for more accurate evaluation of the expected return of a given policy by means of simulation of a learned model of the real system that might not be entirely correct. Two algorithm that uses a weighting factor, called the artificial weight, are proposed for tuning the learned system's model to predict more accurately the parts of the state space visited by the policy being evaluated, and thus improve the estimate of the expected return of this policy. This improved estimate can further be used for policy improvement, as customary for RL algorithms. An empirical evaluation in simulated environments demonstrates the validity of the proposed approach, both for policy evaluation and for policy improvement. \n\nUpdate: I appreciate the clarification about the evaluation task the authors are solving (pendulum swing-up, not stabilization). I also noticed that the other reviewers also found the advantages of the proposed algorithm not all that overwhelming. This, combined with the generally unclear presentation, makes the paper borderline. I am revising my rating to 6, just above borderline - it would still be good to accept this paper, if possible, and depending on the ratings of other papers.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method applies to the case when a data set is collected from a real system by means of an exploration policy, and the new policy to be evaluated would result in a significantly different state occupancy distribution than that of the exploration policy. In cases when the new policy is expensive to evaluate on the new system, and it is instead desired to be evaluated in simulation, using a model learned from the observed data that might have limited modeling capacity, this difference in the state distributions, combined with the possibly incorrect model of the dynamics, would cause incorrect prediction of the system trajectory under the policy, and from there, and incorrect estimate of its expected return on the real system.\n\nAs a solution to this problem, the paper proposes to use a weighting factor called artificial weight, expressed as a ratio of state-action occupancy probabilities, in order to iteratively tune the parameters encoding the dynamics of the learned system model to more accurately predict regions of state space where the policy being evaluated would bring the system to, and from there, the estimate of the expected return of the policy on the real system. In essence, the model of the dynamics ends up focusing on the parts of state space that will actually be used by the evaluated policy, instead of trying to model the dynamics everywhere. A parameter tuning algorithm with convergence guarantees is proposed, along with a simplified version that does not have such guarantees, but is computationally faster. For stability reasons, only a scaled-down value of the weight is used. Both versions of the algorithm are empirically evaluated on a pendulum stabilization problem and policy, and it is evident that the predicted trajectories and the estimate of the expected return of the policy are much closer to that of the real system that what a system model optimized by empirical risk minimization would produce. \n\nMoreover, the improved estimate of the policy's return can further be used for policy improvement, as customary for RL algorithms. Empirical evaluation on the D4RL benchmark suite demonstrates improved performance with respect to the unweighted method. (Although, the comparison is not overwhelmingly in favor of the new method.)  \n\nOverall, the idea appears to be sound, and addresses a major issue with offline MBRL, which itself is a very important RL method for use cases that involve real physical equipment that is difficult, expensive, and slow to operate. It is highly original, and very promising with respect to solving challenging decision and control problems involving physical systems. The proposed work builds upon a lot of related research in ML concerning covariate shift and offline MBRL that is well discussed and referenced by the authors, and the combination of these ideas into the proposed algorithms is the main contribution of this paper. Based on this, I think the paper is a solid contribution to a field of high practical importance and significance, and thus merits acceptance.\n\nAt the same time, the presentation of the ideas and results could be improved. I found the paper fairly difficult to read. The iterative improvement of the parameter $\\theta$ representing the system's dynamics is not explained very well. For example, in Algorithm 1, there is an iteration over index i, but it is not clear how many iterations there are, and what i actually indexes. Is it the successive estimates $\\theta_i$ in lines 4, 5, and 6, respectively $\\theta_{i+1}$ in line 7? If yes, Eq. 10 should probably be updated for consistency, too, replacing $\\theta_0$ with $\\theta_i$ and $\\theta$ with $\\theta_{i+1}$  \n\nIt is also not entirely clear what the objective of the pendulum task is. Is it the pendulum stabilization task around the vertical position (and if yes, which one - up or down), or the much harder problem of pendulum swing-up? The goal state is angle 0 with 0 angular velocity, but does this angle correspond to the unstable up position, or the stable down position? I assume it is the former, but please clarify. It doesn't help that the starting state is not clearly shown, either. The authors say that they are solving the pendulum stabilization task, which is normally defined for relatively small angles around the unstable equilibrium, typically +/-15 degrees. However, Fig. 1 shows a trajectory in phase space (or is it two or more?) starting about pi radians away from the goal state (maybe in the stable equilibrium?) that pumps energy into the system, strongly suggesting a swing-up task with underactuated pendulum. Please clarify the task that the evaluated policy is trying to solve.\n\nMinor typos:\nLine 40: \"the both\" -> \"both\"\nLine 74: \"Preliminary\" -> \"Preliminaries\"\nLine 75: \"a action\" -> \"an action\"\nLine 193: \"with normalizing it\" -> \"and normalizing it\"\nLine 201: \"common logarithm\" -> \"natural logarithm\"?\nLine 228: \"the\" -> \"The\"\nLine 236: \"averaged averaged\" -> \"averaged\"\nLine 237: \"stanrard\" -> \"standard\"\n   \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am not aware of any potential negative social impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "m3crsqgiLLH", "reviewer": "Reviewer_L2Wz", "summary": "This work studies how to estimate a learned model in offline model-based reinforcement learning (MBRL). The authors propose to use \"the artificial weight,\" which is the ratio of offline data and simulated future data, to improve the model estimation. They further propose a new objective function for offline MBRL and an EM-style algorithm to optimize it. The authors evaluate their algorithm in several different tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors devise a new loss function for offline MBRL to alleviate the covariate shift issue. The core idea is weighting with the state-action distribution ratio of offline data and \"simulated\" future data, which is interesting.\nNevertheless, the authors may want to conduct several additional experiments to demonstrate the effectiveness of the proposed approach.\n1. The authors may want to compare the proposed approach with that using \"the natural weight.\" Several popular OPE methods can estimate the natural weight, such as DualDICE [1], GenDICE [2], and Gradient DICE [3].\n2. The author may want to provide a comparison with state-of-the-art offline algorithms, such as CQL [4] and BEAR [5]. \n3. The results in Table 1 seem to show that using \"the artificial weight\" does not improve the performance. I wonder whether the proposed method reduces the error of the fictitious samples. The authors may want to provide more details to explain the results in Table 1. Is it possible the weight $[w^\\pi_{\\theta_0}(s_n,a_n)]^\\alpha$ is so close to 1 that its effect on the learned model is negligible?\n3. The authors may want to show how the hyperparameter $\\alpha$ affects the accuracy of the learned model. For example, the authors may want to illustrate the K-step prediction error during the generation of fictitious samples when $\\alpha$ varies.\n\n[1] Nachum, O., Chow, Y., Dai, B., and Li, L. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. arXiv:1906.04733, 2019.\n\n[2] Zhang, R., Dai, B., Li, L., and Schuurmans, D. GenDICE: Generalized ofﬂine estimation of stationary values. arXiv:2002.09072, 2020.\n\n[3] Zhang, S., Liu, B., and Whiteson, S. Gradientdice: Rethinking generalized ofﬂine estimation of stationary values. In International Conference on Machine Learning, 2020.\n\n[4] Kumar A, Zhou A, Tucker G, et al. Conservative q-learning for offline reinforcement learning. arXiv:2006.04779, 2020.\n\n[5] Kumar A, Fu J, Tucker G, et al. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv:1906.00949, 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers find the problem addressed by the paper important and the proposed idea novel. The most important negative aspect of the paper, however, is with its empirical studies:

There are several issues there:

1) Although the empirical results for the simpler environment (Pendulum) is promising, the results for more difficult set of problems (MuJuCo tasks) are underwhelming. In most cases, there is no significant difference between the proposed method (which corresponds to the choice of $\alpha > 0$, and specifically $\alpha = 0.2$ in the results), and $\alpha = 0$, which would be the case when the method reduces to the usual ERM with no sample weighting.

The lack of significant difference is more exasperated as the results are based on only 5 random seeds and the standard deviation is  large in most cases, and we have a lot of overlap between confidence intervals. It is difficult to say with certainty if the new method actually helps or not. It also appears that there is no really good explanation on why there is not much difference.

I should note that the authors provided new results during the discussion period with 10 runs for a subset of the problems.

Although it is OK if the method is not performing very well, it would make a much stronger paper if we at least learn why that is the case.
I suggest that the authors perform similar experiments, but with larger number of runs and perhaps with proper statistical tests.


In addition to this major issue, there are some other comments:

2) The proposed method follows a sequence of approximation. Ablation studies would be helpful in providing insight in the effect of each approximation.

3) Empirical comparison with methods that use "natural" weights strengthens the paper.

4) There is another comment, which was not raised by reviewers, but is based on my reading of the paper. I do not give much weight to this because the authors have not had a chance to respond to it.

In the derivations leading to Eq. (3), the first inequality uses the telescoping lemma with the choice of distribution of expectation to be w.r.t. the discounted future state distribution induced by the model $P_\theta$ and the value function inside the expectation to be w.r.t. the true model $P^*$. This derivation leads to the introduction of density ratio w_theta in order to change the outer expectation from $d_{P_\theta}$ to $d_{P^*}$ in Eq. (3).

But I believe it was also possible to use the telescoping lemma differently.
If the expectation was taken w.r.t. $d_{P^*}$, the value function inside the expectation would be $V_{P_\theta}$. In the next line, we would still upper bound that value function by $\max_{s,a} |r(s,a)| / (1 - \gamma)$, and we would get the same $B$. The result would be that we wouldn't need any density ratio $w_\theta(s, a)$.

If what I wrote is correct, this means that we could have an upper bound that does not require the introduction of any weights. If so, much of the proposed method would not be needed.

It would be helpful if the authors can clarify this. Perhaps one upper bound is tighter than the other one?

**Evaluation:**
Overall, I would consider this as a *borderline* paper. The suggested method is interesting and might be useful too, but I believe that there is room for improvement and there are straightforward ways that this paper can significantly be improved.
Decision: Accept (Poster)