Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a model-based algorithm for offline reinforcement learning (RL) that attempts to address the covariate shift issue by re-weighting the model losses for different datapoints. Based on an upper bound of the policy evaluation error, a joint loss function is defined over both the policy and the model parameters, and the two are optimized in an alternating fashion. Several simplifications and tricks are described to make the algorithm computationally tractable.

On a simple pendulum environment, the proposed algorithm predicts the policy value much more accurately than a model based on empirical risk minimization (ERM). It also obtained good results on the MuJoCo tasks from the D4RL benchmark.
main_review: The algorithm in the paper is novel to my knowledge. Despite some similarities to MOPO, it approaches the model estimation differently (using weighted losses and training jointly with the policy), leading to a more complicated algorithm but better results on D4RL tasks. Considering that distribution shift is a major challenge in offline RL, this strikes me as a useful contribution.

Given that the algorithm requires a number of simplifications and tricks, it would be useful to ablate those design choices where possible. For example, the authors take a majorization-minimization approach “To make each optimization step easier”, but give no explanation as to what goes wrong if no MM surrogate is used. Does it diverge? Or are the results just not as good?

The algorithm’s claimed performance on the D4RL MuJoCo tasks is strong compared to previous algorithms. However, I had to consult the D4RL paper to determine this. I would recommend putting one or more of the baseline algorithms in the table for ease of comparison.

I found it odd that numerical results on D4RL were not presented for the simplified version of the algorithm. It appears to work similarly to the full version on the pendulum environment as depicted in Figure 1(b), but it would be good to verify that it scales to more challenging environments, especially given that its primary motivation (relative to the full version) is to reduce the computational burden.

The writing of the paper is understandable, although the grammar could be improved in many places, and there are several typos.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies how to estimate a learned model in offline model-based reinforcement learning (MBRL). The authors propose to use "the artificial weight," which is the ratio of offline data and simulated future data, to improve the model estimation. They further propose a new objective function for offline MBRL and an EM-style algorithm to optimize it. The authors evaluate their algorithm in several different tasks.
main_review: The authors devise a new loss function for offline MBRL to alleviate the covariate shift issue. The core idea is weighting with the state-action distribution ratio of offline data and "simulated" future data, which is interesting.
Nevertheless, the authors may want to conduct several additional experiments to demonstrate the effectiveness of the proposed approach.
1. The authors may want to compare the proposed approach with that using "the natural weight." Several popular OPE methods can estimate the natural weight, such as DualDICE [1], GenDICE [2], and Gradient DICE [3].
2. The author may want to provide a comparison with state-of-the-art offline algorithms, such as CQL [4] and BEAR [5]. 
3. The results in Table 1 seem to show that using "the artificial weight" does not improve the performance. I wonder whether the proposed method reduces the error of the fictitious samples. The authors may want to provide more details to explain the results in Table 1. Is it possible the weight $[w^\pi_{\theta_0}(s_n,a_n)]^\alpha$ is so close to 1 that its effect on the learned model is negligible?
3. The authors may want to show how the hyperparameter $\alpha$ affects the accuracy of the learned model. For example, the authors may want to illustrate the K-step prediction error during the generation of fictitious samples when $\alpha$ varies.

[1] Nachum, O., Chow, Y., Dai, B., and Li, L. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. arXiv:1906.04733, 2019.

[2] Zhang, R., Dai, B., Li, L., and Schuurmans, D. GenDICE: Generalized ofﬂine estimation of stationary values. arXiv:2002.09072, 2020.

[3] Zhang, S., Liu, B., and Whiteson, S. Gradientdice: Rethinking generalized ofﬂine estimation of stationary values. In International Conference on Machine Learning, 2020.

[4] Kumar A, Zhou A, Tucker G, et al. Conservative q-learning for offline reinforcement learning. arXiv:2006.04779, 2020.

[5] Kumar A, Fu J, Tucker G, et al. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv:1906.00949, 2019.
limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a model-based policy optimization algorithm for offline reinforcement learning. To account for the mismatch between the available dataset and the experience used during training, the proposed method uses estimation of the density ratio of the state distributions induced by the estimated and the real model. This importance weight, called “artificial”, is in contrast with the one, referred to as “natural”, employed by other off-policy methods involving the densities for two different policies under a same model. The resulting model learning objective is inserted into an expectation-maximization algorithm, which is then evaluated in a standard offline RL benchmark.
main_review: The ideas presented in the paper are appealing: by leveraging a known bound on the performance difference under two different models, a method for model learning based on density ratio estimation techniques can be derived. Similarly, the same objective can be integrated into a policy optimization algorithm, which is reminiscent of recent model-based offline reinforcement learning approaches. The derivation of the algorithm is generally sound, although the practical approximation used in the experiments looses most of the guarantees of the initial counterpart.

**Originality**: The method is, to be the best of my knowledge, a new use of importance ratio estimation for model learning in offline RL. Nevertheless, some reference to relevant related work is missing.

**Quality**: The general quality of the work is mostly okay for the theoretical grounding, but less convincing for the experimental evaluation.

**Clarity**: The clarity of the writing could be improved.

**Significance:** The presented method is a new approach at the intersection between two very active research areas, decision-aware model-based RL and offline RL.

**Major concerns**:

- For the experiment on the D4RL domain, given only 4 runs and largely overlapping error bars, I do not think there is enough experimental evidence to argue that one method is better than the other on this particular domain. Even the explanation provided by the paper for the *Walker2d-medium dataset* is not very convincing for me: the learning process can cause the agent to fall for many other, confounding, reasons (e.g., the random initialization of the function approximators in use) apart from fundamental differences in the two methods and I do not see enough nuances in the experimental design to discern whether that one is the specific cause or not.
- Given the nice properties shown for the method in Section 6.1, I would have expected a clear difference in performance with respect to the baseline in Section 6.2. How can the lack of it be explained? Is the difference in performance in the first experiment due to the limited capacity of the model? What happens if more hidden units are used for the model of the dynamics?
- The exposition is often unclear for a generic reader. To just give two examples, in line 19, it is written that the paper focuses on offline MBRL, without being explicit on the meaning of this expression; in line 187, it is used the expression "policy evaluation", in a sense that is different from the one the typical RL researcher would expect (i.e., computing the value function for a policy).

**Minor concerns**:

- As reported in the related works, the method presented in the paper is related to previous work in decision-aware model learning. By contrast with what is written in the section, the use of a weighted log-likelihood objective for learning a model in the context of offline RL was already employed in (D'Oro, Pierluca, et al. "Gradient-aware model-based policy search." Proceedings of the AAAI Conference on Artificial Intelligence. 2020.), which incorporates a trajectory-based  importance weight to handle the mismatch between behavior and target policies.
- Can you elaborate again on the passage from Equation (3) to Equation (4)? This should not be so important for the paper overall, but is the importance weight (which is unbounded) missing from the second term? Also, in the same derivation, is an absolute value missing from the first inequality?

**Recommendation:** At the current state, I tend to recommend to reject the paper. I do not see enough experimental, nor theoretical, evidence about the advantages of the proposed method over existing baselines. I am open to increase my score if the authors adequately address my points of concern.

——

**After rebuttal:** I thank the authors for their response. After reading it, I tend to recommend to accept the paper. The bug disclosed and fixed by the authors does not change my general opinion, and my updated score.
limitations_and_societal_impact: The main limitations are mostly addressed in the related section of the paper. A question the authors should answer (and that could be a possible limitation for the method) is whether the advantages provided by the model learning procedure are only clear in the presence of model misspecification.

Given the mostly theoretical and algorithmic nature of the paper, I do not foresee any immediate societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the problem of covariate shift in offline memory-based reinforcement learning by proposing a novel method for more accurate evaluation of the expected return of a given policy by means of simulation of a learned model of the real system that might not be entirely correct. Two algorithm that uses a weighting factor, called the artificial weight, are proposed for tuning the learned system's model to predict more accurately the parts of the state space visited by the policy being evaluated, and thus improve the estimate of the expected return of this policy. This improved estimate can further be used for policy improvement, as customary for RL algorithms. An empirical evaluation in simulated environments demonstrates the validity of the proposed approach, both for policy evaluation and for policy improvement. 

Update: I appreciate the clarification about the evaluation task the authors are solving (pendulum swing-up, not stabilization). I also noticed that the other reviewers also found the advantages of the proposed algorithm not all that overwhelming. This, combined with the generally unclear presentation, makes the paper borderline. I am revising my rating to 6, just above borderline - it would still be good to accept this paper, if possible, and depending on the ratings of other papers.
main_review: The method applies to the case when a data set is collected from a real system by means of an exploration policy, and the new policy to be evaluated would result in a significantly different state occupancy distribution than that of the exploration policy. In cases when the new policy is expensive to evaluate on the new system, and it is instead desired to be evaluated in simulation, using a model learned from the observed data that might have limited modeling capacity, this difference in the state distributions, combined with the possibly incorrect model of the dynamics, would cause incorrect prediction of the system trajectory under the policy, and from there, and incorrect estimate of its expected return on the real system.

As a solution to this problem, the paper proposes to use a weighting factor called artificial weight, expressed as a ratio of state-action occupancy probabilities, in order to iteratively tune the parameters encoding the dynamics of the learned system model to more accurately predict regions of state space where the policy being evaluated would bring the system to, and from there, the estimate of the expected return of the policy on the real system. In essence, the model of the dynamics ends up focusing on the parts of state space that will actually be used by the evaluated policy, instead of trying to model the dynamics everywhere. A parameter tuning algorithm with convergence guarantees is proposed, along with a simplified version that does not have such guarantees, but is computationally faster. For stability reasons, only a scaled-down value of the weight is used. Both versions of the algorithm are empirically evaluated on a pendulum stabilization problem and policy, and it is evident that the predicted trajectories and the estimate of the expected return of the policy are much closer to that of the real system that what a system model optimized by empirical risk minimization would produce. 

Moreover, the improved estimate of the policy's return can further be used for policy improvement, as customary for RL algorithms. Empirical evaluation on the D4RL benchmark suite demonstrates improved performance with respect to the unweighted method. (Although, the comparison is not overwhelmingly in favor of the new method.)  

Overall, the idea appears to be sound, and addresses a major issue with offline MBRL, which itself is a very important RL method for use cases that involve real physical equipment that is difficult, expensive, and slow to operate. It is highly original, and very promising with respect to solving challenging decision and control problems involving physical systems. The proposed work builds upon a lot of related research in ML concerning covariate shift and offline MBRL that is well discussed and referenced by the authors, and the combination of these ideas into the proposed algorithms is the main contribution of this paper. Based on this, I think the paper is a solid contribution to a field of high practical importance and significance, and thus merits acceptance.

At the same time, the presentation of the ideas and results could be improved. I found the paper fairly difficult to read. The iterative improvement of the parameter $\theta$ representing the system's dynamics is not explained very well. For example, in Algorithm 1, there is an iteration over index i, but it is not clear how many iterations there are, and what i actually indexes. Is it the successive estimates $\theta_i$ in lines 4, 5, and 6, respectively $\theta_{i+1}$ in line 7? If yes, Eq. 10 should probably be updated for consistency, too, replacing $\theta_0$ with $\theta_i$ and $\theta$ with $\theta_{i+1}$  

It is also not entirely clear what the objective of the pendulum task is. Is it the pendulum stabilization task around the vertical position (and if yes, which one - up or down), or the much harder problem of pendulum swing-up? The goal state is angle 0 with 0 angular velocity, but does this angle correspond to the unstable up position, or the stable down position? I assume it is the former, but please clarify. It doesn't help that the starting state is not clearly shown, either. The authors say that they are solving the pendulum stabilization task, which is normally defined for relatively small angles around the unstable equilibrium, typically +/-15 degrees. However, Fig. 1 shows a trajectory in phase space (or is it two or more?) starting about pi radians away from the goal state (maybe in the stable equilibrium?) that pumps energy into the system, strongly suggesting a swing-up task with underactuated pendulum. Please clarify the task that the evaluated policy is trying to solve.

Minor typos:
Line 40: "the both" -> "both"
Line 74: "Preliminary" -> "Preliminaries"
Line 75: "a action" -> "an action"
Line 193: "with normalizing it" -> "and normalizing it"
Line 201: "common logarithm" -> "natural logarithm"?
Line 228: "the" -> "The"
Line 236: "averaged averaged" -> "averaged"
Line 237: "stanrard" -> "standard"
   

limitations_and_societal_impact: I am not aware of any potential negative social impact of this work.
review_title: null

Meta-review: The reviewers find the problem addressed by the paper important and the proposed idea novel. The most important negative aspect of the paper, however, is with its empirical studies:

There are several issues there:

1) Although the empirical results for the simpler environment (Pendulum) is promising, the results for more difficult set of problems (MuJuCo tasks) are underwhelming. In most cases, there is no significant difference between the proposed method (which corresponds to the choice of $\alpha > 0$, and specifically $\alpha = 0.2$ in the results), and $\alpha = 0$, which would be the case when the method reduces to the usual ERM with no sample weighting.

The lack of significant difference is more exasperated as the results are based on only 5 random seeds and the standard deviation is  large in most cases, and we have a lot of overlap between confidence intervals. It is difficult to say with certainty if the new method actually helps or not. It also appears that there is no really good explanation on why there is not much difference.

I should note that the authors provided new results during the discussion period with 10 runs for a subset of the problems.

Although it is OK if the method is not performing very well, it would make a much stronger paper if we at least learn why that is the case.
I suggest that the authors perform similar experiments, but with larger number of runs and perhaps with proper statistical tests.


In addition to this major issue, there are some other comments:

2) The proposed method follows a sequence of approximation. Ablation studies would be helpful in providing insight in the effect of each approximation.

3) Empirical comparison with methods that use "natural" weights strengthens the paper.

4) There is another comment, which was not raised by reviewers, but is based on my reading of the paper. I do not give much weight to this because the authors have not had a chance to respond to it.

In the derivations leading to Eq. (3), the first inequality uses the telescoping lemma with the choice of distribution of expectation to be w.r.t. the discounted future state distribution induced by the model $P_\theta$ and the value function inside the expectation to be w.r.t. the true model $P^*$. This derivation leads to the introduction of density ratio w_theta in order to change the outer expectation from $d_{P_\theta}$ to $d_{P^*}$ in Eq. (3).

But I believe it was also possible to use the telescoping lemma differently.
If the expectation was taken w.r.t. $d_{P^*}$, the value function inside the expectation would be $V_{P_\theta}$. In the next line, we would still upper bound that value function by $\max_{s,a} |r(s,a)| / (1 - \gamma)$, and we would get the same $B$. The result would be that we wouldn't need any density ratio $w_\theta(s, a)$.

If what I wrote is correct, this means that we could have an upper bound that does not require the introduction of any weights. If so, much of the proposed method would not be needed.

It would be helpful if the authors can clarify this. Perhaps one upper bound is tighter than the other one?

**Evaluation:**
Overall, I would consider this as a *borderline* paper. The suggested method is interesting and might be useful too, but I believe that there is room for improvement and there are straightforward ways that this paper can significantly be improved.
Decision: Accept (Poster)