Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In model-based offline reinforcement learning, a dynamics model is trained from the dataset and subsequently used to produce a decision-making policy which attains high reward (according to the model). However, one must also take care not to allow the policy to exploit the model’s inaccuracies, so we also want it to visit states and take actions which have low model uncertainty. Prior work has managed this tradeoff by linearly combining the two objectives.

This work proposes instead to find many diverse policies along the Pareto front of these two objectives. The algorithm, Pareto policy pool (P3), proceeds as follows:
1. Generate reference vectors which quantify tradeoffs in the Pareto front.
2. In parallel, find a Pareto-optimal policies for each reference vector using “Algorithm 2” to solve a constrained bi-objective optimization problem, where the constraint ties the solution to the reference vector. Algorithm 2 begins by finding a feasible solution and then applies MGDA to improve the objectives.
3. “Local extension”: Each reference vector is perturbed in two opposing directions, and then each perturbed vector is further optimized via Algorithm 2, with intermediate policies being added to the policy pool. (This is just an optimization to reduce the computational cost.)
4. At test time, each policy in the pool is evaluated, and the best one is selected.

The paper includes some theoretical analysis of Algorithm 2, showing convergence to an approximate stationary point. Empirical evaluation is performed on the D4RL benchmark, where P3 achieves good results, particularly on the datasets generated by lower-performing policies.
main_review: The paper is well-motivated, and the idea to approximate the Pareto front between model-estimated performance and uncertainty is novel to my knowledge. The algorithmic choices (e.g. local extension to reduce computational cost of training from scratch) also make sense. Algorithm 2 is intuitive, although not particularly novel as it is essentially just applying MGDA with gradients estimated by ES, plus some small modifications to account for the constraint.

The performance of P3 on the D4RL tasks is good. UWAC is clearly better for cases where there is expert data, but for the lower-quality datasets, P3 handily outperforms the baselines. I also appreciate that the authors have taken efforts to tune the MOPO baseline, in some cases even finding hyperparameters that work better than the original paper’s results.

My main concern is regarding the fairness of the empirical evaluation, where P3 arrives with a large set of policies to be evaluated, unlike the other methods which produce only one policy (or one for each hyperparameter setting). The authors explain that the computational cost of running P3 is similar to the cost of tuning other methods, which I can believe, but even if you don’t tune P3’s hyperparameters, you still have to draw samples to evaluate each policy, and in the offline setting we do not want to be drawing so many additional samples. Thus, while the authors frame the development of many diverse policies as a strength of their method, I feel that it is only a strength when paired with a strategy to choose from this set of policies (i.e. off-policy evaluation), which is not presented in this paper. At the very least, I feel that the authors should control for the number of samples used at model selection time when comparing against prior work, as this is arguably more of a bottleneck than runtime in offline RL. Currently P3 has an advantage at evaluation time because you get to try many policies and take the max.
summary_of_the_review: I think the high-level strategy pursued by the paper is reasonable, and the results are promising. Thus, the paper is a useful contribution overall. However, I have reservations about the setting of the empirical evaluation, and think more care could be taken to ensure a fair comparison.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model-based approach for offline RL that is inspired by multi-objective RL. The approach, called P3, finds a Pareto front of policies that trade off between obtaining high return (with respect to the model trained on the offline dataset) and minimizing the model's uncertainty. To obtain this Pareto front, the approach has two stages: 1) find a few spread-out solutions on the Pareto front using a gradient-based method, and 2) initialize with these solutions to find optimal solutions for the neighboring regions of the Pareto front. The empirical evaluation shows that P3 outperforms existing approaches when the offline dataset is of low or medium quality.
main_review: In terms of strengths, this paper presents a novel perspective on navigating the trade-off between model-based return versus uncertainty, for offline RL. The experiments are thorough in comparing against a range of baselines, including both model-based and non-model-based offline RL algorithms. Figures 1 and 2 do a nice job of visualizing the approach and the results.

In terms of weaknesses, a main weakness is that P3 does not outperform existing approaches for good-quality offline RL datasets: in Table 1, for the medium-expert and expert datasets, it consistently performs substantially worse then the UWAC baseline. Why is this the case? I would expect P3 to perform at least as well, if it is actually recovering the true Pareto front. Another weakness is that the experiments are on a relatively small number of domains—only on three MuJoCo domains, out of the many domains available in D4RL.

Also, in practice, trying out every single (fully-trained) policy in the real environment is still expensive. This should be acknowledged in the paper. If I understand correctly, the results in Table 1 for P3 show the best performance across all policies found. This is significantly more computationally expensive than prior approaches (because at least several of the Pareto policies require separate training from scratch), and also requires more runs on the real environment. So it is not entirely a fair comparison. With that said, I appreciate the ablation studies that make a more fair comparison against ablations of P3.

Finally, the algorithmic design decisions should be motivated more clearly. In particular, when computing the similarity metric between the reference vector and the objective vector in Equation (6), why is KL-divergence used instead of a dot product? The dot product is the typical way of measuring similarity between vectors (as is the case here), whereas KL-divergence is meant to be used for distributions. Also, in Equation (4), does the uncertainty need to be exponentiated and tempered?

Additional comments:
- The Related Work and Conclusion should be in the main paper, not in the Appendix.
- In the Related Work, it would be useful to also discuss non-model-based approaches for offline RL. It is also worth mentioning DiME [1], a recent approach that, similar to P3, also takes a multi-objective perspective for offline RL (although in a non-model-based setting), and obtains state-of-the-art results by doing so.
- There are several typos and grammatical errors. e.g. "computational forbidden" at the bottom of page 3
- In the last paragraph of Section 3.1, the writing is overly verbose and unclear. For instance, it mentions twice that it is inefficient to train multiple policies from scratch to obtain the Pareto front.
- Section 4.2 claims that scalarization does not perform well because it only finds policies on the convex part of the Pareto front, but the Pareto fronts found by P3 (in Figure 8) don’t look concave. Instead, I think the different scales of the objectives is the issue, because only five weight settings for linear scalarization are tried.
- In A.8, for walker2d-random and walker2d-medium, why do these not look like Pareto fronts? In other words, why is there not a conflict between optimizing for model-based return and minimizing model uncertainty for this task?

[1] Abdolmaleki et al. On Multi-objective Policy Optimization as a Tool for Reinforcement Learning. 2021.
summary_of_the_review: This is an interesting and relevant direction to study, with promising empirical results for tasks with low- and medium-quality datasets. However, the algorithmic design decisions and empirical results (for good-quality datasets) require more explanation, and the presentation requires polishing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new algorithm for model-based offline reinforcement learning in which the learner attempts to optimally balance maximizing the reward under the learned model and minimizing the uncertainty of the model. The algorithm is motivated by the fact that offline RL methods suffer from distribution shift that often results in policies/predictions that are overly optimistic about highly uncertain areas of the state-action space. The authors’ proposed method attempts to address this problem by identifying a relevant collection of policies lying on the Pareto frontier that trades off these two criteria. The underlying supposition is that a well-performing policy should be among this collection of “non-dominated” policies on the frontier. Specialized subroutines are proposed in order to make sure the collection is diverse. Some theory is presented to validate certain aspects. Extensive experimental results support that the method performs well in relevant benchmarks compared to a number of state-of-the-art methods.

main_review: Strengths:
- The paper is well written aside from a few typos here and there. The figures are well made and clearly convey relevant information about the problem, method, and experimental results.
- The problem identified by the authors is highly relevant to the RL and offline RL community. While this problem (and similar ones) has been identified before, there is relatively little prior work that has addressed it despite it being an important problem.
- The proposed method seems technically sound and the techniques for multi-objective optimization in RL seem to be of interest to the community. The theory is not particularly new, but it does validate certain parts of the algorithm, which is nice.
- The experiments are extensive and compare against a large pool of recently proposed methods. P3 appears to very clearly out-perform the other methods when low-medium quality trajectories are given in the dataset. The ablation study and heatmaps are particularly informative to understand exactly what the algorithm is achieving.



Weaknesses:
- The algorithm is purported to apply to the case of offline RL, yet there is a critical concession in practice: one must have online access to the simulator/domain to actually select the best policy from the frontier -- the algorithm is unable to determine this with offline data alone. This is concerning for two reasons. (1) In my opinion, this trivializes the model/policy/hyperparameter selection problem since we can simply try many different hyperparameters for a given algorithm and see which one is best directly, just as one can in standard supervised learning e.g. by cross-validation or hold-out sets. (2) I do not see an easy way to remedy this problem (I am certainly open to and curious about any suggestions) because Figure 5 shows that a large portion of frontier policies still have low reward so random sampling probably won’t work reliably and something like OPE requires hyperparameters of its own.
- Given that some online access is permitted to do the final selection for P3, I think a key experiment that is missing is a comparison with hyperparameter sweeps of the benchmark algorithms like MOReL, MOPO, UWAC, etc. in each domain. That is, if one grids for example the regularization weight $\lambda$ mentioned in Section 2 and picks the best for each domain from online evaluation, how does this compare with P3? Otherwise it seems that P3 has an unfair advantage. I think if such a comparison is made, then the previous issue is not as significant.

Minor comments and questions:
- Equation (4): What is the motivation for the exponentiated uncertainty? How do you set kappa?
- Can you define “non-convex regions” on the Pareto frontier?
- Fig 5 is informative to some extent, but I am confused about how to interpret the full frontier. Clearly there are some points in high reward regions, but also many in seemingly arbitrary areas. Can additional discussion be provided to aid interpretation? Why are some of the points very dense while others are not?
- Some important sections like conclusion and related work seem to be in the appendix.
- For the experiments, how many policies are in the set $\mathcal{P}$ in the end?

summary_of_the_review: In summary, I believe there are a lot of positives about this paper. It identifies and studies an important problem in the literature. The technical contributions and main ideas are interesting and appear to be new to this problem setting. The experiments are reasonably thorough. Unfortunately, I feel that the mentioned weaknesses (requiring online access, lack of comparison against hyperparameter sweeping other algorithms) are significant and leave open questions about the practical applicability of the proposed method and comparison to prior algorithms.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new model-based offline reinforcement learning, where a pool of Pareto optimal policies is trained on a model of the environment provided by offline RL data. First, a model of the environment is trained using supervised learning from the dataset of logged experiences and subsequently a set of Pareto optimal policies is trained along the dimension of return in the MDP model and uncertainty in the MDP model. The authors argue that providing a Pareto optimal set of policies is superior to prior methods that relied on regularization of the two objectives into a combined metric, as having a Pareto optimal set of policies enables one too explore a range of behavior that is inherent with trade-off between model return vs model uncertainty in offline model based RL.

The authors then describe their method, Pareto Policy Pool (P3), which addresses the bi-objective optimization problem two stage method. Initially, the authors initialize a set of reference vectors along the Pareto frontiers to ensure a diverse set of Pareto policies can be found. Subsequently, the authors apply "Local Pareto Extension" where first a given policy is corrected to a desired range along the objective dimensions provided by the reference vectors. Once the policies are within the desired range, the method applies MGDA based optimization to further improve the performance of the policy in a Pareto optimal way (ascension stage). In order to alleviate computational cost of training multiple policies, the authors apply a "Pareto extension" method to initialize new policies along the Pareto frontier. 

The authors then perform experiments on D4RL based benchmarks and compare their method to other literature methods and claim outperformance, particularly in lower quality dataset settings. The authors also perform an ablation study pertaining to the various components of their method, as well as an analysis related to why it is challenging to obtain optimal trade-offs in model based offline RL.
main_review: **Strengths** 

I think the paper has the following strengths:
- The paper outlines a novel method, with a thorough and detailed description, for offline model-based reinforcement learning that reformulates a significant problem in a bi-objective manner. 
- The paper provides a thorough explanation of the various components of their method, including algorithms, gradients and relevant parameters and hyper parameters. 
- The method shows general outperformance compared to other methods in lower data quality settings in the D4RL benchmark.

**Weaknesses** 

I think the paper could be strengthened by providing more detail on the following:
- Clarifying how the policies for the score in Table 1 were chosen. Given that you have a Pareto front, did you choose the one with the highest score on the given benchmark? If so, can you provide more detail on how many policies end up in your Pareto front. I am potentially concerned that creating many additional could add significant computation overhead when measuring performance if all policies are tested, so further details could help.
- You mention that you use ES for gradient estimation for the MGDA based optimization. Did you consider other methods and how do you think gradient estimation will impact the quality of solutions along the Pareto frontier that P3 can find?
- Can you provide more detail on how you chose the number of reference vectors for your bi-objective optimization ("n" in Line 5 Algorithm 1). Figure 5 has many (red-dot) Pareto optimal points, so it appears that many reference vectors were chosen. Related to the first question, I am trying to get a sense of the scale of the optimization problem.
- Related question to Figure 5: Based on my assessment it appears that not all red dots representing Pareto policies are non-dominated along the two objective dimensions. Could you clarify if this is the case? 
-  Could you comment on why you think UWAC outperforms P3 in the medium-expert and expert cases?

**Additional Remarks**  

- Reproducibility: While hyperparameters and settings are provided in the appendix, there is no mention that I can find of code release. Could you provide more detail on whether you plan to release any source code?


summary_of_the_review: Overall, I think that the paper provides a novel and valuable method with a satisfactory set of experiments to support the overall claims. I have outlined some of my questions and concerns above and think that further clarification will give me a better sense to re-assess my recommendation during the discussion period. 

----- Discussion Period Comments -----

I am adjusting my score based on the authors' response to my questions, their modifications in the paper draft and response to the feedback of other reviewers. I will follow up with specific technical questions in the individual response.
review_title: null

Meta-review: Previous approaches to model-based offline RL require carefully tuning the trade-off between model return and uncertainty. The authors propose an approach that produces a diverse pool of policies on the Pareto front of this tradeoff. On the D4RL offline RL benchmark, P3 outperforms competing approaches when the experience is collected with low or medium return policies.

Before the rebuttal, reviewers identified the following primary concerns:
* The experimental evaluation of P3 uses many policy evaluations to select the policy, which results in an unfair comparison with existing methods.
* P3 underperforms existing methods on some datasets. Why?

Overall, reviewers were satisfied by the response and raised their scores accordingly. The authors responded by including a modification of P3 that uses FQE to select the policy for evaluation, resolving the first concern. The authors explain that P3 underperforms on high return datasets because it splits its updates across the pool of policies. The authors state, "We believe (and in theory it does hold) that P3 can achieve the same performance of UWAC on high-quality datasets, if provided with more computational budget." I suggest that the authors conduct at least one experiment to verify this claim.

The proposed idea is interesting and the revisions the authors have made resolved the primary concerns from reviewers, so I recommend acceptance. The reviewer/author discussion has many substantial points that I recommend the authors integrate into the revision.
Decision: Accept (Poster)