Venue: NeurIPS 2022 Workshop Offline_RL

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper's contributions are two folds. 1- The authors introduce a dataset for offline Multi-Objective RL with two sets of behavioral policies with different evaluation settings. 2- They propose a novel method called PEDA based on recent works in offline RL. This paper shows that by intelligent preference conditioning, PEDA can achieve performance near the Pareto front of the underlying behavioral policy used to generate the dataset. Overall, I found the paper well-written, including interesting ideas, and useful to the community. 

Major comments:

- There is a claim in the conclusion that "In some scenarios, the learned policies can also generalize to higher target rewards that exceeds the data distribution." somehow was not discussed in the previous sections and needs more evidence. Table 1 shows none of the algorithms can achieve higher performance than the behavioral policy. How the authors reached that conclusion? It could be useful to add more discussion on that. 

- I am not an expert in MORL and had to read the paper twice to understand the difference between an algorithm with/without **P**. It could be helpful to add a few sentences about the difference, why we need to evaluate both cases, and in which scenarios we prefer one over the other.  



review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work presents a benchmark for multi-objective offline RL where at test-time, agent's can be controlled to obtain a desired tradeoff between the various rewards. Variants of prior offline RL methodologies are implemented and compared on the benchmark.

Comments:
* For amateur, why not instead use agents that were not trained to expert level performance?
* I couldn't understand the definition of the Sparsity metric. The manuscript would benefit from better description of Sparsity (with references to works that introduced this metric).
* I think it's very important to also understand how well the methods generalize to preferences outside the set they were trained on.
* Is there room for improvement on your benchmark? If not, it would be important to also have environments where performance is not saturated by methods you have tried. Otherwise, the benchmark would not be useful for future work to evaluate on.
* It would also be good to define a metric that is in the range 0-1, where 1 would indicate that the domain is solved.
* Generally, more complex locomotion tasks (e.g. antmaze from D4RL) or non-locomotion type tasks (e.g. robotic manipulation, health domains, etc.) could also lead to valuable benchmark domains. Unfortunately Mujoco-gym type domains have a tendency to saturate quickly in the literature.
* I didn't understand this sentence: "In principle, the scalarized RTG ˆgt and preference vector ω can recover the vector-valued RTG gt."

review_title: Benchmark for offline MORL; Comparison of various methods for offline MORL;

Decision: Accept