Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Modern day policy (and value) function optimization procedures routinely include terms beyond the typical scalar reward, such as to induce regularization, encourage exploration and perform discriminative feature learning. How must one go about balancing these objectives? The typical scheme is to take a weighted sum; this paper recasts that into a multi-objective optimization question, and proposes an objective that combines policies from various objectives in a weighted KL sense.
main_review: 1. Between weighing the Q-values directly in the policy update vs. weighing KL distances, as DiME does, the paper repeatedly points to the fact that the former is sensitive to the magnitude of rewards. Is there a more fundamental distinction in the tasks one can perform better on vs. the other, especially for a fixed scalar reward (the abstract seems to make a case for this; but the justification for this is not followed up on the paper)? Possibly in terms of some qualitative or quantitative features one might ascribe to the task? 
2. The fact that the proposed approach can discover the pareto frontier  for parameterized rewards is impressive. But, as I understand, both LS (which may have similarities to existing approaches) & DiME are both algorithms proposed in this paper. Is a more direct comparsion possible to an exisiting, published result? MORL is a quite a well established problem in RL. Is there a reason why considering MO-MPO's (for instance) experimental setup beyond humanoid walk-run (for which results are fairly similar) for MORL are prohibitive here?
summary_of_the_review: Futher comparisons on existing benchmarks along with characterization (empirically is more than good enough) of environment where the proposal works would really add value.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes using multi-objective optimization as a tool for tackling challenges in RL. The motivation is that the different additional contraints (or objectives) in the policy optimization step are always in conflict, which is natural to view the existing RL as an instance of MORL problem. Typical MORL applies linear scalarization (LS) by taking a weight sum of the objectives. In this paper, first, the authors identifies the disadvantages of prior work (including LS and MO-MPO) and proposes a new objective called DiME. The proposed DiME fits a separate variational distribution per objective and thus can find a tighter variational lower bound. Second, the authors show case studies of DiME for finetuning and offline RL. Experiments show that DiME outperforms LS on standard MORL problems as well as two case studies (finetuning and offline RL).

main_review: The paper studies a foundamental problem in RL optimization with an interesting view of multi-objective policy optimization, trying to tackle challenges in existing RL approaches. 
The main contributions of this work are in two-fold. First, it provides a novel view that uses MORL as a tool to address fundamental challenges in RL, and in the paper, as case studies, the authors studies the use of MORL in finetuning and offline RL. Second, it proposes a novel MORL approach DiME that outperforms tranditional LS on standard MORL tasks.

The paper did a good derivation and analysis for the proposed DiME approach. Table 1 is nice to show the comparison between prior works. However, it is not theoretically shown that why DiME can outperform LS, though in the paper the authors hypothesize that a separate variational distribution allows DIME to find a tighter variational lower bound. In the experiments, the paper shows that DiME outperforms LS on standard MORL tasks empirically. However, it did not show comparison to MO-MPO --- this is wierd because MO-MPO seems to serve as a better baseline than LS.

In section 5, the paper did case studies of MORL for finetuning and offline RL, and showed how to formulate the algorithms of finetuning and offline RL as MORL problem. The authors provide a general formulation and shows that small modifications to the general objective can recover to existing algorithms. Extensive experiments show that DiME allows better optimization for finetuning algorithms and offline RL algorithms.

summary_of_the_review: Pros:
1. Provide an interesting perspective that regards RL as MORL problems.
2. Propose a better MORL algorithm DiME.
3. Experimental results are interesting and convincing.

Cons:
1. Lack of theoretical results.
2. Did not compare with all baselines, such as MO-MPO.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new algorithm DiME for multi-objective policy optimization. The main target application is in the problems of offline RL and policy finetuning, where some closeness to the behavior policy is often desired, in conjunction with optimizing the value function. Experimentally, the paper shows that DiME outperforms the baseline method of Linear Scalarization (LS) on the above applications, and presents theoretical insights on the comparisons between the proposed DiME and several existing algorithms.
main_review: **Strengths**

The experimental study in this paper is quite extensive, with results on two important applications (offline RL, finetuning) and various task domains. From Table 1 and Table 2, it is quite convincing that DiME outperforms the baseline method of LS (and other existing non multi-objective based algorithms) on offline RL.  

The presentation of this paper is quite clear.

**Weaknesses**

My central concern about this paper is about how DiME compares with the very related existing method of MO-MPO. I have two main questions on this end.
1. Experimentally, MO-MPO is not tested in the main experimental results (Table 2, Figures 2-3). Given that MO-MPO is presented as one of the two main existing algorithms in Table 1 (alongside LS), and LS is tested quite extensively in the above tables/figures, it is somewhat surprising that MO-MPO is not tested in the above results. Only Section 6.1 (multi-objective RL) mentions MO-MPO with results in appendix D, showing that DiME only performs on par with MO-MPO.  
Section 5.2 contains a short discussion on why MO-MPO is not suitable for offline RL. However I find the explanations quite vague and not convincing why it is not implementable or necessarily performs badly in offline RL.

2. Algorithmically, I feel like DiME may actually be not different at all from MO-MPO, especially after taking into account an important implementation trick hidden in the appendix.  
Comparing DiME vs. MO-MPO in Table 1 and Section 3.1, DiME involves choosing both the stepsize $\eta_k$ as well as the weights $\alpha_k$ for the $K$ objectives, whereas for MO-MPO the step-size $\eta_k$ is determined from $\alpha_k$. However, when taking into account the actual implementation details, $\eta_k$ in DiME are actually optimized too, in a fashion very similar to MO-MPO indeed. 
For example, the DiME objective (11) for offline RL has $K=2$, and it chooses $\eta_2=1$ but still leaves $\eta_1=\eta$ a hyperparameter. The main paper and the implementation detail section in Appendix C only provides the initial value of $\eta$, but does not further tell how it is chosen or tuned.  
Going through the appendix, the way to choose $\eta_k$ seems to be hidden in Section B.1, Page 17 (which ought to be a section for the math intuition about variational inference, not for implementation details). From there it seems like $\eta_k$ needs to be optimized with gradient-based optimization on the dual objective (45)---This makes it exactly the same to Eq (4) of the MO-MPO paper (Abdolmaleki et al. 2020). Thus I am afraid that DiME is in essence almost the same as MO-MPO.

Regarding these two questions, I’m curious whether the authors could either provide some more details on how the actual implementation of DiME differs from MO-MPO, and/or present some experiments on the MO-MPO algorithm in conjunction with DiME and LS.

summary_of_the_review: Overall, this is a well-executed paper that presents a new algorithm DiME for multi-objective policy optimization. However, currently, I am afraid it is missing comparisons with the very related prior method MO-MPO, and hiding important implementation details with which the present algorithm becomes almost equivalent to that prior method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors in this paper propose to utilize multi-objective optimization techniques to improve offline reinforcement learning and finetunning. 
Based on the extensive empirical experiments, the proposed method can outperform other baselines which do not view the problem as a multi-objective optimization problem. 

main_review: Strength

- The idea can be naturally applied to the offline rl and finetuning setting, since in these two problems, we can view the original objectives as two different objectives when learning the policy.
- The experiments are conducted thoroughly and demonstrate the effectiveness of the current approach. 

Weakness

- The computational costs. Since the new proposed method requires to sample alpha to learn, it may introduce additional computational costs to learn the multiobjective, especially when we need to learn the whole pareto. Would the authors discuss the computational costs than the single linear combination baseline methods?

- I wonder if it is possible for the authors to compare some other multi objective baselines instead of linear scalarization, such as multiple gradient descent or pcgrad? since I think linear scalarization is the most native baseline for multi objective optimization. 

- I am always curious how the authors implemented the multiobjective optimization part, ever since your first paper. But I never see an opensource code and it is really hard to get the intuition and computation efficiency of the proposed method just from what you described in the appendix. 


summary_of_the_review: Overall I think it is interesting to introduce multiobjective to improve offline rl and policy finetuning tasks, although the computational costs might increase a lot compare with the native approach.
review_title: null

Meta-review: The reviewers in general did not seem to be strongly impressed by the contribution of the paper. As the authors noted, some reviewers seemed to misinterpret the claims of the paper --- the paper is not to design new MORL algorithms that are significantly better on standard MORL benchmarks but is to apply MORL on offline RL and fine-tuning. On the other hand, the AC suspects that the paper's exposition could be more centered around the applications, e.g., arguing why offline RL can be benefited from better MO training, and why the challenge of offline RL is to balance some given notions of risk and return computationally (instead of, e.g., developing the right notion/formula for quantifying the risk and return.) Moreover, I think the paper would be stronger if the evaluation for offline RL setting can be made stronger, e.g., including more tasks and algorithms on the D4RL dataset. If the paper's claim is that MORL is a great tool for offline RL, perhaps it's useful to demonstrate that MORL can achieve SOTA reliably when used on top of existing offline RL algorithms (which almost always have two parts in the objective). In summary, in the AC's opinion, the paper has a valuable contribution to the community but is somewhat boardline for ICLR in the current form, and the AC encourages the authors to resubmit to a top venue conference after addressing some of the reviewers' comments.
Decision: Reject