Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the heterogeneous datasets problem in offline reinforcement learning. The major idea of the proposed method LAPO can be summarized as two points. First, the authors construct the policy with a latent variable to better model multi-modal actions. Second, the advantage-weighted scheme is conducted to highlight the importance of high-reward trajectories during the training. The experimental result shows that LAPO can outperform other compared methods by a clear margin with heterogeneous datasets.
strengths_and_weaknesses: Strengths:

* The paper extends traditional offline reinforcement learning into a practical setting, where the offline datasets are heterogeneous and of diverse quality. This is good for making IL methods applicable in real-world tasks.

* The motivation is clear and the proposed method is technically sound. Moreover, the experiments demonstrate the effectiveness of the proposed method.

Weaknesses:

* The idea of using latent policy to represent multi-modal actions and advantage weighting seems to derive from existing offline RL methods.

* Some related work can be added.
  
  1. The method reminds me of another exponential advantage-weighted method [1] in imitation learning, which also addresses the diverse demonstrations problem. What is the difference in the advantage-weighting scheme between LAPO and [1]?

  2. [2] also addresses the problem of OOD data by weighting in offline reinforcement learning. However, [2] assigns higher weights not to the data *with high advantage* but to the data that is *close to the trained policy distribution*. Is there any contradiction between these two weighting schemes and which one is better?

  [1] Learning to Weight Imperfect Demonstrations, ICML 2021

  [2] Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning, ICML 2021
questions: See Weaknesses.
limitations: The authors have addressed the limitations of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose LAPO, an offline-RL algorithm that is designed to efficiently handle the scenario where the provided dataset has heterogeneity. In this setup, the dataset is composed of trajectories from policies solving tasks that could be different from the target task, and/or trajectories from policies solving the target task but in behaviorally diverse ways. There are two main models in LAPO that are jointly trained – 1.) a latent-variable generative model that maximizes the weighted data log-likelihood, where the weight is computed using the advantage function; 2.) a latent policy that is learned to maximize the long-term discounted cumulative rewards (Q-value). Through experiments on the standard D4RL benchmarks (locomotion, navigation, manipulation), the authors show that LAPO compared favorably to the baseline offline RL methods, especially when the dataset is heterogeneous.
strengths_and_weaknesses: The problem setting considered in this paper – offline-RL with heterogeneous datasets – seems to be under-explored in the community. The paper is well-written, well-motivated (e.g., Figure 2), and the experiments/ablations look thorough. In terms of originality, I am inclined to rate the paper as “incremental” as the proposed method looks to be a derivative of PLAS and prior advantage-weighted BC methods.
questions: 1. The crucial points of differentiation between LAPO and PLAS are: a.) advantage weighting that is applied in equation 4; b.) the iterative training of the generative model. As the authors mention in related-work, the proposal of a latent-variable generative model and a latent-policy was done in PLAS. When these concepts are introduced in Section 3, I find the corresponding references to PLAS missing. 
For quantitative comparisons to PLAS, since LAPO is compared with the “original” PLAS code, could there be other differences in the model implementations (decoder, Q-function, policy) that are having an impact? Is this a reasonable way to ensure parity – run the LAPO code with $\omega=1$ and move the decoder update out of the training loop, keeping everything else the same? 

2. Could the authors provide some intuition as to why the latent policy can avoid OOD actions by using the VAE decoder? The approach is understandable when the action distribution is uni-modal. However, with multi-modal action distributions, could using VAEs – that tend to be mode-covering – potentially lead to sampling OOD actions?

limitations: Yes, the authors have mentioned the limitations in the checklist.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a method, latent-variable advantage-weighted policy optimization (LAPO), that learns a policy on top of an advantage-weighted latent space. In summary, the policy outputs a truncated Gaussian distribution over latent actions, which are decoded by a learned decoder. The key idea is that with this method, the agent can learn good policies from multi-modal data, or data collected by a number of demonstrators possibly completing different tasks. The paper also presents experiments suggesting LAPO does in fact perform well on heterogenous data compared to other offline algorithms.
strengths_and_weaknesses: The paper presents a novel, interesting method that seems to do well empirically in some difficult settings. However, there are issues with presentation that need to be addressed before the paper is ready for publication.

# Strengths

 - Originality: the paper appears to present a novel algorithm. It is most similar to [53], but learns an advantage weighted latent space that empirically performs better that [53] in the domains considered.
 - Significance: The paper considers the offline RL problem, which is highly relevant. A key issue for many offline learning algorithms is dealing with multi-modal data, and this paper seems to take a convincing step towards addressing this problem, making it significant.

# Weaknesses
 - The paper's introduction was confusing. It seemed to be referencing experiments done in a toy environment that are not mentioned elsewhere in the paper. The authors seem to be using this to motivate their method, but the end result was unfocused.
 - The methods section's organization was confusing. The discussion around Equation 3 was fairly confusing without the explanations that came in the later sections, and looking at Algorithm 1 it appears that equation wasn’t actually used anywhere, with LAPO instead alternating between optimizing equations 4, 2/5 and 6.
 - Some of the claims could be better substantiated. Although the results are compelling, it would be helpful to see experiments done with more than 3 random seeds. Also, the fact that AWAC+GMM does not work well does not satisfactorily prove to me that GMMs are not as well fit for this problem as the learned latent embedding.
 - The paper suffers from some issues with presentation quality that distract from the purpose of the work and make it, in its current state, unpublishable. Namely:
1. Figure 3 does not have any axis labels
2. Figures 5 and 6 are scaled sloppily, and the text is warped
3. There is some inconsistency in the results. Why are there different names for the walker2d task between figure 3 and table 1? For Walker2d-mix-jump-v1, why does AWAC win in the table while LAPO wins in the graph?
4. In section 3, it seems as though $Q$ is referring to $Q_\phi$ although that is not written
5. In section 3, $\lambda$ is used multiple times before being introduced.
6. The legend for figure 6 does not agree with the caption
7. The word 'tenerative' should be replaced with 'generative' on line 194
questions: - Was I correct in assuming that multi-walker and walker-mix are the same environment?
- Can you clarify the purpose of the experiments mentioned in the introduction, and possibly provide more details about them?
- In order to prepare the paper for publication, I would strongly suggest addressing all of the presentation issues mentioned above.
limitations: Most of the limitations of the work are related to presentation, which the authors obviously have not addressed. The algorithm itself seems to be novel and the results suggest it could be a significant paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In offline RL, the static datasets are often heterogeneous. 
The standard policies constrained methods can be insufficient to learn from such datasets, since they may artificially assign high probability to out-of-distribution actions, leading to sub-optimal policies.
To address this challenge, this paper proposes to learn a latent variable model that generates high-advantage actions when sampling from a prior distribution on the latent space.
Furthermore, this paper trains a latent policy to obtain state-conditional latent actions, which empirically results in better outcomes compared to sampling from a fixed prior distribution. 
The proposed method, LAPO, performs well on several tested datasets.
strengths_and_weaknesses: ### Strengths
1. It is interesting to see a method to learn a  policy more flexible than the standard deterministic / Gaussian policy.

2. The empirical results are promising.

### Weaknesses
1. Novelty: while the whole algorithm is novel, it seems to be a combination of ideas in PLAS and AWAC. Thus, the novelty is incremental compared with prior work.

2. Presentation: explanation to the proposed method is insufficient. In particular, it will be beneficially to explain the intuition/motivation behind the design of each component. \
Please check the ***Questions*** section for questions I have for this paper.
questions: 
1. L30-31: why learning from heterogeneous settings have the problem of "lack of sufficient high-return trajectories?" Why the dataset "is constructed by re-labeling state-action pairs from some different tasks?" I don't think the tested Maze2d and Antmaze datasets has this relabelling process.
2. It seems from Figure 2 that AWAC_GMM perform competitive to LAPO_VAE? Could you provide some intuition why it underperforms the LAPO in many of the D4RL datasets?
3. L60: How to "construct a state-conditioned latent space that represents high-return actions" if at some state the dataset only contain low-return actions? This can be exacerbated when the dataset only contains few high-return actions, which is what this paper wants to address.
4. For a given targeted task, if the goal is to "select which (of the multiple conflicting action) samples to use," why bother using a "more expressive policy class to represent the multi-modal action distributions?" Since we only need one action choice at each state, why couldn't we just use a deterministic policy + advantage weighted learning? \
As a further discussion, this paper aims at the following two goals: **(1)** represent the multi-modal action distributions, where the multiple modes come from the distribution of conflicting actions; and **(2)** learn a mechanism to select which samples (actions) to use. These two goals themselves seems conflicting and I am confused why we would like to simultaneously achieve both goals. In particular, since this paper use advantage weighted learning, the authors seem to implicitly assume that one action-mode is significantly better the others. Then why do we want to model other inferior action modes?
5. L113-114: why the latent values will be decoded into in-distribution actions? Why couldn't the decoder generate out-of-distribution actions?
6. I am confused by this sentence in L114-116 "The parameter $\theta'$ denotes the parameter of the decoder in earlier versions, which do not provide gradient information and therefore do not interfere with the optimization of the first component."\
If the $p_{\theta'}$ network does not provide gradient information, how can we train the $\pi_v$ network? \
A follow up discussion, the use of $a$ in the first term of Eq. (3) seems unnecessary due the use of $\theta'$. Can we define the Q function as $Q(s,z)$ so that the gradient information can be back-propagated directly to the $\pi_v$ network? I think this one has better semantic meaning that the latent policy, which maximizes the Q-function, does not rely on the action decoder during training. The first term in Eq. (3) is confusing to me since the training of $\pi_v$, and hence the generation of the latent values, entwists with the (lagged) decoder. Will the latent policy change as the decoder change? If so, could you explain the intuition behind this design? \
A even further discussion, since the decoder training already favors high advantage actions, I am not sure if the latent policy can actually *learn* "latent actions which result in high returns after being converted to the original action space using the decoder", as claimed in L148-149. 
7. L119-120, the sentence "while actively prioritize regions ... through the importance weights" is confusing. As shown in Eq. (3), the training of the latent policy $\pi_v$ does not involve the "importance weight". Could you explain this? 
8. L120-121, the sentence "Using the decoder, the latent policy will naturally avoid sampling OOD actions" is confusing. Why the the OOD actions can be *naturally* avoided? How do you define the OOD actions in the latent space, the (function) range of the latent policy?
9. I think Eq. (4) is wrong. Could you double check it with the Eq. (4) of PLAS [1]?
10. I am still unsure how the estimation of the advantage function can be exempted from the OOD actions? Since the Q-function is trained by the TD-error Eq. (2), a wield $a'$ can propagate the error. If so, the training of the decoder $p_\theta$ will be affected by the inaccurate advantage function.

[1] Zhou, Wenxuan, Sujay Bajracharya, and David Held. "PLAS: Latent action space for offline reinforcement learning." arXiv preprint arXiv:2011.07213 (2020).
limitations: The authors briefly discuss the limitations but not potential negative societal impacts.
review_title: null

Meta-review: metareview: The paper studies the setting with heterogeneous datasets in offline reinforcement learning
The paper proposes an offline-RL algorithm that is designed to efficiently handle the scenario. 

The paper received 4 expert reviews with a broad consensus on value of the problem and strengths of the method. While there is a lack of unanimous consensus, all reviewers however agree that the problem setting is of interest to broader community, and while there may be some concerns on the method, it would be to the community's benefit to see this work.
confidence: Certain
Decision: Accept