Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies an offline RL algorithm by utilizing the special policy structure of the behavior policy (which generates the offline data). The paper proposes to define the policy being learned by using a mixture of sub-policies, which is in the form of pi(s, a) = sum_z p(z|s) pi_sub (a | s, z), where z is the introduced latent variables. The sub policy pi_sub is deterministic and can be written as mu(s, z).  The basic idea behind this form is that the mixture policy is less likely to force the policy to cover all modes in the offline data; covering all modes using a single-mode policy is potentially harmful as it may provide support to many OOD actions, which results in overestimation. The paper then proposes to optimize the policy pi by maximizing the variational lower bound. Experiments are presented on commonly seen testing domains in offline RL literature. 
strength_and_weaknesses: Strength: 

1. It seems novel to utilize policy structure to improve offline RL algorithms. 

2. The paper presents its idea reasonably clearly. 

Weaknesses. 

I will keep my comments short. I have several critical concerns below. 

Motivation. The motivation for using mixture distribution is not persuasive. The paper currently says the offline data may be multimodal, and a unimodal distribution policy can result in OOD actions, as shown in Fig 1. But the mixture distribution could also assign higher density to somewhere between two modes (i.e., Fig 1(c), somewhere between two regions could also have high density). I don’t see a clear benefit of using a mixture distribution.  

Lack of insight into the proposed approach. The proposed method has two key components: one is the use of a mixture distribution policy; another one is to use variational lower bound to learn the policy parameters. I expect strong empirical results to show the necessity and effectiveness of the two components. Particularly to show at least the following: 

1. The benefit of the proposed algorithm is not due to a more complicated NN. There is a reason to believe that a very large/complicated NN could be beneficial: it is less likely to touch another action’s density when updating one. So probably, the OOD action’s density can be kept low in the learning process. Why not use a mixture of Gaussian as a policy parameterization (with a relatively large NN) and plug it into an offline RL algorithm as a baseline? 

2. Why use variational lower bound?  

Technical clarification. Eq (3), why such a definition of z? Does it mean pi_gate is also deterministic? 

Empirical results: 
Why is there no implicit Q learning? IQL empirical works really well and should be included. 
clarity,_quality,_novelty_and_reproducibility: See above. 
summary_of_the_review: Due to the concerns about the motivation, insight of the proposed method, and empirical results, I tend to reject the paper for now.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel algorithm for offline RL that works by dividing the state-action space into discrete latent regions, learning a deterministic policy for each, and acting with a mixture of these policies. The authors demonstrate the effectiveness of their method on a series of experiments from the D4RL benchmarking datsets and argue that it helps avoid issues with out-of-distribution actions.
strength_and_weaknesses: # Strengths
The paper is well-written and well motivated, and the algorithm is novel (as far as I can tell).

I also wanted to highlight the comparison between the original results of AWAC and the authors in Appendix E (although more to say about this point below).

# Weaknesses
The main weaknesses of this paper are in evaluation.

## Choice of $|\mathcal{Z}|$
The first point is in the choice of the cardinality of $|\mathcal{Z}|$, which is crucial to the algorithm. I feel that the importance of this choice is not highlighted enough (the first time I saw it discussed was in section 7.2 in page 7). In Algorithm 1, for instance, it is never mentioned that $|\mathcal{Z}|$ needs to be selected.

I appreciate that the effect of the dimensionality choice was evaluated to some extent in section 7.2, but I think this is an analysis that needs to be performed more extensively, as it is one of the core components of the proposed method. Some questions that were left unanswered for me:
* How does one pick $|\mathcal{Z}|$?
* How does one know if it'd be better to pick cV2AE instead (where $|\mathcal{Z}|$ doesn't have to be picked)?
* How sensitive is the algorithm to $|\mathcal{Z}|$ in general?

## Implementation details
In section 7.2 the authors say "For TD3+BC and CQL, we used the author-provided implementations." This makes for a comparison I do not trust. Implementation matters and there are often subtle design choices that have a big impact on performance. For proper "apples-to-apples" comparison, you should really be using the same codebase, otherwise the results are suspect. Furthermore, it seems you used your own implementation of TD3+BC for the toy experiments... Why did you not use the same one for the D4RL benchmark tasks?

Indeed, right below table 4 the authors say: "Remarkably, our implementation of AWAC showed significantly better performance than that reported by Nair et al. (2020)." This is precisely why it's not good to compare across implementations.

## Statistical significance
In the tables presented there are bolded numbers even when improvements are not statistically significant: you should only be bolding numbers when there are no overlapping CIs, which is not the case for most of e.g. Table 3 and Table 5.


# Questions / suggestions
1. In equation (2), it seems $\mathcal{Z}$ has to have a pre-specified size. How is that chosen?
1. In equation (5), to the right of the inequality I think it should be $d^{\beta}(\mathbf{s})$
1. In equations (7) and (10) the $\mathbf{z}$ is not qualified. Are you missing a sum or an integral over $\mathcal{Z}$?
1. At the end of equation (10) shouldn't it be $\mathbf{s_i,a_i}$?
1. In Algorithm 1 you should have $\lbrace\ldots \rbrace$ surrounding $(s, a, s', r)$ to indicate its a set of transitions. You should also have a for loop or something like that saying "For each element $(s, a, s', r)$ in the mini-batch...
1. In the second expectation in equation (12), shouldn't it be $a\sim\pi_{z'}$?
1. In the paragraph above equation (10) it should say "In addition, we also propose a regularization" (not an)
clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear. As far as I can tell, the idea is novel and (mostly) reproducible, modulo the implementation details mentioned above.
summary_of_the_review: This paper presents an interesting and novel algorithm for dealing with out-of-distribution actions. The writing, motivation, and algorithmic description are clear.

My main concern with this paper is in the evaulation of the method, sensitivity analysis of one of its core components, and comparison to baselines (see above for details).

For these reasons I am leaning towards an accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a more expressive policy for offline reinforcement learning. In particular, the paper proposes to use a mixture of deterministic policies.
strength_and_weaknesses: # Strength
* More expressive policies are an interesting direction of offline reinforcement learning research.

# Weaknesses
* The approach lacks novelty. A VAE-style policy learning scheme has already been explored in [1]. While [2] proposes more expressive autoregressive policies. 
* The paper lacks comparisons to the current SOTA results. In particular, [1] and [3] achieve significantly stronger results on the antmaze dataset, which are considered to be more challenging for offline reinforcement learning [4].
* The advantage scaling term (15) relies on the minimum and maximum over minibatches that might be sensitive to outliers.

[1] Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, Chongjie Zhang

[2] EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu

[3] Offline Reinforcement Learning with Implicit Q-Learning
Ilya Kostrikov, Ashvin Nair, Sergey Levine

[4] RvS: What is Essential for Offline RL via Supervised Learning?
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine
clarity,_quality,_novelty_and_reproducibility: In general the paper is easy to follow. However, it lacks novelty.
summary_of_the_review: The paper lacks novelty and omits comparisons to the current SOTA. For these reasons, I believe that they paper falls below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work aims to draw attention to the issue that there may be multiple “modalities” to the behavior data collected for offline RL and it is hypothesized that it is harmful to force a single policy to learn from the data from all those different behavior policies. 

The paper proposed to use V2AE (a VAE inspired method) to separate out the different modalities in the behavior data and use one policy for each of those modalities. They propose a gating policy that selects the latent modality z with the maximum Q value, which in practice is approximated with a softmax distribution inthe variational lower bound. In addition, the authors found that by adding a MI regularization term that encourages diversity of the latent modality per (s,a) pair, they got improved performance. 

In terms of the experiment results, the authors found that their proposal does well on a toy problem where there are two potential goals that the agents can head to. Existing methods exhibits hesitation around forks in the path towards those two goals, whereas the V2AE method solves it perfectly. More encouragingly, such gain is still exists for AntMaze, where the problem setup can be seen as a more complicated version of the toy problem. 

strength_and_weaknesses: 
The paper pointed out an often ignored factor of offline RL, which is that the dataset may come from multiple behavior policies and forcing one policy to adapt to all of them at once may make learning harder. The proposed solution is mathematically grounded and the motivation is clearly shown in the toy problem. The authors also illustrated the connection between policy structure and the critic loss. All of those points above make this paper a strong candidate. 

There are also some weaknesses of the paper. One is that it is unclear what exactly caused fitting the multimodal distribution hard and whether it is a fundamental problem or something that will automatically go away with more advanced network architecture with stronger regularization, without explicitly modeling the latent z. Another area that can be improved is the clarity. (see the next section)

clarity,_quality,_novelty_and_reproducibility: 
The individual techniques used in this paper are not novel (e.g. VAE, modeling clusters of data, MI, softmax distribution given by the Q values) but seeing them combined together in the context of offline RL is.  This aspect makes this work original. 

The paper would benefit from some further clarifications. For example, some details in the equation 7 is not immediately obvious (e.g. why is the expected value in the second term over p(z) not p(z|s_i)?)  and would merit a derivation in the appendix.  Another example is eq 10. I understand how the equation follows from Barber & Agakov (2003) and for the need of an auxiliary distribution. But how is g(z|s,a) connected with q(z|s,a)? Similarly in Algorithm 1, is the actor and the posterior updated in the same gradient step? 

Another clarification question I have is whether for each value of z, you have a different actor network, or do they all share the same network, and the value of z is just a part of the input to the same actor network? Afaiu, it is the latter. Let me know if I am wrong.


summary_of_the_review: I recommend a score of 6 because this paper pointed out a unique perspective to the failure modes of offline RL methods and proposed a preliminary approach that combines various other areas of ML to solve it. It is valuable due to the novel perspective and to the sound motivation as demonstrated in the toy problem. Whether there are easier solutions to the current approach is still yet to be seen, and the clarity can be improved. That is why I did not give a higher score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes using a mixture model for the target policy in offline reinforcement learning. There are similar ideas in the literature, including LAPO presented at NeurIPS this year. The main difference is that this work proposes using the mixture of deterministic policies. The main body of the paper needs to be rewritten to more thoroughly cover the related work for motivate and appreciate the proposed method, specifically a detailed comparison and discussion on why we should better choose the mixture of deterministic policies over other mixture policies. The lack of comparison to related work is undermining the novelty of the paper. 

Related to the above point, it seems that the actual benefit might be coming from the reduction of the variance using determinisic policies, yet making them more expressive by taking the mixture of them. If so, this would also require rewriting the narrative in the paper. 

justification_for_why_not_higher_score: The specific idea - the mixture of deterministic policies - is not well motivated over other mixture models. Why and when should we pursue this architecture over other mixture models? Is the benefit specific to the dataset used in the experiments or does it translated to other datasets and domains? These questions are quite important to be addressed in the main body of the paper.
Decision: Reject