Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: 
The author presents Supported Policy OpTimization (SPOT), a policy constraint method to offline RL built upon an off-the-shelf off-policy RL algorithm. SPOT introduces a pluggable regularization term applied directly to the estimated behavior density. The experiments are conducted in the D4RL benchmark and several SOTA baselines are taken for comparison.

strengths_and_weaknesses: 
Strengths:

1. the article is overall well-written and easy to follow.
2. The motivation of the article, simple and pluggable offline RL, is meaningful in real applications.
3. The solution is simple to implement and reasonable.


Weaknesses

The weaknesses come from the experiment designs, which are unclear to me. I have listed the related questions for the authors to qualify it.

questions: 1. In figure 1(b), why should we compare the performance of SPOT relative to the baselines under the same constraint strength? And why SPOT can achieve better performance with the same constraint strength since the related evidence cannot be found in the current theorem. btw, I think a meaningful comparison is that SPOT can easily adjust the degree of constraint by controlling \lambda, while other solutions do not do this well. Also, why are the other baselines only report a few points (e.g.,  In hopper-medium-replay-v2, PLAS only has one point)?
2. In figure 2, why does the effect become worse when using Gaussian, especially in Walker2d-medium and Hopper-medium. The policies of these datasets are Gaussian distribution, so Gaussian should be the most accurate modeling.
3. In the last paragraph of Section 5.2, you mentioned that the TD3 method is not as stable as SAC, especially in the Ant environment, and Figure 2 illustrates the problem. So, is it possible that the performance improvement of SPOT reported in Table 3 comes from our algorithm choice (TD3 instead of SAC) rather than our constraint mechanism? For a fair comparison, an ablation experiment may be required here.
4. In Section 5.3, the author compares the performance improvement of SPOT and IQL under the online fine-tuning setting, and the author claims that the SPOT algorithm can be online fine-tuned seamlessly. This point of view is a bit confusing to me: if I have an online reinforcement learning algorithm, then, for any offline reinforcement learning algorithm, I only need to pass the parameters of the optimal policy solved offline to the online reinforcement learning algorithm as the initialization parameters of the policy, and then I can achieve “fine-tuned seamlessness”. Is this something only SPOT can do? Further, I think the comparison with IQL is unfair. The author directly uses the constraint mechanism of IQL for online training, while the SPOT algorithm reduces the constraint coefficient. Since conservative updates are unnecessary during online training, this obviously suppresses the ability of IQL-based online-trained policies.
5. In Section 5.4, the authors compare the computation cost of inference of several algorithms and illustrate the inference speed advantage of SPOT. In my opinion, the advantage of SPOT's inference speed should exist in many algorithms, such as some algorithms based on Pluggable Regularization (such as CQL, BRAC, etc.) and most model-based offline reinforcement learning algorithms (such as MOPO). Is my statement correct? If so, I think the description in Section 5.4 is a bit overclaimed: we shouldn't consider it a SPOT-exclusive feature.


limitations: I will consider increasing the score if the authors clarify the above questions or give a better experiment design to demonstrate the proposed method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes supported policy optimization (SPOT), which introduces a "pluggable" regularization term applied to an estimated behavior policy. In the implementation, following VAE, the authors use the ELBO to estimate the behavior policy, and following TD3+BC, add a normalization term to the policy loss. The authors emphasize that SPOT is computationally efficient at inference, obtains excellent performance, and enables effective online fine-tuning. 
strengths_and_weaknesses: Strengths:
+ This paper is well-written.
+ The proposed method is simple and intuitive. 
+ The authors conduct extensive experiments to show the effectiveness of SPOT. 

Weaknesses:
+ The paper lacks theoretical analysis.
+ The method is somewhat incremental. 
questions: 1. The authors state that "our method benefits from a closer connection between theory and algorithm". What does the author mean by "theory"? 
2. I think there is no essential difference between Equation 4 and Equation 3. Furthermore, the authors approximate Equation 4 and get Equation 5. Is the theoretical guarantee still satisfied? 
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper present a regularization method for offline reinforcement learning called SPOT. The main idea is a policy constraint inspired by support constraint on behavior policy. The method is evaluated extensively on standard offline RL benchmarks and achieves state of the art result.
strengths_and_weaknesses: Strengths: The idea of support constraint looks simple and effective as in Sec.4.1, and the empirical result as shown in Figure 1 and Table 2 demonstrate its effectiveness for practical problems. From the empirical result, this work achieves new state-of-the-art results on standard offline RL benchmarks and is beneficial to the related communities.

Weakness: From my perspective, the effectiveness of the proposed algorithm is mainly demonstrated by empirical experiments. It would be more complete if the empirical improvement over previous methods can be justified with some theoretical analysis.
questions: A minor suggestion is on the symbols in $\log\pi_\beta(\pi_\phi(s)|s)$ (4), as $\pi_\beta$ is the probability $\in [0,1]$ while $\pi_\phi$ actually is the action $\in \mathcal{A}$. It may cause some confusion at first glance.
limitations: The authors have addressed the societal impact of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the offline Reinforcement Learning setting, and introduces a loss that allows any off-policy RL algorithm to learn from offline data. The loss addresses the issue of out-of-distribution actions being sampled by the actor being trained. The loss is obtained by first training a (conditional) Variational Autoencoder on state-action tuples. Then, its ELBO loss (used for training the VAE itself) is used as part of the actor training, to keep in close to the support of actions in the dataset. The critic training loss does not seem to have to be modified (Equation 1 is untouched), and leverages the actor to sample actions (so, indirectly, it samples actions in the support of the dataset).

An empirical evaluation in challenging environments shows that the proposed method, SPOT (the loss described above + TD3) outperforms all the baselines being considered. The results are sometimes moderate, sometimes much more significant.
strengths_and_weaknesses: The proposed method is easy to implement, and indeed does not need to change fundamental aspects of an off-policy RL algorithm to make it offline.

Strengths (quality, clarity, somewhat the significance):

* The paper is well-written and easy to read. It flows naturally
* The proposed method is sound, and its derivation from constraint-based optimization is intuitive and motivates the resulting loss well.
* The empirical evaluation is thorough, with the impact of hyper-parameters being well-studied. The environments being used are challenging (not just toy environments), and many baselines are considered.
* Source code is provided! This is a rare event, and greatly helps reproducibility and answering questions such as "does the gradient flow to the actor through the VAE?" (the answer is yes). The code is also clean, self-contained and easy to understand.

There is no big weakness in this work, even though its novelty is maybe a bit lower than the other aspects listed in Strengths. While the proposed method is novel, and quite distinct from the existing literature, it is still using a VAE in combination with some off-policy RL algorithm. BCQ uses the VAE to sample actions (while this work uses the VAE as part of the actor loss). BRAC estimates the behavior policy and uses the KL divergence between the actor and the behavior policy to regularize learning. BEAR is inspired from constraint-based optimization like this work, but uses the maximum mean discrepancy to keep the actor close to the actions in the dataset. So, overall, the originality of this work, albeit satisfactory, cannot be stated as a strength.

Minor comments:

* Section 2 in the appendix is quite interesting, and if room can be made in the paper, would deserve to be in the main text
* The description of how $\lambda$ decreases over time in the online fine-tuning setting is interesting and should also be in the main text (it is in the appendix currently)

**Author response**

The authors mention that they will improve the conclusion of the paper and extend the main text, which were my two minor comments. The other reviewers also seem to be leaning towards acceptance, so I maintain my score of accept.
questions: I don't have any question for the authors
limitations: There does not seem to be any negative societal impact of this paper that should be discussed. The proposed algorithm outperforms the baselines in every case, and is overall strong. As such, the lack of a "Limitations" section, for instance, is not problematic for this paper. A mention of possible areas of future improvement would have been nice in the conclusion, that for the moments sounds a bit like an advertisement.
review_title: null

Meta-review: metareview: This work presents an interesting idea of constraining the policy network in offline reinforcement learning (RL)  to not only be within the support set but also avoid the out-of-distribution actions effectively unlike the standard behavior policy through behavior regularization.  
The proposed Supported Policy OpTimization (SPOT) method leverages the theoretical framework of density-based support constraint. and adopts a VAE-based density estimator to model the supports of behavioral actions. Such a simple method indeed allows effective density-based regularization and can be flexibly be combined with most standard off-policy RL algorithms. Experiments also show that the propose algorithm achieves better performances than SOTA offline RL methods. All the reviewers think that the paper is written carefully, with the ideas explained intuitively, and algorithms tested extensively to showcase the effectiveness of SPOT. Therefore the consensus is to accept this paper for publication at NeurIPS22.

confidence: Certain
Decision: Accept