Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a regularized policy learning algorithm for offline reinforcement learning. The implicit policy is trained by a GAN-like framework, and the regularization loss constrains the distance between learned policy and behavior policy. Experiments and ablation study on the D4RL dataset validate the proposed framework and algorithmic designs.
main_review: Strengths:
1. The idea of learning policy with latent/implicit structure can be promising in offline RL as the latent structure might suffer less from the distribution shift. The paper borrows some method from GAN training which is novel to offline RL and show they are effective.
2. The paper proposed a number of practical enhancement that can be generalized to a large range of offline policy learning algorithms in section 4.2. These methods are well motivated and tested in the ablation study. I think this study is helpful for many offline RL work.

Weakness:
1. The theory in section 3 is weak and closely related to many previous results. In Kumar et al. (2019), Theorem 4.2 also showed a bound on the visitation ratio when the policy distance is small enough. The results is very similar to this in some sense. However both of them are not really enough to support the claim that it is sufficient to require the policy distance to be small to bound the visitation distance. The key part is in k_max. Assuming policy distance smaller than 1/k_max which is probably requiring some exponentially (in horizon) small number, and cannot hold in any meaningful scenarios. In the end, any constant policy difference can still propagate exponentially to the state visitation distance.
2. The key algorithmic contribution, $L_g(\phi)$, seems vague in some sense. Since the state distribution is fixed, and the state samples in x and y are two i.i.d. samples, the loss can be decoupled into two parts: Construct another example $x' = (\tilde{s}, a')$ where $\tilde{s}$ is the same as in $x$ and $a' \sim \pi_b$. The the distance between $x'$ and $y$ is a constant, then the regularization loss is actually optimizing the conditional action distribution between $\pi_\phi$ and $\pi_b$, which seems not so different from previous work, and it is not so clear why the form in paper is more promising.
summary_of_the_review: Overall, this paper proposed a novel algorithmic idea in offline RL. I think the algorithmic contribution in this paper can be helpful to the community. Though some theoretical claim/results in the paper needs to be better explained.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes offline RL algorithms which consist of several key components: (1) use implicit policies instead of Gaussian policies (2) regularize the state-action visitation distribution instead of the policy. The paper provides a theoretical analysis to show the equivalence between regularizing policy and regularizing state-action visitation, and provides an empirical study on several deep RL environments. 
main_review: Strengths
1.	The paper is well written and easy to follow. It provides good motivation for using implicit policies and regularizing the state-action visitation distribution. The visualization in Figure 1 is also helpful. 

Weaknesses
1.	The theoretical analysis assumes $T_{i,j} > 0$. Although the paper claims that the assumption can be satisfied by substituting the zero entries with small numbers, doesn’t that change $||d_\phi – d_b||_1$? Does that bound still hold (how much error would be introduced if we substitute all zero entries)? 
2.	I don’t understand the purpose of the theoretical analysis. The paper mentions that the goal is to show that “this line of prior work is compatible in theory with our proposed framework”, but it seems like the theory just says that regularizing policy implies regularizing the state-action visitation. Is that what the paper trying to show or do I miss anything here? Moreover,  I think similar analysis has been done in the existing works (e.g., Appendix A and B in [1] or Section 4.2 of [2]). Can you explain how the analysis in the paper differs from the existing works? What is novelty in the theoretical analysis in this paper?  
3.	In the empirical section, the paper mentions that “we do not test on the random and expert datasets as they are less practical”. However, I think the point of the empirical study is to understand how the proposed algorithm performs compared to baselines, not how practical the algorithm or the dataset is. Therefore, I think the empirical results would be clearer with all dataset reported. 
4.	The results in Table 1 do not show confidence interval for baselines and the average scores. I find it hard to see a clear conclusion from the table. For example, for many tasks, it is not clear whether the proposed algorithm outperforms baselines significantly or not. Moreover, showing the proposed algorithms have higher average scores seems pointless since the scores are not normalized (please correct me if I am wrong). It is possible that one algorithm wins on one task with a big margin and loses on other tasks with a small margin to have a higher average score. Is this algorithm better?
5.	One of the main points of the paper is about regularizing the state-action visitation distribution.  However, in Section 4.1, the paper uses the state distribution from the dataset to approximate $d_{\phi}(s)$. Doesn’t the algorithm basically reduce to policy regularizing algorithm by doing this approximation? Approximating the state-action visitation of a target policy is still an ongoing topic in the OPE community, so I don’t think it is okay to just say that we can use the dataset to approximate the state distribution. This is my biggest concern of the paper. 

[1] John  Schulman, et al. “Trust region policy optimization.”  
[2] Sergey Levine, et al. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”

summary_of_the_review: I am leaning towards a recommendation to reject this paper since the proposed algorithm does not properly regularize the state-action visitation, and the theoretical and empirical results do not support the main points of this paper as mentioned in the Main Review section. 

---
After the author response, I think the paper provides a good empirical study on state-action joint regularization methods, so I increased my score for the empirical novelty and significance. However, my main concern remains so I am keeping my overall score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: One of the existing Offline RL algorithms is to constrain the learned policy, such as constraining the learned policy to be consistent with the behavior policy itself or the action distribution based on state conditions, or adopting a Gaussian policy. However, in any given state s, the potential action value function in the action space may have multiple local maxima. Therefore, only one point can be used to evaluate whether the current policy is close to the behavior policy in any specific state, which may not well reflect the true difference between the two conditional distributions, and the stochastic policy that leads to determinism or unimodality may only capture one of the local optima leads to ignoring many other high-value actions. Especially when such policies such as CVAE and other models exhibit strong model-recovering behavior, large probability density may be assigned to low data density areas, resulting in exaggerating the density of high-value actions.

In response to the above problems, the method proposed in this article:
- Uses an implicit policy to capture the multi-modality of the action value function
- Controls the <s,a> access frequency in the data set to be as consistent as possible with the <s,a> access frequency of the learned policy as an additional training target, without the need to explicitly construct a behavior policy
- provides the theory that proves, the method of matching behavior policy and current policy is equivalent to matching their corresponding <s,a> access frequency

main_review: $\textbf{Strengths:}$
&nbsp;

Method
- This direct matching state-action joint access alleviates the problem of uncontrollable extrapolation errors in the estimation of the Q-value function, because the proximity of the access frequency leads to a decrease in the chance of estimating the Q-value of the state-action pair far away from the offline data set.
- Since the state-action pairs in the offline data set are all regarded as samples from joint visits, the problem in the policy matching method that fits the distribution of each state-conditional action on only one data point can be avoided.
- This method implicitly encourages the smoothness of state-action mapping, that is, similar states should have similar actions. This smoothing feature helps to ensure that the policy is reliable when promoting to unseen states. 

Experiments
- The authors show the effectiveness of implicit policy, state-action-visitation matching, and their full algorithm in the experimental results. I believe the most valuable experiments are the toy experiments to illustrate their motivations as the findings in the toy experiments may help the policy constraint methods that need to recover the behavior policies to improve their methods further.

Writing
- The paper is well structured and clearly written. I enjoy reading it. 

&nbsp;

$\textbf{Weaknesses:}$
&nbsp;

Experiments
- The experimental results seem not very prominent. One reason may be because the experimental setting does not highlight the proposed method’s advantages as there are too few modalities in the used D4RL offline data. I suggest the authors to verify their method on datasets with more modalities to further discover the method’s full potential.


summary_of_the_review: This paper developed a framework that supports learning a flexible while well-regularized policy in offline RL. The technical novelty is sufficient and significant. The empirical results show the method reach satisfied performance on D4RL while I'm looking forward to seeing potentially better results on datasets with more modalities. Overall, I recommend to accept this paper.
review_title: null

Meta-review: The authors propose to use implicit policies (similar to a conditional GAN) with a GAN-inspired regularizer. Theoretically, they show an equivalence between policy-matching and state-action-visitation matching. Finally, they evaluate their approach on D4RL and showed improved performance as well as ablations.

Reviewers did not find the theoretical contribution to be significant.  While the exact form may be novel, the general result has been shown in previous work and they only use the general result as a loose motivation for their approach. All reviewers acknowledge their empirical improvements as the primary strength of the paper. While a central component of their story is joint state-action regularization, Reviewer Ht1b identified that their proposed approach does not appear to directly regularize the joint state-action distribution, but rather behaves more similarly to existing policy constraint methods. I agree with Reviewer Ht1b and after much back-and-forth discussion (both Reviewer Ht1b and myself) with the authors, I have not been persuaded otherwise.

The paper has a lot of potential - strong empirical results, but the justification and explanation of the method needs to be rewritten in light of the policy constraint regularization or a stronger argument needs to be put forth in support of joint state-action regularization. I don't think this diminishes the results though, but without this substantial revision, I cannot accept the paper at this time.
Decision: Reject