Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Disclaimer: this paper was assigned to me as an emergency review paper, so I might be missing something important.

### Evaluation
I recommend rejection. I think the paper presents an interesting idea to regularize policy updates by the variance of a return. Besides, it proves that the return-variance-regularized policy update leads to a monotonic policy improvement, which I think is novel. That being said, the paper seems to have several issues that decreased my evaluation. First, the clarity of the paper is really low: many typos, ambiguous notations, and confusing presentation of a new algorithm. Second, I could not understand some parts of the paper, especially a proof of the monotonic policy improvement theorem. Third, it is unclear why the proposed regularizer is suitable for offline RL.

### Paper Summary
Offline RL is (said to be) a key to enable RL applications to real world problems. The paper proposes a new algorithm, which regularizes policy updates by the variance of return, for offline RL. The paper proves a monotonic policy improvement when the return-variance-regularized policy update is used. Experiments show moderate performance gain by the regularization.

### Strong Points
1. Interesting idea to regularize policy updates by the variance of a return
2. Moderate performance gain, especially when offline dataset is obtained by a suboptimal policy.

### Weak Points
1. The paper is not well-written. It contains typos, unclear sentences, and ambiguous notations.
2. Some parts of the paper, like the proof of the monotonic policy improvement theorem, seem to contain mistakes. (I might be wrong, though.)
3. The performance gain seems to be moderate, despite a high complexity of the computation of the proposed regularizer.

### Comments to the Authors

If I am missing something, please feel free to let me know. As noted above, I could not spare much time to review the paper.

1. The paper is not well-written. Please revise it again. (I don't point out all ambiguous notations and typos, but later I point out some serious ones.) Besides, please revise the references section. It refers to arxiv versions of papers that were accepted to conferences.
2. In page 3, $\omega_{\pi / \mathcal{D}}$ suddenly appears. What does it mean? Maybe $\omega_{\pi / \mu}$?
3. What does $s \sim \mathcal{D}$ mean? Does it mean $s \sim d_\mu$? Since the dataset $\mathcal{D}$ contains states visited by $\mu$, simply drawing states from $\mathcal{D}$ will be different from $d_\mu$.
4. What is $d_{\mathcal{D}}$ in, for example, Equation 4?
5. The idea to use the Fenchel duality to avoid double-sampling problem seems to be not new (cf SBEED paper). While the paper mentions AlgaeDICE as an algorithm using a similar technique, it does not mention SBEED about the use of Fenchel duality. Why?
6. In the beginning of Section 3.4, the min-max problem is being solved by repeating the inner minimization and outer maximization. As far as I remember (I might be wrong, though!), this way of solving a min-max problem might not find the exact solution. Isn't it a problem?
7. In Equation 6, it seems that $Q^\pi (s, a)$ is rewritten as $E_{(s, a) \sim d_\mathcal{D}} [ \omega (s, a) r (s, a)]$. According to the notation of the paper, isn't $E_{(s, a) \sim d_\mathcal{D}} [ \omega (s, a) r (s, a)]$ be $(1-\gamma) E_{s_0 \sim \beta, a_t \sim \pi} [\sum_{t=0}^\infty \gamma^t r(s_t, a_t)] \neq Q^\pi(s, a)$?
8. Is Lemma 1 different from Lemma 1 of [Bisi et al, 2019](https://arxiv.org/pdf/1912.03193.pdf)? Also what is its meaning? Why is is useful to understand the variance regularizer?
9. How is the beginning of Section 4.2 related to Theorem 1? Theorem 1 seems to be derived based on a different inequality in its proof.
10. As far as I remember, the dual form of the total variation is $\sup_{f \in C_`1} E_{A \sim P} f(A) - E_{B \sim Q} f(B)$, where $C_1$ is the space of all continuous functions bounded by $1$. Therefore, we don't need $\phi$ and can explicitly state the space of $f$ in Equation 12. Am I wrong or missing something? 
11. Why is there no sup over f in Theorem 1?
12. As for Equation 59, how do you get the first line? In addition, do you need $E_{s \sim d_\pi}$? In the second line, the first $E_{s \sim d_{\pi'}, a \sim \pi}$ is sampling an action from $\pi$. Isn't it $\pi'$? In the fourth line, $d_{\pi'}$ changed to $d_\pi$. How is it possible?
13. I don't fully understand what Section 4.3 does. In addition, what are random variables in Theorem 2?
14. I don't understand what the final objective to be optimized is. In Algorithm 1, $J(\theta, \phi, \psi, \nu)$ appears. Is it the same as Equation 6?
review_title: The paper proposes an interesting idea, but it needs revision.

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary:**

This paper proposes a novel algorithm for offline policy optimization. The main idea is to prevent overestimation bias by regularizing against the variance of the importance weighted value estimate. There are two key modifications: (1) using an importance weight from the stationary distribution and (2) using Fenchel duality to introduce a min-max problem to avoid double sampling when estimating the gradient of the variance regularization term. The theory section motivates the use of variance regularization and the experiments show improvements over BCQ when adding the proposed variance regularization algorithm. 

--------------------------------------------------------------------

**Strengths:**

1. The paper provides theoretical motivation for variance regularization. Theorem 2 demonstrates that using variance regularization can be seen as optimizing a lower bound on the true value of a policy. 
2. The algorithm provides a novel way to implement variance regularization. By introducing the $ \nu $ variables, the paper proposes a way to get around double sampling issues in estimating the gradient of the regularizer. 

--------------------------------------------------------------------

**Weaknesses:**

1. There are several technical steps that I am not convinced by. I am not sure if these are mistakes or just statements that require more explanation: 
   1. In the algorithm, $ \nu $ is defined by $ \tilde r $ and $ \tilde r $ is defined by $ \eta $ in equation (7). This definition seems somewhat circular and the paper never clearly explains why this should work. 
   2. The continual updating of $ \tilde r $ seems to make the problem nonstationary for the policy improvement subroutine that depends on the current iterate of the distribution ratio $ \omega_\psi$. The argument made in a remark on page 4 does not seem to sufficiently resolve this issue since even though $ \nu $ has a closed form, that closed form depends on the distribution ratio which depends on the current policy. 
   3. The variance term that shows up in Theorem 1 does not seem to be what is being estimated by $ V_D $ in the algorithm. In light of this, what is the point of Theorem 1 vis a vis the proposed algorithm?
   4. Theorem 2 shows that using the square root of the variance as a regularizer provides a lower bound, but the algorithm uses the variance. What is the explanation for this mismatch?
2. Notation is somewhat confusing and sloppy. For example, $J$ is defined in several different ways, first in equation (1) and then in equation (3). This makes the rest of the paper confusing since it is sometimes unclear which $J$ is being referred to. Another example is that $ V_D $ is defined as a function of $ \pi $ in equation (2) and then a function of $ \omega, \pi $ in equation (3). Equation (4) just refers to $ V $ when I think it means $ V_D$. Also in equation (4) the LHS has no $s,a$ variables but the first two terms on the RHS are a function of $ s,a $. Where do these $s,a$ come from? Is there an expectation missing? In equation (5) it is not clear which $ \omega $ is being referred to. Equation (6) overloads $J$ once more.  Equation (6) could also use a better explanation as to why this is the dual form of (5) rather than just asserting it. The definitions of $ \epsilon_\pi, L_\pi $ are not included in Lemma 2, but are instead in the following paragraph. In equation (12) $ d_\pi $ is denoted by $\beta_{\pi'}$ for no apparent reason. In equation (13), $f$ is supposedly a dual class of functions, is there a sup missing? In equation (15), ${\hat \omega_{\pi /D}}$ is never explicitly defined. This sort of sloppiness pervades the paper and makes it often difficult to understand.
3. The writing is unclear. One major issue is that lemmas are presented with no context or direction. It would be helpful to preface each section with an explanation of where the section is going before presenting technical lemmas. This is a problem for both Lemma 2 and Lemma 3 where it is unclear to the reader what the point of the lemma is until much later in the section. This is also a problem for Lemma 1 which has it's own subsection that does not seem to make any clear claim to connect to the thesis of the paper. The lemmas may be better suited in the appendix rather than the main text of the paper or just require more explanation. 
4. Empirical results are not very convincing. The plots presented in the experimental section seem to show a slight but not large or consistent advantage for the proposed method. Perhaps more worrying, the only type of result reported are the final performance of the algorithms. There is no empirical indication as to how the algorithm is working or whether the variance regulation is indeed having the desired effect of reducing overestimation. Moreover, there is no indication that the Fenchel duality tricks are doing anything useful empirically.
5. Important ablations and baselines are missing. The authors do not include conservative Q learning (Kumar et al., 2020) as a baseline saying that it is too recent to compare. However, this paper came out in June and according to the ICLR reviewer guidelines we are only meant to consider work released in August or later as contemporaneous. So, CQL ought to be included as a baseline and I think generally outperforms the proposed methods. Additionally, while there is one ablation included in the appendix (adding importance weighting to BCQ), more ablations are needed. Specifically, since the main algorithmic innovation of the paper not just using variance regularization, but computing gradients using the Fenchel duality max-min procedure, there should be ablations showing whether this part of the algorithm is indeed necessary and useful. 

--------------------------------------------------------------------

**Recommendation:**

I recommend rejection and gave the paper a score of 4. While there may be a good idea in there, I do not think the paper is ready to publish. A more clear and careful exposition of the algorithm as well as more rigorous theory and experiments are needed.

--------------------------------------------------------------------

**Additional feedback:**

Typos:

- The first sentence of the intro has a space before the period. 
- The references on the top of page 2 seem to be incorrect. "A." is the first initial of the first author, not their last name.
- Throughout the paper there is usually a space preceding each colon. There should be no spaces before each colon.
- After equation two "a as" should be "as a "
- In the first sentence of section 3.4 "leads" should be "lead"
- The paper sometimes refers to the proposed variance regularization as a "framework" (e.g. in the first sentence of the conclusion). It is not a framework, it is an algorithm.


review_title: Review

final_score: rating: 3: Clear rejection
review_content: 
Summary:
This work is based on a recent work of [Bisi et al., 2019], where a per-step reward formulation is presented with some outstanding unresolved problems, e.g., the double-sampling issue and the policy-dependent reward issue. This paper proposes to use the Fenchel duality to solve the double sampling problem and extend it to the behavior-agnostic off-policy setting by leveraging the density ratio estimation technique.

Major concerns:

1 
The derivations in several major equations are WRONG. The objective in Eq 5 is NOT the same as the objective in Eq 6. \E_{s, a ~ d_D}[\omega(s, a)r(s, a)] in Eq 6 is NOT equal to \E_{s~D}[Q^\pi(s, \pi(s)] in Eq 5. 

2. The motivation and empirical demonstration of the variance regularization are unclear.
First, the definition of the variance doesn’t make sense to me. The variance in mean-variance optimization is a long-established term, which refers to the var of the return (either one-step or cumulative). So it is not clear why V_P makes sense without further motivation or reference. Moreover, the variance term defined in Eq. 2 is very weird. It is neither the variance of the return nor the so-termed “variance of the marginalized IS” (since it involves the reward r). By definition, the variance V_D is the variance of d_\pi(s, a)/d_\mu(s, a)r(s, a). This expression involves both \pi, \mu, and r, and its randomness comes from the randomness of (s, a). It is unclear why minimizing this variance is useful.

Second, the empirical results are not convincing. As pointed out by Eq. 3, the overall objective is to achieve a trade-off E[Q] and V_D(\pi) through \lambda. The empirical results, however, do not show this trade-off. Then it becomes unclear where the empirical improvement comes from. I would like to see how changing \lambda influences V_D(\pi). 

3 Several key deductions, which hold in on-policy cases, may NOT be true in this paper’s off-policy setting. For example, Eq . 8. lacks proof. In Bisi’s setting, this holds only for on-policy cases. I don’t think it still holds for off-policy settings. The author needs to prove it.
The MDP setting is unclear. The authors consider an infinite horizon MDP, so what does T mean? The proof of Lemma 1 also seems problematic. First, without a clear definition of T, there is no way to check the proof of Lemma1. Is T a random variable? Second, Eq 33 is wrong. Eq 33 is the same as Eq 24, but the definition of D^\pi is different, so how can they be the same?
Moreover, It is hard follow the inequality in Eq 34. It looks wrong to me. Thee reviewer strongly suggest that the author write it step by step to make it clear? And also, it would be great to show how the products of IS in Eq 34 reduce to the density ratio in Eq 35.

4. Theorem 2, which is the paper's major theoretical contribution,  is obvious and trivial. By definition, the first term of RHS of Eq 16 is exactly J(\pi). So what theorem 2 says is that J(\pi) \geq J(\pi) \sqrt{c * variance of sth}}. This is fairly obvious and does not bring in any insight.

5. The entire Appendix B.2 is wrong, where the 3rd equality (aka, line 2 of Eq. 39) does NOT necessarily hold. The term d_\pi(\theta)’s gradient is not computed at all, and therefore, any results afterward are not correct.

6. There are some missing references as well. Using Fenchel Duality to solve the double-sampling issue in mean-variance optimization using variance as regularization has been solved by previous literature, e.g., Xie et al., (2018) and Zhang et al., (2020). The author should acknowledge this. Also, I encourage the authors to compare with them. Especially I think the authors may want to compare with Zhang et al., (2020). Algorithm 1 is very similar to the offline MVPI in Zhang et al., (2020). There is only a slight difference in computing the augmented reward.

Xie, T., Liu, B., Xu, Y., Ghavamzadeh, M., Chow, Y., Lyu, D., & Yoon, D. (2018). A block coordinate ascent algorithm for mean-variance optimization. In Advances in Neural Information Processing Systems (pp. 1065-1075).
Zhang, S., Liu, B., & Whiteson, S. (2020). Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning. arXiv preprint arXiv:2004.10888.
review_title: The contribution is limited, and there are several technically unsound places

Meta-review: The reviewers are unanimous that the submission does not clear the bar for ICLR.
Decision: Reject