Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces POP3D, an on-policy policy gradient algorithm that is a variant of TRPO and PPO. While TRPO uses a particular penalty function to keep the policy from being updated too aggressively, POP3D uses an alternative objective function that lower bounds the square of the total variance divergence between two policy distributions. The authors argue that this alternative formulation results in an algorithm that is sample-efficient, like PPO, but that is more effective at keeping policy updates from overshooting. The authors also argue that this new formulation helps users to avoid the arguably challenging process of selecting penalty constants, as required (for instance) by TRPO.

Overall, this was an interesting paper. I believe that the idea of lower bounding the square of the total variance divergence is worth being investigated. The experimental results also seem promising, showing that the method may outperform POP and TRPO on a wide range of games. I do have a few questions about the statistical significance of the results, though: some of the score differences may not be significant, the results might be based on too few trials, and the authors only presented mean performances, but not standard deviation or standard error information. I liked the idea that the proposed lower bound solves possible issues arising from using the KL divergence, which is a measure that is not symmetric. On the other hand, after presenting the surrogate objective function's mathematical definition, the authors argued that this new formulation results in better exploration and that it "expands the manifold [of] solutions". I had a hard time following these arguments.

Finally, in my opinion, writing could be improved. A few sentences are hard to read or appear to be presenting incomplete thoughts. For instance:

- "Hessian free strategy: Fisher vector product is utilized to cut down the computing burden." (page 1)
- "(...) the action is taken approximately strongly corrected with the highest probability value." (page 4)
- "Therefore, even if theta outputs theta_old the same high probability for the right action, it's still penalized owing to probabilities mismatch for other uncritical actions." (page 4)
- "From the perspective of the manifold, if the optimal parameters constitute a solution manifold." (page 4)

I believe that this is an interesting work, though possibly still showing only preliminary results. I also think that the paper might be improved by clarifying a few of the authors' key mathematical arguments. As it is, and based on the thoughts presented above (and on the questions below), I would say that more work would benefit the paper and would argue for a weak reject. 

I have a few questions and comments for the authors:

1) the authors state the "[POP3D] dives into the mechanism of PPO's improvement over TRPO by the perspective of solution manifold". Could you please clarify? What is the solution manifold perspective, and how is it used more effectively by POP3D, compared to previous algorithms? At the end of Section 3.3, you present an argument that I found hard to follow: "From the perspective of the manifold, if the optimal parameters constitute a solution manifold.". Later on, you also talked about "[expanding] the solution manifold at least one dimension such as curves to surfaces". I believe that these ideas may be central to the paper. Still, I had difficulty following the arguments in this section.

2) a quick note on notation: \hat{A}_t (Eq6) looks like a random variable denoting the action taken in time t. Perhaps the LHS of this equation is missing parameters specifying a (s,a) pair? In particular, note that the advantage function in Eq6 involves computing delta^V_{t+l}, a TD error for a specific experience; the current notation does not make it clear which experiences are being analyzed. Compare, for example, the definition of advantage used in Eq6 and the one used in Eq3.

3) still regarding notation, I wonder if you could clarify which of s, a, s_t, a_t, are random variables and which ones are realizations of the corresponding random variables. In Eq6, for example, why do you write A(s_t, a) instead of A(s_t, a_t)?

4) in Eq10, please formally define D_{TV}.

5) immediately after Eq11, the authors argue that because the KL divergence is not symmetric, the choice of whether to use D_KL(p1, p2) or D_KL(p2, p1) might affect the algorithm's behavior, and that (for this reason) it might not be the best choice for a divergence measure. Could you please discuss the possible advantages of simply using the symmetrized version of this divergence (https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence)?

6) in Eq12, where is the action "a", on the RHS, coming from? The inputs to D_pp are two distributions, but its definition seems to consider the difference between two particular probabilities. Could you please clarify the notation being used here?

7) regarding your experiments, I wonder if you could discuss whether the score differences observed in Table 1 are statistically significant.

8) your method does seem to perform better in some particular games, but it does not do so well in other games. A discussion/interpretation of these results would be nice: what are, for instance, the properties of the games that are hard for POP3D?

9) regarding Table 3, could you please discuss whether you can draw strong conclusions regarding the performance difference between PPO and POP3D when analyzing just 3 trials? Also, do you have standard deviation information associated with the averages that you could present?

10) one key point that the authors make is that POP3D encourages exploration by "expanding the optimal solution manifold". More efficient exploration is indeed advantageous, but I did not follow the argument being made here. Could you please clarify why POP3D results in better exploration?


review_title: This paper introduces POP3D, an on-policy policy gradient algorithm that is a variant of TRPO and PPO. Overall, this was an interesting work. I believe that the idea of lower bounding the square of the total variance divergence is worth being investigated. The experimental results seem promising, though I do have questions regarding the statistical significance of the results. I also believe that this paper might be improved by clarifying a few of the authors' key mathematical arguments.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors replace the divergence-based constraint in trust region policy optimization model with an alternate distance measure, which is added to the objective function with a multiplier (beta).  In fact, the parameter beta plays a role that is similar to a Lagrange multiplier, if the new distance measure is introduced as a constraint. The authors explain the shortcomings of KL-divergence and the solutions obtained with other methods but they do not provide a sufficient discussion how and why their simple approach overcomes those concerns. For instance, why would the new measure encourage exploration and what is the effect of large beta value on this?
review_title: A simple surrogate objective with almost no insight

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper discusses different methods in robust reinforcement learning including TRPO and PPO then propose a variant of TRPO using a new type of regularizer to quantify the difference between two distributions.

Strength: 
- The paper is easy to understand and provide informative discussion on related methods.
- The point probability distance is simple but appears to work well.
- The numerical experiments are extensive which include discrete and continuous control tasks.

Weakness:
- There is lack of justification why the proposed algorithm converges.
- Although POP3D appears to be better than PPO and others using the mean final scores metric, it seems to be comparable with PPO when using the all episodes mean scores which raise the concern about the clear improvement of POP3D.

I have the following questions and comments:
Q1. As the point probability distance is bounded, there is an issue with the scale effect where different environment can produce different range of the cumulative reward (the first term in the objective). In this situation, choosing a common \beta that works for most environment appears to be challenging to me? How do you deal with this?

Q2. As from Q1, is there a systematic way to select \beta?

Q3. As the KL divergence is the upper bound of the total variation distance while the point probability distance is the lower bound, POP3D does not inherit from the theoretical result. What is your opinion about this?

C1. Equation (10) appears to be different from the one in the TRPO paper? Are you using a different definition of \eta(.) function, it is better to state its definition for clarity.

C2. It would be better to have figures illustrating the performance of 4 algorithms in some continuous tasks over the number of episodes collected to see their learning process.


Small comments: 
- The paper needs to be proofread again for some small typos.
- If the hyper-parameter for POP3D is tuned, it is better if the hyper-parameter for other methods are tuned as well for fair comparison in continuous domain.
- The point probability distance actually depends on the action taken at that iteration. Do you think the formulation needs to be changed to account for that?

Overall, I suggest a weak-reject decision based on the following reason:
- The paper lack theoretical discussion on the convergence of the new method.
- The numerical results do not show significant improvement over existing methods especially PPO as the authors claim.
- The authors claim that the new method removes the need of choosing preferable penalty coefficient while I still believe the parameter \beta needs to be suitably chosen for particular environment.

However, I can adjust my decision once I see the authors’ response.


review_title: Blind Review #4

Meta-review: While the paper contains some interesting ideas, the reviewers felt that overall the paper is not theoretical well supported, and likewise the experiments are not fully convincing. Even after the rebuttal, these concerns still persist.
Decision: Reject