Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
Summary: this paper proposes a new regularized policy optimization (PO) method which is based on Wasserstein distances. Its idea is to use SGD to optimize the dual form of the WD, then used in two different policy search approaches TRPO and Evolution Strategies. The evaluations are carried out on a variety of control tasks from OpenAI Gym. 


Overall, the paper studies an interesting problem in RL. The idea is somewhat interesting. The experiment results also look promising. However, the writing sometimes has unclear descriptions, probably because there are many things packed in this paper. I have some following concerns about it. 

- The idea of using behavior embedding is new in PO. The description in section 3 is a bit unclear. There are lacks of motivation why in the paper there needs BES/BEM, the interplay of policies, trajectories, and embedding maps, and why non-surjective of BEM is still fine in this case, etc.. In addition, the definition of state-based, action-based, reward-based assume only discrete domains? Besides, they seem not to reappear in other places in the paper. 

- Does the choice of the functions \lambda in 4.2 as a function in RKHS, especially when approximated as a linear function with random Gaussian features, limit the representation power of the embedding space? 

- What is the effect of \beta when positive vs. negative?

- Experiments: The proposed methods show a lot of potentials, but the description in this section is sometimes unclear. Is TRPO with KL divergence the standard algorithm defined without using BEM or with BEM? Then one can wonder how more ablations can be added, e.g. BGPG with KL divergence, TRPO with WD distances are compared with the proposed algorithm. In addition, how BGPG is compared with other related work that also uses skill/policy embedding, instead of a flat PO approach like TRPO. Similar questions are also applied to the experiments for BGES.

- It would also be more self-contained if the paper includes experiment settings.

- The paper also comes with theoretical results. The author could also consider mentioning one key result in the main paper, instead, everything is put in the appendix.


* Minor comments:
- The KL definition on page 4 use rho instead of \xi


- are u and v in Eq. 3 functions in \cal C(X) and \cal C(Y), respectively?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work explores two uses of Wasserstein distances (WD) within reinforcement learning: the first is a variant of policy gradient, where WD is used to guide the policy search (instead of alternative such as Trust-region used in TRPO); the second is a variant of evolutionary search where WD is used again to guide the policy updates.

One of the strengths of the work is to clarify the notion of Behavior embeddings (Sec.3), which I expect can have several uses in RL.   In this paper, the behavioral embeddings are assumed to be given; it would be interesting to discuss/explore learning these embeddings.

Section 4 of the paper reviews key concepts related to WD.  This is much harder to follow for an RL researcher, and would be improved by adding some intuition relating the material presented to the concepts of Sec.3.  Furthermore, this confusion carries out in Sec.5.  For example, what is the best way to think of \lambda_1 and \lambda_2?  And the maps s_1 and s_2?  What are necessary/desirable properties of P^\phi_b?   There are also many steps packed in Alg.2 & Alg.3, which are difficult to unpack.  For example, what are the \epsilon (step 1., Alg.3), scalars or vectors, how are they sampled?  It would be helpful to have a discussion of the complexity (both data & compute) of both algorithms.

Section 6 presents empirical results for each proposed algorithm.  Corresponding baselines are presented, but I would be interested to see a wider set of baseline methods. The literature is rich with methods in these classes, both variants of TRPO and ES.  It’s necessary to at least pick a representative sample to show and compare (e.g. GAE, SAC).  I am also puzzled by the actual results presented, for example the Hopper reward shown in Fig.3 seems much worse (by orders of magnitude) compared to that reported in the SAC paper (Haarnoja et al. 2018).



review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary

This paper proposes using Wasserstein Distances to measure the difference between higher-level functions of policies, which this paper terms as "behaviors". For example, one such behavior could be the distribution over final states given the policy, or the distribution over returns given policy. Through the lens of these behavioral embeddings, this paper recovers a few important special cases that are well-known in the literature including WD-based TRPO and distributional RL. This paper shows that the dual formulation of the Wasserstein Distance gives the ability to score individual policies based on a given "behavioral mapping".

Review

The idea of generalizing the trust-region of TRPO by using the WD measure and behavior maps is intriguing. The paper introduces many choices of behavior maps that could lead towards interesting algorithms that merit additional study. I think the connections between the proposed family of algorithms and WD-based TRPO and distributional RL is highly motivating.

There are, however, a several key issues with the empirical study of the proposed methods that make it challenging to assess the value of introducing another partially understood deep policy gradient algorithm. The results show only behavioral studies of the algorithm, not investigating the effects of the WD based regularizer or the effects of the choice of behavioral map. The results are also significantly limited in their statistical significance, making distinguishing between algorithms difficult in most cases. And the comparison of wall-clock time is particularly difficult to assess due to the uncountably many possible sources of noise when comparing wall-clock time of highly complex algorithms. In the following paragraphs, I will expand upon each of these points.

One of the key contributions of this paper is the ability to define regularizers based on the definition of the behavior map. The paper introduces many such behavior maps, those over state visitation, actions, and returns; however, the paper does not investigate the effects this novel choice has on the results. It is unclear if introducing each of these behavior maps leads towards different results or if they each induce roughly the same final performance of the agent. It would be highly valuable for me to see a more careful study of the effect of choosing each of these behavior maps on a single simple environment, clarifying that this formulation leads to a family of useful algorithms. As it stands, BGPG may only beat TRPO on these domains because it had more meta-parameters to choose between (BGPG introduces many new meta-parameters: choice of behavior map, kernel for produce RKHS, the meta-parameters of that kernel, the meta-parameters of the behavior map like trajectory length, the entropy regularization term in the WD, etc.). Without a careful study, or intuitive explanation of any of these parameter choices, it is unclear if BGPG won simply through overfitting to the problem.

The statistical significance of the proposed algorithm is impossible to assess in the given form. The comparisons are made using only five random seeds and the standard error bars are frequently quite large. I refer to Henderson et al. 2017 for further explanation as to why five random seeds is simply too few to provide a meaningful comparison between algorithms. Instead, I'll discuss a few of the results in particular. In figure 3a, I notice that the proposed method has high variance until it plateaus around -300 reward; why? In each of the remaining plots of Figure 3, I notice that the propose method is significantly higher variance than any of its competitors. This greatly leads me to suspect that the proposed method would have much lower average performance if studied across a greater number of random seeds. In the walker domain (Figure 3d), why do BGPG and TRPO both plateau at the same point for many timesteps, then eventually BGPG starts improving again? From Figure 5, the paper somewhat misleadingly states that "BGES is the only method that drives the agent to the goal in both settings." However Figure 5 on the left clearly shows that BGES and NSR-ES have indistinguishable performance, and the error bars indicate that neither significantly outperform NoisyNet-TRPO. In fact, due to the high variance of BGES it is unclear if it would drive the agent towards the goal on average if run over more random seeds. On the right, NoisyNet-TRPO noticeably outperforms BGES and is significantly lower variance. The language used to describe Figure 6 is strong, stating that Figure 6 "proves that the benefits come here not just from introducing the regularizer, but from its particular form." However, I tend to disagree that Figure 6 reliably proves anything. Although it appears that BGES is significantly outperforming other methods, the number of meta-parameters over which it gets to optimize makes it difficult to say if the performance gain is from overfitting to the problem or the form of the regularizer. Additionally, good performance demonstrated across a single problem is hardly proof especially using only five random seeds (Henderson et al. 2017 Figure 5).

Although this plays comparatively little role in my scoring of the paper, I feel it is necessary to discuss briefly. Because it is impossible to determine the source of the speed differences between the particle approximator and BGPG, the results in Figure 4 are extremely difficult to interpret. It simply could be that BGPG uses slightly more optimized code than the particle approximator, or it could be that there is in fact a significant performance difference between the algorithms. The only true way to discuss performance differences between algorithms generally is through computational complexity for these reasons. In the case that these statistical algorithms share the same computational complexity, then further analysis including convergence rates would be able to shed light on the speed difference. However, wall-clock time vs. performance has so many confounding factors, it is a fairly meaningless unit of measure. For an empirical investigation of speed, I would suggest giving each algorithm similar number of learning steps or similar number of updates to their weights and compare performance in this way.

Additional Comments (do not affect score)

I'm curious if this work could be extended to the off-policy case. It does seem like a minor disadvantage that distributional RL is well-defined in both the on-policy and off-policy cases, but the proposed family of methods is not. From my reading of the paper, it appears that there is little preventing this extension. Is this true?

--------
Edit after reading other reviews and rebuttals.

I appreciate the response noting that the meta-parameters of this algorithm were not tuned. I still feel that understanding the sensitivity to these parameters is important to understand how the choice of these parameters impacts the performance. While it is certainly likely that extensive tuning of these parameters could yield even more improved results, it is also quite possible that the chosen parameters have biased the results. If a more intuitive discussion of the choice of meta-parameters could be included in the paper, I would be more willing to concede this point.

I recognize that 3 or 5 random seeds are a standard in the deep RL community, but I think Henderson et al. 2017 make this point better than I could: this is not a standard that we should hold to. I think that using small domains and smaller networks would allow running across more random seeds and providing a much more careful scientific study. This paper does not convincingly show state of the art performance, a metric that is nearly impossibly to define in a field that moves as quickly as ours, so should not strive to follow the same demonstration study design that SOTA papers follow. A paper with strong theoretical motivations (such as this) should compliment with strong empirical understanding of how that theory translates to practice. Instead, having strong theoretical motivations backed by a demonstration that the algorithm still works is much less convincing (to me). I think a more convincing set of experiments that match well with the intended purpose of the paper (theoretical contribution of behavioral embeddings) would be an investigation to _how_ the optimization affects the learned policies or representations. Instead of performance benchmarks, this would provide understanding for the various effects of the new framework.

After the rebuttal phase, I now have an additional new concern about the clarity of the writing and the completeness of the empirical details. Many of the results do not mention most of the important details necessary to replicate or even interpret the results. I had assumed that details were consistent across figures (e.g. figure 3 uses 5 random seeds, so I assumed the same to be true for all other figures that did not specify), but the author response made clear that this is not true. In additional, little to no details are provided in the paper referring to the choices of meta-parameters for any algorithm or the methodology used to chose these meta-parameters. After these details were given in the rebuttal, my concerns were greatly alleviated, but not completely removed. It is unclear if the authors intend to update the paper before the final version, but based on the language of the rebuttal (e.g. indicating it is purely a misunderstanding of the paper), I am forced to assume that the paper will not include these important details.

To summarize. I think the theoretical underpinnings of these paper are extremely motivating and interesting. I do not want to see these innovations lost. However, the current state of the empirical section of the paper leaves too much open for me to be able to recommend an accept at this time. I do not believe a paper should solve everything in one pass (e.g. careful parameter studies, SOTA demonstrations, real-world demonstrations, etc.), but I do not believe this paper demonstrates any one of these convincingly.
review_title: Official Blind Review #2

Meta-review: The authors introduce the idea of using Wasserstein distances over latent "behavioral spaces" to measure the similarity between two polices, for use in RL algorithms.  Depending on the choice of behavioral embedding, this method produces different regularizers for policy optimization, in some cases recovering known algorithms such as TRPO.  This approach generalizes ideas of similarity used in many common algorithms like TRPO, making these ideas widely applicable to many policy optimization approaches.  The reviewers all agree that the core idea is interesting and would likely be useful to the community.  However, a primary concern that was not sufficiently resolved during the rebuttal period was the experimental evaluation -- both the ability of the experiments to be replicated, as well as whether they provide sufficient insight into how/why the algorithm performs.  Thus, I recommend rejection of this paper at this time.
Decision: Reject