Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes Adversarial Robust Policy Optimization (ARPO) that trains a generator to transfer the observation style so that policy taking different actions, the policy optimizes the long-term reward plus a regularization that minimizes the KL divergence between policy outputs given style-transferred image and original image. In this way, ARPO tries to learn underlying semantic representations for better generalization and being robust to the perturbations. The experimental results on some Procgen envs show improved results compared to PPO and RAD.
main_review: Strength
- The idea of combining style transfer and RL to improve generalization is reasonable. As far as I know, the idea is novel in this setting.  

Weakness 
- There are multiple overclaim in this paper. 1) In the abstract, there is no empirical
nor theoretical evidence that ARPO finds optimal policy. It is highly skeptical that
ARPO can find an optimal policy in practice as the added KL regularization will rarely converge to zero. 
- It might be better to discuss whether the policy will converge to the optimal policy, where KL term in eq 3 reaches zero at the end. 
- An important ablation study could be showing how does the KL regularization change during the
training. 
- The three run averages of results present large variance, and it is not very convincing to me ARPO
really achieves statistically better generalization results. Based on the results in Table 1, 
there is no significant improvement from ARPO compared to baselines.
- There is no discussion nor ablations on whether cycle consistency loss finds the
underlying structure of MDP. It would be better at least to provide examples of style-transferred images compared with original images.
summary_of_the_review: This paper proposes a novel idea that combines style transfer with RL to improve the
generalization performance. The experimental results show the proposed approach is competitive with the existing baselines. The results could be better improved by adding more runs and more ablations that approved the effectiveness of ARPO. More theoretical studies are needed to understand whether the algorithm converges to optimal policy.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
To achieve more robust behavior of reinforcement learning algorithms, in the deep learning part,  confounding features need to be removed from the decision-making process. To achieve this, the paper proposed first clustering the state space into $n$ clusters/domains and use StarGAN to style-transfer from one cluster to another.
main_review: To learn robust features for decision-making, the generator of the StarGAN is encouraged to style transfer the state space such that the action distribution under the new transferred state will be distinct in terms of KL divergence with the original policy. The generator is further trained with a cyclic consistency loss.

The domain discriminator tries to tell about different domains.

Using Proximal Policy Optimization (PPO) as an example, the policy network is trained with an augmented loss of minimizing the KL divergence of the action distribution between original and style-transfered state.


Experiments are done on the Procgen environments in comparison to PPO and RAD(data augmentation). Ablation study is done with respect to different linear coefficients, combining the different losses in the adversarial learning.

# Strong points
- The paper integrate style-transfer into the training of reinforcement learning algorithms to enhance the algorithm robustness against spurious features.
- The experimental setting is explained relatively in detail. Experimental results look promising.

#Issues
1. In section 3.3, GMM is only used in the beginning to generate different clusters for usage in Star-GAN learning. Will this be sensitive to initial sample distribution?
2. Could you elaborate on how the GMM clustering is done? Simply pool all state observations and do the clustering? e.g. for a tuple s,a,s', s and s' are treated  equally?

## Minor issues
1. 3rd paragraph in section 1, there is a citation on Anonymous.

# Questions
1. Could you explain the benefit of the gradient penalty term in Eqn. (6)?

summary_of_the_review: I like the innovative point that policy is not only learned to optimize return but also robustness against style transfer of state observation. Compared to data augmentation methods, the proposed methods showed promising methodology and empirical advantage.
Issues in the above section need to be addressed however.
.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: An algorithm is proposed for using a GAN-like objective to generate observation perturbations, thus making RL agents more robust to context shift. This work falls nicely into the recently growing field of data augmentation methods in RL. Results are presented on Procgen. 
main_review: The general idea of this paper is to train a GAN-like architecture to increase the robustness of RL algorithms to shifts in the input distribution. The Policy Network uses a PPO style loss to train the policy, with the addition of a KL term to ensure that perturbed X’ result in similar actions being taken by the policy. This is a good idea and the authors explain it very clearly in section 3, which I found enjoyable to read. Some more specific comments below:


The KL constraint in equation 3 is the most straightforward way to go about ensuring close observations are mapped continuously into the policy space (in the sense that a small delta in observation space causes a small delta in actions). Did you try a simple MLE loss on the mean of the output distribution? There is always some worry that this sort of KL constraint will lead to mode collapse. But in practice that doesn’t seem to be a problem? Did you consider alternative distributions to parameterize the policy? Some sort of catagorical distribution trick as in DreamerV2? Equation 5 also has a similar KL term. Often, I’ve found that in this kind of setup, it can also help to constrain the policy towards a normal distribution (add a KL(N(0,1), pi) term), to help avoid mode collapse. A similar trick is presented in PEARL. 

The clustering pipeline and use of ResNet features was interesting. I think this is a small idea but also novel in my experience.  


Is RAD the SOTA data augmentation technique? I’d be interested in seeing comparisons with DRQV2, SODA, SVEA, PAD, and SAC.  What about Learning Task Informed Abstractions, a recent publication on a similar benchmark?

You compare against PPO rather than something like SAC. I suppose that’s fair because this algorithm is based on a PPO style objective. Is on-policy learning truly needed in this case? Would the algorithm not benefit from keeping a buffer as in SAC? I am a bit concerned that this algorithm only marginally improves on PPO in many environments, which is itself a very weak baseline. 

Results in table 1 are somewhat hard to parse because the scores are not normalized. It would be ideal if the scores could be normalized, or the results could be presented as a percentage. Maybe in an appendix. I obviously understand the desire to present the raw data. 

I think the sample efficiency experiment is important, and I wish more papers included this kind of plot.  

The beta value chosen in these experiments is much higher than I would expect. How sensitive is the algorithm to this value? Figure 6 B looks at different values of beta and seems to suggest that this value doesn’t matter very much? Is there any intuition as to why? What happens if beta is set to 0? Is this term even necessary? 

In general, the related work section could use some help. There is a rich history of style transfer objectives in inverse reinforcement learning that is missed (AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos). This section also runs into the problem of simply listing a lot of different work, but fails to adequately discuss it in the context of the present paper. I’d really like to see this section revised. 
summary_of_the_review: This method represents an incremental improvement gained by developing a novel architecture. This paper — and the success of the proposed methods — was not surprising. But of course the authors of course should receive credit for getting things to work so well. The quality of results generally ranges from acceptable to quite good, although the quality of ablations falls far behind some comparable papers and can be improved.  Overall, a good paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper propose a method for improving generalisation and robustness across visual factors of variation in reinforcement learning, using a style transfer network to adversarially perturb the input to the policy, while the policy is trained to be invariant to this visual perturbation. The style transfer network is trained using StarGAN, based on domains created through a Gaussian Mixture Model clustering of the observations from the policy's experience. The method is demonstrated on OpenAI Procgen easy mode, and compared against PPO and RAD. The method does better than both of these baselines on 8 of the 16 games. Ablations on a single game show that the method is fairly robust to different choices of hyperparameters (number of clusters for the GMM, and weight of the adversarial loss for the style transfer network and policy).
main_review: # Strengths
- The idea of using style transfer combined with adversarial perturbation of the visual input to improve generalisation in RL is interesting and novel.
- The method is benchmarked in a standard way on a standard benchmark, which aids reproducibility and comparison.
- The description of the approach is fairly clear, and if code is released then I believe the method would be quite reproducible

# Weaknesses
- The paper oversells it's empirical results. In the abstract it claims to outperform SOTA algorithms, but SOTA on Procgen is not PPO or RAD, but IDAAC (https://arxiv.org/abs/2102.10330). Further, RAD isn't even the SOTA method which applies data augmentation on Procgen, as that's UCB-DrAC (https://arxiv.org/abs/2006.12862). In many places the authors claim the method consistently outperforms the baselines, but they only perform better than both the baselines in 8 out of 16 games in terms of test performance, and many of those improvements aren't outside the confidence intervals. I think the language around the comparisons to baselines should be made less grand and more accurate.
- Further, it would be beneficial to actually compare to SOTA results, which can be taken from the IDAAC paper https://arxiv.org/abs/2102.10330. Or justify why your method shouldn't be compared against SOTA results
- The presentation of results in Table 1, where their method is bolded if it outperforms a single baseline, is non-standard and unfairly promotes their method (the other methods aren't bolded if they outperform any other method). I think this practice shouldn't be done.
- The method implicitly relies on the visual inputs being able to be clustered into distinct visual themes, which Procgen does have, but many other methods do not. It would be worth investigating whether this method works on benchmarks with less distinct clusters of visual themes.
- In general, no limitations of the method are discussed, and I think they should be to ensure that the method's strengths and weaknesses are fully understood.
- The related work section could use more discussion of other approach which take an adversarial approach to improving generalisation, such as https://arxiv.org/abs/2102.07097, https://arxiv.org/abs/2106.15587 and https://arxiv.org/abs/1703.02702.

# Questions
- In table 1, what do you mean by "the mean of the best test results"?

# Suggestions for improvements.

## Large:
- Evaluating the method on another generalisation benchmark, such as distracting control suite (https://arxiv.org/abs/2101.02722), would better validate the usefulness of this method. I think the idea of using style-transfer for adversarial data augmentation is interesting, and so I don't think that SOTA results are required, but showing improved results on a range of benchmarks would add evidence for the method being generally applicable and useful.
- In particular, the method seems useful from two perspectives: (1) from a sample-efficiency point of view as data augmentation and auxiliary representation learning losses often increases that. (2) from a generalisation perspective, *if* the training environment doesn't contain every combination of visual style and underlying state-space level (as is the case for procgen). I think an evaluation protocol for Distracting Control Suite could disentangle these two perspectives further. 

## Small:
- Typo: Pg 5 "irreverent" -> "irrelevant"
- Reporting min-max normalised score, aggregated across games, and displaying performance profiles, as motivated in https://arxiv.org/abs/2108.13264, would improve the results reporting dramatically, and allow a scalar comparison between their method and the baselines.
- When using it as a verb I think you should write "style-translate", or just "translate the style of..." rather than "style translate" without the hyphen.
- Describing Procgen as "Visually enriched" implies you've changed (enriched) the procgen environment with more visual variety, which I don't think you have. Procgen is an established benchmark, I don't think you need to prefix it with adjectives when you mention it. Further, you describe the evaluation protocol you use as if you created it, but it's the standard evaluation protocol for easy mode in Procgen, which I think is worth mentioning.
- In the related work section, it's unclear why Burda et al 2018 is in the list of approaches using data augmentation (as it targets exploration, not generalisation), and Cobbe et al 2019 is in the list twice, and Kostrikov et al 2020 three times.
summary_of_the_review: While I think the idea motivating this paper is interesting and potentially useful, I think the execution of the idea in this paper, the lack of strong or varied empirical results, the overselling of the effectiveness of the method, and the incomplete discussion of related works makes me recommend the paper is rejected. Several of these criticisms are relatively easy to address, but overall I'm unlikely to recommend acceptance unless the authors add encouraging empirical results on a different benchmark, or vastly improve the results they currently have.

EDIT: after seeing the response from the authors I have raised my score from 3 to 5, and my correctness score from 2 to 3. I believe the improvements are worthwhile but don't make the paper worthy of acceptance: crucially, I think better comparisons with baselines (such as UCB-DrAC, and a more performant SAC implementation), and combining the method with an off-policy method (such as SAC or QT-Opt) would make the empirical justification of the idea more robust and make me more likely to accept the paper.
review_title: null

Meta-review: I thank the authors for their submission and active participation in the discussions. This papers is borderline with reviewers WXXr and eK4b leaning towards acceptance and reviewers f6jT and FV5x leaning towards rejection. On the positive side, reviewers remarked that the paper is interesting [FV5x] and novel [FV5x,f6jT,eK4b,WXXr]. However, there all reviewers found some flaws with respect to the execution and empirical validation [FV5x], specifically around lacking baselines [FV5x,WXXr] and some ablations [f6jT,WXXr]. I side with the comment made by reviewers FV5x as well as WXXr that a comparison to stronger baselines (UCB-DrAC) is warranted. Therefore, I recommend that this paper is not ready for publication at this point and that it will benefit greatly from another iteration with stronger empirical results. I want to very strongly encourage the authors to further improve their paper based on the reviewer feedback.
Decision: Reject