Adversarial Style Transfer for Robust Policy Optimization in Reinforcement Learning
Authors: Md Masudur Rahman
Yexiang Xue
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Reinforcement Learning
Generalization in Reinforcement Learning

Abstract: This paper proposes an algorithm that aims to improve generalization for reinforcement learning agents by removing overfitting to confounding features. Our approach consists of a max-min game theoretic objective. A generator transfers the style of observation during reinforcement learning. An additional goal of the generator is to perturb the observation, which maximizes the agent's probability of taking a different action. In contrast, a policy network updates its parameters to minimize the effect of such perturbations, thus staying robust while maximizing the expected future reward. Based on this setup, we propose a practical deep reinforcement learning algorithm, Adversarial Robust Policy Optimization (ARPO), to find an optimal policy that generalizes to unseen environments. We evaluate our approach on visually enriched and diverse Procgen benchmarks. Empirically, we observed that our agent ARPO performs better in generalization and sample efficiency than a few state-of-the-art algorithms.

Under review as a conference paper at ICLR 2022
ADVERSARIAL STYLE TRANSFER FOR ROBUST POL- ICY OPTIMIZATION IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes an algorithm that aims to improve generalization for rein- forcement learning agents by removing overfitting to confounding features. Our approach consists of max-min game theoretic objective. generator transfers the style of observation during reinforcement learning. An additional goal of the generator is to perturb the observation, which maximizes the agent's s probability of taking a different action. In contrast, policy network updates its parameters to minimize the effect of such perturbations, thus staying robust while maximizing the expected future reward. Based on this setup, propose practical deep rein- forcement learning algorithm, Adversarial Robust Policy Optimization (ARPO), to find robust policy that generalizes to unseen environments. We evaluate approach on Procgen and Distracting Control Suite for generalization and sample efficiency. Empirically, ARPO shows improved performance compared few baseline algorithms, including data augmentation.
INTRODUCTION
The reinforcement learning (RL) environments often provide observation, which is a high- dimensional projection of the true state, complicating policy learning as the deep neural network model might mistakenly correlate reward with irrelevant information. Thus deep neural networks might overfit irrelevant features in the training data due to their high flexibility and the long-stretched time duration of the RL training process, which leads to poor generalization (Hardt et al., 2016
Zhang et al. 2018a Cobbe et al. 2019: 2020 Zhang al. 2018b] Machado et al.) 2018] Gamrian & Goldberg 2019). These irrelevant features (c.g.. background color) usually do not impact the reward; thus, an optimal agent should avoid focusing on them during policy learning. Even worse, this might lead to incorrect state representations, which prevents deep RL agents from performing well even in slightly different environments. Thus, to learn correct state representation, the agent needs to avoid those features that are not important for optimal policy learning.
In recent times, generalization in RL has been explored extensively. zero-shot generalization framework (Zhang et al. 2018b [oogg et al. 2020: Wang et al.) 2019 Packer et al. (018) the agent is trained on finite training set of MDP levels and then evaluated on the full distribution of the MDP levels. The distribution of the MDP levels can be generated in various ways, including changing dy- namics, exploit determinism, and adding irrelevant confounders to high-dimensional observation. An agent trained in these setups overfit any of these factors, resulting in poor generalization performance. Many approaches have been proposed to address this challenges including various data augmentation approaches such as random cropping, adding jitter in image-based observation (Cobbe et al. 2019: Laskin et al. 2020a Raileanu et al. 2020 Kostrikov et al. 2020 Laskin et al.,
2020b).randon noise injection (Igl et al. 2019). network randomization (Osband et al. 2018 Burda et al. 2018 Lee et al. 2020). and regularization (Cobbe et al.) 2019 Kostrikov et al. 2020
let al. 2019 Wang et al. 2020) have shown to improve generalization. The common theme of these approaches is to increase diversity in the training data so as the learned policy would better gener- alize. However, this perturbation is primarily done in isolation might change an essential aspect of the observation, resulting in sub-optimal policy learning. More- over, the random perturbation in various manipulations of observations such as cropping, blocking, or combining two random images from different environment levels might result in unrealistic ob- servations that the agent will less likely observe during testing. Thus these techniques might work poorly in the setup where agents depend on realistic observation for policy learning. It is also de-
the RL reward function, which

---

Under review as a conference paper at ICLR 2022
Observation
Replay Buffer
RL Training
a (go luft)
TE (x)
a (go lert)
update policy to keep same action
Policy π
Adversarial Training
X
x'
π π. (x')
update generator change to different
Style Transfer Generator
nget))
Figure I: Overview of approach. The method consists of minimax game theoretic objective. It first applies clustering approach generate group of clusters based the different visual features in the ob- servation. generator is then used style-translate observation from one cluster to another while maximizing the change in action probabilities of the policy. In contrast, the policy network updates its policy parameters to minimize the translation effect while optimizing for the RL objective that maximizes the expected future reward.
sirable to train the agent with realistic observations, which helps it understand the environments' semantics. Otherwise, the agent might learn unexpected and unsafe behaviors while entirely fo- cusing maximizing rewards even by exploiting flaws in environments such as imperfect reward design.
This paper focuses particular form of generalization where the agent gets observe high- dimensional projection of the true state. In contrast, the latent state dynamics remain the same across all the MDP levels. This assumption has been shown affect generalization (Song al. 2020) in well studied benchmarks (Nichol et al. 2018). It has been observed that the agent overfit the scoreboard or timer and sometimes achieves the best training performance even without looking at the other part of the observation. Consequently, this policy completely fails to generalize in test environments. The high-capacity model even can memorize the whole training environment, thus severely affecting the generalization (Zhang et al. 2018b).
This paper proposes a style transfer-based observation translation method that considers the obser- vation's s content and reward signal. In particular, a generator performs the style transfer while the policy tries to be robust toward such style changes, Note that, style transfer-basee approach has been used to target generalization in Anonymous 202I where the authors propose to train style transfer and reinforcement learning in isolation. In contrast, in this paper, we propose a unified learning approach where the style transfer is integrated into the reinforcement learning training. Thus the generator produces more effective translation targeted toward RL objective, making the RL policy more robust. The trajectory data from the agent's replay buffer is first clustered into different cat- egories, and then observation is translated from one cluster's ' style to another cluster's style. Here the style is determined by the attribute commonality of observation features should be robust toward changes of such features. Moreover, the translated trajectories correspond those that possibly appear testing environments, assisting the agent in adapting to unseen sce- narios.
cluster. The agent
Figure [ shows an overview of our method. Our approach consists of max-min game theoretic objective where generator network modifies the observation by style transferring it to maximize the agent's probability of taking a different action. In contrast, the policy network (agent) updates its parameters to minimize the effect of perturbation while maximizing the expected future reward. Based on this objective, we propose practical deep reinforcement learning algorithm, Adversarial Robust Policy Optimization (ARPO), that aims to find robust policy by mitigating the observational overfitting and eventually generalizes to test environments.
We empirically show the effectiveness of our ARPO agent in generalization and sample efficiency challenging Procgen (Cobbe et al.) 2020) and Distracting Control Suite (Stone et al.) (021) bench- mark with image-based observation. The Procgen leverages procedural generation create diverse
2

---

Under review as a conference paper at ICLR 2022
environments to test the RL agents' generalization capacity. The distracting control adds various vi- sual distractions on top of the Deepmind Control suite (Tassa et al. 2020 to facilitate the evaluation of generalization performance.
Empirically, we observed that our agent ARPO performs better in generalization and sample effi- ciency than the baseline PPO (Schulman et al., 2017) and data sugmentation-basee RAD (Laskin eet al. 2020a) algorithm on the many Procgen environments. Furthermore, APRO performs better compared PPO and SAC (Haarnoja al. 2018 on the evaluated Distracting Control Suite environments in various setups.
In summary, contributions are listed as follows:
We propose Adversarial Robust Policy Optimization (ARPO), deep reinforcement learn- ing algorithm to find a robust policy that generalizes to test environments in zero-shot generalization framework.
We evaluate our approach on challenging Procgen (Cobbe et al. 2020 and Distracting Control Suite (Stone et al. 2021) benchmark in generalization ans sample efficiency set- tings.
Empirically, we observed that our agent ARPO performs better generalization and sample efficiency performance compared to standard PPO (Schulman et al. 2017). SAC (Haarnoja et al. (2088) and data augmentation-basel RAD (Laskin et al. 2020a approach in various settings.
2 PRELIMINARIES AND PROBLEM SETTINGS
Markov Decision Process (MDP) An MDP is denoted by M (S,A,P,r) where every timestep f, from an state E S the agent takes an action from set of actions A. The agent then receives reward Tt and the environment and move to a new state $141 E S based on the transition probability P(s az).
Reinforcement Learning. Reinforcement learning aims to learn a policy π II that maps state to actions. The policy's 's objective is to maximize cumulative reward in an MDP, where II is the set of all possible policies. Thus, the policy which achieves the highest possible cumulative reward the optimal policy n" E II.
Generalization in Reinforcement Learning. In the zero-shot generalization framework (Song et al. 2020), we assume the existence of a distribution of levels D over an MDP and a fixed optimal policy x which can achieve maximal return over levels generated from the distribution D. The levels may differ in observational variety, such different background colors. In this setup, the agent has access to a fixed set of MDP levels during training the policy. The trained agent is then tested the unseen levels to measure the generalization performance of the agent.
Style Transfer.
The task of image-to-image translation is to change a particular aspect of a given image another, such red background green background. Generative adversarial network (GaN)-based method has achieved tremendous success in this task (Kim et al. [017 Isola et al.)
2017 Zhu al., 20177] Choi et al.) 2018). Given training data from two domains, these models learn to style-translate images from one domain to another. The domain is defined as a set of images sharing the same attribute value, such similar background color and texture. The shared attributes in domain are considered as the "style" of that domain. Many translation methods require paired images from two domains, which is not feasible in the reinforcement learning setup. The agent col- lects the data during policy learning, and annotating them in pairs is not feasible. Thus, we leverage unpaired image to image translation, which does not require a one-to-one mapping annotated images from two domains.
In particular, we build upon the work of StarGAN (Choi et al. 2018) which capable of learning mappings among multiple domains efficiently. The model takes in training data of multiple domains and learns the mappings between each pair of available domains using only a single generator. This method automatically captures special characteristics of one image domain and figures out how these characteristics could be translated into the other image collection, making it appealing in the reinforcement learning setup. However, the RI. agent generates experience trajectory data which is
3

---

Under review as a conference paper at ICLR 2022
not separated into domains. Thus, further apply clustering approach which first clusters the trajectory observation into domains, and then we train the generator, which those domains.
among
ADVERSARIAL ROBUST POLICY OPTIMIZATION (ARPO)
Our approach consists of max-min game theoretic objective where a generator network modifies the observation by style transferring it to maximize the agent's probability of taking different action. In contrast, the policy network (agent) updates its parameters to minimize the effect of perturbation while maximizing the expected future reward. Based this objective, we propose practical deep reinforcement learning algorithm, Adversarial Robust Policy Optimization (ARPO), to find robust policy that generalizes to test environments. We now discuss the details of the policy network and generator.
3.1 POLICY NETWORK
The objective of the policy network is two-fold: maximize the cumulative RI. return during training and be robust the perturbation on the observation imposed by the generator. Ideally, the action probability of the translated image should not change from the original image, as the translated observation is assumed to represent the same semantic as the original observation.
The reinforcement learning loss L. can be optimized with various reinforcement learning mecha- nisms, such as policy gradient (optimization). However, we do make any assumption the type of RL algorithms to use for optimizing Lr; thus, any algorithms can be incorporated with our method. Here we discuss the policy optimization formulation, which is based on the proximal policy opti- mization (PPO) (Schulman et al. 2017) objective:
L. -Eel To To(ar(s)) S2) A.]
(1)
Toad (1 81)
A, = + yT
TT-1 + +TT
(2)
where, To is the current policy and To.. is the old policy; A, is estimated from the sampled trajectory by policy Test leveraging a value function estimator denoted as V. Here, both the policy and value network V are represented as neural networks.
An additional objective of the policy is to participate in the adversarial game with the style transfer generator. Thus the final policy loss Le is defined as follows:
Ca Cr + K. Ta (e(1)x()]
The policy parameter is updated to minimize equation For the observation s, at timestep the KL-component measure the distortion in the action distribution of the policy ma due to the per- turbation (x²)). The hyperparameter 31 determines the participation in an adversarial objective with the generator. By minimizing the KL-part, the policy becomes robust to the change in perturbation proposed by the generator.
3.2 GENERATOR NETWORK
takes the observation 2; as input and outputs the style-translated observation x/. The objective of this network is to change the agent's (policy network's) s) probability of taking a different action for the corresponding input observation. However, the content or semantic of the observations needs to be intact, and the only style of the images will differ.
The first step of this process is to curate dataset with different style features such as background color and texture of objects present in the observation. We first use experience trajectory observation data from the environment and separate them into different classes based on visual features (Figure 2). The task of the generator is then translate the images from one class images another class images. In this case, the "style" defined as the commonality among images in a single class. We now discuss details about how separate the trajectory observation to get different clusters.
4

---

Under review as a conference paper at ICLR 2022
A
B
Replay Buffer
clustering
GMM
Generator
style transfer between each pair of clusters
trajectory obs
C
Figure 2: Overview of style transfer generator module. The experience trajectory observation data for the task environment are separated into different classes based on visual features using Gaussian Mixture Model (GMM) clustering algorithm. The task of the generator is then to translate the image from one class image to another classes images. In this case, the "style" is defined the commonality among images in a single class. Given observation, first infers into its (source) cluster using GMM, and then the generator translated it (target) another cluster style. The target cluster taken randomly from the rest cluster.
Clustering Trajectory Observation. The trajectory data is first clustered into n clusters. Then, we use the Gaussian Mixture Model (GMM) for clustering and ResNet (He et al. 2016) for feature extraction and dimension reduction. Firstly, a pre-trained (on ImageNet) ResNet model reduces an RGB image into 1000 dimensional vector. This step extracts useful information from the raw image and reduces the dimension, allowing faster cluster training. Note that this process focuses entirely the visual aspect of the observation. After passing through the ResNet, the observation dataset is clustered using GMM.
Generator Training. Generator G tries fool discriminator D in adversarial setting by gener- ating realistic image represented by the true image distribution. We extend the generator's 's ability, and additional objective of the generator the style of the original observation. For example. changing the background color to different one does not change the semantic of the observation, but the image style would be different. Ideally, the action mapping of the policy needs to be the same on both the original and translated observation. However, policy that focuses on the irrelevant background color will suffer from this translation, and thus its probability distribution deviates because of the style translation. This is defined as the KL-divergence K L (T (-)=;), (.((1=()). where ae is the given observation and of is the translated observation by the generator.
fool the RL agent (policy network) by transforming
The generator tries to generate style that eventually increases the KL quantity, On the other hand, the policy should learn to be robust toward such changes. The policy training described above addresses this and tries to minimize the KL-divergence due to the style transfer, which eventually allows robust policy learning and ultimately prevents the agent from overfitting irrelevant features from the high-dimensional observation.
We build the generator previous works (Choi al. 2018 Zhu al. 2017 which is unified framework for multi-domain image to image translation. The discriminator loss is
LD Ludo Aeta
(4)
which consist of the adversarial loss Ladv and domain classification loss L The discriminator detects a fake image generated by the generator from the real image in the given class data.
On the other hand, the generator loss is
(5) which consists of image translation components and policy component which is the KL-part. From the translation side, the Cadv is adversarial loss with discriminator C, is the loss of detecting fake image. Finally, a cycle consistency loss (Choi et al., 2018: Zhu et al. (017)) Cree is used which makes sure the translated input can be translated back the original input, thus only changing the domain related part and not the semantic. Some more details of these losses can be found in the supplementary materials.
La Lade + Aeta + Areohren LoTKL(Ta(aaa)) To (+(14)
Our addition to the generator objective is the KL-part in equation 5 which tries to challenge the RL agent and fool its policy by providing more challenging translation. However, the eventual goal
5

---

end for Under review as a conference paper at ICLR 2022
of this adversarial game is to help the agent to be robust to any irrelevant features in the observa- tions. The hyperparameter B2 control the participation in the adversarial objective of the generator network.
Now we discuss a practical RL algorithm based on the above objectives.
3.3 ARPO ALGORITHM
The training of the policy network and the generator network happen in an alternating way. First, the policy is trained on a batch of replay buffer trajectory observation, and then the generator parameters are updated based the objectives discussed above. The more overall training progresses, the better the generator at translating observation, and the policy also becomes better and becomes robust to irrelevant style changes. For stable training. the imput images data for the generator is kept fixed initially, and we apply the GMM cluster only at the beginning generate different class images algorithm given Algorithm
(distribution). The pseudocode
Algorithm Adversarial Robust Policy Optimization (ARPO)
1: Initialize parameter vectors for policy network and generator network
2: for each iteration do
3: for each environment step do
4: Gg ~ To [ (2a)
5:
11+1 P( [22, az) ~ R(I,, 0t)
6:
7:
D+ DU (
Ut Te: ((+11
8:
end for
9:
for each observation 2, in D do
10:
ry &- -Generator (xx) // generate translated observation
Compute L, from data DD using Equation] and update policy for RL objective Compute Ca using Equation 3 3// update policy with adversarial KL part
12:
13:
Update generator by computing Equation and
14:
end for
15:
be- come zero when the RL policy is optimal (x*) and the generators only perturbs (translates) the irrelevant part of all observations. In that case, the optimal policy only focuses on the relevant part of the observations, that is, the true state; thus, any changes in irrelevant part due style transfer will be ignored. At that point the KL component become zero as the (.(2)) = गद (-I=(). Note that practice, the algorithm usually trained for limited timesteps, and thus the adversarial training might not converge to optimal. However, empirically we observe that our algorithm ARPO achieves improved performance many challenging Procgen and Distracting control suite tasks.
Discussion on convergence. The adversarial min-max KL component K (xT((()z)
EXPERIMENTS
4.1
SETUP
Our experiments cover both discrete (Procgen) and continuous (Distracting control) action spaces with image-based observation.
Procgen Environment. We evaluate our proposed agent ARPO on the challenging RL generaliza- tion benchmark Procgen (Cobbe et al. 2020). We use the standard evaluation protocol from Cobbe al. (2020) where the agent is allowed to train only 200 levels of each environment, with the difficulty level set as easy. Then the agents are tested over more than 100K levels, full distribution. Thus, to better generalize the test environment, the agent must master the skill and avoid overfit- ting to spurious features in the training environment. The observations are raw pixel images, and the environment varies drastically between levels; some snippets given in Figure
Distracting Control Suite Environment. The distracting control adds various visual distractions on top of the Deepmind Control suite (Tassa al. 2020) to facilitate the evaluation of generalization

---

Under review as a conference paper at ICLR 2022
Climber
Jumper
DodgeBal
StarPilot
BigFish
CaveFlyer
Seen
Unseen:
Figure 3: Some snippets of different Proegen environments (Cobbe et al. 2020). The training (seen) levels vary drastically from the testing (unseen) environment. The agent must master the skill without overfitting irrelevant non-generalizable aspects of the environment perform better in unseen levels.
performance. We experimented on two distracting settings, background (dynamic) and color for easy difficulty mode. In the distracting background, videos played in the background, and for the color distraction, the body of the robot color changes in different agent interactions. The train and test environments differ by different distractions, such as separate videos for train and test. As the agent has to use image-based observation, the distractions make the task challenging. Some snippets of the environments can be found in Figure
Walker walk (background)
Walker walk (color)
Cheetah run (background)
Cheetah run (color)
Figure 4: Some snippets from Distracting control suite
(Stone et al. 2021
Baselines. We compare our algorithm with Proximal Policy Optimization (PPO), an on-policy gorithm motivated by how we quickly optimize our policy while staying close to the current policy. PPO performs effectively many RL benchmarks, which includes the Procgen environ- ments (Cobbe al. 2020). Furthermore,
evaluated our method with data augmentation tech- nique, RAD (Laskin et al. 2020a) on Procgen environments. RAD is plug-and-play module that be used with existing RI. algorithms and shows effective empirical performance in complex RL benchmarks, including some Procgen environments. In particular, we evaluated Cutous Color augmentation technique which has shown better results in many Procgen environments compared to other augmentation techniques evaluated in Laskin et al. (0220a). This augmentation is applied be easily plugged in with these data
on top of standard PPO. Note that our ARPO algorithm methods. However, in this paper, we evaluated how our ARPO agent performs even when no augmentation used. Furthermore, in the Distracting control suite, in addition to PPO, we compare ARPO with Soft Actor-Critte (SAC) (Haarnoja et al. 2018) which is an off-policy algorithm that optimizes stochastic policy.
augmentation-bars
Implementation. We implemented our ARPO on the Ray RLlib framework (Liang et al. 2018) and used a CNN-based model for the policy network for these experiments. For the experiments with ARPO, we set the B 32 20 (in equation 3 and 5). The generator's hyperparameters are set to Acts = 1, and Arec = 10. Number cluster is set to 3. Details of model configurations and parameters given in the Appendix. We will open-source code and data for reproducibility and facilitating further research.
4.2 PROCGEN RESULTS
Generalization. The results in Figure 5 show the test results ARPO and baselines PPO and RAD on 16 Procgen environments with image observation. Overall, ARPO performs better in many environments. We observe that, almost in half the environments, our agent ARPO outperforms both baselines. It also performs comparably in few other environments. Note that in envi- ronments such as Ninja, Plunder, Chaser, Heist, all the agents fail to obtain a reasonable training accuracy (Cobbe et al. 2020) and thus perform poorly during testing (generalization).

---

Under review as a conference paper at ICLR 2022
jamber
tiglahh
cimber
covetlyer
(ours)
PPO
RAD
154 Pruthot
200
TCM
15M
33M
20M
1CM
12M
214
mtr
leapprr
003gehal
Score
2.10
23.0
37.5
164
LDM
204
13M
caltruu
bosstight
Timesteps (Million)
Figure 5: Generalization results. Test performance comparison on Procgen environments with image observa- tion. ARPO performs better in many environments where it outperforms the baselines. The results are averaged over 3 seeds.
Table Generalization results on Procgen environments. The results are the best test result achieved by each agent during the entire training timesteps. The mean value and standard deviation calculated across 3 random seed runs. ARPO achieves the best generalization results in most of the environments compared to PPO and RAD. Best agent is in Bold.
Env
ARPO (ours)
PPO
RAD
jumper
6.5 + 0.06
6.49
0.15
6.35
0.04
bigfish
4.18 + 1.59
2.82 + 0.75
3.22 + 0.23
climber
5.88 + 0.42
5.22 - 0.59
5,53 0.55
caveflyer
5.91 0.08
5.5 4 0.25
5,46 0.34
miner
7.2
6.8 - 0.17
6.21 = 0.36
fruitbot
27.04 0.49
26.53 + 0.57
26.99 + 0.97
leaper
4.19 0.55
4.33 + 0.97
3.22 0.06
ninja
6.49 0.41
6.96 + 0.5
5.82 = 0.29
dodgeball
2.32 + 0.22
2.28 0.05
1.85 - 0.17
maze
6.1
6.23 0.32
5.98 0.04
plunder
8.37 + 0.95
9.39 + 0.47
8.34 + 2.19
starpilot
29.0 + 3.04
31.13 + 0.27
31.59 + 1.28
heist
3.93 + 0.41
3.7 + 0.35
3.75 0.13
coinrun
7.67 + 0.68
8.81 + 0.12
8.25 = 0.22
bossfight
4.08 + 1.27
7.63 - 0.6
7.99 - 0.39
chaser
2.32 0.09
3.11 0.69
2.36 0.04
This result shows that our proposed agent ARPO achieves strong performance and reduces the generalization gap compared the tested baselines on generalization.
We further show how each agent achieves generalization during the entire timesteps. Consequently, we compute the maximum score achieved by each agent the entire duration and take the average best score. Finally we report mean and standard deviation of these best scores across seed runs. TTe generalization results are computed by evaluating the trained agents on test levels (full distri- bution). The performance is computed by evaluating each agent for 10 random episode trials after a training interval of 10. All agents ran for 22 million timesteps for training. Table [ shows the results comparison for all the agents on Procgen environments. Similar to the previous discussion, we see that many of the environments, our ARPO agents perform better than the baselines. ARPO performs better than both the baselines in half of the environment, and in 12 out 16 environments, ARPO performs better than at least on baselines. This result shows that our agent outperforms the baselines in the generalization results Procgen benchmarks.
8

---

Under review as a conference paper at ICLR 2022
Sample Efficiency. We evaluate and compare agent on the sample efficiency during training. This result shows how quickly our agent achieves a better score during training. Learning curve can be found in Appendix B (Figure 7
In many environments, we see our agent ARPO perform better than the baselines PPO and RAD. These results show that despite optimizing for the generalization, our adversarial training helps the ARPO agent learn a robust policy which is also better during training.
4.3 DISTRACTING CONTROL RESULTS
Figure 6 shows the results of ARPO and PPO Walker walk environment from distracting control We observe that ARPO performs better compared to the baseline algorithm PPO in both
suite.
Background distraction
Color distraction
walker walk
walkerr walk
walker walk
walker walk
50
50
50
40
40
40
40
Anno .P0
APP
200K
100K
200K
100K
200K
(a) Train
(b) Test
(c)
(d)
Figure 6: Sample efficiency (Train) and generalization (Test) results on Distracting control Walker walk envi- ronments. The results are averaged over 3 seeds.
sample efficiency (train) and generalization (test) for background and color distraction. Note that we observe large variance (standard deviation) in the results across the run, particularly in test time. These results correspond to the benchmark results where Stone et al. (202T) also find comparatively larger variance in reported results. In some cases (Figure (b). the test performance surpasses the corresponding timestep's 's train reward. This scenario might happen because the videos for train and test environments have different difficulties. Thus in some cases, the agent finds better test performance than the training. However, our ARPO's performance remains better during testing compared to PPO. We also observe that in these settings ARPO (and also PPO) performs better compared to SAC. The results are in the Appendix D (Figure I0). Furthermore, results analysis for Cheetah run environment is in the Appendix
Please see the Appendix for ablation study (Section
E and related work (Section F
5 DISCUSSION
Lack of visual diversity the observation the environment might result poor clustering, which eventually leads to less challenging style translation by the adversarial generator. However, in those scenarios, the reinforcement learning algorithms often perform well as the train and test environ- ment remain consistent (e.g., same background). Furthermore, this setup might lead the agent to overfit the training environment, which results in the agent performing poorly in a slightly different environment (Song et al., 22020) Zhang et al. (2018)). In this paper, we are interested in the scenario where the train and test environment are visually different, which is a more practical setup (Cobbe) al. 2020 Stone et al. 2021).
In summary, we proposed an algorithm, Adversarial Robust Policy Optimization (ARPO), to im- prove generalization for reinforcement learning agents by removing the effect of overfitting in high- dimensional observation space. Our method consists of a max-min game theoretic objective where a generator is used to transfer the style of high-dimensional observation (e.g., image), thus perturb original observation while maximizing the agent's 's probability of taking different action for the corresponding input observation. In contrast, a policy network updates its parameters to minimize the effect of such translation, thus being robust to the perturbation while maximizing the expected future reward. We tested our approaches on Procgen and distracting control benchmarks in gener- alization setup and confirmed the effectiveness of our proposed algorithm. Furthermore, empirical results show that our method generalizes better in unseen test environments than the tested baseline algorithms.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G. Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021a.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Informa- fion Processing Systems, 2021b.
Anonymous. Bootstrap state representations for better generalization in deep reinforcement learn- ing. In Under review, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.128900 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-imaae translation. In Proceedings of the IEEE conference computer vision and pultern recognition, pp. 8789-8797, 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1289. PMLR, 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning. pp. 2048- 2056. PMLR, 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: /802.0/561, 2018
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation. In International Conference on Machine Learning, pp. 2063-2072. PMLR, 2019.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. arXiv preprint arXiv:/704.00028, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning. pp. 2555-2565. PMLR, 2019.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 1225-1234. PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. arXiv preprint arXiv:/603.0112! 2016.
Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. arXiv preprint arXiv: 7777.08777755 2017.
10

---

Under review as a conference paper at ICLR 2022
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Advances in neural information processing systems, pp. 13978-13990, 2019.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In International Conference Ma- chine Learning, pp. 1857-1865. PMLR, 2017.
Iyya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020.
Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In The 2010 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2010.
Sascha Lange. Martin Riedmiller, and Arne Voigtländer. Autonomous reinforcement learning on raw visual input data in a real world application. In The 2012 international joint conference on neural networks (IJCNN), pp. 1-8. IEEE, 2012.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In Advances in neural information processing systems, 2020a.
Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020b.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple tech- nique for generalization in deep reinforcement learning. International Conference on Learning Represeniations, 2020. URL http:tt://peerreieeeeeeee net wttp://wwwwweeeeeeeoneeeeeeeeee. http://oouunni---Hu-eeee--
Bonnie Li, Vincent François-Lavet, Thang Doan, and Joelle Pineau. Domain adversarial reinforce-
ment learning. arXiv preprint arXiv:2102.07099 2021.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pp. 3053-3062, PMLR, 2018
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-5622 2018.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:1804.0374 2018.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8617-8629, 2018.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song- Assessing generalization in deep reinforcement learning. arXiv preprint 1010:.:110..222000 2018.
Lerrel Pinto, James Davidson, and Abhinay Gupta. Supervision via competition: Robot adversaries for learning tasks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1601-1608. IEEE, 2017a
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017b.
11

---

Under review as a conference paper at ICLR 2022
Roberta Raileanu and R. Fergus. Decoupling value and policy for generalization in reinforcement learning. ArXiv, abs/2102.103330 2021.
Roberta Raileanu, Max Goldstein, Denis Yarats, llya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization deep reinforcement learning. arXiv preprint arXiv:2006. 12862, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv: 1910.44492. 2019.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfit- ting in reinforcement learning. In International Conference on Learning Representations, 2020.
https///ppeeeeeeeew wttp:///www...nttoooooooooo-ddo.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting con- trol suite challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722, 2021.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm-control: Software and tasks for continuous control, 2020.
Huan Wang, Stephan Zheng, Caiming Xiong, and Richard Socher. On the generalization gap in reparameterizable reinforcement learning. In International Conference on Machine Learning, pp- 6648-6658. PMLR, 2019.
Kaixin Wang. Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. arXiv preprint arXiv:2010.10814 2020.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937 2018a.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint AXXv:20006.1042.. 2020.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn- ing invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021. URLhtttps:/openeeev/e wet.reurumeet/tum forum?
id 2FCwDKRRBU
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. study on overlitting in deep reinforcement learning. arXiv preprint arXiv:1804.06899, 2018b.
Hanping Zhang and Yuhong Guo. Generalization reinforcement learning with policy-aware ad- versarial data augmentation. arXiv preprint arXiv:2106 15587, 2021.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A APPENDIX
B ADDITIONAL PROCGEN RESULTS
Figure shows the sample efficiency results for ARPO, PPO, and RAD.
12

---

Results for ARPO and PPO Cheetah run with background distraction are in Figure We that both agents perform comparably in both background and color distraction. We observe similar results for both the agents in this setup of Cheetah timestep the PPO agents perform poorly in the training and testing. The highest achievable reward for this environment is much higher (see Stone al. (202T). )) Consequently the the ARPO agent could not improve over the base algorithm PPO. This suggest that the base algorithm needs to trained Under review as a conference paper at ICLR 2022
Jamper
trghtsh
Eimber
Cevatlyer
ARPO (ours)
PPO
RAD
oodgesal
Score
300
124
29
204
24
18M Mtarpake
204
24
10N bwit
10k
34
LON
154
200
54
106
354
34
154
Lonsfight
chaseer
Timesteps (Million)
Figure 7:
Sample efficiency results. Train performance (leaming curve) comparison Procgen environ- ments. Despite using perturbation on the observation, our agent ARPO outperforms the baselines many environments, thus achieving better sample efficiency during training. The results are averaged over seeds.
Procgen aggregated results. further show the aggregated results on all the environments in the Procgen environment. shows the aggregated results comparison all 16 environments of Procgen, which is probabilistic measure that highlights how likely an improvement the algorithm over other proposed in Agarwal et al. (2021b).
Algorithm X ARPO
Algorithm PPO
ARPO
RAD
0.3 0.4 0.5
0.8 0.9 1.0
P(X Y)
Figure 8: Probabilistic measure AAaawwl al. 2021b) of how likely our agent ARPO is to improve over baseline PPO and RAD. We observe that ARPO is up to 54% likely to improve over PPO and up to 72% likely to improve over baseline RAD in the Procgen generalization setup.
C ADDITIONAL DISTRACTING CONTROL RESULTS
environments. Note that in this training
sufficiently to achieve any reasonable reward before we benefit of the adversarial training.
Note that the resulting score is low compared to the reported results in the benchmark paper Stone al. 2021) which suggests the PPO itself could not learn helpful behavior in this environment.
D COMPARISON WITH SAC
Results comparison Distracting Control for ARPO. PPO, and SAC are in Figure 10 (Walker walk), and [ (Cheetah run). We see that ARPO (and PPO) perform better compared to the SAC in both sample efficiency (train) and generalization (test) in these settings.
13

---

Under review as a conference paper at ICLR 2022
Background distraction
Color distraction
cheetah run
cheetah
cheetah run
cheetah
10.0
10.0
7.5
7.5
7.5
5.0
5.0
5.0
5.0
2.5
AREO
A5PO
0.0
100K
200K
300K
0.0
100K
300K
0.0
200K
300K
0.0
100K
300K
Train
Train
(d) Test
Figure 9: Distracting control Cheetah environment results. The results are averaged seeds. (a, c)
Sample efficiency (train), and (b, d) Generalization (test).
Background distraction
Color distraction
lkek walk
walker walk
walker_waal walk
walker walk
30
30
30
20
m0
ARPO
20
J0P0
20
mP
APPO
40K
120K
(a)
(b) Test
(c)
Figure 10: Distracting control Walker walk environment results (including SAC). The results are averaged
over 3 seeds. (a, c) Sample efficiency (train) (b, d) Generalization (test).
Background distraction
Color distraction
cheetah,
cheetah
cheetah
cheetah
7.5
7.5
PRO
1P0
2.5
Nro SAC
2.5
PPO
SAC
SAC
SAC
40K
80K
120K
40K
BDK
120K
40K
B0K
120K
40K
BUK
120K
(a) Train
(b) Test
(c) Train
(d) Teat
Figure 11: Distracting control Cheetah run environment results (including SAC). The results are averaged over 3 seeds. (a, e) Sample efficiency (train), and (b, d) Generalization (test).
E ABLATION STUDY
E. HYPERPARAMETER OF ARPO
We conduct study on how ARPO agent's performance varies due to its hyperparameters: (a) the number of clusters generated by the GMM model for generator training, and (b) 31, .3 which indicate the amount of participation of policy and generator in the adversarial objective.
shoows the ablation results. We observe that our ARPO agent shows improvement in gener- alization (test) performance with the increase in number of cluster on Climber environment (Figure 12ah. As the number of cluster increase, the generator has more option choose for translation, and thus the policy might face hard challenge and thus learn more robust policy. In this paper, report the ARPO agent's 's results on cluster n. nl.custer = 3. On the other hand, and ablation results on different values shown in Figure 12b When B values are large the test performance goes up: however, training performance suffer bit. In this paper, we report the results on 3 = ,3 = 20, which shows balance in train and performance.
E.2 QUALITATIVE ANALYSIS OF ADVERSARIAL STYLE TRANSFER
In this section we discuss how the generator performs in generating adversarially translated obser- vation during agent training.
14

---

Under review as a conference paper at ICLR 2022
climber
climber
10
10
3
-0-
Trah 3
Tran -.0
8
8
-2.02000
Train 3
Trand. -00
Test cu
6
6
Tran 92- 50.0
Score
Score
2-
0
10M
15M
20M
10M
15M
20M
Timesteus (a)
Timesteos
(b)
Figure 12: Ablation results. The results are averaged over seeds. (a) Our ARPO agent's results on different cluster numbers. It improves generalization (test) performance with the increase in cluster number in the Climber environment. (b) Our ARPO agent's results different values which determine participation adversarial optimization.
Figure 13 14
15| and 16 show sample translation from the generator. Please see the correspond- ing caption for detailed discussion.
13
original
Walker Walk
translated
translated original
Cheetah Run
Figure 13: Sample translation of generator on background distraction for Walker-wall environment from Distracted Control Suite benchmark. We see that the translated observations retain the semantic that is the robot pose while changing non-essential parts. Interestingly, non-essential parts of the translated images. This scenario might indicate that the min-max objective between policy network and generator tries to recover the actual state from observation by removing parts irrelevant to the reward.
observe that the adversarial generator blackout
E.3 KL REGULARIZATION
Figure shows the KL regularization results in different policy iteration. We for each environment run the experiment and collect KL value (K ((Tel (-|æ;), ne (+|=())) for each policy iteration. To calculate moving average, we select a window size of 500 and the standard deviation calculated these 500 values, and showed in the shaded area in Figure 17
F RELATED WORK
Data augmentation in RL. In recent time, many approaches have been proposed to address gen-
eralization challenges including various data augmentation approaches (Cobbe et al. 2019 Laskin
al. 2020a Kostrikov et al. 2020) Raileanu al. 2020) Laskin al. 2020b [g] 2019
15

---

Under review as a conference paper at ICLR 2022
Walker Walk (color)
original
transiated
Cheetah Run (color)
origina
Figure 14: Sample translation of generator color distraction for Walker walk environment from Distracted
Control Suite benchmark.
Dodgeball
Climber
original
original
translated
translated
Caveflyer
Coinrun
translated original
translated original
Figure 15: Sample translation of generator on four Procgen environments. We see that the generator retains most of the game semantic while changing the background color and the texture of various essential objects.
Osband al. 2018 Lee 2020, Wang al., 2020 Zhang al. 2021 have shown im- prove generalization. The common theme of these approaches is to increase diversity in the training data; however, these perturbations are primarily performed in isolation of the RL reward function, which might change crucial aspects of the observation, resulting in sub-optimal policy learning. In contrast, we propose a style transfer that tries to generate semantically similar but different visual style observation to train robust policy. Furthermore, our style perturbation takes into account the reward signal from the policy.
Note that the base RL algorithm (e.g., PPO) in ARPO method still uses the original observa- tion from the environment while participating in adversarial objectives (see Figure IT. Thus, data augmentation can be applied to the original observations before feeding them to the base RL algo- rithm to potentially improve the base RL agent, as observed in many data augmentation techniques discussed above.
16

---

Under review as a conference paper at ICLR 2022
Training Timesteps
Original observation Cluster A
Translated Cluster A (same)
Translated Cluster
generator 1241
2669
18275
35785
58259
60622
train iteration
Sample translation of generator during various training phases on background distraction for Walker-walk environment from Distracted Control Suite henchmark. [Second row] We see that when translated back to the parts). while the essential parts remain mostly intact. This scenario might be happening due the regularization of the cycle consistency loss by the KL component in equation [Third row] We see that the translation to a different cluster generates images that changes the irrelevant parts while mostly keeping the semantic intact. In both the second and third row, the translation gets better as time progresses, suggesting the adaptation of both policy and generator networks due the
Figure 16:
cluster (in the second row), the generated images get
distortion (e.g., blackout some
objective.
jumper
bigfish
climber
fruitbot
L.7
1.3
119
Me Jnnda Ho map
1/0
339
Moving
205
375
Moving Arg
M.M.wiingg
Nably Resoeii
Pency ciecattion
KL regularization during RL training for four Proegen environments. At the beginning of the training. both the policy and generator behave randomly; thus, the KL values become nearly zero. Gradually, the KL started to increase as the policy learning progressed with the adversarial generated observation. Eventually, the policy tries adjust to the changes in the observation, and the KL values become stable (bigfish) or going downward (jumper, and climber). We that in fruithot environment, the values started increase at the end, which suggests possible divergence. This scenario might be explanation performance started to drop at the end in Figure[5] possible overfitting. Thus this KL measure might be a tool to detect possible divergence and overfitting. However, each environment's 's stability cutoff time (policy iteration) different, suggesting that the KL regularization behavior might be different depending on the environment complexity. Idcally, with enough training, the KI. values should converge optimal (for true MDP state) and the generator fully recovers the true state from the observation and changes all the irrelevant information from the observation. Note that we limit the policy to train until a cutoff time (22 Million RL timesteps).
Figure 17:
why generalization
(again) if the policy reaches
Style Transfer in RL. Many approaches have been proposed which use style transfer improve reinforcement learning algorithms. Gamrian & Goldberg (2019) use image-to-image translation between domains learn the policy zero-shot learning and imitation learning setup. They use different levels' data and train unpaired GANs between two levels (domains), which requires access to the annotated agent's 's trajectory data in both source and target domains. Furthermore, (Smih et al.) (019) performs pixel-level image translation via CycleGAN to convert the human demonstration to a video of a robot. These translated data can then used to convert the human demonstration into a video of robot used to construct reward function that helps learn complex tasks. However, their CycleGAN operates on two predefined domains, human demonstration and robot video.
In contrast, first automatically generate the domains using clustering algorithm and then train the style-transe translatiiion generator. In our case, we do not need the information of levels; instead, we automatically cluster the trajectory data based visual features of the observations. Additionally,
17

---

Under review as a conference paper at ICLR 2022
visual-based clustering may put observations from multiple levels into a cluster, potentially preventing GAN from overfitting Gamrian & Goldberg (2019) to any particular environment levels.
Adversarial Approach in RL. Adversarial methods have been used in the context of reinforcement learning to improve policy training. Li et al. (202T) proposes to combine data augmentation with auxiliary adversarial loss to improve generalization. In particular, they use adversarial discrimi- nator to predict the label of the observation. However, they need to know the domain label, limiting the applicability it might not be available, and the number of labels might be large, for example, in procedural generation. In contrast, we use an unsupervised clustering that automatically finds out similar visual features later used for generator training. (Pinto et al. 2017b) use adversarial min-mac max training to learn robust policy against forces/disturbances in the system (Pinto et al. (017b)). In addi- tion, adversarial training has been used between two robots to improve object grasping (Pinto al.) 2017a) and in the context of multi-agent self-play scenario (Heinrich & Silver 2016). /Zang & Guo (2021) generate adversarial examples for observations in a trajectory by minimizing the cumulative reward objective of the reinforcement learning agent. In contrast to these methods, we use adver- sarial visual-based style-transfer of the observation to guide robust policy learning by enforving the policy produce similar output action distributions.
Generalization in RL. Many methods have been proposed to reduce the generalization gap be- tween training and unseen test environment in the context of reinforcement learning. State learning approaches (Higgins et al. 2017, Agarwal et al. 2021a ZZang et al. 2020) and auto-encoder based latent state learning (Lange al.] 2012: Lange & Riedmiller] 2010: Hafner al.; (2019) with construction loss have been proposed. The sequential structure in RL (Agarwal et al. 202 and behavioral similarity between states (Zhang et al. 2020)) to learn invariant states have been leveraged to improve RL robustness and generalization. In contrast, we learn the invariant state by providing the RL agent with an additional variant of the same observation while retaining the reward structure using an adversarial max-min objective.
Our primary focus is to demonstrate how the adversarial style transfer helps learn robust policy and improves the robustness of the base RL policy used. Thus our method is orthogonal to many existing algorithms, including strong baseline IDAAC (Raileanu & Fergus] [02T). Combining our method with these algorithms could improve performance; investigating them can be interesting avenue for future work.
G TRAINING LOss OF STARGAN GENERATOR
The adversarial loss is calculated the Wasserstein GAN (Arjovsky et al. 2017) objective with gradient penalty which is defined as
Lodo = E. Drrel
(G c))] App [II [I) 1)²},
(6)
where é is sampled uniformly along a straight line between pair of real and generated fake images. Here, the gradient penalty helps to stabilize the GAN training process and been shown to perform better than traditional GAN training (Choi et al. 2018 We set the hyperparameter Agp 10.
Arjovsky et al. 2017: Gulrajani al. 2017
The classification loss of real image is defined as
S.s = E Dete (d' [x)]
(7)
where (ets(e)lz)) (z) is the probability distribution over all clusteer labels. Similarly, the classification loss of fake generated image is defined
G Ex,el- log Data (c)
(8)
The reconstruction loss with the generator objective defined as
Cree = Exx,c, ([teeellllGGG( G(G(x,o),
(9)
where we use the A1 norm
18

---

Under review as a conference paper at ICLR 2022
Table 2: Hyperparameters for Procgen (RLlib) Experiments
Description
Hypemarameters
Discount factor of the MDP
gammaa 0.999
The (amddaamdda parameter
Iambda : 0.95
The delault learning
5.00
Number of epochs train batch
nam baam.aggd.ae Her :
Total SGD batch size
egd.mamsbatch.aize 2008
Training batch size
frain batch,btcchaeee sise: 16384
Initial crefficient for KI. divergence
kl_cneff 0.00
Tirget value for KL divergence
kl kl.toree ; 0.01
Coeflicintt of the value function loss
of of Joss.coeff 0.5
Cocfliiiett of entropy regularizer
extropy.coe If 0.01
elpp parameter
clsp.poram 0.2
Clip param for the value function
uf.dlipppram 0.2
Clip glohal amount
grad clip 0.5
Default preprocesoos
deepmind
Py Forch Framework
fromework : torch
Settings for Model
custom_model impolo_cnn_toree
Kollout Fragment
vollous./rugment.enghh 256
H ENVIRONMENT DETAILS
Procgen We conducted experiments on OpenAl Proegen Cobbe et al. (2020) benchmark, consist- ing of diverse procedurally-generaae environments with different action sets. This environment has been used to measure how quickly (sample efficiency) a reinforcement learning agent learns generalizable skills. These environments greatly benefit from the use of procedural content gener- ation, the algorithmic creation of a near-infinite supply of highly randomized content. The design principles consist of high diversity, tunable difficulty, shared action, shared observation space, and tunable dependence on exploration. Procedural generation logic directs the level layout and other game-specific details. Thus, to master any of these environments, agents must learn effective policy across all environment variations. We use all 16 environments available in this benchmarks.
All environments discrete 15 dimensional action space which generates 64 64 x 3 RGB image observations. Note that some environments may use no-op actions to accommodate a smaller subset of actions.
I IMPLEMENTATION DETAILS
Procgen Experiments For experimenting on Procgen environments, we used RLlib Liang et al. (2018) to implement our ARPO, and baselines PPO and RAD cutour color algorithms. For all the agents' policy network (model), we use CNN architecture used in IMPALA [speholt et al. (2018) which is the best performing model in Procgen benchmark (Cobbe et al. 2020). We use the same policy parameters for all agents for a fair comparison.
Policy learning hyperparameter settings for all the agents (ARPO, PPO, and RAD) are set same for fair comparison and detect the effect of our proposed method amax-min adversarial objective with perturbation network). These hyperparameters are given in Table 2 Note that only the custom parameters are given here, other defaults parameter values can be found in the RLlib library Liang et al. (2018).
Distracting Control Experiments For experimenting on Distracting Control Suite (Stone et al., 2021) environments, used RLlib Liang al. (2018) to implement ARPO, and baselines PPO and SAC algorithms. We use the PPO's CNN-based policy network (model) from RLlib for ARPO, and PPO. The policy specific parameters for ARPO, and PPO are the same which given in Table 3
For SAC, we use it's CNN-based model available in RLlib. The policy specific parameters given in T Note that only the custom parameters for the RLlib implementation are given here, other defaults parameter values can be found in the RLlib library [iang al. (2018).
19

---

Under review as a conference paper at ICLR 2022
Table 3: ARPO, and PPO Hyperparameters for Distracting Control Experiments
Deseription
Hyperparameters
Number of epochs per train batch
numm_sgd.iter 3
Total SGD batch size
sgd minshatch size 256
Training
bravn_botch.size 8192
Porch Framework
Trammwwokk torch
Table 4: SAC Hyperparameters for Distracting Control Experiments
Description
Hyperparameters croim.bbtchhaize 512
Truining batch
Timestops per iteration
temesteps.per ateration ptrrrrtttoottssstiion 1000
Timesteps per iteration
tearning_starts 5000
Py Torch Framework
framework torch
Computing details. We used the following machine configuration to run our experiments: 20 core- CPU with 256 GB of RAM, CPU Model Name: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, and Nvidia A A100 GPU.
20


Reviews:
[{"review_id": "y_F4LYLSXuS", "reviewer": "Reviewer_FV5x", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- The idea of using style transfer combined with adversarial perturbation of the visual input to improve generalisation in RL is interesting and novel.\n- The method is benchmarked in a standard way on a standard benchmark, which aids reproducibility and comparison.\n- The description of the approach is fairly clear, and if code is released then I believe the method would be quite reproducible\n\n# Weaknesses\n- The paper oversells it's empirical results. In the abstract it claims to outperform SOTA algorithms, but SOTA on Procgen is not PPO or RAD, but IDAAC (https://arxiv.org/abs/2102.10330). Further, RAD isn't even the SOTA method which applies data augmentation on Procgen, as that's UCB-DrAC (https://arxiv.org/abs/2006.12862). In many places the authors claim the method consistently outperforms the baselines, but they only perform better than both the baselines in 8 out of 16 games in terms of test performance, and many of those improvements aren't outside the confidence intervals. I think the language around the comparisons to baselines should be made less grand and more accurate.\n- Further, it would be beneficial to actually compare to SOTA results, which can be taken from the IDAAC paper https://arxiv.org/abs/2102.10330. Or justify why your method shouldn't be compared against SOTA results\n- The presentation of results in Table 1, where their method is bolded if it outperforms a single baseline, is non-standard and unfairly promotes their method (the other methods aren't bolded if they outperform any other method). I think this practice shouldn't be done.\n- The method implicitly relies on the visual inputs being able to be clustered into distinct visual themes, which Procgen does have, but many other methods do not. It would be worth investigating whether this method works on benchmarks with less distinct clusters of visual themes.\n- In general, no limitations of the method are discussed, and I think they should be to ensure that the method's strengths and weaknesses are fully understood.\n- The related work section could use more discussion of other approach which take an adversarial approach to improving generalisation, such as https://arxiv.org/abs/2102.07097, https://arxiv.org/abs/2106.15587 and https://arxiv.org/abs/1703.02702.\n\n# Questions\n- In table 1, what do you mean by \"the mean of the best test results\"?\n\n# Suggestions for improvements.\n\n## Large:\n- Evaluating the method on another generalisation benchmark, such as distracting control suite (https://arxiv.org/abs/2101.02722), would better validate the usefulness of this method. I think the idea of using style-transfer for adversarial data augmentation is interesting, and so I don't think that SOTA results are required, but showing improved results on a range of benchmarks would add evidence for the method being generally applicable and useful.\n- In particular, the method seems useful from two perspectives: (1) from a sample-efficiency point of view as data augmentation and auxiliary representation learning losses often increases that. (2) from a generalisation perspective, *if* the training environment doesn't contain every combination of visual style and underlying state-space level (as is the case for procgen). I think an evaluation protocol for Distracting Control Suite could disentangle these two perspectives further. \n\n## Small:\n- Typo: Pg 5 \"irreverent\" -> \"irrelevant\"\n- Reporting min-max normalised score, aggregated across games, and displaying performance profiles, as motivated in https://arxiv.org/abs/2108.13264, would improve the results reporting dramatically, and allow a scalar comparison between their method and the baselines.\n- When using it as a verb I think you should write \"style-translate\", or just \"translate the style of...\" rather than \"style translate\" without the hyphen.\n- Describing Procgen as \"Visually enriched\" implies you've changed (enriched) the procgen environment with more visual variety, which I don't think you have. Procgen is an established benchmark, I don't think you need to prefix it with adjectives when you mention it. Further, you describe the evaluation protocol you use as if you created it, but it's the standard evaluation protocol for easy mode in Procgen, which I think is worth mentioning.\n- In the related work section, it's unclear why Burda et al 2018 is in the list of approaches using data augmentation (as it targets exploration, not generalisation), and Cobbe et al 2019 is in the list twice, and Kostrikov et al 2020 three times.", "summary_of_the_paper": "The paper propose a method for improving generalisation and robustness across visual factors of variation in reinforcement learning, using a style transfer network to adversarially perturb the input to the policy, while the policy is trained to be invariant to this visual perturbation. The style transfer network is trained using StarGAN, based on domains created through a Gaussian Mixture Model clustering of the observations from the policy's experience. The method is demonstrated on OpenAI Procgen easy mode, and compared against PPO and RAD. The method does better than both of these baselines on 8 of the 16 games. Ablations on a single game show that the method is fairly robust to different choices of hyperparameters (number of clusters for the GMM, and weight of the adversarial loss for the style transfer network and policy).", "summary_of_the_review": "While I think the idea motivating this paper is interesting and potentially useful, I think the execution of the idea in this paper, the lack of strong or varied empirical results, the overselling of the effectiveness of the method, and the incomplete discussion of related works makes me recommend the paper is rejected. Several of these criticisms are relatively easy to address, but overall I'm unlikely to recommend acceptance unless the authors add encouraging empirical results on a different benchmark, or vastly improve the results they currently have.\n\nEDIT: after seeing the response from the authors I have raised my score from 3 to 5, and my correctness score from 2 to 3. I believe the improvements are worthwhile but don't make the paper worthy of acceptance: crucially, I think better comparisons with baselines (such as UCB-DrAC, and a more performant SAC implementation), and combining the method with an off-policy method (such as SAC or QT-Opt) would make the empirical justification of the idea more robust and make me more likely to accept the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ehqOo4IwChG", "reviewer": "Reviewer_eK4b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To learn robust features for decision-making, the generator of the StarGAN is encouraged to style transfer the state space such that the action distribution under the new transferred state will be distinct in terms of KL divergence with the original policy. The generator is further trained with a cyclic consistency loss.\n\nThe domain discriminator tries to tell about different domains.\n\nUsing Proximal Policy Optimization (PPO) as an example, the policy network is trained with an augmented loss of minimizing the KL divergence of the action distribution between original and style-transfered state.\n\n\nExperiments are done on the Procgen environments in comparison to PPO and RAD(data augmentation). Ablation study is done with respect to different linear coefficients, combining the different losses in the adversarial learning.\n\n# Strong points\n- The paper integrate style-transfer into the training of reinforcement learning algorithms to enhance the algorithm robustness against spurious features.\n- The experimental setting is explained relatively in detail. Experimental results look promising.\n\n#Issues\n1. In section 3.3, GMM is only used in the beginning to generate different clusters for usage in Star-GAN learning. Will this be sensitive to initial sample distribution?\n2. Could you elaborate on how the GMM clustering is done? Simply pool all state observations and do the clustering? e.g. for a tuple s,a,s', s and s' are treated  equally?\n\n## Minor issues\n1. 3rd paragraph in section 1, there is a citation on Anonymous.\n\n# Questions\n1. Could you explain the benefit of the gradient penalty term in Eqn. (6)?\n", "summary_of_the_paper": "\nTo achieve more robust behavior of reinforcement learning algorithms, in the deep learning part,  confounding features need to be removed from the decision-making process. To achieve this, the paper proposed first clustering the state space into $n$ clusters/domains and use StarGAN to style-transfer from one cluster to another.", "summary_of_the_review": "I like the innovative point that policy is not only learned to optimize return but also robustness against style transfer of state observation. Compared to data augmentation methods, the proposed methods showed promising methodology and empirical advantage.\nIssues in the above section need to be addressed however.\n.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bT72SLl3gvU", "reviewer": "Reviewer_f6jT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The idea of combining style transfer and RL to improve generalization is reasonable. As far as I know, the idea is novel in this setting.  \n\nWeakness \n- There are multiple overclaim in this paper. 1) In the abstract, there is no empirical\nnor theoretical evidence that ARPO finds optimal policy. It is highly skeptical that\nARPO can find an optimal policy in practice as the added KL regularization will rarely converge to zero. \n- It might be better to discuss whether the policy will converge to the optimal policy, where KL term in eq 3 reaches zero at the end. \n- An important ablation study could be showing how does the KL regularization change during the\ntraining. \n- The three run averages of results present large variance, and it is not very convincing to me ARPO\nreally achieves statistically better generalization results. Based on the results in Table 1, \nthere is no significant improvement from ARPO compared to baselines.\n- There is no discussion nor ablations on whether cycle consistency loss finds the\nunderlying structure of MDP. It would be better at least to provide examples of style-transferred images compared with original images.", "summary_of_the_paper": "This work proposes Adversarial Robust Policy Optimization (ARPO) that trains a generator to transfer the observation style so that policy taking different actions, the policy optimizes the long-term reward plus a regularization that minimizes the KL divergence between policy outputs given style-transferred image and original image. In this way, ARPO tries to learn underlying semantic representations for better generalization and being robust to the perturbations. The experimental results on some Procgen envs show improved results compared to PPO and RAD.", "summary_of_the_review": "This paper proposes a novel idea that combines style transfer with RL to improve the\ngeneralization performance. The experimental results show the proposed approach is competitive with the existing baselines. The results could be better improved by adding more runs and more ablations that approved the effectiveness of ARPO. More theoretical studies are needed to understand whether the algorithm converges to optimal policy. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "14k63Ma4ndB", "reviewer": "Reviewer_WXXr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The general idea of this paper is to train a GAN-like architecture to increase the robustness of RL algorithms to shifts in the input distribution. The Policy Network uses a PPO style loss to train the policy, with the addition of a KL term to ensure that perturbed X’ result in similar actions being taken by the policy. This is a good idea and the authors explain it very clearly in section 3, which I found enjoyable to read. Some more specific comments below:\n\n\nThe KL constraint in equation 3 is the most straightforward way to go about ensuring close observations are mapped continuously into the policy space (in the sense that a small delta in observation space causes a small delta in actions). Did you try a simple MLE loss on the mean of the output distribution? There is always some worry that this sort of KL constraint will lead to mode collapse. But in practice that doesn’t seem to be a problem? Did you consider alternative distributions to parameterize the policy? Some sort of catagorical distribution trick as in DreamerV2? Equation 5 also has a similar KL term. Often, I’ve found that in this kind of setup, it can also help to constrain the policy towards a normal distribution (add a KL(N(0,1), pi) term), to help avoid mode collapse. A similar trick is presented in PEARL. \n\nThe clustering pipeline and use of ResNet features was interesting. I think this is a small idea but also novel in my experience.  \n\n\nIs RAD the SOTA data augmentation technique? I’d be interested in seeing comparisons with DRQV2, SODA, SVEA, PAD, and SAC.  What about Learning Task Informed Abstractions, a recent publication on a similar benchmark?\n\nYou compare against PPO rather than something like SAC. I suppose that’s fair because this algorithm is based on a PPO style objective. Is on-policy learning truly needed in this case? Would the algorithm not benefit from keeping a buffer as in SAC? I am a bit concerned that this algorithm only marginally improves on PPO in many environments, which is itself a very weak baseline. \n\nResults in table 1 are somewhat hard to parse because the scores are not normalized. It would be ideal if the scores could be normalized, or the results could be presented as a percentage. Maybe in an appendix. I obviously understand the desire to present the raw data. \n\nI think the sample efficiency experiment is important, and I wish more papers included this kind of plot.  \n\nThe beta value chosen in these experiments is much higher than I would expect. How sensitive is the algorithm to this value? Figure 6 B looks at different values of beta and seems to suggest that this value doesn’t matter very much? Is there any intuition as to why? What happens if beta is set to 0? Is this term even necessary? \n\nIn general, the related work section could use some help. There is a rich history of style transfer objectives in inverse reinforcement learning that is missed (AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos). This section also runs into the problem of simply listing a lot of different work, but fails to adequately discuss it in the context of the present paper. I’d really like to see this section revised. ", "summary_of_the_paper": "An algorithm is proposed for using a GAN-like objective to generate observation perturbations, thus making RL agents more robust to context shift. This work falls nicely into the recently growing field of data augmentation methods in RL. Results are presented on Procgen. ", "summary_of_the_review": "This method represents an incremental improvement gained by developing a novel architecture. This paper — and the success of the proposed methods — was not surprising. But of course the authors of course should receive credit for getting things to work so well. The quality of results generally ranges from acceptable to quite good, although the quality of ablations falls far behind some comparable papers and can be improved.  Overall, a good paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussions. This papers is borderline with reviewers WXXr and eK4b leaning towards acceptance and reviewers f6jT and FV5x leaning towards rejection. On the positive side, reviewers remarked that the paper is interesting [FV5x] and novel [FV5x,f6jT,eK4b,WXXr]. However, there all reviewers found some flaws with respect to the execution and empirical validation [FV5x], specifically around lacking baselines [FV5x,WXXr] and some ablations [f6jT,WXXr]. I side with the comment made by reviewers FV5x as well as WXXr that a comparison to stronger baselines (UCB-DrAC) is warranted. Therefore, I recommend that this paper is not ready for publication at this point and that it will benefit greatly from another iteration with stronger empirical results. I want to very strongly encourage the authors to further improve their paper based on the reviewer feedback.
Decision: Reject