Robust Policy Optimization in Deep Reinforcement Learning
Authors: Md Masudur Rahman
Yexiang Xue
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Reinforcement Learning
Policy Optimization

Abstract: Entropy can play an essential role in policy optimization by selecting the stochastic policy, which eventually helps better explore the environment in reinforcement learning (RL). A proper balance between exploration and exploitation is challenging and might depend on the particular RL task. However, the stochasticity often reduces as the training progresses; thus, the policy becomes less exploratory. Therefore, in many cases, the policy can converge to sub-optimal due to a lack of representative data during training. Moreover, this issue can even be severe in high-dimensional environments. This paper investigates whether keeping a certain entropy threshold throughout training can help better policy learning. In particular, we propose an algorithm Robust Policy Optimization (RPO), which leverages a perturbed Gaussian distribution to encourage high-entropy actions. We evaluated our methods on various continuous control tasks from DeepMind Control, OpenAI Gym, Pybullet, and IsaacGym. We observed that in many settings, RPO increases the policy entropy early in training and then maintains a certain level of entropy throughout the training period. Eventually, our agent RPO shows consistently improved performance compared to PPO and other techniques such as data augmentation and entropy regularization. Furthermore, in several settings, our method stays robust in performance, while other baseline mechanisms fail to improve and even worsen the performance.

Under review as a conference paper at ICLR 2023
ROBUST POLICY OPTIMIZATION IN DEEP REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Entropy can play an essential role in policy optimization by selecting the stochas- tic policy, which eventually helps better explore the environment in reinforcement learning (RL). A proper balance between exploration and exploitation is challeng- ing and might depend on the particular RI. task. However, the stochasticity of- ten reduces as the training progresses; thus, the policy becomes less exploratory. Therefore, in many cases, the policy can converge to sub-optimal due to a lack of representative data during training. Moreover, this issue high-dimensional environments. This paper investigates whether keeping a certain entropy threshold throughout training can help better policy learning. In particu- lar, we propose an algorithm Robust Policy Optimization (RPO), which leverages perturbed Gaussian distribution encourage high-entropy actions. We evalu- ated our methods on various continuous control tasks from DeepMind Control, OpenAI Gym, Pybullet, and IsaacGym. We observed that in many settings, RPO increases the policy entropy early in training and then maintains a certain level of entropy throughout the training period. Eventually, our agent RPO shows con- sistently improved performance compared to PPO and other techniques such data augmentation and entropy regularization. Furthermore, several settings, our method stays robust in performance, while other baseline mechanisms fail to improve and even worsen the performance.
be severe in
INTRODUCTION
Exploration in a high-dimensional environment is challenging due to the online nature of the task. In reinforcement learning (RL) setup, the agent is responsible for collecting high-quality data. The agent has to decide on taking action which maximizes future return. In deep reinforcement learning, the policy and value functions representing complex functions with continuous action space. If explored well, the learned policy will more likely lead to better data collection and, thus, better policy. However, in high- sigh-dimessional observation space, the possible trajectories are larger; thus, having representative data is challenging. Moreover, it has been observed that deep RL exhibit the primacy bias, where the agent has the tendency to rely heavily on the earlier interaction and might ignore helpful interaction at the later part of the training (Nikishin al. 2022).
often represented neural networks due to their flexibility in
Maintaining stochasticity in policy is considered beneficial, as it can encourage exploration (Mnih al. 2016 Ahmed et al., 2019). Entropy is the randomness of the actions, which is expected to go down as the training progress, and thus the policy becomes less stochastic. However, lack of stochasticity might hamper the exploration, especially in the large dimensional environment (high state and action spaces), as the policy can prematurely converge to suboptimal policy. This scenario might result in low-quality data for agent training- In this paper, we are interested in observing the effeet when maintain certain level of entropy throughout the training and thus encourage exploration.
We focus on policy gradient-based approach with continuous action spaces. A common practice (Schulman et al. 2017 2015)) is to represent continuous action as the Gaussian distribution and learn neural net-
the parameters (μ, and σ) conditioned work, and it takes the state as input and outputs the one Gaussian parameters per action dimension. Then the final action is chosen sample from this distribution. This process inherently introduces
the state. The policy be represented

---

Under review as a conference paper at ICLR 2023
hopper_hop
DeepMind Control Envs
0.4
12.5
Policy Entropy
10.0
7.5
5.0
Normatized
2
0.2
-5.0
PPO
1
0.0
-4
-2
-10.0
-7.5
RPO (D0r5)
X
Timestops
PPO RPO (ours)
Figure 1: [Left] Standard Gaussian and corresponding Perturb Gaussian distribution (ours). We observe that the probability density is less centered around the mean in distribution. [Middle] Policy entropy at different timesteps of training in DeepMind Hopper Hop Environment. Similar patterns are observed for other evaluated environments. The PPO agent who uses Standard Gaussian starts from a particular entropy and becomes less stochastic as the training progresses and reduces the policy entropy. In contrast, our agent RPO uses our perturbed Gaussian distribution, increases the entropy at the initial timestep, and then maintains a certain level of entropy throughout the training. [Right] Our agent RPO shows over 3x performance improvement in normalized return compared to the base PPO agent on 12 DeepMind control environments. The results are averaged over 10 random seed runs.
randomness in action as every time it samples action for the same state, the action value might dif- fer. Though, in expectation, the action value is the same as the mean of the distribution, this process introduces some randomness in the learning process. However, as time progresses, the randomness might reduce, and the policy becomes less stochastic.
We first notice that when this method is used to train PPO-based agent, the entropy of the policy starts to decline as the training progresses, as in Figure [ even when the return performance is not good. Then we pose the question; what if the agent keeps the policy stochastic or entropy throughout the training? The goal is to enable the agent to keep exploring even when it achieves a certain level of learning. This process might help, especially in high-dimensional environments where the state and action spaces often remain unexplored. We developed an algorithm called Robust Policy Optimization (RPO), which maintains stochasticity throughout the training. We notice a consistent improvement in the performance of our method in many continuous control environments compared to standard PPO.
Seeing the data augmentation through the lens of entropy, we observe that empirically, it can help the policy achieve a higher entropy than without data augmentation. However, this process often requires prior knowledge about the environments and a preprocessing step of the agent experience. Moreover, such methods might result in an uncontrolled increase in action entropy, eventually ham- pering the return performance (Raileanu et al. 20200 Rahman & Xue 2022). Another way control entropy is to use an entropy regularizer (Mnih et al.) 2016; Ahmed et al. 2019), which often shows beneficial effects. However, it has been observed that increasing entropy in such a way has little effect in specific environments (Andrychowicz et al. 2020). These results show difficulty in setting proper entropy throughout the agent's training.
To this end, in this paper, we propose a mechanism for maintaining entropy throughout the training. We propose to use new distribution to represent continuous action instead of standard Gaussian. The policy network output is still the Gaussian distribution's 's mean and standard deviation in our setup. However, we add a random perturbation on the mean before taking an action sample. In particular, we add a random value ~ Finally, the action is taken from the perturbed Gaussian distribution a N a)) The resulting distribution is shown in Figure [] We see that the resulting distribution becomes flatter than the standard Gaussian. Thus the sample spread more around the center of the mean than standard Gaussian, whose samples are more concentrated toward means. The uniform random number does not depend on states and policy parameters and thus can help the agent to maintain a certain level of stochasticity throughout the training process. We name approach Robust Policy Optimization (RPO) and compare it with the standard PPO algorithm and other entropy- contopy-oottooled methods such as data augmentation and entropy regularization.
a) to the mean μ to get perturbed mean μí u z.
2

---

Under review as a conference paper at ICLR 2023
We evaluated our method in several environments from DeepMind Control T Tuunyasuvunakoo al. 2020). OpenAI Gym (Brockman et al. 2016). PyBullet (Coumans Bail 2016 2021), and Nvidia IsaacGym Makoviychuk et al., 202I.. We observed that our method RPO performs consistently better than PPO, and the performance improvement shows larger in high-dimensional environments. Moreover, RPO outperform two data augmentation-baset method: RAD (Laskin et al. (020)) and DRAC (Railcanu et al., 2020). In addition to that, our method is simple and free from the type of data augmentation assumption and data preprocessing; still, our method achieves better empirical performance than the data augmentation-ben based methods in many environments.
Further, we tested our method against the entropy regularization method, where the entropy of a policy is controlled by coefficient weight of the policy entropy. Empirically, we observe that our method RPO performs better in most environments, and some choice of coefficient eventually leads to worse performance. Moreover, we observe that even when the agent is trained for large simulation experience, the performance might not stay consistent. In particular, in IsaacGym envi- ronments, the agent has access to large sample due to the high-performing simulation GPU, This abundance of data may not necessarily improve or maintain the performance. In the classic CartPole environment, we observe that PPO agents quickly achieve a certain performance; however, it then fails to maintain it, and the performance quickly drops as we train for more. Similar results have been found in the OpenAl Wapker environments. On the other hand, method RPO shows robustness to more data and keeps improving or maintaining a similar performance as we keep training agents for more data.
In summary, we make the following contributions:
We investigate whether keeping the policy stochastic throughout the training is beneficial for the RL training.
We propose algorithm, Robust Policy Optimization (RPO), that uses a perturbed Gaus- sian distribution to represent actions and empirically shown to maintain a certain level of policy entropy throughout the training.
We evaluate our method on 18 tasks from four RL benchmarks. Evaluation results show that our method RPO consistently performs better than standard PPO, two data argumentation--asse methods RAD and DRAC, and entropy regularization.
2 PRELIMINARIES AND PROBLEM SETTINGS
Markov Decision Process (MDP). An MDP is defined tuple M (S,A,P, R). Here an agent interacts with the environment and take an action a₂ E A at a discrete timestep when at state 8, S. After that the environment moves to next state 81+1 S based on the dynamic transition probabilities P(st+1l St, ax) and the agent recieves a reward r; according to the reward function R.
Reinforcement Learning In a reinforcement learning framework, the agent interacts with the MDP, and the agent's goal is to learn a policy и E II that maximizes cumulative reward, where IT is the set of all possible policies. Given a state, the agent prescribes an action according to the policy n, and an optimal policy π² e II has the highest cumulative rewards.
Policy Gradient. The policy gradient method is a class of reinforcement learning algorithms where the objective is formulated to optimize the cumulative future return directly. In a deep reinforcement learning setup. the policy can be represented as a neural network that takes the state input and out- put actions. The objective of this policy network is to optimize for cumulative rewards. In this paper, we focus on a particular policy gradient method, Proximal Policy Optimization (PPO) (Schulman et al. 2017), which is similar to Natural Policy Gradient (Kakade (001)) and Trust Region Policy Optimization (TRPO) (Schulman et al. 2015). The following is the objective of the PPO:
Cx -E, Ta(a st)
A A(se; dt)]
where (e(a)(s:) is the probability of choosing action ae given state 8, at timestep using the current policy parameterized by 6. On the other hand, the Te (a,|s:) refer the probabilities of using an old policy parameterized by previous parameter values oota- The advantage A(s,, a,) is an estimation which is the advantage of taking action a at 8 compared to average return from that state.
3

---

Under review as a conference paper at ICLR 2023
In PPO, a surrogate objective is optimized by applying clipping on the current and old policy ratio on equation [ In the case of the discrete action, the output is often used as the categorical distribution, where the dimension is the number of actions. On the other hand, in continuous action cases, the output can correspond to parameters of Gaussian distribution, where each action dimension is represented by mean and variance. Finally, the action is taken the sample from this Gaussian distribution. In this paper, we mainly focus on the continuous action case and assume the Gaussian distribution for action.
Entropy Measure Entropy is a measure of uncertainty in the random variable. In the context of our discussion, we measure the entropy of a policy from the action distribution it produces given a state. The more entropic an action distribution is, the more stochastic the decision would be as the final action eventually a sample from the output action distribution. This measure can be seen as the policy's ' stochasticity, and higher entropy distribution allows agents to explore the state space more. Thus, entropy can play role in controlling the amount of exploration the agent performs during learning. In this paper, we focus on the entropy measure and refer to the stochasticity of policy by this measure.
3. ROBUST POLICY OPTIMIZATION (RPO)
Exploration at the start of training can help agent to collect diverse and representative data. How- ever, as time progresses, the agent might start to reduce the exploration and try to be more certain about state. In standard PPO, we observe that the agent shows more randomness in action at the beginning (see Figure T.. and it becomes smaller as the training progresses. We measure this randomness in the form of entropy, which indicates the randomness of the actions taken. In the con- tinuous case, a practical approach is to represent the action as a parameter of Gaussian distribution. The entropy of this distribution represents the amount of randomness in action, which also indicates how much the policy can explore new states.
However, in Gaussian distribution, the samples are concentrated toward the mean; thus, it can limit the exploration in some RL environments, This paper presents an alternative to this Gaussian distri- bution and proposes a new distribution that accounts for higher entropy than the standard Gaussian. The goal is to keep the entropy higher or at some levels throughout the agent training. In partic- ular, we propose to combine Gaussian Distribution and Uniform distribution. At each timestep of the algorithm training, we perturb the mean μu of Gaussian N N(H, σ) by adding a random number drawn from the Uniform u(-a, a) and get the perturbed mean μl. Then the sample for action is taken from the new distribution N [(u, a). In this setup, the Gaussian parameters still depend on the state, similar to the standard setup. However, the Uniform distribution does not depend on the states. Thus as training progresses, the standard Gaussian distribution can be less entropic the policy op- timizes to be more confident in particular states. However, the state-independent perturbation using the Uniform distribution still enables the policy maintain stochasticity.
Figure [ shows the diagram of the resulting Gaussian distributions. This diagram is generated by first sampling data from a Gaussian distribution with mean μ 0.0 and standard deviation a 1.0 and then corresponding perturbed mean μ² = + and standard deviation a = 1.0, where 2 ~ 2((33.0, 3.0). We see that the values are less centered around the mean for our Perturb Gaussian than the standard. Therefore, samples from our proposed distribution should have more entropy than the standard Gaussian.
Algorithms [ shows the details procedure of our RPO method. The agent first collects experience trajectory data using the current policy and stores in buffer 'D (lines to 10). Next, the agent uses the standard Gaussian to sample the action in these interaction steps (line 6). Finally, these experience data are used to update the policy, optimizing for policy parameter 0. In this case, given state St, the policy network outputs μ and for each action. Then the μ is perturbed by adding value 2, which samples from a Uniform distribution 4(-a, a.) (lines 13 and 14 in blue). After that, the probability is computed from the new distribution, A ((p',, .), which is eventually used to calculate action log probabilities that are used for computing loss Ly. In the case of the a hyperparameter, report results using 0.5 unless otherwise specified.
4

---

Under review as a conference paper at ICLR 2023
Algorithm Robust Policy Optimization (RPO)
Initialize parameter vectors for policy network.
2: for each iteration do
3: D (
4: for each environment step do
5:
p, + (T(-) o)
6:
7:
8:+1 P(s az)
8:
R(s,, at)
9:
D + DU ((8,, at, Tt, 8(+1)
10:
end for
11:
for each observation 8; in D do
P40 a (s(ss)))) 2~U(-a,a) p' p'+++2
13:
15:
prob + N
logp t prob(aa)
17:
Compute RI. loss T. using logp, as, and value function.
18:
end for
19:
end for
4 EXPERIMENTS
4.1 SETUP
Environments We conducted experiments on continuous control task from four reinforcement learning benchmarks: DeepMind Control (Tunyasuvunakool al., 2020), OpenAl Gym (Brock- man et al. 2016), PyBullet (Coumans & Bail 2016 2021), and Nvidia IsaacGym (Makoviychuk al. 2021). These benchmarks contain diverse environments with many different tasks, from low to high-dimensional environments (observations and actions space). Thus our evaluation contains a diverse set of tasks with various difficulties. The IsaaeGym contains environments that run in GPU, thus enabling fast simulation, which eventually helps collect large amount of simulation experience quickly, and faster RL training with GPU enables deep reinforcement learning models.
Baselines We compare our method RPO with the PPO (Schulman et al. 2017) algorithm. Here our method RPO uses the perturbed Gaussian distribution to represent the action output from the policy network, described in Section In contrast, the PPO uses standard Gaussian distribution to represent its action output. Further, we observe that the data augmentation method can help in- crease the policy's entropy by often randomly perturbing observations. This process might improve the performance where higher entropy is preferred. Thus, we compare our method with two data augmentation-baser methods: RAD (Laskin al. (2020), and DRAC (Raileanu al., 2020). Here, The pure data augmentation baseline RAD uses data processing before passing it to the agent, and the DRAC uses data augmentation to regularize the value and policy network. Both of these data augmentation methods use PPO as their base RL policy. Another common approach to increase the RI. objective. A coefficient determines how
entropy to use the Entropy Regularization much weight the policy would give to the entropy. We observe that various weighting might result in different levels of entropy increment. We use the entropy coefficient 0.0. 0.01, 0.05, 0.5, 1.0, and 10.0 and compare their performance in entropy and, in return, with algorithm's 's RPO. Note that our method does not introduce any additional myper-parameters and does not use the entropy coefficient dyper-parameter
The implementation details of algorithm and baselines are in the Appendix. To account for the stochasticity in the environment and policy, we run each experiment each several times and report the mean and standard deviation. Unless otherwise specified, experiment with 10 random seeds.
4.2 RESULTS
We compare the return performance of our method with baselines and show their entropy to contrast the results with the entropy of the policy.
5

---

Under review as a conference paper at ICLR 2023
humanoid stand
humanold run
hopper hop
quedruped walk
Tain Retur
PR0 RPO (OUIS)
PPC RPO lours)
1000
120
EPO tours
500
500
HPD (tuns)
20
300
200
2M
7M
GH
Timesteps
Timesteps
Timesteps
Timesstapp
quadruped esccpe
walcor stand
Walkerr
100
fish swimm
400
P10
PMQ
Tten fteton
90
(oun)
010
touns)
(ouns)
Trein Beloon
(surs)
9110
Tren
400
100
Timesteps
Timesteps
Timesteps
Timesteps
Figure 2: Results on DeepMind Control Environments. PPO agent fails to learn any useful behavior and thus results in low episodic return in some environments (humanoid stand, humanoid run, and hopper hop). Overall, our method RPO performs better and achieves a much higher episodic return. The RPO also shows better mean return than the PPO in other environments. In many settings, PPO agents stop improving their performance after around 2M timestep while RPO consistently improves over the entire training time.
Comparison with PPO This evaluation is the direct form of comparison where no method uses any aid (such as data augmentation or entropy regularization) in the entropy value. Figure D results comparison on DeepMind Control Environments. Results more environments are in Appendix Figure TT The entropy comparison is given in the Appendix Figure [2] In most scenarios, our method RPO shows consistent performance improvement in these environments compared to the PPO. In some environments, such as humanoid stand, humanoid run, and hopper hop, the PPO agent fails learn any useful behavior and thus results in low episodic return. In contrast, our method RPO shows better performance and achieves a much higher episodic return. The RPO also shows a better mean return than the PPO in other environments. In many settings, such as in quadruped (walk, run, and escape), walker (stand, walk, run), fish swim, acrobot swingup, the PPO agents stop improving the performance after around 2M timestep. In contrast, our agent RPO shows consistent improvement over time of the training. This performance gain might be due to the proper management of policy entropy, as our setup, the agent encouraged to keep exploring as the training progress. On the other hand, the PPO agent might settle in sub-optimal performance as the policy entropy, in this case, decreases as the agent trains for more timesteps. These results show the effectiveness of our method in diverse control tasks with varying complexity.
Results of OpenAl Gym environments: Pendulum and BipedalWalker are in Figure 3 Overall, our method, RPO, performs better compared to the PPO. In Pendulum environments, the PPO agent fails to learn any useful behavior in this setup. In contrast, RPO consistently learns with the increase in timestep and eventually learns the task. We see the policy entropy of RPO increases initially and eventually remains at certain threshold, which might help the policy to stay exploratory and collect more data. In contrast, the PPO policy entropy decreases over time, and thus eventually, the performance remains the same, and the policy stops learning. This scenario might contribute to the bad performance of the PPO.
In the BipedalWalker environment, we see that both PPO and RPO learn up to a certain reward quickly. However, as we keep training both policies, we observe that after a certain period, the PPO's performance drops and even starts to become worse. In contrast, the RPO stays robust we train for more and eventually keep improving the performance. These results show the robustness of our method when ample train time is available. The entropy plot shows similar pattern, as PPO decreases entropy over time, and RPO keeps the entropy at a certain threshold.
Figure shows results comparison on PyBullet environments: Ant, and Minituar. We observe that our method RPO performs better than the PPO in the Ant environment. In the Minitaur environment, PPO quickly (at around 2M) learns up to certain reward and remains the same performance as time progresses. In contrast, RPO starts from a lower performance, eventually surpassing the PPO's s performance as time progresses. These results show the robustness of consistently improving the

---

Under review as a conference paper at ICLR 2023
Pendulum v1
Pendutum
Bipeda/Wallier
Bipeda/Walker
Retum
-100
-200
Policy Entrapy
Thin Bee
150
Palkyy Entrapy
-600
louns
(sum)
102
-1000
PRC
PAC RPo (ouns)
RPo toun))
1400
I0M
VW
SCM
10M
200
4MM
Tipe
Timestops
Timesteps
Figure 3: Results on OpenAI Gym Environments. The PPO agent fails to learn any useful behavior the Pendulum environment. In contrast, RPO consistently learns with the increase in timestep and eventually achieves higher scores. The entropy of RPO policy increases initially and eventually remains at a certain threshold. In contrast, the PPO policy entropy decreases over time. In Bipedal- Walker, the PPO fails to progress after achieving particular reward, ad eventually, the performance degrades with more training. In contrast, our RPO agent consistently improves performance over the entire training time.
AntBullctEnvvvO
AntBuiletEnv-vG
MinitaurBulitEnnvvv
MriitauBuuletEEww.o
2000
Trenn Betor
Polccy Entrery
Trein Deturn
Policy Entreryy
1200
PPO
8 E
mPo RPO (ouns
1000
RPO jours)
500
Timesteos
Timesteps
Timesteps
Timestops
Figure 4: Results on PyBullet Environments. Our method RPO consistently performs better than the PPO in the Ant environment. On the other hand, in the Minitaur environment, PPO quickly
learns up to a particular reward and remains on the same performance as time progresses while RPO surpasses the PPO's performance.
900
Carrpoae-saacdyy
Cartpole-liaccyym
Bail GyylBalance-kaacaCC
Bullallanca-saacdyy
Then hetom
Policy Entropy
Thein Bettrr
Policy Entrapr
FPO
(suns
1.0
200
0.5
and (ours)
Timesteps
Timesteps
Timesteps
Timesteps
Figure 5: Results on IsaacGym Environments. the Cartpole environment, the performance of PPO started to degrade over time, and the policy entropy kept decreasing. In contrast, our RPO agent keeps improving the performance over the entire training time. These results show the robustness of our method RPO over PPO even when an abundance of simulation is available. In the BallBalance environment, our method RPO achieves a slight performance improvement compared to PPO.
policy of the RPO method. The entropy pattern remains the same in both cases; PPO reduces entropy while RPO keeps the entropy at certain threshold which it learns automatically an environment.
Figure 5 shows results comparison on IsaacGym environments: Cartpole, and BallBalance. In this setup, we run the simulation up to 100M timesteps which take around 30 minutes for each run in each environment in a Quadro RTX 4000 GPU. We see that for Cartpole, both PPO and RPO learn the reward quickly, around 450. However, as we kept training for long. the performance of PPO started to degrade over time, and the policy entropy kept decreasing. On the other hand, our RPO agent keeps improving the performance; notably, the performance never degrades over time. The policy entropy shows that the entropy remains at threshold. This exploratory nature of RPO's policy might help keep learning and get better rewards. These results show the robustness of our method RPO over PPO even when abundance of simulation is available. Interestingly, more simulation data might not always be good for RL agents. In our setup, the PPO even suffers from further training in the Cartpole environment. In the BallBalance environment (results are averaged over 3 random seed runs), our method RPO achieves a slight performance improvement over PPO. Overall, our method RPO performs better than the PPO in the two IsaacGym environments.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Result comparison with data augmentation on DeepMind control environments. Our method RPO performs better in mean episodic return in most environments than PPO and other data augmentation baselines RAD and DRAC. Moreover, in some environments, the data augmen- tation baselines worsen the performance compared to the base PPO. The results are after training the agent for 8M timesteps. The mean and standard deviations are over 10 seed runs.
Eav
PPO
RAD-PPO
DRAC-PPO
RPO (ours)
acrobot swingup
23.93 #60
13.41 19.44
21.32
40.46 =4.01
fish swim
87,39 #80
81.53 46.33
87.5 47.74
119.42 4:3846
humunoid stand
+0.17
6.14 4037
55.22 #35.34
humanoid run
1.55 #1 59
1.1 20.04
13.13 1166
13.26 =9,45
pendulum swingup
518.63 2:79.73
23.04 1:1708
320.51 1:319,76
699.79
1:3.99
quadriped walk
216.67 666.5
195.07 112.99
374.78 197.98
437.66
1:1
quadruped run
183.14
=3844
158.01 #26.84
247. 027.0222 #83,25
256.75
49%.18 18
quadruped escape
22.51 47.0
54.0 4:22.13
32.62 21.88
53.37
4:25.7
walker stand
346.55 + 104.27
558.92
#:11.0M
361.99
4-93,44
652.7
#202.47
walker walk
329.61 67.92
333.22
150.52 425.31
611.88
1:770,16
walker run
123.63
253.5
141.43
=34.77
1 137.18
291.83
10888
quadruped, Tun
quadruped run
walker walker_stand stand
walker_ standd
ID ant coel--
Fing int_Satta
Toal Detum
250
300
on Soetwood
600
500
EoCCoed E3
IFl hault
Eno Eng_Sweek
120
Timeztops
2.0%
Timesteps
4.PM
8.MM
25
6M Thmestogs
2.5M
Thmresopp
4:50
6.7M
K0N
Figure 6: Comparison with Entropy Regularization. Sometimes (e.g., walker stand), the perfor- mance improves with a proper setup of tuned coefficient value. In other cases, the improper coef- ficient value often worsens the performance. In contrast, our method RPO consistently performs better than PPO and the different entropy coefficient baselines.
Comparison with Data Augmentation. In results on all DeepMind Control environments are shown in Table D Our method performs better in mean episodic return in most environments than PPO and other data augmentation baselines RAD and DRAC.
The return and policy entropy curve are shown in Appendix, Figure 10 We observe that the data augmentation slightly improved the base PPO algorithms, and the policy entropy shows higher than the base PPO. However, our method RPO still maintains a better mean return than all the baselines. The entropy of method shows an increase the initial timestep of the training. However, it eventually becomes stable at a particular value. The data augmentation method, especially RAD, shows an increase in entropy throughout the training process. However, this increase does not trans- late to the return performance. Moreover, data augmentation assumes some prior knowledge about the environment observation, and selecting a suitable data augmentation method requires domain knowledge. Improper handling of the data augmentation may result in worsen performance.
In addition, the augmentation method poses an additional processing time which can contribute to longer training time. In contrast, our method RPO does not require such domain knowledge and is thus readily applicable to any RL, environment. Moreover, in return performance, our method shows better performance compared to the base PPO and these data augmentation methods.
Comparison with Entropy Regularization
Due to the variety in the environments, the entropy requirement might be different. Thus, improper setting of the entropy coefficient might result in bad training performance. In the tested environ- ments, we observe e that sometimes the performance improves with a proper setup of tuned coeffi- cient value and often worsens the performance (in Fig However, our method RPO consistently performs better than PPO and the different entropy coefficient baselines. Importantly, our method does not use these coefficient param-ersseterss and still consistently performs better in various en- vironments, which shows our method's 's robustness in various environment variability.
We observe that in some environments, the coefficient 0.01 improves the performance of standard PPO with coefficient 0.0 while increasing the entropy. However, an increase in entropy to 0.05 and
8

---

Under review as a conference paper at ICLR 2023
quedruped run
quedruped
Jun
walker Jun
Thinn Putuur
1.00
Palcy Emacpy
Policy Emorgy
120
280
200
50
251
44
CH
2.N. 4.N 6.3H 0:314
254
DM
2.00 4.0% 5.0H 3.5M Timesleps
Timessepps
Timmsstess
Figure 7: Ablation on alpha values of the uniform distribution for RPO. An a value between 0.1 to 3 often results in better performance, while large value often results in worse performance.
above results an unbounded entropy increase. Thus, the performance worsens in most scenarios where the agent fails to achieve reasonable return. Results with all the coefficient variants in the Appendix Figure Overall, our method RPO achieves better performance in the evaluated environ- ments in our setup. Moreover, method does not use the entropy coefficient hyperparameter and controls the entropy level automatically in each environment.
Ablation Study We conducted experiments the value ranges in the Uniform distribution. Fig- ure 7 shows the return and policy comparison. We observe that the value of a affects the policy entropy and, thus, return performance. A smaller value of (e.g.. 0.001) seems to behave similarly to PPO, where policy entropy decreases over time, thus hampered performance. Higher entropy values, such as 1000.0, make the policy somewhat random the uniform distribution dominates over the Gaussian distribution (as in Algorithm [ This scenario keeps the entropy somewhat at constant level; thus, the performance is hampered. Overall, a value between 0.1 often results in better performance. Due to overall performance advantage, in this paper, we report results with 0.5 for all 18 environments. Learning curve for more environments are in Appendix Figure
5 RELATED WORK
Since the inception of the practical policy optimization method PPO (TRPO and Natural Policy Gradient), several studies have investigated different different algorithmic components (Engstrom et al.) 2019
Andrychowicz al. 2020; Ahmed et al.) 2019; Ilyas al. 2019). Entropy regularization enjoys
faster convergence and improves policy optimization (Williams & Peng. 1991 Mei et al., 2020
Mnih al. 2016 Ahmed et al. 2019). However, some empirical findings observe the difficulty in setting proper entropy setup and observe no performance gain in many environments (Andrychowiez
al. 2020 Gaussian distribution is commonly used to represent continuous action (Schulman
et al. 2015: 2017). Thus many follow-ups to implementation also use this Gaussian distribution
as s default setup (Huang et al.) [022] (021]. contrast, in this paper, we propose perturbed Gaussian distribution to represent the continuous action to maintain policy entropy. Furthermore, in this paper, we showed the empirical advantage of our method several benchmark environments compared to the standard Gaussian- method. Due to space limitations, we moved some related work discussions to the Appendix.
CONCLUSION
This paper investigates the effect of entropy in policy optimization-based reinforcement learning. We design a method that, when applied to a policy network, can keep the entropy of the policy to a certain threshold through training. Our approach uses a perturbed Gaussian distribution by random number sample from a uniform distribution to represent the policy. We observe that keep- ing certain entropy threshold throughout using our method can help better policy learning. Our proposed algorithm Robust Policy Optimization (RPO), performs better than the standard PPO on 18 continuous control tasks from four RL benchmarks, DeepMind Control, OpenAl Gym, Pybullet, and IsaacGym. Further, our method performs better in many settings than other approaches, en- couraging higher entropy, such as data augmentation and entropy regularization. Overall, we show our method's 's effectiveness and robustness where standard PPO or other baseline methods fail to maintain or even worsen the performance over time.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pp. 151-160. PMLR, 2019.
Marcin Andrychowiez, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et What matters for on- on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations, 2020.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybbllet..ogg 2016-2021
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International conference on learning representations, 2019.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, and Jeff Braga. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. 2021.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022. URL
io htt.thhub..//202//033333 http://iccrr-ooootreeeeeee 25/ 25/ppo implimmeeee wwwpplteeettttt. condeetalss/ implemmntatioo-dettalls/
Andrew Iyyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In International Con- ference on Learning Representations, 2019.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems, 33, 2020.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance pas-based physics simulation for robot learning, 2021.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference Machine Learning, pp- 6820-6822 PMLR, 2020.
Volodymyr Mnih Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Evgenii Nikishin, Max Schwarzer, Pierluca D' D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pp. 16828-16847. PMLR, 2022.
Md Masudur Rahman and Yexiang Xue. Bootstrap state representation using style transfer for better generalization in deep reinforcement learning. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2022), 2022.
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization deep reinforcement learning. arXiv preprint arXiv:2006. 12862, 2020.
10

---

Under review as a conference paper at ICLR 2023
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1899 PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-
dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv1707.06347, 2017.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi:
https///do.org/10.0016// http:///......00...............0 100022. URL wttL:ttttt///ww.siieeetee /ttp://wwinneeieeeetttttt com/
science/article//iL. /tt.rraaa.eeaaee///2222333332000
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcmeett learning
algorithms. Connection Science, 3(3):241-268, 1991.
A APPENDIX
A.I ADDITIONAL RELATED WORK
Another form of approach which adds entropy to the policy is data augmentation. We observe that the policy entropy often increases when the data is augmented often by some form of random processes. This improvement in entropy can eventually lead to better exploration in some environ- ments. However, the type of data augmentation might be environment specific and thus requires domain knowledge as to what kind of data augmentation can be applied. In contrast, to this method, our method maintains an entropy threshold during entire policy learning. Moreover, we evaluated our algorithm with two kinds of data augmentation-based methods RAD (Laskin et al. (020) and DRAC (Raileanu et al. (020). Finally, we evaluate with an effective data augmentation approach random amplitude modulation, which was found to be worked well in vector-based states set up.
Many other forms of improvement have been investigated policy optimization, which results in better empirical success, such Generalized Advantage Estimation (Schulman al. 2016) Normalization of Advantages (Andrychowicz et al., 2200)), and clipped policy and value objective (Schulman et al., 2017: Engstrom et al.) 2019; Andrychowicz et al. 2020). These improvements
eventually led to overall improvement and are now used in many standard implementations such as (Huang al. 2022 2021). In our implementation, we leverage these implementation tricks when possible. Thus our method works on top of these essential improvements. Furthermore, our method complementary to these approaches and thus can be used along with this method. In this paper, our implementation of baselines also used these implementation tricks, which is for a fair comparison.
A.2 EXPERIMENTS
Implementation Details: Our algorithm and the baselines are based on the PPO (Schulman et al. 2017) implementation available in (Huang al. 2022 2021). This implementation incorporated many important advancements from existing literature in recent years on policy gradient (e.g.. Or- thogonal Initialization, GAE, Entropy Regularization). We refer reader to (Huang et al. 2022 for further references. The pure data augmentation bascline RAD (Laskin et al.) (020)) uses data process- ing before passing it to the agent, and the DRAC (Raileanu al. (0200) uses data augmentation to regularize the loss of value and policy network. We experimented with rector-based states and used random amplitude scaling proposed in RAD (Laskin et al. 2020) as a data augmentation method for RAD and DRAC. In the random amplitude scaling, the state values are multiplied with random values generated uniformly between a range a to B. We used the suggested (Laskin et al. (020) and better performing range = 0.6 to B 1.2 for all the experiments. Moreover, both RAD and DRAC use PPO as their base algorithm. However, our RPO method does not use any form of data augmentation.
11

---

Under review as a conference paper at ICLR 2023
Table 2: Hyperparameters for the experiments.
Description
Value
Number of rollout steps
2048
Learning rate
3e 4
Discount factor gamma
0.99
Lambda for the GAE
0.95
Number of mini-batches
32
Epochs to update the policy
Advantages normalization
True
Surrogate clipping coefficient
0.2
Clip value loss
Value loss coefficient
0.5
We use the hyperparameters reportte in the PPO implementation of continuous action spaces (Huang et al. 2022, 202T), which incorporate best practices in the continuous control task. Furthermore, to mitigate the effect of hyperparameters choice, we keep them the same for all the environments. Further, we keep the same hyperparameters for all agents for fair comparison. The common hyperparameters can be found in Table 2
Entropy Regularization The results comparison of RPO with entropy coefficient are in Figure
Data augmentation return curve and entropy: Figure 10 shows data augmentation return curve and entropy comparison with RPO.
Results on DeepMind Control is in Figure
Entropy Plot for DeepMind Control is in Figure 12
Ablation Study Ablation on a values of RPO are shown in Figure 13
12

---

Under review as a conference paper at ICLR 2023
quadruped run
quadruped run
200
Coof-o Ent, Coaf-6.00 01
Coef=0
Train Return
300
PPO Ent Coaf-0.05 0.05 (ours)
Policy Entropy
PPO Ent_ Coef=0.01
PPO Ent_ Coef=0005
RPO (ours)
0
2M
4M
6M
8M
0.0
2.0M
4.0M
6.0M
8.0M
Timesteps
Timesteps
walker sttrrrssandddddd
bolker_stande
900
100
800
PPO Ent Coef=0
PPO Ent_ Coef=0
Train Return
700
PPO Ent Coef=0.01
Policy Entropy
Ent, Coef=0.01
600
PPO Ent Coef=0.05
PPO Ent Ent_ Coef=0.05
RPO (ours)
RPO (ours)
500
40
400
20
300
100
0
2M
4M
6M
8M
0,0
2.0M
4.0M
6.0M
8.0M
Timesteps
Timesteps
walker walk
walker walk
800
700
Coof=0.01
Coof-O
PPO Ent, Coef=0
Train Return
600
PPO Ent, Coof-0.05
Policy Entropy
60
Ent_ Coef=0.01
RPD (ours)
PPO Ent Coef=0.05
500
40
RPO (ours)
400
300
20
200
0
100
-20
0
2M
4M Timesteps
6M
8M
0.0
2.0M
4.0M
6.0M
8.0M
Timesteps
fish swim
fish swim
160
40
Ent Coefoooo
PPO Ent_ Coef=0
Train Return
140
Ent Coefoo
Policy Entropy
PPO Ent Coef=0.01
RPO (ours)
PPO Ent_ Coef=0.05
120
20
RPO (ours)
100
80
60
0
2M
4M
6M
8M
0.0
2.0M
4.0M Timesteps
6.0M
8.0M
Timesteps
humanoid stand
humanoid stand
100
300
PPO Ent Coaf-o
Coaf-6.01 Ent
Policy Entropy
Train Return
60
RPO (ours)
200
PPO Ent. int_Coef=0
PPO Ent Coef=0.01
PPO Coef=0.05
100
RPO (ours)
0
0
2M
4M
6M
8M
0.0
2.0M
4.0M
6.0M
8.0M
Timesteps
Timesteps
Figure 8: Comparison with Entropy Regularization.

---

Under review as a conference paper at ICLR 2023
quadruped run
quadruped run
200
300
Coof-o
Codfee
Train Return
PPO Ent Coaf-0.00 05
Ent, Coaf=6.00 01
Coef=0.5
Policy Entropy
250
200
PRO ENt 01
PRO Ent. Cost- 05
PFO tni, Cocle
Ert, Coef-1.8
Ent, Coaf-1.0
PPO Ent coof-10.0
150
100
50
0
100
0.0
0.2M
0.4M
0.6M
0.8M
1.0M
0.0
0.2M
0.4M
0.6M
0.8M
1.0M
Timesteps
Timesteps
fish_swim swim
fish swhi_swim
100
Ent_Coef=0
PR Cod_C PPO Ent Co
Ent, Coefoo 01
120
Train Return
90
PPO Ent, Coefoo 05
Policy Entropy
100
PPO trt Cod
Fez Comf-B
Ent Coe.=00
Coct-1.0
80
Ent Coef=10.0
60
40
70
20
0.0
0.2M
0.4M
0.6M
0.8M
1.0M
0.0
0.2M
0.4M
0.6M
0.8M
1.0M
Timesteps
Timesteps
walker stand
walker stand
550
160
500
Coof-O
140
MTC bit Cocleg
PRO Ent Coe--00)
Train Return
450
400
PP0 Ent, Coof-0.5
PPO Ent Coof-0.05
Policy Entropy
120
350
Coef=1.0
100
Cocf-1000
300
80
250
60
200
150
100
50
0.0
0.0
0.2M
0.4M
0.6M
0.8M
1.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Timesteps
Timesteps
walker walk
walker walk
400
160
Ent Cot_ooooo
PP Set Conf-
350
Train Retur
300
PPO Ent, Coef=0.5
Ent Coefoo
Policy Entropy
120
Cod-+tu5
Coef=1.0
100
Coef-160
200
60
150
40
100
0
0.0
-20
0.2M
0.4M
0.6M
0.8M
1.0M
0.0
0.2M
0.4M
0.6M
0.8M
1.0M
Timesteps
Timesteps
humanoid stand
humanoid stand
8
600
PPO Ent Coef-o
Coaf-0.01
Policy Entropy
500
ca 82
Train Return
PPO Ent Couf-0.5
400
Cont-1
PPO Ent Coof-1.0
bit Cout-10.2
5
Coaf-10.
300
4
200
2
100
b.o
0
0.2M
0.4M
0.6M
0.8M
1.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Timesteps
Timesteps
14
Figure 9: Comparison with Entropy Regularization on different coefficient values.

---

Under review as a conference paper at ICLR 2023
acrobot, swingup
acrobot swingup
pendulum swingup
pendulum swingup
Thain Beeur
Policy Brtopy
Train Return
Palicy Ertopy
rro
BRAC
KDD #PO
MPU tours)
BRACcmo
DRAC-910
W-d (ouns)
220
DBAC.990
nno leurs)
0n0 tours)
7M
414 Timestops
GH
BM
2M
IM
3M
3M
4V Timestops
GH
DM
0
2M
IM
GN
BM
Timestops
Timestops
fishh
hooper
hopper hop
Then hetumn
140
AAn mr
Policy Entrosry
100
Mn
Polkyy Entrapry
110 jorn
110 joun
PPG
RAD-mn0
Tram
0442- P91
10
(sun)
190 100
imesteps
limestepo
Timesteps
limesteps
Figure 10: Comparison with PPO and data augmentation RAD, and DRAC.
100
humanoid stand
30
humanold, run
140
hopper hop
700
quadruped, walk
Trai Return
20
Tran Retur
Train Retur
tours)
Tain Retur
60
13
10c
10
100
limesteps
Timesteps
Timesteps
limestepp
quadruppd run
cuadruped escape
walker stand
walker walk
PED
800
FRD
Thein Retumm
RPO loura)
RPO
1nd teurss
300
200
200
100
R-- (ourt)
2M
JM4 Timestops
64
BM
0
25
401 Timesseps
CM
3M
200
29
GM
DM
2M
GM
SM
BM
Timestops
Timestops
walker
fish swim
pendulum_swingue
acrobot_swill swingup
420
P7O
P1D
Tein heturr
lours)
Tali heture
140
RPO lours)
500
Tain Nettor
420
100
100
MP
(surs)
(ours)
Timesteps
Timesteps
limesteps
Timesteps
Figure 11: Results on DeepMind Control Environments. PPO agent fails to learn any useful behavior and thus results in low episodic return in some environments (humanoid stand, humanoid run, and hopper hop). Overall, our method RPO performs better and achieves a much higher episodic return. The RPO also shows better mean return than the PPO in other environments. In many settings, PPO agents stop improving their performance after around 2M timestep while RPO consistently improves over the entire training time.
15

---

Under review as a conference paper at ICLR 2023
70
humanold, stand
humerold run
hopper hop
quadruped walk
Polcy Entropy
Falcy Entog
Palcy Entrapy
75
PPO
Rno louns)
50
HPO
6n0 (surs)
255
PPO
R70 (oun)
-3.0
pa
PM
4V
Timesteps
Timestipss
Timesteps ps
Timesteps
quadruped
quadiruped ascape
walker stand
lort walk
Polkyy Entrgyy
Policy Entooyy
Policy Extropy
Polcy Sane
mo
Jeurs)
1903
RPD lours)
RF0 (ound)
-30
271
ON
2M
-20
2M
CM
SM
201
4H
64
BM
2H
4V Timesteps
CM
6M
Timesteps
Timesteps
Timesteps
walker
fish
pendulum swingup
scrobot
Policy Entropy
25
Policy Entropy
Pulcy Entropy
Pulcy Entrapy
hho (oun)
tours)
isurs
Timestops
Timestops
Timestops
Timestops
Figure 12: Results on DeepMind Control Environments.
quedruped
uped
32
Thin Baturr
100
0 250
320
10
200
150
300
2M1
4M
6M
EH
b0 2.N koN ON BCM Timesteps
0
28
JM Timestaps
8M
0M
6c 2.01 4.0M S 3 Timesteps
Timesteps
200
we Ieer walk
walker welt
300
humenold, stand
humenoldd stand
720
800
Tras Ratuur
350
300
10
"bo Timestegs
Timestops
Timestops
Tiresteps
quadruped escape
quadruped escape
Tal Return
Foltyy Ett
214
104
154
200 4:00 RMM KIN limesteps
Timestaps
Figure 13: Ablation alpha values of the uniform distribution for RPO.
16


Reviews:
[{"review_id": "kkNxrTsw6Ot", "reviewer": "Reviewer_fgod", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper argues for stochastic policies that maintain high entropy for high RL performance. In particular, the authors propose to randomly perturb the mean of the output Gaussian distribution to produce more diverse actions. In the experimental evaluation, this perturbation scheme is compared to PPO (with and without entropy regularization) and data augmentation techniques including RAD and DRAC.", "summary_of_the_review": "Overall, the novelty of the work is not clear to me. The experimental evaluation also lacks relevant baselines, and some details of their evaluation are currently unclear. Therefore, I am recommending a reject.", "strength_and_weaknesses": "The paper is overall written well. However, it’s not clear to me how adding action noise is different from existing ways of incorporating entropy regularization. For example, this is equivalent to enforcing that each component of \\sigma is at least some threshold. This clipping already exists in many PPO implementations I believe.\n\nI also have some questions and concerns in regards to the experimental evaluation:\n- Figure 2 shows that PPO fails to learn in these environments, but this is probably due to the entropy regularization being removed. A fairer comparison would be to PPO with the standard entropy regularization.\n- The comparison to the data augmentation methods is a bit confusing to me since these are methods that perform image augmentation for image-based RL tasks. Are the experimental tasks being solved from images?\n- There seems to be more useful comparisons to include, such as Soft Actor-Critic, which optimizes the maximum-entropy RL objective and automatically tunes the entropy coefficient, and exploration strategies for RL, such as Random Network Distillation.\n- What are the action spaces for each environment? It is my understanding that each action component is between -1 and 1 for the DM Control Suite tasks. Wouldn’t alpha values larger than or equal to 1 essentially make the actions uniform-random?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "TEn0E-BDKV", "reviewer": "Reviewer_JnzH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new algorithm RPO which builds on top of PPO. The algorithm takes the mean output of a Gaussian distribution, adds a uniform perturbation to the mean, then samples actions from the resulting Gaussian distribution with the perturbed mean. The proposed method is shown to be effective on a variety of high dimensional continuous control tasks.", "summary_of_the_review": "Overall, while the introduced method does appear to be novel and shows evidence of empirical improvements on a set of benchmarks, I don’t believe the paper in its current state advances our understanding of the field as a whole and therefore I cannot recommend the paper for acceptance at this time.\n", "strength_and_weaknesses": "Strength:\n- Proposed method is extremely straightforward, easy to implement, and appears to show good results.\n- Paper was easy to read and clearly written.\n\nWeakness:\n-  My initial problem with the paper is the word “robust” since the term itself is quite overloaded and can have different meanings in different fields (e.g. robust control has a very specific meaning). The authors’ use of the robustness throughout the paper seem quite vague and note well defined.\n- While the experiments are quite detailed and do show improvements on a set of challenging tasks over the benchmark, I do not believe they provided us with a better understanding of why the proposed approach worked well. Especially since the idea that its beneficial to maintain a certain degree of high entropy through training seems somewhat counter-intuitive in many scenarios. I believe some toy examples would be more helpful in this regard in showing us why/under what conditions the proposed method works well, I think this is something that would greatly enhance the quality of the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4Q4fK38f5gn", "reviewer": "Reviewer_JmVS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work the authors proposed a new robust policy optimization method in which the main approach is to randomly perturb the mean of the Gaussian policy to improve exploration. The main motivation is that stochastic policies that have high entropy tend to produce better RL performance, and so the authors propose to constantly perturb this policy during training. Empirically they showed that this new policy outperforms standard RL methods including the ones with data augmentation schemes (e.g., PPO, RAD, DRAC).\n", "summary_of_the_review": "Authors propose a new randomization scheme for RL policy training in order to improve exploration, and thus the performance of RL (PPO).\nWhile this method may show some empirical benefits in some small domain, there has not been much theoretical backing of this approach justified in the paper. Furthermore, it is unclear whether this method will also work with other setting in which a non-Gaussian policy is used. Finally, the experiments are not fully convincing (for example in some cases the proposed method does not strongly beat the baseline, and more SOTA baselines are needed to really showcase the potential superiority of the proposed method).\n\n", "strength_and_weaknesses": "Strengths:\nPaper is generally clear in explaining the ideas of  how a perturbed Gaussian policy policy can maintain policy entropy throughout training and help with RL performance.\nThe paper includes pseudo-code and detailed hyper-parameters for readers to re-implement their work. This work should be reproducible. \nThe authors evaluated the RPO method on quite a number of domains and in some cases the proposed method is more powerful than SOTA.\n\nWeaknesses:\nThe justification of this method is mainly empirical, there are not theoretical justifications of this work in terms of why the proposed method would work. \nIf the main contribution is on increasing entropy for better exploration, then there is not enough baseline comparisons with other RL methods that perturb the policy to improve robustness as well as performance during training. For example, one such algorithm is NoisyNet, which also perturbs Gaussian policy network to improve robustness. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2gqBp1XNVL2", "reviewer": "Reviewer_NT5y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a Robust Policy Optimization (RPO) method, a simple extension of the popular Proximal Policy Optimization (PPO) method, by adding a uniform random number to the action mean, sampled from a normal distribution. It is argued that std. normal distribution used to parameterize continuous action is not suitable if more exploration is expected from the agent. Authors modify the gaussian to have more support than the std. gaussian r.v. \n\nEmpirical results on std. RL benchmarks demonstrate that the proposed method has higher exploration throughout the training. ", "summary_of_the_review": "- Authors propose a modification of the existing PPO method for continuous actions cases, by increasing the support of the policy parameterization distribution. \n- They add a uniform random number to the mean of the gaussian distribution used to parameterize the policy, thereby increasing its support. \n- There is no theoretical justification for using the particular distribution; related works on using std. exploration ($\\eps$-greedy etc) is not discussed. \n- Other distributions with wider support around the mean are also not discussed in the paper. ", "strength_and_weaknesses": "Some strengths of the paper: \n\n- The idea is easy to understand, and extremely easy to implement. Basically just adding a random number from uniform distribution would work. \n- The results show that the agent trained with the proposed method explores more than the baseline PPO method. \n\nSome weaknesses of the paper: \n\n- The idea of exploration-exploitation is very old in the RL/bandit community. \n- Authors didn't compare the std. exploration methods in the literature. \n- There is no theoretical justification for choosing this particular distribution. How do other distributions with wider support work in place of gaussian? \n- The effect of changing the distribution to the constraint of the PPO method is also not discussed. How does the constraint function behave when the policy parameterization is changed? \n- Algorithm 1 is not written very well. what is \"logp\", \"prob\"? Please make it more clear.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject