Regulatory Focus: Promotion and Prevention Inclinations in Policy Search
Authors: Lanxin Lei
Zhizhong Li
Xiaoyang Li
Cong Qiu
Dahua Lin
Venue:  (2020)
Keywords: Reinforcement Learning
Regulatory Focus
Promotion and Prevention
Exploration

Abstract: The estimation of advantage is crucial for a number of reinforcement learning algorithms, as it directly influences the choices of future paths. In this work, we propose a family of estimates based on the order statistics over the path ensemble, which allows one to flexibly drive the learning process in a promotion focus or prevention focus. On top of this formulation, we systematically study the impacts of different regulatory focuses. Our findings reveal that regulatory focus, when chosen appropriately, can result in significant benefits. In particular, for the environments with sparse rewards, promotion focus would lead to more efficient exploration of the policy space; while for those where individual actions can have critical impacts, prevention focus is preferable. On various benchmarks, including MuJoCo continuous control, Terrain locomotion, Atari games, and sparse-reward environments, the proposed schemes consistently demonstrate improvement over mainstream methods, not only accelerating the learning process but also obtaining substantial performance gains.

Under review as a conference paper at ICLR 2020
PROMOTION AND PREVENTION INCLINATIONS IN POLICY SEARCH
REGULATORY Focus:
Anonymous authors Paper under double-bling review
ABSTRACT
The estimation of advantage is crucial for a number of reinforcement learning algorithms, as it directly influences the choices of future paths. In this work, we propose a family of estimates based on the order statisties over the path ensemble, which allows one to flexibly drive the learning process in a promotion focus or prevention focus. On top of this formulation, we systematically study the impacts of different regulatory focuses. Our findings reveal that regulatory focus, when chosen appropriately, can result significant benefits. In particular, for the environments with sparse rewards, promotion focus would lead to more efficient exploration of the policy space; while for those where individual actions can have critical impacts, prevention focus is preferable. On various benchmarks, including MuJoCo continuous control, Terrain locomotion, Atari games, and sparse-reward environments, the proposed schemes consistently demonstrate improvement over mainstream methods, not only accelerating the learning process but also obtaining substantial performance gains.
INTRODUCTION
It is not enough to do good; one must do it the right way.
The best-known motivational principle that has been recognized in psychology (Gray, 1982; Atkinson, 1964) is the hedonic principle that people approach pleasure and avoid pain. However the principle does not explain how people approach and avoid. Higgins (1997) proposes the regulatory focus theory to address this question. A promotion focus is concerned with advancement, growth, and accomplishment, and a prevention focus is concerned with security, safety, and responsibility (Crowe & Higgins, 1997). The theory hypothesized that the promotion focus inclination insures against errors of omission, and the prevention focus inclination insures against errors of commission. Furthermore, the regulatory fit theory (Higgins, 2000; Higgins et al., 2003) proposes that when people pursue goal in a manner that fit their regulatory orientation, they are stronger motivated during goal pursuit. For example, Lee & Aaker (2004) showed that ads are more persuasive if there is regulatory fit: use gain frames (e.g. "great looks and exceptional engineering") for individuals who are eager to approach positive outcomes, and use loss frames (e.g. "don't be stranded with disabled vehicle without an emergency safety kit") for individuals who are vigilant to avoid negative outcomes.
We study the phenomenon of regulatory fit in the context of reinforcement learning (RL). Be aware that humans own real emotions and are fundamentally more complex than RL agents, thus we borrow terminologies in psychology while trying retain the resemblance. We hypothesize that different RL environments are more fit with agents of different regulatory focus. For example, in the environments with sparse rewards, it might be advisable to actively exploring those policies that show potentials without being discouraged by a few failed trials; while in the environments that are fragile, e.g. those where wrong choice of action can lead to catastrophic consequences down the path, more conservative, risk-averse stance might be preferable. There are three questions need to answer: (1) How to automatically determine the suitable regulatory focus for an environment; (2) How to train agent with specified regulatory focus; (3) Is there any benefit when training the agent with the regulatory focus that fits the environment? In this article, we aim to answer the second and third questions and leave the first question open.
The research on deep reinforcement learning is gaining momentum in recent years. A number of learning methods, such as Proximal Policy Optimization algorithm (PPO) (Schulman al., 2017),

---

Under review as a conference paper at ICLR 2020
Trust-Region Policy Optimization algorithm (TRPO) (Schulman et al., 2015a), and Advantage Actor- Critic algorithm (A2C) (Mnih et al., 2016), have been developed. These methods and their variants have achieved great success in challenging problems, e.g. continuous control (Schulman et al., 2015b), locomotion (Heess et al., 2017), and video games (Vinyals et al., 2017). The core of all these methods is the estimation of the advantage A(st, at), i.e. the gain in the expected cumulative reward relative to the state value ((se) if a certain action at is taken. A common practice is to use the n-step estimate over the sampled trajectories. This way has been widely used in actor-critic algorithms such as the A2C. Schulman et al. presents a generalization called Generalized Advantage Estimator (GAE) (Schulman al., 2015b), which blends weights, thus reducing the variance of policy gradients. For convenience, we refer to the set of 1 to n-step estimators along a trajectory as the path ensemble.
n-step estimators with exponentially decayed
We propose to take the maximum over path ensemble to implement the promotion focus inclination, and take the minimum over path ensemble to implement to prevention inclination during training. Intuitively, when taking the maximum minimum, the agent is comparing a list of counterfactual speculations. When it looks at a k-step estimation where k S n, it is thinking what if didn't took those actions after step k?? For promotion focus, in order to insure hits and insure against omission, it would favor the estimation with largest value because it has the possibility of being that good. Similarly, for a prevention focus, it prefers the minimum insure against errors of commission.
We systematically studied different estimation schemes on various environments, including the sparse-reward environments (Fu et al., 2017), the Terrain RL Simulator (Berseth et al., 2018) for biped locomotion, the Atari games (Bellemare et al., 2013), and the set of MuJoCo environments for continuous control (Brockman et al., 2016). Our study shows that the proposed estimation strategies, when chosen appropriately, can substantially outperform the standard way. In particular, the promotion focus inclination, i.e. taking the maximum over path ensemble, greatly improves the learning efficiency in the environments with sparse rewards. On the other hand, the prevention focus inclination, i.e. taking the minimum, can effectively stabilize the learning by avoiding risky actions.
It is noteworthy that the proposed strategic inclinations differ essentially from the recent two lines of work on robustness (Smirnova et al., 2019; Delage Mannor, 2007) and risk-sensitivity (Tamar et al., 2012; Chow & Ghavanzadeh, 2014). Robust MDP optimizes for the worst case when there exist uncertainties in the parameters; risk-sensitive MDP optimizes the value of a risk measure. Our method does not modify the training objective. Instead, it controls the policy gradient towards policies with promotion or prevention inclinations through the alternative ways of estimating the advantage.
2 RELATED WORK
Estimation of Advantage and Action-Value The n-step advantage estimation is derived from the n-step return, which is used in the n-step TD or the n-step Sarsa method (Sutton & Barto, 2011). Similarly, the generalized advantage estimator (Schulman et al., 2015b) is analogous to the X-return (Watkins, 1989) in the TD(A) method (Seijen & Sutton, 2014). They are all lincar combinations over the path ensemble. A nonlinear combination scheme is seen in the work of Twin Delayed DDPG (TD3) (Fujimoto et al., 2018), which uses the min of two critics to estimate Q. However, it aims mitigate the overestimation problem in deep Q-learning, and the objects be combined are the network outputs instead of the estimators based on the cumulative rewards. Another nonlinear combination is the Positive Temporal Difference (PTD) (Van Hasselt & Wiering, 2007; van Hasselt, 2012), where the advantage is set as 1 when the 1-step estimate of advantage is positive, and is set otherwise. The benefit of limiting policy updates only toward those actions that have positive advantages is increasing the stability of learning.
Robustness and Risk-Sensitivity in RL In the assumption of robust MDPs (Nilim & El Ghaoui, 2004; Xu & Mannor, 2007; Delage & Mannor, 2010; Mannor et al., 2012; Smirnova et al., 2019), the parameters of the problem lie in an uncertainty set, and the target is to find a solution that performs best under the worst cases. On the other hand, risk-sensitive MDPs consider the uncertainty of the rewards. The objective is to minimize the miskureasure, which is defined by the exponential utility (Howard & Matheson, 1972), the Conditional Value-at-Risk (CVaR) (Chow & Ghavamzadeh, 2014; Prashanth, 2014; Chow et al., 2015; Tamar et al., 2015), the percentile (Delage & Mannor, 2007) the variance (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013) of the cumulative
2

---

Under review as a conference paper at ICLR 2020
rewards. The statistics over the path ensemble is not the same as those statistics for advantage. However, using suitable nonlinear combination of the numbers in the path ensemble, the exploration strategy in algorithms bias the learning toward risk-averse risk-seeking policies. Unlike the robust and risk-sensitive approaches, which typically introduce involved computations, our approach straightforward implement and requires minimal changes incorporate the idea into existing algorithms such as A2C, PPO, TRPO and their variants.
Distributional RL The value distribution the object of study in distributional RL. Since the full distribution of return is difficult to estimate, researchers have adopted nonparametric meth- ods (Morimura ct al., 2010) or used simple distributions (Bellemare et al., 2017; Dabney et al., 2018; Barth-Maron et al., 2018) to approximate it. The RL algorithms can be formulated with any criterion based on the distribution of return, such as the aforementioned risk-sensitivity measures. Again, the distribution formed by elements in the path ensemble does not represent the full distribution of return, but some joint distribution that we can exploit.
3 REGULATORY FOCUS BY ORDER STATISTICS OVER PATH ENSEMBLE
In this section, we first review the preliminary knowledge on RL. Then we introduce the central concept of this article: regulatory focus implemented by estimators over path ensembles. Specifically, we focus on the family of order statistics. Next, we give an illustrative study on the max statistics over the path ensemble, where we show that the induced estimator of action-values would influence the optimization process of RL algorithms. With this specific example in mind, we give practical algorithm to incorporate the general estimators into existing RL algorithms in the last section.
3.1 PRELIMINARY
We consider the standard formulation of RL, which is typically modeled an MDP (8, R). where S is the state space, AA is the action space, T (.) S.S. E AA is the transition probabilities, 7 E (0, 1] the discount factor, and R is the reward function. At timestep t, the agent in state interacts with the environment by choosing an action 0r ~ T($)) following the policy тT, and receives a reward Re from the environment. The environment then transits to the next state $1+1- The discounted return is defined as G, := >T-1 7 -i-tR, and the goal of RL is to maximize the expected return J = Esg~Sp Sp Go sol. where So the distribution of initial states. The action-value function Q" (s:, az) == E GG St, a.] under a policy π is the expected return of taking action 0 (S:) in state st. In policy- parameters 9 is optimized via gradient ascent. An estimation of the gradient is the policy gradient
model-free deep RL algorithms, policy no with
Vana(ur 18s) To(a 5t)
A(si, an)
voJ(O) = E, ET
(I)
where = {50, 00, 81, o1,...,
87, WT} is trajectory following the policy. A(st, a.) Q( ar) ((te) is the advantage function. The n-step look ahead estimation of advantage function is given by
n-1
A(m) = Σ (P/res) + + (((tt)))))))))))))))) VV(t,, n= 1,2, 22....,--.. T
(2)
3.2 ORDER STATISTICS OVER THE PATH ENSEMBLE
Given a trajectory, the set of stsp estimators E = (A(1) -4(0) .... is defined as the path ensemble for the pair (st, a.) We have briefly mentioned that the maximum over path ensemble achieve the promotion focus, and the minimum over path ensemble can achieve the prevention focus in Introduction. The max statistics and the min statistics over the path ensemble respectively computed by
Amax = max [A()).
Arrin : := min {100
For many environments that are neither sparse nor fragile, whether they are more close to promotion focus prevention focus is unclear without in-depth inspection. They also might be too complex to
3

---

Under review as a conference paper at ICLR 2020
be simply described by any one of the two regulatory types. For these cases, we propose generalized order statistics which is a hybrid of the maximum and minimum. We call it the max-abs statistics which took the element with the maximum absolute value,
A Amax-abs = argmax [A].
(4)
A
To understand how elements in the path ensemble relates to regulatory focus, we interpret the k-step estimation as counterfactual speculations when k S n. Since k-step estimation only involves the first k states after the evaluated state, all actions and the followed observations are dropped and be replaced by the vaate-valeeaa at step-k. This underlies a counterfactual thinking that what if the agent didn't do the action sequence d++k+1, 0t+k+2, ... and replace the situation by the average behavior? Then different regulatory focus would treat the set of counterfactual speculations different ways.
The max Statisties and the Promotion Focus A promotion focus is concerned with advancement, growth, and accomplishment. The promotion focus inclination is to insure hits and insure against errors of omission. Facing the set of counterfactual speculations, it would choose the one with the maximum value to insure any action at in 9; that is possibly to behave good to have more importance. The max statisties leads optimistic estimation of advantages. It beneficial for sparse-reward environments as actions that may lead to large returns will not be buried by the later bad actions.
The min Statisties and the Prevention Focus A prevention focus is concerned with security, safety, and responsibility. The prevention focus inclination is to insure correct rejections and insure against errors of commission. Facing the set of counterfactual speculations, it would choose the one with the minimum value to avoid wrongly acknowledge action that only temporarily leads to high rewards. The min statistics implies the risk-averse bias. It avoids actions that may cause bad states later. By decreasing the advantage estimation of those actions, it makes the optimization direction away from them. This property is useful for fragile environments, such as the biped locomotion environments which are sensitive to joint motions of the characters.
The max-abs Statistics and Overreaction In the max-abs statistics, the estimator with the largest absolute value in the path ensemble is chosen. This means that an action evaluated either overly good when the largest positive advantage is selected, or overly bad when the smallest negative advantage selected. We refer to this behavior as overreaction. Overreaction not a terminology in regulatory focus theory, but a descriptive term for the phenomenon concerning the effects of the max-abs statistics. This statistics implements a heuristic that makes the good paths look better and the bad paths look worse. As will be shown in experiments, this heuristic generally improves sample efficiency for the MuJoCo and Atari environments, where their regulatory focuses are unclear.
Both the maximum and the minimum belong to the family of order statistics, where the k-th order statisties equals to the k-th smallest value. From computational perspective, it is also viable to test whether the more general order statistics are effective for training, although lacking strong explanations. We put the study on general order statics in Appendix A.
3.3 AN ILLUSTRATIVE STUDY ON THE max STATISTICS
Before delving into the discussion of estimating advantages, we first look at concrete example of how the max statistics of Q-values affect the learning process of the policy iteration algorithm (Sutton & Barto, 2011, Section 4.3). In Figure 1(a), an MDP with states drawn. For simplicity, we assume that both the state transitions and the rewards are deterministic, and the discount factor is In this example, only the two actions in state 83 can get rewards, where R(83, 41, 45) = 2 and R(s3, 02 $6) 2. At the initialization step of the policy iteration algorithm, the policy is initialized as the uniform random policy πΙ. In the first step of the policy iteration, we evaluate the Q function and value function V of this random policy ni Under policy nT, all possible trajectories are also shown in Figure 1(a). We compute that QT ($1, 01) = 0, and QTI ($1. az) = O. So in the next step, the greedy policy π2 for state $ is still at random. We compute Quma using the max statistics,
Quax a1) max { (1,, 01 = 0,
imax (s1, a2) = E max (1 | $1, 02 =
(5)
4

---

Under review as a conference paper at ICLR 2020
MDP
Trajectories
6 max
MDP
Trajectories
MDP
Trajectrriss
0.5
O
Deterninisti transitions rewards
(0) Uncortainty in transitons
Uncortainty rewards
Toy examples using the max path ensemble. (a) When caleulating return of the second trajectory, the max chooses G(D), that the bad action d1 at state Sy is blacklisted. This leads to an optimistic view when evaluating the long-term returns of actions. The dashed circle means that it ignored in the computation of return. (b) and (c) demonstrate the over-estimation problem of the max statistics when uncertainty exists in transitions or rewards.
Figure 1:
Using QmI the greedy policy at state $1 is argmax, Q a) = a2. Actually, the algorithm with max statistics finds the optimal policy within only 1 step in this example.
We see that the max statistics over the path ensembles converges faster the toy example because it discovers the potentially good action d2 at state $1 earlier than using the standard way. As the computation of the second trajectory in Figure ((a) shows, the max statistics blacklists the bad action aj in state 83. In trajectory, when action is so bad that the later actions cannot compensate for the caused loss, it will be replaced by the average actions. This good-action discovering ability is characteristic of promotion focus, and is suitable for sparse-reward environments. In the early exploration stage in the training, the agent barely receives positive signals. Many trajectories might happens act well at some time, but then go bad states later due wrong actions. The max statistics can highlight actions that are possible to get high rewards in any middle step of the trajectory. By effectively discovering good actions, this method is expected to improve the sampling efficiency.
The foundational position of advantage in actor-critic algorithms is attributed by the policy gradient theorem in Equation (1). A sample (st, 0e) makes the policy network To adjust its parameters according to the estimation of advantage (s at). The probability of (T{)))) = ag is increased when the advantage is positive, and decreased when the advantage is negative. If the estimation of advantage Ás(Se, 0e) is manipulated to be larger, the learned policy is then biased toward the action a at state st- Thus when the estimation of certain advantages A (sy, az) is manipulated properly, the leamedd policy can be biased toward the desired regulatory orientation.
3.4 INCORPORATING REGULATORY FOCUS INTO RL. ALGORITHMS
We should be careful that in the presence of uncertainty noise, applying the max statistics in policy iteration may fail to improve policy. We give two examples when the estimation is overly optimistic. The first example, which is shown in Figure 1(b), is caused by the uncertainty in state transitions. We have Q (1),a1) a1) = 0.5 and Q a2) 1, which implies that action 02 should be chosen in state However, the optimal strategy is to select a1. This is caused by the the max statistics' ignorance on the coupling risks. When taking an action, both good and bad may happen. The bad cases is ignored in the estimation. However, we cannot avoid bad next-state when selecting that action. The second example, shown in Figure ((c), is a case when the reward owns randomness. The problem is that the maximum operation over the ensemble causes overestimation, a symptom that also troubles the Q-learning algorithm as discussed TD3 (Fujimoto et al., 2018).
We overcome this shortcoming from two aspects: (1) implement the regulatory focus exploration strategy. Namely, the estimation from path ensemble is used with a probability p. We call p as the regulatory ratio. (2) using small ensemble size. We argue that small ensemble size is sufficient for practical use because elements in the path ensemble are inter-related. Roughly speaking,
the difference between the i-step advantage estimator A(") and the j-step estimator A! in path ensemble is discounted exponentially with the minimum step index min(i.j) of the two. In fact, let j > i, then
A A =
Σ (r"re4o4)) + + - V ((OCHt)
(6)
5

---

Under review as a conference paper at ICLR 2020
DoonMMylllaHtmee
203
-142
06
100
-200
01
250
375
500
230
522
1020
Updklie
Updana
Updkies
Figure 2: The max and the min statistics on three different types of sparse-reward environments. The max statistics were significantly better than the original EX2 algorithm on all three environments. The min statistics were not effective for sparse reward tasks. All settings were run for seeds.
The sum in the big bracket can be assumed to be bounded. Based this observation, the values of the estimation A(4) would not change too much when & becomes large. In practice using subset of the full ensemble also reduces the computational cost. For example, we used an ensemble with only
four elements & = (A A(16) A((4) A ((2048) the MuJoCo physies simulator experiments.
Summarizing the above discussion, our algorithm is designed as follows.
Algorithm 1 Regulatory Focus by Order Statistics over Path Ensembles
1: Parameters: the ensemble set the order statistics Åunder over &, and the regulatory ratio p.
2: for each iteration do
3: Collect trajectories.
4: Compute the advantage Å and the estimation Åorder
5: Update the policy network according to PPO, A2C, or TRPO, which Åander is used with
probability p. and Á is used otherwise.
6: Update the critic network using the estimation Q value, which equals to V + Å.
7:
end for
Our method can be easily plugged into any actor-critic algorithms with minimal code change and small computational overhead.
4
EXPERIMENT
We evaluate the performance of our algorithm four different types of problems, including the environment with sparse reward, biped locomotion, continuous control and Atari games.
4.1 SPARSE REWARD AND THE PROMOTION FOcUS INCLINATION
The amortized EX2 algorithm (Fu et al., 2017) is designed for tasks with sparse rewards. Three environments were chosen from their paper, which represent three types of problems. The order statistics over path ensemble was implemented by modifying the advantage estimation procedure of the authors' original code, where the TRPO is used for policy optimization. The path ensemble was composed of k-step estimators where & E [1, 16, 64, 4000) for Maze and Doom, and k {1, 16, 64, 50003 for SparseHallCheeetah The numbers 4000 and 5000 originated from the respective batch sizes. The regulatory ratio for both the max and the min statistics was set to p 0.5 for Maze and Doom, and p = 0.3 for SparseHalfCheetah. All pareerrrraeeeerss followed the settings of the amortized EX2. The result is shown in Figure 2.
2D Maze This environment provides sparse reward function, where the agent can only get the reward when within a small radius of the goal. From the figure, we observed that the max statistics started gain rewards at the very early stage, which indicates the promotion focus provides a supreme sample efficiency over other methods. After 500 updates, the average episodic reward of much higher than that of the EX2 algorithm. The min statisties worse than EX2.

---

Under review as a conference paper at ICLR 2020
Biped2D Inciievv
Bipd2DD, Flat-10
1o, 10
PO Fla-vo
1000
6000
6000
3000
2000
1000
1M
2.M
D:M
1
2M
3.M
6M
Trrostape
Timeetogs
Figure 3: Left: screenshot of the flat and incline environment for biped walking. Right: the performance of PPO and PPO with min and max statistics. We ran 5 seeds for each setting.
20
3000
2008
1300
410
2.0M0
1:M
1.0M
2M
2.M
1:
1.0M
0.0
1M
100
0.0
1M
18:
18:0
1M
18M
Figure 4: Experiments on MuloCo environments. Three order statistics max, min and max-abs were compared with the vanilla PPO algorithm. Each setting was run with seeds.
Doom MymayHome is a vision-based maze navigation benchmark, where the agent required to navigate through a series of interconnected rooms before reaching the goal. A +1 reward is given for reaching the goal before timeout. The task is challenging because the inputs are realistic images in the first-person perspective. The max statistics was very effective for this challenge and reached nearly 100% success rate. In comparison, EX2 only got average of 0.6 episode rewards.
SparseHalfCheetah It is a challenging continuous control task with sparse reward. Our promotion focus significantly improved the sample efficiency, and the reward after 1000 updates was 40% higher than the EX2 algorithm. Also, the min statistics was less effective than EX2 in all three benchmarks, which demonstrates that prevention focus is not effective sparse environments.
4.2 BIPED LOCOMOTION AND THE PREVENTION FOCUS INCLINATION
The prevention focus is very useful for environments that are fragile to actions, i.e., one wrong action would lead to catastrophic results later on. The Terrain RL simulator (Berseth et al., 2018) provides such environments. Two biped walking tasks were selected, where a character needs to learn how to walk steadily forward in two different terrains, the flat and the incline, as shown in Figure 3. The character continuously receives rewards until it fells down or stuck, in which case the episode terminates. This task is challenging because the observation only contains the pose of the character, which forces the character to learn how to walk without memorizing actions based on its location. In contrast, the locomotion tasks in MuJoCo environments contain absolute world coordinates in the observation. The action space is 11 11-dimensional, which corresponds to the joints of the character. The environment is fragile. If wrong action is performed, the character might lose balance and fall down. We used PPO in this experiment. The hyper- parameters were borrowed from those designed for MuJoCo environments also listed in Appendix. The min and max statistics were implemented on top of the PPO algorithm. The path ensemble consisted of k-step estimators where & E {1, 16, 64, 2048), and the regulatory ratio is P 0.3. The results are shown in Figure 3. The prevention focus inclination via the min statistics successfully mastered the task while the vanilla PPO algorithm failed. The promotion focus inclination by the max not very effective in these environments as the low reward curves indicate that the character fells down at the beginning of the episode. We conclude that prevention focus by the min statistics of the path ensemble helps the agent to learn in fragile environments.
the baselines' (Dhariwal et al., 2017) PPO implementation and are
7

---

Under review as a conference paper at ICLR 2020
1300
3000
1450
1800
25M5
SM
M
1030
23:0
5MM
7
1000
24:01
5.M
75M
100
0
2:50
6.5
7.5M
25.
5M
75M
10
Figure 5: Experiments on the Atari environments. A2C' was the original A2C algorithm without mini-batch, and A2C was using the same mini-batch configuration as other three settings: max, min and max-abs. Each setting was run for seeds. (This figure best viewed in color.)
4.3 CONTINUOUS CONTROL AND THE OVERREACTION INCLINATION
We tested various advantage estimations on 5 continuous control benchmarks based on the MuJoCo physics simulator. They are not sparse-reware environments, and most of them are also not sensitive to individual actions. For example, the design of the HalfCheetah agent makes it seldom fall down or be stuck, We used PPO in this set of experiments. The implementation was based on the baselines (Dhariwal et al., 2017) code repository, and the default hyper- parameers were adopted. We tested three order statistics including the max, the min and the max-abs statistics. In all settings, the path ensemble had index set {1, 16, 64, 2048), and the regulatory ratio was set to p 0.4. Results are shown in Figure 4. We observed that the max-abs statistics was consistently better than the baseline PPO algorithm, while the performances of the max and the min statisties depended upon the specific enviroument. So we conclude that the overreaction inclination by the max-abs statistics is generally effective for wide range of environments.
4.4 ATARI GAMES AND THE OVERREACTION INCLINATION
The new advantage estimations were also tested on subset of Atari game environments. We used the A2C algorithm with 4 paralleled threads. The implementation was based on the A2C codes in baselines (Dhariwal et al., 2017). Hyper-parameters were the defaults for Atari games. Since the sampled trajectory length was 5 in the default setting, the path ensemble had fewer elements, which was composed of (1,3, 3, 5). The maximum trajectory length of 5 in a batch was too small to gather enough data on k-step estimations when k >> 1. For example, only the first state has valid 5-step advantage estimation in a length-5 trajectory. This affects the power of the path ensemble. To circumvent this limitation, estimators using these longer trajectories. Since the batch size was n times of the original setting, it was split into mini-batches. For Seaquest and Breakout, n 20; for Pong and Qbert, n 10; and for BeamRider, n 60. The result is shown in Figure 5. In these environments, the sparsity and fragility were unknown. Obviously, the overreaction inclination generally improves the performance. whereas the max and the min statistics are only effective for a subset of environments.
collected paths of length 5n, and then computed the advantage
5 CONCLUSION
According to the regulatory fit theory in psychology. when people engage in goal pursuit activities in a manner that fits their regulatory orientation, they feel right about what they are doing. In this paper, we proposed simple yet effective way of exploration using the idea of regulatory fit. The regulatory focus is implemented in the context of reinforcement learning via the order statistics over path ensembles. which formed nonlinear combinations of different n n-step advantage estimators. The maximum, the minimum, and the element with the maximum absolute value were studied in detail. We incorporated these advantage estimators into three widely used actor-critic algorithms including A2C, TRPO and PPO. When the promotion prevention focus inclination fits the environment, the proposed algorithm could be effective in solving sparse-reward environments, or the fragile environments which are sensitive to individual actions. We verified the effectiveness of our approach by extensive experiments on various domains, including the continuous control, locomotion, video games, and sparse-reward environments.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
John William Atkinson. An introduction to motivation. 1964.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprini arXiv: 1804008177, 2018.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In Proceedings of the 34th International Conference on Machine Cearningg-ollumee 70, pp. 449-458. JMLR. org. 2017.
Glen Berseth, Xue Bin Peng, and Michiel van de Panne. Terrain simulator. CoRR, 0hbs/1804.064222 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization mdps. In Advances in neural information processing systems, pp. 3509-3517, 2014.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. In Advances in Neural Information Processing Systems, pp. 1522-1530, 2015.
Ellen Crowe and E Tory Higgins. Regulatory focus and strategic inclinations: Promotion and prevention in decision-mann making. Organizational behavior and human decision processes, 69(2):
117-132, 1997.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, pp- 1104-111,, 2018
Erick Delage and Shie Mannor. Percentile optimization in uncertain markov decision processes with application to efficient exploration. In Proceedings of the 24th international conference on Machine learning, pp. 232. ACM, 2007.
Erick Delage and Shie Mannor. Percentile optimization for markov decision processes with parameter uncertainty. Operations research, 58(1):203-213, 2010.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https:
//github. wwt.c/om//ooaaacccooo petpeenniiaaeeeeeese... leee,,nneneeee.m 2017.
Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan. and R. Garnett Advances in Neural Information Processing Systems 30. pp. 2577-2587. Curran Associates, Inc., 2017. URL ht hwtppaaappaeeeaaaaeeeeeeeeaeeeep n nips www.ippsccpppppp...ppepp 6851-ex2- pdf.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.88027777771 2018.
Jeffrey Gray. Précis of the neuropsychology of anxiety: An enquiry into the functions of the septo-hippocampal system. Behavioral and Brain Sciences, 5(3):469 484, 1982.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv 022.08 2017.

---

Under review as a conference paper at ICLR 2020
E Tory Higgins. Beyond pleasure and pain. American psychologist, 52(12)) 1997.
Tory Higgins. Making good decision: value from fit. American psychologist, 55(11): 2000.
E Tory Higgins, Lorraine Chen Idson, Antonio L Freitas, Scott Spiegel, and Daniel C Molden. Transfer of value from fit. Journal of personality and social psychology, 84(6):1140, 2003.
Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 18(7):356 369, 1972.
Angela Y Lee and Jennifer Aaker. Bringing the frame into focus: the influence of regulatory fit on processing fluency and persuasion. Journal of personality and social psychology, 86(2):205, 2004.
Shie Mannor, Ofir Mebel, and Huan Xu. Lightning does not strike twice: robust mdps with coupled uncertainty. In Proceedings of the 29th International Coference on International Conference Machine Learning, pp. 451-458. Omnipress, 2012.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In Proceedings of the 27th International Conference on Machine Learning (ICML-1O), pp. 799-806, 2010.
Arnab Nilim and Laurent El Ghaoui. Robustness in markov decision problems with uncertain transition matrices. In Advances in Neural Information Processing Systems, pp. 839-846, 2004.
LA Prashanth. Policy gradients for cvar-constrained mdps. In International Conference on Algorith- mic Learning Theory, pp. 155-169. Springer, 2014.
LA Prashanth and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. In Advances in neural information processing systems, pp. 252-260. 2013.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arX.v:1506.024336 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Harm Seijen and Rich Sutton. True online td (lambda). International Conference on Machine Learning, pp. 692-700, 2014.
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprini arXiv: 19Xi::9022.07088 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2011.
Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1651-1658. Omnipress, 2012.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling. In Twenty-Ninth
AAAI Conference on Artificial Intelligence, 2015.
Hado van Hasselt. Reinforcement Learning in Continuous State and Action Spaces, pp. 207-251. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-27645--3 doi: 10.1007/ URL ht //o.i httg//1..100////10 100.11100097783330
10

---

Under review as a conference paper at ICLR 2020
6000
3250
420
5000
2700
1900
700
1200
2200
1400
kd
2/1
Mn
1.10
3/5
Mar
57
2nd
3
M4c
2.14
M4n
3.10
2/1
Figure 6: The order statisties the path ensemble in MuJoCo environments. r-axis labels the different order statistics. The four series in the sub-figures were varying the regulatory ratio and the flat line indicated performance of the baseline PPO algorithm. Each data point was averaged over 5 runs.
Hado Van Hasselt and Marco A Wiering. Reinforcement learning in continuous action spaces. In 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, pp. 272-279. IEEE, 2007.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv: 170..04788.7 2017.
Christopher John Coriihh Hellaby Watkins. Learning from delayed rewards. 1989.
Huan Xu and Shie Mannor. The aobustncss-performaaee tradcoff in markov decision processes. In Advances in Neural Information Processing Systems, pp. 1537-1544, 2007.
A ABLATION STUDY
In this section we study the behavior of general order statistics on MuloCo environments, and then discuss the sensitivity of the algorithms to hyper-paan parameters.
The General Order Statistics As shown in Figure 6, in three (InverteDDuubeeeendllun) Ant, and Humanoid) of the five environments, the max and min were better than the 2nd and the 3rd order statistics. This means that some environments welcome both optimistic and risk-averse exploration. Overall, the intermediate order statistics was better than the baseline, which used the GAE estimator.
Generally speaking, the regulatory ratio p had a great influ- ence on the performance. As shown in Figure 6, the range [0.2, 0.4] is plausible choice for most experiments. If the ratio further increased, the performance may degrade. This is consistent with our analysis, and shows the necessity of using regulatory focus an exploration strategy.
Sensitivity the Regulatory Ratio
Effect of Ensemble Size We built another path ensemble whose index consisted of 12 elements. They were 21, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048). Final performances of the three order statisties, max, min and max-abs, were tested under the regulatory ratio p 0.4. For the ensem- ble with 4 elements (1, 16, 64, 2048), the average episode reward in the end of the training was 928, 927, 974, respectively; and for the ensemble of size 12, the numbers were 767, 854 and 867. shows that small ensemble size can already get most of the performance boost.
B COMPARE WITH C- G-GREEDY EXPLORATION
As exploration method, it is helpful to compare the regulatory focus exploration with the basic e-greedy exploration. The results are shown on Mujoco environments in Figure 7. We can see that completely random exploration, in small proportion, makes the results worse.
11

---

Under review as a conference paper at ICLR 2020
Figure 7: The e-greedy exploration. Red: MaxAbs, Purple: PPO, Rest: c-greedy explorations, where values of 0.05, 0.1, 0.2, 0.3, 0.4. Each experiment has runs.
FD, d2po inclee-v
PD Biped20 Flonve
2006
City pysets
4000
2000
2000
2:4 Tinastip
3:1
Treedrae
Figure 8: Conservative policy via clipping. Red: the Min path-ensemble with the default clipping parameter 0.2, Purple: PPO with the default clipping parameter 0.2, Rest: PPO with smaller clipping constants 0.1 and 0.05. Each experiment has 5 runs.
C COMPARE WITH DIFFERENT CLIP THRESHOLD OF PPO
The clipping threshold of PPO also affects the estimation of advantage. A smaller clipping threshold would result in a more conservative policy iteration. We compare the effect of smaller clipping threshold with the min statistics in Figure 8. Clipping is adding a global minimum constraint, meaning that all trajectories share the same clipping constant. In contrast, the min of path ensemble gives prevention focus inclination in momentary manner, i.e. per trajectory. The performance of min path ensemble is much better.
path ensemble the biped locomotion task. Results shown
D PATH ENSEMBLE OF GAES
The generalized advantage estimator GAB(?, A) is an exponentially-weigteed average of different n-step estimators,
GAE(T, =- (1 (1 (A + = -
x2
With different dyper-parameterr X, the estimator focus different time ranges. For example, if is small, then the decay would be very fast and effectively only the first steps in the trajectory is then GAE takes longer time ranges into account. Having the above
considered; if À is close observation, we can use the path ensemble of GAEs with different As as an alternate version of the path ensemble of n-step advantages. The GAE formulation would have a smaller variance and may be beneficial for some environments. We tested the formulation of ensembling GAEs on the Mujoco environments and found substantial improvements on the Ant-v2 and Humanoid-v2 environments. Note also that HalfCheetah-v2 and Walker2d-v2, performance drops compared to ensembling n-step advantages. The results are drawn in Figure
E DETAILED EXPERIMENT SETTINGS
The detailed hyper-parament settings of the experiments are listed in Table
12

---

Under review as a conference paper at ICLR 2020
Figure Ensemble As. Green: MaxAbs, Blue: PPO, Orange: Max Às 0.3,, 0.3, 0.0, 0.95).
Here the regulatory ratio p = 0.4. Each experiment has 5 runs.
Table 1: Detailed syper-paramettr settings of the experiments.
Environment
Env Name
Hyper-Parametee Value
Alg.
Type
elements of path ensemble=
11. 16, 64, 2048)
InveredDDubbdPonduuum-22
num timesteps=2266
HalfCheetah-v2
seeds=(0,,12,3,4)
Walker2d-v2
num num_cnvs==
Mujoco
Ant-v2
gamma=0.99
PPO
Humanoid-v2
lam=0.95
optimizer-Adam
Biped2D-Incline
Ir=lambda f: 3e-4 "f
TerrainRL
Biped2D-Flat
cliprange=022
value_netwoote fllue_network='coppy
rho=0.4
elements of path ensemble=
2D Maze
{1, 16, 64, 4000)
Doom
Sparse Mujoco
rho=0.5
EX2
elements path ensemble=
Spahsehal
{1, 16, 64, 5000)
rho=0.3
minibatch=10
PongNoFrameskip-v4
lengthOfTrajectory=550
rho=0.1
QbertNoFramnskippv4
elements path (1,
numberOtfarralllEnveal
minibatch=20
ScaquestNoFrameskipvvv
lengthOffrraecctoyy:000
Atari
rho=0.1
A2C
BreakouNNFrramssiip-4
elements path mnsemble=(1,,,,
numberOfParalllEEnva/
minibatch=60
lengthOffrrjeccorr==300
BeamRiderNoFrameskipok
rho=0.1
elements of path 10semblc=(113,55
mmberrOrParalllEEnva.
13


Reviews:
[{"review_id": "r1eAZ3hCFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a modification to policy gradient methods that are computed from advantage function estimates. For a given trajectory of n steps, there are n different advantage function estimates: from 1-step to n-step. GAE (Schulman, 2016) proposes to take an exponentially weighted average of these estimates to compute the policy gradients. This paper proposes instead to use order statistics to compute the policy gradient; e.g. the most optimistic estimate, the most pessimistic estimate, or the most extreme estimate.  The paper introduces a regulatory ratio: the probability of using the averaged advantage estimate vs using the order statistic, for computing the policy gradients. This hyper-parameter is justified on the optimistic case (max advantage), as a way to prevent overtly optimistic estimates. The paper conducts experiments on  different domains (sparse and dense rewards, discrete and continuous actions, fully observable and partially observable environments) which show the effect of choosing different order statistics and regulatory ratio on the policy performance.\n\nThis paper could be accepted as it presents an interesting idea with extensive experiments showing where it works and where it fails, along with some justification for the hyperparameter choices.  But there are a couple concerns about the validity of the method.\n\nOne concern is that the regulatory ratio is only justified for the max case, but not for the min or max-abs case. In addition, the choice of regulatory ratio seems to have a wildly varying impact depending on the choice of the order statistic and environment, for which we get little insights from the paper.\n\nAnother concern is that most of the results are reported only for an ensemble of 4 n-step estimators. In the appendix, the papers reports a comparison with a larger ensemble (12 n-steps estimators), which results in lower performance. This is a bit confusing: using the order statistics (max, min and max-abs), and following the reasoning presented in the paper, I would expect that increasing the ensemble size would result in better performance ( the max/min of the 4 ensemble is n upper/lower bound of the max/min of the 12 ensemble). This paper could be improved with more detailed results on the effect of the ensemble size. While the paper provides arguments for why small ensembles might suffice, it does not explain how the ensemble should be chosen, and what would happen as the ensemble size increases.\n\nFinally, the parallels to human psychology are  bit superfluous. I understand how it might serve as an inspiration for algorithm design, but it is a bit distracting from the technical contribution of the paper.\n\nThings to improve:\n\nWhy is max-abs missing from Figure 2?\nDid you try a Rainbow-style training (changing the regulatory focus periodically over training)?\n\nIn some sentences, it looks as if insure was written in place of ensure. In general, this paper would benefit from proof reading by a proficient English speaker.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxnCx6Ytr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the problem of advantage estimation for actor-critic RL algorithms. The key observation is that the advantage can be computed using 1-step returns, 2-step returns, etc. The paper suggests that, instead of choosing a fixed n, we should aggregate these advantageous together. If the maximum is taken, the resulting policy will be exploratory (i.e., have a \"promotion focus\"); if the minimum advantage is taken, the resulting policy will be risk sensitive (i.e., have a \"regulatory focus\").\n\nThe paper presents results on a number of tasks. A few toy examples show instances where the proposed method works better. Experiments on sparse reward environments show that taking the max advantage provides for exploration that outperforms a baseline (EX2). On a walking talk show that the min-approach can outperform state-of-the-art on-policy RL (PPO); the paper suggests this is because the min-approach is implicitly risk sensitive. Experiments on some of the Mujoco control tasks and some of the Atari games show that taking the advantage with the maximum absolute value performs well, as compared to PPO.\n\nOverall, I think the main contribution of this paper is the finding that there exist smarter ways of computing the advantage. A second contribution is connecting the ideas of risk-sensitive and risk-seeking control with ideas from psychology (regulatory fit theory). \n\nI am leaning towards rejecting this paper. On the one hand, the paper is well written, well motivated, and the experiments quite thorough. On the other hand, I don't think the paper is particularly useful, either from a theory or algorithmic perspective. It is not surprising that if we add an additional hyperparater to an existing RL algorithm and tune that hyperparameter, we'll do better than the existing RL algorithm. The ablation experiments in Fig 6 and Fig 7 suggest that how the advantage estimates are combined, and how they are \"mixed\" with a standard advantage estimate, matter a lot. Further, the fact that using a smaller number of advantage estimates worked better (point #2 on pg 5, Effect of Ensemble Size in Appendix A) suggests that the ensemble size is an important hyperparameter, and that risk-seeking / risk-aversion (i.e., regulatory vs promotion focus) cannot alone explain why the proposed method works. I think that, for this idea to be useful, it must be equipped with a fixed value for these hyperparameters that works well across a wide range of tasks (e.g., a learning rate of 3e-4 works well for Adam on most tasks), or an automated strategy for choosing this hyperparameter (e.g., the automatic entropy tuning in SAC). I would consider increasing my review if either of these were included.\n\nA second concern is that, despite the close connections with risk-sensitive and risk-seeking control (discussed in Section 2), none of these prior works are compared against. Many of these prior works include a temperature parameter for trading off risk-seeking vs risk-aversion (e.g., \\beta in Eq 11 of [Mihatsch 2002]), which is arguably a more transparent (to the user) and easier to analyze (for the RL researcher) than the order statistics used in this paper. Given the wealth of prior work on risk-sensitive and risk-seeking control, I think it's important to know whether these prior methods already solve the problem at hand (choosing between risk-seeking and risk-aversion). I would consider increasing my review if one of these methods were included as a baseline.\n\nMinor comments\n* \"that humans own\" -> \"that humans' own\"\n* The max strategy seems quite closely related to UCB-based exploration. I'd be curious to learn about some discussion on the similarities/differences\n* Fig 6 -- Can you add error bars to show the variance across random seeds?\n* Appendix D -- How does this approach and the results shown in Fig 9 differ from standard GAE? \n\n----------------------- UPDATE AFTER (NO) AUTHOR RESPONSE ----------------------------\nThe authors did not post a response, so I will maintain my vote to \"weak reject\" this paper. I would encourage the authors to incorporate the feedback in all reviews and submit the paper to a future conference.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xOY14FtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on risk-aware reinforcement learning, where an agent could be encouraged to take more risk (high reward, high variance states) or avoid risk (low variance states). Risk control is instantiated by different ways of estimating the advantage of a state (max/min instead of average). Experiments on several environments show good performance of the proposed algorithm.\n\nThe paper is written clearly and the approach is straightforward. However, it's unclear what is the objective function the algorithm is optimizing by using a biased estimation of the advantage. It seems to work pretty well in practice, but I wonder how it compares to other risk-sensitive RL algorithms (e.g. those cited in the related work section).\n\nOverall, this paper presents a simple heuristic to steer the policy towards risk-seeking / risk-avoiding directions, but could benefit from either more theoretical analysis or more empirical comparison with other methods.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors take inspiration from regulatory fit theory and propose a new parameter for policy gradient algorithms in RL that can manage the "regulatory focus" of an agent.  They hypothesize that this can affect performance in a problem-specific way, especially when trading off between broad exploration and risk.  The reviewers expressed concerns about the usefulness of the proposed algorithm in practice and a lack of thorough empirical comparisons or theoretical results.  Unfortunately, the authors did not provide a rebuttal, so no further discussion of these issues was possible; thus, I recommend to reject.
Decision: Reject