Learning with Social Influence through  Interior Policy Differentiation
Authors: Hao Sun
Bo Dai
Jiankai Sun
Zhenghao Peng
Guodong Xu
Dahua Lin
Bolei Zhou
Venue:  (2020)
Keywords: Reinforcement Learning
Social Uniqueness
Policy Differentiation

Abstract: Animals develop novel skills not only through the interaction with the environment but also from the influence of the others. In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers. Specifically, we first define a metric to measure the distance between policies then quantitatively derive the definition of uniqueness. Unlike previous precarious joint optimization approaches, the social uniqueness motivation in our work is imposed as a constraint to encourage the agent to learn a policy different from the existing agents while still solve the primal task. The resulting algorithm, namely Interior Policy Differentiation (IPD), brings about performance improvement as well as a collection of policies that solve a given task with distinct behaviors

Under review as a conference paper at ICLR 2020
LEARNING WITH SOCIAL INFLUENCE THROUGH INTERIOR POLICY DIFFERENTIATION
Anonymous authors Paper under double-blind review
ABSTRACT
Animals develop novel skills not only through the interaction with the environ- ment but also from the influence of the others. In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peen!]] Specifically, we first define a metric to measure the distance between policies then quantitatively derive the def- inition of uniqueness. Unlike previous precarious joint optimization approaches, the social uniqueness motivation in our work is imposed as constraint to encour- age the agent to learn policy different from the existing agents while still solve the primal task. The resulting algorithm, namely Interior Policy Differentiation (IPD), is able to learn a collection of policies that can solve a given task with distinct behaviors and brings about performance improvement byproduct in some cases? ?
INTRODUCTION
The paradigm of Reinforcement Learning (RL). inspired by cogmition and animal stud- ies (Thorndike) 2017, Schultzz et al. 1997). can be described as learning by interacting with the environment to maximize cumulative reward (Sutton al., 1998). From the perspective of ecol- ogy. biodiversity as well as the development of various skills are crucial to the continuation and evolution of species (Darwin 1859; Pianka 1970). Thus the behavioral diversity becomes rising topic in RL. Previous works have tried to encourage the emergence of behavioral diversity in RL with two approaches: The first approach is to design interactive environments which contain suffi- cient richness and diversity. For example, Heess et al. (017)) show that rich environments enable agents to learn different locomotion skills even using the standard RL algorithms. Yet designing a complex environment requires manual efforts, and the diversity is limited by the obstacle classes. The second approach to increase behavioral diversity is to motivate agents to explore beyond just maximizing the reward for the given task. Zhhang al. (2019) proposed to maximize a heuristi- cally defined novelty metric between policies through task-novelty joint optimization, but the final performance of agents is not guaranteed.
In this work, we address the topic of policy differentiation in RL. i.e., to improve the diversity of RL agents while keeping their ability to solve the primal task. We draw the inspiration from the Social Influence in animal society (Rogoff) 1990, Ryan & Deci 2000 van Schaik & Burkart 2011 Henrich 2017 Harari, 2014) and formulate the concept of social influence in the reinforcement learning paradigm. Our learning scheme is illustrated in Fig [] The target agent not only learns to interact with the environment to maximize the reward but also differentiate the actions it takes in
order to be different from other existing agents.
Since the social influence often acts on people passively as sort of peer pressure, we implement the social influence in terms of social uniqueness motivation (Chan et al.) 2012) and consider it as a constrained optimization problem. In the following of our work, we first define a rigorous policy distance metric in the policy space to compare the similarity of the agents. Then develop op- timization constraint using the proposed metric, which brings immediate rather than episodic feed- back in the learning process. A novel method, namely Interior Policy Differentiation (IPD), is further
IIn this work, we use the term "peer" to denote a population of RL. agents.
"Code will be made available soon

---

Under review as a conference paper at ICLR 2020
Environment
should fast
Primal Task
Agent
Apon
Agen
Agent 4
Peers
I should try to be different
Social Uniqueness
The illustration of learning with social influence. Instead of focusing only the primal task, an additional constraint is introduced to the target agent, motivating to not only perform well in the primal task but also take actions differently to other existing agents.
Figure 1:
proposed better solution for the constrained policy optimization problem. We benchmark our method on several locomotion tasks and show it can learn various diverse and well-behaved policies for the given tasks based on the standard Proximal Policy Optimization (PPO) algorithm (Schulman et al. 2017).
2 RELATED WORK
Intrinsic motivation methods. The Variational Information Maximizing Exploration (VIME) method is designed by HHouhhooft et al. (2016) to tackle the sparse reward problems. In VIME, an intrinsic reward term based on the maximization information gains added to contemporary RL algorithms encourage exploration. The curiosity-driven methods, proposed by Pathak et al. (017) and Burda et al. (2018a) define intrinsic rewards according to prediction errors of neural networks. i.e., when taking previous unseen states inputs, networks trained with previous states will tend to predict with low accuracy, so that such prediction errors can be viewed as rewards. Burda al. (2018b) proposed Random Network Distillation (RND) quantify intrinsic reward by prediction differences between a fixed random initialized network and another randomly initialized network trained with previous state information. Liu al. (2019) proposed Competitive Experience Replay (CER), in which they use two actors and a centralized critic, and defined an intrinsic reward by the state coincidence of two actors. The values of intrinsic rewards are fixed to be +1 for the two actors separately. All of those approaches leverage the weighted sum of the external rewards, i... the primal rewards provided by environments, and intrinsic rewards that provided by different heuristics. A challenging problem is the trade-off' between external rewards and intrinsic rewards. The Task-Novelty Bisector (TNB) learning method introduced by Zhang et al. (2019) aims to solve such problem by jointly optimize the extrinsic rewards and intrinsic rewards. Specifically, TNB updates the policy in the direction of the angular bisector the two gradients, i.c., gradicnts of the extrinsic and intrinsic objective functions. However, the foundation of such joint optimization not solid, Besides, creating an extra intrinsic reward function and evaluating the novelty of states or policies always requires additional neural networks such auto-encoders. Thus extra computation expenses are needed (Zhang et al. 2019).
Diverse behaviors from rich environments and algorithms. Heess et al. (2017) introduce the Distributed Proximal Policy Optimization (DPPO) method and enable agents with simulated bodies learn complex locomotion skills a diverse set of challenging environments. Although the learning reward they utilize is straightforward, the skills their policy learned are quite impressive and effective traveling terrains and obstacles. Their work shows that rich environments can encourage the emergence of different locomotion behaviors, but extra manual efforts are required in designing such environments. The research Such et al. (2018) shows that different RL algorithms may converge to different policies for the same task. The authors find that algorithms based policy gradient tend to converge to the same local optimum in the game Pitfall, while off off-policy and balue-based algorithms prone to learn sophisticated strategies. On the contrary, in this paper, are more interested how to learn different policies through a single learning algorithm and learn the capability of avoiding local optimum.
2

---

Under review as a conference paper at ICLR 2020
Population-based novelty-seeking methods. Pugh et al. (2016) establish standard framework for understanding and comparing different approaches to searching for quality diversity (QD). Conti
et al. (2018) investigate adding novelty search (NS) and QD to evolution strategies (ES) to avoid
local optima as well as achieve higher performance. Lehman & Stanley (2011) 2008) conclude
open-ended search algorithm that operates without pressure towards the ultimate objective is possible, suggesting ignoring the objective may often benefit the search itself. The work of Wang et al. (2019) yields new kind of open-ended algorithm which indicates the solution to one environment might be stepping stone to new level of performance in another. [Such et al. (2017) evolve DNN with a population-base based genetic algorithm (GA) for challenging RL tasks. By improving the vanilla TRPO algorithm (Schulman et al. 2015). Kurutach et al. (2018) maintains model uncertainty given the data collected from the environment via an ensemble of deep neural networks.
that deriving
3 QUANTIFYING THE DISTANCE BETWEEN POLICIES
To encourage the emergence of behavioral diversity in RL, we first define metric to measure the difference between policies, which is the foundation for the later algorithm we propose. We denote the learned policies as TTo 02 E O, 1, 1..). wherein 0; represents parameters of the i-th policy, θ denotes the whole parameter space. In the following. omit ı and denote policy no, as 0g for simplicity unless stated otherwise.
3.1 DEFINITION
Mathematically, metric should satisfy three important properties, namely the identity, the symme- try as well as the triangle inequality.
Definition 1 A metric space is ordered pair (M, d) where M is set and d is metric M, i.e. a function d: M x M + R such that for any x, y, 2 E M, the following holds: 1. d(r, y) 0, d(xr,y)
0 + y.
d(r, v) d(y, x). d(x, 2) S d(x, y) d(y,2).
We use the Total Variance Divergence Drv (Schulman et al. 2015) to measure the distance be- tween policies. Concretely, for discrete probability distributions p and g. this distance is defined as Dry (p, 9) = E. PP: 4:l
Theorem (Metric Space (9, DTv)) The expectation of Drv(.)) of two policies over any state distribution p(s):
Drv 0,) == (
(1)
a metric on θ thus (e, Dfv) is a metric space.
The proof of Theorem 1 is in Appendix A It is worth mentioning that, although TVD is used in our work, we can easily extend the result to use other distance between distributions as substitutes of TVD (e.g. Jensen Shannon divergence D,s or Wasserstein metric Dw) (Endres & Schindelin, 2003 Fuglede & Topsoe, 2004
Villani 2008), and similar results can be get
Corollary Let Dug == (9, ('s) and (O, Dwy also metric spaces.
and D = Es (W)())))))((
On top of the metric space (0, (T,)), we could then compute the uniqueness of a policy.
Definition 2 (Uniqueness of Policy) Given reference policy set Ong such that Ore = (0re5
1,), Ore C θ the uniqueness U(8 Orer) of policy is the minimal difference between and all policy in the reference policy set, i.e., U(OlOng) min 6,
'It can he extended to continuous state and action spaces by replacing the sums with integrals. "The factor 1/ in Schulman et al. (2015) omitted in our work for conciseness.
3

---

Under review as a conference paper at ICLR 2020
Consequently, to motivate RL with the social uniqueness, hope method maximize the uniqueness of a new policy, i.e. maxę U(o (rer)), where the Gre includes all the existing policies.
3.2 ESTIMATION OF Dry (0,,
In practice, the calculation of Dry (0,, (0,,0,) is based on Monte Carlo estimation. i.c., we need to sample from p(s). Although in finite state space ergodicity, problem arises when we are facing continuous state cases. i.e. is difficult to efficiently get enough samples.
gel precise estimation after establishing
Formally, we denote the domain of p(s) as S and denote the domain of (a(s) as Sg C S, where
po(s) p(s|s 0) and in finite time horizon problems p(s|s 0) P(so s P(s)
s|0) +... + P(8T s s). As we only care about the reachable regions, the domain S can be divided by S = limn o U Su..
In order improve the sample efficiency, propose to approximate (Tv 02) with (O,, 0z); where a certain fixed behavior policy that irrelevant 0,, 0j. Such approximation requires necessary condition:
Condition 1 The domain of possible states are similar between different policies:
P(ss (Sa S (Sa Sa,))
Σ
P(s E (Sa (e, Vj.
(3)
SES
sES
When such condition holds, we can use p(s|s ~ 0) as our choice of p(s). and the properties in Definition 1 still holds.
In practice the Condition I always holds as we can ensure this by adding sufficiently large noise on 8, while the permitted state space is always limited. And for more general cases, to satisfy the properties in Definition 1, we must sample s from Sg U So accordingly.
(0, Es-(Sous, (PrV (s)))
-E S., (Drv (0(s) (s)) + (5- 1Drv (r((s)))))
(4)
[S )x. N. (Drv (,)),(s))
where N represents random action when policy have never been trained visited such state domain. Plugging Eq.4 into Eq. 2). the objective function of policy differentiation is
( (0, 02) max min
(5)
E (s.us, Ms, DDv (f(s) + (S, Ns, (I,
While the first two terms are related to the policy 0, the last term is only related to the domain Sa- If enable sufficient exploration in training well in the initialization of 0, the last term will disappear (i.e. Sø, Sp). Hence in training of 0g as long as sufficient exploration is guaranteed.
also use DTa (0,, 0,) as approximation of (r 0y)
Proposition 1 (Unbiased Single Trajectory Estimation) The estimation of po(s) using single trajectory T unbiased.
The proof of Proposition I is in Appendix B Given the definition of uniqueness and practically unbiased sampling method, the next step is to develop an efficient learning algorithm.
INTERIOR POLICY DIFFERENTIATION
In the traditional RL paradigm, maximizing the expectation of cumulative rewards Ei=o ''r, is commonly used as the objective. i.c. maxaee E,<olg), where т denotes trajectory T sampled from the policy 0 using Monte Carlo methods.
4

---

Under review as a conference paper at ICLR 2020
To improve the behavioral diversity of different agents, the learning objective must take both reward from the primal task and the policy uniqueness into consideration. Previous approaches (Houthooft ctt al., 2016; Pathak et al. 2017; Burda et al., 20188 b Liu et al,, 2019) often directly write the weighted sum of the reward from the primal task and the intrinsic reward Zint = Ei= 7 Tint, where r denotes the intrinsic reward (e.g., Tint ming,ee... Dr 00) the uniqueness reward in our case) as follows,
max AEB
E Prwal9hotin = max E Jrask + (1 - a)) Dine), AEB
(6)
where 0 < < is weight parameter. Such an objective is sensitive to the selection of as well as the formulation of rint- For example, our case formulating the intrinsic reward rint as ming, (0, 02), exp [ming, (0, 0,)] and - exp ming, Drv((0, (0, 0;)] will result in sig- nificantly different results. Besides, trade-off arises in the selection of a: while large may undermine the contribution of intrinsic reward, a small a could ignore the importance of the reward, leading to the failure of agent in solving the primal task.
To tackle these issues, we draw inspiration from the observation that social uniqueness motivates people in passive ways. In other words, it plays more like constraint rather than an additional target. Therefore, we change the multi-objective optimization problem in Eq into a constrained optimization problem as:
max 8E
s.t. Time, - ro 2 0, Vt 102t=1,2..... 1,2,...,
where ro is threshold indicating minimal permitted uniqueness, and Pint, denotes moving aver- age of fint,e- Further discussion on the selection of ro will be deliberated in Appendix()
From the perspective of optimization, Eq. can be viewed the constrained optimization problem in Eq. with the penalty term Timt and the penalty coeffi-
penalty method which replaces
cient 1-a > 0, where the difficulty lies in the selection of a. The work of Zhang al. 2019 a tackles this challenge by the Task Novel Bisector (TNB) in the form of Feasible Direction Methods (FDMs) (Zoutendijk] 1960). As a heuristic approximation, that approach requires reward shaping and intensive emphasis on Tint,t. Instead, in this work we propose to solve the constrained opti- mization problem (7) by resembling the Interior Point Methods (IPMs) (Potra & Wright 2000) Dantzig & Thapa 2006). In vanilla IPMs, the constrained optimization problem in E is solved by reforming it to an unconstrained form with an additional barrier term in the objective as
E Bf-o(gtask [Task
Σ a log (Fint,t ro)]-
(8)
AE
The limit of Eq. when a - 0 then leads to the solution of Eq. 7 Readers please refer to Appendix G more discussion on the correspondence between those novel policy seeking methods and constrained optimization methods.
However, directly applying the IPMs is computationally challenging and numerically unstable, pecially when a is small. Luckily, in our proposed RL paradigm where the behavior of an agent influenced by its peers, more natural way can be used. Precisely, since the learning process is based on sampled transitions, we can simply bound the collected transitions in the feasible region by permitting previous trained M policies f; E O i = 1,2,.... M sending termination signals during the training process of new agents. In other words. we implicitly bound the feasible region Consequently, during the training process, all
by terminating any new agent that steps outside valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies. At the end of the training, we then naturally obtain new policy that has sufficient uniqueness. In this way, we no longer need to consider the trade-off prob- lem between intrinsic and extrinsic rewards deliberately. The learning process our method is thus more robust and no longer suffer from objective inconsistency. As formulation of the con- strained optimization problem Eq. (7) is inspired by IPMs, we name our approach as Interior Policy Differentiation (IPD) method.
5

---

Under review as a conference paper at ICLR 2020
2045.40
Nchord: 20/5.83
senord: 2397.83
PPO Baselne
1 A
AAAAAL
Rehord: 2030.15
17 1
d
Al l
wehord: 2016.7
2279.99
Our Mathod
192299
Reword: 93
Cewords 12
PPO Baseine
Reword:
2063558
Reword: 2514.11
Method
Figure 2: Results policy differentiation on Walker2d-v3 and HalfCheetah--3. Compared to the PPO baseline, our method significantly diversifies trained policies while maintaining their perfor- mances.
EXPERIMENTS
The MuJoCo environment We demonstrate our proposed method on the OpenAl Gym where the
physics engine based on MuJoCo (Brockman et al. 2016 Todorov et al. 2012). Concretely, we test three locomotion environments, the Hopper-v3 (II observations and 3 actions), Walker2d- v3 (11 observations and 2 actions), and HalfCheetah-v3 (17 observations and actions). our experiments, all the environment parameters set default values.
Uniqueness beyond intrinsic stochasticity Experiments in Henderson et al. (2018) show that poli- cies that perform differently be produced by simply selecting different random seeds before training. Before applying our method to improve behavior diversity, we firstly benchmark how much uniqueness can be generated from the stochasticity in the training process of vanilla RL algo- rithms as well as the random weight initialization. In this work, we mainly demonstrate our proposed method based PPO(Schulman al. 2017). The extension other popular algorithms is straight- forward. We also compare our proposed method with the TNB and weighted sum reward (WSR) approaches as different ways combine the goal of the task and the uniqueness motivation (Zhang et al. 2019). More implementation details are depicted in Appendix D
5.1 UNIQUENESS AND PERFORMANCE COMPARISON
According to Theorem 2, the uniqueness Tine in equation under our uniqueness metric can be un- biased approximated by Vin = ming, c Dhy (8(s)) 0, i.e., we utilize the metric directly in learning policies instead of applying any kind of reshaping.
We implement WSR. TNB, and our method in the same experimental settings and for each method, 10 different policies trained and try to be unique with regard all previously trained policies

---

Under review as a conference paper at ICLR 2020
Hopper-v3
Walker2d-v3
HallCheetah--3
Method + PPO WSR
3000
2500
2000
TNB
Reward
2000
Ours
1500
1000
1000
500
0
100
1.5
2.0
0.8
1.0
1.2
1.4
1.6
0.8
1.2
Rolative Uniquaness
Relative Uniquaness
Rolative Uniqueness
Figure 3: The scatter plot of different policies in terms Uniqueness and Performance in Hopper- v3, Walker2d-v3 and HalfCheetah-v3 environments. The value of uniqueness is normalized rela- tive uniqueness by regarding the averaged uniqueness of PPO policies as the baseline.
Table 1: The reward and success rate learned 10 policies using different methods
Reward
Success Rate
Method
Hopper
Walker2d
HalfCheetah
Hopper
Walker2d
HalfCheetah
PPO PPO+WSR
839 + 753
1611 467
913 + 134
1.0
1.0
0.7
1083 + 768
1429 = 692
603 = 407
1.0
0.7
0.4
PPO+TNB
1064 + 644
1160 484
592 = 384
1.0
0.9
0.5
PPO+Ours
1858 + 744
1506 541
1442 + 588
1.0
1.0
0.9
sequentially. Concretely, the 1st policy is trained by ordinary PPO without any social influence. The 2nd policy should be different from 1st policy, and the 3rd should be different from the previous two policies, and so on. Fig shows the qualitative results of our method. We visualize the motion of agents by drawing multiple frames representing the pose of agents at different time steps in the same row. The horizontal interval between consecutive frames is proportional to the velocity of agents. The settings of the frequency of highlighted frames and the correlation between interval and velocity fixed for each environment. The visualization starts from the beginning of each episode and therefore the readers can get sense of the process of acceleration as well as the pattern of motion of agents clearly.
Fig. shows our experimental results in terms of uniqueness (the x-axis) and the performance (the y-axis). Policies in the upper right are the more unique ones with higher performance. In Hopper and HalfCheetah, our proposed method distinctively outperforms other methods. In Walker2d, both WSR and our method work well in improving the uniqueness of policies, but none of the three methods can find way to surpass the performance of PPO apparently. Detailed comparison on the task related rewards are carried out in Table [ A box figure depicting the performance of each trained policy and their reward gaining curve are disposed in Fig and in Appendix C And F in Appendix CPrroides more detailed results from the view of uniqueness.
5.2 SUCCESS RATE OF EACH METHOD
In addition to averaged reward, we also use success rate as another metrics to compare the perfor- mance of different approaches. In this work, we consider policy is success when its performance is at least as good as the averaged performance of policies trained without social influences. To be specific, we use the averaged final performance PPO as the baseline. If new policy, which aims at performing differently to solve the same task, surpasses the baseline during its training process, will be regarded as a successful policy. Through the success rate, we know the policy does not learn unique behavior at the expense of performance. Table Ishows the success rate of all the meth- ods, including the PPO baseline. The results show that our method can always surpass the average baseline during training. Thus the performance of method can always be insured.

---

Under review as a conference paper at ICLR 2020
Hopper
Welhedd
Hal HalhOhanhh
208
293
1759
2000
17a
158
238
1259
199
509
14
Figure 4: Performance curves along with the number of peers. The results are averaged over repetitions of individual experiments.
5.3 BETTER POLICY DISCOVERY
In our experiments, we observed noticeable performance improvements the Hopper and the HalfCheetah environments. For the environment of Hopper, in many cases, the agents trained with PPO tend to learn a policy that jumps as far as possible and then fall to the ground and terminate this episode (please refer to Fig 11 in Appendix ED. Our proposed method can prevent new policies from always falling into the same local minimum. After the first policy being trapped in a local minimum, the following policies will try other approaches to avoid the same behavior, explore other feasible action patterns, and thereafter the performance may get improved. Such property shows that our method can be helpful enhancement of the traditional RL scheme, which can be epito- mized as policies could make mistakes, but they should explore more instead of hanging around the same local minimum. The similar feature attributes to the reward growth in the environment of HalfCheetah.
Moreover, we can illuminate the performance improvement of HalfCheetah from another perspec- tive. The environment of HalfCheetah is quite different from the other two for there is no explicit termination signal in its default settings (i.e., no explicit action like falling to the ground would trig- ger termination). At the beginning of the learning process, agent will act randomly, resulting in massive repeat, trivial samples as well as large control costs. In our learning scheme, since the agent also interacts with the peers, much effort acting randomly. During the learning process in our method, an agent will first learn to terminate itself then learns to behave differently to pursue higher reward. From this point of view, such learning process can be regarded kind of implicit curriculum.
receive termination signals from the peers to prevent wasting too
as possible to avoid heavy control costs by imitating previous policies and
5.4 SCALE OF THE INFLUENCE
As the number of policies learned with social influence grows, the difficulty of finding unique policy may also increase. Later policies must keep away from all previous solutions. The results of ablation study how the performance changes under different scales of social influence (i.e., the number of peers) is shown in Fig. 4 where the thresholds are selected according to our previous ablation study in Sec. [ The performance decrease is more obvious in Hopper than the other two environments for the action space of Hopper is only 3 dimensional. Thus the number of possible diverse policies be discovered is limited.
CONCLUSION
In this work, develop efficient approach to motivate RI. to learn diverse strategies inspired by social influence. After defining the distance between policies, we introduce the definition of policy uniqueness. Regarding the problem constrained optimization problem, our proposed method, Interior Policy Differentiation (IPD), draws the key insight of the Interior Point Methods. And our experimental results demonstrate IPD can learn various well-behaved policies, and our approach can help agents to avoid local minimum and can be interpreted as kind of implicit curriculum learning certain cases.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study curiosity drivven learning. arXiv preprint 1rXiv:1808.04355 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894, 2018b.
Cindy Chan, Jonah Berger, and Leaf Van Boven. Identifiable but not identical: Combining social identity and uniqueness motives in choice. Journal of Consumer research, 39(3):561-573, 2012.
Conn, Nick Gould, and Ph Toint. A globally convergent lagrangian barrier algorithm for opti- mization with general inequality constraints and simple bounds. Mathematiccs of Computation of the American Mathematical Society, 66(217):261-28, 1997.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a pop- ulation of novelty-seeking agents. In Advances in Neural Information Processing Systems, pp- 5027-5038, 2018.
George B Dantzig and Mukund N Thapa. Linear programming 2: theory and extensions. Springer Science & Business Media, 2006.
Charles Darwin. On the origin of species, 1859. Routledge, 1859.
Dominik Maria Endres and Johannes Schindelin. A new metric for probability distributions. IEEE Transactions on Information theory, 2003.
Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In International Symposium oninformation Theory, 2004. ISIT 2004. Proceedings., pp. 31. IEEE, 2004.
Yuval Noah Harari. Sapiens: brief history of lumankind. Random House, 2014.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 1707.022,, 2017.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference Artificial Intel- ligence, 2018
Joseph Henrich. The secret of our success: How culture is driving human evolution, domesticating our species, and making smarter. Princeton University Press, 2017.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Variational information maximizing exploration. 2016.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 1802. 2018.
Joel Lehman and Kenneth O Stanley. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336. 2008.
Joel Lehman and Kenneth Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 2011.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. CoRR, abs/1902.000288. 2019. uttp::tt://aaxx
aaa//b////5 /ets/192...0 00528
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.

---

Under review as a conference paper at ICLR 2020
Eric R Pianka. On r-and k-selection. The american naturalist, 104(940):592-59)) 1970.
Florian A Potra and Stephen J Wright. Interior-point methods. Journal of Computational and
Applied Mathematics, 124(1-2):281-302 2000.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolu- tionary computation. Frontiers Robotics and AI, 3:40, 2016.
Barbara Rogoff. Apprenticeship thinking: Cognitive development in social context. Oxford university press, 1990.
Richard M Ryan and Edward L Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology, 25(1):54-67, 2000.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Wolfram Schultz. Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, (535530))115331153 1997.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.066 065677, 2017.
Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yu- lun Li, Ludwig Schubert, Mare Bellemare, Jeff Clune, and Joel Lehman. An atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. arXiv preprint arXiv: 2018.
Richard Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 2. MIT press Cambridge, 1998.
Edward Thorndike. Animal intelligence: Experimental studies. Routledge, 2017.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033. IEEE, 2012. ISBN 978-1-4673-1137-55. URL http: 7/dblp. uni trier.rr.... de/ /bbe//bb/aa wwtp:///iisosrooo/ooee htt.::hoottmooooootoooo #oo.oooveoooL.o.oo
Carel van Schaik and Judith M Burkart. Social learning and evolution: the cultural intelligence hypothesis. Philosophical Transactions of the Royal Society B: Biological Sciences, 366(1567): 1008-1016, 2011.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint 19X1v:1901.010 00755,, 2019.
Stephen Wright. On the convergence of the caato//log-bbrriee method. Mathematical Program- ming, 90(1):71-100, 2001.
Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. CoRR, 055.1995500552, 2019. URL htttt:////axxvv www.aaiwwww.a.aaaaaaoooooooooooe abs/1905. 05252
Guus Zoutendijk. Methods of feasible directions: study in linear and non-linear programming. Elsevier, 1960.
10

---

Under review as a conference paper at ICLR 2020
A PROOF OF THEOREM
The first two properties are obviously guaranteed by Dry. As for the triangle inequality,
LA
(zv E p(3)
[0,(s) (s(s)
A
E sss)(s)
10,(s) 0,(s) +0)(s 0x(s))
(A1
S R Papp(s)
10:(s) 0,(60 + 10)(s) O2 (+)D]
[AI
IA
R p0))
10:(s) 0,(+)] E
105(s) (s)))
1=1
1=1
Expp(s) E (PT)((())))) Ea-p( (TT((()) (x))
B PROOF OF PROPOSITION 1
pa(s)= = P(so = s|0) + P(1 s|0) +... + P(sT = s(0)
Σ ((50-
s/Ti)
E 1(11 s|Ti)
E
1ST = S/T.)
L.L.N
lim
N
N
N
Ej=o Σ N
[(s) S(Ti)
ii=l
lim N
N
T
T(8 s/T))
(s)
N
i=
E Elpo (8) - ps() =0 0

---

Under review as a conference paper at ICLR 2020
C
DETAILS OF UNIQUENESS AND PERFORMANCE
19294NO
7254
1115
1280
400
Mechoo
Figure 5: The performance different methods in the Hopper, Walker and HalfCheetah environ- ments. The results are collected from 10 learned policies based on PPO. The box extends from the lower to upper quartile values of data, with a line at the median. The whiskers extend from the box to show the range of the data. Flier points those past the end of the whiskers.
Hopper v3 PPO
Hopper
2000
1500
Map
600
400
sead?
500
1000
2000
3000
4000
5050
6000
7000
8000
2000
4000
6000
805o
10000
12000 14000 16000
Training Episode
Training Egisode
Walker2d-43 PPO
Walker2d-43 Ours
1200
1250
1000
1000
800
750
500
250
200
0-
2003
4000
6000
5000
2000
4003
6003
8000
10000
Training Episode
Training Egisode
HallChoetah-v3 PPO
HallChestah-v3 Ours
seed?
seed2
1000
-500
500
1000
1500
2000
2500
3000
2000
4000
8000
8000
10000
12000
Trairing Spande
Training Epicnd
Figure 6: Avoid local minimum in Hopper and HalfCheetah: the left two figures show 10 policies generated by PPO each environment, the right two figures show 10 policies generated by our method in each environment
12

---

Under review as a conference paper at ICLR 2020
Papored
Amie
Figure 7: Maximal and minimal between policy uniqueness in Hopper, Walker2d and HalfCheetah environments. The results are averaged over all possible combinations of 10 policies. As TNB and WSR optimize the uniqueness reward directly, their uniqueness sometimes can exceed our proposed method. However, such direct optimization will lead to decreasing in task related performance as cost. To tackle the trade-off problem, carefully hyper-parameter tuning and reward shaping is always a must. Detailed comparison on the task related rewards are carried out in Table
D IMPLEMENTATION DETAILS
Calculation of DTV use deterministic part of policies in the calculation of Drv, i.e. we remove the Gaussian noise the action space in PPO and use Drv a2) = |ai -22.
Network Structure We use MLP with hidden layers as our actor models in PPO. The first hidden layer is fixed to have 32 units. Our ablation study the choice of unit number in the second layer detailed in T2 Table3] and Fig Moreover, we choose to use 10, 64 and 256 hidden units for the three tasks respectively in all of the main experiments, after taking the success rate (Table performance (Table 3 and computation expense (i.e. the preference to use less unit when the other two factors similar) into consideration.
Hepperai
Ui
Watbel ofiidde Mo
Namberr 12de wet.ee AA
Figure 8: Uniqueness increases a little as network complexity rises
Table The success rate of learned novel policies by our method in different environments under different thresholds
Thresood
hicllen
hidden
hridenn
0.5
Hopper
0.7
0.6
0.9
0.7
0.4
0.7
0.8
0.5
0.5
0.3
1.0
0.2
0.8
1.0
1.2
0.9
1.0
1.3
0.2
0.8
0.6
1.4
0.8
1.1
0.3
0.9
0.8
HailCheetab
1.2
1.3
1.4
0.4
0.9
1.0
1.5
0.2
0.2
0.7
Training Timesteps We fix the training timesteps in our experiments. The timesteps are fixed to be 1M in H3pper-v3, 1.6M for Walker2d-v3 and 3M for HalfCheetah.
13

---

Under review as a conference paper at ICLR 2020
Table 3: The final training performance of learned 10 novel policies by our method in different
environments under different thresholds, for Hopper and Walker, hn 10, h = 64, h3 = 256; for HalfCheetah, hn 64, h2 256, h3 512
Environment
Threshold
Average Performance
Top 30% Performance
hid mum
h1
Ag
hg
h1
Ag
hg
450 +
735
1202
737
0.6
1868 + 744
1040
+ 914
1626 869
2719 + 108
2263
764
2628
1: 207
Hopper
0.7
1180
740
593
150
785 = 462
2188 510
768
17
1287
- 580
0.8
744
1594
896
0.9
154
835
1347
387
1.0
187 + 220
206 256
499 = 764
490 + 194
648 # 141
1294 1 979
1611 + 467
1504 502
1724 584
2163 +
2018
2311
1.0
725 + 457
1174 + 5.9
1571 + 692
1346 = 466
2042 = 77
2270 = 37
Walker
1.1
1.2
725 + 654
1506 +
1453 480
1561 398
2079 +: 44
1903 104
1.3
405 + 647
1138 + 691
995 + 420
1124 = 796
1954 197
1523 + 313
1.4
393 + 518
831 400
1333 558
945 652
1352 273
2004 4: 106
0
1210 + 391
1278 + 373
1236 317
1606 = 296
1728 + 179
1601 + 241
1.1
434 +: 415
1055 + 265
914 = 427
967 365
1276 - 42
1330 1 207
HaleCheetah
+
1.3
830
+
2081
68
1.4
379 + 655
1224 + 412
1302 + 300
1187 = 305
1534
= 40
1659 + 107
1.5
257 1 442
527 727
780 593
765 642
1480 646
1550
92
Threshold Selection In our proposed method, we can control the magnitude of policy uniqueness flexibly by adjusting the constraint threshold ro- Choosing different thresholds will lead different policy behaviors. Concretely., larger threshold may drive the agent perform more differently while smaller threshold imposes a lighter constraint on the behavior of the agent. Intuitively, larger threshold will lead to relatively poor performance for the learning algorithm is less likely to find a feasible solution to (7.
Besides, we do not use constraints in the form of Eq.(7 need not force every single action of a new agent to be different from others. Instead, we are more care about the long term differences. Therefore, the cumulative uniqueness constraints,
max 8EB
E,-o
DE (r ro) 2 0, VT 1,2,
s.t.
We test method with different choices of threshold values. The performance of agents under different thresholds are shown in Fig. and more detailed analysis of their success rate is presented Table.
2335
200
1:8
380
38
Figure 9: The performance methods under different threshold selection, in the Hopper, Walker and HalfCheetah environments. the results are collected from 10 learned policies.
14

---

Under review as a conference paper at ICLR 2020
E
MORE QUALITATIVE RESULTS
32
Caard: 1020.85
Renord: 1451.09
1696.22
Beand 227
Renoid 2330113
2501.02
Reord: 2730.11
Deaord: 28-0.03
Figure 10: The visualization of policy behaviors of agents trained by our method in Hopper-v3 environment. Agents learn to jump with different strides.
Weoor:: 272.65
Reword 391 26
Rewdd: 461.16
417109
Wetord 539.11
werdr: 78370
wewrr: 785.42
Reword 2231.39
WeRord: 2577.11
Figure II: The visualization of policy behaviors of agents trained PPO in Hopper-v3 environ- ment. Most agents learn a policy that can be described as Jump as far as possible and fall down, leading to relative poor performance.
15

---

Under review as a conference paper at ICLR 2020
Resord: 1811.60
Reward: 1840.18
1881.77
Benord: 1891.80
Reword: 1906,70
1975.59
Rehord: 2051.08
Renord: 2063:56
2314.11
2377.01
Figure 12: The visualization of policy behaviors of agents trained by our method in HalfCheetah-v3
environment. Our agents run much faster compared to PPO agents and at mean time several patterns of motion have emerged.
scword: 1122.71
Beaud 1219.78
Reaord 1285.28
Baardd 1325.53
Roodd 1327.99
1348.8
Renord: T493.22
Renord: 1601.82
Figure 13: The visualization of policy behaviors of agents trained by PPO in HalfChectah-v2 envi- ronment. Since we only draw fixed number of frames in each line, in the limited time steps the PPO agents can not run enough distance to leave the range of our drawing, which shows that our agents run much faster.
16

---

Under review as a conference paper at ICLR 2020
1975-22
1995.03
A
Teauid 2030 19
neword: P0MMES
Renord: 20/7.93
1
A
Renord: 2279.93
2010.30
Figure 14: The visualization of policy behaviors of agents trained by our method in Walker2d-v3 environment. Instead of bouncing the ground using both legs, our agents learns to use both legs to step forward.
BeNONd 1429.18
Nehord: 1652.5
2040.40
ROAOO 2070.85
4
RchoNd 2086.05
2397.83
2620.10
Figure 15: The visualization of policy behaviors of agents trained by PPO in Walker2d-v3 environ- ment. Most of the PPO agents only learn to both legs to support the body and jump forward.
IMPLEMENTATION OF EQ.
We do not use constraints in the form of Eq. as we need not force every single action of new agent be different from others. Instead, we more care about the long term differences. There- fore, we use the cumulative uniqueness constraints. Moreover, the constraints can be applied after the first ts timesteps (e.g. ts 20) for the consideration of similar starting sequences.
max BE6
By-olgtaskl
s.t.
t=ts (Fint,t - ro) 2 0, S,..,
17

---

Under review as a conference paper at ICLR 2020
G RELATION BETWEEN DIFFERENT APPROACHES AND CONSTRAINED OPTIMIZATION METHODS
We note here, the WSR, TNB and IPD methods correspond to three approaches in constrained optimization problem. For simplicity, we consider Hq. (9) with more concise notion Jiut,t Ao.: 2 0, where Pint,t t Lint,t, i...
max
f(0) Er [task
BEB
(10)
8.1.
(e( = Pint,t - 90. 2 0, 1.2,..,,
As the optimization of policy is based on batches of trajectory samples and is implemented with stochastic gradient descent, (O. can be further simplified as:
max f(0) E,~o [9task]
#EB
(11)
s.t.
g J. (0) 2
where (0) denotes the average over a trajectory.
WSR: Penalty Method The Penalty Method considers the constraints of Eq. [I by putting con- straint g0) into penalty term, and then solve the unconstrained problem
1 1 min (g(0),
max BEO
((0)
(12)
using an iterative manner, and the limit when a + lead to the solution of the primal constrained problem. As an approximation, WSR choose a fixed weight term and use the gradient of Vof + 1-2 Veg instead of Vof + I-a V min(g(0), 0}, thus the final solution will intensely rely on the
selection of a.
TNB: Feasible Direction Method
The Taylor series of g(0) at point 0 is
g(ō + Ap) ( + V O(1A)))
(13)
The Feasible Direction Method (FDM) considers the constraints of Eq. 11 by first finding direc- tion 5 satisfies
VoST p> 0 V VogT. -> 0 if 9
(14)
so that for small X, we have
g(ō + Ap) 9 + AV g(0)= =0 if g =0
(15)
and
0 + Aτ) ( + 7>> 0 if ( > 0
(16)
The TNB method, by using the bisector of gradients Vef and Vag. select to be
Vaf
V agg (Vof. Vog) if cos (Vaf. V Vog) I 0 if cos (Vff. Vog) > 0
(17)
Vof
Vag
V.g
Clearly, (1)) satisfies Eq. [4). but it is more strict than Eq. as the Vag term always exists during the optimization of TNB. In TNB, the learning stride is fixed to be [Vaf++/Vagl leading to problem when Vaf + 0. which shows the final optimization result will heavily rely on the selection of g- i.e., the shape of g is crucial for the success of TNB.
18

---

Under review as a conference paper at ICLR 2020
IPD: Interior Point Methods (IPMs) In vanilla IPMs, the constrained optimization problem in Eq.011 is solved by reforming it to an unconstrained form with an additional barrier term the objective as
in
(()
f(0) + g(0)
(18)
or use the barrier term of -a log 9(0) instead:
f(0) a
(19)
where a, the barrier factor, is small positive number. As is small, the barrier term will introduce only minuscule influence on the objective. On the other hand, when 0 get closer to the barrier, the objective will increase fast. It is clear that the solution of the objective with barrier term will get closer to the primal objective sequence {ai} such that < Og < 0'+11 and a; - as k + oo The limit of Eq, (8). Eq. 19) when + then leads the solution of Eq. [I]. The work of Conn al. (1997); Wright 20011) provide proofs of the convergence.
getting smaller. Thus in practice, such methods will choose
Directly applying this method is computationally challenging and numerically unstable, especially when a small. A more natural way can be used: since the learning process is based on sampled transitions,
simply bound the collected transitions in the feasible region by permitting pre- vious trained M policies 0; Over, 1, 2.... .M sending termination signals during the training process of new agents. other words, we implicitly bound the feasible region by terminating any new agent that steps outside it.
Consequently, during the training process, all valid samples collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies. At the end of the training, we then naturally obtain a new policy that has sufficient uniqueness. In this way, we no longer need to consider the trade-off problem between intrinsic and extrinsic rewards deliberately. The learning process our method is thus more robust and no longer suffer from objective inconsistency. Algorithm [ shows the pseudo code of IPD based on PPO, where the blue lines show the addition to primal PPO algorithm.
Algorithm IPD with PPO, Actor-Critic Style
Require
a behavior policy 6old
a set of previous policies ().j 1,2,..., M
a uniqueness metric U0, [0)]lp) U0, (0,)17) ming, (7,-(0,0,)
a uniqueness threshold ro. starting point ts
Initialize Oold
for iteration 1, 2, do
for actor = 1, 2.., N. do
for 2, T do
Run policy Oau in environment, get trajectory
if U (0.aaa (0, - < 0, AND > Ly then
done True end if
if done then
break
end if end for Compute advantage estimates Å Åi,.., AT end for
Optimize surrogate CLIP w.n.t. 9 with K epochs and minibatch size M S NT Oald <- 0 end for
19


Reviews:
[{"review_id": "r1lkFUpCKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new way to incentivize diverse policy learning in RL agents: the key idea is that each agent receives an implicit negative reward (in the form of an early episode termination signal) from previous agents when an episode begins to resemble prior agents too much (as measured by the total variational distance measure between the two policy outputs). \n\nResults on three Mujoco tasks are mixed: when PPO is combined with the proposed objective for training diverse policies, it results in very strong performance boosts on Hopper and HalfCheetah, but falls significantly short of standard PPO on Walker 2D. I would have liked to see a deeper analysis of what makes the approach work in some environments and not in others. \n\nExperimental comparisons in the paper are only against alternative approaches to optimize the same diversity objective as the proposed approach (with weighted sum of rewards (WSR) or task novel bisection(TNB)). Given that this notion of diversity is itself being claimed as a contribution, I would expect to see comparisons against prior methods, such as in DIAYN. There are other methods that have been proposed before in similar spirit to induce diversity in the policies learned. Aside from the evolutionary approaches covered in related work, within RL too, there have been methods such as the max-entropy method proposed in Eysenbach et al, \"Diversity is All You Need...\". These methods, evolutionary and RL, could be compared against to make a more convincing experimental case for the proposed approach.\n\nThe experimental setting is also not fully clear to me: throughout experiments, are the diversity methods being evaluated for the average performance over all the policies learned in sequence to be different from prior policies? Or only the performance of the last policy? Related, I would be curious to know, if K policies are trained, the reward vs the training order k of the K policies. This is close to, but not identical to the study in Fig 4, to my understanding.\n\nAside from the above points being unclear, the paper in general could overall be better presented. While I am not an expert in this area, I would still expect to be able to understand and evaluate the paper better than I did. \n- Sec 3.1 makes a big deal of metric distance, but never quite explains how this is key to the method.\n- The exact baselines used in experiments are unhelpfully labeled \"TNB\" (with no nearby expansion) and \"weighted sum of rewards (WSR)\", with further description moved to appendix. In general, there are a few too many references to appendices.\n- The results in Fig 2 are difficult to assess for diversity, and this is also true for the video in the authors' comment.\n- There is an odd leap in the paper above Eq 7, where it claims that \"social uniqueness motivates people in passive ways\", which therefore suggests that \"it plays more like a constraint than an additional target\". \n- Sec 5.1 at one point points to Table 1 for \"detailed comparison on task related rewards\" but says nothing about any important conclusions from the table.\n- There are grammar errors throughout. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1g9D58Rtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for learning diverse policies in RL environments, with the ultimate goal of increasing reward. The paper develops a novel method, called interior policy differentiation (IPD), that constrains trained policy to be sufficiently different from one another. They test on 3 Mujoco domains, showing improved diversity in all of them and improved performance in 2 of them.\n\nOverall, this paper is very well executed. The explanation of the method is thorough, and the paper is well-written and polished. I like the idea of enforcing a constraint on the policy diversity via manipulating the transitions that the agents can learn from.  The experiments section compares to two other methods of increasing policy diversity and IPD outperforms both of them. I think this is a solid contribution to the literature on improving policy diversity.\n\nThat being said, I have some concerns about the paper:\n1) The motivation for explicitly encouraging diverse policies is a bit confusing, and isn’t very convincing. The paper draws inspiration from social influence in animal society, and say they formulate social influence in RL. First, the term social influence already has an established meaning in RL (see e.g. Jaques et al. (2018)) and refers to agents explicitly influencing others in a causal way in a multi-agent environment. Second, I think calling policy diversity a form of social influence is a bit of a stretch (and anthropomorphizes the agents unnecessarily). I think the paper should scrap the ‘social influence’ angle and instead frame it as ‘increasing policy diversity’. \n\nThe paper also motivates itself in comparison to Heess et al. (2017), which uses a set of environments to get diverse policies. However, the goal of these works are different: in Heess et al., the goal is to train agents that can exhibit complex behaviours in relatively simple environments (the focus is more on complexity of behaviours vs. the fact that agents in the same environment learn diverse policies). In this work, the goal is not to develop any more complex policies, but to have different agents on the same task learn diverse policies (and since the experiments are in Mujoco, the degree of diversity is limited). Thus, while the works are related, I don’t think the Heess et al. paper is good motivation for this work. \n\nI think the primary motivation that makes sense for explicitly encouraging diversity is to improve final performance on the task. Thus, I think it would be best for the paper to clarify the introduction by focusing on this. The paper could also give some reasons why having diverse policies is inherently a good thing (maybe for some applications with humans-in-the-loop it could be helpful?), but currently this is absent. \n\n2) Given that improving the final reward of an RL agent is the main goal, it’s not clear that the experiments (in 3 simple Mujoco settings) are enough to show this reliably. Specifically, it’s unclear whether encouraging diversity in this way will generalize to more complex tasks or domains (e.g. tasks in Mujoco with sparser reward, or environments with larger state spaces). It is possible that the success of the technique is most prevalent when there is only a small observation space. \n\n3) I’d like to see more discussion / analysis of *why* we’d expect diverse policies to lead to better rewards. In work on intrinsic motivation / curiosity for better exploration, it’s clear that encouraging agents to visit unseen states will lead to a better exploration of the state space, and thus will make them more likely to stumble upon rare rewards. But is this also true for policy diversity? Currently, the paper speculates that encouraging diversity could help agents not all fall into the same failure mode. But I could also imagine that it could lead agents to avoid a successful strategy that another agent learned. For example, if a certain sequence of moves is necessary at the beginning to avoid termination, the first agent could find this sequence of moves, but the other agents might avoid this sequence for the sake of diversity (depending on the threshold). Does something like this happen in practice? In my opinion, the environments considered aren’t rich enough to know. \n\n4) There are also some inconsistencies in results section. Specifically:\n- There seems to be a disagreement between the results in Table 1 (which uses 10 peers) and the ablation over number of peers in Figure 4, which shows that the performance with 10 agents is roughly the same as it is with 1 agent (and overall shows little positive trend between the number of peers and performance). \n- If ‘success rate’ means ‘percentage of time beating average PPO policy’, why does PPO sometimes get 100% in Table 1? \n\nGiven the concerns above, I’d assess the paper as being borderline for accept. I’m currently erring on the side of rejection, but I’d consider changing my score if some of the above points are addressed. \n\n\nSmaller concerns and questions:\n- There are a couple of instances where I found the claims of the paper with respect to related work to be over-stated. For example:\n‘Yet designing a complex environment requires a huge amount of manual efforts’ -> not necessarily. There is an initial engineering overhead, but it’s possible to generate environments programmatically with different properties, resulting in different agent behaviours. \nAlso:\nOn the Task-Novelty Bisector method of (Zhang et al., 2019): ‘the foundation of such joint optimization is not solid’. This is given without any explanation --- how is it not solid?\n\n- In the TNB and WSR implementation, what metric is being used? Is it the same as is defined in Section 3?\n\n- It would be nice to have some videos of the agents behavior to be able to more easily assess the learned policy diversity. \n\n\nSmall fixes:\n‘and similar results can be get’ -> and get similar results. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklTKgR-qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new algorithm for maximizing the diversity of different policies learned for a given task. The diversity is quantified using a metric, where in this case the total variation is used. A policy is different from a set of other policy if its minimum distance to all the other policies is high. The authors formulate a new constraint optimization problem where the diversity to previous policies is lower bounded in order to avoid a tedious search for combining task reward and diversity reward. The algorithm is evaluated on different Mojoco locomotion tasks. \n\nPositive Points:\n- The idea of maximizing the minimum total variation is novel and interesting\n- The approach seems to work better than current SOTA approaches for generating diverse behavior\n\nNegative Points:\n- The paper needs to be improved in terms of writing as in particular some of the main parts of the algorithm are unclear\n- The definition of Eq 7 does not make too much sense to me (see below)\n- The results have high variance and some conclusion drawn from it are hard to verify given the plots\n\nMore comments:\n- Eq 7 does not seem to be a very good choice to me. Why does the total variation needs to be different at *every* time step? We can certainly generate very diverse behavior even if the policy is exactly the same for some states. It could even be the case that for some states, only one action does not lead to a failure. In this case, Eq 7 would completely fail to produce any valid policy (?)\n- In general the writing is clear, it gets however quite unclear for the main part of the algorithm (after Eq. 7). It is unclear how equation 8 is obtained and why the limit of alpha going to 0 should lead to the same solution as Eq 7 (if alpha is 0 than it should be the same as optimizing just the task reward??). While this might be obvious for experts of the interior point method, it needs to be explained in much more detail in this paper. I think it is always a good strategy to make a paper self-contained, in particular for the main parts of the algorithm.\n- Also the termination mechanism needs to be much better explained. What reward is given in this case? The current formulation sounds quite heuristic to me, but maybe a better explanation can fix that.\n- while Fig 3 shows a clear advantage of the method, the section about better policy discovery would need better data to verify their claims. Fig 4 shows very noisy results and while for the hopper there might be a clear improvement of performance for number of policies > 2, this does not seem to be very significant for half cheetah. Given the amount of noise in the results many more trials would be needed to really make such statements.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a mechanism for obtaining diverse policies for solving a task by posing it as a multi-agent problem, and incentivizing the agents to be different from each other via maximizing total variation.

The reviewers agreed that this is an interesting idea, but had issues with the placement and exact motivations -- precisely what kind of diversity is the work after, why, and what accordingly related approaches does it need to be compared to.
Some reviewers also found the technical and exposition clarity to be lacking.

Given the consensus, I recommend rejection at this time, but encourage the authors to take the reviewers' feedback into account and resubmit to another venue.
Decision: Reject