INTERIOR POLICY DIFFERENTIATION
Year: 2019

Abstract:
Animals develop novel skills not only through the interaction with the environment but also from the influence of the others. In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers1. Specifically, we first define a metric to measure the distance between policies then quantitatively derive the definition of uniqueness. Unlike previous precarious joint optimization approaches, the social uniqueness motivation in our work is imposed as a constraint to encourage the agent to learn a policy different from the existing agents while still solve the primal task. The resulting algorithm, namely Interior Policy Differentiation (IPD), is able to learn a collection of policies that can solve a given task with distinct behaviors and brings about performance improvement as a byproduct in some cases.2.

heading: 1 INTRODUCTION
text: The paradigm of Reinforcement Learning (RL), inspired by cognition and animal studies (Thorndike, 2017; Schultz et al., 1997), can be described as learning by interacting with the environment to maximize a cumulative reward (Sutton et al., 1998). From the perspective of ecology, biodiversity as well as the development of various skills are crucial to the continuation and evolution of species (Darwin, 1859; Pianka, 1970). Thus the behavioral diversity becomes a rising topic in RL. Previous works have tried to encourage the emergence of behavioral diversity in RL with two approaches: The first approach is to design interactive environments which contain sufficient richness and diversity. For example, Heess et al. (2017) show that rich environments enable agents to learn different locomotion skills even using the standard RL algorithms. Yet designing a complex environment requires manual efforts, and the diversity is limited by the obstacle classes. The second approach to increase behavioral diversity is to motivate agents to explore beyond just maximizing the reward for the given task. Zhang et al. (2019) proposed to maximize a heuristically defined novelty metric between policies through task-novelty joint optimization, but the final performance of agents is not guaranteed.
In this work, we address the topic of policy differentiation in RL, i.e., to improve the diversity of RL agents while keeping their ability to solve the primal task. We draw the inspiration from the Social Influence in animal society (Rogoff, 1990; Ryan & Deci, 2000; van Schaik & Burkart, 2011; Henrich, 2017; Harari, 2014) and formulate the concept of social influence in the reinforcement learning paradigm. Our learning scheme is illustrated in Fig 1. The target agent not only learns to interact with the environment to maximize the reward but also differentiate the actions it takes in order to be different from other existing agents.
Since the social influence often acts on people passively as a sort of peer pressure, we implement the social influence in terms of social uniqueness motivation (Chan et al., 2012) and consider it as a constrained optimization problem. In the following of our work, we first define a rigorous policy distance metric in the policy space to compare the similarity of the agents. Then we develop an optimization constraint using the proposed metric, which brings immediate rather than episodic feedback in the learning process. A novel method, namely Interior Policy Differentiation (IPD), is further
1In this work, we use the term “peer” to denote a population of RL agents. 2Code will be made available soon
Run
proposed as a better solution for the constrained policy optimization problem. We benchmark our method on several locomotion tasks and show it can learn various diverse and well-behaved policies for the given tasks based on the standard Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017).

heading: 2 RELATED WORK
text: Intrinsic motivation methods. The Variational Information Maximizing Exploration (VIME) method is designed by Houthooft et al. (2016) to tackle the sparse reward problems. In VIME, an intrinsic reward term based on the maximization of information gains is added to contemporary RL algorithms to encourage exploration. The curiosity-driven methods, proposed by Pathak et al. (2017) and Burda et al. (2018a) define intrinsic rewards according to prediction errors of neural networks. i.e., when taking previous unseen states as inputs, networks trained with previous states will tend to predict with low accuracy, so that such prediction errors can be viewed as rewards. Burda et al. (2018b) proposed Random Network Distillation (RND) to quantify intrinsic reward by prediction differences between a fixed random initialized network and another randomly initialized network trained with previous state information. Liu et al. (2019) proposed Competitive Experience Replay (CER), in which they use two actors and a centralized critic, and defined an intrinsic reward by the state coincidence of two actors. The values of intrinsic rewards are fixed to be ±1 for the two actors separately. All of those approaches leverage the weighted sum of the external rewards, i.e., the primal rewards provided by environments, and intrinsic rewards that provided by different heuristics. A challenging problem is the trade-off between external rewards and intrinsic rewards. The Task-Novelty Bisector (TNB) learning method introduced by Zhang et al. (2019) aims to solve such problem by jointly optimize the extrinsic rewards and intrinsic rewards. Specifically, TNB updates the policy in the direction of the angular bisector of the two gradients, i.e., gradients of the extrinsic and intrinsic objective functions. However, the foundation of such joint optimization is not solid. Besides, creating an extra intrinsic reward function and evaluating the novelty of states or policies always requires additional neural networks such as auto-encoders. Thus extra computation expenses are needed (Zhang et al., 2019) .
Diverse behaviors from rich environments and algorithms. Heess et al. (2017) introduce the Distributed Proximal Policy Optimization (DPPO) method and enable agents with simulated bodies to learn complex locomotion skills in a diverse set of challenging environments. Although the learning reward they utilize is straightforward, the skills their policy learned are quite impressive and effective in traveling terrains and obstacles. Their work shows that rich environments can encourage the emergence of different locomotion behaviors, but extra manual efforts are required in designing such environments. The research of Such et al. (2018) shows that different RL algorithms may converge to different policies for the same task. The authors find that algorithms based on policy gradient tend to converge to the same local optimum in the game of Pitfall, while off-policy and value-based algorithms are prone to learn sophisticated strategies. On the contrary, in this paper, we are more interested in how to learn different policies through a single learning algorithm and learn the capability of avoiding local optimum.
Population-based novelty-seeking methods. Pugh et al. (2016) establish a standard framework for understanding and comparing different approaches to searching for quality diversity (QD). Conti et al. (2018) investigate adding novelty search (NS) and QD to evolution strategies (ES) to avoid local optima as well as achieve higher performance. Lehman & Stanley (2011; 2008) conclude that deriving an open-ended search algorithm that operates without pressure towards the ultimate objective is possible, suggesting ignoring the objective may often benefit the search itself. The work of Wang et al. (2019) yields a new kind of open-ended algorithm which indicates the solution to one environment might be a stepping stone to a new level of performance in another. Such et al. (2017) evolve a DNN with a population-based genetic algorithm (GA) for challenging RL tasks. By improving the vanilla TRPO algorithm (Schulman et al., 2015), Kurutach et al. (2018) maintains model uncertainty given the data collected from the environment via an ensemble of deep neural networks.

heading: 3 QUANTIFYING THE DISTANCE BETWEEN POLICIES
text: To encourage the emergence of behavioral diversity in RL, we first define a metric to measure the difference between policies, which is the foundation for the later algorithm we propose. We denote the learned policies as {πθi ; θi ∈ Θ, i = 1, 2, ...}, wherein θi represents parameters of the i-th policy, Θ denotes the whole parameter space. In the following, we omit π and denote a policy πθi as θi for simplicity unless stated otherwise.

heading: 3.1 DEFINITION
text: Mathematically, a metric should satisfy three important properties, namely the identity, the symmetry as well as the triangle inequality.
Definition 1 A metric space is an ordered pair (M,d) where M is a set and d is a metric on M , i.e., a function d : M ×M → R such that for any x, y, z ∈M , the following holds: 1. d(x, y) ≥ 0, d(x, y) = 0⇔ x = y, 2. d(x, y) = d(y, x), 3. d(x, z) ≤ d(x, y) + d(y, z).
We use the Total Variance Divergence DTV (Schulman et al., 2015) to measure the distance between policies. Concretely, for discrete probability distributions p and q, this distance is defined as DTV (p, q) = ∑ i |pi − qi|. 34
Theorem 1 (Metric Space (Θ, DρTV )) The expectation of DTV (·, ·) of two policies over any state distribution ρ(s):
D ρ
TV (θi, θj) := Es∼ρ(s)[DTV (θi(s), θj(s))], (1) is a metric on Θ, thus (Θ, D ρ
TV ) is a metric space.
The proof of Theorem 1 is in Appendix A. It is worth mentioning that, although TVD is used in our work, we can easily extend the result to use other distance between distributions as substitutes of TVD (e.g. Jensen Shannon divergence DJS or Wasserstein metric DW ) (Endres & Schindelin, 2003; Fuglede & Topsoe, 2004; Villani, 2008), and similar results can be get
Corollary 1 Let DρJS := Es∼ρ(s)[DJS(θi(s), θj(s))] and D ρ
W := Es∼ρ(s)[DW (θi(s), θj(s))], (Θ, D ρ
JS) and (Θ, D ρ W ) are also metric spaces.
On top of the metric space (Θ, D ρ
TV ), we could then compute the uniqueness of a policy.
Definition 2 (Uniqueness of Policy) Given a reference policy set Θref such that Θref = {θrefi , i = 1, 2, ...},Θref ⊂ Θ, the uniqueness U(θ|Θref) of policy θ is the minimal difference between θ and all policy in the reference policy set, i.e.,
U(θ|Θref) := min θj∈Θref
D ρ
TV (θ, θj). (2)
3It can be extended to continuous state and action spaces by replacing the sums with integrals. 4The factor 1
2 in Schulman et al. (2015) is omitted in our work for conciseness.
Consequently, to motivate RL with the social uniqueness, we hope our method can maximize the uniqueness of a new policy, i.e., maxθ U(θ|Θref ), where the Θref includes all the existing policies.
3.2 ESTIMATION OF D ρ
TV (θi, θj)
In practice, the calculation of D ρ
TV (θi, θj) is based on Monte Carlo estimation. i.e., we need to sample s from ρ(s). Although in finite state space we can get precise estimation after establishing ergodicity, problem arises when we are facing continuous state cases. i.e. it is difficult to efficiently get enough samples.
Formally, we denote the domain of ρ(s) as S and denote the domain of ρθ(s) as Sθ ⊂ S , where ρθ(s) := ρ(s|s ∼ θ) and in finite time horizon problems ρ(s|s ∼ θ) = P (s0 = s|θ) + P (s1 = s|θ) + ...+P (sT = s|θ). As we only care about the reachable regions, the domain S can be divided by S = limN→∞ ⋃N i=1 Sθi .
In order to improve the sample efficiency, we propose to approximate D ρ
TV (θi, θj) with D ρθ TV (θi, θj), where θ is a certain fixed behavior policy that irrelevant to θi, θj . Such approximation requires a necessary condition:
Condition 1 The domain of possible states are similar between different policies:∑ s∈S P (s ∈ (Sθ ∪ Sθj ) \ (Sθ ∩ Sθj )) ∑ s∈S P (s ∈ (Sθ ∩ Sθj )),∀j. (3)
When such condition holds, we can use ρ(s|s ∼ θ) as our choice of ρ(s), and the properties in Definition 1 still holds.
In practice, the Condition 1 always holds as we can ensure this by adding sufficiently large noise on θ, while the permitted state space is always limited. And for more general cases, to satisfy the properties in Definition 1, we must sample s from Sθ ∪ Sθj , accordingly,
D ρ
TV (θ, θj) = Es∼(Sθ∪Sj)[DTV (θ(s), θj(s))] = Es∼(Sθ∩Sθj )[DTV (θ(s), θj(s))] + Es∼(Sθ∪Sθj )\Sθj [DTV (θ(s),N )]+
Es∼(Sθ∪Sθj )\Sθ [DTV (N , θj(s))] (4)
where N represents random action when a policy have never been trained or visited such state domain. Plugging Eq.(4) into Eq.(2), the objective function of policy differentiation is
max θ min θj∈Θref
D ρ
TV (θ, θj) = Es∼(Sθ∩Sθj )[DTV (θ(s), θj(s))]
+ Es∼(Sθ∪Sθj )\Sθj [DTV (θ(s),N )] + Es∼(Sθ∪Sθj )\Sθ [DTV (N , θj(s))] (5)
While the first two terms are related to the policy θ, the last term is only related to the domain Sθ. If we enable sufficient exploration in training as well as in the initialization of θ, the last term will disappear (i.e. Sθj ⊂ Sθ). Hence we can also use D ρθi TV (θi, θj) as an approximation of D ρ
TV (θi, θj) in training of θi as long as sufficient exploration is guaranteed.
Proposition 1 (Unbiased Single Trajectory Estimation) The estimation of ρθ(s) using a single trajectory τ is unbiased.
The proof of Proposition 1 is in Appendix B. Given the definition of uniqueness and a practically unbiased sampling method, the next step is to develop an efficient learning algorithm.

heading: 4 INTERIOR POLICY DIFFERENTIATION
text: In the traditional RL paradigm, maximizing the expectation of cumulative rewards g = ∑ t=0 γ
trt is commonly used as the objective. i.e. maxθ∈Θ Eτ∼θ[g], where τ ∼ θ denotes a trajectory τ sampled from the policy θ using Monte Carlo methods.
To improve the behavioral diversity of different agents, the learning objective must take both reward from the primal task and the policy uniqueness into consideration. Previous approaches (Houthooft et al., 2016; Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019) often directly write the weighted sum of the reward from the primal task and the intrinsic reward gint = ∑ t=0 γ
trint,t, where rint,t denotes the intrinsic reward (e.g., rint = minθj∈Θref D ρ
TV (θ, θj) as the uniqueness reward in our case) as follows,
max θ∈Θ Eτ∼θ[gtotal] = max θ∈Θ Eτ∼θ[α · gtask + (1− α) · gint], (6)
where 0 < α < 1 is a weight parameter. Such an objective is sensitive to the selection of α as well as the formulation of rint. For example, in our case formulating the intrinsic reward rint as minθj D ρ TV (θ, θj), exp [minθj D ρ TV (θ, θj)] and − exp [−minθj D ρ
TV (θ, θj)] will result in significantly different results. Besides, a trade-off arises in the selection of α: while a large α may undermine the contribution of intrinsic reward, a small α could ignore the importance of the reward, leading to the failure of agent in solving the primal task.
To tackle these issues, we draw inspiration from the observation that social uniqueness motivates people in passive ways. In other words, it plays more like a constraint rather than an additional target. Therefore, we change the multi-objective optimization problem in Eq.(6) into a constrained optimization problem as:
max θ∈Θ
Eτ∼θ[gtask],
s.t. rint,t − r0 ≥ 0,∀t = 1, 2, ..., T, (7)
where r0 is a threshold indicating minimal permitted uniqueness, and rint,t denotes a moving average of rint,t. Further discussion on the selection of r0 will be deliberated in Appendix D.
From the perspective of optimization, Eq.(6) can be viewed as a penalty method which replaces the constrained optimization problem in Eq.(7) with the penalty term rint and the penalty coefficient 1−αα > 0, where the difficulty lies in the selection of α. The work of Zhang et al. (2019)) tackles this challenge by the Task Novel Bisector (TNB) in the form of Feasible Direction Methods (FDMs) (Zoutendijk, 1960). As a heuristic approximation, that approach requires reward shaping and intensive emphasis on rint,t. Instead, in this work we propose to solve the constrained optimization problem Eq.(7) by resembling the Interior Point Methods (IPMs) (Potra & Wright, 2000; Dantzig & Thapa, 2006). In vanilla IPMs, the constrained optimization problem in Eq.(7) is solved by reforming it to an unconstrained form with an additional barrier term in the objective as
max θ∈Θ Eτ∼θ[gtask + T∑ t=0 α log (rint,t − r0)]. (8)
The limit of Eq.(8) when α → 0 then leads to the solution of Eq.(7). Readers please refer to Appendix G for more discussion on the correspondence between those novel policy seeking methods and constrained optimization methods.
However, directly applying the IPMs is computationally challenging and numerically unstable, especially when α is small. Luckily, in our proposed RL paradigm where the behavior of an agent is influenced by its peers, a more natural way can be used. Precisely, since the learning process is based on sampled transitions, we can simply bound the collected transitions in the feasible region by permitting previous trained M policies θi ∈ Θref, i = 1, 2, ...,M sending termination signals during the training process of new agents. In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it. Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies. At the end of the training, we then naturally obtain a new policy that has sufficient uniqueness. In this way, we no longer need to consider the trade-off problem between intrinsic and extrinsic rewards deliberately. The learning process of our method is thus more robust and no longer suffer from objective inconsistency. As our formulation of the constrained optimization problem Eq.(7) is inspired by IPMs, we name our approach as Interior Policy Differentiation (IPD) method.

heading: 5 EXPERIMENTS
text: The MuJoCo environment We demonstrate our proposed method on the OpenAI Gym where the physics engine is based on MuJoCo (Brockman et al., 2016; Todorov et al., 2012). Concretely, we test on three locomotion environments, the Hopper-v3 (11 observations and 3 actions), Walker2dv3 (11 observations and 2 actions), and HalfCheetah-v3 (17 observations and 6 actions). In our experiments, all the environment parameters are set as default values.
Uniqueness beyond intrinsic stochasticity Experiments in Henderson et al. (2018) show that policies that perform differently can be produced by simply selecting different random seeds before training. Before applying our method to improve behavior diversity, we firstly benchmark how much uniqueness can be generated from the stochasticity in the training process of vanilla RL algorithms as well as the random weight initialization. In this work, we mainly demonstrate our proposed method based on PPO(Schulman et al., 2017). The extension to other popular algorithms is straightforward. We also compare our proposed method with the TNB and weighted sum reward (WSR) approaches as different ways to combine the goal of the task and the uniqueness motivation (Zhang et al., 2019). More implementation details are depicted in Appendix D.

heading: 5.1 UNIQUENESS AND PERFORMANCE COMPARISON
text: According to Theorem 2, the uniqueness rint in equation (7) under our uniqueness metric can be unbiased approximated by rint = minθj∈Θref D ρθ TV (θ(st), θj(st)). i.e., we utilize the metric directly in learning new policies instead of applying any kind of reshaping.
We implement WSR, TNB, and our method in the same experimental settings and for each method, 10 different policies are trained and try to be unique with regard to all previously trained policies
sequentially. Concretely, the 1st policy is trained by ordinary PPO without any social influence. The 2nd policy should be different from 1st policy, and the 3rd should be different from the previous two policies, and so on. Fig.2 shows the qualitative results of our method. We visualize the motion of agents by drawing multiple frames representing the pose of agents at different time steps in the same row. The horizontal interval between consecutive frames is proportional to the velocity of agents. The settings of the frequency of highlighted frames and the correlation between interval and velocity are fixed for each environment. The visualization starts from the beginning of each episode and therefore the readers can get sense of the process of acceleration as well as the pattern of motion of agents clearly.
Fig. 3 shows our experimental results in terms of uniqueness (the x-axis) and the performance (the y-axis). Policies in the upper right are the more unique ones with higher performance. In Hopper and HalfCheetah, our proposed method distinctively outperforms other methods. In Walker2d, both WSR and our method work well in improving the uniqueness of policies, but none of the three methods can find way to surpass the performance of PPO apparently. Detailed comparison on the task related rewards are carried out in Table 1. A box figure depicting the performance of each trained policy and their reward gaining curve are disposed in Fig.5 and Fig.6 in Appendix C. And Fig.7 in Appendix C provides more detailed results from the view of uniqueness.

heading: 5.2 SUCCESS RATE OF EACH METHOD
text: In addition to averaged reward, we also use success rate as another metrics to compare the performance of different approaches. In this work, we consider a policy is success when its performance is at least as good as the averaged performance of policies trained without social influences. To be specific, we use the averaged final performance of PPO as the baseline. If a new policy, which aims at performing differently to solve the same task, surpasses the baseline during its training process, it will be regarded as a successful policy. Through the success rate, we know the policy does not learn unique behavior at the expense of performance. Table 1 shows the success rate of all the methods, including the PPO baseline. The results show that our method can always surpass the average baseline during training. Thus the performance of our method can always be insured.
Hopper
Walker2d
HalfCheetah

heading: 5.3 BETTER POLICY DISCOVERY
text: In our experiments, we observed noticeable performance improvements in the Hopper and the HalfCheetah environments. For the environment of Hopper, in many cases, the agents trained with PPO tend to learn a policy that jumps as far as possible and then fall to the ground and terminate this episode (please refer to Fig.11 in Appendix E). Our proposed method can prevent new policies from always falling into the same local minimum. After the first policy being trapped in a local minimum, the following policies will try other approaches to avoid the same behavior, explore other feasible action patterns, and thereafter the performance may get improved. Such property shows that our method can be a helpful enhancement of the traditional RL scheme, which can be epitomized as policies could make mistakes, but they should explore more instead of hanging around the same local minimum. The similar feature attributes to the reward growth in the environment of HalfCheetah.
Moreover, we can illuminate the performance improvement of HalfCheetah from another perspective. The environment of HalfCheetah is quite different from the other two for there is no explicit termination signal in its default settings (i.e., no explicit action like falling to the ground would trigger termination). At the beginning of the learning process, an agent will act randomly, resulting in massive repeat, trivial samples as well as large control costs. In our learning scheme, since the agent also interacts with the peers, it can receive termination signals from the peers to prevent wasting too much effort acting randomly. During the learning process in our method, an agent will first learn to terminate itself as soon as possible to avoid heavy control costs by imitating previous policies and then learns to behave differently to pursue higher reward. From this point of view, such learning process can be regarded as a kind of implicit curriculum.

heading: 5.4 SCALE OF THE INFLUENCE
text: As the number of policies learned with social influence grows, the difficulty of finding a unique policy may also increase. Later policies must keep away from all previous solutions. The results of our ablation study on how the performance changes under different scales of social influence (i.e., the number of peers) is shown in Fig. 4, where the thresholds are selected according to our previous ablation study in Sec. D. The performance decrease is more obvious in Hopper than the other two environments for the action space of Hopper is only 3 dimensional. Thus the number of possible diverse policies can be discovered is limited.

heading: 6 CONCLUSION
text: In this work, we develop an efficient approach to motivate RL to learn diverse strategies inspired by social influence. After defining the distance between policies, we introduce the definition of policy uniqueness. Regarding the problem as constrained optimization problem, our proposed method, Interior Policy Differentiation (IPD), draws the key insight of the Interior Point Methods. And our experimental results demonstrate IPD can learn various well-behaved policies, and our approach can help agents to avoid local minimum and can be interpreted as a kind of implicit curriculum learning in certain cases.

heading: A PROOF OF THEOREM 1
text: The first two properties are obviously guaranteed by D ρ
TV . As for the triangle inequality,
Es∼ρ(s)[DTV (θi(s), θk(s)] = Es∼ρ(s)[ |A|∑ l=1 |θi(s)− θk(s)|]
= Es∼ρ(s)[ |A|∑ l=1 |θi(s)− θj(s) + θj(s)− θk(s)|] ≤ Es∼ρ(s)[ (|A|∑ l=1 |θi(s)− θj(s)|+ |θj(s)− θk(s)|)] = Es∼ρ(s)[ |A|∑ l=1 |θi(s)− θj(s)|] + Es∼ρ(s)[ |A|∑ l=1 |θj(s)− θk(s)|]
= Es∼ρ(s)[DTV (θi(s), θj(s)] + Es∼ρ(s)[DTV (θj(s), θk(s)]

heading: B PROOF OF PROPOSITION 1
text: ρθ(s) = P (s0 = s|θ) + P (s1 = s|θ) + ...+ P (sT = s|θ)
L.L.N. = lim
N→∞
∑N i=1 I(s0 = s|τi)
N +
∑N i=1 I(s1 = s|τi)
N + ...+
∑N i=1 I(sT = s|τi)
N
= lim N→∞
∑T j=0 ∑N i=1 I(sj = s|τi) N
ρθ(s) = N∑ i=1 T∑ j=0 I(sj = s|τi) N
E[ρθ(s)− ρθ(s)] = 0

heading: C DETAILS OF UNIQUENESS AND PERFORMANCE
text: Hopper-v3
Walker2d-v3
HalfCheetah-v3
D IMPLEMENTATION DETAILS
Calculation of DTV We use deterministic part of policies in the calculation of DTV , i.e., we remove the Gaussian noise on the action space in PPO and use DTV (a1, a2) = |a1 − a2|.
Network Structure We use MLP with 2 hidden layers as our actor models in PPO. The first hidden layer is fixed to have 32 units. Our ablation study on the choice of unit number in the second layer is detailed in Table.2, Table3 and Fig.8. Moreover, we choose to use 10, 64 and 256 hidden units for the three tasks respectively in all of the main experiments, after taking the success rate (Table.2), performance (Table.3) and computation expense (i.e. the preference to use less unit when the other two factors are similar) into consideration.
Training Timesteps We fix the training timesteps in our experiments. The timesteps are fixed to be 1M in Hopper-v3, 1.6M for Walker2d-v3 and 3M for HalfCheetah.
Threshold Selection In our proposed method, we can control the magnitude of policy uniqueness flexibly by adjusting the constraint threshold r0. Choosing different thresholds will lead to different policy behaviors. Concretely., a larger threshold may drive the agent to perform more differently while smaller threshold imposes a lighter constraint on the behavior of the agent. Intuitively, a larger threshold will lead to relatively poor performance for the learning algorithm is less likely to find a feasible solution to Eq.(7).
Besides, we do not use constraints in the form of Eq.(7) as we need not force every single action of a new agent to be different from others. Instead, we are more care about the long term differences. Therefore, we use the cumulative uniqueness as constraints,
max θ∈Θ
Eτ∼θ[gtask],
s.t. ∑t=τ t=0(rint,t − r0) ≥ 0,∀τ = 1, 2, ..., T,
We test our method with different choices of threshold values. The performance of agents under different thresholds are shown in Fig. 9 and more detailed analysis of their success rate is presented in Table. 2.
Hopper-v3
Walker2d-v3
HalfCheetah-v3

heading: E MORE QUALITATIVE RESULTS
text: F IMPLEMENTATION OF EQ.(7)
We do not use constraints in the form of Eq.(7) as we need not force every single action of a new agent to be different from others. Instead, we are more care about the long term differences. Therefore, we use the cumulative uniqueness as constraints. Moreover, the constraints can be applied after the first tS timesteps (e.g. tS = 20) for the consideration of similar starting sequences.
max θ∈Θ
Eτ∼θ[gtask],
s.t. ∑t=τ t=tS (rint,t − r0) ≥ 0, τ = S, ..., T, (9)

heading: G RELATION BETWEEN DIFFERENT APPROACHES AND CONSTRAINED OPTIMIZATION METHODS
text: We note here, the WSR, TNB and IPD methods correspond to three approaches in constrained optimization problem. For simplicity, we consider Eq.(9) with a more concise notion gint,t − g0,t ≥ 0, where gint,t = ∑t t=0 rint,t, i.e.,
max θ∈Θ
f(θ) = Eτ∼θ[gtask]
s.t. gt(θ) = gint,t − g0,t ≥ 0, t = 1, 2, ..., T (10)
As the optimization of policy is based on batches of trajectory samples and is implemented with stochastic gradient descent, Eq.(10) can be further simplified as:
max θ∈Θ
f(θ) = Eτ∼θ[gtask]
s.t. g(θ) = gt(θ) ≥ 0 (11)
where gt(θ) denotes the average over a trajectory.
WSR: Penalty Method The Penalty Method considers the constraints of Eq.(11) by putting constraint g(θ) into a penalty term, and then solve the unconstrained problem
max θ∈Θ f(θ) + 1− α α min{g(θ), 0} (12)
using an iterative manner, and the limit when α → 0 lead to the solution of the primal constrained problem. As an approximation, WSR choose a fixed weight term α, and use the gradient of ∇θf + 1−α α ∇θg instead of ∇θf + 1−α α ∇θ min{g(θ), 0}, thus the final solution will intensely rely on the selection of α.
TNB: Feasible Direction Method The Taylor series of g(θ) at point θ̄ is
g(θ̄ + λ~p) = g(θ̄) +∇θg(θ̄)Tλ~p+O(||λ~p||) (13)
The Feasible Direction Method (FDM) considers the constraints of Eq.(11) by first finding a direction ~p satisfies
∇θfT · ~p > 0 ∇θgT · ~p > 0 if g = 0
(14)
so that for small λ, we have
g(θ̄ + λ~p) = g(θ̄) + λ∇θg(θ̄)T~p > g(θ̄) = 0 if g(θ̄) = 0 (15)
and g(θ̄ + λ~p) = g(θ̄) + λ∇θg(θ̄)T~p > 0 if g(θ̄) > 0 (16)
The TNB method, by using the bisector of gradients∇θf and∇θg, select ~p to be
~p = { ∇θf + |∇θf ||∇θg|∇θg · cos (∇θf,∇θg) if cos (∇θf,∇θg) ≤ 0 ∇θf + |∇θf ||∇θg|∇θg if cos (∇θf,∇θg) > 0
(17)
Clearly, Eq.(17) satisfies Eq.(14), but it is more strict than Eq.(14) as the ∇θg term always exists during the optimization of TNB. In TNB, the learning stride is fixed to be |∇θf |+|∇θg|2 , leading to problem when∇θf → 0, which shows the final optimization result will heavily rely on the selection of g. i.e., the shape of g is crucial for the success of TNB.
IPD: Interior Point Methods (IPMs) In vanilla IPMs, the constrained optimization problem in Eq.(11) is solved by reforming it to an unconstrained form with an additional barrier term α 1g(θ) in the objective as
max θ∈Θ
f(θ) + α 1
g(θ) (18)
or use the barrier term of −α log g(θ) instead:
max θ∈Θ
f(θ)− α log g(θ) (19)
where α, the barrier factor, is a small positive number. As α is small, the barrier term will introduce only minuscule influence on the objective. On the other hand, when θ get closer to the barrier, the objective will increase fast. It is clear that the solution of the objective with barrier term will get closer to the primal objective as α getting smaller. Thus in practice, such methods will choose a sequence of {αi} such that 0 < αi < αk+1 and αi → 0 as k → ∞ The limit of Eq.(18), Eq.(19) when α → 0 then leads to the solution of Eq.(11). The work of Conn et al. (1997); Wright (2001) provide proofs of the convergence.
Directly applying this method is computationally challenging and numerically unstable, especially when α is small. A more natural way can be used: since the learning process is based on sampled transitions, we can simply bound the collected transitions in the feasible region by permitting previous trained M policies θi ∈ Θref, i = 1, 2, ...,M sending termination signals during the training process of new agents. In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it.
Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies. At the end of the training, we then naturally obtain a new policy that has sufficient uniqueness. In this way, we no longer need to consider the trade-off problem between intrinsic and extrinsic rewards deliberately. The learning process of our method is thus more robust and no longer suffer from objective inconsistency. Algorithm.1 shows the pseudo code of IPD based on PPO, where the blue lines show the addition to primal PPO algorithm.
Algorithm 1 IPD with PPO, Actor-Critic Style Require • a behavior policy θold • a set of previous policies {j}, j = 1, 2, ...,M • a uniqueness metric U(θ, {θj}|ρ) = U(θ, {θj}|τ) = minθj D τ TV (θ, θj)
• a uniqueness threshold r0, starting point tS Initialize θold for iteration = 1, 2, ... do
for actor = 1, 2, ..., N do for t = 1, 2, ..., T do
Run policy θold in environment, get trajectory τ if U(θold, {θj}|τ)− r0 < 0, AND t > tS then
done = True end if if done then
break end if
end for Compute advantage estimates Â1, ..., ÂT
end for Optimize surrogate LCLIP w.r.t. θ, with K epochs and minibatch size M ≤ NT θold ← θ
end for

Review summaries:
This paper proposes a new method for learning diverse policies in RL environments, with the ultimate goal of increasing reward. The paper develops a novel method, called interior policy differentiation (IPD), that constrains trained policy to be sufficiently different from one another. They test on 3 Mujoco domains, showing improved diversity in all of them and improved performance in 2 of them.

The paper presents a new algorithm for maximizing the diversity of different policies learned for a given task. The diversity is quantified using a metric, where in this case the total variation is used. A policy is different from a set of other policy if its minimum distance to all the other policies is high. The authors formulate a new constraint optimization problem where the diversity to previous policies is lower bounded in order to avoid a tedious search for combining task reward and diversity reward. The algorithm is evaluated on different Mojoco locomotion tasks.