ERL-Re$^2$: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation 
Authors: Jianye HAO
Pengyi Li
Hongyao Tang
YAN ZHENG
Xian Fu
Zhaopeng Meng
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Evolutionary Algorithm
Representation

Abstract: Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithm (EA) are two major paradigms of policy optimization with distinct learning principles, i.e., gradient-based v.s. gradient-free. An appealing research direction is integrating Deep RL and EA to devise new methods by fusing their complementary advantages. However, existing works on combining Deep RL and EA have two common drawbacks:1) the RL agent and EA agents learn their policies individually, neglecting efficient sharing of useful common knowledge; 2) parameter-level policy optimization guarantees no semantic level of behavior evolution for the EA side. In this paper, we propose Evolutionary Reinforcement Learning with Two-scale State Representation and Policy Representation (ERL-Re$^2$), a novel solution to the aforementioned two drawbacks. The key idea of ERL-Re$^2$ is two-scale representation: all EA and RL policies share the same nonlinear state representation while maintaining individual linear policy representations. The state representation conveys expressive common features of the environment learned by all the agents collectively; the linear policy representation provides a favorable space for efficient policy optimization, where novel behavior-level crossover and mutation operations can be performed. Moreover, the linear policy representation allows convenient generalization of policy fitness with the help of Policy-extended Value Function Approximator (PeVFA), further improving the sample efficiency of fitness estimation. The experiments on a range of continuous control tasks show that ERL-Re$^2$ consistently outperforms advanced baselines and achieves the State Of The Art (SOTA). Our code is available on  https://github.com/yeshenpy/ERL-Re2.

Published as a conference paper at ICLR 2023
ERL-RE EFFICIENT EVOLUTIONARY REINFORCE- MENT LEARNING WITH SHARED STATE REPRESENTA- TION AND INDIVIDUAL POLICY REPRESENTATION
Jianye Hao, Pengyi Li, Hongyao Tang, Yan Zheng, Xian Fu, Zhaopeng Meng College of Intelligence and Computing, Tianjin University, China
ABSTRACT
Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithms (EA) are two major paradigms of policy optimization with distinct learning principles, i.e., basde v.s. gradient-free. An appealing research direction is integrating Deep RL and EA to devise new methods by fusing their complementary advan- tages. However, existing works on combining Deep RL and EA have two common drawbacks: 1) the RL agent and EA agents learn their policies individually, ne- glecting efficient sharing of useful common knowledge; 2) parameter optimization guarantees no semantic level of behavior evolution for the EA side. In this paper, we propose Evolutionary Reinforcement Learning with Two-scale State Representation and Policy Representation (ERL-Re²), a novel solution the aforemenniooed two drawbacks. The key idea of ERL-Re² is two-scale repre- sentation; all EA and RL policies share the same nonlinear state representation while maintaining individual linear policy representations. The state representation conveys expressive common features of the environment learned by all the agents collectively; the linear policy representation provides favorable space for efficient policy optimization, where novel behavior-level crossover and mutation operations can be performed. Moreover, the linear policy representation allows convenient generalization of policy fitness with the help of the Approximator (PeVFA), further improving the sample efficiency of fitness estima- tion. The experiments on a range of continuous control tasks show that ERL-Re" consistently outperforms advanced baselines and achieves the State Of The Art (SOTA). Our code available on https//gghhubboooyyyeenn///////
policy
extended Value Function
INTRODUCTION
Reinforcement learning (RL) has achieved many successes in robot control (Yuan et al., 2022), game Al (Hao et al., 2022; 2019), supply chain (Ni et al., 2021) and etc (Hao et al., 2020). With function approximation like deep neural networks, the policy can be learned efficiently by tial-and-error with reliable gradient updates. However, RL is widely known to be unstable, poor in exploration, and struggling when the gradient signals are noisy and less informative. By contrast, Evolutionary class of black-box optimization methods, which is
Algorithms (E-A) (Bäck & Schwefel, 1993) demonstrated to be competitive with RL (Such et al., 2017). EA model natural evolution processes by maintaining a population of individuals and searching for favorable solutions by iteration. In each iteration, individuals with high fitness are selected to produce offspring by inheritance and variation, while those with low fitness are eliminated. Different from RI, EA are gradient- and offers several strengths: strong exploration ability, robustness, and stable convergence (Sigaud, 2022). Despite the advantages, one major bottleneck of EA is the low sample efficiency due to the iterative evaluation of the population. This issue becomes more stringent when the policy space large (Sigaud, 2022).
Since EA and RL have distinct and complementary advantages, a natural idea is to combine these two heterogeneous policy optimization approaches and devise better policy optimization algorithms. Many efforts in recent years have been made in this direction (Khadka & Tumer, 2018: Khadka et al., 2019: Bodnar et al., 2020; Wang et al., 2022; Shen et al., 2020). One representative work ERL (Khadka & Tumer, 2018) which combines Genetic Algorithm (GA) (Mitchell, 1998) and DDPG (Lillicrap et al., 2016). ERL maintains an evolution population and a RL agent meanwhile. The population and the RL agent interact with each other in coherent framework: the RI. agent

---

Published as a conference paper at ICLR 2023
learns by DDPG with diverse off oof-po experiences collected by the population; while the population includes a copy of the RL agent periodically among which genetic evolution operates. In this way, EA and RL cooperate during policy optimization. Subsequently, many variants and improvements of ERL are proposed, e.g.. to incorporate Cross-Entropy Method (CEM) (Pourchot & Sigaud, 2019) rather than GA (Pourchot & Sigaud, 2019), to devise gradient-baser genetic operators (Gangwani & Peng, 2018), to use multiple parallel RI. agents (Khadka et al., 2019) and etc. However, we observe that most existing methods seldom break the performance ceiling of either their EA or RI. components (e.g., Swimmer and Humanoid on MuJoCo are dominated by EA and RL respectively). This indicates that the strengths of EA and RL are not sufficiently blended. We attribute this to two major draw backs. First, each agent of EA and RI. learns its policy individually. The state representation learned by individuals can inevitably be redundant yet specialized (Dabney et al., 2021), thus slowing down the learning and limiting the convergence performance. Second, typical evolutionary variation occurs at the level of the parameter (e.g., network weights). It guarantees no semantic level of evolution and may induce policy crash (Bodnar et al., 2020).
In the literature of linear approximation RL (Sutton & Barto, 1998) and state representation learn- ing (Chung et al., 2019; Dabney et al., 2021; Kumar et al., 2021), policy is usually understood as the composition of nonlinear state features and linear policy weights. Taking this inspiration, we propose new approach named Evolutionary Reinforcement Learning with Two-scale State Representation and Policy Representation (ERL-Re²) to address the aforementioned two drawbacks. ERL-Re is devised based on a novel concept, i.e., /wo-scale representation: all EA and RL agents maintained in ERI-Re are composed of a shared nonlinear state representation and an individual linear policy representation. The shared state representation takes the responsibility of learning general and expressive features of the environment, which is not specific to any single policy, e.g., the common decision-related knowledge. In particular, it is learned by following unifying update direction derived from value function maximization regarding all EA and RI. agents collectively. Thanks to the expressivity of the shared state representation. the individual policy representation can have a simple linear form. It leads to a fundamental distinction of ERL-Re evolution and reinforcement occur in the linear policy representation space rather than in a nonlinear parameter (e.g.. policy network) space as the convention. Thus, policy optimization with ERL-Re". In addition, we propose novel behavior-leve/ crossover and mutation that allow to imposing variations on designated dimensions of action while incurring no interference on the others. Compared to parameter-level operators, our behavior-level operators have clear genetic semantics of behavior, thus are more effective and stable. Moreover, we further reduce the sample cost of EA by introducing a new surrogate of fitmess, based on the convenient incorporation of Policy-extended Value Function Approximator (PeVFA) favored by the linear policy representations. Without loss of generality, we use GA and TD3 (and DDPG) for the concrete choices of EA and RL algorithms. Finally, we evaluate ERL-Re" on MuJoCo continuous control tasks with strong ERL baselines and typical RL algorithms, along with comprehensive study ablation, hyperparameter analysis, etc.
2
be more efficient
We summarize our major contributions below: 1) We propose a novel approach ERL-Re2 to integrate EA and RL based on the concept of two-scale representation; 2) We devise behavior-leyel crossover and mutation which have clear genetic semantics: 3) We empirically show that ERL-Re² outperforms other related methods and achieves state-of-the-arr performance.
2 BACKGROUND
Reinforcement Learning Consider a Markov decision process (MDP), defined by a tuple (S. P, R, y, T). At each step f, the agent uses policy to select an action 0e T(:)) A according to the state s, E S and the environment transits to the next state St+1 according to transition function P(se, a:) and the agent receives reward ra R(ss,, az). The return is defined as the dis- counted cumulative reward, denoted by R2 >" - where y E [0, 1) is the discount factor and T is the maximum episode horizon. The goal of RL is to learn an optimal policy T² that maximizes the expected return. DDPG (Lillicrap et al., 2016) is representative off- policy Actor-Critic algorithm, consisting of a deterministic policy Tue (i.e., the actor) and a state-action value function approximation Q (i.e.. the critic), with the parameters and & respectively. The critic is optimized with the Tempo- ral Difference (TD) (Sutton & Barto, 1998) loss and the actor is updated by maximizing the estimated Q value. The loss functions are defined as: [(()- Ep [r + 72=r ( Tw (s') - Q (s, a)) and C(w) E (s, (s)]] where the experiences (s, a, r, s') are sampled from the replay buffer D, v and w' are the parameters of the target networks. TD3 (Fujimoto et al., 2018) improves DDPG
by addressing overestimation issue mainly by clipped double-Q learning.
2

---

Published as a conference paper at ICLR 2023
Conventional value functions are defined on specific policy. Recently, new extension called Policy-extended Value Function Approximator (PeVFA) (Tang et al., 2022) is proposed to preserve the values of multiple policies. Concretely, given some representation Xx of policy n, a PeVFA parameterized by 9 takes as input Xx additionally, i.e., Q2 a, X=r) Through the explicit policy representation Xx. one appealing characteristic of PeVFA is the value generalization among policies (or policy space). PeVFA is originally proposed to leverage the local value generalization along the policy improvement path to improve online RI. (we refer the reader to their original paper). In our work, we adopt PeVFA for the value estimation of the EA population, which naturally fits the ability of PeVFA well. Additionally, different from the on-policy learning of PeVFA adopted in (Tang et al., 2022), we propose a new off-policy learning algorithm of PeVFA which is described later.
Evolutionary Algorithm Evolutionary Algorithms (EA) (Bäck & Schwefel, 1993) are a class of black-box optimization methods. EA maintains a population of policies IP = {TI, T2 .... Tn J in which policy evolution is iteratively performed. In each iteration, all agents interact with the environment for episodes to obtain Monte Carlo (MC) estimates of policy fitness (f(T)) (x.)).... ( where
[(m) Ei Tt. TT). The policy with higher fitness is more likely to be selected as parents to produce the next generation in many ways such as Genetic Algorithm (GA) (Mitchell, 1998) and Cross-Entropy Method (CEM) (Pourchot & Sigaud, 2019). With GA, offspring are generated by applying genetic operators: the parents T. and F are selected randomly to produce offspring गद and " by performing the crossover operator, i.e., T.. " Crossover (TT, TT) or the mutation operator T Mutat i (..)). In most prior methods, the crossover and mutation operate at the parameter level. Typically, k-point crossover randomly exchange segment-wise (network) parameters of parents while Gaussian mutation adds Gaussian noises to the parameters. Thanks to the diversity brought by abundant candidates and consistent variation, EA has strong exploration ability and convergence.
RELATED WORK
Recently, an emergent research field is integrating the advantages of EA and RL from different angles to devise new methods (Sigaud, 2022), for example, combining EA and RL for efficient policy optimization (Khadka Tumer, 2018; Bodnar et al., 2020), using EA to approximate the greedy action selection in continuous action space (Kalashnikov et al., 2018; Simmons-Edie Shi & Singh. 2021; Shao et al.. 2021: Ma et al., 2022). population based hyperparameter tuning of RL (Jaderberg et al., 2017; Pretorius & Pillay, 2021), and genetic programming for interpretable RL policies (Hein et al., 2018: Hein, 2019). In this work, we focus on combining EAA and RI. for efficient policy optimization. ERL (Khadka& & Tumer, 2018) first proposes a hybrid framework where a DDPG agent is trained alongside genetic population. The RL agent benefits from the diverse experiences collected by the EA population, while the population periodically includes a copy of the RL agent. In parallel, CEM-RL (Pourchot & Sigaud, 2019) integrates CEM and TD3. In particular, the critic function of TD3 is used to provide update gradients for half of the individuals in the CEM population. Later, ERL serves as a popular framework upon which many improvements are made. CERL (Khadka et al., 2019) extends the single RL agent to multiple ones with different hyperparameter settings to make better use of the RL side. GPO (Gangwani & Peng, 2018) devises gradient-basee crossover and mutation by policy distillation and policy gradient algorithms, respectively. Further, PDERL (Bodnar et al., 2020) devises the Q-iiltered distillation crossover and Proximal mutation to alleviate the policy crash caused by conventional genetic operators at the parameter level. All these works adopt no sharing among agents and each agent of EA and RL learns its own state representation, which is inefficient and specialized. By contrast, our approach ERL-Re² makes use of a expressive state representation function which is shared and learned by all agents. Another common point of these works is, evolution variations are imposed at the parameter level (i.e., policy network). Despite the existence of GPO and PDERL, the semanties of genetic operators on policy behavior cannot be guaranteed due to the nonlinearity nature of policy parameters. In ERL-Re?. we propose behavior- level crossover and mutation operators that have clear semantics with linear policy representations.
et al., 2019;
4 REPRESENTAAIIN EVOLUTIONARY REINFOR LEARNING
In this section, we introduce the overview of 2 to gain the holistic understanding of the key concept. In addition, we introduce how ERL-Re2 can be realized by general form of algorithm framework. We defer the concrete implementation details in Sec. 5.
3

---

Published as a conference paper at ICLR 2023
4.1 THE CONCEPT OF TWO-SCALE STATE REPRESENTATION AND POLICY REPRESENTATION
The previous algorithms that integrate EA and RL for pol-
ERL Framework
ERL-Re² Framework
Evolution-based based optimization
Evolution- based optimization
icy optimization primarily fol- low the ERL interaction architec- ture shown on the left of Fig. I, where the population's 's policies offer a variety of experiences for RL training and the RL side in- jects its policy into the popula- tion to participate in the iterative evolution. However, two signif- icant issues exist: 1) each agent maintains an independent nonlin- car policy and searches in param- eter space, which inefficient because each agent has inde- pendently and repeatedly learn common and useful knowledge. 2) Parameter-level perturbations might result in catastrophic fail-
2
R
W1
W
W
EA side
EA side
Conveyy common useful knowledge by
Evaluation
Evaluation
Inject policy
Inject
Z
policy
the shared state representation
Experiences
Experiences
Gradient-based optimization
Gradient- baaeed optimization
TTyt
Wvt
RL side
RL side
Figure I: The left represents ERL framework and the right rep- resents ERL-Re framework. In ERL-Re all the policies are composed the nonlinear shared state representation Za and an individual linear policy representation W. ures, making parameter-level evolution exceedingly unstable.
To address the aforementioned problems, we propose Two-scale Representation- based policy con- struction, based which maintain and optimize the EA population and RL agent. The policy construction is illustrated on the right of Fig. 1. Specifically, the policies for EA and RI. agents are all composed of shared nonlinear state representation = 7z(se) E TRd (given a state 8,) and an indi- vidual linear policy representation W E g(d+1)x|A|, We refer to the different representation scopes (shared/individual + state/policy representation) of the policy construction as the two scales. The agent i makes decisions by combining the shared state representation and the policy representation:
(st) = act(Zz [e[[ + Wi.(++1) E RlA!
where W Wm((a)] denotes the slice of matrix W that consists of row m (to n) and act(-) de- notes some activation function (e.g., tanh) In turn, we also denote the EA population by P = (W1, W2 ... Wn) and the RI. agent by Wrt- Intuitively, we expect the shared state repre- sentation Ze to be useful to all possible policies encountered during the learning process. It ought to contain the general decision-related features of the environment, c.g., common knowledge, while not specific any single policy. By sharing the state representation Z it does not require each agent to learn how to represent the state independently. Thus, higher efficiency and more expressive state representation can be fulfilled through learning in a collective manner with the EA population and RL agent. Since Zg is responsible for expressivity, each individual policy representation can have a straightforward linear form that is easy optimize and evaluate (with PFAF.).
4.2 THE ALGORITHM FRA MEWOKK OF 2
Due to the representation- based policy construction, the state representation function Zg determines policy space denoted by [[)). where we optimize the individual representations of the EA and RL agents. The optimization flow of ERL-Re" is shown in Fig. The top and bottom panels depict the learning dynamics of Zg and the agents, respectively. In each iteration t, the agents in the EA population IP and the RL agent W, evolve or reinforce their representations in the policy space (()))) provided by Zo, (Sec. 5.2). After the optimization at the scale of individual representation, the shared state representation is optimized (i.e., + 0:+1) 1 towards unifying direction, derived from value function maximization regarding all the EA and RL agents (Sec. 5. By this means, the shared state representation is optimized in the direction of a superior policy space for successive policy optimization. an iterative manner, the shared state representation and the individual policy representations play distinct roles and complement each other in optimizing the EA and RL agents.
In principle, ERL-Re² is general framework that can be implemented with different EA and RL algorithms. For the side of EA, we mainly consider Genetic Algorithm (GA) (Mitchell, 1998)
We use deterministic policy for demonstration while the construction is compatible with stochastic policy.
4

---

Published as a conference paper at ICLR 2023
P111
Optimize
Optimize
Pw
Ow
4Pce2
PWz
Optimized Agents
Optimized Agents
Policy Space (((h)) Derive by
Policy Space ('dbus) Derived tc-1
Superior policy region
W
W2 W
Individual optimization
Integrated optimization
EA agent O RL agent
Optimized
representation
Wa
O Optimized state rupresentation regarding W,
Figure 2: The optimization flow of ERL-Re². In an iterative fashion, the shared state representation and individual linear policy representations are optimized two scales.
as a representative choice while another popular choice CEM (Pourchot & Sigaud, 2019) is also discussed in Appendix F. Our linear policy representation can realized the genetic operations with clear semantics. For the side of RL, use DDPG (Lillicrap al., 2016) and TD3 (Fujimoto et al., 2018). For efficient knowledge sharing and policy optimization, we learn a shared state representation by all the agents, then evolve and reinforce the policies in the policy representation space rather than in the original policy parameter space.
A general pseudo-code of ERR-RRe is shown in Algorithm 1. In each iteration, the algorithm proceeds across three phases (denoted by blue). First, each agent of EA and RL interacts with the environment and collects the experiences. Specifically, the agents in the EA population TP have the probability 1 p rollout partially and estimate the surrogate fitness with higher sample efficiency (Line 5-6), in contrast to the conventional way (Line 7-8). Next, evolution and reinforcement update /optimize their policy representations in the linear policy space (Line 13-14), The agents are optimized by EA and RL, where RL agent learns with additional off-policy experiences collected by the agents in (Line 14) and periodically injects its policy to P (Line 15). Finally, the shared state representation is updated provide superior policy space for the following iteration (Line 17).
EVOLUTION AND REINFORCEMENT IN REPRESENTATION SPACE
In this section, we present the algorithm details of how to optimize the two-scale representation. In addition, we introduce a new surrogate policy fitness to further improve the sample efficiency.
5.1 OPTIMIZING THE SHARED STATE REPRESENTATION FOR A SUPERIOR POLICY SPACE
The shared state representation Z designates the policy space IT., in which EA and RI. are conducted. As discussed the previous section, the shared state representation takes the responsibility of learning useful features of the environment from the overall policy learning. We argue that this is superior the prior way, i.e., each agent learns its individual state representation which can be less efficient and limited in expressivity. In this paper, we propose to learn the shared state representation by the first principle of value function maximization regarding all EA and RL agents. To be specific, detail the value function approximation for both EA and RL agents. For the EA agents, we learn a PeVFA (s a, W;) based the linear policy representation W, in EA population IP; for the RL agent, we learn a critic Q (s, a). In principle, RL can use PeVFA as its critic. We experimentally show that both approaches can achieve similar performance in Appendix.D.. D.4. The loss functions of Qo and Q are formulated below:
(9) = ((s.a.r.") (r+ 12 (s), W.) Qo (s, W.))
(I)
(1) = E(sans)) 7 (((()) - Qu (s, (s).)))]
where D is the experience buffer collected by both the EA and RL agents, 6', v denote the target networks of the PeVFA and the RI critic, n/ denote the target actor with policy representation W' and recall (,(s) act act (s) W.,n + (a((+1) E El Note that we make Q9 and Qy take the raw state input rather than (s). Since sharing state representation between actor and critic
5

---

Published as a conference paper at ICLR 2023
Algorithm 1: ERL with Two-scale State Representation and Policy Representation (ERL-Rea)
Input:
the EA population size n. the probability p of using MC estimate, the partial rollout length H Initialize: a replay buffer D. the shared state representation function Ze. the RL agent Wrt. the EA
population P = (W1, ,W,,, the critic Qu and the PeVFA Qo (target networks are omitted here)
repeat
# Rollout the EA and RL agents with Zo and estimate the (surrogate) fitness Random Number > p then
Rollout each agent in for H steps and evaluate its fitness by the surrogate (W) see Eq. 3
else
Rollout each agent in for one episode and evaluate its fitness by MC estimate
Rollout the RI. agent for one episode
10
Store the experiences generated by P and Wrz to D # Individual scale: evolution and reinforeement in the policy space Train PeVFA Qs and RL critic Qu with D
sCC Eq. 1
13
Optimize the EA population: perform the genetic operators (i.e., selection, crossover and mutation) the behavior level for
(W1.-.., W..)
see Eq. 4
14
Optimize the RL agent: update W,, (by e.g.. DDPG and TD3) according to Q
see Eq. 5
15
Inject RL agent to the population P' periodically
16
# Common scale: improving the policy space through optimizing Z
17
Update the shared state representation: optimize Ze with a unifying gradient direction derived from see Eq. 2
value function maximization regarding Qy and Q+
18
until reaching maximum steps;
may induce interference and degenerated performance as designated by recent studies (Cobbe et al., 2021; Raileanu & Fergus, 2021). Another thing train PEA in an off-policy fashion (Eq. 1). We discuss more on this in Appendix F.
to notice is, to our knowledge, we are the first to
For each agent of EA and RL, an individual update direction of the shared state representation Zø is now ready to obtain by Qo (s), W;) for any W; E P or (s (T)(s) through n. and ni respectively. This the value function maximization principle where we adjust Z, to induce superior policy (space) for the corresponding agent. To be expressive, Zo should not take either individual update direction solely; instead, the natural way to take a unifying update direction regarding all the agents (i.e., the currently targeted policies). Finally, the loss function of 2o is defined:
Cz( EE (W, 15
DW,,,,, ,P [Qv (s, (s)) + Σ Qo (s. T. (s), W
(2)
where K is the size of the sampled subset of EA agents that engage in updating Zg. By minimizing Eq. 2, the shared state representation Z is optimized towards a superior policy space II pertaining to all the EA and RL agents iteratively, as the learning dynamics depicted in Fig. 2. Note that the value maximization is not the only possible choice. For a step further, we also investigate on the incorporation of self-supervised state representation learning in Appendix D.
5.2 OPTIMIZING THE POLICY REPRESENTATION BY EVOLUTION AND REINFOR
Given the shared state representation Zg. all the agents of EA and RL optimize their individual policy representation in the policy space (o))). The fundamental distinction here is, that the evolution and reinforcement occur in the linear policy representation space rather than in nonlinear policy network parameter space as the convention. Thus, policy optimization can be efficiently conducted. Moreover, the linear policy representation has a special characteristic: it allows performing genetic operations the behavior level. We detail the policy representation optimization of EA and RL below.
The evolution of the EA population P mainly consists of: 1) interaction and selection, 2) genetic evolution. For the process of interaction and selection, most prior methods rollout each agent in P for one or several episodes and calculate the MC fitness. The incurred sample cost is regarded as one major bottleneck of EA especially when the population is large. To this end, we propose new surrogate fitness based on the PeVFA Q9 (Eq, 1) and partial rollout. At the beginning of each iteration, we have a probability p rollout the EA population for и steps. For each agent W,, the surrogate fitness is estimated by H H-step bootstrapping (Sutton & Barto, 1998):
1-1
f(W.) = Σ
r' + + Qo(SH, (o(s W.).
(3)

---

Published as a conference paper at ICLR 2023
Thanks to the PeVFA Qo. the surrogate fitness can be conveniently estimated and reduce the sample cost effectively. Moreover, one key point is that f(W:) is free of the off-policy bias in the surrogate proposed in the concurrent work (Wang et al., 2022). To be concrete, the proposed surrogate fitness for EA agents in (Wang et al., 2022) is estimated by bootstrapping based the RL critic and uniformly samples from the replay buffer. Apparently, the off off-policy bias in their surrogate comes from the mismatch of value functions between specific EA agents and the RL agent, and the mismatch between initial state distribution and the replay buffer. By contrast, our surrogate fitness obviates such mismatches by bootstrapping from PeVFA and using partial rollout. We experimentally prove that f(W) is more efficient. Therefore, according to the fitness estimated by MC or our surrogate, parents with high fitness are more likely to be selected as parents. The details of the selection process follow PDERL (Bodnar et al., 2020).
For the process of genetic evolution, two genetic operators are involved: crossover and mutation. Typical k-point crossover and Gaussian mutation that directly operate at the parameter level can easily lead to large fluctuations in the decision and even cause crash to policy (Gangwani & Peng, 2018; Bodnar et al., 2020). The reason behind this is that the policy parameter is highly nonlinear to behavior, thus the natural semantics could be hardly guaranteed by such parameter-level operators. In contrast, recall the linear policy representation W in matrix form. Each row Wi determines the i-th dimension of action and thus the change to Wis does not affect the behavior of other dimensions. Based on the granularity of clearer semantics, and mutation, as the illustration in Fig.3. For the behavior- level crossover (t-Crossovez)) the offspring are produced by inheriting the behaviors of the parents at the corresponding randomly selected dimensions. For the behavior-level mutation (b-Mutati i on), each dimension of behavior has the probability a to be perturbed. Formally, we formulate the two operations below:
propose novel behavior-level crossover
(War: Wea) = (Wp, 0d, Wp2 Wp &dg Wp,) b-Crossover Wpa). W mm W,, 0d, Pi b- M-Mutation(W,))
where d1, d2, d1 are the randomly sampled sub- sets of all dimension indices, P1 is the ran- domly generated perturbation matrix, and we use A a B to denote A being replaced by B at the corresponding rows in d. For the dimension selection and perturbation generation, we fol- low the same approach used in (Bodnar et al., 2020) (detailed Appendix C) and we analyze the hyperparameter choices specific to our ap- proach in Sec. 6.3. By imposing variations on specific behavior dimensions while incurring no interference on the others, our proposed op- erators (i.e., Crossover and (-ou) can be more effective and stable, as well as more informative in the sense of genetic semantics.
Wc1
Wm
Crossover
Mutation
P2
Figure 3: Behavior-level crossover and mutation op- erators in ERL-Re2 The offspring and parents indexed by 01, C22 m1, m2 and pi, P2 respectively.
As to the learning of the RL agent Wri, resembles the conventional circumstance except done with respect to the linear policy representation. Taking DDPG (Lillicrap et al., 2016) for typical example, the loss function of Wrr is defined below, based on the RI. critic Q2. (learned by Eq. 1):
LR. -EEWD Qu (s. n., (s))
(5)
The RL agent learns from the off-policy experience in the buffer 1) collected also by the EA population. Meanwhile, the EA population incorporates the RL policy representation W,, at the end of each iteration. By such interaction, EA and RL complement each other consistently and the respective advantages of gradient- and gradient- policy optimization are merged effectively.
EXPERIMENTS
6.1 EXPERIMENTAL SETUPS
We evaluate 2 on six MuJoCo (Todorov et al., 2012) continuous control tasks commonly used in the literature: HalfCheetach. Swinmer, Hopper, Ant, Walker. Humanoid (all in version 2). For a comprehensive evaluation, we implement two instances of ERL-Re based on DDPG (Lillicrap al., 2016) and TD3 (Fujimoto et al., 2018), respectively. We compare ERL-Re2 with the following
7

---

Published as a conference paper at ICLR 2023
Safmmer
exce
vice
100
SRUTCH
undscou
PFC
1D3
200
1300
Thme
Humanoo d
Happer
13304
Ha IChheth
Un3icauutts Ricum
Undiccanttd frdiumm
23
1x
2000
Tiee Stige Balts
Thm Shap Cal
Figure 4: Performance comparison between ERL-Re" and baselines (all in the TD3 version).
354
Salmmer
6X1
Undicounted lecum
Via
1 X
CCM clps
5X0
R
Banc
3004
Time
Humaneid
Hoppor
HallChoctach
11:00
Undipouutes Rebuu
Undizouunte fotumn
1000
43:00
13300
12000
J000
Time Stage Dolk
Thme Stap Cal)
Figure Performance comparison between ERL-Re and baselines (all in the DDPG version).
baselines: 1) basic baselines, i.e., PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), DDPG, TD3 and EA (Mitchell, 1998); 2) ERL-related baselines, including ERL (Khadka & Tumer, 201 8), CERL (Khadka et al., 2019), PDERL (Bodnar et al., 2020), CEM-RL (Pourchot & Sigaud, 2019). The ERL-related baselines are originally built on different RL algorithms (either DDPG or TD3), thus we modify them to obtain both the two versions for each of them. We use the official implementation and stable-baseling for the mentioned baselines in our experiments. For a fair comparison, we compare the methods built on the same RL algorithm (i.e., DDPG and TD3) and fine-tune them in each task to provide the best performance. All statistics are obtained from 5 independent runs. This is consistent with the setting in ERL and PDERL. We report the average with 95% confidence regions. For the population size n, we consider the common choices and use the best one in (5, 10} for each concerned method. We implement our method ERI-Re based on the codebase of PDERL and the common hyperparameters remain the same. For the hyperparameters specific to ERI-Re2 (both DDPG and TD3 version), we set a to 1.0 and select H from {50, 200). K from {1,3), from {0.3, 0.5, 0.7, 0.8) for different tasks. All implementation details are provided in Appendix E.
6.2 PERFORMANCE EVALUATION
We evaluate ERL-Re and baselines in TD3 and DDPG versions separately. The results in Fig.4 and Fig.5 show that both ERL-Re2 (TD3) and ERL-Re (DDPG) significantly outperform other methods in most tasks. It is worth noting that, to our knowledge, ERI-Re is the first algorithm that outperforms EA in Swimmer. We can see that both ERL-Re² (TD3) and ERL-Re² (DDPG) achieve a 5x improvement in convergence rate compared to EA and obtain higher and more stable performance. Moreover, in some more difficult environments such Humanoid which is dominated by RL algorithms, ERL-Re2 also achieves significant improvements. Overall, ERL-Re2 is an effective and general framework that significantly improves both EA and RL in all the six tasks while other methods fails. Beyond MuJoCo, we also demonstrate the performance improvement achieved by ERL-Re² in several visual control tasks of DMC (Tassa et al., 2018) in Appendix D.5.
6.3 SUPERIORITY OF COMPONENTS & PARAMETER ANALYSIS
We conduct experiments to compare our proposed genetic operators and surrogate fitness with other related methods. In Fig.6a, we compare the behavior-level operators with other alternatives. The
8

---

Published as a conference paper at ICLR 2023
Setmmer
Vi'e
Undbcouueed Retum 11c
3000
3000
5000
-090
1000
undiccuuttin
3000
cang beance calaaiinng throko
e-ong saain
P02RA sperton
Oun sperston
1090
3090
syter (hemtk
butte (Diti
diccuuited reward
Time imprrino
(a) Different genetic operators
(b) Different choices of surrogale fitness
manrr
6.05
Andoccunted Secuum
Uhdcount
D Oftc PraPA
A000
324
4
v00
3000
2000
ETL MRy
HI RefoRI
2IC4
103
1000
120
Stom
1204
Time Smgaiae
Teme Srow
(c) Ablation study Ze (terms in Eq. 2) & Different K
(d) Ablation study on surrogate fitness
Figure 6: Comparative evaluation of the components in ERL-Re and ablation study.
Undigqurt. Rutum
5300
5000
UndESourrc
3000
2000
p-0.3
L000
p-0.8
The St00
Tren Lin
Thee (1AO
Thme Stape Ca0
(a) Probability a of each action mutation
(b) Probability p to our surrogate fitness
Figure 7: Analysis of main hyperparameters.
results show that the behavior-level operators are more effective than other operators. Next, we examine whether the surrogate fitness estimate fWW) (i.e., H-step bootstrapping (PeVFA)) is better than using the RL critic to evaluate fitness based on the latest state from replay buffer (Wang et al., 2022) (i.e.. buffer (Critic)). The results in Fig.6b show that using PeVFA is more effective than using the RL critic. This is because the RL critic cannot elude the off off-policy bias while Pe thanks to the generalized approximation. Second, we provide ablation study on how to update Z and analyze the effect of different K. We evaluate ERL-Re2 (TD3) with different K E [1,3] and ERL-Re (TD3) with only PeVFA critic to optimize the shared state representation. The results in Fig.6c demonstrate the conclusion of two aspects: 1) K affects the results and appropriate tuning are beneficial; 2) both the RL critic and PeVFA play important roles. Only using the RL critic or PeVFA leads to inferior shared state representation, crippling the overall performance. Third, we perform ablation experiments on (WW).. The results in Fig. 6d show that using f(W,) can deliver performance gains and achieve further improvement to ERL-Re² especially in Swimmer.
free of
For hyperparameter analysis, the results in Fig.7a show that larger a is more effective and stable, which is mainly since that larger can lead to stronger exploration. Thus we use a = 1.0 on all tasks. Finally, the results in Fig. 7b demonstrate that p has an impact performance and appropriate tuning for different tasks is necessary. It is worth noting that setting p to 1.0 also gives competitive performances, i.e., do not use the surrogate fitness f(W)) (see blue in Fig. 6d).
Others Due to space limitations, more experiments the hyperparameter analysis of H, 8, differ- ent combinations of genetic operators, incorporating self-supervised state representation learning, the interplay between the EA population and the RI. agent and etc. are placed in Appendix D.
CONCLUSION
We propose a novel approach called ERL-Re2 to fuse the distinct advantages of EA and RL for efficient policy optimization. In ERL-Re", all agents are composed of the shared state representation and an individual policy representation. The shared state representation is endowed with expressive and useful features of the environment by value function maximization regarding all the agents. For the optimization of individual policy representation, we propose behavior-level genetic operators and a new surrogate of policy fitness to improve effectiivness and efficiency. In our experiments, we have demonstrated the significant superiority of ERL-Re compared with various baselines.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation China (Grant 10.62106172), the "New Generation of Artificial Intelligence" Major Project of Science Technology 2030 (Grant No.2022ZD01 I 6402),, and the Science and Technology Information Systems Engineering Laboratory (Grant No. WDZC20235250409 No. WDZZ2200055040077
REFERENCES
Bäck and H. Schwefel. An overview of evolutionary algorithms for parameter optimization. Evol. Comput., 1(1):1-23, 1993.
Belghazi, Rajeswar, A. Baratin, R. D. Hjelm, and A. C. Courville. Mutual information neural estimation. In International conference on machine learning, pp. 531-540. PMLR, 2018.
C. Bodnar, B. Day, and P. Lió. Proximal distilled evolutionary reinforcement learning. In AAAI, pp. 3283-3290, 2020.
W. Chung, S. Nath, A. Joseph, and M. White. Two-timescale networks for nonlinear value function approximation. In ICLR, 2019.
K. Cobbe, J. Hilton, O. Klimov, and J. Schulman. Phasic policy gradient. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 2020-2027, 2021.
W. Dabney, A. Barreto, M. Rowland, R. Dadashi, J. Quan, M. G. Bellemare, and D. Silver. The value-impres inlooeemproveeentt path: Towards better representations for reinforcement learning. In AAAI, pp. 7160-7168, 2021.
F. Faccio, L. Kirsch, and J. Schmidhuber. Parameter-based value functions. In ICLR, 2021.
M. C. Fontaine and S. Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, 2021.
Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591, 2018.
T. Gangwani and J. Peng. Policy optimization by genetic distillation. In ICLR, 2018.
D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31; Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, 2018
Haarooja, Zhou, Abbeel, and Levine. Soft actor-eritic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, 2018.
Xiaotian Hao, Weixun Wang, Jianye Hao, and Yaodong Yang. Independent generative adversarial self-imitation learning in cooperative multiagent systems. arXiv preprint arXiv:/909. 1468, 2019.
Xiaotian Hao, Zhaoqing Peng, Yi Ma. Guan Wang, Junqi Jin, Jianye Hao, Shan Chen, Rongquan Bai, Mingzhou Xie, Miao Xu, al. Dynamic knapsack optimization towards efficient multi-channel sequential advertising. In International Conference on Machine Learning, pp. 4070- PMIR, 2020.
Xiaotian Hao, Weixun Wang, Hangyu Mao, Yaodong Yang, Dong Li, Yan Zheng, Zhen Wang, and Jianye Hao. Api: Boosting multi-agent reinforcement learning via pennnn-eeemaaiiinnnniiiia networks. arXiv preprint 22X.v:2203.05285, 2022.
D. Hein. Interpretable Reinforcement Learning Policies by Evolutionary Computation. PhD thesis, 2019.
10

---

Published as a conference paper at ICLR 2023
D. Hein, S. Udluft, and T. A. Runkler. Interpretable policies for reinforcement learning by genetic programming. Eng. Appl. Artif. Intell., 2018.
M. Jaderberg, V. Dalibard, S. Osindero. W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population based training of neural networks. arXiv preprint arXiv:111.0994/ 2017.
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In CoRL, volume 87 of Proceedings of Machine Learning Research, pp. 651-673, 2018.
S. Khadka and K. Tumer. Evolution-guie -gideed policy gradient in reinforcement learning. In NeurIPS, pp. 1196-1208, 2018.
S. Khadka, S. Majumdar, T. Nassar, Z. Dwiel, E. Tumer, S. Miret, Y. Liu, and K. Tumer. Collaborative evolutionary reinforcement learning. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 3341-3350, 2019.
A. Kumar, Agarwal, T. Ma, A. C. Courville, G. Tucker, and S. Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint arXiv:2112.04716, 2021.
M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning. In ICLR, 2016.
Y. Ma, T. Liu, B. Wei, Y. Liu, K. Xu, and W. Li. Evolutionary action selection for madieen--based policy learning, arXiv preprint arXiv:2201.0422 2022.
S. Majumdar, S. Khadka, S. Miret, S. McAleer, and K. Tumer. Evolutionary reinforcement learning for sample-efficient multiagent coordination. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. PMLR, 2020.
M. Mitchell. An introduction to genetic algorithmss. MIT Press, 1998.
V. Mnih, K. Kavukeuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. A. Ricdmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5660 2013.
F. Ni, J. Hao, J. Lu, X. Tong, M. Yuan, J. Duan, Y. Ma, and K. He. A multi-graph attributed reinforcement learning based optimization algorithm for large-scale hybrid flow shop scheduling problem. In KDD, pp. 13451, 2021.
J. Marker-Holder, A. Pacchiano, K. M. Choromanski. and S. J. Roberts. Effective diversity in population based reinforeement learning. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
B. Peng. T. Rashid, C. de Witt, Kamienny, P. H. S. Torr, W. Bochmer, and Whiteson. FACMAC: factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 12208-12221, 2021.
A. Pourchot and O. Sigaud. CEM-RL: combining evolutionary and gradient-based methods for policy search. In ICLR, 2019.
K. W. Pretorius and N. Pillay. Population based reinforcement learning. In SSCI. pp. 1-8, 2021.
R. Raileanu and R. Fergus. Decoupling value and policy for generalization in reinforcement learning.
In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8787-8798, 2021.
11

---

Published as a conference paper at ICLR 2023
M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C. Hung, P. H. S. Torr, J. N. Foerster, and S. Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019, 2019.
Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.0063777447 2017.
L. Shao, Y. You, M. Yan, S. Yuan, Q Sun, and J. Bohg. GRAC: self-guided and self-regularized actor-critic. In CoRL, volume 164 Proceedings of Machine Learning Research, pp. 267-276, 2021.
Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, and Yang Liu. Generating Sehavior-diverss game ais with evolutionary multi-objective deep reinforcement learn- ing. In Christian Bessiere (ed.). Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3371 13771-377. ijcai.org, 2020.
Z. Shi and S. P. N. Singh. Soft actor-critic with cross-entropy policy optimization. arXiv preprint arXiv:2112.11115. 2021.
O. Sigaud. Combining evolution and deep reinforcement learning for policy search: survey. arXiv preprint arXiv:2203. 2022.
R. Caamons--Edeer B. Eisner, E. Mitchell, H. S. Seung, and D. D. Lee. Q-learning for continuous 2019.
actions with cross-entropy guided policies. arXiv preprint
P. Such, Madhavan, Conti. J. Lehman, K. O. Stanley, and Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning, arXiv preprint urXiv:1712.06567 2017.
R. Sutton and A. G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press, 1998.
H. Tang, Z. Meng. J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Yang, W. Tao, and L. Wang. What about inputting policy in value function: Policy representation and policy-extended value function approximator. In AAAI, pp. 8441-8449, 2022.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018.
E. Todorov, Erez, and Y. Tassa, Mujoco: A physics engine for model-based control. In IROS, pp- 5026-5033, 2012.
Y. Wang. K. Xue, and C. Qian. Evolutionary diversity optimization with clustering bussed for reinforcement learning. In International Conference on Learning Representationss 2021.
based selection
Y. Wang, T. Zhang, Y. Chang, B. Liang, X. Wang, and B. Yuan. A surrogate-assisted controller for expensive evolutionary reinforcement learning. Information Sciences, 2022.
Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Jinyi Liu, Yingfeng Chen, and Changjie Fan. Euclid: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model. arXiv preprint 2rXiv:2200.004220 2022.
12

---

Published as a conference paper at ICLR 2023
A LIMITATIONS & FUTURE WORK
For limitations and future work, firstly our work is empirical proof of the effectiveness of the ERL- Re² idea and we provide no theory optimality, convergence, and complexity. Secondly, one critical point is the expressivity of the shared state representation, which not optimally addressed by this work. Self-supervised state representation learning and nonlinear policy representation with behavior semanties be potential directions. Thirdly, diversity is key aspect of EA while in ERL-Re² we currently do not consider an explicit mechanism or objective to optimize the diversity of EA population (and also RL policy). One important future work is integrate Quality-Diversity principle Parker-Holders et al., 2020; Fontaine Nikolaidis, 2021; Wang et al., 2021) for further development of ERL-Re". Fourthly, most current ERL, methods including our ERL-Re² are model-free. Obviously, the model-free paradigm limits the advance sample efficiency. One promising direction can be ERL with a world model, as pioneer work (Ha & Schmidhuber, 2018) has demonstrated the feasibility with CMA-ES. For the final one, follow the convention in the literature and evaluate ERL-Re² on MuJoCo the main text and additional tasks DMC (Tassa et al.. 2018) with image inputs in Appendix D.5; while performance in other environments, e.g.. Atari (Mnih et al., 2013), not thoroughly studied.
B COMPARISON BETWEEN ERL AND ERL-RE"
In this section, we compare ERL (Khadka & Tumer, 2018) and ERL-Re² in detail. The ERL framework currently one popular framework to integrate EA and RL for policy optimization, We illustrate the difference between the two frameworks intuitively in Fig.8. On the foundation of the interaction schema between EA and RL in ERL, ERL-Re" makes all policies composed of the shared state representation and individual policy representations. Based on the shared state representation, useful common knowledge be delivered efficiently among agents. Mcanwhile, the linear policy representations allow for imposing the novel behavior-leve crossover and mutation operations, which are more stable and effective.
We detail how to update the shared state representation by the principle of value function maximization. Here we illustrate the process in Fig. 9 to help the readers understand it more intuitively. After interacting with the environment for steps, optimize the shared state representation times. In each update, we maximize (s, a) the RL agent on the one hand. On the other hand, we sample K policies representation optimized toward unifying direction according to Eq. 2.
from the population to maximize Qss, a, W.). Finally, the shared state
EA based on Policy Representation
EA based on actor
Evaluation
Evaluation
Inject policy EA population
Inject RL policy
Shared State Representation
Env
Env
EA population
Experiences
Experiences
RL based on Policy Representation
RL based on actor
Figure 8: Comparison between ERL (Khadka & Tumer, 2018) and ERL-Re² The left shows the interaction between EA and RL ERL, and the right shows the interaction between EA and RL in ERL-Re2.
C DETAILS OF GENETIC ALGORITHM IN ERL-RE²
In this section, we provide the genetic algorithm flow and the corresponding pseudo-code in Al- gorithm 2. Since ERI-Re built on the official code PDERL. (Bodnar et al., 2020), thus all hyperparameters and genetic algorithm flow remain the same.
13

---

Published as a conference paper at ICLR 2023
Shared
Wyl
RL
Shared State Representation Network
5₁
2t
Critic
gradients
Population
Policy Representation
Cl2
PeVFA
Q ag wi)
W1
sample
W/
W;
Figure 9: The illustration on how update shared state representations. each update, we sample K policy representations from the population. Based on this, ERL-Re" updates the shared state representation network with PeVFA and critic according to Eq. 2.
Specifically, we first get the fitness all policies the population and divide all policies into three groups: elites, winners, and discarders. We choose the best policies the elites E (In PDERI., is different across tasks. We choose for tasks ERL-Re"), Then we randomly select policies from the non-elites policies and choose the best one as the winner. Then we repeat the process for times and get the set of these policies Wi without duplicates. The policies that as the elites and winners are denoted as discarders Dis. After sorting the categories, we perform the crossover operator. Specifically, sampled policies from and Win respectively, For each action dimension a,, the weights and biases corresponding to the 0,-th action of W will be replaced by those of W p2 with probability 50%. Conversely, the weights and biases of W pa will be replaced by those W After the exchange, we get two offssring and replace the policies in Dis with these offspring. We repeat the above crossover process until all the policies in Dis are replaced. Then we perform mutation on all policies except the elites with a probability 90%. For the selected agent, we perturb the weights corresponding to each action with probability percentage (i.e.. B) parameters. Specifically, we perform a small Gaussian perturbation, a large Gaussian perturbation, reset these parameters with probability 90%, 5%, and 5%, respectively. By completing the above operation, one iteration of population evolution is completed. Except for a and 3 we introduced, we use the same settings of the hyperparameters as in PDERL.
selected
obtain the parents Wp. and W pu by cloning the randomly
For each selected action, the same kind of perturbation is added to certain
D COMPLETE EXPERIMENTAL DETAILS AND FULL RESULTS
This session provides more experiments help understand ERL-Re more comprehensively. We report the average with 95% confidence region based on 5 different seeds. A navigation summary of additional experiments is provided as follows:
Exp Ablation study behavior-level crossover and mutation operators.
Exp Evaluation of different choices of H used in the calculation of our surrogate fitness in Eq.
repeated episodes used the calculation of our surrogate fitness in Eq. Exp Evaluation of different choices of the mutation ratio 3 Algorithm 2.
Exp Analysis
Exp Evaluation of different choices of population size.
Exp 6 Experiments on using the idea of ERL-Re² to improve EA, i.e., Improve EA with the shared state representation and individual policy representations.
Exp A comparative experiment participation rates of the elites and discarders.
Exp Experiments the change of the vverag/maximmm/miiimnn return of the population before and after optimizing the shared state representation.
Exp Further exploration on unsupervised learning to improve shared state representation.
Exp 10 Further exploration on the Value Improvement Path (VIP) (Dabney et al., 2021) to improve shared state representation.
14

---

Published as a conference paper at ICLR 2023
Algorithm 2: Genetic Algorithm in E
Input: the EA population size the probability and 8 Initialize: the shared state representation function Zz, the EA population IP (Wi,... ..., W... repeat
If Obtain Fitness
Rollout the EA policies with Ze and get the (surrogate) fitness
# Perform selection operators
Rank the populations based fitness and select the best several (1 in ERL-Re") policies the elites Randomly seleet 3 policies other than the elites and retain the best policies as the winner, and repeat the process collect
of winners as the winners Win
Policies are not selected as the elites winners are collected as discarders Dis. # Perform crossover operators b- D-Crossover (W Wwgg)
10
11
while 121 a is not empty do select two discarders Wa,Wa, from Dis random select W; from E, and clone as one parent W random select W3 from win clone Wj as the other parent Wp. for index a; in action dimension do if random number < 0.5 then Wp Wp, P1 Ca Wp2
13
14
15
else
Pp p2 Ca, Wp,
use WPp P1 and Wp to replace Wa, and Way and remove Wa, and Wa, from Dis 20
# Perform mutation operators b- MMtaatinn(W, for in all non-elic agents if random number <0.9 then for index a; in action dimension do if random number < a then Add minor (90%), drastic (5%) Gaussian perturbations, reset parameters (5%) to randomly selected 3 parameters from W
22
23
24
25
28
29
until reaching maximun training steps;
Exp 11
Evaluation of different architectures, i.e., maintaining PeVFA and RL Critic for EA and RL separately or only a PeVEA for both EA and RL.
Exp 12
Time-consuming analysis of ERL-Re² (TD3) and other related algorithms.
Exp 13
Analysis of the number of parameters of ERL-Re (TD3) and other related algorithms.
Exp 14 Experiments the influence of other factors i.e., layer normalization, discount reward, and actor structure.
Exp 15
More experiments on Deepmind Control Suite (Tassa et al., 2018) with pixel-level inputs.
Exp 16
More experiments Starcraft II (Samvelyan al., 2019).
Exp 17
The experiments MuJoCo with longer timesteps.
Exp 18 Combine ERL-Re2 with SAC and evaluate on MuloCo.
D.1 ADDITIONAL ABLATION & HYPERPARAMETER ANALYSIS
In this subsection, we provide experiments on ablation and hyperparameter analysis Exp 1, Exp 2, Exp 3, Exp 4, Exp 5, Exp 6.
Exp demonstrate the respective roles of operators, we perform ablation experiments on behavior- crossover and mutation operators. ERL-Re² (TD3) w/o crossover(mutationn means using parameter-level crossover(mutation)) ERL replace the behavior-level crossover(mutation)) The results in Fig. 10 show that both behavior-level mutation and crossover operators can deliver performance gains and the best performance be obtained by using combination of both.
15

---

Published as a conference paper at ICLR 2023
Walker
Swimmer
Undiscounted Return
350
Undiscounted Return
5000
4000
250
3000
150
2000
ERL-Re' (TD3)
(TD3) w/o crossover
1000
50
ERL-Re' w/o mutation
0
26
Steps (1e6)
145
Time Steps (1e6)
Figure 10: Ablation study on behavior-level crossover and mutation operators in ERL-Re² (TD3). The results show that both the mutation and crossover can bring performance gains and the best performance can be obtained by using a combination of both.
Swimmer
Walker
350
Undiscounted Return
Undiscounted Return
5000
250
3000
150
2000
H=200
H=50
50
H=500
Steps
Steps (1e6)
166
Figure 11: Parameter analysis on ERL-Re (TD3) with different H. The results show that ERL-Re (TD3) is not sensitive H but proper adjustment can get better results.
Exp 2: To analyze the effect of hyperparameter H on performance, we evaluate ERL-Re² (TD3) with different H on Swimmer and Walker. The results in Fig. 11 show that ERL-Re" is not sensitive H, but appropriate adjustments can deliver some performance gains.
Exp 3: the surrogate function f(W;) in Eq. provide experiments to perform the analysis. The results in Fig. 12 show that one episode is often enough to achieve good performance. To reduce hyperparameters and simplify the tuning process, we set the episode to all tasks.
do not consider the repeated episodes. Here
Exp 4: To analyze the effect of hyperparameter 3 performance, we evaluate ERL-Re (TD3) with different on Swimmer and Walker. The results Fig. 13 show ERL-Re" is not very sensitive to 8. The 8 determines the degree of mutation for an action. A large 3 means strong exploration. stable environments such as Swimmer and HalfCheetach,
experience, in larger 3 is generally chosen, while in some unstable environments such as Walker and Hopper, a smaller 3 is chosen.
According
Exp 5: Although the population size remains consistent 5 across all tasks our experiments, we give some additional experiments to explore the effect of population size on performance. The results in Fig. 14 show that 5 more effective than 10 ERL-Re (TD3). The main reason behind this may come from the fact that when the population is too large, the number of steps that the RL agent interacts with the environment is drastically reduced, which may lead decrease in RL performance. This problem can be solved by considering a more efficient mechanism for evaluating the individuals reduce the total interaction of EA properly to improve the number of steps of RL interactions. For a fair comparison, we also evaluate other algorithms based on different population size 5 and 10 and report the best performance.
Exp 6: We provide an ablation study on ERL-Re² without the RL side, which can verify whether the idea of ERL-Re2 can improve ÉA.. Specifically, we construct the policies with the share state representation and individual policy representations in EA and maintain PeVFA share the useful
16

---

Published as a conference paper at ICLR 2023
Ant
Walker
Undiscounted Return
6000
Undiscounted Return
5000
5000
4000
4000
200
2000
2000
1x
10 x 20
1000
1000
Steps
Steps
Figure 12:
Experiments on the repeated episodes with different H as the surrogate the fitness. 10 20 means that repeat 10 episodes with 20 steps (i.e., H 20) and get the average the fitness. In the results, 1 x is effective in the tasks. To reduce hyperparameters and simplify the tuning process, we use 1 episode in all tasks.
Swimmer
Hopper
Undiscounted Return
Undiscounted Return
3500
300
3000
250
200
2000
150
8=0.2
1500
0=
0-0.5
1000
0=0.7
0=0,7
50
1.00
500
B=10
Time Steps (1e6)
Time Steps (1e6)
1n6
Figure 13: Parameter analysis of the mutation ratio in ERL-Re (TD3). The results show that ERL-Re² (TD3) is not sensitive to but proper adjustment can get better results..
common knowledge in the population. The results in Fig. show that the idea of ERL-Re" can significantly improve EA, which demonstrates the validity of the idea of ERL-Re".
D.2 FURTHER UNDERSTANDING ERL-REE
In this subsection, we provide experiments to help readers better understand the advantages of ERL-Re in Exp 7 and Exp 8.
Exp 7: To further investigate the reason why ERL-Re2 is more efficient than ERL and PDERL, provide comparative experiment elites and discarded rates of RL agents. The results in Fig. 16 show that RI agent in ERL-Re2 (TD3) is selected as elites with a larger probability and as discarders with a smaller probability than RL agent in ERL(TD3) and PDERL which indicates that both EA and RL in E (TD3) have more competitive effects on population evolution than in ERL(TD3) and PDERL(TD3).
Exp 8: To further study the effect of the state representation the population, record the added undiscounted return after optimizing the shared state representation. Specifically, we record the change in average return, maximum return, and minimum return of the population before and after optimization the shared state representation. The results Fig. 17 show the improvement of shared state representations can improve the average reward of the population, where the major improvement comes from poor policies. This demonstrates that the optimization of shared state representations can construct a better policy space for all policies, thus facilitating the evolution of the population.
17

---

Published as a conference paper at ICLR 2023
Ant
Walker
6000
Undiscounted Return
Undiscounted Return
5000
5000
4000
4000
3000
3000
2000
2000
10
1000
Time Steps
Steps
Figure 14: Comparison of E (TD3) with different population sizes. 5 is more effective than 10 in ERL-Re² (TD3). Thus we set for population size in all tasks. For fair comparison, we also evaluate other baselines with 5 and 10 and report the best performance.
5000
Undiscounted Returr
Undiscounted Return
Undiscounted
5000
230
130
6000
2000
1000
10
Time CLin1
(300)
1ime 500es
Figure 15: Comparison of EA and EA with Rea With Re² the performance of EA can be further improved, which indicates that the idea of ERL-Re" can be integrated with EA and further improve EA.
D.3 ADVANCED STATE REPRESENTATION
In this subsection, we further explore how to learn more effective shared state representation in Exp 9 and Exp 10.
Exp 9: We conduct additional exploration to enable further improvements of shared state represen- tations by unsupervised learning. To make the representation have dynamic information about the environment, we add the mutual information (MI) loss to the first layer of the shared state represen- tation network. Specifically, When given current state and the next state s', we want to make the representation based
and s' have high correlation with the corresponding action a, which can be donated as I(x, 8'; a). We use the MINE (Belghazi et al., 2018) to estimate MI. MINE is a neural network to estimate the MI of any two random variables. To make the shared state representation contain the information, we obtain one variable by combining the representations of and from the first layer of the shared state representation network and use the representations and as the inputs to MINE. We evaluate ERL-Re2 (TD3) with the MI loss and ERL-Re on Ant and Walker. The results in Fig. 18 show that the MI loss can bring some performance improvement in Ant and fail in Walker. We find that the MI loss has both good and bad effects on the final performance. A more stable and efficient way to improve the performance is left as subsequent work.
Exp 10: Inspired by the idea of VIP (Dabney al., 2021), conduct experiments optimize the shared state representation with the previous policy representations. Specifically, we optimize the shared state representation with policy representations not only current but also the previous generation. The results in Fig. 19 show that this approach does not guarantee performance gains. This is just an initial attempt to see how the shared state representation can be further improved by self-supervised learning, which deserves subsequent study. We consider this future work.
D.4 DISCUSSION OF OTHER ASPECTS
In this subsection, we further discuss some other aspects of ERL-Re² such as the architectures selection, time consumption, the number of parameters and other factors in Exp 11, Exp 12, Exp 13, Exp 14.
18

---

Published as a conference paper at ICLR 2023
Swimmer
Swimmer
0.8
Discarded Percentage
0.8
Elites Percentage
0.6
0.6
0.5
0.4
0.4
0.3
ERL(TD3)
0.2
PDERL(TD3)
0.1
ERL-Re2(TD3)
0.0
0.0
Steps (1e6)
145
Time Steps (1e6)
Figure 16: Comparisons of probabilities of RL agent being selected as the elite (left) and discarder (right) of the population in ERL-Re² (TD3), ERL(TD3), and PDERL(TD3). The results show that both EA and RL in ERL-Re² (TD3) have more competitive effeets on population evolution than in ERL((DD3) and PTERL(TD33.
HalfCheetach
7000
Max
Incremental Return
6000
Mean
5000
Min
4000
3000
2000
1000
-1000
100000 200000
300000 400000
500000 600000 700000 80000
Time Steps (1e6)
Figure 17: the policy performance change in the EA population after optimizing the shared state representation. Max/Min/MMen means the change of the best/worst/average performance in the population before and after the optimization. The results demonstrate that the optimization of shared state representations can construct better policy space for all policies, thus facilitating the evolution of the population.
Exp 11: In ERL-Re2, we introduce PeVFA for EA while maintaining the RL critic for RL. In principle, RL can also be improved by the PeVFA. leading only one VeVFA for both the EA and RL agents. Both architectures can achieve useful common knowledge sharing. To further verify the effectiveness, version. The results in Fig. 20 show that similar performance can be obtained with both architectures. But to preserve the flexibility and corresponding characteristics of RL methods, retain the original critic. From the perspective of RL, our architecture does not make changes to the original structure, e.g., maintaining the architecture of the value function, and this separately maintained approach is more convenient analyze and combine. Therefore, we choose this architecture as the final solution.
conduct comparison experiments between these
architectures in the TD3
We evaluate the time consumption of different algorithms on Walker. The experiment is carried out NVIDIA GTX 2080 Ti GPU with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. We evaluate the time consumption of each algorithm individually with no additional programs running the device. As shown in Table 1, ERL-Re incurs other methods. The main time consumption comes from the training PeVFA and shared state representation. 94.73% of the total time is spent on training. If the researchers sensitive to time overhead, distributed approach can be used reduce the time overhead.
Exp 12:
additional time consumption compared to
Exp 13: We provide the number of parameters (i.e., weights and biases) on HalfCheetach in Table 2. The results show that ERI-Re2 (TD3) is competitive with other methods in terms of the number
19

---

Published as a conference paper at ICLR 2023
Walker
Ant
Undiscounted Return
6000
Undiscounted Return
5000
5000
4000
3000
2000
2000
ERL-Re((TD3)
1000
1000
ERL3Re w/ MI loss
0
0.4
Steps (1e6)
146
Time Steps (1e6)
Figure 18: Comparisons of ERL-Re² (TD3) and ERL-Re (TD3) with MI loss (i.c., Self-supervised loss). MI loss brings certain performance improvements in Ant, which is worth further research in the future.
of parameters. The number of parameters ERL-Re² (TD3) is mainly derived from PeVFA, i.c., 47.9%. In addition, ERL-Re² has better scalability capability than other ERL-related methods the population size increases. Since ERL-Re² only introduces a linear policy instead of an independent nonlinear policy network for each agent. For example, on HalfCheetach, when the population size increase to 100, ERL-Re² only need additional 22.4% parameters (i.e.. 171570), while other methods such as CERL(TD3) and CEM-TD3 needs additional 1047378600 parameters, which is not feasible.
We provide more experiments eliminate the influence of some other factors. In the official codebase of ERL and PDERL, layer normalization is used for both actor and critic. Our code is built on the codebase of PDERL. thus ERL-Re² also uses layer normalization. But in the official codebase of TD3, layer normalization is not employed. To exclude the effect of layer normalization, we provide experiments of TD3 with layer normalization. The results in Fig.21 show that TD3 with layer normalization has similar performance to TD3, which demonstrates that the strengths of ERL-Re2 do not come from the layer normalization.
Exp 14:
Second, we want to exclude the influence of discount rewards on performance. Thus provide experiments by comparing ERL-Re² (TD3) using MC return one episode as the surrogate of the fitness and ERL-Re (TD3) using discount MC return of one episode as surrogate of the fitness. The results in Fig. 22 show that only using discounted MC return as the surrogate of the fitness not effective as using undiscounted MC return, which indicates that the advantages of ERL-Re" do not come from the discounted form in the surrogate fitness f(W.).
Third, the network structure of the actor of ERL and PDERL is not consistent with the structure of the TD3 paper. Thus we provide a comprehensive comparison, one maintains the structure of the original paper and one is consistent with the structure in the TD3 paper. The results in Fig. 23 and Fig. 24 show that the algorithms with the structure of the TD3 paper improve the performance in some environments and degrade in others. We can find that large network structure (i.e., the structure of TD3 paper) leads to large fluctuations in ERL such as Swimmer, where EA plays a more critical role than RL. The main reason behind this is that parameter-level crossover and mutation large networks are more unstable than in small networks. In summary, ERL-Re² (TD3) still significantly better than the best performance in both structures.
Table 1: Time consumption of different algorithms on Walker2d every 10000 steps. The additional time consuming comes mainly from the training of PeVFA and shared state representations. Dis- tributed training can solve this problem.
Algorithm
ERL-Re (TD3)
TD3
PDERL(TD3)
seconds
396.14
215.5
251.37
algorithm
ERL(TD3)
CERL(TD3)
CEM-TD3
seconds
223.86
502.45
241.82
20

---

Published as a conference paper at ICLR 2023
Ant
Walker
6000
Undiscounted Return
Undiscounted Return
5000
5000
4000
3000
3000
2000
ERL-Re<(TD3)
2000
1000
version 1
1000
version
Steps (1e6)
146
Time Steps (1e6)
Figure 19: Comparisons of ERL-Re² (TD3) and ERL-Re² (TD3) with VIP (Dabney et al., 2021). We design two versions. The version I means firstly optimizing PeVFA with the current and previous generation' policy representations, then optimizing the state representations with these policy representations. The version 2 means directly optimizing the state representations with the current and previous generation' policy representations.
Swimmer
Hopper
Undicceuntr Mettures
310
320
Undisounted Return
6000
Undicounted Relurn
2000
230
23/0
2:0
6000
2000
110
1.000
2000
-000
PCVFA both and
PoUTA
Rl
2000
500
Time Steps (1991
Time Staps (1.8))
Time Steas (205)
Figure 20: The comparative experiment of two architectures, Only PeVFA for both EA and RL is using the PeVFA to update RL without introducing the RL Critic. PeVTA for EA and Critic for RL is the architecture chosen in this paper. The results show that both architectures are feasible and can achieve competitive performances.
D.5 EXPERIMENTS WITH VISUAL STATE INPUTS
We evaluate whether ERL-Re² could bring improvement in Deepmind Control Suite (Tassa et al., 2018) with visual state inputs in Exp 15.
We evaluate ERI-Re on four tasks. In these tasks, we take pixel-level images as inputs. Specifically, implement ERI-Re² based on RAD (Laskin et al., 2020) (which can be simply viewed as SAC + Image Augmentation for image inputs). We follow the conventional setting and provide the performance comparison in 100k environment steps. We use 4 representative DMC tasks, covering different robot morphology. The results shown Table are means and stds over 5 seeds.
Exp 15:
We can observe that ERL-Re" also leads to significant improvements in visual control, which demonstrates the effectiveness of ERL-Re", while other methods fail. The main reason is that population evolution is very inefficient without knowledge sharing with image inputs, since each individual needs learn common information from images independently. which is more difficult to learn than the state-level tasks. This also indicates that the techniques of addressing visual states (e.g.. image augmentation) are orthogonal to our method, which is worthwhile for further development in the future.
D.6 EXPERIMENTS IN HIGH-DIMENSIONAL COMPLEX TASKS
For complex high- nigh-dimensional tasks, we choose the multi-agent the StarCraft micromanagement (SMAC) benchmark (Samvelyan et al., 2019) as the testbed for its rich environments and high complexity of control. The SMAC benchmark requires learning policies Agents can move in four cardinal directions, stop. take noop (do nothing), or select an enemy to attack at each timestep. Therefore, if there are n. enemies in the map, the action space for each allied unit contains n. + 6 discrete actions. To solve these high complexity tasks, we need to integrate E with Multi-Agent Reinforcement Learning (MARL) algorithm. Fortunately, ERL-Re can
large action space.
21

---

Published as a conference paper at ICLR 2023
Table 2: The number of parameters (i.e., weights and biases) for different algorithms on HalfCheetach. For ERL and PDERL, we use to separate the results for two implementations: the left one uses the same structure as in the original paper, and the right one is aligned with the settings in the TD3 paper.
Algorithm ERL-Re" (TD3) number
CERL(TD3)
PDERL(TD3)
CEM-TD3
ERL(TD3)
765508
1553462
456302 1553462
1553462
456302 1553462
Ant
Walker
Swimmer
5020
Returr
120
6030
2002
Undiccoanmtan
Undiscounted
2502
3020
60
2010
2002
1020
layer
TD3 we byyo nommalzazion
502
-20
Time stops doen
1 Stots (308)
Time Stere (108)
Hopper
Humanoid
HalfCheetach
3350
10030
Undiccouted Return
3300
Undiscoutedd Ieeturn
Undiscounted Return
2300
2000
6020
3500
4020
3300
300
2020
2016-
Time Stepe (1:4)
Time Stape (1AO)
Time Stops (160
Figure 21: The comparative experiment of TD3 with and without layer normalization. The results demonstrate that the strengths of ERL-Re? do not come from the layer normalization.
be combined with the MARL algorithms easily. Specifically, we need to maintain population of teams (i.e., Multiple policies that control multiple agents for collaboration are treated as a team). Each policy in the team needs to follow the two-scale oppresetattion-basee policy construction. Thus knowledge can be efficiently conveyed through shared representations and reinforcement and evolution occur in the linear policy space.
For the base MARL algorithm, Learning algorithm published in NeurIPS 2021 (Peng et al., 2021). Then we combine ERL-Re² with FACMAC and evaluate it on six tasks. Our implementation is based on the official code of FACMAC and the hyperparameters are kept consistent. To simplify hyperparameter introduced by ERL-Re2, we set p to 1.0, K to on all tasks and select 3 from (0.2, 0.05, 0.01 For baselines, we compare ERL-Re with MERL (Majumdar et al., 2020) except FACMAC and EA. To the best of our knowledge, MERL is the only work that combines EA and MARL for policy search. The difference is that MERL is applied to environments with dense agent-specific rewards (optimized by RL) as well as sparse team rewards (optimized by EA). For comparison, we use EA and MARL(i.e.. FACMAC) to jointly optimize the team rewards of SMAC. The results in Figure 25 show that ERL-Re can significantly improve FACMAC in terms of convergence speed and final performance. It is worth mwnymmaaaalllllaa baneling, MM, 3s5z and 2c _e_us____ ___2g is greater than 100 dimensions, which demonstrates that ERL-Re can bring improvements to the original al algorithm in nigh-dimensional complex control tasks. Through the above experiments, we further demonstrated the efficiency and generalization of ERL-Re?.
choose FACMAC, advanced Multi-Agent Reinforcement
noting that the joint action space of so many
D.7 EXPERIMENTS ON MCJoCo WITH LONGER TIMESTEPS
In Figure 4, we provide the experiments on MuJoCo with 1 million timesteps, which is mainly consistent with the training timestep in SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018). To demonstrate the competitiveness of ERL-Re" in longer timesteps, we provide the experiments in the TD3 version with 3 million timesteps. Since Swimmer and Hopper have already achieved convergence performance, we mainly provide the experiments on the remaining four tasks. The results may differ slightly from those in the main text, as they are obtained by running on different
22

---

Published as a conference paper at ICLR 2023
Walker
Humanoid
Undiscounted Return
5000
Undiscounted Return
5000
4000
4000
3000
3000
2000
1000
Discounted MC
1000
MC
Time Steps
Steps
146
Figure 22: Comparison experiment using one episode MC return and one episode discounted MC return. The experiments do not use f(W:) as a surrogate of the fitness. The results indicate the strengths of ERL-Re do not come from the discounted design.
Table 3: Comparisons Deepmind Control Suite with image inputs.
Environments
Cartpole Swingup 854,453 17.582
Cheetah Run 508.619 22.622
Finger Turn hard
Walker Walk
ERL-Re (RAD)
197.35 83.388
567.417 + 139.278
RAD
728.66 + 149.219
472.594 = 29.595
126.725 + 48.954
498.867 = 142.509
ERL(RAD)
527.81 + 126.48
329.709 132.674
93.6 + 130.53
287.969 57.168
PDERL(RAD)
471.164 + 220.221
331.903 + 29.614
150 + 70.711
252.492 + 93.01
CEM-RAD
694.909 = 44.74
421.574 + 20.891
50.25 + 67.67
66.042 + 19.23
Steps
100k
100k
100k
100k
servers. The results in Figure 26 show FRI-Re2 still significantly outperforms other algorithms with 3 million timesteps, which further verifies the efficiency of ERL-Re".
E METHOD IMPLEMENTATION DETAILS
All experiments are carried out on NVIDIA GTX 2080 Ti GPU with Intel(R) Xcon(R) CPU E5-2680 v4 @ 2.40GHz.
E. IMPLEMENTATION OF BASELINES
For baseline algorithms, we use the official implementation. In our paper, there are two main implementations, one combines TD3 and the other combines DDPG, Many of these official imple- mentations contain implementations such as CEM-RL' and CERL For methods that do not have two implementations such as ERL$ and PDERL (both based on DDPG), we modify the DDPG to TD3 which is very simple and straightforward extension. For the basic algorithm TD3?, official implementation. For EA, we use the official implementation we use the implementation from t We fine-tuned ERL-related baselines, mainly including population size, episodes of agents interacting with the environment in the population, elite rate, injection frequency, etc.
the PDERL. For SAC and PPO,
E.2 NETWORK ARCHITECTURE
This section details the architecture of the networks. Our code is built on the official codebase of PDERL. Most of the structure remains the same. For structures specific to framework, the shared
hhapss//gituub.co///neaaeeee
hhtps://gituub.com// "htps://gttuu..com/// R- R--RM/stalee-hsslinees
23

---

Published as a conference paper at ICLR 2023
Hopper
Walker
S020
Undiscounted Return
4003
Undigcountad
6020
Undiscourted
2502
2000
2002
3020
1202
2000
2010
1020
Original
1000
TD3 paper
Time Siops (heb)
Tim Soara (a0))
Time Staze (245)
HellCheetach
Undiscurtzd Retarn
Undiscaunted Return
#300
220-
200
anficcuuteed
3002
150-
4200
2003
200
30
2002
Thm Stope (3e0)
Thee Stepe (end)
Tim Stace (3a))
Figure 23: The comparative experiment of ERL with different actor structures. With the structure of the TD3 paper, ERL improves performance some tasks and degrades in other tasks. The parameter-level operators in a large network can make performance casily crash. (See Swimmer). In summary, ERL-Re" (TD3) is still significantly better than the best performance of the two structures.
Ant
Hopper
Walker
1502
Undiscounted Retum
0000
5000
3003
Undicourta
Undizrounted
2302
3000
2000
2010
2000
1020
Original atructree
Strudture of TD3 paper
2000
34
0.2
21
03
C2
12
Swimmer
HellCheetach
Humanoid
400
12300
Unatscounzed Retur
335
Undiscounted Retum Retum
30000
250
8000
300-
6300
1003
:c0
4200
2003
2002
Figure 24: The comparative experiment of PDERL with different actor structures. With the structure of the TD3 paper. PDERL improves performance in some tasks and degrades in other tasks. In summary, ERL-Re2 (TD3) is still significantly better than the best performance of the two structures.
state representation network is constructed by two fully connected layers with 400 and 300 units. The policy representation is constructed by one fully connected layer with action dim units.
PeVFA takes state, action and policy representation inputs and maintains double Q networks which are similar to TD3. The policy representation can be regarded as a combination of matrix with shape [300, act ion cetttiinndii dim] (i.e., weights) and a vector with shape [act dim] (i.e., biases) which can be concatenated
dim. We first encode each vector with shape [300 + 1] of the policy representations with 3 fully connected layers with units 64 and leaky_re preeerllu activation function. Thus can get embedding list with shape [64, act on dim] and get the final policy embedding with shape [64] by taking the mean value of the embedding list in the action dimension. With the policy embedding, we concatenate the policy embedding, states, and actions as the input to an MLP with fully connected layers with units 400 and 300 and get the predicted value by PeVFA. The activation functions in PeVFA all use relu. We list structures Table 4 and 5. For other structures, we take the settings directly from the codebase of PDERL. The network structure is the same in ERL-Re? (TD3) and ERI-Re (DDPG).
a matrix with shape [300- 1,
24

---

Published as a conference paper at ICLR 2023
many. baneling
5.7
38
3.6-
Test
3.4
0.4
FACMAC
MERL
0.2
2.5
5.
03
430
5.0
638
3s.vs_
3.0
2.6
Test win
0.6
0.6
145
2.4
3.0
9.0
5.0
Staps 12c5)
Figure 25: Performance comparison between ERL-Re² and baselines (all the FACMAC version).
HalfCheetah
Ant
14000
7000
Undiscounted Return Undiscounted Return
12000
Undiscounted Return
6000
10000
5000
4000
ERL-Re² (TD3))
6000
CEM-RL
4000
ERL
2000
PDERL
2000
1000
TD3
0
Time Steps (1e6)
Time Steps (1e6)
Humanoid
Walker
6000
7000
Undiscounted Return
5000
va
4000
5000
3000
3000
2000
1000
1000
0.
0.5
1.0
2.0
2.5
3.0
3.5
1.0
3.5
2.0
2.5
3.0
Time Steps (1e6)
Time Steps (1e6)
146
Figure 26: Performance comparison between ERL-Re and baselines (all the TD3 version with million timesteps).
E.3 HYPERPARAMETERS
This section details the hyperparameters across different tasks. Some hyperparameters are kept consistent across all tasks. Population size is 5 for both 2 (TD3) and ERL-Re² (DDPG). for mutation operators is 1.0 for both ERL-Re" (TD3) and ERL-Re (DDPG). The number of episodes of interaction with the environment for surrogate fitness 1. is 0.99 except 0.999 for Swimmer (all baselines follow this setting). Since ERL-Re2 is built on PDERL, other settings remain
25

---

Published as a conference paper at ICLR 2023
Table 4: The structures the shared state representation network and policy representations.
Shared State Representation Network
Policy Representation actiion _n_dim)
(state_ __tee___m____ 400)
tanh (400,300) tanh
tanh
Table 5: The structure of PeVFA.
PeVFA
(state-action_dim + 64, 400) leaky_relu (400, 300)
(301, 64)
leluy_relu (64,
leaky relu (300, 1)
leaky_rell (64, 64)
the same. addition, we list other hyperparameters specific to ERL-Re" which varied across tasks in Table 6 and Table 7. In addition this, we find that ERL-Re" is very stable, and if you don't want tune too many hyperparameters, competitive performance can be obtained when keeping and 3 consistent with the settings in the table and do not use our surrogate fitness (i.e., p=1.0). We prove this in Fig. 6d.
Table 6: Details of the hyperparameters of ERL-Re--TD: that are varied across tasks.
Env name
p
3
H
K
HalfCheetach
0.3
1.0
200
Walker
0.8
0.2
50
1
Swimmer
0.3
1.0
200
3
Hopper
0.8
0.2
50
3
Ant
0.5
0.7
200
Humanoid
0.5
0.5
200
F ADDITIONAL DISCUSSION
F.1 THE ALTERNATIVE OF REPLACING GA BY CEM
In this paper, use GA as the basic evolutionary algorithm to combine with different RL algorithms. ERL-Re can utilize different evolutionary algorithms to exploit the advantages of these algorithms. Here we discuss how to use CEM (Pourchot & Sigaud, 2019) to replace GA in ERL-Re
The pseudo-code is shown in Algorithm 3. To achieve this, need to modify population generation. 2) population evolution. For the first aspect, we need to use CEM to generate the population instead of pre-defining the entire population by constructing mean policy W,, to sample the population based on a predefined covariance matrix I.stI. For the second aspect, we need to select the top T policies to update W and Σ mutation operators.
aspects: 1)
Tint instead of the crossover and
F.2 TRAINING PEVFA AN OFF-POLICY FASHION
To our knowledge, PeVTA (Tang et al., 2022) and related variants are trained in an on-policy fashion in prior works. In the original paper of PeVFA, newly implemented PPO algorithm, called PPO-PeVFA is studied. In VEA, the algorithm saves states, returns and the corresponding policies in replay buffer; then PeVFA Vo(s. is trained by the Monte-Carlo method, based on the experiences collected the replay buffer with some policy representation X Note that although historical experiences are replayed, each policy uses its own collected experiences to train Ve. thus being on-policy. Training the PeVFA Va in off-policy manner needs introduce additional mechanisms such as importance sampling method, which are not implemented in their work. Similarly, this case can be also be found in (Faccio et al., 2021).
26

---

Published as a conference paper at ICLR 2023
Table 7: Details of the hyperparameters of FERL-Re--DDPE that are varied across tasks.
Env name
p
1
H
K
HalfCheetach
0.5
1.0
200
1
Walker
0.8
0.2
50
1
Swimmer
0.3
0.5
200
3
Hopper
0.8
0.7
50
3
Ant
0.7
0.5
200
1
Humanoid
0.7
0.5
200
1
Algorithm 3: ERL-Re" with Cross Entropy Method (CEM)
Input: E Tinit I and T of CEM, the EA population size n, the probability of using MC estimate, the partial rollout length H
Initialize: Initialize the mean policy W, p of CEM. replay buffer D, the shared state representation function Zo. the RL agent Wr,, the RL critic Qa and the PcVFA Qo (target networks are omitted here) repeat
# Draw the CEM population PP Draw the population IP = (Wi,..., W. from (T Σ) # Rollout the EA and RL agents with Zg and estimate the (surrogate) fitness if Random Number p then
Rollout each agent in for H steps and evaluate its fitness by the surrogate f(W) see Eq. 3 cise
10
L Rollout each agent in P for one episode and evaluate its fitness by MC estimate
11
Rollout the RL agent for one episode
12
Store the experiences generated by and W. # Individual scale: evolution and reinforcement in the policy space Train PeVFA Qs and RL critic Qu with /)
13
14
D see Eq. 1
15
Optimize CEM: rank all policies with fitness and use the top T policies to update CEM. Optimize the RL agent: update W., (by e.g.. DDPG and TD3) according to Q # Common scale: improving the policy space through optimizing Z
16
see Eq. 5
Update the shared state representation: optimize Ze with unifying gradient direction derived from Eq. 2
value function maximization regarding Th and Q+
19 until reaching maximum training steps;
In this work, we train PeVFA in an off-policy manner for the first time. We achieve this in more efficient way by training PeVFA Qs a, W)) with 1-step TD learning and the linear policy representation W. That is, the value function of each policy can be learned (with its representation W: and Qo) from any off-policy experience (s, 8', r) according Eq. 1. We emphasize that this is appealing, and we believe that this can realize effective training of VeVFA. In turn, the ability of PeVFA in conveying the value generalization among policies can be better rendered. We will further investigate the potential of this point the future.
We use PeVFA to provide value estimates for each agent (in the EA population). One may note that the same purpose can be achieved by maintaining conventional Q network for each agent individually. However, this means suffers from the issue of scalability, resulting in significant time and resource overhead. In addition, this approach does not policy representations and thus cannot take advantage of value function generalization to improve value function learning.
27


Reviews:
[{"review_id": "wfxCAxftWSr", "reviewer": "Reviewer_oDSW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out the problems of the previous ERL frameworks, i.e., the parameters of different policies are independent and the EA operators often fail. The paper then proposes a new ERL framework that splits the policy network into two parts, i.e., the shared nonlinear state representation and the independent linear policy representation, to overcome the former problem, and proposes the behavior-level EA operators to overcome the latter problem. Experimental results show that the proposed method performs better than other ERL, EA, and RL methods in all tasks.", "summary_of_the_review": "This paper proposes a new ERL framework that splits the policy into two parts, and proposes the behavior-level EA operators. The idea is novel and reasonable. The paper conducts extensive experiments and ablation studies to show the performance of the framework and the effects of each part of the framework. Despite some limitations, it is overall a good work.", "strength_and_weaknesses": "- It is a good and reasonable idea to split the policy into the shared state representation and the independent policy representation, which may be of independent interest.\n\n- The method performs significantly better than other methods in the experiments, and the ablation studies are extensive.\n\n- The paper is well-written and easy to follow.\n\n- The method may be further improved by taking full advantage of EAs, e.g., keeping the diversity of population.\n\nMinor problems: There are some confusing symbols in the paper, e.g., the symbols of the sets in Section 6.1.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "w5lOBGuPx1", "reviewer": "Reviewer_vR7r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a strategy for combining aspects of reinforcement learning (RL) and evolutionary algorithms (EA). A shared representation is learned for all actors (EA policies and an RL policy), with each EA actor maintaining only its own linear head. The EA policies are recombined using proposed behavior-level genetic operators that act on rows of the linear head, while the RL policy is updated by maximizing the Q function as usual. The authors show that EA agent performance can be estimated without full rollouts by using the learned value function. The proposed algorithm, dubbed ERL-Re$^2$, displays strong performance on various control tasks.", "summary_of_the_review": "Overall, I think the paper is a useful contribution owing to its strong empirical results. The RL community would benefit from seeing that ideas from EA can be effectively combined with RL to improve performance.", "strength_and_weaknesses": "Strengths\n* The core idea of utilizing a shared representation across agents and only applying EA to the last layer makes intuitive sense.\n* On all the tasks used in evaluation, ERL-Re$^2$ outperforms existing RL and EA algorithms.\n* The appendix includes extensive ablations investigating the effects of various hyperparameters and design choices. I appreciate that the authors have reported negative results for different ideas they tried; this may save a grad student some time later.\n\nWeaknesses\n* Since the algorithm combines elements of RL and EA, it has many hyperparameters to tune.\n* The performance gains resulting from the proposed modifications (behavior-level genetic operators and fitness evaluation metric) are not very large.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "v2sGBMAMmy", "reviewer": "Reviewer_ouqq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Combinations of evolutionary algorithm (EA) and deep reinforcement learning (DRL) have been investigated in the literature to take advantage of EA (exploration ability, robustness and stability) while maintaining the advantage of DRL (sample efficiency). However, in the previous work on the combination of EA and DRL, we observe that the performance of the combination is sometimes dominated by EA-only or DRL-only approaches, for example on Swimmer and Humanoid on MuJoCo environments.\n\nThis paper proposed a novel combination, namely Evolutionary Reinforcement Learning with Two-scale State Representation and Policy Representation (ERL-Re2). In the proposed approach, the policies of EA and RL agents are linear policies on a non-linear state-representation that is shared over all EA and RL agents. The state-representation is trained using a replay buffer storing all the experiences generated by EA and RL. Linear policies are trained independently by EA and RL. To improve the sample efficiency of EA part, the authors introduced a shared critic network taking the policy parameter as its input and use it as a surrogate function of the cumulative reward, leading to reduction of the number of interaction for policy evaluation in EA. Moreover, to make the crossover and mutation in EA part more sense, the authors introduced novel operators, named behavior-level mutation and crossover. \n\nNumerical experiments on MuJoCo environments show its efficiency as well as superior final performance over existing ERL variants and RL baselines and EA. ", "summary_of_the_review": "This paper improved the performance of ERL significantly as summarized in the strength and weaknesses section. Each algorithmic component have been carefully evaluated in experiment (in appendix). ", "strength_and_weaknesses": "# Strength\n\n* Strong performance has been observed on MuJoCo environments in comparison with PPO, SAC, DDPG, TD3 (RL baselines), ERL, CERL, PDERL, CEM-RL (ERL baselines).\n* Overall, the paper is well-written. The algorithm and experimental details are provided sufficiently.\n\n# Weaknesses\n\n* From the description of the algorithm written in the main text, the introduction of the shared state representation appears to be aimed at improving search efficiency. However in the experiments, the final performance was improved on some environments. Is this because the training was not finished? A possible criticism is that as the training is not finished, the final performance of the proposed approach might be lower than the other approaches. A careful discussion is required.\n\n# Comment / Question\n\n* The proposed behavior-level operators seem to make sense. However, similar effect (considering the correlation between parameters) could be realized by just using CMA-ES or CEM. Why do you want to employ a GA-like approaches?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dQRIdxxOI7", "reviewer": "Reviewer_eTSz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Following on from the direction proposed by ERL at NeurIPS 2018, this work seeks to combine evolution and off policy RL into a hybrid algorithm, achieving the best of both. This direction remains promising albeit a little saturated after several years, with many variants proposed. The novel component of this work is to maintain two separate policies which share a representation space, and evolve the evolution policy in the behavior space. Empirical results show gains in the MuJoCo suite for both TD3 and DDPG.", "summary_of_the_review": "This paper presents a method for general RL tasks. Since the experiments are toy MuJoCo domains, with no analysis over how or why it has gains there, it is hard to know if it is useful for problems we actually care about in the RL community. I thus vote to reject the paper based on the breadth and depth of the experimental results. ", "strength_and_weaknesses": "### Strengths\n\n* The general direction of combining evolution and RL seems promising, and maybe under explored despite many similar works in the past few years. \n* This paper proposes a slightly different approach, keeping track of two separate policies, and some tricks to share data that may be useful for others.\n* The empirical gains are strong in MuJoCo and ablations seem convincing that the method is not overly sensitive.\n* It is good to see a limitations section even though it is 80% just discussing future work.\n\n### Weaknesses\n\n* This would have been a good paper in 2019, but right now the field has moved on so far from just showing gains on MuJoCo to provide a useful contribution. Since 2020 it has become common practice to use pixel based tasks, which is no longer expensive to run. Further, there has been a shift in focus towards generalization of agents (see Kirk et al 2021 for a survey), and tasks requiring more complex behaviors. Why does this matter? It is very hard to know if the insights in this work will translate to settings at the forefront of research, and thus the impact seems limited.\n* For a pure methods paper, which is solely based on experimental results, I would expect to see at least two distinct domains. Having both TD3 and DDPG doesn't seem additive since they are essentially the same algorithm with a few tricks. This seems like a bit of a duplication which superficially makes the method look more general.\n* The method seems to rely heavily on a linear behavior representation. How does this work if we move to a higher dimensional setting? We already know from Mania 2018 that a linear policy can solve MuJoCo, but how about something higher dimensional?\n* There is no intuitive analysis as to why this method outperforms where it does. Why is it so good on Swimmer? More broadly, what can we take away from this other than knowing it is great on MuJoCo from proprioceptive states with a 1M timestep budget?\n\nMinor Issues:\n* There are a few places where acronyms should be in the parentheses with the reference, e.g. (RL, Sutton 1998). The paper has it like (RL) (Sutton, 1998).\n* There are some grammatical mistakes for example \"Evolutionary Algorithm\" should be plural.\n* In the DDPG figure the Swimmer plot has the wrong title.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster