Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection
Authors: Jiajun Fan
Yuzheng Zhuang
Yuecheng Liu
Jianye HAO
Bin Wang
Jiangcheng Zhu
Hao Wang
Shu-Tao Xia
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Deep Reinforcement Learning
The Arcade Learning Environment
Human World Records
Behavioral Control

Abstract: The exploration problem is one of the main challenges in deep reinforcement learning (RL). Recent promising works tried to handle the problem with population-based methods, which collect samples with diverse behaviors derived from a population of different exploratory policies. Adaptive policy selection has been adopted for behavior control. However, the behavior selection space is largely limited by the predefined policy population, which further limits behavior diversity.  In this paper, we propose a general framework called Learnable Behavioral Control (LBC) to address the limitation, which a) enables a significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies; b) constructs a unified learnable process for behavior selection. We introduce LBC into distributed off-policy actor-critic methods and achieve behavior control via optimizing the selection of the behavior mappings with bandit-based meta-controllers. Our agents have achieved 10077.52% mean human normalized score and surpassed 24 human world records within 1B training frames in the Arcade Learning Environment, which demonstrates our significant state-of-the-art (SOTA) performance without degrading the sample efficiency.

Published as a conference paper at ICLR 2023
LEARNABLE BEHAVIOR CONTROL: BREAKING ATARI HUMAN WORLD RECORDS VIA SAMPLE- EFFICIENT BE- HAVIOR SELECTION
Jiajun Fan", Yuzheng Zhuang', Yuecheng Liu², Jianye Hao", Bin Wang"
Jiangcheng Zhu", Hao Wang*. Shutao Xia
1 TTinggua Shenzhen International Graduate School, Tsinghua University
2 Huawei Noah's Ark Lab, Huawei Cloud, 4 ZZejiang University
fanjj21 @ maails.tiighuu..cdu.cc xiast viast@szzi wt.nnniinnghaa.ccc..nnn haohaow @ @zju.edu.cn, 2,3 (zhuangyuzheng, liuyuechengl, haojianye, wangbin158, 158, zhujiangcheng/ @ cum
ABSTRACT
The exploration problem is one of the main challenges in deep reinforcement Recent promising works tried to handle the problem with population- based methods, which collect samples with diverse behaviors derived from a population of different exploratory policies. Adaptive policy selection has been adopted for behavior control. However, the behavior selection space is largely limited by the predefined policy population, which further limits behavior diversity. In this paper, we propose a general framework called Learnable Behavioral Control (LBC) to address the limitation, which a) enables a significantly enlarged behavior selection space via formulating hybrid behavior mapping from all policies; b) constructs a unified learnable process for behavior selection. We introduce LBC into distributed off-policy actor-critic methods and achieve behavior control via optimizing the selection of the behavior mappings with bandit-based meta- controllers. Our agents have achieved (0077.52% mean human normalized score and surpassed 24 human world records within 1B training frames in the Arcade Learning Environment, which demonstrates our significant state-of-the-arr (SOTA) performance without degrading the sample efficiency.
learning Recent
LBC (Ours)
MEME (2022)
AgentS7 (2020)
100
10
1008
10:0
10
5
25
5
10a
16.
.6.
Number Samples
Figure 1: Performance on the 57 Atari games. Our method achieves the highest mean human normal- ized scores (Badia al. 2020a), is the first to breakthrough 24 human world records (Toromanofi et al. 2019). and demands the least training data.
INTRODUCTION
Reinforcement learning (RL) has led to tremendous progress in a variety of domains ranging from
video games (Mnih et al. 2015) robotics (Schulman et al. 2015 2017). However, efficient exploration remains one of the significant challenges. Recent prominent works tried to address the problem with population-basee training (Jaderberg al. 2017 PBT) wherein population of policies with different degrees of exploration is jointly trained to keep both the long-term and short- term exploration capabilities throughout the learning process. A set of actors is created acquire
Work done as a research intern at Huawei Noah's 's Ark Lab.
"orresponning authors.

---

Published as a conference paper at ICLR 2023
Made Naviattonn
Actos
Mata contrclee
Framonat
Actar
Endretment
Motticotticerr sagesegfl
Erwronrrae
2
Masping Sp.
Mode lraring
Duta Collction
Figure 2: A General Architecture of Our Algorithm.
diverse behaviors derived from the policy population (Badia et al. 2020b a). Despite the significant improvement in the performance. these methods suffer from the aggravated high sample complexity due to the joint training on the whole population while keeping the diversity property. To acquire diverse behaviors, NGU (Badia et al., 2020b) uniformly selects policies in the population regardless of their contribution to the learning progress (Badia et al.) 2020b). As an improvement, Agent57 adopts an adaptive policy selection mechanism that each behavior used for sampling is periodically selected from the population according to a meta-controller (Badia al.) (020a).. Although Agent57 achieved significantly better results on the Arcade Learning Environment (ALE) benchmark, it costs tens of billions of environment interactions as much as NGU. To handle this drawback, GDI (Fan & Xiao 2022) adaptively combines multiple advantage functions learned from a single policy to obtain an enlarged behavior space without increasing policy population size. However, the population-ban scenarios with more than one learned policy has not been widely explored yet. Taking a further step from GDI, we try to enable a larger and non-degenerate behavior space by learning different combinations across a population of different learned policies.
In this paper, we attempt to further improve the sample efficiency of population- bbsed reinforcement learning methods by taking a step towards a more challenging setting to control behaviors with significantly enlarged behavior space with a population of different learned policies. Differing from all of the existing works where each behavior is derived from a single selected learned policy, we formulate the process of getting behaviors from all learned policies as hybrid behuvior mapping, and the behavior control problem is directly transformed into selecting appropriate mapping functions. By combining all policies, the behavior selection space increases exponentially along with the population size. As a special case that population size degrades to one, diverse behaviors can also be obtained by choosing different behavior mappings. This two-fold mechanism enables tremendous larger space for behavior selection. By properly parameterizing the mapping functions, our method enables a unified learnable process, and we call this general framework Learnable Behavior Control.
We use the Arcade Learning Environment (ALE) evaluate the performance of the proposed methods, which is an important testing ground that requires a broad set of skills such as perception, exploration, and control (Badia et al. 2020a). Previous works use the normalized human score to summarize the performance on ALE and claim superhuman performance (Bellemare et al. 2013). However, the human baseline is far from representative of the best human player, which greatly underestimates the ability of humanity. In this paper, we introduce more challenging baseline, i.e., the human world records baseline (see [Toromanoff et al. (2019): ): Hafner et al.) (2021) for more information on Atari human world records). We summarize the number of games that agents can outperform the human world records (i.e., HWRB, see Figs. D to claim a real superhuman performance in these games, inducing a more challenging and fair comparison with human intelligence. Experimental results show that the sample efficiency of our method also outperforms the concurrent work MEME Kapturowski ct al. (2022), which is 200x faster than Agent57. In summary, our contributions
follows:
1. A data-efficient RL framework named LBC. We propose a general framework called Learnable Behavior Control (LBC), which enables a significantly enlarged behavior selec- tion space without increasing the policy population size via formulating hybrid behavior mapping from all policies, and constructs unified learnable process for behavior selection.
2. A family of LBC-based RL algorithms. We provide a family of LBC-based algorithms by combining LBC with existing distributed off-policy RL algorithms, which shows the generality and scalability of the proposed method.
3. The state-of-the-ae performance with superior sample efficiency. From Figs. [ our method has achieved 10077.52% mean human normalized score (HNS) and surpassed 24
2

---

Published as a conference paper at ICLR 2023
human world records within IB training frames in the Arcade Learning Environment (ALE), which demonstrates our state-of-the thr (SOTA) sample efficiency.
2 BACKGROUND
2.1 REINFORCEMENT LEARNING
MDP) defined by (S, A. p, r, 7/ Po). Considering discounted episodic MDP, the initial state so sampled from the initial distribution (s(s) S + P(S), where we use P to represent the probability distribution. At each time t, the agent chooses an action at E A according to the policy (Tay : P(AA) at state 8, E S. The environment receives at, produces the reward Tt r(8,a) : S A - R and transfers to the next state 8t+11 according to the transition distribution p (s' (.0) : x A P(S). The process continues until the agent reaches a terminal state or maximum time step. Define the discounted state visitation distribution as d (s) = (1 - E
RL can be formulated as a Markov Decision Process (Howard 1960,
= s|so]]. Define return G2 Σk= k=0 rerk+ wherein E (0, 1) is the discount factor. The goal of reinforcement learning is to find the optimal policy x² that maximizes the expected sum of discounted rewards Gt:
argmaxxBss.. Ead Ex G.
Tetelss
(1)
2.2 BEHAVIOR CONTROL FOR REINFORCEMENT LEARNING
In value-based methods, a behavior policy can be derived from a state-action value function Q7 (s, a) via e-greedy. In policy-based methods, a behavior policy can be derived from the policy logits & (Li et al., (018) via Boltzmann operator. For convenience, we define that a behavior policy can be derived from the learned policy model to.h via a behavior mapping, which normally maps a single policy model to a behavior, e.g., c-greedy (rrreddy@Fo...) (fo,.. In PBT-based methods, there would be set of policy models { (Do,,,i .bi. &gg.hy J. each of which is parameterized by 0, and trained under its own hyper-parameters h;, wherein O; E θ {O).... 0x} and h, E H (hi,..,, hr)
The behavior control population- based methods normally achieved two steps: i) select a policy model Le.h h from the population. ii) applying behavior mapping to the selected policy model. When the behavior mapping is rule-based for each actor (e.g.. e-greedy with rule-based c) the behavior control can be transformed into the policy model selection (See Proposition D Therefore, the optimization of the selection of the policy models becomes one the critical problems in achieving effective behavior control. Following the literature on PBRL, NGU adopts uniform selection, which is unoptimized and inefficient. Built upon NGU, Agent57 adopts a meta-controller to adaptively selected a policy model from the population to generate the behavior for each actor, which is implemented by non-stationary multi-arm bandit algorithm. However, the policy model selection requires maintaining a large number of different policy models, which is particularly data-consuming since each policy model in the population holds heterogeneous training objectives.
To handle this problem, recent notable work GDI-H3 Fan & Xiao 2022) enables obtain an enlarged behavior space via adaptively controls the temperature of the softmax operation over the weighted advantage functions. However, since the advantage functions are derived from the same target policy under different reward scales, the distributions derived from them may tend to be similar (e.g., See App. N. thus would lead to degradation of the behavior space. Differing from all of the existing works where each behavior is derived from a single selected learned policy, in this paper, try to handle this problem via three-fold: i) we bridge the relationship between the learned policies and each behavior via hybrid behavior mapping, ii) propose general way build deg-engeee large behavior space for population-bast based methods in Sec. [..] iii) we propose a way to optimize the hybrid behavior mappings from a population of different learned models in Proposition. 2
LEARNABLE BEHAVIOR CONTROL
In this section, we first formulate the behavior control problem and decouple it into two aab-problems: behavior space construction and behavior selection. Then, we discuss how to construct the behavior
3

---

Published as a conference paper at ICLR 2023
space and select behaviors based on the formulation. By integrating both, obtain general framework to achieve behavior control in RL, called learnable behavior control (LBC).
3.1 BEHAVIOR CONTROL FORMULATION
Behavior Mapping Define behavior mapping FF mapping from some policy model(s) behavior. In previous works, behavior policy is typically obtained using a single policy model. In this paper, as generalization, we define two kinds of F according how many policy models they take as input to get behavior. The first one, individual behavior mapping, is defined as mapping from a single model to behavior that widely used prior works, e-g., e-greedy and Boltzmann Strategy for discrete action space and Gaussian Strategy for continuous action space; And the second one, hybrid behavior mapping, is defined to map all policy models to a single behavior, ie., F(Ø,, _b ..... byy. The hybrid behavior mapping enables us to get hybrid behavior by combining all policies together, which provides a greater degree of freedom to acquire a larger behavior space. For any behavior mapping F parameterized by 2b, there exists family of behavior mappings Fv (F E D that hold the same parametrization form with Fap. where L R R is parameter set that contains all possible parameter ab.
described above, in our work, a behavior can be acquired by applying a behavior mapping F& some policy model(s). For the individual behavior mapping case, a behavior can be formulated as jee = Fup (o.n). which also the most used case in previous works. As for the hybrid behavior mapping case, behavior is formulated Me,H. + = Fup ( wherein @ H = (..,, bonn........u... ho is policy model set containing all policy models.
Behavior Formulation
Behavior Control Formulation Behavior control can be decoupled into sub-problems: 1) which behaviors can be selected for each actor at each training time, namely the behavior space construction. 2) how select proper behaviors, namely the behavior selection. Based on the behavior formulation, we can formulate these sub-problems:
Definition 3.1 (Behavior Space Construction). Considering the RL problem that behaviors μ are generated from some policy model(s). We can acquire family of realizable behaviors by applying family of behavior mappings Fv to these policy model(s). Define the set that contains all of these realizable behaviors the behavior space, which can be formulated
M (uon ( O, H. } for individual behavior mapping (HEHH F F }) for hybrid behavior mapping
(2)
Definition 3.2 (Behavior Selection). Behavior selection can be formulated as finding optimal selec- tion distribution PiMe.. n. some optimization target Lp, wherein Lp the optimization target of behavior selection:
select the behaviors μ from behavior space Me.H, H and maximizing
Pie
argmax Lp
PMo
3.2 BEHAVIOR SPACE CONSTRUCTION
In this section, we further simplify the equation?? 2 and discuss how to construct the behavior space. Assumption Assume all policy models share the same network structure, and h; can uniquely index a policy model Pe.h,, Then, Denh be abbreviated as Tb.
Unless otherwise specified, this paper, we assume Assumption holds. Under Assumption 1 the behavior space defined in equation caa be simplified as,
H. 2b HHn F ( h (1H F. (P E V for hybrid behavior mapping
)) for individual behavior mapping
MF
According equation 4 four factors need to be considered when constructing behavior space: the network structure Q the form of behavior mapping F. the parammer set H and the parameter set V. Many notable representation learning approaches have explored how to design the
4

---

Published as a conference paper at ICLR 2023
network structure (Chen et al. 2021 Inie al. 2021). ). but it is not the focus of our work. In this paper, we do not make any assumptions about the model structure, which means it can be applied to any model structure. Hence, there remains three factors, which will be discussed below.
For cases that behavior space constructed with individual behavior mappings, there are two things to be considered if one want to select specific behavior from the behavior space: the policy model D, and behavior mapping Fi.. Prior methods have tried to realize behavior control via selecting a policy model &h; from the population { ( Pny (See Proposition 1.. The main drawback of this approach is that only one policy model is considered to generate behavior, leaving other policy models in the population unused. this paper, we argue that we can tackle this problem via hybrid behavior mapping. wherein the hybrid behavior is generated based on all policy models.
In this paper, we only consider the case that all of the N policy models are used for behavior generating, Le., HH. Fu (H)). Now there is only one thing to be considered, i.e., the behavior mapping function Fg. and the behavior control problem will be transformed into the optimization of the behavior mapping (See Proposition 2). We also do not make any assumptions about the form of the mapping. As an example, one could acquire a hybrid behavior from all policy models via network distillation, parameter fusion, mixture models, etc.
3.3 BEHAVIOR SELECTION
According to equation 4 each behavior can be indexed by and sb for individual behavior mapping cases, and when the &b not learned for each actor, the behavior selection can be cast the selection of h (see Proposition M] As for the hybrid behavior mapping cases, since each behavior can be indexed by nb, the behavior selection can be cast into the selection of ub (see Proposition 2). Moreover, according to equation 3. there are two keys in behavior selection: 1) Optimization Target Lp. 2) The optimization algorithm to learn the selection distribution РMH,v and maximize Cp. In this section, we will discuss them sequentially.
Optimization Target
Two core factors have be considered for the optimization target: the diversity- measurement VTD (Eysenbach et al. 2019) and the value-based measurement VTV Parker-Holder et al. 2020). By integrating both, the optimization target can be formulated as:
Cp R + C- CC-Dpan D
(5)
-BnNPMa [VTV +c. VTP).
wherein, Rym-Pag and D Pu~PasH.v is the expectation of value and diversity of behavior p over the selection distribution РMH When Fip is unlearned and deterministic for each actor, behavior selection for each actor can simplified into the selection of the policy model:
Proposition 1 (Policy Model Selection). When Fv is deterministic and individual behavior mapping for each actor at each training step (wall-clock), e.g., Agent57, the behavior for each actor can be uniquely indexed by h, so equation 5 can be simplified into
Lp = En-PH [V'LV + VTT1].
(6)
where Ph is selection distribution of h E H {hi...., hx). For each actor, the behavior is generated from a selected policy model Tb. with a pre-defined behavior mapping Fwz
In Proposition]] I the behavior space size is controlled by the policy model population size (i.e., [H)). However, maintaining a large population of different policy models is contt-oonnumnnn.. Hence, we try control behaviors via optimizing the selection of behavior mappings:
Proposition 2 (Behavior Mapping Optimization). When all the policy models are used to generate each behavior, e.g., ue = F (ee. a) for single policy model cases or Hay F ( ....... Dox .us) for N policy models cases, each behavior can be uniquely indexed by F and equation 5 can be simplified into:
Lp = Ev-Pu VVTV + c. VTD]
where Ps a selection distribution of V.
In Proposition 2 the behavior space is majorly controlled by |1), which could be a continuous parameter space. Hence, larger behavior space be enabled.
5

---

Published as a conference paper at ICLR 2023
Selection Distribution Optimization Given the optimization target Lp, seek to find the optimal
behavior selection distribution Pj that maximizes Lp:
P PMH.. argmaxxp
argmaxCp
PMH
Ph
(8)
argmaxCp,
Pv
where (I) and (2) hold because have Proposition ! and can be solved with existing optimizers, e.g., evolutionary algorithm (Jaderberg et al. 2017). multi-arm bandits (MAB) (Badia et al. 2020a), etc.
respectively. This optimization problem
4 LBC-BM: A BOLTZMANN MIXTURE BASED IMPLEMENTATION FOR LBC
In this section, provide example of improving the behavior control of off- off-policy actor-critic methods (Espeholt et al. 2018) via optimizing the behavior mappings as Proposition? 2 practical design of hybrid behavior mapping, inducing an implementation of LBC, which we call Boltzmann Mixture based LBC, namely LBC-BM.. By choosing different H and V, we can obtain family of implementations of LBC-B.M with different behavior spaces (see Sec. 5.4).
We provide a
4.1 BOLTZMANN MIXTURE BASED BEHAVIOR SPACE CONSTRUCTION
In this section, we provide a general hybrid behavior mapping design including three sub-processes:
Generalized Policy Selection In Agent57, behavior control is achieved by selecting a single policy from the policy population at each iteration. Following this idea, we generalize the method to the case where multiple policies can be selected. More specifically, we introduce a importance weights vector describe how much each policy will contribute to the generated behavior, w [WI, WN],40 wi 2
0, E Wi = 1, where W; represents the importance of ith policy in the population (i.e., Pn:). In particular, if w is one-hot vector, i.e., ži E {1, ..., NJ. then the policy selection becomes a single policy selection as Proposition [ Therefore, it can seen generalization of single policy selection, and call this process generalized policy selection.
...,
L;Vj E (1,2, .... N} * i, wy = O,
Policy-Wiss Entropy Control In our work, we propose to use entropy control (which is typically rule-based controlled in previous works) to make better trade-off between exploration and exploita- tion. For a policy model Ln, from the population, we will apply entropy control function fr. (-), i.e.
Th..7: = fr. (..)). where hh.,. is the new policy after entropy control, and fr. (.) is parameterized by Ti. Here we should note that the entropy of all the policies from the population is controlled in policy-wise manner. Thus there would be a set entropy control functions to be considered, which is parameterized by [TI,..,,NN TN).
Behavior Distillation from Multiple Policies Different from previous methods where only one policy is used to generate the behavior, in our approach, we combine N policies [Th,.: ...., Thy mx together with their importance weights w = [w],...,w]] WN). Specially, in order to make full use of these policies according to their importance, we introduce behavior distillation function g which takes both the policies and importance weights as input, i.e., HH.T.w = g(Thi.t1 ..., Thx.7s w). The distillation function g(, w) can be implemented in different ways, e.g., knowledge distillation (supervised learning), parameters fusion, etc. In conclusion, the behavior space can be constructed as,
MH = (g (Fr. (Pm.).... (P wI.,., WN) x }
(hi,... Note that this a general approach which can be applied to different tasks and algorithms by simply selecting different entropy control function f-. (-) and behavior distillation function g(-, w). As an example, for Atari task, we model the policy as a Boltzmann distribution, i.e., Th.:: (als) = (a)n(()))) E.. Th (a')a) 1s), where E (0, oo). The entropy can thus be controlled by controlling the temperature. As for the behavior distillation function, we are inspired by the behavior design of GDI, which takes a weighted sum of two softmax distributions derived from two advantage functions. We can further extend this approach
wherein = ( (TI...., TN. w...... ".......,);;; wx)).

---

Published as a conference paper at ICLR 2023
to the case to do combination across different policies, i.e., H Σ w WI This formula actually a form of mixture model, where the importance weights play the role of mixture weights of the mixture model. Then the behavior space becomes,
N
MH {HH
w, (Sh. (2b)) E }
(10)
4.2 MAB BASED BEIIAVIR SELECTION
According to Proposition 2 the behavior selection over behavior space 10 can be simplified to the selection of tb In this paper, use MAB-based meta-controller to select Tb L. Since V is continuous multidimensional space, we discretize V into regions (VI, Vx and each region corresponds arm of MAB. At the beginning of trajectory i, I-th actor will use MAB sample region V indexed by arm k according to Pv softmax(Scoree
Beurce,
We adopt UCB
score as Scorey, = Vi + C. log(l + Exen No,)
to tackle the ceward-diversity trade-off problem
1+No,
in equation 7 (Garivier & Moulines 2011). Ng. means the number of the visit of V indexed by arm k. Ve, is calculated by the expectation of the undiscounted episodic returns to measure the value of each Le, and the UCB item is used to avoid selecting the same arm repeatedly and ensure sufficient diverse behavior mappings can be selected boost the behavior diversity. After an V is sampled, zb will be uniformly sampled from x corresponding to a behavior mapping F&.. With Frg. we can obtain a behavior Ue according to equation 10 Then, the I-th actor acts μ4 to obtain a trajectory т; and the undiscounted episodic return G., then G3 is used to update the reward model Vv, of region R indexed by arm k. As for the nonstationary problem, we are inspired from GDI, which ensembles several MAB with different learning rates and discretization accuracy. We can extend to handle the nonstationary problem by jointly training a population of bandits from very exploratory purely exploitative (i.e., different of the UCB item, similar to the policy population of Agent57). Moreover, we will periodically replace the members of the MAB population to ease the nonstationary problem further. More details of implementations of MAB can be found in App. E Moreover, the mechanism of the UCB item for behavior control has not been widely studied in prior works, and we will demonstrate how boosts behavior diversity in App. K.
EXPERIMENT
In this section, we design our experiment answer the following questions:
Whether our methods can outperform prior SOTA RL algorithms in both sample efficiency
and final performance in Atari IB Benchmarks (See Sec. 5.2 and Figs. 3
Can our methods adaptively adjust the exploration exxporratnnnnoo trade-off (See Figs.
How enlarge narrow down the behavior space? What is the performance of methods with different behavior spaces (See Sec. 55.4
How much performance will be degraded without proper behavior selection (See Figs. 5)
5.1 EXPERIMENTAL SETUP
5.1.1 EXPERIMENTAL DETAILS
We conduct our experiments in ALE (Bellemare et al. 2013). The standard pre-processing settings of Atari are identical to those of Agent57 (Badia et al. (020a). and related parameters have been concluded in App. We employ a separate evaluation process to record scores continuously. We record the undiscounted episodic returns averaged over five seeds using a windowed mean over 32 episodes. To avoid any issues that aggregated metrics may have, App. pprovides full learning curves for all games and detailed comparison tables of raw and normalized scores. Apart from the mean and median HNS, we also report how many human worlds records our agents have broken to emphasize the superhuman performance of methods. For more experimental details, App.
7

---

Published as a conference paper at ICLR 2023
Human World Records Breakthrough
Mean Humon Nomalzzd Score
Modian Hu man Normalized Score
10000.0000
20
5000.
1500, 0%
1000,
10
4000. 0%
0.0%
0.0%
0.9
0.2
9.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Ours@BB
Muzerog208/20201
Agent576 100BB12011
MEME s 18(2022)
CDI- 200-H
Co-Explonegi1000 08(20211
Figure 3: The learning curves in Atari. Curves smoothed with moving average over 5 points.
DemonAttack
1n5
jamesbond
Phoenix
1eG
2.5
2.0
0.8
2.0
Scores
1.5
1.5
0.4
0.5
0.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.0
1.0
0.0
52
0.4
0.5
0.0
1.0
Training Scale (Frames)
1c9
lcb
1c9
Scores
Entropy
Figure 4: Behavior entropy and scores curve across training for different games where we achieved unprecedented performance. The names of the axes are the same as that of the leftmost figure.
5.1.2 IMPLEMENTATION DETAILS
We jointly train three polices, and each policy can be indexed by the hyper-parameters h; (r, RS,), wherein RS, is reward shaping method (Badia et al. (020a)). and 7 is the discounted factor. Each policy model Pb, adopts the dueling network structure (Wang et al. (016)) where
in Ab. = Qh, Vn,, More details of the network structure can be found in App. L To
correct for harmful discrepancy of off-policy learning, we adopt V-Trace (Espeholt et al. 2018 and ReTrace (Munos et al. (016) to learn V1, and Qn. respectively. The policy is learned by policy gradient (Schulman et al. 2017). Based on equation [0] we could build behavior space with
hybrid mapping as MH
(1HHv Di softmax, (#h,, ).. wherein H {h1, b2 ha). ( = (T1,W1. T2, w2, T3, (g) W3)) T. E (0, (0,T
Wj 1}. The behavior selection is achieved by MAB described Sec. [4.2] and more details can see App. E Finally, we could obtain an implementation of LBC-BM, which is our main algorithm. The target policy for AT and A in
GDI-H3 3 is the same, while in our work the target policy for A7 æ, softmax(AA)..
5.2 SUMMARY RESULTS
Results on Atari Benchmark The aggregated results across games are reported in Figs. 3 Among the algorithms with superb final performance, our agents achieve the best mean HNS and surpass the most human world records across 57 games of the Atari benchmark with relatively minimal training frames, leading to the best learning efficiency. Noting that Agent77 reported the maximum scores across training as the final score, and if report performance in the same manner, median is 1934%, which is higher than Agent57 and demonstrates our superior performance.
Discussion of Results With LBC, we can understand the mechanisms underlying the performance of " more clearly: i) GDI-H has high-capacity behavior space and a meta-controller to optimize the behavior selection ii) only single target policy learned, which enables stable learning and fast converge (See the case study of KL divergence in App. N.. Compared to GDI-H², to ensure the behavior space will not degenerate, LBC maintains a population of diverse policies and, a price, sacrifices some sample efficiency. Nevertheless, LBC can continuously maintain significantly larger behavior space with hybrid behavior mapping, which enables RI. agents to continuously explore and get improvement.
5.3
CASE STUDY: BEHAVIOR CONTROL
To further explore the mechanisms underlying the success of behavior control our method, we adopt case study to showcase our control process of behaviors. As shown in Figs. 4 in most tasks,
8

---

Published as a conference paper at ICLR 2023
Scaled Vtan iB55
Scales Median IFC
320 m9
120.0%
Scaled HINS
30
19%
52%
PUM
50 18.0%
59%
66
10.39
11.0%
50.0%
0.9
Mun Algeriam
Recesing
Pellacin und
Pandem cction
0.0%
Figure 5: Ablation Results. All the results are scaled by the main algorithm to improve readability.
agents prefer exploratory behaviors first (i.e., high stochasticity policies with high entropy), and, as training progresses, the agents shift into producing experience from more exploitative behaviors. On the verge of peaking, the entropy of the behaviors could be maintained at certain level (task-wise) instead of collapsing swiftly to zero to avoid converging prematurely to sub-optimal policies.
5.4 ABLATION STUDY
In this section, we investigate several properties of our method. For more details, App.
Behavior Space Decomposition To explore the effect of different behavior spaces, decompose the behavior space of our main algorithm via reducing H and T
Reducing H. When we set all the policy models our main algorithm the same, the behavior space transforms from F(Pn. Ph Phg) into F( Dn, Pn,) ). H degenerates from {h1, h2, hs} into {h1}. We can obtain a control group with a smaller behavior space by reducing H.
Reddcigg H and V. Based on the control group reducing H. we can further reduce v to further narrow down the behavior space. Specially, we can directly adopt a individual behavior mapping to build the behavior space as MH. v = {pg = softmax, (P,,) },, where T degenerates from {wi, W2, w3, Ti, T2, T3) to {T) and H = (hr). Then, we can obtain a control group with the smallest behavior space by reducing H and V
The performance these methods is illustrated Figs. 5 and from left to right, the behavior space of the first three algorithms decreases in turn (According to Corollary I App. []. It is evident that narrowing the behavior space via reducing H or y will degrade the performance. On the contrary, the performance can be boosted by enlarging the behavior space, which could be promising way to improve the performance of existing methods.
Behavior Selection To highlight the importance of an appropriate behavior selection, we replace the meta-controller of our main algorithm with a random selection. The ablation results are illustrated in Figs. []] from which it is evident that, with the same behavior space, not learning an appropriate selection distribution of behaviors will significantly degrade the performance. We conduct a t-SNE analysis in App. K.3 to demonstrate that our methods can acquire more diverse behaviors than the control group with pre-defined behavior mapping. Another ablation study that removed the UCB item has been conducted in App. K.3 to demonstrate the behavior diversity may be boosted by the UCB item, which can encourage the agents to select more different behavior mappings.
CONCLUSION
We present the first deep reinforcement learning agent break human world records in Atari using only IB training frames. To achieve this, we propose a general framework called LBC, which enables significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies, and constructs a unified learnable process for behavior selection. We introduced LBC into off-policy actor-critic methods and obtained a family of implementations. of experiments on Atari have been conducted to demonstrate the effectiveness of our methods empirically. Apart from the full results, we do detailed ablation studies to examine the effectiveness of the proposed components. While there many improvements and extensions to be explored going forward, we believe that the ability of LBC to enhance the control process of behaviors results in powerful platform propel future research.
large number

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is majorly supported by the National Key R&D Program of China (Grant Number 20212D01 10400). In addition, this work is partly supported by the National Natural Science Foundation of China under Grant 62171248, and the R&D Program of Shenzhen under Grant JCY/20220188100112200
We are very grateful for the careful reading and insightful reviews of meta-reviewers and reviewers.
REPRODUCIBILITY STATEMENT
Open-sourced code will be implemented with Mindspore MS 2022 and released on our website
REFERENCES
Mindspore, 2022. URL
cn/
Software available from
https//www.mmndssorreccm
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In P Proceedings of the 37th International Conference on Machine Learning, ICML, 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 507-517. URL http:htt:///pootte
PMLR, 2020a.
ngs ml r. www.ppp.ssse..... www.v11/baaaa 19/badi a20a htmll
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. URI. htt.sttts:://
http:::://///eeeevveeeeeeeee /ttp:ttttsss//peereeeee n forum diw.ummmmdssyyeeeee x
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253-279, 2013. doi:
10.1613/jair.9912. [tRLhhtps: http://oo..rro///ooo 100 1113 jair. 3912
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems,
2021.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssuun Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1406-1415. PMLR, 2018. URL http: /proccedingg mlr. .ww.sresssseeemmmmmmmmm.mm..else
prwss / v800 http///80///eeeeee http//ppphhtt.a....e html
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL net forum i d33xx63 RRqFm
htttps://openrev
Jiajun Fan and Changnan Xiao. Generalized data distribution iteration. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Muchine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 6103 6103-6184. PMLR, 2022. URL ltttps:/pproeedii www.::tssss......am.em.mmmmreeee
mlr www.sprssssseeee /ttee////////////2..22ccccc html
fan22c.
Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In Jyrki Kivinen, Csaba Szepesvári, Esko Ukkonen, and Thomas Zeugmann (eds.), Algorithmic Learning Theory- 22nd International Conference, ALT 2011, Espoo, Finland, Oc- tober 5-7, 2011 Proceedings, volume 6925 of Lecture Notes in Computer Science, pp. 188.
10

---

Published as a conference paper at ICLR 2023
Springer, 2011. doi: 10.1007799783--222244111 URL https:////ol
Danijar Hafner, Timothy P. Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In 9th International Conference on Learning Representations, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttpt::/opeereeeeeee
net/ seet/forum?id=Dobbwyy
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.). Proceedings of the Phirty-Seconi AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3215-3222. AAAI Press, 2018. URL https: www. http://wwwwaa.orr..... ocs/ index.
Ronald A Howard. Dynamic programming and markov processes. John Wiley, 1960.
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear trans- formers with recurrent fast weight programmers. Advances in Neural Information Processing
Systems, 34:7703-7777,, 2021.
Max Jaderberg. Valentin Dalibard, Simon Osindero. Wojeiech M. Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, abs/1711 1.0996,,, 2017.
URL http: org/ abs /tabs/1111 09846
Steven Kapturowski, Georg Ostrovski. John Quan, Rémi Munos. and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL/Bttts://opeeeeeee
net. fett/forummd=aaa d=r jA
Steven Kapturowski, Victor Campos, Ray Jiang, Nemanja Rakićević, Hado Hasselt, Charles Blundell, and Adrià Puigdomènech Badia. Human-level atari 200x faster. CoRR, abs/2209.07550, 2022. URL http:tts:: anttps://arxiv. org abs 2209. 07550
Jinke Li, Ruonan Rao, and Jun Shi. Learning to trade with deep actor critic methods. In 2018 11th International Symposium Computational Intelligence and Design (ISCID), volume 2, pp. 66-71. IEEE, 2018
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. J. Artif. Intell. Res., 61:523-562, 2018. doi: 10. 10.1613/jair.5699 URL
httpps://oi.. http:/di..oggg//// 1613 5699
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep re- inforcement learning. Nat., 518(7540):529-555 533, 2015. doi: 10. 10.3038/nature 14236. URL https://doi. /t0g 1038 htt///nauree4222
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off- policy reinforcement learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1046-1054 1054, 2016. hash/c33900 tathhh3392e998cccco 2bd 3ac.aaaa......aaae1..o...aaa3333
http:/uru.ppppcccaaaaa/////2111 pappp/20ee/////////// Abstract. htmll
Jack Parker-Holder, Aldo Pacchiano, Krzysztof Marcin Choromanski, and Stephen J.
Roberts. Effective diversity in population based reinforcement learning. In Hugo
11

---

Published as a conference paper at ICLR 2023
Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Confer- ence Neural Information Processing Systems 2020, NeurIPS 2020. December 6-12,
2020, virtual, 2020. URL htt.:: httpttssstpppccpsppiitt neurips www.eris..ccppp.eeeeee https//cc/cppp///0///e/aaaaa
Sepp Hochreiter; Jürgen Schmidhuber. Long short-term memory. Neural Computation. 1997.
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-pollcy actor-critic with shared experi- ence replay. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Re- search, pp. 8545-8554 PMLR, 2020. URL schmitt20a.1 html
mlr. http://www......../se....mm.. press/v119/
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1889-1897. JMLR.org, 2015. URL http:///// /ttp:///roceeeeeeoe ngs m wtt..mm........//..... wtress//v3/// schu 1an man 15. html
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL htttp://axii... org /bs/
1707. 06347
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-6 URL https
www. worldcat. ottp.ccc////299332000000
Marin Toromanoff, Émilie Wirbel, and Fabien Moutarde. Is deep reinforcement learning really superhuman on atari? CoRR, abs/1908.0460 04683, 2019. /BLL/tttp///xxx..oor///aa/
04683
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1995-2003. 1995-20033JMMIRR.0000 Jor.. 2016. URL httt:::/proceee..ii....
mlr. wwwpp///./wwaaeeeeep /ttp://////////a/1l..........
Ronald Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229-256, 1992. doi: 10. 10.1007/BF00999266 URL http::tps:.
12


Reviews:
[{"review_id": "hCKz6LxJdw-", "reviewer": "Reviewer_Spxv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers the exploration problem in RL. Building upon prior work on population based training, this work aims to increase behavior space without increasing the population size in order to improve sample efficiency. \nThe method is hybrid behavior mapping which learns mapping functions to get sampling behaviors from policies. Experimental results show this method achieves new state-of-the-art performance on the ALE benchmark. \n", "summary_of_the_review": "This paper proposes a novel and effective exploration method that achieves good results on Atari games. \nThere are some actionable non-critical issues that I hope could be addressed in the next version. \n", "strength_and_weaknesses": "- Paper seems well written, key concepts are explained, and claims are supported. \n\n- The results are very good. As many games of the ALE benchmark require exploration, the significantly improved results show that the method is highly effective at exploring. I'm impressed by the effectiveness of this approach. \n\n- The behavior construction and selection method is novel to me although seems too complicated to get it to work well in practice. But the authors provide important hyperparameters and pseudo algorithms. I think this is a good work that demonstrates advances of exploration leads to unprecedented performance on Atari games. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KGEMlgQtpw", "reviewer": "Reviewer_PtKB", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission proposes a mechanism that it calls learnable behavioral control (LBC). LBC aggregates the policies of a population into a single policy using a bandit-based meta controller. The submission shows that LBC achieves very strong results on Atari.", "summary_of_the_review": "The main methodological contribution of the submission has already been proposed by existing work, which was first arxived over a year ago and recently published in ICML; furthermore, this existing work achieves similar empirical results to the submission. Because the submission does not acknowledge this existing work, and instead presents its methodology as a novel contribution, I do not think it is acceptable in its current form.\n\n---\n\nAfter discussion with the authors and revisions of the submission, I raised my score.", "strength_and_weaknesses": "# Main Concern\n\nThis situation is unfortunate. **The submission is excellent** (simple approach, great empirical results, nice writing), and the authors clearly spent a large amount of time to make it as good as it is. **But there is existing work that was arxived (https://arxiv.org/abs/2106.06232v1) more than a year ago and recently published in ICML (https://proceedings.mlr.press/v162/fan22c) that proposed essentially the same methodology and achieves similar empirical results.** I will refer to this existing work as GDI.\n\n### Irresponsibility of the omission\n\nGDI was published by a relatively lesser known research group among the reinforcement learning community and has not received as much attention as it deserves, so I can see how the authors may not have known about it. However, even if this omission was unintentional, I feel it was still irresponsible: When making a submission with claims of novelty, the submitters are staking their credibility on the claims of novelty being factual; the submitters appear to have made such claims without having done due diligence.\n\n### Work is not concurrent\n\nI do not think the submission can claim concurrency with GDI since GDI has been on arxiv since June 2021.\n\n### Similarity of LBC and GDI implementations\n\nIn their implemented algorithms, both works use continuous MAB algorithms to optimize the temperatures of softmax functions and mixture parameters over policies. There are many instantiation-level differences, but I feel that the implementation in the submission can meaningfully be described as an instance of GDI (and vice versa). If the authors disagree with this characterization, I ask that they provide detailed evidence for their claims.\n\n### Similarity of empirical results\n\nAcross various metrics, the submission reports results (e.g., 10077.52% mean human normalized score and 24 human world records) that are a bit better than GDI (e.g., 9620.98% mean human normalized score and 22 human world records) after 5 times more samples (1B for LBC vs. 200M for GDI). It is hard to make a precise comparison, but, if anything, I might say that GDI's empirical results are a bit more impressive, given that it achieved almost as good results with 5x fewer samples.\n\n### Value of submission\n\nAlthough I feel that there is not algorithmic novelty and that the empirical results are not substantially better, I think the submission as is offers value in the sense that it is much more clearly written than GDI. Since the idea behind GDI/LBC is, in my view, quite important for modern RL but not currently widely understood by the community, I think it deserves to have a paper where it is explained more clearly than it was in GDI. That said, I do not think it would be fair to the authors of GDI to re-brand the same idea under a different name and merely mention GDI as related work.\n\n## Other thoughts\n\n### Measurement notation\n\nThe submission chooses to break the optimization target into two terms that it calls a diversity-based measurement and a value-based measurement. I am not sure what purpose doing this serves, as the submission never actually comes back to this point when discussing the actual algorithm it implemented. I think the best way to resolve the issue would be to use a single symbol V to refer to a measurement and note in the text that this measurement may be decomposable into some weighted combination of diversity and value. Alternatively, if the authors do not want to do that, I think they should at least describe the algorithm that they implemented in terms of these two measurements at some point in the submission so that readers are able to make the connection.\n\n### \"Goal-directed\" terminology\n\nThe submission repeatedly uses the terminology \"goal-directed\". To me, this feels like unnecessary jargon. The submission suggests that Agent57 uses a goal-directed meta controller, but I could not find any usage of \"goal-directed\" in the Agent57 paper. If the submission does not want to remove this terminology, I think the submission should at least make it clear exactly what it means by \"goal-directed\" and how this usage departs from or is the same as that in existing reinforcement learning literature.\n\n### EfficientZero final performance\n\nThe submission states:\n> From Figs. 3, EfficientZero (Ye et al., 2021) achieves remarkable learning efficiency in smaller data volumes but fails to obtain comparable final performance as other SOTA methods like Agent57 or Muzero.\n\nThis statement may be superficially true, but it is misleading to readers because it is comparing apples to oranges. The numbers reported in the submission for LBC use **four orders of magnitude!** more samples than EfficientZero. It is entirely plausible that the final performance of EfficientZero would be comparable to that of MuZero given the same number of samples. Thus, saying that it does not achieve the same \"final performance\" does not strike me as appropriate.\n\nOn this note, I do not think it is even necessary to include EfficientZero(100k) in these plots. It is not a relevant baseline for the submission due to the drastically different sample usages.\n\n## Closing Thoughts\n\nAs articulated above, I feel that the submission is a great piece of work and, even given existing work, offers value to the community. I hope that, through conversing with the authors, we will be able to agree on appropriate revisions that will make the submission acceptance-worthy. However, I anticipate that these changes may be quite substantial -- i.e., going well beyond simply mentioning GDI as related work. I think they would likely require the submission to be re-written from the perspective of providing additional clarity and empirical confirmation for GDI, rather than that of proposing a novel methodology. To me, it is important that the authors of GDI not be denied credit for having had an important idea simply because they are not well-known RL researchers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B9tOhQqgQt", "reviewer": "Reviewer_qVG5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new technique for enhancing population-based RL (PBRL), and shows state-of-the-art results on the Atari Learning Environment (ALE). The technique expands upon previous work in PBRL, which used a multi-armed bandit (MAB) meta-controller to select the best policy from a population of policies. This paper also uses the MAB approach, but proposes to combine multiple policies from the population by summing their softmax outputs, creating a mixture model. This allows them to achieve the highest number of world records in Atari of any previous algorithm. ", "summary_of_the_review": "In summary, the paper should be accepted given that the results are impressive and are of significant interest to the community. However, the organization and technical communication in the paper could be significantly improved to enhance its potential impact and reproducibility. \n\nIf it were not for the issues with technical communication pointed out above, I would increase my score to indicate that the paper should be highlighted at the conference. ", "strength_and_weaknesses": "A key strength of the paper is in the significance of the results. Setting a new state-of-the-art in Atari is a significant accomplishment, and will be of interest to the RL community. \n\nA weakness of the paper is in a lack of clarity around key technical details, combined with a lot of redundant and unnecessary explanation characterizing the space of possible techniques. The paper spends 3-4 pages (p. 3-6) on redundant and repetitive explanations of behavior space vs. behavior mapping. Many parts of these pages are repeated (for example, the \"Behavior Formulation\" paragraph on p.4 is fully subsumed in other parts of the section and does not need to be repeated). However, the paper spends relatively little time justifying or giving intuition for why the central contribution of the paper (Eq. 9) is the right approach. Most importantly, the explanation of how the MAB behavior selection interacts with Eq. 9 is left very unclear. It would seem that to apply Eq. 9 the MAB would need to select N policies, but this is not stated in the text, which instead says that Eq. 9 will be applied to a single $\\Phi_k$, which does not make sense. This lack of precision makes the paper hard to replicate. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%