Adapting Behaviour for Learning Progress
Authors: Tom Schaul
Diana Borsa
David Ding
David Szepesvari
Georg Ostrovski
Will Dabney
Simon Osindero
Venue:  (2020)
Keywords: adaptation
behaviour
reinforcement learning
modulated behaviour
exploration
deep reinforcement learning

Abstract: Determining what experience to generate to best facilitate learning (i.e. exploration) is one of the distinguishing features and open challenges in reinforcement learning. The advent of distributed agents that interact with parallel instances of the environment has enabled larger scale and greater flexibility, but has not removed the need to tune or tailor exploration to the task, because the ideal data for the learning algorithm necessarily depends on its process of learning. We propose to dynamically adapt the data generation by using a non-stationary multi-armed bandit to optimize a proxy of the learning progress. The data distribution is controlled via modulating multiple parameters of the policy (such as stochasticity, consistency or optimism) without significant overhead. The adaptation speed of the bandit can be increased by exploiting the factored modulation structure. We demonstrate on a suite of Atari 2600 games how this unified approach produces results comparable to per-task tuning at a fraction of the cost.

Under review as a conference paper at ICLR 2020
ADAPTING BEHAVIOUR FOR LEARNING PROGRESS
Anonymous authors
Paper under double-blind review
ABSTRACT
Determining what experience to generate to best facilitate learning (i.e. explo- ration) is one of the distinguishing features and open challenges in reinforcement learning. The advent of distributed agents that interact with parallel instances of the environment has enabled larger scale and greater flexibility, but has not re- moved the need to tune or tailor exploration to the task, because the ideal data for the learning algorithm necessarily depends on its process of learning. We propose to dynamically adapt the data generation by using non-stationary multi-armed bandit to optimize a proxy of the learning progress. The data distribution is con- trolled via modulating multiple parameters of the policy (such as stochasticity, consistency or optimism) without significant overhead. The adaptation speed of the bandit can be increased by exploiting the factored modulation structure. We demonstrate on a suite of Atari 2600 games how this unified approach produces results comparable per-task tuning at fraction of the cost.
INTRODUCTION
Reinforcement learning (RI.) a general formalism modelling sequential decision making. It as- pires to be broadly applicable, making minimal assumptions about the task at hand and reducing the need for prior knowledge. By learning behaviour from scratch, it has the potential to surpass human expertise or tackle complex domains where human intuition not applicable. In practice, however, generality is often traded for performance and efficiency, with RL practitioners tuning algorithms, architectures and syyer--arameee that the resulting methods can be brittle, or difficult to reliably reproduce (Nagaaajan et al. 2018).
to the task at hand (Hessel et al. 2019). A side-effect of this
Exploration is one of the main aspects commonly designed or tuned specifically for the task be- ing solved. Previous work has shown that large sample-efficinncy gains are possible, for example, when the exploratory behaviour's level of stochasticity is adjusted to the environment's hazard rate (García & Fernández] (2015), or when an appropriate prior is used in large action spaces (Dulac- Arnold et al. 20155; Czarnecki et al., 2018 Vinyals et al.] 2019). Ideal exploration in the presence of function approximation should be agent-centred. It ought to focus more generating data that supports the learning of agent at its current parameters 0. rather than making progress on objective measurements of information gathering. A useful notion here is learning progress (LP), defined as the improvement of the learned policy Ta (Section 3)
The agent's source of data is its behaviour policy. Beyond the conventional RL setting of a single stream of experience, distributed agents that interact with parallel copies of the environment can have multiple such data sources (Horgan al. 2018g. In this paper, we restrict ourselves to the setting where all behaviour policies are derived from single set of learned parameters 8, for ex- ample when f parameterises an action-value function Qs. Consequently the behaviour policies are given by π (Qo, =). where each modulation leads to meaningfully different behaviour: This can be guaranteed is semantic (e.g. degree of stochasticity) and consistent across multiple time-steps. The latter is achieved by holding fixed throughout each episode (Section) 2).
We propose to estimate a proxy that is indicative of future learning progress, f() (Section separately for each modulation and to adapt the distribution over modulations maximize J. using a non-stationary multi-armed bandit that can exploit the factored structure of the modulations (Section 4). Figure [ shows diagram of all these components. This results in an autonomous adaptation of behaviour to the agent's 's stage of learning (Section 5). varying across tasks and across time, and reducing the need for hyper-paramest tuning.

---

Under review as a conference paper at ICLR 2020
sampled modulation
x(a|s,2)
environment
replay
bandit
episode trajectories
fitness Sz(=)
Qs(
learner
train
Figure The overall architecture is based on a distributed deep RL agent (black). addition, modulations are sampled once per episode and modulate the behaviour policy. A bandit (purple) adapts the distribution over z based on fitness fz(2) that approximates learning progress (cyan).
MODULATED BEHAVIOUR
As usual in RL, the objective of the agent is find policy that maximises the 7-discounted expected return G, = 00 7 Re+i- where R; is the reward obtained during the transition from time t 1. common way to address this problem to use methods that compute the action- value function Q" given by (2* ( a) = EE a), i.c. the expected return when starting from state 8 with action a and then following
(Puterman 1994).
A richer representation of QF that aims to capture more information about the underlying distribu- tion of G, has been proposed by |Bellemare al. (2017), and extended by Dabneyy al.] (2018). Instead of approximating only the mean of the return distribution,
approximate discrete set of 1 }) such that IP(Q" S 4x) = v. Outside the benefits in performance and representation learning (Such et al. 2018), these quantile estimates pro- vide a way of inducing risk-sensitiv behaviour. We approximate all qu using a single deep neural network with parameters 0, and define the evaluation policy the greedy with respect the mean estimate:
n quantile values 4v (where v E ( 2n ..... 2m
To (-(+s) arg max Σ (s.
The behaviour policy is the central element of exploration: it generates exploratory behaviour (and experience therefrom) which used learn тh: ideally in such way to reduce the total amount of experience required to achieve good performance. Instead of a single monolithic behaviour policy, we propose use modulated policy support parameterized variation. Its modulations should satisfy the following criteria: they need to (i) be impactful, having direct and meaningful effect on generated behaviour; (ii) have small dimensionality, as to quickly adapt to the needs of the learning algorithm, and interpretable semantics to ease the choice of viable ranges and initialisation; and (iii) be frugal, in the sense that they are relatively simple and computationally inexpensive to apply. In this work, we consider five concrete types of such modulations:
Temperature: a Boltzmann softmax policy based on action-logits, modulated by temperature, T.
Flat stochasticity: with probability the agent ignores the action distribution produced by the softmax, and samples an action uniformly at random (e-greedy).
Per-action biases: action-logit offsets, b. to bias the agent to prefer some actions.
Action-repeat probability: with probability p. the previous action repeated (Machado et al., 2017 This produces chains of repeated actions with expected length T
Optimism: as the value function is represented by quantiles qu. the aggregate estimate Q. can be parameterised by an optimism exponent w, such that w = recovers the default flat average, while positive values of w imply optimism and negative ones pessimism. When near risk-neutral, our simple risk measure produces qualitatively similar transforms to those Wang (2000).
We combine the above modulations produce the overall z-modulated policy
+ (Q.a()))))))
+ é(1 - p)
<als, 2) = (1 e)(1 p)
+ plaama
EeeeettQuuenn (
A
2

---

Under review as a conference paper at ICLR 2020
where 2 = b,,,, Il is the indicator function, and the optimism-aggregaun value is
Qu Ev wa qu
Ev
Now that the behaviour policy be modulated, the following two sections discuss the criteria and mechanisms for choosing modulations 2.
3 EXPLORATION & THE EFFECTIVE ACQUISITION OF INFORMATION
A key component of a successful reinforcement learning algorithm is the ability to acquire expe- rience (information) that allows it make expeditious progress towards its objective of learning to act in the environment in such way as to optimise returns over the relevant (potentially dis- counted) horizon. The types of experience that most benefit an agent's 's ultimate performance may differ qualitatively throughout the course of learning the beginning of training often enough does not carry over to the end, as illustrated by the analysis in Figure However, this analysis conducted in hindsight, and in general how to generate such experience optimally - optimal exploration any environment - remains an open problem.
behaviour modulatin that is beneficial in
One approach is to require exploration to be in service of the agent's 8 future learning progress (LP), and to optimise this quantity during learning. Although there are multiple ways of defining learning progress, in this work we opted for task-related measure, namely the improvement of the policy in terms of expected return. This choice of measure corresponds to the local steepness of the learning curve of the evaluation policy no.
LPA(A0) = E [V*: 20 (80) - V (80)),
where the expectation is over start states so, the value (s) Ex Ralso s] is the 7- discounted return one would expected to obtain, starting in state s and following policy π afterwards, and Ae is the change in the agent's 's parameters. Note that this is still limited criterion, as it is myopic and might be prone to local optima.
E,
As prefaced in the last section, our goal here is to define mechanism that can switch between different behaviour modulations depending on which of them seems most promising at this point in the training process. Thus in order to adapt the distribution over modulations z, we want to assess the expected LP when learning from data generated according to z-modulated behaviour:
(2)
[LP,(A((T,t))),
TNiTo (+)
with A0(T, () the weight-change of learning from trajectory 7 at time L. This a subjective utility measure, quantifying how useful is for particular learning algorithm, this stage in training.
Proxies for learning progress: While Lz((z) is simple and clear progress metric, is not readily available during training, so that in practice, a proxy fimess fz( (2) needs be used. A key practical challenge is to construct St from inexpensively measurable proxies, in a way that sufficiently informative to effectively adapt the distribution over z, while being robust noise, approximation error, state distribution shift and mismatch between the proxies and learning progress. The ideal choice of f(2) is a matter of empirical study, and this paper only scratches the surface on this topic.
After some initial experimentation, we opted for the simple proxy of empirical (undiscounted) episodic return: fi(2) =
Ea,nr(Qu, fi(=)==En<<( R, This is trivial to estimate, but it departs from LP(2) in a number of ways. First, it does not contain learner-subjective information, but this partly mitigated through the joint use of with prioritized replay (see Section (.I) that over-samples high error experience. Another potential mechanism by which the episodic return can be indicative of future learning is because an improved policy tends to be preceded by some higher-return episodes in general, there is a lag between best-seen performance and reliably reproducing it. Second, the fitness is based on absolute returns not differences in returns suggested by Equation this makes no difference to the relative orderings of z (and the resulting probabilities induced by the bandit), but it has the benefit that the non-stationarity takes different form: difference- based metric will
3

---

Under review as a conference paper at ICLR 2020
stationary behaviour policios
non-stationary setup
expacted reward
expeected reward
10
10"
other flxed
fxed unitorm
banett (factored)
2/tg) bansit (lat))
LPtx) bandit
2000
6000
10000
episodes
ep episodes
Figure 2: LavaWorld experiments, with 31 distinct modulations 2. Left: If the candidate behaviours stationary (not affected by learning), the bandit nearly recovers the performance of the best fixed choice. The factored variant (blue) is more effective than a discretization (green). Using the true LP(=) instead of a less informative proxy (red) has a distinct effect: in the stationary sparse-reward case this makes the bandit equivalent to uniform sampling. Right: a non-stationary setting where the modulated Q-values are learned over time, the dynamics with ideal fitness again comes close to the best fixed choice, while uniform or proxy-based variants struggle more (see Appendix B
complex, and even the bandit
appear stationary if the policy performance keeps increasing at a steady rate, but such a policy must be changing significantly to achieve that progress, and therefore the selection mechanism should keep revisiting other modulations. In contrast, our absolute fitness naturally has this effect when paired with non-stationary bandit, described in the section.
NON-STATIONARY BANDIT TAILORED TO LEARNING PROGRESS
The most effective modulation scheme may differ throughout the course of learning. Instead of applying a single fixed modulation or fixed blend, we propose an adaptive scheme, in which the choice of modulation is dynamically based learning progress. The adaptation process is based on a non-stationary multi-armed bandit (Besbes et al. 2014: Raj & Kalyani (017]) where each arm corresponds behaviour modulation z. The non -ta-stationarityy reflects the nature the learning progress LP, (z) which depends on the time in training through the parameters A,.
Because non- non-stationarity. the core challenge for such bandit is identify good modulation arms quickly, while only having access to noisy, indirect proxy fz(z) of the quantity of interest (z) However, our setting also presents an unusual advantage: the bandit does not need to identify the best 2, as in practice it suffices to spread probability among all arms that produce reasonably useful experience for learning.
Concretely, our bandit samples modulation {z....., #x} according to the probability that it results in higher than usual fitness (measured as the mean over a recent length-h window):
t-1
P.(2) P(f((=) 2 m²), where m. = Σ Se(ze)-
d'=th
Note that me depends the payoffs of the actually sampled modulations 21-hot-1 allowing the bandit to become progressively more selective (if m keeps increasing).
Estimation: For simplicity, (z)) is inferred based the empirical data within time window of the same horizon h that is used to compute my Concretely, Pz(z) = pe(z)/ Σn (x') with the preferences pz(2) P (f)((z) > m²) defined as
1. 1/ - A I.
where n(2, h) is the number of times that z was chosen in the corresponding time window. We encode prior preference of in the absence of other evidence, additional (fictitious) sample.
4

---

Under review as a conference paper at ICLR 2020
asterit
10.3
12.0
0.0k
40.0%
12.04
18.00
$2.00
1.04
1.24
6.0k
30.0%
8.04
6.8k
0.00
1.0
10.00
2.04
0.0k
1.0
155
2.0
5.5
1.0
1.5
0k
0.5
1.0
1.5
2.0
0.0K
0.5
3,0
1.5
2.0
0.5
1.0
2,0
1.0 rrames
1.5
20
frames
1ed
frames
led
frames
1m
frames
1+0
trames
365
Figure 3: The best fixed choices of a modulation = vary per game, and this result holds for multiple classes of modulations. Left: Repeat probabilities p. Right: Optimism w.
The choice of h can be tuned hyper-parameter, but in order to remove all hyper-parameters from the bandit, we adapt it online instead. The update is based on a regression ac- curacy criterion, weighted by how often the arm is pulled. For the full description, see Appendix [A]
Adaptive horizon:
Factored structure: As we have seen in Section _ our concrete modulations = have additional factored structure that can be exploited. For that we propose to use a separate sub-bundit (each defined as above) for each dimension j of z. The full modulation is assembled from the 2j independently sampled from the sub-bandits. This way, denoting by K2 the number of arms for Zj. the total number of arms to model is Σ, Kj. which is significant reduction from the number of arms in the single flattened space II, Kj. This allows for dramatically faster adaptation in the bandit (see Figure 2. On the other hand, from the perspective of each sub-bandit, there is now another source of non-stationarity due to other sub-bandits shifting their distributions.
EXPERIMENTS
The central claim of this paper is that the best fixed hyper-parameterrs in hindsight for behaviour differ widely across tasks, and that an adaptive approach obtains similar performance to the best choice without costly per-task tuning. We report a broad collection of empirical results on Atari 2600 (Bellemare et al. 2013) that substantiate this claim, and validate the effectiveness of the pro- posed components. From our results, we distill qualitative descriptions of the adaptation dynamics. To isolate effects, independent experiments may use all or subsets of the dimensions of 2.
Two initial experiments in toy grid-world world setting are reported in Figure 2 They demonstrate that the proposed bandit works well in both stationary and non-stationary settings. Moreover, they highlight the benefits of using the exact learning progress T.P(z), and the gap incurred when using less informative proxies f22). They also indicate that the factored approach can deliver substantial speed-up. Details of this setting are described in Appendix B
5.1 EXPERIMENTAL SETUP: ATARI
Our Atari agent is distributed system inspired by Impala (Fispeholt et al., 2018) and Ape-X (Horgan let al. 2018), consisting of one learner (on GPU), multiple actors (on CPUs), and a bandit providing modulations the actors. On each episode f, an actor queries the bandit for a modulation 2t, and the learner for the latest network weights O,. At episode end, it reports a fitness value fz(z) to the replay table for the learner. For stability and reliability,
bandit, and adds the collected experience we enforce a fixed ratio between experience generated and learning steps, making actors and learner run at the same pace. Our agents learn a policy from 200 million environment frames in 10-12h wall-clock time (compared to a GPU-week for the state-of-art Rainbow agent (Hessel et al. 2018
Besides distributed experience collection (i.e., improved experimental turnaround time), algorithmic elements of the learner are similar to Rainbow: the updates use multi-step double Q-learning. with distributional quantile regression (Dabney et al.) (0188) and prioritized experience replay (Schaul et al. 2015). All hyper-parameters (besides those determined by 2) are kept fixed across all games and all experiments; these are listed in Appendix C aloongside default values of 2. These allow us to generate competitive baseline results (118 + 6% median human-normalicn normalised score) with so-called reference setting (solid black in all learning curves) which sets the exploration parameters to that is
most commonly used in the literature 0.01, w 0, T 0. b 0, p = 0).
5

---

Under review as a conference paper at ICLR 2020
Temperature
Epsilon
repeats
Optimism
relative rank
I-
T-n.0
5-0.1
1-10
T-mo
groandr
0.00
P-10
P-0 70
pbnnd
Figure 4: No fixed arm always wins. Shown are normalized relative ranks (see text) of different
fixed arms (and curated bandit), where ranking is done separately per modulation class (subplot). The score of would be achieved by an arm that performs best across all seeds (0 it is always the worst). Dashed lines ranks of an oracle that could pick the best fixed arm in hindsight per game. The gap between the dashed line and the maximum indicates the effect of inter-seed variability, and the gap between the dashed line and the highest bar indicates that different games require different settings. In all four modulation classes, the bandit performance is comparable to one of the best fixed choices.
If not mentioned otherwise, all aggregate results are across 15 games listed in Appendix D and at least N independent runs (seeds). Learning curves shown are evaluations of the greedy pol- icy after the agent has experienced the corresponding number of environment frames. To aggregate scores across these fifteen games we use relative rank, an ordinal statistic that weighs each game equally (despite different cretely, the performance outcome G(game, seed, variant) is defined as the average return of the greedy policy across the last 10% of the (20 million frames). All outcomes G(game, ., then jointly ranked, and the corresponding ranks are averaged across seeds. The averaged ranks are normalized to fall between 0 and 1, such that normalized rank of corresponds to all N seeds of a variant being ranked at the top N positions in the joint ranking. Finally, the relative ranks for each variant are averaged across all games. See also Appendix D
scales) and highlights relative differences between variants. Con-
5.2 QUANTIFYING THE TUNING CHALLENGES
It is widely appreciated that the best hyper- hyper-paranen parameterss differ per Atari game. illustrates this point for multiple classes of modulations (different arms come out on top different games), while Figure 4 quantifies this phenomenon across 15 games and 4 modulation classes and finds that this effect holds general.
If early performance were indicative of final performance, the cost of tuning could be reduced. We quantify how much performance would be lost if the best fixed arm were based on the first 10% the run. Ssows that the mismatch is often substantial. This also indicates the best choice is stationary: what is good early learning may not be good later on - an issue sometimes addressed by hand-crafted schedules (c.g., DQN linearly decreases the value of (Mnih et al. 2015)).
Another approach is to choose not to choose, that is, feed experience from the full set of choices the learner, approach taken, e.g., in (Horgan al. 2018). However, this merely shifts the problem, as it in turn necessitates tuning this set of choices. Figure 6 shows that the difference between naive and carefully curated set can indeed very large (Table 4 in Appendix C lists all these sets).
5.3 ADAPTING INSTEAD OF TUNING
turns out that adapting the distribution over & as learning progresses effectively addresses the three tuning challenges discussed above (per-task differences, early-late mismatch, handling sets). Figure 6 shows that the bandit can quickly suppress the choices of harmful elements in non- curated set; in other words, the does not need to be carefully tuned. At the same time, game-

---

Under review as a conference paper at ICLR 2020
Performance drop when picking best fixed early
asterix
breakout
demon_attace
frostbite
hero
ms_pacman
private eye
qbert
seaquest
space_invaders invaders
space_
star_ _ar_gunner
tennis
venture
yars_revenge
zaxxon
0.0
0.5
1.0
Figure 5: Tuning exploration by early performance is non-trivial. The bar- bar-plot shows the perfor- mance drop when choosing a fixed & based on initial performance (first 10% the run) as compared the best performance in hindsight (scores see Appendix D). This works well for some games but not others, and better for some modulation classes than others (colour-coded in Figure 6b. but overall its not reliable method.
normalized between best and worst final outcome,
Tempe rature
bandit curated
Epsilon
Action Repeat
Optimism
Combe
uniform curated
bandit extended
uniform extended
Figure 6: Tuning exploration by choosing set of modulations is non-trivial. This figure contrasts four settings: the effect of naive ('extended") versus curated 2-sets, crossed with uniformly sampling using the bandit, and for each of the four settings results given for multiple classes of modu- lations (colour-coded). The length of the bars are normalized relative ranks (higher is better). Note how uniform does well when the is curated, but performance drops on extended z-sets (except for w-modulations, which are never catastrophic); however, the bandit recovers similar performance the curated set by suppressing harmful modulations (the third group closer the first than to the fourth). The 'combo' results are for the combinatorial c, space, where the effect is even more pronounced.
specific schedule emerges from the non-stationary adaptation, for example recovering an e-schedule reminiscent of the hand-crafted one in DQN (Mnih et al.) 2015) (see Figure 17 in Appendix B Finally, the overall performance of the bandit is similar to that of the best fixed choice, and not far from an "oracle" that picks the best fixed 2 per game in hindsight (Figure
number of other interesting qualitative dynamics emerge in our setting (Appendix E) action biases used initially and later suppressed (e.g., SEAQUEST, Figure 19); the usefulness of action

---

Under review as a conference paper at ICLR 2020
3500
18004
50000
7-2
3-00
1=0.2
8000
X-03
20003
13
12300
ue-20
19300
3500
0-20
-0
0-333
GA 1008k
37
0.5
2.6
8.3
23
20
frames
Figure 7: A few games cherry-picked to show that the combinatorial c, P. bandit (thick red) can sometimes outperform all the fixed settings of 2. The thick black line shows the fixed reference setting 0.01.
repeats varies across training (e.g., on H.E.R.O., Figure 18 Figure 16 looks at additional bandit baselines and finds that addressing the non-stationarity is critical (see Appendix E.3).
Finally, our approach generalizes beyond a single class of modulations; all proposed dimensions can adapt simultaneously within single run, using factored bandit handle the combinatorial space. Figure [3] shows this yields similar performance to adapting within one class. In a few games this outperforms the best fixed choice" in hindsight; see Figure 6 ('combo") and Figure 7 presumably because of the added dynamic adaptation to the learning process. On the entire set of 57 Atari games, the bandit achieves similar performance (113 = 2% median human- normalzee score) to our fixed, tuned reference setting (118 + 6%), despite operating on 60 different combinations of modulations.
6 RELATED WORK
Here we focus on two facets of our research: its relation to exploration, and hyper-paramend parameter tuning.
First, our work can be seen building rich literature exploration through intrinsic motivation aimed at maximising learning progress. As the true learning progress is not readily available during training, much of this work targets one of number of proxies: empirical return (Jaderberg al.,
2017]; change in parameters, policy, or value function (Itti & Baldi 2006); magnitude of training
loss (Mirolli & Baldassarre 2013 Schmidhuber] [191];; error reduction or derivative (Schmidhuber)
1991 Oudeyer et al., (200);; expected accuracy improvement (Misra et al.) 2018); compression
progress (Schmidhuber 22088; reduction uncertainty; improvement of value accuracy: or change in distribution of encountered states. Some of these have the desirable property that if the proxy is zero, so is T.P. However, these proxies themselves may only be available in approximated form, and these approximations tend to be highly dependent on the state distribution under which they are evaluated, which subject to continual shift due to the changes in policy. As a result, direct comparison between different learning algorithms under these proxies tends to be precarious.
Second, our adaptive behaviour modulation can be viewed as an alternative to per-task hyper- parameter tuning. or hyper-parameter tuning with cross-task transfer (Golovin et al., (017). and can be compared to other works attempting to reduce the need for this common practice. (Note that hyper-
the best-fixed-arm in our experiments is equivalent to explicitly tuning the modulations parameters.) Though often performed manually, hyper-parameter tuning can be improved by ran- dom search (Bergstra et al. (011), but in either requires many full training cycles, whereas our work optimises the modulations on-the-fly during a single training run.
Like our method, Population Based Training (PBT, Jaderberg et al. 2017) and meta-gradient RL (Andrychowicz et al.) 2016 Xu et al., 2018) share the property of dynamically adapting hyper- parameters throughout agent training. However. these methods exist in a distinctly different problem setting: PBT assumes the ability to run multiple independent learners in parallel with separate ex- perience. Its cost grows linearly with the population size (typically > 10), but can tune other hyper-parameters than our approach (such as learning rates). Meta- deta-gradieet RL, on the other hand,
'Since it is too expensive to investigate all individual combinations in the joint modulation space, we only vary 2-5 along single dimension time.
8

---

Under review as a conference paper at ICLR 2020
assumes that the fitness is differentiable function of the hold for exploration ayper-parameter..
which may not generally
While method focuses modulating behaviour in order to shape the experience stream for effective learning, a related but complementary approach is to filter or prioritize the generated ex- perience when sampling from replay. Classically, replay prioritization has been based on TD error, a simple proxy for the learning progress conferred by an experience sample (Schaul et al.) (015). More recently, however, learned and thereby more adaptive prioritization schemes have been pro- posed (Zha et al. 2019). with (approximate) learning progress as the objective function.
DISCUSSION & FUTURE WORK
Reiterating one of our key observations: the qualitative properties of experience generated by an agent impact its learning, in way that depends on characteristics of the task, current learning parameters, and the design of the agent and its learning algorithm. We have demonstrated that by adaptively using simple, direct modulations of the way an agent generates experience, we can improve the efficiency of learning by adapting to the dynamics of the learning process and the specific requirements of the task. Our proposed method has the potential to accelerate RI. research by reducing the burden of typer-parameter tuning the requirement for hand-designed strategies, and does so without incurring the computational overhead of some of the alternatives.
The work presented in this paper represents first stab exploiting adaptive modulations to the dynamics of learning, and there are many natural ways of extending this work. For instance, such an approach need not be constrained draw only from experiences generated by the agent; the agent can also leverage demonstrations provided by humans or by other agents. Having an adaptive system control the use of data relieves system designers of the need to curate such data to be of high quality an adaptive system can learn to simply ignore data sources that are not useful (or which have outlived their usefulness), as our bandit has done in the case of choosing modulations to generate experiences with (e.g., Figures| 17
18 19p.
A potential limitation of our proposal is the assumption that a modulation remains fixed for the duration of an episode. This restriction could be lifted, and one can imagine scenarios in which the modulation used might depend on time or the underlying state. For example, an agent might generate more useful exploratory experiences by having low stochasticity in the initial part of an episode, but switching to have higher entropy once reaches an unexplored region of state space.
There is also considerable scope to expand the set of modulations used. A particularly promising avenue might be to consider adding noise in parameter space, and controlling the variance (Fortunato) let al. 2018 Plappert et al.) 2018). In addition, previous works have shown that agents can learn diverse behaviours conditioned on a latent policy embedding (Eysenbach et al.] 2018 Haarnoja et al. 2018), goal (Ghosh et al., 2018 Nair et al., 2018) task specification (Borsa et al. 2019). A bandit could potentially be exposed to modulating the choices in abstract task space, which could be powerful driver for more directed exploration.
ACKNOWLEDGEMENTS
(omitted for anonymity)
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. CoRR, abs/1 abs/1606.04474, 2016.
a166.//.../66......../ 04474
Peter Auer. Using confidence bounds for exploitation-exxloraiio trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002. 2002.
TThe code for bandit implementations will be made available online when this paper is published, facilitating community extension and reproducibility.
9

---

Under review as a conference paper at ICLR 2020
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- 70, pp. 449-458. JMLR. org, 2017.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandi problem with non- stationary rewards. In Advances in Neural Information Processing Systems, pp. 199-207, 2014.
Diana Borsa, Andre Barreto, John Quan, Daniel Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In International Conference on Learning Representations, 2019.
Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems, pp. 2249-2257, 2011.
Wojeiech Marian Czarnecki, Siddhant M Jayakumar, Max Jaderberg. Leonard Hasenclever, Yee Whye Teh, Simon Osindero, Nicolas Heess, and Razvan Pascanu. Mix&match-agent cur- ricula for reinforcement learning. arXiv preprint
2018.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Gabriel Dulac-Amold, Richard Evans, Hado Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep rein- forcement learning in large discrete action spaces. arXiv preprint arXiv: 152.00776 2015.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: 1802.0166/ 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprint arXiv: /802.0070, 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learning Representations, 2018.
Javier Gareía and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(44):1437-1480, 2015. URL http://t:// orrg/
papers/vl 6/ /ag/riaaaale html
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies. arXiv preprint 1rXiv:1811.078/9 2018.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John El- liot Karro, and D. Sculley (eds.). Google Vizier: A Service for Black-Box Op- timization. 2017. moogle-viziee swwvice- -ooooooockkk.... box wwt.opiaiiinniionn
URL http::/ /ttt:://www.r kdd. org http://kkoo11////////e//ee
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. arXiv preprint 1804v:1804.0240 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
10

---

Under review as a conference paper at ICLR 2020
Matteo Hessel, Hado van Hasselt, Joseph Modayil, and David Silver. On inductive biases in deep reinforcement learning. CoRR, 0tbs//997.029900 2019.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:8805 00033, 2018.
Laurent Itti and Pierre F Baldi. Bayesian surprise attracts human attention. Advances Neural Information Processing Systems, pp. 547-554, 2006.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, abs/171 abs/1711.09846, .09846, 2017.
Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial intelligence and statistics, pp. 592 692-600,200 2012.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. CoRR, abs/1709.06009, 2017.
Marco Mirolli and Gianluca Baldassarre. Functions and mechanisms of intrinsic motivations. In Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 49-72. Springer, 2013.
Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens Van Der Maaten. Learning by asking questions. In Proceedings of the IEEE Conference on Com- puler Vision and Pallern Recognition, pp. 11-20, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 518(7540):529, 2015.
Prabhat Nagarajan, Garrett Warnell, and Peter Stone. The impact of nondeterminism repro- ducibility in deep reinforcement learning. 2nd Reproducibility in Machine Learning Workshop at ICML 2018, July 2018
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
Pierre Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2):265- 286. 2007.
Matthias Plappert. Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018.
Martin I.. Puterman. Markov Decision Processes- Discrete Stochastic Dynamic Programming. John Wiley Sons, Inc., 1994.
Vishnu Raj and Sheetal Kalyani. Taming non-stationary bandits: A Bayesian approach. arXiv preprint arXiv:1707.09727 2017.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.0599: 2015.
Jürgen Schmidhuber. Curious model-buildigg control systems. In Proc. International Joint Confer- ence on Neural Networks, pp. 1458-1463, 1991.
Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. Workshop on anticipatory behavior in adaptive learning systems, pp. 48-76. Springer, 2008.
11

---

Under review as a conference paper at ICLR 2020
Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Ludwig Schubert, Marc Bellemare, Jeff Clune, and Joel Lehman. An Atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. arXiv preprint
arXiv: 1812.0009, 2018.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of samples. Biometrika, 25(3/4):285-299 1933.
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang. Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level StarCraft using multi-agent reinforcement learning. Nature, 575(7782): 350-354, 2019. doi: 10.1038//4 11885/9915---11-7722
Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, 67(1):15-36, 2000.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradiens reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2396-2407, 2018.
Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu Experience replay optimization. In Proceedings of the Tighty-Eigtth International Joint Conference on Artificial Intelligence, IJCAI- 19, pp. 4243 4249, 2019.
12

---

Under review as a conference paper at ICLR 2020
A ADAPTIVE BANDIT
In this section we will briefly revisit the adaptive bandit proposed Section 4 and provide more of the details behind the adaptability of its horizon. For clarity, start by restating the setting and key quantities: the reference fitness my, the active horizon fgg and the observed fitness function fe Z + R. where
{21,-... =K} defines modulation class.
The bandit samples a modulation E {=1,-.., ² according to the probability that this will result higher than average fitness (within recent length-h window):
t-1
(z(z) x P(f((2) 2 m²), where my = h ドーt-h
Σ felze).
Adapting z-probabilities. For simplicity, Pz(z) is inferred based on the empirical data within recent time window of the same horizon h that is used to compute my- Concretely, Pz())) = (x())/ Σ.' pe(²)) with the preferences (4(2) & P(f((z) 2 m,) defined as
[e(2) Ei 12.
1+ ((,,h
where (2,, is the number of times that z was chosen the corresponding time window. We encode prior preference of in the absence of other evidence, additional (fictitious) sample.
A dapting the horizon. As motivated in Section 4 the discrete horizon size hr = lte + adapted in order to improve regression accuracy (h):
- fex(z))f((z))) (z)( (h) 2
where f(2) is the fitness of the modulation 2, chosen at time t, and
Je((z) mc + Σt fe(ze )ll,
1 1 1=2,
This objective is not differentiable w.r.t. h, cannot grow beyond the amount of available data, the finieediifference is not symmetric around ht. Concretely, every step evaluate two candidates: according the current horizon hr, (h;), and one proposing shrinkage in the effective horizon, (h(),, where the new candidate horizon is given by:
perform
step. As the horizon
N' = max(2K, (1 n)))).
Thus the new horizon proposes a shrinkage of up to 17 = 2% per step, but is never allowed to shrink beyond twice the number of arms K. Given a current sample of the fitness function fe(=)), probe which of these two candidates ht or best explains it, by comparing L, (h+) and (.()). If the shorter horizon seems explain the new data point better, we interpret this sign of non-stationarity in the process and propose a shrinkage proportional to the relative error prediction reduction ((( namely:
2((h.)
2K, (1 (((h)) C.(h") )h1) if L.(h)) (hh))) (((hg)
max
h41
h4
otherwise.
Factored bandits. In the case of factored sub-bandits, they each maintain their own independent horizon h; we have not investigated whether sharing would beneficial.
B LAVAWORLD EXPERIMENTS
In this section we will describe in further details the experiments behind Figure These were conducted LavaWorld, small (96 states), deterministic sour-rooms-style navigation domain, with dcadly lava instead of walls. We chose this domain to illustrate what our proposed adaptive mechanism (Section 4 would do under somewhat idealised conditions where the learning is tabular
13

---

Under review as a conference paper at ICLR 2020
and we compute the ground-truth LP(2) and assess oracle performance. This investigation allows us to first see how well the bandit can deal with the kind non-stationary arising from an RL learning process (entangled with exploration).
In this setting, we consider three modulation classes, 6, b. where can boost the logits for any of the 4 available actions. The sets of modulations are c, (0.01, 0.1, and b, E {0, 0.13 resulting in 31 unique modulated (stochastic) policies for each Q-function (see Figure B). Q-functions are look-up tables of size (96 4) this domain contains 96 unique states. The single start state is in the top-left corner, and the single rewarding state is in the top-left corner of the top-right room,
and is also absorbing. We treat the discount y 0.99 as probability of continuation, and terminate episodes stochastically based on this, or when the agent hits lava.
Figure 8: Example visitation densities LavaWorld of different modulated policies that all use the same set of Q-values.
We study the behaviour of the system in two settings: one stationary, one non-stationary.
In the stationary setting (Figure) 2 left), considered modulation behaviours т (-Iss that do not change over time. They are computed based on a 'dummy' action-value function Q, as described in Section _ but this value does not change over time. The learning process is tabular and independent of this behaviour generating value Q. In this case, we compute the cumulative probability that an executed policy encounters the single sparse reward ('expected reward') ) function of the number of episodes, where we assume that the policy will be perfect after the first reward event. The LP(()) signal given the oracle bandit the true (stationary) expectation of this event for every modulation z. Given the extreme reward sparsity and the absence of learning, there is no obvious choice for proxy measure f. the non-oracle bandit reverts to uniform choice over The reference Q-values are the optimal ones for the task. The results presented are averaged over 10 runs.
Secondly, we considered non-stationary setting (Figure right) similar to the one above, but where the Q-values behind the modulated behaviour are learned over time. This is akin to the actual regime of operation this system would encounter in practice, although the learning of these values is idealised. These are initialised at zero, and the only update (given the absence of rewards) to suppress the Q-value of any encountered transition that hits lava by 0.1; this learning update happens instantaneously In other words, the agent does random walk, but over time it dies in many different ways. The reported "expected reward" is again the probability that the policy induced by Qi encounters the final reward. In this case, reasonable proxy f(2) for LP(2) is the binary signal on whether something was learned from the last episode (by encountering a new into-lava transition), which obtains large fraction of the oracle's performance.
14

---

Under review as a conference paper at ICLR 2020
C ATARI HYPER-PARAMETERR
In this section we record the setting used in our Atari experiments: hyperparameters of the agents (Tables and 3p, environment and preprocessing (Table [ and modulation sets for our behaviours (Table
Reference modulations Unless specified otherwise, we use the following modulations by default: 0.01, temperature 7 0.00001 (for tie fie-breaking between equal-valued actions), biases b O, optimism w = O, repeat probability = 0. This corresponds to the most commonly used settings in the literature. On learning curve plots, this fixed setting always shown in black.
Modulation sets The sets of curated and non-curated modulations we use are described in Table Curated values were chosen based on the results Figure
Fixed hyperparameters The hyper-parametert used by our Atari agent are close defaults used in the literature, with a few modification to improve learning stability in our highly distributed setting. For preprocessing and agent architecture, we use DQN settings detailed in Table and hyper-parameters parameters used by our Atari agent.
Table Table 3 summarizes the other hyper-ppaaeee
Parameter
value
Grey-scaling
True
Observation down-sampling
(84, 84)
Frames stacked
Reward clipping
[-1, 1]
Action repeats (when p = 0)
Episode termination on loss of life
False
Max frames per episode
108K
Table 1: Environment and preprocessing hyperparameters.
Parameter
value
channels filter sizes stride hidden units
32, 64, 64 8 8x8,4xx
4, 3 x 3
2,
512
Table Q network hyperparameters.
D EVALUATION
In this section we detail the evaluation settings used for our Atari experiments, well as the metrics used to aggregate results across games in Figure 5 in the main text.
of 15 games chosen for their different learning characteristies: As- TERIX, BREAKOUT, DEMON ATTACK, FROSTBITE, H.E.R.O. Ms. PAC-MAN, PRIVATE EYE. **BERT, SEAQUEST, SPACE INVADERS. STAR GUNNER, TENNIS, VENTURE, YARS' REVENGE and ZAXXON.
Games We evaluate
Human- normaliee scores are computed following the procedure in (Mnih et 2015), but differ- ing in that we evaluate online (without interrupting training), average over policies with different weights O, (not freezing them), and aggregate over 20 million frames per point instead of million.
The relative rank statistic in Section 5.1 normalized to fall between 0 and for any variant)) set of outcomes G(game, variant) for any number of seeds N. For this we simply scale the raw average ranks by their minimal and maximal values: N+1 and N+ + N+1 where N is the number of other outcomes this variant is jointly ranked with. 2
Metrics
15

---

Under review as a conference paper at ICLR 2020
Parameter
value
n for n-step learning
number of quantile values
11
prioritization type
proportional to absolute TD error
prioritization exponent
0.6
prioritization importance sampling exponent learning rate
0.3
0.0001
optimization algorithm
Adam
Adam setting
10
1-2 regularization coefficient
10-6
target network update period
IK learner updates (= 250K environment frames)
replay memory size min history to start learning batch size
IM transitions
80K frames
512
samples to insertion ratio
8
Huber loss parameter max gradient norm discount factor number of actors
10
0.99
40
Table 3: Agent hyperparameters.
Modulation
Curated set
Extended
Temperature
0.0001, 0.001, 0.01
0.00001, 0.0001, 0.001, 0.01, 0.1, 10
0, 0.001, 0.01, 0.1
0. 0.001, 0.01, 0.1, 0.2, 0.5.
Repeat probability p Optimism w
0, 0.25, 0.5
0, 0.25, 0.5, 0.66, 0.75, 0.8, 0.9
-1, 0, 1, 2, 10
-10, -, 2,
Action biases b
0
-1,0,
Table 4: Modulation sets.
different metric compare the effect of chosen modulation early into the learning process. For this we compute E, (G the average episode return of modulation at the end of training across all seeds E S, and 2p, the modulation with highest average episode returns at the beginning of training (first 10% of the run). Based on this, we compute the normalised drop in performance resulting from committing prematurely to zo:
In Figure
E. [_ E.IG
Performance drop(z)
E. ((
(
where = arg max,ez E. [. x arg mingez EE, |G! and Z the modulation class consid- ered in the study (c's, temperatures T. action repeat probabilities and optimism w).
E ADDITIONAL. ATARI RESULTS
E. TER-TASK NON-STATIONARITY
In this section we report detailed results which were presented aggregate in Figure 4 Specifically, these results show that (1) the most effective set of modulations varies by game, (2) different mod- ulations are preferable on different games, and (3) the non-stationary bandit performs comparably with the best choice of modulation class all games.
In the next few figures we give per-game performance comparing fixed-arm modulations with the adaptive bandit behaviour for modulation epsilon (Figure 9), temperature (Figure IO), action re- peats (Figure ITp. and optimism (Figure 12). For reference, we include the performance of uniform bandit (dashed-line in the figures), the same modulation set as the bandits, well as the best parameter setting across games (reference solid black line in the figures).
16

---

Under review as a conference paper at ICLR 2020
25.0k
asterix
250
breakout
7.0k
demon, attack
6.0k
frostbite
30. Ok
hero
6.0k
20.0k
200
15.0k
+-2.1
+-35
2-22
150
5.0k
4.0%
4.0%
3.0k
15.0k
20.0k
10.0k
Bunds
100
10.0k
1.0c
5.0k
b.ok
0.5
2.0
3.5
2.0
0.5
1.0
1.5
2.0
0.5
3.0 obert
3.5
1.0
0.5
3.0
3-3
ms_pacman
108
private_eye eye
1n
1=84.00
1ef
space_
4.0k
1.0c
14.0k
1.6k
3.5k
0.8k
22.0k
1.4k
10.0k
1.0k
2.5k
0.4k
0.00
6.0k
0.8k
1.5k
0.0k
4.0%
4.0K
2.04
0.2k
2.8
4k
2.00
4.00
k.okk
1.0
60.0k
700
1=Bo.00
1.9g.0k
50.0k
2.0
6.00
20.00
0.0k-
7.0k
40.0k
500
400
50.00
30.0k
300
10.0k
4.0k
20.0k
-10
200
20.00 o2
3.0k
10.0k
-20
100
1.0k
0.0k
2.0
30
2.0
Ok
frames
frames
frames
leB
framess
lsB
frames
Figure 9: Fixed arms, bandit and uniform over the curated sets for epsilon.
asterix
breakout
8.0k
demon_attee
frostbite
30.00
hero
F-3
3W0000
7.08
5.00
5.0%
3.0k
20.0k
4.0%
15.0k
10.0k
Bunds
1.00
3.0k
2.0g
10.0k
5.0k
2.04
1.0c
0.0k
2.0
0.5
1.5
0..0
10 ok
6.0k
mas Pac_pa
3e03.0c
privalie eye
1e
obert
1els.oc 5.0k
seaqueest
168
3.0k
Space inw.ddeedesss 1ell
2.5k
14.00
4.0k
1.5k
10.0k
2.0k
2.0k
0.5k
6.0%
1.0k
4.0k
2.0k
0.0k
0.5
1.0
3.5
0k
0.5
1.0
1.5
2.5
0.5
3.0
2.5
0.5
1.0
2.5 0k
0.5
3.0
3.5
60.0k
star gun.gunnrr
1el
50
tonnis
168
4.50
venture no
1=Bo.oc
yars, revengeegggg
1efo.ok okkk
20300n
1eB
4.00
M
P0.00
350
60.0e
8.0k
40.0k
10
250
3.00
ok
6.0k
30.0k
330.00
20.0k
-10
150
10.0k
-20
100
10.0k
0.5
3.0
3.5
2.0
0.5
1.0
1.5
2.0
0.5
3.0
1.5
2.5
0.5
1.0
1.5
0.5
3.0
1.5
frames
108
frammss
leB
frames
1eB
frames
1eB
frames
16B
Figure 10: Fixed arms, bandit and uniform over the curated sets for temperatures.
E.2 COMBINATORIAL BANDITS
In this section we include additional results for the different combinatorial bandits run on the cu- rated and extended sets (see Table 4). Most of these experiments were subsets of the cu- rated/extended sets across modulations, rather than the full Cartesian product. As convention, whenever modulation class is omitted from the experiment name, the value for this class is set to the default reference value reported in Section C (Reference modulations). Thus, for instance refer per-class-modulattee bandit, say optimism w, the modulations z for this class would
17

---

Under review as a conference paper at ICLR 2020
25.0k
asterix
250
breakout
8.0k
demon, attack
6.0k
frostbite
30. 0k
hero
20.0k
200
6.0%
7.04
5.0c
25.0k
15.0k
10.0k
p-0.75
bunds
Um Ftrm
rnforence
150
100
4.0%
2.0k
3.0k
20.0k
15.0k
10.0k
2.04
1.0k
1.0c
5.0k
5.0k
10.0k
0.5
2.0
2.0
0.5
1.0
1.5
2.0
0.5
2.0
3.5
0.5
3.0
3-3
iss_pacman
1=
private_eye
obert
1+86.00
seaqqees
space_invaders
16.0k
4.00
1.4k
14.0k
52.04
1.4k
1.2k
10.0k
8.0k
0.00
1.0k
6.0k
6.0k
4.0k
4.0k
0.6k
1.0k
2.0k
2.0k
2.8 4k
28
k.ok
2.8
1.0
9:00
1*Bo.0
1*fo.ok
50.0k
3.5
800
700
20.00
40.0k
8:00
60.00
5:0
30.0k
4.00
30.0%
4.0k
20.0k
300
10.0k
200
20.00 00
2.0k
-15
1.00
0.0k
2020
1.5
2.0
2.0
0k
frames
fraimes
frames
1eB
frames
l68
frames
Figure 11: Fixed arms, bandit and uniform over the curated sets for action repeats.
25.0k
asterix
250
breakout
10.0k
demon, attack
7.0k
frostbite
35.0k .k
hero
6.0k
30.0k
20.0k
200
8.0k
5.0k
5..0k
15.0k
2=3
--10
0.0k
4.0%
20.0k
10.0k
2-20
100
4.0k
3.0 0k
15.0k
2.0k
10.0k
1.0c
b.0k
0.5
3.0
3.5
2.0
0.5
1.0
1.5
2.0
0.5
3.0
1.5
2.0
0.5 1.0
1.5
2.3
0.5
3.0
13
4.0k
ms pac.paacann
1e0
private
legs
obert
1e0g
503qu
1e9
space invaders 1eB
3.5k
1.0k
20.0k
26.0%
1.6k
1.4k
1.2.0k
15.0k
0.00
0.5k
1.0k
0.0k
6.0k
4.0c
0.6k
2.0k
0.4k
2k
0.5
3.0
3,5
2.0
0.5
1.0
1.5
2.0
0.5
3.0
3.5
2.0
0.5
1.0
2.0
0.5
3.0
3.5
2.0
60.0k
star_gunne
1efl
1w8f
1<Bo
years revenge
1e%%.0k
20
30.0c
8.0k
50.0k
w0.00
7.0k
30.0k
3.0
30.0k
5.0k
4.0k
20.0k
200
20.0k
10.0k
-15
1.00
1.0k
1-20
0k
1.0
2.0
1.0 frames
frames
framms
framess
frames
Figure 12: Fixed arms, bandit and uniform over the curated sets for optimism.
be the ones reported in Table (line 4), while all other modulation dimensions would be kept fixed to their reference values.
Figure 13 shows that the combined bandit performs competitively compared with per-factor bandits (the same adaptive bandit but restricted to one class of modulation). In particular, it is worth noting that the per-factor bandit that performs best is game dependent. Nevertheless, the combined ban- dit, considering modulations across many of these dimensions, manages to recover a competitive performance across most games.
18

---

Under review as a conference paper at ICLR 2020
20
Figure 13: Comparing the combinatorial e, wandiiit to the per-modulation-class bandits.
In Figure 14 we include a comparison plot between the combinatorial bandit on the curated set of 3 modulation classes (e, p,w), its uniform counterpart on the same set and the reference fixed arm across games. First thing to notice is that on the curated set the uniform bandit is quite competitive, validating our initial observation that the problem of tuming can be shifted level above, by carefully curating set of good candidates. We can additionally see that the adaptive mechanism tends to fall in-between these two extremes: an uninformed arm selection and tuned arm selection. We can see that the adaptive mechanism can recover a behaviour close to uniform in some games (H.E.R. R.R.R.... R.O., YARS' REVENGE), while maintaining the ability to recover something akin to best arm identification in other games (see ASTERIX). Moreover there are (rare) instances, see ZAXXON, where the bandit outperforms both of these extremes.
19/03
aster'x
breakout
demon_attick
frestalte
140
horo
30.
12:03
560
2:.
13303
0.03
J0c3
1400
1003
1000
1300
1003
1010
148
548
14000
100
STA/ Siiteer
velliire
yani_revenge
200400
4000
$000
15303
4230
122
15030
Trammi
Figure 14: The learning curves for the uniform and the -anddit on the curated set.
19

---

Under review as a conference paper at ICLR 2020
In Figure 15 include plot comparing the performance of combinatorial bandit on the full curated and extended modulation sets. These are bandits acting on across all modulation classes outlined in Table As a reference, as in Figure T3 The bias modulation class was omitted as modulating exclusively within this class leads very poor performance as policies tend lock into a partieular preference. We can also see a negative impact on the overall performance when adding a bias set to the set of modulations the bandit operates on, as one can see from Figure 15 (magenta line). This is why we opted not to include this set in the extended bandit experiments reported in Figure 6 and restricted ourselves to the other extended modulation sets.
include the performance of the per-class modulation bandits,
100
Fristlilee
1000
CST purner
ventire
yarr, Hwwrng
2300001
Tannk
290
580
Zichuve
Excluee her Coctenttedd
110744
mamec
15/140
Figure 15: Bandits on extended sets versus curated sets For single-class modulation, the gaps are small overall (as shown in Figure (). One negative result is that on the fully combinatorial space T, c. b, p, the bandit performance drops substantially (compare red to magenta). We assume this due to the dominant effect of strong action biases b that effectively impoverish the behaviour for many sampled z.
E.3 OTHER BANDIT BASELINES
16 we provide a comparison to other more established bandit algorithms, UCB (Auer 2002 Kaufmann et al. 2012) and Thompson Sampling (Thompson 1933 Chapelle & Li 201 I). that would need to learn with modulation to use. The results in this figure are averaged across 5 seeds, and jointly modulate across three classes (e, p, w). We also include resulting learning curves for our proposed adaptation mechanism, the bandit described in Section 4 as well as uniform. The first thing notice is that the stationary bandits, UCB and Thompson Sampling, are sometimes significantly worse than uniform, indicating that they prematurely lock into using modulations that may be good initially, but don't help for long-term performance. We have already seen signs of this non-stationarity in the analysis in Figure 5 which shows that early commitment based on the evi- dence seen in the first part of training can be premature and might hinder the overall performance. In contrast, proposed bandit can adapt the non-stationarity present in the learning process, resulting in performance that is on par or better than these baselines in most these games (with only one exception, in YARS' REVENGE, where it matches the performance of the uniform bandit). In that context, it is worth highlighting that the best alternative baseline (UCB, Thompson Sampling, Uniform) differs from game to game, outperforming all of them is significant result. Another point is that UCB and Thompson Sampling still require some hyper-paramedt tuning (we report the results for the best setting we found), and thus add extra tuning complexity, while approach is hyper-parameter parameter-freee
Finally, in Figure
20

---

Under review as a conference paper at ICLR 2020
1400
asterft
14
breakout
demon sttack
2000
frostbite
35000
hero
1200
P2
1000
14-
35000
2000
19000
12:4
3300
ITC, h.c.paomaan
pfivote_
15000
abort
30000
Scoquest
1340
spacc_ Jtwoders
X-
3500
1:0000
4000
3000
6.0
3100
2000
49000
Star qunner
Banris
1300
venture
40000
yars revenge
9020
Zakion
4000
8000
3000
14:
30000
6.00
31000
2000
1990
3000
200
350
3000
Promium
Pramas
Figure 16: Comparison of several bandits (and uniform) on the combinatorial space spanned by the extended set of modulations e, p. w. The performance of uniform (gray), UCB (green) and Thompson Sampling (blue) varies considerably from one game to another, while our proposed adap- tive mechanism ('bandit', red) seems to deal quite effectively with this variability and achieves very good performance across almost all games. Note that the bandit and uniform results here correspond to the extended 'combo' results in Figure
E.4 BEHAVIOUR OF THE NON-STATIONARY BANDIT
In the previous section we saw that the bandit is able to effectively modulate the behaviour policy to give robust performance across games. In this section we dive in deeper analyse the precise modulations applied by the bandit over time and per-game. We see that the bandit does indeed adaptively change the modulation over time and in a game-specific way.
With these results we aim to look into the choices of the bandit across the learning process, and in multiple games. Most of the effect seems be present in early stages of training.
Epsilon schedules: Figure shows the evolution of the value of by the bandit over time. We see that this gives rise to adaptive type of epsilon schedule, where early in training (usually the first few million frames) large values are preferred, and as training progresses smaller values are preferred. This leads to gradually increasingly greedy behaviour policy.
Action repeats decay: Figure 18 shows the bandit modulated values for action repeats. We observe that as the agent progresses it can benefit from more resolution the policy. Thus, the agent adaptively moves to increasingly prefer low action repeat probability over time, with prolonged period of non -oeefereee early in training.
Figure I9 shows the evolution of the sampling distributions of a combined bandit with access to all modulation dimensions: T. e, b, p, and w. Despite having over 7.5 million combinations of modulation values, the bandit can efficiently learn the quality of different arms. For instance, the agent quickly learns to prefer the down action over the up action, to avoid extreme left/right biases, and to avoid suppressing the fire action, which is consistent with our intuition (in SEAQUEST, the player must move below the sea level to receive points, avoid the left and right boundaries, and fire at incoming enemies). Moreover, in the case of single arm bandits discussed above, the combined bandit prefers more stochastic choices of c and temperature in the beginning of training and more deterministic settings later in training, and the action repeat probability decays over time.
SEaQuEST:
21

---

Under review as a conference paper at ICLR 2020
0.3
0.3
0:30
0.70
0.29
6.0
0:03
0.8
000
0.20
1.0
2.2
0.40
IS5 pecman
2.43
pewaat e/t
0.4
spert
0.4
senquest
0.45
space Itanders
0.20
0.25
0.23
033
0.22
0.50
0.59
0:57
630
0.00
37
107
335
033
0.0
0:27
020
0.03
0.0%
Figure 17: Choices of by the bandit across time (log-scale) the extended set of Note that in general the values of e start high and gradually diminish, but the emerging schedules are quite different across games.
0.
0.0
0.-
0.
0.
30'
10P
10P
30
30
10*
[16. (66_02cmaan
privato_cye
q2ort
sobquest
space, iwadd's
55
0.4
0.
0.4
65
1.0
gurner
3.0
senture
1.0
10
Ttarin
Yarres
Figure 18: Choices of action repeat over time for different games on the extended set of p.
22

---

Under review as a conference paper at ICLR 2020
artion
arlion ban
Artior
0.30
0.30
0.5
0.5
0.70
0.28
0.4
0.10
0.10
6:18
0.18
opailes
Esmperruure
Up--miom
OC50n repeal probubritty
P-5
0.5
0.0
04.00
0.00
008
0.50
0.10
033
030
Figure 19: Probabilitics of selected action bias scales (first four subplots) and other modulation dimensions across training, on the game of SEAQUEST.
23


Reviews:
[{"review_id": "rygtop9XcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This papers studies how to explore, in order to generate experience for faster learning of policies in context of RL. RL methods typically employ simple hand-tuned exploration schedules (such as epsilon greedy exploration, and changing the epsilon as training proceeds). This paper proposes a scheme for learning this schedule. The paper does this by modeling this as a non-stationary multi-arm bandit problem. Different exploration settings (tuple of choice of exploration, and the exact hyper-parameter), are considered as different non-stationary multi-arm bandits (while also employing some factorization) and expected returns are maintained over training. Arm (exploration strategy and hyper-parameter) is picked according to the return. The paper demonstrates results on the Atari suite of RL benchmarks, and shows results that demonstrate that their proposed search leads to faster learning.\n\nStrength:\n1. The paper tackles an interesting and important problem. The proposed solution is simple, yet effective.\n\nShortcomings:\n1. The presentation is somewhat convoluted. The paper motivates the problem that we need to pick out an exploration sequence that optimizes learning progress, but then approximates it as simply measuring the return. Given there is no theoretical justification for the approximation, I believe the paper claims more than what it delivers and should change the presentation, so as not to claim that it is measuring and capturing learning progress to learn faster.\n\n2. I am confused by Figure 4, and in general with the relative rank metrics. Specifically, in Figure 4, is it that the proposed bandit approach not as good as picking a single hyper-parameter for the different settings (T=0.01, eps=0.01, omega=2.0)? Similarly, for Figure 2, a singe fixed z, seems to do better than the bandit versions. Why doesn't the proposed bandit algorithm not pick out the best hyper-parameter? How well would a simpler hyper-parameter search procedure (picking the best hyper-parameter after the first 2000 episodes)?\n\n3. This apart, I think that the experiment section is pretty hard to read, given all the metrics and methodology is in the Appendix. An alternate organization that presents all the main results in the main body in a self-contained manner will help.\n\n4. Comparison with past works. I believe there are other existing works that should be cited and compared to. Using bandits to decide between different hyper-parameters is common (for example, see [A] for a service to do this with ML models), [B] uses improvements in accuracy as a way to pick between which question type to train on. Such past works should be cited and compared against.\n\n[A] https://ai.google/research/pubs/pub46180\n[B] Learning by Asking Questions\nIshan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta and Laurens van der Maaten", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1eP2ufv9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops a multi-arm bandit-based algorithm to dynamically adapt the exploration policy for reinforcement learning. The arms of the bandit are parameters of the policy such as exploration noise, per-action biases etc. A proxy fitness metric is defined that measures the return of the trajectories upon perturbations of the policy z; the bandit then samples perturbations z that are better than the average fitness of the past few perturbations.\n\nI think this paper is just below the acceptance threshold. My reservations and comments are as follows.\n\n1. While I see the value in designing an automatic exploration mechanism, the complexity of the underlying approach makes the contribution of the bandit-based algorithm difficult to discern from the large number of other bells and whistles in the experiments. For instance, the authors use Rainbow as  the base algorithm upon which they add on the exploration. Rainbow itself is an extremely complicated algorithm, how can one be certain that the improvements in performance are caused by the improved exploration and not a combination of the bandit’s actions with the specifics of Rainbow?\n2. I don’t understand Figure 4. The score defined in Appendix is the average over games for which seed performs better. Why is the random seed being used to compare the performance of different arms? Do you instead mean that s and s’ are two values of the arm in Figure 4? If not, how should one interpret Figure 4, no fixed arm is always good because the performance varies across the seeds. The curated bandit does not seem to be doing any better than a fixed arm.\n\nI have a few more questions that I would like the authors to address in their rebuttal or the paper.\n\n1. The proxy f(z) does not bear any resemblance to LP(z). Why discuss the LP(z) then. The way f(z) is defined, it is just the value function averaged over perturbations  of the policy. If one were to consider z as an additional action space that is available to the agent during exploration, f(z) is the value function itself. The exploration policy is chosen not to maximize the E_z [f(z)] directly but to maximize the lower bound in Markov’s inequality (P(f(z) >= t) <= E_z [f(z)]/t) in Section 4.\n2. Can you elaborate more on the metric for measuring the learning progress LP? Why does the myopic metric make sense in spite of the there being plateaus in the training curves?\n3. The key contribution of the paper that the authors could highlight better is that they do not add new hyper-parameters. In this aspect, the auto-tuner for exploration is a plug-and-play procedure in other RL algorithms.\n4. From Figure 6 and Figure 8-11, it looks like the bandit is more or less on par with fixed exploration policies. What is the benefit of the added complexity?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxurON6FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an adaptive exploration scheme that can reduce the complexity of per-task tuning. This goal is achieve by formulating the adapting scheme as a multi-arm bandit problem with the actual \"learning progress\" as a feedback signal.\n\nThe paper is well written and easy to be understood.\n\nThe strength of this paper is that 1) the proposed method is new in the sense that it invents an automatic way for exploration. 2) The algorithm is simple yet effective by the experiment results the authors provide. \n\nWeakness: the presentation of the tables/bar charts in the experiment is a bit unclear. More explanations are needed.   ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJl2JOl9cr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This in an interesting paper as it tries to alleviate the burden of hyper-parameters tuning for exploration strategies Deep Reinforcement learning.\nThe paper proposes an adaptive behaviour in order to shape the data generation process for effective learning. The paper considers a behaviour policy that is parametrized by a set of variables z called modulations: for example the Boltzmann softmax temperature, the probability epsilon for epsilon-greedy, per-action biases, ..\nThe author frame the modulations search into a non-stationary multi-armed bandit problem and proposes to adapt the modulations according to a proxy to the learning progress. The author provides thorough experimental results.\n\nComments:\n\n- All the variations considered for the behaviour policy performs only myopic exploration and thus provably inefficient in RL. \n- The proposed proxy is simply the empirical episodic return. It is not well explained in the paper how this proxy correlates with the Learning progress criteria.   \n- The proxy seems to encourage selecting modulations that lead to generate most rewarding trajectories. How this proxy incentives the agent to explore poorly-understood regions? In other terms, how this proxy help to tradeoff between exploration and exploitation ? \n-  The modulation adaptation problem is framed into non-stationary multi-armed bandit problem but the authors present a heuristic to solve it instead of using provably efficient bandit algorithm such as exponential weight methods (Besbes et al 2014) or Thompson sampling (Raj & Kalyani 2017) cited in the paper. \n- The way the authors adapt the modulation z (or at least its description in the paper) seems not technically sounded for me. They estimate a certain probability at time step t by empirical frequency based on data from previous time steps. But as the parameters change during the learning, the f_t’(z) at time t’ < t is not distributed as f_t(z). This introduces a biases in the estimate.\n- I appreciate the thorough empirical results and ablation studies in the main paper and the appendix. They are really interesting. \n- I am confused what is the fixed reference in Figure 6. It is not explained in the main paper. Is it a baseline with the best hyperprameters in hindsight? \n-  From the plots of learning curves in appendix, the proposed methods doesn’t seem to show a huge boost of performance comparing to the uniform bandit. Could you show aggregated comparison between the proposed method and uniform bandit similarly to what is done in Figure 4 ? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a non-stationary bandit strategy for adapting the exploration rate in Deep RL algorithms. They consider exploration algorithms with a tunable parameter (e.g. the epsilon probability in epsilon-greedy) and attempt to adjust this parameter in an online fashion using a proxy to the learning progress. The proposed approach is empirically compared with using fixed exploration parameters and adjusting the parameter using a bandit strategy that doesn't model the learning process.

Unfortunately, the proposed approach is not theoretically grounded and the experiments lack comparison with good baselines in order to be convincing. A comparison with other, provably efficient, non-stationary bandit algorithms such as exponential weight methods (Besbes et al 2014) or Thompson sampling (Raj & Kalyani 2017), which are cited in the paper, is missing. Moreover, given the whole set of results and how they are presented, the improvement due to the proposed method is not clear. In light of these concerns I recommend to reject this paper.
Decision: Reject