Learning Synthetic Environments and Reward Networks for Reinforcement Learning
Authors: Fabio Ferreira
Thomas Nierhoff
Andreas Sälinger
Frank Hutter
Venue: ICLR 2022 Poster (2022)
Keywords: Synthetic Environments
Synthetic Data
Meta-Learning
Reinforcement Learning
Evolution Strategies
Reward Shaping

Abstract: We introduce Synthetic Environments (SEs) and Reward Networks (RNs), represented by neural networks, as proxy environment models for training Reinforcement Learning (RL) agents. We show that an agent, after being trained exclusively on the SE, is able to solve the corresponding real environment. While an SE acts as a full proxy to a real environment by learning about its state dynamics and rewards, an RN is a partial proxy that learns to augment or replace rewards. We use bi-level optimization to evolve SEs and RNs: the inner loop trains the RL agent, and the outer loop trains the parameters of the SE / RN via an evolution strategy. We evaluate our proposed new concept on a broad range of RL algorithms and classic control environments. In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only on the real environment. However, once such an SE has been learned, we do not need any interactions with the real environment to train new agents. Moreover, the learned SE proxies allow us to train agents with fewer interactions while maintaining the original task performance. Our empirical results suggest that SEs achieve this result by learning informed representations that bias the agents towards relevant states. Moreover, we find that these proxies are robust against hyperparameter variation and can also transfer to unseen agents.

Published as a conference paper at ICLR 2022
LEARNING SYNTHETIC ENVIRONMENTS AND RE- WARD NETWORKS FOR REINFORCEMENT LEARNING
Fabio Ferreira' Thomas Nierhoff' Andreas Sälinger' Frank Hutter 1.2
University of Freiburg Bosch Center for Artificial Intelligence (ferreira, fh] unw.unii--ee.... ani--rriiuuue bereerurgdeee
ABSTRACT
We introduce Synthetic Environments (SEs) and Reward Networks (RNs), repre- sented by neural networks, as proxy environment models for training Reinforce- ment Learning (RL) agents. We show that an agent, after being trained exclusively on the SE, is able solve the corresponding real environment. While an SE acts as a full proxy a real environment by learning about its state dynamics and rewards, an RN is partial proxy that learns to augment or replace rewards. We use li-level optimization evolve SEs and RNs: the inner loop trains the RL agent, and the outer loop trains the parameters of the SE / RN via an evolution strategy. We evaluate proposed new concept control environments. In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only the real environment. However, once such an SE has been learned, we do not need any interactions with the real environment train new agents. Moreover, the learned SE proxies allow us to train agents with fewer interactions while maintaining the original task performance. Our empirical results suggest that SEs achieve this result by learning informed representations that bias the agents towards relevant states. Moreover, we find that these proxies are robust against hyperparameter variation and can also transfer to unseen agents.
broad range of RL algorithms and classic
INTRODUCTION
Generating synthetic data addresses the question what data is required to achieve a rich learning experience in machine learning. Next increasing the amount of available data, synthetic data can enable higher training efficiency that opens up new applications for Neural Architecture Search (Such et al. 2020), may improve algorithm analysis facilitate custom datasets (Jhang al. 2020).
In this paper, we consider learning neural synthetic data generators for Reinforcement Learning (RL.. We investigate the question of whether real (target) environment which is capable of producing synthetic data to allow effective and more efficient agent training, that is, to achieve similar or higher performance more quickly compared to when training purely on the real environment. When learning to produce both states and rewards, we refer to these neural network proxies as synthetic environments (SEs). Additionally, we investigate the same question for learning reward proxies that do not learn about the state dynamics and which we refer to as a Reward Networks (RNs).
learn synthetic Markov Decision Process of
We depict our procedure in Figure T which resembles a bi-level optimization scheme consisting of an outer and inner loop. The inner loop trains the agent on an SE or RN. Since our method is agnostic to both domain and agent, we can interchangeably adopt standard RL algorithms in the inner loop. In the outer loop, we assess the agent's performance by evaluating it on the real environment; we then take the collected reward inner loop. In this way, the SE/RN is gradually updated such that an agent being trained on it, scores higher real environment. For the outer loop Salimans et al.) 2017) with population of SE/RN parameters.
score update the SE's RN's neural parameters used in the
Evolution Strategies (Rechenberg 1973
After discussing related work (Section
2 we make the following contributions:

---

Published as a conference paper at ICLR 2022
Synthetic Environments (SEs)
Reward Networks (RNs)
train
train
cpdlads me
agens
Treeze policy
agent
rividd net
agent
freeze policy
agent
shate
rrwoart
? action
whabe
action
state
action
reasedd
?
action
Nilu
buter loop
ouler loop
New
P
ralidd 5.ggg
Weai of on 100g
Figure I: We use an agent-agnostic meta-learning approach to learn neural proxy RL environments (left) and reward networks (right) for a target task. In the inner loop, we train RL agents on the proxy and use the evaluation performance on the target task in the outer loop to evolve the proxy.
We introduce synthetic environments (Section 3 a novel concept that focuses on the en- vironment instead of agent learning using bi-level optimization scheme, which is guided purely by the agent performance. This concept goes beyond the usual learning of a one- time internal environment model inside an agent (such as in Dyna (Sutton| 1990)).
As sub-problem of SEs, we investigate reward networks (Section 4. contrasting several types of potential- based reward shaping (Ng et al. 1999) variants.
We show that it possible to learn SEs (Section 5) and RNs (Section that, when used for agent training, yield agents that successfully solve the Gym tasks (Brockman et al. 2016) CartPole and Acrobot (SEs), as well as Cliff' Walking, CartPole, MountainCarContinuuus, and HalfCheetah (RNs).
We show that SEs and RNs are efficient and robust in training agents, require fewer training steps compared to training on the real environment and are able to train unseen agents
We report empirical evidence showing that SEs and RNs achieve their efficiency gains for training new agents through condensed and informed state and reward representations.
Overall, we find noteworthy that is actually possible to learn such proxies, and we believe our research will improve their understanding. Since these learned proxies can train agents quickly and transfer to unseen agents, we believe this work might open up possible avenues of research in RL. Possible future applications include cheap-to-run environments for AutoML (Hutter et al. (019) or for robotics when training on targets is expensive, as well as agent and task analysis, or efficient RL agent pre-trae tre-training. Our PyTorch Paszke et al. 2019) code and models are made available publicly
2 RELATED WORK
Synthetic Environments In the context of RL, learning synthetic environments is related to model-based RL (MBRL) where the dynamics model can be viewed as an SE. In MBRL, one jointly leams both a dynamics model and policy as in Dyna (Sutton| 1990) an existing dynamics model used with planning methods to learn a policy (Silver et al. 2017 Moerland et al. 2020). Our work does not involve planning, does not mix synthetic and real data during policy learning. Instead, we use purely synthetic data to train agents similar to World Models (Ha & Schmidhuber, 2018) but the cumulative rewards from the real environment in bi-level optimization for learning our model jointly with our agent. For more extensive discussion on the differences between our work and model-based RL, we refer the reader to Appendix D
does it use supervised learning for the dynamics model and it
Analogous learning SEs Procedural Content Generation TToeelius et al. 2011 and Curriculum Learning that concerns automatically selecting (Matiien et al. 2020) or generating the content of training environments (Volz et al. 2018 (Sakeer et 2016; Wang et al. 2019) Cobbe et al. 2020
with the closest work being Generative Playing Networks (GPNs) (Bontrager & Togelius 2020 GPNs learn an environment generator that creates increasingly difficult SEs according to what critic's value function estimates challenging. While GPNs are method to generate environment curricula, our approach studies learning an SE for effective and efficient agent training by compress- ing the relevant information into single model of the environment. We also more generally
https::/ www.::tww......ooo..oto /ea www.//aaannnnnn raainnnng ning envi ronments
2

---

Published as a conference paper at ICLR 2022
applicable objective that does not rely on actor-critic formulations but purely the achieved cu- mulative reward. Less related areas are methods like domain randomization (Tobin et al. 2017) that generate environments regardless of agent performance or minimax methods that adversarially (instead of cooperatively) generate environments based on the difference between the reward of two competing agents that trying to solve the generated environment (Dennis et al. 2020).
Related to our work, and inspiring it, are Generative Teaching Networks (Such et al. 2020). While
similarly use a bi-level optimization learn synthetic data generator, our approach different in central aspects: we use Evolution Strategies to avoid the need for explicitly computing Hessians, we do not use noise vectors as input our SEs, and we target sequential decision-makee maniin problems instead of supervised learning.
Reward Networks Reward shaping concerns the question of how to enhance the reward signal to allow agents to be trained more effectively efficiently. Common learned reward shaping ap- proaches are curiosity or count-based exploration (Pathak et al.] 2017; Burda et al. 2019; [Singh
al. 2010, Bellemare al.. 2016 Tang et al., 2017). Others achieve reward shaping with prior knowledge through expert demonstrations (Judah et al. 2014 Brys et al. 2015 Ibarz et al.) 2018). In contrast our work, these contributions all apply single-level optimization. When using a bi- level optimization, the reward shaping fuunction is usually learned in the outer loop while the policy using the learned rewards is optimized in the inner loop. Here, one way to meta-learn the param- eterization of reward functions (Faust et al. 2019: Hu et al.) [0200] Jaderberg et al. 2019). Another way to learn a neural network that resembles the reward function.
While learning full synthetic environments is entirely novel, there exists prior work on learning reward shaping networks. The most related works are (Zheng et al., 2018) for single tasks, (Zou et al., (209)) for entire task distributions, or (Zheng et 2020) that additionally take into account the entire lifetime of an agent to learn "statefulness across episodes" rewaad function. Despite the similarities, some noteworthy differences exist. Importantly, the approaches in (Zheng et al. 2018 Zou et al. 2019) are not agent-agnostic, making it less straightforward to exchange agents as in our work. Moreover, the transferability of learned shaped rewards is studied only limitedly (Zheng al. 2018), not at all (Zou et al. 2019) or only for grid world-like environments (Zheng et al. 2020).
3 LEARNING SYNTHETIC ENVIRONMENTS
Problem Statement Let A, R) be a Markov Decision Process (MDP) with the set of states S. the set of actions A. the state transition probabilities P and the immediate rewards R when transitioning from state s E S to the next state s' E S through action E A. The MDPs we consider are either (SE) represented by a neural network with parameters t/. Interfacing with the environments is identical both cases, i.e. s', r = E(s, a). The crucial difference is that for SEs, the state dynamics and rewards are learned. The main objective of an RL agent when acting on MDP Ereat is to find a policy Te parameterized by 0 that maximizes the cumulative expected reward F(0; Ereat). We consider the following bi-level optimization problem: find the parameters vi", such that the agent policy Ne parameterized with 0 that results from training on Eeyn, achieves the highest reward on target environment Ereat- Formally that is:
designed environments Ereal or learned synthetic environments Exyr.
4b arg max F(O"(w); Ereat)
(I)
s.t. 8" (0())= arg max F(O: Esyn.v).
We use standard RL algorithms for optimizing the agents on the SE in the inner loop. Although gradient-based optimization methods can be applied in the outer loop, we chose Natural Evolution Strategies (Wierstra et al., 2008) (NES) to allow the optimization to be independent of the choice of the agent in the inner loop and to avoid potentially expensive, unstable meta-gradie getaiggadssnts (Metz et al. 2019.. Additional advantages of NES are that it is better suited for long episodes, sparse or delayed rewards, and parallelization (Salimans et al. 2017).
Algorithm We now explain our method. The overall scheme is adopted from (aalmaans et al. It consists of an Evolutionary Strategy in the outer loop
2017) and depicted in Algorithm [
3

---

Published as a conference paper at ICLR 2022
to learn the SE and inner loop which trains RL agents. The performances of the trained agents are then used in the outer loop to update the SE. We instantiate the population search distribution as multivariate Gaussian with mean 0 and fixed covariance I²I. The main difference to (Salimans et al,, 2017) is that, while they maintain a population over agent pa- rameter vectors, our population consists of SE parameter vectors. Moreover, our approach in- volves two optimizations (the agent and the SE parameters) instead of one (agent parameters). Our algorithm first stochastically per- turbs each population member which
Algorithm 1: Learning Synthetic Env. with NES
results in 4/ (Line 5). Then, a new randomly initialized agent is trained on the SE parameterized by 1/; for ne
Input: initial SE parameters v, real environment Ereat, NES noise std. dev. Ø, number of episodes
ne: population size ny, NES step size
episodes (1.). The trained agent with
2 repeat
fixed parameters is then tested across 10 episodes on the real environment (L7). yielding the average cumulative reward which we use score Fe.i- Finally, we update v with a stochastic gradient estimate based
foreach member of pop. = 1, ... "p do 4 ~ x2)
5
v = vb +
0. TrainAgent(O,, Esym. 1ee)
7
Fasi = EvaluateAgent(0), Ereat)
all mem- ber scores (L8). We use a parallelized version of the algorithm and an early stopping heuristic to stop in fewer than ne episodes when progress plateaus (more in Appendix A.1
Update SE: v + 3 + a noo
Σ. "p F7E
until ni steps
LEARNING REWARD NETWORKS
Learning both the state dynamics and rewards can be challenging (see Section To reduce compu- tational complexity but still be able to achieve training efficiency, one can therefore reduce the prob- lem formulation to only learn (to augment) the rewards and make use of the real environment for the state dynamics (and original rewards) as illustrated in Figure [ We reuse the formulation of Eggn and Eyeat from the problem statement (Eq. [ and describe learning reward networks as getting the next state and reward s', r Ereat(s, (8, a) and a corresponding synthetic reward Tsyn = -Er,, (8, a, 8') with a (reward) environment represented by a neural network sb. In the domain of reward shaping, Teyn is usually added to r to generate an augmented reward Tang = r + Tsyn- A popular approach in reward shaping is potential-based reward shaping (PBRS) (Ng et al. 1999). which defines Eyn as (s, a, s') = =
R and a discount factor E (0, 1]. We adopt PBRS for finding Eyn.gi, represent &b as a neural network with parameters d and use Algorithm I learn gb. PBRS has the useful property of preserving optimality of poli- cies (Ng et al. 1999) but we also introduce additional variants of Eyn,,, listed in Table which are motivated by PBRS but are not typically referred to as PBRS as they may not have such guarantees. These variants are motivated by the idea that fewer constraints may enable better reward learning (c.g., in case the difference of potential functions imposes a too strong constraint on the hypothesis space). For simplicity, we refer to the output of Reward Networks (RNs) as Taug-
(s) with potential function + S
We now point out the differences between the RN and the SE case. When training SEs, we use a heuristic for determining the number of re- quired training episodes since both rewards and states are synthetic (see Appendix [..]p. In the RN case, the real states allow us to directly ob- serve episode termination, thus such a heuristic is not needed. Another difference to SEs lies in the EvaluateA function of Algorithm [ Optimizing SEs for the maximum cumulative reward au- tomatically yielded fast training of agents as a side-product, likely due to efficient synthetic state dynamics. Since we do not learn state dynamics in RNs and motivated by the efficiency gains, we now directly optimize the number of training steps Thy the TrainAgent function requires to solve the real environment when being trained on an RN. Additionally, we include the cumulative reward threshold of the real environment Caal and the cumulative reward at the end of training C Cim. yield- ing the final objective Ber + Woa maa O, Csat Cfin) (to be minimized) where Wsot R gives
Table 1: Overview of Reward Network variants.
Reward Network Additive Potential RN (Ng al. 1999)
Taug
r+ r+) ((s)
Exclusive Potential RN Additive Non-Potential RN
>D(s') - -(s))
r +(s')
Exclusive Non-Potential RN ((s')
4

---

Published as a conference paper at ICLR 2022
control over efficiency and effectiivnness an RN should achieve. In Appendix B.4 study the influence alternative RN optimization objectives.
5 EXPERIMENTS WITH SAHHSSYNNN SYNTHETIC ENVIRONMENTS
Experimental Setup So far we have described proposed method before we start with individual SE experiments we describe the experimental setup. In our work, we refer to the process of optimizing for suitable SEs with Algorithm | as SE training and the process of training agents on SEs as agen! training). For both SE and agent training on the discrete-action- space CartPole-vO and Acrobot-vl environments,
abstract level and
DDQN (van Hasselt et al., 2016). We also address generalization from an algorithmic viewpoint by varying the agent hyperparameters during SE training (Line 6 after each outer loop iteration. Also, we study how robustly found SEs can train agents under varying agent hyperparameters and how well they transfer in training unseen agents. For studying transferability of SEs, we use Dueling DDQN (Wang et af. 2016) and TD3 (Fujimoto & H. Hoof 2018). TD3 is chosen because it does not solely rely Q-Learning and is algorithm of the actor-critic family. Both our tasks employ a discrete action space and to be able to use TD3 we use Humbel-Softmas distribution (Jang et al. 2017).
We wrap our algorithm another outer loop to optimize some of agent and NES HPs with the multi-fidelity Bayesian optimization method BOHB (Falkner et al., 2018) to identify stable HPs. The optimized HPs are reported in Table 3 in the appendix. We did not optimize some of the HPs that would negatively affect time (e.g., population size, Table 4 in the appendix). After identifying the agent and NES HPs, we removed BOHB and used the found HPs for SE training. Once SEs were found, reverted from specific agent HPs that worked best for training SEs to default agent HPs as reported in Table (a to test out-of-the-box applicability.
Feasibility After introducing the core concept of learning SEs, we now inves- tigate its efficacy of learning SEs for the CartPole and Acrobot tasks. First, we identified stable DDQN and NES hyperpa- rameters. Then, we ran Algorithm [] with 16 population members for 200 NES outer loop iterations to generate the SEs. We de-
200
100
Gamulative reward
200
pict the result Figure 2 which shows the average of the 16 members' evaluation scores (given by EvaluateAgent in Algo- rithm TD as function of the NES outer loop iterations for multiple NES optimiza- tion runs (thin lines). The thick solid lines show the average of these. Notice that 50 NES iterations
-00
CartPole tirsshak CartPole-on
Pcrobon-v1 solved threshold Acrebon-v1
25
50
100
125
175
ES
Figure 2: Multiple NES runs of Alg. for CartPole (top) and Acrobot (bottom) which can teach agents the task effectively. Each thin line is the average of 16 worker scores from EvaluateAgen of such run.
often sufficient find SEs that teach agents to solve a task (for more details, refer the reader the Ap- pendix Section A).
Performance of SE-trained Agents As we have shown in the last section, learning SEs that can teach agents the CartPole and Acrobot task feasible with the proposed concept. Now, we investigate how efficient and hyperparameter- sensitive the generated SEs are. For this, we dis- tinguish three different cases: 1) train: real, 2) train: synth., HPs: varied, and 3) train: synth., HPs: fixed. The first is our baseline for which we train agents only on the real environment with- out any involvement of SEs. The second case denotes SEs for which we randomly sampled agent hyperparameters (HPs) from the ranges in Table before each TrainAgent call during SE training. Lastly, the third case is similar to the second except that we did not randomly sample agent HPs but rather used the optimized ones reported in Table which we kept, fixed during SE training.
Table 2:
Agent hyperparameter ranges sampled from for training SEs and agents.
Agent hyperparam.
Value range
log scale
learning rate
10 //3, 10
batch size hidden size
42, 384
[42
hidden layer
[1.3]
5

---

Published as a conference paper at ICLR 2022
DDQN on DDQNtraineo SEs (CartPule-v))
Transfer: Duding DDQN DDQN trainet SEs
Transfer: TD3 en DDQW-raned
trainrod
trann syrth vadee
train: syrth. iPP foed
17811 syrth HPs vades test
0.035
0.030
89302
steps
eshodss
20302
10302-
19207
150.70
175.46
179101
188.39
176.35
187.6
151.4
101
9504
46.31 PTS: Bood
P97: warted
H7s: lices
N1s var od
18' varte 12 beat)
Figure 3: Top row: Densities based on each 4000 cumulative test rewards collected by DDQN (left), Dueling DDQN (center), and discrete TD3 (right) agents on CartPole. We show three settings: agents trained real environment without any involvement of SEs (blue, baseline), SEs where the agent HPs were fixed during SE training (green), and on SEs when agent HPs were varied during SE training (orange). After training, the evaluation is always done the real environment. In all three settings, we randomly sample the agent HPs before agent training. Bottom row: Average train steps and episodes corresponding to the densities along with the mean reward (below bars). When training on SEs, we train 30-65% faster compared to training on the real environment.
We first trained 40 SEs for each of the cases 2) and 3); then, we evaluated all approaches with the following procedure: on each SE we trained 10 DDQN agents with randomly sampled HPs according Table 2 After agent training, we evaluated each agent on the real environment across 10 test episodes, resulting in 4000 evaluations. For our baseline, where no SE is involved, the procedure was identical except that we used real environment instantiations instead of SEs to obtain the same number of evaluations. We then used the cumulative rewards from the test episodes for generating the densities in Figure 3 and report the average episodes and training steps required until the heuristic for stopping agent training SEs/real environment is triggered (Appendix A.I)
Training DDQN agents on DDQN-trained SEs without varying the agent's 's HPs (green) during SE training clearly yielded the worst results. We attribute this to overfitting of the SEs to the specific agent HPs Instead, when varying the agent HPs (orange) during SE training, we observe that the SEs are consistently able train agents using ~60% fewer steps (6818 vs. 16887) average while being more stable than the baseline (fewer train steps / lower episode standard deviations and little mass density tails). Lastly, the SEs also show little sensitivity HP variations.
Transferability of SEs In the previous section we evaluated the performance of DDQN-trained SES in teaching DDQN agents. However, since many environment observations are needed for SE training, it would be beneficial if DDQN-trained SEs are capable of effectively training unseen agents as well. To study this question, we reuse the DDQN-trained SEs from above but now train Dueling DDQN agents on the SEs. From Figure 3 (top and bottom center) we conclude that the transfer to the Dueling DDQN agent succeeds and it facilitates a ~50% faster (6781 vs. 12745 train steps), more effective (higher reward), and noticeably more stable training (lower std. dev.; smaller tails) on average when compared to the baseline (orange vs. bluc). As before, not varying the DDQN agent HPs during SE training reduces transfer performance (green).
We also analyzed the transfer from Q-Learning to a discrete version of TD3 (see Section 3) The result shown in Figure 3 (top right) indicates a limited transferability. We believe this may be due to the different learning dynamics actor-critics compared to learning with DDQN. However, we also noticed that individual SE models can consistently train TD3 agents successfully while remaining efficient. To show this, we repeated the evaluation procedure from Figure 3 individually for each SE model (see (p...ldx(F...I]] and selected five SE models that yielded high rewards when training TD3 agents. As a consequence, we can again significantly improve the performance (black curve) with 655% % speed-up (6287 vs. 17874 train steps; 187 vs. 152 reward) over the baseline.
We note that even in the "HPs: varied" case not evaluating whether SEs extrapolate to new HP ranges but rather how sensitive SEs react HP variation within known ranges.

---

Published as a conference paper at ICLR 2022
Getts ITN 19EA) Fesl
Ord deV (Part
synds
Ada
em 3091
copurerte
20
20
10'
sludal
vabelly traot slabel (17r]
pole siaial [nad]
(ower)
oyith dev, crart vww. Adla
GIE en. 178 syell covible
test Sends ceviira
120
20"
12
10'
3
Jant
Figure 4: Top row: Histograms of next state s' and reward r produced by 10 DDQN agents when trained on a CartPole SE (blue) and afterwards tested for 10 episodes on a real environment (orange). We also show the SE responses when fed with real environment data seen during testing (green). Bottom row: The same for Acrobot (subset of state dimensions shown due to brevity).
We executed these experiments with similar results for the Acrobot task (solved with a reward 2- 100, see Figure 7 in the appendix). In short, we find that the transfer to Dueling DDQN facilitates a ~38 faster training on average (18376 vs. 29531 steps, mean reward: -111). The discrete TD3 transfer shows a 8-2% (14408 vs. 80837 steps, five best models) speed-up but many of the runs also fail (reward: -343) which partly may be due to a sub-optimal hyperparameter selection. Simultaneously, we see that SE-learned policies in the Dueling DDQN transfer case are substantially better on average than those learned on the real environment. All in all, we believe these results are intriguing and showcase the potential of our concept to learn proxy RL environments.
Analyzing SE Behavior In the following, we try to illuminate the inner workings to explain the efficacy of the learned SEs. We exploit CartPole's 's and Acrobot's small state space and visualize an approximation of the state and reward distributions generated by agents trained on SEs and evaluated on real environments. First, we trained 10 DDQN agents on randomly selected SE with default HPs and logged all (s, a, r, r') tuples. Second, we evaluated the SB-trained DDQN agents on the real environment for 10 test episodes each and again logged the tuples. Lastly, we visualized the histograms of the collected next states and rewards in Figure E We observe distribution shifts between the SE and the real environment on both tasks, indicating the agent encounters states and rewards it has rarely seen during training, yet it can solve the task (reward: 199.3 on CartPole and -91.62 on Acrobot). Moreover, some of the synthetic state distributions are narrower or wider than their real counterparts, e.g., the real pole angle in CartPole is box- inside + 0.418 rad but the SEs may benefit from not having such a constraint. The synthetic reward distribution is wider than the real one, indicating that the sparse reward distribution becomes dense. The green histograms show the SE responses when fed with real environment data based on the logged state-action the agents have seen during testing. Since the green distributions align better with the blue than the orange ones, we conclude it is more likely the shifts are generated by the SE than the agents itself.
Based on these findings, we hypothesize the SEs produce an informed representation of the target environment by modifying the state distributions to bias agents towards relevant states. These results are likely explain the efficacy of SEs and can be understood as efficient agent "guiding systems". We also observed similar patterns with Dueling DDQN and discrete TD3 (see Appendix A.4)
6 EXPERIMENTS WITH REWARD NETWORKS
Experimental Setup To better understand our RN experiments, we first describe similarities and dissimilarities to the experimental setup used for SEs. Similar to SEs, to learn RNs, we first identified stable agent and NES HPs (see Appendix B.3) for each of total of four environments. Contrary to training SEs, we do not vary the agent HPs during RN training as we did not observe improvements in doing so. Now, given these HPs, we then trained multiple RNs for 50 NES outer loop iterations with 16 workers for each environment. For CartPole-vO vO we used DDQN for RN training and, similar to SEs, evaluated the transferability of DDON-trained RNs to train Dueling DDQN agents. For the custom environment Cliff Walking (Sutton & Barto 2018) (see Appendix B.2 for more details) we used Q-Learning (Watkins 1989) and show transfer to SARSA (Rummery & Niranjan 1994). We
7

---

Published as a conference paper at ICLR 2022
CUI Whrking
CartholevD
MountairCartoome au
100
1.0
-93
120
100
gal
-20
onl
Panaact
(453)
1001
-200
1000
2003
3000
4000
5000
20000
20000
30000
42800
50000
50300
L00000
208002
200000
253030
CIP Wolking Varied Hype paramerrs
CartPolo vo Varicd Hypergaramcters
Mountain www.maaincaroornnouuu... vo Varied Hyporparometerr
-10
-27
100
CL- ot3 pot
2001
#80 201 FM
-20
122 adc oct
-20
2001
1000
2003
1000
2000
S200
.O300
20000
30000
40000
50000
50:000
L00000
150080 staps
200000
253030
Kago
CIT Wolking Tansler
Corthole- Trandfe
Nour Cocrittaacn Transfer
-10
200
100
-13
150
SAECK got.
PO knC TM
POD enc Ioe qut IN RV
-093
1000
2003
$200
SECOE
206ces 353900
Figure 5: The average cumulative test rewards of agents when trained on different RN variants for one episode and evaluated on the real environment for one episode alternatingly. Top row: Training and evaluation with the same agent using default agent hyperparameters (HPs). Center row: With varied agent HPs. Bottom row: Transfer to unseen agents. We show the std. error of the mean and the flat curves in TD3 are due to filling the replay buffer hefore training.
also include MountainCarCCntinuuus (Brockman et al. (0116) and HalfCheetahh-3 (Todorov & Tassa 2012) with TD3 and transfer to PPO (Schulman et al.J (017). With these, we cover wide range of variation (discrete and continuous state/action spaces, sparse/dense rewards) and computa- tional complexity for training an RN (see Appendix C for details). All RN plots show the average cumulative lest reward as function of RN train steps. We alternated episodes of training on the RN and evaluation on the real environment and performed 100 runs (10 RNs x 10 agents) per curve on each environment except for HalfCheetah where used 25 runs (5 RNs 5 agents).
Baselines For all environments, we compare our approach multiple baselines, namely the bare agent and the bare agent extended by an Intrinsic Curiosity Module (ICM) (Pathak et al., 2017) that uses the error of predicted action effects as the synthetic reward. Additionally, for Cliff Walking we use count-based exploration (Strehl & Littman 2008) with a synthetic reward 3 n(s.a) to encourage exploration of less visited (s,a)-pairs with 3 E R20 and n : S x A + No a visitation count. For HalfCheetah, we also consider the observation vectors and v' with the position and velocity of the current and next state that HalfCheetah provides. We use these as input to the RNs with ( v) or d o'), combine them with the existing variants from Table I and tag them with augm.".
Evaluating the Performance of RN-trained Agents Like in the SE case, we study how efficient and hyperparameter sensstivveeeee the RNs are. Consider the top and middle row of Figure 5 that show how the RNs perform in training seen agents (used to optimize the RNs) as well as the performance when we vary the agents' HPs (i.e., learning rate, discount factor, etc., Appendix B.3)
8

---

Published as a conference paper at ICLR 2022
For all environments, see that there always exists an RN variant that offers significant speed-upe ups compared to the baselines not involving RNs. The variant additive potential RN (r+yd(s') ((s) implementing the original PBRS formulation (Ng al. 1999) seems to be good default choice. However, it often does not outperform the baseline, presumably trading reliability and formal guar- antees with performance. The variant additive non-potential RN (r ((s')) seems to often be the better choice, as it never failed completely and often outperformed the original PBRS variant. We ar- gue that additive non-potential RN can be seen as an undiscounted additive potential RN ( = 1 vs. 0.99) that may be better in sparse-reward and long-trajecte trajectory tasks as in Continuous Car. In contrast, exclusive potential RN (s'( hypothesize that not involving the real reward may result in challenging RN optimization. Across all tasks, RNs do not seem to overfit to the HPs (Figure 5 2nd row). The results and a discussion for HalfCheetah are given in Appendix B.1
((s) shows an increased risk of failing and we
Transferability of RNs In this experiment, we investigate the ability of RNs train unseen agents, For this, consider the bottom row of Figure 3 (Appendix Figure(for HalfCheetah). We used the default agent HPs reported in Appendix B.3 without varying them. The results show that RNs can also train unseen RL agents efficiently. However, for both Cliff Walking and CartPole some RN variants cause a deterioration of the learned policies after initially teaching well-performing policies. While the reason for this is unclear, workaround is to apply early-stopping since additive potential RN and additive non-potert non-potential RN deliver significant (Cliff W., CartPole) to moderate (Mountain- Car) speedups or higher final performance (HalfCheetah) even in the agent- scenario.
We noticed that RNs are generally less prone to overfitting than SEs and are overall easier to train. However, the components of the RN variants seem to induce important inductive biases that beg deeper understanding and may require a more isolated analysis. Nevertheless, we see clear evidence of the benefits of using RNs for efficient RI. agent training. All in all, we argue that the RN effi- ciency, like for SEs, stems from a learned informed representation that biases agents towards helpful states for completing the task quickly. In the same spirit of the qualitative analysis of SEs in Section S, similar study for Cliff Walking RNs that supports this hypothesis is in Appendix B.2
LIMITATIONS OF THE APPROACH
As reported in (Salimans et al. 2017). NES methods strongly depend on the number of workers and require lot of parallel computational resources. We observed this limitation in preliminary SE experiments when we applied our method to more complex environments, such as the HalfCheetah or Humanoid task. Here, 16 workers were insufficient to learn SEs able to solve them. Moreover, non-Markovian states and partial observability may add further complexity to the optimization of SEs and RNs. In contrast to the SE case, in the RN case, scaling our method to more complex tasks was straightforward since learning rewards is intuitively easier than learning complex state dynamics. Nevertheless, the success of learning efficient proxy models also depends significantly on hyperparameter optimization. Lastly, we observed that optimized NES HPs were transferable among different target environments in the RN case but usually not in the SE case.
8 CONCLUSION
We proposed a novel method for learning proxy models outside the current learning schemes. We apply it learn synthetic environments and reward networks for RL environments. When training agents on SEs without involving the real environment at all, our results on two environments show significant reductions in the number of training steps while maintaining the target task performance. In the RN case, we also noticed similar benefits in three of four environments. Our experiments showed that the proxies produce narrower and shifted states as well as dense reward distributions that resemble informed representations to bias agents towards relevant states. We illustrated that these proxies are robust against hyperparameter variation and can transfer to unseen agents, allowing for various applications (e.g., agent- agnostic, cheap-to- environments for AutoML, for agent and task analysis or agent pre-traini Overall, we see our method as a basis for future research on more complex tasks and we believe that our results open exciting new avenue of research.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
The authors acknowledge funding by Robert Bosch GmbH, by the Deutsche Forschungsgemein- schaft (DFG, German Research Foundation) under grant number 417962828, by the state of Baden- Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG, and by TAILOR, project funded by the EU Horizon 2020 research and innovation programme under GA No 952215.
ETHICS STATEMENT
As learned data generating processes that teach learners, our proxies may have ethical and soci- etal consequences. On the application side, re-purposed environments, for example, that allow efficiently learning policies for weapon systems. Ways to control this may include ethically reviewing both environment releases (staging releases) and the used optimization objectives, as well as regulations to enforce their disclosure. In contrast, proxies can also be used for good-natured applications, such as elderly care robotics to help analyze human safety.
the risk of learning proxies maliciously
Methodologically, we see risks in generating proxies that are purely optimized for efficiency or effectiveness they may elicit agent policies that achieve the objective by neglecting established societal and moral rules. Effects like these may reinforced by the design of the target environ- ment. While the latter could be addressed through the democratized design of target environments, the former could be tackled by using multi-objective optimization during proxy learning and by adopting fair-efficient rewards (Jabbari et al., 2017). The flexibility that the proxy optimization al- lows can also be beneficial to society, for example, by using objectives that (autonomous) vehicles. Moreover, our proxies can have positive effect the environment they target sample inefficiency in RL, e.g.. by using them for efficiently pre-training RL agents like in supervised learning and, addressed in our work, by targeting proxy transferability exploit in- vested resources across agent families. In summary, despite the possible negative implications of learning environment proxies, we find that there exist possibilities successfully address them and that the laid out merits outweigh the potential harms.
fuel-saving in
REPRODUCIBILITY STATEMENT
In our repository, we include the full code, SE / RN models, hyperparameter configurations, and instructions to reproduce all figures reported in the main paper. All our experiments use random seeds and all results are based on multiple random runs for which standard deviations are reported. Moreover, dedicate individual experiments to investigate the robustness variation of randomly sampled hyperparameters in results denoted by vary HP in Sections Jand Additionally, we report total run-time for SEs and RNs Sections 5 and 6 and discuss the computational complexity in more detail in Appendix C. Our code can be found here: httpps: http::////ttubbbcoo////tt/ttt
Learningeetviromeent
REFERENCES
M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count- based exploration and intrinsic motivation. In Proc. of NeurlPS 16, pp. 1471-1479, 2016.
Philip Bontrager and Julian Togelius. Fully differentiable procedural content generation through generative playing networks. arXiv:2002.052599 2020.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, Tang, and W. Zaremba. Ope-
nAI Gym. CoRR, abs//b06.01sa13 2016.
T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Nowé. Reinforcement learning from demonstration through shaping. Proc. of IJCAI' 2015.
Y. Burda. H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. In Proc. of ICLR' 2019.
10

---

Published as a conference paper at ICLR 2022
K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proc. of ICML'222 20, pp. 2048-2056, 2020.
M. Dennis, N. Jaques, E. Vinitsky, A. M. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and shot transfer via unsupervised environment design. In Proc. of NewrlPS 20, pp. 11349-130611 2020.
S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In Proc. of ICML 18, pp. 1437-1446, 2018.
A. Faust, A. Francis, and D. Mehta. Evolving rewards to automate reinforcement learning. In ICML 2019 Workshop on Automated Machine Learning, 2019.
Fujimoto and D. Meger H. Hoof. Addressing Function Approximation Error in Actor-Critic Methods. In Proc. of ICML 1M, pp. 1582-1591, 2018.
David Ha and Jürgen Schmidhuber. World models. arXiv: 10122, 2018.
Y. Hu, Wang, H. Jia, Y. Wang, Y. Chen, Hao, F. Wu, and C. Fan. Learning to utilize shaping rewards: A new approach of reward shaping. In Proc. of NeurIPS'20, 2020.
F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019.
B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations atari. Proc. of NeurlPS 18, pp. 8022-8034, 2018.
S. Jabbari, M. J., M. Kearns, J. Morgenstern, and A. Roth. Fairness reinforcement learning. In Proc. of ICML'' pp. 1617-1626, 2017.
M. Jaderberg al. Human-level performance in 3d multiplayer games with population- rein- forcement learning. Science, 364(6443):889-8866 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with softel- In Proc. of ICLR' 17, 2017.
You-Cyuan Jhang et al. Training a performant object detection ML model on synthetic data using Unity Perception tools.
a
uni
Sep 2020.
K. Judah, A. Fern, P. Tadepalli, and R. Goetschalckx. Imitation learning with demonstrations and shaping rewards. In Proc. of AAAI'A4 pp. 1890-1896, 2014.
Matüisen, A. Oliver, T. Cohen, and Schulman. Teacher-student curriculum learning. IEEE Trans. Neural Networks Learn. Syst. 31(9):3732-3740, 2020.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In Proc. of ICML''1, 2019.
M. Moerland, J. Broekens, and M. Jonker. Model-based reinforcement learning: A survey. arXiv:2006. 167 2020.
A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proc. of ICM1.'99, '99, pp. 278-287, 1999.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, Kopf, E. Yang. Z. DeVito. M. Raison, A. Tejani, Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. Py'oorch: An imperative style, nigh-peeffrmance deep learning library. Proc. of NeurlPS' /9, pp. 8024-8033, 2019.
L. Antiga, A. Desmaison,
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In Proc. of CVPR'17, pp. 488-489, 2017.

---

Published as a conference paper at ICLR 2022
Ingo Rechenberg. Evolutionsstrategie. Optimierung technischer systeme nach prinzipien der biolo- gischen evolution. frommann holmaann-oolzbg, stuttgart, 1973. PPSN 1973, 1973.
G. A. Rummery and M. Niranjan. On-line Q-learning using connectionist systems, volume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994.
T. Salimans, J. Ho, X. Chen, and I. Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv:/703.0388 2017.
Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1177.066:: 170..06347 [cs. 2017.
N. Shaker, Togelius, and M. J. Nelson. Procedural content generation in games. Springer, 2016.
D. Silver, J. Schrittwieser, K. Simonyan, Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, Bolton, et Mastering the game of go without human knowledge. nature, 550(7676): 354-359, 2017.
Singh, L. Lewis, A. G. Barto, and J. Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2010.
A. L. Strehl and M. L. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-133) 2008. ISSN 0022-0000.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, and Jeffrey Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In Proc. of ICML 2020.
R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In B. Porter and R. Mooney (eds.), Machine Learning Proceedings 1990, pp. 216-224. Morgan Kaufmann, San Francisco (CA), 1990.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Proc. of NeurlPS' 20, 2020.
H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, Schulman, F. De Turck, and P. Abbeel. #exploration: study of count-based exploration for deep reinforcement learning. In Proc. of NeurlPS' 17, pp. 2753-2762, 2017.
J. Tobin, R. Fong, A. Ray, J. Schneider. W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Proc. of IROS' 17, pp- 23-30. 2017.
Todorov and Erezand Tassa. Mujoco: physics engine for model-based control. In Proc. of IROS 12, pp. 5026-5033, 2012.
Togelius, G. Yannakakis. K. O. Stanley, and C. Browne. Search based procedural content generation: A taxonomy and survey. IEEE Transactions on Computational Intelligence and Al in Games, 3(3): 2011.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proc. of AAAI'1,, pp. 2094-2100 2016.
V. Volz, J. Schrum, J. Liu, S. M. Lucas, A. Smith, and S. Risi. Evolving mario levels in the latent space of deep convolutional generative adversarial network. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 221-228 2018.
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. 1rXiv:1901.0133 2019.
12

---

Published as a conference paper at ICLR 2022
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanetot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In Proc. of ICML' 16, pp. 1995- 2003, 2016.
Christopher John Cornish Hellaby Watkins, Learning from delayed rewards. PhD thesis, King's College, 1989.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In Proc. of CEC*08, pp. 3381-3387, 2008.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Proc. of NeurlPS' 18, pp. 4644 4654, 2018.
Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In Proc. of ICML'2 200 pp. 11436-11446 2020.
H. Zou, Ren, D. Yan, H. Su, and J. Zhu, Reward shaping via meta-learning. arXiv: urXiv:190.009330 2019.
13

---

Published as a conference paper at ICLR 2022
APPENDIX A SYNTHETIC ENVIRONMENTS
A.1 HEURISTICS FOR AGENT TRAINING AND EVALUATION
Determining the number of required training episodes ne on an SE is challenging as the rewards of the SE may not provide information about the current agent's performance the real environment. Thus, we use a heuristic to early-stop training once the agent's training performance on the SE converged. Specifically, let Ck be the cumulative reward of the k-th training episode. The two values C2 and C24 maintain a non-overlapping moving average of the cumulative rewards over the last d and 2d episodes up to episode k. Now, if [Ca-Czal [Caal S Caifs the training is stopped. In all SE experiments we choose = 10 and Caiss = 0.01. Agent training on real environments is stopped when the average cumulative reward across the last d test episodes exceeds the solved reward threshold. In case no heuristic is triggered, we train for n, = 1000 episodes at most on SEs train agent
and real environments. Independent of whether assess the actual agent performance is equivalent: we run the agent on 10 test episodes from Ereal for fixed number of task task-specific steps (200 on CartPole and 500 on Acrobot) and use the cumulative rewards as the performance.
&reat or Esyn, the process to
A.2 DISCUSSION ON THE FEASIBILITY OF LEARNING SES WITH OUR ALGORITHM
After identifying stable DDQN and NES hyperparameters (HPs), we Algorithm T in parallel with 16 workers for 200 NES outer loop iterations for both CartPole and Acrobot. Each worker had one Intel Xeon Gold 6242 CPU core its disposal, resulting in an overall runtime of 6-7h on Acrobot and 5-6h on CartPole for 200 NES outer loop iterations. For reference, we note that (Salimans et al., 2017) used 80 workers each having access to 18 cores for solving the Humanoid task.
In Figure 6 it becomes evident that the proposed method with the given resources and used hy- perparameters given by Table 3 allows identifying SEs that can teach agents to solve CartPole and Acrobot tasks respectively. Each thin line in the plot corresponds to the average of 16 worker eval- uation scores given by EvaluateAgent in Algorithm aas function of the NES outer loop iteration. We repeat this for 40 separate NES optimization runs and visualize the average across the thin lines as a thick line for each task. We note that we only show a random subset of the 40 thin lines for better visualization and randomly sample the seeds at all times in this work. We believe that the stochasticity introduced by this may lead to the variance visible in the plot when searching for good SEs.
Both the stochasticity of natural gradients and the sensitivity of RL agents to seeds and parameter initializations may additionally contribute this effect. Notice, it is often sufficient run approxi- mately 50 NES outer loops to find SEs that solve a task. Besides early-stopping, other regularization techniques (e.g. regularizing the SE parameters) can be applied to address overfitting which we likely observe in the advanced training of the Acrobot task.
A.3 EVALUATION OF PERFORMANCE AND TRANSFERABILITY OF SES ON THE ACROBOT TASK
Similar to Figure 3 we visualize the performance and transferability evaluation results of SEs for the Acrobot-vl task. Ágain, the top row depicts the density functions of the average cumulative rewards collected by DDQN, Dueling DDQN and discrete TD3 agents based on 4k evaluations per density and DDQN-trained SEs. Like the CartPole case, we show three different SE training settings: agents trained on real env. with varying agent HPs (blue), on DDQN-trained SEs when varying agent HPs during NES runs (orange), on DDQN-trained SEs where the agent HPss were fixed during training of the SEs (green). The bottom row visualizes the average reward, train steps, and episodes across the 4k evaluations and each bar correspond to one the shown densities. We achieve on Dueling DDQN more stable training compared to the baseline. However, we point out that many of the runs with discrete TD3 also fail (mean reward: -343). This is likely caused by the limited transferability already observed in the Cartpole case but may also due to sub-optimal hyperparameter selection since reused the DDQN HPs found Cartpole for Acrobot (as well the HPs and ranges for
38% (18376 vs. 29531 train steps) and on discrete TD3 82% faster and
14

---

Published as a conference paper at ICLR 2022
200
100
curulaatvv reward
-100
-200
-300
CartPole-vC solved threshold
-400
CartPole-vO
Acrobot-v1 solved threshold
500
Acrobot-v1
0 25 50
75
100
125
150
175
ES outer loop Iteration
Figure 6: Results from 40 different NES runs with 16 workers each (using random seeds) show that our method able to identify SEs that allow agents to solve the target tasks. Each thin line corre- sponds to the average of 16 worker evaluation scores returned by EvaluateAgent in our algorithm as a function the NES outer loop iterations.
We further point out that training on SEs consistently yields a more stable training can be in the standard deviations of the train steps and episodes bars.
variation from Table 2)
Dueling ODQN on SE:
0.020
0.034
tmit: life unied bec)
-10
3303
2200
-300
15:0000
102000
153000
25C
52000
100000
45% varicc
MPS fieed
4% vinee
HSS feee
HR:: viree
HSS feed
Figure 7: Evaluation of performance and transferability of SEs on the Acrobot-vl task.
A.4 QUALITATIVE STUDY OF SYNTHETIC ENVIRONMENTS FOR THE CARTPOLE AND ACROBOT TASKS
To extend the qualitative analysis of SE behavior from Section 9 depict in Figure additional histograms for the approximate next state s' and reward r distributions produced by 10 agents with random seeds and default hyperparameters from Table 5 for each histogram plot. We here addi- tionally show the behavior for Acrobot and also for Dueling DDQN and discrete TD3 agents. For the histograms with discrete TD3 we chose well-performing but arbitrary SE model according the individual model analysis, see Section F.I.I and F.1.2) We chose well-performing model in the case of discrete TD3 since we assumed that the significant number of failing runs would deteriorate the discrete TD3 performance and distort the comparison to the other (well- (well-perfommigg) DDQN and Dueling DDQN agents. For the DDQN and Dueling DDQN agents, we chose the SE model com- pletely at random, independent of their performance. In summary,
the same patterns that
15

---

Published as a conference paper at ICLR 2022
observed for CartPole and DDQN in Section 9 also with Dueling DDQN and discrete TD3 on both tasks: distribution shift, narrower synthetie distributions (and stabilitai or control behavior, meaning that the number of large positive and negative states are rarely encountered, for example, this can be seen in the cart velocity (CartPole) and the joint angular velocity (Acrobot) plots), and wider/dense reward distributions. The histograms in green show the SE responses when fed with real environment data based on the logged state-action the agents have seen during testing. Since the green distributions align better with the blue than the orange ones, we conclude it is more likely the shifts are generated by the SE than the agent. For CartPole we show all state dimensions and for Acrobot we show four of the six state dimensions due to brevity.
10
pptEs ETN
10
systh enz dira
107 SOmmmm
Syrt
EyrTs anrisl
2010)
357
Orenn
10"
Dest
10
100
30P
1UF
20F
carl postiio (nest stano) In!
+3
cart atlocity (nest stann) [mr]]]
pole angle Oveect STaTel [rad]
sudad butia)
0.01
rewand
1.0
10"
10"
(est)
byrth
syath een, dera
aynts an da hale ava
30
30
1FF
10%
0.0
0.5
1.0
24
0.5
3.0
3.5
carl postiio state) [m]
cart uelocity (nast state)) [1100
pow angle OvEect statel [rwd]
dlate bratt)
10a
10r0% ETN IORIA cie ascind 4..%
10
syets WON
10'
EI ow E1 em cata
32*
th oum
200
32
-0.5
0.0
0.5
30
0/6 0.5
1.5
0.5
1.5
0.0
0.5
15
cart posdlt jeact stare) (m]
cart by joact stare] [mvh)]
pcha amalie (rant state) (144]
rbward
quet neu SUAM
(SARE LiV sock
10
10"
wuth ver Aula
Syrth auree en Cit
cooumonne
syels on corido
10
12
100
101
100
-10
(oan: lood)
Jonn 3 [Mr]
pani (urd
Jan1 1 (1S1
reward
20"
10"
with
1.'
2010
10"
syets em tren tyeth anc.
10"
1e-th raadd
cyeth 4nv.
zytT, anival
200
101-
20-
200
20
427
20F
John: 1 lood)
joint 1 1or3
joint [om)
Joit TUS1
mwwadd
sytth ena (Ea0))
SyrPh
syelt em tran tyets enc.
10
syyet uen clita
syath any. 2tth
Ayrth anieal
20P
10t
120
20
10P
11F
102
127
20F
-2
-10
Jant 1 [cza]
1aln 1 1972
Joint [cm)
reward
Figure 8: Histograms of approximate next state s' and reward r distributions produced by 10 DDQN or Dueling DDQN agents when trained on an SE (blue) and when afterwards tested for 10 episodes on real environment (orange) for each task. 1st row: CartPole and DDQN, 2nd row: CartPole and Dueling DDQN, 3rd row: CartPole and discrete TD3, 4th row: Acrobot and DDQN, 5th row: Acrobot and Dueling DDQN, 6th row: Acrobot and discrete TD3
The average cumulative rewards across the 10 runs for each agent variant and task are:
CartPole and DDQN (Ist row): 199.3
CartPole and Dueling DDQN (2nd row): 193.75
CartPole and discrete TD3 (3rd row): 197.26
16

---

Published as a conference paper at ICLR 2022
Acrobot and DDQN (4th row): -91.62
Acrobot and Dueling DDQN (5th row): -88.56
Acrobot and discrete TD3 (6th row): -93.17
A.5 SYNTHETIC ENVIRONMENT HYPERPARAMETERS
The following table provides an overview of the agent and NES hyperparameter (HP) ranges that we optimized in an additional outer loop of our algorithm. The HPs in the 2nd and 3rd columns represent the results of this optimization and which we used learning SEs. Note that in experiments where sampled agent HPs from ranges given by Table we overwrite a small subset (namely DDQN learning rate, batch size, DDQN no. of hidden layers, and DDQN hidden layer size) of the listed HPs below (runs denoted by "HPs: varied").
Table 3: Hyperparameter configuration spaces optimized for learning SEs.
Hyperparameter
CartPole-vO
Acrobot-vl
Optimization value range
log. scale
NES step (a))
0.148
0.727
0.1
True
NES noise std. dev. (a)
0.0124
0.0114
0.01 -1
True
NES mirrored sampling
True
True
False/True
NES score transformation
All Better 2
All Better 2
see Appendix Section
E.
NES SE of hidden layers
1
1
False
NES hidden layer size
83
167
192
True
NES SE activation function
LReLU 1
PReLU
Tanh/ReLULRRLLLPRRRLL
DDQN initial episodes
20
20
True
DDQN hatch size
199
149
64 256
False
DDQN learning
0.000304
0.00222
0.0001 0.005
True
DDQN target net update rate
0.00848
0.0209
0.005 0.05
True
DDQN discount factor
0.988
0.991
0.9 0.999
True (inv.)
DDQN initial epsilon
0.809
0.904
0.8 - -1
True
DDQN minimal epsilon
0.0371
0.0471
0.005 0.05
True
DDQN epsilon decay factor
0.961
0.899
0.8 0.99
True (inv.)
DDQN no. hidden layers
1
False
DDQN hidden layer size
57
112
48 - 192
True
DDQN activation function
Tanh
LReLU
Tanb/ReLU/LRLLUPRR.i
The following table lists the hyperparameters that we always kept fixed and which were never opti- mized:
Table Subset of the hyperparameter configuration spaces that we kept fixed (not optimized) for learning SEs.
Hyperparamcter
Symbol
CartPole-vO
Acrobot-vi
NES number of outer loops
200
200
NES number of train episodes
ne
1000
1000
NES number test episodes
10
NES population size
16
16
DDQN replay buffer size
100000
100000
DDQN early out number
10
10
DDQN carly out difference env. max. episode length env. solved reward
Casti
0.01
0.01
200
500
195
-100
17

---

Published as a conference paper at ICLR 2022
The table below lists the agent hyperparameters (HPs) that chose default. We used these HPs in all experiments after SEs were found, i.. for evaluations Figures 3 and 7 Note that in some cases we sampled agent HPs from ranges given by Table 2 In these cases (runs denoted by "HPs: varied", "HPs: fixed", and also the individual SE. models plots below) the HPs of the table overwritten by the sampled ones.
Table 5:
Default agent hyperparameters for the evaluations depicted in Figure and early out number and DDQN early out difference are equivalent to Table
DDQN
Agent hyperparameter
Default value
initial episodes
10
batch size
128
learning rate (DDQN D.DDQN/ TD3)
0.001 0.0005
target network update rate
0.01
discount factor
0.99
epsilon decay factor number of hidden layers
0.9
hidden layer size activation function (DDQN & D. DDQN / TD33)
ReLU Tanh
replay buffer size
100000
max. train episodes
1000
Gumbel Softmax start temperature heeeeoo--ecddd actions (TD3)
1 False
18

---

Published as a conference paper at ICLR 2022
APPENDIX B REWARD NETWORKS
B.I HALFCHEETAH RNS PLOTS
For HalfCheetah, we see that additive non-potential RN (and also exclusive non-poteen con-potential RN) trains well perf-oreron agents the fastest but saturates at a cumulative reward of ~3k which corresponds to the used solved reward threshold. This may be due to the used objective function since its term mar (0, Cool
Cyin) does not incentivize RNs to improve after reaching the threshold.
HallChessah-v3
HaftChesah-2 +2 Varied Pyperparometers
HallChoctah +2 Transler
3300
3000
102 eBc nen pot
3002
TDS
3000
TEE ASS pit
(03 esc, pet Bil sagn
TOO
100
500
1003
TDD 1u.
TES
1003
TOS eagn
-500
-1000
-1000
1100
and
Strps
142
Figure 9: Performance of different RN variants for the HalfCheetah-v3 environment. Left: perfor- mance of RNs that train the known TD3 agent with default hyperparameters. Center: performance of RNs when the hyperparameters of known agents are varied. Right: Transfer performance to the unseen PPO agents with default hyperparameters. All curves denote the mean cumulative reward across 25 runs (5 agents X 5 RNs). Shaded areas show the standard eπтоr of the mean and the flat curves in TD3 stem from fixed number of episodes that are used fill the replay buffer before training. Notice that using the observation vector in the RN variants (denoted by "+augm.") only marginally yields a higher train efficiency.
HolfCheetah-3
HortChoetah v3 Veried Hyperparoneters
HolChhetth v3 Transter
4333
0931
3001
4300
4000
3900
-1000
20000
40000
69003
SCOCo
J80085
-1000
20300
40000
60000
aCOCO
100003
0.55
0.25
0/90
6.75
1.00 stape
125
150
135
2.00
Viga
Mago
Figure 10: This is the zoomed in version of Figure g
B.2 QUALITATIVE STUDY OF REWARD NETWORKS ON THE CLIFF WALKING TASK
Like in Section A.4 we again make use of the small state and action space and analyze the learned augmented rewards of the proposed different RN variants across 50 generated RN models Cliff Walking environment (Sutton et al. 2020). Cliff Walking consists of a two-dimensional grid with 4x 12 states in which the agent, starting at the start state (S), has to reach the goal state (G) by moving up, down, left, or right to adjacent states. 10 of the states in the lower row are cliffs. The agent receives a reward of 1 for every action it takes and a reward of -000 if it enters a cliff state. The episode terminates if the agent enters either the goal state or a cliff state. The environment is further bounded by walls, e.g. if the agent moves left when being in the state (S), it stays in the state (S).
we depict a visual representation of the learned rewards for different RN variants averaged across 50 RN models taken from the RN experiment in Section 6 (i.e. trained with a Q-Learning agent with non-varying HPs, no early stopping criterion in the outer loop, 50 outer loop iterations). To do this, we color-coded the rewards we received from the RNs for all possible state-action pairs. To visualize the rewards corresponding to each of the four actions, every state is split up into four triangles corresponding to the different actions (move up, down, left, right) that the agent can perform when being in this state. We show small, average, and high rewards in red, yellow, and green, respectively. The left column of Figure TT the grids with all rewards
In Figure I1
19

---

Published as a conference paper at ICLR 2022
where observe that the negative rewards of 100 of the additive RNs distort the color plot such that the fine-grained reward structure learned by the RNs becomes invisible. To address this, the right column features the same plots while ignoring any rewards I -50. This implies for the additive RN variants, that the rewards the cliff states are ignored (white color) since the additive RN variants produce large negative rewards and lead to the described distortion of fine-grained rewards. For the goal state we further note that, although actions were never executed in this terminal state, the (neural network of the) RNs still compute rewards in this case which we did ignore in the plot simply because it did not lead to distortions.
Matching the performances shown in Figure 5 (top row, left), the reward structure learned by the exclusive potential RN is sub-optimal the high rewards are nearly randomly distributed. In con- trast, the additive potential RNs clearly assign low rewards to any state-action pair that lets the agent enter cliff state. The RNs further output low reward for transitions that go from the third row up to the second row and high reward for the contrarily directed transitions. This keeps the agents close to the cliff and reduces the exploration space which results faster training. The exclusive non-potential RNs assign useful rewards around the start and goal states but fail adequately shape the reward along the shortest path from the start to the goal state. Lastly, the additive non potenttial RNs clearly assign high rewards that guide the agent along the shortest path towards the goal state while avoiding cliff states. As expected, the well-performing RN variants seem to learn informed reward representations of the task that guide the agent towards relevant states for completing the task like in the SE case.
B.3 REWARD NETWORK HYPERPARAMETERS
B.3.1 GENERAL NES AND ENVIRONMENT HYPERPARAMETERS
The below table lists all HPs for learning RNs that we kept fixed (not optimized) and which are identical across all RN experiments:
Table 6: Hyperparameters that kept fixed (not optimized) for learning RNs.
Hyperparameter
Symbol
Value
NES numher of outer loops
no
50
NES numher of test episodes
Nic 16
NES population size
Up
NES unsolved weight
what
100
Below table lists HPs for learning RNs that we optimized and which are identical across all experiments. The NES score transformations are described in detail in Appendix Section E. and we also report the results for the BOHB hyperparameter optimization of different score transformation variants in Appendix Section E.2
Table 7: Hyperparameters that optimized and which are identical across all RN experiments
Hyperparameter
Symbol
Optimized value
NES mirrored sampling
True
NES score transformation
Better (see Appendix Section E.1
NES step size
a
0.5 0.1
NES noise std. dev.
Below table lists the environment-specife HPs:
Table 8:
Environment-specffcc hyperparameter
Hyperparameter max. episode length solved reward
Cliff Walking
CartPole-vO
MountaiinaarrComniiuuuuu---
HalfCheetah-v3
1000
-20
195
90
3000
NES max. number
100 (TD3)
100 (TD3)
100
100
of train episodes
1000 (PPO)
1000 (PPO)
20

---

Published as a conference paper at ICLR 2022
exclusive potential reward network
exclusive potential reward network (only rewards -50)
(5)
cliff
(G)
(5)
cliff
(G)
additive potential reward network
additive potential reward network (only rewards -50)
[S)
cliff
(G)
(S)
cliff
GG)
exclusive non-potential reward network
exclusive non-potential reward network (only rewards -50)
(S]
cliff
(G)
(S
clff
(G)
additive non-potential reward network
additive non-potential reward network (only rewards -50)
(S)
cliff
(G)
(5)
clfff
Figure 11: Learned Cliff Walking reward for different reward network types. Shown is the start state (S), goal state (G), all cliff states the lower row and the learned reward each state-action pair (red: low reward, yellow: medium reward, green: high reward). The plots on the left side show averaged values across 50 reward networks when considering all rewards, the plots on the right side ignore all rewards that are < -50. Values have been normalized to be in [0.1] range in both cases (left with full reward range, right without ignored rewards range).
B.3.2 VARIED AGENT HYPERPARAMETERS
The following tables list the for Q-Leariin, DDQN and TD3 agent hyperparameter ranges from which we sampled in the HP variation experiment.
21

---

Published as a conference paper at ICLR 2022
Table Hyperparameter sampling ranges for the reward networks HP variation experiment
QL hyperparameter
Value Range
log. scale
learning rate
0.1
False
discount factor
0.1
False
DIQON hyperparameter
Value range
log. scale
learning rate
8.3
10 7.5 10
True
batch size
11 - 96
True
hidden size
21 192
True
hidden layer
1 2
False
TD3 hyperparameter
Value range
log, scale
learning rate
4 10
True
batch size
85 768
True
hidden size
42 - 384
True
hidden layer
1-3
False
B.3.3 AGENT AND NES HYPERPARAMETERS FOR CLIFF WALKING
In Table TO list the NES and Q-Learning hyperparameter configuration spaces that we optimized for learning RNs on Cliff Walking. In Table IT we list the default Q-Learning and SARSA (transfer) hyperparameters that used once RNs have been learned. In Table 12 we list the hyperparameter configuration spaces that we used (and optimized over) for the count- count-based Q-Learning baseline.
Table 10: Hyperparameter configuration spaces optimized for learning RNs on Cliff Walking.
Hyperparameter
Optimized value
Value range
log. scale
NES RN number of hidden layers
False
NES RN hidden layer size
32
32 192
True
NES RN activation function
PReLU
Tanb/ReLU//LRLLUPRROO
Q-Learning learning rate
0.001
True True (inv.)
Q O-Leaning discount factor
0.8
0.001
O-Learnin initial epsilon
0.01
0.01
Truc
O-Learning minimal epsilon
0.01
0.01
Truc
O-Learning epsilon decay factor
0.0
0.001
True (inv.)
Table 11: Default Q-Learning and SARSA hyperparameters that we used after learning RNs.
Hyperparameter
Default value
O-Learning & SARSA learning rate
O-Lernnin & SARSA discount factor
0.8
Q Learning & SARSA initial epsilon
0.1
Learning SARSA minimal epsilon
Q- Learning & SARSA epsilon decay factor
0.0
Table 12: Used baseline hyperparameter configuration spaces
Hyperparameter
Optimized value
Value range
log. scale
Count- based Q-I .earming 3
0.1
0.0001
2
False
22

---

Published as a conference paper at ICLR 2022
B.3.4 AGENT AND NES HYPERPARAMETERS FOR CARTPOLE-VO
In Table 13 we list the NES and DDQN hyperparameter configuration spaces that we optimized for learning RNs on CartPole. We do not list the optimization value range this table since they are equivalent as in the learning SE case described in Table 3 In Table 14 we list the default DDQN and Dueling DDQN (transfer) hyperparameters that we used once RNs have been learned. In Table 13 we list the hyperparameter configuration spaces that we used (and optimized over) for the ICM baseline (Pathak et al. 2017). We note that, despite optimizing the ICM HPs, we did not find that the optimized ICM HPs yielded better results than the default ones which is why we use the default ICM HPs in all experiments that involve ICM.
Table 13: Optimized hyperparameters for learning RNs on CartPole. The value ranges are the same as in Table Hyperparameters not listed were not optimized.
Hyperparameter
Optimized value
NES RN number of hidden layers
NES RN hidden layer size
64
NES activation function
PReLU
DDQN initial episodes
DDQN batch size
192
DDQN learning rate
0.003
DDQN target network update
0.01
DDQN discount factor
0.99
DDQN initial epsilon
0.8
DDQN minimal epsilon
0.03
DDQN epsilon decay factor
0.95
DDQN number of hidden layers
1
DDQN hidden layer size
64
DDQN activation function
LReLU
DDQN replay buffer size
1000000
Table 14: Default DDQN and Dueling DDQN hyperparameters that used after learning RNs.
Hyperparameter
Default value
DDQN & Dueling DDQN
initial episodes
DDQN Dueling DDQN batch size
32
DDQN & Dueling DDQN learning rate
0.00025
DDQN & Ducling DDQN target network update rate
0.01
DDQN & Ducling DDQN discount factor
0.99
DDQN Dueling DDQN initial epsilon
1.0
DDQN & Ducling DDQN minimal epsilon
0.1
DDQN & Dueling DDQN epsilon decay factor
0.9
DDQN & Dueling DDQN number of hidden layers
1
DDQN & Dueling DDQN hidden layer size
611
DDQN Dueling DDQN activation function
ReLU
DDQN& & Dueling DDQN replay buffer size
1000000
Dueling DDQN feature dimension
128
Dueling DDQN value and adv. stream no. hidden layers
1
Ducling DDQN value and adv. stream hidden layer size
128
B.3.5
AGENT AND NES HYPERPARAMETER FOR MOUNTAINCARCNNTINUOOS VO AND HALFCHEETAH-V3
In Table 16 we list the NES and TD3 hyperparameter configuration spaces that we optimized for learning RNs MountainCarContinuur (CMC) and HalfCheetah (HC). In Table 17 we list the default TD3 hyperparameters that we used once RNs have been learned. Table 18 shows the PPO agent hyperparameterss we used for showing transfer of RNs to train PPO agents. In Table 19 list the hyperparameter configuration spaces that we used (and optimized over) for the ICM base- line (Pathak et al. 2017). We note that, despite optimizing the ICM HPs, did not find that the optimized ICM HP' yielded better results than the default ones which is why we use the default ICM HPs in all experiments that involve ICM.
23

---

Published as a conference paper at ICLR 2022
Table 15: Used ICM (Pathak 2017) baseline hyperparameter configuration spaces optimized with CartPole and DDQN. We additionally report the default values taken from the ICM reference implementation.
Hyperparameter
Default value
Optimized value
Value range
log. scale
ICM learning rate
0.0001
0.00001
0.00001
0.001
True
ICM g
0.2
0.05
0.001
True
ICM
0.5
0.03
0.001
True
ICM feature dimension
64
32
16-256
True
ICM hidden layer size
128
128
16-256
True
(forward, feature and inverse model)
ICM number of hidden layers
not optimized
(forward, feature and inverse model)
ICM activation function
LReLU
optimized
ICM number residual blocks
not optimized
Table 16: Optimized hyperparameters for learning RNs on MountainCarrCnniiunni (short
"CMC") and HalfCheetah-v3 (short "HC"). Hyperparameters not listed were not optimized.
Hyperparameter
Optimized value
Value range
log. scale
NES RN number of hidden layers
1-2
False
NES RN hidden layer size
128
48-192
True
RN activation function
PReLU
Tath/Ree.U/LRRLLUUPRLLLU
TD3 initial episodes (CMC/HC)
50 / 20
1-50
True
TD3 batch size
192
64 256
False
TD3 learning rate
0.003
0.0001 0.005
True
TD3 target network update rate
0.01
0,005 0.05
True
TD3 discount factor (CMC/HC)
0.99 0.98
0.9 0.99 1-3
True
TD3 policy update delay factor
False
TD3 number of action repeats (CMC/HC)
2/1
1-3
False
TD3 action noise std. dev.
0.05
0.05 0.2
True
TD3 policy noise dev.
0.2
0.1 0.4
True
TD3 policy noise std. dev. clipping
0.5
0.25
True
TD3 number of hidden layers
2
1-2
False
TD3 hidden layer size
48 192 LRcLU ReLU Taab/ReLU/LLeLL/PRLLL not optmized
128
True
TD3 activation function (CMC/HC)
TD3 replay buffer size
100000
Table 17: Default TD3 hyperparameters that we
used after learning MountainCarConinnouss (short "CMC") and HalrCheetah-v3 (short "HC").
RNs on
Hyperparameter
Default value
TD3 initial episodes (CMC/HC)
507 20
TD3 batch size
TD3 learning rate
0.0003
TD3 target network update rate
0.005
TD3 discount factor
0.99
TD3 policy update delay factor
TD3 number of action repeats (CMC/HC)
2/
TD3 action noise std. dev.
0.1
TD3 policy noise std. dev.
0.2
TD3 policy noise std. dev. clipping
0.5
TD3 number hidden layers
TD3 hidden layer size
128
TD3 activation function
ReLU
TD3 replay buffer size
100000
24

---

Published as a conference paper at ICLR 2022
Table 18:
PPO (clip) hyperparameters that used for showing the transfer of RNs on MounaiinCarContinuuss-O0 (short "CMC") and HalfCheetah-v3 (short "HC").
Hyperparameter
Default value
PPO epochs (CMC/HC)
80 / 10
PPO learning rate
0.0003 0.00001
PPO number episode updates (CMC/HC)
10/
PPO discount factor
0.99
PPO policy update delay factor
2
PPO number of action repeats (CMC/HC)
5/1
PPO value function coefficient
1
PPO entropy coefficient (CMC/HC)
0.01 0.001
PPO clip ratio
0.2
PPO number of hidden layers
2
PPO hidden layer size (CMC/HC)
64/128
PPO activation function (CMC/HC)
RcLU / tanh
Table 19: Used ICM (Pathak 2017) baseline hyperparameter configuration spaces optimized on CorootinnnCConiiuuuuus.... (short "CMC") and HalfCheetah-v3 (short "HC") with TD3. We additionally report the default values taken from the ICM reference implementation.
Hyperparameter
Default value
Optimized value
Value range
log scale
ICM learning rate (CMC/HC)
0.0001
0.0005 0.00001
0.00001 0.001
True
ICM 3 (CMC/HC)
0.2
0.1 / 0.001 0.01 / 0.1
0.001
True
ICM (CMC/HC)
0.5
0.001
True
ICM feature dimension
64
32
16-256
True
ICM hidden layer size
128
128
16-256
True
(forward, feature and inverse model)
ICM number of hidden layers
not optimized
(forward, feature and inverse model)
ICM activation function
I.ReLU
optimized
ICM number residual blocks
not optimized
25

---

Published as a conference paper at ICLR 2022
B.4 STUDYING THE INFLUENCE OF DIFFERENT REWARD NETWORK OPTIMIZATION OBJECTIVES
After we noticed the efficiency improvements of SEs, tested whether RNs could also yield such improvements by directly optimizing for efficiency. Another reason for optimizing for efficiency directly was that the real environment is involved in the RN case and we pursued the goal to reduce real environment interactions. Based on this, we developed the RN objective reported in the paper in Section which we refer to as the Reward Threshold objective. However, to analyse whether different objectives would yield similar results, we conducted small study on the environments to assess the influence of different objectives.
For this, we considered the rrea-under-the-cure of the cumulative returns objective (AUC) as well as the cumulative reward maximization objective (max reward) which is the reward threshold objective with just the Cinn term.
The overall conclusion from these results is that an AUC objective generally yields similar results as for the reward threshold objective, but in some cases it seems to minorly reduce the efficiency. However, the overall significant efficiency gains remain. Moreover, in contrast to SEs, efficiency improvements in the RN case seem to arise more often when explicitly optimized for it (compare AUC vs. reward threshold below) while maximizing reward (only tested for CartPole and Cliff en- vironments) leads efficiency improvements only in the CartPole but not in the Cliff environment.
B.4.1 CLIFF
Clif Warking
CI Walking Voried Hypersormmccars
O Walking Trarsfer
-10
-10
-23
-20
-20
-40
-40
1000
2003
3000
4030
SC00
-100
1000
2000
30C0
6000
5200
-100
1000
2000
3000
4000
5903
Kaya
staps
Figure 12: The Cliff environment RNs optimized with the AUC objective.
CMll Warking
Varking Vemted Hyp ramernamaa
Wallingg Therader
-12
-10
-10
-33
-30
gel.
Pannact
325 rarpes KM
net put
SAPAA
1000
2003
2000
4030
5000
1000
2003
3000
4000
5200
1000
2000
3000
4000
5203
Figure 13: The Cliff environment RNs optimized with the max reward objective.
WiNking
Vakiing Hynerpprameees
-43
-00
1000
2002
3000
-0.00
5000
1000
2033
5000
5000
5090
1000
2000
3020
4000
5293
Figure 14: The Cliff environment RNs optimized with the reward threshold objective (similar to the results reported in Section 6).
26

---

Published as a conference paper at ICLR 2022
B.4.2 CARTPOLE
CartPole
Cortisle vl Varled Hyporparamotoos
Corthale vb Trandfer
75
75
Buc 2003
10000
20000
4C0C0
10392
33030
ACOCO
6000
stpo
Mage
Figure 15: The CartPole environment RNs optimized with the AUC objective.
CartPole-00
Cart/sle-vo Varied Hyperpar amsters
CarcPsle-vD Transter
17%-
75-
0009
Dut
abll
25
1831
11
18813
10020
20000
33303
4c000
50000
28292
20000
33000
4DECO
50000
2000
4000
0000
10000
S106
S00P6
Stops
Figure 16: The CartPole environment RNs optimized with the max reward objective.
Cantoodeud
Cart-ite-O Vasted Hyperppppeeee smeters
CartPieeeD Trwnsder
1.7
1.0
1.50
125
125
125
203
100
100
75
30-
3000
10000
20000
32303
aCOCE
2000
J0300
20000
33000
ALOCO
50000
10303
20000
20000
4D000
S4000
1000
Kaga
staa
Figure 17: The CartPole environment RNs optimized with the reward threshold objective (similar to the results reported in Section 6).
27

---

Published as a conference paper at ICLR 2022
B.4.3
MOUNNAIICARCCNTINNOUSS
N5 Uounnta nCanContnuuuss V0 Vorted Hyporio timttoss
CountonCarComtuuuuu vO Transfer
97-
03
00
43-
CUP www.ttteee
T00
100
-03
171
-00
50000
329030
150000
200000
2SC0CC
50330
100000
20000
252033
100000
156582
20000
253030
2005
Maps
saga
Figure 18: The MountainCurCCntimmuus environment RNs optimized with the AUC objective.
Mounta mncarCort nubus-vO Varied Hyper barumeters
Mounta Car.innaaannnnaaaooaoo Transter
100
33
MeWll Nivtt Aaly heteerry
8
50020
323020
150000
200003
250000
53232
120000
1500ec
200000
252023
50303
100000 2500ez 200000 252030
S106
5006
Stops
Figure 19: The MountanCCrrCnninouus environment RNs optimized with the reward threshold objective (similar to the results reported in Section 6)
B.5 HALFCHEETAI
3
IarCCheraah-ecc Veried Eyperperametton
HallCheetae-C -D3 Thatsler
0302
0000
5303
5000
4302
3353
TDO und pal
5003
TOO und pal,
(S0+ aus raniac
3u3 manject Mi
.303
TO3 HCM
TDD sigs.
1000
TED
TO3
sag's
1116
-509
-1003
100000
200030
200000
090000
SEC000
-1000
100000
230030
200CeC
00000
-1000
and
stpo
Mags
Figure 20: The HalfCheetah environment RNs optimized with the AUC objective.
HallChettah-v
HullChartah- vd Varied Hyperparantters
HallChoota* vid TTanster
0302
5030
3332
2000
4302
TD3 acd gat
4000
3303
LSE Jack Fanract
3000
AGk
2303
TDS HCM
TOD sigs.
2000
TOO edd menves FM
1003
102
TOS
TOO eSC
-1003
100000
209030
300000
300300
5E0000
1000
100000
230030
200000
550033
1000
Zapa
Eaga
staps
Figure 21: The HalfCheetah environment RNs optimized with the reward threshold objective (simi- lar to the results reported in Appendix Section B.I.
28

---

Published as a conference paper at ICLR 2022
APPENDIX C COMPUTATIONAL COMPLEXITY FOR TRAINING SYNTHETIC ENVIRONMENTS AND REWARD NETWORKS
Training both SEs and RNs can be computationally demanding, depending on the environment com- plexity and the available resources per worker. Table 20 lists approximate timings for a single NES outer loop iteration based our log files. Every population member had one Intel Xeon Gold 6242 CPU core at its disposal.
Table 20: Approximate timings for one NES outer loop for SEs and RNs
Environment
Synthetic Environment
Reward Network
Cliff Walking
10s
Acrobot-v1
100s
CartPole-vO
100s
100s
MountainCarConninuuus--0
500s
HaliCheetah-v3
2000s
Given 200 NES outer loop iterations in the SE case and 50 steps in the RN case, these timings result in the following overall runtime for finding one SE or RN, respectively:
SEs:
Acrobot: ~ 6-7 hours
CartPole: 5-6 hours
RNs:
Cliff Walking: ~ 8 minutes
CartPole: 7 hours
MountainCarContinuunn 7 hours
HalfCheetah: 28 hours
We note that the required time can vary by more than a magnitude, depending on how quickly the RL agent can solve the environment in the inner loop.
APPENDIX D SUPERVISED LEARNING AND SYNTHETIC ENVIRONMENTS
One might ask how our proposed way of learning synthetic environments relates and compares to purely learning a synthetic environment model through a supervised learning (or model- based RL (MBRL)) objective. In this Section, discuss the differences from our view point and show results of a small experiment that aimed at giving some answers to this question.
D.1 CONTRASTING BOTH APPROACHES
First, our interpretation can be summarized with the following table:
We now elaborate in more detail what we believe are the core differences between both concepts and their resulting effects.
In the case the supervised learning of the model as jointly learned mapping from (state, action) pairs to the next state and the reward, we are limited by the human prior in regards to what is beneficial and informative to achieve a certain given task. This prior typically is encoded in the reward. Whereas the states of problem almost always arise from physical (e.g. Cartpole) logical/rule based (e.g. board games) foundation, the reward to a much higher degree based on a human perspective. This limits the supervised approach to almost only replication of the problem designer's is choices. Now, it might be argued that therefore it a pure approach for modeling world environment, but by shifting the perspective towards the artificial nature of reward signal,
29

---

Published as a conference paper at ICLR 2022
Table 21: Comparing attributes between supervised learning and SE learning.
Synthetic Environments
Supervised Learning
Learning Objective
Mapping (S.A) + (S, Raefinea)
Mapping (S, + (S, /scoe/ssial)
Signal of Information
Performance of agent on given real environment
Error in mapping like MSE, MLE
A functional mapping same input-output dimensionality as in the supervised case but defined to increase agents performance; Quality in terms of reward given real environment
Learned Function
mapping defined by environment; quality in terms of closeness
Can imitate state transitions but is not restricted to this and can learn very different environment that nevertheless helps quickly train agents do well on the real environment
Imitaterreplicate the given environment
Potential
the choice of reward might as well be source of misguidance or least a potential hindrance to learning in a effective / efficient, and more unbiased fashion.
We believe that our approach reduces the potential shortcomings of human bias and choice of design to some degree by taking more "task-aware" perspective. In our approach, the feedback signal for the model training comes from the given environment (similar to supervised model learning), but does not constrict the model to explicitly learn approximate duplicate mapping. Rather, by using the performance of the agent as the only information, the model gains the ability to construct a mapping of states actions to the next states and a reward signal of which the sole purpose is to improve the future performance of the agent (task-awareness). With this, we allow the model to converge different kind of distribution (mapping) and potentially one that might squeeze or extend either or both state and rewards. This could be seen as a compression of information on the one hand and (by having a larger space) also storing and extending additional information explicitly. Of course, our approach is not completely unbiased by the choice of reward because here as well the duman--esigna reward influences the performance signal on which the model is trained.
D.2 COMPARING SUPERVISED LEARNING WITH SYNTHETIC ENVIRONMENTS ON CARTPOLE
We have conducted an experiment the CartPole environment to compare SEs to models trained purely with supervised learning. Before we show results, we will explain our setup:
Setup: To train such supervised learning model and also maintain comparability to SEs, used the data coming from SE runs (i.e. executing Alg. [ More precisely, we recorded all the real environment data from each outer loop step which agents evaluated (10 test episodes), across 5 populations with each 16 workers/agents over 25-50 outer loop steps. The (s, a, s',r) tuples coming from each population run were used to fit supervised model, yielding in total models. We note that the amount of data that each supervised model has been trained with corresponds to the same amount that we would use to learn an SE. The models have the identical architecture as the SEs and we used an MSE loss to optimize the error between predicted (s', and ground truth (s', r) pairs given (s, a) as input. We trained the models for 100 epochs with Adam and a batch size of 1024, finally yielding an average error of 5e-07.
After fitting the supervised baseline models, we executed the same train and evaluation pipeline for SEs that we describe in Section First, we trained agents with randomly sampled hyperparameters on the supervised model. Second, we evaluated them on the real environment and used the resulting rewards to fit the density plot curves. We hereby ensured the same number of reward data points (4000) as for the curves given in the paper by sampling more agent configurations. To also show transfer performance of the supervised baseline models, executed the same procedure for unseen agents (Dueling DDQN & discrete TD3).
observe that the supervised baseline underperforms the other models significantly in terms of the reward distribution and the average cumulative reward (15.01 vs. 190.7 for SEs). In terms of training efficiency see low number of steps and episodes but this
Result: In Figure 22
30

---

Published as a conference paper at ICLR 2022
stems from inferior policies that cause early termination. We also experiment where only considered two of the "best" models which we depict in Figure 23 Hereby "best" means that the majority of samples in the dataset come from high-quality trajectories achieving rewards 2 190 (i.e. which effectively solve the environment). However, in this case, we can only observe a slight increase in performance compared to using all five models (15.01 17.73 average cumulative reward).
The reasons for underperformance be manifold and their analysis were not part of this experi- ment. Potential explanations for this are (but are not limited to):
The supervised baseline models may be subject to overfitting, resulting in inaccurate dy- namics models and rendering policy learning difficult task (it also may increase agent hyperparameter sensitivity).
Catastrophic forgetting sensitivity to the order of seen trajectories during baseline training may not allow to recover accurate dynamics at, for example, early stages of policy learning.
The supervised baseline models may entail other sources of inaccuracies, for example, smoothness or noise, resulting in an inferior approximation of the real environment that may also hinder policy learning.
In the case of SEs, the task-aware meta-loss (and the meta-learning itself) may be good regularizer for preventing the above failure cases.
DDON-trained (CartPole-vO)
Transter. DDQN-tralned
TD3 DDQN-tralned SEs
2.08
Hih
2.06
Hhh howd
mbri HtS: varied
sy HP:: warted best)
2.82
3.00
30000
10000
19379
17507
3:51
THET sytth
THERS mebi
Figure 22: Top: Comparing the cumulative test reward densities of agents trained SEs (green and orange), supervised baseline (purple), and baseline on real environment (blue). Agents trained the supervised model underperform the SE models and the real baseline. Bottom: Comparing the needed number test steps and test episodes to achieve above cumulative rewards. Left: real baseline, center: SEs, right: supervised (MBRL) baseline.
31

---

Published as a conference paper at ICLR 2022
DOON-trained (CartPale-v0)
Transter: DDGN
TD3
2.25
5.70
HiS fowd
Hil: carned
Drnh
0.15
moni. HPS varied besti sy RPP var best)
2.20
2.25
2.00
30000
mreentrnn
1000C
19379
175.77
3:35
Tratr
Figure 23: Top: Comparing the cumulative test reward densities of agents trained on SEs (green and orange), supervised baseline (purple), and baseline on real environment (blue). Agents trained on the supervised model underperform the SE models and the real baseline and using only the best two supervised models (purple) does not seem to change this. Bottom: Comparing the needed number of test steps and test episodes to achieve above cumulative rewards only using the best two of the five models. Left: real baseline, center: SEs, right: supervised (MBRL.) baseline
APPENDIX E NES SCORE TRANSFORMATION
E.1 SCORE TRANSFORMATION VARIANTS
The following overview lists eight score transformations for calculating F; based on the expected cumulative reward (ECR) K; the associated rank Rg. These score transformations where also used in the hyperparameter optimization configuration space for finding SEs and RNs (e.g, in Tables alddd] 7] The rank R, is defined the number of population members that have a lower ECR than the i-th member, i.c. the rank associated with the lowest ECR is O, the one with the highest ECR Mp 1 (the population size minus 1). We denote the ECR associated with the current SE RN incumbent as Kg..
Linear Transformation: The linear transformation maps all fitness values the [0; 1] range as
K: min(K,)
F. max(K,) min(K,) 4
(2)
with min(K;) K2) as the lowest of all ECRs and max( as the highest of all ECRs.
Rank Transformation: For the rank transform the individual rewards are first ranked with
respect to each other, resulting in R. The ranks then linearly mapped the |0; 1] range as
R min(R,)
(3)
max R2) min(R))
with min( (R;) as the lowest rank and max R2) as the highest rank.
NES: Originally published in WWeerstra et al. 2008) and hence named after the paper "Nat- ural Evolution Strategies" (NES), this method first transforms the rewards to their ranks Rú in similar manner as the rank transformation. In the next step, the fitness values are calculated from the individual ranks as
max (0, log( " + 1) - log(R.))
1
F
(4)
max (0, log(
ny 2 1) (R))
ni
F
F. max [F:
(5)
32

---

Published as a conference paper at ICLR 2022
with np as the size of the population. Whereas Eq. (4) constitutes the essential transforma- tion, Eq. (5) acts solely as a scaling to the range [x; Tx<oo and is used for convenience. The
NES transformation also allows negative fitness values F; through the Ry
term in Eq. (4
NES unnormalized: We also consider an unnormalized NES score transformation variant.
Eq. (4) and Eq. (5) here become
max (0, log( 2 + 1) - log(
max (0, log( 2 + 1) loogoo(( (2,))
F
(7)
max (1
Although not directly visible from the equations above, the worse half of all fitness values becomes O.
Single Best: We can induce strong bias towards the best performing population member by only assigning nonzero weight to the incumbent the current ES iteration. The corresponding transformation scheme can be expressed by
if R; max RR;),
0 else,
with (R.) the best rank.
Single Better: It might be advantageous to update the SE and RN parameters only if their performance supposed to improve. By calculating the ECR K2 associated with the parameters, the adapted "single best" transformation becomes
[ 1 if K and K, max(K Ki), 0 else.
F
All Better 1: The "single best" transformation relies at most on a single population mem- ber update the SE RN parameters in each ES iteration. A more regularized version considers all population members whose ECR K; is better than the ECR K2 of the current SE or RN parameters. more mathematical notation, this can be expressed as
K,-Ky if K; K max( K2
F;
(10)
elsee
A
F
(11)
max
F
with Kep being evaluated similarly as the individual ECRs K₃.
All Better 2: This transformation is almost identical to the "all better 1" transformation, except that we divide by the sum of all R; values when calculating the F; values and not by the maximum E. value:
K,-K
if KG K
A
max(K.)-KK Ky
(12)
0 else,
F
F
(13)
Σ F
E.2 EVALUATION OF SCORE TRANSFORMATION VARIANTS
Below we visualize the BOHB hyperparameter optimization results of different score transformation variants for SEs and plot these function of the NES step size with (left) and without (right)
33

---

Published as a conference paper at ICLR 2022
mirrored sampling. The colors indicate the number of outer loop iterations of Algorithm required to solve the target task (here: 2x2 grid world environment). Green signifies O, yellow 25, and red 50 outer loop iterations. Each of the roughly 3500 dots corresponds single evaluation. We can observe multiple things:
none of the rank transformations fails completely for the given simple environment
mirrored sampling is only marginally better (if at all) while doubling the computational load (at least this simple 2x2 grid world environment)
noise: good performing samples in regions with bad performing samples and vice
the "all better score transformation seems to perform best
w| mirrored sampling
We mirrored sampling
11
636-1
22E
3.21
146-1
1.60-1
15
54
8.0-2
796-2
4.0
4.0E
2.0E
2.0E
1.08-2
1.00-3
Inser trand.
rank trans.
NES-
NES urnam
singl bust
s better
al battor
ai bottor
Imaar trnn.
rank trand-
NES-
NES unnomm
shole best
snile bettor
al bottor
al botoor
Figure 24: Evaluation of different score transformation schemes for synthetic environments: Shown is the required number of NES outer loop iterations (green: 0, yellow: 25, red: 50) for different hyperparameter configurations until an SE is found that solves the real environment (here: 2x2 grid world environment).
APPENDIX F ADDITIONAL PLOTS
F.I SYNTHETIC ENVIRONMENTS
F.1.1 EVALUATION OF INDIVIDUAL SE MODELS (CARTPOLE)
Below we show the individual or model-based violin plots for the 40 SE models which we cal- culated based on the cumulative rewards of 1000 randomly sampled agents that were each eval- uated across 10 test episodes. For agent sampling we again chose the hyperparameters ranges denoted in Table 2 As the best five models for discrete TD3 (Figure 3 we chose the models CartPole-vO 25 4I271D, CartFole-vO 22 22_CW9CSH, CartPole-vO 21 ___NNK, CartPole- v0 IF, and CartPole-vOl v0_17
-0_. WPWK3R.
34

---

Published as a conference paper at ICLR 2022
Figure 25: Model-wise evaluation: 1000 randomly sampled DDQN agents à 10 test episodes per SE model.
Figure 26: Model-wise evaluation based on the cumulative rewards of 1000 randomly sampled Dueling DDQN agents à 10 test episodes per SE model.
F.1.2 EVALUATION OF INDIVIDUAL SE MODELS (ACROBOT)
Below show the individual model-based violin plots for the 40 SE models which calculated based on the cumulative rewards of 1000 randomly sampled agents that were each evaluated across 10 test episodes. For agent sampling we again chose the hyperparame- ters ranges denoted in Table 2 As the best five models for discrete TD3 (Figure 7 we chose the models Acrobot-vo 223R5W, Acrobet-v1_2xN044 Acrobot-v1_7D9-2. BGrobSt-vi_9WAAGE
9MA9G8 and 1
35

---

Published as a conference paper at ICLR 2022
Figure 27: Model-wise evaluation based on the cumulative rewards of 1000 randomly sampled discrete TD3 agents à 10 test episodes per SE model.
Figure 28: Model-wise evaluation based on the cumulative rewards of 1000 randomly sampled DDQN agents 10 test episodes per SE model.
Figure 29: Model-wise evaluation based on the cumulative rewards of 1000 randomly sampled Dueling DDQN agents à 10 test episodes per SE model.
36

---

Published as a conference paper at ICLR 2022
Figure 30: Model-wise evaluation based on the cumulative rewards of 1000 randomly sampled discrete TD3 agents à test episodes per SE model.
37


Reviews:
[{"review_id": "s4BAoDkpYWz", "reviewer": "Reviewer_Q4vZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- This paper extends prior work on meta-learning MDP transition dynamics for improving training performance on a target environment, by introducing a method that aims to learn both the state and reward transitions. This formulation is more general than prior works.\n- Experimental results are provided for multiple formulations of meta-learned MDP transition dynamics, spanning both joint state and reward networks (SEs) as well as various formulations of a reward network (potential and non-potential-based reward shaping parameterizations).\n- This paper includes experiments on transferring to different agent architectures and hyperparameter settings, which seems to be a novel experimental setting for meta-learning MDP dynamics.\n\n### Weaknesses\n- The motivation for learning SEs and RNs for singleton target environments is not convincing. While there seem to be some marginal gains in sample-efficiency, it would seem these gains are largely made irrelevant due to the additional training needed to train the environment in the first place. Moreover, if SEs and RNs require first training the agent multiple times on synthetic environments, why not just copy weights or use a form of policy distillation, e.g. kickstarting (Schmitt et al, 2018), rather than train the agent again? Further, it seems the basic ICM baseline already matches or outperforms the proposed RN approaches across experimental settings, further weakening the argument for using the more computationally expensive method proposed in this work.\n- Given the wide design space of possible ways of learning the state and reward transition dynamics for a target real environment, it is unclear why only these two (SE and RN) formulations were studied. Given that this is an empirical paper, the value of the findings rests largely on a clean experimental design that provides a convincing recommendation based on comprehensive experiments. The seemingly arbitrary choice for only considering the SE and RN formulation does not provide such a convincing recommendation for using either of these formulations.\n- RNs are effectively an ablation of SEs, where the state is not learned. However, the current experimental design does not neatly present these results as a clean ablation. This is because the RN experiments also separately consider various kinds of potential and non-potential-based reward shaping parameterizations for the learned reward. These parameterizations are not studied in combination with SEs.\n- Likewise, there is no corresponding ablation for when the reward is not learned, but only the state is learned, i.e. a \"state-transition network\" or perhaps a \"state-augmentation network.\"\n- It is not made clear why exactly SEs and RNs have to use different objectives when training.\n- The RN objective seems strange, as it seems to be optimizing for sample efficiency rather than performance. In contrast the SE objective optimizes for performance. This seems like a significant inconsistency between the training methology for the two types of models, making it hard to compare the relative merits of SEs and RNs.\n- The reward difference term of the RN objective also caps the maximum performance improvement compared to training on the real environment. The motivation for this objective design is not clear, as not taking the max between the reward difference and 0 should encourage learning RNs that lead to outperforming training on the real environment, rather than simply matching it.\n- Further, it seems that there is not a consistent ranking in terms of which parameterization for the RNs improves training performance on the real environment the most, and similarly, no consistent ranking between non-transfer and transfer scenarios for each environment.\n- Meanwhile, the SE results on CartPole and Acrobot would benefit from a direct comparison to the training curves for agents using the best hyperparameters, trained on the real environment. This would directly separate out the performance gains due to training on the  SE.\n- The comparison to previous work such as Zheng et al, 2020 seems quite hand-wavey. Zheng et al, 2020 in particular is quite similar to this work in terms of analyzing the effect of learning an RN and also includes more comprehensive experimental results in a more complex setting requiring recurrent RNs. The novel contributions in relation to these prior works seems to be around analyzing the impact of varying agent hyperparameters on transferring to alternative agent settings and architectures during evaluation. However, these transfer results do not seem consistently strong for any one method proposed.\n- Sampling agent hyperparameter settings from a diverse range seems crucial for the learned SEs and RNs to transfer to other agent configurations. The paper does not discuss how sensitive the proposed methods are to the agent hyperparameter ranges used during training.", "summary_of_the_paper": "This paper aims to learn proxy environments (synthetic environments or SEs) and reward functions (reward networks or RNs), parameterized as neural networks, such that these proxy models provide beneficial transitions to make it more sample-efficient to learn a policy for a fixed target environment (referred to as the real environment). SEs replace the observed state and rewards during training, while reward networks provide a synthetic reward that is a function of the real reward, current state, and next state. The proposed method formulates this problem as a bi-level optimization, where the inner loop consists of standard RL under the proxy model, and the outer loop consists of NES with the aim of optimizing either the performance on the true, target environment (SEs) or a the number of training steps needed to reach a certain return threshold on the real environment. ", "summary_of_the_review": "While this work proposes an interesting generalization of prior work on meta-learning MDP dynamics, the case made for these methods is weak, and the experimental results—especially for SEs—are uncompelling. Given the additional compute required by these methods, it seems that various forms of policy distillation or even training from scratch seem more efficient. Further, it is not clear why only certain formulations (SEs and RNs) of meta-learning neural transition dynamics were investigated, while others (like a state-only neural transition network) were not. Given these points, I recommend this paper in its current form for rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kG-za7eEsmv", "reviewer": "Reviewer_1Zys", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting line of work and initial investigation. The exposition of the method and description of experiments are quite clear and complete. However, there are a few weaknesses.\n\nThe motivation for this work deserves more attention. The authors suggest that SE’s could be used for AutoML, iterating on algorithms in expensive environments like robotics, agent or task analysis, or agent pre-training. However, I’m not sure these are particularly well aligned with the authors’ statement of the problem, implementation, or experiments.\nMost applications are predicated on the relative performance of various algorithms, when trained in the SE, being correlated with performance in the true environment.\nHowever, the problem formulation in eqn (1) describes maximising performance absolutely.\nThis is approximated in Alg. 1 with a generic TrainAgent function that approximates the argmax_theta. But it seems the specific TrainAgent functions chosen are highly relevant to potential use cases – it is not a generic stand-in for an argmax.\nThe authors certainly understand the relevance of this, as they study generalisation of SE’s to different algorithms and their hyperparameters, and even address generalisation algorithmically by varying hyperparameters during training of the SEs. However, this is introduced as a brief comment in the experiment descriptions, rather than being central to the method.\nIt feels that to use SE’s for the potential applications discussed, it would make more sense to reformulate the problem statement to explicitly account for different inner-loop algorithms.\nFurther, simply maximising the final performance of all inner-loop algorithms seems quite limited when applications involve comparing different algorithms.\nIt might be interesting to algorithmically address this use-case by changing the fitness function to account for the relative performance of different algorithms.\nHowever, it would also be useful to address this empirically given the current formulation of the objective, and the authors likely already have the data to do so, at least partially: does the performance of different HPs/algorithms *in the SEs* correlate with their performance on the true environment?\n\nIt may also be possible to motivate the work from a more traditional model-based RL point of view, which might be more closely aligned to the problem statement given in the paper. A closely related work is that of Nikishin et al [1] (who use implicit differentiation to address the bi-level optimisation). NB I don’t fault the authors for not citing this, I believe it is only available as preprint. They argue that models learned only to further optimisation of a policy (as in this work's problem statement) may perform better than traditional models when capacity is limited. However, the ES-based bi-level optimisation is expensive, so it may be difficult to make a strong case for this approach.\n\nAnother weakness of the study is the simplicity of the environments used. I understand the authors may have limited computational resources, but it is difficult to know how highly to weight an empirical feasibility study carried out on such small-scale tasks. In the absence of more compelling empirical findings, it is more critical to explore the algorithmic choices and motivations.\n\nIn this vein, I did appreciate the investigation of which inductive biases might help during reward learning. Overall, I find the case for RNs is perhaps more intuitive: dense-ifying rewards once in an expensive optimisation to allow fast iteration later. Using an additive potential form even guarantees that this will not change the optimal behaviour – sadly, it seems this type of RN did not outperform the baseline much in the experiments.\n\nIn the RN case optimising for the speed of training makes sense, but I still find the chosen objective strange. Why not optimise the area-under-curve returns for the full inner-loop? Clipping at a somewhat arbitrary solution threshold seems like it would result in suboptimal RNs (this appears to have occurred in the half-cheetah experiment in the appendix).\n\nMinor comments:\n - In the SE experiments, it would be good to show the performance of the ‘fixed’ hyperparams evaluated with the same fixed hyperparams: this would show the ‘generalisation gap’ compared to evaluating with varied hyperparms as is currently shown.\n - Overall, many figures are a bit hard to parse. Try using more smoothing or subsampling for curves. For the barplots, maybe re-arrange them so the comparable bars (i.e. measuring the same metric, ‘steps’ or ‘episodes’, are next to each other).\n\n[1] “Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation” Nikishin et al 2021\n\n----------------------\n\nUpdate: the authors have addressed many of my concerns, ran some additional useful experiments, and clarified their reasoning on several points in the overall discussion with reviewers. I still think there is a bit of a gap between the big-picture motivation for the work and the instantiation we see studied empirically; the authors are working on an empirical investigation to help make that connection. However, even in its absence, I am sufficiently positive about the paper overall now to lean towards acceptance -- I found the study interesting and creative.\n\n", "summary_of_the_paper": "This paper proposes to learn neural environment models they term Synthetic Environments, such that planning in the model with model-free RL results in good performance in the real environment.\nThe authors study learning full models as well as reward-only models they call Reward Networks, investigating the feasibility of learning these models as well as their robustness to different inner-loop algorithms and hyperparameters.", "summary_of_the_review": "While the work is interesting, I find the motivation a little strained at not well aligned with the problem statement or algorithmic instantiation. The empirical study is fairly clear but limited by the simplicity of the domains and the restriction largely to assessing feasibility rather than giving clearer indications of how the method could be used in practice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GbSZnNGlnN1", "reviewer": "Reviewer_pG56", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nFirst and foremost, let me start by saying that I *really* like this manuscript.\n\n1. The writing is extremely clear & detailed, and makes the paper a pleasure to read. I also found that the authors tended to answer questions that were popping as I was reading paragraphs immediately thereafter, which is a likely sign of writing that has gone through a lot of careful iterations.\n\n2. The authors have gone through a lot of work to justify the choice of optimization methods, how and why the framework was setup in certain ways, and how the hyperparameters for a now fairly complex learning system were chosen. Considering the code release and the extremely detailed appendices, I suspect the work will be extremely easy to reproduce. Truly commendable, and a great example of what good ML research looks like (I also particularly enjoyed the frank discussion about the work's limitation due to choosing ES as an optimization method for the outer loop).\n\n3. The central idea of the method is relatively simple, but seems to be quite effective; it honestly borrows from a lot of previous literature (reward learning, population-based optimization, etc.), whilst effectively producing a novel take on learning syntethic environment models.\n\nSo, overall, I am extremely happy to argue for acceptance as it is. Now, let's look at what I perceive to be some of its weaknesses:\n\n4. The manuscript at times feels very dense, especially when looking at methodology and experimental details, and its understanding is greatly helped from the presence of a sizable appendix. It very much feels like the authors could have a feasibly split the paper into two, one on ES and a follow up on RN, providing more space to experimentally analyse ES and RN separately and make for a cleaner and easier review process.\n\n5. Some of the hypotheses presented to justify the improved agent training feel relatively handwave-y. Consider for instance the following quote (emphasis mine):\n\n>Optimizing SEs for the maximum cumulative reward automatically yielded fast training of agents as a side-product, likely due to **efficient synthetic state dynamics**.\n\nIt is implied that this meta-learning approach leads to generally discover better data-producing dynamics for the agents used in the system, but I wonder whether this is primarily due to the structure of the tasks employed in the experimental settings (as state space and transition dynamics of these simple control problems are _generally_ not very informative, and the _useful_ -- policy learning wise -- parts of these MDP are instead both easily discoverable via search and extremely informative). \n\nIn short, I wonder if we'd see similar improvements for tasks / environments where task difficulty is extremely affected by the emerging complexity of the environment, or whether we can actually show that this is a property of SEs / RNs when trained in this manner (which would be quite outstanding!). Although that said, the effecitveness of RNs do seem to indeed point to the latter hypothesis.\n\n\n### Nits / Additional comments\n\nFigure 2: the labels are inverted (dotted lines should be the thresholds).\n\nSection 6 (but generally in many parts of the paper): it is claimed that learning rewards is generally easier than learning transition functions -- would it be possible to find a reference for this? My personal opinion is that it should indeed be often the case, but that generally it is not true (e.g. imagine an MDP with a small transition matrix but an extremely stochastic reward function).", "summary_of_the_paper": "The authors propose a framework for learning to synthesize a proxy models for usually non-learned components of an RL loop: namely transition dynamics and reward as defined in the interacting environment. They learn the parameters of these model by \"meta-learning\" onto the real environment with a learning agent (with slight detail variants depending on what the learning objective is), showing that not only can this setup lead to successfully learn the proxy parameters, but also improve learning performance for the agents.\n", "summary_of_the_review": "\nThis is a good paper. It contains excellent writing, good research, and makes for a great example of what an ICLR paper should look like. The research problem and proposed methods are interesting and well placed in the literature, and the experimental section is exhaustive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6ihfnhFq4V7", "reviewer": "Reviewer_ktr8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Results:\n- In Figure 2, the performance seems very unstable: In particular, the reward for individual agents seems to fluctuate widely and even the average starts drooping after the 50th iteration in the arcobot task. It would be great to know what the source of this instability is. \n\nConceptual:\n- There are repeated claims in the paper that the synthetic environment \"focuses the learning\" on the relevant states or learns \"representations\" of the target environment (\"an informed representation of the target environment by\nmodifying the state distributions to bias agents towards relevant states\"). This claim is entirely unclear to me at the moment. In particular, there simply doesn't seem to be any reason that the SE should have much in common with the ground truth environment. The only condition is that _optimal policies_ for the two environments match. In particular, there will be many environments in which the optimal policy is likely much easier to learn than in the original Env. \nFor example, you can imagine a state transition dynamic that is entirely random but provides a large reward for the optimal action in each of the given states. Another option is to learn environment dynamics where the optimal action in each state deterministically transitions to an arbitrary next state, such that the agent will see all required states during training. \nIt would be great to provide any type of evidence that there is actually semantic meaning in the SE that matches the underlying ground truth environment.  \n\nOther comments:\n- In Figure 2, it seems like the legend is switched? I.e. dashed line should be the solved threshold?", "summary_of_the_paper": "This paper introduces Synthetic Environments and Synthetic Rewards as a bi-level optimization problem. The main idea is to find SEs and REs such that an RL agent trained under them will do well in a given, known target environment, without further training. ", "summary_of_the_review": "A very interesting paper with early results in a potentially interesting direction. It is currently unclear what this method allows to do that was previously impossible from a practical point of view, but there seems to be promise. \nI'd be open to raising my score if my concerns are addressed by the authors. \n\n\n\nUpdate: \nBased on the discussion phase and the rebuttal I have updated the score to a 6. I believe this paper has enough merit to be published at ICLR, if there is space. It is an intriguing piece of work, even though the practical utility is very unclear at this point. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for generating synthetic environments and reward networks for reinforcement learning tasks. This happens as a nested process: policies are learned in an inner loop, and environments are evolved in an outer loop. The environment representation is quite simple: the parameters of an MDP. Similarly, the reward networks are simply neural networks. Results show that the the learned environments and reward networks are reasonably good at decreasing policy training time by RL. The proposed method appears to be simple and quite general, and it would be interesting to see how it scales up to more complex environment representations.

The discussion around the paper centered on understanding various details of the method, and on the quality of the results. The reviewers generally agree that the paper is easy to read, and vary in their assessment of the significance of the results. It was pointed out that the generated environments are not necessarily similar to the base tasks, but it was nowhere claimed in the paper that they were. (In fact, it could be argued that the dissimilarity makes the method more interesting, given the good results of policy training.)

I'm happy to recommend the paper for poster acceptance. If the results would have been more impressive, it could have been accepted for a more prominent presentation form; however, I believe that the method can yield better results in the future with more sophisticated environment representations.
Decision: Accept (Poster)