Emergence of Collective Policies Inside Simulations with Biased Representations
Authors: Jooyeon Kim
Alice Oh
Venue:  (2020)
Keywords: collective policy
biased representation
model-based RL
simulation
imagination
virtual environment

Abstract: We consider a setting where biases are involved when agents internalise an environment.  Agents have different biases, all of which resulting in imperfect evidence collected for taking optimal actions.  Throughout the interactions, each agent asynchronously internalises their own predictive model of the environment and forms a virtual simulation within which the agent plays trials of the episodes in entirety. In this research, we focus on developing a collective policy trained solely inside agents' simulations, which can then be transferred to the real-world environment. The key idea is to let agents imagine together; make them take turns to host virtual episodes within which all agents participate and interact with their own biased representations. Since agents' biases vary, the collective policy developed while sequentially visiting the internal simulations complement one another's shortcomings. In our experiment, the collective policies consistently achieve significantly higher returns than the best individually trained policies.

Under review as a conference paper at ICLR 2020
EMERGENCE OF COLLECTIVE POLICIES INSIDE SIMULATIONS WITH BIASED REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We consider setting where biases involved when agents internalise envi- ronment, Agents have different biases, all of which resulting in imperfect evidence collected for taking optimal actions. Throughout the interactions, each agent asyn- chronously internalises their own predictive model of the environment and forms virtual simulation within which the agent plays trials of the episodes in entirety. In this research, we focus on developing collective policy trained solely inside agents' simulations, which can then be transferred to the real-world environment. The key idea is to let agents imagine together, make them take turns to host virtual episodes within which all agents participate and interact with their own biased representations. Since agents' biases vary. the collective policy developed while sequentially visiting the internal simulations complement one another's shortcom- ings. In our experiment, the collective policies consistently achieve significantly higher returns than the best individually trained policies.
INTRODUCTION
We look at the world through our own lenses. Two people with the same exact observations create different internalizations of the world through their own mental models, collect different amounts of evidence, take different actions, and end up with different outcomes. For example, students attending a lecture look at different parts of the presentation slides, and doctors looking at the clinical log of patient focus different data and statistics and make different diagnoses.
In model-based reinforcement learning (RL.) (Kaelbling et al. 1996 SStton & Barto 1998), agents interact with environment through models that are analogous the human mental models (Hamrick) 2019). However, just as our decisions governed by our mental models are often far from perfect, the agents' policies that are designed upon the models suboptimal (Box 1976 1979). as the inductive biases used to learn such models are not guaranteed to be correct (Grifiths et al. 2010).
In this research, we propose learning algorithm of collective policy that leverages multiple agents' biased representations encoded from their internal models. Like humans who often compensate for their cognitive biases through collaboration (Woolley al. 2010:; Muchnik al.) 2013), the goal of the collective policy is to make use of multiple internal models so that the biased representation formulated by each internal model complements one another achieve higher returns than the best- performing individual policy. The collective policy in this research has the following characteristics:
Asynchronous shaping of agents' internal models: The collective policy operates on agents' preestablished internal models. Prior to learning the collective policy, agents construct their internal models over the course of their own interactions with the environment, which can be done asyn- chronously and independent of other agents. Note that this learning process of the collective policy does not fall into the conventional multi-agent RL settings, where agents are concurrently situated in the same environment (Littman 1994; Panait & Luke 2005 Bu et al. 2008 Buşoniu et al. 2010).
Collective policy learned solely inside agents' internal simulations: model-free and model-based RL, policy learning typically requires a large amount of interaction with the environment, which hampers feasibility, time-efficiency (Guez et al. 2018). and safety (Garcia & Fernández 2013). This is particularly prohibitive in setting, as during the collective policy learning, need to force the agents to go through the same rollouts together, coordinate their actions, and interact with the environment in algorithmically intended ways (see e.g., Yahya et al. (007)); Rubenstein al. (2014)

---

Under review as a conference paper at ICLR 2020
Nagpal (0166). To overcome this, we propose learning algorithm of the collective policy that involves zero interaction with the real environment; instead, the collective policy is trained solely inside agents' simulations, analogous to human's 's mental simulation or imagination (Hamrick) 2019). In our learning algorithm, agents take turns to be the host that generates a virtual rollout, within which all agents participate in and learn the collective policy together.
- Minimal number of parameters to learn the collective policy: In the proposed algorithm, the number of parameters and the training iterations of the collective policy function same as those of the individual policy functions. This not only allows the efficient training, but also assures that the performance edge of the collective policy over the individual policies not attributable the model complexity but to the complementing evidences provided by the multiple internal models. Moreover, learning of the collective policy can precede learning of the individual policies.
We make the following contributions in this paper:
I. We present a novel problem setting in model--ased RL that resembles the human cognitive bias of imperfect information. Note that our setting is clearly different from the partial observation settings in RL (Astrom 1965; Kaelbling et al. 1998); in our setting, two agents at the same exact time and location in the environment encode different representations from the same observation.
II. We compare the real environment and the internal simulation generated by agent models for training policy. With appropriate early stopping epoch and т, we achieve much better scores at a fraction of the training time.
III. We present learning algorithm of the collective policy that operates on the simulations generated from internal models of agents and complements their biased representations, thereby achieving higher scores than that of the best-performing individual policy.
IV. We experiment with VIZDOOM, different environment in which observe an unexpected behavior of agents becoming delusional, i.e., having the false belief about the reward system, preventing us from learning the collective policy in simulation.
V. We release our source code, visual demonstrations, additional results, and in-depth discussions in a separate link
COLLECTIVE POLICY TRAINING ALGORITHM
Here, we describe the process of training a collective policy by leveraging multiple agents' internal models that create their own biased representations given the same observation inputs. We start by describing the components of agents' internal models and explain how each agent creates its own biased representation of an observation inside the model. Then, we introduce additional components required for training the collective policy inside the agents' simulations.
2.1 INTERNAL MODELS OF AGENTS
We use world models (Ha & Schmidhuber 2018a Schmidhuber 2015) as agents' internal models. World model abstracts apatio-temporiy features of the environment in a compact representation that used for policy learning after the model training. The left panel of Figure I illustrates the two components of the world model along with the environment and the controller C that learns the policy after the world model is trained. First, at each time step I, the visual sensory component abstracts a high dimensional input observation from the environment into compressed representation. Variational autoencoder (VAE) (Kingma& & Welling 2014, Rezende et al. 2014)) is used to encode 2D image input frame into latent vector 24 Then, the latent vector 24 well the following action Ge are used as inputs for the memory component M, which outputs a prediction of the next encoding 2 Mixture-density recurrent neural network (MD-RNN) (Bishop 1995 1994 Graves 2013 Ha & Eck 2018 Graves 2015) is used for this component for the stochastic prediction of the future, i.e., P(=+1 [rt /t, a,) ), where ht is the hidden state variable time t. V and M use sequential image frame data from multiple rollouts from a random policy and apply variational inference and backpropagation algorithm for inference and learning.
After the two components of the world model are trained, the parameters of C are optimised online dur- ing the following rollouts using covariance matrix adaptation evolution strategy (CMA-ES) (Hansen
http: /ttp://s000 t tingpplnnnyllpoooo www.hyywlll...coomooomm
2

---

Under review as a conference paper at ICLR 2020
Env.
Host
M
M
h
2,
Zq he
(a) Policy learning
(b) Policy learning
in environment
in simulation
(c) Collective policy learning in simulation
Figure 1:
Flow diagrams of policy learning of (a) an individual policy while interacting with an environment, (b) individual policy learning inside internal simulation, (c) collective policy learning involving multiple agents' internal models. Note that for (b) and (c), there is no interaction with the environment. For collective world model, we omit the decoding part of V for clear presentation.
Hansen 2006 2016) which is shown be the method-offchoice for small solution spaces to few thousand parameters (Salimans et al. 2017 Loshchilov & Hutter 2016).
Ostermeier 2001;
Training policy inside an agent's simulation The memory component M predicts the latent representation the next frame and allows an agent to take an action based on the agent's prediction of the near future as well as the current situation. Further, one can generate multiple future frames, even the whole episode from start to finish (Figure lbp. Then, C is trained inside this virtually generated environment (Ha & Schmidhuber] 2018a b). MD-RNN, which is the realisation of M, can endow a desirable level of uncertainty in the simulation by adjusting the temperature parameter т.
Biased representations We hypothesize that when an agent's 's model internalizes the observation through V, the agent from collecting the full evidence needed to learn the optimal policy because of bias. The bias, which persists throughout the experiment, can be that certain parts of the frame are left out (occlusion), only certain part of the frame is accessible (magnification).
Asynchronous shaping of agents' world models Prior our merging of agents' simulations, assume that agents preestablish their internal models (M and V) asynchronously, based on their imperfect evidence accumulated. However, that all agents share, c.g., few frames that all agents co-observe, to train the translator component. This will be further explained in the following subsection. Unlike the typical distributed ML systems (Chen et al., 2012, Deisenroth & Ng 2015; Low et al. 2015), each internal model can be trained independently without broadcasting or establishing the communication protocol among agents, i.e., each agent's encoded representations from V can be done using independent VAEs, each with different initialisations. This is because the collective controller and all individual controllers trained independently of each other Cs, as well as the remaining components V and M.
necessary ensure there are environment states
2.2 COMPONENTS FOR LEARNING THE COLLECTIVE POLICY
We gather up agents' biased representations created inside the individual world models and train collective policy in the internal simulations generated from the agent models so that each model's biased representation complements others' shortfalls and the virtual return per episode is maximised in a collective manner. Here, the key idea to make agents be in the same internal simulation at the same time, where each agent's model takes a turn to be the host of the virtual rollout using her respective memory component M (Figure Tc). Throughout this section, we suppose that there are total of N internal models in system, and the hosting agent per virtual rollout is denoted by H. Added components for the collective policy learning are explained as follows:
By design, our algorithm can operate different kinds of biases, e.g., temporal distortions, although we did not empirically test the performance under such conditions.
3

---

Under review as a conference paper at ICLR 2020
Rule component (R) When learning policy in the environment, automatically retrieve reward signals from the environment. However, when the (collective) policy is trained in simulation, we need to learn how the reward is given using previous and current state representations. Within each agent's model, we only preserve the memory component and introduce an additional component that understands the rule of the reward system. For every virtual rollout have hosting agent H whose rule component Ry returns a virtual score which is instantly transferred to the controller component so that C can assess the quality of the current policy solely inside simulation and without the need to interact with the actual environment. For this component, we use RNN that takes previous and current biased representations as inputs and returns the estimated reward for each time step.
Translator component (T) In our setting, agents' visual sensory components create different repre- sentations from the same observation. While we train the collective policy inside agents' simulations, the translator component T converts biased representations between two agents. For every virtual roll- out generated by the hosting agent H, we need N 1 translators Th.a for E {1, 2,...... NN {H), which are the functions (H,a(2H) Za. In setting, the internal representations in dimensionality than the original observations, and representations are highly intercorrelated among different internal models. Therefore, we apply meta-learning (Finn et al., 2017; 2018 Kim al. 2018 Nichol et al. 2018) to speed-up the training of T and reduce the training data.
much smaller
Aggregator component (A) To train a collective policy, there is aggregator component A which is a permutation invariint function that takes representations from agents' internal models as inputs and returns their summarisations of the same shape. Here, we take the summation function as an aggregator which simple but effective (Ali et al. 2018 Garnelo et al. 2018a b Soelch et al., 2019)
Algorithm 1:
Collective policy learning algorithm
Input: N trained memory components components
N trained rule components R
2 trained translator components T
Initialize parameters for the collective policy function; while not converged do
Set RewardT otal + 0; for hosting agent .,...,NV do Set the hosting agent + a; Retrieve initial encoding for the first time step 2H,0:
for time t until episode terminates do
Generate encodings of other agents 2a,t, Va (1, (H) using 2H, from T;
11
Generate ha, for all agents using their respective M and aggregate N concatenated vectors [za,t bo,r] using A to return [Ze,ū he.]]
12
Insert the aggregated vector he.,] in the controller C to get action at time t; Execute the action inside the hosting agent's simulation M infer the next encoding
13
21,++1 and to get reward from R;
14
end
15
Set RewardTooal + RewaRdd otal
end
17
Update the parameters for the collective policy using CMA-ES and RewardT olal;
18
end
Training scheme for C We use CMA-ES that operates a state space to optimise the parameters of C. To use the simulations of all hosting agents, for each training iteration, we define a single episode by concatenating multiple virtual rollouts and use the sum of the cumulative rewards calculated from the hosting agent's 's rule component Rg. Aside from making use of all agents' simulations, combining multiple rollouts single episode is shown to stabilise policy learning throughout the training (Ha 2017). We highlight that the number of parameters for C as well as the number of training iterations identical to those of the individual controllers.
For each iteration, the controller component C operates on exactly the same input space as the individual model as take the summation function and [[=.; he]] = [[za; h.]]. Va E (1,...,NT
4

---

Under review as a conference paper at ICLR 2020
Ewryy wwrggngggggggg epech
400
5000
100
6.6
x7
1.
8.8
Temparalure
Temporatore
(a)
(b)
Figure 2: (a): Relationship between the temperature parameter T of MD-RNN in internal simulation and the score received from the real environment. governs the stochasticitiy of the virtual environ- ment generation, and there an optimal value, 1.7 that gives the best score. (b): As T increases, the simulated environment becomes more unpredictable and we accordingly need more training epochs.
Table 1: Comparison between training the policy inside real environment and inside an agent's internal simulation. is followed by one standard error.
# Env. interaction
Time / epoch (min.)
Epoch
Pop. size
Mean
Median
Model tr.
Policy
Env. Sim.
10
1.84
1,800
42
64
900.0
5.2
911.9
10 K
0 225
0.8
32
851.9
9.8
875.1
Therefore, the only added terms in the time complexity of collective world model come from T and A, where for T, we need forward passes of RNNs and that aggregates N input vectors.
In algorithm 1
we summarise how the learning of the collective policy works inside agents' simula- tions using trained components M, R, and T along with the aggregator A.
3 EXPERIMENT
In this section, answer the following questions: First, how does training policy in internal simulation compare to training a policy in the real environment? What specifications do we need for comparable performance in the simulation? Second, how much do gain from the collective policy trained within the simulations of agents' internal models whose biased representations complement one another? What causes the collective policy to perform better than the individual policies? In addition, we introduce another environment where when bias involved during the internalization, agents in the environment become delusional, not being able to understand the rule of the environment.
3.1
EXPERIMENTAL SETUP
Agents learn to drive in OpenAl gym CARRACING environment (Brockman et al.) 2016 Klimov 2016). Each trial consists of 1,000 image frames, and while agents are driving car, the score calculated as the total number of tiles, i.c., the gray area Figure 3al the car visited. Thus, agents need keep moving forward without derailing from the track until the trial ends in order to maximise the return. For each time step, agents can take three continuous actions: steer left/right and accelerate. Throughout the experiment, we report average
over 100 trials with standard error.
Following the previous setting by Ha and Schmidhuber (Ha & Schmidhuber 2018a b), we train each internal model's components using 10,000 rollouts with random policy. As displayed in Figure 3a we make each agent's observations imperfect by I) occluding a certain part of the frame or 2) only showing a magnified portion of the frame. Here, we conserve the car image (in red) for all agents' observations, as this image portion is critical for the rule component to calculate the virtual score. Also, we highlight that this gatheggtteringggg phase the only time interacting with the real environment throughout the experiment, except for the testing trials. Finally, we keep the same number parameters and maximum training iterations for both individual and collective policies.
3.2 EXPERIMENTAL RESULTS
Policy learning in simulation Figure describes how the temperature parameter T of MD-RNN (M component) affect the policy learning inside simulation. As increases, the virtual environment
5

---

Under review as a conference paper at ICLR 2020
Table 2: Results of the collective policy learning. When multiple agents' internal models provide complementary evidences from their biased representations, the collective policy achieves higher real-environment scores than that of the best performing individual internal model.
Bias
Indiv. policy
Collective Policy
# M
Overlap (%)
Cov. (%)
type
Worst
Mean
Best
Occ.
0
100
615.2 +: 22.7
618.4
621.6 7.9
750.8 12.3
Mag.
12.5
37.5
494.3 = 16.5
558.7
623.0 = 22.9
733.8 + 19.9
Mag.
18.8
37.5
494.3 + 16.5
602.1
688.7 = 17.4
744.6 10.2
O.,M.
100
100
494.3
16.5
581.0
623.0 22.9
742.1 08.8
Occ.
2
100
62.5
468.4 + 21.9
541.8
615.2 + 22.7
629.1 + 15.5
Mag.
2
100
39.1
571.9 + 21.4
678.3
784.7 + 9.3
769.8 + 16.3
600
200
Indi
Mald
WM1
VM3
(b)
(c)
Figure 3: (a): Original observation (top) with decodings of the biased representations of different internal models (occlusion magnification). (b) (c): Virtual scores when the policy trained inside its internal simulation (left bars), when using the policy that is trained in other's internal simulation (center bars), and when the collective policy used (right bars). other's policy used in the virtual environment generated by oneself, the virtual score decreases catastrophically, whereas the collective policy gives reasonable scores inside all simulations. (d): "Joint" the collective policy without T and A components; when the collective policy is trained without co-observation, performs worse than individual policies.
generated by the internal model becomes more unpredictable. In Figure 2a we observe that there is an optimal temperature parameter, which is 1.7 in CARRACING environment, that gives the score when the policy learned in simulation is transferred to the real environment. This result is in accordance with the result reported by Ha & Schmidhuber (2018a), where they experimented in the VIzDOOM environment. In Figure 26) we observe that there is an additional trend that incorporates т: With all т, we observe that when we train the policy in simulation for too long, the policy overfits the virtual environment so that the virtual score monotonically increases as the training epoch increases but performs worse when the policy is transferred to the real environment. Thus, the early stopping is required when training the policy in simulation, and the early stopping epoch generally increases as we increase т, meaning that we need more training iterations until we transfer the policy to the real environment when the virtual environment is more unpredictable.
From Table We report that with an appropriate early stopping epoch and т, we can achieve a score of 851 9.8 in CARRACING environment with the policy that is trained only in simulation and then transfered to the real environment, compared to the score of 900.0 + 5.2 when trained in the real environment. When the policy trained simulation, the training time per epoch is approximately 53 times faster, total train time changes from 52 days to 3 hours, and the number of rollouts generated the environment reduced from 1.84 million to zero during the policy training.
Collective policy Table shows the results of the collective policy compared with the individual policies whose internal models that are integrated. Here, #M is the number of internal models used for training, Overlap is the maximum percentage of frame overlaps between agents' biased representations in terms of pixel count, and Cov. is the percentage of total coverage when all biased representations are superimposed
terms of pixel count. From the first four rows in Table 2 we
3 To our knowledge, there is no previous research that trained policies for single task in both real and virtual game enviromments; in the original world models paper, the CARR ACING experiment is done in the actual environment, and the V1zDooM experiment is done inside simulation, but the other way around.

---

Under review as a conference paper at ICLR 2020
1942
Valud WV3
1941 Tak
Whbel lare Telh
100 1642
(b)
(d)
(C)
Figure 4: (a): Biased representation (occlusion) in VIZDOOM environment. (b) - (e): 4 different agents' internal models with the bias setting given in (a). Although trained under exactly the same setting, internal models (b) and (d) learns reasonable policies inside simulation, while (c) and (d) became delusional. For all four internal models, their virtual scores are similar (blue bar), but the actual score significantly varies between normal and delusional agents. Also, once agent's 's model is trained, delusional agents do not become normal during the policy training phase and vice versa.
confirm that when we train the collective policy that integrates multiple internal models whose biased representations complement one another, we can achieve higher scores than that achieved by the pest-performing individual policy in the real environment. In the last two rows, we do not observe any significant increase in the score when the collective policy is compared with the individual policies, because one internal model's biased representation is a proper superset of that of the other's.
Figures 3b 3c show the virtual scores of individual and collective policies. When individual policy is trained inside the simulation generated by its own model, we achieve the best score. However, when the trained policy is executed in the simulation generated by another model, the score decreases catastrophically. In comparison, the collective policy returns reasonable scores in all simulations, although the score is less then the one achieved by the individual policy in its own simulation.
Figure 3d highlights the necessity of the model structures proposed in this research for learning the real-environment score achieved by the collective
collective policy. Here, "Joint" corresponds policy jointly trained by sequentially visiting multiple internal simulations, but while doing so, the agents other then the host does not participate in the simulation. In other words, its the collective policy without the T and A components. From this result, we confirm that the main performance edge of the collective policy comes from the "co-observation" during the policy learning.
Delusional agents We report that in the VIzDOOM environment (Paquette 2016 Kempka et al. 2016), environment where the rules of the game and the other aspects of the environment tightly coupled, agents randomly become delusional when the bias is involved in their internal models (4) Here, the rule that governs the reward system of the game is highly intertwined with physics of the system (movement and perspective of the fireball, explosion of the fireball when the ball contacts the agent) and is potentially affected by every pixel of the frame, and therefore the rule component R inevitably has to be incorporated within the memory component M (this phenomenon is extensively studied by Ellefsen al. (2019)). In this case, some agents' internal model will falsely believe that they will be rewarded not by dodging the fireball but by going inside the occluded region and stay there until the trial ends. To our knowledge, this type of delusion has previously not been observed; in the world models paper, agents will sometimes cheat inside simulation by creating an easy virtual environment when the temperature parameter set to a low value but clearly it a different type of behaviour. This abnormal behaviour is remarkable in two ways. First, an agent falling into a delusion happens completely at random at the training phase of the memory component. Thus, for the same exact training data, methods, and iterations, some agent will behave normally (treating the distortion as part of the environment) while some will be delusional. Second, once the memory component is trained and the agent shows delusional behaviour, it can never get out of this however many times train the policy with different initialisations. On the other hand, in the car racing task, this issue can be resolved because the reward system is only governed by a regional area surrounding the car; therefore, if we do not distort this part and train R and M separately, we can prevent agents from falling into this type of delusion. In our collective setting, this issue particularly problematic because, for delusional agents, the degree of agents' bias do not directly transfer to their performances.
4 RELATED WORK
Machine learning (ML) and RL researchers propose internal models of the agents that is analogous to humans' mental models (see the work of Hamrick (2019) for the review), and the concept of learning policy solely inside agent's simulation using recurrent structures was proposed by Schmidhuber
7

---

Under review as a conference paper at ICLR 2020
1991| 1990) and recently been reviewed in the name of learning to think (Schmidhuber 2015: Lake et al. (017). Also, [Racaière et al. (2017) develop a neural architecture that leverage internalisations of the world on top of the real-environment interactions, and Ha & Schmidhuber (2018a b) propose world models that enable full operationalisation inside simulation. Additionally, the idea of internalising the world is adopted for meta-control and to optimise and computation based on task difficulty and compexity (Hamrick et al., 2017: Pascanu ct al. 2017). Finally, there is a recent approach in model-based RL that further develops and applies the Idea of simulation or imagination to achieve better results. Nair et al. (2018) and Haatner et al. (2019) make use of agents' internal models to pre-determine the long-term outcomes (e.g.. imagined outcomes) and develops a policy based on such outcomes.
in 1990s (Schmidhuber 1970
In this research, the environments that we experiment on has state spaces that are accommodable for the internal models to capture with random policies. Orthogonal to research, |Hanner et al. (2019) advances the idea of training a policy inside a simulation by proposing an end-to-end framework that incorporates simulations in environments that cannot be fully explored and described by random policies, and by doing so, balances out the optimizations of the exploration policies and the model learning while minimizing the number of interactions with the environment.
Close to our research in terms of conceptual proximity and motivation is study by Bard et al. (2019), In their work, authors focus on a board game called Hanabi in which agents operate in a cooperative fashion to solve a task. Similar to our work, agents in the Hanabi game only has access to imperfect information, and their information can be complemented only by interacting with other agents, Here, the most critical diverging point is that the Hanabi challenge seeks an optimal policy for each individual, which ideally mimics human's 's sophisticated behaviours such as the theory of mind (Premack & Woodruff 1978), and the authors do so by interacting with the actual environment. On the other hand, we focus on designing a specific model structure that can be solely operationalised in multiple agents' simulations and deriving a collective policy that gives better performance than individual ones. On a related note, [Battagia et al. (2018) proposes a general framework that discovers structural, relational, and compositional representations of an arbitrary system that builds upon graph networks and complements each entity's 's inductive biases, which can be applied to our setting with more entities and with relational information.
DISCUSSIONS
We postulate scenario in which each agent's internalisation of the world is biased, resulting in imperfect evidence collected at each timestep. Our goal is to simultancously superimpose agents' biased representations portrayed by their internal models to learn collective policy that agents' defective simulations can be complemented by others. In our experiment, we used the CARRACING platform with maximum of four agents and showed that the collective policy with integrated simulations give better performance for the car driving task. However, a few outstanding questions remain that need further exploration to better comprehend the overall process of learning a collective policy inside simulation and make it more effective and scalable.
The first direction is to find an optimal group composition that maximises the score of the collective world model when the number of agents is fixed. Coordinating an ideal group is a non-trivial task because the collective performance depends not only on each individual's 's ability but also on the degree of each agent's 's feature complementing one another (Kim et al. 2016) and the needs for optimal coordination among agents will become more salient in massive settings (Suarez et al. 2019). In our task, we tried to investigate the relationship between the number of total pixels covered by the group of agents and the performance but were unsuccessful at finding any significant correlations. The second approach is improving the model architecture overcome the performance degrade, especially in the aggregator component. We conjecture that our current setting of naively superimposing internal representations might potentially "dilute" useful bits of information as the number of agents grows. Recently, simple aggregator architectures. such as summation or mean functions have been implemented previous research (Ali et al. 2018 Garnelo et al. 2018a b) and extended by adopting e.g., attention mechanisms (Bahdanau et al.
2015 Xu et al. 2015 Wang et al., 2018 Vaswani et al. 2017) as by Kim et al. (Kim et al.) 2019) or tower representation architectures by Eslami et al. (Ali et al. 2018).. Finally, we tried finding the complete observation "frame- wiiee" (rather than directly finding the optimal collective policy), with agents' scores and their encodings at every time step and using Bayesian optimisation (Mockus) 2012 (Shahriari et al. 2015 Močkus 1975) with neural processes (Garnelo et al., 2018a Kim et al. 2019) as the surrogate function, but was unsuccessful because of time complexity.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eslami Ali, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari Morcos, Marta Garelo,, Avraham Ruderman, Andrei Rusu, Ivo Danihelka, Karol Gregor, David Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene represen- tation and rendering. Science, 360(6399): 1210, 2018.
Karl Astrom. Optimal control of Markov processes with incomplete state information. Journal of mathematical analysis and applications, 1741):77 1965.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In ICLR, 2015.
Nolan Bard, Jakob Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare Bellemare, and Michael Bowling. The hanabi challenge: new frontier for ai research. arXiv preprint arXiv:1902.00553 2019.
Peter Battaglia, Jessica Hamrick, Victor Bapst, Alvaro Sanchez-Gonzaler Vinicius Zambaldi, Ma- teusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: (ati:::0060....... 2018.
Christopher Bishop. Mixture density networks. Technical report, Citeseer, 1994.
Christopher Bishop. Neural networks for pattern recognition (chapter 6). Oxford university press, 1995.
George Box. Science and statistics. Journal of the American Statistical A 711359):791-799 1976.
George Box. Robustness in the strategy of scientific model building. In Robustness in statistics. 1979.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Lucian Bu, Robert Babu, Bart De Schutter, et al. A comprchensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 18(2):156177,, 2008.
Lucian Buşoniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning: An overview. In Innovations in multi-agent systems and applications-11 2010.
Jie Chen, Kian Hsiang Low, Colin Keng- Yan Tan, Ali Oran, Patrick Jaillet, John Dolan, and Gaurav Sukhatme. Decentralized data fusion and active sensing with mobile sensors for modeling and predicting spatiotemporal traffic phenomena. In UAI, 2012.
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint 17X.v:1704.02254, 2017.
Paul Cisek and John Kalaska. Neural correlates of mental rehearsal in dorsal premotor cortex. Nature, (31(7011):99)) 2004.
Jean Decety and Julie Grèzes. The power of simulation; imagining one's own and other's behavior. Brain research, 1079(1):4-14, 2006.
Mare Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In NewrIPS, 2017.
Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. In ICLR, 2017.
James Driskell, Carolyn Copper, and Aidan Moran. Does mental practice enhance performance? Journal of applied psychology, 79(4),481, 1994.

---

Under review as a conference paper at ICLR 2020
Kai Ellefsen, Charles Martin, and Jim Torresen. How do mixture density RNNs predict the future? arXiv preprint arXiv:1901.078.00 2019.
Raphael Feraud, Reda Alami, and Romain Laroche. Decentralized exploration in nulti--amedd bandits. In ICML, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-lcarning. In NeurIPS, 2018.
Javier Gareia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480 2015.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and Ali Eslami. Conditional neural processes. In ICML., 20.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, Ali Eslami, and Yee Whye Tch. Neural processes. arXiv preprint 1rXiv:1807.01622. 2018b.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint 188.00000..0800
2013.
Alex Graves. Hallucination with recurrent neural networks.
https://youtu.br
SXDSYeDHbg 2015.
Thomas Griffiths, Nick Chater, Charles Kemp, Amy Perfors, and Joshua Tenenbaum. Probabilistic models of cognition: Exploring representations and inductive biases. Trends in cognitive sciences, 14(8):357-364, 2010.
Arthur Guez, Théophane Weber, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wierstra, Rémi Munos, and David Silver. Learning to search with metsnets. arXiv preprint wrXivv1802.04699 2018.
Matthew Guzdial, Boyang Li, and Mark Riedl. Game engine learning from video. In IJCAI, 2017.
David Ha. Evolving Stable Strategies. evolving- strategies/ 2017.
David Ha and Douglas Eck. A neural representation of sketch drawings. ICLR, 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS. 2018a.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint
2018b.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In JCML, 2019.
Jessica Hamrick. Analogues of mental simulation and imagination in deep learning. Current Opinion in Behavioral Sciences, 29:8-16. 2019.
Jessica Hamrick, Andrew Ballard, Razvan Pascanu, Oriol Vinyals, Nicolas Heess, and Peter Battaglia. Metacontrol for adaptive imagination-base optimization. In ICLR, 2017.
Nikolaus Hansen. The CMA evolution strategy: a comparing review. Towards a new evolutionary computation, pp. 75-102. Springer, 2006.
Nikolaus Hansen. The CMA evolution strategy: a tutorial. arXiv preprint arXiv:1604.00722 2016.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2) 2001.
10

---

Under review as a conference paper at ICLR 2020
Minh Hoang, Nghia Hoang. Kian Hsiang Low, and Carleton Kingsford. Collective model fusion for multiple black-box experts. In ICML, 2019a.
Nghia Hoang, Minh Hoang, Kian Hsiang Low, and Jonathan How. Collective online learning of Gaussian processes in massive multi- -agett systems. In AAAI, 2019b.
Leslie Kaelbling, Michael Littman, and Andrew Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Leslie Pack Kaelbling, Michael Littman, and Anthony Cassandra. Planning and acting partially observable stochastic domains. Artificial intelligence, 101(1-2):99-334 1998.
Michat Kempka, Marck Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Vizdoom:
A doom-based Al research platform for visual reinforcement learning. In CIG, 2016.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In ICLR, 2019.
Jooyeon Kim, Brian Keegan, Sungjoon Park, and Alice Oh. The proficieney-congrenny dilemma: Virtual team design and performance multiplayer online games. CHI, 2016.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian madel-agnostic meta-learning. In NeurIPS, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Oleg Klimov. Carracing-vll v0. https::/ggm.oppeeee http://wppaaaa..coe///eeee///aaa 2016.
Brenden Lake, Tomer Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Fatimah Lateef. Simulation-based learning: Just like the real thing. Journal of Emergencies, Trauma and Shock, 3(4):488, 2010.
Felix Leibfried, Nate Kushman, and Katja Hofmann. A deep learning approach for joint video frame and reward prediction in Atari games. In ICLR. 2017.
Michael Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML, 1994.
Ilya Loshchilov and Frank Hutter. CMA-ES for hyperparameter optimization of deep neural networks. arXiv preprint arXiv:1604.0777 1604.07269, 2016.
Kian Hsiang Low, Jiangbo Yu, Jie Chen, and Patrick Jaillet. Parallel Gaussian process regression for big data: Low-rank representation meets Markov approximation. In AAAI, 2015.
Ingo Meister, Timo Krings, Henrik Foltys, Babak Boroojerdi, Mareike Müller, Rudolf Töpper, and Armin Thron. Playing piano in the mind-an fmri study on music imagery and performance in pianists. Cognitive Brain Research, 19(3):219 2228, 2004.
Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference, 1975.
Jonas Mockus. Bayesian approach global optimization: Theory and applications, volume 37. Springer Science & Business Media, 2012.
Lev Muchnik, Sinan Aral, and Sean Taylor. Social influence bias: A randomized experiment. Science, 341(6146):647-6511 2013.
Radhika Nagpal. Taming the swarm collective artificial intelligence. httpss//youtu.be/ LHcVRO1zFJc// 2016.
Ashvin Nair, Vitchyr Pong. Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine, Visual reinforcement learning with imagined goals. In NewrlPS, 2018.
11

---

Under review as a conference paper at ICLR 2020
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint 1.13v18800009990 2018.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in Atari games. In NeurIPS, 2015.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In NeurIPS, 2017.
Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art Aulonomous agents and multi-agent systems, 11(3)3387-434, 2005.
Philip Paquette.
Doomtakeccverr-v..
https://gya..peeaaa.coo//
DoonfTakeCover-vv/ 2016.
Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebastien Racanière, David Reichert, Théophane Weber, Daan Wierstra, and Peter Battaglia. Learning model-based planning from scratch. arXiv preprint arXiv: 02.006170 2017.
David Premack and Guy Woodruff. Does the chimpanzee have theory mind? Behavioral and brain sciences, 1(4):515-526, 1978.
Chao Qu, Shie Mannor, Huan Xu, Yuan Qi, Le Song, and Junwu Xiong. Value propagation for decentralized networked deep multi-agent reinforcement learning. In NeurIPS, 2019.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination-augmentee agents for deep reinforcement learning. In NeurIPS, 2017.
Danilo Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In ICML, 2014.
Michael Rubenstein, Alejandro Cornejo, and Radhika Nagpal. Programmable self-assembly in thousand-robot swarm. Science, 345(6198):795-799 2014.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 2017.
Jürgen Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments (TR FKI-126-90). Technical report, NA, 02 1970.
Jürgen Schmidhuber. An on-line algorithm for dynamic reinforcement learning and planning in reactive environments. In IJCNN, 1990.
Jürgen Schmidhuber. Reinforcement learning in Markovian and non-Markovian environments. In NewrlPS, 1991.
Jürgen Schmidhuber. On learning to think: Algorithmic information theory for novel combina- tions of reinforcement learning controllers and recurrent neural world models. arXiv preprint arXiv:1511.09249, 2015.
Bobak Shahriari, Kevin Swersky, Ziyu Wang. Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148-175 2015.
David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Amold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. In ICML, 2017.
Maximilian Soelch, Adnan Akhundov, Patrick van der Smagt, and Justin Bayer. On deep set learning and the choice of aggregations. arXiv preprint arXiv: arrXiv/903.000 1903.0734330 2019.
12

---

Under review as a conference paper at ICLR 2020
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural MMO: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint vrXiv:1933.00784, 2019.
Richard Sutton. Integrated architectures for learning, planning, and reacting based approximating dynamic programming. In Machine Learning Proceedings. 1990.
Richard Sutton and Andrew Barto. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks, In NeurlPS, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In ICCV, 2018.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning physies simulator from video. In NeurIPS, 2017.
Paul Werbos. Learning how the world works: Specifications for predictive networks in robots and brains. In IEEE SMC, 1987.
Anita Woolley, Christopher Chabris, Alex Pentland, Nada Hashmi, and Thomas Malone, Evidence for collective intelligence factor in the performance of human groups. Science, 330(6004):6866666 2010.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML,, 2015.
Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In IROS, 2017.
13

---

Under review as a conference paper at ICLR 2020
A
LINKS FOR VIDEO TRIALS AND SIMULATIONS
In supplementary material, we attach videos for actual trials and simulations.
A.1 CARRACING
CarRacing collective trial carracing catptig__oolletv_______ and carracing cacpioggooollttteeeeee 2_.gge files are the ceal-enviromment trial using collective policy with agents' internal models. Since we cannot visualise the decoding of the superimposed biased representations, we present the viewpoint of each agent. The performance of this collective policy is reflected in the main draft.
CarRacing individual trial carracing_individuulJ1 f.gif individual internal model.
f file is the real environment trials using
A.2 VIZDOOM
We demonstrate the delusion of agents in simulation under biased representations, which is reported and discussed in section 3.2.
Normal agent in simulation Besddom_normal_com caaillllggnntmmpppp file is simulation of normal agent with occlusion.
Delusional agent in simulation vizdoom deceoonddddussoualeemm dellusionallagent.mmgn file is simulation of delusional agent with occlusion. These two agents are trained with exactly the same specification. We tried the same specification for not just these two agents, but with 6 more, confirmed that this delusion occurs randomly.
B ADDITIONAL RELATED WORK
Nascent development of simulation in ML In reinforcement learning (RL) (Kaelbling et al. 1996. Sutton & Barto 1998). the concept of simulation develops from prediction. Model-based RL builds predictive model that abstracts key features of observations and leverage the state space to predict future observations for planning. For example, there is line of work that uses such predictive powers of the world for short-step lookaheads using FNNs (Guzdial al. 2017) and RNNs (Sutton) 1990) Werbos 1987 Dosovitskiy & Koltun, 2017), while another branch focuses on generating longer-range predictions conditioned time step wise actions using pixel image frames inputs. but without policy learning (Chiappa et al. 2017 Oh et al.) 2015) Leibfried et al. 2017: Denton & Birodkar
2017 Watters et al.) (017). Model-free RL also benefits from exploiting the predictive power by
directly predicting future rewards or values using deep neural nets (Silver et al. 2017 Oh et al. 2017
Tamar al. 2016).
Humans' efficacy of imagination For humans, the benefit of imagination, in the names of mental practice, mental training and cognitive rehearsal, is empirically demonstrated in varying fields of
learning e.g., musical instruments (Meister al. 2004). physical actions (Premack Woodruff 1978).
and social interactions (Decety & Grèzes 2006 Lateef 2010) and validated with neurological Ciiek
Kalaska 2004) and cognitive (Driskell al. 1994) evidence.
Distributed ML Finally, in terms of model architecture, our research falls into the branch of distributed ML, and specifically, into recently proposed model fusion (Hoang et al.) 2019a) and collective machine learning (Hoang et al., 2019b). Similar to ours, Hoang et focuses on setting where there are multiple independent experts, and the objective is to uncover the communication protocol among them in a distributed manner to reach the consensus prediction. However, the target domains of this line of research and other recent advances in distributed machine learning (Feraud et al. 2019 Qu et al. 2019) mainly lie in privacy and security (e.g., a setting where each expert treats confidential information) reducing the overload of the central server. On the other hand, in our research, the communication protocol issue among agents' internal models is structurally negated since although train each agents' V component (VAE) independently, there's collective
14

---

Under review as a conference paper at ICLR 2020
controller C that's also trained independently to all the individual V components and also to the individual C components.
C TRAINING SPECIFICATIONS
We used the same setting as in the original world models paper for training the individual components V and M.
C.I CONTROLLER COMPONENT C
For C. we used CMA-ES for training. We concatenated 16 rollouts to define one episode, and each virtual rollout hosted by an agent the collective system. We made sure that the number of rollouts in the episode can be exactly divided by the total number of agents in collective world model with remainder of 0, so that all agents can be hosts the same number of times. Also, with this setting, we manage to keep the training iteration of collective policy exactly the same with the individual policies. For example, there are 8 agents in total, each can host simulations 2 times in a single episode; there are 4 agents in total, each can host simulations 4 times in single episode, and so forth.
We set the population size to 32, and each population is assigned with a single CPU, We set the initial sigma value of CMA to 0.1, and set all solution values to 0 for initialisation. Finally, for every training, we used random seed numbers.
C.2 RCLE COMPONENT R
For the rule component R, we used RNN with 2 hidden layers, each layer with the same size of 256. We used Adam as optimizer, with initial learning rate of 0.001, and used mean squared error for the loss function. We also used dropout for all hidden layers with the value 0.1, gradient clipping value of 0.25, batch size of 100, and training epochs of 500. For inputs, for every time step é, we used generated encodings the current as well as the 10 previously generated encodings, so in total of 352-dimensional input space was used, and scalar value of the instant reward was returned.
generously set. Since training of this component not the bottleneck of our computation, did not try to shrink the model architecture, as only focused on the model returning low training/testinng errors. Therefore, the current model size might be overly large, and we believe that this size be drastically reduced without significant loss in accuracy.
Here, we note that in our setting, the model complexity
C.3 TRANSLATOR COMPONENT
For the trainslator component T, we used FNN with 4 hidden layers, each layer with sizes 1024, 512, 256, and 128. We used Adam as optimizer, with initial learning rate of 0.001, and used mean squared error for the loss function. We also used dropout for all hidden layers with the value 0. batch size of 10,000, and training epochs of 250. Inputs and outputs are both 32-dimensions, which corresponds to the size of the encoding z.
As in the rule component R, we believe that model size of T can also be drastically reduced.
C.4 COMPUTING INFRASTRUCTURE
Our model is programmed using python Tensorflow.
Training C is done in multi-CPU environment, as CMA-ES can be effectively implemented in parallel computations. We used 32 CPU cores for training both collective and individual C, where each core assigned with single population in CMA-ES.
For training V (VAE), M (MD-RNN), T(FNN), R(RNN), we used a single GPU GTX- 1080 Ti for training each component.
15


Reviews:
[{"review_id": "rJlsPbo0KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approach to conducting RL: it proposes to first train a collection of individual agents independently, and then exploit them to learn one collective policy. Each agent has an observation space that is differently impaired, such as by blocking out some specific set of pixels in its image observations. Each such agent learns its own dynamics and reward models based on its own observations, and goes on to train a policy based on simulated rollouts using these learned models. In the second phase, a collective policy is trained as follows: every training episode happens inside a different \"host\" agent's simulation, and the collective policy is thus effectively trained on this mixture of variously inaccurate simulations. Since different individuals have different observations, the collective policy receives an input observation that is the aggregate of all agents' individual observations. Throughout, this paper builds on top of the \"world models\" approach proposed in Ha & Schmidhuber 2018.\n\nMost sections of this paper are well-written and the high-level ideas are novel and interesting. My main issues with the paper are listed below, in rough order of priority:\n(1) I find the experimental evaluations short of convincing.\n  * Baselines: The proposed collective policy is evaluated against the individual policies of agents in its population, which is a very weak baseline since the individual agents are artificially impaired. It is also evaluated against an upper bound: a  policy trained directly in the real world, and shown to be only slightly weaker in performance. I would suggest the following additional baseline:\n     - individual agents without observation impairments, trained in simulation (essentially the world model paper this approach builds on). [\"world models\"]\n     - the average of the individual policies [\"policy ensemble''].\n     - a policy trained with a model that is a composite average of the individual world models [\"model ensemble\"]. This is the in the spirit of prior work on model ensembling, such as Chua et al 2018, \" ... handful of trials.\"\n  * Environments: The results presented so far indicate the proposed approach works on one environment (CarRacing) and does not on another (VizDoom). I would like to see experiments on more environments to figure out whether CarRacing is the exception or the rule.\n  * choice of observations for individual agents: in a general setting, how are these to be engineered? \n\n(2) It does not sufficiently acknowledge or clarify connections to other work in the field, such as on model bootstrapping in model ensembles (e.g. Chua et al 2018, \"... handful of trials.\") or evaluate against them. In particular, I think the proposed approach is a novel way to achieve model bootstrapping, plus some additional tweaks: rather than train different models on different subsets of training experience, it trains different models on different feature views of the same training episodes. \n\n(3) It does not sufficiently motivate its setting: when is it true in realistic settings that it would make sense for different agents in an environment to be artificially impaired in different ways? Experiments are only conducted in contrived settings where portions of the environment are deliberately removed from the observation for different agents. I also have a somewhat related suggestion: perhaps future versions of the paper might focus on using different modalities (such as RGB images and depth) for the different agents.\n\n(4) From my understanding, there seems to be a tradeoff between how much overlap there exists between different agents' observations and how translatable different agents' views are to each other. In other words, the method requires the translator T to take one view of an agent and transform it into (a feature representation of) the view of another agent. This seems ill-defined in general settings. \n\n(5) Related to the above, how is the aggregated vector [z_{ct}, h_{ct}] generated for the collective policy when it is eventually evaluated in the real world? Are the features computed separately by each agent before aggregation, or is one agent selected and the translator T used again? If the former, then does this induce a domain shift between training time when the policy is trained on predicted features from the translator, and testing time when the policy sees true features corresponding to different agents?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJlU5046OH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the settings where multiple agents each act independently in different copies of an environment, without interacting with each other. Each agent uses model-based learning, learning a representation of the world, then learning a controller against the learned world model (learned with an evolutionary strategy of CMA-ES). Different agents will naturally learn world models with different biases - this paper proposes a method that learns a collective policy from the biased models of the different agents, by letting each agent observe imagined rollouts from other agents' world models, training on that data. They find this improves performance over learning within each agent's individual model.\n\nFrom a style perspective, I found this paper hard to read. Many terms are immediately abbreviated into non-conventional single letter abbreviations (C for controller, T for Translator). This extends to the results table - it's quite difficult to understand the results in Table 1 and Table 2 without a careful reading to find what what the abbreviations mean, and I still don't understand what the Overlap (%) and Cov. (%) columns mean in Table 2, despite reading the paragraph explaining it several times. The analysis section also makes a distinction between delusional agents and cheating agents, which I also didn't understand. The GIFs provided demonstrate that the model moves towards the occluded region, and this is described as a novel delusion that hasn't been observed before, distinct from the cheating behavior from Ha and Schmidhuber's world models paper. I don't understand why these are different - in both cases, the agent travels to a region of space where no fireballs appear, and since reward is defined as not getting hit by a fireball, this is optimal behavior for the imperfect world model.\n\nTo created biased representations, the authors apply cutout or zoom in on specific parts of the state. I found this surprising - shouldn't different models naturally lead to biased representation by themselves? The VAEs used should naturally be inaccurate enough to lead to different imperfections.\n\nThere are very few details for the translator T, that translates latent z_i from world model M_i into the correspond latent x_j for world model M_j.  This seems like a very important detail of the method, I could see different translators T leading to very different results, and would have like to see ablations for how different Ts affected the training procedure.\n\nOverall, the question I found myself asking the entire paper was, \"what does this add on top of existing work on training against an ensemble of world models?\" This question has been studied in previous work (a quick search turned up https://arxiv.org/abs/1809.05214 and https://arxiv.org/abs/1802.10592). In these prior works, instead of learning a translator T between representations, we decode an imagined rollout from the VAE for each world model, in the original high-dimensional input state (images, in the case of this paper), and simply train on that data instead. This does not require learning O(N^2) translator models translating z_i to z_j for each pair of world models. I would have liked to see a comparison to this approach, because right now I don't see why you would want to learn translators T in the first place. Simply aggregating rollouts across models would still lead to individually biased world models that can be aggregated to learn a single controller that performs better than any single model.\n\nEdit: have read author reply, no changes to review.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkg92xTnFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model-based RL architecture based on World Models which additionally makes use of an ensemble of models. Specifically, each model in the ensemble is independently trained models and has different amounts and types of occlusions (i.e. part of the image cropped out or a small piece of the image is magnified). The ensemble of models are used to train what is referred to as a “collective” policy in the CarRacing and VizDoom environments. The collective policy achieves better performance than policies trained using the individual models and achieves higher data efficiency than model-free RL.\n\nWhile I think the paper tackles an interesting question---how to leverage multiple models of the world that may be trained on different data and with different capabilities---I unfortunately do not feel that this paper is ready for publication at ICLR, and thus recommend rejection. My justifications are that (1) the paper suffers from clarity issues and several cases makes claims or statements which are incorrect, (2) the proposed evaluation is somewhat contrived and the method does not seem to generalize well to other domains, and (3) there is not much engagement with the rest of the literature on model-based RL. I think that addressing #1 and #3 alone would substantially improve the paper. An additional change that would make the paper much more compelling would be to train the models on more ecologically valid observations, such as from multiple modalities or different camera angles (see below for discussion).\n\n\nMy first concern with the paper has to do with clarity. Overall, I found the paper quite difficult to read as it uses a lot of unusual terminology to explain what it does. For example, the paper uses “rule component” to refer to what would normally be a “reward function”. Similarly, although this is an ensemble method, the term “ensemble” is never used even once. The word “bias” is also used in a way which is different from how it is normally used---here it seems to refer to the fact that the agents see different things, whereas in ML bias typically has a precise meaning in terms of the bias-variance tradeoff. I also noticed several places throughout the paper where citations or related work seem to be misunderstood:\n \n- 1st page, 2nd paragraph: this is not what Griffiths (2010) is about, and is not really an appropriate use of the term “inductive bias”\n- 2nd page, contribution 1: the paper claims that it is not dealing with a partially observable setting, but this is exactly what the paper is doing. Specifically, each model in the ensemble receives different partially observed observations from the environment. Although in traditional RL partially observed usually means “one agent, one observation of a state”, there is nothing limiting the general formulation of partial observability to cover the case of “multiple agents, multiple observations of the same state”.\n- 8th page, line 2: Lake et al (2017) is not really about “learning to think”.\n- Section 4, last paragraph: this seems to be not very relevant to this paper, as the present paper is not about a multiagent system or about graph-based representations.\n \nThere are some additional issues with clarity of the model explanation; for example, it’s not clear how the translator component is trained (the paper states that it is trained with meta-learning, but this is vague).\n \nMy second concern is that the approach is somewhat contrived and does not seem to generalize well to other domains. In particular, under what circumstances would you want to train multiple models that receive inputs that are missing different pieces? I can see the appeal of learning different models in some cases (e.g. different camera angles, or different sensory modalities) but these have not been evaluated here which makes the whole thing seem a bit arbitrary. Moreover, while it seems to work ok in the CarRacing domain (though the ensemble-trained policy does not perform as well as a non-ensemble agent which is trained with full observations) it does not seem to work well in the VizDoom environment. On the basis of these two issues it is not at all clear to me how well the proposed method will work in more ecologically valid settings requiring multiple models.\n \nFinally, my third concern is that the engagement with the rest of the model-based RL literature is lacking. As mentioned above, this method is an ensemble method but there is no discussion of other model-based RL ensemble methods (e.g. [1] and [2]). Similarly, the paper dismisses other work on partial observations but this work is quite relevant to the present paper which does indeed include partial observations (see above). While there is a lot of text spent discussing other work such as literature from cognitive science, this is not really that relevant to the present paper and that space would be better spent discussing related work from model-based RL.\n\nSome additional minor comments:\n\nIt would be interesting to see a comparison between the ensemble policy trained with models that always receive the same (full) observations versus the present approach (where they receive partial observations). This might still provide some benefit as the models will still result in slightly different latent representations due to different initialization and training data, so it would be interesting to see whether this helps as well even if the observations are the same.\n\nWhat does “model tr” and “policy lr” mean in Table 1?\n\nIn the various tables and figures, how are the error bars computed? Are they over episodes, or agent seeds? In general, how many seeds were used to perform evaluations? (Ideally it would be at least 3).\n\nPage 2, section 2.1: “V and M use sequential image frame data from multiple rollouts from a random policy and apply variational inference and backpropagation algorithm for inference and learning.” → this is imprecise; they do not apply both variational inference and backpropagation; backpropagation is used to implement variational inference.\n\nI found the results of Figure 3d quite interesting, though am wondering if you controlled for the number of evaluations of each model? E.g. if you only train the policy on one model at a time versus training them on all models at a time, do you train for longer in the first case so that the policy ultimately receives the same amount of information from all the models?\n\n[1] Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018). Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592.\n[2] Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (pp. 4754-4765).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an ensemble method for reinforcement learning.  The method trains an ensemble of transition and reward models.  Each element of this ensemble has a different view of the data (for example, ablated observation pixels) and a different latent space for its models.  A single (collective) policy is then trained, by learning from trajectories generated from each of the models in the ensemble.  The collective policy makes direct use of the latent spaces and models in the ensemble by means of a translator that maps one latent space into all the other latent spaces, and an aggregator that combines all the model outputs.  The method is evaluated on the CarRacing and VizDoom environments.  

The reviewers raised several concerns about the paper. The evaluations were not convincing with artificially weak baselines and only worked well in one of the two tested environments (reviewer 2). The paper does not adequately connect to related work on model-based RL (reviewer 1 and 2). The paper does not motivate its artificial setting (reviewer 2 and 1).  The paper's presentation lacks clarity from using non-standard terminology and notation without adequate explanation (reviewer 1 and 3).  Technical aspects of the translator component were also unclear to multiple reviewers (reviewers 1, 2 and 3).  The authors found the review comments to be helpful for future work, but provided no additional clarifications.

The paper is not ready for publication.
Decision: Reject