Title: EMERGENCE OF COLLECTIVE POLICIES INSIDE SIMULATIONS WITH BIASED REPRESENTATIONS

Abstract: We consider a setting where biases are involved when agents internalise an environment. Agents have different biases, all of which resulting in imperfect evidence collected for taking optimal actions. Throughout the interactions, each agent asynchronously internalises their own predictive model of the environment and forms a virtual simulation within which the agent plays trials of the episodes in entirety. In this research, we focus on developing a collective policy trained solely inside agents' simulations, which can then be transferred to the real-world environment.
The key idea is to let agents imagine together; make them take turns to host virtual episodes within which all agents participate and interact with their own biased representations. Since agents' biases vary, the collective policy developed while sequentially visiting the internal simulations complement one another's shortcomings. In our experiment, the collective policies consistently achieve significantly higher returns than the best individually trained policies.

1. INTRODUCTION:

We look at the world through our own lenses. Two people with the same exact observations create different internalizations of the world through their own mental models, collect different amounts of evidence, take different actions, and end up with different outcomes. For example, students attending a lecture look at different parts of the presentation slides, and doctors looking at the clinical log of a patient focus on different data and statistics and make different diagnoses.

In model-based reinforcement learning (RL) (Kaelbling et al., 1996; Sutton & Barto, 1998) , agents interact with an environment through models that are analogous to the human mental models (Hamrick, 2019) . However, just as our decisions governed by our mental models are often far from perfect, the agents' policies that are designed upon the models are suboptimal (Box, 1976; 1979) , as the inductive biases used to learn such models are not guaranteed to be correct (Griffiths et al., 2010) .

In this research, we propose a learning algorithm of a collective policy that leverages multiple agents' biased representations encoded from their internal models. Like humans who often compensate for their cognitive biases through collaboration (Woolley et al., 2010; Muchnik et al., 2013) , the goal of the collective policy is to make use of multiple internal models so that the biased representation formulated by each internal model complements one another to achieve higher returns than the bestperforming individual policy. The collective policy in this research has the following characteristics:

-Asynchronous shaping of agents' internal models: The collective policy operates on agents' preestablished internal models. Prior to learning the collective policy, agents construct their internal models over the course of their own interactions with the environment, which can be done asynchronously and independent of other agents. Note that this learning process of the collective policy does not fall into the conventional multi-agent RL settings, where agents are concurrently situated in the same environment (Littman, 1994; Panait & Luke, 2005; Bu et al., 2008; Buşoniu et al., 2010) .

-Collective policy learned solely inside agents' internal simulations: In model-free and model-based RL, policy learning typically requires a large amount of interaction with the environment, which hampers feasibility, time-efficiency (Guez et al., 2018) , and safety (Garcıa & Fernández, 2015) . This is particularly prohibitive in our setting, as during the collective policy learning, we need to force the agents to go through the same rollouts together, coordinate their actions, and interact with the environment in algorithmically intended ways (see e.g., Yahya et al. (2017) ; Rubenstein et al. (2014) ; Nagpal (2016) ). To overcome this, we propose a learning algorithm of the collective policy that involves zero interaction with the real environment; instead, the collective policy is trained solely inside agents' simulations, analogous to human's mental simulation or imagination (Hamrick, 2019) . In our learning algorithm, agents take turns to be the host that generates a virtual rollout, within which all agents participate in and learn the collective policy together.

-Minimal number of parameters to learn the collective policy: In the proposed algorithm, the number of parameters and the training iterations of the collective policy function is same as those of the individual policy functions. This not only allows the efficient training, but also assures that the performance edge of the collective policy over the individual policies is not attributable to the model complexity but to the complementing evidences provided by the multiple internal models. Moreover, learning of the collective policy can precede learning of the individual policies.

We make the following contributions in this paper: I. We present a novel problem setting in model-based RL that resembles the human cognitive bias of imperfect information. Note that our setting is clearly different from the partial observation settings in RL (Astrom, 1965; Kaelbling et al., 1998) ; in our setting, two agents at the same exact time and location in the environment encode different representations from the same observation.

II. We compare the real environment and the internal simulation generated by agent models for training a policy. With appropriate early stopping epoch and τ , we can achieve much better scores at a fraction of the training time.

III. We present a learning algorithm of the collective policy that operates on the simulations generated from internal models of agents and complements their biased representations, thereby achieving higher scores than that of the best-performing individual policy.

IV. We experiment with VIZDOOM, a different environment in which we observe an unexpected behavior of agents becoming delusional, i.e., having the false belief about the reward system, preventing us from learning the collective policy in simulation.

V. We release our source code, visual demonstrations, additional results, and in-depth discussions in a separate link1 .

2. COLLECTIVE POLICY TRAINING ALGORITHM:

Here, we describe the process of training a collective policy by leveraging multiple agents' internal models that create their own biased representations given the same observation as inputs. We start by describing the components of agents' internal models and explain how each agent creates its own biased representation of an observation inside the model. Then, we introduce additional components required for training the collective policy inside the agents' simulations.

2.1. INTERNAL MODELS OF AGENTS:

We use world models (Ha & Schmidhuber, 2018a; b; Schmidhuber, 2015) as agents' internal models. World model abstracts spatio-temporal features of the environment in a compact representation that is used for policy learning after the model training. The left panel of Figure 1 illustrates the two components of the world model along with the environment and the controller C that learns the policy after the world model is trained. First, at each time step t, the visual sensory component V abstracts a high dimensional input observation from the environment into a compressed representation. Variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) is used to encode a 2D image input frame into a latent vector z t . Then, the latent vector z t as well as the following action a t are used as inputs for the memory component M, which outputs a prediction of the next encoding z t+1 . Mixture-density recurrent neural network (MD-RNN) (Bishop, 1995; 1994; Graves, 2013; Ha & Eck, 2018; Graves, 2015) is used for this component for the stochastic prediction of the future, i.e., P (z t+1 |z t , h t , a t ), where h t is the hidden state variable at time t. V and M use sequential image frame data from multiple rollouts from a random policy and apply variational inference and backpropagation algorithm for inference and learning.

After the two components of the world model are trained, the parameters of C are optimised online during the following rollouts using covariance matrix adaptation evolution strategy (CMA-ES) (Hansen & Ostermeier, 2001; Hansen, 2006; 2016) which is shown to be the method-of-choice for small solution spaces up to few thousand parameters (Salimans et al., 2017; Loshchilov & Hutter, 2016) .

Training policy inside an agent's simulation The memory component M predicts the latent representation of the next frame and allows an agent to take an action based on the agent's prediction of the near future as well as the current situation. Further, one can generate multiple future frames, even the whole episode from start to finish (Figure 1b ). Then, C is trained inside this virtually generated environment (Ha & Schmidhuber, 2018a; b) . MD-RNN, which is the realisation of M, can endow a desirable level of uncertainty in the simulation by adjusting the temperature parameter τ .

Biased representations:

We hypothesize that when an agent's model internalizes the observation through V, the agent from collecting the full evidence needed to learn the optimal policy because of bias. The bias, which persists throughout the experiment, can be that certain parts of the frame are left out (occlusion), or only a certain part of the frame is accessible (magnification).2 

Asynchronous shaping of agents' world models Prior to our merging of agents' simulations, we assume that agents preestablish their internal models (M and V) asynchronously, based on their imperfect evidence accumulated. However, it is necessary to ensure there are environment states that all agents share, e.g., few frames that all agents co-observe, to train the translator component. This will be further explained in the following subsection. Unlike the typical distributed ML systems (Chen et al., 2012; Deisenroth & Ng, 2015; Low et al., 2015) , each internal model can be trained independently without broadcasting or establishing the communication protocol among agents, i.e., each agent's encoded representations from V can be done using independent VAEs, each with different initialisations. This is because the collective controller and all individual controllers are trained independently of each other Cs, as well as the remaining components V and M.

2.2. COMPONENTS FOR LEARNING THE COLLECTIVE POLICY:

We gather up agents' biased representations created inside the individual world models and train a collective policy in the internal simulations generated from the agent models so that each model's biased representation complements others' shortfalls and the virtual return per episode is maximised in a collective manner. Here, the key idea is to make agents be in the same internal simulation at the same time, where each agent's model takes a turn to be the host of the virtual rollout using her respective memory component M (Figure 1c ). Throughout this section, we suppose that there are a total of N internal models in a system, and the hosting agent per virtual rollout is denoted by H. Added components for the collective policy learning are explained as follows:

Rule component (R) When learning a policy in the environment, we automatically retrieve reward signals from the environment. However, when the (collective) policy is trained in simulation, we need to learn how the reward is given using previous and current state representations. Within each agent's model, we only preserve the memory component M and introduce an additional component R that understands the rule of the reward system. For every virtual rollout we have a hosting agent H whose rule component R H returns a virtual score which is instantly transferred to the controller component C so that C can assess the quality of the current policy solely inside simulation and without the need to interact with the actual environment. For this component, we use RNN that takes previous and current biased representations as inputs and returns the estimated reward for each time step.

Translator component (T) In our setting, agents' visual sensory components create different representations from the same observation. While we train the collective policy inside agents' simulations, the translator component T converts biased representations between two agents. For every virtual rollout generated by the hosting agent H, we need N -1 translators T H,a for a ∈ {1, 2, . . . , N }\{H}, which are the functions T H,a (z H ) = z a . In our setting, the internal representations are much smaller in dimensionality than the original observations, and representations are highly intercorrelated among different internal models. Therefore, we apply meta-learning (Finn et al., 2017; 2018; Kim et al., 2018; Nichol et al., 2018) to speed-up the training of T and reduce the training data.

Aggregator component (A):

To train a collective policy, there is an aggregator component A which is a permutation-invariant function that takes representations from agents' internal models as inputs and returns their summarisations of the same shape. Here, we take the summation function as an aggregator which simple but effective (Ali et al., 2018; Garnelo et al., 2018a; b; Soelch et al., 2019) . Retrieve an initial encoding for the first time step z H,0 ; for time t until episode terminates do Generate encodings of other agents z a,t , ∀a ∈ {1, 2, . . . , N }\{H} using z H,t from T; Generate h a,t for all agents using their respective M and aggregate N concatenated vectors [z a,t ; h a,t ] using A to return [z c,t ; h c,t ];

Insert the aggregated vector[z c,t ; h c,t ] in the controller C to get an action at time t;

Execute the action inside the hosting agent's simulation M to infer the next encoding z H,t+1 and to get a reward r from R;

end Set RewardT otal ← RewardT otal + r;

end Update the parameters for the collective policy using CMA-ES and RewardT otal; end Training scheme for C We use CMA-ES that operates on a state space to optimise the parameters of C. To use the simulations of all hosting agents, for each training iteration, we define a single episode by concatenating multiple virtual rollouts and use the sum of the cumulative rewards calculated from the hosting agent's rule component R H . Aside from making use of all agents' simulations, combining multiple rollouts as a single episode is shown to stabilise policy learning throughout the training (Ha, 2017) . We highlight that the number of parameters for C as well as the number of training iterations are identical to those of the individual controllers.

For each iteration, the controller component C operates on exactly the same input space as the individual model as we take the summation function and Therefore, the only added terms in the time complexity of collective world model come from T and A, where for T, we need N -1 forward passes of RNNs and A that aggregates N input vectors.

|[z c ; h c ]| = |[z a ; h a ]|, ∀a ∈ {1, . . . , N }.

In algorithm 1, we summarise how the learning of the collective policy works inside agents' simulations using trained components M, R, and T along with the aggregator A.

3. EXPERIMENT:

In this section, we answer the following questions: First, how does training a policy in internal simulation compare to training a policy in the real environment? What specifications do we need for comparable performance in the simulation? Second, how much do we gain from the collective policy trained within the simulations of agents' internal models whose biased representations complement one another? What causes the collective policy to perform better than the individual policies? In addition, we introduce another environment where when bias is involved during the internalization, agents in the environment become delusional, not being able to understand the rule of the environment.

3.1. EXPERIMENTAL SETUP:

Agents learn to drive in OpenAI gym CARRACING environment (Brockman et al., 2016; Klimov, 2016) . Each trial consists of 1,000 image frames, and while agents are driving a car, the score is calculated as the total number of tiles, i.e., the gray area in Figure 3a , the car visited. Thus, agents need to keep moving forward without derailing from the track until the trial ends in order to maximise the return. For each time step, agents can take three continuous actions: steer left/right and accelerate.

Throughout the experiment, we report an average score over 100 trials with one standard error. Following the previous setting by Ha and Schmidhuber (Ha & Schmidhuber, 2018a; b) , we train each internal model's components using 10,000 rollouts with random policy. As displayed in Figure 3a , we make each agent's observations imperfect by 1) occluding a certain part of the frame or 2) only showing a magnified portion of the frame. Here, we conserve the car image (in red) for all agents' observations, as this image portion is critical for the rule component to calculate the virtual score. Also, we highlight that this data-gathering phase is the only time interacting with the real environment throughout the experiment, except for the testing trials. Finally, we keep the same number of parameters and maximum training iterations for both individual and collective policies. generated by the internal model becomes more unpredictable. In Figure 2a , we observe that there is an optimal temperature parameter, which is 1.7 in CARRACING environment, that gives the score when the policy learned in simulation is transferred to the real environment. This result is in accordance with the result reported by Ha & Schmidhuber (2018a) , where they experimented in the VIZDOOM environment. In Figure 2b , we observe that there is an additional trend that incorporates τ ; With all τ , we observe that when we train the policy in simulation for too long, the policy overfits the virtual environment so that the virtual score monotonically increases as the training epoch increases but performs worse when the policy is transferred to the real environment. Thus, the early stopping is required when training the policy in simulation, and the early stopping epoch generally increases as we increase τ , meaning that we need more training iterations until we transfer the policy to the real environment when the virtual environment is more unpredictable.

3.2. EXPERIMENTAL RESULTS:



Policy learning in simulation:

From Table 1 , We report that with an appropriate early stopping epoch and τ , we can achieve a score of 851 ± 9.8 in CARRACING environment with the policy that is trained only in simulation and then transfered to the real environment, compared to the score of 900.0 ± 5.2 when trained in the real environment. When the policy is trained in simulation, the training time per epoch is approximately 53 times faster, total train time changes from 52 days to 3 hours, and the number of rollouts generated in the environment is reduced from 1.84 million to zero during the policy training.3 

Collective policy Table 2 shows the results of the collective policy compared with the individual policies whose internal models that are integrated. Here, #M is the number of internal models used for training, Overlap is the maximum percentage of frame overlaps between agents' biased representations in terms of pixel count, and Cov. is the percentage of total coverage when all biased representations are superimposed in terms of pixel count. From the first four rows in Table 2 , we confirm that when we train the collective policy that integrates multiple internal models whose biased representations complement one another, we can achieve higher scores than that achieved by the best-performing individual policy in the real environment. In the last two rows, we do not observe any significant increase in the score when the collective policy is compared with the individual policies, because one internal model's biased representation is a proper superset of that of the other's.

Figures 3b -3c show the virtual scores of individual and collective policies. When an individual policy is trained inside the simulation generated by its own model, we achieve the best score. However, when the trained policy is executed in the simulation generated by another model, the score decreases catastrophically. In comparison, the collective policy returns reasonable scores in all simulations, although the score is less then the one achieved by the individual policy in its own simulation. Figure 3d highlights the necessity of the model structures proposed in this research for learning the collective policy. Here, "Joint" corresponds to real-environment score achieved by the collective policy jointly trained by sequentially visiting multiple internal simulations, but while doing so, the agents other then the host does not participate in the simulation. In other words, its the collective policy without the T and A components. From this result, we confirm that the main performance edge of the collective policy comes from the "co-observation" during the policy learning.

Delusional agents We report that in the VIZDOOM environment (Paquette, 2016; Kempka et al., 2016) , an environment where the rules of the game and the other aspects of the environment are tightly coupled, agents randomly become delusional when the bias is involved in their internal models (4). Here, the rule that governs the reward system of the game is highly intertwined with physics of the system (movement and perspective of the fireball, explosion of the fireball when the ball contacts the agent) and is potentially affected by every pixel of the frame, and therefore the rule component R inevitably has to be incorporated within the memory component M (this phenomenon is extensively studied by Ellefsen et al. (2019) ). In this case, some agents' internal model will falsely believe that they will be rewarded not by dodging the fireball but by going inside the occluded region and stay there until the trial ends. To our knowledge, this type of delusion has previously not been observed; in the world models paper, agents will sometimes cheat inside simulation by creating an easy virtual environment when the temperature parameter is set to a low value but clearly it is a different type of behaviour. This abnormal behaviour is remarkable in two ways. First, an agent falling into a delusion happens completely at random at the training phase of the memory component. Thus, for the same exact training data, methods, and iterations, some agent will behave normally (treating the distortion as part of the environment) while some will be delusional. Second, once the memory component is trained and the agent shows delusional behaviour, it can never get out of this however many times we train the policy with different initialisations. On the other hand, in the car racing task, this issue can be resolved because the reward system is only governed by a regional area surrounding the car; therefore, if we do not distort this part and train R and M separately, we can prevent agents from falling into this type of delusion. In our collective setting, this issue is particularly problematic because, for delusional agents, the degree of agents' bias do not directly transfer to their performances.

4. RELATED WORK:

Machine learning (ML) and RL researchers propose internal models of the agents that is analogous to humans' mental models (see the work of Hamrick (2019) for the review), and the concept of learning a policy solely inside an agent's simulation using recurrent structures was proposed by Schmidhuber in 1990s (Schmidhuber, 1970; 1991; 1990) and recently been reviewed in the name of learning to think (Schmidhuber, 2015; Lake et al., 2017) . Also, Racanière et al. (2017) develop a neural architecture that leverage internalisations of the world on top of the real-environment interactions, and Ha & Schmidhuber (2018a; b) propose world models that enable full operationalisation inside simulation. Additionally, the idea of internalising the world is adopted for meta-control and to optimise and computation based on task difficulty and compexity (Hamrick et al., 2017; Pascanu et al., 2017) . Finally, there is a recent approach in model-based RL that further develops and applies the idea of simulation or imagination to achieve better results. Nair et al. (2018) and Hafner et al. (2019) make use of agents' internal models to pre-determine the long-term outcomes (e.g., imagined outcomes) and develops a policy based on such outcomes. In this research, the environments that we experiment on has state spaces that are accommodable for the internal models to capture with random policies. Orthogonal to our research, Hafner et al. (2019) advances the idea of training a policy inside a simulation by proposing an end-to-end framework that incorporates simulations in environments that cannot be fully explored and described by random policies, and by doing so, balances out the optimizations of the exploration policies and the model learning while minimizing the number of interactions with the environment.

Close to our research in terms of conceptual proximity and motivation is a study by Bard et al. (2019) , In their work, authors focus on a board game called Hanabi in which agents operate in a cooperative fashion to solve a task. Similar to our work, agents in the Hanabi game only has access to imperfect information, and their information can be complemented only by interacting with other agents. Here, the most critical diverging point is that the Hanabi challenge seeks an optimal policy for each individual, which ideally mimics human's sophisticated behaviours such as the theory of mind (Premack & Woodruff, 1978) , and the authors do so by interacting with the actual environment.

On the other hand, we focus on designing a specific model structure that can be solely operationalised in multiple agents' simulations and deriving a collective policy that gives better performance than individual ones. On a related note, Battaglia et al. (2018) proposes a general framework that discovers structural, relational, and compositional representations of an arbitrary system that builds upon graph networks and complements each entity's inductive biases, which can be applied to our setting with more entities and with relational information.

5. DISCUSSIONS:

We postulate a scenario in which each agent's internalisation of the world is biased, resulting in imperfect evidence collected at each timestep. Our goal is to simultaneously superimpose agents' biased representations portrayed by their internal models to learn a collective policy so that agents' defective simulations can be complemented by others. In our experiment, we used the CARRACING platform with a maximum of four agents and showed that the collective policy with integrated simulations give better performance for the car driving task. However, a few outstanding questions remain that need further exploration to better comprehend the overall process of learning a collective policy inside simulation and make it more effective and scalable. The first direction is to find an optimal group composition that maximises the score of the collective world model when the number of agents is fixed. Coordinating an ideal group is a non-trivial task because the collective performance depends not only on each individual's ability but also on the degree of each agent's feature complementing one another (Kim et al., 2016) and the needs for optimal coordination among agents will become more salient in massive settings (Suarez et al., 2019) . In our task, we tried to investigate the relationship between the number of total pixels covered by the group of agents and the performance but were unsuccessful at finding any significant correlations. The second approach is improving the model architecture to overcome the performance degrade, especially in the aggregator component. We conjecture that our current setting of naively superimposing internal representations might potentially "dilute" useful bits of information as the number of agents grows. Recently, simple aggregator architectures, such as summation or mean functions have been implemented in previous research (Ali et al., 2018; Garnelo et al., 2018a; b) and extended by adopting e.g., attention mechanisms (Bahdanau et al., 2015; Xu et al., 2015; Wang et al., 2018; Vaswani et al., 2017) as by Kim et al. (Kim et al., 2019) or tower representation architectures by Eslami et al. (Ali et al., 2018) . Finally, we tried finding the complete observation "frame-wise" (rather than directly finding the optimal collective policy), with agents' scores and their encodings at every time step and using Bayesian optimisation (Mockus, 2012; Shahriari et al., 2015; Močkus, 1975) with neural processes (Garnelo et al., 2018a; b; Kim et al., 2019) as the surrogate function, but was unsuccessful because of time complexity.

controller C that's also trained independently to all the individual V components and also to the individual C components.

C TRAINING SPECIFICATIONS:

We used the same setting as in the original world models paper for training the individual components V and M.

C.1 CONTROLLER COMPONENT C:

For C, we used CMA-ES for training. We concatenated 16 rollouts to define one episode, and each virtual rollout is hosted by an agent in the collective system. We made sure that the number of rollouts in the episode can be exactly divided by the total number of agents in collective world model with remainder of 0, so that all agents can be hosts the same number of times. Also, with this setting, we manage to keep the training iteration of collective policy exactly the same with the individual policies. For example, if there are 8 agents in total, each can host simulations 2 times in a single episode; if there are 4 agents in total, each can host simulations 4 times in a single episode, and so forth.

We set the population size to 32, and each population is assigned with a single CPU. We set the initial sigma value of CMA to 0.1, and set all solution values to 0 for initialisation. Finally, for every training, we used random seed numbers.

C.2 RULE COMPONENT R:

For the rule component R, we used RNN with 2 hidden layers, each layer with the same size of 256. We used Adam as optimizer, with initial learning rate of 0.001, and used mean squared error for the loss function. We also used dropout for all hidden layers with the value 0.1, gradient clipping value of 0.25, batch size of 100, and training epochs of 500. For inputs, for every time step t, we used generated encodings of the current as well as the 10 previously generated encodings, so in total of 352-dimensional input space was used, and scalar value of the instant reward was returned.

Here, we note that in our setting, the model complexity of R is generously set. Since training of this component was not the bottleneck of our computation, we did not try to shrink the model architecture, as only focused on the model returning low training/testing errors. Therefore, the current model size might be overly large, and we believe that this size can be drastically reduced without significant loss in accuracy.

C.3 TRANSLATOR COMPONENT T:

For the trainslator component T, we used FNN with 4 hidden layers, each layer with sizes 1024, 512, 256, and 128. We used Adam as optimizer, with initial learning rate of 0.001, and used mean squared error for the loss function. We also used dropout for all hidden layers with the value 0.1, batch size of 10,000, and training epochs of 250. Inputs and outputs are both 32-dimensions, which corresponds to the size of the encoding z.

As in the rule component R, we believe that model size of T can also be drastically reduced.

C.4 COMPUTING INFRASTRUCTURE:

Our model is programmed using python Tensorflow.

Training C is done in multi-CPU environment, as CMA-ES can be effectively implemented in parallel computations. We used 32 CPU cores for training both collective and individual C, where each core is assigned with a single population in CMA-ES.

For training V (VAE), M (MD-RNN), T(FNN), R(RNN), we used a single GPU GTX-1080 Ti for training each component.

:

http://s000.tinyupload.com/?file_id=54935721167326296555

By design, our algorithm can operate on different kinds of biases, e.g., temporal distortions, although we did not empirically test the performance under such conditions.

To our knowledge, there is no previous research that trained policies for a single task in both real and virtual game environments; in the original world models paper, the CARRACING experiment is done in the actual environment, and the VIZDOOM experiment is done inside simulation, but not the other way around.

Figures and Tables:

Figure1: Flow diagrams of policy learning of (a) an individual policy while interacting with an environment, (b) individual policy learning inside internal simulation, (c) collective policy learning involving multiple agents' internal models. Note that for (b) and (c), there is no interaction with the environment. For collective world model, we omit the decoding part of V for clear presentation.

Collective policy learning algorithm Input: N trained memory components components M N trained rule components R N 2 trained translator components T Initialize parameters for the collective policy function; while not converged do Set RewardT otal ← 0; for hosting agent a = 1, . . . , N do Set the hosting agent H ← a;

Figure2: (a): Relationship between the temperature parameter τ of MD-RNN in internal simulation and the score received from the real environment. τ governs the stochasticitiy of the virtual environment generation, and there is an optimal value, 1.7 that gives the best score. (b): As τ increases, the simulated environment becomes more unpredictable and we accordingly need more training epochs.

Figure 2 describes how the temperature parameter τ of MD-RNN (M component) affect the policy learning inside simulation. As τ increases, the virtual environment

Figure 3: (a): Original observation (top) with decodings of the biased representations of different internal models (occlusion and magnification). (b) -(c): Virtual scores when the policy is trained inside its own internal simulation (left bars), when using the policy that is trained in other's internal simulation (center bars), and when the collective policy is used (right bars). If other's policy is used in the virtual environment generated by oneself, the virtual score decreases catastrophically, whereas the collective policy gives reasonable scores inside all simulations. (d): "Joint" is the collective policy without T and A components; when the collective policy is trained without co-observation, it performs worse than individual policies.

Figure 4: (a): Biased representation (occlusion) in VIZDOOM environment. (b) -(e): 4 different agents' internal models with the bias setting given in (a). Although trained under exactly the same setting, internal models (b) and (d) learns reasonable policies inside simulation, while (c) and (d) became delusional. For all four internal models, their virtual scores are similar (blue bar), but the actual score significantly varies between normal and delusional agents. Also, once an agent's model is trained, delusional agents do not become normal during the policy training phase and vice versa.

Comparison between training the policy inside real environment and inside an agent's internal simulation. ± is followed by one standard error.

Table content: <table><tr><td/><td colspan="3"># Env. interaction # Epoch Model tr. Policy lr.</td><td colspan="2">Time / Pop. size epoch (min.)</td><td>Mean</td><td>Median</td></tr><tr><td>Env.</td><td>10 K</td><td>1.84 M</td><td>1,800</td><td>42</td><td colspan="2">64 900.0 ± 5.2</td><td>911.9</td></tr><tr><td>Sim.</td><td>10 K</td><td>0</td><td>225</td><td>0.8</td><td colspan="2">32 851.9 ± 9.8</td><td>875.1</td></tr></table>

Results of the collective policy learning. When multiple agents' internal models provide complementary evidences from their biased representations, the collective policy achieves higher real-environment scores than that of the best performing individual internal model.

Table content: <table><tr><td>Bias type</td><td colspan="3"># M Overlap (%) Cov. (%)</td><td>Worst</td><td>Indiv. policy Mean</td><td>Best</td><td>Collective Policy</td></tr><tr><td>Occ.</td><td>2</td><td>0</td><td colspan="4">100 615.2 ± 22.7 618.4 621.6 ± 7.9 750.8 ± 12.3</td></tr><tr><td>Mag.</td><td>2</td><td>12.5</td><td colspan="4">37.5 494.3 ± 16.5 558.7 623.0 ± 22.9 733.8 ± 19.9</td></tr><tr><td>Mag.</td><td>3</td><td>18.8</td><td colspan="4">37.5 494.3 ± 16.5 602.1 688.7 ± 17.4 744.6 ± 10.2</td></tr><tr><td>O.,M.</td><td>4</td><td>100</td><td colspan="4">100 494.3 ± 16.5 581.0 623.0 ± 22.9 742.1 ± 08.8</td></tr><tr><td>Occ.</td><td>2</td><td>100</td><td colspan="4">62.5 468.4 ± 21.9 541.8 615.2 ± 22.7 629.1 ± 15.5</td></tr><tr><td>Mag.</td><td>2</td><td>100</td><td colspan="4">39.1 571.9 ± 21.4 678.3 784.7 ± 9.3 769.8 ± 16.3</td></tr></table>

Peer review (iclr2020):
This paper proposes a new approach to conducting RL: it proposes to first train a collection of individual agents independently, and then exploit them to learn one collective policy. Each agent has an observation space that is differently impaired, such as by blocking out some specific set of pixels in its image observations. Each such agent learns its own dynamics and reward models based on its own observations, and goes on to train a policy based on simulated rollouts using these learned models. In the second phase, a collective policy is trained as follows: every training episode happens inside a different *host* agent*s simulation, and the collective policy is thus effectively trained on this mixture of variously inaccurate simulations. Since different individuals have different observations, the collective policy receives an input observation that is the aggregate of all agents* individual observations. Throughout, this paper builds on top of the *world models* approach proposed in Ha & Schmidhuber 2018. Most sections of this paper are well-written and the high-level ideas are novel and interesting. My main issues with the paper are listed below, in rough order of priority: (1) I find the experimental evaluations short of convincing. * Baselines: The proposed collective policy is evaluated against the individual policies of agents in its population, which is a very weak baseline since the individual agents are artificially impaired. It is also evaluated against an upper bound: a policy trained directly in the real world, and shown to be only slightly weaker in performance. I would suggest the following additional baseline: - individual agents without observation impairments, trained in simulation (essentially the world model paper this approach builds on). [*world models*] - the average of the individual policies [*policy ensemble**]. - a policy trained with a model that is a composite average of the individual world models [*model ensemble*]. This is the in the spirit of prior work on model ensembling, such as Chua et al 2018, * ... handful of trials.* * Environments: The results presented so far indicate the proposed approach works on one environment (CarRacing) and does not on another (VizDoom). I would like to see experiments on more environments to figure out whether CarRacing is the exception or the rule. * choice of observations for individual agents: in a general setting, how are these to be engineered? (2) It does not sufficiently acknowledge or clarify connections to other work in the field, such as on model bootstrapping in model ensembles (e.g. Chua et al 2018, *... handful of trials.*) or evaluate against them. In particular, I think the proposed approach is a novel way to achieve model bootstrapping, plus some additional tweaks: rather than train different models on different subsets of training experience, it trains different models on different feature views of the same training episodes. (3) It does not sufficiently motivate its setting: when is it true in realistic settings that it would make sense for different agents in an environment to be artificially impaired in different ways? Experiments are only conducted in contrived settings where portions of the environment are deliberately removed from the observation for different agents. I also have a somewhat related suggestion: perhaps future versions of the paper might focus on using different modalities (such as RGB images and depth) for the different agents. (4) From my understanding, there seems to be a tradeoff between how much overlap there exists between different agents* observations and how translatable different agents* views are to each other. In other words, the method requires the translator T to take one view of an agent and transform it into (a feature representation of) the view of another agent. This seems ill-defined in general settings. (5) Related to the above, how is the aggregated vector [z_{ct}, h_{ct}] generated for the collective policy when it is eventually evaluated in the real world? Are the features computed separately by each agent before aggregation, or is one agent selected and the translator T used again? If the former, then does this induce a domain shift between training time when the policy is trained on predicted features from the translator, and testing time when the policy sees true features corresponding to different agents?