Multi-Source Policy Aggregation in
Heterogeneous and Private Environmental Dynamics
Mohammadamin Barekatain∗
Technical University of Munich
Munich, Germany

Ryo Yonetani
OMRON SINIC X
Tokyo, Japan

Masashi Hamaya
OMRON SINIC X
Tokyo, Japanm.barekatain@tum.deryo.yonetani@sinicx.com

masashi.hamaya@sinicx.com

1. Introduction
We envision a future scenario where robotic agentsworking in diverse and private environments help a newagent in an unknown environment to learn its policy efficiently. For instance, imagine various types of pick-andplace robotic agents working in a factory. While the agentsare involved in the same task, dynamics of the environment
in which the task is performed is different based on eachrobot’s kinematics (e.g., degree of freedom, link length, andjoint orientations) and dynamics (e.g., joint damping, armature, and friction) [1]. Moreover, no prior knowledge
about the dynamics of environments as well as the specification of agents policies can be available for a new agent
due to the confidentiality of products and processes in thefactory. Other relevant scenarios include autonomous vehicles on private land and home assistants interacting withpeople privately.
The problem setting shown above makes it hard to adopt
many existing approaches for efficient learning of a target
agent’s policy. For instance, meta-learning approaches typically require an agent to be trained on a diverse task distribution [4], which is not possible here due to the privacy ofenvironments. Also, existing transfer learning approaches
that focus on the transfer of policies between dynamics, require prior information about the environments [1] or policy configuration (e.g., actor network weights and agent’s
value function [2]) which are both unavailable.
In such scenarios, we argue that the target agent can get
information from other private agents through their policies (hereafter source policies) that act as a black-box function mapping states to actions. Specifically, we propose a
new sample efficient approach named MULTI-source POLicy AggRegation (MULTIPOLAR). Much like a multipolar
neuron that can integrate information coming from otherneurons, our MULTIPOLAR aggregates the actions produced by the source policies to serve a robust baseline action. It also learns an additional policy to predict a ‘residual’∗ Work done as an intern at OMRON SINIC Xstfθb

θwµ1µ... µK

1KPKµi (st )i=1 θwi+at ∼ πθ (st )Figure 1: The proposed MULTIPOLAR policy network.

around the baseline actions to mitigate the unseen dynamics
of the target agent’s environment. As a result, MULTIPOLAR achieves training sample efficiency since the aggregation of source actions provides a strong inductive bias.
As a preliminary experiment, we evaluate MULTIPOLAR with two public simulated environments with continuous and discrete action spaces: Roboschool Hopper1 andOpenAI Acrobot2 . Our experimental results demonstratethat MULTIPOLAR allows a new agent to learn its policy significantly faster on average compared to when it istrained from scratch.

2. Proposed Method
Preliminaries We formulate our policy aggregation problem under the standard Reinforcement Learning (RL)
framework, where an agent interacts with its environment
modeled by a Markov Decision Process (MDP). An MDP
is represented by a 6-tuple (ρ0 , γ, S, A, R, T ) where ρ0 isthe initial state distribution and γ ∈ (0, 1] is the discount1 https://github.com/openai/roboschool
2 https://gym.openai.com/envs/Acrobot-v1

Figure 2: Example learning curves for Acrobot and Hopper averaged over trials. The shaded area represents 1 standard error.
factor. At each timestep t, given the current state st ∈ S,
the agent executes an action at ∈ A based on its policyπθ (at |st ) which is parameterized by θ. The environment
returns a reward rt = R(st , at ) and transitions to the next
state st0 based on the environment state transition distribution T (st0 |st , at ). In this framework, RL aims to maximize
the expected return with respect to the policy parameters θ.
In this work, we use ε to denote a particular environment
such as Hopper which the agent is interacting with and εi
to denote different unknown environmental dynamics forthat particular environment ε. Specifically, ρ0 , γ, S, A, R
are the same for εi s but the state transition distribution
Ti (st0 |st , at ) is different.
Problem statement Given an environment ε and a library (set) of K deterministic source polices L ={µ1 , µ2 , ..., µK }, where each µi is acquired from εi ,quickly learn an optimal policy πθ for εtarget by exploiting knowledge from L. Each source policy µi can be parameterized (e.g. learned from interacting with an environment) or non-parameterized (e.g., heuristically designed byhumans). Either way, we assume no prior knowledge about
the representations of µi (e.g. their network architectures)
as well as their environmental dynamics and we only haveaccess to their predicted deterministic actions ai,t = µi (st ).
Moreover, µi ’s are not necessarily optimal on the εi they
were acquired from.
Policy aggregation Our proposed MULTIPOLAR policy
network πθ , illustrated in Figure 1, aims at improving sample efficiency of training an agent on a particular environment with unknown dynamics target by leveraging knowledge from L. The main idea is to learn how useful each action of each source policy is for target by learning a stateindependent scale vector θwi for each source policy µi . To
ensure the optimality of MULTIPOLAR policy, a residual
policy fθb is learned around the average of scaled sourceactions. Specifically, the mean actions at timestep t is:uθ (st ) = fθb (st ) +K
1 X

K i=1

(for discrete action space environments) and final actions
are sampled from the corresponding distribution. MULTIPOLAR can be trained with any model-free policy gradientmethods given that we can take the gradient of policy performance with respect to θ.

3. Experiments
We aim to empirically demonstrate the importance of aggregating source policies for achieving training sample efficiency. We evaluate the effectiveness of MULTIPOLAR
with four source policies (K = 4) on Hopper with continuous action space and Acrobot with discrete action space.
In both environments we compare MULTIPOLAR policy
to the standard MultiLayer Perceptron (MLP) policy network typically used in RL. To have a fair comparison, bothbaseline MLP and MULTIPOLAR are trained with Proximal Policy Optimization algorithm [3] with the same hyperparameters and same network size for MLP and fθb .
Results In both Hopper and Acrobot experiments, we designed 100 target environments by randomly sampling theenvironmental dynamics parameters such as links lengthand mass. For each target environment, we trained MULTIPOLAR three times with different sets of source policies acquired from random i ’s. Each training was done
three times with different random seeds. Figure 2 shows thelearning curves of MULTIPOLAR policy (averaged over 3
choices of source policies × 3 random seeds = 9 trials) andthat of the baseline policy (averaged over the same 3 randomseeds) for different target ’s. It clearly shows that MULTIPOLAR outperforms the baseline in terms of sample efficiency and sometimes the final episodic reward. It is also
noteworthy that MULTIPOLAR learning curves are significantly more consistent across different trials, given that theirstandard errors are much smaller than the baseline.

References[1] T. Chen, A. Murali, and A. Gupta. Hardware Conditioned Policies for MultiRobot Transfer Learning. In NeurIPS, 2018. 1

θwiµi (st )(1)where θ = {θb , θw1 , . . . , θwK }. uθ (st ) is the estimatedmean of a multivariate Gaussian distribution (for continuous action space environments) or a categorical distribution[2] F. L. Da Silva and A. H. R. Costa. A Survey on Transfer Learning for Multiagent
Reinforcement Learning Systems. J. Artif. Int. Res., 64(1):645–703, 2019. 1[3] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal
Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017. 2[4] J. Vanschoren. Meta-Learning: A Survey. arXiv preprint arXiv:1810.03548,
2018. 1