Learning to Control Self-Assembling Morphologies

A Study of Generalization via Modularity
Software starts simple, but hardware remains fixed & complex!

(* equal contribution)

Generalization via Modularity

Dynamic Graph Networks (DGN)

DG1 (w/ Psgs)
DG1 (nR Psgs)
16000

0eDn 5ewDrGs

Idea 1: Separate policy for each limb
with shared params

0eDn 5ewDrGs

14000
12000

How to adapt when hardware changes?
10000
8000
6000
4000
2000
0250

500

750

1000

1250

1500

1750

1uPber Rf trDining steSs(a) Standing Up§ Self-supervision§ Curious exploration§ Learning “common sense”§ Multiple tasks§ Expert demos§ Rewards, labels

Bottom-up
message out

Node......

Node

Node

Node

Nodeinput

Node
Node

sharedobjective

Nodesingle à multicellular(i.e. competition à collaboration)

Node

Compositionality is useful in language (Andreaset.al. 2016). However, parser is fixed à tractable.
How to implement compositionality in hardware?

Modular Co-evolution of Control and Morphology
CylinderConfigurable
Motor Joint

Potential
Magnetic Joint

Nodesharedpolicy𝜋"
Bottom-upfunctionsharedpolicy𝜋"
input

same
dimension

ActionIteration tAdapt by Finetuning?

Ø Output = Torques, Link, Unlink

Acts as single agent upon joining.

Rewards are shared!

10000
8000
6000

12000
10000
8000
6000

4000

4000

2000

2000

12000
10000
8000
6000
4000

8

6

6

4

20

20

2000
−20

Message-passing
helps!

42500

500

200750400
10001250600 1500

800
1750

1000

1uPberRf trDining
1uPberRf trDining
steSs steSs(b)(a) Standing Up w/ Wind0

−20 200

500
400

1000
600

1500
800

2000
1000

1uPberRf trDining
1uPberRf trDining
steSs steSs(c) Locomotion(b) Standing
Up w/ Wind

2500

0500

1000

1500

2000

2500

1uPber Rf trDining steSs(c) Locomotion

Curriculum learning
but in hardware.

Iteration t+1(b) No message passing: Note
some environments
tasks,
the context
from theorothernodes(b)that
Nofor
messagepassing: Noteor
thatfor someenvironments
tasks,
the context from the other nodes
might not be a necessary requirementcontrol.
In such for
scenarios,
messagemight notfor
be effective
a necessaryrequirement
effectivecontrol.passing
In suchmight
scenarios, message passing might
create extra overhead for training
DGN.overhead
Importantly,
even withno messages,
DGNevenstillwithallowscreatea extra
for training
a DGN.
Importantly,
no formessages, DGN still allows forcoordination between limbs.coordination
This is similar
to a typicalmulti-agent
setup[28], where
betweenlimbs.cooperative
This is similar
to a typicalcooperative
multi-agent setup [28], where
each limb makes its own decisions
in response
the previousof thetootheragents. However,each limb
makes itstoown
decisionsactions
in response
the previous
actions of the other agents. However,
our setup differs in that our agents
maydiffers
physicallyratherthanjust coordinating
behavior.
our setupin thatjoin
ourup,
agentsmay
physicallyjoin up, rather
than just coordinating behavior.

Too slow as shape changes every iteration!

Message Passing:Children pass messages toparents for communication.

message
input

Primitive Agents: “limbs”
Ø Input = Local Sensory State

12000

8Top-downfunction

Idea 2: Neural Networks as reusable LEGO Blocks
output

14000

14000

Ci
iii
network information about its
childrennodes. Weabout
redefine⇡✓i as ⇡✓inodes.
: [sit , m! [ait , mwherenetwork
information
its children
We⇡✓iit ]as⇡✓i : [sit , mCt ]redefinet ] ! [at , mt ] whereCi
Cii
mit is the output message ofm
policy
that
goes
into
theparentlimb
andmis theaggregatedinput
istheoutput
messageofpolicy
that
goes
into
theparentlimbandmt
t is the aggregated input
tPP
Ci
Ci
cc
messages from all the children
nodes, i.e,
mtall =. If i has(i.e,i mroot),messagesfromthe children
i.e,nomtchildren= c2C
has no children (i.e, root), a vector
c2Ci mtnodes,
t . Ifai vector
iiof zeros is passed in mC
passedinrecursively
until thenode.
An alternative
wayroot
is node. An alternative way isof zeros areis passedmC
areroot
passedrecursively
until thet . Messages
t . Messages
to start from the root node andrecursively
pass
until
theandmessagesreachpass
the until
leaf nodes.
to startfrom theroot
noderecursively
the messages reach the leaf nodes.

4

message
output

16000

14000

control and morphology for threetasks:andlearning
to standfor
upthree(left),tasks:
standing
up intothe
presenceof wind(center)
controlmorphology
learning
standup (left),standingup in the presence of wind (center)and locomotion in bumpy terrainand(right).
These policies
to novel
scenarios
asgeneralize
shown in thetables.scenarios as shown in the tables.
locomotion
in bumpygeneralize
terrain (right).
Thesepolicies
to novel

Bottom-up Top-down
message in message out

input

16000

0RnRlithic PRlicy, DynDPic GrDSh
0RnRlithic PRlicy, 6tDtic GrDSh

Figure 4: Training self-assembling
agents:
We show
the performanceof different
methods
for joint trainingof methods for joint training ofFigure4: Training
self-assembling
agents:
We show
the performanceof different

Node

Node16000

DG1 (w/ Psgs)
DG1 (nR Psgs)

Top-down
message inoutput

Node

DG1(w/ Psgs)0RnRlithic
0RnRlithic
DynDPic
DG1 (w/Psgs)
PRlicy, PRlicy,
DynDPic
GrDSh GrDSh
DG1(nR Psgs)0RnRlithic
0RnRlithic
6tDtic GrDSh
DG1 (nRPsgs)
PRlicy, PRlicy,
6tDtic GrDSh

DG1 (w/ Psgs)
PRlicy, GrDSh
DynDPic GrDSh
DG1 (w/ Psgs)
0RnRlithic0RnRlithic
PRlicy, DynDPic
DG1 (nR Psgs)
PRlicy,6tDtic GrDSh
DG1 (nR Psgs)
0RnRlithic0RnRlithicPRlicy, 6tDtic
GrDSh

0RnRlithic PRlicy, DynDPic GrDSh
0RnRlithic PRlicy, 6tDtic GrDSh

0eDn 5ewDrGs

UC Berkeley

How to train a robot to act?

Alexei A. Efros

0eDn 5ewDrGs

Trevor Darrell Phillip Isola

0eDn RewDrGs

Chris Lu*

0eDn RewDrGs

Deepak Pathak*

Code Released!
https:// pathak22.github.io/modular-assemblies/

Dynamic + Graph è Dynamic Graph Nets

Experiments

4Experiments

We test the co-evolution of morphology
and control across
two primary
whereself-assemblingWe test the co-evolutionof morphology
andtasks
controlacrosstwo primary tasks where self-assembling
agents learn to: (a) stand up,agentsand (b)perform
Limbs
start each
episode disconnectedlearn
to: (a)locomotion.
stand up, and(b) perform
locomotion.
Limbs start each episode disconnected
and located just above the groundplane justat random
locations,shownin Figurelocations,
3. In theas
absenceand locatedabove thegroundasplaneat randomshown in Figure 3. In the absenceof an edge, input messages aresetedge,
to 0 andthemessagesoutput ones
Actionspace
is arecontinuousof aninput
are areset ignored.
to 0 and theoutput
ones
ignored. Action space is continuous
raw torque values. Across all
thetorquetasks, values.
the numberof limbs
training
is kept fixed
to 6.atWetake is kept fixed to 6. We takeraw
Acrossall theattasks,
the numberof limbs
training
the model from each time stepevaluate
it on time
50 episoderuns
to plotitmean
std-deviation
the andmodel
from eachstep andevaluate
on 50andepisoderuns to plot mean and std-deviation
confidence interval in training
curves. interval
At test, wereport themean At
rewardacross50 episodes
confidencein training
curves.
test, wereportthe mean ofreward across 50 episodes of1200 environment steps. The1200
mainenvironment
focus of oursteps.
investigation
is focus
to evaluate
the emergedismodular
The mainof ourifinvestigation
to evaluate if the emerged modular
controller generalizes to novel
morphologies
and environments.
Video is on
project website
andis on the project website andcontroller
generalizes
to novel morphologies
andtheenvironments.
Video
implementation details are inimplementation
Section 1.1 of thesupplementary.
details
are in Section 1.1 of the supplementary.
Baselines We further compare
how wellthesedynamic
morphologies
performin comparison
to a perform in comparison to aBaselines
Wefurthercomparehow well these
dynamic
morphologies
learned monolithic policy forlearned
both dynamic
and fixed
In particular,
compare to aIn(a)particular, we compare to a (a)monolithic
policymorphologies.
for both dynamic
and fixedwe
morphologies.
Monolithic Policy, Dynamic Monolithic
Graph: Baseline
agentsare still
dynamicand agentscan self-assemble,
Policy,where
Dynamic
Graph:
Baseline
whereare still dynamic and can self-assemble,
but their controller is represented
by acontroller
single monolithic
policyby
thattakes asinput the policy
combinedbut theiris represented
a singlemonolithic
that statetakes as input the combined stateof all agents and outputs actions
foragents
each of(b) actions
Monolithic
Policy,Similar Policy,
single Fixed Graph: Similar singleof allandthem.
outputs
for eachof Fixed
them. Graph:(b) Monolithic
monolithic policy as the previous
baseline,butas
thethemorphology
is hand-designed
constructedisfrom
monolithic
policyprevious baseline,
but the morphology
hand-designed constructed fromthe limbs and kept fixed and thestatic
duringandandtest.
Thisduring
is analogous
a standard
robotics
limbs
andtraining
kept fixedstatictrainingtoand
test. Thisis analogous to a standard robotics“vanilla RL” setup in which “vanillamorphology
predefined,
then a policy
is learned tocontrol
We is learned to control it. We
RL” issetup
in whichandmorphology
is predefined,
andthen ait.policy
chose the fixed morphology to
be a thestraightof 6-limbs
inaallthe experiments.
This linear-chainchose
fixedchain
morphology
to bestraight
chain of 6-limbs
in all the experiments. This linear-chain
may be optimal for standing may
as tallbeasoptimal
possible,
it is notasnecessarily
optimalto stand;optimal for learning to stand;forbutstandingtall as possible,
butforit islearning
not necessarily
the same would hold for locomotion.
wefor
confirmedthat bothstanding
and locomotion
tasksstanding and locomotion tasks
the sameHowever,would holdlocomotion.
However,
we confirmedthat bothare solvable with linear-chainare
morphology(shownin Figure
3 and video(shownon the inprojectsolvable withlinear-chain
morphology
Figurewebsite).
3 and video on the project website).
Although the monolithic policy
is moreof all limbs),
isAlthoughtheexpressive
monolithic(complete
policy is statemoreinformation
expressive (complete
stateit informationof all limbs), it isalso harder to train as we increase
the number
becauseobservation
andbecause
action spaces
also harder
to trainofaslimbs,
we increase
thethenumberof limbs,
the observation and action spaces
increase in dimensionality. Indeed,
thiswhat we find inIndeed,
Figurethis
1 ofissupplementary:monolithic
increasein isdimensionality.
what we find inthe
Figure1 of supplementary: the monolithic
policy can perform well on up
to threelimbs butwell
doesonnot
thelimbs
optimum
on foursix limbs.
policycan performupreachto threebut does
not toreachthe optimum on four to six limbs.
In contrast, the DGN limb policy(sharedthe
between
all limbs)
a fixedsize observation
In contrast,
DGN limb
policyhas(sharedbetween
all limbs) andhas action
a fixed size observation and action
space, independent of the numberlimbs underofcontrol.
space,ofindependent
the number of limbs under control.

Self-Assembling Robots
in the Real World?

6 Lab at UPenn]6[Mark Yim’s[Daniela Rus's Lab at MIT]

[Modular Snake Robot – Howie Choset’s Lab at CMU]