Learning to Control Self-Assembling Morphologies

Generalization via Modularity

DeepakPathak*ChrisLu*

TrevorDarrell* equal contribution

NeurIPS 2019

Phillip
IsolaAlyosha
Efros

How do we train a robot?






Multiple tasks
Expert demonstrations
Rewards, labels
…






Self-supervision
Curious exploration
Learning “common sense”
…





Multiple tasks
Expert demonstrations
Rewards, labels
…

...

...

… even earlier?

Single to Multicellular

Single to Multicellularcompetition Æ collaboration

Single to Multicellularcompetition Æ collaboration

shared objectiveCompositionality has been useful in language …

[Andreas et. al. 2016]

How to implement compositionality in hardware?

Modular Co-evolution of Control and Morphology

Modular Co-evolution of Control and Morphology

Cylindrical Limb

Modular Co-evolution of Control and Morphology

Cylindrical Limb

Configurable Motor JointModular Co-evolution of Control and Morphology

Modular Co-evolution of Control and Morphology

Modular Co-evolution of Control and Morphology

Potential
Magnetic JointModular Co-evolution of Control and Morphology

Potential
Magnetic JointModular Co-evolution of Control and Morphology

Acts as single agent upon joining
Rewards are shared!

Potential
Magnetic JointModular Co-evolution of Control and Morphology

Acts as single agent upon joining
Rewards are shared!

¾ Input = Local Sensory State
Potential
Magnetic Joint¾ Output = Torques, Link, Unlink

Modular Co-evolution of Control and Morphology

Acts as single agent upon joining
Rewards are shared!

¾ Input = Local Sensory State
Potential
Magnetic Joint¾ Output = Torques, Link, Unlink

Consider the task of “standing up” …

How to learn compositional controllers?

Idea: Shared policy network across limbs
Bottom-up
message out

Top-down
message inBottom-up
functionTop-downfunction

Node

Node

Node
Node

Node

Node

Node

Nodeinput

Node
Node

Node
Node

Node

Bottom-up Top-down
message in message outAction

Idea: Shared policy network across limbs
Bottom-up
message out

Top-down
message inoutput

Node

Node

Node
Node

Node

Node

Node

Node
Node

Node
Node

Node

Node
input

sharedBottom-up
Top-downfunctionpolicy function𝜋𝜃
Bottom-up

Top-down

message in message out
input

ActionHow to adapt when morphology changes?

How to adapt when morphology changes?

Network as reusable LEGO Blocks

Network as reusable LEGO Blocks

outputsharedpolicy𝜋𝜃
input

Network as reusable LEGO Blocks

outputmessage
outputsharedpolicy𝜋𝜃
inputmessageinput

Network as reusable LEGO Blocks

outputsharedpolicy𝜋𝜃
inputmessage
outputsame
dimension
messageinputNetwork as reusable LEGO Blocks

outputmessage
outputsharedpolicy𝜋𝜃
inputmessageinputNetwork as reusable LEGO Blocks

outputmessage
outputsharedpolicy𝜋𝜃
inputmessageinputNetwork as reusable LEGO Blocks𝜋𝜃𝜋𝜃outputmessage
outputsharedpolicy𝜋𝜃
inputmessage
input𝜋𝜃Network as reusable LEGO Blocks𝜋𝜃𝜋𝜃

outputmessageoutput

sharedpolicy𝜋𝜃
inputmessage
input𝜋𝜃Network as reusable LEGO Blocks𝜋𝜃𝜋𝜃

outputmessageoutput

sharedpolicy𝜋𝜃
inputmessage
input𝜋𝜃Network as reusable LEGO Blocks𝜋𝜃𝜋𝜃

outputmessage
outputsharedpolicycut

𝜋𝜃input𝜋𝜃messageinputNetwork as reusable LEGO Blocks𝜋𝜃𝜋𝜃

outputmessageoutput𝜋𝜃sharedpolicycut and paste𝜋𝜃
input𝜋𝜃message
input𝜋𝜃𝜋𝜃Network as reusable LEGO Blocks𝜋𝜃𝜋𝜃

outputmessage
outputsharedpolicyadaptation byconditioning𝜋𝜃input𝜋𝜃cut and paste𝜋𝜃messageinput𝜋𝜃𝜋𝜃Dynamic Graph Networks

BTW, basically curriculum learning but in hardware

How well does it generalize?

...

...

a bit crazy… is it even possible in real world?

Self-Assembling Robots in the Real World[Mark Yim’s Lab at UPenn]

Also: [Modular Snake Robot – Howie Choset’s Lab at CMU]

[Daniela Rus's Lab at MIT]

code & data at
https://people.eecs.berkeley.edu/~pathak/

Poster # 197
Today (Tues)!!

Thank You!