Toward Natural Language Supervision
AAAI 2023

Jacob Andreasnlplingo.csail.mit.edu

How do we learn?
fromobservations

fromexploration

Overall structure of the SAYCam dataset. (b) The classes and the number of frames
in the labeled dataset. (c) Random images from the labeled dataset with the category
ted on the left. Note that the labeled data are a curated subset of the data from child S.

[Sullivan et al. 2020]

develop a novel self-supervised learning objective for learning high-level visual repreations from video data based on the principle of temporal invariance (Földiák, 1991;
kott and Sejnowski, 2002) and show that this objective yields better representations thane-of-the-art image-based and temporal contrastive self-supervised learning objectives on
SAYCam dataset.
m SAYCam, we curate a large, developmentally realistic dataset of labeled images foruating self-supervised models and analyze the learned visual representations.

[Legare 2012]

fromdemonstrations

fromlanguage[Buchsbaum et al. 2010][Morgan et al. 2015]

2

What do we learn from language?
facts

procedures

picture: a classroom

language!

picture: a cookbook

3

m

i

t

Language in human decision-making[Hermer-Vazquez, Spelke, Katznelson 1999]

4

m

i

t

Language in human decision-making

44%
31%
24%

[Hermer-Vazquez, Spelke, Katznelson 1999]

5

m

i

t

Language in human decision-making

8%

30%

50%

12%

[Hermer-Vazquez, Spelke, Katznelson 1999]

6

m

i

t

Language in human decision-making

0%

95%

5%

0%

[Hermer-Vazquez, Spelke, Katznelson 1999]

7

m

i

t

Language in human decision-making

6%

43%

38%

13%

[Hermer-Vazquez, Spelke, Katznelson 1999]

8

How do machines learn?
fromobservations

fromexploration

Overall structure of the SAYCam dataset. (b) The classes and the number of frames
in the labeled dataset. (c) Random images from the labeled dataset with the category
ted on the left. Note that the labeled data are a curated subset of the data from child S.

develop a novel self-supervised learning objective for learning high-level visual repreations from video data based on the principle of temporal invariance (Földiák, 1991;
kott and Sejnowski, 2002) and show that this objective yields better representations thane-of-the-art image-based and temporal contrastive self-supervised learning objectives on
SAYCam dataset.
m SAYCam, we curate a large, developmentally realistic dataset of labeled images foruating self-supervised models and analyze the learned visual representations.

[Legare 2012]

fromdemonstrations

fromlanguage[Buchsbaum et al. 2010]

???
9

m

i

t

Today’s talkLearning to act

Learning to program

cook aneggturn(90);pick(pan);goto(stove);place(pan)…

(f24 5 (λ(X) (get/set(λ (y) (f2 1(f41 5 y)))x)) z)

Learning to explain
dog faces& wheels

10Learning skills from demonstrations
and instructions

Pratyusha
Sharma+ Antonio Torralba[Skill Induction & Planning w/ Latent Language. ACL 2022.]

https://www.dailymail.co.uk/sciencetech/article-6591753

[Shridhar et al. 2020]

it

put a sliced tomato on the
kitchen counter

Can we train a xed model
to map goals → actions?
turn(left)

14
fi

m

Learning to act

it

put a sliced tomato on the
kitchen counter

Can we train a xed model
to map goals → actions?
turn(left)

0% success rate!

15
fi

m

Learning to act

m

i

t

Long horizon tasks
put a sliced tomato on the kitchen counter

16

m

i

Hierarchical policiestExplicitly decomposing tasks into subtasks makes this problem easier…

put the knife in the drawerfind(knife)turn(left)

forward(10)grasp(knife)turn(right)

open(box)[Hand-engineered hierarchies: Parr & Russell, 1998; Andre & Russell, 2002][Supervised training of sub-policies: Kearns & Singh 02, Kulkarni et al. 16]

fi[Fully unsupervised: Stolle & Precup 02, Fox & Krishnan et al. 16]

grasp(knife)

...
...

…but existing methods
require lots of domainspeci c engineering.

17

How can language help?

fi

m

i

t

A hierarchical policy with latent language
put the knife in the drawer
Language Model Plannerturn(left)

nd a knife

grab the knife

Learned Skill

Learned Skill

forward(10)turn(right)

open(box)...

...
grasp(knife)

19

fi

m

i

t

Language as a repr. of composable skills
put the knife in the drawer
Language Model Plannerturn(left)

nd a knife

grab the knife

Learned Skill

Learned Skill

forward(10)turn(right)

open(box)...

...
grasp(knife)

20

fi

m

i

t

Instructions as a source of easy supervision
put the knife in the drawer
Language Model Plannerturn(left)

nd a knife

grab the knife

Learned Skill

Learned Skill

forward(10)turn(right)

open(box)...

...
grasp(knife)

21

fi

m

i

t

Text corpora as priors on plausible plans
put the knife in the drawer
Language Model Plannerturn(left)

nd a knife

grab the knife

Learned Skill

Learned Skill

forward(10)turn(right)

open(box)...

...
grasp(knife)

22How can language help?
Representation of composable skills
Supervision for a planning model
Generalization to novel goals

m

i

t

Learning to plan and act with language
put the knife in the drawerturn(left)

forward(10)turn(right)

open(box)...
grasp(knife)

24

fi

m

i

t

Learning to plan and act with language
put the knife in the drawer

nd a knife

grab knife

go to the drawer...

?
turn(left)

forward(10)turn(right)

open(box)...
grasp(knife)

25

fi

m

i

t

Learning to plan and act with language

put the knife in the drawer
nd a knifeturn(left)

grab knife

...

...
forward(10)Annotateddemonstrations

cook the egg

grasp(egg)...
forward(9)Unannotateddemonstrations (x9)
26

fi

m

i

t

Learning to plan and act with language

put the knife in the drawer
nd a knifeturn(left)

grab knife

...

...
forward(10)Latent alignments!

cook the egg

grasp(egg)...
forward(9)Latent plans andalignments!
27

fi

fi

m

i

t

Learning to plan and act with language

put the knife in the drawer
nd a knifeturn(left)

grab knife

...

...
forward(10)Latent alignments!

cook the egg
nd an egg

grasp(egg)locate stove...
forward(9)Latent plans andalignments!
28

fi

m

i

t

3
(SL) : Semi-supervised skill learning w/ latent lang.

cook the egg

nd an egg

grasp(egg)locate stove

forward(9)
29

m

i

t

3
(SL) : Semi-supervised skill learning w/ latent lang.

argmaxθalignments
subtaskslog pθ ( goal , subtasks ,alignments, actions )annotated demos

subtasksgoal(+ log pθ,,alignments

actions),

unannotated demos

30

m

i

t

3
(SL) : Semi-supervised skill learning w/ latent lang.

alignments

skills

31

fi

m

i

t

3
(SL) : Semi-supervised skill learning w/ latent lang.

nd a knife

grab it

α=1

α=1

α=2

turn(left)

forward(10)graspalignments

1. Improve alignments
skills

32

fi

m

i

t

3
(SL) : Semi-supervised skill learning w/ latent lang.

nd a knife

grab itturn(left), forward(10)grasp

Looks like inference in an HMM with every
NL string as a possible hidden state!

alignments

2. Improve labels
skills

33

m

i

t

3
(SL) : Semi-supervised skill learning w/ latent lang.

Just gradient descent.

alignments

3. Improve parameters
skills

34

m

i

t

Inferred task decompositions

35

m

i

End to end success ratest

20.1
16.5 17.2
0behavior (SL)3 HLSM FILM
cloning 10[Blukis
[Min
ann

2021]

2021]

+

+

%

37

m

i

End to end success ratest

16.5 17.2 20.1
0behavior (SL)3 HLSM FILM
cloning 10% [Blukis
[Min
ann

2021]

2021]

+

+

38

m

i

End to end success ratest

16.5 17.2 20.1
0behavior (SL)3 HLSM FILM
cloning 10% [Blukis+ [Min
ann

2021]

2021]

+

39

m

i

End to end success ratest

20.1
16.5 17.2
0behavior (SL)3 HLSM FILM
cloning 10% [Blukis+ [Min+
ann

2021]

2021]

40

m

i

End to end success ratest

20.1
16.5 17.2
0behavior (SL)3 HLSM FILM
cloning 10% [Blukis+ [Min+
ann

2021]

2021]

7.3

12.6
11.1

HiTUT

ET

ABP

[Zhang+ [Pashevich+ [Kim+
2021]
2021]
2021]

41

m

i

t

Subtask success rates (excl. navigation)
56

58

(SL)3

3
(SL)

10%
ann

100%
ann

27

behavior
cloning

42

m

t

i

Subtask success rates (excl. navigation)
56

58

(SL)3

3
(SL)

10%
ann

100%
ann

27

+

+

+

behavior
cloningMod
3[Shridhar (SL)[Singh2020]
2020][Corona
2021]

43

m

i

t

Subtask success rates (excl. navigation)
56

58

(SL)3

3
(SL)

10%
ann

100%
ann

27

behavior
cloning

44

m

i

t

Subtask success rates (excl. navigation)
56

58
42

27

behavior
cloning

(SL)3

3
(SL)

10%
ann

100%
ann

no
pretrain
(SL)3
45

m

i

t

Subtask success rates (excl. navigation)
56

58

27

behavior
cloning

(SL)3

3
(SL)

10%
ann

100%
ann

42

39

no
pretrainseq2seq2seq[e.g. SayCan]

(SL)3

46

fi

m

i

t

Planning with latent language?
put the knife in the drawer

nd a knifeturn(left)

forward(10)grab itturn(right)

open(box)...

...
grasp(knife)

47

i

t

cook the egg

nd a knife

π1grab a knife

π2

put the egg in the microwave

π3

put the egg in the pan

π4cook with the microwave

π5

cook using the stove

π6

...

fi

m

An implicit “library” of reusable skills48

rement and reproducibility, our main experiments will
hods on real Baxter
robotsincollaboration
with
Prof.
Learning from text corpora
e proposal (see letter of collaboration).
mit

onstraints

is the use
nd knowlmples andntext, ones informahow goals
ard house,
container

Query

Prediction

The color of a banana is [?].
I can use a [?] to chop a carrot.
I can use a [?] to scrub a carrot.
Plates are found in the [?] room.
If I drop a glass, it will [?].

green
knife
brush
diningexplodeTable 1: Predictions from the BERT language model[43]. The [Devlinright column
shows
the
word
predicted
et al. NAACL 2019]as the most probable replacement for the [?] token.
While imperfect, these predictions encode informa-

49

fi

m

i

t

A hierarchical policy with latent language
put the knife in the drawer
Language Model Plannerturn(left)

nd a knife

grab it

Learned Skill

Learned Skill

forward(10)turn(right)

open(box)...

...
grasp(knife)

50

m

i

t

Approach: learning from text corpora

51

Learning skills: summary
What:

Hierarchical policy learning from demonstrations with(sparse) natural language supervision.

How:

Automatic “parsing” of annotated & unannotated demoswith dynamic programs for alignment andinference of string-valued latent variables.

Why:

Instructions are easy to collect; training with <1k of themgives performance comparable to state-of-the-art models
evaluated with ground truth plans.

Learning functions from denotations
and descriptions

Lio
Wong

+ Josh Tenenbaum[Leveraging Language for Program Search and Abstraction Learning. ICML 2021.]

itpreviously:

now:

Predict a program to execute
given a high-level goal.

Infer a program given the
results of execution.

cooked(egg)(f24 5 (λ (X) (get/set (λ (y)(f2 1 (f41 5 y))) x)) z)

grasp(egg)forward(9)

54
fi

m

Inferring programs from speci cations

tMany learning problems are naturally formulated as program synthesis.

s/Figure/Fig./g(f24 5 (λ (X) (get/set (λ (y)(f2 1 (f41 5 y))) x)) z)

Figure 1 → Fig. 1

as in Figure 6a → as in Fig. 6aa striking gure → a striking gure

fi

55
fi

i

fi

m

Inferring programs from speci cations

m

i

t

Language & program abstractions

Programs are compositional.

for i in range(8):for j in range(7):pendown()
forward(1cm)
penup()
rotate(129)rotate(45)for i in range(5):for j in range(5):pendown()
forward(2cm)
penup()
rotate(108)rotate(72)f1(8,f2(7, 1cm))f1(5,f2(5, 2cm))

56

m

i

Language & program abstractionstPrograms are compositional.
pick up a knife, nd a tomato, then
go to the counter and slice it
find_knife
find_tomato

This compositional structure is re ected in language!
a pinwheel madeof 8 heptagons

a pinwheel madeof 5 pentagons

pinwheel(8,gon(7, 1cm))

pinwheel(5,gon(5, 2cm))

goto_counter
slice

fl

fi

57

fi

m

i

t

Language & program abstractions
nd a knifeturn(left)

forward(10)grab itgrasp

Can we use language to learn
from denotations the same
way we used it to learn with fullsupervision?

a pinwheel made of

5pentagons

for i in range(5):for j in range(5):pendown()
forward(2cm)
penup()
rotate(72)rotate(72)

58

fi

m

i

t

Language & program abstractions
nd a knifeturn(left)

forward(10)grab itgrasp

Can we use language to learn
from denotations the same
way we used it to learn with fullsupervision?
Key challenge: we need to inferprograms along with all theother latent vars!

a pinwheel made of

5pentagons

for i in range(5):for j in range(5):pendown()
forward(2cm)
penup()
rotate(72)rotate(72)

59

m

i

t

LAPS: lang. for abstraction & program searcha pinwheel made ofannotation

Program generatoralignment
lambda x, f:for i in range(x):f()
rotate(360/x)

program

Execution model

output60

small triangle

a five sided gon beside aseven sided gon

long triangles as arms

a medium square
one medium square

m

i

t

An explicit library of reusable functions
a medium eight gon
octogonfor

Execution model
1
2

six small five gons in a row
six overlapped pentagons
going left to rightseven sided snowflake with ashort space and a short line
and a short space and asmall triangle as arms
a seven sided snowflake withseven triangles and line

Generative model of programs....and example program abstractions learned with language shown with learned high probability p(word | primitive)f0=(λ (x y z) (for x (λ(u v) (move z y v))))move

lambda x, f: pen-up
for i in range(x):f()
rotate(360/x)+

a small triangle connected by a
big line to a medium triangle
a small triangle with a long line
and a medium triangle

a greek spiral with eight turns
a long line that curls in on
itself at right angles

a pinwheel made of

D. Example initialgraphics primitives
generator

a small nine gon separated
by a big space from a small
circlenine gon on left with small
circle on right not connecteda four stepped zigzag
four step ladder going from
top to bottom

a big circle
just a circle

Programfour stacked squares

move pen inparameterized loop

f9=(f0 ∞ ε)
0.07 | semicircle

smooth curvea small semicircle(f19 (f9 0 x))f4=(λ (x y z) (f0 x (/ 2πy) 1 z))0.09 | small

0.67 | separated
0.06 | spacerotates and draws aunit linef5=(λ (x y) (f4 x x y))
0.27 | gon
0.22 | small...

rotational symmetry by
number of sides

a small five gon(f5 5 x)f17=(λ (x) (pen-up (λ (y)(f16 x y))))...

lift pen between
consecutive shapes

f24=(λ (x y) (f23 (λ (z u)(f21 y 0 x u))))0.09 | snowflake
0.09 | arms

rotate shapes
around axis

eight sided snowflake with asmall seven gon as arms
(f24 7 8 x)composes functions from a library of
program fragments given words

a medium semicircle(f3 (f9 0 x))
a big semicircle(f9 (* (/ ε 1) 5) x)

a small nine gon(f5 9 x)a medium seven gon(f5 2 (f20 7 x))

five sided snowflake with a short line
and a medium five gon as arms

(f24 5 (λ (x) (get/set (λ (y)(f2 1 (f41 5 y)))x)) z)

. ..

61

m

i

t

LAPS: lang. for abstraction & program searcha pinwheel made ofannotation

Library learning
max

Program generator

θ̂

library fns

alignment
lambda x, f:for i in range(x):f()
rotate(360/x)

LAPS

program

Program searc

Execution modelmax
π̂

program

Segmentationmax
α̂alignments

output

h

62

i

t

[c.f. Ellis et al. 21, DreamCoder.]Param updatemaxθ̂library fns

Program searc

Segmentationπ̂alignments

max
programa small square

a medium squaremaxα̂

4 nested squares
63

h

m

Growing the library and the set of solved programs

it[c.f. Ellis et al. 21, DreamCoder.]Param updatemaxθ̂library fns

a small square

Program searc

Segmentationπ̂alignmentsmax

for i in range(4):pendown()
forward(1)penup()
rotate(90)programa medium squaremaxα̂

4 nested squares
64

h

m

Growing the library and the set of solved programs

it[c.f. Ellis et al. 21, DreamCoder.]Param updatemaxθ̂library fns

a small square

Program searc

Segmentationπ̂alignmentsmax

for i in range(4):pendown()
forward(1)penup()
rotate(90)programa medium squaremaxα̂

4 nested squares
65

h

m

Growing the library and the set of solved programs

it[c.f. Ellis et al. 21, DreamCoder.]squareParam updatedef fn0(fn):for i in range(4):fn()
rotate(90)maxθ̂library fns

a small square

Program searc

Segmentationπ̂alignments

max

fn0(
lambda: (pendown();
forward(1);penup()))

programa medium squaremaxα̂

4 nested squares
66

h

m

Growing the library and the set of solved programs

it[c.f. Ellis et al. 21, DreamCoder.]squareParam updatedef fn0(fn):for i in range(4):fn()
rotate(90)fn0(
lambda: (pendown();
forward(1);penup()))

a small squaremaxθ̂library fns

fn0(
lambda: (pendown();
forward(2);penup()))

a medium square

Program searc

Segmentationπ̂alignments

max
programmax
α̂

4 nested squares
67

h

m

Growing the library and the set of solved programs

it[c.f. Ellis et al. 21, DreamCoder.]squaredef fn0(fn):for i in range(4):fn()
rotate(90)Param updatedef fn1(x):pendown()
forward(x)penup()

maxθ̂library fns

Program searc

Segmentationπ̂alignments

max
programsquare(fn1(1))

a small squaremaxα̂square(fn1(2))

a medium square

4 nested squares
68

h

m

Growing the library and the set of solved programs

mi

Data: inverse graphics

2

t

0

metal

What number of gray rubber cubes are there?
how many grey rubber cubes do you see

There is another thing that is the samecolor as the large rubber thing; what is it
made of?
what material is the other object that is
the same color as the large rubber object

What if the gray sphere became a small
green metal sphere?

If you removed the red things, how many
spheres would be left?

what if the grey ball morphed into a small
green ballcount the spheres would be left after
removing the red things

200 training images:

C. Compositional Graphics (shown with random human description of n=3)
Simple shapes

Compositional objects and relations

Complex objects
a seven pointed stara seven sided snowflake withlong triangles as arms

a small triangle
small triangle
a medium square
one medium square
a medium eight gon
octogona big circle
just a circle

D. Example initialgraphics primitives
fora small five gon next to asmall seven gon
a five sided gon beside aseven sided gona small nine gon separated
by a big space from a small
circlenine gon on left with small
circle on right not connecteda four stepped zigzag
four step ladder going from
top to bottom

a greek spiral with eight turns
a long line that curls in on
itself at right angles

a small triangle connected by a
big line to a medium triangle
a small triangle with a long line
and a medium triangle

four nested squares
four stacked squares
six small five gons in a row
six overlapped pentagons
going left to rightseven sided snowflake with ashort space and a short line
and a short space and asmall triangle as arms
a seven sided snowflake withseven triangles and line....and example program abstractions learned with language shown with learned high probability p(word | primitive)f0=(λ (x y z) (for x (λ(u v) (move z y v))))f4=(λ (x y z) (f0 x (/ 2πy) 1 z))...

f17=(λ (x) (pen-up (λ (y)(f16 x y))))69

(u v)Synth(move train/testz y v))))

46.00

ve

-up

-

s

m

i

52.20
t57.0054.60

26.12
move pen in
Synth train/test
parameterized loop
92.79
Synth train/test
86.49
f9=(f0∞ ε)
Synthtrain/test
98.19
Synth train/test

DreamCoder (no language)
23.20
76.50
49.5
0.67
|
separated
0.09 | small
rotates and draws
a generative translation model)
Multimodal(nolift pen between
0.06 | space
unit line
consecutive shapes
52.93 LAPS in neural 95.6
88.1
searchf24=(λ (x y)82.3(f23 (λ (z u)
80.18
96.5
...
mutual exclusivity(f21 y 0 x u))))f5=(λ (x y) LAPS
(f4 x x+y))81.98 LAPS + ME + language-program
95.6
95.9
compression

33.4
46.00

89.20 test
Synth train/human
58.55
train/human test
aHumansmall semicircle– LAPS + ME + language-program
97.4–
compression–
95.6–+ ME + language-program
compressiona small five LAPS
gon
eight sided snowflake with a

54.60
48.60

Results: inverse graphics

0.07 | semicircle

54.60
48.60

smooth curve(f19 (f9 0 x))

0.06
27.2
33.2
52.4

No
languageattest
a medium semicircle(f3 (f9 0 x)) 0.00
No language on train/test
a big semicircle
41.44
No(*
languagetrain/test(f9(/ ε 1) on
5) x)
62.16
No lang at test
No lang at test91.0

0.27 | gon
0.22 | small

rotational symmetry by
number of sides(f5 5 x)a small nine gon(f5 9 x)0.09 | snowflake
0.09 | arms

rotate shapes
around axis

52.20
57.00
54.60

small seven gon as arms
(f24 7 8 x)

five sided snowflakewith a short line
– Original DSL; Enumerative
27.8–
and a medium five gon as arms
5 (λ (x) –(get/set (λ (y)– seven53.6
a mediumgonDreamCoder(best
library): (f24
Enumerative(f2 1 (f41 5 y)))x)) z)(f5 2 (f20 7 x))– LAPS (best library):
93.04Enumerative–– LAPS (best library):
95.6 example-only neural– synthesis

0.06
27.2
33.2
52.4

LAPS + ME+ lang. compressionLAPS in neural seDreamCoder(no language)
LAPS + mutual
exclusivity Multimodal(nogenerative)B, C) Example tasks from all three synthesis domains shown with synthetic and sample human language annotations
nthesis domains are shown with a random subset (n=3) of the paired input/output examples. Human language annotations
omly sampled (all domains were annotated by multiple people for a broader range of language.) (D) Representative
m primitives and library abstractions learned with LAPS for the graphics domain. Shown with example tasks solved withrograms containing the learned abstractions and high probability natural language learned from the joint model.
% Solved (0 – 100%)

APS in neural search# Learning Iterations (0 – 27)

70

m

i

t

Language guides discovery of program abstractions

a seven stepped staircase

a small five gon(f5 5 x)a small nine gon

Original DSL primitives

New primitives added through abstraction learning

f4=(λ (x y z) (f0 x (/ 2πy) 1 z))pen-up0.09 | small. ..
0.31 | line
0.31 | short
0.09 | a2

f0=(λ (x y z) (for x (λ(u v) (move z y v))))move pen inparameterized
loop

rotates and draws aunit line

f9=(f0 ∞ ε)
0.07 | semicircle

30.91 | three
0.98 | triangle
4(f5 2 (f6 1 4 4 x))

a four stepped staircase

six small five gons in a row(f32 4 (get/set (λ (x) x) y))(f6 1 6 5 x)

a five stepped zigzag(f25 (λ (x) x) 3 8 (f32 5 y)(f5 2 (f20 7 x))

move

1

(f5 9 x)a medium seven gon

for(f32 7 (get/set (λ (x) x) y))four small squares in a row

f6=(λ (x y z u) (for y (λ(v w) (f5 z (f5 x w))) u))f5=(λ (x y) (f4 x x y))
0.27 | gon
0.22 | small

rotational
symmetry by
number of sides

f14=(λ (x y) (for 7 (λ (zu) (f9 x u)) y))

1.0 | stepped
0.64 | staircase
0.36 | zigzag...

0.16 | circle
0.08 | turns
0.09 | nested
a big circle

f17=(λ (x) (pen-up (λ (y)(f16 x y))))0.67 | separated
0.15 | next
0.06 | space(f19 (f9 0 x))(f14 (logo_DIVL 1 4) x)

a small circle next to a small
six gona medium semicircle

a small circle(f14 ε (f14 ε (f17 2(f5 6 x))))a small semicircle...

f32=(λ (x) (for x (λ (yz) (move 1 (/ 2π 4) (move 1(- 2π (/ 2π 4)) z)))))

0.94 | four
0.89 | square(f3 (f9 0 x))(f14 (logo_DIVL ε 1) x)

Learned translation
probabilities p(π | u)a big semicircle

two nested circles(f9 (* (/ ε 1) 5) x)(f14 ε (f14 ε (f16 x)))a small nine gon next to a
medium square(f5 9 (f5 1 (f17 1 (f204 x))))...

f24=(λ (x y) (f23 (λ (zu) (f21 y 0 x u))))0.09 | snowflake
0.09 | arms
eight sided snowflake with asmall seven gon as arms
(f24 7 8 x)

five sided snowflake with ashort line and a medium fivegon as arms
(f24 5 (λ (x) (get/set(λ (y) (f2 1 (f41 5 y)))x)) z)

71

t

Lbase

2%/3"3%0*'2,/2*+0'!"#$%$"*(

L1

L2

L3

!%"'(
!"#$"%& blue_block()
red_block()...

floor(window,
door,
window)...

window()...
row(6)...

house(h=2,w=3)
roof(6)!%"'(!"#$"%& rotate(line(s=1),
th=pi/6,
r=1)...
circle(scale=0.5)
rotate(circle(s=0.1),
th=pi/8,
r=0.7)...

hexagon(s=1)
circle(s=0.5)
rotate(circle(s=0.1),
th=pi/8,
r=0.7)...

hexagon(s=1)
circle(s=0.5)
med_ring(
circle(s=0.1),n=8)

hex_circ_med_ring(n=8)

300.-/01)-$!"#$%&%'()#$

250
200

-2.0

240

-2.5

220

-3.0

-4.0

-4.5
50

L2
!:

140

L3
!;

Base
8%(*

'30,*-$
-2.2

160

-4.5

L1
!9

L2
!:

L3
!;

-2.6
120
-2.8

160

-2.8

140

-2.9

120

-3.0

100

-3.1

-3.0

-4.0

L1
!9

L2
!:

L3
!;

L3
!;

Base
8%(*

L1
!9

L2
!:

L3
!;

-3.5

175

250

-3.0

-4.0

150

Base
8%(*

-2.5

225

L1
!9

L2
!:

L3
!;

1+$#)-$

-2.8

200

-3.0

175

-3.2

-3.5
150

150

-3.4

-4.0
125

-3.6

-4.5

100

60

200

125

-3.2
-3.3

-3.0

-3.8

100

Base
8%(*

-2.7

80

L2
!:

225

200

140

100

-3.2

-3.6

120

-2.5

-3.0

-3.4
140

275
250

/("$-$

180

-2.4

L1
!9

160

10#0-$

180

Base
8%(*

180

160

-5.0

L1
!9

-3.5

-2.8

180

-4.0

100

2"3!0#"3-

200

200

-3.5

80

!"#$%$&'(")*

260

-3.0

150

Base
8%(*

-2.0

-1.5

-2.5

*+,*-#$!"#$%$&42-45-6%#70%$&'%0".1/*12'!$()*"'i!"#$%$&'(")*'+',$-.$%/'0*1.23'!"#$%&"'

m

Library learning as a scienti c tool

100

Base
8%(*

-3.8

-5.0

50

75

L1
!9

L2
!:

L3
!;

Base
8%(*

L1
!9

L2
!:

L3
!;

+$,-$%.'!*/-01

Figure 4: Relationship between concept libraries {Lbase , L1 , L2 , and L3 } (x-axis); combined library size and averag
program length in that library (dashed); and library-to-vocabulary alignment (solid).

Figure 3: Graphics libraries were defined by progressively
adding subroutines at higher levels of abstraction, resulting in more efficient expression of any particular programand hierarchical structure with human descriptions. We those produced in reference games where coarser distin
find that the length of people’s descriptions varies with
tions between whole objects are sufficient, with the aim
atthe
expenseofalargerlibrary.
[Wong*, McCarthy*, Grand* et al. Identifying concept libraries from language about object structure. CogSci 2022.]fi

the length of an object’s generative program, establish-

72 thof understanding how task goals and context shapeLearning libraries: summary
What:

Inductive program synthesis with natural language
guidance.

How:

Discovery of reusable program fragments using language to
guide a library learning procedure.

Why:

With only 100s of annotations, solve 72% more program
synthesis tasks than a leading synthesizer.

Learning fromunannotated text alone

Belinda
Li

Will
Chen

Pratyusha
Sharma[LaMPP: Language Models as Probabilistic Priors for Perception and Action. arXiv 2023]

m

i

t

Language as a latent variable, LMs as priors
a pinwheel made ofProgram generator

lambda x, f:for i in range(x):f()
rotate(360/x)
Execution model75

m

LM priors for vision & beyond!

tiSemantic
segmentation!

Householdnavigation!

LaMPP: Language Models as Probabilistic Priors
for Perception
and Action
LaMPP:
Language Models
as Pr

c Priors for Perception and Action

bedroom

bed

y1

r

TV

room
p(

nightstand

true
label

a nightstand looks
p()
like a table

bed

d1

x1

table

g

goal
objecta nightstand in)a bedroom

y2

d2

noisy
label

x2
image
segments

Activityrecognition!

living
room
true

r

(crack egg

agent
locationgoal object
present?

x

agent
observation

whisk)

Dirichletpriorpancakes,= p( to makeAegg
rolloutof) this policy thus consi
crack
then whiskp(a TV in a living room)ymake reachedtask
ready
the goal objectpancakes
is judged to be present, the ep

crack
egg

igure 2. Generative model for image semantic segmentation.
FigureGenerative model for object navigation. Specifically,
mages originate in a room r, which generates the objects
y1 , y3.
2 in
he room, which generate noisy object labels d1 , d2 given
representing
a goal object g, we depict a decomposition of the agent’s

transition
igation actions, interleavedwith aprobabilities
ery observation obtained while nachoosing
actionsrequiry1
y2 effectively
actionunobserved property of environmen
object is in fact present
near the avideo
x1
x2 frame
property with a latent variable yagent must infer the room that is mgoal object. When selecting, the aits current perception is reliable.

76
We normalize the low-level policy’noisyase models. We focus
bed d1
table d2 labelagLMpriorsimprove
generalization
zero-shot(ZS),
out-ofmtilivingilistic Priors for Perception and Actionlobution (ID). The type of
room
Model
mIoU Best/Worst Object ( IoU)
the availability
and
disp(
a
T
Base model
47.8
ID
Model
chaining
37.5
showercurtain(+16.9)ations focus on the casetoilet ( 37.2)
x2
y just forLAcomponentsofNightstand
go
x
MPP
48.3
shower curtain (+18.9)
Bed
1
true
pr
objects), but no informa-desk ( 2.16)
image
Base model
33.8
OOD configurations ofsegments)bilistic(e.g.,
LA
MPP
34.0
nightstand(+8.92)
Priors for Perception and Action
sofa ( 2.50)
Figure 2. Generative model for image semantic segm
OOD evaluations focus
Success rate
in a room r, which generates the objects
Model
Class
Freq. Best/Worst
Object( originateSR)
articular
label
combinaTable
1. Imagesemantic
segmentation
resultsImagesforID andOOD
generalization.
(IoU)
for each
Base model We report
52.7 Intersection-over-Union
53.8 theroom,
which generate noisy object labels d1 , d2 repd).model:ID
evaluations
focus
ag
the basea modelchaining
Uniform
prior model,
52.1
51.7
- approach, and L A MPP.
perceptuallysimilarobjects.
Finally,eachdgenerates
Model
chaining
61.2averaged
65.3 overToilet(+20.9)We
report
mIoU
(IoUs
each
objectcategory),
asiistribution
is
known
and
ob
TV
Monitor(4.2)
well as the most- and least-improved object from each method

r

y

x

segmentx,acontinuous
regionof
image
pixelsdepic
irelative to the base model (and the corresponding IoU). L A MPP
L A MPP
66.5
65.9
TV Monitor (+33.0)
improves semantic segmentation dramatically on
certain categories,
object.
Rooms
r,
true
labels
y
,
and
noisy
labelsdaiiPlant ( 0.0)
while having minimal effect on all other categories.
77
while image segments xi are observed.

LM priors: summaryWhy:

Big increases on accuracy on rare labels, input
con gurations.
fi

How:

Query LMs to parameterize domain-speci c graphicalmodels.

fi

What:Language as a source of background knowledge in generalprobabilistic models.

Learning interactively

Jesse
Mu

[Compositional Explanationsof Neurons. NeurIPS 2020.]

Evan
Hernandez

Teona
Bagashvili

Sarah
Schwettmann+ David Bau and Antonio Torralba[Natural Language Descriptions of
Deep Visual Features. ICLR 2022.]

m

i

t

Understanding deep networks
What has this network learned?

⇒⇒ Poodle

80

m

i

t

Understanding features in deep networks
What is the function of this neuron?

⇒⇒ Poodle

81Labeling neurons with visualizations

Top activated imagesmit

mples from the Broden Dataset. The ground truth for each concept is a pixel-wise dense anno

ed images

Train
non
Collection

Segmented images using the binarized unit activation map

d images using the binarized unit activation map
Semantic segmentation annotations

egmentation annotations
Segmented annotations

Idea: determine aTraining
Network
Supervneuron's function by
none
AlexNetidentifying input (regions)
AlexNet
GoogLeNetthat activate it.
Supervised

VGG-16
Sel
ResNet-152
Extremely labor-intensive!
DenseNet-161
Self

AlexNetto com82
the baswater IoU .14

m

i

t

water OR river IoU .15

Labeling neurons with language(water OR river)
AND NOT blue IoU .16(a) inputs x

river IoU .08(b) neuron f483(x)blue IoU .006

NOT blue IoU .004

λ(e) logical forms L(x)Intersection

Neuron + Concept(c) neuron masks M483(x)pmi( ⋅ ; ⋅ (f)) IoU

water that is
not blue(d) concepts C(x)Figure 1: Given a set of inputs (a) and scalar neuron activations (b) converted into binary masks (cwe generate an explanation via beam search, starting with an inventory of primitive concepts (d), theincrementally building up more complex logical forms (e). We attempt to maximize the IoU score oan explanation (f); depicted is the IoU of M483 (x) and (water OR river) AND NOT blue.

max
log
p(description
∣
mask)
−
log
p(description)
quality and interpretability of these learned concepts relate to model performance? Third, can we us

the logical concepts encoded by neurons to control model behavior in predictable ways? We find83tha

Under review as a conference paper at ICLR 2022m

it

Machine-generated neuron descriptions

84

m

i

t

Editing models

85

i

t

⇒ chihuahua

⇒ car

frog

ship
Delete neurons labeled as text recognizer→ 12% decrease in error rate!

86
s

m

Editing models

spurious trained model model detect text, hurting its
mance on performanceon the test
set.
Editingmodels
8% accu-spurious
olutional
nfirming
capacity
7c shows
rline andest accu4
or rate).
ng proceurons acm

i

t

87

does not eliminate, units sead, nose, eyes, Auditing
lective forunblurred faces.
modelsd images).
mit

Blurred ImageNet [Yang et al. 2021]
A Study of Face Obfuscation in ImageNet

(a)

(b)

Figure 1. Most categories in ImageNet Challenge (Russakovsky et al., 2015) are not people cmany people co-occurring with the object of interest, posing a potential privacy threat. These arehusky, beer bottle, volleyball and military uniform.
88ision aims to construct privacy-aware datasets, e.g. by deakage of information Auditing
about specificindividuals
intotrainedmodelsat extent does this form of anonymization actually reduce
We wish to understand if models
eatures that can identify specificemographic categories. A core
the number
ne Blurring
learningreduces
is to enable
auditingof face-sensitivee, weapply MILANneurons
to investigate
across12models…
et privacy.
mitsubset of convolutional units in
ge classification on the blurred
ese models are distributed by the89mageNet,
Auditing
modelsan faces. (a)AN iden…but not entirely!
LAN canFigure 1. Most categories in ImageNet Challenge (Russakovsky et al., 2015) are not people categories. Howeve
red andmany people co-occurring with the object of interest, posing a potential privacy threat. These are example images fro
husky, beer bottle, volleyball and military uniform.
eals that (b)Effects of face obfuscation on classification accuracy.
well as both face-centric and face-agnos
ns selecObfuscating sensitive image areas is widely used
for pre-of people
Faces
In all of the 4 tasks, models pretraine
serving privacy (McPherson et al., 2016). Using our face
images perform closely with models pree to useannotations and a typical obfuscation strategy: blurring
images. We do not see a statistically sig
(Fig.
1),
we
constructaface-blurred
versionof
ILSVRC.
urons dibetween them, suggesting that visual fea
What are the effects of using it for image classification?
face-blurred pretraining are equally trans
(c)
At
first
glance,itseems
inconsequential—one
shouldstill
not simencourages us to adopt face obfuscatio
recognize a car even when the people inside have their faces
protection on visual recognition dataset
blurred. However, to the best of our knowledge, this Human
has
faces
cted atabout
detrimental effects on the dataset’
not been thoroughly analyzed. By benchmarking various
deep neural networks on original images and face-blurred
Blurring
contributions are
142 face-selective
neurons
12 modelstrainedContributions.
on blurredOur
faces.
images, we reportinsightsacrossabout the effectsof face blurring.
mit

Figure 6: (a) The blurred ImageNet dataset.
90
obtain accurate face annotations in ILS

m

i

t

Adversarial examples

91Unit 39
Adversarial examples
in NLPfor vision. For each scene (with 3 example images atmticonnection weight) are shown, along with their length
ons to crudely modify an input image and change thehe top-right corner, the left-most image is presented toshown); we modify the image to the right-most image,

7

OR

6

(nobody in hypothesis)Uni

hyp:nobody AND (NOT pre:hair) AND (NOT
pre:RB) AND (NOT pre:’s)
IoU 0.465
wentail -0.117 wneutral -0.053 wcontra 0.047

NEIG
hyp:
IoU 0

Pre
Three women prepare a meal in a kitchen.
Orig Hyp The ladies are cooking.
Adv Hyp Nobody but the ladies are cooking.
adv
adv
True entail → neutral Pred entail → contra

Pre
Orig
Adv
True

Unit 133 (couch words in hypothesis)

Unit 15 (sitting only in hypothesis)

Unihypothesis
contains: synonyms
of couch or one
NEIGHBORS(hyp:couch)OR hyp:inside
OR ofinside,indoors,home, eating. OR hype:eating
hyp:home
OR hyp:indoors
IoU 0.202
wentail -0.125 wneutral -0.024 wcontra 0.088

hypothesis
eating, sitting,sleeping, or sitsOR
and hyp:hyp:eatingcontains
OR hyp:sitting
OR hyp:sleeping
premisedoes
containsits
hyp:sits
ANDnot(NOT
pre:sits)
hyp:
IoU 0.239
wentail -0.083 wneutral -0.059
wcontra 0.086
IoU 0

Pre
5 women sit around a table doing some crafts.
Orig Hyp 5 women sit around a table.
Adv Hyp 5 women sit around a table near a couch.
adv
adv
True entail → neutral Pred entail → contra

Unit 941 (inside/indoors in hypothesis)
hyp:inside OR hyp:not OR hyp:indoors OR
hyp:moving OR hyp:something
IoU 0.151
wentail 0.086 wneutral -0.030
wcontra -0.023
Orig Pre Two people are sitting in a station.

Orig Pre A blond woman is holding 2 golf balls while
reaching down into a golf hole.
Adv Pre A blond woman is holding 2 golf balls.
Hyp
A blond woman is sitting down.
adv
adv
True contra → neutral Pred contra → contra

Orig
Adv
Hyp
True

Figure 9: “copy-paste” adversarial examples for NLI. T
an adversarial (adv) premise or hypothesis which chan
adv
model prediction (original label/prediction ! adversa

sensible; water, foliage, and rivers contribute to a swimm
92Explainability
and
model
accuracy
bute to model accuracy?
t

0.8

SNLI acc

0.88
0.6

0.4

0.84
0.80
0.76

0.1

0.2

IoU

0.2

Description match

0.4

0.6

IoU

−0.50

0.300

ion

ricallyurons,
above.
of theeuron
hat iswhere
d thatmore

NLI
Language

VisionAccuracy when firing

i

ion

m

93

Explainability
and
model
accuracy
bute to model accuracy?
t

0.8

0.6

0.4

↓

worse predictions!

SNLI acc

better explanations

0.88
0.84
0.80
0.76

0.1

0.2

0.2

IoU

0.4

0.6

IoU
−0.50

0.300

ion

ricallyurons,
above.
of theeuron
hat iswhere
d thatmore

NLI
Language

VisionAccuracy when firing

i

ion

m

94

t

i

 

 

 

 

 

 

 

 

 

 

 

 

 

 

fl

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

fi

 

 

 

 

 

 

 

fl

 

 

.

 

 

Green and yellow animals, a yellow smiley face, and are ghter's head and jacket.
The heads of animals
about animals
foods in the packet
Dog and fox eyes.
This is a animal head.
Body part of the birds
The bottom portions of faces of animals.
Dog heads with black and white.
Anything that has the color blue in it.
These are animal heads.
These are animal heads.
about animals
This is showing both parts of animals and parts ofwheels
The color white
These are diagonal lines.
Shifting contrast colors, either light-dark-light or darklight-dark.
Doors, windows, and see-through objects.
fi

fi

fl

m

The next challenge: relevant explanation
Red clothing, vehicles, plants, and objects.
Human skin
The black areas are highlighted in the images.
The images show body coverings of animals including
fur, feathers, hair, and claws.
It shows an image that has a bit of white in it.
These are owers and animal fur.
This regions is that is being highlighted are spots.
eyes and mouth
face of all animals
This is fruit and other circles.
Face of dogs
Dog faces and bodies.
The face area is highlighted in the images.
eyes and beakPeople's faces and other body parts.
Green grass, plants, and objects.
This is black and white grids.
yellow spots surrounded by uniform colors
body of the dogs
archesall the above are in green color

This is text.
The regions depict lines,
center.
They are the west or 9 oobjects that contain conc
This is the area above do
They are brownish fur.
Texts and blue or yellow
This is very natural areaside angle part of all obje
letters of the all images
Objects with curved edge
Core hours are set the tim
outside in the of ce.The
exible.
All images are made up
They are circular objects
Blue colored objects.
Animal skins are in the im
The shiny white part of v
They are the midsections
Eyes of various animals,

Generating explanations: summary
What:

Automatic natural language labels for every neuron in a
deep network.

How:

Textual summaries of neuron visualizations using program
synthesis & image captioning techniques.

Why:

Neuron explanations surface adversarial vulnerabilities,
expose sensitive features, improve model robustness.

m

i

t

Toward natural language supervision
Learning fromdemonstrations
turn(90)
grasp

Learning fromobservations…

(f24 5 (λ(X) (get/set(λ (y) (f2 1(f41 5 y)))x)) z)

Learning to
explaindog
faces &
wheels

E ective & e cient natural language supervision is possible for lots ofinteresting learning problems!
ffi

ff

97

Lio
Wong

Pratyusha
SharmaThank you!

Evan
Hernandez

Jesse
Mu

Teona
Bagashvili

Sarah
Schwettmannitoriginal(a) “purple flowers”(b) “horizontal lines”

original(c) “cloudy white sky”(d) “empty road”

Force neurons with the desired label to activate
controlled manipulation of image content!
99:m

Text-based image editing

m

i

t

Text-based image editing by activating descriptions
original(a) “purple flowers”(b) “horizontal lines”

original(c) “cloudy white sky”(d) “empty road”

100