Synthesizing Environment-Aware Activities via Activity Sketches
Yuan-Hong Liao1,2⇤ , Xavier Puig3⇤ , Marko Boben4,5 , Antonio Torralba3 , Sanja Fidler1,2
1
University of Toronto, 2 Vector Institute, 3 MIT, 4 University of Ljubljana, 5 Abelium d.o.o{andrew, fidler}@cs.toronto.edu, {xavierpuig,torralba}@csail.mit.edu, marko.boben@fri.uni-lj.si

Abstract
In order to learn to perform activities from demonstrations or descriptions, agents need to distill what the essenceof the given activity is, and how it can be adapted to newenvironments. In this work, we address the problem ofenvironment-aware program generation. Given a visualdemonstration or a description of an activity, we generate program sketches representing the essential instructions
and propose a model to transform these into full programs
representing the actions needed to perform the activity under the presented environmental constraints. To this end,
we build upon VirtualHome [17] to create a new dataset
VirtualHome-Env, where we collect program sketches torepresent activities and match programs with environmentsthat can afford them. Furthermore, we construct a knowledge base to sample realistic environments and anotherknowledge base to seek out the programs under the sampledenvironments. Finally, we propose ResActGraph, a networkthat generates a program from a given sketch and an environment graph and tracks the changes in the environment
induced by the program.

1. Introduction
We want agents to be able to perform everyday tasks(such as setting up the table, preparing coffee, or even sit on
the couch and watch TV). An agent should learn to performthese tasks from high-level descriptions or visual demonstrations. The challenge is on how to generalize the acquired knowledge to new environments. For instance, if wewant to learn to make coffee, an agent could first watch avideo of someone making coffee in order to extract the sequence of steps (program) that need to be executed. However, when trying to make coffee in an environment that differs from the environment in which the demonstration took
place, the agent has to adjust the program so that it can be
executed. For instance, the agent could find that the coffee
machine is unplugged, or that it is in the living room instead⇤ indicates equal contribution

Activity (a): Watch TV
Description (d a):
Go to the living room. Sit on the couch and watch TV.
Visual Demonstration (ia):Activity Sketch (sa)
Sit

Couch

Watch

TV

(a)
New Environment (e)Environment-aware
Program (p)
Walk

(b)

Changes: Cat on sofa

Living room

Find

TV

Find

Couch

Grab

Cat

Put

Cat

Sit

Couch

Watch

TV

Figure 1. Overview of the environment-aware program generation.
Our goal is (a) generating a sketch sa distilling the essential stepsof the given demonstration ia or description da and (b) given a
new environment e, generating a program p, adapting the sketch
to e. The program contains the instructions to perform the activity(blue blocks) as well as instructions to deal with the environment(red blocks, grabbing the cat to sit).

of in the kitchen, or that someone else is currently using it
and the agent needs to wait. Being able to perform these adjustments requires the access to a common-sense databaseof knowledge that allows the agent to decide which steps
in the demonstration are essential in the task definition, andhow the program needs to be modified (by adding/removing
steps) in order to accomplish the task in a new environment.
To address the generalization problem we represent activities with sketches, representations inspired from work inprogramming languages [18, 15]. Sketches are high-level
representations of the steps needed to perform a task but
leaving holes that need to be completed for the sketch to
become executable in a particular environment.
Fig. 1 illustrates our goal. Given a visual demonstration
or a description of someone going to watch TV, we want to
extract the sketch of the activity (fig. 1a). In this example,
the sketch consists of two steps: Sit Couch, Watch TV.
The fact that the demonstration had the person going to theliving-room is not important. Watching TV requires us to
go to the room that contains the TV. Executing this activity
in a new environment requires expanding the sketch into acomplete program (fig. 1b). The sketch expansion depends
on the state of the environment. In this example, it turns outthat there is a cat on the couch. To sit on the couch, we needto push away the cat first. We show the automatically generated program that includes all the steps needed to complete
the task and to deal with the cat.
To expand sketches into programs, it is necessary tohave access to commonsense knowledge about the worldand how to deal with typical situations. This knowledge
provides ways to solve situations that can be reused in a
multitude of tasks. Since no such information, with diverse programs and environments, exists to date, we collect a new dataset, built upon VirtualHome [17], containing around 3k home activities. We extend the VirtualHome
dataset to include more actions and over 30k programs andcollect sketches of the activities by crowdsourcing. Eachenvironment is represented as a graph with 300 objects and4000 spatial relations on average.
We then propose a model to generate programs by selecting, for every step, a node in the environment graphrepresenting an object of interaction. To do so, we exploit
Graph Neural Networks (GNNs) to reason about the states
and relations between the objects in the environments. Furthermore, we propose ResActGraph, a model that reasons
about the changes in the graph induced by the agents previous steps to generate the goal program.
The main contributions of this work are: introducing
sketches as environment-independent representations of anactivity, a database of commonsense knowledge of activities, sketches and how to deal with a variety of situations,and a method to generate programs from sketches that accomplish the task in a new environment. We show results in
VirtualHome [17].

2. Related Work
Learning from demonstrations. Learning from visualdemonstrations or language descriptions has been of increasing interest in both robotics and computer vision.
However, this has been mostly focused on learning lowlevel tasks rather than the high semantic level activities that
we tackle in this work. For example, [20, 13, 1] focus onlearning to navigate in environments or manipulate objects,
while in visual imitation learning, multiple works have usedvideos to learn to manipulate objects under low supervision regimes [16, 7] or imitate kinetic human behaviors[14]. Our work is more closely related to semantic planning [10, 22, 3], which focus on modeling sequences ofcomposite and semantically loaded actions. Learning thoserequires inferring and modeling the sub-goals of a giventask. [10] represents such goals as a graph, with nodes be-

ing actions and edges being precondition states, while we
propose program representations. Furthermore, to perform
such tasks, it is necessary to know what the constraints ofthe given environment where it will be executed are. Similarly to [3], we propose to encode knowledge of the environments and how they can constrain the activities to be
performed.
Program Synthesis by Sketching. Our approach forenvironment-aware program generation is partially inspiredby performing program synthesis by sketching. In [18], asketch expresses the high-level structure of an implementation but leaves holes in place of low-level details, whichcorresponds to our model that derives the details based on
the environmental constraints so as to execute the programs
smoothly.A recent body of work has developed neural approaches to program generation using user-provided examples [8, 4], visual demonstrations [19], and descriptions [6].
The work [15] is the most related to ours. They learn amodel that predicts sketches of programs relevant to a label
and the predicted sketches are concretized into code using
combinatorial techniques. The main difference between ourwork and theirs is not only that our sketches are inferredfrom visual or textual data, but that we focus on how toincorporate the environmental constraints in program generation.

3. Problem formulation
The goal of environment-aware program generation is topredict, given a demonstration or description of an activity
and an environment, a program that can execute the activity in such environment. We define this task and the corresponding notation in this section.
Let A and E be the universe of activities and environments. An activity a (e.g. watch TV) can be represented asa set of programs Pa containing a sequence of instructions(e.g. TurnOn TV, Sit Sofa, Watch TV), which varydepending on how the activity is performed. Let (p, e) be
an indicator function determining if p can be executed in e(e.g. an agent can not grab cups inside a closed cabinet).
Given an activity a 2 A and an environment e 2 E, ourgoal is to learn a model that generates p such thatp 2 Pa , (p, e) = 1(1)

We use the corresponding visual demonstrations ia 2 Ia
or descriptions da 2 Da to specify a. However, ia and da
are implicitly conditioned on certain environments whichmight differ from the current one e. Therefore, directly inferring p̂ does not ensure that p̂ satisfies eq. 1 given e.
Inspired by [18], we introduce program sketches sa 2 S
as environment-independent representations of the activities. We thus change the constraint in eq. 1 to be:

(b)

(a)



propagation












 

GRU enc


 
 

Copy

…

…

…

…


h

h

h

GRUdecGRU dec

GRU dec







 







Learned state changes

Differentiable path

Non-differential path

Figure 2. (a) We extract the ground truth environment graph from VirtualHome and perform message passing on the graph. (b) At every
time step, the decoder perform sequential classification over the hidden states of the graphs (top row). The selected nodes are shown inbold border blocks. We also model the environment changes induced by the generated programs (solid red arrows).

p 2 Psa , (p, e) = 1(2)

With the program sketches as proxy representations, we
can divide the task into two sub-problems: a model thatpredicts a sketch sˆa from a demonstration ia or a description
da and another model that predicts a program p̂ given thepredicted sketch sˆa and an environment e:p̂ = fsketch2prog (sˆa , e), wheresˆa = fdemo2sketch (ia ) or sˆa = fdesc2sketch (da )(3)Here, p and sa are a sequence of instructions. Each instruction is represented by an action and up to two arguments
representing objects of interaction (↵, 1 , 2 ).1

4. ModelIn this section, we present our approach to theenvironment-aware program generation task. First, we introduce ResActGraph to generate programs from sketches
and target environments. Later on, we describe how we predict sketches from demonstrations or descriptions.

4.1. Program generation from sketches and graphs
We frame the program generation task as a seq2seq problem, where an encoder encodes the input sketch and the decoder generates the target program one instruction at a time,composed by an action and object arguments. Given that
1 The number of arguments depends on the type of the action.

the program must be grounded in a target environment, instead of predicting the objects from a fixed taxonomy, themodel predicts for each instruction object instances that arepresent in the environment. This has two benefits: (1) It
avoids referring to object instances that do not exist in theenvironment. (2) It allows the model to use information ofeach instance within the environment, such as its state or
relations with other objects, to predict the appropriate instruction.
To do that, we encode the scene as a graph G = (V, R)
modeling the dependencies of the object instances. The
node v 2 V indicates the object instance and each node
has a label, including the object class cv , its states lv , andproperties propv . Note that V includes a node for the agent
itself. The edge r 2 R encodes the spatial relations, including ON, IN OBJ, IN ROOM, CLOSE TO, and FACE AT,
between every two object instances.
The node labels and relations are used to obtain vector
embeddings for each instance which are used by the decoder
to predict the environment-aware program, as we describe
in the following section.

4.2. ResActGraph
We adopt the GGNN [12] framework to obtain the hidden states of the nodes and capture the object relations in
the environment graph. The hidden states of each node v
are initialized by its label (cv , lv , propv ):h0v = tanh(ginit ([Wc cv , Wl lv , Wprop propv ])(4)

We apply one-hot encoding to the label and set

Wc , Wl , Wprop as learnable weights. ginit is a network
composed of fully connected layers that combine all the information.
At propagation step k, each node’s incoming information xkv is determined by aggregating the hidden states of its
neighbors v 0 2 N (v) at the previous step k 1:
X
X
xkv =
Wpj hkv0 1 + bpj(5)
j2L(R) v 0 2Nj (v)

L(R) denotes the set of edge labels and the linear layer Wpj
and bias bpj are shared across all nodes.
After aggregating the information, the hidden states of
the nodes are updated through a gating mechanism similar
to Gated Recurrent Unit (GRU) [5] as follows:
zvk = ⇢(Wz xkv + Uz hkv 1 + bz ),rvk = ⇢(Wr xkv + Ur hkv 1 + br ),hˆk = tanh(Wh xk + Uh (rk hk 1 ) + bh ),vv

hkv = (1

zvk )

v

v

hkv 1 + zvk

hˆkv

(6)

This results in a vector embedding for each object hkv , withinformation about its state and relationship with the environment.
We use one GRU to encode the sketches and another one
to generate the program one instruction at a time. For timet, let f eatsa = enc(sa ) be the output of the sketch encoder,and htdec the hidden states of the decoder. To predict the
program instruction, (↵ˆt , ˆt1 , ˆt2 ), we predict the first object
argument t1 over the graph nodes, use it to predict the action ↵ˆt and combine this information to predict the second
argument ˆt2 :ˆ1 = arg max (g 1 (ht , hK , f eat ))
sa
t
dec
v
v2V

↵ˆt = arg max (g↵ (htdec , hKˆ1 , f eatsa ))
↵2A

(7)

t

ˆ2 = arg max (g 2 (ht , hK , f eat , hK , ↵ˆ ))
t
ˆ1
sa
t
dec
v
v2V

t

where A is all the possible actions anddenotes thesoftmax function.
Note that so far the hidden states of the nodes hK
v are
constant over t, but we would like them to change according
to the program being executed. To do that, at time t, we
1 , ˆ2
use the previously generated instructions (↵ˆ<t , ˆ<t<t ) toupdate the hidden states of the nodes. We set the initialKt
0state of each node as hK= hKvv and update the state hv
at time t if v is interacted by the agent at the previous time
step or is the agent itself. For example, if the instruction
at the previous time step is grab mug, we use the action
embeddings of grab to change the hidden states of agent

and mug. Let v̂ correspond to the agent node or one of theKprevious arguments t1ˆ 1 , t2ˆ 1 . The state hv̂ t 1 is updated
as follows:
K

t 1
t
hK
+r
v̂ = hv̂

K

r = tanh(gres (hv̂ t 1

ge (Emb(↵tˆ 1 ), mt 1 )))(8)

where Emb(↵tˆ 1 ) is the embedding of ↵tˆ 1 and mt 1 isa one-hot encoding denoting if v is the subject or the object
Kof ↵tˆ 1 . ge and gres consider the change of the hv̂ t 1 andpredict the residuals. Note that since the node agent isinvolved at every time step, it tracks the progress throughthe generation. The model overview is shown in Fig. 2.
Learning. We use the cross-entropy loss function for program prediction. The GGNN and GRUs are then trainedwith the back-propagation through time (BPTT).

4.3. Inferring sketches
Activities specified by demonstrations. We use key
frames i = [in ]n=1:Ndemo as the representations of thedemonstrations, where Ndemo is the length of the key
frames. To be specific, we take the bird-eye view of eachin . Besides, we also use the ground truth semantic segmentation map iseg = [iseg n ]n=1:Ndemo as input. Two CNNs
are used to extract the features separately, and we apply latefusion to extract the nth visual features f eatn as follows:
f eatn = gf use ([CN Ni (in ), CN Nseg (iseg n )])(9)where [, ] denotes concatenation. Later on, we max-pool thefeatures over the different steps time steps and apply a GRU
to decode the sketches.
Activities specified by descriptions. We adopt the seq2seq
model with a GRU encoding each word in the description
and a GRU to decode each of the sketch instructions.

5. Dataset: VirtualHome-Env
Our goal is to generate a sketch sa from a demonstration
ia or description da and induce a program p from sa and atarget environment e. In this section, we describe how we
obtain the dataset containing (a, ia , da , sa , e, p). We briefly
introduce VirtualHome as the playground for the dataset
collection, we describe how we extend the dataset with activity sketches and finally describe how we pair the programs with arbitrary environments where they can be performed, motivating the creation of a common sense knowledge base of activities2 .
Background. VirtualHome is a dataset and simulator torepresent human household activities. The dataset was collected by asking annotators to come up with various activities and provide a description for them. In a second
2 The details of the collected knowledge base are described in 8.6

(1)Programs

Description:
Pick up my dirty clothes
from the bedroom, load thewasher, add detergent, andturn on washer

(2)

Walk

B edroom

Walk

Clothes

Grab

ClothesWalk

E ntrance hall

Walk

Washing machine

Washing machine inentrance hallOpen

Washing machine

Washing machine off

P ut

Clothes

No clothes insidewashing machine

Grab

Detergent

P ut

Detergent

R un

Washing machine

Clothes in bedroom

Detergent near
washing machine

Preconditions

Figure 3. The annotators label the descriptions and programswith certain environments in mind (bottom left), resulting in theenvironment-dependent descriptions and programs. The blocks
colored in blue are considered as environment-dependent components.

stage, annotators were shown a description of an activity
and were asked to write a program from it. With activities
represented as programs, a Unity simulator executes themin some predefined apartments and renders the programs asvideos. This allows us to obtain activities and programs together with descriptions and demonstrations (a, ia , da , p).

5.1. Collecting sketches
When collecting each activity in VirtualHome, annotators imagine an environment where the activity could takeplace and provide a description according to it. As a result, the description and subsequent program is specific to agiven environment, but may not be doable when presented
with new environments or constraints (see Fig. 3). Therefore, we need a more abstract representation of an activity,
which can be consistent with multiple environments and theinformation (a, ia , da , p).
Inspired by [18], we collect the sketches of the activities to abstract out the components that are environmentdependent and informally define the sketches as the environment independent representations. Different from programming languages, it is highly non-trivial to define thesketches of the activities since they depend on the commonsense of each individual. Therefore, we manually collectthe sketches and get the information (a, ia , da , sa , p).

5.2. Pairing programs with environments
We finally need to add an environment that pairs with theactivity programs. Since we do not know the environmentthat each annotator had in mind, we need to infer it from theprogram. We first extract the preconditions of the programs
and use them to sample feasible environments. We define
preconditions of a program as the conditions that have to betrue in the environment in order to execute the program in it.

For example, in order to execute watch TV, the tv should
be on. We construct a function that infers the preconditions from p and sample e from the set of environments thatsatisfy such preconditions E (p) :e ⇠ E (p) ⇢ E, s.t. 8j 2(p), e satisfies j(10)

The above is a weak constraint since it does not inform
about objects that are not specified in (p). To get realistic environments, these objects should follow some priors.
For example, couches can be occupied, but they can not becold. Apples can be stored inside fridges, but they are seldom found in bathtubs. We build collect these rules in aknowledge base, KB-RealEnv and use them to build the environment, starting from the environments provided in VirtualHome.

5.3. Extending programs to diverse environments
We now have the information (a, ia , da , sa , e, p). However, relying solely on the original programs results in a limited set of preconditions and thus environments. One possible reason is that when describing activities, annotators tend
to assume the simplest setting to perform the given activity.
For example, when thinking of doing the laundry, it is common to imagine that the washing machine is idle or empty.
To address that, we build a simulator
that takes a program p and environment graph e and outputs the graph corresponding to the environment after executing the program,or raises an exception if the program is not executable at a
certain step. Given a program with preconditions (p), we
start by randomly perturbing them into (p)0 and use eq. 10
to obtain e0 as an environment satisfying (p)0 . Then, we
execute p in the simulator with the environment e0 . Given
that the environment and preconditions have changed but
the program is still the same, as the program is executed,
some exception will be raised from the simulator. Then,a subroutine is called to modify the program p into p0 , byinserting or removing instructions to correct the exception,
obtaining the extended program.
For example, when we executing Sit Sofa, if Sofa
is occupied, the subroutine is expected to perform actions
to remove things on the Sofa until there is enough spaceto Sit. We manually compose the subroutines based on
different types of exception, forming the knowledge base,
KB-ExceptionHandler. We show more details of how we
augment the programs in the supplementary materials.
This way, we augment over 30k tuples of sketches, environments, and programs (sa , e0 , p0 ). Note that the sketches
sa are environment-independent, so there is no need tochange them after applying the subroutines.

Figure 4. The effect of the dataset augmentation: changes in the
distribution of preconditions for the objects in the environment.

5.4. Dataset Analysis
From the original 2807 programs in VirtualHome, we
trim out the programs that can not be executed in the simulator with the environment sampled via preconditions, obtaining 1387 executable programs. Using the process describedin Sec. 5.3, we extend these programs to our final datasetwith around 30k programs. The significant increase in the
program length is induced by the modified preconditions.
For example, the agent needs to open containers to reachobjects or make space to sit on a sofa. In Table 6, we show
the statistics of the new dataset. Fig. 4 shows the change
in the distribution of preconditions of the cup and the sauce
pan. The programs after augmentation show a less skewed
distribution of preconditions and therefore more diverse environments. Finally, we use the simulator to generate snapshots of the environment after executing each instruction ofa program, as show in fig. 12. Note that some of the objects
in the program do not have a model in the simulator, so wegenerate frames for a subset of 8421 programs.

6. Experiments
We split the dataset into train and test set in terms of different types of activities with ratio 7:3 and leave one apartment for the test set. We aim to test the capability of ourmodel with novel activities and environments. We follow
the same split for sketch prediction, where we only keep theoriginal programs for the desc2sketch, since they containthe collected descriptions, and use the available frames forthe demo2sketch task.
In this section, we describe the evaluation metrics, baselines. Next, we show the extensive experiment results of
ResActGraph. Finally, we analyze the extent to which theproposed method is environment-aware. We will describe
the implementation details in supplementary materials.

6.1. Evaluation Metrics
We analyze the performance of sketch prediction andprogram generation by measuring the normalized longest
common subsequence (LCS) between the generated andground truth sequences. LCS is sensitive to the order of thesequences and allows gaps in between. To further measure
if the generated programs achieve the specified activities,
we compute the differences between the final environment
graphs Ĝ = (p̂, e) and G = (p, e) using F1 scores3 . In
particular, we only compared the sub-graph containing theobject instances mentioned in p and p̂. We describe the details of F1 (Ĝ, G) in the supplementary materials. Inspiredby [2], we also compute F1 -state and F1 -relation.
Furthermore, inspired by program synthesis, we care
whether the generated programs are “compilable” as well.
We evaluate if the generated programs can be parsed(parsibility) and executed (executability) by the simulators.
We will describe the detailed definition of them in the supplementary materials.

6.2. Baselines
We implement five different baselines to compare withthe proposed ResActGraph.
Nearest Neighbors: For every example in the testing set,
we retrieve the training sample that has a sketch with thehighest LCS. In case of a tie, we pick the one with the mostsimilar initial graph.
Unaries: We set K = 0 in Eq. 7. This model does notconsider the relations of the objects. We use it to showcase
the benefits of modeling object relations.
Graph: This model does not consider the change of graphs
induced by programs (Eq. 8).
FCActGraph:
This model uses a FC layer tomodel the graph changes.
Specifically, it takes theKt[hv t 1 , Emb(↵tˆ 1 ), mt 1 ] as inputs and outputs hKv .
GRUActGraph: This model treats the graph changes
as another sequence and uses a GRU to ingest
K[Emb(↵tˆ 1 ), mt 1 ] as inputs and considers hv t 1 asthidden state to output hKv .

6.3. Results
We show the results of ResActGraph quantitatively andqualitatively. Next, we show the ablation study of the number of the graph propagation steps. Finally, we show theprediction results of the whole system.
Program generation from sketches and graphs. The
results are shown in Table 1. By comparing the Graph andUnaries, we show that aggregating information from neighboring nodes increases performances in nearly all metrics.
The three bottom rows of Table 1 show the results ofmodels that consider graph changes induced by programs.
The F1 scores and executability benefit the most, whichis expected. For example, suppose there is a glass nearan opened cabinet in the environment and the model predicts (Grab Glass, Put Glass Cabinet, Close
3 If the generated programs cannot be parsed or cannot be executed, theF1 is set to be 0.

LCS

F1 -relation

F1 -state

F1

Executability

Parsability

Nearest Neighbors

0.127

0.019

0.288

0.041

-

-

Unaries
Graph
FCActGraph
GRUActGraph
ResActGraph

0.372
0.4
0.469
0.508
0.516

0.16
0.171
0.261
0.41
0.41

0.142
0.171
0.273
0.408
0.42

0.159
0.172
0.263
0.411
0.413

24.8%
23.1%
33.7%
48.9%
49.3%

75.3%
82.2%
88.6%
87.9%
85.3%

Table 1. Induce program from ground truth sketches and ground truth graphs. (K = 2)
SketchGT Program

Generated Program[Open] <washing machine>[Put] <basket> <washing machine>[Put] <soap> <washing machine>[SwitchOn] <washing machine>[Walk] <bedroom> (273)[Walk] <basket> (1000)[Find] <basket> (1000)[Grab] <basket> (1000)[Walk] <bathroom> (1)[Walk] <washing machine> (1001)
Environment[Find] <washing machine> (1001)[Open] <washing machine> (1001)
Washing machine (1001) is closed[Put] <basket> (1000) <washing machine> (1001)
Washing machine (1001) is off[Find] <soap> (1002)
Washing machine (1001) is unplugged[Grab] <soap> (1002)
Washing machine (1001) in bathroom (1)Soap (1002) inside Washing machine (1001) [Put] <soap> (1002) <washing machine> (1001)[Find] <washing machine> (1001)[Close] <washing machine> (1001)[PlugIn] <washing machine> (1001)[SwitchOn] <washing machine> (1001)[Walk] <bedroom> (273)[Walk] <basket> (1000)[Find] <basket > (1000)[Grab] <basket > (1000)[Find] <washing machine> (1001)[Open] <washing machine> (1001)[Put] <basket> (1000) <washing machine> (1001)[Find] <soap> (1002)[Grab] <soap> (1002)[Put] <soap> (1002) <washing machine> (1001)[Close] <washing machine> (1001)[Plugin] <washing machine> (1001)[SwitchOn] <washing machine> (1001)

Figure 5. An example of the prediction of ResActGraph. We colore the LCS between the prediction and ground truth in light green. Notethat the sketch is environment agnostic, so it does not specify the ‘id’ (the number in the parentheses) of the object instances.

Unaries (K=0)
ResActGraph (K=1)
ResActGraph (K=2)
ResActGraph (K=3)

LCS
0.372
0.427
0.516
0.513

F1 -relation
0.16
0.262
0.41
0.399

F1 -state
0.142
0.271
0.42
0.407

F1
0.59
0.264
0.413
0.401

Table 2. Ablation study of the propagation steps K.

Cabinet) at the first three steps, if the model wants tograb other things from the cabinet without opening it aftert 4, it fails since the cabinet is closed at t = 3.
Among the three bottom rows of Table 1, the proposed
ResActGraph performs the best in F1 . The reason is that using the residual architecture is easier for the model to learn
the state “changes” compared to using FC. Using GRU to
encode the state changes is also an alternative, but we observe that it converges slower since it has more number ofparameters to learn and does not perform better.
In Fig. 5, we show the qualitative results of ResActGraph. Even though the generated program does not exactly
match the ground truth, it reaches nearly the same environment state. In Fig. 6, we show the results with the same
sketch, but different initial environment states. Note that weonly show the states and relations related to the programs.
The model correctly induces correct actions w.r.t. the environment changes and the two generated programs nearlyreach the same environment states.
Ablation studies. We show the effect of the number of
propagation steps K in Table 2. Both the baseline and the
proposed model benefit as K increases, and the proposedmodel performs better than the baseline regardless of different K. We found that the performance saturates whenK = 2, so we fixed it for all other experiments.
Combining predicted sketches with ResActGraph.
The LCS of fdemo2sketch and fdesc2sketch are 0.15 and 0.27
respectively. The reason why the LCS is low is that they are
significantly shorter (on average 2.4 instructions) than the
programs (on average 18.79). This makes the sketch prediction a quite challenging task where LCS is highly penalizedeven under small errors.
With the trained model, we can directly generate the programs from the demonstrations or descriptions. Note that
we do not re-train the program generation model. In Table 3 and Table 4, we show the results of the program generation with the sketches predicted from descriptions anddemonstrations respectively. The performance gap between

Sketch[Sit] <sofa>[Read] <book>Environment 1

Generated Program 1

Environment 2

Generated Program 2

Book (263) in bedroom (23)Bookmark (27) near book (263)Bookmark (27) in bedroom (23)
Sofa (101) in bedroom (23)
Sofa (101) near bookmark (275)
Sofa (101) is freeBook (263) near sofa (1001)[Walk] <bedroom> (23)[Walk] <book> (263)[Find] <book> (263)[Grab] <book> (263)[Find] <sofa> (101)[Sit] <sofa> (101)[Find] <bookmark> (27)[Read] <book> (263)

Sofa (101) in bedroom (23)Book (263) in bedroom (23)Bookmark (27) in bedroom (23)
Sofa (101) occupied
Phone (75) on sofa (101)
Cards (261) on sofa (101)Game (231) on sofa (101)Check (213) on sofa (101)[Walk] <bedroom> (23)[Walk] <sofa> (101)[Find] <sofa> (101)[Find] <phone> (75)[Grab] <phone> (75)[Release] <phone> (75)[Find] <cards> (261)[Grab] <cards> (261)[Release] <cards> (261)[Find] <game> (231)[Grab] <game> (231)[Release] <game> (231)[Find] <check> (213)[Grab] <check> (213)[Release] <check> (213)[Sit] <sofa> (101)[Standup][Walk] <book> (263)[Find] <book> (263)[Grab] <book> (263)[Read] <book> (263)

Figure 6. An example of the prediction of ResActGraph with the same sketch, but different environments. We highlight the difference
between two environments with orange and color the LCS between two predictions in light green.
Demonstration

Description

Environment

Generated Sketch Generated Program

Phone (10) in living room (1)[Grab] <phone>

Environment

Generated Sketch Generated Program

Walk into the home office. Walk up to Chair (29) close to keyboard (2)the chair, sit down in the chair. Type Chair (29) is free
Keyboard (2) close to computerwith the keyboard.
Computer (31) in living room (1)[Walk] <living room> (1) [Walk] <phone> (10)[Walk] <phone> (10)[TurnTo] <phone> (10)[Find] <phone> (10)[LookAt] <phone> (10)[Grab] <phone> (10)[Sit] <chair>[Walk] <living room> (1) [Sit] <chair> (29)[Type] <keyboard> [Walk] <desk> (137)[Find] <keyboard> (2)[Find] <chair> (29)[Type] <keyboard> (2)Figure 7. Predictions from the ResActGraph given sketches from descriptions and demonstrations.
UnariesGraph
ResActGraph

LCS
0.289
0.297
0.331

F1 -relation
0.188
0.24
0.347

F1 -state
0.191
0.233
0.339

F1
0.189
0.241
0.348

Executability
44.4%
43.9%
63.3%

Parsability
73.4%
89.9%
92.5%

Table 3. Induce program from sketches predicted from decsriptions and ground truth graphs. (K = 2)
UnariesGraph
ResActGraph

LCS
0.257
0.284
0.327

F1 -relation
0.099
0.202
0.316

F1 -state
0.09
0.202
0.323

F1
0.098
0.203
0.318

Executability
25.6%
34.3%
54.7%

Parsability
74%
82.8%
86%

Table 4. Inducing program from sketches predicted from demonstrations and ground truth graphs. (K = 2)

the proposed model and the baselines becomes small. The
reason is that the model is confused when the non-perfectsketches are given, resulting in similar performance. Notethat all models still perform better than Unaries. Qualitative
results are shown in Fig. 7, showing that the model predicts
the plausible sketch and ResActGraph generates plausible
programs w.r.t to the sketch and the graph.

7. Conclusion
In this work, we propose the environment-aware
program generation task.
We introduce sketches asenvironment-independent activity representations and address the problem in two steps: generating sketches fromdemonstrations or descriptions and generating programsfrom sketches and graphs. To this end, we propose a novel
model, ResActGraph and create a dataset VirtualHome-Env,
with sketches, environments, and programs.
The environment-aware program generation is far frombeing solved and opens exciting research directions. While
we access the truth state of the environment graph, one natural extension would be to predict it from environment observations [21], but this would still require an oracle providing images of the interior of closed objects or unexplored
areas. Additionally, though the ResActGraph updates thehidden states of nodes at each time step, the graph structure
is not changed, which impedes from performing message
passing at each time step. Finally, our setting assumes afully observable environment. Working with partially observable environments opens up exciting directions, such as
dealing with environment changes due to factors external tothe agent or modelling the agent’s theory of mind.
Acknowledgement The authors would like to thank Chaoqi Wang, TsunHsuan Wang, and Meng-Li Shih for helpful discussions and feedback forthe early draft. X.P. is supported by La Caixa Fellowship.

References[1] M. Alomari, P. Duckworth, M. Hawasly, D. C. Hogg, andA. G. Cohn. Natural language grounding and grammar induction for robotic manipulation commands. In Proceedingsof the First Workshop on Language Grounding for Robotics,
pages 35–43, 2017. 2[2] P. Anderson, B. Fernando, M. Johnson, and S. Gould. Spice:
Semantic propositional image caption evaluation. In European Conference on Computer Vision, pages 382–398.
Springer, 2016. 6[3] M. Beetz, U. Klank, I. Kresse, A. Maldonado,
L. Mösenlechner, D. Pangercic, T. Rühr, and M. Tenorth.
Robotic roommates making pancakes.
In Humanoid
Robots (Humanoids), 2011 11th IEEE-RAS International
Conference on, pages 529–536. IEEE, 2011. 2[4] M. Bošnjak, T. Rocktäschel, J. Naradowsky, and S. Riedel.
Programming with a differentiable forth interpreter. arXiv
preprint arXiv:1605.06640, 2016. 2[5] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio.
On the properties of neural machine translation: Encoderdecoder approaches. arXiv preprint arXiv:1409.1259, 2014.
4[6] L. Dong and M. Lapata. Coarse-to-fine decoding for neuralsemantic parsing. arXiv preprint arXiv:1805.04793, 2018. 2[7] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shotvisual imitation learning via meta-learning. In Conferenceon Robot Learning, pages 357–368, 2017. 2[8] A. L. Gaunt, M. Brockschmidt, R. Singh, N. Kushman,
P. Kohli, J. Taylor, and D. Tarlow. Terpret: A probabilistic programming language for program induction. arXiv
preprint arXiv:1608.04428, 2016. 2[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
770–778, 2016. 10[10] D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei,
S. Savarese, and J. C. Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration,
2018. 2[11] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980, 2014. 10[12] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel.
Gated graph sequence neural networks. arXiv preprint
arXiv:1511.05493, 2015. 3[13] H. Mei, M. Bansal, and M. R. Walter. Listen, attend, andwalk: Neural mapping of navigational instructions to action
sequences, 2015. 2[14] J. Merel, Y. Tassa, S. Srinivasan, J. Lemmon, Z. Wang,
G. Wayne, and N. Heess. Learning human behaviors frommotion capture by adversarial imitation. arXiv preprint
arXiv:1707.02201, 2017. 2[15] V. Murali, L. Qi, S. Chaudhuri, and C. Jermaine. Neuralsketch learning for conditional program generation. arXiv
preprint arXiv:1703.05698, 2017. 1, 2
[16] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine. Combining self-supervised learning andimitation for vision-based rope manipulation. In Robotics
and Automation (ICRA), 2017 IEEE International Conference on, pages 2146–2153. IEEE, 2017. 2[17] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, andA. Torralba. Virtualhome: Simulating household activities
via programs. arXiv preprint arXiv:1806.07011, 2018. 1, 2
[18] A. Solar-Lezama. Program synthesis by sketching. Citeseer,
2008. 1, 2, 5
[19] S.-H. Sun, H. Noh, S. Somasundaram, and J. Lim. Neuralprogram synthesis from diverse demonstration videos. In
Proceedings of the 35th International Conference on Machine Learning, 2018. 2[20] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, and N. Roy. Understanding natural language
commands for robotic navigation and mobile manipulation.
In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI’11, pages 1507–1514. AAAI Press,
2011. 2[21] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 5410–5419, 2017. 8[22] Y. Zhu, D. Gordon, E. Kolve, D. Fox, L. Fei-Fei, A. Gupta,
R. Mottaghi, and A. Farhadi. Visual semantic planning using
deep successor representations. In Proceedings of the IEEE
International Conference on Computer Vision, pages 483–
492, 2017. 2

Input(size: 240, 320, 3)

Conv 1+BN
Kernel: 5*5
Stride: 1
Padding: 2
Channel: 16

Conv 2+BN
Kernel: 5*5
Stride: 1
Padding: 2
Channel: 32

Conv 3+BN
Kernel: 5*5
Stride: 3
Padding: 2
Channel: 32

FC
256

Table 5. The model architecture of the CNN that ingests the semantic segmentation map.

# programs
Average length
Average # nodes
Average # edges

VirtualHome
2807
13.26
-

Executable
1387
9.61
289
4393

Augmented
27284
18.58
290
4305

VirtualHome-Env
32761
18.79
289
4319

Table 6. The statistics of VirtualHome-Env. The fact that some
programs in VirtualHome are noisy results in less number of executable programs.

8. Appendices8.1. Implementation details
To train ResActGraph, we use the ground truth graphs extracted from
Unity to represent the environments. We set the propagation time K = 2
except for the ablation studies. To train the model that ingests demonstrations to generate sketches, we use a finetuned Resnet-18 [9] to extract
image features and a 3-layer CNN to extract features from semantic segmentation map. All the GRUs are with 256 hidden units except for thosein GGNN are with 32.
The following is the details of the model ResActGraph, Desc2sketch,and Demo2sketch:
RNN-ResActGraph. The hidden units of GRU is 256 and the hidden
units of GGNN is 32. We use Adam [11] with 3 ⇤ 10 4 as initial learning
rate and mini-batch size 32. Schedule sampling is applied to mitigate theexposure bias. We use linear schedule with max probability 1.0 and minprobability 0.3 and decrease in 20000 steps.
Desc2sketch. We use seq2seq model with GRU. The hidden units of
GRU is 256. We use Adam with 3 ⇤ 10 4 as initial learning rate and minibatch size 64. Schedule sampling is applied to mitigate the exposure bias.
We use linear schedule with max probability 1. and min probability 0. anddecrease in 20000 steps.
Demo2sketch. We finetune Resnet-18 to extract image features andbuild another CNN to ingest the ground truth semantic segmentation maps.
The CNN structure is shown in Table. 5. On the other hand, we use GRU to
decode the sketches. The hidden units of GRU is 256. We use Adam with3⇤10 4 as initial learning rate and mini-batch size 16. Schedule samplingis applied to mitigate the exposure bias. We use linear schedule with max
probability 1. and min probability 0. and decrease in 10000 steps.

8.2. Dataset statistics
We show in statistics of the collected dataset in table 6

8.3. Method for dataset augmentation
We formally describe how we augment the dataset of programs using
exceptions. Let p be the original program and e0 the modified environment.
Let ⇣ be the function that for some exception ✓, takes a program p andenvironment e0 corrects the program into p0 . Finally, let ⌦ be the functionthat takes p, e as the input and outputs its post-conditions4 . We formallydefine the program extension as:
4 A post-condition is a condition or predicate that must always be true
just after the execution of some section of code.

p0 =ef inal =0⇣ (p, e ),
0 0

(11)

(e , p )s.t. 8j 2 ⌦(p), ef inal satisfies j

The algorithm is shown in 1:

8.4. Details of the Evaluation Metrics
We compute the F1 (Ĝ, G) as follows:P(Ĝ, G) =kT(Ĝ)k ⌦ kT(G)kkT(Ĝ)kkT(Ĝ)k ⌦ kT(G)kR(Ĝ, G) =kT(G)kF1 (Ĝ, G) =2 · P(Ĝ, G) · R(Ĝ, G)P(Ĝ, G) + R(Ĝ, G)

where ⌦, P, and R are the binary matching function, precision and recall
respectively. T is a function that converts graphs to tuples.
Parsability refers to the syntactic correctness of the programs, evaluating whether the number of object arguments given the action is valid.
Sleep TV is invalid, since Sleep does not expect any object. Executability refers to the semantic correctness, meaning that 1) the combination of objects and actions is valid (e.g., Grab Bedroom is invalid)and 2) the instruction can be performed under a given environment state(e.g., Grab Mug is only valid if the mug is close).

8.5. Further analysis of the experiments
The performance of the sketch prediction from descriptions is far better
than from demonstrations (shown in Sec. 6.3). We find that only 7% of thetesting vase are more easily predicted from demonstrations than descriptions, especially for the activities pet cat and pick up phone. One potential

Algorithm 1 Pseudo-code of the method described inSec. 5.3. Given a program p with preconditions (p), theprocedure produces at most n new programs.
1: procedure AUGMENT(p, (p), n)2:
perturbed-set
set()
3:
while len(perturbed-set) < n do
. Perturb preconditions
4:zrand(0, 1)
5:(p)0perturb( (p), z)
06:e ⇠ E (p)0
. Sample with eq. 10
7:perturbed-set.add(e0 )8:end while
9:
final-progs
list()
10:p0
p11:
foreach e0
perturbed-set do
12:
iter
1
13:
valid
F alse14:
while not valid and iter < maxiter do
15:
try
16:
(p0 , e0 ). Execute in simulator
17:
final-progs.add(p0 )
18:
valid
T rue
19:
catch ExceptionType
0 0
20:
p0
. Solve with eq. 11
ExceptionType (p , e )21:
iter
iter + 1
22:
end try23:
end while24:
end foreach25:
return final-progs
26: end procedurereason is that the average length of the descriptions of these activities (15.3
words) is longer than those of other activities (13.4).

8.6. Knowledge base
8.6.1

RealEnv

We collected two knowledge bases that aim to set up our environments.
The first one, ObjectStates KB, contains information about which statescorrespond to which objects. We collected 29 different states and 9 ofthem are considered by the simulator. Table 8 shows the collected stateswith an indication of whether they are included in the simulator.
The second knowledge base, ObjectRelations KB, contains typical spatial relations between objects that aims at obtaining realistic environment
layouts to set up the scene for each activity. We show a few examples ofspatial relations in Table 7. In total, it contains 505 types of spatial relations included.

8.6.2

ExceptionHandler

We show the exceptions and subroutines used to augment the dataset in
Table 9. Each row shows the name of the exception, the case when the
exception fires and a description of the subroutine that is called to correct
the given exception.

8.7. Additional Qualitative Results
We now show additional qualitative results of our RNN-ResActGraph.
In Fig. 8 and Fig. 9, we use the ground truth sketches as the input to themodel. Although the generated programs are not exactly the same as theground truth programs, the generated program does achieve similar environment state as the environment state of ground truth.
In Fig. 10, we show how the generated program changes when using
the same sketches but different graphs as input. We highlight the difference
between two environments with orange color. From both cases, we observethat the generated programs achieve almost the same environment stateeven thought they start from different environment states.
In Fig. 11, we show the generated programs from the sketches predicted from descriptions. It shows that sometimes our model cannot generate the exact same program as ground truth, but the generated program does
achieve similar environment state as the environment state of the groundtruth. This is also shown in the main paper where the F1 scores of the
proposed model are higher than others.
Lastly, we show one example of the of our simulators in Fig. 12

States
On/OffPlugged/Unplugged
Open/Closed
Free/Occupied
Dirty/Clean
Full/Empty
GrabbedFlushed
Sharpened
Wet/Dry
Peeled/Sliced/Squeezed
Burning
FrozenFolded/Unfolded
Hot/Cold
Cooked/Raw/Rotten

ExamplesTelevision, Computer, Microwave, Oven, Stereo
Television, Computer, Microwave, Oven, Stereo
Fridge, Cabinet, Folder, Bag, Purse, Blender
Sofa, Table, Sink, Toilet, Piano, Shoe Rack, Bed
Apple, Floor, Bowl, Socks, Plate, Cupboard
Washing machine, Dishwasher, Measuring Cup
Apple, Towel, Book, Glass, Plate, Toy, Cat, Pot
ToiletKnife
Towel, Plate, Glass, Napkin, Shirt, Toothbrush
Banana, Vegetable, Fish, Fruit, Cake
Plate, Pot, Pan, Fish, Water, Candle, Chicken
Water, Vegetable, Ice cream, Wine, Hamburger
Sheets, Shirt, Jacket, Ironing Board, Placemat
Plate, Bowl, Water, Hamburger, Oven, Microwave
Egg, Fish, Vegetable, Meat, Fruit, Pasta, Pizza

In simulator
X
X
X
X
X

Table 7. ObjectStates KB

ObjectWine Glass
Milk
Remote Control
Pot
Razor
Cd Player
ChickenOven mitts
Document
SheetsIn relations
Cupboard, Cabinet, Dishwasher
Fridge, Microwave, Trashcan, Sauce pan
Dresser
Microwave, Cupboard, Cabinet
Cabinet, Trashcan, Bag, Box
Closet, Cabinet, Bag
Fridge, Trashcan, Freezer, Sauce Pan, OvenOven, Cabinet, Basket for clothes
Folderm Garbage Can, Cabinet, Filing Cabinet
Basket for clothes, Dresser, Washing Machine, Cabinet

On relations
Coffee table, Dishrack, Kitchen Counter
Kitchen counter, Table, Pantry
Sofa, Love Seat, Bed, Table, Mousepad
Dishrack, Kitchen Counter, Stove
Table, Bathroom counter
Table, Couch
Kitchen Counter, Cutting Board, Plate
Table, Kitchen counter
Desk, Sofa, Bookshelf, Music Stand
Couch, Love Seat, Bed

Table 8. ObjectRelations KB

Nearby Relations
Dvd player
StereoException name
IS OPENIS CLOSEDSTANDING
SITTINGNOT CLOSENOT FACING
IS ON
IS OFF
PLUGGEDUNPLUGGED
OCCUPIED
INSIDE CLOSEDFREE HAND

When is raised
Character aims to open something that is alreadyopen
Character aims to close something that is closed
1) Character aims to stand up but is not sitting,2) Character aims to sleep but is not lying nor
sitting
1) Character aims to Sit but is sitting, 2) Character aims to walk but is sitting
Character wants to interact with an object but it
is not close enoughCharacter wants to Watch and object (e.g. TV)
but is not facing it.
1) Character switches on an object that is alreadyon 2) Character aims to open a utility (e.g. washing machine) before turning it off
1) Character switches off an object that is off, 2)
character watches TV but it is off
1) Character plugs in an object that is alreadyplugged1) Character plugs out an object that is pluggedout 2) Character switches on an object that isplugged out
There are objects in a surface where the character aims to sit or lie
The character aims to interact an object that is
inside some closed one (e.g. a fridge)
The Character wants to interact with an object
but both of their hands are occupied

Subroutine
Remove the Open instruction
Remove the Close instruction
1) remove Stand Up, 2) add instruction Sit or Lie
1) Remove instruction Sit, 2) Add instruction
Stand up
Add instruction Walk towards the object
Add instruction texitTurnTo the object
1) Remove instruction Turn On 2) Insert an instruction Switch Off.
1) Remove instruction Switch Off, 2) Insert
Switch On
Remove instruction PlugIn1) Remove instruction PlugOut 2) Insert instruction PlugIn
Obtain all the objects in the occupied surface andinsert Grab and Release instructions to remove
them
Obtain the object containing the target object,and open it
Leave one object from the hands, Interact with
object and grab object again

Table 9. Exceptions handled to augment the programs dataset, based on perturbation of the environment

SketchGT Program

Generated Program[Grab] <tooth paste>[Put] <tooth paste> <toothbrush holder>[Switchon] <faucet>[Rinse] <toothbrush>[Walk] <bathroom> (32)
[Walk] <sink> (9)[Find] <toothbrush holder> (63)[Turnto] <toothbrush holder> (63)
[Lookat] <toothbrush holder> (63)[Find] <tooth paste> (15)[Grab] <tooth paste> (15)[Find] <toothbrush> (2)[Grab] <toothbrush> (2)[Walk] <toothbrush holder> (63)[Put] <tooth paste> (15) <toothbrush holder> (63)[Find] <faucet> (70)
[Rinse] <toothbrush> (2)[Walk] <toothbrush holder> (63)[Put] <toothbrush> (2) <toothbrush holder> (63)[Walk] <bathroom> (32)
[Walk] <sink> (9)[Find] <toothbrush holder> (63)[Turnto] <toothbrush holder> (63)
[Lookat] <toothbrush holder> (63)[Find] <tooth paste> (15)[Grab] <tooth paste> (15)[Find] <toothbrush> (53)[Grab] <toothbrush> (53)[Walk] <toothbrush holder> (63)[Put] <tooth paste> (15) <toothbrush holder> (63)[Rinse] <toothbrush> (2)[Walk] <faucet> (70)
[Switchoff] <faucet> (70)

SketchGT Program

Generated Program[Wash] <hands>[Grab] <soap>[Rinse] <hands>[Wipe] <hands>

Faucet (10) is off
Faucet (10) near Sink (93)
Faucet (10) in Bathroom (41)
Soap (77) in Bathroom (41)Towel (8) in Bathroom (41)
Hands (92) near Sink (93)[Walk] <bathroom> (41)
[Walk] <sink> (93)[Find] <sink> (93)[Turnto] <sink> (93)[Find] <faucet> (10)[Switchon] <faucet> (10)[Find] <hands> (92)
[Wash] <hands> (92)[Find] <soap> (77)[Grab] <soap> (77)
[Scrub] <hands> (92)
[Putobjback] <soap> (77)
[Rinse] <hands> (92)
[Switchoff] <faucet> (10)[Find] <towel> (8)[Grab] <towel> (8)[Wipe] <hands> (92)
[Putobjback] <towel> (8)[Walk] <bathroom> (41)
[Walk] <sink> (93)[Find] <sink> (93)[Turnto] <sink> (93)[Find] <faucet> (10)[Switchon] <faucet> (10)[Find] <hands> (92)
[Rinse] <hands> (92)[Find] <soap> (77)[Grab] <soap> (77)[Squeeze] <soap> (77)
[Wash] <hands> (92)
[Rinse] <hands> (92)
[Switchoff] <faucet> (10)[Find] <towel> (8)[Grab] <towel> (8)[Wipe] <hands> (92)
[Putobjback] <towel> (8)SketchGT Program

Generated Program[Grab] <paper towel>[Wipe] <hands>[Walk] <bathroom> (81)[Walk] <paper towel> (54)[Find] <paper towel> (54)[Turnto] <paper towel> (54)
[Lookat] <paper towel> (54)[Grab] <paper towel> (54)[Find] <hands> (4)[Wipe] <hands> (4)[Walk] <bathroom> (81)[Walk] <paper towel> (54)[Find] <paper towel> (54)[Grab] <paper towel> (54)[Find] <hands> (4)[Wipe] <hands> (4)[Putobjback] <paper towel> (54)

Environment
Faucet (70) near Sink (9)Faucet (70) is on
Faucet (70) in Bathroom (32)
Tooth paste (15) near Sink (9)Toothbrush holder (63) in Sink (9)Toothbrush (2) in Bathroom (32)

Environment

Environment
Hands (4) near Paper Towel (54)Paper Towel (54) in Bathroom (81)Figure 8. Predictions from the RNN-ResActGraph from the GT sketch and the environment, along with the GT programs. We color the
longest common subsequence in light green.

SketchGT Program

Generated Program[Putoff] <clothes jacket>[Put] <clothes jacket> <hanger>[Walk] <bedroom> (28)[Walk] <clothes jacket> (19)[Find] <clothes jacket> (19)[Find] <clothes jacket> (19)[Putoff] <clothes jacket> (19)[Grab] <clothes jacket> (19)[Find] <hanger> (18)[Put] <clothes jacket> (19) <hanger> (18)

[Walk] <bedroom> (28)[Walk] <clothes jacket> (19)[Find] <clothes jacket> (19)[Find] <clothes jacket> (19)[Putoff] <clothes jacket> (16)[Grab] <clothes jacket> (16)[Find] <bed> (4)[Put] <clothes jacket> (19) <bed> (4)SketchGT Program

Generated Program[Lie] <bed>[Sleep][Walk] <bedroom> (6)[Walk] <lamp> (72)
[Switchoff] <lamp> (72)[Walk] <bed> (79)[Lie] <bed> (79)[Sleep][Walk] <bedroom> (6)[Walk] <bed> (79)[Find] <lamp> (72)[Lie] <bed> (79)[Sleep]SketchGT Program

Generated Program[Greet] <child>[Greet] <man>[Greet] <woman>[Walk] <living room> (12)[Walk] <child> (67)
[Find] <child> (67)[Greet] <child> (67)
[Find] <man> (48)[Greet] <man> (48)
[Find] <woman> (59)[Greet] <woman> (59)[Walk] <living room> (12)[Walk] <child> (67)
[Find] <child> (67)[Greet] <child> (67)
[Find] <woman> (59)[Greet] <woman> (59)[Find] <man> (48)[Greet] <man> (48)

SketchGT Program

Generated Program[Switchon] <faucet>[Put] <glass> <sink>[Drink] <glass>[Walk] <dining room> (72)[Walk] <sink> (22)[Find] <glass> (99)[Grab] <glass> (99)
[Find] <faucet> (15)
[Switchon] <faucet> (15)[Find] <water> (5)[Put] <glass> (99) <sink> (22)[Grab] <glass> (99)
[Switchoff] <faucet> (15)[Drink] <glass> (99)

[Walk] <dining room> (72)[Walk] <glass> (99)
[Find] <glass> (99)[Grab] <glass> (99)
[Find] <faucet> (15)
[Switchon] <faucet> (15)[Find] <water> (5)[Find] <sink> (22)[Put] <glass> (99) <sink> (22)[Grab] <glass> (99)
[Switchoff] <faucet> (15)[Drink] <glass> (99)

Environment
Hanger (18) near Clothes Jacket (19)Hanger (18) in Bedroom (28)
Bed (4) in Bedroom (28)
Clothes Jacket (19) in Character

Environment
Lamp (72) is on
Lamp (72) near Bed (79)
Bed (79) in Bedroom (6)Bed (79) is free

Environment
Man (48) near Child (67)
Woman (59) near Child (67)Man (48) in Living Room (12)Environment
Faucet (13) near Sink (68)
Faucet (13) in Dining Room (72)
Water (8) near Sink (68)
Glass (99) in Sink (68)Figure 9. Predictions from the RNN-ResActGraph from the GT sketch and the environment, along with the GT programs. We color the
longest common subsequence in light green.

[open] <washing_machine>[putback] <basket> <washing_machine>[putback] <soap> <washing_machine>[switchon] <washing_machine>

SketchEnvironment 1

Generated Program 1

Environment 2

Generated Program 2

Soap (1002) near washingmachine (1001)
Soap (1002) inside washingmachine (1001)
Washing machine (1001) is
openedWashing machine (1001) is on
Washing machine (1001) isplugged[Walk] <bedroom> (273)[Walk] <basket> (1000)[Find] <basket> (1000)[Grab] <basket> (1000)[Find] <washing_machine>(1001)[SwitchOff] <washing_machine>(1001)[Open] <washing_machine>(1001)[Put] <basket> (1000)<washing_machine> (1001)[Find] <soap> (1002)[Grab] <soap> (1002)[Put] <soap> (1002)<washing_machine> (1001)[Close] <washing_machine>(1001)[SwitchOn] <washing_machine>(1001)

Soap (1002) near washingmachine (1001)
Washing machine (1001) isclosedWashing machine (1001) is off
Washing machine (1001) is
unplugged[Walk] <bedroom> (273)[Walk] <basket> (1000)[Find] <basket> (1000)[Grab] <basket> (1000)[Find] <washing_machine> (1001)[Open] <washing_machine> (1001)[Put] <basket> (1000) <washing_machine>(1001)[Find] <soap> (1002)[Grab] <soap> (1002)[Put] <soap> (1002) <washing_machine>(1001)[Close] <washing_machine> (1001)[Plugin] <washing_machine> (1001)[SwitchOn] <washing_machine> (1001)[putback] <groceries> <fridge>

SketchEnvironment 1

Generated Program 1

Environment 2

Generated Program 2

Grocery (1001) near fridge (1000)
Fridge (1000) in dining room (47)Grocery (1001) in dining room (47)
Fridge (1000) is closed[Walk] <dining_room> (47)
[Walk] <fridge> (1000)[Find] <fridge> (1000)[Open] <fridge> (1000)[Find] <groceries> (1001)[Grab] <groceries> (1001)[Put] <groceries> (1001)<fridge> (1000)[Close] <fridge> (1000)

Grocery (1001) near fridge (1000)
Fridge (1000) in dining room (47)Grocery (1001) in dining room (47)
Fridge (1000) is opened[Walk] <dining_room> (47)
[Walk] <fridge> (1000)[Find] <fridge> (1000)[Find] <groceries> (1001)[Grab] <groceries> (1001)[Put] <groceries> (1001) <fridge> (1000)[Close] <fridge> (1000)

Figure 10. Predictions from the RNN-ResActGraph model from the same sketches but different environments. We color the difference
between environments in orange and the longest common subsequence in light green.

Description

pick up book. sit on sofa. open
book. read

Description

turn on reading light. bring book to couch. sit on couch. <unk>pillow. put pillow <unk> my back. read my book.

Sketch[sit] <chair>, [read] <book>

Sketch[sit] <sofa>, [read] <book>

GT Program [walk] <living_room> (21)[walk] <book> (1001)[find] <book> (1001)[grab] <book> (1001)[find] <chair> (1002)[sit] <chair> (1002)[read] <book> (1001)

Generated
Program[walk] <living_room> (21)[walk] <book> (1001)[find] <book> (1001)[grab] <book> (1001)[find] <chair> (1002)[sit] <chair> (1002)[read] <book> (1001)

GT Program [walk] <bedroom> (276)[walk] <book> (1001)[find] <book> (1001)[grab] <book> (1001)[find] <sofa> (201)[sit] <sofa> (201)[standup][walk] <bookmark> (1004)[find] <bookmark> (1004)[read] <book> (1001)

Generated
Program[walk] <bedroom> (276)[walk] <book> (1001)[find] <book> (1001)[grab] <book> (1001)[find] <sofa> (201)[sit] <sofa> (201)[read] <book> (1001)

Figure 11. Predictions from the RNN-ResActGraph from the predicted sketch (from descriptions) and the environment, along with the GT
programs. We color the longest common subsequence in light green.

Start

Walk office

Walk chair

Sit Chair

Grab mouse

Grab keyboard

Figure 12. An example of snapshots generated in VirtualHome for a given program.