ProTo: Program-Guided Transformer for Program-Guided Tasks
Authors: Zelin Zhao
Karan Samel
Binghong Chen
Le Song
Venue: NeurIPS 2021 Poster (2021)
Keywords: program-guided tasks
learning to execute
program-guided agent
neural symbolic reasoning

Abstract: Programs, consisting of semantic and structural information, play an important role in the communication between humans and agents. Towards learning general program executors to unify perception, reasoning, and decision making, we formulate program-guided tasks which require learning to execute a given program on the observed task specification. Furthermore, we propose Program-Guided Transformers (ProTo), which integrates both semantic and structural guidance of a program by leveraging cross-attention and masked self-attention to pass messages between the specification and routines in the program. ProTo executes a program in a learned latent space and enjoys stronger representation ability than previous neural-symbolic approaches. We demonstrate that ProTo significantly outperforms the previous state-of-the-art methods on GQA visual reasoning and 2D Minecraft policy learning datasets. Additionally, ProTo demonstrates better generalization to unseen, complex, and human-written programs.

ProTo: Program-Guided Transformer for Program-Guided Tasks
Zelin Zhao"
Karan Samel
The Chinese University of Hong Kong zelin@link cuhk. edu. hk
Georgia Institute of Technology ksamel waaneeh.gaecc....mmm edu
Binghong Chen Georgia Institute of Technology binghongegatech. edu
Le Song
Biomap and MBZUAI dasongle@gmail. com
Abstract
Programs, consisting of semantic and structural information, play an important role in the communication between humans and agents. Towards learning general program executors to unify perception, reasoning, and decision making, we for- mulate program-guidee guided tasks which require learning execute given program on the observed task specification. Furthermore, we propose Program-Guided Transformer (ProTo), which integrates both semantic and structural guidance of program by leveraging cross-altention and masked self-attention to pass messages between the specification and routines the program. ProTo executes program in a learned latent space and enjoys stronger representation ability than previous symrolic approaches. We demonstrate that ProTo significantly outperforms the previous state-of-the-art methods on GQA visual reasoning and 2D Minecraft policy learning datasets. Additionally, ProTTo demonstrates better generalization unseen, complex, and human-writte programs.
Introduction
Programs are the natural interface for the communication between machines and humans In comparison instructing machines via demonstrations 41 52
via natural language guiding agents by programs has multiple benefits. First, programs are explicit and much cleaner than other instructions such languages [8] Second, programs are structured with loops and branches | so they can express complex reasoning processes [99] Finally, programs are compositional, promoting the generalization and scalability of neural models [15
14
32 3
20) However,
while program synthesis and program induction have been deeply explored [1
18 22 39 49).
very few works focuses on learning follow program guidance [8] ZT]] Furthermore, previous work designs ad-hoe program executors for different functions in different tasks hinders the generalization and scalability of developed models.
29 23
which
To pursue general program executors to unify perception, reasoning, and decision making, we formulate program-guied tasks, which require the agent follow the given program perform tasks conditioned on task specifications. The programs may come from a program synthesis model [ or be written by human [8] Two exemplar tasks shown in Figure T Program-guided tasks are challenging because the agent needs to jointly follow the complex structure of the program [
perceive the specification, and ground the program semantics on the specification 40)
Inspired by the recent significant advance of transformers in diverse domains [24
25 59
we present
the Program- guuided Transformer (ProTo) for general program-gut -uidedd tasks. ProTo combines the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Task
Task 2
Program-guided Visual Reasoning
Program-guided Policy Learning
If IsThere(river):
Filter(gir)
Mine(wood)
Relate(left)
BuildBridge(
Select(table)
Else:
Relate(on)
Goto(4, 2)
Filter(fries)
While Env[Gold] > O:
Query_color()
Mine(Gold)
Specification
Program
Specification
Program
Goal: reason according to the program
Goal: leam a policy to follow the program
Figure 1: Illustration of two exemplar program -uuuddd tasks. The first task is program gegramm-uuiedd visual reasoning [47], where the model needs learn execute visual program on the specification (image) to get predicted answer. The second task is program-guides policy learning [8I] where agent learns a policy perform tasks based on the observed specification following the program guidance.
strong representation ability of transformers with symbolic program control flow. We propose to separately leverage the program semantics and the explicit structure of the given program via efficient attention mechanisms. ProTo enjoys strong representation ability by executing the program a learned latent space. In addition, ProTo either learn from reward signals execution supervision [15].
81 or from dense
We evaluate ProTo on two tasks, program-guided visual reasoning and program-guided policy learning (corresponding to Figure Tleft and Figure riihh).. The former task requires the model to learn to reason given image following program guidance. We experiment the GQA dataset 47 where the programs are translated from natural language questions via a trained program synthesis model. The evaluation on the public test server shows that we outperform the previous state-of-the-an program- based method by 4.31% in terms of accuracy. Our generalization experiments show that ProTo capable of following human program guidance. The latter task asks the agent learn multi-task policy to interact with the environment according to the program. We experiment on a 2D Minecraft environment M where the programs randomly sampled from the domain-specific language (DSL). We find that ProTo has a stronger ability to scale to long and complex programs than previous methods. ProTo significantly outperforms the vanilla transformer [87] which does explicitly leverage the structural guidance of the program. We will release the code and pre-trained models after publishing.
In summary, our contributions are threefold. First, we formulate and highlight program -uiddedg tasks, which are generalized from program -uiddedd visual reasoning 199 64] and program guindddggddddddde policy learning [81] Second, we propose the program-guice guided transformer for program- program-guided -uidedd tasks. Third, conduct extensive experiments two domains and show promising results towards solving program program-guided tasks.
Related Work
Program Induction, Synthesis and Interpretation Program induction methods learn an underly- ing input-output mapping from given examples [54
50, and no programs are explicitly predicted. Differently. program synthesis targets at predicting symbolic programs from task specifications
22
23
12
18
17
Gll
However, these approaches do not learn to execute domain digit and string operation (adding, copying, sorting, etc.), Neural Program learn to compose lower-level programs to express higher-level programs use attentional processes with external memory infer
27l
93
85
16
80 58 86
programs.
Interpreter (NPI)
7
92
75
37l while Neural Turing Machines
551
simple algorithms. Recently,
[
proposes to guide the agent in 2D Minecraft via programs. 65

---

detects repeated entries in the image and uses programs to guide image manipulation. They neither attempt to formulate general moogram-guides tasks nor propose unified model for different tasks.
Visual Reasoning Visual reasoning requires joint modeling of natural language and vision. A typical visual reasoning task is the visual question answering (VQA)
[3] Attention mechanisms have
been widely used in the offte-of-the--eee art VQA models [60] 2
74,
104 4d
72). Neural module networks
and the eeural-symbolic approaches 19 64] propose to infer programs from natural languages and execute programs on visual contents to get the answer. A large-scale dataset GQA A77, which addresses the low diversity problem in the synthetic CLEVR dataset 48 is proposed to evaluate the state-of-the-arr visual reasoning models. Neural state machine
45 is proposed by leveraging modular reasoning over a probabilistic graph, which does not leverage the symbolic program. Our model is similar to the concurrent work, meta module network [15]] in that we use shared parameters for different function executors. Nevertheless, we propose a novel attention- architecture and extend our model to the policy learning domain.
Transformer Transformer was firstly proposed machine translation After that, transformers demonstrate their power on many different domains, such as cross- modsl reasoning 83]
large-scale
unsupervised pretraining [4 93 and multi-task representation learning 77C 67
Recently,
transformers appear to be competitive models in many fundamental vision tasks, such as image
classification 25 101 (6), object detection [3 and segmentation (50) [R], While we share the same belief that transformers are strong, unified, elegant models for various deep learning tasks, we propose novel transformer architecture for program program-guides guided tasks. We discuss a few similar unpublished works in Appendix A
Policy Learning With Programs Previous policy learning literature explored the benefits of pro- grams in different aspects. First, some pre-defined routines containing human prior knowledge could help reinforcement learning, and planning (30 68
[00). Second, programs enable interpretable and verifiable reinforcement learning [9 Our approach follows recent line of work that learns to interpret program guidance [8 4] However, we adopt a novel, simple and uniform architecture that demonstrates superior performance, thus conceptually related to multitask reinforcement learning
84 92 and hierarchical reinforcement learning
3 Program-guided Tasks
Unlike natural language that is flexible, complex, and noisy, programs are structured, clean and formal لл. Therefore, programs can serve as a powerful intermediate representation for human- computer interactions [I]. Different from previous work that learss to synthesis programs from data 09] 7 we study learning to execute given programs [8]] based on three motivations. First, instructing machines via explicit programs instead of noisy natural languages enjoys better efficiency, and accuracy 81. Second, learned program executors have stronger representation ability than hard- coded ones 199 15 Third, we attempt to develop unified model to integrate perception, reasoning, and decision by learning to execute, which heads towards core direction in the neural-symbolic Al 36 35
A program-guse guided task is specified by tuple (S where S is the space of specifications, T denotes the program space formed by domain-specific language (DSL) for a task, O is the space for execution results, and G is the goal function for the task. For each instance of the program-gun -uided lask, a program P E I' is given. An executor & is required by the task to execute the program on the observed specification. Note that different from hard-coded non- parameeerii executors used in some previous work [99] 2 the executor Eg is parametrized by $. For convenience, we define a routine to be the minimum execution unit of a program (e.g. Filter (girl) in Figure Tis routine). We denote Pt: to be the k-th routine in P and [P] to be the number of routines in P. According to the DSL., the program should begin at the entrance routine and finish at one of the exiting routines (e.g. svery_colore
Filter (girl) in Figure is an entrance routine and Query color () is an exiting routine). At each execution step т, the executor executes the current routine Pp E P on the specification s(r) E and produces an output (T)) O. The execution of P finishes if one of the exiting routines finishes execution. The goal of a program-guided task to produce desired execution results to achieve the goal G.
3

---

Algorithm 1: Pro'To Execution
Algorithm 2: UpdatePointer(p. olt), Pp)
Result:
Execution results {o!T)
J==1
Pp is an If-routine then
Initialize 7 = and the pointer p;
Point p to the first routine in the T/F branch if o!T) TIF;
Build P according to Eq. 0
3 while not reach an exiting routine do
else if P. is Whv Le-routne then Point p to the first routine
Observe 8(7) and set T
Build pd(r) according to Eq.
inside/outside the loop if ol is T/F;
alt) = ProTolnferrss P P ,p);
5 else if Pp ends a loop then Point p to the loop condition routine.
8
if Pp. finishes execution then
7 else
UpdatePointer(y, o(T), pp)
Point p to the subsequent routine;
10 end
end
4 Program-guided Transformer
A good model for program- guided task should fulfiil the following merits. First of all, it should leverage both the semantics and structures provided by the program (refer to Figure 2 Second, it would better be multi-task architecture with shared parameters for different routines ensure efficiency and generalization [5 90 70) Third, it would be better if the model does not leverage external memory for the efficiency of batched training [4] To these ends, present the Program- guided Transformer (ProTo). ProTo treats routines as objects and leverages attention mechanisms to model interactions among routines and specifications. The execution process of ProTo is presented in Algorithm| ! In each execution timestep, ProTo leverages the semantic guidance ps and the current structure guidance p((r) (Line 2&5 in Algorithm I detailed Sec 4.1 and infers for one detailed in Sec
4.2). When routine finishes execution, pointer p E (1,2,... 2, ...,. |P] indicating the current routine Pp (Pp is the pth routine in P) would be updated
step (Line 6 in Algorithm D
(Line 8&9 in Algorithm detailed in Algorithm 2 and Sec 4.3 The results of ProTo are denoted
as {o(T))T_, where T is the total number of execution timesteps. ProTo supports several training
schemes, which are listed in Sec 4.4
4.1 Disentangled Program Representation
As shown in the left of Figure 2
we leverage both the semantics and structure of the program in ProTo. The semantic part of the program Pa E P|xl is an embedding matrix for all routines where R is the real coordinate space and d is hyperparameter specifying the feature dimension. Note P remains the same for all execution timesteps. Denote { {w to be the words in Ps where L is the maximum number of words in one routine (padding words are added if Ps does have L words), we construct the k-th row of Pa corresponding to Ps via a concatenation of all token embeddings:
P WordFimbed(w!?
where || represents the concatenation function and WordEmbed maps token to embedding with dimension dm where we set d = x dm-
The structure part of the program p*(r) E RIP is transition mask calculated at each execution timestep T to pass messages from the previous execution timestep to the current exccution timestep. Although most types of the routines only need to get information from the previously executed routines, some logical routines such as Compare_Coo caaree_Coood () rely on more than one routines' results. We denote Parents(P,) to be the set of routines whose results would be taken as input by the current routine Pp. We can easily derive Parents(Pp) (pp) from the program P, which is detailed in the Appendix C.1 The transition mask p((r) defined by the following equation:
0 if (P3 E (P,e and P; = Pω)) or ((=
(2)
-00 else,
where we set diagonal elements of (*(r) to zeros to preserve each routine's self information to the next timestep. Positional encoding is added following the standard transformer [87]

---

Disentangled Program Representation
ProTo Inference
(Select(bag)
@Select(wine)
Last Result Matrix
Program
@Verift_relationn(ef))
Semantic Guidence
Cross-Attention
Select
bag
Layer Norm
Structure Guidence
Guided Self-attention
Select
wine
T=2
wn
Layer Norm
Verify_reeation
left
Specification
Cross-Attention
T=8
New Result Matrix
Semantic Guidenee
Structure Guidance
Output
Main components of prngram-guidee Transformer (ProTTo). Left: we propose a disentangled program representation leverage both semantics and structures of the program. The shown program selects the bag and the wine from the image and verifies whether the bag is on the left to the wine. Right: given structure and semantic guidance offered by a program, ProTo updates the result matrix leveraging the program guidance and the specification.
Figure 2:
4.2 ProTo Inference
Given the disentangled program guidance P p*(T)) and the current observed specification s(r) Naxd where N₂ is the number of objects in the specification, ProTo infers the execution result o(r) via stacked attention blocks as shown on the right of Figure 2 which is presented in detail in this subsection. Note we assume the specification represented in an object-centric manner, which might be obtained via pre-trained object detector
2
or image patch encoder
25
ProTo maintains a result embedding matrix Z E routines. The result embedding matrix is initialized as zero matrix and is updated at each timestep. During Pro inference, first conduct cross-attention to inject semantic information of routines to the hidden results via
RR Pixd which stores latent execution results all
show the architecture of transformer executors the right of Figure 2
ZP
H CrossAtt; (Z, P*) softmax
P
Vd
where H1 P is the first intermediate latent results.
propagate the information of the previous results to the current routine. The masked self-attention leverages the transition mask to restrict the product of queries and keys before softmax (queries, keys and values are the same in the self-attention). Formally, acquire the second intermediate results H2 RIP[xd by
We propose adopt masked self-attention
H,H, + H
H2 MaskedSelfAt(HL, p*()) = softmax
(4)
Vd
After that, we apply cross-attention again to update results based on the specification (F);
Hzs(7)T (T())
Z (rossAAtt(H2,555 softmax
(5)
Vd
The above equations for self-attention and cross-attention show only one head of self-attention and coos-attention for simplicity. However, experiments, we use multi-head attention with eight heads 87/. The resulting embedding corresponding to the pointed routine Zp would be decoded via an MLP to produce explicit output (T7)) MLP(ZZ)).

---

4.3 Pointer Update
After each execution timestep, we would update the pointer p if P. finishes execution. Note that some routines cannot be finished in execution timestep (e.g.. the routine Mine (gold) requires the agent to navigate to the gold and then take Mine action). We know the execution of Pp is finished if the execution result ol7) meets the ending condition of Pp (e.g. when ofr) is the Mine action and the agent successfully mines a gold, we judge that Mine (gold) is finished). A full list of ending
conditions for all routines is in the Appendix B
The pointer p is updated according to the control flow 18 movement is determined by the execution result o(r) the type of Pp. and the location of Pp- We outline the detailed procedure of pointer update in Algorithm
81 of the program. The pointer's
Parallel Execution Transformer has one powerful ability that it can conduct sequential prediction in parallel [87 In our case, we can execute routines that do not have result dependency in parallel. We only need to modify the masked self-attention in Eq. 4 to support passing multiple routines' messages in parallel. Furthermore, multiple pointers would be adopted and updated in parallel, while multiple results would be output in parallel in execution timestep. We detail the paralleled version of Al Algorithm|| | and Algorithm 2 in the Appendix C.2
4.4 ProTo Training Targets
After deriving the exccution results of all routines denoted via (o( ProTo can be trained via the following three types of training targets. (1) Dense Supervision LD. When the ground truth known, we can use a dense loss LD. which the L2 distance between {o(r))²=1 and (T-))==-- (2) Partial Supervision Lp. Knowing the final execution result of the whole program o(T) ProTo can be learned through partial supervision Cp which measures the L2 distance between oT and GTT), (3) RL Target CR. When a program is successfully executed, the environment gives the agent sparse reward of +1. Otherwise, the agent gets a zero reward. In this case, ProTo can be optimized via reinforcement learning loss CR [51 81 In experiments, we follow the same type of supervision as the corresponding baselines, which varies from task to task.
of all execution results of all routines {o
5 Experiments
5.1 Program-guided Visual Reasoning
Task Description Program-guides visual reasoning requires the agent to follow program guidance to reason about an image. It is formed by tuple (S", r", O" G") where v denotes the visual reasoning task. One specification s E S is the object-centric representation of an image formed via a pre-trained object detector 2. which is not updated from time time. In other words, g(r)
((++1) holds for each timestep r. The design of the program space I follows the previous work 4v The output set O" is the possible results for all routines for all the programs (all types of results are encoded to fixed-length vectors explained in Appendix D.I. We define answer to a program P to be the final execution result of the program P, and the goal of program- guided visual reasoning g" is predict the correct answer to the program. As for the training target, we adopt the dense supervision Lp as described in Sec 4.4 to train ProTo following previous approaches 15 42 [6]
Dataset Setup
We conduct experiments of program-guidee visual reasoning based on the public GQA dataset [4] consisting of 22 million questions over 140 thousand images. It is divided into training, validation, and testing splits. The ground true answers, programs, and scene graphs are provided in the training and validation split but not in the test split. We use the provided balanced training split to control data bias. On the training split, we train transformer- based seq2seq model 87 to parse a question into program. For validation and testing, we use this trained seq2seq model to acquire a program from a question Besides answer accuracy, the GQA dataset Air offers three more metrics evaluating the consistency, validity, and plausibility of learned models.
we found that simple seq2seq model can achieve 98.1 validation accuracy to convert natural language question into program. The concurrent work [15] also found a similar fact.
In the GQA dataset 4

---

Table Comparison of ProTo with previous models on the test2019 split of the GQA dataset. In the collum of training Signal, we use QA. SG, Prog to denote question-ans ansterr pairs, scene graphs, and programs. As for the metrics, Cons., Plaus., Valid., Distr., Acc. represent consistency, plausibility, validity, distribution, and accuracy correspondingly.
Model
Signal
Binary
Open
Cons.
Plaus.
Valid.
Distr.
Acc.
Human
47
91.20
87.40
98.40
97.20
98.90
89.30
Bottom Up
QA
66.64
34.83
78.71
84.57
96.18
5.98
49.74
MAC
44
QA
71.23
38.91
81.59
84.48
96.16
5.34
54.06
LXMERT
QA
77.16
45.47
89.59
84.53
96.35
5.69
60.33
NSM 45
SG
78.94
49.25
93.25
84.28
96.41
3.71
63.17
PVR 66
Prog 78.02
43.75
91.43
84.77
96.50
6.00
59.81
SNMN
Prog 73,40
40.82
85.11
84.79
96.37
5.14
56.09
MMN 15
Prog 78.90
44.89
92.49
84.55
96.19
5.54
60.83
ProTo
Prog 79.12
51.45
93.45
86.12
96.52
3.66
65.14
Table 2: Accuracy of the ablation study on the validation split of the GQA dataset.
Table 3: Results of the generalization experiments on the validation split of GQA dataset.
Model
Acc.
Generalization
Model
Acc.
Pro'To Full Model
64.47
MMN [5]
59.47
Human Program Guidance
ProTo
70.33
No Structure Guidance
55.16
No Semantic Guidance
33.82
MMN 15
61.88
Unseen Programs
GAT Encoding
58.58
ProTo
65.34
Partial Supervision
60.28
MMN 15
52.39
Restricted Data Regime
NS- NS-VQA 1971
29.57
ProTo
58.31
IPA-GNN 10
47.10
MMN 15
60.40
Experiment Details We take N = 50 object features (provided by the GQA dataset) with d 2048 dimension. The optimizer is BERT Adam optimizer [24] with base learning rate x 10-4, which decayed by factor of 0.3 every epoch. To alleviate over-fitting, we adopt an L2 weight decay of 0.01. The model is trained for 20 epochs on the training split, and the best model evaluated on the validation split is submitted to the public evaluation server to get testing results. The testing results of baselines are taken from the corresponding published literature [
45 15
the leaderboard.
Following 15 45
we do not list unpublished methods or methods leveraging external datasets.
We present the results on the testing split of GQA comparing to previous baselines in Table ProTT surpasses all previous baselines considerable margin, successfully demonstrat- ing the effectiveness of leverage program guidance in reasoning. The superior performance of ours over the concurrent work meta module network (MMN) 15 reveals ProTo's 's strong modeling ability. More visualizations are in Appendix D.I
Testing Results
Ablation Study We ablate our full model to the following variants to study the importance of different components. Despite the described changes, other parts remain the same with the full model. (1) No Structure Guidance. Only the semantic guidance is used, and p((r) is set to the all-zeros matrix in Eq (2) No Semantic Guidance. The semantic guidance P is set to the all -elzzros matrix to disable semantics guidance. (3) GAT Encoding. We use graph attention networks [88] to encode the program and fuse the program feature with result embedding matrix and specification feature via two consecutive cross attention modules (refer Appendix D. for details). (4) Partial Supervision. We only supervise the predicted answer to the program but do not give dense intermediate supervision (refer 4.4)
Table shows the results, and the validation accuracy of two program-based baselines is listed for reference. The results demonstrate that both structure guidance and semantic guidance contribute significantly to the overall performance. ProTo also better than the GNN baseline because of the strong cross-mit modal representation learning ability of the transformer
43
And the extremely
7

---

low validation accuracy of NS- VQA 19, which is reported by its authors in 1971, reveals that the hard-coded program executors are not as powerful as the learned transformer executors.
Generalization Experiments We conduct systematical experiments to evaluate whether humans can guide the reasoning process via programs. More details are in the Appendix [D.]] (1) Human Program Guidance. We test whether humans can guide the reasoning process via programs on a collected CCQA-Human-Proorram dataset. We ask volunteers write 500 programs and corresponding answers on 500 random pictures taken from the GQA validation split. No natural language questions are collected. All the models are trained on the training split of GQA and tested on the GQA- Program dataset. (2) Unseen Programs, Following [5 we remove from the training split all the programs containing the function verify_ _hape, and we evaluate the models on the instances containing verify_shaae on the validation split. (3) Restricted Data Regime, We restrict the models only to use 10% uniformly sampled training data test the data efficiency of models.
Results are presented on Table E We found that ProTo can successfully generalize its learned program execution ability to human written programs, surpassing the previous state-of-the-art neural module network approach by over 10 points. ProTo can also generalize to unseen programs, again verifying the compositional generalization ability of our neural-symbolic transformer model 22 20 Besides, ProTo is more data-efficient than MMN (.5. We also found that ProTTo is much more effective than the recent learning-to-exen -o-exccute approach IPA-GNN
5.2 Program-guided Policy Learning
Task Description Program-guided policy learning requires the agent to learn a policy perform tasks following a given program [I We denote this task as a tuple (8", r", O", g") where p stands for the policy learning task. Since we experimenting environment, the specification 8 E S is the feature embeddings of Ng objects placed in N₂
0.8
grid-world
Reward
0.7
0.6
grids. Unlike the visual reasoning task, the spec- ification is updated by the environment each timestep r after the agent takes an action. The design of the program space I follows [Р]] The output space OP consisting of several types: (I) Boolean results (True or False); (2) Motor Actions (e.g. Up); (3) Interactive Actions (e.g. Mine and Build). Note the agent can only in- teract with the grid it stands on. In this task, the agent should learn from reward signal L (described in Sec 4.4 while we also experiment applying the dense supervision. The goal GD is to maximize the task completion rates
9.5
ProTo(OURS)
PGA
0.3
Transformer
0.2
TreeRNN
Environment Steps
Figure 3: Training curves on the 2D Minecraft environments in comparison to several baselines. Both mean and standard errors over ten random agents are shown the figure.
Experiment Details Following 77 we conduct experiments on a 2D Minecraft Environmen? Programs are sampled from the DSL and divided into training and testing splits (4000 for training and 500 for testing). For each instance, the agent must follow a given program to navigate the grid world, mine resources, sell mined resources from its inventory, or place marks. The baselines include Program- guided Agent (PGA) Σ8), the naïve Transformer [87] and TreeRNN [22. PGA separately learns perception and policy modules. Transformer and TreeRNN encode the input program in
token by token manner and output an action distribution. We ensure that the number of parameters for different methods is comparable. We use the same manner of encoding the objects in the grid into features as [81), which are projected to d-dimension features via an MLP. The policy is optimized via the actor-critic (A2C) algorithm [51], and we use the same policy learning hyperparameters with PGA 11 which are detailed in the Appendix D.2 for reference. When using dense supervision, the ground-trus execution traces come from hard-coded planner. More details are in the Appendix D.2
Training Curves, Testing Results and Visualization We first show the training curves under the RL. target in Figure We observe that ProTo surpasses Program-guided Agent (PGA) [8].
The implementation of the environment, the dataset, and the baselines are provided by the authors of g1

---

Table 4: Task completion rates on 2D MineCraft under different test settings. We repeat each experiments on 10 different random seeds and we show both averaged rates and their standard deviations. Baseline numbers are slightly different from [81] due to random noises.
Supervision Model
RL Target
Dense Supervision ProTTo
Transformer 87
PGA 81
ProTo
PGA [1
Standard Testing
50,1
94.2110.8
97.312.1
96.910099 92.1+0.77
99.1
Longer Programs
41 41.2+33.5
86.1+-0.9
91.2++3.3
94.4+10.8
Complex Programs
4..
89.7++0.3
95.0++2.5
91.2++0.5
96.3++100
Df Agent[Golda 3:
Right
Left
Right Rigght Right Bridge
Mine
Else:
While Env[Gold] >
F
(ioe(Gold)
BuildBridge0
(4) Program
Execution Results Specifications
Figure An example result on the 2D Minecraft environment. The given program shown in (a). Each timestep's execution results and specifications are shown in (b) top and (b) bottom, respectively. Due to space limitations, we only show some specifications along the timeline, and the arrows in (b) denote the places of the specifications in the timeline.
which demonstrates the power leveraging disentangled program guidance transformers. The fact that ProTo outperforms the vanilla end-to-end transformer by large margin demonstrates the effectiveness of explicitly leveraging program structure guidance.
We test the trained agent in different settings. Despite the Standard Testing split offered by 81. we sample two more splits from the DSL while ensuring the testing cases are not seen in the training split: Longer Programs and Complex Programs. All programs in Longer Programs contain more than eighty tokens, while all programs in Complex Programs include more than four Furthermore, we add execution losses on the training split and test the baseline method PGA and our method. The results on test splits are shown Table We find that ProTo performs better than PGA 8 all testing settings. ProTo scales better than PGA to longer and complex programs, demonstrating the strong ability ProTo to leverage the program structure. We also observe that ProTo has superior performance when dense execution supervision is provided. A demonstration of the test split is provided in Figure 4 where we observe ProTo successfully learns to execute the program and develops good policy to follow the program guidance.
While tokens.
6 Conclusion and Future Work
In this paper, we formulated program-guided tasks, which asked the agent to learn to interpret and execute the given program on observed specifications. We presented the Program-guides Transformer (ProTo), which addressed program-gutee guidedd tasks by executing the program in hidden space. ProTo provides new state-of-the-ard performance versus previous datasct--peciffc methods in program- guided visual reasoning and program-gune guided policy learning.
Our work suggests multiple research directions. First, it's intriguing explore more advanced and program-guided
challenging program guided tasks, such as program -uided embodied reasoning (1] and program- guided robotic applications [69]
[[2].. Second, we are learning separate parameters for different tasks, while building general and powerful program executors across tasks with shared
96
95
parameters very promising 44
92
90 Additionally, improving transformer executors with more
hierarchy 599 and better efficiency
100
is a meaningful future direction.

---

7 Acknowledgement
We thank Shao-Hua Sun for sharing his codes. This work is supported in part by the DARPA LwLL Program (contract 1.8750-19-2-0201
10

---

References
[1] Frances E Allen. Control flow analysis. ACM Sigplan Notices, 5(7):1-19, 1970.
[2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6077 7676, 2018.
[3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and- ann-language navigation: Interpreting visually-grounde grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
[4] David Andre and Stuart J Russell. Programmable reinforcement learning agents. PhD thesis, University of Califoria, Berkeley, 2003.
[5] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision. pp. 2425 2433, 2015.
[6] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot lcarning from demonstration. Robotics and autonomous systems, (7(5):469-488 2009.
[7] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1):41-77,200 2003.
[8] Osbert Bastani, Yewen Pu, and Armando Golam-Lezama. Verifiable reinforcement learning via policy extraction. arXiv preprint 18XYv:8805.0888 2018
[9] Tarek R. Besold, Artur S. d' Avila Garcez, Sebastian Bader, Howard Bowman, Pedro M. Domingos. Pascal Hitzler, Kai-Uwe Kühnberger, Luis C. Lamb, Daniel Lowd, Priscila Machado Vieira Lima, Leo de Penning, Gadi Pinkas, Hoifung Poon, and Gerson Zaverucha. -eural-symbolic learning and reasoning: A survey and interpretation. CoRR, abs/171110090 2017. URL httpp//arxiv org/ /a/abb//171 1711
03902
[10] David Bieber. Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute programs with instruction pointer attention graph neural networks. CoRR, abs/201001162)) 2020. URL URiphttps!
//arxiv. wrgg/bs////0oooo/oo 12621
[11] Paul Booth. An introduction to human compntcc interaction (psychology revivals). Psychology Press, 2014.
[12] Rudy Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. arXiv preprint 1805.0605 2018.
[13] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
[14] David Chen and Raymond Mooney, Learning to interpret natural language navigation instructions from observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, 201
[15] Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu, Meta module network for compositional visual reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 355-664,22 2021.
[16] Xinyun Chen, Chang Liu, Richard Shin, Dawn Song, and Mingcheng Chen. Latent attention for if-then program synthesis. arXiv preprint 161 1,01667, 2016.
[17) Xinyun Chen, Chang Liu, and Dawn Song. Towards synthesizing complex programs from input-output examples. arXiv preprint arXiv:17066 777700002444 2017.
[18] Xinyun Chen, Chang Liu, and Dawn Song. Execution-gue guided neural program synthesis. In International Conference on Learning Representations, 2018
[19] Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference Learning Representations, 2019.
15

---

[20] Xinyun Chen, Chen Liang. Adams Wei Yu, Dawn Song, and Denny Zhou. Compositional generalization via neural-symbolic stack machines. arXiv preprint WxXiv:208806662,, 2020.
[21] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em- boliiedqustionnnsvveiing In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-10, 2018.
[22] Jacob Devlin, Rudy Bunel, Rishabh Singh, Matthew Hausknecht, and Pushmect Kohli. Neural program meta-induction. arXiv preprint 11Xvv:1710.04157, 2017.
[23] Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Push- meet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference machine learning. pp.
PMLR, 2017.
[24] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[25] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthier, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recngnition at scale. arXiv preprint arXiv:2010. 11929, 2020.
[26] František Duchoñ, Andrej Babinec, Martin Kajan, Peter Beño, Martin Florek, Tomáš Fico, and Ladislav Jurišica. Path planning with modified a star algorithm for mobile robot. Procedia Engineering, 96: 59 69, 2014.
[27] Kevin Ellis, Daniel Ritchie, Armando Solar- and Joshua B Tenenbaum. Learning to infer graphics programs from hand-drawn images. arXiv preprint /xxiv:/77.0096: 2017.
[28] Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, and Xuanjing Huang. Mask attention networks: Rethinking and strengthen transformer. arXiv preprint arXiv: 103. 102.1133.977 13597, 2021.
[29] John K Feser, Mare Brockschmidt, Alexander L Gaunt, and Daniel Tarlow. Differentiable functional program interpreters. arXiv preprint 1.Xi::6611001880 2016.
[30] Richard E Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189 198-0,888 1971.
[31] Roy Fox, Richard Shin, Sanjay Krishnan, Ken Goldherg, Dawn Song, and Ion Stoica. Parametrized hierarchical procedures for neural programming. ICLR 2018, 2018.
[32] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirkpairka,,ric Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for vision-and- lnn-uug navigation. arXiv preprint 1806 02724, 2018.
[33] Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation network. CoRR, abs/2106.01 2021. URL https://arxixi www.aawwwww....raaaaaaeaae abo///0............/.. 2001/100000101
[34] Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lun Yiu Nie, and Xin Xia. Code structure guided transformer for source code summarization. CoRR, ahs/2104.09340, 2021. URI. htt.itttts://axiv.oom org
abs /b04/ 09340
[35] Artur d' Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv: (ate:::0500000000 06088, 2019.
[36] Artur SD Avila Garcez, Luis Lamb, and Dov M Gabbay. Neural-symbolic cognitive reasoning. Springer Science& & Business Media, 2008.
[37] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. Neural Computation, 1999.
[38] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 1rX.:.1410.5401, 2014.
[39] Sumit Gulwani. Program synthesis. Software Systems Safety, pp. 43-75. 2014.
[40] Stevan Harnad. Grounding symbols in the analog world with neural nets. Think, 2(1):12-78, 1993.
16

---

[41] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. NKK::1606603476. 2016.
arXiv preprint
[42] Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pp. 53-69, 2018.
[43] Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. arXiv preprint arXiv:1909.00960 00964, 2019.
[44] Wenlong Huang. Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. International Conference on Machine Learning, pp. 4455-4464. PMLR, 2020.
[45] Drew Hudson and Christopher D Manning. Learning by abstraction: The ncural state machine. In Advances in Neural Information Processing Systems, pp. 5903-5916, 2019.
[46] Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. International Conference on Learning Representations, 2018.
[47] Drew Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6700-6709, 2019.
[48] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-20122000 29100 2017.
[49] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zit- nick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE International Conference Computer Vision, pp. 2989-29999999 2998, 2017.
[50] Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv: arXiv:1311.0022 2015.
[51] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances neural information processing systems, pp. 1008-1014. Citeseer, 2000)
[52] George Konidaris, Scott Kuindersma, Roderic Grupen, Andrew Barto. Robot learning from demon- stration by constructing skill trees. The International Journal of Robotics Research, 31(33:660-37 2012.
[53] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson. Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision. 123(1):32-73. 2017.
[54] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332 1338, 2015.
[55] Brenden M Lake, Tomer D Ullman, B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[56] Guohao Li, Xin Wang, and Wenwu Zhu. Percoptual visual reasoning with knowledge propagation. Proceedings of the 27th ACM International Conference on Multimedia, MM '19, pp. 530-538, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 9781450368886. doi: 10. 015553343031.335000
http:://pp:oiiii wtt.://///.0.11/33333330333 3350922
[57] Michael I. Littman. Markov games framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
[58] Yunchao Liu and Zheng Wu. Learning describe seenes with programs. In International Conference on Learning Representations, 2019.
[59] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 140300000.. 2021.
17

---

[60] Jiasen Lu, Jianwei Yang. Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. arXiv preprint urXiv:1606.00000 2016.
[61] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic 1908.02265, 2019.
representations for vision-and-language tasks. arXiv preprint arXiv: /
[62] Aleksander Madry, Alcksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1700:0/066..3333 2017.
[63] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa- Bianchi, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates. Inc., 2018. URL wtwptttsss//////// https::/pproceedinnnee file/ /ttp///aeeeeeeeeeeeee/6666662220 Paper...pppppp...p
newrips cc/paper//118////////
[64] Jiayuan Mao, Chuang Gan, Pushmect Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro-Symbolic Concept Leamnr: Interpreting Scenes, Words, and Sentences From Natural Supervision. In Interna- tional Conference on Learning Representations, 2019. URL URLhttts://////ttt /ttps:://pperevvllii wwwppperrreeiee.eeeertrueeee nwt/forummii
rJgMIhRctm
[65] Jiayuan Mao, Xiuming Zhang, Yikai Li. William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Program-guide image manipulators. CoRR, abss/199.00211 16, 2019. URL httppt/////a/iii wwwpsxivvoorr/aao abs/ 1909. 02116
[66]
Mingyuan Mao, Renrui Zhang, Honghui Zheng, Peng Gao, Teli Ma. Yan Peng, Errui Ding, and Shumin Han. Dual-stream network for visual recognition. arXiv preprint arXiv:2105.14734 2021.
[67]
Saif M Mohammad. Sentiment analysis: Detecting valence, emotions, and other affectual states from text. In Emotion measurement, pp. 201-237. Elsevier, 2016.
[68] Ronald Parr and Stuart Russell. Reinforeement learning with hierarchies of machines. Advances in neural information processing systems, pp. 1043-1049, 1998.
[69] Dean A Pomerleau, Jay Gowdy, and Charles E Thorpe. Combining artificial neural networks and symbolic processing for autonomous robot guidance. Engineering Applications of Artificial Intelligence, 4(4): 279-288 285, 1991.
[70] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
[71] Scott Reed and Nando De Freitas. Neural programmeer-nnterreeer.. arXiv preprint arXiv:1511..0277) 2015.
[72] Karan Samel, Zelin Zhao, Binghong Chen, Kuan Wang. Robin Luo, and Le Song. How to design sample and computationally efficient vqa models. arXiv preprint arXiv:2103. 153.7 2021.
[73] Stefan Schaal et al. Learning from demonstration. Advances in neural information processing systems, pp. 1040-1046, 1997.
[74] Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4613-4621, 2016.
[75] Richard Shin, Illia Polosukhin, and Dawn Song. Towards specification-direetedd program repair, 2018. wet./foruml.dd-1122222.
URL htttthttttttssstie
[76] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484448) 2016.
[77] Sungryull Sohn, Junhyuk Oh, and Honglak Lec. Hicrarchical reinforcement learning for zero-shot generalization with subtask dependencies. arXiv preprint arXiv: 1807.0766. 2018
[78] Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang. Mask-guided contrastive attention model for person re-identification. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[79] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp.
2019.
18

---

[80] Shao-Hua Sun. Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthesis from diverse demonstration videos. In International Conference on Machine Learning, pp. 4790-4799. PMLR, 2018.
[81] Shao-Hua Sun, Te-Lii Wu, and Joseph Lim. Program guided agent. In International Conference on Learning Representations, 2019.
[82] Sheng Tai. Richard Socher, and Christopher Manning. Improved semantic representations from troe-structured long short-term memory networks. arXiv preprint arXiv: 1503.00075, 2015.
[83] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP), pp. 5103-5114 114,, 2019.
[84] Yee Whye Tch, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint arXiv: 00775,,, 2017.
[85] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William Freeman, Joshua B Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. arXiv preprint arXiv:1901.02877 2019.
[86] Dweep Trivedi, Jesse Zhang. Shao- Hua Sun, and Joseph J. Lim. Learning to synthesize programs as interpretable and generalizable policies, 2021.
[87] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017.
[88] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
[89] Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Pro- grammatically interpretable reinforcement learning. In International Conference on Machine Learning. pp. 5045-5054. PMLR, 2018.
[90] Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 7(2)777799, 2002.
[91] Huiyu Wang. Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Max-deeplab: End- to-end panoptic segmentation with mask transformers. CoRR, abs/2012.00759, 2020. URL https
// www.v/orrgabb///22..oe
[92] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement lcarning: a hierarchical bayesian approach. In Proceedings of the 24th international conference Machine learning, pp. 1015-1022, 2007.
[93] Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 699 707, 2017.
[94] Da Xiao, Jo-Yu Liao, and Xingyuan Yuan. Improving the universality and lcarnaillity of neural progranmer-ineerpreeers with combinator abstraction. arXiv preprint 1rXiv:1802.026693 2018
[95] Wenbin Xu, Xudong Li, Wenda Xu, Liang Gong, Yixiang Huang. Zelin Zhao, Lujie Zhao, Binhao Chen, Haozhe Yang. Li Cao, et al. Human-robot interaction oriented human -eeeeei---theeeee the- real-time motion imitation on a humamoid tri-co robot. In 2018 3rd International Conference on Advanced Robotics and Mechatronics (ICARM), pp. 781-786. IEFF, 2018
[96] Wenbin Xu, Xudong Li, Liang Gong, Yixiang Huang. Zeyuan Zheng, Zelin Zhao, Lujie Zhao, Binhao Chen, Haozhe Yang. Li Cao, et al. Natural teaching for humanoid robot via human-in-the-loog scene- motion cross-modal perception. Industrial Robot: the international journal of robotics research and application, 2019.
[97] Jianwei Yang, Jiayuan Mao, Jiajun Wu, Devi Parikh, David D Cox, Joshua Tenenbaum, and Chuang Gan. Object-centric diagnosis of visual reasoning. arXiv preprini arXiv:2012.11552 2020.
[98] Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 190.009900 2019.
19

---

[99] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. arXiv preprint arXiv:1810.02338, 2018
[100] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song. James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXivv1904.0000
2019.
[101] QingLong Zhang and Yubin Yang. Rest: An efficient transformer for visual recognition. CoRR, abs/2105.13677, 2021. URL https:///xxxxiii wtt.:////..ab......////// 13677
[102] Zelin Zhao, Gao Peng, Haoyu Wang, Hao-Shu Fang, Chengkun Li, Cewu Estimating 6d pose from localizing designated surface keypoints. arXiv preprint urXiv:1812.013335 2018.
[103] Zelin Zhao, Chuang Gan, Jiajun Wu, Xiaoxiao Guo, and Joshua Tenenbaum. Augmenting policy learning with routines discovered from a demonstration. arXiv preprint arXiv:2012.12469 2020.
[104] Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Simple baseline for visual question answering. arXiv preprint arXiv: 2015.
20


Reviews:
[{"review_id": "xXlL7Ltm7w6", "reviewer": "Reviewer_5PzE", "summary": "This paper explores how learning agents can follow program guidance to solve visual reasoning and policy learning tasks. While most prior works are only designed to address one of the two tasks, this paper aims to propose a unified framework that can tackle both two problems. To this end, the paper proposes a Transformer-based framework (ProTo) that learns to interpret and execute programs on observed specifications by learning to execute a program in a latent space with cross attention and masked attention mechanisms. The experiments on the visual reasoning task (GQA dataset) show that ProTo outperforms baselines using different learning signals such as question-answer pairs, scene graphs, and programs. The experiments on the program-guided policy learning in a 2D-Minecraft environment show that ProTo outperforms baselines and generalizes better to long and complex programs. I believe this work studies a promising research direction and proposes a framework that can address program-guided tasks in two domains. Therefore, I am leaning toward accepting this paper yet still have some concerns (see below) which I hope to be addressed in the rebuttal.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Paper strengths and contributions\n**Motivation**\n- The motivation for using programs as a structured, clean and formal representation to guide learning agents for task-solving is convincing.\n\n**Novelty and technical contributions**\n- To the best of my knowledge, the idea of unifying two program-guided task domains, including visual reasoning and policy learning, is novel.\n- The idea of separately leveraging the program semantics and the explicit structure of the given program is convincing, and this paper proposes a way to implement this idea.\n- Treating routines (sub-tasks in programs) as objects and leverages attention mechanisms to model interactions among routines and specifications using Transformer seems effective.\n- Execute programs in a learned latent space for strong representation ability seems to contribute to the performance gain and generalization capability.\n- The proposed framework can be trained using different levels of supervisions, including dense execution traces, a goal specification with distance measurement, and sparse rewards. This provides great flexibility.\n\n**Clarity**\nThe writing of the introduction and the related work is clear. The problem and the proposed framework are well motivated and the contributions are explicitly stated.\n\n**Experimental results**\n- The presentation of the experimental results is clear.\n- Visual reasoning task on the GQA dataset:\n\t- ProTo outperforms all baselines using learning signals including question-answer pairs, scene graphs, and programs on all the metrics.\n\t- The generalization experiments evaluate how well the models can generalize to the programs that are written by humans. The results show that ProTo outperforms the meta module network.\n\t- The low data regime experiment evaluates if how well the models can perform with only 10% of training data. The result shows that ProTo outperforms the meta module network.\n- Policy learning task on the 2D-Minecraft domain:\n\t- ProTo learns more efficiently compared to the program-guided agent baseline and end-to-end learning models.\n\t- The experiments on longer and more complex programs show the better generalization performance of ProTo.\n\n**Ablation study**\nThe ablation studies on the GQA dataset show that:\n- Leveraging both the structure guidance and the semantic guidance is the key to the superior performance of ProTo.\n- Utilizing a Transformer to encode inputs is better than using a graph neural network.\n\n**Reproducibility**\nThe code is provided, which helps understand the details of the proposed framework.\n\n## Paper weaknesses & questions\n\n**Clarity: method section**\nThe writing of the method section is unclear and is not easy to follow. I have read it three times and still am not sure I correctly understand what modules are learned to do what, what input/output representations they use, how they are used for tasks from two domains. With my partial understanding of the proposed method, it is not easy for me to judge it. I believe a significant revision of the method section is required. \n\n**Walk-through example or flow chart**\nTogether with the point mentioned above, I believe it would be better to either provide a walk-through example or a flow chart that explicitly shows the learning modules and the input/output representations. It would be even better if walk-through examples from both domains are shown. While Figure 2 is supposed to serve this purpose, it is not still clear to me.\n\n**Table 2 bottom rows**\nI do not know how to interpret the bottom two rows of Table 2. Do the numbers indicate the performance of NS-VQA and MMN with partial supervision? If this is the case, I suggest the authors move the midrule a row higher so that three models (ProTo, NS-VQA, and MMN) learning from partial supervision and put them together for comparison. If not, the authors should probably provide a better description at the end of Section 5.1.\n\n**Unseen programs**\nI do not understand the unseen program generalization experiment on the GQA dataset. Does it mean that the model has never seen verify type tokens during training and can still work when tested on a program with such tokens without further training? Also, why are the verify type of instances chosen not others? It would be better if the authors offer intuitions for this.\n\n**External memory**\nWhile the authors believe that it would be better if a model does not leverage external memory for program-guided tasks for more efficient training, \nI actually believe incorporating external memories would allow the framework to be applied to a wider scope of program-guided applications. \n\n**Human-written programs and model-synthesized/GQA programs**\nThe generalization experiment evaluates how well ProTo and MMN can generalize to human-written programs, which are supposed to be different from model-synthesized programs and the programs in the GQA dataset. However, without some examples of all types of programs (human-written programs, model-synthesized programs, and GQA programs), it is not easy to judge if the distribution gap exists or how large the gap is, making it even harder to judge the generalization ability of ProTo and MMN.\n\n**Missing related works**\nWhile the related work section is fairly sufficient, including the following works would make it more comprehensive:\n- Neural Scene De-rendering\n- Neural Program Synthesis from Diverse Demonstration Videos\n- Learning to Describe Scenes with Programs\n- Learning to Infer and Execute 3D Shape Programs\n\n**Minor issues**\n- duplicated citations: [57] & [58]\n- L155: Note \"that\"\n- Many other grammar errors\n\n## Other metrics\n\n### Relevance and significance\n- Reasonable contribution to a minor problem\n\n### Novelty\n- One idea that surprised me by its originality, solid contributions otherwise\n\n### Technical quality\n- Technically strong, highly general results, advanced techniques\n\n### Experimental evaluation\n- Solid, informative evaluation w.r.t all 5 criteria\n\n### Clarity\n- This paper is difficult to understand in places because of typos, lack of organization, or other flaws. The paper would need significant/major improvement in writing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Described in the main review section.", "review_text": "", "extra_scores": {"ethical_concerns": "I currently do not recognize any potential negative impact or ethical concerns for this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "gF_uRMO4cz3", "reviewer": "Reviewer_LuHc", "summary": "This paper presents ProTo (Program-Guided Transformer), a modified \nTransformer-based architecture that aims to learn a (fully neurally-parameterized)\n execution model for symbolic programs. The authors propose a parameterization scheme that encodes the program semantics (decomposed along minimal executable components of the symbolic program) and program syntax (using a transition mask defined based on the program tree); and describe multiple training regimes (varying from dense supervision based on ground truth or symbolic executions, to a fully RL-based distant supervision.) \nThe authors present empirical evaluations on two domains -- visual question answering (GQA) and policy execution (2D Minecraft) with respect to other comparable neural architectures that leverage symbolic program traces. They find comparable performance with respect to the SOTA, and demonstrate improvements in generalization to longer and novel programs. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths (soundness; theoretical grounding; empirical evaluation; significance and novelty; relevance to NeurIPS community)*:\n- Application is interesting and of significant interest to the NeurIPS community. Domains used demonstrate promise for future work. Learning executors that can handle formal programming languages is an area of of interest with important potential applications for generalization and interpretability, and the authors demonstrate their model on two domains -- VQA and policy learning -- that highlight this. Further, while the authors focus on learning execution engines for formal program representations, it seems possible and of interest to test the generalizability of the same architecture (perhaps pre-trained on programs as a form of ‘synthetic data’ to natural language processing -- and indeed, the authors do approach this in their GQA approach (which uses a Seq2Seq model to perform the initial transformation from natural language into the DSL used here.)\n\nThe DSLs used are fairly complex -- the planning DSL has looping constructs.\n\n- Architecture proposed is reasonably novel and lays the groundwork for future work. The proposed architecture is based on Transformers -- it uses attention blocks as its primary neural architectural component -- but takes interesting, reasonable seeming, (and I believe, novel) approaches to leverage the inherent semantic and syntactic structure of the program representation. This seems promising as an avenue for future work using similar architectures, and I think if the authors wished they could attempt to describe the most general instantiation of this architecture while signposting particular choices made in their concrete model that could be modified in the future (e.g the authors choose routines as minimal executable units, presumably for supervision purposes. But it seems like changes could be made so that `Filter(girl)` could be decomposed further for greater compositionality in the future.)\n\n- Empirical evaluations show comparable performance to other SOTA architectures (like meta-module networks), and generalization experiments show significant performance improvements. The results from human-specified programs are particularly interesting. The authors compare against a good range of strong baselines and ablations.\n\nFor the generalization experiments, it would be nice to see a continuous graph (rather than bucketing the programs into standard testing, longer programs, and complex programs, for instance, in Table 4) that really shows the fall-off in performance between length of training programs vs. generalization performance -- perhaps this would even more clearly highlight the contributions of this model. \n\nSimilarly, the authors also present intriguing data using ‘unseen programs’ on the GQA dataset. I could personally use some clarification on how this worked -- was there additional ‘training’ at test time to learn to use the `verify` instances? Otherwise, how does the model know at all how to execute these primitives -- but it could be interesting and useful to see a similar continuous graph showing the fall-off in generalization performance as additional new primitives are introduced into the dataset, to really showcase compositionality or graceful robustness to new primitives. \n\n*Weaknesses (soundness; theoretical grounding; empirical evaluation; significance and novelty; relevance to NeurIPS community)*:\n- Discussion of generalization performance is promising but could be more rigorous, especially with respect to symbolic / formal baselines when they exist, or mixed neuro-symbolic methods. The authors mention several times that their approach -- learning the program executor entirely -- is based on prior work showing that learned executors often outperform symbolic ones.\nThis certainly may be the case on many domains of interest, and the authors convincingly show strong improvements over a model (NS-VQA) that uses a symbolic execution model on visual question answering, a domain that does seem to require some learnable component and that seems well-suited for this application.\n\nOn the second domain (Minecraft), however, we do in fact have a fully symbolic model -- the symbolic planner used to construct programs (and that provides the dense supervision.) In the strengths section above, I laid out some figures / generalization experiments that would help give a clearer picture of how this model performs relative to a fully-symbolic executor on two axes that seem most important to emulate: (a) modularity compositionality with respect to new program components introduced into the DSL; and (b) generalization to long programs and programs of arbitrary length.\nEspecially because the authors strongly suggest that a core goal of this model is to replace or outperform symbolic models on important future domains, I think it is important to have a clear picture of how well it can perform on fully symbolic tasks relative to symbolic baselines. \n \n- Empirical performance in comparison to SOTA is comparable and competitive, but in many cases, except on generalization or complex programs, it is not very large. It suggests that a harder dataset would be necessary to really showcase the performance of this model, and on GQA, all models still seem to fall well short of human performance.\n- (More minor): discussion of failure cases, especially wrt longer programs and control flow. Discussion of intermediate execution states and disentanglement. The GQA dataset does not have control flow. Loops and other more complex, characteristically programmatic structures are one area where I would have expected to see degraded generalization performance with respect to a symbolic model (e.g. performing more inaccurately with greater numbers of loops). The authors show results specifically on complex programs (with more than four If/While tokens) which are promising -- but again, I see this as an area that could have specifically probed where this model successfully learns to emulate the desired flow and when it may not. Perhaps some metric other than ground-truth accuracy -- such as an analysis of the intermediate execution state accuracy -- would also serve this purpose. \n- Use of dense supervision. This is discussed in the limitations section (Appendix E), but raises questions about the scalability of this model with respect to its ultimate end goal. However, it could be interesting to see other ‘Partial Superivsion’ training regimes other than the one in the paper (knowing the final execution result of the whole program) -- what about supervision on some program components but not others (suggested by the GQA training regime -- which starts with a pre-trained object detector)? This could be an interesting area for testing compositionality and robustness to new program components as well.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a reasonable discussion of limitations and broader impact in the appendix (E and F).\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not have significant ethical concerns about this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "bk2AzjQpgXv", "reviewer": "Reviewer_SyKX", "summary": "This paper proposes a new model for learning to execute programs on program-guided tasks. The model is based on transformer and uses cross-attention and self-attention mechanisms. It takes as inputs both the program structure (control flow information) and program semantics (through token embeddings), as well as task specifications. The model is evaluated on a visual reasoning benchmark and a 2D Minecraft RL benchmark. Experiments show that the proposed model achieves better performance than previous ad-hoc approaches for learning to execute programs in these domains. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a general approach for learning to execute programs on program-guided tasks. Using programs to guide learning is a promising direction and has received increasing attention, so the paper is targeting an important problem. The paper is well-written and easy to follow. The experimental evaluation is good, covering two distinct domains including visual reasoning and RL, and showed consistent improvement over prior ad-hoc approaches designed for each domain. Overall, this is a well-written paper.\n\nQuestions:\n\n- What are the reasons for the performance improvement by ProTo over the prior ad-hoc approaches? Is it utilizing more information from the program, or just due to the better modeling power of a large transformer model? It would be helpful to see some analysis on the difference between ProTo and prior approaches to help understand why ProTo is better. \n\n- There are some prior work that proposes general techniques for learning to execute programs, which is most closely related to the theme of the paper: Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks, David Bieber et.al. 2020. \nIt would be good if the authors can compare with these approaches.\n\n- ProTo model feeds the last result matrix into the current step, therefore inducing a recurrent structure. How is this treated during training? Since for RL tasks with long horizon and only reward signal, back-propagating through the whole episode seems to be a high computation burden.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact is discussed in the Appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2oVVQrnKZhe", "reviewer": "Reviewer_JPZw", "summary": "This work proposes Program-Guided Transformer (ProTo) for program execution tasks. Specifically, instead of simply encoding the program as a token sequence, they disentangle the program representation into a semantic representation and a structural representation, and the structural representation takes the program control flows into accounts. They modify the attention mechanisms of the vanilla Transformer architecture to encode the disentangled program representation. They evaluate their approach on two tasks: (1) program-guided visual question answering on GQA; and (2) program-guided policy learning task proposed in [64]. They show that ProTo outperforms other baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning to execute programs is an important topic. In particular, executing programs on visual inputs and understanding routines in programs are challenging problems. For both GQA and policy learning tasks, the authors present good results and comparisons to strong baselines. The proposed Transformer architecture properly utilizes the program structure. Although the programs evaluated in this work are in restricted domain-specific languages, the ProTo architecture might be extended to more complicated programming languages.\n\nI have a couple of questions about the approach design and experiments:\n\n1. From Algorithms 1 and 2, the pointer update depends on the execution results. According to the equation for computing the output o, I suppose the execution output vector should represent a probability distribution over all possible execution results. When you train or evaluate the model, do you take the argmax to determine the execution output for the pointer update, e.g., selecting T/F for conditional statements and selecting actions for executing routines? If this is the case, is the model hard to train since wrong intermediate execution results can mislead the execution of all remaining program statements?\n\n2. Could you present the distribution of program lengths in your experiments? What is the largest program length ProTo can handle? From the attention computation and the structural representation, the transition mask and some intermediate results are |P|x|P|-dimensional, thus the model might not scale to larger programs without modification.\n\n3. A general suggestion on related work and baseline discussion: I feel that the main point of this work is about neural-symbolic reasoning and program understanding, because the modification of the Transformer architecture itself is not extremely novel. Therefore, in the main body of the paper, the authors can include more discussion of existing work on neural-symbolic learning, and the key differences between this work and baseline models that lead to better performance.\n\n4. I am confused about the results of Unseen Programs in GQA experiments. For removing \"verify\"-type problems, do you mean that you only remove them for training the ProTo model, or do you also remove them for training the program synthesis model? If you completely remove all these problems for training, I don't know why the model can generalize. Specifically, how should the model understand these new functions if they never appear in training?\n\n5. What is the point of collecting additional human-written programs for evaluation? Why not simply evaluate the model performance on ground truth programs in GQA? Could you provide the statistics of the human program dataset you collect, and explain what are the key differences between these new programs and those in GQA?\n\n6. In the policy learning experiment, it says that \"Transformer and TreeRNN encode the input program in a token by token manner\". However, I suppose the TreeRNN model should encode the program as a tree. Could you explain more about how you use TreeRNN here?\n\n7. In experiments of Longer Programs and Complex Programs, do you only train the models on shorter programs with fewer control flow constructs, or do you still train on programs of all different complexities, but present the test results on harder programs? I wonder whether these results show the generalization of models to programs that are more complex than training samples, or they are more about breakdown results varying the program complexities.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciated the novel approach for learning to execute symbolic programs using both program syntax and semantics. Adding additional experimental evaluation and comparisons to IPA-GNN and symbolic planner for Minecraft was also greatly appreciated. It would be great to incorporate the feedback, and the experimental and architectural clarifications from the reviews and author responses in the final version.
Decision: Accept (Poster)