ProTo: Program-Guided Transformer for Program-Guided Tasks

1 Introduction . Programs are the natural interface for the communication between machines and humans [ 11 ] . In comparison to instructing machines via demonstrations [ 41 , 52 , 73 , 6 ] or via natural language [ 14 , 32 , 3 ] , guiding agents by programs has multiple benefits . First , programs are explicit and much cleaner than other instructions such as languages [ 81 ] . Second , programs are structured with loops and branches [ 1 ] so they can express complex reasoning processes [ 99 ] . Finally , programs are compositional , promoting the generalization and scalability of neural models [ 15 , 64 , 20 ] . However , while program synthesis and program induction have been deeply explored [ 12 , 17 , 18 , 22 , 39 , 49 ] , very few works focuses on learning to follow program guidance [ 81 , 71 ] . Furthermore , previous work designs ad-hoc program executors for different functions in different tasks [ 81 , 29 , 23 ] , which hinders the generalization and scalability of developed models . To pursue general program executors to unify perception , reasoning , and decision making , we formulate program-guided tasks , which require the agent to follow the given program to perform tasks conditioned on task specifications . The programs may come from a program synthesis model [ 99 ] or be written by human [ 81 ] . Two exemplar tasks are shown in Figure 1 . Program-guided tasks are challenging because the agent needs to jointly follow the complex structure of the program [ 1 ] , perceive the specification , and ground the program semantics on the specification [ 40 ] . Inspired by the recent significant advance of transformers in diverse domains [ 24 , 25 , 59 ] , we present the Program-guided Transformer ( ProTo ) for general program-guided tasks . ProTo combines the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . strong representation ability of transformers with symbolic program control flow . We propose to separately leverage the program semantics and the explicit structure of the given program via efficient attention mechanisms . ProTo enjoys strong representation ability by executing the program in a learned latent space . In addition , ProTo can either learn from reward signals [ 64 , 81 ] or from dense execution supervision [ 15 ] . We evaluate ProTo on two tasks , program-guided visual reasoning and program-guided policy learning ( corresponding to Figure 1 left and Figure 1 right ) . The former task requires the model to learn to reason over a given image following program guidance . We experiment on the GQA dataset [ 47 ] where the programs are translated from natural language questions via a trained program synthesis model . The evaluation on the public test server shows that we outperform the previous state-of-the-art program-based method by 4.31 % in terms of accuracy . Our generalization experiments show that ProTo is capable of following human program guidance . The latter task asks the agent to learn a multi-task policy to interact with the environment according to the program . We experiment on a 2D Minecraft environment [ 81 ] where the programs are randomly sampled from the domain-specific language ( DSL ) . We find that ProTo has a stronger ability to scale to long and complex programs than previous methods . ProTo significantly outperforms the vanilla transformer [ 87 ] which does not explicitly leverage the structural guidance of the program . We will release the code and pre-trained models after publishing . In summary , our contributions are threefold . First , we formulate and highlight program-guided tasks , which are generalized from program-guided visual reasoning [ 99 , 64 ] and program-guided policy learning [ 81 ] . Second , we propose the program-guided transformer for program-guided tasks . Third , we conduct extensive experiments on two domains and show promising results towards solving program-guided tasks . 2 Related Work . Program Induction , Synthesis and Interpretation Program induction methods learn an underlying input-output mapping from given examples [ 54 , 22 , 50 ] , and no programs are explicitly predicted . Differently , program synthesis targets at predicting symbolic programs from task specifications [ 23 , 12 , 18 , 17 , 31 , 27 , 93 , 85 , 16 , 80 , 58 , 86 ] . However , these approaches do not learn to execute programs . In the domain of digit and string operation ( adding , copying , sorting , etc . ) , Neural Program Interpreter ( NPI ) [ 71 , 94 , 75 ] learn to compose lower-level programs to express higher-level programs [ 37 ] while Neural Turing Machines [ 38 , 55 ] use attentional processes with external memory to infer simple algorithms . Recently , [ 81 ] proposes to guide the agent in 2D Minecraft via programs . [ 65 ] detects repeated entries in the image and uses programs to guide image manipulation . They neither attempt to formulate general program-guided tasks nor propose a unified model for different tasks . Visual Reasoning Visual reasoning requires joint modeling of natural language and vision . A typical visual reasoning task is the visual question answering ( VQA ) [ 5 ] . Attention mechanisms have been widely used in the state-of-the-art VQA models [ 60 , 2 , 74 , 104 , 46 , 72 ] . Neural module networks and the neural-symbolic approaches [ 99 , 64 ] propose to infer programs from natural languages and execute programs on visual contents to get the answer . A large-scale dataset GQA [ 47 ] , which addresses the low diversity problem in the synthetic CLEVR dataset [ 48 ] , is proposed to evaluate the state-of-the-art visual reasoning models . Neural state machine [ 45 ] is proposed by leveraging modular reasoning over a probabilistic graph , which does not leverage the symbolic program . Our model is similar to the concurrent work , meta module network [ 15 ] in that we use shared parameters for different function executors . Nevertheless , we propose a novel attention-based architecture and extend our model to the policy learning domain . Transformer Transformer was firstly proposed in machine translation [ 87 ] . After that , transformers demonstrate their power on many different domains , such as cross-modal reasoning [ 83 ] , large-scale unsupervised pretraining [ 24 , 79 , 98 ] , and multi-task representation learning [ 70 , 67 ] . Recently , transformers appear to be competitive models in many fundamental vision tasks , such as image classification [ 25 , 101 , 66 ] , object detection [ 13 ] and segmentation [ 59 , 33 ] . While we share the same belief that transformers are strong , unified , elegant models for various deep learning tasks , we propose novel transformer architecture for program-guided tasks . We discuss a few similar unpublished works in Appendix A . Policy Learning With Programs Previous policy learning literature explored the benefits of programs in different aspects . First , some pre-defined routines containing human prior knowledge could help reinforcement learning , and planning [ 30 , 68 , 4 , 103 ] . Second , programs enable interpretable and verifiable reinforcement learning [ 89 , 8 ] . Our approach follows a recent line of work that learns to interpret program guidance [ 81 , 4 ] . However , we adopt a novel , simple and uniform architecture that demonstrates superior performance , thus conceptually related to multitask reinforcement learning [ 84 , 92 ] and hierarchical reinforcement learning [ 7 ] . 3 Program-guided Tasks . Unlike natural language that is flexible , complex , and noisy , programs are structured , clean and formal [ 81 ] . Therefore , programs can serve as a powerful intermediate representation for humancomputer interactions [ 11 ] . Different from previous work that learns to synthesis programs from data [ 99 , 85 , 27 ] , we study learning to execute given programs [ 81 ] based on three motivations . First , instructing machines via explicit programs instead of noisy natural languages enjoys better efficiency , and accuracy [ 81 ] . Second , learned program executors have stronger representation ability than hardcoded ones [ 99 , 15 ] . Third , we attempt to develop a unified model to integrate perception , reasoning , and decision by learning to execute , which heads towards a core direction in the neural-symbolic AI [ 36 , 35 ] . A program-guided task is specified by a tuple ( S , , O , G ) where S is the space of specifications , denotes the program space formed by a domain-specific language ( DSL ) for a task , O is the space for execution results , and G is the goal function for the task . For each instance of the program-guided task , a program P 2 is given . An executor E is required by the task to execute the program on the observed specification . Note that different from hard-coded non-parametric executors used in some previous work [ 99 , 23 ] , the executor E is parametrized by . For convenience , we define a routine to be the minimum execution unit of a program ( e.g . Filter ( girl ) in Figure 1 is a routine ) . We denote Pk to be the k-th routine in P and |P | to be the number of routines in P . According to the DSL , the program should begin at the entrance routine and finish at one of the exiting routines ( e.g . Filter ( girl ) in Figure 1 is an entrance routine and Query_color ( ) is an exiting routine ) . At each execution step ⌧ , the executor executes the current routine Pp 2 P on the specification s ( ⌧ ) 2 S and produces an output o ( ⌧ ) 2 O . The execution of P finishes if one of the exiting routines finishes execution . The goal of a program-guided task is to produce desired execution results to achieve the goal G. Algorithm 1 : ProTo Execution Result : Execution results { o ( ⌧ ) } T⌧=1 1 Initialize ⌧ = 0 and the pointer p ; 2 Build Ps according to Eq . 1 ; 3 while not reach an exiting routine do 4 Observe s ( ⌧ ) and set ⌧ = ⌧ + 1 ; 5 Build Pt ( ⌧ ) according to Eq . 2 ; 6 o ( ⌧ ) = ProToInfer ( s ( ⌧ ) , Ps , Pt ( ⌧ ) , p ) ; 7 Output o ( ⌧ ) ; 8 if Pp finishes execution then 9 UpdatePointer ( p , o ( ⌧ ) , Pp ) ; 10 end Algorithm 2 : UpdatePointer ( p , o ( ⌧ ) , Pp ) 1 if Pp is an If-routine then 2 Point p to the first routine in the T/F branch if o ( ⌧ ) is T /F ; 3 else if Pp is a While-routine then 4 Point p to the first routine inside/outside the loop if o ( ⌧ ) is T /F ; 5 else if Pp ends a loop then 6 Point p to the loop condition routine . 7 else 8 Point p to the subsequent routine ; 9 end 4 Program-guided Transformer . A good model for a program-guided task should fulfill the following merits . First of all , it should leverage both the semantics and structures provided by the program ( refer to Figure 2 ) . Second , it would better be a multi-task architecture with shared parameters for different routines to ensure efficiency and generalization [ 15 , 90 , 70 ] . Third , it would be better if the model does not leverage external memory for the efficiency of batched training [ 64 ] . To these ends , we present the Programguided Transformer ( ProTo ) . ProTo treats routines as objects and leverages attention mechanisms to model interactions among routines and specifications . The execution process of ProTo is presented in Algorithm 1 . In each execution timestep , ProTo leverages the semantic guidance Ps and the current structure guidance Pt ( ⌧ ) ( Line 2 & 5 in Algorithm 1 , detailed in Sec 4.1 ) and infers for one step ( Line 6 in Algorithm 1 , detailed in Sec 4.2 ) . When a routine finishes execution , a pointer p 2 { 1 , 2 , ... , |P | } indicating the current routine Pp ( Pp is the pth routine in P ) would be updated ( Line 8 & 9 in Algorithm 1 , detailed in Algorithm 2 and Sec 4.3 ) . The results of ProTo are denoted as { o ( ⌧ ) } T⌧=1 where T is the total number of execution timesteps . ProTo supports several training schemes , which are listed in Sec 4.4 .

Summary: This paper explores how learning agents can follow program guidance to solve visual reasoning and policy learning tasks. While most prior works are only designed to address one of the two tasks, this paper aims to propose a unified framework that can tackle both two problems. To this end, the paper proposes a Transformer-based framework (ProTo) that learns to interpret and execute programs on observed specifications by learning to execute a program in a latent space with cross attention and masked attention mechanisms. The experiments on the visual reasoning task (GQA dataset) show that ProTo outperforms baselines using different learning signals such as question-answer pairs, scene graphs, and programs. The experiments on the program-guided policy learning in a 2D-Minecraft environment show that ProTo outperforms baselines and generalizes better to long and complex programs. I believe this work studies a promising research direction and proposes a framework that can address program-guided tasks in two domains. Therefore, I am leaning toward accepting this paper yet still have some concerns (see below) which I hope to be addressed in the rebuttal.