ProTo: Program-Guided Transformer for Program-Guided Tasks

1 Introduction . Programs are the natural interface for the communication between machines and humans [ 11 ] . In comparison to instructing machines via demonstrations [ 41 , 52 , 73 , 6 ] or via natural language [ 14 , 32 , 3 ] , guiding agents by programs has multiple benefits . First , programs are explicit and much cleaner than other instructions such as languages [ 81 ] . Second , programs are structured with loops and branches [ 1 ] so they can express complex reasoning processes [ 99 ] . Finally , programs are compositional , promoting the generalization and scalability of neural models [ 15 , 64 , 20 ] . However , while program synthesis and program induction have been deeply explored [ 12 , 17 , 18 , 22 , 39 , 49 ] , very few works focuses on learning to follow program guidance [ 81 , 71 ] . Furthermore , previous work designs ad-hoc program executors for different functions in different tasks [ 81 , 29 , 23 ] , which hinders the generalization and scalability of developed models . To pursue general program executors to unify perception , reasoning , and decision making , we formulate program-guided tasks , which require the agent to follow the given program to perform tasks conditioned on task specifications . The programs may come from a program synthesis model [ 99 ] or be written by human [ 81 ] . Two exemplar tasks are shown in Figure 1 . Program-guided tasks are challenging because the agent needs to jointly follow the complex structure of the program [ 1 ] , perceive the specification , and ground the program semantics on the specification [ 40 ] . Inspired by the recent significant advance of transformers in diverse domains [ 24 , 25 , 59 ] , we present the Program-guided Transformer ( ProTo ) for general program-guided tasks . ProTo combines the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . strong representation ability of transformers with symbolic program control flow . We propose to separately leverage the program semantics and the explicit structure of the given program via efficient attention mechanisms . ProTo enjoys strong representation ability by executing the program in a learned latent space . In addition , ProTo can either learn from reward signals [ 64 , 81 ] or from dense execution supervision [ 15 ] . We evaluate ProTo on two tasks , program-guided visual reasoning and program-guided policy learning ( corresponding to Figure 1 left and Figure 1 right ) . The former task requires the model to learn to reason over a given image following program guidance . We experiment on the GQA dataset [ 47 ] where the programs are translated from natural language questions via a trained program synthesis model . The evaluation on the public test server shows that we outperform the previous state-of-the-art program-based method by 4.31 % in terms of accuracy . Our generalization experiments show that ProTo is capable of following human program guidance . The latter task asks the agent to learn a multi-task policy to interact with the environment according to the program . We experiment on a 2D Minecraft environment [ 81 ] where the programs are randomly sampled from the domain-specific language ( DSL ) . We find that ProTo has a stronger ability to scale to long and complex programs than previous methods . ProTo significantly outperforms the vanilla transformer [ 87 ] which does not explicitly leverage the structural guidance of the program . We will release the code and pre-trained models after publishing . In summary , our contributions are threefold . First , we formulate and highlight program-guided tasks , which are generalized from program-guided visual reasoning [ 99 , 64 ] and program-guided policy learning [ 81 ] . Second , we propose the program-guided transformer for program-guided tasks . Third , we conduct extensive experiments on two domains and show promising results towards solving program-guided tasks . 2 Related Work . Program Induction , Synthesis and Interpretation Program induction methods learn an underlying input-output mapping from given examples [ 54 , 22 , 50 ] , and no programs are explicitly predicted . Differently , program synthesis targets at predicting symbolic programs from task specifications [ 23 , 12 , 18 , 17 , 31 , 27 , 93 , 85 , 16 , 80 , 58 , 86 ] . However , these approaches do not learn to execute programs . In the domain of digit and string operation ( adding , copying , sorting , etc . ) , Neural Program Interpreter ( NPI ) [ 71 , 94 , 75 ] learn to compose lower-level programs to express higher-level programs [ 37 ] while Neural Turing Machines [ 38 , 55 ] use attentional processes with external memory to infer simple algorithms . Recently , [ 81 ] proposes to guide the agent in 2D Minecraft via programs . [ 65 ] detects repeated entries in the image and uses programs to guide image manipulation . They neither attempt to formulate general program-guided tasks nor propose a unified model for different tasks . Visual Reasoning Visual reasoning requires joint modeling of natural language and vision . A typical visual reasoning task is the visual question answering ( VQA ) [ 5 ] . Attention mechanisms have been widely used in the state-of-the-art VQA models [ 60 , 2 , 74 , 104 , 46 , 72 ] . Neural module networks and the neural-symbolic approaches [ 99 , 64 ] propose to infer programs from natural languages and execute programs on visual contents to get the answer . A large-scale dataset GQA [ 47 ] , which addresses the low diversity problem in the synthetic CLEVR dataset [ 48 ] , is proposed to evaluate the state-of-the-art visual reasoning models . Neural state machine [ 45 ] is proposed by leveraging modular reasoning over a probabilistic graph , which does not leverage the symbolic program . Our model is similar to the concurrent work , meta module network [ 15 ] in that we use shared parameters for different function executors . Nevertheless , we propose a novel attention-based architecture and extend our model to the policy learning domain . Transformer Transformer was firstly proposed in machine translation [ 87 ] . After that , transformers demonstrate their power on many different domains , such as cross-modal reasoning [ 83 ] , large-scale unsupervised pretraining [ 24 , 79 , 98 ] , and multi-task representation learning [ 70 , 67 ] . Recently , transformers appear to be competitive models in many fundamental vision tasks , such as image classification [ 25 , 101 , 66 ] , object detection [ 13 ] and segmentation [ 59 , 33 ] . While we share the same belief that transformers are strong , unified , elegant models for various deep learning tasks , we propose novel transformer architecture for program-guided tasks . We discuss a few similar unpublished works in Appendix A . Policy Learning With Programs Previous policy learning literature explored the benefits of programs in different aspects . First , some pre-defined routines containing human prior knowledge could help reinforcement learning , and planning [ 30 , 68 , 4 , 103 ] . Second , programs enable interpretable and verifiable reinforcement learning [ 89 , 8 ] . Our approach follows a recent line of work that learns to interpret program guidance [ 81 , 4 ] . However , we adopt a novel , simple and uniform architecture that demonstrates superior performance , thus conceptually related to multitask reinforcement learning [ 84 , 92 ] and hierarchical reinforcement learning [ 7 ] . 3 Program-guided Tasks . Unlike natural language that is flexible , complex , and noisy , programs are structured , clean and formal [ 81 ] . Therefore , programs can serve as a powerful intermediate representation for humancomputer interactions [ 11 ] . Different from previous work that learns to synthesis programs from data [ 99 , 85 , 27 ] , we study learning to execute given programs [ 81 ] based on three motivations . First , instructing machines via explicit programs instead of noisy natural languages enjoys better efficiency , and accuracy [ 81 ] . Second , learned program executors have stronger representation ability than hardcoded ones [ 99 , 15 ] . Third , we attempt to develop a unified model to integrate perception , reasoning , and decision by learning to execute , which heads towards a core direction in the neural-symbolic AI [ 36 , 35 ] . A program-guided task is specified by a tuple ( S , , O , G ) where S is the space of specifications , denotes the program space formed by a domain-specific language ( DSL ) for a task , O is the space for execution results , and G is the goal function for the task . For each instance of the program-guided task , a program P 2 is given . An executor E is required by the task to execute the program on the observed specification . Note that different from hard-coded non-parametric executors used in some previous work [ 99 , 23 ] , the executor E is parametrized by . For convenience , we define a routine to be the minimum execution unit of a program ( e.g . Filter ( girl ) in Figure 1 is a routine ) . We denote Pk to be the k-th routine in P and |P | to be the number of routines in P . According to the DSL , the program should begin at the entrance routine and finish at one of the exiting routines ( e.g . Filter ( girl ) in Figure 1 is an entrance routine and Query_color ( ) is an exiting routine ) . At each execution step ⌧ , the executor executes the current routine Pp 2 P on the specification s ( ⌧ ) 2 S and produces an output o ( ⌧ ) 2 O . The execution of P finishes if one of the exiting routines finishes execution . The goal of a program-guided task is to produce desired execution results to achieve the goal G. Algorithm 1 : ProTo Execution Result : Execution results { o ( ⌧ ) } T⌧=1 1 Initialize ⌧ = 0 and the pointer p ; 2 Build Ps according to Eq . 1 ; 3 while not reach an exiting routine do 4 Observe s ( ⌧ ) and set ⌧ = ⌧ + 1 ; 5 Build Pt ( ⌧ ) according to Eq . 2 ; 6 o ( ⌧ ) = ProToInfer ( s ( ⌧ ) , Ps , Pt ( ⌧ ) , p ) ; 7 Output o ( ⌧ ) ; 8 if Pp finishes execution then 9 UpdatePointer ( p , o ( ⌧ ) , Pp ) ; 10 end Algorithm 2 : UpdatePointer ( p , o ( ⌧ ) , Pp ) 1 if Pp is an If-routine then 2 Point p to the first routine in the T/F branch if o ( ⌧ ) is T /F ; 3 else if Pp is a While-routine then 4 Point p to the first routine inside/outside the loop if o ( ⌧ ) is T /F ; 5 else if Pp ends a loop then 6 Point p to the loop condition routine . 7 else 8 Point p to the subsequent routine ; 9 end 4 Program-guided Transformer . A good model for a program-guided task should fulfill the following merits . First of all , it should leverage both the semantics and structures provided by the program ( refer to Figure 2 ) . Second , it would better be a multi-task architecture with shared parameters for different routines to ensure efficiency and generalization [ 15 , 90 , 70 ] . Third , it would be better if the model does not leverage external memory for the efficiency of batched training [ 64 ] . To these ends , we present the Programguided Transformer ( ProTo ) . ProTo treats routines as objects and leverages attention mechanisms to model interactions among routines and specifications . The execution process of ProTo is presented in Algorithm 1 . In each execution timestep , ProTo leverages the semantic guidance Ps and the current structure guidance Pt ( ⌧ ) ( Line 2 & 5 in Algorithm 1 , detailed in Sec 4.1 ) and infers for one step ( Line 6 in Algorithm 1 , detailed in Sec 4.2 ) . When a routine finishes execution , a pointer p 2 { 1 , 2 , ... , |P | } indicating the current routine Pp ( Pp is the pth routine in P ) would be updated ( Line 8 & 9 in Algorithm 1 , detailed in Algorithm 2 and Sec 4.3 ) . The results of ProTo are denoted as { o ( ⌧ ) } T⌧=1 where T is the total number of execution timesteps . ProTo supports several training schemes , which are listed in Sec 4.4 .

Summary: This paper proposes a new model for learning to execute programs on program-guided tasks. The model is based on transformer and uses cross-attention and self-attention mechanisms. It takes as inputs both the program structure (control flow information) and program semantics (through token embeddings), as well as task specifications. The model is evaluated on a visual reasoning benchmark and a 2D Minecraft RL benchmark. Experiments show that the proposed model achieves better performance than previous ad-hoc approaches for learning to execute programs in these domains.