Communicating Natural Programs to Humans and Machines

1 INTRODUCTION . Humans solve a range of procedural tasks such as cooking , tying shoes , and programming . Although current AI systems achieve super-human proficiency at certain narrowly specified tasks ( Silver et al. , 2017 ; Lerer et al. , 2020 ) , their reasoning is domain-specific and fails to generalize to novel situations ( Lake et al. , 2017 ) . The Abstraction and Reasoning Corpus ( ARC ) introduced by Chollet ( 2019 ) presents a set of procedural tasks constructed expressly to benchmark fundamental capacities associated with human general intelligence , including abstraction , generalization , object categories , and the capacity to communicate ( Chi et al. , 2014 ; Harlow , 1949 ; Lake et al. , 2017 ; Lake & Piantadosi , 2020 ; Bartlett , 1932 ; Tian et al. , 2020 ; Lombrozo , 2006 ) . Specifically , ARC requires one to infer a procedure consistent with a small number of abstract input-output examples and apply it to a new input to generate an unseen answer , see Figure 1 . How do we build systems that are capable of solving general , procedural tasks such as ARC ? Traditional approaches of program synthesis ( Parisotto et al. , 2016 ; Ellis et al. , 2019 ; Solar-Lezama et al. , 2006 ; Devlin et al. , 2017b ) and semantic parsing ( Artzi & Zettlemoyer , 2013 ; Ye et al. , 2020 ; Wang et al. , 2015 ; Marzoev et al. , 2020 ; Guu et al. , 2017 ; Kulal et al. , 2019 ) assume the tasks are DSLclosed – aor any task , there exists a program , written in a predefined Domain Specific Language ( DSL ) , that solves the task . The ARC benchmark is uniquely designed to be DSL-open – it does not come with a predefined DSL capable of solving all the tasks . This is both reasonable – most real life tasks , such as cooking and assembling furniture , are DSL-open – and challenging – how can one build an intelligent system that can solve tasks from few examples without a DSL ? To illustrate , what might a DSL that would allow one to program all the ARC tasks in Figure 1 look like ? This question is difficult to answer ; a recent Kaggle competition found that the best AI systems solve at most 20 % of the tasks , while Johnson et al . ( 2021 ) found that most humans easily solve over 80 % 1 . Given that humans greatly outperform the best AI systems in solving ARC tasks , studying the human ’ s cognitive processes ( for instance , which set of concepts do human use to represent these tasks ? ) can shed light on how to build similarly intelligent systems . As these thought processes are not observable directly , we study natural programs – instructions that humans give to each other , as a window into these latent cognitive processes . Like computer programs , these instructions can be reliably interpreted ( by another human ) to produce the intended output . Unlike computer programs , which must be stated in a specific style , natural programs can be stated in any form – such as verbal instructions or input-output examples – as long as another human can execute them . In this work , we study a particular form of natural programs , that of natural language instructions . We show that analyzing these natural programs – with explicit comparisons to computer programs – can both shed light on how humans communicate and interpret procedures ( Spelke et al. , 1992 ; McCarthy et al. , 2021 ; Clark et al. , 1983 ; Clark & Wilkes-Gibbs , 1986 ) and inform how one may build AI systems for challenging , DSL-open domains such as ARC . We present the Language-complete Abstraction and Reasoning Corpus ( LARC ) of natural language instructions elicited from a two-player communication game , where 88 % of the ARC tasks can be successfully communicated . LARC tasks are language-complete : The successful instructions contain all the relevant information , even in absence of the original input-output examples ( see Figure 2 ) . This is important in several ways : First , one can use LARC to study how humans use language to communicate abstract procedures , as humans clearly have the capacity to both generate and execute these natural programs ; Second , one can directly see what concepts an intelligent system must be aware of ( such as colors and numbers ) ; Third , as people readily generate natural programs , studying them will provide insights on building interactive systems . We perform linguistic analysis on LARC , finding that humans readily leverage algorithmic concepts without being explicitly instructed to do so . These concepts range from domain general ones , such as loops , to domain-specific concepts such as flood-fill . However , natural programs in LARC are distinct from typical computer programs in two ways : ( 1 ) They contain a wide range of concepts , in contrast to the few primitives in a typical DSL ; ( 2 ) The majority of effort is spent on providing clarifications and validations , as opposed to stating the procedure verbatim . We apply standard program synthesis algorithms on LARC , finding that while existing approaches can benefit from the additional language annotations , the two aforementioned distinctions pose significant challenges 1Humans were evaluated on a subset of the training tasks ; the Kaggle competition used a private test set . to standard program synthesis approaches . We conclude by providing concrete suggestions on how to build the next generation program synthesizers . 2 COMMUNICATING AND INTERPRETING PROGRAMS . In communicating procedures , a programmer constructs a program in a suitable language , which is then executed on an interpreter , producing a desirable behaviour . For instance , a person can instruct another person to carry out a certain task ( Fig . 3 top-left ) , or directly program a machine to solve tasks using code ( Fig . 3 top-right ) . Program synthesis takes in an instruction , and reformulates it as executable code , insulating the person from the programming process ( Fig . 3 bot ) . How do we build machines that are capable of solving challenging tasks given in a natural way ? Typically , one follows a “ DSL-first ” approach . In the DSL-first approach , one first defines a programming language and builds a corresponding interpreter , so that a skilled programmer may express the tasks as programs . Then , one naturalizes the initial DSL with a synthesizer , allowing end-users to describe tasks using language ( Artzi & Zettlemoyer , 2013 ; Artzi et al. , 2014 ; Ye et al. , 2020 ; Wang et al. , 2015 ; 2016 ; Marzoev et al. , 2020 ) , or by giving examples ( Ellis et al. , 2019 ; Solar-Lezama et al. , 2006 ; Pu et al. , 2020 ) . While this DSL-first workflow has yielded impressive results , the DSL itself is also a single point of failure . It is difficult to design DSL with the right scope , so that it is easy to write programs for the domain , without bloating the language with redundant concepts ( Gulwani et al. , 2015 ; Bruce et al. , 2020 ; Soto-Valero et al. , 2021 ) . For synthesis systems , one must ensure that the DSL aligns reasonably to human instructions ( Liang , 2016 ; Shin et al. , 2021 ) , while simultaneously being efficient when used by the synthesizer ( Ellis et al. , 2019 ; 2020 ) . These challenges may explain why ARC , and other DSL-open domains , pose significant challenges in building intelligent systems using the DSL-first approach . In this work , we suggest that starting with human instructions ( Fig 3 top-left ) can yield insights for building systems on challenging , DSL-open domains and highlight deficiencies in DSL-first approaches . We define a natural program as instructions constructed by a person that can be interpreted by another person to produce a specific output . This program is natural–it can be understood by speakers of the language2 without a prior consensus–but behaves as a program , in that it produces a definitive output , which can be unambiguously checked for correctness . For instance , the original ARC ( Chollet , 2019 ) tasks are natural programs : Given a program consisting of input-output examples , a fellow human can readily interpret this program to produce an output on a new input , which can be checked for correctness . Contrary the DSL-first approach , by starting with ( linguistic ) natural programs , one can directly observe the set of concepts and strategies necessary to master a domain ( such as ARC ) , without having to first build an interpreter . 2language here is to be understood loosely as any medium of communication between people 3 LARC : LANGUAGE-COMPLETE ABSTRACTION AND REASONING CORPUS . We present a dataset that augments the original ARC tasks from Chollet ( 2019 ) with languagecomplete instructions : they can be demonstrably interpreted by other human users to correctly produce the intended outputs without any additional contexts ( including the original input-output examples ) . Thus , LARC tasks ( Fig 2 ) , like their counterparts in ARC , meet the definition of natural program while containing only natural language descriptions . To collect this dataset , we introduce a communication game were human describers produce linguistic instructions for unseen downstream human builders asked to solve the same tasks ; and deploy this experiment using a novel bandit algorithm to efficiently collect verifiable natural programs . The final dataset augments 88 % of the original ARC tasks ( 354/400 ) with at least one verifiable natural program description that could be successfully interpreted by another human participant to solve the task . Fig . 5 ( C-D ) shows the distribution of success rates for participants acting as describers and builders over time . 3.1 HUMAN ANNOTATION DETAILS . We recruited 373 subjects via Amazon Mechanical Turk who were paid for 45 minutes of work . Fifty individuals were excluded for failing to complete the task , so the final analysis included 323 subjects . The study was approved by our institution ’ s Institutional Review Board , did not collect personally identifiable information , and did not pose risks to participants . Subjects were paid $ 6.00 and a $ 0.25 bonus for every successful communication . Subjects averaged 5.5 communications , bringing their expected hourly wage to $ 9.83 . For interface and consent form see Appendix A.2 . 3.2 TWO-PLAYER COMMUNICATION GAME . For each task , a participant may be assigned one of two roles : a describer or a builder . The describer plays the role of a human synthesizer , who reformulates input-output examples ( of ARC ) to natural language descriptions . The builder plays a role of a human interpreter , who must construct the correct output on a new input without access to the original examples ( Fig 4 ) . The description is structured into three sections to incentivize consistency : ( 1 ) what the builder should expect to see in the input , ( 2 ) the output grid size , and ( 3 ) what the builder should do to create the output ( Fig 2 ) . After the description was submitted , we verify the describer ’ s own understanding by asking them to build it , and discarding the submission if the describer fails . The describer was shown all previous verified descriptions for a task , alleviating challenge of solving the task from scratch . Participants construct the output using actions defined in ARC , such as paint ( color , x , y ) , copy/paste , and floodfill . All action sequences are recorded .

Summary: The paper presents the language-complete ARC to study how human language can affect abstraction and reasoning capability.  It collects the natural program described by human annotators and applies program synthesis techniques to analyze the gap between communicating to humans and machines. The motivation is sound and the contribution is solid with the effort of annotating LARC and benchmarking the program synthesis approaches.