Iterated learning for emergent systematicity in VQA

1 INTRODUCTION . Although great progress has been made in visual question-answering ( VQA ) , recent methods still struggle to generalize systematically to inputs coming from a distribution different from that seen during training ( Bahdanau et al. , 2019b ; a ) . Neural module networks ( NMNs ) present a natural solution to improve generalization in VQA , using a symbolic layout or program to arrange neural computational modules into computation graphs . If these modules are learned to be specialized , they can be composed in arbitrary legal layouts to produce different processing flows . However , for modules to learn specialized roles , programs must support this type of compositionality ; if programs reuse modules in non-compositional ways , modules are unlikely to become layout-invariant . This poses a substantial challenge for the training of NMNs . Although Bahdanau et al . ( 2019b ) and Bahdanau et al . ( 2019a ) both observe that NMNs can systematically generalize if given humandesigned ground-truth programs , creating these programs imposes substantial practical costs . It becomes natural to jointly learn a program generator alongside the modules ( Johnson et al. , 2017b ; Hu et al. , 2017 ; Vedantam et al. , 2019 ) , but the generated programs often fail to generalize systematically and lead to worse performance ( Bahdanau et al. , 2019b ) . Iterated learning ( IL ) offers one way to address this problem . Originating in cognitive science , IL explains how language evolves to become more compositional and easier-to-acquire in a repeated transmission process , where each new generation acquires the previous generation ’ s language through a limited number of samples ( Kirby et al. , 2014 ) . Early works with human participants ( Kirby et al. , 2008 ) as well as agent-based simulations ( Zuidema , 2003 ) support this hypoth- ∗Correspondance at : ankit.vani @ umontreal.ca . esis . The machine learning community has also recently shown an increasing interest in applying IL towards emergent communication ( Guo et al. , 2019 ; Li & Bowling , 2019 ; Cogswell et al. , 2019 ; Dagan et al. , 2020 ; Ren et al. , 2020 ) . Different from previous works , we believe that IL is an algorithmic principle that is equally applicable to recovering compositional structure in more general tasks . We thus propose treating NMN programs as samples from a “ layout language ” and applying IL to the challenging problem of VQA . Our efforts highlight the potential of IL for broader machine learning applications beyond the previously-explored scope of language emergence and preservation ( Lu et al. , 2020 ) . To demonstrate our method , we introduce a lightweight benchmark for systematic generalization research based on the popular SHAPES dataset ( Andreas et al. , 2016 ) called SHAPES-SyGeT ( SHAPES Systematic Generalization Test ) . Our experiments on SHAPES-SyGeT , CLEVR ( Johnson et al. , 2017a ) , and CLOSURE ( Bahdanau et al. , 2019a ) show that our IL algorithm accelerates the learning of compositional program structure , leading to better generalization to both unseen questions from the training question templates and unseen question templates . Using only 100 ground-truth programs for supervision , our method achieves CLEVR performance comparable to Johnson et al . ( 2017b ) and Vedantam et al . ( 2019 ) , which use 18000 and 1000 programs for supervision respectively . 2 RELATED WORK . Systematic generalization . Systematicity was first proposed as a topic of research in neural networks by Fodor & Pylyshyn ( 1988 ) , who argue that cognitive capabilities exhibit certain symmetries , and that representations of mental states have combinatorial syntactic and semantic structure . Whether or not neural networks can exhibit systematic compositionality has been a subject of much debate in the research community ( Fodor & Pylyshyn , 1988 ; Christiansen & Chater , 1994 ; Marcus , 1998 ; Phillips , 1998 ; Chang , 2002 ; Marcus , 2018 ; van der Velde et al. , 2004 ; Botvinick & Plaut , 2009 ; Bowers et al. , 2009 ; Brakel & Frank , 2009 ; Fodor & Lepore , 2002 ; Marcus , 2018 ; Calvo & Symons , 2014 ) . Bahdanau et al . ( 2019b ) investigate various VQA architectures such as neural module networks ( NMNs ) ( Andreas et al. , 2016 ) , MAC ( Hudson & Manning , 2018 ) , FiLM ( Perez et al. , 2018 ) , and relation networks ( Santoro et al. , 2017 ) on their ability to systematically generalize on a new synthetic dataset called SQOOP . They show that only NMNs are able to robustly solve test problems , but succeed only when a fixed tree-structured layout is provided . When learning to infer the module network layout , robust tree-structured layouts only emerged if given a strong prior to do so . The authors conclude that explicit regularization and stronger priors are required for the development of the right layout structure . CLEVR ( Johnson et al. , 2017a ) is a popular VQA dataset , and various benchmarks achieve almostperfect CLEVR validation scores ( Hu et al. , 2017 ; Hudson & Manning , 2018 ; Perez et al. , 2018 ; Santoro et al. , 2017 ) . Bahdanau et al . ( 2019a ) proposed an extension of CLEVR with a new evaluation dataset called CLOSURE , containing novel combinations of linguistic concepts found in CLEVR . The authors found that many of the existing models in the literature fail to systematically generalize to CLOSURE . Moreover , there is a significant gap between the performance achieved with ground-truth layouts and learned layouts on CLOSURE . Language emergence and compositionality . Agents interacting in a cooperative environment can learn a language to communicate to solve a particular task . The emergence of such a communication protocol has been studied extensively in multi-agent referential games . In these games , one agent must describe what it saw to another agent , which is tasked with figuring out what the first agent saw ( Lewis , 2008 ; Skyrms , 2010 ; Steels & Loetzsch , 2012 ) . To encourage a dialogue between agents , several multi-stage variants of such a game have also been proposed ( Kottur et al. , 2017 ; Evtimova et al. , 2018 ) . Most approaches to learning a discrete communication protocol between agents use reinforcement learning ( Foerster et al. , 2016 ; Lazaridou et al. , 2017 ; Kottur et al. , 2017 ; Jorge et al. , 2016 ; Havrylov & Titov , 2017 ) . However , the Gumbel straight-through estimator ( Jang et al. , 2017 ) can also be used ( Havrylov & Titov , 2017 ) , as can backpropagation when the language in question is continuous ( Foerster et al. , 2016 ; Sukhbaatar & Fergus , 2016 ; Singh et al. , 2019 ) . Several works in the literature have found that compositionality only arises in emergent languages if appropriate environmental pressures are present ( Kottur et al. , 2017 ; Choi et al. , 2018 ; Lazaridou et al. , 2018 ; Chaabouni et al. , 2020 ) . While generalization pressure is not sufficient to guarantee compositionality , compositional languages tend to exhibit better systematic generalization ( Bahdanau et al. , 2019b ; Chaabouni et al. , 2020 ) . The community still lacks strong research indicating what general conditions are necessary or sufficient for compositional language emergence . Iterated learning . The origins of the compositionality of human language , which leads to an astounding open-ended expressive power , have attracted much interest over the years . Kirby ( 2001 ) suggests that this phenomenon is a result of a learning bottleneck arising from the need to learn a highly expressive language with only a limited set of supervised learning data . The iterated application of this bottleneck , as instantiated by IL , has been demonstrated to cause artificial languages to develop structure in experiments with human participants ( Kirby et al. , 2008 ; Silvey et al. , 2015 ) . Ren et al . ( 2020 ) present neural IL following the principles of Kirby ( 2001 ) , where neural agents play a referential game and evolve a communication protocol through IL . They use topographic similarity ( Brighton & Kirby , 2006 ) to quantify compositionality , and find that high topographic similarity improves the learning speed of neural agents , allows the listener to recognize more objects using less data , and increases validation performance . However , these experiments are limited to domains with extremely simple object and message structure . Several ablation studies ( Li & Bowling , 2019 ; Ren et al. , 2020 ) have found that re-initializing the speaker and the listener between generations is necessary to reap the benefits of compositionality from IL . However , seeded IL ( Lu et al. , 2020 ) proposes to seed a new agent with the previous generation ’ s parameters at the end of the learning phase of a new generation . Since self-play has not yet fine-tuned this initialization , it has not had the opportunity to develop a non-compositional language to fit the training data . The authors find that seeded IL helps counter language drift in a translation game , and hypothesize that IL maintains the compositionality of natural language . 3 METHOD . We are interested in solving the task of visual question-answering ( VQA ) . Let X be the space of images about which our model will be required to answer questions . Next , let Q be the space of natural-language questions and Y the space of all possible answers to the questions . Additionally , we consider a space Z of programs , which represent computation graphs of operations that can be performed on an image in X to produce an output in Y . We consider a question template T to be a set of tuples ( q , z ) , where q ∈ Q and z ∈ Z . Each question template contains questions with the same structure but varying primitive values . For example , the questions “ Is a triangle blue ” and “ Is a square red ” belong to a template “ Is a SHAPE COLOR. ” The program z corresponding to the question q in a template defines a computation graph of operations that would produce the correct answer in Y to q for any input image x ∈ X . Finally , let T be a finite set of question templates . The dataset for training our model and evaluating VQA performance constitutes tuples of the form ( q , z , x , y ) . First , a template T ∈ T is sampled and a tuple ( q , z ) is sampled from T . Then , an image x ∈ X is sampled and the answer y is produced by passing x through the program z . These collected variables ( q , z , x , y ) form a single example in the task dataset . To evaluate our model ’ s performance on unseen question templates , we define Ttrain ⊂ T to be a subset of training templates and Ttest = T − Ttrain to be the subset of test templates . The training dataset D is prepared from templates in Ttrain and the out-of-distribution test dataset Dtest from templates in Ttest . We allow a program z to be absent in D , in which case it is not used for auxiliary supervision during training . Our goal of systematic generalization is to learn a model p ( Y | X , Q ) that performs well on the dataset Dtest created using unseen question templates , where Y , X , and Q are random variables taking values in Y , X , and Q . We define our model to be a composition of a program generator PGθ ( Z | Q ) and an execution engine EEφ ( Y |X , Z ) , parameterized by θ and φ respectively .

Summary: The authors apply iterated learning - a procedure originating in CogSci analyses of how human languages might develop - to the training of neural module networks. The goal is for iterated learning to encourage these networks to develop compositional structures that support systematic generalization without requiring explicit pressures for compositional structures (in past work, such explicit pressures have generally been necessary). The proposed approach brings substantial improvements in systematic generalization across two datasets, SHAPES and CLEVR.