Iterated learning for emergent systematicity in VQA
Authors: Ankit Vani
Max Schwarzer
Yuchen Lu
Eeshan Dhekane
Aaron Courville
Venue: ICLR 2021 Oral (2021)
Keywords: iterated learning
cultural transmission
neural module network
clevr
shapes
vqa
visual question answering
systematic generalization
compositionality

Abstract: Although neural module networks have an architectural bias towards compositionality, they require gold standard layouts to generalize systematically in practice. When instead learning layouts and modules jointly, compositionality does not arise automatically and an explicit pressure is necessary for the emergence of layouts exhibiting the right structure. We propose to address this problem using iterated learning, a cognitive science theory of the emergence of compositional languages in nature that has primarily been applied to simple referential games in machine learning. Considering the layouts of module networks as samples from an emergent language, we use iterated learning to encourage the development of structure within this language. We show that the resulting layouts support systematic generalization in neural agents solving the more complex task of visual question-answering. Our regularized iterated learning method can outperform baselines without iterated learning on SHAPES-SyGeT (SHAPES Systematic Generalization Test), a new split of the SHAPES dataset we introduce to evaluate systematic generalization, and on CLOSURE, an extension of CLEVR also designed to test systematic generalization. We demonstrate superior performance in recovering ground-truth compositional program structure with limited supervision on both SHAPES-SyGeT and CLEVR.

Published as a conference paper at ICLR 2021
ITERATED LEARNING FOR EMERGENT SYSTEMATICITY IN VQA
Ankit Vani"
Max Schwarzer
Yuchen Lu Mila, Université de Montréal
Mila, Université de Montréal
Mila, Université de Montréal
Eeshan Dhekane Mila, Université de Montréal
Aaron Courville
Mila, Université de Montréal, CIFAR Fellow
ABSTRACT
Although neural module networks have an architectural bias towards composi- tionality, they require gold standard layouts to generalize systematically in prac- tice. When instead learning layouts and modules jointly, compositionality does not arise automatically and an explicit pressure is necessary for the emergence layouts exhibiting the right structure. We propose address this problem using iterated learning, a cognitive science theory of the emergence of compositional languages in nature that has primarily been applied to simple referential games in machine learning. Considering the layouts of module networks as samples from emergent language, we use iterated learning encourage the development of structure within this language. We show that the resulting layouts support sys- tematic generalization in neural agents solving the more complex task of visual presstonn-answeingg Our regularized iterated learning method can outperform baselines without iterated learning on SHAPES-SyGeT (SHAPES Systematic Generalization Test), new split of the SHAPES dataset we introduce to eval- uate systematic generalization, and on CLOSURE, an extension of CLEVR also designed test systematic generalization. We demonstrate superior performance in recovering ground-truth compositional program structure with limited supervi- sion on both CHAPES-SyGel) and CLEVR.
INTRODUCTION
Although great progress has been made in visual question-answering (VQA), recent methods still struggle to generalize systematically to inputs coming from a distribution different from that seen during training (Bahdanau al. 2019b a). Neural module networks (NMNs) present natural solution to improve generalization in VQA, using a symbolic layoul or program to arrange neural computational modules into computation graphs. If these modules are learned to be specialized, they can be composed in arbitrary legal layouts to produce different processing flows. However, for modules learn specialized roles, programs must support this type of compositionality: programs reuse modules in non-compositional ways, modules are unlikely to become layout-invariant.
This poses substantial challenge for the training of NMNs. Although Bahdanau al. (0019)) and Bahdanau et al. (2019a) both observe that NMNs can systematically generalize if given human- designed ground-truth programs, creating these programs imposes substantial practical costs.
becomes natural to jointly learn program generutor alongside the modules (Johnson et al. 2017b Hu et al. 2017: Vedantam et al.) 2019), but the generated programs often fail to generalize system- atically and lead worse performance (Bahdanau et al. 2019b).
Iterated learning (IL) offers one way to address this problem. Originating in cognitive science, IL explains how language evolves to become more compositional and easier-to-acquire in peated transmission process, where each new generation acquires the previous generation's 's lan- guage through limited number of samples (Kirby et al.) 2014). Early works with human partici- pants (Kirby et al. 2008) as well as agent-based simulations (Zuidema 2003) support this hypoth-
"Correspondance ank vani @umontreal.

---

Published as a conference paper at ICLR 2021
yes
Classifier
is a green shape left of square?
Bidirectional RNN encoder
and
RNN decoder with attention
color[green
transform[Left.co of]
program generator
shapefsquare]
and color| [green] scene
transforn[left__ggg shupe[square] )
Scene
execution engine
Feature extractor
Figure 1: An overview of neural module networks (NMNs). A question q is read by the program generator produce a program 2. The execution engine assembles neural modules according to the layout & and feeds the input image æ into the assembled module network. A classifier takes the output of the top-level module produce answer for the given (q, x) pair.
esis. The machine learning community has also recently shown an increasing interest in applying IL towards emergent communication (Guo et al. 2019; Li & Bowling, 2019 Cogswell et al. 2019
Dagan et al. 2020 Ren et al. 2020 Different from previous works. we believe that IL is an al- gorithmic principle that is equally applicable to recovering compositional structure in more general tasks. We thus propose treating NMN programs as samples from a "layout language" and apply- ing IL to the challenging problem of VQA. Our efforts highlight the potential of II. for broader machine learning applications beyond the previously-explored scope of language emergence and preservation (Lu et al. 2020).
To demonstrate our method, we introduce a lightweight benchmark for systematic generalization research based on the popular SHAPES dataset (Andreas et al., 2016) called SHAPES-SyGe (SHAPES Systematic Generalization Test). Our experiments on SHAPES-SyGed CLEVR (John-) son et al. 2017a), and CLOSURE (Bahdanau et al. 2019a) show that our IL algorithm accelerates the learning of compositional program structure, leading to better generalization to both unseen questions from the training question templates and unseen question templates. Using only 100 ground-truth programs for supervision, our method achieves CLEVR performance comparable to Johnson et al. (2017b) and Vedantam al. (2019), which use 18000 and 1000 programs for super- vision respectively.
RELATED WORK
Systematic generalization. Systematicity was first proposed as a topic of research in neural net- works by Fodor Pylyshyn (1988), who argue that cognitive capabilities exhibit certain symme- tries, and that representations of mental states have combinatorial syntactic and semantic structure. Whether or not neural networks can exhibit systematic compositionality has been subject of much
debate in the research community (Fodor & Pylyshyn] 1988 Christiansen & Chater 1994 Marcus
1998
Phillips 1998 Chang 2002 [002|Mareuss 2018 van der Velde et al. 20044; Botvinick Plaut
2009
Bowers al. 2009 Brakel & Frank 2009 Fodor & Lepore 2002 Marcus 2018 Calvo &
Symons 2014).
Bahdanau et al. (019bb investigate various VQA architectures such as neural module networks (NMNs) (Andreas et al. 2016). MAC (Hudson & Manning 2018), FiLM (Perez et al. 2018). and new syn- thetic dataset called SQOOP. They show that only NMNs are able to robustly solve test problems, but succeed only when fixed tree-structured layout is provided. When learning to infer the module
relation networks (Santoro et al. 2017) their ability to systematically generalize
2

---

Published as a conference paper at ICLR 2021
network layout, robust tree-structured layouts only emerged if given strong prior to do The authors conclude that explicit regularization and stronger priors are required for the development of the right layout structure.
CLEVR (Johnson et al. 2017a) is a popular VQA dataset, and various benchmarks achieve almost-
perfect CLEVR validation scores (Hu et al. 2017 Hudson & Manning 2018 Perez et al. 2018 Santoro et al. 2017). Bahdanauu et al. (2019a) proposed an extension of CLEVR with a new eval- uation dataset called CLOSURE, containing novel combinations of linguistic concepts found in CLEVR. The authors found that many of the existing models in the literature fail to systematically generalize to CLOSURE. Moreover, there is a significant gap between the performance achieved with ground-truth layouts and learned layouts on CLOSURE.
Language emergence and compositionality. Agents interacting a cooperative environment can learn a language to communicate to solve particular task. The emergence of such a communication protocol has been studied extensively multi-agent referential games. In these games, one agent must describe what it saw to another agent, which is tasked with figuring out what the first agent saw (Lewis) 2008; Skyrms] 2010] Steels & Loetzsch (012). To encourage a dialogue between agents, several multi-stage variants of such game have also been proposed (Kottur et al.
2017 Evtimoval
et al. 2018). Most approaches to learning a discrete communication protocol between agents use
reinforcement learning (Foerster el. 2016 Lazaridou et al. [017 Kottur et al.]
[017 Jorge al.,
2016 Havrylov & Titov (2017). However, the Gumbel straight- werriightthhrugghh estimator (Jang et al. 2017)
can also be used (Havrylov & Titov] 2017), as can backpropagation when the language in question
is continuous (Foerster et al. (0166; Sukhbaatar & Fergus 2016; Singh et al. 2019).
Several works in the literature have found that compositionality only arises in emergent languages if appropriate environmental pressures are present (Kottur et al.) (2017 [Cho et al.) 2018 Lazaridou
jet al. 2018) Chaabouni et al., 2020). While generalization pressure is not sufficient to guarantee compositionality. compositional languages tend to exhibit better systematic generalization (Bah-) danau et al. 2019b; Chaabouni et al. 2020). The community still lacks strong research indicating what general conditions are necessary sufficient for compositional language emergence.
Iterated learning. The origins of the compositionality of human language, which leads to an as- tounding open-ended expressive power, have attracted much interest the years. Kirby (2001) suggests that this phenomenon is a result of a learning bottleneck arising from the need to learn a highly expressive language with only limited set of supervised learning data. The iterated applica- tion of this bottleneck, as instantiated by IL, has been demonstrated to cause artificial languages to
develop structure in experiments with human participants (Kirby al. 2008 Silvey al. 2015
Ren et al. 2020) present neural IL following the principles of Kirby 2001). where neural agents play referential game and evolve a communication protocol through IL. They use topographic similarity (Brighton & Kirby 2006) to quantify compositionality, and find that high topographic similarity improves the learning speed of neural agents, allows the listener to recognize more objects using less data, and increases validation performance. However, these experiments are limited to domains with extremely simple object and message structure.
Several ablation studies (Li & Bowling 2019 Ren et al. 2020) have found that re-initializing the speaker and the listener between generations is necessary to reap the benefits compositionality from IL. However, seeded IL (Lu et al.] 2020) proposes to seed new agent with the previous generation's 's parameters at the end of the learning phase of new generation. Since self-play has not yet fine-tuned this initialization, it has not had the opportunity to develop non-compositional language to fit the training data. The authors find that seeded II. helps counter language drift in a translation game, and hypothesize that IL maintains the compositionality of natural language.
METHOD
We are interested in solving the task of visual question-answering (VQA). Let X' be the space of images about which our model will be required to answer questions. Next, let Q be the space of latural-language questions and y the space of all possible answers to the questions. Additionally, we consider a space Z of programs, which represent computation graphs of operations that can be performed image in X' to produce output in Y. We consider question template T to be
3

---

Published as a conference paper at ICLR 2021
set of tuples (q, 2). where Q and Z. Each question template contains questions with the same structure but varying primitive values. For example, the questions "Is a triangle blue" and "Is square red" belong to template "Is SHAPE COLOR." The program corresponding to the question q in a template defines a computation graph of operations that would produce the correct answer in y' to q for any input image æ E X. Finally, let T be a finite set of question templates.
The dataset for training our model and evaluating VQA performance constitutes tuples of the form (q, 2; 2,, y). First, a template T T is sampled and a tuple (q. is sampled from T. Then, an image x E X' is sampled and the answer is produced by passing æ through the program 2. These collected variables (q. 2, 2. y) form a single example in the task dataset. To evaluate our model's performance on unseen question templates, we define Terain C. T to be a subset of training templates and Trast = T Terain to be the subset of test templates. The training dataset 'D is prepared from templates in Ttrain and the a program z to be absent in D, in which case it not used for auxiliary supervision during training-
test dataset Drest from templates in Trest- We allow
Our goal of systematic generalization is to learn model p(Y X, Q) that performs well on the dataset Drest created using unseen question templates, where Y. X, and Q are random variables taking values in V, X, and Q. We define our model to be composition of program generator [Z Q) and an execution engine EY X, Z), parameterized by 9 and & respectively.
3.1 MODEL ARCHITECTURE
Our overall model architecture is illustrated in Figure I We use the attention- sequence-to sequence model from Bahdanau et al. (2019a) as the program generator, which translates input question q into a sampled program e. The execution engine assembles modules into a layout dictated by á to instantiate explore three execution engine architecture choices, including Tensor-MMN, which is the module architecture from Johnson al. (2017b), and Vector-NMN from Bahdanauu al. (2019a). We also experiment with a novel hybrid of these architectures that performs better on SHAPES-SyGeT, which we dub Tensor- FiLM- Appendix[ elaborates the details of our model architecture.
NMN that takes the input image to predict answer ý. In this work, we
The goal of the Tensor-FiL F-LL---MMMM- architecture is to combine the minimalist, FiLM -baeedaaedddd defini- tion of modules in Vector-NMI with the spatial representational power of Tensor-NMN. As such, the modules in Censor-FFiLM-NMM use FiL.M to condition global operations on module-specific em- beddings as in fector-NMN, but the module inputs and outputs are tensor-valued feature maps like in TenSor-NMN. The following equations illustrate the working of Tensor-FiLM-NNN module:
h = 7h C h1 θ Br,
h2 bb=7a Yh @ h2 ① Br e- [(T,a @ x @ Bo); max ha); (hi - h2)l
(2)
g W1 * e + bj
(3)
y = ReLU(W, * (Yg C ReLLU(gg cumsum.left(g)) cumsum @ 3y) @ b2 + te.
(4)
Here, x is the input image. hi and h2 are the module inputs, and y is the module output. Yo and G. are FiLM parameters computed using different 2-layer MLPs per layer on the module-specific embeddings. The weights W. and biases b. are shared across all modules. We further strengthen our model's 's ability for spatial reasoning by using cumulative sums of an intermediate representation across locations in left-to-right and top-to-bottom directions. This allows our model, for example, to select regions to the 'left of" or 'below' objects through appropriate scaling and thresholding.
3.2 ITERATED LEARNING FOR NMNS
We can view the program generator and the execution engine as communicating agents in a cooper- ative VQA game where the programs passed between agents are messages drawn from an emergent language. Introducing more compositional structure to this language, such as reusing low-level concepts using the same tokens and high-level concepts using the same sequence of tokens, helps address the combinatorial complexity of the question space, allowing agents to perform better on new question templates containing previously-unsedd -unsen combinations of known concepts.
We use IL to encourage the emergence of structure in the generated programs. We iteratively spawn new program generator and execution engine agents, train them the VQA task, and transfer their
4

---

Published as a conference paper at ICLR 2021
PG
D
PG
(nrw)
(towe)
PG (now)
(a) Interacting phase.
(b)
Transmitting
(c) Program genera- tor learning phase.
(d) Execution engine learn- ing phase.
phase.
Figure 2: Phases of II for emergent module layouts. Solid arrows indicate forward pass through the model, and dashed lines indicate the cross-entropy loss between predictions and targets. Af- ter proceeding through phases (a- the new program generator and execution engine begin interacting phase (a) of a new generation.
knowledge to the next generation of agents. Limiting this transmission of knowledge between gen- erations imposes learning bottleneck, where only the easy-to-learn linguistic concepts survive. Since the central hypothesis of neural IL is that structure is easier for neural networks to learn than idiomatic non-compositional rules (Li Bowling, 2019; Ren al. 2020). our IL algorithm pushes the language of the programs to be more compositional. The combination of learning composi- tional structure and performing well on the VQA training task engenders systematic generalization.
We follow Ren et al. (2020) in dividing our method into three stages, an interacting phase, trans- mitting phase, and a learning phase, which are cycled through itcratively until the end of training. Figure illustrates the phases of our method for training module networks with compositional emer- gent layouts and Appendix presents a formal algorithm for the same.
Interacting phase. The program generator and the execution engine must work together. Without programs that consistently use the same tokens to mean the same operations, the execution engine cannot assign the correct semantics to modules. Simultaneously, the program generator depends the language grounding provided by the execution engine the form reinforcement signal for beneficial layouts. To make the problem more tractable, is common provide the model a small set of ground-truth programs (Johnson et al., 20176) Vedantam et al. 2019). We find that providing program supervision with small number of ground-tre truth programs throughout the interacting phase helps in generalization. This can be seen as form of supervised self -lay (Lowe et al. 2020). and has the effect of simulating an inductive bias towards the desired program language.
The agents are jointly trained for 7; steps to minimize answer prediction error on data sampled from PGo(g) ;): ý = EE (æx, é). We train E.F,s by
the training dataset D. with predictions given by minimizing the cross-entropy loss L between the true answer label y and the predicted distribution ý. Additionally, we estimate the gradients for the parameters using REINFORCE:
VgLpa (f) Eg~pe((l) [clip(-L., 5, V log ( q)]
(5)
Here, Pa is the distribution over programs returned by PG)(q). We use the negative cross-entropy
loss - clipped between - and the reward for the program generator. When ground-truth program z is available, we additionally minimize a weighted cross-entropy loss between z and the generated program using teacher forcing. In practice, the model operates on minibatches of data, and subset of every minibatch is subsampled from the examples with ground-truth programs.
Transmitting phase. During the transmitting phase, new dataset with T₁ samples is prepared for use during the learning phase. Questions q. well as ground-truth programs z if available, are sampled from D. For data examples without ground-truth programs, a program the execution engine using q. Finally, question- proorrmm pairs (g, z), if is available, or (g, 2) are added to D. By transmitting the ground- truth programs when available, we continue simulating the inductive bias towards desirable programs during the learning phase.
sampled from
Learning phase. Program generators and execution engines are initialized in the learning phase, forming new generation of agents play the VQA game. The new program generator then quires the previous agents' language from the transmitted data D. However, it does so imperfectly due to learning bottleneck that exerts pressure towards compositionality (Ren et al.) 2020)
5

---

Published as a conference paper at ICLR 2021
We implement this learning bottleneck primarily by limiting the number of gradient updates in the learning phase, effectively performing early stopping. A smaller number of program generator gradient updates Tp leads to an underfit program generator that could waste computation during the interacting phase to re-learn global linguistic rules. On the other hand, setting Tp to a high value can lead overfitting and learning of the previous generation's ' idiosyncrasies, losing the benefits of the learning bottleneck. In addition to limiting the number of training iterations, we optionally further regularize the model by applying spectral normalization (Miyato et al. 2018) on the program generator's s decoder LSTM parameters.
We train the new program generator PGg by minimizing the cross-entropy loss between model samples Σ ~ (G)(g) and transmitted programs & corresponding to q in D for Tp gradient steps. Then, the new execution engine F.F is trained with programs from the new program generator pG using the entire dataset DD for T. steps. The forward pass in the execution engine learning phase is similar to that in the interacting phase, but the backward pass only updates the execution engine parameters Φ) Adapting the new execution engine to the new program generator ensures stable training in the upcoming interacting phase.
4 SYSTEMATIC GENERALIZATION TEST FOR SHAPES
The SHAPES dataset (Andreas et al. 2016) is a popular yet simple VQA dataset. The lower image and question complexities of SHAPES relative to CLEVR make an attractive choice for experi- mentation, as training can be faster and require fewer resources. Each of the 15616 data points in SHAPES contains a unique image. However, there are only 244 unique questions. Although the validation and test splits of SHAPES contain unique questions not present in any of the other splits, the questions are of the same form as the ones in the training split.
To the best of our knowledge, there is no published systematic generalization evaluation setup for SHAPES. Thus, we present SHAPES-SyGeT, a SHAPES Systematic Generalization Test" To un- derstand the distribution of question types in SHAPES, we categorized questions into 12 standard templates, out of which 7 are used for training, and 5 to test systematic generalization.
To evaluate the in-distribution and out-of-distribution generalization performance of our models, we prepare the SHAPES-SS SyGeT training set with only a subset of the questions under each train template and use the rest in-distribution validation set (Val-IID). Questions belonging to the evaluation templates are used as an -oto---distriuutinn validation set (Val-OOD). Please see Ap- pendix D for further details about the question templates and the dataset splits.
EXPERIMENTS
In this section, we present results on SHAPES SyGeT, CLEVR, and CLOSURE. For our preliminary experiments on GQA (Hudson & Manning 2019), please see Appendix G The SHAPES-SyGes experiments are run with different seeds and the CLEVR experiments use 8 seeds. We report the mean and standard deviation of metrics over these trials. CLEVR experiments use representations of images from a pre-traine pre-trained ResNet- 101 (He et al. 2016) as input, whereas the CHAPES-SyGet runs use standardized raw images. Please see Appendix A for the hyperparameters
use.
For our NMN baselines without IL, we still utilize the multi-task objective of using REINFORCE to backpropagate gradients to the program generator and using cross-entropy loss to provide program supervision when available. We find this method of training NMNs to generalize better than pre- training on the available programs and then interacting without supervision is done in Johnson et al. (2017)). One can also view our baselines as executing one long interacting phase without any other phases or resetting of the model.
5.1 SHAPES-SYGRE
Due to its lightweight nature, use SHAPES- SyySet for a series of experiments designed to illumi- nate the essential components of our method. For all SHAPES-SyCaa experiments, we record the
SHAPES--SyGdl be downloaded from: https:: thub.
SyGe'

---

Published as a conference paper at ICLR 2021
Table I: SHAPES SHAPES-SyGeT accuracies. NMNs are trained with 20 135 ground-truth programs;
FiLM (Perez et al. 2018) and MAC (Hudson & Manning 2018) do not use program supervision.
Model
Val-IID
Val-OOD
FiLM
0.720 +
0.609 0.01
MAC
0.730 1 0.01
0.605 0.01
#GTT 20
#GT 135
#GT 20
#GT 135
Tensor-NKN
0.645 0.01
0.700 0.01
0.616 0.01
0.641 0.03
Sensor-NMN+HL
0.756 0.07
0.763 + 0.04
0.648 0.02
0.661 = 0.02
Tensor-F NM-
0.649 = 0.02
0.851 0.01
0.605 0.01
0.692 = 0.06
Tensor-F -MN+IL
0.954 + 0.07
1.000 0.00
0.858 = 0.15
0.971 4 0.02
1.0
Val-IID without IL
Accuracy
Val-OOD without IL
0.8
Val-IID with II.
Val-OOD with IL
0.6
10
20
30
40
135
No. of GT programs
Figure 3: SHAPES--yGel answer accuracies of models with varying number of ground-truw -tunh pro- grams, with and without II.. All models use a FiLNN- execution engine.
in-distribution Val-IID and the outttf-diistibuttion Val-OOD accuracies. However, model selection and early stopping are done based on Val-IID. For our experiments, we use spectral normaliza- tion of the program generator during the learning phase, and we study its utility in Appendix E We do not report results on Vector-NMN, which performs very poorly. We believe that this is be- cause SHAPES-SY requires intermediate module outputs contain detailed spatial information, which Vector-NMN has an inductive bias against thanks to its module outputs being vectors.
We note that the reported results SHAPES in the literature make of NMN architectures with specially designed modules for each operation (Andreas et al.) [2166] Hu et al.) 2017;] Vedantam et al.) 2019). In contrast, minimal semantic priors, making our results hard to compare directly to prior work using SHAPES.
generic modules in order to study compositional layout emergence with
In-distribution and out-of-distribution accuracies. Table 1 illustrates the difference in the per- formance of various configurations our models. All the reported models achieve perfect accuracy on the training data but generalize differently to Val-IID and Val-OOD, We find that Tensor-FiLM NMN generalizes better than Tensor-NMN trained with supervision of both 20 and 135 ground-truth programs. Moreover, we find that IL improves generalization across configurations. We further evaluate the relationship between program supervision and performance in Figure 3 and find that IL improves performance all levels of supervision, effectively improving data efficiency.
Learning bottleneck. Although all models achieve perfect training accuracy in under 5000 steps, we notice that only II. leads to gradually increasing systematic generalization as training progresses. Figure 4 shows the effect of the learning bottleneck of II., comparing two module architectures with and without IL. In the presence of some ground-trute program supervision that the emergent language should abide by, there is a stricter notion of correct programs Thus, in calculating the program accuracy, we consider a program to be correct it matches the ground -hound-truth program ex- actly and incorrect otherwise. We find that the program accuracy increases steadily through the
It is unlikely that diverse set of programs provided for supervision could fit into a compositional structure significantly different from the ground-truth. Furthermore, small error in the serialized programs could result in drastic di sifferences the parsed computation graph structure and semantics.

---

Published as a conference paper at ICLR 2021
1.0
0.8
Actursyy
0.7
0.6
Tensor-FiLMan Fiiu----MMMMM-.IL.
Tensor FILM NMN
Tensor FILM NMN
Tunsor-NMi
Temoor-MMN
20000
40000
60000
80000
100000
20000
40000
60000
B2000
100000
Steps
(a) Task accuracy. Solid lines are training and dashed lines are Val-IID.
(b) Program accuracy.
Figure 4:
Learning curves of models with and without IL SHAPES-SyGeff using 20 ground- truth programs. The IL curves use global gradient step counter across all phases. The dips in the IL training curves indicate the beginning of new generations.
influence of the learning bottleneck in the case of IL training progresses, indicating increasing structure in the language of the programs.
Ablation tests. We use SHAPES-SyGes to perform a series ablations tests and report the full results for these experiments in Appendix [ We first examine the importance of reinitializing the execution engine. We consider two alternatives to reinitializing it from scratch at every genera- tion: seeded IL (Lu al. 2020), which uses the parameters the end of the previous execution engine learning phase for re-initialization, and simply retaining the execution engine without any re-initialization. We find that both of these methods harm generalization performance compared to the standard setting.
Next, we study the effect of spectral normalization applied to the program generator decoder. With- out II., spectral normalization has marginal effect on generalization, indicating it alone is not sufficient to achieve the generalization improvements we observe. However, improves the Val-IID and Val-OOD accuracies substantially with IL in the standard execution engine setting. Finally, we observe that retaining the previous program generator a new generation without re-training greatly harms the performance of our best model, indicating that the learning phase is crucial.
Curiously, we note that the effects of spectral normalization and not resetting the program generator small reversed when the execution engine is not re-initialized from scratch. This can be explained by an empirical observation that a partially trained execution engine finds it easier to overfit to the input images when the programs noisy in the case of small dataset like SHAPES- SyGeT, instead of waiting for the program generator to catch up to the existing module semantics.
5.2 CLEVR AND CLOSURE
CLEVR is significantly larger and more complex than SHAPES-SyGeT. It takes an execution engine over day reach 95% task accuracy on the CLEVR validation on a Nvidia RTX-8000 GPU, even when trained with ground-truth programs without a program generator. Re-training the exe- cution engine from scratch for every generation of is thus computationally infeasible. Between using seeded II. and not resetting the execution engine at all, we find not resetting the execution engine generalizes better. CLEVR contains 699989 training questions, and we provide 100 ground- truth programs for program supervision, one-tenth of that used by Vedantam al. (2019). We find that Lensor-FiLM-NMN does not improve over Vector-NMN for CLEVR. Thus, we report results only Tensor-NMN and Vector-NMN for conciseness.
Figure 5 fillustrate the training dynamics with and without IL on CLEVR, and Table reports model performance the CLEVR validation set and the -istrf-disttibution CLOSURE categories. The CLEVR validation curves through training are presented in Appendix F Similar to Bahdanau et al. 2019a), find that Vector-NMN generalizes better than Tensor-NMN on CLEVR without II..
8

---

Published as a conference paper at ICLR 2021
Table 2: Task accuracy on the CLEVR validation set and each CLOSURE category for models trained with and without IL. using 100 ground-truth programs.
Tensor-NMN
Vector-NMN
Evaluation
Without IL
With IL
Without IL
With IL
CLEVR-Val
0.912 1 0.07
0.964 1 0.01
0.960 + 0.01
0.964 0.00
and_mátt_spa
0.278 + 0.17
0.16
0.400 = 0.13
0.335 + 0.18
or
0.327 0.11
0.481 0.24
0.367 0.11
0.563 0.23
or or.mat_apa
0.286 + 0.13
0.405 + 0.22
0.330 0.11
0.444 0.24
compare.mat
0.793 0.11
0.851 + 0.17
0.660 + 0.16
0.873 = 0.12
compare.mat.spe
0.746 + 0.13
0.853 0.15
0.677 + 0.14
0.871 0.12
embed spa. mat mat
0.824 + 0.07
0.947 0.03
0.863 = 0.07
0.900 0.08
embed_mat.epa.
0.739 + 0.14
0.941 + 0.02
0.894 = 0.03
0.936 0.03
1.0
1.0
6.9
0.8
Accurary
0.6
0.4
Vector-
Vector NMN
6.5
NMN
0.2
NMN+IL
Tensor NIIN+LL
Tonson-NMN4
0.4
0.0
A00n0
8,0000
1200000
200000
120000
180000
200000
Steps
Stops
(a) Task training accuracy.
(b) Program accuracy.
Figure 5: Learning curves of models trained with and without IL on CLEVR using 100 ground-truth programs. The IL curves global gradient step counter across all phases. The dips in the IL training curves indicate the beginning of new generations.
However, Tensor-NMN systematically generalizes substantially better with IL. Across both models, using IL improves generalization CLOSURE, greatly increases program accuracy, and achieves validation performance on CLEVR close to ProbNMN (Vedantam et al., 2019) despite using far fewer programs for supervision.
CONCLUSION
We establish IL as a practical tool for the emergence of structure in machine learning by demonstrat- ing its utility in challenging VQA tasks using NMNs. Our work shows that IL leads to improved performance with less supervision and facilitates systematic generalization. As the study IL. in neural agents is relatively young and has largely been explored in the narrow domain of language emergence in referential games, our setting presents several important new challenges for IL meth- ods. Unlike in simple games (Ren al. 2020), the emergence of compositional language without any supervision is thus far infeasible in VQA due to the difficult joint optimization of the program generator and the execution engine. However, by exploring learning phase regularization, suitable model architectures, and the learning dynamics of IL, we are able to dramatically reduce the amount of ground-trult supervision necessary; the surprising success of spectral normalization in particular should motivate further research into the role of regularization during the learning phase. We hope that this progress will spur future work to improve IL methods and their theoretical understanding. as well extend them to other challenging tasks in machine learning.
ACKNOWLEDGMENTS
We acknowledge the financial support of Samsung, Microsoft Research, and CIFAR.

---

Published as a conference paper at ICLR 2021
REFERENCES
Jacoh Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Dzmitry Bahdanau, Harm de Vries, Timothy O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. CLOSURE: assessing systematic generalization of clevr models. arXiv preprint
2019a.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2019b.
Matthew M Botvinick and David C Plaut. Empirical and computational support for context- dependent representations of serial order: Reply to bowers, damian, and davis (2009). 2009.
Jeffrey S Bowers, Markus F Damian, and Colin J Davis. A fundamental limitation of the conjunctive codes learned in pdp models cognition: Comment on botvinick and plaut (2006). 2009.
Philémon Brakel and Stefan Frank. Strong systematicity in sentence processing by simple recurrent networks. In Proceedings of the Annual Meeting of the Cognifive Science Society, volume 31, 2009.
Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life. 12(2):229-25 242, 2006.
Paco Calvo and John Symons. The architecture of cognition: Rethinking Fodor and Pylyshyn's systematicity challenge. MIT Press, 2014.
Rahma Chaabouni, Hugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 1427-4442. Association for Computational Linguistics, 2020.
Franklin Chang. Symbolically speaking: connectionist model of sentence production. Cognitive science, 26(5):609-651, 2002.
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent compositional communication learning from raw visual input. In International Conference on Learning Representations, 2018.
M Christiansen and Nick Chater. Generalization and connectionist language learning. Mind and Language, 9(3), 1994.
Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of composi- tional language with deep generational transmission. arXiv preprint arXiv:1904.00 090609094. 2019.
Gautier Dagan, Dicuwke Hupkes, and Elia Bruni. Co-evolution of language and agents in referential games. arXiv preprint vrXiv:2001.03300 2020.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in a multi-modal, multi-step referential game. In International Conference on Learning Repre- sentations, 2018.
Jerry A Fodor and Ernest Lepore. The compositionality papers. Oxford University Press, 2002.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analy- sis. Cognition, 28(1-2):3-71, 1988.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in neural information processing systems, pp. 2137-2145, 2016.
10

---

Published as a conference paper at ICLR 2021
Shangmin Guo, Yi Ren, Serhii Havrylov, Stella Frank, Ivan Titov, and Kenny Smith. The emergence of compositional languages for numeric concepts through iterated learning in neural agents. arXiv preprint arXiv:1910.05291,, 2019.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agen games: Learning to com- municate with sequences of symbols. In Advances in neural information processing systems, pp- 2149-2159, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 804-813, 2017.
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine rea- soning. International Conference on Learning Representations, 2018.
Drew A Hudson and Christopher D Manning. GQA: new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax, In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei. C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2901-2910, 2017a.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reason- ing. In Proceedings of the IEEE International Conference on Computer Vision pp. 2989 2998, 2017b.
Emilio Jorge, Mikael Kågebäck, Fredrik D Johansson, and Emil Gustavsson. Learning to play guess who? and inventing a grounded language as a consequence. arXiv preprint 1:Xiv:661..03211 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learning model of the emer- gence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, 5(2): 110, 2001.
Simon Kirby, Hannah Cornish, and Kenny Smith. Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language. Proceedings of the National Academy of Sciences, 105(31) 10611-1666666 2008
Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution of language.
Current opinion in neurobiology, 28:1 121-108444 114, 2014.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'nat- urally' multi-agen dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural I Language Processing, pp. 2962-2967, 2017.

---

Published as a conference paper at ICLR 2021
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi- ageit cooperation and the emergence of (natural) language. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings, 2017.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Confer- ence on Learning Representations, 2018.
David Lewis. Convention: philosophical study. John Wiley & Sons, 2008
Fushan Li and Michael Bowling. Fase-of-teaching and language structure from emergent commu- nication. In Advances in Neural Information Processing Systems, pp. 15851-15861, 2019.
Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication. In International Conference on Learning Representations, 2020.
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. Countering language drift with seeded iterated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 6437-6447. PMLR, 2020.
Gary Marcus. Rethinking eliminative connectionism. Cognitive psychology, 37(3):243-282, 1998.
Gary Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Steven Phillips. Are feedforward and recurrent networks systematic? analysis and implications for connectionist cognitive architecture. Connection Science, 10(2):137-160, 1998.
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B Cohen, and Simon Kirby. Compositional lan- guages emerge in a neural iterated learning model. In International Conference on Learning Representations, 2020.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
Catriona Silvey, Simon Kirby, and Kenny Smith. Word meanings evolve to selectively preserve distinctions salient dimensions. Cogmitive science, 39(1):212-226, 2015.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Individualized controlled continuous communication model for multiagent cooperative and competitive tasks. In International Confer- ence on Learning Representations, 2019.
Brian Skyrms. Signals: Evolution, learning, and information. Oxford University Press, 2010.
Luc Steels and Martin Loetzsch. The grounded naming game. Experiments in cultural language evolution, 3:41-59, 2012.
Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropagation. In Advances in neural information processing systems, pp. 2254-2252, 2016.
Frank van der Velde, Gwendid T van der Voort van der Kleij, and Marc de Kamps. Lack of combi- natorial productivity in language processing with simple recurrent networks. Connection Science, 16(1):21-46, 2004.
12

---

Published as a conference paper at ICLR 2021
Ramakrishna Vedantam, Karan Desai, Stefan Lee, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Probabilistic neural symbolic models for interpretable visual question answering. In Proceedings of the 36th International Conference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6428- 6437. PMLR, 2019.
Willem H Zuidema. How the poverty of the stimulus solves the poverty of the stimulus. In Advances in neural information processing systems, pp. 51-58, 2003.
13

---

Published as a conference paper at ICLR 2021
Table 3: IL hyperparameters used in our experiments.
Hyperparameter
Value
Optimizer
Adam (Kingma Ba 2015)
PG learning rate
0.0001
EE learning rate
0.0001 for CLEVR, 0.0005 for SHAPES-SyGeT Tensor-
NMN, 0.001 for Tensor-FiLM-NMN
Batch size
128
GT programs in batch
4
PG REINFORCE weight
10.0
PG GT cross-entropy weight
1.0
PG spectral normalization
On for SHAPES-SyGET, off for CLEVR
Interacting phase length T;
2000 for SHAPES-SGGeT, 5000 for CLEVR
Transmitting dataset size T;
2000 x batch size = 256000
PG learning phase length T,
2000
EE learning phase length T
For SHAPES-SyGeT, 250 when re-initializating EE from scratch, 200 for seeded IL; For both SHAPES-SyGeTT and CLEVR, 50 when not resetting EE
A HYPERPARAMETERS
Table 3 details the hyperparameters used in experiments in Section for both SHAPES-SyGeT and CLEVR.
B MODEL ARCHITECTURE DETAILS
B. PROGRAM GENERATOR
The program generator is seq2seq model that translates input questions programs. We the Seq2SeqAtt model Bahdanau et al.) (2019a). where unlike Johnson et al. (2017b). the decoder uses attention mechanism (Bahdanau et al. 2015) over the encoder hidden states. We found this model to generalize better on held-out in-distribution questions in preliminary experiments. Ideally, the program generator's target must be tree-structured since the output program represents a computation graph. Since seq2seq model cannot directly model graphs. we instead model serialized representation of the programs. Like Johnson al. (2017b), we chose to use the Polish (or prefix) notation to serialize the syntax tree.
Considering the program generator parameterized by 0, the probability distribution over layouts is given by
(2 q) II (e(2s 214-1.9). 2=1
(6)
During training, we sample programs from this distribution but take the arg max at each timestep when evaluating model. We LSTM (Hochreiter & Schmidhuberi 1997) trained using
teacher forcing to represent the distribution pe (2, 2 9) at every timestep To condition on
q for each decoder step, we perform attention the hidden states of bidirectional LSTM encoder over the question
B.2 EXECUTION ENGINE
To run the program produced by the program generator, the execution engine assembles neu- ral modules according to into a neural network that takes input to produce answer ÿ. We consider this execution engine EE, &) to be parameterized by o. To illustrate a for- ward pass through the module network, consider the program 'and color [green] scene trans form [left of] shape [qquare] scene' from Figure "scene' is special mod- ule which represents the input image features from CNN feature extractor. According to the
14

---

Published as a conference paper at ICLR 2021
annotators, the 'color [green]' module should filter green shapes from the scene, and the "shape [square] module should filter squares. "transform [eoor__eeeofo]]] should select re- gions to the left of the filtered squares. In Figure the final 'and' module should then compute the intersection of green shapes and the region to the left of squares. A classifier takes this output from the top-level module to produce distribution ÿ over answers. We consider the final answer be the arg max of
Each module corresponds one token of the program and can be reused in different configurations. Like Johnson et al. (0177)) our modules are not hand-designed to represent certain functions but have a similar architecture for all operators. Other than the 'scene' module which has arity 0, all other modules are implemented as a CNN on the module inputs to produce module output. In this work, we explore three module architecture choices to make up the execution engine: Tensor-NMN, Vector-NMN, and Siusor-FilMM-MM. Here, the Tensor-NMN module architecture is the module network proposed by Johnson et al. (2017b), Vector-NMN is the architecture from Bahdanau et al. 2019a), and Tensor-FiLM-NMMN is described in Section 3.1
C ITERATED LEARNING ALGORITHM
We present the full IL algorithm described in Section 3.2 in Algorithm]
|
D SHAPES-SYGET TEMPLATES AND SPLITS
The images in SHAPES are arranged 3 x 3 grid, where each grid cell can contain a triangle, square, circle that is either red, green, blue, or the cell be empty. Table 4a shows the standard splits for training and evaluating SHAPES used in the literature (Andreas et al. 2016 Hu al. 2017 Vedantam al. 2019). We categorize SHAPES questions into 12 standard templates, listed in Table 5 We then derive new splits for the SHAPES data, such that each split has the same primitives, but different ways in which the primitives are combined. The final split of the SHAPES-SyGeT dataset is presented in Table 4b
E ABLATION EXPERIMENTS
We perform ablation experiments on SHAPES- TyGeT rather than CLEVRICLOSURE as the com- putational requirements of experiments on this dataset order of magnitude lighter. Results are which we find benefits far more presented in Table 6 We focus primarily on ground-truth programs for supervision, choice from IL than Tensor-NMN. In all cases, which we find to clearly reflect the differences between the various settings considered. We describe the important observations from our ablation study in Section 5.1
20
F CLEVR VALIDATION CURVES
The validation curves for CLEVR, illustrated in Figure 6 are similar the training curves in Fig- ure [5] Although CLEVR training exhibits a lower amount of overfitting compared to SHAPES- SyGeT, observe higher systematic generalization performance CLOSURE with models trained using II.
G PRELIMINARY EXPERIMENTS WITH GQA
To study if II can offer benefits for larger datasets without synthetic images and questions, we choose to evaluate our IL method on the GQA dataset (Hudson & Manning 2019). As preliminary
setup, we remain as close as possible to the architectural setup described in Section 3 We only use question text, spatial image features from a pre-trained ResNet-101 (He et al. 2016). and programs, all without object annotations. We serialize the programs according to the Polish notation as we do in the case of SHAPES Syeee and CLEVR, which can result in duplicated token sequences GQA programs do not always form strict directed acyclic graphs. Although these limitations allow us to easily apply the presented method to GQA, they also prevent us from getting competitive results
15

---

Published as a conference paper at ICLR 2021
Algorithm 1: Iterated learning for compositional NMN layout emergence.
Data: Nepochs: number of epochs, (T; Tú, T Te): length of each phase, D: training dataset. Initialize PG parameters 0 and EE parameters dj:
for to Nepocha do /. Interacting phase
for i T, do
Sample new tuple (q, (q, v) from D; PGo. (g):
I. < cross-entropy-loss(yy EE (a. 2)):
is available then + + cross-entropy-loss(az 2):
Update On and on to minimize L;
end
Transmitting phase D' Ø; for to T, do
10
11
sample new tuple (q, a. from D; if is available then 2;
13
else & ~ PGo, (q): Add (g. 2) to D': end
15
16
/+ Program generator learning phase Initialize PG parameters 8n-1:
17
for i to T do
19
Sample new tuple (q, 2) from D': P. (q):
20
+ cross-entropy-loss(22.
22
Update 8,+1 to minimize L with spectral normalization; end
23
/* Execution engine learning phase Initialize EE parameters Φn+1;
24
for i to T. do
26
Sample new tuple (q, x, x. v) from D;
27
2 (G)
L + cross-entropy-loss(yy Ey,, (x, 2));
29
Update Φ.+1 to minimize L;
30
end
31 end
16

---

12. is COLOR shape COLOR Published as a conference paper at ICLR 2021
Total questions
Unique questions
Total questions
Unique questions
Split
Split
Train
13568
212
Train
7560
135
Val
1024
16
Val-III
1080
135
Test
1024
16
Val-OOD
6976
109
(a) SHAPES dataset.
(b) SHAPES- SyyeeT dataset.
Table 4: Split of questions in the SHAPES and SHAPES-SyGell SSeTT datasets.
Total questions
Unique questions
Template
Train templates
8640
135
1. is a COLOR shape RELATIVE (1) a COLOR shape
2304
36
2. is SHAPE RELATIVE (1) COLOR shape
2304
36
3. is a SHAPE RELATIVE (2) a COLOR shape
960
15
4.
is SHAPE RELATIVE (2) a SHAPE
1344
21
5. is a COLOR shape a SHAPE
576
9
6. is SHAPE COLOR
576
7. is SHAPE a SHAPE
576
9
Evaluation templates
6976
109
8.
is COLOR shape RELATIVE (2) a COLOR shape
960
15
9.
is COLOR shape RELATIVE (1) SHAPE
2304
36
10.
is a COLOR shape RELATIVE (2) a SHAPE
832
13
11.
is SHAPE RELATIVE (1) SHAPE
2304
36
576
9
Table 5: Overview and splits of the SHAPES- templates. The placeholder COLOR can take values in 'red,' 'green,' and 'blue,' and SHAPE can be either 'circle,' 'triangle, or 'square.' RELATIVE (1) placeholder for one of the positional prepositions 'above," "below," 'left of,' or 'right of," whereas RELATIVE (2) is combination of two RELATIVE (1) s, such as 'above left of.'
the dataset. However, goal is not to pursue state-of-the-att performance GQA, but instead to study if we can still observe the advantage of IL a limited ground-truth program setting for NMN setup. Still, due to the significantly higher complexity of GQA, few modifications and special considerations become necessary.
G.1 CHANGES TO THE MODEL. ARCHITECTURE
In both SHAPES-SyGeT and CLEVR, the programs are sequences of tokens where each token represents an operation, corresponding one module of the assembled NMN. Each possible op- eration is carried out on the module input, the input image, or a combination of both, and has fixed arity. However, the structure of programs in GQA has additional levels of granularity. Each timestep has an operation (e.g. exist. query), weather). an integer arity, and list of argument tuples. Each argument tuple this list con- tains an argument (e.g. horse, water) negated. We ignore additional details in the programs provided in the GQA dataset. As an illus- tration, the question "does the lady to the left of the player wear skirt?" translates the pro- gram "verify rel (skirt, wearing, o) {1} relate (lady, to to.the.left.on s) {1} select (player) {0}", containing three tokens with arities 1, 1, and O respectively. The op- erations and the corresponding sub-operations are separated by period (.), and the parentheses contain the list of arguments, none of which are negated in this example.
optional sub-operation (e.g. material,
well as a Boolean value indicating if the argument is
Since every timestep of the program sequence is given by a tuple (Op, Subop, Args. ArgNegs, Arity), modify the program generator produce this tuple instead of a single operation. For Args and
17

---

Published as a conference paper at ICLR 2021
PG: Seq2SeqAtt
EE: Tensor-NMN
VaI-IID
Val-OOD
+FullSN
0.646 = 0.03
0.614 0.01
+NoSN
0.645 + 0.01
0.616 0.01
EE: Tensor-FiLM-NNN
+FulISN
0.642 0.02
0 0.02
+NoSN
0.649 = 0.02
0.605
+IL
+IL
0.954 = 0.07
0.858 0.15
+IL+NNSN
+IL
0.803 + 0.13
0.630 0.07
+lL+NoRetrain
+IL
0.834 0.13
0.664
+IL
+eLdee
0.792 = 0.05
0.626
+IL+NoSN
Seedee
0.744=0...
0.647
+II+NNRetrriin
Seeded
0.794 + 0.12
1662+0.1
NoRset
0.624 + 0.02
0.586 0.01
+ +IL+NoSN
0.629 + 0.01
0.588 + 0.01
+ IL+NoRetrain
NoRett
0.682 + 0.05
0.600 + 0.01
Table 6: Ablation experiments for models SHAPES-SyGed trained with 20 ground-truth pro- grams. 'FullSN" spectral normalization applied throughout training:"NoSN" no spectral normal- ization in any phase; 'NoRetrain': new program generator is taken from the previous generation without any re-training: 'Seeded': new execution engine is initialized to the state after the previous learning phase; 'NoReset': execution engine re-initialized at the start of a generation.
1.0
0.9
0.8
Acturry
0.5
0.6
Tenem-NNMX
Veclon-NMK +I.
NMN
40000
80000
120000
1600C0
200000
Steps
Figure 6: Validation curves of models trained with and without IL on CLEVR using 100 ground- truth programs.
ArgNegs, we consider a fixed argument list length of 3, padding with the argument <NULL> and no negation when the argument list is shorter than 3. This allows to represent the distribution over the tokens at various levels of granularity for every timestep using a fixed number of logits.
Finally, it is longer feasible have dedicated modules for every possible unique program step, as the number possible (Op. Subop, Args, ArgNegs Sifeessssccoom combinations very large. Thus, we restrict ourselves using the Vector-NMN architecture, where instead of using FiLM embedding for the operation the module represents, we concatenate embeddings for the Op, Subop, Args, and ArgNegs generate timestep-specific FiLM embeddings.
G.2 CHANGES TO THE METHOD
Unlike the question and program structures of SHAPES SyGeT and CLEVR, the structures in GQA significantly more diverse and include larger vocabularies illustrated in Table This makes training the program generator to a decent performance, even when trained on ground-truth pro- grams directly, prohibitively slow. As result, it becomes impractical train the program generator
Arriyy is consumed during the assembly of the NMN.
18

---

Published as a conference paper at ICLR 2021
Dataset
Token type
Vocabulary size
Question
18
SHAPES-SyGer
Program
16
Question
93
CLEVR
Program
44
Question
2939
Program Op
16
GQA
Program Subop
55
Program Arg
2659
Table 7: Vocabulary sizes for SHAPES-SyGer CLEVR, and GQA token types after pre-processing.
#GT programs
Model
Accuracy
Vector-NMN on GT programs
0.556 + 0.002
943000
Vector-NMN
0.486 + 0.002
4000
Vector NMN
0.455 + 0,006
4000
0.469 + 0.007
4000
0.470 + 0.007
4000
Vector NMN+IL
0.480 + 0.003
Table GQA validation accuracies. Vector-NMN on GT programs, the execution engine always assembles modules according to the ground-trute programs, including at test time. For all the models except Bector-NMN+IL, the program generator retained between generations. 'ResetEE': reset parameters except the FiLM embeddings; FFinetunEEE:: train the execution engine on frozen program generator before joint training; 'IL': the full IL algorithm.
from scratch in every generation of IL. We thus need to explore strategies of resetting the program generator between generations while maintaining the benefits the learning bottleneck IL.
We hypothesize that due to the larger vocabulary sizes, learning good embeddings for question tokens and the various program tokens at all levels of granularity constitutes large portion of the program generator training time. Following this intuition, output embeddings of the questions and the programs while resetting other parameters between generations. Experimentally,
find it helpful to retain the input and
verified that this method of resetting generalizes better than resetting the program generator at all during IL. It also outperforms an alternate strategy of choosing new parameters based on an interpolation of freshly initialized program generator and the final program generator from a generation.
Finally, despite search over the Vector-NMN hyperparameters, we find that the optimal hyper- parameters have tendency to exhibit overfitting on the GQA images. The strategy of retaining embeddings and resetting the rest of the parameters also works well to combat this overfitting for the execution engine, where the embeddings we retain are the various FiLM embeddings. In Sec- we compare our method with baselines that only implement this partial reset of the execution engine regularizer.
tion G.3
G.3 RESULTS
For our GQA experiments, results respectively. For each run, we report the mean and standard deviation across 3 trials. To study the effect of IL with limited ground-truth programs, ground-truth programs, which constitutes only 0.4% of all available training programs.
the balanced train and validation splits for training and reporting
our IL experiments with 4000
Table presents the results for different configurations of models. A Vector-NMN execution engine that assembles modules using ground-truth programs during both training and validation provides upper bound of performance over learned program generator. When learning the
19

---

Published as a conference paper at ICLR 2021
1.00
1.00
0.95
0.85
0.90
Acsuracy
0.90
0.85
0.85
Veclor-NMIN
Vecter-NMY
Veclor-NMNN- RE Vector NMN 6 RE FizetunnEE
Vecler-NMEN IbweetER
Vecter NDES RusstEE FinetuneEE NMN++L
Vector
50000
100000
150000
2000 00
250000
Scoc0a
50000
100000
150000
200000
250000
300000
Steps
(a) Op.
(b) Spbopp
Accurazy
Vector ReseeEE
Voctor NMNN + ResetEE
Vector NMNN-RsSeEE- PiReeinEEE
Voctor B Einetuncet
NMN-I--NMN-IL
Vector NMN-IL
50000
100000
150000
20000n
250000
30000a
5onon
10000n
Scóca
200000
250000
30000n
(c) SuboptArgg
(d) OpttSubpp-ArgtArrNee
Vector MN Resettil
Voctor N3VV BesetEE Firotune
Vector NMN+IL
50000
100000
130000
200000
250000
3000000
(e) Op+-SsbbottarrrArggggg--tttt
Figure 7: Program accuracies at various levels of granularity of models trained on GQA using 4000 ground-truth programs. Here, (e) uses the strictest notion of program accuracy. The models with any form of IL use global gradient step counter across all phases. The dips in the curves indicate the beginning of new generations.
20

---

Published as a conference paper at ICLR 2021
0.40
Acturacy
0.46
0.6
0.46
Vector KMN
Vector-NMX
0.2
Vartor-NM N-rr--MNN-RR-ooE
0.42
Vecter- NMN Ileeel
Vector 4
Vector- FinetunedE
0.40
50000
10n0no
1500nn
200c0n
250000
50006
130000
200000
2500000
306000
Steps
Steps
(a) Training accuracy.
(b) Validation accuracy.
Figure 8: Training and validation curves of models trained on GQA using 4000 ground-truth pro- grams. The models with any form of IL use a global gradient step counter across all phases. The dips in the training curves indicate the beginning of new generations.
program generator, find that using 4000 ground-truth programs with IL performs almost well as using 943000 ground-truth programs without IL. Due the tendency of the execution engine to overfit the GQA images, also compare regularization through iterated partial resetting of the execution engine. While we find this method to generalize better than standard training, still find it necessary to employ the learning bottleneck through the learning phase of the program generator to achieve the best accuracy in the limited- program- mograa--upervision setting.
IL model with stronger baselines that perform
We see a clear advantage of IL in learning the correct program structure in Figure 7 which re- ports program accuracy at various levels of granularity. We notice that this increase the program accuracy correlates with an improved generalization performance in Figure 8, even though the train- ing accuracy of all the iterated models is similar. These preliminary experiments on GQA indicate that a learning bottleneck can be beneficial even in larger non-synthetic datasets, but may require additional considerations such partial resetting to make the training time tractable.
21


Reviews:
[{"review_id": "njolLhrWLWe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors apply iterated learning - a procedure originating in CogSci analyses of how human languages might develop - to the training of neural module networks. The goal is for iterated learning to encourage these networks to develop compositional structures that support systematic generalization without requiring explicit pressures for compositional structures (in past work, such explicit pressures have generally been necessary). The proposed approach brings substantial improvements in systematic generalization across two datasets, SHAPES and CLEVR.\n\nStrengths:\n\n1. The approach is well-motivated, including impressive coverage of prior literature in both ML and CogSci.\n\n2. The approach brings impressive gains in an area that is one of the major weaknesses of current ML systems, namely systematic generalization. \n\n3. In addition to the gains in accuracy, one particularly impressive benefit of this approach is the decreased amount of supervision that it requires compared to past approaches.\n\n4. The paper is generally well-written and easy to follow. \n\nWeaknesses:\n\n1. One of the motivations is to expand the use of iterated learning beyond toy datasets. While SHAPES and CLEVR may be not as toy-ish as datasets used in the past, they still are pretty toy-ish, so I’m not sure if this paper can reasonably claim that one of its contributions is to expand iterated learning to realistic domains.\n\n2. Though the paper in general was very clear, I found Section 3.2 to be a bit hard to follow, and that section is important as it is the part that describes the structure of the iterated learning framing. I think this section would benefit from starting each subpart with a more high-level, intuitive description of what that stage accomplished, before diving into the details.\n\nMinor comments:\n\n1. Fodor et al only has 2 authors - Fodor and Pylyshyn\n\n2. Page 3: “Although, the Gumbel straight-through estimator”: this use of “although” is usually frowned upon - better to use “However”\n\n3. Page 5: typo: “minimzing”\n\n4. In general, for the bibliography, check to see if a paper has been published at a conference or journal; if so, cite that version instead of the arXiv version. E.g., “Neural machine translation by jointly learning to align and translate.” was published in ICLR 2015, and “Systematic generalization: what is required and can it be learned?” was published at ICLR 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hHAsr1Et-0k", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to combine iterated learning (the process of repeated language transmission from a ‘parent’ agent to a ‘child’ agent) with neural module networks (NMNs), in order to emerge NMN layouts that perform better at systematic generalization. The paper evaluates on their new variant of the SHAPES dataset that tests systematic generalization, and on CLEVR / CLOSURE, showing improved systematic generalization performance while requiring a small amount of ground-truth layout supervision.\n\nPros:\n- I think the idea of combining IL with NMNs is really clever (heh). Treating the program generator and execution engine as two agents that need to coordinate through a shared language (NMN layouts) is really interesting. If it were to work without layout supervision, it could open up new doors to applying IL + NMNs to many other tasks\n\n- The paper is quite well-written, and easy to follow\n\n- The related work section is thorough\n\n- The experiments on SHAPES-SyGeT, show that IL helps significantly for generalization of NMN models \n\n- I appreciate the ablations in the Appendix. \n\n\nCons: \n- One of the main questions I have about this approach is whether it will provide any benefit on more complex problems (e.g. large-scale VQA). There are a few reasons to think it might not be able to do so:\n1) As alluded to in the paper, the IL procedure requires a lot of compute, which could be used to train larger models on more pre-training data\n2) The improvement in validation performance on the more complex CLEVR dataset is fairly modest (though the program accuracy increase is large). While CLEVR is more complex than SHAPES, it is still a fairly artificial dataset targeted ‘compositional’ in nature, compared to general VQA. This suggests that it might be hard to get IL+NMN to work well on harder problems.\n3) The method still requires (a small amount of) ground-truth layout supervision, which is not obtainable in general VQA or most other tasks.\n\n- I would also like to have seen a bit more analysis / description of why the method currently fails without any ground-truth layout supervision. I think this would improve the paper a lot, as it would help other researchers improve upon the method to address this problem.\n\n\n\nOverall:\nI think the ideas in this paper are interesting enough, and the execution good enough, to warrant acceptance. While I have some concerns about whether this approach will scale, these questions will have to be answered in subsequent works and with further research.\n\n\nSmall typos:\n“each new-born child need” -> needs\n\n“Recently machine learning community also show” -> the machine learning community also shows\n\n“Minimzing” -> minimizing \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2W0yhGNtsm", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review:\nThe authors address methods to encourage the emergence of the layout expression structures on the frameworks of neural module networks (NMN) for the visual QA problems. The methods are motivated from the works on language emergence for communication between multi-agents and the language acquisition of new-born babies from parents, which achieved with limited data. The methods, ‘iterative learning’ (IL) are designed as forming two agents (program generators and execution engines) to play VQA games. Basic architectures and learning methods seem to be very similar to the approach of semi-supervised learning introduced in [ICCV17]. \nThis paper deals with one of very interesting topics, the language emergence among cooperative multi-agent environments and the compositionality of human language as recent related studies are well-surveyed in related work. In particular, the main idea of problem formation, layout expressions in NMN as emergent languages is very fresh and interesting.\nThe main claims are as follows: (1) the proposed approach of IL improves generalization performance for visual QA, and it is shown experimentally by comparing the ablation results of IL. (2) the language structures in the ground-truth data are recovered with only limited supervisions and the superiority is validated on two datasets – SHAPES-SyGeT and CLOSURE. However, I believe that the evidences for their claims are insufficient. Specifically, the authors do not provide enough information of language structure such as the superiority compared to other methods and the structure similarity of recovery levels.\nI recommend 'ok, but not good enough – reject’ for this paper. \n\n\nPros:\n- The authors propose novel interesting problem and their solutions. Arguably, it seems potentially to be on one of important research flows to make influence to lots of works for academia in the future.\n- They find and report good performance for out-of-distribution accuracies for visual QA datasets.\n\nConcerns:\n- It is not clear which parts in the proposed methods are novel with respect to previous works. Those make vague which parts are the authors’ contributions.\n- I think IL should be clarified from semi-supervised learning approaches on them for visual QA. Also, for reproducibility, it should be specified which parts are with/without IL. What is ‘learning bottleneck’ on this approach? Also, it is not enough how program generators and execution engines are specified, even though some explanations are in appendix. I think it needs more links for reference or explanation.\n- As mentioned above, the supports for main claims are not appropriate or unclear. It needs to theoretically or experimentally show the results of comparison with other methods and the similarity of the recovery level of language structures.\n- Table 1 reports the result of comparative methods such as MAC and FiLM without program supervision. How are they configured in the experiments?\n- I think that it would be better understandable to show usability and superiority with the experimental results on realistic visual QA such as VQA and GQA.\n\nMinors:\n- In Section 3.1,  γ. -> γ,  β. -> β\n- It needs the description for operators and symbols for the formulae in Section 3.1\n\n[ICCV17] Johnson et al., Inferring and executing programs for visual reasoning, ICCV 2017.\n\n------------------------------------------------------------------------------------------------------------------------------\nAfter rebuttal\n\nFrom the revised version of this manuscript, the authors resolve my major concerns such as clarity/reproducibility of the method, differentiation from the previous works including semi-supervised learning, and scalability. So, I've raised my score to 6. \nThank you for the contributions!\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an original perspective on how to learn layouts and modules of neural module networks jointly, in a way that encourages the emergence of compositional solutions. In particular, layouts are treated as messages from an emergent language, and iterated learning is used to encourage the emergence of structure. The paper shows good performance in inducing compositional structure in two datasets.

Summarizing the reviewers' doubts, one is that the idea is tested on relatively toyish data sets, and it is not clear how it would scale up. The second, coming from one reviewer, concerns a lack of originality that, honestly, I do not see. If anything, this is probably the most original paper in my pool.

Concerning the first point, that is a fair objection, but I think that getting good results on program learning on datasets such as CLEVER is more than encouraging for a paper that is introducing quite a novel idea for the first time.

Finally, the authors added new text and new experiments strenghtening their conclusion during the discussion.

I am strongly in favour of accepting this paper.

Decision: Accept (Oral)