The Role of Learning Regime, Architecture and Dataset Structure on Systematic Generalization in Simple Neural Networks
Authors: Devon Jarvis
Richard Klein
Benjamin Rosman
Andrew M Saxe
Venue: ICLR 2022 Submitted (2022)
Keywords: Systematic Generalization
Iterated Learning
Linear Neural Networks

Abstract: Humans often systematically generalize in situations where standard deep neural networks do not. Empirical studies have shown that the learning procedure and network architecture can influence systematicity in deep networks, but the underlying reasons for this influence remain unclear. Here we theoretically study the acquisition of systematic knowledge by simple neural networks. We introduce a minimal space of datasets with systematic and non-systematic features in both the input and output. For shallow and deep linear networks, we derive learning trajectories for all datasets in this space. The solutions reveal that both shallow and deep networks rely on non-systematic inputs to the same extent throughout learning, such that even with early stopping, no networks learn a fully systematic mapping. Turning to the impact of architecture, we show that modularity improves extraction of systematic structure, but only achieves perfect systematicity in the trivial setting where systematic mappings are fully segregated from non-systematic information. Finally, we analyze iterated learning, a procedure in which generations of networks learn from languages generated by earlier learners. Here we find that networks with output modularity successfully converge over generations to a fully systematic `language’ starting from any dataset in our space. Our results contribute to clarifying the role of learning regime, architecture, and dataset structure in promoting systematic generalization, and provide theoretical support for empirical observations that iterated learning can improve systematicity.

Under review as a conference paper at ICLR 2022
THE ROLE OF LEARNING REGIME, ARCHITECTURE AND DATASET STRUCTURE ON SYSTEMATIC GENERALIZATION IN SIMPLE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Humans often systematically generalize in situations where standard deep neural networks do not. Empirical studies have shown that the learning procedure and network architecture can influence systematicity in deep networks, but the under- lying reasons for this influence remain unclear. Here we theoretically study the acquisition of systematic knowledge by simple neural networks. We introduce minimal space of datasets with systematic and non-systematic features in both the input and output. For shallow and deep linear networks, we derive learning trajec- tories for all datasets in this space. The solutions reveal that both shallow and deep networks rely on non-systematic inputs to the same extent throughout learning, such that even with early stopping, no networks learn a fully systematic mapping. Turning the impact of architecture, we show that modularity improves extrac- tion of systematic structure, but only achieves perfect systematicity in the trivial setting where systematic mappings are fully segregated from non-systematic in- formation. Finally, we analyze iterated learning, a procedure in which generations networks learn from languages generated by carlier learners. Here we find that networks with output modularity successfully converge a fully systematic 'language' starting from any dataset in our space. Our results contribute to clarifying the role of learning regime, architecture, and dataset struc- ture in promoting systematic generalization, and provide theoretical support for empirical observations that iterated learning can improve systematicity.
generations
INTRODUCTION
Humans frequently display the ability to systematically generalize, that to leverage specific learn- ing experiences in diverse new settings (Lake et al., 2019). For instance, exploiting the approximate compositionality natural language, humans can combine a finite set of words or phonemes into near-infinite set of sentences, words, and meanings. Someone who understands "brown dog" and "black cat" also likely understands "brown cat," to take one example from Szabó (2012). The result that human's ability reason about situations or phenomena extends far beyond their ability to directly experience and learn from all such situations or phenomena.
Deep learning techniques have made great strides in tasks like machine translation and language pre- diction, providing proof of principle that they can succeed in quasi-compositional domains. How- ever, these methods typically data hungry and the same networks often fail to generalize in even simple settings when training data are scarce (Lake & Baroni 2018 Lake et al. 2019). Empirically, deep networks influenced by many factors. One possibility that the learning dynamics in a deep network could impart an implicit inductive bias toward systematic structure (Hupkes et al. 20200;; however, a number of studies have identified situations where depth alone is insufficient for structured generalization (Lake & Baroni 2018 8018: Niklasson & Sharkey 1992] Pollack [990] Phillips & Wiles 1993). Another significant factor is architectural modularity, which can enable system to generalize when modules are appropriately configured (Vani et al. 2021 Phillips 1995). However, identifying the right modularity through learning remains challenging (Bahdanau et al., 2019). A third possibility builds on Iterated Learning (IL), method in which generations of agents train briefly on a 'language" produced by their parent, and then generate a new language for their child (Kalish al. 2007). If systematic components easier learn
the degree of systematicity

---

Under review as a conference paper at ICLR 2022
than non-systematic ones, this process successively refine language toward systematic struc- ture, a process which has been hypothesized to be the cause of the compositional nature of natural language (Kibyy 2001, Kirby et al. 2008). In spite of these (and many other) possibilities for im- proving systematicity (Hupkes et al. 2020), it remains unclear when standard deep neural networks will exhibit systematic generalization (Dankers et al. 202T), reflecting long-standing theoretical debate stretching back to the first wave of connectionist deep networks (Rumelhart & McClelland] 1986 Fodor & Pylyshyn
1988 Smolensky, 1991 1990
Hadley 1994).
In this work we theoretically investigate the acquisition of systematic knowledge by simple neural networks (NNs). We introduce a simple space of datasets that contain systematic and non-systematic features, and examine the impact of implicit biases, architectural modularity, and iterated learning on the learned input-output mappings of shallow and deep linear networks. In particular,
We derive exact training dynamics for shallow and deep linear networks as function of the dataset parameters.
We show that for all datasets in the space, despite the possibility of learning a fully system- atic mapping, neither shallow nor deep networks do so under gradient descent dynamics.
We show that modular network architectures can learn fully systematic network mappings. but only when the modularity segregates systematic and non-systematic features.
We show that iterated learning can converge to fully systematic 'language" when com- bined with the weaker architectural constraint of output modularity.
In Section 7 we consider how our findings, which rely simplified setting for mathematical tractability, generalize to more complicated datasets and non-linear architectures by training a con- volutional neural network to label handwritten digits between and 999. Overall, our results help clarify the diverse factors impacting systematic behavior in neural networks, and suggest that iter- ated learning can improve systematicity.
BACKGROUND
Systematic generalization has been proposed key feature of intelligent learning agents which can generalize to novel stimuli in their environment (Hockett & Hockett 1960 Fodor & Pylyshyn 1988 [Hadley] 1993] Kirby al.) 22115 Lake et al. (017). In particular, the closely related concept of compositional structure has been shown to have benefits for both learning speed (Ren et al. 2019) and generalizability (Lazaridou et al. 2018). There are, however, counter-examples which find only a weak correlation between compositionality and generalization (Andreas) 2018) or learning speed (Kharitonov & Baroni 2020). In most cases neural networks do not manage to generalize systematically (Ruis al. 2020), or systematic generalization occurs only with the addition of modular architectures, explicit regularizers or a degree of supervision of the learned features.
Neural Module Networks (NMNs) (Andreas al. 2016 Hu et al. 2017 2018) have become one successful method of creating network architectures which generalize systematically. By (jointly) training individual neural modules on particular subsets of data or perform particular subtasks, the modules will specialize. These modules can then be combined in new ways when an unseen datapoint is input to the model. Thus, through the composition of the modules, the model will systematically generalize, assuming that the correct modules can be structured together. [Bahdanau al. (2019) show, however, that strong regularizers are required for the correct module structures to be learned. Thus, without regularizers, compositional mappings do not emerge with NMNs.
Iterated learning (IL) approaches suggest that systematicity can emerge over generations of agents that learn from each other. In some works, based primarily on the topological structure of language (Brighton & Kirby] 2006), compositional structure does emerge from II. (Ren et af. 2019). When paired with NMNs, IL has previously been used to refine the module structures to be compositional (Vani et al. I by treating the module structure as an output "language". Then each "word" in the language represents a particular module composed to form "sentence" which describes the model architecture (the relative arrangement of the modules in Polish notation). Thus, by phrasing the problem as language learning task, IL is able to refine this language of architectures to become modular. This work also highlights the fact that IL is applicable outside of traditional language learning tasks.
2

---

Under review as a conference paper at ICLR 2022
"Sold"ve vv "Stripe"
fled vs "Blue"
My car the ferrari, red with white stripe.
"Vollswaaeen"
Extract
Name
"Tyyaio"
Parschee
Ferraii
TMin
Example of systematicity preference in language.
Initial "Words"/Lablls
Final "Words"/Labels
Data (Logical Form)
(Phonetic Form)
(Phonetic Form)
Jeep
Sald vs Stripe
Nowkswggnn
Sald Stripe
vs Blue
Vo Nokksagenn
va
Toyeta
Opel
leep
Rorrnn
Volva
Mini
Toyeta
Opel
jeep
Porsche
Fomran
Volva
Mini
-11
-1
0
0
0
0
0
0
0
0
7
0
0
0
0
0
0
0
0
0
11
0
0
0
0
0
0
0
0
0
0
-1
O
2
0
11
0
7
0
11
0
2
0
0
2
0
O
0
7
1
-1
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
1
0
O
O
O
0
0
2
1
O
0
0
0
2
0
0
0
0
0
.7
22y
r
2
r y
I y
x
y
(b) Dataset with the settings n2, 3, ny 2, k2 1, ki = 1 and r 2.
Figure 1: Problem setting and dataset space. (a) Upon viewing an object, an agent might extract various input features, mapping these to output features or "words" in order to refer to the item (Brighton & Kirby, 2006). (b) We schematize this setting with a space of datasets containing sys- tematic (52) and non- systematic (I) features in the input (left panel) and output (middle panel). Rows contain examples and columns contain features. In this case cars are named with both a systematic component (based on features: presence of stripe, colour and number of doors) and non-systematic component (based on brand). In iterated learning, the names objects change from the initial labels over generations until they stabilize at the refined final labels (right panel. Sty, f,y).
IL has also been successfully used with NNs as part of larger pipeline. In particular Seeded Iterated Learning (SIL) (Lu et al. 2020a) uses IL to counter language drift. Language drift occurs when a pretrained language model is then used to complete a language based task. While training on the task, the model begins to lose structure in the learned language that is not useful for performing the task. By training a "teacher" model on task completion and then using the teacher supervise pretrained "student"", [u et al. (2020a) show that the student network is able to learn to complete the task while maintaining the initially learned language. SIL is extended by [Lu et al.] (20206) to include supervised selfplay in the teacher task-learning step. This is known as Supervised SIL (SSIL). Thus, parts of the data used to pretrain the teacher are also replayed while the teacher learns to perform the task. This also aids in avoiding language drift. While these previous works reflect the influence of certain initial data properties, regularizers or complex architectures on the emergence of compositional structure, the necessity of structured architectures with modular designs remains unclear.
3 A SPACE OF DATASETS WITH COMBINATORIAL SYSTEMATICITY
The notion of systematic generalization is broad, and has been assessed using a variety of datasets and paradigms (Hupkes et al.) 2020). Here we introduce a simple setting reminiscent of learning to refer to objects in an environment, as depicted in Fig. [ For instance, we might see a particular car (Fig. ()) extracting various perceptual input features such as whether it has a stripe, is red, or has a particular hood ornament. This scenario affords several ways in which we could refer to the car. We could systematically map individual input features to individual outputs by saying sentence like "My car the red one with the white stripe;" or we could choose a single word to refer to some combination of properties, saying something like "My car is the Ferrari." This task has type
3

---

Under review as a conference paper at ICLR 2022
of combinatorial systematic structure, because individual input features can link independently to individual output features. More generally, an agent first observes an object and forms an internal "logical form". The task is then to map from this logical form to a "phonetic form" by providing name for each object (Brighton & Kirby, (006). As illustrated in this example, real world settings often allow redundant expressions and mappings, such that the same object could be individuated with a variety of expressions that vary in systematicity.
To formalize this setting, we define a parametric space of datasets with input and output matrices x x [22 ]=]" and Y = [ r'y respectively, where fa, fly kz ky, r E IR+ are the parameters that define a specific dataset. Figure Dp visualizes one dataset in the space. The systematic inpur feature matrix f2 E {-1.1}"a x2" consists of all binary patterns with ny bits. Here ny is a key parameter determining the number of bits in the systematic input structure. Overall, the dataset con- tains 2 examples. The systematic output feature matrix fy E {-1, 1 x=**=*** is sampling of ny features (rows) from S2g. and is the systematic component of the output matrix. Intuitively, the binary combinatorial structure in S2g and Sty is meant to reflect systematic mappings from input features (the color red or blue) to output features (the words "red" or "blue"). Next, the non-rystematic input feature matrix F2 [r1 ... rI,, consists of k₂ scaled identity matrices, I3 E 1). 222 x2" Simi- larly, the non-systematic output matrix r = [r ... r/yy] has ky scaled identity matrices, with scale factor r. These identity matrices provide a single feature for each pattern which is only on for that pattern. Intuitively, these features are meant to reflect idiosyncratic features (like hood ornament, a distinctive scratch, or a proper noun name) or features reflecting specific nonlinear combinations of systematic features tred-striped-2-door Together kæy ky and control the frequency and intensity (for example the size or perceptual prominence of the brand's logo) of the non-systematic features, which are both factors that can promote non-compositional language being used by humans (Rogers) ct al. 2004).
This space of datasets is consistent with numerical notions of compositionality in previous works (Andreas) 2018), which define compositionality as a homomorphism between the observation space and the naming space (since the input-output mappings are linear they are homomorphic). The amount of systematic structure can be titrated by adjusting ng and ny: and the prevalence of non- systematic structure by adjusting kz, ky and r. Importantly, and novel to our analysis, datasets in this space allow redundant solutions: the systematic output features can be generated based on systematic input features alone, but they equally be generated using non-systematic features alone, or some mixture of the two. Exploiting this fact, we now ask how reliance on systematic or non-systematic structure is influenced by implicit biases, architecture, and learning regime.
4 LEARNING DYNAMICS IN SHALLOW AND DEEP LINEAR NETWORKS
The generalization abilities of deep networks depend on a complex interplay of learning dynamics, architecture, initialization, and dataset statistics. In this section we ask how the implicit inductive bias in gradient descent influences systematicity. We consider training both shallow and deep linear networks on datasets in our space. While deep linear networks can only represent linear input-output mappings, the dynamics of learning change dramatically with the introduction of one or more hidden layers (Fukumizu) 1998 Saxe et al.] 2014 2019, Arora et al., 2018 [ampinen & Ganguli] 2019). and the learning problem becomes non-convex (Baldi & Hornik 1989). They therefore serve as a tractable model of the influence of depth specifically on learning dynamics, which prior work has shown to impart a low-rank inductive bias on the linear mapping (Huh et al. (2021).
We leverage known exact solutions to the dynamics of learning from small random weights in deep linear networks (Saxe et al. 2014 2019) to describe the full learning trajectory analytically for every dataset in our space. In particular, consider a single hidden layer network computing output =W?W' in response to an input x, trained to minimize the mean squared error loss using full batch gradient descent with small learning rate c (full details and technical assumptions given in Appendix A). The network's total input-output map after t epochs of training is
(1) where A(t) is a diagonal matrix of singular values. The dynamics of A(t). as well as the orthogo- nal matrices U and V, depend on the singular value decomposition of the input- and input-output correlations in the dataset. If the input- and input-output correlations can be expressed as
(t (t)= = U UA(T)TT,
== = = FXX
VDVT.
E
F x = USVT
4

---

Under review as a conference paper at ICLR 2022
where U and V are orthogonal matrices of singular vectors and S are diagonal matrices of sin- gular values/eigenvalues, then the diagonal elements of A(t)aa = Tx evolve through time as
Aa/ /0
To (8)
Am Bo To
exp
21
where >a and 6a are the associated input-output singular value and input eigenvalue (Saa and Deva I respectively), To denotes the singular value at initialization, and 2 is the learning time constant. These dynamics describe a trajectory which begins at the initial value To when = 0 and increases Aa/ /5, as l
In essence, the network's 's total input-output mapping at all times in training is a function of the singular value decomposition of the dataset statistics. We therefore analytically obtained this de- composition in terms of the dataset parameters for our space of datasets. We find that there are three distinct input-output singular values which we denote A1 22 and X3; two distinct input singular values 81 and d2; and therefore three asymptotes πi T2 and nz.
(Rirr2 2m (Rig + 2
kyr? + 2
\1
(4)
ai A1101
(5)
22
ligh + 2m
(kir12 + n (k)
A2
(6)
72" -2/011
(7)
22m
1ig 2 + 2
kakkyr
kig
A3
(8)
33 >a/82
(9)
22mz
kin
Substituting these expressions into the dynamics yields the full learning trajectories for all datasets in the space. We note that each distinct singular value occurs multiple times in the dataset: A has multiplicity nhe Ry. X2 has multiplicity ny, and Ag has multiplicity 2 F2p. Full derivations,
including explicit expressions for singular vectors, are deferred to Appendix B
A similar derivation for a shallow network (no hidden layer) shows that the singular values of the model's 's mapping follow the trajectory
na (t) Aa (1 exp (-5,t/r) + To exp exp (-5,t/T),
(10)
such that the time course depends on the singular values of the input covariance matrix, Σ². The unique singular values are
2 + 2m
61
(1I)
82
(12)
2
2n
We empirically verify these equations by simulating the full training dynamics for deep and shallow linear networks trained using gradient descent on an instantiation from the space of datasets in Figure While training, we compute the singular values of the network after each epoch of training. These simulations of the training dynamics for each unique singular value are then compared to the predicted dynamics. We see close agreement between the predicted and simulated trajectories!
4.1 THE EVOLUTION OF SYSTEMATICITY OVER LEARNING
To understand the extent to which a network comes to rely on systematic or non-systematic input features, and the timing with which it produces systematic non-systematic output features, calculate the Frobenius norm of the input-output mapping between different subsets of features. In particular we partition the input-output mapping into four components: systematic inputs to sys- tematic outputs; non-systematic inputs to systematic outputs; systematic inputs to non-systematic outputs; and non-systematic outputs non-systematic outputs. Analytical expressions for these norms over training (which rely on both singular value dynamics and the structure of the singular
All experiments run using the Jax library (Bradbury et al. 2018). Full code for reproducing all figures contained in the Supplementary Material.
5

---

Under review as a conference paper at ICLR 2022
1-0
Porater 2L11
Peddction
Porsted E.
P Sesttte TLL
E,,, Nerr
3.2
mneeme Tilchom
2.30
10.00
2.00
210
2.0
Eauth surber
(a)
(b)
1.0
Prodecion
Ped to
123
1.5
Prodecoo
Medt
C4
Bed
Predecion
Mois Pacital F.T.
0.20
wanber
(c)
(d)
Figure 2: Analytical learning dynamics for deep (panels a-b) and shallow (panels c-d) linear net- works. (a,c) Comparison of predicted and actual singular value trajectories over learning, for the three unique dataset singular values. (b,d) Comparison of predicted and actual Frobenius norms of the input-output mapping to/from systematic (22,, S2y,) and non-systematic (T r,)) features. Deep networks show distinct stages of improvement over learning. However, at no point is purely sys- tematic mapping learned. Parameters: nz 3, ny = k2 = 3, ky 1,r
vectors) are given in Appendix [7]0duuee to space constraints. Figure depicts these dy- namics for one specific dataset. Crucially, we find that, first, systematic outputs are generated using substantial contributions from non-systematic inputs, regardless of network depth (see r L. curve); and second, contributions to non-systematic outputs y arise simultaneously with contributions to systematic outputs 2, (all curves initially rise together). Hence perfectly systematic mappings never arise any point in the learning process, either for shallow deep networks. For these datasets in which a given mapping can be implemented using either systematic or non-systematic features, the implicit bias in learning dynamics settles mixed solution.
While the results in Figure 2 are for one specific dataset, our analytical results offer more general insight throughout our space of datasets. We note that in the deep network, the timescale of learning each unique singular value is roughly O(1AA), such that larger singular values are learned faster (Saxe et al. 2019). In Appendix A we show that X > X2 A for any setting of the dataset parameters, and _ 2 n.² > T assuming that Kg 2 ky (i.e., the non-systematic output dimension larger than the non-systematic input dimension). Further, the three classes of singular vectors associated with each singular value occur due to specific properties of the dataset. Aj. and by extension 11 arises from the systematic input and output features, and makes contributions to all four norms. X2 occurs due to the left-out systematic output features, and contributes only to the norms of the mappings to non-systematic outputs. Lastly, A3 occurs because of the non-systematic features, and appears only in the norm from non-systematic inputs non-systematic outputs. Thus, for all datasets in the space, the mixed systematic/non-Ss systtmmmttstttttiicc singular value mode πi is the fastest singular value to learn, while the purely non-systematic singular value Ty is the second largest but slowest to learn by the network.
In the shallow network. ó1 drives the dynamics of the mixed sestematic/non-systemmade mode and 82 contributes to the non-systematic mappings. We can show that for any setting of the dataset parameters 8n > 62 and so the mixed systematic/non--ystemaaii mapping again learned faster

---

Under review as a conference paper at ICLR 2022
Dense
Outout Partitioned
Fully Partitioned
T
y1
y
7
7
3
2
3
12
1
Systematic Non-systematic Combination
Mapping
Mapping
Mespiigg
Figure 3: Impact of architectural biases. Architectures that partition systematic and non-systematic features in different ways with the corresponding graphical representation of the resulting network mappings. The dynamical modes T1, 72, and T3 contain contributions from systematic and non- systematic input components, depicted the color of the bottom half of the each mode; and they make contributions to systematic and non non-systematie output features, depicted as the color of the top half of each mode (red for systematic, green for non-systematic, and orange for mixed). To learn systematic mapping, the fastest mode т must be systematic. The output partitioned network is able to obtain output systematicity but not input systematicity. Only the fully partitioned network obtains complete systematicity.
than the purely non-systematic one. However, because the shallow dynamics exhibit exponential approach to their asymptote, both modes are learned at overlapping times.
In sum, a part of the non-systematic mapping will be learned at the same speed the systematic mapping, for any setting of our dataset. Thus, there is no epoch at which training could be stopped that would completely remove mappings from the non-systematic component of the input and out- put. In our setting, the implicit bias arising from depth, small random initialization, and gradient descent is insufficient learn fully systematic mappings for any datasets in this space.
5 MODULARITY AND NETWORK ARCHITECTURE
We now turn to modularity and network architecture, another prominent approach for promoting systematicity in network's mapping (Vani et al., 2021 Bahdanau et al. 2019). Architectures such as Neural Module Networks (Andreas et al. 2016 Hu et al.]2017| learn reconfigurable modules that implement specific aspects of larger problem. By rearranging existing modules to process novel input, they can generalize far beyond their training set. Here we investigate whether simple forms of additional architectural structure can yield strong enough inductive biases to learn fully systematic mappings.
In particular, instead of a dense network, we consider architectures in which systematic and non- systematic features are processed in separate processing streams, depicted in Figure The learn- ing dynamics for several of these different settings can be obtained by combining the dynamics of other points in the space of datasets. To illustrate the approach, consider the output- cntput-partitioned net- work. The pathway leading to the systematic outputs effectively learns from a dataset with ky = 0, and the pathway leading the non-systematic outputs effectively learns from dataset with ny - 0. By combining these dynamics, we can reconstruct the full solution, as shown in Figure [ of Ap- pendix E We note that this approach cannot be taken for the input partitioned network. which has interactions in the output layer. We summarize our findings with the graphical representations of the network mappings in Figure 3 The same three modes drive learning dynamics, but they contain subtly different mixtures of contributions from systematic and non-systematic inputs and outputs. network exhibits output systematicity but not input systematicity,
Notably, the output-partitionel significant point that we will return to in the iterated learning section. However, only the fully partitioned network achieves full systematicity, such that early stopping could completely prune non-systematic features. Hence find that architectural biases can enforce systematicity, but only in the heavy-hander and relatively trivial case where the bias perfectly segregates systematic and non-systematic information.
7

---

Under review as a conference paper at ICLR 2022
r
Relabel
2 r
Relabel
f2
Sop fir Bookk
14-
She Nacuork
Y
Spitt Nicwo
Der: Salt Net Early Stpp
10-
Dense/ Shallow Network
Densel Shallow Network
Dense/ Shallow Network
Train
Train
Train
0.
X 12
X
00
Generation
Generation
Generation
Mamber (Genoratinn
(a)
(b)
Figure 4: Iterated learning dynamics. (a) Generations of agents learn from languages generated by their parent, and pass on their acquired language to their children. (b) Norm of non-systematic output mapping over 20 generations of the IL procedure using deep, shallow and output-split networks. Dashed line shows the best early-stopping can achieve for the starting dataset (stopping at epoch 30 with the output-split network). Even though the networks are run for 40 epochs per generation, IL is still able to out-perform early stopping with the output-split network. It is important to note that the dense and split networks maintain the original systematic mapping by epoch 40, and these results show the closest each network can get to removing the non-systematic component under this restriction. The shallow network is unable to fully maintain the systematic mapping with any form of IL. thus the decrease in its norm is at the expense of the systematic mapping. Parameters: nr 3, ny = 2, lig 1, ky = 1 and r y= 2.
ITERATED LEARNING
Finally, we consider the impact of iterated learning on extracting systematic structure, alongside architectural partitioning. In iterated learning, the focus is on refining the output 'language' toward systematicity over generations of learners (Kirby 2001 Kalish al. (207)) as illustrated in Figure 1b middle and right panel. Each generation learns from the language acquired by the previous generation (Figure 4a). To instantiate this setting, we start from particular dataset in our space, but halt training before full convergence after pre-defined number of training steps. We then use the network's s output (logits) as the target outputs for the next generation. Throughout learning, the network's ' input-output mapping takes the same form as the dataset's 's singular value decomposition, but with different singular values. Therefore, the generated language always has the same singular vectors, but with different singular values corresponding to the amount of learning progress made by a given agent. This fact permits straightforward analysis of iterated learning dynamics throughout space of datasets.
Figure 4p depicts the refinement of the output language over generations networks with differ- architectures. In partieular, iterated learning effective reducing or eliminating the non- systematic output features, and can do so more effectively than carefully chosen carly-stopping on the original dataset. We make several observations. First, while deep and shallow networks do not leam perfectly systematic mappings in the first generation, the deep network does learn a relatively more systematic mapping, such that yields more rapid benefits. Second, iterated learning in dense networks can reduce non-systematic structure relative to early stopping, but does not eliminate it. Third, and most notably, iterated learning can exploit weak architectural priors: with optimal early stopping, the output partitioned network does not fully remove non-systematic output features, but over generations of iterated learning it completely climinates non-systematt output structure.
This effect arises because the systematic and non-systematic mappings depend on entirely different modes. Since Ti is used by the systematic mapping and is learned the fastest for the space datasets, in fact IL in the nutput-partitionn dataset in the space of datasets. Whereas early-stop sarly-stopping has varying performance over the space of datasets, II. with the output-split network will always be able to converge to systematic language
network completely removes the non-systematic mapping for any
8

---

Under review as a conference paper at ICLR 2022
Normalized Error
Error
Dense Systematic Error
1.4
Error
Split Systematic Error
1.2
Splrt Non systematic Erro
1.0
0.6
0.8
Dense Systematic Errr
Nan-systamatic Error
Norma
0.4
Norma
0.6
Split systemati tirer
Split systematic Error
0.4
150
250
150
250
Number
Number
(a)
(h)
Figure 5: (a) Normalized training loss, (b) test loss of a deep CNN with ReLU activation on the Compositional- -NNIST dataset averaged over 10 runs. Error bars reflects one hundard-deviation
(dataset labels). However, we note that still, this language will be produced in part from mappings from non non-systematic input features. Thus IL only achieves output systematicity.
COMPOSITIONAL MNIST (CMNIST)
To evaluate how well our results generalize non-linear networks and more complex datasets, in this section we train a deep Convolutional Neural Network (CNN) to learn a compositional variant of MNIST (CMNIST). In this dataset three digits from MNIST are stacked horizontally, resulting in a value between 0 and 999. The systematic output encodes each digit in 10-way one-hot vector, resulting in a 30-dimensional vector. The non-systematic output encodes the number as a whole with a 1000-dimensional one-hot vector. This task is similar to the SVHN dataset (Netzer et al.) 2011) with systematic and non-systematic output labels.
We compare results of using single dense CNN and split CNN, in which two parallel sets of convolutional layers with half the convolutional filters of the dense network each that connect sep- arately systematic and non-systematic labels. Full details of the two network architectures and hyper-pane hyper-parameters used for the CMNIST experiments are given in Appendix G The effect predicted by our theoretical work is that the redundancy the labels will interfere with the network's learned hidden representations and decrease the systematic generalizability of the network. This predic- tion is shown be true in Figure 5, which shows the mean-squared error for the systematic and non-systematic network predictions over the course of training. normalized so that the initial error at 1.0. Firstly, in dense networks the error of the systematic mapping is tied to the error of the non-systematic mapping. This is seen in Figure 5(a) where the blue curve cannot converge until the orange curve has become sufficiently low. This effect is not observed with the split network. Secondly, comparing Figure 5(a) and 5(b) we demonstrate the lack of generalization which occurs when using non-systematic features. This seen as the orange and red curves achieving a lower training error while the test eтror increases. Lastly. again by comparing Figure 5(a) and 5(b), see that the systematic mapping of the split architecture is the only mapping which generalizes well (near 0 training and test error). Thus, even in this complex setting, see the negative effect shared hidden layer has on the generalization of the network (comparing the blue and green curves).
DISCUSSION
In this work we have theoretically and empirically studied the ability of simple NNs to acquire sys- tematic knowledge. We found that this ability is challenging even in our simple setting. Neither implicit biases learning dynamics, nor all but the most stringent modularity, caused networks to ignore non-systematic inputs. However, iterated learning-when combined with weaker architectural priors-cam converge to strongly systematic output features. Our results complement recent empir- ical studies, helping highlight the complex factors influencing systematic generalization. We hope greater understanding of these phenomena will ultimately aid the design of improved learning systems that can leverage specific learning in diverse new ways.

---

Under review as a conference paper at ICLR 2022
9 REPRODUCIBILITY STATEMENT
To aid the reproducibility of this work, we have included all source code used to obtain the results in the supplementary material as well as instructions on using the code. In this work we introduced the space of datasets used in Sections and [S] for our theoretical analysis with linear networks, as well as the Compositional MNIST dataset. Included in the source code are methods of constructing the space of datasets and for loading the CMNIST dataset. No pre-pi pre-processing steps are used on either dataset before being used to train the networks. In Appendix G we have also included all hyper-parameters and network architectures used for the CMNIST experiments. Finally, for our theoretical results, the full details and technical assumptions can found in Appendix
REFERENCES
Jacob Andreas. Measuring compositionality in representation learning, In International Conference Learning Representations, 2018.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
S. Arora, N. Cohen, and E. Hazan. On the optimization of deep networks: Implicit acceleration by overparameterivation. 35th International Conference on Machine Learning, ICML, 2018, 1: 372-389, 2018. arXiv: 1802.06509 ISBN: 9781510867963.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2019.
P. Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):53-58, January 1989. ISSN 08936080. doi: 10.
1016/0899-600000899904422 URL http:://inkke http://wwww.eeeeee.....eeeeeeeee cttp:/tteeeeeeee////ee/eeeete 011/089300009900144
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao
Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//gi www.thhub.cooooooo cottubbecmm/ggooge//////
Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life, 12(2):229-245) 2006.
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. The paradox of the compositionaliiy of natural language: a neural machine translation case study. arXiv preprint arXiv:2108.05885, 2021. URL
https://arxivy: org/ abs /00.////..0080355333 05885
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analy- sis. Cognition, 28(1-2):3377) 1988.
K. Fukumizu. Effect of Batch Learning In Multilayer Neural Networks. In Proceedings of the 5th International Conference on Neural Information Processing, pp. 67-70, 1998.
Robert F Hadley. Connectionism, explicit rules, and symbolic manipulation. Minds and machines, 3(2):183-200, 1993.
Robert F Hadley. Systematicity in connectionist language learning. Mind & Language, 9(3):247- 272, 1994.
Charles F Hockett and Charles Hockett. The origin of speech. Scientific American, 203(3):88-97, 1960.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference Computer Vision, pp. 804-813, 2017.
10

---

Under review as a conference paper at ICLR 2022
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pp. 53-69, 2018.
Minyoung Huh, Hossein Mobahi, Richard Zhang. Brian Cheung, Pulkit Agrawal, and Phillip Isola. The lowkrank simplicity bias in deep networks. arXiv preprint 2tXhv:2103.1042, 2021.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Michael I. Kalish, Thomas I. Griffiths, and Stephan Lewandowsky. Iterated learning: Intergenera- tional knowledge transmission reveals inductive biases. Psychonomic Bulletin & Review, 14(2): 288-294, 2007.
Eugene Kharitonov and Marco Baroni. Emergent language generalization and acquisition speed are not tied compositionality. arXiv preprint arXiv:2004.0344 2020.
Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learning model of the emer- gence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, 5(2):102- 110, 2001.
Simon Kirby, Hannah Cornish, and Kenny Smith. Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language. Proceedings of the National Academy of Sciences,
1-6666 2008.
Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. Compression and communica- tion in the cultural evolution of linguistic structure. Cognition, 141:87-102, 2015.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of gegueece-to-sequenee recurrent networks. In Proceedings tfe 35th International Conference on Machine Learning, pp. 1487-4499) International Machine Learning Society (IMLS), 2018.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Brenden M Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional in- structions. In Proceedings of the 4/st Annual Conference of the Cognitive Science Society, 2019.
A.K. Lampinen and S. Ganguli. An analytic theory of generalization dynamies and transfer learning in deep linear networks. In T. Sainath (ed.). International Conference on Learning Representations, 2019. ISBN 0311-5518, doi: 10. 1111000001115511888611900000 //arxiv.org/abs///
URL http:
10374
arXiv: 1809.10374.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Confer- ence on Learning Representations, 2018
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. Countering language drift with seeded iterated learning. In International Conference on Machine Learning. pp. 6437-6447. PMLR, 2020a.
Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron Courville. Supervised seeded iterated learning for interactive language learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3962-3970, 2020b.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Lars Niklasson and Noel Sharkey. Systematicity and generalisation in connectionist compositional representations. Citeseer, 1992.
11

---

Under review as a conference paper at ICLR 2022
Steven Phillips and Janet Wiles. Exponential generalizations from polynomial number of examples in combinatorial domain. Proceedings of 1993 International Conference on Neural Networks (IJCNN-93 -N-Noyya Japan), volume 1, pp. 505-508. IEEE, 1993.
Steven Andrew Phillips. Connectionism and the problem of systematicity, PhD thesis, University of Queensland, 1995.
Jordan B Pollack. Recursive distributed representations. Artificial Intelligence, 46(1-2):77-105, 1990.
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B Cohen, and Simon Kirby. Compositional lan- guages emerge in neural iterated learning model. In International Conference Learning Representations, 2019.
Timothy T Rogers, James L McClelland, et al. Semantic cognition: A parallel distributed processing approach. MIT press, 2004.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden Lake, A benchmark for systematic generalization in grounded language understanding. Advances in Neural Informa- fion Processing Systems, 33, 2020.
D. E. Rumelhart and L. McClelland. On Learning the Past Tenses of English Verbs, volume II, MIT Press, Cambridge, MA, USA, 1986. ISBN 0262132184.
pp.
A.M. Saxe, J.L. McClelland, and S. Ganguli. Exact solutions the nonlinear dynamies of learning in deep linear neural networks. In Y. Bengio and Y. LeCun (eds.), International Conference on Learning Representations, Banff, Canada, 2014. Oral presentation. arXiv: 1312.6120v3.
Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116
153):11537-11566, 2019.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159-22 216, 1990.
Paul Smolensky. Connectionism, constituency, and the language of thought. In Barry M. Loewer and Georges Rey (eds.), Meaning in Mind: Fodor and His Critics. Blackwell, 1991.
Zoltan Szabó. The case for compositionality. The Oxford handbook of compositionality, 64:80, 2012.
Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville. Iterated learning for emergent systematicity in vqa. In International Conference on Learning Representations, 2021.
12

---

Under review as a conference paper at ICLR 2022
A LEARNING DYNAMICS IN SHALLOW AND DEEP LINEAR NETWORKS
The dynamics of learning for shallow and deep linear networks are derived in Saxe et al. 2019 We state full details of our specific setting here. We train linear network with one hidden
layer to minimize the quadratic loss L(W2, 22) 2h Y w2 using gradient de- scent. This gives the learning rules F|AW"] 2nr (Y
W (Y w and E|AW?]
2
(W"' By using a small learning rate and taking the continuous time limit, the mean change in weights is given by - W w2" (Σ wz and T # W (E W " where Σ- = E[XX²] is the input correlation matrix, Σ = Y is the input-output correlation matrix and t = 2=se. Here, t measures units of learning epochs. It is helpful to note that since we are using small learning rate the full batch gradient de- scent and stochastic gradient descent dynamics will be the same. [aaxe et al.] (2019) has shown that the learning dynamics depend on the singular value decomposition of E Ex Sminnnn 2 My k2y" \ ga" and Σ = VDVT = = 8a 00 pa" To solve for the dynamics we require that the right singular vectors V of E are also the singular vectors of Σ". This is the case for any dataset in our space, as shown in Appendix(Bi Note, we assume that the net- work has at least 2 hidden neurons (the number of singular values in the input-output covariance matrix) so that it can learn the desired mapping perfectly. If this is not the case then the model will learn the top n. singular values of the input-output mapping where ny is the number of hidden rons (Saxe et al. 2014). Given the SVDs of the two correlation matrices the learning dynamics can be described explicitly ())())+
_
USVT
U UA(()
Eme (m, +. 2 my + Ry 2nx) To (4) (y)
where At) is the effective singular value matrix of the network's 's mapping, The trajectory of each Aa/
From these dynamics it is helpful to note that the time-course of the trajectory is only dependent on the Σve singular values. Thus, E affects the stable point of the network singular values but not the time-course of learning. In addition for the y singular values we have Theorem
singular value in Att) is described as Ta (t) 1- .. exp(
Theorem For all points in the space of datasets: Dzz ny kiz Ky E IR+ the input-output covari- ance matrix Eye singular values will be ordered as: x > x2 As-
Proof:
Firstly we prove that d > A2:
\1 > A2
(karr + 2 (kg + 22xx)
(kor + 2m (ky
>
22n
22n=
((ksp2 + 2n (ky 2² + 22x²)) > ((kir2 + 2n (kk 2k))22)
kyy² + 2n h
n >
2d > is true by definition since n. R+ and, thus, A1 > Aa for all points in our space of datasets.
Now we prove that A2 > Ag:
A2 > A3
(ker2 + 2th (ky
7/
k kekypt
22nz
22m
(Ror? + 2m )kx
> kirk
kinky + 2th ky > ki kinkk
2 ky > 0
13

---

Under review as a conference paper at ICLR 2022
2d kyr 0 is true by definition since ney ke, E R* and, thus, X2 Ay for all points in space of datasets. Thus, using the transitivity of inequality: A1 > Ag > Ag for all points the space of datasets.
B SINGULAR VALUE DECOMPOSITION EQUATIONS
In this section provide the general formulas for the singular value decomposition of the 2 and n covariance matrices for any dataset in the space of datasets, As stated in Section 4 the right singular vectors of 2 must match the singular vectors of 2 which is the V matrix below. Thus, I = VDVT and E USVT.
Let:
= (T,
B 2252
C = N.
THP (K, 22x 2
k
1
20
122
S2
Where THP is the SVD of K,
1 x n
22
Then the following are the
k.
matrix formulas for the components of the SVD for E
and Σ
2
Ony xk 20
2d (b, 202)
U
(13)
pst
B
(C B)
T
2 (ky, +2"z)
2 (kynd)
200 (Ais +2nx>)
x²
V
Xnx
2 (Kiz +2
(14)
Ok. 20x xnxx
) pT
Kg
222 11,, 22 2 2m 20nz
+
(Me 2 (Kor
AA
On xkx
8=
22n
O x 20x
(lipky) ) 2n
1 x
(15)
(ke (ky, 2
In x
On xkx
D
(16)
k.t²
2k Xnx
12nx x2
C SHALLOW NETWORK SIMULATIONS
In this section we show the simulations of shallow network trained on instance from the space of datasets. We again see strong agreement between the predicted and simulated time-courses for the training dynamics. Importantly for our findings in Section the modes of variation for shallow network are all learned at roughly the same time. This can be seen in Figure where time- courses, and by extension norms, are all learned simultaneously. Thus, adding depth to network necessary for IL to be even partially effective at removing non non-systematic output labels while maintaining the systematic outputs.
D INPUT AND OUTPUT PARTITIONED FROBENIUS NORMS
We partition the input-output mapping along the systematic and non-systematic input and output components. The time-courses for these norms can be seen in Equations [7 to 20 From these equations that the mappings both components of the output rely on all inputs. Thus, the non-systematic inputs still offer some benefit to the systematic output and are used in the systematic mapping. Likewise the systematic inputs are used for the non-systematic mapping.
22n (1))
2g- =
(17)
2 + 2 )kuy 2 + 2n=)
14

---

Under review as a conference paper at ICLR 2022
My-Noms
0,8
1.4
Predicted Norm
Prediction
Ty-Norm
1taal
1.2
Predicted N-ror
Singiar Valce
1.00
Prodiction
Real
0.4
Tocal
Prediction
Frebenuu
0.8
0.4
340
1000
13U
200
2001
3001
Cooch number
Epoch number
(a) Learning trajectories of the three unique Singular Values learned by the shallow network.
(b) Time paths of the shallow network mappings
Figure Singular Values and Frobenius norms of the shallow network mappings
RyHlyhon
D.U.-Nomm
Predicted 0yB, Norm
Predicted QoIy Norm
N2ry-Nerm
T
Predicted Term Nerm
Predicted T,T Norm
0.8
Frobonis Norm
0.4
0.0
100
160
200
250
50
150
200
300
Epoch rumber
(a) Systematic output norm partitioned by the system- Non-systematic output partitioned by sys- atic non-systematic input components.
tematic and non-systematic input components.
Figure 7: Frobenius Norm of the systematic and non-systematic deep network mapping partitioned by the systematic and non-systematic inputs on a dense linear network.
2 n2 (77
I In--2y-Norm = [kip + 2 (ko, 2 2n.))
(18)
2 ki 2 (x7
n (nz ngg) kir + 2n
Qg-IyyNomm
+
T
(19)
(kig + 2m (gn=)(ka))
(ki 2nx)
(
R nyr (k:r + 2n, [kk 2na)
(no ny)kkr? k2r + 2ns
I Tg-Tyy-Norm-
+
T Tt(() ---==22h) (t)
(20)
Figure shows the simulated and predicted training dynamics for these norms on a deep linear network. Figure 7a reflects that the mapping to the systematic output relies evenly on both the systematic and non-systematic inputs, with a slight preference to the systematic inputs. Likewise, Figure reflects that the mapping to the non-systematic outputs also uses the systematic and non- systematic inputs evenly, with a preference towards the non-systematic inputs towards the end of training.
15

---

Under review as a conference paper at ICLR 2022
Non Systematic Neem Parcentoo Progrts
Pracerte
Slariiw Network
Diep Splt "Netook
Port by
Pradecie
Lyfom 1,
Deep spit Netaors 10 spoctt
Shallzel Network 10 Spacns
Parte
Baloe
Powimbe
110
0.
044
003
2310
0.2
Tooch
Systermatz Norm Tercaniage Progreso
(a) Learning trajectories of the three (h) Time paths of the split network unique Singular Values learned by mappings the output-split network.
(c) Phase diagram of the split net- work shallow network Frobenius
norms
Figure 8: Singular Values and Frobenius norms of the split network mappings
We can also consider the Frobenius norm from the full input to either the systematic or non- systematic components of the output (obtained by summing over input features). This takes simpler
form,
$
2 ml iit
-ay-Norm
(21)
kyy + 2
ky (
1
-orm
kuy 20= (ng ny + (2 n2))
(22)
E MODULARITY AND ARCHITECTURE
In Figure S we depict the Frobenius norm of the mapping from the entire input to the systematic and non-systematic output, for the partttt-artttoneed network. The dynamics of this case can equally be seen as using one neural module responsible for the systematic mapping and another for the non-systematic mapping. Now, from the perspective of the non-systematic mapping, none of systematic features in the input present in the output, and the mapping will no longer be depen- dent on TT. The systematic mapping is still only dependent on the mi effective singular value. The time-courses for the resultant Frobenius norms given in Equations
23 and 24
Norm = (nymi(()) NyrNorm (1) - (2" (1)))
(23)
(24)
F IMPERFECT OUTPUT PARTITIONS
We now investigate the case where the split network architecture does not perfectly partition the output into the systematic and non-systematic components. In this case some of the son-systematic identity output blocks are grouped with the systematic outputs (we only consider partitioning along the systematic and con-syytemaaic blocks to keep the closed form solutions tractable). Thus, we separate the number of non-systematic outputs ky into the number of non-systematic outputs of ky. We also the left network branch xllett and right network branch kijehh such that leet + 1.iggtt examine the mapping from the full input to aid in readability.
With this setup three distinct Frobenius norms emerge, as shown in Equations 25 In the extreme cases when flell K. we recover the dense network equations shown Equations 21 and 22 since T2 8 for Equation 27 This is apparent from the singular value equations for πo and
shown in Equations and 9 with ky in the numerator which is replaced by g. in this case. Thus,
16

---

Under review as a conference paper at ICLR 2022
Equation 27 falls away completely. In the other extreme case of rightt ky recover the split
network equations shown in Equations 23 and 24 This is again because T2 = My = but this time
from the left network branch's 's perspective. By definition klch 0 in this case and so all components
of Equation 26 fall away. leaving just Equations 25 and 27
26 anddd 27
2 12 7
2y-Norm-
(25)
k + 2
")
My x3()) hleny 2n
T -o-meft-Norm
+ (f2g -ny) + (2 n2 m2)
(26)
y-righe-Norm (2 + +2"T -2-m-m mmm)
(27)
G CMNIST ARCHITECTURE AND HYPER-PARAMETERS
In this section we provide the network architectures for the CMNIST experiments as well as other details of the experimental setup. We scale the non-systematic output labels to help the network learn these labels. For the results of this section a scale of 10 was applied, however, the results are consistent for wide range of scale values. Increasing or decreasing the scale merely changes the time taken for the same effects to occur. Both the dense and split networks are trained using stochastic gradient descent from random initial weights sampled from an isotropic normal distribution. No regularization, learning rate decay or momentum is used. We aim to keep the setup simple as possible while reducing the effects of other implicit or explicit regularizers the results, since we are comparing the systematic generalization of the networks. The simplicity also aids the comparison between the dense and split network architectures which is the goal of the experiment. Table [ shows the hyper-parameters used to train both networks, which have the same
and the two architectures are shown in Figures 9 (dense architecture) and 10 (split architecture).
Table 1: Table showing the papee--prrmeter
used for the CMNIST experiments.
Value
Step Size
2e
Batch Size
16
Initialization Variance
0.01
We describe some further observations. Consistent with results in linear networks, when training dense network there is a portion of the systematic mapping which is learned the same time as the non-systematic mapping (particularly from around epoch 150). In contrast, when using a split architecture the systematic mapping is learned independently of the non-systematic mapping and faster, while the non-systematic mapping struggles to learn. The non-systematic mapping fails to reach a near-zero error and plateaus around 0.4 regardless of how long training lasts. Thus, without systematic mapping sharing the same hidden layer and helping learning, the non-systematic mapping ineffective even fitting the training data in a reasonable amount of time.
Turning to Figure 55] we see that for both the dense and split networks the non-systematic mapping does not generalize to test data. This to be expected as unlikely that the same numbers in the training data would ever appear in the test data. Notably, comparing the systematic mappings we see that the converged dense network generalizes far worse than the converged split network. It is interesting to note that initially both networks generalize equally well, indicating the potential for IL to be partially effective for a dense network. However, findings, IL on the dense network would not come without cost since the network is not able to fully learn the systematic mapping while still maintaining a low generalization error. Since the final portion of the systematic mapping is learned at the same time the non-systematic mapping, the earliest epoch during training when IL can fully preserve the systematic labels will already have learned some of the non-systematic mapping. However, by learning the non-systematic mapping
agreement with our theoretical
17

---

Under review as a conference paper at ICLR 2022
Conv(9x27)
Conv(7x21)
Conv(5x15)
Flatten
Dense
5
84
58
28
20
38
14
32
32
1030
7680
Figure 9: Dense network architecture trained to perform the CMNIST classification task.
Conv(9x27)
Conv(7x21)
Conv(5x15)
Flatten
Dense
58
38
28
24
1
20
16
16
10
16
30
557
3840
Conv(9x27)
Conv(7x21)
Conv(5x15)
Flatten
Dense
84
28
58
24
20
16
16
10
1000
3840
Figure 10: Split network architecture trained to perform the CMNIST classification task.
18

---

Under review as a conference paper at ICLR 2022
the network's hidden layer will become worse for generalization, mapping. In contrast, the split network architecture sees no generalization gap and maintains near- zero test error from early in the training. Thus, it clear that the benefit of using the split network architecture that it avoids the conflict in its hidden layers from learning to accommodate both the systematic and non-systematic mappings.
for the learned systematic
19


Reviews:
[{"review_id": "boOmHdgg8R7", "reviewer": "Reviewer_u1Hs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper aims at investigating the impact of implicit biases on the learned network mappings both with the means of theoretical and empirical analysis. The impact of prior assumptions is a fundamental problem in learning compositionality. I understand that in order to have full control over the studies system the authors decided on the use of a synthetic dataset, which allows them to derive exact training dynamics.\n\nHowever, one of my concerns lies exactly with the construction of this dataset and its relevance to the studied problem. The dataset is built from artificial systematic features, which are just all binary combinations over a specified number of bits. The non-systematic features are represented by identity matrixes, where an entry on the diagonal corresponds to a combination of “features”. The target representation follows the exact same definition, with the difference only in the number of sampled “features”. The task, if I understand correctly, is to compute a mapping from the input to the output features. However, without any additional bias or task constraints, it is unclear why a systematic mapping between those features should be favored or even desired. In this light, the observation that such a mapping is not learned, i.e. that non-systematic inputs contribute to the systematic outputs, seems just like an observation of the non-uniqueness of the problem (for example, one could learn something close to an identity mapping or use the non-systematic features as a basis for the outputs. Without any additional biases both those solutions seem to be good fits, so why unsystematic mappings should not be desired in such a case?). Similarly, the observations about the impact of the network architecture (or rather the lack of such) look quite straightforward, and again, without any prior assumptions, it is unclear why the network should even partition the computation. \n\nFinally, beyond the linear study, litter commentary is given on how to address the nonlinear case, apart from a simple CMNIST experiment. I understand that the focus of the paper was on the linear case, however, it questions how impactful the findings are (especially in the light of the above critique) in real-world scenarios.\n\nTechnical questions: What is the use of k_x>1? It seems to me that for any k_i, the same identity matrix is used, so the information is not even redundant, but simply the same.  \n\nMinor Issues:\nLack of citations in line 8 of the second paragraph in Introductions.\nFigure 3: The caption seems not to be on pair with the image (it mentions rows while there are none in the image).", "summary_of_the_paper": "The paper studies the relation of learning dynamics and architecture structure to the acquisition of systematic knowledge in shallow and deep linear neural networks. To this end the authors study the network behavior on a space of parametric datasets composed of systematic and non-systematic features and assess the impact of training regime, architecture, and iterated learning on the learned by the network functions. It is observed that a fully systematic mapping, thought theoretically possible to learn in such a regime, is not achieved by the shallow nor deep networks, and is not encouraged by a modular architecture (except for the most trivial, fully separated case).  Authors also observe that, in the studied framework, the iterated learning can lead to systematic functions when combined with the output modularity constraint. ", "summary_of_the_review": "As the problem of the role of implicit biases in the systematic generalization seems very interesting, some incorporated constructions and problem designs seem artificial, and the conclusions lack significance, especially in the light of their limitation to the linear case study. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R7f7j6m6hU", "reviewer": "Reviewer_9QCY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n========\nThe paper investigates the subject of systematic generalization in neural networks, which is important and not well understood yet.\nIt builds on existing research, and performs reasonable approximations and simplifications in the setting: no non-linearities, synthetic low-dimensional data, gradient descent with a constant learning rate.\n\nMetrics\n\nWeaknesses\n==========\nClarity could be improved on different levels:\n- The research questions are not clearly stated, and neither are the conclusions in the \"discussion\" section\n- The links between systematicity, the reported metrics, and the general principle of generalization are not obvious. The influence of the non-systematic inputs are quantified, but does such an influence mean that generalization is worse? A network might have learn a way to represent non-systematic inputs in a systematic way in an intermediate layer, I would expect that to be _desirable_. Why would \"ignoring the non-systematic inputs\" be the right way to generalize?\n- It's unusual (and maybe misleading) to refer to 1-hidden-layer neural nets as \"deep\". There may not be a formal definition, but I'd say 3 hidden layers would be a minimum for \"deep\" networks.\n- It is not clear what questions the experiments on CMNIST are addressing, or what the answers are. Would the iterated learning procedure be applicable in that case?\n- I'm not sure what the point of the brand logos vs. brand names are in Figure 1b. Is there a difference between the two? Aren't these labels represented as 1-hot vectors in any case? The caption of Figure 1 mentions that \"the name of objects change [...] over generations\", but I did not catch any reference to \"names\" or \"spelling\" in Section 6.\n\nThe paper introduces a whole family of datasets, with factors of variations. However, I don't see them explored, although it would be helpful to see if some results are specific to the variant used. Figure 2 and Figure 4 use different variants, but I'm not sure what's the impact. Similarly, the size of the hidden layer (when there is one) may have an impact on the learning dynamics, and it may be good to address it.\n\nThe end of section 4 mentions \"the bias the implicit bias arizing from [...] gradient descent\". It might be useful to compare against stochastic gradient descent, because it is often argued that SGD has an implicit bias encouraging better-generalizing solutions (compared to full-batch GD).\n", "summary_of_the_paper": "The paper introduces a synthetic dataset where examples are composed of both \"systematic\" inputs (indicating whether one specific feature is present) and \"non-systematic\" ones (indicating whether a specific combination of all features are present), and where the targets are a subset of systematic inputs, plus all non-systematic ones.\nIt then studies the training of a few neural network architectures in terms of how the two parts of the inputs are used in predicting each part of the output, theoretically and in practice.", "summary_of_the_review": "Overall, the reported results are interesting, but the findings are too few and too narrow.\nThis paper does provide a benchmark and preliminary findings, that could be interesting to build on, but I think this paper does not have enough substance yet to be published at a conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Hw4ek6dxXed", "reviewer": "Reviewer_2WAW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The publication has a good motivation but lacks in presentation and results. Systematic generalisation in neural networks is not a new topic and there has been a lot of work on this subject. E.g., Hadley, Christansen, Chater, Niklasson, Phillips, Gelder, Smolensky have in my opinion done very similar analysis in the early 90s. Thus, I find the results up to page 6 not particularly novel or interesting. The authors introduction and related work section consists mostly of very recent work so a more systematic reading of the work following Fodor and Pylyshyn's critique may be adequate. \n\nThat said, I found the iterated learning section interesting, albeit rather brief and I'm not sure I fully understood the setup. At which point is training halted and why? Are the logits used to train the next generation or the feature classes? The iterated learning method describes how the description of an observation refines into a more structured language as it is passed through several participants/generations. But in the presented setup, the output neurons of the model have specific meaning since they are tied to specific features. It is not clear to me how this conundrum is resolved. \n\nWhat are the non-systematic network predictions in the cmnist experiments? The split network is more biased towards separate functions, it's not surprising to me that this built-in bias results in a more systematic function.\n", "summary_of_the_paper": "This work aims to study the influences that lead to the systematic generalisation in linear models. They theoretically and empirically explain why a linear model doesn't converge to a systematic solution. They also study iterated learning and find that that with sufficient output modularity the models do converge to systematic solutions. \n", "summary_of_the_review": "I find that the paper tackles some interesting questions but does not provide enough value to be accepted. The presentation is lacking. The figures could be improved and the language could be more precise in places. I also find that the work fails to connect with early work on (systematic) generalisation in connectionist models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8K3r5lpDN0M", "reviewer": "Reviewer_ZgRW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths\n\n1. Understanding and guaranteeing systematicity is an important problem in the design of neural network models. Moreover, a negative result like this is valuable in demonstrating that systematicity is hard in a technical sense.\n\n1. The submission formalizes a simple data setting in which systematicity is controllable. Controlling systematicity in the dataset in this way is a nice approach that I haven't seen taken before.\n\n#### Weaknesses\n\n1. Simplifying assumptions limit the applicability of the theoretical insights to standard machine learning pipelines: binary features, deep linear nets. In addition, the empirical study in Section 7 is not tied closely to the theoretical investigations, so its significance is not clear.\n\n1. The formalization is adapted from the proof technique of Saxe et al. (2019), so the technical novelty there is smaller, although the theoretical goal is distinct.\n\n1. The paper could use many improvements in clarity in critical sections and several attributions to prior work (see minor comments below). In addition, many critical explanations are relegated to the Appendix.\n\n#### Minor comments\n\n1. \"...however, a number of studies have identified situations where depth alone is insufficient for structured generalization ()\" missing reference?\n1. \"If systematic components are easier to learn than non-systematic ones, this process [of iterated learning] can successively refine a language toward a systematic structure...\" I'm not sure that I agree with this claim in isolation, or at least \"learnability\" needs to be better defined.\n1. \"the closely related concept [to systematicity] of compositional structure\" Can you give precise definitions of systematicity and compositional structure?\n1. \"Thus, without regularizers, compositional mappings do not emerge with NMNs.\" It should also be noted that NMNs require auxiliary data (i.e, the string, parse, etc. associated with a scene).\n1. \"Iterated learning (IL) approaches...\" Iterated learning should be intuitively defined before this point. I also found the rest of this paragraph hard to understand at this point in the paper.\n1. I think the clarity of the formalization paragraph could be increased. It was hard to understand what is part of the underlying data-generating process vs. what is given to a model as observation, and also what role is played by the \"non-systematic input feature matrix\" vs. the \"non-systematic output feature matrix\". It is not clear what the \"frequency and intensity\" of non-systematic features means, and similarly for \"the amount\" of systematic structure—I can believe that these notions should have an effect on the learning dynamics, but these notions are not precisely defined.\n1. \"This space of datasets is consistent with numerical notions of compositionality in previous works (Andreas, 2018), which define compositionality as a homomorphism between the observation space and the naming space.\" This should be better explained to argue more clearly for the specific decisions made in the present formalization.\n1. \"The generalization abilities of deep networks depend on a complex interplay of learning dynamics, architecture, initialization, and dataset statistics.\" Citations should be added as there is much work on this topic.\n1. \"...which prior work has shown to impart a low-rank inductive bias on the linear mapping.\" Specific citation missing.\n1. Are Eqs. (1) to (12) from prior work, or newly proven in this paper? Please clarify what is re-used from e.g., Saxe and what are new developments.\n1. Before Eq. (1), all assumptions on the specific training setup (architecture, initialization) should be (even briefly) mentioned in the text. I don't think leaving these to the appendix is sufficient, since they are necessary to scope the impact of the result.\n1. It is not clearly explained in the text of Section 4.1 what is the significance of the Frobenius norm and singular value trajectories in measuring usage of systematic vs. non-systematic features. Similarly, it is not quite clear at this point what are systematic inputs vs. outputs, and systematic features have not yet been very precisely defined. These clarity issues make it quite hard to understand the implications of the plots in Figure 2.\n1. In Section 5, it is not explained how the \"output-\" and \"input-partitioned networks\" are constructed. I found interpreting these results extremely difficult because the setup is under-explained.\n1. Section 6 is nice in exploring iterated learning as a regularizer, but I also found this section very difficult to understand because it builds on technical structure from prior sections which is insufficiently explained.\n1. It is mentioned in Sections 5 & 7 that \"[optimal] early stopping\" is used; is this using an appropriate validation (not test) split? If not, this optimal early stopping is not achievable in practice, and so there is not evidence that non-systematic features could be eliminated. ", "summary_of_the_paper": "The submission studies what properties of a training setup impact systematicity of a neural network. The theoretical setting uses the deep linear neural networks setup of Saxe et al. (2014; 2019) and derives the effect of the degree of systematicity of a dataset on a model's systematic behavior. The theoretical results demonstrate that systematicity is difficult to guarantee, but that structural modularity and iterated learning are helpful to improve systematicity.", "summary_of_the_review": "My main concern is the applicability of the insights to more standard settings (W1). The submission also needs major improvements in clarity (W3).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The Authors study the emergence of systematic generalization in neural networks. The paper studies a timely topic and presents a set of concrete results. For example, reviewer ZgRW emphasizes that a key strength of the paper is constructing simple datasets where systematicity emerges. I think indeed it is valuable, as systematicity is sometimes poorly defined and understood, so building a theoretical testbed might be very helpful.

However, the reviewers found important issues, which the rebuttal was unable to address. Perhaps the key issue (raised e.g. by reviewer 9QCY) is that results do not clearly generalize to more practically relevant settings. What is somewhat missing is a clear set of guidelines or implications for how to improve systematicity in more practically relevant neural networks.

Based on this and other issues raised by the reviewers, unfortunately, I have to recommend rejecting the paper. Thank you for your submission, and I hope that the review process will help you improve the work.
Decision: Reject