Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission studies what properties of a training setup impact systematicity of a neural network. The theoretical setting uses the deep linear neural networks setup of Saxe et al. (2014; 2019) and derives the effect of the degree of systematicity of a dataset on a model's systematic behavior. The theoretical results demonstrate that systematicity is difficult to guarantee, but that structural modularity and iterated learning are helpful to improve systematicity.
main_review: #### Strengths

1. Understanding and guaranteeing systematicity is an important problem in the design of neural network models. Moreover, a negative result like this is valuable in demonstrating that systematicity is hard in a technical sense.

1. The submission formalizes a simple data setting in which systematicity is controllable. Controlling systematicity in the dataset in this way is a nice approach that I haven't seen taken before.

#### Weaknesses

1. Simplifying assumptions limit the applicability of the theoretical insights to standard machine learning pipelines: binary features, deep linear nets. In addition, the empirical study in Section 7 is not tied closely to the theoretical investigations, so its significance is not clear.

1. The formalization is adapted from the proof technique of Saxe et al. (2019), so the technical novelty there is smaller, although the theoretical goal is distinct.

1. The paper could use many improvements in clarity in critical sections and several attributions to prior work (see minor comments below). In addition, many critical explanations are relegated to the Appendix.

#### Minor comments

1. "...however, a number of studies have identified situations where depth alone is insufficient for structured generalization ()" missing reference?
1. "If systematic components are easier to learn than non-systematic ones, this process [of iterated learning] can successively refine a language toward a systematic structure..." I'm not sure that I agree with this claim in isolation, or at least "learnability" needs to be better defined.
1. "the closely related concept [to systematicity] of compositional structure" Can you give precise definitions of systematicity and compositional structure?
1. "Thus, without regularizers, compositional mappings do not emerge with NMNs." It should also be noted that NMNs require auxiliary data (i.e, the string, parse, etc. associated with a scene).
1. "Iterated learning (IL) approaches..." Iterated learning should be intuitively defined before this point. I also found the rest of this paragraph hard to understand at this point in the paper.
1. I think the clarity of the formalization paragraph could be increased. It was hard to understand what is part of the underlying data-generating process vs. what is given to a model as observation, and also what role is played by the "non-systematic input feature matrix" vs. the "non-systematic output feature matrix". It is not clear what the "frequency and intensity" of non-systematic features means, and similarly for "the amount" of systematic structure—I can believe that these notions should have an effect on the learning dynamics, but these notions are not precisely defined.
1. "This space of datasets is consistent with numerical notions of compositionality in previous works (Andreas, 2018), which define compositionality as a homomorphism between the observation space and the naming space." This should be better explained to argue more clearly for the specific decisions made in the present formalization.
1. "The generalization abilities of deep networks depend on a complex interplay of learning dynamics, architecture, initialization, and dataset statistics." Citations should be added as there is much work on this topic.
1. "...which prior work has shown to impart a low-rank inductive bias on the linear mapping." Specific citation missing.
1. Are Eqs. (1) to (12) from prior work, or newly proven in this paper? Please clarify what is re-used from e.g., Saxe and what are new developments.
1. Before Eq. (1), all assumptions on the specific training setup (architecture, initialization) should be (even briefly) mentioned in the text. I don't think leaving these to the appendix is sufficient, since they are necessary to scope the impact of the result.
1. It is not clearly explained in the text of Section 4.1 what is the significance of the Frobenius norm and singular value trajectories in measuring usage of systematic vs. non-systematic features. Similarly, it is not quite clear at this point what are systematic inputs vs. outputs, and systematic features have not yet been very precisely defined. These clarity issues make it quite hard to understand the implications of the plots in Figure 2.
1. In Section 5, it is not explained how the "output-" and "input-partitioned networks" are constructed. I found interpreting these results extremely difficult because the setup is under-explained.
1. Section 6 is nice in exploring iterated learning as a regularizer, but I also found this section very difficult to understand because it builds on technical structure from prior sections which is insufficiently explained.
1. It is mentioned in Sections 5 & 7 that "[optimal] early stopping" is used; is this using an appropriate validation (not test) split? If not, this optimal early stopping is not achievable in practice, and so there is not evidence that non-systematic features could be eliminated. 
summary_of_the_review: My main concern is the applicability of the insights to more standard settings (W1). The submission also needs major improvements in clarity (W3).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the relation of learning dynamics and architecture structure to the acquisition of systematic knowledge in shallow and deep linear neural networks. To this end the authors study the network behavior on a space of parametric datasets composed of systematic and non-systematic features and assess the impact of training regime, architecture, and iterated learning on the learned by the network functions. It is observed that a fully systematic mapping, thought theoretically possible to learn in such a regime, is not achieved by the shallow nor deep networks, and is not encouraged by a modular architecture (except for the most trivial, fully separated case).  Authors also observe that, in the studied framework, the iterated learning can lead to systematic functions when combined with the output modularity constraint. 
main_review: The paper aims at investigating the impact of implicit biases on the learned network mappings both with the means of theoretical and empirical analysis. The impact of prior assumptions is a fundamental problem in learning compositionality. I understand that in order to have full control over the studies system the authors decided on the use of a synthetic dataset, which allows them to derive exact training dynamics.

However, one of my concerns lies exactly with the construction of this dataset and its relevance to the studied problem. The dataset is built from artificial systematic features, which are just all binary combinations over a specified number of bits. The non-systematic features are represented by identity matrixes, where an entry on the diagonal corresponds to a combination of “features”. The target representation follows the exact same definition, with the difference only in the number of sampled “features”. The task, if I understand correctly, is to compute a mapping from the input to the output features. However, without any additional bias or task constraints, it is unclear why a systematic mapping between those features should be favored or even desired. In this light, the observation that such a mapping is not learned, i.e. that non-systematic inputs contribute to the systematic outputs, seems just like an observation of the non-uniqueness of the problem (for example, one could learn something close to an identity mapping or use the non-systematic features as a basis for the outputs. Without any additional biases both those solutions seem to be good fits, so why unsystematic mappings should not be desired in such a case?). Similarly, the observations about the impact of the network architecture (or rather the lack of such) look quite straightforward, and again, without any prior assumptions, it is unclear why the network should even partition the computation. 

Finally, beyond the linear study, litter commentary is given on how to address the nonlinear case, apart from a simple CMNIST experiment. I understand that the focus of the paper was on the linear case, however, it questions how impactful the findings are (especially in the light of the above critique) in real-world scenarios.

Technical questions: What is the use of k_x>1? It seems to me that for any k_i, the same identity matrix is used, so the information is not even redundant, but simply the same.  

Minor Issues:
Lack of citations in line 8 of the second paragraph in Introductions.
Figure 3: The caption seems not to be on pair with the image (it mentions rows while there are none in the image).
summary_of_the_review: As the problem of the role of implicit biases in the systematic generalization seems very interesting, some incorporated constructions and problem designs seem artificial, and the conclusions lack significance, especially in the light of their limitation to the linear case study.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work aims to study the influences that lead to the systematic generalisation in linear models. They theoretically and empirically explain why a linear model doesn't converge to a systematic solution. They also study iterated learning and find that that with sufficient output modularity the models do converge to systematic solutions. 

main_review: The publication has a good motivation but lacks in presentation and results. Systematic generalisation in neural networks is not a new topic and there has been a lot of work on this subject. E.g., Hadley, Christansen, Chater, Niklasson, Phillips, Gelder, Smolensky have in my opinion done very similar analysis in the early 90s. Thus, I find the results up to page 6 not particularly novel or interesting. The authors introduction and related work section consists mostly of very recent work so a more systematic reading of the work following Fodor and Pylyshyn's critique may be adequate. 

That said, I found the iterated learning section interesting, albeit rather brief and I'm not sure I fully understood the setup. At which point is training halted and why? Are the logits used to train the next generation or the feature classes? The iterated learning method describes how the description of an observation refines into a more structured language as it is passed through several participants/generations. But in the presented setup, the output neurons of the model have specific meaning since they are tied to specific features. It is not clear to me how this conundrum is resolved. 

What are the non-systematic network predictions in the cmnist experiments? The split network is more biased towards separate functions, it's not surprising to me that this built-in bias results in a more systematic function.

summary_of_the_review: I find that the paper tackles some interesting questions but does not provide enough value to be accepted. The presentation is lacking. The figures could be improved and the language could be more precise in places. I also find that the work fails to connect with early work on (systematic) generalisation in connectionist models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a synthetic dataset where examples are composed of both "systematic" inputs (indicating whether one specific feature is present) and "non-systematic" ones (indicating whether a specific combination of all features are present), and where the targets are a subset of systematic inputs, plus all non-systematic ones.
It then studies the training of a few neural network architectures in terms of how the two parts of the inputs are used in predicting each part of the output, theoretically and in practice.
main_review: Strengths
========
The paper investigates the subject of systematic generalization in neural networks, which is important and not well understood yet.
It builds on existing research, and performs reasonable approximations and simplifications in the setting: no non-linearities, synthetic low-dimensional data, gradient descent with a constant learning rate.

Metrics

Weaknesses
==========
Clarity could be improved on different levels:
- The research questions are not clearly stated, and neither are the conclusions in the "discussion" section
- The links between systematicity, the reported metrics, and the general principle of generalization are not obvious. The influence of the non-systematic inputs are quantified, but does such an influence mean that generalization is worse? A network might have learn a way to represent non-systematic inputs in a systematic way in an intermediate layer, I would expect that to be _desirable_. Why would "ignoring the non-systematic inputs" be the right way to generalize?
- It's unusual (and maybe misleading) to refer to 1-hidden-layer neural nets as "deep". There may not be a formal definition, but I'd say 3 hidden layers would be a minimum for "deep" networks.
- It is not clear what questions the experiments on CMNIST are addressing, or what the answers are. Would the iterated learning procedure be applicable in that case?
- I'm not sure what the point of the brand logos vs. brand names are in Figure 1b. Is there a difference between the two? Aren't these labels represented as 1-hot vectors in any case? The caption of Figure 1 mentions that "the name of objects change [...] over generations", but I did not catch any reference to "names" or "spelling" in Section 6.

The paper introduces a whole family of datasets, with factors of variations. However, I don't see them explored, although it would be helpful to see if some results are specific to the variant used. Figure 2 and Figure 4 use different variants, but I'm not sure what's the impact. Similarly, the size of the hidden layer (when there is one) may have an impact on the learning dynamics, and it may be good to address it.

The end of section 4 mentions "the bias the implicit bias arizing from [...] gradient descent". It might be useful to compare against stochastic gradient descent, because it is often argued that SGD has an implicit bias encouraging better-generalizing solutions (compared to full-batch GD).

summary_of_the_review: Overall, the reported results are interesting, but the findings are too few and too narrow.
This paper does provide a benchmark and preliminary findings, that could be interesting to build on, but I think this paper does not have enough substance yet to be published at a conference.
review_title: null

Meta-review: The Authors study the emergence of systematic generalization in neural networks. The paper studies a timely topic and presents a set of concrete results. For example, reviewer ZgRW emphasizes that a key strength of the paper is constructing simple datasets where systematicity emerges. I think indeed it is valuable, as systematicity is sometimes poorly defined and understood, so building a theoretical testbed might be very helpful.

However, the reviewers found important issues, which the rebuttal was unable to address. Perhaps the key issue (raised e.g. by reviewer 9QCY) is that results do not clearly generalize to more practically relevant settings. What is somewhat missing is a clear set of guidelines or implications for how to improve systematicity in more practically relevant neural networks.

Based on this and other issues raised by the reviewers, unfortunately, I have to recommend rejecting the paper. Thank you for your submission, and I hope that the review process will help you improve the work.
Decision: Reject