Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies systematic generalization and the extent to which modular networks are better at learning systematic mappings relative to non-systematic mappings. Leveraging prior results on deep linear networks, the authors demonstrate that while non-modular networks learn a combined systematic and non-systematic mapping, a partitioned network can be biased towards learning a systematic mapping. Finally, the authors demonstrate that modularity enhances generalization on the CMNIST task.
strength_and_weaknesses: **Strengths**
The paper is quite original. It provides a mathematical formalization of systematicity which to my knowledge is the first of its kind. Moreover, it demonstrates theoretically that modularity enhances the learning of systematic mappings in deep linear networks. Finally, the authors extend the results empirically to nonlinear networks, which boosts its significance. Overall, the paper appears very relevant to the field of systematic generalization.

**Weaknesses**
The theoretical results are understandably limited to deep linear networks on simple datasets. However, the authors may wish to expand on how the definition of systematicity (3.1) could be interpreted for more complex datasets beyond the one presented in section 3. Similarly, the authors may wish to further discuss what the mapping partitions in section 5 could correspond to in more complex datasets.

clarity,_quality,_novelty_and_reproducibility: **Quality**
Overall, the quality of the theoretical and empirical results is high. The claims made in the paper are backed up by solid evidence.

**Clarity**
The text is generally clear and figures are generally well-illustrated. As mentioned above, the authors may wish to further expand on the mapping between the concepts in the paper as described for the simple dataset of section 3 to more complex datasets; this can help clarify the significance of the work to the broader field.

**Originality**
To my knowledge, the paper is quite original. It is the first to propose a general mathematical formalization of systematicity, and may have a strong impact on how systematicity is viewed in the field.
summary_of_the_review: Overall, the paper makes a novel contribution to the field of systematic generalization by mathematically formulating systematicity, and showing theoretically and empirically a link between modularity and systematicity. I believe this paper can be a strong contribution to the field.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a theoretical framework for studying systematic generalization that considers the relationship between the structure in the dataset and the neural network architecture used to learn its input-output mapping. It defines two kinds of input/output features, either combinatorial features that are binary values {-1, 1} values, or non-combinatorial features that map all combinatorial features to one large 2^n one-hot vector according to the binary string. The output corresponding to the input is defined as a random subset of the combinatorial features and the non-combinatorial features. In essence the task is to learn the identity function where the subset of features is unknown apriori, where both input and output space contain redundant representations of the same information (the one-hot vector exactly identifies the combinatorial features in the input).

Given this description of the input-output X, Y matrices, they define systematic generalization as leveraging a low-rank substructure of the correlation matrix YX^t. They then proceed to provide the optimization dynamics for deep linear networks using gradient flow type of analysis, and show how the dynamics depend on the singular values of the correlation matrices YX^t and XX^t, for which they provide an exact closed form description relating the singular values to the number of feature types the dataset is constructed of. Using these training dynamics, they proceed showing how the correlation between different input-output splits changes during training (combinatorial to combinatorial feature, combinatorial to non-combinatorial, etc.), and come to the conclusion that dense networks cannot learn a systematic mapping between combinatorial features (i.e., that does not depend on the non-combinatorial inputs). However, when defining a simplified form of a neural module network, where the network completely separates the handling of the two kinds of features, then the network is able to learn the systematic mapping. Finally, the authors propose a simple empirical demonstration of these ideas in the form of predicting 3 MNIST digits, where the authors consider two cases, where the goal is to predict the 1000 combinations as 1000 classes or as 3 separate 10 class problems. The authors test this setup both for the dense case as well as for the split-network case where networks process each digit separately. The experiments show a clear benefit to learning problems when they are presented in a decomposable fashion, both for the input-output representation and for the chosen architecture.
strength_and_weaknesses: Strengths:

* The topic is of significant importance to the community. Many works have tackle this curcial topic of decomposable vs "end to end" learning, or systematic generalization as this paper calls it, and it is still an area where our understanding is lacking. Prior works have used more restrictive analysis, which this work (based on the recent development of gradient flow analysis) alleviates.

* The paper contains many intermediate results that could be of independent interest for further investigating this topic.

Weaknesses:

* The paper is too dense and might be better served as a journal paper without a strict page limit. Most of the theoretical analysis is demoted to the appendices, and just reading the body of the paper can be very confusing. Some sections end without any point or context, for example, the training dynamics solely discuss the dynamics of the singular values, but they should've mentioned first why we care about the singular values (discussed in the next section), and how the asymptotic solution converges to the correct function for predicting Y given X (It's possible to show this with some basic linear algebra, but it's never mentioned). Alternatively, the authors should consider removing and simplifying its various sections, while providing proof sketches for the analysis. Even the problem setup is hard to understand and there are many things that are left poorly described or explained, e.g., what is the difference between the various non-combinatorial features (aren't they all the same identity matrices?)? How are the subset of the combinatorial features sampled, and how does this distribution affect the results? Does the number of non-combinatorial output features depend on the number of combinatorial output features?

* The basic setup of the paper seems needlessly intricate. As I see it, this is simply the problem of learning the identity function, where we are given "side information" on the *preferred* binary representation of the input, and you examine how these input-outputs are represented and the relation to the architecture. The extra copies of the non-combinatorial features, or the subset selection seem like complications that don't really add anything to the discussion. Moreover, I think that there is some subtlety that the authors need to discuss more carefully. Namely, in their theoretical setup they merely consider learning the identity (though it is not presented as such), but simple random linear networks would "learn" such mapping over the "challenging" one-hot case immediately -- specifically, if W_1 is normally distributed with 1/sqrt(d) standard deviation, where d is the input dimension, and W_2 is constrained s.t. W_2 = W_1^T, then W_1 W_2 is approximately the identity even with the number of hidden units is as low as log(d). In other words, a randomly initialized network (not that far from the conventional initialization scheme) could approximately solve this case.

* The paper does not cite or discuss highly related prior works. As mentioned above, there is not much difference between the topic of "systematic generalization" / compositionality to the slightly "older" (in ML terms) topic of "decomposability vs end-to-end learning" that has been studied by others (e.g., [1, 2, 3]). In fact, a very similar setup and experiment has been proposed in [2], and analysed both theoretically and empirically, albeit using a different analysis and a more limiting set of assumptions. A thorough discussion of these works is missing.

[1] - Shalev-Shwartz et al., On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training, arXiv preprint arXiv:1604.06915, 2016.

[2] - Shalev-Shwartz et al., Failures of Gradient-Based Deep Learning, ICML 2017.

[3] - Wies et al., Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks, arXiv preprint arXiv:2204.02892, 2022. (Given it's a relative new preprint I don't expect it to be cited, but nevertheless I place it here for context).
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is too hard to read, as it tries to compress too much information in the body while skipping over most of the actual analysis (that is in the appendices). The setup is also needlessly complex, and could be simplified greatly while keeping the core message intact.

Quality: As far as I can tell, the proofs appear to be correct, though I only skimmed some of them. However, many results are given without any proof and we only get to see the final answer, e.g., the derivation of the singular value decomposition equations in App. C, or the partitions of the forbinuous norm in App. D. The authors should provide more details on how they got to these results (though I did verify parts of it on my own for simple cases, so it seems to be correct).

Novelty: There is novelty in the simple setup that allows us to pin-point the issue of decomposability vs end-to-end learning. However, it is not completely novel as there are similar approaches in prior works, though the analysis here appears to be a bit more general.

Reproducibility: As mentioned, many details are missing from the technical proofs that would make it difficult to verify. As for the experiments, they were straightforward and simple.
summary_of_the_review: In its current state, the submission is a bit below the threshold for acceptance. It has severe clarity issues that obscure its theoretical contributions, though I believe these could be corrected with a revision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper works on expanding the exact training dynamics analysis of Saxe et al 2019, which considers interrelated semantic groupings, to also cover systematic behavior. For this analysis, they consider the effect of combinatorial behavior in the dataset on the resulting gradient dynamics. In particular, they consider how a deep linear model without internal modular structure compares to one with modular structure in its trajectories. The result is an attempt to precisely characterize the importance of implicitly modular network structure in systematic generalization.

strength_and_weaknesses: Strengths:
- The idea here is really exciting to me. I would love to see more of this kind of exact analytic result around compositional behavior. I remember originally reading Saxe 2019 and wondering how it would interact with compositional or systematic behavior, and this approach seems like a really solid attempt at that.
- A number of steps taken to formalize modularity and systematicity, and the techniques for analysis that result, are innovative and likely to be useful in future work. In particular, I appreciate the use of the compositional properties of the dataset as a parameter when calculating gradient trajectories.
- The authors do test whether the predictions theoretically made in the linear setting also apply in a nonlinear setting.
- “For example, a module with no hidden layer connections to the non-compositional output components is learning on a dataset with ky = 0.” This is such a clean way of formalizing modularity!
- Again, I want to restate the serious potential of the formalisms that they offer and the techniques that they are using in their analysis, even if I think their presentation definitely needs work and the definitions might need some refinement.

Weaknesses:
- The theoretical results are only in the linear setting. I think that they might still be really valuable as a starting point for understanding more realistic networks, but it’s certainly a limitation in this work, and it’s not obvious to me how future work might adapt these methods, given their reliance on the linearity of module interaction. Although there are empirical results in a nonlinear setting that, given that it is only MNIST, these interactions are likely to be more linear than they would be in a network that trained for a longer period of time, so they might align better with the predictions made in the theoretical setting than a modern network would.
- “Neither implicit biases in learning dynamics, nor all but the most stringent modularity, caused networks to exploit compositional sub-structure in the data.” — again I’m not sure whether this is going to apply outside of these relatively low resource, somewhat linear, empirical settings.
- I often struggled to follow the intuitions behind the definitions. Examples of dataset that are combinatorial only in the input or only in the output, and ones that are combinatorial in both input and output, would have helped a lot with this. Could you give me some examples now? I’m really still stuck on the intuitions behind some of these definitions.
- I’m not even entirely clear on the intuition of the difference between the combinatorial and non combinatorial features, that could also really do with an example.
- Likewise, it was hard for me to grasp the intuitions yielded by the trajectory results (eq 5-9). What are the implications in terms of data efficiency? Let’s assume that most readers are not particularly effective at simulating and comparing function curves in their heads. All we have to go on is Fig 2, and it’s clear something is going on there but it’s a lot harder to understand what it is that you want me to take away.
- In general, I would really like to see examples of intuitions. Using the inspiration of Saxe 2019 as an example, there are running examples of specific hierarchical semantic classes throughout the text. This work would be greatly supported by an equivalent running example of compositional data of varying sorts.
- In definition 3.1, it’s specified that systematic generalization is only present if it “facilitates generalization from fewer training examples”. However, the results give in don’t seem to have anything to do with data scarcity, so I’m not sure how they link into this part of the definition. Could you spell out the implications in particular of these results in figure 2 with respect to the data efficiency alluded to in definition 3.1?
- I think that you really need to be testing generalization behavior or the amount of data required. In particular, you should be looking at splits that constrain the combinations exposed during training, as is typical in systematicity research.

Questions and minor issues:
- Are the rules for each of these datasets randomly generated?
- Without a more solid intuition about your definitions, it’s hard for me to say whether 3.1 would apply to behavior I wouldn’t think of as systematic personally. For example, if the matrix is a sum of the (non combinatorial?) “true” data and a noise matrix formed the covariance matrix, would that end up fitting into this definition? It doesn’t seem that effectively representing the underlying true data is a systematic behavior, but it seems like it might fit into this definition.
- “it is possible to have a near infinite number of resultant combinations” — really?
- Figure 3: There might be some notational inconsistency here. Previously, only the asymptotic bound was defined with $\pi^{ss}$, and I don’t think I’ve seen this particular variable name show up already.
- “To do this we calculate the Frobenius norm of the network’s mapping between different subsets of input and output components.” This needs a much more clear explanation of what form the mapping takes. What are you defining as a mapping?
- I really don’t like how you index figure references! Every time I see “Figure 3b)” I cringe at the unmatched parenthesis.
clarity,_quality,_novelty_and_reproducibility: - The clarity is the main flaw of the paper, to a significant enough degree that my lack of intuition around the definitions they offer leaves me unable to assess the quality/correctness of the paper properly.
- Very novel to my knowledge.
- The lack of clarity might pose a challenge for reproduction of the results.
summary_of_the_review: Because of the lack of clarity around the intuitions underlying the math, I don’t feel confident assessing the correctness of the paper. This is reflected in my confidence score, although I find the concepts and the general approach of the paper very exciting so my score reflects both the promising concept and the need for grounding and intuition.

**After discussion:**

I think I agree with reviewer 83KP that this would be better served by a journal where the authors could fully explain their work within the main body of the paper. I have raised my score from 5 to 6 in response to the authors' attempts to clarify the writing, but I just think that there is a ceiling on how good this paper can get with only nine pages. The work here is impressive, and I would rather see it published in a form that serves it well.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper investigates how to achieve systematic generalization via neural module specialization from theoretical perspective. Build upon previous results on deep linear network, this work demonstrates that a partitioned network can be biased towards learning a systematic mapping, and modularity enhances generalization on the CMNIST task.

Strengths:
- The theoretical study on systematic generalization is original. To my knowledge, this is the first work that provides a mathematic definition of systematicity.
- The claims are backed up by solid evidence and will inspire future work.
- The results on deep linear network is extended to nonlinear network empirically.
- The topic is of significant importance to the community.

Weaknesses:
- The theoretical results are limited. It does not provide theoretical study on more complicate datasets and nonlinear networks.
- The paper lacks some intuitions behind the definitions and experimental results.
- The experimental setting is not very realistic.
- The work is not fully explained in the main body. Some details are moved to appendix due to the limitation of 9 main pages.
justification_for_why_not_higher_score: Although the authors conducted theoretical study on linear DNN, they do not extend the study to more complicate datasets and nonlinear networks.  The reviewers also raised some valuable concerns such as lacking intuitions behind the definitions and experimental results. The authors are encouraged to make the necessary changes to the best of their ability in the final camera-ready version of the paper.
justification_for_why_not_lower_score: The theoretical study on systematic generalization is original and of significant importance to the community. To my knowledge, this is the first work that provides a mathematic definition of systematicity.
The claims are backed up by solid evidence and will inspire future work.
Decision: Accept: poster