Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an empirical study of whether modularity can emerge within neural networks. It starts by proposing a novel definition of modularity that identifies modules by their functionality. To discover the module that implements a specific target functionality, the paper proposes to first pretrain the full network on the original task, then freeze the pretrained weights, and train a binary mask for each weight using Gumbel-Sigmoid. The training objective for the masks is given by the target functionality (e.g., a subtask of the original task), plus some sparsity regularization. The paper then investigates the discovered modules in terms of specialization, reusability, and compositionality. The main findings are: (1) Neural nets tend to satisfy specialization but not reusability; (2) Weight sharing between modules tends to be affected more by whether I/O are shared than by task similarity, and there tends to be less sharing in larger networks; (3) When trained on algorithmic tasks, neural nets fail to learn compositional rules, and thus generalize poorly; (4) CNNs trained for image classification contain class-specific, non-shared weights in the feature detectors.

Pros
- The experiments are comprehensive, covering many neural net architectures and datasets. The results seem consistent across these architectures and datasets. Also, source code is provided, and the supplementary material provides sufficient details to reproduce the results.

- The paper investigates natural emergence of functional modules, which is a novel perspective on modularity.

Cons
- Some concepts are not rigorously defined. For example, in P_{specialize} and P_{reuse}, when should two modules be considered the same? The paper seems to define a module as a subset of weights. Does this imply that two different subsets (potentially differ by only one element) will correspond to two different modules? If this is the case, then P_{reuse} is extremely hard to achieve, and it can never be true if the input/output neurons of two modules are different (i.e., separate I/O considered in Section 3.2). Hence, the experiments in Section 3.2 seem meaningless. One may argue that the weights in the input/output layer should be excluded from the module. But then the functionality of the module is not well defined. Another confusion I had is: What can be a target functionality and what cannot? Take the addition/multiplication task as an example. The task is basically to learn the function f(a,b,s), where a and b are two numbers, and s is a switch indicating whether addition or multiplication should be performed for a and b. The paper suggests f(a,b,s=+) and f(a,b,s=*) as two target functionalities, which are the original function restricted to two subsets of the input space. Is it reasonable to consider f(a=99,b,s) a target functionality? Would the result be different? Also, all other experiments in the paper seem to construct target functionalities by similarly restricting the original functions. Are there other ways to define target functionalities?

I am somewhat skeptical about investigating properties like compositionality and generalization after network pruning. It is well known that training performance does not reflect these properties. So even if training performance only slightly drops after network pruning, the pruned network may lose some properties that the full network has.

Minor Comments
- It is mentioned that separate I/O biases the network at initialization time to not share weights, but to me it seems to only bias the input/output layers rather than the hidden layers. It would be interesting to see if shared I/O will also lead to unshared hidden weights.

- Fig. 13 did not mention which LSTM variant was used.

- In Section 3.3, why is increased sharing in the first layer undesirable? It might not be possible to undo the permutation with only one layer.
review_title: Interesting investigation on an important problem. Some approach can be defined more rigorously.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper investigates the functional modularity of neural networks using a simple yet efficient end-to-end method. The paper is well written and clearly articulates a contribution to the literature. The proposed method is intuitive and straightforward. The experimental evidence is provided for both synthetic, language, and image classification tasks. Most of the related works are cited.

Concerns: The biggest concern I had is whether the conclusion reached in the paper is invariant to different neural network architectures, the size of the network, and the complexity of the task. As depicted in Figure 6 (a), there is a huge difference in the relative drop in performance for simple CNN, simple CNN without drouput and ResNet-110. It seems that a larger and more complex network tends to not sharing weights.

Besides, the paper uses accuracy drop after masking the weights as the main metric, which is related to the number of masked weights. It might be better to learn the binary mask for each subtask with a certain accuracy objective (e.g., less than 1-2% lower than the original network) for each subtask and compare the learned mask of different subtasks. In addition, it is useful to see the results on a more complex task such as ImageNet. In ImageNet, there exist many similar subtasks as many categories of the images are actually belong to one big category. I am wondering whether these subtasks can share the weights.
Minor comments: The paper claims the advantage of using Gumbel-Sigmoid than a simple threshold function. However, the state-of-the-art binarized neural networks are trained using a simple sign function with a straight-through estimator. Is there a significant difference (e.g., the stability of the training) in training the binary mask with the Gumbel-Sigmoid and threshold function?

Reasons for score: I vote for accepting. I like the finding that most neural networks have non-overlapped functional modules. However, I still have some concerns about the generality of the conclusion, and also the number of shared weights is not calculated with respect to a uniform accuracy requirement. I would consider raising my score if the authors can address my concerns.
review_title: This work presents a simple yet effective method to investigate the functional modularity of neural networks. Through many carefully designed experiments, the authors found that most of the weights are specialized for a specific function and not shared across different tasks.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
In its current form, I feel that the paper should be disqualified because it contains some results essential to its claims in the appendix (referenced in the second paragraph, page 5). However, this can be easily addressed - thus my full review below:

Summary of the paper:
The paper aims to analyze if and how neural networks learn modular representations. Modularity under the paper's definition means that the network learns representations that (1) specialize (using different modules for different functions) and that (2) compose in a re-usable fashion - i.e., that functions are used in diverse tasks.
To do so, the authors train different probabistic masks (using a Gumbel-Sigmoid) on the weights of a neural network over a series of different tasks. They incentivize these masks to be sparse by regularizing the number of "active" elements in a mask. 
They then compare masks learned for different tasks (including simple arithmetic tasks, and permuted MNIST), and then compare the usage of the parameter masks over different tasks. They find that NNs learn to specialize only, without reuse.
The paper continues by postulating that the reason is either that the network learned "bad" representations, or that the network did not learn the correct composition. They argue that results on the SCAN dataset show that the representations are of a sufficient quality, concluding that the network did not learn the correct composition.

Commentary:
The question the paper tries to answer is very relevant, on two levels. The first is that we do not understand how an NN learns sufficiently well. The second is that compositional modularity is a highly desireable property, and it is important to know if neural network exhibit it.

Strengths:
- The paper does some interesting analysis 
- In general, the paper is well-written, and easily understood.

Unfortunately, the paper suffers from major drawbacks:
- (this is a minor point I'm putting here to facilitate my discussion below) The paper is not positioning itself correctly in the literature, thereby using confusing terminology
While modularity is not a main focus in neural network research, there exists some meaningful research that has established some terminology. In essence, the paper talks about compositional modularity (and combinatorial generalization), but does not use this terminology. This makes the paper a little difficult to follow, if one is familiar with this literature. Within this terminology, specialize would be called "modularize" and Preuse would be called "compose", a terminology I will use in the following.  
- The conclusions are not convincing
The core argument in the paper is that neural networks fail to modularize because they either learn insufficient representations, or because they fail to learn to compose (to learn the "algorithm" required to utilize the modules correctly). Because the network learns re-usable representations, they modularize, and must thus fail to compose. 
While this is probably true, I cannot help to feel a little underwhelmed. It is well known that neural networks are overparameterized (see the "lottery ticket" literature). For this analysis, this means that it appears to be easier for the model to re-learn using the available capacity than to re-use the existing modules.
This is not particularly surpsising either, because this is, at its core, overfitting: the model does not generalize (which, in effect, is a "softer" way to re-use), but instead learns something akint to a separate function for different inputs.
What the paper does not investigate is why a neural network not re-uses capabilities, even those should be a good fit for a problem. This would be a really interesting analysis, one which I would very strongly argue for acceptance in any venue.
- The novelty of some parts is overstated
This is particularly true for using binary masks for multi-task like learning. See "Bengio, E., Bacon, P. L., Pineau, J., & Precup, D. (2015). Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297."
Additionally, the paper does not sufficiently relate their insights to the (cited) work around inducing modularity and compositionaly in networks, some of which does already come to similar results.

review_title: In its current version disqualified for formal reasons; Lacks discussion of several areas of relevant literature; Main result cannot convince

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper studies weight modularity in neural networks (NNs). In particular, given a NN trained to perform a task, a subset of weights are identified which in isolation perform well on a subtask of the original task. Such subsets are inspected to understand the extent to which they are specialized or reused across different subtasks of the original task. To identify subtask specific weights, a mask is learned that minimizes loss over a subtask when applied to the original NN's frozen weights. This process is carried out using gradient based optimization techniques (Adam). Extensive experiments are performed across various datasets and architectures. The paper concludes that while NNs seem to exhibit module specialization, they fail to exhibit reuse.

### Strengths

1. Understanding modularity within NNs and its relation to failures of systematic generalization is an important research direction.
2. The precise masking method proposed here is novel to the best of my knowledge.
3. The functional view of modularity has clear advantages over clustering based approaches.
4. I believe this work could be extended in interesting ways. For example, it seems like the methods presented here could naturally be extended to encourage modularity, rather than measure it.

### Weaknesses

1. The paper addresses a well known issue, that NNs often fail to generalize systematically. Why this occurs and potential solutions are left unaddressed. While the paper does put forward a hypothesis regarding why, that the difficulty of learning routing operation in NNs may lead current approaches to fail to discover solutions which generalize systematically, little evidence is provided to support this conclusion. As such, I believe claims such as, "our approach can also be applied in these settings to provide additional insight regarding the underlying reason for the observed failure at systematic generalization" should be relaxed, or clarified if there is evidence for this hypothesis presented that I am failing to connect.

2. How do we know the problem is with the full model, rather than the mask? Recent literature (cited in the paper) demonstrates that performant models can be obtained even when severe weight constraints are imposed [1, 2]. Indeed, [2] demonstrates masking alone is sufficient to adapt a NN to a completely new task. Given these observations, I'm concerned about the validity of assuming the mask optimization procedure identifies an independent functional module in the context of the computation performed by the full model. It seems plausible that the mask performs well on the subtask given the fixed model weights, but is unrelated to the function of the full model. If this was the case, I believe a number of the conclusions in the paper would not be justified.

  The paper does provide some evidence this may not be occurring. For example, the double-addition experiment (Figure 2) shows that inverted masks perform well on the other pair. This suggests the mask and inverse mask may correspond to distinct functionality in the full model. However, Figure 10 tells a different story, where inverse masks do not perform well on the other task. Is this due to the shared IO in the addition/multiplication task? Is there any relation between the addition mask with shared weights removed and the inverted multiplication mask with shared weights removed?

   Ideally, the paper would provide more evidence to support this assumption. Are units correlated in the masked and unmasked model? Could the analysis here be supported by explicitly constructing a model that exhibits reuse, and verifying it can be identified?

3. The paper argues in several places that a weight level analysis is necessary for reasoning about functional modularity, e.g., "without considering the contribution of individual weights it is not possible to reason about *functional* modularity." I believe this statement conflates two separate issues; (1) if modules should be defined in terms of units or weights, and (2) whether modules should be defined by functionality (as done in this work) or using techniques such as clustering. On the latter point, I agree with the statements in the paper and believe the functional approach is more closely aligned with intuitive notions of what is meant by modularity. However, I don't believe a weight vs unit level perspective is necessarily mutually exclusive. For example, consider a simple affine layer $h = f(Wx)$. Defining a module as a subset of elements of $h$ would be equivalent to defining a module as a subset of rows in $W$. 

4. Figures 2, 10, 14 would be much clearer if presented in a tabular format. I had to write out the results this way myself to aid understanding.

5. I found the use of a non-symmetric sharing metric extremely confusing. I spent a while pondering over Figure 1 trying to figure out what the weights were shared with since both tasks are represented in the figure. I believe this issue could be easily remedied, for example, by using something like IoU (Jaccard index). The paper uses this metric in Appendix B.1 to determine the amount of sharing between different masks trained on the same network and task.

6. Permuted MNIST Experiment

   1. The paper states "it suffices to re-train a new first layer to undo the permutation so that later layers can be reused." This makes sense. However, given that the procedure is "freeze the occupied weights," lower level weights cannot perform this simple permutation by construction since they are frozen (I've assumed that occupied means a non-zero mask value).
   2. Why was the choice made to "train masks and weights simultaneously" in this setting? This introduces an additional source of variation from previous experiments, and I'm unsure what the benefit is.

7. SCAN experiments

   1. The paper states "if the masking process removes any important weights, then the solution is pattern-recognition like instead of being based on reusable rules, confirming explanation." This is related to point (2) above, and I'm not convinced this is the only possible conclusion. While the mask has identified a subnetwork whose solution is pattern-recognition like, I'm not sure that the logical jump to conclude the model as a whole performs identically is sound.
   2. Regarding the output weight analysis, I'm also unsure why it should be the case that the final layer is sufficiently powerful for unbinding bound variables. Isn't it also possible that the mask learns to ignore those weights because it is not important for the task it is trained on?

### Recommendation

I recommend acceptance. Although I have several concerns with the paper, I found the ideas and analysis presented very interesting. I believe the community would benefit from further discussion, scrutiny, and exploration of the ideas presented.

### Post Author Response Period Update

Most of my concerns have been addressed by the additional experiments and updated language in the latest revision. I believe the techniques and analysis presented here for assessing reuse could be an important step between observations and explanations for the failure of NNs to generalize systematically. I have raised my score accordingly.

### Minor Issues

1. Appendix C.2: "We always check the **choosen** hyperparameters" => "We always check the **chosen** hyperparameters"

### References

1. Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 67–82, 2018.
2. Adam Gaier and David Ha. Weight agnostic neural networks. In Advances in Neural Information Processing Systems, pp. 5364–5378, 2019.
review_title: Interesting exploration of modularity in Neural Networks, although some conclusions not fully justified

Meta-review: This is a paper that is actively discussed.  The general sentiment is that this paper aims to address an important set of questions. While the technique could be improved with more novelty, the empirical study is extensive. The concerns are about how to interpret the results, or rather whether the empirical evidence fully supports the the claim/hypothesis.  After discussion and rebuttal, the reviewers improved their scores (and one reviewer remained at "weaker marginally above threshold").

The AC read the paper and the discussion. One value the AC sees that the discussion threads between the authors and the reviewers provide a significant amount of scientific value -- the questions to be answered are hard and might indeed require further refinements in framing and conceptualization, better techniques,  strong power in experimental designs to rule exclusively various hypotheses.  Thus, the AC recommends acceptance.
Decision: Accept (Poster)