Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
Authors: Róbert Csordás
Sjoerd van Steenkiste
Jürgen Schmidhuber
Venue: ICLR 2021 Poster (2021)
Keywords: modularity
systematic generalization
compositionality

Abstract: Neural networks (NNs) whose subnetworks implement reusable functions are expected to offer numerous advantages, including compositionality through efficient recombination of functional building blocks, interpretability, preventing catastrophic interference, etc. Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality. In this paper, we present a novel method based on learning binary weight masks to identify individual weights and subnets responsible for specific functions. Using this powerful tool, we contribute an extensive study of emerging modularity in NNs that covers several standard architectures and datasets. We demonstrate how common NNs fail to reuse submodules and offer new insights into the related issue of systematic generalization on language tasks.

Published as a conference paper at ICLR 2021
ARE NEURAL NETS MODULAR? INSPECTING FUNC- TIONAL MODULARITY THROUGH DIFFERENTIABLE WEIGHT MASKS
Róbert Csordás
Sjoerd van Steenkiste IDSIA USI / SUPSI ejcerd@idiia
Jürgen Schmidhuber IDSIA / USI SUPSI NNAISENSE juergenSicsia.ce
IDSIA / USI / SUPSI
sobert@idsia.
ABSTRACT
Neural networks (NNs) whose subnetworks implement reusable functions are expected to offer numerous advantages, including compositionality through ef- ficient recombination of functional building blocks, interpretability, preventing catastrophic interference, etc. Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality. In this paper, we present novel method based on learning binary weight masks to identify individual weights and subnets responsible for specific functions. Using this powerful tool, contribute an exten- sive study of emerging modularity in NNs that covers several standard architectures and datasets. We demonstrate how common NNs fail reuse submodules and offer new insights into the related issue of systematic generalization on language tasks.
INTRODUCTION
Modularity is an important organization principle in both artificial (Ballard 1987 Baldwin & Clark
2000) and biological (von Dassow & Munro 1999, Lorenz al. 2011 Clune et al. 2013) systems. It provides a natural way of achieving compositionality. which appears essential for systematic generalization, one of the areas where typical artificial neural networks (NNs) do not yet perform
well (Fodor et al. 1988 Marcus 1998:
Lake & Baroni 2018 Hupkes et al. 2020).
Recently, NNs with explicitly designed modules have demonstrated superior generalization capabil- ities (Clune et al. 2013 Andreas et al.] [2016] Kirsch et al., [2018] Chang et al. [019] BBahdanau et al.,
2019: Goyal et al. 2021bb, which support this intuition. An implicit assumption behind such models that NNs without hand-designed modularity do not learn to become modular by themselves. In contrast, it was recently shown that certain types of modular structures do emerge in standard NNs (Watanabe 2019 Filan al.] (020). However, due to defining modules in terms of activation statisties or clustering connectivity, it remains unclear whether these correspond to a functional decomposition.
This paper contributes new insights into the generalization capabilities of popular neural networks by investigating whether modules implementing specific functionality emerge and to what extent they enable compositionality. This calls for a functional definition of modules, which has not previously been studied in prior work. In particular, we consider functional modules given by subsets of weights (i.e. subnetworks) responsible for performing a specific 'target functionality'. such as solving subtask of the original task. By associating modules with performing a specific function they become easier interpret. Moreover, depending on the chosen target functionality, modules at multiple different levels of granularity can be considered.
To unveil whether a NN has learned to acquire functional modules we propose a novel analysis tool that works on pre-trained NNs. Given an auxiliary task corresponding to a particular target function of interest (e.g., train only on a specific subset of the samples from the original dataset), we train probabilistic, binary, but differentiable masks for all weights (while the NN's weights remain frozen). The result is a binary mask exhibiting the module necessary to perform the target function. Our approach is simple yet general, which readily enables to analyze several popular NN architectures on a variety of tasks in this way, including recurrent NNs (RNNs), Transformers (Vaswani et al. 2017), feedforward NNs (FNNs) and convolutional NNs (CNNs).

---

Published as a conference paper at ICLR 2021
To investigate whether the discovered functional modules part of compositional solution, analyze whether the NN has the following two desirable properties: (P (Pspeciatize) it uses different modules for very different functions, and (P reuse) it uses the same module for identical functions that may have to be performed multiple times" Here we treat P Pspecialize and Pruse as continuous quantities, which lets us focus the degree which functional modularity emerges. Further, since for many tasks it is unclear what precise amount of sharing is desirable, we will measure P Pspocialize and Presse by considering the change in performance as a result of applying different masks corresponding to target function. This yields an easy to interpret metric that does not assume precise knowledge about the desired level of weight sharing. We experimentally show that many typical NNs exhibit P Epeecialize but not Preuse By additionally analyzing the capacity for transfer learning, we provide further insight into this issue. We offer a possible explanation: while simple data routing between modules in standard NNs is often highly desirable, it is hard to learn since the weights must also implement the data transformation. Indeed, our findings suggest that standard NNs have no bias towards separating these conceptually different goals of data transformation and information routing.
We also demonstrate how the functional modules discovered by typical NNs do not tend to encourage compositional solutions. For example, we analyze recoderd LSTMs (Hochreiter & Schmid huber 1997) and Transformers (Vaswani et al. 2017) on the SCAN dataset (Lake & Baroni (2018) designed to test systematic generalization based on textual commands. We show that combination- specific weights are learned to deal with certain command combinations, even when they are governed by the same rules as the other combinations. The existence of such weights indicates that the learned solution is composoooiii and fails at performing the more symbolic manipulation required for sys- tematic generalization on SCAN. To demonstrate that this issue present even in more real-workle sce- narios, we highlight identical behavior on the challenging Mathematics Dataset (Saxton et al. 2019).
Finally, we study whether functional modules emerge in CNNs trained for image classification, which are thought to rely heavily on shared features. Surprisingly, we can identify subsets of weights solely responsible for single classes: when removing these weights the performance on its class drops signif- icantly. By analyzing the resulting confusion matrices, we identify classes relying on similar features.
2 DISCOVERING MODULES VIA WEIGHT-LEVEL INTROSPECTION
To investigate whether functional modules emerge in neural networks one must perform a weight-level analysis. This precludes the use of existing methods, which discover modular structure in NNs based on clustering individual units according to their similarity (Watanabe) 2019 Filan et al. 2020) and that may not always be enough to draw meaningful conclusions. Units can be shared even when their weights, which perform the actual computation, are not. Indeed, units can be viewed as mere "wires" for transmitting information. Consider for example gated RNN, such as an LSTM, where gates can be controlled either by the inputs or the state, yet make use of different weights to project to the same gating units. To overcome this limitation, we propose novel method to inspect pre-traiie pre-trained NNs at the level of individual weights. It works as follows. First, we formulate a target task corresponding to the specific function for which we want investigate if a module has been learned. For example, this can be a subset of the original problem (i.e. a subtask), or based on a particular dataset split, e.g. to test generalization. Next, we train weight mask on this target task while keeping the weights themselves frozen. The resulting mask then reveals the module (subnetwork) responsible for the target task.
To train the mask, we treat all N weights seperately of each other. Let i E NJ to denote the weight index. The mask's s probabilities are represented as learned logits 4, E IR which are initialized to keep the weights with high probability (0.9). If one were to apply continuous masks to the weights it would be possible to scale them arbitrarily, thereby potentially modifying the function the network performs. To prevent this, we binarize masks, which only provides for keeping or removing individual weights. The binarization is achieved using Gumbel-Sigmoid with a straight-through estimator, which we derive from the
(Jang et al. 2017: Maddison et al. 2017) in Appendix A. A
'e emphasize the distinction between the ability to reuse modules and the ability to compose them: a compositional solution may fail to reuse a module to implement the same behavior multiple times. Similarly, weights can be reused without them being composed to yield a compositional solution. Further, we consider specialization of modules a special case of modularization where modules are specialized to implement particular functionality that is semantically meaningful.
2

---

Published as a conference paper at ICLR 2021
1.0
1.0
Propartion
loU
loMin
Propartion
lol
loMin
0.5
0.5
0.0
0.0
layer
layer
layer
layer
aulput
1stm 0.hhh
lstm.0.1t 0.1h
Istm.1.MM
lstm.11 Lihh
oulput
Figure 1: Proportion of shared weights per layer on addition/muuliillication Left: FNN, right: LSTM.
sample 8, E [0, from the mask can be drawn as follows:
8; ((4 log (log / log U2)) /r) with U1 U(O,1),
where r E (0, is the temperature and (x)
Ite is the sigmoid function. Next, we can Bengio et al. 2013) to obtain a binarized sample
use straight-through estimator (Hinton 2012 E {0,
[1,5 stlstop + 84s
where I is the indicator funetion and supp is an operator for preventing backward gradient flow. In random variable (proof in Appendix A..3). The
this case the b, are samples from a Bernoulli(o masks are applied elementwise: w/ = w, *b, Training is done by applying the loss function defined by the target task and backpropagating into the logits 4-- Typically multiple (between 4 8) binary masks are sampled and applied to different parts a batch to improve the quality of the estimated gradient.
The goal of the masking is to remove weights that not necessary to perform the target function. Thus, the logits 4 should be regularized, such that the probability for weight w; to be active small unless Wt is necessary for the task. This is achieved by adding regularization term r a Σ: the responsible for the strength of the regularization. How to
loss, where a E [0, 20) is a hyper-paramett best choose a is described in detail in Appendix C.3 At the end of the training process, deterministic binary masks M, E {0, 1} for weights are obtained via thresholding M; = ll.(.)) >0.5 Applying the full mask M then uncovers the module responsible for the target task. prelimmary study confirmed that the mask training process is stable and thereby suitable for inspection (Appendix BB1I.
In the following sections we will analyze several standard NNs using this technique of mask- toosk-training] Throughout our experiments we will avoid drawing conclusions based the measured amount of sharing alone as much as possible, since it is unclear what degree of sharing can be expected or desirable. Rather, will analyze the performance drop caused by removing weights corresponding to certain functionality, which offers a more consistent and easier to interpret metrid For example, to show that a module is responsible for particular subtask (A) but not for another (B), we train masks on A and test on both. performance drop is expected on task only. In contrast, to show that this module is exclusively needed for a particular subtask, we can invert the masks and test on both tasks. The inverted masks are expected to perform well on the complementary task, but not on the original one. However, we note that this mask inversion method is limited to analyzing entirely disjoint weights.
We analyze weight sharing between two tasks using two different metrics: one is Intersection over Union (IoU), which measures how much the weights used for solving the tasks overlap. We call the other Intersection over Minimum (loMin), which measures the number of overlapping weights (intersection) divided by the minimum of the total number of weights used for each task. In that sense JoMin measure of "subseenness". Intuitively, if weights are shared, both IoU and IoMin are zero. If all weights are shared, both loU and loMin are one. However, when the weights needed for one task are strict subset of the other, then loMin one, while IoU < 1.
In general we find that concentrates either or and so thresholding is safe (see also Fig. 9 A complete overview of all experimental details is available in Appendix Mcan and standard deviations shown in the figures are calculated over 10 runs unless otherwise noted.
'Code for all experiments is available at whhttps: // tiggthu.. com/ http://berttaarrass//eoo
"Exceptions only include cases where the observed amount of weight sharing can be clearly interpreted. However, in these cases, our analysis will focus general trends rather than the precise amounts observed.
3

---

Published as a conference paper at ICLR 2021
3 ANALYZING FUNDAMENTAL PROPERTIES OF MODULES
Let us consider P Pspecialize and Press (defined in Sec. T in more detail, as they reflect the advantages of modular compositional design. According to our notion of functional modularity, an NN is not modular without Papecialize- Moreover, disjoint modules prevent catastrophic interference (McCloskey & Cohen 1989 Rosenbaum et al.; 2019). since changing the weights responsible for specitic function does not affect the others. P rouse also has multiple advantages. It increases data efficiency by processing all relevant data using the same module, which thus receives additional training when a module can be reused. It also helps with generalization. For example, consider processing the expressions * b and (c + d) where a, b, and d are sampled from the same range. By reusing the multiplier it will be able to perform a*b on wider range of inputs then it would otherwise be trained for.
In this section conduct several experiments using synthetic datasets test whether NNs have natural inductive bias supporting Pspecialize and Prewse These experiments are designed to be as simple as possible to isolate the property of interest. Let's assume the network consists of compositional mod- ules. The input of such modules can come from multiple sources within the network. Similarly, their output could be connected to different parts of the network. For example, the previous arithmetic ex- pression, the first operand of the multiplier can come directly from the input or the output of the adder. The same holds for the outputs. Therefore we consider cases where the imputs and outputs are shared between the modules of the ideal solution (shared 1/O) and where they are separated (sepurate I/O),
We construct two different datasets for analyzing P Pspecialize and Preuse. For P Pspecialize: we use shared I/O and two different target functions (additiionmuutiilicaatoo task in section 3.T). The shared I/O biases the network towards weight sharing by default. Thus we use this dataset to test whether there is a bias for specializing different computations (functions) to separate weights. In contrast, to test for Preuse. we construct dataset where the same function should be performed twice, but using separate 1/O (double Since separate I/O biases the network to not share weights at initialization
addition task in section time, we will make use of this dataset to test whether NNs exhibit a bias for reusing computation. Here reusing weights is expected, since information routing is assumed be easier to learn than the actual function (addition). We emphasize that this initial bias due to different choices for I/O arises naturally in any network composing multiple different internal modules to arrive at a solution.
The conclusions are surprising: typical NNs tend to satisfy Pspecialize but not Preuse- Our experiments suggest that weight sharing across tasks is mostly driven by shared I/O rather than task similarity, which results in redundancies and lack of data efficiency.
3.1 ADOTTIN//ULTIPPICATON EXPERIMENTS
The additiinnmulliplication dataset is designed to test P Pspecializee The task is to add multiply numbers (modulo 100). The input and output units are the same for both operations. An additional one-hot input specifies the operation. The numbers are two-digit and encoded as two 10-way one-hot vectors, each representing a digit. Thus, the total input is 42-dimensional,, and the output is 20.
First, we train the network to perform this task without any masking. Once the performance is nearly perfect, we freeze its weights. We perform two stages of mask training: first, we train mask on addition (multiplication examples excluded), then we repeat this procedure for multiplication.
We analyze FNN and LSTM on this task. For LSTM, we present the full input for fixed number of timesteps. The result is the output at the final step. No loss applied at intermediate steps. Regardless of the architecture, we found the same general tendencies: There is more sharing in the input and output layers and less in the hidden layers (Fig. Dp. We also found that the multiplication uses 3.8 times more weights than the addition (Fig. [2]. which partially explains the low loU in this case. We conclude that there does appear to be some bias towards specializing weights according to different functions. On the other hand, the separation might still be inadequate to prevent interference and catastrophic forgetting. Increased sharing in I/O layers could be due to a switching/routing procedure used to select which operation perform.
We further analyzed how performance breaks down on the task for which the mask was not trained on. Here the behavior of the FNN and the RNN differ. The FNN tends to ignore the function description and performs the operation for which the mask was trained, while the LSTM tends to produce invalid outputs, suggesting that it learned solution where the two operations are more intertwined (Fig. 13
4

---

Published as a conference paper at ICLR 2021
Full
Pair
-Pair 1
Pair
-Pair
Pair
100 0.0
100 0.0
20 12.7
1 0.1
92 + 10.5
FNN
Pair 2
100 + 0.0
1 + 0.1
94 + 6.7
100 + 0.0
21 + 11.0
Pair
100 = 0.0
100 0.0
2.5
1 0.1
99 3.0
LSTM
Pair 2
100 = 0.0
1 1 0.1
100 + 0.2
100 = 0.0
2 0.3
Pair
100 = 0.0
100 = 0.0
4 0.8
1 + 0.1
99 0.7
LSTM (forced)
Pair
100 + 0.1
1 0.1
96 4.1
100 = 0.0
0.6
Table I: Double-addition task: accuracy [%] of LSTMs and FNN on the pairs. In case of LSTM (forced) only one input is presented at a time (to prevent interference). The header shows on which pair the mask was trained on. denotes an inverted mask.
3.2
DOUBLE-ADDITION EXPERIMENTS
1.0
IoU loMin
loU loMin
Proportion
Proportion
0.5
0.5
layer
layer 2
layer 3
layer
output
Istm.0.Jh
Istm.0.hh Istm.1.Jn in lstm.1.hm
output
(a) FNN
(b) LSTM
Figure 2: Double addition task: proportion of weights shared per operation in case of (a) feedforward network, (b) LSTM, both inputs presented together. The first and last layers have no shared weights.
The couble-addition experiment designed test property P Preuse- The task to perform modulo 100 addition twice using separate I/O (different units) for each of the two instances. Using imputs b, and d, the network should output + b and c d. This simulates the realistic scenario of having different data sources within network when composing modules dynamically, without considering the additional problem of finding the right composition. Since the operation is the same and the operands' data distributions are exact matches, this simple setup encourages sharing. The encoding is the same as in section 3.I resulting in 80 input and 40 output units.
We first train the network until convergence the full task, then freeze its weights. We train mask on + b, followed by c + d. We analyze both FNN and LSTM architectures. FNN needs special care to avoid activation interference. When both operations have to be performed simultaneously. sharing is impossible. Thus, for the FNN, we perform two forward passes. In each pass, we feed only one pair of numbers to the network (either a, b or c, d), while zeroing out the other. With LSTM, investigate two different settings. In the first, both pairs are presented together for fixed number of steps, and the result the final output. Hence, the LSTM allowed to schedule the execution of the operations freely. In the second setting, called LSTM (forced), we remove any incentive for solving the pairs simultaneously by feeding a single pair for multiple steps with the other zeroed out, and then read its output. This procedure is then repeated for the second pair without resetting the state.
All experiments' results are consistent: weight sharing low (Fig. 2). To assert the modules' independence, we invert masks trained on pair I, removing all weights needed for pair 1. We test the resulting network on pair 2, where the performance decreases only slightly, suggesting that they are independent (Tab. [ further analysis is provided in Appendix C. No difference between the two LSTM variants was observed.
These observations show that Preuse is violated even in this simple case. Realistic scenarios tend to be more complex as the data distribution for different operation instances might be different (with overlaps), providing even fewer incentive to share. Furthermore. comparing the results to those of section 3.1 it apparent that sharing depends more on the location the inputs/outputs than on the similarity of the performed operations. This behavior is undesired and calls for further research.
5

---

Published as a conference paper at ICLR 2021
1.0
Figure 3: Proportion of the weights of task shared with any of the previous tasks. Every second task on permuted MNIST. Each task corresponds to permu- tation. The last layer has the lowest capacity, filling up first, forcing subsequent runs to share weights.
Proportion
T8
T10
T8
0.5
0.0
Layer
Layer 3
Layer
3.3
TRANSFER LEARNING EXPERIMENTS
Let us now consider a more complex setting to assess the degree to which property Preuse is violated (see sec. (.2). Here, we will measure the amount of possible transfer in a continual learning setup using the popular permuted MNIST benchmark (Kirkpatrick et al., 2017 Golkar et al., 2019 Kolouri ett a 2019). A sequence of tasks is created by applying different permutations to MNIST images (LeCun al. 20TO). Spatially close pixels may no longer be observed in nearby locations in this case, which leads us to train a FNN (as opposed to a CNN) sequentially on all permutations (tasks),
Continual learning closely related transfer learning, which is additionally concerned with transferring knowledge between tasks to improve learning and use fewer parameters. Typical approaches revolve around freezing used weights via masking when a new task is added (Fernando itt al. 2017 Mallya & Lazebnik 2018 Golkar et al. 2019). We adjust our method accordingly: We train on a single task and freeze the occupied weights. In particular, to be able to bias the network towards weight sharing, we train masks and weights simultaneously in this case. The free weights are then reinitialized and a new mask is allocated for the next task to obtain a mask for each permutation.
Note that since each task differs only by the input's permutation, it suffices to re-train a new 'first layer' to undo the permutation so that later layers can be reused. Indeed, since a significant portion of the weights is in the hidden layers, knowledge transfer between the permutations is possible and expected to be beneficial. However, re-learning the first layer may not always possible in practice since the required weights could already have been occupied to address previous permutation. To ensure that this does not happen, we always reset the first layer and do not freeze any of its elements. Notice how, while this departs from the standard transfer learning setting, it still provides us with an upper bound on the amount of transfer that is possible when no such conflict occurs. Even with these modifications, we observed only little weight sharing when sufficient free space was available (Fig. 3 Only once all the capacity is saturated, weights become shared. This effect is especially apparent for the output layer.
We additionally conducted experiment in which explicitly bias the network towards sharing. We initialized elements of new masks corresponding to occupied weights by significantly higher probability (P x 0.88) compared to the unused ones (P 0.27). Intuitively, this encourages reusing the old, frozen network and adds new weights with low probability. This was able to force the network to significantly share in later layers (see Fig. [4.. However, we emphasize that knowledge about which weights have to be reused between which samples is usually not available and explicitly biasing the network in this way is therefore generally not possible.
Together these observations re-affirm that Preuse does not emerge naturally and that the same functionality re-learned. This both redundant and potentially harmful as we investigate in Sec.
3.4 A POTENTIAL EXPLANATION FOR LACK OF WEIGHT SHARING
Let us consider a possible explanation for the lack of weight-sharing observed in sections 3.2 and 3.3 which is that data routing is difficult in standard NNs. Indeed, inputs and outputs must be correctly routed to different sources/targets to reuse modules in different compositions. In routing networks (Kirsch al. 2018 Rosenbaum et al., 2019) Chang et al. 2019), this is achieved through hand-designed mechanisms. Without those, routing can only occur through the weights of the NN. However, such a 'routing transformation' would change the data representation alongside the routing unless the weights have a special structure that we empirically find is hard to learn. Indeed, our experiments suggest that NNs find hard to learn to represent data similarly along different information routes that can in principle be processed by single module. We argue that this is an important issue and that additional research on suitable inductive biases is needed to address this. Further discussion the potential role of attention mitigate this is provided in Appendix B.4

---

Published as a conference paper at ICLR 2021
Accuracy [%)
100
Flemoeed
25
LTTUNN
JJUMP
LRUN
EOS
Turn Left
Jump
Length
LTURN RIGHT
LWWLKK
LLOOK
(a) Performance on different splits
(b) Weights removed from last layer on "Add jump"
Figure 4: Results of experiments on SCAN. (a) Test accuracy on split shown on r-axis with masks trained on the full problem (blue, orange) and with masks trained on split shown on æ-axis (green, red). LSTM: blue, green, Transformer: orange, red (b) Percentage of weights removed per token from the output layer of the LSTT decoder trained on the "Add jump" split.
4 ANALYZING SYSTEMATIC GENERALIZATION ON ALGORITHMIC TASKS
Let us now consider the known issue of systematic generalization in light of our previous observations. First, what is meant by systematic generalization? Once NN has learned perform certain operations and some of their combinations, it should perform well on unseen combinations governed by the same algorithmic rules. Hence, it requires Preuse to hold. The failure of typical NNs to generalize systematically is one of the central issues of current-day NN research. The SCAN dataset (Lake & Baroni] 2018) is designed for analyzing the degree which NNs can generalize systematically. It consists of compositional commands (e.g. "jump twice"), to be translated into primitive output moves (e.g. "JUMP JUMP"). The "simple" data split is IID, the "length" split has shorter training samples than test samples, and the "add primitive" splits have particular command presented in the training set but no compositions of this command with others (as in the test set).
It was previously shown that typical NNs generalize poorly on data splits systematically different from the train set (Lake & Baroni 2018 Saxton et al. 2019). The root of the problem is unclear, however. In fact, there might be two explanations: (a) The NN might have learned the correct algorithm for solving the problem, but failed to pick up on certain symmetries between concepts due scarce evidence in the train set. For example, in the "add primitive" split, the NN might be unable to form an analogy between the additional primitive and the well-perforning ones. This can also be understood as representation problem: this case the NN has failed to represent the new primitive in a way that lets them be used for problem solving in similar manner following the acquired solution. However, the NN not pressured improve since the learned solution suffices for solving the training set. (b) Alternatively. the NN might not have learned the correct algorithm to solve the problem. For example, may have learned to recognize patterns determining when an output token should be produced in place of reusable rules. In this case, new weights are required to solve new problems of the same kind, since they correspond to different patterns. Only this case, we argue, has the NN failed to leverage the problem's 's compositional nature. Note that (a) requires Preuse to hold such that weights responsible for performing each individual operation are shared between different samples, while (b) does not.
We have tested two networks: the baseline 2 layer LSTM encoder-decodee model from Lake & Baroni 2018) and Transformer (Vaswani al. (2017) (see Appendix (.7.1). We pretrain the model on the IID data, which ensures that the learned weights are capable of solving the full problem and that potential absence of sufficient evidence for learning about the correct symmetries between concepts is not an issue. Hence, this rules out explanation (a) being the only issue. For each split, we train a mask on its train set and measure the discovered subnetwork's performance on the corresponding systematically different test set. This process removes the weights that are not required to solve the train set for a given split. However, all splits' train sets contain sufficient information about the full set of rules required to perform well on any split. Hence, should the masking process remove any important weights, then we argue that the solution is likely pattern-recognittee like rather than based on reusable rules, providing evidence for explanation (b). Indeed, our experimental results demonstrate precisely this behavior as can be seen from the large generalization gap in Fig. 4a Note that while this gap is consistent with the findings Lake & Baroni (2018), we additionally able to provide evidence that the learned algorithm is likely inherently non non-compositional, i.e. by eliminating explanation (a) being the only issue as possibility.
7

---

Published as a conference paper at ICLR 2021
Accuracy [%]
100
Figure Accuracy on the "hard" test set of differ- ent tasks of the Mathematics Dataset: model without masks, masks trained on IID data and masks trained on "easy" set. A performance drop can be observed, because of the sample-specific weights. 5 seeds/task.
Add or sub
Differentiat
Poly. Collect
Linear 1D
Sort
Reathro drop (S)
100
10.0
otherfagge
7.5
50-
5.0
2.5
0.0
akplana
bird
dloor
trog
ahip
Arrjane
Sind
deef
fmo
ship
automoblle
6at
dog
horee
truck
autommobilo
cat
dog
horse
truck
(a) Relative performance drop per class
(b) Largest drop in non- non-target class (relative)
Figure 6: (a) Relative drop in performance for simple CNN, simple CNN without dropout and
ResNet-110. (b) Largest performance drop in a non-target class relative to the drop in the target class.
To assess if the same behavior can be observed in a more complex real- world setting, we conduct similar experiment on the challenging Mathematics Dataset (Saxton et al. 2019). Here, we generated difficulty-baseee splits for tasks like differentiation, solving linear equations, sorting, etc. (further details in Appendix C.7.2). In Fig. 5 a consistent performance drop can be observed when applying the inferred subnetwork on the "hard" split using mask trained the "easy" split. This demonstrates that samples in the "hard" split depend on exclusive weights, despite those being governed by the same underlying rules, which is consistent with results on SCAN.
The weight level analysis provided by our method enables us to gain further insight. We inspect the LSTM decoder's weights on the "add jump" split of SCAN and note that the most apparent difference is in the output layer. Almost half of the weights corresponding to "LJUMP" are removed (Fig. 4b). suggesting that the network learned to detect patterns of cases when "I JUMP" should be the output, and the last layer puzzles them together. In contrast, we hypothesize that the generalizing algorithm for solving such problems necessitates proper variable manipulation (Garnelo Shanahan 2019)
5 ANALYZING CONVOLUTIONAL NEURAL NETWORKS
As a final case-study, we consider whether we are also able to observe lack of weight-sharing in CNNs. By conducting a leight-level analysis using our tool, we are able highlight sets of non-shared weights solely responsible for individual classes. We consider multiple CNN architectures trained on CIFARIO (Krizhessky et al. 2009): simple CNN with dropout (an ablation is provided in Appendix C.8.2 and ResNet- 110 (He al. (2016). Full details proceed as follows. First, we train a "control mask' on the full dataset to highlight all used weights. Next, we train mask with a single class removed so that the weights solely responsible for this class will be absent from the resulting mask. Here we avoid removing all weights responsible for the this class from the output layer (leaving no connection to the corresponding output unit) by fixing its mask to one trained on the full dataset. This corresponds to inspecting the feature detector layers as opposed to the classifier. We repeat this process for all classes to obtain a total of 11 masks.
available in Appendix C.8). We
We compute the confusion matrix on the full validation set at the end of each stage. Then we calculate the difference between the confusion matrices with and without the removed class, which unveils how the removal changes the classification. Interestingly, the performance of the target class drops significantly (Fig. 6a), only a small drop in performance (possibly due noise when mask sampling) is observed for non-target classes (Fig. (b). This indicates a large dependence on class-exclusive, non-shared weights the feature detectors. These findings, which assume that the network has sufficient capacity relative to dataset size, are in line with those observed in sections 3 Analyzing the difference in misclassification rates yields further insights: as the true positive rate drops, certain other classes predicted instead that appear to rely on similar shared features. For example, removing "airplane" causes images to be classified as "birds" and "ships" instead, which have a blue background in common. Additional insights are reported for other classes in Fig. 18 in Appendix C.8
8

---

Published as a conference paper at ICLR 2021
6 RELATED WORK
There have been few other attempts at analyzing emerging modularity in NNs. Filan et al. (2020) iden- tifies groups of neurons with strong internal and weak external connectivity via clustering, while others group neurons based on their connectivity pattern (Watanabe et al. 2018) or cluster them hierarchically based on activation statistics [Watanabe] 2019). However, as we have argued, without considering the contribution of individual weights it is not always possible to reason about functional modularity. Davis et al. (2020) considers an alternative approach based on mutual information to detects salient pathways in NNs that could in principle allow for this. However, the discovered pathways are not grounded with respect to particular functionality, nor is it analyzed whether they support composi- tionality. [Bengio et al. 2013) formulate adaptive mask learning reinforcement learning problem, with the main goal of accelerating inference via conditional execution. However, the masking is unit level and trained together with network weights. Similarly, functional modularity is not considered.
It has repeatedly been argued that NNs lack compositionality due to their failure at systematic generalization (Lake Baroni 2018 8 Bahdanau al., 2019, Barrett et al. 2018 Hupkes et al.) 2020) Hill et al.) 2020). Typically this analysis proceeds by evaluating NNs on a dataset that exhibits some systematic difference from the training data, yet can be solved through clever recombination of known concepts based on inferred rules. Here we were additionally able to show that the learned solution incorporates combination-specife weights even on the training set, suggesting that this issue runs deeper than simply being unable to learn about symmetries between well-know and novel concepts. Andreas 2019) introduced a direct measure compositionality by reconstructing NN representations from set of primitive representations and learned composition function. However this method is concerned about the data representation and not about the modularity of the performed computation.
Finally, we note how many transfer and continual learning methods make use of weight freezing via masking to prevent catastrophic forgetting (Fernando et al. 2017 Mallya & Lazebnik] [018]] Golkar
et al. 2019 Yang al. 2020). Determining the importance of individual weights has been studied in [
network pruning (LeCun et al.
1990; Hassibi & Stork 1993 Li et al. [017] [Frankle & Carbin 2019)
Gaier& & Ha 2019) and feature attribution (Simonyan et al. 2013 Springenberg et al. 2015 Sundarara
jan et al. 20 1/177//rrr et al. (017)) often using weight and/or gradients magnitudes. Differentiable
binary weight masks have also been explored in the multi task setting (Mallya et al. [2018), albeit deter- ministically in contrast to the Gumbel-Sigmoid used here. It should also be mentioned how many ex- plicitly modular architectures have been proposed improve generalization (e.g. Clune et al. (2013); Andreas et al.] (2016); Kirsch et al. (2018); Chang et al. (2019): Goyal et al. (2021b)) and data eff- ciency (Purushwalkam et al. 2019). Rather than engineering an explicitly modular solution, our goal is to let this emerge naturally. We believe that our current findings help take step in that direction.
CONCLUSION
Our new method for inspecting modularity in neural networks is the first to identify modules by their functionality. It is a powerful tool for analyzing how the NNs share or separate weights based on the performed computation. By analyzing diverse sets of neural networks (FNNs, CNNs, RNNs, Trans- formers), could draw significant novel conclusions: in typical current NNs, weight sharing between modules does not reflect task similarity (as desired) but can mostly be explained by rather trivial shared IO interfaces of solution-impa implementing modules. The lack of weight sharing between multiple uses of the same function makes the learning data inefficient since it has to be re- re-leamed repeatedly. Moreover, NNs trained on algorithmic tasks appear fail to learn general, modular, compositional algorithms. Rather, we have shown that they require specific subset weights to solve a particular combination of the input tokens, even when the same rules govern both the solution as the other samples. Our discoveries call for future research: function dependent weight sharing in the neural networks should vastly improve data efficiency, and encouraging algorithmic solutions should improve generalization.
ACKNOWLEDGMENT
We wish to thank Aleksandar Stanić, Francesco Faccio, Kazuki Irie & Louis Kirsch for their construc- tive feedback. This research was supported by European Research Council Advanced grant (no: 742870), two Swiss National Science Foundation grants (no: 200021 200021_16567511, 165675/1, 200021 11922_611)36)) and hardware donations from NVIDIA & IBM. We thank Weights & Biases for free academic license.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Jacob Andreas. Mcasuring compositionality in representation learning, In International Conference on Learning Representations (ICLR), 2019.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference Learning Representations (ICLR), 2019.
Carliss Young. Baldwin and Kim B. Clark. Design rules / the power of modularity. MIT Press, 2000.
Dana H Ballard. Modular learning in neural networks. In Proc. AAAI, pp. 279-284, 1987.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pp. 511-520, 2018.
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:/511.06220 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 2013.
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. Automatically composing representation transformations as means for generalization. In International Conference on Learning Representations (ICLR), 2019.
Jeff Clune, Jean-Baptiste Mouret, and Hod Lipson. The evolutionary origins of modularity. Proceed- ings of the Royal Society B: Biological Sciences, 280, 2013.
Brian Davis, Umang Bhatt, Kartikeya Bhardwaj, Radu Marculescu, and José MF Moura. On network science and mutual information for explaining deep neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8399-8403, 2020.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations (ICLR), 2019.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 771.08874, 2017.
Daniel Filan, Shlomi Hod, Cody Wild, Andrew Critch, and Stuart Russell. Neural networks are surprisingly modular. arXiv preprint arXiv:2003.0488) 2020.
Jerry A Fodor, Zenon W Pylyshyn, al. Connectionism and cognitive architecture: A critical analysis. Cognition, 28:3-71, 1988.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural
networks. In International Conference on Learning Representations (ICLR), 2019.
Adam Gaier and David Ha. Weight agnostic neural networks. In Advances in Neural Information Processing Systems, pp. 5364-5378, 2019.
Marta Gameeoo and Murray Shanahan. Reconciling deep learning with symbolic artificial intelligence: representing objects and relations. Current Opinion in Behavioral Sciences, 29:17-23, 2019.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. In NeurlPS 2019 Workshop Neuro AI, 2019.
10

---

Published as a conference paper at ICLR 2021
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Charles Blundell, Sergey Levine, Yoshua Bengio, and Michael Mozer. Factorizing declarative and procedural knowledge in struc- tured, dynamical environments. In International Conference Learning Representations, 2021a.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations, 2021b.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05200 2020.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, pp. 164-171, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. In International Conference on Learning Representutions (ICLR), 2020.
Geoffrey Hinton. Neural networks for machine learning. Coursera, video lectures., 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1735-1780, 1997.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrizzaion with gumbel-softmax. In International Conference Learning Representations (ICLR), 2017. 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 11:: 3521-3526, 2017.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning decompose neural computation. In Advances in Neural Information Processing Systems, pp.
2018.
Soheil Kolouri, Nicholas Ketz, Xinyun Zou, Jeffrey Krichmar, and Praveen Pilly. Attention- based structural plasticr.. arXiv preprint arXiv: 1903. 06600000000 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, 2009.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequens recurrent networks. Proc. International Conference on Machine Learning (ICML), volume 80, pp. 1879-28883 2018.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598 598-609- 605, 1990.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available:
wttp.//cccc..cooo////misssss 2, 2010.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations (ICLR), 2017.
Dirk M. Lorenz, Alice Jeng, and Michael W. Deem. The emergence of modularity in biological systems. Physics of life reviews, 8:129-160, 2011.

---

Published as a conference paper at ICLR 2021
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proc. Conference Computer Vision and Pattern Recognition (CVPR), pp. 7765-7773, 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 67-82, 2018
Gary Marcus. Rethinking eliminative connectionism. Cognitive psychology, 37:243-2822 1998.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. In Proc. International Conference on Machine Learning (ICML), 2020.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference International Conference Machine Learning, pp. 807-814, 4,,,7777 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035, 2019.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. Task- driven modular networks for zero-shot compositional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv:1904.1277 2019.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations (ICLR), 2019.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3145- 1153,2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013.
Jost Tobias Springenberg. A. Dosovitskiy, T. Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. International Conference on Learning Representations (ICLR), Workshop Track, 2015.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Learning-Volume Volume 70, pp. 3319-3328.
Proceedings of the 34th International Conference on Machine Learning- 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998 6008, 2017.
12

---

Published as a conference paper at ICLR 2021
G Dassow and E Munro. Modularity in animal development and evolution: elements of conceptual framework for evodevo. The Journal of experimental zoology, 285:307- 1225, 1999.
Chr. von der Malsburg. Self-organization of orientation sensitive cells in the striate cortex. Kybernetik, 14:85-100, 1973.
Chihiro Watanabe. Interpreting layered neural networks via hierarchical modular representation. In International Conference on Neural Information Processing, pp. 376-388, 2019.
Chihiro Watanabe, Kaoru Hiramatsu, and Kunio Kashino. Modular representation of layered neural networks. Neural Networks, 97:62-73, 2018.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. arXiv preprint arXiv:2003. 13661, 2020.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, pp. 3592-3603 2019.
13

---

Published as a conference paper at ICLR 2021
A DERIVATIONS
A.1 FROM GUMBEL-SOFTMAR TO HUMBEL-SIGMOII
In what follows, probabilities are y E 1 is a sample vector from the Gumbel-Softuad distribution (also called Concrete distribution by Maddison
the notation by Jang al. 2017): k is the number of categories, class
al. (2.17)) Individual components of y are denoted by yi, E [1, k]. We will refer to 4 = log as logits. We show how to sample from the Gumbel-Sigmoid distribution, the special case of ko 2,
0 of the Gumbel-Softmax distribution.
First, we show that the sigmoid is equivalent to first element /1 E R of the output vector of the two element softmax with 1 = x; x E IR. and //
(xz)
6 e0
eh el2
91-
According to Jang et al. 2017)) the sample vector y E IR from the Gumbel-Softmax distribution can be drawn as follows:
e (4+9))
y/
EØ=1 e+ (15+))))
(4)
where di (umbel(O, 1) are independent samples from the Gumbel distribution. We are interested the special case of sigmoid, which we showed to be equivalent the yı 2, l2 case:
e (11+)))
(5)
e (9)) 4
This can be rearranged as:
(1 - 922)
(6)
1 - +1 -
Writing out the inverse transformation sampling formula for 9: Gumbel 1); 9: - -log( - log U:), were Us ~ U(0, 1) are independent samples from the uniform distribution, we get:
91 σ (11 - log - (olog U1) + log(-og(oo log U2))
log U1
0 1 log log U₂
Finally, by renaming = and = 1 (we have just single logit), we obtain the sampling formula for -umbel-Sigmoid:
1
log log U1 log U2 )
(8)
A.2 STRAIGIT-TIIROGGI ESTIMATOR
Samples from the Gumbel-Softmax distribution can directly be converted to a sample from the categorical distribution
C 1l arg max, Ve
Applying the straight-through estimator (Hinton 2012 Bengio al. 2013) provide 'hard'
samples while permitting gradient flow (1-)sop is an operator for blocking backward gradient flow):
= [1,=arg max, vs Vilstop +
(10)
Since in the Gumbel-Sigmoid we have k = 2 categories and E, We = 1, the arg max can be replaced by testing whether ys >> 0.5:
G 11, vilstop + Y/.
(11)
Since we defined the sample to be 1/1 (i.e. we are interested in the = case) the index can be omitted. The variable has Bernouli distribution (see Section A.3
substitution b ci
to get:
b [1s00.5 s)soop +
(12)
14

---

Published as a conference paper at ICLR 2021
A.3 THE EXPECTED VALUE OF THE SAMPLES
Let us analyze the distribution governing the samples b. Each sample is binary and independent since it is generated using independent samples from a uniform distribution. Since the sampling process stationary they must therefore be Bernoulli distributed. Next, we show that their mean is μ = a (1).
We are interested in:
μ= P(6 1) = ([1.00. s]stop + + = 1).
(13)
The straight- estimator does not change the numerical value b, so we can ignore it:
P(b 1) (1.20.5 1) (s 0.5),
(14)
where (7
(3( (1-
log
log U1 log U2
Let simplify the condition 0.5:
( ( log log U1 log U2 )) 0.5
(15)
is monotonically increasing and (0) 0.5, so:
1 1 log log U2
log U1
0.
(16)
By multiplying both sides with r > 0 and re-ordering we obtain:
log U1
I log
(17)
log U2
Since e" is monotonically increasing, we can exponentiate both sides to obtain:
log U1
(18)
log U
The samples U; uniform random samples from the range U2 E (0, Hence, it follows that log U, < O. Multiplying both sides by log U2 we get:
e' log U2 < log U1-
(19)
Exponentiating once again leads to:
Ug < U1.
(20)
Let us return to the original problem, which is now takes a much simpler form:
P(b 1) P(Us' U1).
(21)
Using the definition of the mean, we get:
-
(22)
Using the definition of expectation:
P(U2) [ P(Ui) <v, dU
(23)
Since U4 ~ U(0, 1) are samples form uniform distribution with range (0. 1). P(U.) in interval
U, E (0, 1) and 0 otherwise. This enables us to change the boundaries of the integrals:
n= S. S/' "'u'<o,d dU,
(24)
Since the value of 1 Ug <U, is 1 when U1 > Ug' and 0 otherwise, we can tighten the bounds of integration and eliminate the indicator function:
IdU dU, L/ [Uilege dU2.
11
Jug
(25)
15

---

Published as a conference paper at ICLR 2021
S U2 dU'
1
U2
dU
0
1
1e'1 ge't1
U2
=1
e ++ 1
el +1 e +1
(26)
0
1
+1
1 e
=
e
e'11
e 4+1
e++1 cl+1
=(I)
A.4 CHOOSING THE TEMPERATURE
Notice that μ (I) does not depend on the temperature (Appendix A.3) The binarized sample, will have the same output regardless of the value of т. thus will have the same gradients. The logit 1, however, has a gradient scaled by L. but which can mitigated by normalization in the Adam optimizer. Thus, can choose 7 freely. We set T 1 for all experiments of our paper.
B ADDITIONAL DISCUSSION
B. STABILITY OF THE MASKS
Multiple sources randomness could affect the final masks discovered by our method. These include sampling the mask each iteration, different data for each target task, and the order in which data is used for training. To verify that the masks discovered by our method are consistent we considered pairs of CIFARIO classes target tasks in combination with simple CNN without dropout (Appendix C.8 Pairs are chosen instead of the leave-one-out scheme used in Section 5o increase the sparsity of the masks as much as possible (potentially making them even more unstable). We trained 10 CNNs and analyzed 10 random pairs of classes for each of them. For each pair we trained two separate masks and calculate the Intersection over Union (IoU), resulting in a total of 100 data points.
We found that the mean IoU is 93.26 + 0.96%, which confirms the discovered masks' stability. Note that in case multiple redundant weight configurations are present in the network. different mask seeds will find a different subset of them, so their IoU will be less then 100% even in the optimally stable case. Using dropout would encourage such cases.
B.1.1 POTENTIAL ERRORS INTRODUCED BY THE STRAIGHT- ESTIMATOR
The traight-through estimator introduces approximations in the gradient calculation. Fortunately, the inaccuracies do not build up through multiple estimation steps, since the masking and straight-through estimator are applied directly to the network's parameters. Indeed, on each gradient path, there is at most a single Bruight-througe estimator present.
B.2 DOES MASKING CHANGE THE PERFORMED OPERATION?
The recent work of Zhou et al. 2019) demonstrated how it is possible to achieve non trivial perfor- mance by training binary masks on neural network with frozen weights that were randomly chosen. This raises the question whether the masking process in our method changes the performed operation after the weights are frozen and could thus cause misleading observations.
To investigate this possibility, we randomly selected some of the networks and datasets used through- out the paper, and trained as usual. After both the weights and the masks are learned, we performed the following experiment:
applied masks to roughly half of the networks weights, while leaving the remainder unmasked. In one variant of this experiment, early layers near to the input are masked, while the later layers, including the output, are not. In the other variant, the opposite is true. If the net- work demonstrates compatibility between the masked and non- mon-mesked layers for these experiments, then this is strong indication that has not altered the performed operation significantly.
The outcome of these experiments are shown on Fig. 7 where we report the performance drop for transformer SCAN dataset, FNN addition//ulliipccatin dataset, LSTM,, big (4 layers of 2000
16

---

Published as a conference paper at ICLR 2021
units) and small (4 layers of 800 units) FNNs on the double-addition experiments and the small CNN on CIFAR 10.
For almost all configurations, operations performed by the network remain mostly the same under the masking process. The only exception we found is the big FNN on the double-addition task, when the early layers masked. Note, however, that its performance is well above the chance level (P = 10-+), Since this network is severely overparametrized, we speculate how this might be the reason for this observed difference. For example, it could have learned to solve the problem by combining multiple alternative pathways, all of which contribute to the output. If the masking process removes some of those pathways from the layer near the input, but leaves them in the layers near the output, the pathways are cut in half. Thus, they might produce erroneous outputs. To further analyze this we therefore also trained a smaller version of the same network, which we observe behaves similarly to all other networks, suggesting that also in this case the masking does not alter the performed computation significantly.
observe only a low drop in performance, indicating that the
Finally, we note the variant where the early layers are masked appears considerably more difficult than other way around. This might be because some of the inputs of the unmasked later layers are removed by masking the early layers. Normally, if all layers are masked as well, such weights of later layers would be removed together with the ones in the early layers, thus not affecting the result.
100
Accuracy drop
Early masked
Late masked
SCAN trafo
+/s LSTM +/+ FNN Big
+/+ FNN Small
+/4 FNN
CIFAR 10 simple
Figure 7: Accuracy drop for masks applied to half of the weights. See Sec. B.2 for details.
There may be multiple reasons for these different findings compared to Zhou et al. 2019). First, we use well trained networks instead a randomly initialized ones. Untrained network are believed to contain many random subnetworks which can useful for performing any task. However, the fully trained network has its subnetworks tuned the task, likely decreasing the possibility of further subnetworks existing that implement radically different operations. Second, we are training the masks on a subset of the train set, which does not encourage changing the performed operations either: the highest performance can be achieved by selecting the correct subnetwork already performing the operation well. Finally, the experiments of Zhou et al. (2019) indicated that the performance of the best found network decreases with the task complexity, and performs best on MNIST. Some of the experiments considered here are significantly more complex.
B.3 CHOOSING TARGET FUNCTIONALITY
In principle any operation that the network is able perform can be used as a target functionality. This includes partitions of the dataset, even novel tasks if the network can generalize them. The resulting masks will highlight which weights are responsible for performing them. For our experiments, we always chose subset of the training set of the weights target functionality. This ensures that no generalization is required from the network to solve the problem, and that the subset of the original weights required solve this subproblem is highlighted. The discovered module then corresponds to functionality that the network should already have learned in the original training phase.
Interesting target functionality should be chosen such that removing the discovered set of weights or its inverse can be expected to lead measurable performance difference on some test set. This test set should ideally be subset of the original training set used for the weights. In this way, one does not have to directly consider the amount of sharing, and can measure (the difference in) accuracy, which we find more reliable and easy to interpret (see also Sec. 2 for further details) However, such a choice is not available and the amount of sharing has to be analyzed directly. then we recommend drawing conclusions only when the observed difference compared to some reference score is sufficiently high. For example in the Permuted MNIST experiments, the sharing of < 20% is significantly lower than the expected 100%.
17

---

Published as a conference paper at ICLR 2021
B.4 Is ATTENTION THE SOLUTION?
Could a form of attention (Bahdanau et al., 2015) solve the problem discussed in Section 3.4p At least the current use of attention does not seem promising, In theory, attention-based Transformers (Vaswani et al. (017) can reuse the same modules in parallel, but only if they are executed in the same layer. For a b + c* d the multiplier is reusable, but for a * b it is not, since the second multiplication requires the result of the first: that is, different layers are needed. In recurrent models, such as RNNs with attention (Bahdanau et al. 2013) and Universal Transformers (Dehghani et al., 2019), attention does not permit routing between functional modules but is just used to route data to the input of monolithic transformation block which processes the information. Emerging functional modules must in that case appear within the processing blocks. However, since attention is neither able to rewire the block's 's internal data flow, nor to permute elements of the attended vector, it does not help with the routing between modules emerging inside the block. Indeed, we showed empirically in Section that Transformers suffer from the same generalization issues as LSTM on the SCAN dataset, and they did not generalize on the more complex Mathematies Dataset either. Attention might help though, if all the input and output interfaces of functional blocks overlap, and a single processing step executes single function. However, as our experiments show, the separation between modules tends to be inadequate in the case of shared interfaces (Sec 3.1). Moreover, there is no control over executing single function per time step (c.g., the whole a b c block could be executed in a single step).
In order to help with data routing between emerging functional modules, attention has to be able to focus on arbitrary parts of the activation vectors. This would enable information exchange between such modules, but it is unclear how this could be implemented. For example, in the double addition experiment (Section 3.2), the task requires to process disjoint subsets of the input, which is not possible with the attention mechanism. In general, attention-based solutions would require to store one "concept" in a single vector so that they can separately attended to. However, what makes a good "concept" in this case is unclear, especially since different processing stages might require a different granularity- - for example, sorting tuples of numbers based on the first element of the tuple requires accessing individual elements of it but also treating the tuple whole. For broader discussion on dynamic information routing and the problem of variable binding in neural networks, we refer the reader Greff al. (2020).
B.5 EXPLICITLY MODULAR NETWORKS
At first glance, explicitly modular networks (Clune et al. 2013 Andreas et al. 2016 Kirsch et al. 2018)CCang et al. 2019: Bahdanau et 2019 could provide a solution for the discovered problems. In what follows we will call hardcoded modules "blocks" to distinguish from functional modules, which we call "modules". Routing networks, in addition to the problems described by (Rosenbamm et al. (2019) restricted to exchange information between blocks Because all the information has to be stored in this vector (such as different variables), either different parts of the vector should be responsible for different stored variables, or they have to be stored in superposition, e.g., by projecting them in a space where they are orthogonal to each other. Either way, this requires the blocks not only to perform a given operation, but also to be aware which variable they want to access. Thus, the blocks are not universal since operating on different variables encoded in the state require different modules. For example, in the double addition experiment (Section 3.2). the task requires to process disjoint subsets of the input. This is true in general: different subsets of the net- work state may require independent processing. Routing networks consist of simple modules stacked sequentially, which is obviously not good fit for this type of data. Alternatively, RIMs (Mittal et al.) 2020 Goyal et al., 2021b]ai a) attend to the data, making it possible to execute multiple modules in par- based attention. which also has its limitations (Sec. (B.4). These difficulties
single fixed-size activation vector.
allel. However, they let us believe that a general inductive bias towards function- based specialization in generic neural net- works would be preferable solution compared to explicit modality and motivates this paper's 'ss topic.
18

---

Published as a conference paper at ICLR 2021
C ADDITIONAL RESULTS AND EXPERIMENTAL DETAILS
C.I SANITY CHECKING THE MASK DISCOVERY PROCESS
Our method frequently discovers resistance against weight sharing. Perhaps this could raise the question whether our method is able to discover shared weights at all. We ran additional experiments to verify this.
We used the double-addition experiments from Sec. 3.2 before, but after the weight training phase, we copy the input and output weights one pair to the part of the weight matrix corresponding the other. This ensures that the hidden layers can not see any difference between the two pairs. We use the FNN variant since can be used without any modification, while the LSTM would require changes avoid state conflicts.
Specifically, we trained the network as
In Fig. 8 it can be seen how our method accurately discovers that the sharing is almost perfect in this case, which further justifies approach. Compare this to the identical setup of Fig. Note that the first and last layers are still not shared: they contain identical, but non-shared copies of the weights.
1.0
IoU loMin
Proportion
0.5
0.0
layer
layer
layer
layer 4
output
Figure 8: Double addition task with manually edited input/output weight matrices to reuse the hidden layers. Proportion of weights shared per operation in case of FFN.
C.2 COMMON HYPERPARAMETER CHOICES
Our method is implemented PyTorch (Paszke et al. 2019), and available at hhttps://githii
com Robe rt Csordas modu les Unless otherwise noted we the Adam optimizer (Kingma
& Ba 2015), batch size of 128, a learning rate of 10-3, and gradient clipping of 1. To improve the
quality of the masks divide batch into four parts that each
different mask sample. For
non-LSTM networks, we use the ReLU activation function (von der Malsburg
1973 Nair & Hinton 2010) for the activations intermediate layers. The Gumbel-sigmoid always has a temperature of t = 1 (the reason for this is explained in Appendix A.4). For most of our experiments, the regularization coefficient a specified as bo, where b the batch size used for training the masks. Otherwise we will mention separately. All figures in this paper, unless noted otherwise, show mean and standard deviation calculated over 10 runs with different seeds.
C.3 CHOOSING THE REGULARIZZTION HYPERPARAMETER
Choosing the regularization hyperparameter critical to obtain valid conclusions. Too low a might yield the false impression that no modules exist or that they share more weights than they really do. Too strong regularization may degrade performance on the target task, discarding essential weights.
Fortunately, there is simple and consistent heuristic for choosing a, which follows from training mask on the full task. We increase a as long the performance does not start to drop. Then, we reduce a slightly until the performance is adequate, e.g. > 95% of the original performance. This method is not very sensitive to the exact value a and transfers well across different network sizes (Fig. 10). We find that it is less critical but still important to tune the learning rate and the number of steps of the mask training process. We always check the chosen hyperparameters' validity by training mask the full, unmodified problem, where expect
only slight drop in performance.
Note that underfitting NNs tend to share more weights. Indeed, in our experiments we found that choosing a sufficiently large network size is essential to avoid false conclusions about the reason for sharing. Fig. [T] shows an example how the amount of weight sharing changes as a function of network capacity.
19

---

Published as a conference paper at ICLR 2021
+/+ FNI
+/s FNN
1000
LSTM
1/4 LSTM
-0-
6/6 FNN
1/+ LSTM
4/4
500
1.0
0.6
1.0
mask value
mask volue
(a) Linear scale, small values cut off
(h) Log scale, values
Figure 9: Histogram (normalized as 500-bin PDF) of expected values of the mask elements
(14, = (44) different tasks. (a) Shown linear scale. Values 0.0002 (bottom 10% of the first bin) are removed from the calculation because their number vastly exceeds the number of kept weights for most of the networks, making the details invisible. (b) All mask means, Pú, (without small values removed) shown on log-scale.
100
100
50
50
Accurecy
90
Accuracy
80
40
40
60
80
70
30
30
10
10
10
(1 Du
2 bi
(a) Big network
(b) Medium network
Figure 10: Sensitivity analysis for hyperparameter B ba (b is the batch size) addition/multi- plication experiments. Note the logarithmic æ-axis. The color indicates the total amount of sharing [%]. The red line and the star indicate the value chosen for our experiments. Each point is mean of 10 independent seeds. The network is not very sensitive to the exact choice of 3. (a) Big network. with 5 layers of size 2000. (b) Medium network, with 5 layers of size 800. It can be seen that the hyperparameter transfers well between network sizes.
20

---

Published as a conference paper at ICLR 2021
C.4 ADDITION/ULTIPPICATTON EXPERIMENTS
Since preliminary experiments indicated that modulo 100 multiplication require lots of weights, used reasonably large networks for this experiment. The FNN is 5 layers deep, each layer having 2000 units and the LSTM hidden state size of 256 (further increase resulted in overfitting.. network was trained for 20k steps on the full task before freezing. The following mask training phase takes an additional 20k steps for each mask. Mask training uses learning rate of 10-2 and 3 10 for regularization. For the LSTM we use 3 time steps where the input is repeated for every step. The dataset is generated by sampling numbers and operations uniformly at random.
Fig. 11 demonstrates that even if we use large network (the small, 3 layer networks of 400, 400, 200 can solve the task), the percentage shared weights still changes when increasing the network size.
In Sec. 3.1 we showed that even though there is certain level of natural separation between the modules responsible for addition and multiplication, there still a significant proportion of shared weights. To analyze the importance of those shared weights, we tested the network with inverted masks as in Sec. 3.2 Tab, shows the results. Given the high proportion of sharing, especially the input and output layers, the results expected: inverted masks do not perform well, showing that the separation of the modules is limited.
60
Total sharod [N]
40
20
small
medium
big
huge
Figure 11: Additionn/uultiplccaiioo task: the total proportion of shared weights for the "add" operation for different network sizes. "small" means 4 layer network with hidden sizes of 400, 400, 200, "medium" 5 layers/ hidden sizes of 800, "big" 5 layers/ 2000, "huge" layers / 4000.
Full
100 0.0
100 0.0
13 5.5
0.00
- 7.0 17
FNN
100 - 0.2
0.0
69 9.7
100 0.0
+
100 - 0.0
100 0.0
0.6
0.6
1 0.2
LSTM
100 0.1
3 1.2
6.0.88
100 - 0.0
2 1.2
Table 2: Accuracy of additiion/muliilicattin task addition and multiplication with FNN and LSTM. The header shows on what the applied mask was trained on. denotes an inverted mask
x100
No. of weights
No. of weights
layer
layer 2
layer 3
layer
output
Istm_0.ih 1stm_0.ht Istm_1_ih
Istm_1.hh
output
FNN
(b) LSTM
Figure 12: Addition/multiilicationn taks: number of weights per operation for each layer (a) feedforward network, (b) LSTM.
21

---

Published as a conference paper at ICLR 2021
100
100
100
True
0
1
0=
1
True
2 100 1
98
3
0
100 0
=
0
0
none
none
none
Predicted
Predicted
Predicted
(a) FNN: Mask trained on + and
(b) FNN: Mask trained on +
(c) FNN: Mask trained on
Thue
99
1 1
0
100
99
1
0
100
True
26
70
1
100
0
0
99
0
21 8
3=
75
0
100
0:0
0
none
none
none
Predicted
Predicted
Predicted
(d) LSTM: Mask trained on + and *
(e) LSTM: Mask trained on +
(1) LSTM: Mask trained on
Figure 13: Analysis of FNN (a, b. c) and LSTM (d, f) performance degradation on the addition/ multiplication task. The y-axis shows the target operation. The æ-axis shows the actual operation performed. "none" means the predicted number is neither the result of addition nor multiplication. The FNN ignores the operator specification and performs the one corresponding the mask; in contrast, the LSTM tends to perform invalid operations.
C.5 DOUBLE ADDITION EXPERIMENTS
The training protocol for double-addition experiments is identical the one in Appendix C.4 except that the FNN variant uses mask regularizer of 3 = 4 10 The LSTM uses 6 steps in total in this case (3 steps per operation).
the full input case, both tuples are presented for 6 steps and the output is read from the last step. In the case where shown for the first steps, resulting in an output at the 3rd step, the second tuple is presented for the next steps, resulting an output the 6th step.
tuple is presented at time, the first tuple is
C.5.1 ADDITIONAL INVERTED MASK EXPERIMENTS
Following the inverted-mask mask experiments done in Sec. 3.2 we investigated how well the separation holds if we consider only the hidden layers. This is achieved by using inverted masks for the hidden layers, while using the mask trained on the full task without inversion for the input and output layers. Hence, in this case the inputs and outputs contain all the connections needed for both tasks. Our findings are shown Tab. 3 and are consistent with Tab. I can be seen how the performance of the inverted mask tends to work well on the opposite task, while its performance is significantly lower on the original task (note that chance is at P = 0.01 for these experiments), suggesting that this effect is not due their inputs/outputs being disjoint.
Further we experimented with leaving the input and output layers unmasked, while inverting the discovered masks for the hidden layers. Surprisingly, in this case the inverted masks perform well on both tasks (around 90%), even on the task for which the mask was inverted. This suggests that the network contains ensemble of subnetworks individually capable of solving the problem with good performance. However based on the findings in Tab. 3 these subnetworks in the hidden layers appear to be mostly independent of each other: the performance nonzero on the original task only if both the original weights and the ones corresponding to the inverted masks are now included. It remains unclear what causes this particular behavior this setting, which we believe an interesting direction for future research.
C.6 TRANSFER LEARNING EXPERIMENTS
In the transfer learning setup, train 11 permutations of MNIST using the same network. Training the weights and masks together is more difficult than the usual setup. order to improve the quality of the mask gradients use 8 mask samples per batch instead of the standard 4. Each phase takes 30k steps. The learning rate is 10 10-2 The network is 4 layers deep, with hidden sizes of 10
800, 800, 64. We using mask loss of
Fig. 14 demonstrates the number of shared weights per layer for a network that has its masks initialized such that it prefers to reuse the old weights. The mask logits corresponding to weights
22

---

Published as a conference paper at ICLR 2021
Full
Pair
-Pair
Pair
-Pair
Pair
100 = 0.4
100 0.0
7 4.0
1 0.1
63 + 15.9
FNN
Pair 2
100 + 0.1
1 + 0.1
62 16.9
100 + 0.0
+ 5.0
Pair
100 = 0.0
100 0.0
16 16
1 0.1
99 = 1.3
LSTM
Pair 2
100 = 0.0
1 1 0.0
97 + 4.9
100 = 0.0
16 = 5.9
Pair
100 = 0.0
100 = 0.0
25 - 6.1
1 + 0.1
76 + 10.0
LSTM (forced)
Pair
100 + 0.1
0.1.
94 4.2
100 = 0.0
42 + 14.7
Table 3: Double-addition task: accuracy [%] of LSTMs and FNN on the pairs. In case of LSTM (forced) only one input is presented at a time (to prevent interference). The header shows on which pair the mask mask (for the full task) is applied to the input and output layers. For further details please refer to Sec. C
trained on. denotes inverted mask for the hidden layers, while the regular
of the previous task initialized 2 (corresponding P 0.88), the logits for newly initialized weights to either (P = 0.5, Fig. [4a) or -1 (P x 0.27, Fig. 14b). Compared to Fig. the sharing significantly increased.
T10
0.8
T4
T8
0.8
Proportion
Proportion
0.6
0.6
T2
T6
T10
0.2
0.2
T4
T8
Layer 2
Layer 3
Layer 4
Layer 2
Layer
Layer 4
(a) New weights with P 0.5.
(b) New weights with P Rs 0.27.
Figure 14: Proportion of weights shared per layer after every second task permuted MNIST, for a network with masks initialized to prefer reusing the old weights. Old weights are sampled with P s 0.88 probability. Each task corresponds permutation. Decreasing the probability of new weights forces increased sharing.
C.7 EXPERIMENTS ON ALGORITHMIC TASKS
C.7.1 SCAN EXPERIMENTS
In preliminary experiments we observed that the full-size word embeddings used for the baseline in Lake Baroni (018 yielded many possible redundant input-to-hidden weight configurations that have a greatly reduced probability of being sampled. This caused the input-to-hidden layer to be removed by the thresholding procedure. Therefore, we appropriately reduced the size of word embeddings to 16 (note that SCAN has only 13 input and 6 output tokens and they are not shared). When using the reduced embedding we do not suffer from the aforementioned problems. Teacher forcing was used for each batch with 50% probability.
The Transformer network is based on PyTorch's 's internal implementation, with modifications needed to apply multiple masks more effectively. We dmodel 100, inner-layer dimensionality of d 200, h = 4 heads, and 3 layers both the encoder and decoder. The network always trained with teacher forcing. An end token is applied the end of each input sequence, and decoding starts with start token. The sinusoidal positional embedding (Vaswani et al. (2077) is applied to the inputs of the transformer in both encoder and decoder.
The training procedure uses batch size of 256 and a gradient clipping 5. The mask learning rate 3 10-5 for the L.TTM experiments and / 10-3 for the Transformers.
is 10-2 and
23

---

Published as a conference paper at ICLR 2021
Test accuracy [%)
100
Test accuracy [%)
100
50
50
0
Turn Left
Jump
Length
Turn Left
Jump
Length
(a) LSTM
(b) Transformer
Figure 15: The networks' performance when it is directly trained and tested on the splits indicated on the z-axis, without masking. This is the standard setup from Lake & Baroni (2018). Performance when the network is trained on IID split, then masks are trained on train split indicated on the z-axis. can be seen that although training on the IID set helps compared to the basic setup, the network still needs task-specific weights, which hurt performance when removed by the masks.
We train the networks for 25k steps without masks before freezing and for another 25k steps for each mask training phase.
We train the network weights on the IID dataset (the "simple" split), and only the masks on the rest of the data splits. Figure [3] [3] shows that this process marginally improves the performance of all splits, compared to when the network is directly trained on the corresponding train split. However, the performance degrades significantly when removing weights that are unnecessary for the given training split. This allows Notice that the remaining weights still have significantly better performance than training the network solely the training set of the given split without masking.
conclude that the learned solution requires tasks-specific weights.
The word embeddings are excluded from the masking process and remain unmodified after initial training keep the learned word representations unchanged.
C.7.2 EXPERIMENTS ON THE MATHEMATICS DATASET
The Mathematics Dataset (Saxton et al. 2019) is a dataset intended to test the mathematical reasoning skills of NNs. It consists of 56 tasks from different areas mathematics, on high school-level. Each task consists of train set divided into 3 levels of difficulty (easy, medium, and hard) and an IID test set. All questions and answers are provided only a human-readable text format (see examples in Fig. 16
We train the network on individual tasks, in contrast to the method of Saxton et al. 2019), where all tasks are trained together. The main reason for this is interference between the tasks. We chose five different tasks to analyze, based on their difficulty: the chosen tasks should have good performance without masking but should be nontrivial. Thus choose "arithmetic: add_or_st toooo_suu or_sub",, "algebra: linear_ ld", "calculus: differentiate". "comparison: sort" and "polynomials: collect". Note that the performance of network might be less than reported in Saxton et al. (2019) since no transfer between tasks is possible, and we train for significantly fewer iterations because of limited computational resources.
computation and to prevent possible
We split the official easy, medium, and hard train sets to obtain new train and validation sets for each difficulty level. We randomly choose 10k samples for the new validation set; the rest is used as the new train set. We filter for repetitions, making sure that no sample appears twice. This way, we get train and validation set for each difficulty level. We ignore the official test sets because of the missing distinction in difficulty. This treatment is needed because all difficulty levels but also the masks only on the easy split. Additionally, we want to evaluate its performance the hard difficulty. In this way, are able to determine whether specific weights needed exclusively for performing the hard split. Note that the same rules govern the samples in all sets.
want to be able to train the network on
24

---

Published as a conference paper at ICLR 2021
What is the difference between 1801791.2422 and -0.7? 1801791.9422
Solve -719+0 + 3179#0 + 135275 = -2280 777#0 for o.
-35
What is the derivative of 30595=je*4 + 254xj+=3 + 1559873 wrt j? 122380+j** + 762+j*+2
Sort -3/5, 1355.6, 703, 2, 223 in ascending order.
1335566, -2/3, -3/5, 2. 703
Collect the terms in -66wV 67 294v 12+v - 155. 15xv 225
Figure 16: Examples from Mathematics Dataset. One sample for every task we use.
First, we train the network on all difficulty levels (easy, medium, and hard). Then we freeze its weights. Next, we train masks the easy split and test the hard split. If this results in performance drop, then this indicates that the network requires a separate set of weights for different difficulty levels, which is undesirable. Nonetheless. we observe precisely this behavior (Fig. 5 which confirms once more that NNs tend to violate Preuse-
Interpreting the size of the drop is nontrivial due how the easy, medium, and hard splits differ. The more difficult splits may include some samples from the easier splits, but never the other way around. This means that the hard test set's performance will be nonzero even if none of the hard samples are solved correctly. This behavior is inherent to the original dataset and can not be changed without regenerating
We use the Transformer /Vaswani et al. 2017) model from Saxton et al. (2019) It has dmmode = 256, inner-layer dimensionality d = 512,
heads. Both the encoder and decoder have layers. The word embeddings of the encoder and decoder shared, and the output layer is tied to the word embedding. The network always trained with teacher forcing. We use the Adam optimizer 10-9 31 0.9, 32 0.995 and gradient clipping of 1. We use
with learning rate of 10-4 masks samples for each batch. We found that some tasks require linear learning rate warmup for 5k iterations the beginning of network training in order converge. No warmup is used for training the masks. Individual tasks use different hyperparameters, listed in Table Batch sizes are chosen so that the experiments fit on single GPU with 16Gb of VRAM (2 GPUs for "Poly. collect").
Hyperparameter
Add or sub
Linear 1D
Differentiate
Sort
Poly. Collect
Batch size (net)
256
512
128
256
128
Batch size (mask)
256
400
128
256
256
Mask regularizer (3)
10-5
10-6
10-3
3 10-6
10-6
Training iters (net)
30k
200k
40k
30k
200k
Training iters (masks)
30k
50k
40k
30k
50k
Learning rate (masks)
0.03
0.02
0.03
0.03
0.02
Warmup steps
5k
5k
Table 4: Hyperparameters for different tasks on the Mathematics Dataset
25

---

Published as a conference paper at ICLR 2021
C.8 CNN EXPERIMENTS ON CIFAR10
C.8.1 SIMPLE CNN
We use a learning rate of 10 and 8 10-4. We train the network for 20k steps before freezing its weights and then use an additional 20k steps for training each of the masks, including the reference mask. See Table 3 for details regarding the architecture.
Index
Operation
Inputs
Outputs
Kernel
Padding
Activation
Dropout
Conv
3
32
3x3
ReLU
2
Max pooling
32
32
2x2
Conv
32
64
3x3
ReLU
4
Max pooling
64
64
2x2
0
Conv
64
128
3x3
ReLU
0.25
Max pooling
128
128
2x2
0
Conv
128
256
3x3
ReLU 0.5
8
Spatial average
256
256
Feedforward
256
10
Softmax
Table 5: Architecture of the simple CNN used for CIFAR 10 experiments
Fig. 18 shows the confusion matrix difference for classes of CIFAR 10. The most surprising observation is that the decrease in performance for each of the classes is substantial, ranging from 40 to 60%. This shows the heavy reliance on class-exclusive features.
Analyzing confusion matrix differences yields interesting insights. "Airplane" is confused with"bird' and "ship", which likely due to having a similar blue background. Classes "cat" and "dog" tend to be confused with each other-removing exclusive feature detectors for one improves the performance of the other. "Truck" and "car" are highly related, likely due to the similarities terms of shape, such having tires, and similar backgrounds, such the road.
C.8.2 SIMPLE CNN WITHOUT DROPOUT
The CNN architecture used for experiments Section Jsees dropout, as shown Table 5 natural question to ask is how this affects the modularity of the resulting network. Figure 17 indicates that, as expected, removing dropout results in few percent of performance loss. When comparing Figures 18 and 19 it can be seen that adding dropout causes higher degradation in the class performance when the class-exclusive feature detectors are removed (roughly 30%-40% higher drop per class). This indicates that network with dropout depends more on class-specific modules, which is in line with findings presented Filan et al. 2020).
C.8.3 RESNET-111 110
To demonstrate that these behaviors apply to more complex models, we train a ResNet-110 (He et al. 2016) model which achieves competitive 93% validation accuracy following /attps://gittee
www.lasssiiiatiioo ion The network is built from non-bottleneck blocks ("BasicBlocks" Fig. 5, left Vaswani al. (2017). ). It is trained with SGD using weight decay of 10-4, batch size of 128 and a starting learning rate of 0. The learning rate is divided by 10 at iterations 32 000 and 48 000 (corresponding roughly to epoch 81 and 122). The network trained for 64000 iterations (164 epochs). Data augmentation of random horizontal flipping and random crop (with padding and output size of 32x32) is used. Masks are trained with Adam, batch size of 256, learning of 0.03, during the initial stage of training the weights, but the usual clipping to norm of 1.0 is applied when training the masks.
hton/bearrpa//yytoorr.
10 10-5 for 30 000 iterations each. Gradient clipping is not applied
As Figures and show, the performance drop per class is even more dramatic than in the simple CNN case, reaching almost 100%.
26

---

Published as a conference paper at ICLR 2021
86
1-00
8:41
1-0
1:10
0-10
1:0
1:5
6:1
2-0
0+00
1:00
The BMA
0
0:00
0:00
5:1
5:41
0
4
17:3
3
12
521
312
31
1:0
140
0-00
1-0
Rog
0+00
8+
1+0
0+0
1-0
0+0
2
3
3+1
3+1
0-0
0
1-0
142
010
Simple CNN
4+1
5+2
2=0
0+0
120
0:0
0:0
1:0
010
2:2
6:
1:10
5:1
1:1
-01
0:10
5:2
1
2
261
012
140
1.0
Thee
0.10
4:1
14:2
411
110
hons
0+0
2+1
0+0
1-1
2+
1
1
0
D-0
0-0
0+0
11
Piad
(b) Simple CNN without dropout
0+0
0+0
0+0
0:0
00:2
2
2:1
2:0
1:0
0:0
0:00
P41
1+0
Thun Dbe
0:10
2:1
0.0
1
0:10
2
7
140
0
140
0
0.0
0.0
24
2
010
0.0
0.00
81p
1+0
0
0
0
0-
0-0
0+0
0
Pool ene libel
(c)
ResNet-110
Figure 17: Confusion matrix on CIFARIO with masks trained on all classes. It can be seen that performance without dropout is few percent lower, as expected. ResNet- 110 has significantly better performance in all classes.
Inspecting the confusion matrix differences of different architectures as seen in Figures 18 highlight their similarity. This suggests that the interdependence between classes previously observed is mostly data driven an independent of the actual network architecture.
19 and 20
27

---

Published as a conference paper at ICLR 2021
60
7+9
3+1
6-0
146
31
0.10
14.
0.22
140
011
210
0.0
01
0_00
24
0-2
241
0-2
012
0.0
der
2.00
0:10
01.1
011
211
0.0
511
0.1
210
010
311
0-0
213
0-1
+112
6:0
015
010
011
0.0
1.0
0.43
0.00
-14
860
1x1
0x1
260
0.0
999
244
0-0
261
0-2
340
6-1
6.5
0-00
0.4
0-0
0.1
01:1
010
040
0.0
110
0.0
0.00
241
0-1
310
040
0..0
0-0
8-0
0-0
3+0
0-1
19
60
Predited loa
Prodcted
(a) airplane
(b) automobile
0.3
1
0.10
DAC
0.00
0.00
261
0.0
10
213319
111
0..0
241
-1.0
200
D.0
011
021
0.00
0.41
0..0
0.00
R1D
0.00
21.0
0.0
2.0
0.1
211
0-11
210
0-00
210
0.0
01.0
01.0
011
0-1
20
16+3
0.00
+1
12-2
10.4
11-2
161
0-0
2+11
0-0
162
--11
3+1
2-
1-4
0-00
0+3
0-0
The Thee
0-:1
0.40
12
142
040
21.1
240
0.0
0-0
411
34
111
32
1-1
0-0
20
d0g
-0
10g
0:.0
0:1
-1:1
0:0
510
10g
215
210
012
0.0
0-0
0+00
-2+0
0+8
6+1
1-1
846
1-1
8+0
0+0
100
0-0
3+8
3-1
5+0
2-1
6-8
6-0
0-11
0.1
-120
0.0
010
0-0
010
0.0
211
0.0
211
0_00
210
-1_0
010
0.0
010
010
01.
0.0
021
DLD
0-1
210
0100
Reddel kodl
(c) bird
(d) cat
0-11
0.4
0m1
0.4
-100
0-0
0-0
0-:0
361
0-0
0-0
0-0
20
The D00
der-
01.1
4:1
0:1
0.1
1923
11.1
10:3
113
-2-1
4733
F:2
612
1914
120
510
020
0.0
20
08
01
215
0:0
0_00
231
111
1113
121
311
-311
12
011
B1:
012
011
0:0
9
0.0
0.1
0.40
611
1.0
0.1
-2.0
-11
142
516
0.0
121
510
240
010
0.0
40
59
215
0.0
0_0
211
243
4113
1-1
210
241
-281
22
111
7-2
012
0.0
0.0
80
0+4
0-0
160
0-0
0-0
260
2-1
160
---
0-0
back
80
-00
Predide kool
Prodiced
(c) deer
(I) dog
01:1
0.2
12.
0.0
DAC
0.20
211
0.0
3100194
0.00
211
0-0
210
0.0
911
0.0
0.0
0.1
010
010
D10
0.0
-110
0.00
210
0.1
311
0.1
218
0-0
110
0.0
011
0100
911
0.0
0
0.4
4+2
041
0-1
-541
0-1
260
0.0
04
24
0-0
161
0-1
360
0-0
-1
0+2
0.0
0.1
1-1
212
-111
0.0
510
0.0
0.0
211
2-2
110
-3.1
0.0
0+3
1-1
3+1
0-0
20
0-0
3+1
0-1
5-9
10
002
10g
1gg
99
311
80
0-0
0+43
6-6
044
B41
6-1
540
0-1
560
0-0
0-0
361
6-1
79-7
03+3
0+3
30
011
0.1
010
010
D10
0.0
510
0.0
111
0.0
311
0.0
218
0-0
218
0:0
011
010
011
010
0:00
021
8:0
0:1
010
0.1
0-1
0:0
240
0:0
21.
0-00
0100
(g) frog
(h) horse
3:1
031
650
010
BEG
0:0
516
0:0
411
030
0:00
211
0:0
350
D:0
0:5
0:0
011
8-6
20
0-0
3-0
0:2
020
1:2
211
1:2
315
dear
0-00
6-6
0+1
6+1
6-3
8+6
6-3
110
0-0
0-0
361
0-1
3+2
6-0
8-1
6-1
6+1
6-0
40g
0.0
0.1
01.0
011
010
0-2
510
0.1
210
0.0
215
0_00
211
0-1
318
D-1
015
011
011
-1.0
0.00
011
011
010
010
0.0
210
0.0
0.0
210
0.0
310
0:0
010
011
0.0
0.4
0.0
0.0
0-1
240
0.0
0-00
240
0-1
111
0-0
Pedided hool
Prodiced and
(i) ship
(i) truck
Figure 18: Simple CNN: The change in confusion matrix for all CIFAR1O classes, when class indicated by the caption, is removed.
28

---

Published as a conference paper at ICLR 2021
1423
211
120
120
141
12-3
52T
10
313394
215
0-0
225
010
222
022
388703ia
210
1:1
0100
010
610
0.3
510
0.3
1:0
0.1
31.1
110
1-0
310
6:0
111
010
51
17.4
693
0.40
4+2
0.0
0-1
540
0-3
1.0
0.0
-1
14:
0-0
340
0-0
5+5
0-0
6-0
TheeWe
0.40
141
0-1
110
0.0
240
1-1
240
012
0.0
0
1-1
0-0
3+1
0-1
3+1
002
39
103
151
10g
310
0-0
6-0
0+8
8+0
6-8
8+8
6-1
8+0
0-0
100
0-0
3+8
0-0
3+0
6-0
8+3
0+3
6-0
411
011
010
010
010
0.0
810
0.0
412
0.0
311
-2-1
210
0-0
110
0-0
010
010
111
0.0
0100
0.0
01.0
510
0.19
1
-7
740
0-0
310
0-00
Prediced hool
Prodited
(a) airplane
(b) automobile
2=1
0:40 0.49
DW0
0.0
0.0
0.0
0-:
2
300
0-0
DN
0:10
5:0
10
411
121
dear
0-0
0+00
1+11
141
2+1
1-1
8+0
6-1
5+0
0+0
D80
2+5
6-0
548
2+2
2-1
6+1
6-1
0+3
6-0
-10
0.00
0.10
-42.
141
010
2-2
510
0.1
210
0.0
-2
59g
215
0_00
140
-1912
111
1212
111
110
012
0.0
0.1
-4:1
010
010
0.0
510
010
0.0
111
-3.2
311
221
011
0.0
0.40
-141
140
140
0.0
0-00
240
-3-1
0.0
hedied kod
Prodcced
(c) bird
(d) cat
0111
013
010
0.00
0.0
210
010
10
315319
0.00
211
0_0
210
012
0:0
0.1
0:0
011
01.
0:0
010
0:30
8:0
0:1
A.trmbde
211
0111
310
0:0
310
0:00
01.
0100
011
0:0
0.0
0.1
11
1.0
1E21
0.1
2100
141
2.0
0.0
04
21.1
0
24
1-1
111
-5_1
0.5
14.1
011
0.0
0.0
0.1
11#3
121
312
742
66
1:1
510
010
0-0
10
0
0-0
111
26
111
1=1
11
1+2
1112
011
0.0
0-0
20
10
1ng
00g
80
0-0
0+0
8+0
140
4-11
0-3
3-1
860
0+0
MM
0-0
360
1-1
3+0
4-1
0+2
0+2
0-0
0:00
0:1
B2D
010
D1.
0:33
510
0.2
0.00
215
0:0
210
0:0
310
0:0
011
020
011
0:0
0-00
0+
6-0
0-0
0-00
0-6
84.
0-0
0-1
0-1
1e0
0-0
3
0-00
0-5
6-0
0-3
6-1
Predited
(c) deer
(I) dog
0-:1
0.4
0
010
D10
-110
0.0
111
0.0
10
01.1
0.0
240
0.0
210
020
-1.0
012
0.0
18
0-0
0+0
1+0
0-0
0-0
3+0
0-0
1+0
0-0
0
6:11
310
0-0
6-1
141
1+1
6-3
3+1
6-0
3+0
0-0
0-0
161
1-0
3+2
1-0
6+3
5-1
0+2
6-0
0.0
0.1
010
1:2
010
1:1
-211
0.0
210
010
515
0_0
218
1-1
110
512
01.
-521
012
0.0
1:0
1:81
10:33
12:4
611
4:22
34:4
1:0
151
1:0
20
510
0:0
258
0:1
310
0:0
-1:0
011
0:0
010
610
210
0.0
0-0
211
4-1
11-2
32.02
2-1
Pedided kow
Pholibel
(g) frog
(h) horse
abpane-
3:2
011
01.
3120210
310
320
0-0
1+
6-0
0+8
6+6
0-3
8+0
0-3
2+1
0-0
4-2
3+0
0-0
3+5
6-0
6-5
6-00
0.0
0.1
11.
0.1
010
0.0
510
0.0
-11.
0.0
211
0.00
111
0-0
210
0-0
0.5
01.1
011
0.0
0:00
0.1
010
1:11
010
51.1
0.10
-111
0.0
0.00
210
0-1
310
01.
011
-22
0.49
0-:1
0.0
0-0
0
0-00
241
0-0
111
0-0
10
530
10
kog
1gg
0
0
8a0
040
D&C
041
0
1099
0-0
368
0-0
245
0-0
042
18
6+2
1-1
2+1
1+0
0-3
1+0
0-0
1-0
3+0
0-0
3+6
6-0
6-5
6-0
044
0-0
246
0-0
23+7
1e0
1-1
2
0-00
3+3
31-
(i) ship
(i) truck
Figure 19: Simple CNN without dropout: The change in confusion matrix for all CIFARIO classes,
when class indicated by the caption, is removed. The network has the same architecture as Table 5 but without the dropout layers. The performance drop is reduced by roughly 30%-40% compared to the same architecture with dropout (Fig. [8).
29

---

Published as a conference paper at ICLR 2021
412
231
21
121
211
0-0
510
60
0.23
141
-21
011
210
0.0
01
0-00
241
-2.2
241
1-0
0-
0-1
0+1
142
0-1
1+0
0+0
D00
0-0
361
6-0
143
6-1
8+3
0+3
0-0
1
0.43
0-11
-142
060
143
540
0-1
260
0.0
999
244
0-0
240
-122
361
0-2
061
0.4
0-0
0-0
0.43
1-1
041
610
0-1
312
0-0
5+0
0-0
0-0
341
0-1
340
6-0
8-1
6-0
6+3
0-0
0.0
0-0
2+0
0-0
3+0
0-1
0-0
80
Pidddeed Moo
Prodeced
(a) airplane
(b) automobile
Aapase
3-2
0.4
0.00
D&C
0.00
0.00
241
213311
0.:0
241
E-0
0++
6-0
0-0
0+00
0-0
1-0
0-1
0-1
3+0
0-0
3-0
0-0
0-3
6-1
25
0.+0
20
10-3
9-2
16-4
2-1
161
1+0
2+
0-0
165
2-1
1-0
1=1
0-11
0+4
0-0
+20
The Tre
0:0
0.19
-1:1
139
123
B11
021
230
030
1:0
113
112
5253
10:2
21-
2-1
10
0.00
0-0
40
-20
10g
0:1
-2:1
518
0:0
10g
211
-2:1
210
0
0.4
-140
041
1411
0.2
240
0.0
100
0-0
245
-1a1
261
141
0.0
682-
0-11
0.1
0LD
010
B10
0-0
510
0.0
141
-1.0
64
11-1
0_00
210
0-0
110
0-0
01.
D10
111
-111
E-0
0+0
0+0
0-0
040
0-0
160
1-1
2+c
1-1
240
0-0
0-0
0-6
0-00
0+1
1.9
kod
(c) bird
(d) cat
0:2
014
611
010
010
510
0.0
111
010
20
0.00
0.0
0.0
011
01
011
110
010
01
0
21.
211
012
2-1
0+0
23
17
20
24
R14
160
0-0
0-0
26
1=1
142
2-1
0+2
0-0
0.0
0.0
611
-142
110
142
510
0.0
240
0.0
599
141
0_0
263
4.2
211
7-2
012
0.0
96
0-0
0+0
0+4
0+0
8-1
0+1
-161
5+0
0-8
8+1
0-0
160
2+0
0+0
0-0
60
5+5
0-0
0-0
260
361
121
1=1
140
3+8
060
2-1
141
6-0
8+1
0-0
6-0
40
back
Prediadd bos
Peedcdd shot
(c) deer
(I) dog
sapass
0111
013
011
010
B10
0.0
010
0.0
210
02
312314
212
0.00
211
0_0
210
0.0
010
01.0
011
-1.1
0-0
0+
0-0
0+0
040
0-0
540
0-0
160
0-1
2+4
0-1
160
0-0
360
0-0
5+
0-0
0+1
111
0.4
142
-210
0.0
110
0.0
0.0
142
0-1
140
-1.0
0.0
0+4
8-1
5+0
0-0
0-4
1+0
1-0
10g
7-1
0.4
1:+0
0-11
847
D41
7-3
9641
0.0
260
362
6+4
0.0
02
39
0-0
0-0
361
541
6-1
143
345
141
6-0
-2-3
114
-1.0
6-0
0+1
012
0-0
6-0
40
0-0
0.4
640
041
B41
0-1
240
0-1
260
0+0
40
5+5
1-1
1+3
6-2
96-7
1:48
5-5
41
0+1
9-1
-0
0:11
011
010
010
010
0.0
111
0.1
0:0
218
0:0
020
021
Peddieed Roe
Poddied Wbo
(g) frog
(h) horse
80
315331
0-0
1+
0-0
0+6
0+0
0-3
5-0
2-1
3+0
0-0
3+0
0-0
0-
50
0::1
0.3
122
-121
010
0.11
510
0.0
210
0.0
D4
111
0.0
-112
0:1
210
D21
011
01.0
011
0.0
The Tee-000
0-00
0.00
1-1
224
151
0-2
511
0-1
2-0
0-0
515
0-0
361
0-3
341
1-2
8-1
6-1
011
110
0.1
011
-117
0.1
210
0.0
0-0
211
0-1
313
012
0.0
70
22
10g
1gg
012
010
2.5
0.0
100
0-00
215
0-1
211
0.0
682
15.4
211
111
110
0-0
311
1.00
26
12
111
0_00
218
0_0
110
0-0
011
D10
111
-1.0
Bu.
040
0-0
240
0.0
15
248
2-1
345
0=
0
140
42
8
(i) ship
truck
Figure 20: ResNet- 10: The change in confusion matrix for all CIFAR 10 classes, when class indicated by the caption, is removed.
30


Reviews:
[{"review_id": "hNhH2WnPXLA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies weight modularity in neural networks (NNs). In particular, given a NN trained to perform a task, a subset of weights are identified which in isolation perform well on a subtask of the original task. Such subsets are inspected to understand the extent to which they are specialized or reused across different subtasks of the original task. To identify subtask specific weights, a mask is learned that minimizes loss over a subtask when applied to the original NN's frozen weights. This process is carried out using gradient based optimization techniques (Adam). Extensive experiments are performed across various datasets and architectures. The paper concludes that while NNs seem to exhibit module specialization, they fail to exhibit reuse.\n\n### Strengths\n\n1. Understanding modularity within NNs and its relation to failures of systematic generalization is an important research direction.\n2. The precise masking method proposed here is novel to the best of my knowledge.\n3. The functional view of modularity has clear advantages over clustering based approaches.\n4. I believe this work could be extended in interesting ways. For example, it seems like the methods presented here could naturally be extended to encourage modularity, rather than measure it.\n\n### Weaknesses\n\n1. The paper addresses a well known issue, that NNs often fail to generalize systematically. Why this occurs and potential solutions are left unaddressed. While the paper does put forward a hypothesis regarding why, that the difficulty of learning routing operation in NNs may lead current approaches to fail to discover solutions which generalize systematically, little evidence is provided to support this conclusion. As such, I believe claims such as, \"our approach can also be applied in these settings to provide additional insight regarding the underlying reason for the observed failure at systematic generalization\" should be relaxed, or clarified if there is evidence for this hypothesis presented that I am failing to connect.\n\n2. How do we know the problem is with the full model, rather than the mask? Recent literature (cited in the paper) demonstrates that performant models can be obtained even when severe weight constraints are imposed [1, 2]. Indeed, [2] demonstrates masking alone is sufficient to adapt a NN to a completely new task. Given these observations, I'm concerned about the validity of assuming the mask optimization procedure identifies an independent functional module in the context of the computation performed by the full model. It seems plausible that the mask performs well on the subtask given the fixed model weights, but is unrelated to the function of the full model. If this was the case, I believe a number of the conclusions in the paper would not be justified.\n\n  The paper does provide some evidence this may not be occurring. For example, the double-addition experiment (Figure 2) shows that inverted masks perform well on the other pair. This suggests the mask and inverse mask may correspond to distinct functionality in the full model. However, Figure 10 tells a different story, where inverse masks do not perform well on the other task. Is this due to the shared IO in the addition/multiplication task? Is there any relation between the addition mask with shared weights removed and the inverted multiplication mask with shared weights removed?\n\n   Ideally, the paper would provide more evidence to support this assumption. Are units correlated in the masked and unmasked model? Could the analysis here be supported by explicitly constructing a model that exhibits reuse, and verifying it can be identified?\n\n3. The paper argues in several places that a weight level analysis is necessary for reasoning about functional modularity, e.g., \"without considering the contribution of individual weights it is not possible to reason about *functional* modularity.\" I believe this statement conflates two separate issues; (1) if modules should be defined in terms of units or weights, and (2) whether modules should be defined by functionality (as done in this work) or using techniques such as clustering. On the latter point, I agree with the statements in the paper and believe the functional approach is more closely aligned with intuitive notions of what is meant by modularity. However, I don't believe a weight vs unit level perspective is necessarily mutually exclusive. For example, consider a simple affine layer $h = f(Wx)$. Defining a module as a subset of elements of $h$ would be equivalent to defining a module as a subset of rows in $W$. \n\n4. Figures 2, 10, 14 would be much clearer if presented in a tabular format. I had to write out the results this way myself to aid understanding.\n\n5. I found the use of a non-symmetric sharing metric extremely confusing. I spent a while pondering over Figure 1 trying to figure out what the weights were shared with since both tasks are represented in the figure. I believe this issue could be easily remedied, for example, by using something like IoU (Jaccard index). The paper uses this metric in Appendix B.1 to determine the amount of sharing between different masks trained on the same network and task.\n\n6. Permuted MNIST Experiment\n\n   1. The paper states \"it suffices to re-train a new first layer to undo the permutation so that later layers can be reused.\" This makes sense. However, given that the procedure is \"freeze the occupied weights,\" lower level weights cannot perform this simple permutation by construction since they are frozen (I've assumed that occupied means a non-zero mask value).\n   2. Why was the choice made to \"train masks and weights simultaneously\" in this setting? This introduces an additional source of variation from previous experiments, and I'm unsure what the benefit is.\n\n7. SCAN experiments\n\n   1. The paper states \"if the masking process removes any important weights, then the solution is pattern-recognition like instead of being based on reusable rules, confirming explanation.\" This is related to point (2) above, and I'm not convinced this is the only possible conclusion. While the mask has identified a subnetwork whose solution is pattern-recognition like, I'm not sure that the logical jump to conclude the model as a whole performs identically is sound.\n   2. Regarding the output weight analysis, I'm also unsure why it should be the case that the final layer is sufficiently powerful for unbinding bound variables. Isn't it also possible that the mask learns to ignore those weights because it is not important for the task it is trained on?\n\n### Recommendation\n\nI recommend acceptance. Although I have several concerns with the paper, I found the ideas and analysis presented very interesting. I believe the community would benefit from further discussion, scrutiny, and exploration of the ideas presented.\n\n### Post Author Response Period Update\n\nMost of my concerns have been addressed by the additional experiments and updated language in the latest revision. I believe the techniques and analysis presented here for assessing reuse could be an important step between observations and explanations for the failure of NNs to generalize systematically. I have raised my score accordingly.\n\n### Minor Issues\n\n1. Appendix C.2: \"We always check the **choosen** hyperparameters\" => \"We always check the **chosen** hyperparameters\"\n\n### References\n\n1. Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 67–82, 2018.\n2. Adam Gaier and David Ha. Weight agnostic neural networks. In Advances in Neural Information Processing Systems, pp. 5364–5378, 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eEX1zHPIUx5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an empirical study of whether modularity can emerge within neural networks. It starts by proposing a novel definition of modularity that identifies modules by their functionality. To discover the module that implements a specific target functionality, the paper proposes to first pretrain the full network on the original task, then freeze the pretrained weights, and train a binary mask for each weight using Gumbel-Sigmoid. The training objective for the masks is given by the target functionality (e.g., a subtask of the original task), plus some sparsity regularization. The paper then investigates the discovered modules in terms of specialization, reusability, and compositionality. The main findings are: (1) Neural nets tend to satisfy specialization but not reusability; (2) Weight sharing between modules tends to be affected more by whether I/O are shared than by task similarity, and there tends to be less sharing in larger networks; (3) When trained on algorithmic tasks, neural nets fail to learn compositional rules, and thus generalize poorly; (4) CNNs trained for image classification contain class-specific, non-shared weights in the feature detectors.\n\nPros\n- The experiments are comprehensive, covering many neural net architectures and datasets. The results seem consistent across these architectures and datasets. Also, source code is provided, and the supplementary material provides sufficient details to reproduce the results.\n\n- The paper investigates natural emergence of functional modules, which is a novel perspective on modularity.\n\nCons\n- Some concepts are not rigorously defined. For example, in P_{specialize} and P_{reuse}, when should two modules be considered the same? The paper seems to define a module as a subset of weights. Does this imply that two different subsets (potentially differ by only one element) will correspond to two different modules? If this is the case, then P_{reuse} is extremely hard to achieve, and it can never be true if the input/output neurons of two modules are different (i.e., separate I/O considered in Section 3.2). Hence, the experiments in Section 3.2 seem meaningless. One may argue that the weights in the input/output layer should be excluded from the module. But then the functionality of the module is not well defined. Another confusion I had is: What can be a target functionality and what cannot? Take the addition/multiplication task as an example. The task is basically to learn the function f(a,b,s), where a and b are two numbers, and s is a switch indicating whether addition or multiplication should be performed for a and b. The paper suggests f(a,b,s=+) and f(a,b,s=*) as two target functionalities, which are the original function restricted to two subsets of the input space. Is it reasonable to consider f(a=99,b,s) a target functionality? Would the result be different? Also, all other experiments in the paper seem to construct target functionalities by similarly restricting the original functions. Are there other ways to define target functionalities?\n\nI am somewhat skeptical about investigating properties like compositionality and generalization after network pruning. It is well known that training performance does not reflect these properties. So even if training performance only slightly drops after network pruning, the pruned network may lose some properties that the full network has.\n\nMinor Comments\n- It is mentioned that separate I/O biases the network at initialization time to not share weights, but to me it seems to only bias the input/output layers rather than the hidden layers. It would be interesting to see if shared I/O will also lead to unshared hidden weights.\n\n- Fig. 13 did not mention which LSTM variant was used.\n\n- In Section 3.3, why is increased sharing in the first layer undesirable? It might not be possible to undo the permutation with only one layer.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YY7qxIzAMsi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn its current form, I feel that the paper should be disqualified because it contains some results essential to its claims in the appendix (referenced in the second paragraph, page 5). However, this can be easily addressed - thus my full review below:\n\nSummary of the paper:\nThe paper aims to analyze if and how neural networks learn modular representations. Modularity under the paper's definition means that the network learns representations that (1) specialize (using different modules for different functions) and that (2) compose in a re-usable fashion - i.e., that functions are used in diverse tasks.\nTo do so, the authors train different probabistic masks (using a Gumbel-Sigmoid) on the weights of a neural network over a series of different tasks. They incentivize these masks to be sparse by regularizing the number of \"active\" elements in a mask. \nThey then compare masks learned for different tasks (including simple arithmetic tasks, and permuted MNIST), and then compare the usage of the parameter masks over different tasks. They find that NNs learn to specialize only, without reuse.\nThe paper continues by postulating that the reason is either that the network learned \"bad\" representations, or that the network did not learn the correct composition. They argue that results on the SCAN dataset show that the representations are of a sufficient quality, concluding that the network did not learn the correct composition.\n\nCommentary:\nThe question the paper tries to answer is very relevant, on two levels. The first is that we do not understand how an NN learns sufficiently well. The second is that compositional modularity is a highly desireable property, and it is important to know if neural network exhibit it.\n\nStrengths:\n- The paper does some interesting analysis \n- In general, the paper is well-written, and easily understood.\n\nUnfortunately, the paper suffers from major drawbacks:\n- (this is a minor point I'm putting here to facilitate my discussion below) The paper is not positioning itself correctly in the literature, thereby using confusing terminology\nWhile modularity is not a main focus in neural network research, there exists some meaningful research that has established some terminology. In essence, the paper talks about compositional modularity (and combinatorial generalization), but does not use this terminology. This makes the paper a little difficult to follow, if one is familiar with this literature. Within this terminology, specialize would be called \"modularize\" and Preuse would be called \"compose\", a terminology I will use in the following.  \n- The conclusions are not convincing\nThe core argument in the paper is that neural networks fail to modularize because they either learn insufficient representations, or because they fail to learn to compose (to learn the \"algorithm\" required to utilize the modules correctly). Because the network learns re-usable representations, they modularize, and must thus fail to compose. \nWhile this is probably true, I cannot help to feel a little underwhelmed. It is well known that neural networks are overparameterized (see the \"lottery ticket\" literature). For this analysis, this means that it appears to be easier for the model to re-learn using the available capacity than to re-use the existing modules.\nThis is not particularly surpsising either, because this is, at its core, overfitting: the model does not generalize (which, in effect, is a \"softer\" way to re-use), but instead learns something akint to a separate function for different inputs.\nWhat the paper does not investigate is why a neural network not re-uses capabilities, even those should be a good fit for a problem. This would be a really interesting analysis, one which I would very strongly argue for acceptance in any venue.\n- The novelty of some parts is overstated\nThis is particularly true for using binary masks for multi-task like learning. See \"Bengio, E., Bacon, P. L., Pineau, J., & Precup, D. (2015). Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297.\"\nAdditionally, the paper does not sufficiently relate their insights to the (cited) work around inducing modularity and compositionaly in networks, some of which does already come to similar results.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KYPSF7TExQA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the functional modularity of neural networks using a simple yet efficient end-to-end method. The paper is well written and clearly articulates a contribution to the literature. The proposed method is intuitive and straightforward. The experimental evidence is provided for both synthetic, language, and image classification tasks. Most of the related works are cited.\n\nConcerns: The biggest concern I had is whether the conclusion reached in the paper is invariant to different neural network architectures, the size of the network, and the complexity of the task. As depicted in Figure 6 (a), there is a huge difference in the relative drop in performance for simple CNN, simple CNN without drouput and ResNet-110. It seems that a larger and more complex network tends to not sharing weights.\n\nBesides, the paper uses accuracy drop after masking the weights as the main metric, which is related to the number of masked weights. It might be better to learn the binary mask for each subtask with a certain accuracy objective (e.g., less than 1-2% lower than the original network) for each subtask and compare the learned mask of different subtasks. In addition, it is useful to see the results on a more complex task such as ImageNet. In ImageNet, there exist many similar subtasks as many categories of the images are actually belong to one big category. I am wondering whether these subtasks can share the weights.\nMinor comments: The paper claims the advantage of using Gumbel-Sigmoid than a simple threshold function. However, the state-of-the-art binarized neural networks are trained using a simple sign function with a straight-through estimator. Is there a significant difference (e.g., the stability of the training) in training the binary mask with the Gumbel-Sigmoid and threshold function?\n\nReasons for score: I vote for accepting. I like the finding that most neural networks have non-overlapped functional modules. However, I still have some concerns about the generality of the conclusion, and also the number of shared weights is not calculated with respect to a uniform accuracy requirement. I would consider raising my score if the authors can address my concerns.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a paper that is actively discussed.  The general sentiment is that this paper aims to address an important set of questions. While the technique could be improved with more novelty, the empirical study is extensive. The concerns are about how to interpret the results, or rather whether the empirical evidence fully supports the the claim/hypothesis.  After discussion and rebuttal, the reviewers improved their scores (and one reviewer remained at "weaker marginally above threshold").

The AC read the paper and the discussion. One value the AC sees that the discussion threads between the authors and the reviewers provide a significant amount of scientific value -- the questions to be answered are hard and might indeed require further refinements in framing and conceptualization, better techniques,  strong power in experimental designs to rule exclusively various hypotheses.  Thus, the AC recommends acceptance.
Decision: Accept (Poster)