The Multiple Subnetwork Hypothesis: Enabling Multidomain Learning by Isolating Task-Specific Subnetworks in Feedforward Neural Networks
Authors: Jacob William Renn
Ian Sotnek
Benjamin Harvey
Brian Caffo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Networks
Multitask Learning
Pruning

Abstract: Neural networks have seen an explosion of usage and research in the past decade, particularly within the domains of computer vision and natural language processing. However, only recently have advancements in neural networks yielded performance improvements beyond narrow applications and translated to expanded multitask models capable of generalizing across multiple data types and modalities. Simultaneously, it has been shown that neural networks are overparameterized to a high degree, and pruning techniques have proved capable of significantly reducing the number of active weights within the network while largely preserving performance. In this work, we identify a methodology and network representational structure which allows a pruned network to employ previously unused weights to learn subsequent tasks. We employ these methodologies on well-known benchmarking datasets for testing purposes and show that networks trained using our approaches are able to learn multiple tasks, which may be related or unrelated, in parallel or in sequence without sacrificing performance on any task or exhibiting catastrophic forgetting.

Under review as a conference paper at ICLR 2023
THE MULTIPLE SUBNETWORK HYPOTHESIS ENABLING MULTIDOMAIN LEARNING BY ISOLATING SPECCICCCIICC SUBNET- WORKS IN FEEDFORWARD NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Neural networks have seen an explosion of usage and research in the past decade, particularly within the domains of computer vision and natural language process- ing. However, only recently have advancements in neural networks yielded perfor- mance improvements beyond narrow applications and translated to expanded mul- titask models capable of generalizing across multiple data types and modalities. Simultancously, it has been shown that neural networks are overparameterized a high degree, and pruning techniques have proved capable of significantly reducing the number of active weights within the network while largely preserving perfor- mance. In this work, we identify methodology and network representational structure which allows a pruned network to employ previously unused weights to learn subsequent tasks. We employ these methodologies on well-known bench- marking datasets for testing purposes and show that networks trained using our approaches are able to learn multiple tasks, which may be related unrelated, in parallel or in sequence without sacrificing performance on any task or exhibiting catastrophic forgetting.
INTRODUCTION
It is well known and documented that artificial neural networks (ANNs) are often overparameterized, resulting in computational inefficiency (LeCun al.) 1990] Liu al. 2019). Applying unstructured pruning to an ANN pares down the number of active weights by identifying which subset of weights an ANN is most important for the model's predictive performance and discarding those which are less important or even entirely unnecessary. This technique has been shown to reduce the compu- tational cost of using and storing a model without necessarily affecting model accuracy (Frankle & Carbin 2019 Suzuki et al. 2001 Han et al. 2016: Lis et al. 2019 Wang et al. 2021). This phe- nomenon naturally raises a corollary question: are pruned weights entirely useless, or could weights identified as being unnecessary for one task be retained and used to learn other tasks? Further, can the ANN's performance on learned tasks be preserved while these weights learn to perform new
tasks?
The exploration of these questions leads us to propose the multitple subnetwork hypothesis that dense, randomly-nntializedd feedforward ANN contains within its architecture multiple disjoint subnetworks which can be utilized together to learn and make accurate predictions on multiple tasks, regardless of the degree of similarity between tasks or input types. Instead of focusing on matching or surpassing state of the art results on the data sets and tasks presented in this study, we focus on testing the multiple subnetwork hypothesis on a set of standardized network architectures and compare multitask model performance with traditionally trained, single-task models of identical architectures.
An obstacle to developing multitask ANNs is the tendency of ANNs to exhibit catastrophic for- getting (CF), during which they destroy internal state representations used in learning previously acquired tasks when learning new task (French) 1999 Goodfellow et al. 2013 Pfülb & Gepperth,
2.19). CF is especially pronounced in continuous learning paradigms with a low degree of inter- task relatedness (Aljundi et al. 2017 Ma et al. 2018 Masana et al.) 202I), posing challenge to creating multitask models able to perform prediction or classification across very different tasks, input data types, input shapes.

---

Under review as a conference paper at ICLR 2023
Isolating parameters for given task, thereby fixing parameter subsets for given task, is means
of minimizing the effects of CF in a continuous learning context (Delange et al. 2021). Parameter isolation methods localize the performance of learned tasks over physically (Yonn et al. 2018
Rusu et al. 2016] Mallya & Lazebnik 2018) or logically (Liu et al. 2019 Serra et al. 2018 Mallya
al.] (2018) isolated region of an ANN, reducing eliminating the interaction of weights responsi- ble for a given task, therefore minimizing the opportunity for the acquisition of new tasks to interfere with the ability of an ANN to continue to perform previously learned tasks. While the isolation of
subnetworks is the surest way to prevent interference arising from learning new tasks, this method can scale poorly under memory and computational constraints and therefore demands a flexible approach to limiting network capacity (Yoon et al.) 2018) minimizing the size of the network utilized for each task. Our methodology aims to negate the high computational cost of parameter isolation by representing multidomain multitask models as sparse tensors, thereby minimizing their computational footprint. this work, we demonstrate the implications of the multiple subnetwork hypothesis through a set of experiments, the results of which demonstrate that our model training procedure and multitask representational structure can be used to create multiclass models with reduced computational footprints which capable of overcoming CF.
2
METHODOLOGY
In this section, we will discuss how enable a single model to learn multiple tasks across multiple domains and datatypes through both a task-specific weight representational structure and modified training procedure which selects disjoint subsets of network weights for each individual task.
2.1 MULTITASK REPRESENTATIONAL STRUCTURE
The weights of fully connected neural network layers traditionally consist of a kernel tensor, k. of shape m x and a bias vector, b, of length n, where denotes the number of columns in the input data and n denotes the number of neurons within the layer. To process input data J. the decision function is thus ((kr + b), where & denotes the layer's activation function. Suppose then that there are inputs from two different distributions, Fi and 22.. Given the woodimenniooall structure of the kernel tensor in this traditional representation, there is no way to alter the decision function given the distribution currently selected.
To address this problem, we add a new dimension, t, the kernel tensor and the bias vector within the layer to denote which distribution task the input data belongs to, leaving the kernel tensor with a shape of x m x n and transforming the bias vector into bias matrix of shape n. The decision function for this new layer is thus altered to that in Equation 1
F(E, i) = (kk + b:)
(1)
We take similar approach for convolutional layers. For two-dimensional convolutions with color channels, the kernel is of shape 81 X 82 X x f, where 8 and 82 denote the height and width of the convolutional filters, represents the number of channels in the input, and f corresponds the number of filters. The bias vector in this scenario has length f. In our multitask representation of the convolutional layer, we once more add a new dimension the front of both the kernel tensor and the bias vector to denote the task.
2.2 SPARSIFICATION AND SUBNETWORK IDENTIFICATION
While the multitask representational structure described above could theoretically be used to enable network to learn multiple tasks as-is, it alone does not help us answer our hypothesis. Alone, this structure allows multiple networks to be combined into a single model, as the number of param- eters within each layer is increased by factor of the number of tasks. To fully test our multiple subnetwork hypothesis, we devised the Reduction of Sub-Network Neuroplasticity (RSN2) training procedure, which ensures that only one weight along the task dimension is active for all other fixed indices in multitask layers. In other words, due to the nature of RSN2's s pruning schema, no two fixed weights along the task dimension, I, are active after pruning.
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1 RSN2 Training Procedure
Input: Neural Network F(x), Training data for multiple tasks, X,, Y,, E N,,
for i {1, .., ] do
Choose Ps E (0, 100). ensuring Σ i= P: <= 100
Mask all weights which have been used for previous tasks
Unmask all weights which have not been used for previous tasks
Select (zz, Mt) as a subset of training data and labels for the current task Calculate weight gradients with respect to (I:, 3k)
Set all masked weight gradients to 0
Identify the quantity q, such that p% of weight gradients with respect to (æz, 9/) is less than ga Deactivate all weights with gradients less than q, and do not allow them to train Train the network as usual, only altering identified active weights and keeping all other weights masked
Save all weight values for the specified task end for
At inference time for task i, unmask only stored weights for task and perform inference as usual
((x) {
{
1, if 0, otherwise
Mathematically, we utilize the indicator function from Equation as activation function to indi- cate whether an individual weight is active or inactive within a layer. Using this function, we can therefore demonstrate whether disjoint subnetworks are active for all tasks by taking the sum of the indicator function across the task dimension for all other fixed indices. If for all fixed indices, and the condition present in Equation 3 holds, then each weight is only active once across all tasks, meaning the task dimension adds no additional active weights to the network layer.
T(Ka,b.e) S 1; b, c fixed
We utilize binary masking in our training procedure to impose these conditions. These masks there- fore ensure only select weights are active during training through the bitwise multiplication of the weight value and the mask value.
For weight selection and pruning, we utilize gradient-based based approach to identify which weights are to be selected for each task. By performing pruning in this manner, all pruning is done in one-shot manner after initialization rather than iteratively as the model trains, with the pruning rate hyperparameter selected at training time. The entirety of the RSN2 training procedure for single network layer can be found in Algorithm [ For performing inference, the masks used for training can be removed, thus reducing the number of active weights maximum value of t ++ W, where t represents the number of tasks, the maximum proportion of weights active for any one task, and W the number of weights which would be present in the network had traditional, one-task structure been used. Note that to satisfy Equation [ t * S 1. As a result, the RSN2 procedure, when coupled with the multitask representational structure defined above, identifies and separates disjoint subnetworks within a traditional network to allow each to individually perform separate tasks. Due the lack of interaction between these subnetworks in the resulting model, similarity between tasks is not required, and no transfer learning will occur between tasks. For a visual representation of the pruning methodology, we refer the reader to Appendix
2.3 EXPERIMENTS
We test our methodologies across variety of experiments utilizing multiple datasets and various model architectures to test the multiple subnetwork hypothesis and RSN2 training procedure. In our first experiment, leamn the same task multiple times. To test this, we create a single convolutional multitask network and isolate five separate subnetworks within it, each utilizing disjoint 20% of the weights within
test the merits of the hypothesis by identifying whether a single network can
3

---

Under review as a conference paper at ICLR 2023
the overall network architecture. We then train each of these subnetworks the Fashion MNIST dataset (Xiao et al. 2017). The performances of each of these subnetworks, as well as the ensemble of all five subnetworks, are then compared to the performance of dedicated network of identical architecture.
For our second experiment, train fully-connected network perform both the MNIST Digit Recognition (LeCun & Cortes) 2010) and Fashion MNIST tasks and compare the performance of a single network two dedicated networks. This experiment is the first we perform test whether single network can truly learn different tasks from multiple different domains and distributions, and it is the first experiment designed show whether fully connected network can perform multiple tasks.
In our third experiment, we train convolutional network perform both MNIST tasks from the previous experiment. This network is identical in architecture to all convolutional architectures trained previously, and once again this network is compared to two dedicated networks of the same architecture.
Our fourth experiment tests whether a single network can learn multiple tasks across an even wider arry of input data types. For this experiment, we train single fully-connected network to perform four tasks: both MNIST tasks, the Boston housing regression task, and an IMDB reviews sentiment classification task (Maas et al. 2011).
Our fifth experiment seeks to explore the feasibility of our methodologies in the context of more complicated tasks. For this experiment, we trained single convolutional model to perform four tasks. The first three of these are the age, gender, and ethnicity prediction tasks from the UTK- Face dataset (Zhang et al., 2017). The fourth task is the CIFAR1O task (Krizhevsky et al.). This experimental network is tested against four dedicated networks of identical architectures.
Our final experiment tests training procedure with a primary focus on testing the pruning and optimization capabilities of RSN2. For this ex- periment, we tested an unpruned transformer-based model's performance on the Reuters-21578, Distribution 1.0 dataset accessed via the UCI Machine Learning Repository (Dua & Graff (017))
single-task transformer-based architecture,
RESULTS
In this section, we present the results for each experiment conducted. Firstly, we present the results for each control model for all tasks, as well describe the model architectures and training sched- ules, as well as any data preprocessing steps which were taken. For control results, each control model was created only once and will therefore be presented with greater detail upon first presenta- tion. but its results will be included in every pertinent experiment. For more detailed indicators of performance for all models, as well as mappings between integer labels and their respective classes, we refer the reader to the appendices.
3.1 EXPERIMENT 1: MNIST FASHION ENSEMBLE, CONVOLUTIONAL ARCHITECTURE
Our first experiment involved creating single convolutional network with five disjoint subnetworks each trained on the MNIST Fashion dataset. We utilized a convolutional architecture with two convolutional blocks, with the first block containing two convolutional layers with thirty-two 3 x filters and ReLU activation (Agarap) 2018) followed by maximum pooling over a 2 x 2 filter area. The second convolutional block was identical to the first, except each of the convolutional layers utilized sixty-four filters. The output to the convolutional blocks were then flattened and fed into two fully-connected layers with 256 neurons each and ReLU activation. A final fully-connected layer with ten neurons and softmax activation was used to provide the final outputs. The only preprocessing which was done on input data was a division of pixel values by 255 to ensure all input values were within the interval from 0 to 1. The model was trained using a batch size of 512, and early stopping was initialized after three epochs with no improvement of greater than 0.01 in loss on validation data. The same architecture, preprocessing steps, and training procedure were used in the experimental case as well, with each subnetwork pruned so that 20% of the network weights were allocated to each task. A summary of results can be found in Table
4

---

Under review as a conference paper at ICLR 2023
Table 1: Experiment Results.
NETWORK
ACCURACY EXPERIMENTAL
PRUNING RATE
SUBNETWORK
90%
80%
SUBNETWORK 2
90%
80%
SUBNETWORK 3
90%
80%
SUBNETWORK 4
90%
80%
SUBNETWORK 5
92%
80%
ENSEMBLE
92%
80% EACH
CONTROL
MNIST FASHION
92%
N/A
Table 2: Experiment 2 results.
NETWORK
ACCURACY EXPERIMENTAL
PRUNING RATE
MNIST DIGIT
97%
90%
MNIST FASHION
86%
90%
CONTROL
MNIST DIGIT
97%
N/A
MNIST FASHION
88%
N/A
3.2 EXPERIMENT 2: MNIST DIGITS AND MNIST FASHION MODEL, FULLY-CONNECTED ARCHITECTURE
Our second experiment is the first experiment truly test whether a single model can learn disparate tasks across a variety of data modalities. To do this, we trained single collecconeeeed network to perform both the MNIST Digit Recognition and the MNIST Fashion Recognition tasks. For this experiment, we preprocessed the data for both tasks once more by dividing each pixel value by 255. Each image was then flattened into woo-dimensional vector. The images would then be passed through six fully connected layers each containing 1000 artificial neurons each and activated using the ReLU activation function. The final layer contained ten neurons which were activated with the softmax activation function. The model
trained with the same early stopping conditions all previous models, and a batch size of 512 was used once more. The experimental model was trained using the procedure and with the architecture, with each task pruned utilize 10% of the network weights. The results of this experiment can be found in Table
3.3 EXPERIMENT 3: MNIST FASHION Two-TAN TASK MODEL, CONVOLUTIONAL ARCHITECTURE
For our third experiment, we utilized an identical architecture, data preprocessing, and training pro- cedure to the previous experiment but trained two individual subnetworks within that architecture. the five classes which were most easily identified by the control
We trained the first subnetwork model. measured by Fl score, and we trained the second subnetwork on the five classes which were the most difficult for the control model to identify, also measured by F1 score. The results of the experiment can be found Table B
3.4 EXPERIMENT 4: MNIST DIGIT, MNIST FASHION, BOSTON HOUSING, AND IMDB REVIEWS MODEL, FULLY-CONNECTIO ARCHITECTURE
Our fourth experiment provides perhaps the most wide-ranging set of tasks for single network, and as a result it is perhaps the best test of our multiple subnetwork hypothesis. In this experi- ment, utilized single fully-connected neural network and trained the network on four separate tasks: the first two of these tasks, both the MNIST Digit and MNIST Fashion tasks, were trained simultaneously. The third task. the Boston Housing regression task, then trained. Finally, the IMDB reviews sentiment classification task was trained. The same architecture as previously used in all fully-connected experments used in this experiment. For the MNIST and Boston Housing
5

---

Under review as a conference paper at ICLR 2023
Table 3: Experiment 3 results.
NETWORK
ACCURACY EXPERIMENTAL
PRUNING RATE
EASY TASK
98%
80%
HARD TASK
84%
80%
CONTROL
MNIST FASHION
92%
N/A
Table 4: Experiment 4 results. In the performance column, percentages correspond to accuracy percentage test data, while decimals represent mean squared error test data.
NETWORK
PERFORMANCE EXPERIMENTAL
PRUNING RATE
MINST DIGIT
97%
90%
MNIST FASHION
87%
90%
BOSTON HOUSING
0.011
90%
IMDB
82%
90%
CONTROL
MNIST DIGIT
97%
N/A
MNIST FASHION
88%
N/A
BOSTON HOUSING
0.017
N/A
IMDB
80%
N/A
tasks, the same preprocessing and reshaping efforts took place as previous experiments with fully- connected architectures. For the IMDB task, only the most common 10000 words were utilized and sequences were padded (or truncated) to 128 words, with both padding and truncating occurring at the end the review. Furthermore, each of the input tokens was passed through an embedding layer, which embedded the token in two-dimensional vector space. The embedded tensors then flattened into a two-dimensional vector and passed through the architecture used throughout this experimen[] For the multitask model in this experiment, the first fully-connected layer which processed inputs from the IMDB task was dedicated solely to the IMDB task due to the differences in input shapes between all tasks. During training, the same early stopping criteria were used for all training iterations, batches sizes of 512 were used for the MNIST and IMDB tasks, while batch size of 32 was used for the Boston Housing task. The results for this experiment can be found in Table and task losses can be found in the appendices Section B The model was pruned such that each task utilizes 10% of the available weights in fully-connected layers; the embedding layer was not pruned.
3.5 EXPERIMENT 5: UTKFACE AND CIFAR10 FOUR-TASK MODEL, CONVOLUTIONAL ARCHITECTURE
In our fifth experiment, we tested the RSN2 training procedure on larger network with more com- plex tasks involved. We created a single network with convolutional architecture on four individual tasks, three tasks from the UTKFace dataset (Zhang et al., 2017) and one multiclass classification task from CIFAR10 (Krizhevsky et al.). For the age task in the UTKFace data, we grouped values into classes by decade. excluding a final class containing all individuals with age greater than 90.
Architecturally, the model trained in this experiment contains three convolutional blocks consisting of 16, 32, and 64 3 x 3 filters, respectively. Each block contains two convolutional layers with padding to preserve shape, followed by max pooling layer. These blocks then feed into fully connected architecture of three layers, each with 128 neurons each and activated using ReLU acti- vation Aggarpp 2018), followed by an output layer of the required shape perform the specified task.
For the multitask model, utilized shared convolutional embedding between all FTCe tasks. This resulted in a two-task structure for convolutional layers within the model, with one task chan-
This architecture involves six fully-connected layers of 1000 neurons and ReLU activation.

---

Under review as a conference paper at ICLR 2023
Table 5: Experiment 5 results.
NETWORK
ACCURACY
PRUNING RATE
EXPERIMENTAL
UTKFACE AGE
55%
75%
UTKFACE GENDER
89%
75%
UTKFACE ETHNICITY
77%
75%
CIFARIO
50%
75%
CONTROL
UTKFACE AGE
32%
N/A
UTKFACE GENDER
48%
N/A
UTKFACE ETHNICITY
14%
N/A
CIFAR10
10%
N/A
Table 6: Experiment 6 results.
NETWORK
ACCURACY
PRUNING RATE
CONTROL
75%
N/A
EXPERIMENTAL
79%
90%
nel processing the UTKFace images and the other task channel processing the CIFAR1O images. For the fully-commected layers, each of the UTKFace tasks was isolated and a four-task architec- ture was used. Along both the convolutional and the fully-connected architectures, each task was pruned to utilize only 25% of the model's available weights per task, meaning 50% of the weights in the convolutional layers were utilized and 100% of the weights across the shared fully-connected architecture was utilized All four tasks were trained simultancously, with early stopping criteria utilized with patience of live epochs. A summary of the results of this experiment can be found in T In addition to these performance results, which interestingly showed that the dedicated con- trol models were unable to converge but that the multitask model was able to converge, was also found that the multitask model only utilized 130M B of disk space when saved, while the combined single-task models utilized 203M of disk space, over 56% more than the multitask model.
3.6 EXPERIMENT 6: TRANSFORMER BEAEEEEEERE---DDDDD-- ARCHITECTURE
For our final experiment, we tested the applicability of our RSN2 training procedure training a transformer-basee architecture (Vaswani al.) 2017). For this experiment, we created model with sequence input length of 512 and vocabulary of 30000, attention heads, a feed-forward dimension of 1028, and dropout rate of 10%. Both models have approximately 35M parameters. We trained our control and experimental models on the Reuters-21577 Distribution 1.0 dataset accessed via the UCI Machine Learning Repository (Dua & Graff 2017) with batch size of 256 and utilizing 20% of the training dataset as hold-out validation data for early stopping.
For the experimental model, utilized multitask representational structure for feed-forward layers within the multi-headed attention components of the model, and these layers as well as all downstream fully-connected layers were pruned. Due the complexity of the embedding layers within the model architectures, however, instead of applying pruning immediately before training, we allowed our experimental model be trained for two epochs before applying pruning. Once this initial training occurred, we pruned the model using our gradient-based approach directly such that of 90% of weights within the aforementioned pruned layers were active.
A summary of this experiment's 's results can be found in T In addition to these results, we also saw a significant reduction of the size of the resulting models. The experimental model required 133M B of disk space, while the control model required 397 M B, nearly 200% more than the pruned model.
This due to the two-task structure of the convolutional layers, as mentioned above. This due the four-task structure of the fully-connected layers, as mentioned above.
7

---

Under review as a conference paper at ICLR 2023
3.7 LIMITATIONS
There are number of limitations to this study which limit the conclusions we can draw from the results. Firstly, our experiments focus primarily on computer vision related tasks in this work, with all experiments containing some image classification task. While we do also include experiments which test our methods on both tabular input and natural language inputs, further studies should still be conducted more rigorously test hypotheses and methods various input and data types.
Another limitation to this study is the limited size of both the datasets and models which were studied. Many state of the art models have been trained on millions of data samples, and the models themselves consist of many tens or hundreds of millions of parameters. Our study, on the other hand, considered labeled data with up tens of thousands of samples and models which contained hundreds of thousands to low-tens-of-million cases) than the state of the art in most cases. It is therefore unknown whether these methods would need to be modified to accommodate larger modeling paradigms.
of parameters, orders of magnitude fewer (in both
4 DISCUSSION AND CONCLUSIONS
This study was designed to test our multiple subnetwork hypothesis, which proposed that weights that might otherwise be pruned can be repurposed, ultimately yielding ANNs which can effectively perform multiple tasks across multiple data domains. To test our hypothesis, we developed the RSN2 training procedure to help identify subsets weights within a network which are well-suited for a particular tasks. To combat CF, then developed customized ANN representational structure designed to isolate individual subnetworks within the overall architecture of the global model on layer-by-layee basis. We then utilized these techniques in series of experiments across range of datasets utilizing a variety of common feedforward model architectures. Across each of these experiments, we consistently saw that single network was able to converge on multiple tasks by utilizing only a small fraction of network weights for each task. Furthermore, multitask models were able to perform each task with near identical performance to dedicated models with all weights active, even exceeding dedicated model performance in a few cases. Additionally, we saw that there needed be no similarity between tasks, as our methodologies do not utilize transfer learning across subnetworks within the network architecture.
tasks but instead rely on completely logically-isolal In our most extreme experiment, we were able to teach a single model to perform two separate image classification tasks and sentiment classification task on natural language.
In addition to these results, we also saw performance improvements terms of saved model sizes, which recorded for our larger models in our final two experiments. While also maintaining performance compared to dedicated models for all tasks, our multitask convolutional model utilized only 64% of the disk space of all of the dedicated, unpruned models. Similarly, our transformer- based model required only 33.5% of the space the unpruned model required. All of these size reduction numbers were achieved without the explicit use of sparse tensors in TensorFlow (Abadi et al. 2015).
Despite the limitations this study, there are still a number of conclusions results. Each multitask model was clearly able to converge on all its assigned tasks, achieving performance similar to, and in some cases exceeding, individual dedicated learners. These results therefore confirm our Multiple Subnetwork Hypothesis, showing that a single neural architecture is capable of utilizing small portions of itself to learn individual tasks. Additionally, we showed that our methodology for identifying and logically separating network weights is robust to CF. and we showed that we were therefore able to train multitask models across various domains and tasks both in parallel or in sequence without any adverse effects on previously-leamed tasks.
draw from its
Because the resulting model would have the ability to robustly perform multiple tasks, we can also conclude that a single model trained with these techniques could substitute multiple separate models in a deployed scenario. This substitution would therefore reduce resource requirements for organiza- tions deploying and utilizing models, thus saving costs and simplifying the deployment architectures these organizations use. These models would also be able to acquire more downstream tasks over
In particular, it is unknown whether our one-shot sparsification technique would generalize to much larger models.
8

---

Under review as a conference paper at ICLR 2023
their traditionally ttriined counterparts after previously being trained initial tasks, meaning adding new predictive analytics would be a much simpler undertaking as well, as it would only involve up- dating an
model and not require creating completely new deployment for every
new analytic.
4.1 FUTURE WORK
Following studies should be designed address some of the limitations of this work well as to improve the methods used within. Future studies should provide more thorough analysis on how these methods perform on a larger variety of data modalities, including data types, task types, data availability, and model sizes. Secondly, intend to identify whether more precise prun- ing method can be applied as training occurs instead of one-shot manner as was done this study. If pruning can be applied in more intelligent manner, we believe the resulting model would potentially be able to more consistently surpass unpruned model performance while also achieving greater sparsification rates than possible with one-shot techniques.
Additionally, this study does not perform any analysis to identify the maximum amount tasks which be learned single network. Future work should therefore address finding the upper bounds of the number tasks an individual model can support. Finally, the methods this work do allow for any transfer learning across tasks; instead, assumed that new tasks should utilize a completely disjoint subset of weights relative to all previously-eaared tasks. Additional studies should be conducted to identify whether transfer learning can leveraged during the acquisition of similar tasks to enable further reduction in the number of active weights and the quicker acquisition of new tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz. Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https: https://www... tensor low org/ Software available from tensorflow.org.
Abien Fred Agarap. Deep learning using rectified linear units arXiv:1803.08335. 2018.
(relu).
arXiv preprint
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3366-3375, 2017.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt:t///rcciive ics. uci, edu/ml
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URI. https://
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3: 128-135, 05 1999. doi: -dd-r_1-b3errr
Spenreview.nete forum htt///ooummmdrrr__--
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:131..6211,200
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: /ttp///avvvvvv.////vv///////.alv org. abs/ 1510.00144
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). URL http: www. www.:.cs.toon...oo toronto. edu/-kris cifarrbre html
Yann LeCun and Corinna Cortes. 2010.
MNIST handwritten digit database.
URL http://yann airaan.leecun. com/ exdo/
mn mnist/
Yann LeCun, John Denker, and Sara Solla.
Optimal brain damage. In D. Touret- zky (ed.). Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1990. URL
Gc5822baacee03322225111177777---
pdf
Mieszko Lis, Maximilian Golub, and Guy Lemieux. Full deep neural network training on a pruned weight budget. Proceedings of Machine Learning and Systems, 1:252-263, 2019.
Zhuang Liu, Mingjic Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruming. In International Conference on Learning Representations, 2019. https::::
//openrevies ew. wwtp:/eeetooommmeeememmm net.:
10

---

Under review as a conference paper at ICLR 2023
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relation- ships in multi-task learning with multi-gate mixture-of-expert In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1930-1939, 2018.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguisties. URL http: 7/ //www.c ac web. http://thhoolly/1111111111
antho
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network multi- ple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 67-82, 2018
Mare Masana, Tinne Tuytelaars, and Joost van de Weijer. Ternary feature masks: sero-forgetting for task-incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2570-3579 2021.
B. Pfülb and A. Gepperth. A comprehensive, application-oriented study of catastrophic forgetting in DNNs. In International Conference on Learning Representations, 2019. URL https:// openreview. net htt.:e//ooorrmmm/d..loosss/////s
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint (eli:1600.067777 2016.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548 4557. PMLR, 2018
Kenji Suzuki, Isao Horiba, and Noboru Sugie. A simple neural network pruning algorithm with application to filter synthesis. Neural processing letters, 13(1):33-53, 2001.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httpps: hwwriis.ee.p.pp.pp cc/ httpppaaer/21////lee haa.a//22577caa11bb0aaaccccaaaaa
pdf
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Emerging paradigms of neural network pruning. arXiv preprint arXiv:2103. 064600 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. fishion-mniste a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018. URL .nt/ http://wwwwwweeetteeueeeeeeeeeee http://foummmmikkkksfsfttt
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
11

---

Under review as a conference paper at ICLR 2023
A VISUAL REPRESENTATION OF RSN2
Figure 1: Diagram of the RSN2 training procedure and multitask representational structure for single fully-connected layer with two tasks. On the left, we begin with the initialized layer weight matrix (center) and create two binary mask matrices (top and bottom). These binary masks then bitwise-multiphe to the weight matrix, resulting in two disjoint subsets of weights which are then separated across the task dimension in representational structure. Note to the right, two corresponding parameters are nonzero.
12

---

Under review as a conference paper at ICLR 2023
TASK LOSSES, EXPERIMENT 4 B
10
MNIST Digit Test Loss
MNIST Fashion Test Loss
Boston Housing Test MSE Loss
IMDB Test Loss
Loss
0.4
Both MNIST
Boston Housing
IMDB
Tasks Trained
Figure 2: Task losses obtained on test data for multitask model after training each set of tasks.
13

---

Under review as a conference paper at ICLR 2023
C
CLASS LABELS FOR CLASSIFICATION TASKS.
Table Class labels for MNIST tasks.
DATASET
INTEGER
LABEL
MNIST DIGIT
DIGIT "0"
DIGIT "1"
3
DIGIT "2"
4
DIGIT "3"
DIGIT "4"
DIGIT "5"
DIGIT "6"
8
DIGIT "7"
DIGIT "8"
DIGIT "9"
MNIST FASHION
T-SHIRT/TOP TROUSER
PULLOVER DRESS
5
COAT
SANDAL SHIRT
SNEAKER
BAG
10
ANKLE BOOT
14

---

Under review as a conference paper at ICLR 2023
Table 8: Class labels for CIFARIO and UTKFaae tasks.
DATASET CIFAR-10
INTEGER
LABEL
AIRPLANE
2
AUTOMOBILE
3
BIRD
CAT
5
DEER
6
DOG
7
FROG
HORSE
9
SHIP
10
TRUCK
UTKFACE AGE
0 10
2
11 20
3
21 30
4
31
41
6
51 60
7
61 70
8
71 80
81
10
91+
UTKFACE GENDER
MALE
FEMALE
UTKFACE ETHNICITY
WHITE BLACK
3
ASIAN
4
INDIAN
5
"OTHER" (DEFINED BY DATASET CREATORS)
Table 9: Class labels for IMDB task.
INTEGER 1
LABEL
NEGATIVE REVIEW
2
POSITIVE REVIEW
15

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL MODEL PERFORMANCE METRICS.
When conducting our study, we collected additional performance measures for all models which were not included in the main manuscript above. These metrics are presented in the following tables.
16

---

Under review as a conference paper at ICLR 2023
Table 10: Per-class and overall accuracy scores for control computer vision models. "FC" indicates fully-connected architecture. "Conv" indicates convolutional architecture.
ACCURACY
.97
.99
.88
.92
.47
.32
48
.14
10
10
.96
.98
.96
.97
.52
0
N/A
N/A
0
9
.96
.99
.97
.99
.55
0
N/A
NA
0
8
.97
.99
.95
.97
.53
N/A
N/A
7
.98
.99
.69
.76
51
0
N/A
N/A
0
CLASS NUMBER
6
.97
.99
.97
.98
.39
0
N/A
N/A
18
5
.98
.99
.79
.86
.38
N/A
0 0
.97
.99
.88
.92
.26
0
N/A
0
0
.97
.98
.79
.87
.35
.49
N/A
25
0
.99
.99
.98
.99
.59
0
98
99
84
85
.55
UTKFACE ETHNICITY CONV
CIFAR-I0 -0 CONV (EXP 6)
UTKFACE GENDER CONV
CIFAR-10 CONV (EXP 8)
MNIST FASHION FC MNIST FASHION CONV
UTKFACE AGE CONV
MNIST DIGIT FC MNIST DIGIT CONV
NETWORK
17

---

Under review as a conference paper at ICLR 2023
ACCURACY
0.90
0.90
0.90
0.90
0.92
0.98
0.84
0.98
0.87
0.99
0.90
0.97
0.87
0.98
0.89
0.60
0.97
0.87
0.55
0.89
0.77
0.50
Table 11: Per-class Fl and overall accuracy scores for experimental computer vision models.
10
0.97
0.96
0.97
0.97
0.97
0.97
0.96
N/A
0.96
0.95
0.98
0.96
0.96
0.95
0.97
0.96
0.69
0.96
0.95
0.08
N/A
N/A
0.52
0.97
0.98
0.97
0.98
0.98
0.98
0.99
N/A
0.97
0.97
0.98
0.97
0.97
0.96
0.98
0.97
0.70
0.96
0.95
0.45
N/A
N/A 0.61
9
0.96
0.96
0.96
0.96
0.96
0.97
0.96
N/A
0.98
0.94
0.99
0.96
0.98
0.94
0.98
0.95
0.70
0.96
0.93
0.29
N/A
N/A 0.54
0.72
0.72
0.72
0.72
0.78
0.76
N/A
0.73
0.98
0.62
0.99
0.71
0.97
0.68
0.98
0.70
0.69
0.97
0.64
0.34
N/A
N/A 0.53
7
CLASS NUMBER
0.98
0.98
0.98
0.98
0.98
0.98
0.97
N/A
0.97
0.96
0.99
0.98
0.97
0.95
0.98
0.97
0.49
0.96
0.95
0.44
N/A N/A 0.47
0.84
0.83
0.83
0.84
0.88
0.86
N/A
0.85
0.97
0.79
0.99
0.85
0.97
0.78
0.98
0.83
0.49
0.97
0,78
0.14
N/A
0.27
0.44
5
0.90
0.89
0.88
0.90
0.91
0.91
N/A
0.92
0.97
0.87
0.99
0.90
0.97
0.88
0.98
0.89
0.43
0.96
0.87
0.37
N/A
0.68
0.35
0.85
0.84
0.82
0.84
0.88
0.87
N/A 0.87
0.98
0.77
0.99
0.85
0.97
0.78
0.98
0.83
0.46
0.98
0.78
0.68
N/A
0.79
0.27
3
0.98
0.97
0.98
0.98
0.99
0.98
1,00
N/A
0.99
0.97
0.99
0.98
0.99
0.98
0.99
0.98
0.72
0.99
0.97
0.45
0.88
0.83
0.63
2
0.86
0.85
0.84
0.85
0.87
0.87
N/A
0.85
0.99
0.82
0.99
0.85
0.98
0 0.82
0.98
0.84
0.65
0.98
0.82
0.89
0.90
0.84
0.58
EASY SUBNETWORK
HARD SUBNETWORK
UTKF ETHNICITY
NETWORK
SUBNETWORK, ORK 2
SUBNETWORK, 3
MNIST FASHION
MNIST FASHION
MNIST FASHION
MNIST FASHION
MNIST FASHION
UTKF GENDER
SUBNETWORE ORK
SUB TUBNETWORK
SUBNETWORK
MNIST DIGIT
MNIST DIGIT
MNIST DIGIT
MNIST DIGIT
MNIST DIGIT
ENSEMBLE
CIFAR- 10
UTKF AGE
CIFAR- 10
EXP
EXP
EXP 4
EXP 5
EXP 6
EXP 7
EXP 8
EXP
18

---

Under review as a conference paper at ICLR 2023
Table 12: Per-class Fl and overall accuracy scores for experimental and control IMDB sentiment analysis models.
NETWORK
CLASS NUMBER
ACCURACY
EXPERIMENT 7 IMDB
0,83
0.81
0.82
CONTROL IMDB
0.80
0.80
0.80
Table Boston Housing model losses.
NETWORK
LOSS
EXPERIMENT 5
0.016
EXPERIMENT 7
0.011
CONTROL
0.017
19

---

Under review as a conference paper at ICLR 2023
Table 14: Transformer performances. Per-class performance is reported as Fl score.
CLASS NUMBER
CONTROL PERFORMANCE EXPERIMENTAL PERFORMANCE
0,48
0.70
0.80
0.75
0.48
0.63
0,92
0.93
0.83
0.85
0,00
0.00
0.57
0.96
0.00
0.00
0.41
0.69
10
0.80
0.88
11
0.88
0.90
0.64
0.73
13
0.00
0.40
14
0,59
0.56
15
0,00
0,40
0.14
0.17
0.70
0.74
18
0.33
0.32
19
0.45
0.56
20
0,66
0.68
0.41
0.55
22
0.54
0.67
23
0.00
0.00
24
0.12
0.24
0,50
0.49
0.54
0.76
27
0.75
0.35
28
0.00
0.67
29
0,20
0.37
0,00
0.33
31
0.33
0.32
32
0.32
0.50
33
0.24
0.95
34
0,00
0.80
0.35
0.73
36
0.00
0.50
37
0.14
0.70
38
0.00
0.00
0.50
0.00
0.00
0.13
41
0.00
0.46
42
0.46
0.62
43
0,00
0.00
0,00
0.00
45
0.67
0.89
46
0.00
0.67
AUCURAAACCCUACA
0.75
0.79
20


Reviews:
[{"review_id": "h8PTMWjE7A7", "reviewer": "Reviewer_aGRW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper builds on existing work showing that artificial neural networks often require only a small fraction of their parameters to perform well. The paper proposes the 'Multiple Subnetwork Hypothesis', which extends this observation to conclude that the weights of a single network could be separated into disjoint subnetworks that could each be trained for separate tasks. This allows a single network to be trained on many different tasks without using increased memory or suffering from catastrohpic forgetting. The authors propose an algorithm called RSN2 to train a single network in this disjoint fashion. This algorithm works by copying the model weights across an additional 'task' dimension, then ensuring that only one weight can be active at a time across the task dimension by masking the weights based on an initial pruning performed at the beginning of training. They test their hypothesis in a number of multitask settings by comparing the performance of a control model on a single task to their multiple subnetwork model across many tasks, and show comparable performance to the control with significant memory savings.", "summary_of_the_review": "Overall I think this paper proposed an interesting idea, but failed to make a convincing case for why it would actually be useful. There were no empirical comparisons done to other existing approaches, the results were not statistically significant, and the authors were not clear about what the intended benefit of their approach was and how it compared to other similar methods. Many of the experiment settings and tables were unclear, and I think it is difficult to conclude much about their approach from the results they presented. I think the paper would need significant rewriting including at a minimum:\na) direct comparisons of the memory/energy savings of their method to other existing approaches\nb) justification for why they expect a performance increase from their method (if this is a claimed benefit)\nc) experiments with multiple trials, including comparisons to other weight pruning approaches and proper comparisons against the baseline\nd) a clearer description of the experimental settings", "strength_and_weaknesses": "This paper makes an interesting observation - if a network requires only a fraction of its weights to perform well, can the others be put to use for other tasks? The idea of performing multi-task learning this way is certainly interesting, and in a few cases they even show slightly improved performance with it over the control. I like the idea behind the approach, and I think a thorough evaluation of using ensembles of models trained in this way to compare performance against the base model could lead in interesting directions. I do have a number of reservations, however.\n\nWhile this method is interesting, I am not sure what concrete benefit it actually offers. The ability to store multiple trained networks within one network without using additional memory is certainly interesting, but all of those networks are constrained to using the same architecture, and since there is no interaction between the separate tasks it is hard to see how there could be any type of transfer learning or improved performance. The memory savings of this approach are certainly interesting, but it is unclear to me whether how this memory/energy savings compares to the memory/energy savings of existing methods for pruning neural networks. Perhaps the benefit is that existing weight pruning methods require dedicated hardware/software for dealing with sparse matrices to be fully realized? If so however, this is not made fully clear in the paper, nor is it clear that the proposed architecture would avoid having the same limitations. \nAs such, I do not fully understand what the actual contribution here is meant to be. Since there is no comparison done against any other similar approaches for memory/energy savings, it is hard to see if this is a significant contribution. And since each subnetwork is entirely disjoint, I do not see how this method would be likely to lead to improved performance. The results shown are largely comparable to the control in accuracy, save for a few examples where the proposed approach leads to slight improvements - but there is no statistical analysis done to show whether these small improvements are significant or not. There is also no comparison done to the performance of other weight pruning approaches on the same tasks and architectures, either in terms of memory use or model accuracy.\n\nThe most compelling example given is experiment 6, using a transformer architecture. The setting of this experiment is highly unclear to me, however. What is the actual task being performed here - simply text classification? If so, how is the multi-task approach being used, as there is only one task under consideration? Is the multi-task model using an ensemble of models each learning to perform the same task? Are they split by class somehow? How many models are being used? Is the final result shown in table 6 an ensemble across multiple models, or an average, or something else? The table and description make it appear as though only one model is being used, but the proposed approach is explicitly a method for training a single model's weights on multiple tasks.\n\nAs an additional note, the table given for experiment 3 seems highly incomplete, as there is no way to directly compare the authors' model to the baseline. The table should include either the results of the baseline model split across the two task settings, or some sort of ensemble of the experimental model evaluated on the whole dataset. As is, the values in the table are meaningless as there is no way to compare them.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cSLtY7htOA", "reviewer": "Reviewer_PPCR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a hypothesis and network structure that allows a pruned network to employ previously unused weights to learn subsequent tasks. Specifically, the authors develop a customized ANN representational structure to isolate individual sub-networks and devise a training procedure named Reduction of Sub-Network Neuroplasticity (RSN2) to train the proposed model. Experimental results show that the proposed method can train an ANN without sacrificing performance or catastrophic forgetting.\n", "summary_of_the_review": "The idea is interesting. The experiments can be further improved.", "strength_and_weaknesses": "Strengths:\n1. The multiple subnetwork hypothesis is very interesting. The proposed multi-task training method based on this hypothesis is reasonable.\n2. This paper provides sufficient and detailed implementation details. The pseudocode in Algorithm 1 is clear.\n3. The limitations of the proposed method and future work are clearly discussed.\n\nWeaknesses:\n1. This paper evaluates the proposed method on both convolutional networks and transformer architectures. However, all the considered architectures do not belong to any popular architecture family. It would be better to conduct experiments on some popular architectures, e.g., ResNet [A] and Swin Transformer [B].\n\n2. It is unclear what the essential differences are between the proposed RSN2 and existing pruning methods. Moreover, the comparisons among these methods are still missing. \n\n3. This paper mainly conducts experiments on small datasets, such as MNIST and CIFAR. It would be better to consider some large-scale datasets, e.g., ImageNet.\n\n4. As mentioned in the paper, some weights of subnetworks may be unnecessary for one task but can be used to learn other tasks. In other words, the performance of subnetworks may vary a lot among different tasks. However, such differences are still unclear in the paper. It would be interesting to visualize the performance distribution of subnetworks on different tasks, similar to what is shown in [C]. Moreover, it is also very interesting to see how large the ranking of subnetworks changes from one task to another one (e.g., the best subnetwork on one task may be only located at the median on another task).\n\n\n5. As mentioned in Section 1, the main goal is to“negate the high computational cost of parameter isolation”. Nevertheless, it seems that there are no results to show how much computational cost can be reduced.\n\n[A] Deep Residual Learning for Image Recognition, CVPR 2016.\n\n[B] Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows, ICCV 2021.\n\n[C] Improving Robustness by Enhancing Weak Subnets, ECCV 2022.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VXQ3Xec6gN", "reviewer": "Reviewer_yCzy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper builds upon the lottery ticket hypothesis and posits the multiple subnetwork hypothesis in which weights of a single network are disjointly allocated to multiple tasks to avoid catastrophic forgetting while achieving competitive task-wise performance on all tasks.", "summary_of_the_review": "I recommend reject for this paper due to the lack of novelty and insight in the proposed method for using disjoint weights of a single network to learn multiple tasks. There is no sharing at all between tasks unless introduced in an ad-hoc manner manually for specific tasks.  The experiments are also limited and of low quality. ", "strength_and_weaknesses": "Pros:\n- Perhaps unsurprisingly, the authors are able to demonstrate that disjoint subsets of network weights can be used for different tasks without loss in performance.\n- The approach to more efficient multi-task learning with a single network is simple. \n\nCons:\n- The authors do not adequately discuss related work in multi-task learning.  Learning Sparse Sharing Architectures for Multiple Tasks by Sun et al., 2019 is especially relevant and not discussed at all.\n- The proposed approach is not very interesting since there is no knowledge sharing across tasks by design with disjoint subset of weights to avoid catastrophic forgetting. There is obvious benefit to sharing as recognized by the authors in Experiment 5 in which the UTKFace tasks share convolutional embeddings.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RwS5QG3_ue", "reviewer": "Reviewer_3BZv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a method that enables a pruned network to employ previously unused weights to learn subsequent tasks. The exploration along this work leads to multiple subnetwork hypothesis-- that a dense, randomly initialized feedforward ANN contains within its architecture multiple disjoint subnetworks which can be utilized together to learn and make accurate predictions on multiple tasks.", "summary_of_the_review": "The paper needs significant improvement before being accepted by any major conferences. First, please elaborate why this method is different from traditional approaches like lifelong learning with lottery ticket [1], and multitask learning with sparsely gated mixture-of-experts. In a MoE network, one or a few experts are selected for each data, which is a more general formulation that subsumes the proposed method in this paper. The paper should consider improving the baselines and evaluation benchmarks. For example, how does this proposed method improve against traditional lifelong learning methods and multitask methods. If the emphasis is on lifelong learning, then the paper should focus on a streaming of tasks.  If multitask is the priority, then should compare against approaches with stronger performance on multitask learning, and sparsely gated MoE can be a good one.\n\n[1]: https://openreview.net/forum?id=LXMSvPmsm0g ", "strength_and_weaknesses": "The proposed idea looks very similar to lottery ticket hypothesis [1], such that there is a winning ticket for every task. The problem formulation, the expanded kernel tensor, looks very much like a mixture-of-expert layer while the sparsification and subnetwork identification are very much like the routing function used in a mixture-of-expert network. In this sense, this paper has very limited novelty and add-on contribution compared to related work. At least, the paper should explain why it is different from related work and why this method address multitask learning better.  \n\nEvaluations are not through by only comparing on tiny datasets like MNIST. The tables are extremely hard to interpret. \n\n[1]: https://arxiv.org/abs/1803.03635", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject