Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks
Authors: Andrey Zhmoginov
Dina Bashkirova
Mark Sandler
Venue:  (2021)
Keywords: modular networks
transfer learning
domain adaptation
self-organization

Abstract: Conditional computation and modular networks have been recently proposed for multitask learning and other problems as a way to decompose problem solving into multiple reusable computational blocks. We propose a novel fully-differentiable approach for learning modular networks.In our method, the modules can be invoked repeatedly and allow knowledge transfer to novel tasks by adjusting the order of computation. This allows soft weight sharing between tasks with only a small increase in the number of parameters. We show that our method leads to interpretable self-organization of modules in case of multi-task learning, transfer learning and domain adaptation while achieving competitive results on those tasks. From practical perspective, our approach allows to: (a) reuse existing modules for learning new task by adjusting the computation order, (b) use it for unsupervised multi-source domain adaptation to illustrate that adaptation to unseen data can be achieved by only manipulating the order of pretrained modules, (c) show how our approach can be used to increase accuracy of existing architectures for image classification tasks such as ImageNet, without any parameter increase, by reusing the same block multiple times.

Under review as a conference paper at ICLR 2021
COMPOSITIONAL MODELS: MULTI-TASK LEARNING AND KNOWLEDGE TRANSFER WITH MODULAR NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Conditional computation and modular networks have been recently proposed for multitask learning and other problems as a way to decompose problem solving into multiple reusable computational blocks. We propose novel dully-diffrrentiable approach for learning modular networks. In our method, the modules can be in- voked repeatedly and allow knowledge transfer to novel tasks by adjusting the order of computation. This allows soft weight sharing between tasks with only a small increase in the number of parameters. We show that our method leads interpretable self-organization of modules in case of multi-task learning, transfer learning and domain adaptation while achieving competitive results on those tasks. From practical perspective, our approach allows to: (a) reuse existing modules for learning new task by adjusting the computation order, (b) use for unsupervised multi-source domain adaptation to illustrate that adaptation to unseen data can be achieved by only manipulating the order of pretrained modules, (c) show how our approach can be used to increase accuracy of existing architectures for image clas- sification tasks such as IMAGENET, without any parameter increase, by reusing the same block multiple times.
INTRODUCTION
Most modern interpretations of neural networks treat layers as transformations that convert lower- level features, such as pixels, into higher-level abstractions. Adopting this view, many designs that break up monolithic models into interacting modular components localize them to particular processing stages thus disallowing parameter sharing across layers. This restriction, however, can be limiting when we need to continually grow the model, be that for transferring knowledge to new task preventing catastrophic forgetting. It also prohibits module reuse within model even though certain parameter-efficient architectures and complex decision making processes may involve recurrent components (Kaiser & Sutskever 2016: Randazzo et al. (020)) and feedback loops (Herzog et al. 2020 Yan et al. 2019 Kar et al. 2019).
Here propose simple alternative design that represents entire model as mixture of modules each of which can contribute to computation at any processing stage. As opposed to many other approaches (Zoph & Le (206)); Bengio (2016): :); Kirsch al. (2018a); Rosenbaum et al. (2019) etc.), we use a soft mixture of modules to obtain a more flexible model and produce a differentiable opti- mization objective that can be optimized end-to-end and does not involve high-variance estimators. Following our approach, the parameters of every block (or layer) of the network is computed as a linear combination of set of "template" block parameters thus representing the entire model as: (a) a databank of template blocks and (b) vectors of "mixture weights" that are used to generate weights for every layer. This simple design can utilized for a variety of applications from producing com- pact networks and training multi-task models capable of re-using individual modules to knowledge transfer and domain adaptation. The experimental results show that: (a) when used for multi-task training, our model organizes its modules where tasks share first few layers while specializing closer to the head, while (b) in domain adaptation problems modules instead specialize on processing the image, while sharing later processing stages. Moreover, our self-organizing model achieves promis- ing results in multi-task learning and model personalization. The rest of the paper is organized as follows: in Section 2 we go over the related literature and discuss the existing approaches to modular

---

Under review as a conference paper at ICLR 2021
networks and neural architecture search; in Section 3 we describe the architecture of our conversa- tional model and various aspects of learning the best module sequence for each specific task; we introduce the experimental results in single-task setting, multi-task learning, continual learning and domain adaptation in Section 4 and provide a final discussion in Section 5
PRIOR WORK
The idea of constructing a deep neural network as a composition of reusable computation blocks has been extensively explored in recent literature. First conditioned computation methods used evolution algorithms (Wierstra et al. (2005): Floreano et al. (2008); Mikkkulinen et al. (2019)) to determine a suitable model architecture; later on, reinforcement learning was used to optimize the model layout and parameters (Zoph & Le 2016); [engio (016)); Baker et al. (2066). Recently, Kirsch et al. 2018a) proposed end-to-end modular neural network framework that learns to choose the best among several module candidates at each layer according to the input data.
Conditional computation routing is even more appealing for the multitask learning application, as it allows to both learn reusable computation blocks and adjust the model to each specific task with minimal network alteration. For instance, Misra al. (2016) use an additional set of modules for each task and enforce similarity between the corresponding task-specific modules weights. [ooee- baum et al. (2017) 2019) used reinforcement learning to perform task- and data-specific routing; Sun et al. (2019) train both the model weights and the task-specifi policy that determines which layers should be shared for a given task; Ma al. (2019); Maziarz et al. (2019) introduced a multi- task model where each layer's output is computed as a weighted sum of set or candidate modules outputs, which is similar to our method, with two important differences: a) in the works by Ma ett al. (2019;; Maziarz et al. (2019), the output is computed outputs, not module weights, and b) the module candidates in their methods are tied to the specific location in the network. Maninis al. (2019) added task-specific residual adapters specialize the feature extractor to each task, The method proposed by PPurushwalkmm et al. (2019) performs zero-shot multitask learning by finding task-specific routing using gating mechanism. This ap- proach is somewhat similar to ours, but is less efficient in terms of the number of parameters since a) the method in PPurushwalkmm et al. (2019) uses modular network on top of a classical ResNet (He et al. (2016)) while our method uses only about 100 additional parameters and b) the modules used in Purushwalkam et al. (2019) are layer-specific and not reusable while there is no such constraint in our architecture. Wu et al. (2018); 8); Newell et al. (2019); Guo et al. (2020) perform task task-specific model compression. There also exists a number of studies on the effectiveness of modular networks in the context of visual question answering. One of the early papers Andreas et al. (2016) used natural language parser to determine the layout of the composite network consisting of predefined modules that solve different kinds of subtasks, such as find, measure, describe etc; further works im- proved this approach by switching from an external parser to a fixed (Hu et al. (2017b) or arbitrary query structure (Hu et al. 2017a 2018): Pahuja et al. (209)).
linear combination of candidates
3 COMPOSITIONAL. MODELSS AND MODULE MIXTURES
3.1 COMPOSITIONAL MODELS
In conventional convolutional deep neural networks, most layers are different from each other and do not conform to a particular fixed design. Various model blocks may have different resolutions and different numbers of filters, some blocks may have or lack residual connection and on. This makes most model activations at different layers incompatible with each other. Recently, it has been demonstrated (Sandler et al. 2019) that high performing convolutional networks can nevertheless be composed of identical blocks. Such networks that were called isometric networks essentially iterate on the same activation space. The architecture of an isometric network includes the following core components: (a) input adapter that maps model input E I into the activation space Z: (b) model body. a sequence of blocks all sharing the same architecture and mapping the space Z to itself: (c) output adapter mapping the output of the last block into the embedding space E, and finally (d) logits layer for classification models, mapping embedding space E into the final predicted probabilities.
2

---

Under review as a conference paper at ICLR 2021
Prediction A
Prediction
Prediction
Prediction
layer templates
Layer
Layer k
Layer k
Laver
Layer 1-1
Layer le-1
Layer
Layer 2
Layer 2
Layer
Layer
41
Layer
Task
Task H
Task
Task B
Compositional model (our)
Conventional weight sharing
Figure I: Left: conventional hard weight sharing approach. The model consists of a shared feature extractor followed by the task-specific task solvers (e.g. logits layer in case of classification). The only place where the model can specialize to each task is in its solver, which might not be enough for sufficiently distinct tasks. Right: Compositional model. We ules (we call them templates) available for all tasks; each layer's weights are generated as a linear combination of the templates weights, and the linear combination weights (mixture weights { task-specific. In this case, the templates can not only be shared across different tasks, they also can be reused multiple times within the same task. (best viewed in color).
shared set of trainable mod-
While isometric networks achieve near SOTA performance, the fact that they iterate on the same internal activation space Z makes them perfect candidate for studying modular architectures. Specifically, instead of treating each block as independently trainable mapping, entire model body as a composition will denote set of trainable modules as F {f1, f2,..... fx)
may view the modules f& each of which may appear in a sequence more
than once. In the following, and their composition f Jm, o ... o fm, forming a model body l will frequently be denoted as m(.........)) with (m (mi,....,7p being called signature of this model.
3.2 LEARNING COMPOSITIONAL MODELS
Mixture modules Compositional models discussed in Section 3.1 can be trained on particular task for any predefined choice of the model signature. But, while the study of hand-designed mod- ular architectures is interesting exploration in itself, one may want to go further and identify sequence modules (m1, .2...... m)) together with the modules F themselves that is the most advantageous choice for solving particular problem. In some previous explorations of modular chitectures, authors used probabilistic approach where the model body (m.,... m.) was sampled for every input batch or sample from the optimized probability distribution p(m1, m2,- mz). For example, in Kirsch et al. (2018b), module were sampled from a distribution defined by per-layer conditional probabilities p(mppxa11 with xz-1 being the output the previous layer (E 1), i.c.. Iz-1 f(mm....m. (i). Since discrete nature of such a choice limits the flexibility of the model and since practice learning such a model may be complicated due to high-variance estimator noise, we instead replace composition of blocks from JF with a composition of mixture modules, each which is essentially a smooth superposition blocks from F. More precisely, each mixture module he with & E IR K has the same architecture as blocks from F. but every parameter 0 in this mixture module (be that some convolutional kernel, or a batch normalization f or parameter) is linear combination ΣK each block contains multiple convolutions with different nonlinearities, mixture module obtained by linearly combining weights from F is not a linear combination of modules from F. It is true however that if & di for some fixed j (which we later refer to as "one-hot" vectors), mixture module bg becomes identical to fy F. Due to the fact that hg mixes parameters from F, we will
k-k1 E& of the corresponding parameters 0% from fi E F. Notice that since
with L is the total number of layers in the model
hhee each ma is actually set of several chosen modules the output of which is combined
Herr 81g is Kronecker delta

---

Under review as a conference paper at ICLR 2021
sometimes refer to blocks from F as templates, which are mixed together to finally form actual unit of computation.
After we choose particular task and replace the model body with composition of mixture modules heg ..... o heg. we obtain a differentiable objective that can be optimized with respect to a model signature = (f1..... ε1) and parameters of F. In our experiments, we constrained each of & by using a softmax nonlinearity, which allowed us to guarantee that (Ee);; = 1 for every layer l. Of course, even with this choice of the nonlinearity, we are not guaranteed that the final signature = will contain only one-hot vectors & that can be interpreted as blocks from F.
Regularization of mixture weights While in our experiments, mixture weights & frequently ap- proached a set of approximately one-hot vectors one can use additional regularizers to ensure that all & are one-hot and the resulting model can be represented as a composition of individual template modules. In Appendix tion that penalizes (E); < 1, but does not involve template weights and (b) a regularizer that penalizes E\r and also effectively attracts individual templates towards the generated mixture weights to which they contribute the most.
[ we outline two choices that we experimented with: (a) entropy regulariza-
Weight generation interpretation Consider single block of our isometric network and let be function family containing single-block tensor transformations corresponding to all possible choices of block weights. All neural networks obtained by composing L such blocks without any weight- tying form a function family 8 Now, instead, consider a model composed of mixture-weight blocks with templates from F. In this case, the family of all possible single-block transformations is only controlled by a vector of mixture weights & and narrows down to family further denoted as Jr. A composition of weiihtt blocks is thus 83. In a sense, choosing templates F, we introduce small function family that can be parameterized by just handful of real-valued parameters E. This perspective shows a connection of the proposed approach to other approaches parameterizing neural network weights like, for example, HYPERNETWORKS (Ha et al. 2017).
4 APPLICATIONS AND EXPERIMENTAL RESULTS
4.1 ISOMETRIC MODEL DETAILS
Before discussing our experimental results, we first need to briefly describe the architecture of the Isometric network used as basis of these experiments, As described in Section 3.1 our implemen- tation of the modular Isometric network contains the following four components (see Appendix [B.] for details): (a) input adapter, nonlincar convolution with the kernel and the stride of the same size (typically 4 to 16), which effectively downsamples the image while increasing the number of channels in the tensor to 40g, where @ is the model depth multiplier controlling the complexity of the network; (b) model block, a typical MobileNetV3 (Howard et al.] 2019), or Isometric model residual block (Sandler et al., 2019) with the "expansion", depthwise and "projection" convolutions and squeeze-and-excite block (Hu et al. (020); (c) output adapter, a sequence of two nonlinear kernel-l convolutions with an average pooling layer in between and finally (d) logits layer, a fully-connecter layer producing model predictions. In the following, we will rely on two specific designs: (a) larger design more suitable for complex tasks that uses 40-channel tensors (or o = 1) and the embedding size of 1280 (as produced by the output adapter) and (b) smaller design typically using 24-channel tensors and the embedding size of 256.
In our implementation of networks with mixture modules, we used mixture weights & E RK for mixing all variables appearing in every component of the block except for batch normalization (BN) parameters: this included all convolutional weights and biases, and all components of the squeeze- and-excite block. In some experiments, we also used the same & to produce a linear combination of batch normalization 3 and " parameters (but never moving mean variables). In the following, will refer to such experiments as experiments with "mixed batch normalization parameters".
Notice that generating block parameters from templates and mixture weights can be expensive for large number of templates in F. However, since in this work we perform this calculation for an entire
'i..., a state with max; & Rs for most layers f We used the number of channels divisible by 8 for training efficiency.
4

---

Under review as a conference paper at ICLR 2021
Table I: IMAGENET results, comparing performance of different module sequences. We use 8 and 16-layer isometric models from Sandler et al. (2019). For our method we use the same models, but the modules are arranged into 32 and 48 layers as described in Section 4.2 The models are of the same size as baseline.
Isometric-8
8/16 model
Isometric-16
16/48 model
68.3
70.6 (+2.3%
71.7
72.8 (+1.1%%
batch and not for individual samples, this procedure did not introduce significant computational overhcad.
4.2 LEARNING A SINGLE TASK
We began our study of networks with mixture weights by training these models on individual super- vised tasks. In our initial experiments, we studied models with 4 to 32 layers and 2 to 16 templates forming F. We trained these models on the CIFAR-100 dataset using 24-channel tensors and smaller output adapter (see Appendix B.I.
When training model on single task, we observe that even in the absence of regularization, the mixture weights E frequently approach collection of one-hot vectors, i.c., (5x; = for most layers f. Discovered model signatures (with one regularized example having a signature (1, 2, 3, 3,3, 3, 4, 4,4, 4, 4, 4)) are discussed in Appendix C in detail. After studying multiple such signatures, observed that they tend repeat the same modules in contiguous uninterrupted quences and the number of repetitions grows towards the end of the model. In other words, early modules use fewer repetitions. We call such pattern progressive.
Once the optimal signature is identified, we investigated whether this particular model architecture truly optimal for solving particular task. We compared networks with mixture weights those with heiined signatures of three types: (a) sequential signatures (Fk/ ..... (fLijk) repeating each of K templates L/ K times, where L is the number of layers; (b) cyclic signatures corresponding to (ʃx o f1) L/K and (c) random signatures. Our experimental results (for details see Appendix () appear to indicate that mixture-weight results are comparable to those obtained with a sequential signature and overall outperform results for other considered model signatures. However, it remains an open question whether progressive signatures are actually optimal, or are an artifact of the modular network training.
In Table we show our results of improving IMAGENET accuracy by increasing the number of layers while keeping the number of parameters intact. Due to hardware and time limitations we only experiment with progressive pattern and leave more detailed study for future work. More details on IMAGENET architecture included in the Appendix.
4.3 MULTI-TASK AND TRANSFER LEARNING
Multi-task learning Another potential application of compositional models is in multi-tat task learn- ing, where expect the computation be partially shared across different tasks. In this application, each task T, uses its own private set of mixture weights E,, but the template blocks F and input, output adapters are shared across all tasks. As the number of tasks grows, the total number of blocks in F should also be scaled up to achieve sufficiently high accuracy on all {Tu). Also note that modular networks provide convenient tool for training arguments like those, for example, used in basic visual question answering modules (finding specific objects, transforming and filtering attention regions, etc.) (Hu et al. 2018 2017a).
of tasks parameterized by additional
Notice that in its current form, this approach is generalization of ADASHARE (un et al. 2019). in which reused modules always retain their relative order with respect to each othe Since multi- task training is not the primary application explored in this paper and the direct comparison is not possible without re-implementing the method, we do not compare our results with ADASHARE.
While the tasks } could be very diverse and along with conventional supervised tasks could also include unsupervised tasks, or even generative and image-to-image translation objectives, here we
and cannot he reused later in the same network
5

---

Under review as a conference paper at ICLR 2021
Dataset
Original
Boundary
Blur
2
8
E-MNIST
ImageNet
P
8
14
10
Layers
Places365
K-MNIST
Layers
16
75
24
4
Sun397
MNIST
/
1
0
1
32
Templates
Templates
(a)
(b)
Figure 2: Self-organization of modules for multi-task and domain domain-adaption. In (a) co-training natural image datasets with 32 modules (CIFAR- 100 and FOOD 1 not shown). first 5 layers were reused across all learned models (shown with dashed rectangles). In (b) co-training modules tasks: 3 handwritten digit datasets EMNIST, MNIST and KMNIST with the original images and two image augmentations (boundary extraction and blurring). The model learned use different input transformation for each image augmentation, while sharing the top layers for each task. On the other hand, for each fixed augmentation, the model learned share first few layers across all different tasks.
will only consider a collection image classification tasks. For our experiments, we chose a col- lection of natural image datasets including: IMAGENET, CIFAR- 100, PLACES365, SUN397 and FOOD101 101, Here we used model depth multiplier of 1 and larger output adapter (see Section We compared multiple different ways of co-training different classification tasks: baseline model with or without per-task batch normalization 3 and variables (see linear patches in Mudrakartal (2019)) and (b) model with mixture-weights with or without per-task batch normalization variables.
In experiments summarized in Appendix D noticed that the mixture-weight model with K = 32 templates achieved a lower overall cross-entropy loss and thus fitted training data better than the baseline models. This did translate superior IMAGENET performance, but in some cases resulted in worse accuracies for simpler datasets, on which overfitting seemed to occur.
Solf-Organization of Computational Blocks Perhaps one the most interesting aspects of net- works trained with mixture modules that they essentially involve self-organization of atomic com- putational blocks. The properties of emergent block configurations are insightful both when we train a model on single task and when performing multi-tase learning.
In the case of multi-task learning, we discover that the location of blocks re-used across tasks agrees with our intuition regarding the nature of the task similarity. Specifically, if all tasks operate on similar domain, first few layers of all models appear to share the same mixture weights (see Figure a)) thus sharing early-stage image processing. For similar tasks operating on different image domains, we see the opposite situation (see Figure 2))) where the first few layers differ, but the late sample processing is shared across all tasks performing inference on the same dataset (EMNIST, MNIST and KMNIST in experiments). In this case, first few layers essentially adapt network input to a particular common format shared across all related tasks and processed by the following layers.

---

Under review as a conference paper at ICLR 2021
Table 2: Transfer learning results from model trained on multiple initial datasets (IMAGENET, CIFAR- 1RR-000, 00, PLACES365, SUN397 and FOOD101) onto new datasets. For baseline column we train multi-task model following Mudrakarta et al. (2019), with shared backbone and per-task logits and batch normalization parameters. We then transfer to the target datasets by fine-tuning logits and batch normalization parameters. Other columns show results obtained by our method, where we use equivalent architecture, but with mixture weights both for initial model and each transfer task. Each column shows results with mixture coefficients initialized from the corresponding training task. Our method adds less than 500 additional parameters for each transfer task.
Dataset AIRCRAFT
Baseline
IMAGENET
CIFAR
FOOD
43.3
43.9
42.6
39.4
CARS
49.2
51.0
46.7
42.2
EMNIST
85.5
85.0
85.2
83.6
STANFORD DOGS
55.6
57.6
42.4
36.5
Transfer learning A set of blocks F pretrained on collection of tasks can later be used for transferring extracted knowledge onto a new task. Adjusting mixture weights to form a new model potentially gives us more flexibility than tuning logits or batch normalization parameters alone.
In our experiments, we used models pretrained on 5 datasets: IMAGENET, CIFAR-100 PLACES365, SUN397 and FOOD101 (see Appendix D). We then used learned collection of tem- plates F for transferring knowledge to different new tasks such as STANFORD DOGS, AIRCRAFT, CARS and EMNIST. We compared our results with the baseline obtained by fine-tuning logits and BN parameters of a baseline model trained with a common model backhone, but per-task logit layers and BN parameters ("Baseline with patches" in Table 7 Proper initialization of mixture weights =\t proved to be extremely important for achieving best performance. Model initialized with random E performed significantly worse than the models initialized with E corresponding to one of the datasets in the original mixture. In the following, we will denote by E, a set of mixture weights used for a task v in the original mixture- module network trained on a collection of datasets. Initializ- ing the model at some particular E,, we compared model performance with and without fine-tuning (fine-tuning BN parameters and logits in both cases).
The final results presented in Table 2 Experimental observations suggest that a proper choice of the dataset v for E E, is extremely important: using E\t values from the FOOD model resulted in model that achieved the same > 99% training accuracy, but would frequently lead to 10% - 20% smaller validation accuracy. We also observe that the validation accuracy improvement in the final model can be partly attributed to higher validation accuracy the chosen model E, and partly to the result of fine-tuning this E as Ey value.
4.4 DOMAIN ADAPTATION
In this section, we explore the use of compositional models for unsupervised multi-source domain adaptation which aims to minimize the domain shift between multiple labeled source domains and an unlabeled target domain. To our knowledge, we are the first to show that it is possible to per- form domain adaptation with modular networks by simply changing the order of modules in the network. To illustrate the ability of compositional networks to perform distribution alignment, we adopted two existing unsupervised domain adaptation methods: Adversarial Diseriminative Domain Adaptation (A (ADDA) by Tyzznng et al. (2017) and Moment Matching for Multi-Source Domain Adap- tation (M3SDA) by Peng et al. (2019). We performed experiments with two sets of datasets: digits datasets (MNIST, corrupted MNIST, SVHN and USPS) and DomainNet dataset introduced by Peng al. (2019). The code used for the domain adaptation experiments can be found on our project page (to be added if accepted). The details of implementation of the compositional domain adaptation and a review of related work on domain adaptation can be found in the Appendix
Multi-source adversarial classification In the compositional adversarial classification method, the shared classifier is trained to perform accurate classification on all source datasets. Additionally, domain discriminator is trained to distinguish between the late features of source and target do- mains (in our implementation, the output of the layer before the output adapter). As in the multitask learning scenario, the model contains separate mixture weights for each domain; the mixture weights
7

---

Under review as a conference paper at ICLR 2021
Table 3: Multi-source domain adaptation target test accuracy results with compositional models on Digits domains (MNIST, corrupted MNIST (shear), corrupted MNIST (scale), corrupted MNIST (shot noise). SVHN + USPS) and DomainNet domains (clipart, infograph, quickdraw, painting, real + sketch). Baseline methods: Adversarial Discriminative method and Multi-Source Moment Matching method. We pretrained the baseline models on source domains and fine-tuned only the input adapter (IA) containing & 1000 parameters. We applied the same domain adaptation tech- niques for our conversational model and fine-tuned only mixture weights (MW) consisting of 64 parameters.
Method
parameters
Digits
DomainNet
Source-only
85.8
13.6
Adversarial
680
93.4
14.5
Compositional Adversarial (our)
64
92.2
13.8
Moment Matching
680
94.2
15.6
Compositional Moment Matching (our)
64
94.4
14.4
of source domains are trained along with templates to minimize the classification loss, while the tar- get domain mixture weights trained to "fool" the domain discriminator by aligning the target feature with the source features.
Moment Matching As in the original paper introduced by Peng et al. (2119), the method performs domain adaptation by minimizing the distance between the source and target domain statistics (more details can be found in the original paper). In the compositional moment matching setting, only tar- get domain mixture weights are trained to perform moment matching. addition to that, the model has two logits layer instead of one which are used to perform minmax discrepancy optimization (see Peng et al. (2019) for more detail). the compositional case, only the mixture weights are affected by the discrepancy minimization.
Results As we see in the Table 3] fine-tuning only the mixture weights for the domain adap- tation objective improves the classification accuracy on target domain. To compare the results of conversational domain adaptation, we trained the adversarial and moment matching models with isometric model without weight sharing as a backbone, and for fair comparison we only fine-tuned the input adapter (Saito et al. (2019) point out that finetuning the first layers is appropriate for the case when the domain shift is mostly in the low-level visual features). The results show that although only fine-tuning mixture weights of the conversational model does not result in the same adaptation efficiency as the baseline probably due to the fact that mixture weights contain more than ten times parameters than the input adapter, such approach can still significantly increase the target accuracy.
5 DISCUSSION
In this paper, introduced novel approach to modular neural networks that decouples modules from their position in the network thus allowing reuse of modules not only across tasks, but also across layers. Our method beats the baseline multitask learning with additional templates and achieves better transfer learning results on most of the datasets we explored, which we partly at- tribute to less aggressive parameter over-sharing and partly the ability of model to fine-tune the module order itself. We then apply our method to training parameter-efficient networks and show that we can scale SOTA IMAGENET classification models achieve higher accuracy while keeping the number of parameters fixed. We also show that simple fine-tuning of mixture weights allows unsupervised domain adaptation improving model's performance on target domain by large margin.
Since, by design, the compositional models divides the task into simple subtasks, it can provide greater understanding of the computation process. For example, we notice that co-training multiple tasks on the same image domain, all trained models share early image processing modules. At the same time, when the dataset contains multiple image domains of the otherwise identical task, early layers bring network activations to the common format, while the final stage of the computation defined by the sequence of participating modules shared.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39,48, 2016.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. arXiv preprint ar (.02 2016.
Emmanuel Bengio. On reinforcement learning for deep neural architectures. 2016.
Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7354-7362, 2019.
Dario Floreano, Peter Dür, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary intelligence, 1(1):47-62, 62, 2008.
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. arXiv preprint arXiv:2006.00 01895, 2020.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. In 5th International Confer- ence on Leurning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference net forum?io
Track Proceedings. OpenReview.net, 2017. URL
rkpACellx
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Sebastian Herzog, Christian Tetzlaff, and Florentin Wörgötter. Evolving artificial neural networks with feedback. Neural Networks, 123: 162, 2020. doi: 10.1016/).euunet2011 12.004. URL
111.0...1111110 www.eeuneee...m 201. 12.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998, 2018.
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Search- ing for mobilenetv3. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2. 2019, pp. 1324-1324. IEEE, 2019. doi: doi org 10. 1109/ ICCV 2019.
10.1109/CCC22211.000010
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. saueeze-and-exxitaie networks. IEEE Trans. Pattern Anal. Mach. Intell., 42(8) 1-2023, 2020. doi: 10. 1091////PPM...019
2913372. URL U htt.t:ttp////tm
.rrg/ 0 1 09/ TPAMT. 2019 9 2913372
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 804-813, 2017a.
Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. Modeling rela- tionships in referential expressions with compositional modular networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1115-1124, 2017b.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pp. 53-69, 2018
Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. http: org. abs/ 1511 08228

---

Under review as a conference paper at ICLR 2021
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral streams execution of core object recognition behavior. Nature neuroscience, 22(6))974-983, 2019.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. In Advances in Neural Information Processing Systems, pp. 2408 2418, 2018a.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decom- pose neural computation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Gameet (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 2414-2423, 2018b. 7508 modular networks- learning- decempose neural-
URL http::/ nips. wtpp.cccpppprr
Ming-Yi Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pp. 2208-2217, 2017.
Jiaqi Ma, Zhe Zhao, Jilin Chen, Ang Li, Lichan Hong, and Ed H Chi. Snr: Sub-network routing for flexible parameter sharing in multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 216-223, 2019.
PokissK Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multi- ple tasks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1860, 2019.
Krzysztof Maziarz, Ef Kokiopoulou, Andrea Gesmundo, Luciano Shaiz, Gabor Bartok, and Jesse Berent. Gumbel-matrix routing for flexible multi-task learning. arXiv preprint arXiv:1910.04913 04915,,,,, 2019.
Risto Mikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293 312. Elsevier, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994 4003, 2016
Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew G. Howard. K for the price of 1: Parameter-efficient multi-task and transfer learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. (RLL:tttps://ppennee
/ d=BBxxEhhCFO
Alejandro Newell, Lu Jiang, Chong Wang. Li-Jia Li, and Jia Deng. Feature partitioning for efficient multi-task architectures. arXiv preprint arXiv: /tt.:/9888.999. 04339, 2019.
Vardaan Pahuja. Jie Fu, Sarath Chandar, and Christopher Pal. Structure learning for neural module networks. arXiv preprint arXiv: 1905. / 1532, 2019.
Xingchao Peng and Kate Saenko. Synthetic to real adaptation with generative correlation alignment networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp- 1982-1991. IEEF, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision pp. 1406-1415, 2019.
10

---

Under review as a conference paper at ICLR 2021
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Mare' Aurelio Ranzato. Task-driven modular networks for zero-shot compositional learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 3593-3602, 2019.
Ettore Randazzo, Alexander Mordvintsev, Eyvind Niklasson, Michael Levin, and Sam Grey-
danus. Self-classifying mnist digits. Distill, 2020. doi:
https:/ddiiiilllpp//22200s//oogo
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, Atticus Geiger, Lauri Karttunen, Joshua D Greene, Dan Jurafsky, and Christopher Potts. Dispatched routing networks. Technical report, Technical Report Stanford AI Lab, NLP Group Tech Report 2019-1, Stanford 2019.
Kuniaki Saito. Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong- distribution align- ment for adaptive object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6956 6965, 2019.
Mark Sandler, Jonathan Baccash, Andrey Zhmoginov, and Andrew Howard. Non-discriminative data weak model? on the relative importance of data and model resolution. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea
(South), October 27-28, 2019, pp. 1036-1044, IBEE, 2019. doi: 10.109/1CCVV.2219.0011 URL https: //oo. org /10 wwt.:11111///// 201 00133
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Thirtieth AAAI Conference on Artificial Intelligence, 2016
Ximeng Sun, Rameswar Panda, and Rogerio Feris. Adashare: Learning what to share for efficient deep multi-task learning. arXiv preprint arXiv: 1.t.:9911..22,,, /2423, 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:
2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
Daan Wierstra, Faustino Gomez, and Jürgen Schmidhuber. Modeling systems with internal state using evolino. In Proceedings of the 7th annual conference Genetic and evolutionary compu- tation, pp. 1795-1802, 2005.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8817-8826, 2018.
Siming Yan, Xuyang Fang, Bowen Xiao, Harold Rockwell, Yimeng Zhang, and Tai Sing Lee. Recurrent feedback improves feedforward representations in deep neural networks. CoRR, 10489, 2019.
10489
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz Central moment discrepancy (cmd) for domain-invariant representation learn- ing. arXiv preprint arXiv:1702.00 (et.11702..881111 2017.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imagee translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.01178. 2016.
11


Reviews:
[{"review_id": "ciqKYsh5em", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==========\n\nSummary: \nThis paper proposes a modular neural network architecture that consists of module templates that can be invoked repeatedly in different layers and combined using mixture weights.\nIt can be trained end-to-end by linearly combining parameters from different modules. It shows promising results on multiple applications such as transfer learning and domain adaptation\n\n==========\n\nStrength:\n\n- Modular architecture allows parameter efficient and interpretable models and I believe this is an important direction to explore for many applications, including the applications considered in this paper.\n\n- Proposes a general formulation where module templates can be reused multiple times in different layers and can be trained end-to-end by softly combining module parameters.\n\n- Self-organization of modules and how they are reused, as shown in Figure 2, is interesting and can be used to interpret what each module is learning.\n\n- Method is simple, clear, and well written.\n\n==========\n\nWeakeness:\n\n- Sec 4.2 & appendix C. Quantitative results do not sufficiently show if the proposed method is actually better. The accuracies are pretty close to each other and the numbers are from a single run so they are not statistically convincing.\n\n- Sec 4.3 multi-task learning & appendix D, Section 4.4. same as above.\n\n- Table 2. It says results are also from Places365 and sun397, but they are not shown in the table\n\n- Section 4.3. It requires a manual choice of initial template weights that is learned from some dataset, and otherwise, it can result in worse performance. This is a strong limiting factor and if the model is able to utilize the template modules, it should be able to learn the weights. \n\n==========\n\nThis paper proposes an interesting way of composing neural modules for a number of applications, but the experimental results do not sufficiently demonstrate the advantage of using the proposed approach. \nI believe more exploration in this direction and making the results concrete by showing how this model can be better than baseline models would make the paper much stronger. \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XLoG_7KB95E", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to design network architectures as a combination of network components. Their idea consists in imposing an architecture that is a sequence of identical network blocks, and learn a series of templates, which provide an instantiation of model weights for one such network block. Model weights are then estimated for a specific task as a linear combination of a set of template weights. \n\n\nSTRENGTHS\n\nThe idea of using modular networks to share knowledge across tasks and domain is appealing, as is the potential interpretability with regards to which templates are preferred.   \nThe concept of weight templates is interesting, as it affords to share knowledge implicitly, and provides a flexible setting for network construction.   \nAuthors evaluate their approach on a very large set of problems and datasets, and provide promising performance. The experiment studying similarities across network signature with respect to the problem setting is particularly interesting. \nThe domain adaptation setting, with a significantly lower number of parameters is additionally quite promising. \n\n\nWEAKNESSES\n\n-One of the main limitations of the work is the unclear or incremental novelty with respect to other works performing soft weight sharing, and that implement modular networks techniques. Differences are not clearly stated, in particular with respect to the works of Ma et al, Maziarz et al (linear combination of module candidates), Kirsch et al (probabilistic sampling of modules), and Sun et al (shared modules for domain adaptation). A clear list of contributions as well as a dedicated section analysing main innovations with respect to these highly similar works would be very beneficial.  \nIn particular, a claimed innovation/main difference in section 3.2 is the use of a mixture of templates to compute the weights of a specific network layer. However in practice, mixture weights are regularised to be closest to one-hot encodings, leading to a composition of blocks rather than a composition of mixtures of blocks.  Can authors please comment on this point?\n\n-The regularisation strategy is an important aspect of the model and its performance. Details regarding the adopted method and proposed solution should be present in the main method description, and the two strategies adopted (vs no regularisation) should be evaluated in a dedicated ablation sections. Can authors comment on whether templates are regularised to comprise different sets of weights, and whether the problem of templates being too similar arises in practice? \n\n-Authors have made a lot of effort towards evaluating their approach in different scenarios and settings. Unfortunately, the experimental evaluation section needs substantial modifications to provide thorough and reproducible evaluation of the work. \nThe first issue relates to the first two set of experiments, (single task and multi-task) for which only vague descriptions are provided: experimental setting, datasets, network architectures (number of layers, number of templates)…Most of these important details are either absent from the paper or described only in the supplementary material. This makes it very challenging to understand exactly the experimental setting, and further more concerning is the fact that results are only provided and discussed in the supplementary material. In particular, while the datasets used are, for the most part, very common, providing a description in the supplementary material is important for reproducibility (regardless of code sharing).  \nThe issue is present throughout the whole experimental section, with key details missing from the experiment descriptions (notably number of templates vs number of layers), and some important details are only provided in table captions. \n\n-In addition, the evaluation is missing important comparisons to closest works. Problems like knowledge transfer and multi-task learning are evaluated using novel settings that differ from standard benchmarks in the fields, preventing from providing context and directly comparing to similar approaches. For example, the multi-task learning section claims unavailability to compare to similar work Adashare, despite the fact that this work has been evaluated on standard, publicly available benchmarks. Can the authors justify this decision? Why not use standard benchmarks to evaluate the proposed method?    \n\n-The domain adaptation/transfer learning sections are interesting, providing promising results. However, wouldn’t it be more advantageous to learn templates across multiple datasets, which would allow to extract common knowledge and facilitate transfer to a new dataset/domain?  \n\nRECOMMENDATION\n\nIn summary, the idea of learning modular, compositional architectures, with a set of pre-trained templates that can be reused in different problem settings is quite exciting. \nHowever, the paper in its current form suffer from a set of important flaws, and would need substantial modifications. I would recommend 2 key changes:  \n\n1-\tA clarification of novelty/contributions, in particular with respect to pre-existing modular architecture works and soft sharing methods. I would also recommend focusing a little more on the regularisation strategy.   \n2-\tRevamping the evaluation section in a significant way: a) first by providing a setting where the approach can be compared to similar works on a standard benchmark, b) providing a clear description of a set of experiments/problem settings, with all main results and interpretation of results available in the main paper; c) providing ablation experiments, in particular with regards to the regularisation strategy ; c) moving experiments whose results do not fit in the main paper to the appendix, where all details and results can be discussed.   \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P6cxbAri23Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper considers “modular multi-task learning” where parameters in each layer/task are generated as a (layer/task-specific) linear (mixture) combination of a common pool of parameters. Exploring this idea, several observations are made: (1) single task isometric model performance on ImageNet can be improved, (2) Multi-task learning is supported and parameter sharing (selecting same mixture components) emerges in early layers, with specialisation emerging in later layers. With multi-domain learning, the opposite effect is achieved with domain-wise specificity arising in earlier layers, and sharing in later layers. (3) Parameter-efficient transfer learning is supported by fine-tuning the task-specific weights for new tasks. (4) Parameter-efficient domain adaptation is supported by optimising the task-specific weights for new domains. \n\nOverall impression: It’s a neat exploration of parameter sharing in isometric networks. However the novelty is over claimed/prior literature missed, and the experiments are not good enough to make up for this limited novelty with empirical insights discovered or SotA performance reached.\n\nStrengths: \n+ Overall the idea is reasonable, and the results make sense. \n+ By taking the idea of isometric networks seriously, this paper explores a richer space of parameter sharing than some existing papers that use the same idea. \n+ By using soft modularisation, the framework is easier to train than other methods that need to use gradient estimators for hard sharing.\n+ It’s good to have a framework with flexibility for single-task, multi-task, and cross-task/domain transfer.\n\nWeaknesses: \n1. Novelty/Related work. This paper makes a stronger claim about novelty than is justified. \n- The proposed method is a special case of tensor-factorisation based parameter sharing that has already been widely studied in several papers. Specifically, where a vanilla baseline uses a parameter set that can be described as a tensor X of size (W*H*C1*C2*L)  (here L = layers for STL or tasks/domains for MTL), then the proposed method instead uses a parameter set (Y,Z) where X~=Y*Z and Y is a (W*H*C1*C2*K) tensor of K “shared modules” and Z is a (K*L) tensor of “task/layer-specific mixture parameters”, and “*” is the corresponding tensor contraction. This kind setting has already been widely analysed in papers such as [A,B] among others.  The minor difference is the softmax constraint on the factor Z, which gives it the specific “mixture” interpretation, but this was already studied in related methods such as [F].  Parametrically, it also seems to be a special case of methods such as [G].\n- With regard to observations about changes in the per-layer sharing strength in multi-task/multi-domain learning, similar observations were made in [A] among others.\n- With regard to transfer learning by re-training task-specific factors in such a factorisation setup, this was done by lots of papers including [B,C,D].\n- With regard to cross-layer parameter sharing specifically, this is somewhat less commonly studied, but was included in [B,G].\n\n2. Evaluation\n- Sec 4.2. It’s neat that increasing layers with a fixed number of parameters can improve ImageNet performance. But this is not quite unprecedented, and it’s not compared with other related approaches to achieving the same such as HyperNets or [B,G] etc. So we don’t know how impressed to be.\n- Sec 4.3. Fig 2 is neat. But again, it’s not quantitatively or qualitatively compared to the literature on soft-sharing for deep MTL, which is now very large across both the ML and CVPR communities. \n- Sec 4.3. Tab 2 is also not compared to any of the numerous alternative methods for transfer learning out there. Neither in the broader TL field where there are many alternatives, nor among highly related methods such as [B,C,E,etc]\n\n3. Insight. Prior papers that have placed an emphasis on modularity usually resort to significantly more complex optimization strategies [G,etc]. But such papers often start by mentioning and discarding variants of the simple \"soft\" weighting strategy used here before moving on to more complex setups, e.g., for optimising \"hard\" module selection. It’s good that this paper makes modularity work with a very easy setup, but it would be good to have more insight on what are the key tricks that are required to achieve it with simple soft blending and optimisation. Presumably something “obvious” was missed by prior work that resorted to much more complex setups. \n\n\n[A] ICLR’17, “Deep Multi-Task Representation Learning: A Tensor Factorisation Approach”\n[B] arXiv’19/AAAI’20 “Incremental Multi-domain Learning with Network Latent Tensor Factorization”.\n[C] IJCAI’17, \"Tensor Based Knowledge Transfer Across Skill Categories for Robot Control”\n[D] IEEE Trans ASL’15, ”Cluster Adaptive Training for Deep Neural Network Based Acoustic Model” \n[E] NeurIPS’17, “Learning multiple visual domains with residual adapters”\n[F] ICLR’18, “Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering”\n[G] NeurIPS’19, “Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains”\n\nA. Minor: \n- Reference Kirsch 2018 is repeated.\n- Fig 2 caption calls it “domain adaptation”, but to my understanding this experiment might better be called multi-domain learning, than domain-adaptation per-se. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8gekuNA9vA1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method of composing stacked neural network blocks by linearly combining module \"template\" weights.  The work extends \"isometric networks\", in which each block in the stack has the same operational structure, by parameterizing each block using a mixture of the weights from a bank of K blocks (\"templates\").  In multitask learning experiments, the mixtures naturally learn to share common components between tasks, while learning task-specific components where needed.  Further experiments describe behavior of the system applied to transfer learning and domain adaptation scenarios.\n\nThe most interesting findings are in the naturally arising patterns of template sharing:  When these occur, and why, including the occurrence of the \"progressive\" pattern of repeated blocks.  I feel this can be even further developed, and tied to existing architectures in the discussion -- for example, resnet architectures are progressive, and this model learns progressive blocks on its own:  this suggests the model converged to a similar structure, reinforcing the importance of this trait in classification deep nets.\n\nThe other two applications, transfer learning and domain adaptation, seem underdeveloped.  The transfer learning experiment compares only against a baseline model with similarly constructed initialization but fewer transfer-learned parameters, so it's hard to know what to take away from this (see also comments and questions below).  Even though the mixture weights are relatively few parameters, it's not clear that these are usefully fewer in most settings:  full\n\nThe main multitask investigation has much interesting discussion relegated to supplementary materials.  The appendices should be in the main PDF (I almost didn't see them in the supplemental zip file; particularly because they are referred to as \"appendices\" but were not appended).  I also think some of this material, particularly much of appendices C and D, would be better suited to the main text in order to flesh out the multitask behavior.\n\nOverall I find the approach and the investigation of multitask training very interesting.  However, the text also lacks clear descriptions of many of the setups and comparisons, and sacrifices a deeper dive into the multitask behavior and ties to existing architectures, in order to include additional tasks that do not have clear takeaways.\n\n\n\n\nAdditional comments and questions:\n\n- at inference time, it seems one could construct model weights \\theta using the mixture and use these directly, discarding the templates.  This would probably be more efficient for most scenarios; it would be good add a comment about this, I don't see it explicitly mentioned.\n\n- p.3: what is the definition of m_l ?\n\n- fig 2a:  how are the columns sorted ?\n\n- fig 2b:  I slightly disagree with the last sentence of the caption, \"for each augmentation ... learned to share first few layers across all different tasks\" --- in fact, this looks to be the case only for the \"boundary\" augmentation, not each of the augmentations.  Nevertheless, the fact that it happened in this case seems significant, and the claim could be revised to e.g. \"in the case of the boundary augmentation ...\".\n\n- how quickly do the mixtures converge to mostly 1-hot, how do they evolve over the course of training?\n\n- table 1:  I don't understand what are the \"8/16\" and \"16/48\" models --- what do the 8 and 16 in \"8/16\" correspond to?  sec 4.2 doesn't have this notation\n\n- fig 2:  why are cifar10 and food101 not shown?  they would be interesting to see, e.g. if first layers are different due to different resolution, for example\n\n- some mixtures, especially learned via sgd, can suffer from partial collapse where just one or a few components that happen to perform slightly better at the start of training \"take over\" and starve out potential use of remaining components, due to a self-reinforcing loop of component model weights being updated to perform better, leading to its mixture coefficient increasing, which leads to further model improvement, larger coefficient, etc.  Have you seen this model suffer from a problem like this?\n\n- For transfer learning experiment:  This can have more comparisons.  I think the baseline uses the same initial model (using \\xi_\\nu) as imagenet, and fine-tunes just the batchnorm and last logits layers on the new dataset.  This one point of comparison, but I'm not convinced it is the strongest baseline --- one could also train all model weights, for example.\n\n- Are there any benefits to constraining weight updates to just the mixture coefficients \\xi compared to training all weights?  Yet another strategy could be to initialize model weights using the best initial \\xi, but instead of learning a new mixture, construct the model with mixed weights \\theta = sum(\\xi_i \\theta_i), and fine-tune the model weights \\theta of each layer directly on the new dataset (eliminating the mixture entirely at this point).\n\n- Are there any transfer datasets for which a different initial model choice performs better than imagenet?  What about cifar10, is it better using imagenet or cifar100?\n\n\n- Domain adaptation experiment:  I find this experiment very interesting, that such similar performance can be obtained just by tuning the mixtures.  I think it could be developed further:  Are the templates that are changed more at the first layers or at all layers?  And how does this compare to learning all weights instead of just the mixture?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for modular multi-task learning. All the reviewers believe the goals are appealing and the idea is reasonable. However, R2 and R4 raise concerns with respect to novelty. There are also strong concerns regarding experiments. The concerns vary from reproducibility to small improvements and right baselines. The rebuttal fails to provide any new experiments or handle the reviewer concerns. All reviewers and AC agree that paper is not yet ready for publication.
Decision: Reject