Feature Partitioning for Efficient Multi-Task Architectures
Authors: Alejandro Newell
Lu Jiang
Chong Wang
Li-Jia Li
Jia Deng
Venue:  (2020)
Keywords: multi-task learning
neural architecture search
multi-task architecture search

Abstract: Multi-task learning promises to use less data, parameters, and time than training separate single-task models. But realizing these benefits in practice is challenging. In particular, it is difficult to define a suitable architecture that has enough capacity to support many tasks while not requiring excessive compute for each individual task. There are difficult trade-offs when deciding how to allocate parameters and layers across a large set of tasks. To address this, we propose a method for automatically searching over multi-task architectures that accounts for resource constraints. We define a parameterization of feature sharing strategies for effective coverage and sampling of architectures. We also present a method for quick evaluation of such architectures with feature distillation. Together these contributions allow us to quickly optimize for parameter-efficient multi-task models. We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify architectures that effectively make trade-offs between task resource requirements while maintaining a high level of final performance.

Under review as a conference paper at ICLR 2020
FEATURE PARTITIONING FOR EFFICIENT MULTI-TASK ARCHITECTURES
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-task learning promises to less data, parameters, and time than training separate single-task models. But realizing these benefits in practice is challeng- ing. In particular, capacity to support many tasks while not requiring excessive compute for each individual task. There are difficult trade-offs when deciding how allocate pa- rameters and layers across a large set of tasks. To address this, we propose a method for automatically searching over multi-task task architectures that accounts for resource constraints. We define parameterization of feature sharing strategies for effective coverage and sampling of architectures. We also present method for quick evaluation of such architectures with feature distillation. Together these contributions allow us to quickly optimize for parameter-efficien multi-task mod- automatically
difficult to define a suitable architecture that has enough
els. We benchmark on Visual Decathlon, demonstrating that search for and identify architectures that effectively make trade-offs between task resource requirements while maintaining high level of final performance.
INTRODUCTION
Multi-task learning allows models to leverage similarities across tasks and avoid overfitting to the particular features of any one task (Caruana 1997 Zamir et al. 2018). This can result better gen- eralization and more robust feature representations. While this makes multi-task learning appealing for its potential performance improvements, there are also benefits in terms of resource efficiency. Training a multi-task model should require less data, fewer training iterations, and fewer total pa- rameters than training an equivalent set of task-specific models. this work we investigate how to automatically search over high performing multi-task architectures while taking such resource constraints into account.
Finding architectures that offer the best accuracy possible given particular resource constraints is nontrivial. There are subtle trade-offs in performance when increasing or reducing use of parameters and operations. Furthermore, with multiple tasks, must take into account the impact of shared operations. There is a large space of options for tweaking such architectures, in fact so large that it is difficult to tune optimal configuration manually. Neural architecture search (NAS) allows researchers to automatically search for models that offer the best performance trade-offs relative to some metric efficiency.
Here we define multi-task architecture as a single network that supports separate outputs for mul- tiple tasks. These outputs are produced by unique execution paths through the model. In neural network, such path is made up of a subset of the total nodes and operations in the model. This subset may or may not overlap with those of other tasks. During inference, unused parts of the network can be ignored by either pruning out nodes or zeroing out their activations (Figure [ Such architectures mean improved parameter efficiency because redundant operations and features can be consolidated and shared across set of tasks.
We seek to optimize for the computational efficiency multi-task architectures by finding models that perform as well as possible while reducing average node use per task. Different tasks will re- quire different capacities to do well, so reducing average use requires effectively identifying which tasks will ask more of the model and which tasks perform well with less. In addition, perfor- mance is affected by how nodes are shared across tasks. is unclear when allocating resources whether sets of tasks would benefit from sharing parameters or would instead interfere.

---

Under review as a conference paper at ICLR 2020
0000000000ooo
Feature partitioning be used control how much network capacity is used by tasks, and how much sharing is done across tasks. this work we identify effective partitioning strategies maximize performance while reducing average computation per task.
Figure/ 1:
When searching over architectures, differences resource use can be compared at different levels of granularity. Most existing work in NAS and multi-task learning searches over the allocation and use of entire layers (Zoph & Le 2016: Fernando et al. 2017) Rosenbaum et al., 2017), we instead partition out individual feature channels within layer. This offers greater degree of control over both the computation required by each task and the sharing that takes place between tasks.
The main obstacle address in searching for effective multi-task architectures is the vast number of possibilities for performing feature partitioning as well as the significant amount of computation required evaluate and compare arrangements. A naive brute search over different partitioning strategies is prohibitively expensive. We leverage our knowledge of the search space to explore more effectively. We propose a parameterization of partitioning strategies reduce the size of the search space by eliminating unnecessary redundancies and more compactly expressing the key features that distinguish different architectures.
In addition, the main source of overhead in NAS is evaluation of sampled architectures. It is com- mon to define surrogate operation that can used in place training a full model convergence. Often smaller model will be trained for much shorter number of iterations with the hope that the differences in accuracy that emerge early on correlate with the final performance of the full model. We propose strategy for evaluating multi-task architectures using feature distillation which pro- vides much faster feedback on the effectiveness of a proposed partitioning strategy while correlating well with final validation accuracy.
In this work we provide:
a parameterization that aids automatic architecture search by providing direct and com- pact representation of the space of sharing strategies in multi-task architectures.
an efficient method for evaluating proposed parameterizations using feature distillation to further accelerate the search process.
results Visual Decathlon (Rebuffi al., 2017) demonstrate that search strategy al- lows us to effectively identify trade-off' between parameter use and performance on diverse and challenging image classification datasets.
RELATED WORK
Multi-Task Learning:
There is a wide body of work on multi-task learning spanning vision, lan- guage, and reinforcement learning. The following discussion will center around designing multi-task architectures for deep learning vision (Ruder 2017 Caruana 1997 Zhang Yang 2017). There are many obstacles to overcome in multi-task architecture design, but the most pressing concerns depend largely on how the problem setting has been defined. Two distinguishing factors include:
Are all tasks available at all times or are they presented one after the other? Fixed Learned Strategies: Is uniform strategy applied across tasks or is a task-specific solution learned?
Task Ordering:
The former is important as work in which tasks are presented sequentially must address catastrophic forgetting (French) 1999). This is less of a concern our work as we train on all tasks at once. As for the latter, finding a solution tuned to specific set of tasks requires the same sort of outer-loop optimization seen in neural architecture search (Zoph & Le (016) which time-consuming and expensive. The contributions presented in this work seek make this process manageable.
2

---

Under review as a conference paper at ICLR 2020
Multi-Task Architectures: A strong baseline for multi-task architectures is the use of single shared network (Caruana 1997: Kaiser et al., 2017). Deep networks are overparameterized in such way that the same layers can be applied across different domains while producing features that are useful to different ends. Using a shared architecture is common in reinforcement learning to train single agent to perform many tasks with uniform observation and action space (Espeholt itt al., 2018, Sharma et al., (0177). A common technique to train single shared models well in both reinforcement learning and vision is distillation of multiple models into one (Beaulieu et al. 2018
Yim et al. 2017 Rusu et al. 2015 He et al., 2018 Chou et al. 2018).
In work where tasks are presented sequentially, the focus is often to build on top of an existing network while not disrupting its ability to solve its original task (Mallya & Lazebnik [011 Rebuffi
jet al. 2018, Rosenfeld Tsotsos 2017)- Currently, many methods freeze the network's weights so they are not changed while learning a new task. Examples include masking out specific filter weights (Mallya & Lazebnik, 2018) or introducing auxiliary layers (Rebuffi et al.) 2018). Another approach is to dynamically expand network with additional capacity for each new task (Yoon et al.) 2018). All of these methods build on top of a fixed model, meaning that new tasks must perform the computation required for the original task well as take additional steps to be task- task-specific.
It is also common to build multi-tast architectures from sets of layers that are run in paralle! (Misra ct al. 2016 Rosenbaum al. 2017 Fernando et al. 2017: Meyerson & Miikkulainen 2017). Cross-stitch networks compute activations for each task as a learned weighted sum across these layers (Misra et al.) 2016). This sort of soft attention over features be seen in other multi-task architecture work as well (Liu et al. 2018 Ruder et al. (017). There are approaches to search over paths through these layers such that each task has unique, optimal execution path (Rosenbaum et al. 2017; Fernando el al.] 2017). Similar to work in single-task NAS, the best path is found by either reinforcement learning or evolutionary algorithms (Fernando et al.] 2017 Liang al. 2018). The optimal trade-offs in parameter sharing may occur at a more fine- fine-graine grained level than entire layers, so instead of working with parallel blocks of layers we divide up individual feature channels.
Neural Architecture Search: There are three main areas in which contributions are made for more effective architecture search: search space, optimization, and sample evaluation.
Search space: With a well-designed search space, it is possible to randomly sample and arrive at high performing solutions (Li & Talwalkar 2019; Liu et al., 2017b). In general, NAS operates in a discrete space where entire layers are included or not. We instead propose continuous search space where slight changes can be made in how resources are allocated across tasks. This allows alternatives for optimization that would not apply in other NAS work.
Optimization: Leading approaches either use reinforcement learning or genetic algorithms for NAS Zoph & Le, 2016; Real et al. 2018 Pham et al.] 2018). This search is difficult and the trade- offs between approaches are unclear (Li & Tar 2019). We test the effectiveness of random
sampling and evolutionary strategies optimization (Mania et al. 2018 Wierstra al., 2008).
Evaluating Samples: Training a model to convergence is time-consuming and resource intensive. It is not realistic to sample thousands of architectures and train them all. Instead one must use a cheaper form of evaluation. Some options include preserving weights across samples for faster training (Pham et al., 2018), successive halving (Kumar et al., 2018), progressive steps to increase complexity (Liu et al. 2017a), . as well as techniques to model the expected performance of sampled architectures (Deng
al.) 2017]] Brock et al., 2017,, Baker et al., (017). It is unclear how well surrogate functions correlate with final model performance (Zela et al. 2018). We investigate the use of distillation for performing this evaluation.
3 MULTI-TASK FEATURE PARTITIONING
Sharing in the context of multi-task architecture search is often adjusted at the level of individual layers (Rosenbaum et al. 2017: Fernando et al. 2017), but given that state-of-the-art architectures work so well across datasets and tasks we choose to preserve the layer level execution path for each task and focus sub-architectural changes that can be made. In this case, by deciding whether or not individual feature channels within each layer are available. This way, every task experiences the exact same organization of layers, but unique calculation of intermediate layer features.
3

---

Under review as a conference paper at ICLR 2020
Concretely, given a feature tensor F E Rexkxw define a binary mask my E {0,1] for each task, and during a forward pass of the network multiply F by my to zero out all channels not associated with a particular task. We further define mask for the backward pass, m, E {0, 1 whose non-zero elements are a subset of the non-zero elements in my. Gradients are calculated as usual through standard backpropagation, but any weights we wish to leave unchanged will have their gradients zeroed out according to ma..
Together, these masks can capture the training dynamics seen in many existing multi-task architec- ture designs (Rosenbaum et al. 2017 [Rebulfi et al. 2018). For example, one can devote an outsized proportion of features task like ImageNet classification, then make these features available dur- ing forward inference on new smaller dataset. A backward mask, mà, can then be defined to ensure that the ImageNet weights remain untouched when finetuning on the new task.
There are a number of advantages to allocating resources at the channel level. There is enough flexi- bility to allow fine-gran one-grained control allotting specific weights to partieular subsets of tasks. And after training, it is straightforward to prune the network according to any mask configuration. Meaning for simple tasks that only require a small subset of channels we can use a fraction of compute at test time while still leveraging the advantages of joint training with other tasks.
An important implementation detail that masks are applied every other layer. Consider, for ex- ample, making task use half the model. You might think to set half the values of m to and apply to each layer. But that would mean & inputs and & outputs at each layer which only uses one quarter of the original model. Instead, applying mask at every other layer produces the desired behavior of allocating half the model to the task.
3.1 PARTITIONING PARAMETERIZATION
Now that we have decided to partition up feature channels, how do we go about finding the best masks for each task? Consider defining a binary matrix that specifies all partitioning masks: M E 10,e,, where is the number of feature channels and n is the total number of tasks. A direct search over this matrix is problematic. It is not straightforward to optimize over space of many discrete values, and one must account for significant redundancy given that all permutations of channels are equivalent. Moreover, naive random sampling would never cover the full space of partitioning strategies (consider the probability of randomly sampling two masks m that were mu- tually exclusive). In order see diverse degrees of feature sharing, the overlap of channels between masks must be explicitly accounted for.
Thus instead of searching over M, we propose searching directly over the features of M that de- termine performance: 1) the number of feature channels used by each task, and 2) the amount of sharing between each pair of tasks. The former decides the overall capacity available for the task while the latter shapes how tasks help or interfere with each other. We explicitly parameterize these factors matrix P. We can compute P 1M" M, where the diagonal elements of pro- vide the percentage of feature channels used by each task and the off-diagonal elements define the percentage of overlapping features between task pairs.
When sampling new partitioning strategies, we sample directly from P and identify corresponding mask M to match it. To remove some ill posed parts of the space we take additional steps adjust P. More details of this process as well as how we derive M from P can be found in the appendix.
This representation has a number of distinet advantages. in a given layer, so a single parameterization can be used for layers of different sizes. It is low dimensional, particularly since n is typically much smaller than c. And it is interpretable, providing clear impression of which tasks require more less network capacity and which tasks train well together. Moreover we get an immediate and direct measurement of average node usage per task, this is simply the mean of the diagonal of P. We will use this metric to compare the resource efficiency of different proposed partitioning strategies.
not tied to the number of channels
4 OPTIMIZATION STRATEGY
In order to optimize over different parameterizations P. there are two key ideas to cover: how we choose samples from the space and how we evaluate and compare different samples.
4

---

Under review as a conference paper at ICLR 2020
4.1 SEARCH STRATEGIES
We treat our search setting as black box optimization problem where given a particular parameter- ization we have function which returns score assessing its quality. Based on this score we can then choose how to further sample new parameterizations. We investigate two strategies for finding good constraint matrices.
Random sampling: The first is to simply randomly sample values. This has already been demon- strated to serve as a strong baseline in some architecture search work (Li & Talwalkar 2019] Liu et al. 2017a). With the low dimensionality of the matrix as well the additional steps taken to prepro- cess constraints, it is not unreasonable that much of the space can be covered with random samples. Random samples serve well to map out large swaths of the search space and identify the principle choices that affect final performance. Concretely, a random matrix P can be sampled with values taken uniformly from 0 to 1. If a particular resource target is desired, it is trivial to bias or restrict samples to a specific range of parameter usage.
Evolutionary strategies: Because P is continuous, it is possible to search over parameterizations with gradient-based optimization. We run experiments using evolutionary strategies More specif- ically, we use simple implementation with the modifications as described by Mania et al. (018 A gradient direction is approximated by sampling several random directions in the parameter space and computing finite differences to see which directions seem most promising. A weighted average is then computed across all directions to calculate the gradient to update the current parameters.
key feature of our approach is that we modify the algorithm to prioritize parameterizations that use as few channels as necessary per task. An additional L2 weight regularization term is added to the parameters on the diagonal of P. This serves to reduce the number of channels used by each task, in particular those that can be pulled down without affecting the overall accuracy and performance of the model. By controlling the strength of this regularization we can tune the importance of resource efficiency in the search process.
Using this optimization strategy is only possible because of the parameterization defined in 3.1 Approximating gradients make sense in the context of the continuous constraints defined in P. and we can more effectively explore the space of multi-task architectures using this signal. This is different from existing architecture search work where search decisions correspond to the coarse selection of entire computational blocks and their connections to each other.
4.2 SAMPLE EVALUATION
Finally, we must evaluate different partitioning schemes. But as discussed, determining the relative effectiveness of one partitioning over another by training models to convergence expensive. One possible strategy is to train models for a short period of time assuming that the relative differences in performance that appear early in training should correlate well with differences in performance when trained for longer. We instead propose to use feature distillation to observe the representa- tional capacity of a partitioned layer. We test how well shared multi-task layers can reproduce the activations of corresponding single-task layers. By only focusing computation and the number of weights that need to be tuned. In addition, directly distilling to intermediate layer activations provides a more direct training signal than a final classification loss.
few layers, reduce total
Given a proposed partitioning mask, we initialize new layers to be distilled and load reference mod- els for each target task. Input the layer is generated by passing through the task-specific pretrained model up to target depth. The resulting features are then passed through the subsequent layers of the pretrained model well as the new shared layers. In the new layers, intermediate features are masked according to the proposed partitioning. This procedure is illustrated in Figure 2
We use a mean-squared error loss supervise the shared layers such that their output features match those produced by the reference teacher models. We can measure the effectiveness of this distillation by replacing the original pretrained layers with the new shared layers and measuring the updated model accuracy.
'This can also be referred to as "random search" Mania et al. 2018). but we will instead use "evolutionary strategies" to avoid confusion with random sampling which is also called random search in existing NAS work L & Talwalkar 2019).
5

---

Under review as a conference paper at ICLR 2020
pretrained layers
shared layers with task mask
distillation loss
Figure 2: Multi-task distillation: At given step, teacher network pretrained single task is chosen from the available tasks. Features at target depth are produced then passed through the next residual block of both the teacher and student (where mask is applied). Features are then compared using a standard MSE loss. This process can be applied to multiple layers at once.
important to emphasize that we are not using distillation to get a final multi-task model as we do not want to be limited by the performance of individual pre-trained models. Instead, distillation serves as a proxy task for quickly evaluating partitioning strategies. We do not run the distilla- tion process convergence, only for brief interval, and it serves sufficient signal provide feedback on different parameterizations. This leads to dramatic reduction the time required to evaluate particular masking strategy.
EXPERIMENTS
We run number of experiments to investigate the role our proposed parameterization and distilla- tion play in finding multi-task architectures that minimize task computation and parameter use while achieving high accuracy. All experiments are performed using the Visual Decathlon dataset (Rebuffi al. 2017). Visual Decathlon composed many well-established computer vision classífication datasets of various sizes and respective difficulties. There is sufficient diversity that it is difficult to determine which datasets would benefit from more or less network capacity and parameter sharing.
We investigate how a model performs when trained on nine Decathlon tasks at once (all datasets except for ImageNet). We initialize a shared ResNet model with separate fully connected layer output for each task. To simplify experiments, we freeze the first two-thirds of the model and only apply feature partioning to the last third. For training, we alternate bini-batches sampled from each dataset and apply standard cross-entropy classification loss at the appropriate task specific output. More thorough implementation and experiment details can be found in the appendix.
It is unclear that performance will necessarily be better with any feature restriction opposed to using the full model for all tasks. One question is whether partitioning well leads to a reduction in interference across tasks and perhaps improved performance. In addition, wish to see the overall relationship between performance and feature restriction in this multi-task setting. What's the best performance possible average feature use is reduced further and further?
5.1 DISTILLATION
Before performing our search we need to know that given a sampled set of feature masks M, distil- lation performance correlates well with the final accuracy of model trained to convergence. This will determine whether our proposed distillation is a reasonable surrogate in lieu of full training. The higher the correlation between the two, the more confidence can place in our search process.
When performing distillation we initialize the child layers with pretrained layers from an ImageNet model. since the parent single-task networks have also been initialized from the same model. This accelerates the distillation process. The whole process takes just one minute on a P100 GPU. Further details available in the appendix.
We sample many random partitioning masks and run both the distillation procedure and full training to convergence. As baseline, also see how well final validation accuracy compares to accuracies seen earlier in training. We compare to the accuracies after 5k and 10k iterations (corresponding to 5 and 10 minutes of training). As seen in Table the distillation procedure (which takes fraction

---

Under review as a conference paper at ICLR 2020
aircratt, r-0.758
cffar100.--..544 r-0.954
daimerpeddks, r-0.477
Dataset
5k iters
10k iters
dist
aircraft
0.658
0.690
0.758
cifar100
0.928
0.927
0.954
daimlerpedcls
0.018
0.160
0.477
did, m0.478
glarb, r+0.221
Vog flowers. r-0.758
dtd
0.368
0.436
0.478
# for v vetaadk
gtsrb
0.714
0.687
0.821
vgg-flowers
0.703
0.692
0.758
omniglol, 00.801
svin, ro.676
udf101, .0.940
omniglot 0.871
0.906
0.801
svhn
0.485
0.761
0.676
ucf101
0.811
0.732
0.940
Figure 3: Correlation between distillation ac- curacy and final validation accuracy.
Table I: Correlation to final validation accuracy (distillation training 5k and 10k iterations).
Distribution samples
sharing
Evolutionary strateg optimization
0.28
0.5
weight dbca 2012 enlght decay:
endght decay
param
Figure 4: (left) Distribution of performance of random partitioning strategies; (middle) Distribu- tion of performance when fixing channel use but varying sharing across tasks; (right) Optimization trajectories of ES with different degrees of regularization
of the time) correlates higher with final accuracy. This allows us to sample and compare many more parameterizations during search process and have more confidence that the top performing parameterizations will do well when training a full model.
5.2 ARCHITECTURE SEARCH
Randomly sampling parameterizations: To map out the performance of different partitioning strategies we sample random parameterizations and plot distillation performance against the aver- age percentage of allocated features (the mean of the diagonal of P) in Figure 4 (left). From the distribution random samples we get an impression of the best performance possible at different degrees of resource use. The combination of fast feedback with distillation plus effective search space coverage with our proposed parameterization produces this information with less samples and in less time. At high levels of average feature use, choice of partitioning can only make much of difference. We are interested in the opposite how well we can do when restricting task computation as much as possible. Here, partitioning well is necessary achieve high performance.
Also it is important to note that explicitly specifying the degree of sharing between tasks is critical. This is shown in the middle plot of Figure 4 We evaluate different partitioning strategies with three fixed sets of values for the diagonal of P. only adjusting the amount of sharing that takes place between tasks. There can be significant difference expected, sharing affects performance more and more as average parameter use goes down as there more flexibility when choosing how to overlap features. is important that feature sharing parameterized when doing any sort of optimization.
performance in each of these cases, and as
Finally, we look at per-task results (Figure55). In this particular setting, every task benefits from using as many features possible. This may have to do with using model pretrained ImageNet, but makes sense that tasks benefit from using as many features as they can. An important facet to this is how much of those features shared. As average parameter usage increases across tasks (indicated by a lighter color), individual tasks suffer as they now share more of their features and must deal with the interference of other tasks.

---

Under review as a conference paper at ICLR 2020
1.0
200re
0.3
03
0.8
0.4
58
0.2
0.3
0.6
04
0.
05
03
0.50
1.00
0.25
0.7
0.00
0.90
0.50
0.50
100
Flwr
OGIt
SVHN
UCF
Teak per param (h)
0.0
Dvsttatten 2o0r
0.
0.6
0.1
044
033
Avg porom
Ue
04
6.2
02
2.00
0.25 0.50
0.75
1.40
0.00
0.25
0.50
0.75
1,02
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
3
Tank peruor LSe
Teok paromm
purrmm Lsve [N]
Task paraam LOM (76)
Figure 5: Distillation performance per task as a function of parameters allocated to that task. Color indicates the average parameters used across all tasks, notice more parameters are used overall (lighter color) individual task performance drops.
Partitioning
Airc.
C100
DPed
DTD
GTSR
Flwr
OGIl
SVHN
UCF
Total
independent
532 1
74.1
97.8 03
49.4
0.3
99.7 1 0.0
18.3 0.2
87.7
54.501
77.1 0.3
0..
share half
53.1 +
9.2400.1
49.5
0.3
0.
78.6
0.3
87.7 0.1
94.0 0.1
78.2 0.2
+94+0..1
share
76.3
50.0
99.7
78.9
94.1
79.0
es (wd 1o-3)
54
7 0.1
97.
49.7 0.1
99.8 0:0
0.4
88.3 0.0
940
79.4
0.1.0.01
Table 2: Validation results on Visual Decathlon.
Evolutionary strategies: As mentioned above, the distribution of random samples gives im- mediate impression of the best level of performance possible as a function of average parameter use. Evolutionary strategies provides a means to more directly push this edge of performance even further. We visualize the search process by plotting samples over the course of optimization over- laid over the distribution of samples found by random sampling (Figure 4 (right)). ES optimization quickly identifies samples that provide the best accuracy given their current level of parameter use and densely samples in this space, making slight changes for any last available improvements to performance. Furthermore, adjusting the weight decay penalty used during optimization controls the resource use of the final partitioning strategy. This allows us easily tune the optimization to reach the best architecture that meets specific resource needs.
The best parameterization found with evolutionary strategies outperforms number of baselines for We compare across several strategies with different de-
performing partitioning as seen in Table 2 grees of feature use and sharing. We measure validation accuracy of models trained to convergence (averaged over five trials). These baselines include: independent partitions that split features evenly across tasks, sharing half of available feature channels and splitting the rest, and finally, sharing all feature channels. In line with our random sampling, the more channels given across tasks, the better performance. Sharing everything does the best amongst these baselines. However, using the parameterization found from our optimization both reduces average channel use and achieves better performance overall.
We that there exist partitioning strategies that cut down average feature dramatically while still maintaining the same overall performance. This is in large part due to simple tasks that only need small fraction of channels (DPed for example Fig 5). By taking away the interference caused by these simpler tasks, harder tasks stand to gain more and that can be seen in Table 2 with tasks like CIFAR100. 100, Flowers, and Omniglot seeing the largest gains from an effective partitioning strategy.
6 CONCLUSION
In this work we investigate efficient multi-task architecture search to quickly find models that achieve high performance under limited per-task budget. We propose a novel strategy for searching over feature partitioning that automatically determines how much network capacity should be used by each task and how many parameters should be shared between tasks. We design compact rep- resentation to serve as a search space, and show that we can quickly estimate the performance of different partitioning schemes by using feature distillation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705. 2017.
Shawn LE Beaulieu, Sam Kriegman, and Josh C Bongard. Combating catastrophic forgetting with developmental compression. arXiv preprint arXiv: 04280, 2018.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint (00i:::700005444 2017.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Yi-Min Chou, Yi-Ming Chan, Jia-Hong Lee, Chih-Yi Chiu, and Chu-Song Chen. Unifying and merging well-trained deep neural networks for inference stage. arXiv preprint arXiv:/805. 885.04980, 2018.
Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before train- ing. arXiv preprint arXiv:1712.03333 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arxiv:1802.01566 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 110111001..777 2017.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Xiaoxi He, Zimu Zhou, and Lothar Thiele. Multi-task zipping via layer-wise neuron sharing. arXiv preprint arXiv:
2018
Lu Jiang, Zhengyuan Zhou, Thomas Leung. Li-Jia Li, and Li Fei-Fei. Mentomet:: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018
Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017.
Manoj Kumar, George E Dahl, Vijay Vasudevan, and Mohammad Norouzi. Parallel archi- tecture and hyperparameter search via successive halving and classification. arXiv preprint vrXiv:1805.10255 2018.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv: 100.0038,,, 2019.
Jason Liang, Elliot Mcyerson, and Risto Miikkulainen. Evolutionary architecture search for deep multitask networks. arXiv preprint 1833v:1803.03345, 2018.
Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. arXiv preprint arXiv: 172.0059,, 2017a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search. arXiv preprint arXiv: 2017b.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention.
arXiv preprint arXiv: 0000.8000.04 2018.
Arun Mallya and Svetlana Lazebnik. Piggyback: Adding multiple tasks single, fixed network by learning to mask. arXiv preprint 1801:01801.0651, 2018.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach reinforcement learning. arXiv preprint arXiv:1803.07055 2018.

---

Under review as a conference paper at ICLR 2020
Elliot Meyerson and Risto Mikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint arXiv:1711.00108, 2017.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, 2016.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint
2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01545 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Sylvestre-Alvias Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8119-8127, 2018.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv: 2017.
Amir Rosenfeld and John Tsotsos. Incremental learning through deep adaptation. arXiv preprint
1700:042550 2017.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 1505:09706.050088 2017.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Learning what share between loosely related tasks. arXiv preprint arXiv: 1705.0:705.088 7705.08142 2017.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell, Policy distil- lation. arXiv preprint arXiv:1511.0620 2015.
Sahil Sharma, Ashutosh Jha, Parikshit Hegde, and Balaraman Ravindran. Learning multi-task by active sampling. arXiv preprint arXiv: /00i:://700.005553 2017.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In Evolutionary Computation, 2008. CEC 2008. World Congress on Computational Intelli- IEEE, 2008
gence). IEEE Congress on, pp.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In The IEEE Conference on Computer Vision and Pattern Recognition, volume 2, 2017.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3712-3722, 2018
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv: 107,v.1807.06906, 2018.
Yu Zhang and Qiang Yang. A survey multi-task learning. arXiv preprint arXiv: 2017.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. CoRR, abs/161 2016.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 PARTITIONING PARAMETERIZATION
We define simple set of operations that convert from the raw search space P constraint matrix P that is more likely to correspond to feasible masks. Given the knowledge that pairwise values of P are conditioned on its diagonal terms, it is not possible for there be more overlap between two tasks than the channels used by any one task. That is, no off-diagonal element My should be greater than the corresponding diagonal elements M, and Mij-
Refining P:
We remap all off-diagonal elements to appropriate values determined by the diagonal of the matrix. This means that for any off off-diagonal element in P, now maps to the minimum possible overlap and to the maximum possible overlap of the two tasks. The procedure defined as follows:
D = diag( P)1"* E RRXR
(1)
Pmm max(0, D + D JJ
(2)
P maz min( D")
(3)
(P @ (Pmar - Pmmi)) + Pmin) ② (J -)
(4)
where and J are the column vector and the matrix of ones respectively, and @ represents Hadamard product.
Deriving M from P: Now, must find feasible mask M satisfy the constraints specified in P. This can formulated mixed integer programming problem. To show that, let P [0, 1|mx" denote given parameterization of constraints, we are interested in determining binary mask M by minimizing:
minimize
Ee
subject to:
(5)
Mai Pis + di
Vi [1,
Σ Likk & Pii + Eig k=1
v.j E [1,n]
Lijk < / (Mai + Mxj)
W.j E [1,n]
(8)
tizk 2 MR + Mkj- -
W.j E [1
(9)
lijk, MR (0, 1), des Eig 2 0
vk [1,c]
(10)
We employ two techniques. First we introduce two slack variables 6 and to relax the unary task
constraint in equation and pairwise task constraints equation?? 7 Second, we convert the nonlinear constraints between tasks by introducing the binary auxiliary variable I. For each pair of tasks (i,j),
in perfect solution without slack variables, we have:
tigk
(11)
Two tricks are used: auxiliary variable for non-linear constraint and slack variables. The search space now is significantly reduced, and this becomes mixed integer programming problem which can be conveniently solved by off-the-shelf solvers.
11

---

Under review as a conference paper at ICLR 2020
Feature corvistion
Festure commiation thatogram)
Feaker commration
Feature corriation thatograng
Feature comstatinn
Feature comnetation peologaan)
danaeeedts
amn
Layer
Layer 2
Layer
Figure 6: Measuring correlation of feature activations across three blocks of task-specific ResNet models given shared input image. Even though no restriction is made when finetuning individual models, the features produced are highly correlated through the first two-thirds of the model, and greater task differentiation occur due to different batch normalization statistics across datasets, but that is controlled for here.
seen until the end. Differentiation in early blocks would normally
A.2 ADDITIONAL EXPERIMENT DETAILS
Multi-task training: For full model training, we use a batchsize of 64 with SGD and momentum at a learning rate of 0.05 for 100k iterations, dropping to a learning rate of 0.005 at iteration 75k. All training was done single Nvidia P100 GPU. We followed the exact training, validation, and test splits provided by Visual Decathlon.
Several steps taken ensure that model trained simultaneously multiple tasks converges well:
Batch normalization: We maintain separate batch normalization statistics per task as done in (Rebuffi et al. 2017). This adds minimal parameter overhead and accelerates training.
Momentum: We maintain separate gradient statistics when using momentum with SGD. This is important given use of feature partitioning. At any given training step, do not want the unused weights associated with other tasks to updated.
Training curriculum: Rather than uniformly sampling across tasks we apply a simple strat- egy to choose mini-batches from tasks inversely proportional to their current training accu- racy. Tasks that lag behind get sampled more often. Evidence for this approach has been demonstrated in multi-task reinforcement learning setting (Sharma al., 2017). We find a curriculum over tasks more effective than curriculum over individual samples (Jiang et al. 2018).
Pretrained ImageNet model: All experiments are performed with a pretrained ImageNet model. The model is trained using the Py PyToocc implementation made available by Rebuffii et al. (2018). Because ImageNet is orders of magnitude larger than any of the other datasets in Decathlon and takes much longer to train we exclude it in our partitioning experiments to focus on the interactions of other datasets.
The main hyperparameters that determine performance were learning rate and temperature term that controlled the task sampling curriculum. This temperature term determines whether mini- batches are sampled uniformly across tasks or whether tasks with low training accuracy are weighted more heavily. For both hyperparameters we arrive at the final value by simple grid search.
Final validation accuracy reported in Table 2 (in the main paper) is averaged across 5 trials.
To further simplify experiments, we freeze and share the first two-thirds of the network. Partitioning is thus only performed on the last third of the model. By only updating the weights of the last block, we focus attention on the layers where task-specific features are most important without restricting the model's representational capacity to fit each task.
Frozen layers:
In all of our experiments we use an ImageNet-pretrainee ResNet model made up of three computa- tional blocks with four layers each. We freeze the first two computational blocks and only perform partitioning on the last set of layers. The justification for this stems from analysis performed with in- dividual single task models. We compare feature differences across finetuned task-specifi models. These models were trained with no restrictions initialized from an ImageNet model until converging to high accuracy on some target task. Because start with pretrained model we make mean-
12

---

Under review as a conference paper at ICLR 2020
ingful comparisons of each model's channel activations to how task feature diverges after finetuning.
We compare intermediate task features after passing in shared image into every model. An impor- tant detail here is that we control for the batch normalization statistics associated with the dataset that the image is sampled from. The subsequent features produced by each model almost identical all the way up through the first two-thirds of the model. Aside from subtle differences, speciic differentiation did not occur until the final third of the model where features were still somewhat correlated but differed dramatically model to model. This is visualized in Figure 6 Because of this we decided the task-specific differentiation afforded by feature partitioning would not be as impor- tant in earlier stages of the model, and experiments would be more informative and also faster to run while focusing only on the last set of layers.
Distillation details: We do not use the accuracy based curriculum used in normal training during distillation and instead alternate mini-batches uniformly across each task. Distillation training done for brief 3000 iterations with batch size of 4 and learning rate of which is dropped by factor of 10 at iteration 2000.
Distillation is done on the last four ResNet layers at once match the final training setting as closely as possible. All scores reported when performing distillation are averaged across three trials.
Evolutionary strategies details: The optimization curves shown in the paper are from runs that have each taken 1000 samples, these were performed on machines with PI00 GPUs. Given that sample evaluation takes roughly minute, the whole procedure takes just over four hours.
At each step, 16 random parameter directions are sampled and these are both added and subtracted from the current parameterization P to produce 32 new samples to evaluate. A gradient calculated based on the results these samples, and a gradient descent step is applied to the current parameters with learning rate of 0.1. Both clipping and sigmoid operation were tested to ensure that values remain between and with no discernible difference optimization performance.
13


Reviews:
[{"review_id": "rkxwhDbbqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for learning multi-task convolutional neural networks. For each layer of the network, the proposed algorithm assigns a subset of the layer's channels to each of the tasks. This is in contrast to existing methods that assign whole layers to tasks. There are two key ideas here: (1) instead of searching in the space of binary assignments of layers to tasks, search in the continuous space of fractions of channels assigned to each layer, subject to some consistency constraints; this allows for using finite differences for gradient estimation which can be fed into a black-box optimization procedure; (2) the use of distillation to estimate the performance of a given assignment, rather than retraining many models. Experimentally, the proposed framework performs relatively well on the Visual Decathlon benchmark.\n\nOverall, I do like the paper's key ideas. However, I am not convinced by the presentation of the paper and by the lack of multi-task learning baselines in the experiments. I will consider changing my score if the following comments are addressed.\n\n- Testing your main premise: \"we instead partition out individual feature channels within a layer. This offers a greater degree of control over both the computation required by each task and the sharing that takes place between tasks.\" \nActually, based on the paper, it is not clear that assigning channels rather than layers brings about additional gains. To prove that hypothesis, you should carry out experiments comparing your method to existing multi-task learning methods that you surveyed in the paper.\n\n- \"share all\" performance: this baseline seems to do quite well in Table 2, beating es in 5/10 w.r.t. the average accuracy. Why does this happen?\n\n- Too many hacks hidden in the appendix and it is not clear what works why. Have these values been found by cross-validation? The reader should be able to understand how your method works *exactly*. Examples of these magical values:\n-- Number of samples for the ES\n-- Number of parameter directions\n-- \"Distillation training is done for a brief 3000 iterations with a batch size of 4 and a learning rate of 1 which is dropped by a factor of 10 at iteration 2000.\"\n\nClarification questions:\n- I don't understand what \"strategy\" means here: \"Fixed vs Learned Strategies: Is a uniform strategy applied across tasks or is a task-specific solution learned?\"\n\n- Section 4.2 needs to be rewritten to give a more structured exposition of the distillation process. Please add pseudocode if needed. The Figure is not very informative.\n\n- MIP: Generally, this is a very important component of your method, yet it is lacking in detail and left to the appendix. Some questions:\n\n-- how long does it take to solve? Which commercial solver do you use? Please provide more details.\n-- constraint (8) is for which k? Is it for all k? Please fix. Also for (10), it should be for all i and j as well as k.\n-- what is the \"non-linear constraint\" that (8-9) are linearizing? Please provide the original MIP formulation (without slack or linearization), and explain the final formulation accordingly.\n-- MIP: are you adding slack variables to allow for slightly infeasible solutions in case no M can be derived from \\tilde{P}?\n\nMinor:\n- \"cut down average feature dramatically\": unclear, rephrase as needed.\n- Figure 6 is low-resolution.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1g5G3esKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies multi-task learning (MTL) from the deep learning perspective where a number of layers are shared between tasks followed by specific heads for each task. One of the main challenges in this problem is to decide the best configuration among a large number of possible ones (e.g., the number of layers , number of neurons, when to stop the shared part of the network). In this paper, the authors fix the network architecture, and learn which filters (among the already learned ones) should be dedicated to (and hence fine-tuned for) a specific, and which ones should be shared between multiple tasks. \n\nInstead of deciding on other hyper-parameters such as the number of layers, the authors chose to study how to efficiently share the capacity of the network: to decide which filters should be used for which tasks, and which filters should be shared between tasks. \nSpecifically, this is controlled by task specific binary vectors which get multiplied with feature activations for each task, hence blocking or allowing the signal to pass for a specific filter. In addition, they define a different set of binary vectors for the foreground and background passes. This allows simpler tasks to benefit from features learnt from more complicated tasks such as ImageNet classification while avoiding ‘catastrophic forgetting’ at the same time.\n\nMoreover, the authors develop a simple yet elegant strategy to reduce their parameter search space (by using the matrix P which controls the percentage of filters used per task + the percentage of filters shared between each pair of tasks) and quickly evaluate the performance of each configuration (using distillation). The advantages of these approaches are well discussed and validated quantitatively.\n\nThe paper is well written and the approach itself appears to be sound and it led to improvement over independent task estimator.  However, I am mostly concerned about the experimental setting: there are no comparisons with any other MTL algorithm. \n\nThe authors perform a search over the matrix P, which is similar to neural architecture search over the entire possible ways of sharing the capacity of a network. This could potentially lead to improvement beyond multi-task learning. Experimental comparison on this could be provided.\nI think the paper will make a strong case if it is compared with existing deep MTL algorithms including [Misra et al: Cross-stitch networks for multi-task learning]. In addition, the network seems to share a similar spirit with [Mallya et al: PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning], in that they also share the capacity of the network between tasks, and hence a comparison here seems reasonable. \n\nOverall, I think this paper makes a borderline case.\n\nOther comments: \nIn the supplementary material, providing a detailed description of the algorithm (e.g., pseudo code and an accompanying discussion) that calculates the matrices M from P could help reproduce and build upon the experiments reported in the paper. I wonder if M is uniquely defined from M.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJl0NTdaKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a feature partitioning scheme for efficient multi-task neural architecture search.  The proposed scheme automatically determines the network capacity that should be shared across tasks and kept exclusive for each task.\n\nOverall, this is a well written paper. \n\nThe problem of neural architecture search is important and beneficial to deep learning community to be able to extract the best out of these methods. Multi-task learning is no exception. However, I am not sure why the problem of NAS is so different for multiple outs than a single output. Given multiple outputs, we need to either look at a weighted combination where weights could be provided by user to reflect the priority over the tasks, or individual outputs in a multi-objective approach such as Pareto optimality manner. I find the approach of this paper a heuristic.\n\nUsually, due to negative transfer learning, too much sharing of the weights is detrimental for the performance of an individual task. I could not see this anywhere in the results. Especially, I missed any baselines where such problems were present, which would then be improved due to using this method. How would the results be if all tasks are given equal weight and NAS is performed with respect to their average output (of course normalised appropriately)?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers how to create efficient architectures for multi-task neural networks. R1 recommends Weak Reject, identifying concerns about the clarity of writing, unsupported claims, and missing or unclear technical details. R2 recommends Weak Accept but calls this a "borderline" case, and has concerns about experiments and comparisons to baselines. R3 also has concerns about experiments and baselines, and feels the approach is somewhat ad hoc. The authors submitted a response that addressed some of these issues, but the authors chose to maintain their decisions. The AC feels the paper has merit but given these slightly negative to borderline reviews, we cannot recommend acceptance at this time. We hope the reviewer comments help the authors to prepare a revision for another venue.
Decision: Reject