AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning
Authors: Lijun Zhang
Xiao Liu
Hui Guan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Applications: Multi-Task Learning
Infrastructure: Programming Framework for MTL

Abstract: Multi-task learning (MTL) jointly learns a set of tasks by sharing parameters among tasks. It is a promising approach for reducing storage costs while improving task accuracy for many computer vision tasks. The effective adoption of MTL faces two main challenges. The first challenge is to determine what parameters to share across tasks to optimize for both memory efficiency and task accuracy. The second challenge is to automatically apply MTL algorithms to an arbitrary CNN backbone without requiring time-consuming manual re-implementation and significant domain expertise. This paper addresses the challenges by developing the first programming framework AutoMTL that automates efficient MTL model development for vision tasks. AutoMTL takes as inputs an arbitrary backbone convolutional neural network (CNN) and a set of tasks to learn, and automatically produces a multi-task model that achieves high accuracy and small memory footprint simultaneously. Experiments on three popular MTL benchmarks (CityScapes, NYUv2, Tiny-Taskonomy) demonstrate the effectiveness of AutoMTL over state-of-the-art approaches as well as the generalizability of AutoMTL across CNNs. AutoMTL is open-sourced and available at https://github.com/zhanglijun95/AutoMTL.

device AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning
Lijun Zhang
Xiao Liu
College of Information & Computer Sciences University of Massachusetts Amherst
College of Information & Computer Sciences University of Massachusetts Amherst
Amherst, MA, 01003
Amherst, MA, 01003
1i jajunzhangoss. umass. edu
xiaoliu1990@cs umass edu
Hui Guan
College of Information & Computer Sciences University of Massachusetts Amherst
Amherst, MA, 01003 huiguan@cs. umass. edu
Abstract
Multi-task learning (MTL) jointly learns a set of tasks by sharing parameters among tasks. It is promising approach for reducing storage costs while improving task accuracy for many computer vision tasks. The effective adoption of MTL faces two main challenges. The first challenge is to determine what parameters to share across tasks to optimize for both memory efficiency and task accuracy. The second chal- lenge is automatically apply MTL algorithms an arbitrary CNN backbone with- out requiring time-consuning manual re-implementation and significant domain expertise. This paper addresses the challenges by developing the first programming framework AutoMTL that automates efficient MTL model development for vision tasks. AutoMTL takes as inputs arbitrary backbone convolutional neural network (CNN) and a set of tasks to learn, and automatically produces a multi-task model that achieves high accuracy and small memory footprint simultaneously. Experi- ments on three popular MTL benchmarks (CityScapes, NYUv2, (inyTTakkonomy demonstrate the effectiveness of AutoMTL over state-of-the-ard approaches as well as the generalizability of AutoMTL across CNNs. AutoMTL is open-sourced and available https: //g:: thub. com http://gnnin////nuoootrr..t
Introduction
Al-powered applications increasingly adopt Convolutional Neural Networks (CNNs) for solving many vision-related tasks (e.g., semantic segmentation, object detection). leading to more than one CNNs rumning resource-constrained devices. Supporting many models simultaneously is challenging due to the linearly increased computation, energy, and storage costs. An effective approach to address the problem is multi-task learning (MTL) where set of tasks learned jointly to allow some parameter sharing among tasks. MTL. creates multi-task models based on CNN architectures called backbone models, and has shown significantly reduced inference costs and improved generalization performance in many computer vision applications [24] (3].
The effective adoption of MTL faces main challenges. The first challenge is the resource efficcien
architecture design-that is, to determine what parameters of a backbone model to share across tasks
to optimize for both resource efficiency and task accuracy. Many prior works 220 22 B3 22 36 rely on masiine MTL model architectures which share several initial layers and then branch
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

out at an ad hoc point for all tasks. They often result in unsatisfactory solutions due to the enormous architecture search space. Several recent efforts 44 [9 shift towards learning to share parameters across tasks. They emhed policy-learning components into a backbone CNN and train the policy to determine which blocks in the network should be shared across which task or where to branch out for different tasks. Their architecture search spaces lack the flexibility to dynamically adjust model capacity based on given tasks, leading to sub-optimal solutions as the number of tasks grows.
The second major challenge is the automation. Manual architecture design calls for significant domain expertise when tweaking neural network architectures for every possible combination of learning tasks. Although neural architecture search (NAS) approaches automate the model design to some extent, the implementation of these works is deeply coupled with specific backbone model. Some of them [ 3 4 [9] could theoretically support broader types of CNNs. They, however, require significant manual efforts and expertise to re-implement the proposed algorithms whenever the backbone changes. Our user study (Section 4.2) suggests that it takes machine learning practitioners with proficient PyTorch skills 20 to 40 hours to re-implement Adashare [5] a state- of-the-art NAS-based MTL approach, on MobileNet backbone. The learning curve is expected to be much longer and more difficult for general programmers with less ML expertise. The lack of automation prohibits the effective adoption of MTL in practice.
In this paper, we address the two challenges by developing AutoMTL, the first programming frame- work that automates -eoourceefffiiient MTL model development for vision tasks. AutoMTL takes as inputs an arbitrary backbone CNN and a set of tasks to learn, and then produces a multi-task model that achieves high task accuracy and small memory footprint (measured by the number of parameters). backbone CNN essentially specifies a computation graph where each node is an operator. The key insight is to treat each operator as a basic unit for sharing. Each task can select which operators to use to determine the sharing patterns with other tasks. The operator-level sharing granularity not only enables the automatic support of arbitrary CNN backbone architectures, but also leads to stretchable architecture search space that contains multi-task models with a wide range of model capacity. To our best knowledge, we are the first work considering parameter sharing in MTL at the operator level.
AutoMTL features a source-to- to-sure compiler that automatically transforms a user user-provided back- bone CNN to a supermodel that encodes the multi- architecture search space in the operator-level granularity. It also offers a set of PyForch-based APIs to allow users to flexibly specify the input backbone model. AutoMTL then performs gradient-basee based architecture searches on the supermodel to identify the optimal sharing patterns among tasks. Our experiments on several MTL benchmarks with different number of tasks show that AutoMTL could produce efficient multi-task models with smaller memory footprint and higher task accuracy compared to state-of-the-an methods. AutoMTL also automatically supports arbitrary CNN backbones without any re-implemennation efforts and improves the accessibility of MTL to general programmers.
The main contributions of our work are as follows:
We propose a Multi-Task Supermodel Compiler - which transforms a user- provided backbone CNN into multi-task supermodel that encodes the architecture search space. The compiler decouples architecture search with the backbone CNN, removing the manual efforts in re-implementing MTL on new backbone models.
We propose a Stretchable Architecture Search Space that offers flexibility in deriving multi- task models with a wide range of model capacity based on task difficulties and interference. We further propose a novel data structure called Virtual Computation Node to encode the search space and enable compiler-based multi-task supermodel transformation.
Built on top of the supermodel, we adopt the Gumbel-Softmax approximation and standard prop-propa to jointly optimize sharing policies and network weights. Under this context, we propose a policy regularization term on the sharing policy to promote parameter sharing for high memory efficiency.
We implement the AutoMTL system that seamlessly integrates the MTS- MTS-Compiled a set of PyTorch-based APIs, the architecture search algorithm, and training pipeline. AutoMTL
provides an easy-to-use solution for sesource-efffcienn multi-task model development.
Experiments on three popular MTL benchmarks (CityScapes [10. NYUv2 41), Tiny-
Taskonomy [50) using three common CNNs (Deeplab- Net333 [7]. MobileNetV2 [39].
MNasNet (451 demonstrate that the multi-task model produced by AutoMTL outperforms
state-of-the tite-of-thh-art-ce approaches in terms of model size and task accuracy.
2

---

2 Related Work
In hard parameter sharing, a set of parameters in the backbone model are shared among tasks. In soft parameter sharing 18, each task has its own set of parameters. Task information is shared by enforcing the weights of the model for each task to be similar. In this paper, we focus on hard parameter sharing as produces memory-efficient multi-task models.
Multi-task learning (MTL) uses hard or soft parameter sharing 37 5
47
52
34 38
Manual Design and Task Grouping. One of the widely-used hard parameter sharing strategies is proposed by Caruana 6 []. which shares the bottom layers of model across tasks. Following this paradigm, early works [32 35 44 9 22 26) rely on domain expertise to decide which layers should be shared across tasks and which ones should be task-specific. Due to the enormous architecture search space, such approaches are difficult to find an optimal solution. In recent years, several methods attempt to integrate task relationships or similarities to facilitate multi-task model design. What-to-Share [46] measures the task affinity by analyzing the representation similarity between independent models, then recommending the architecture with the minimum total task dissimilarity. Some other works 44 [6 focus on identifying the best task groupings in terms of task performance under the memory budget, whose architectures share the feature extractors within each group.
NAS-based Approach. Recent works attempt to learn the sharing patterns across tasks. Deep Elastic Network (DEN) (1 and Stochastic Filter Groups (SFGs) [3] determine whether each filter in convolutions should be shared via reinforcement learning or variational inference respectively. AdaShare learns task-specific policies that select which network blocks to execute for given task. However, these works cannot dynamically adapt multi-task model capacity based on given tasks. BMTAS 4 and Learn to Branch [19 focus on constructing tree-like structures for multi-tast models via differentiable neural architecture search. Some other works [17]4] explore feature fusion opportunities across tasks. It is worth mentioning that existing studies focus on search algorithms but ignore the ease of programming and extensibility. Their implementation is highly coupled with specific backbone models. It is time-consuming for general programmers to re-implement the algorithms once the backbone models change, hindering their adoption in broader applications.
MTL Optimization. Significant efforts have been invested to improve multi-task optimization strategies, an orthogonal direction to architecture design. There are two major branches to solve such a multi-objective optimization problem 40 28 Some works study single surrogate loss consisting of linear combination of task losses, in which the suitable task weights are derived from different criteria, such as task uncertainty [23]) task loss magnitudes [3T], dynamic task relationships
30. Other works focus on directly modifying task gradients during the multi-task model training 8 49 29). Note that, on top of our AutoMTL framework, users are free to use existing optimization methods to further improve the task performance of multi-task models.
3 AutoMTL
AutoMTI. allows users to provide an arbitrary backbone CNN and a set of vision tasks, and then automatically generate a multi-task model with high accuracy and low memory footprint by sharing parameters among tasks. Figure [ iillutraatss the workflow of AutoMTL. Given a backbone model (Figure I aa)). a user can specify the model using either the AutoMTL APIs or in the prototxi format (Figure [b)). The model specification will be parsed by the MTS MTS-compiler to generate a multi-task supermodel that encodes the entire search space [ AutoMTL then identifies the optimal multi-task model architecture (Figure
from the supermodel using maddient-based architecture search algorithms implemented in the Architecture Search component. AutoMTL supports the model specification in the format of protoixi as it is general enough to support various CNN architectures and also simple for our compiler to analyze. protoixt files are serialized using Google's Protocol Buffers serialization library. AutoMTI. API is currently implemented on top of PyTorch. We next elaborate on the two major components of this framework, MTS- Compiler and Architecture Search.
3.1 Multi-Task Supermodel Compiler
The Multi-Task Supermodel Compiler (MTS-Complln transforms the input backbone model into a multi-task supermodel that encodes the architecture search space. The challenge is how to design the architecture search space and the multi-task supermodel so that (1) the search space allows the
3

---

inddeeentt did Inl, Conodd Oxen-ll VON
AMGMTLAPI
VON
Architettrr Search
YOK Camdi
Mpy Dook
Task
faine "oyy" betaes
VON
9p. Daan
falne "ops" bettess Tpp.
Backhooe Model
nalne "opy App. (-gnioin
Compler
FON
PCN.
O
(60 MeddadddSpeiiiiie Specificrtian
60A Mutt Tad:: Supermofet wilh Search Spaces for Trnkk
Final Mubii-Takk Madel
Figure 1: Illustaations of (a) an input backbone model, (b) two types of model specifications, (c) the multi-task supermodel with search space produced by our AutoMTL APIs and the proposed MTS-Compile and (d) the final multi-task model found by AutoMTL. VCN, ~ VCN, represent the proposed data structure Virtual Computation Node (VCN). In each VCN, op is the original operator in the backbone model, while sp" and sk* are the task-specific copy of op and the skip connection for task t; respectively. ph is variable (a.k.a policy) that determines which operator will be executed for task i,
multi-task model capacity to be flexibly adjusted based on the set of tasks to avoid task interference and (2) the transformation can be fully automated and support an arbitrary CNN backbone.
To address the challenge, we propose a Stretchable Architecture Search Space that contains multi-task models with wide range of model capacities by treating each operator in the backbone model the basic unit for sharing. The compiler duplicates each operator in the backbone model so that each task determine whether it wants to share parameters with other tasks by selecting which operator to use. We further design a novel data structure called Virtual Computation Node to embed the search space and enable compiler-based automatic transformation of an arbitrary CNN multi-task supermodel.
Stretchable Architecture Search Space. As shown in Figure Jc). for each operator in the backbone model, each task can choose from one of the three options to indicate whether it wants to share the operator with other tasks: (1) the backbone operator, (2) task-specific copy of the backbone operator, and (3) a skip connection. Skip connection is an identify function if the input and output dimension match
down-ssmmle function otherwise. The motivation for skip connection option is that a task can skip the operator to improve inference efficiency.
Formally, assume a set of N tasks T = (41,42 fi,..,,,... | defined over a dataset. For the i-th task 4 and the 1-th backbone operator opt, the task may select the backbone operator itself, implying that it can share the parameters in this operator with other tasks. Otherwise, it can select the task-specific copy spt;, or the skip connection sk, as shown in Figure (c). Given a set of N tasks and a backbone model with I. operators, the size of our search space is 3NXL, Suppose the backbone capacity is C (measured by the number of parameters), the capacity of multi-task model in our search space would be in the range of (0, C' x N). where C' x N indicates all tasks choose to use their own operators (i.e., independent models) while 0 represents all the skip connections are selected.
The proposed search space has the following three major benefits compared to existing NAS -ased MTL methods. First, compared to AdaShare 43 and DEN [I] which attempt to pack multiple tasks into single CNN backbone, it can extend the representation power of the backbone model if needed by preserving more task-specific operators. This capability effectively avoids performance degradation caused by task interference as the number of tasks increases (See Section 4.2p. Second, compared to a more general search space [48] that is defined without requiring user-provided backbone model, it still provides users a certain degree of control over the size of the searched multi-task model one can specify smaller backbone model if the computation resource is limited. Last but not least, in terms of search efficiency, although our search space is larger than AdaShare, we still have comparable low search cost by adopting gradient-basee search algorithms. When comparing to a general search space in FBNetV5 [88. our search space can be explored 13 ~ 133X faster (Detailed in Section 4.2)
Multi-Tast Supermodel. A suitable multi-task supermodel abstraction is necessary to allow au- tomatic transformation of an arbitrary CNN backbone to a multi-task supermodel. Our idea is to represent the multi-task supermodel as a computation graph whose topology remains the same as that of the backbone model but nodes are replaced with Virtual Computation Nodes (VCNs). The

---

MTS- CTTS-Coppller first parses the input backbone into list of operators and then iterates the operators initialize the corresponding VCNs.
Specifically, for each operator in the given backhone model, the corresponding VCN in the multi-task supermodel contains: (1) list of parent VCN nodes, recording where inputs come from; (2) the backbone operator; (3) task-specitt copies of the backbone operator, one for each task; (4) skip connections, one for each task; (5) policy variables, one for each task determining which operator to execute for each task. (1) and (2) encode the computation graph of the backbone model. (2), (3), and (4) together encode the architecture search space. (5) determines the sharing patterns across tasks. Figure I () illustrates multi-task supermodel. Details about the policy variable will be presented in Section 3.2
3.2
Architecture Search
The architecture search component aims at efficiently exploring the search space encoded in the multi-task supermodel. We adopt the differentiable policy approximation to enable joint training of sharing policy and the supermodel. Under this context, we propose a policy regularization mechanism to promote parameter sharing for memory efficiency.
Policy Approximation. We introduce trainable policy variable P/: to determine which operator to use for the i-th task & and the (-th VCN in the multi-task supermodel. pi: is zero if the backbone operator opt is used for the task, one if the task-specific copy spi; is adopted, and two if the skip connection skf is selected. Architecture search is to find the optimal sharing policy P = (P < L, S N]. that yields the best overall performance over the set of N tasks T given multi-task supermodel with L VCNs. As the number of potential configurations for P is 3 (i.e., the size of the search space) which grows exponentially with the number of operators and tasks, it is not practical to manually find such P to get the optimal sharing pattern. Therefore, we adopt gradient-base based architecture search algorithm that optimizes the sharing policy and the multi-task model parameters jointly via standard prok-propagation. Gradient-based searches usually allow faster architecture search compared with traditional reinforcement learning [ or evolutionary algorithm-based approaches 27
Because the policy variable P E discrete and thus non- con-differentiable, we apply Gumbel-Softmax Approximation [21] and derives a soft differentiable policy:
exp((Gx + log((T))//)
('()) = E EkE(0.1.) exp( + log(Tx))/7))
(1)
where & E (0, 1,2) represents the three operator options, G₃ ~ Gumbel 1).
After learning the distribution the discrete task-specific policy P is sampled from the learned n to decide which operator execute each VCN for each task and the multi-task architecture can be constructed accordingly. Figure d) illustrates multi-task model given a sharing policy.
Policy Regularization. We propose policy regularization term Greg to encourage sharing operators across tasks to reduce the memory overhead. Specifically, for the soft policy P' = (P < L, < N}. we minimize the sum of the SoftPlus [12] of P' (1) - P": (0) and (2) (2)- (0) for each task in each VCN, where PM (0). the task-specific copy. and the skip connection for the i-th task in the (-th VCN respectively. To further reduce the computation cost, the regularization term is weighted for different operators to promote the parameter sharing of bottom layers. More formally, we define Lreg as,
("; are the probability of selecting the shared operator,
Creg ΣΣ', ISNISL L
[In(1 + exp PP-tt0))) + In(1 + exp ('p (1000))))
(2)
where In(1 + exp") is the SoftPlus function. I is the depth of the current VCN and L is the maximum depth. Finally, the overall loss L is defined as,
L
XiL + Aregfreg:
where L, represents the sask-specific loss, by is a hyperparameter controlling how much each task contributes the overall loss, and Xreg is a thype--parameter balancing task-specific losses and Creg-
5

---

Training Pipelines. AutoMTL implements three-stage training pipeline to generate rell-trained multi-task model. The first stage pre-train aims at obtaining a good initialization for the multi-tase supermodel by pre-training on tasks jointly [5I]. During pre-training, for each task, the output of each VCN is the average of the three operator options (i.e, the backbone operator, the task-specific copy, and the skip connection) so that all the parameters could get warmed up together. The second stage policy-train jointly optimizes the sharing policy and the model parameters. The model parameters and the policy distribution parameters are trained alternately to stabilize the training process, After the policy distribution parameters converge, AutoMTL samples a sharing policy from the distribution to generate a multi-task model. The last stage post-frain frain trains the identified multi-task model until it converges. The model parameters are trained from scratch while the sharing policy is fixed.
4 Experiments
We conduct a set of experiments to examine the superiority of AutoMTL compared to several state-of-the-art approaches in terms of task accuracy, model size, and inference time.
4.1 Experiment Settings
Datasets and Tasks. Our experiments use three popular datasets in multi-task learning (MTL), CityScapes [0]. NYUv2 [41) and Tiny-Taskonomy [50]. CityScapes contains street-view images and two tasks, semantic segmentation and depth estimation. The NYUv2 dataset consists of RGB- D indoor scene images and three tasks, 13-class semantic segmentation defined in [.] depth estimation whose ground truth is recorded by depth cameras from Microsoft Kinect, and surface normal prediction with labels provided in [14]. Tiny-Taskonomy contains indoor images and its five representative tasks are semantic segmentation, surface normal prediction, depth estimation, keypoint detection, and edge detection. All the data splits follow the experimental settings AdaShare (3)
Loss Functions and Evaluation Metries. Semantic segmentation uses a pixel-wise cross-entropy loss for each predicted class label. Surface normal prediction uses the inverse of cosine similarity between the normalized prediction and ground truth. All other tasks use the L1 loss. Semantic segmentation evaluated using mean Intersection over Union and Pixel Accuracy (mIoU and Pixel Acc, the higher the better) in both City and NYUv2. Surface normal prediction is evaluated using mean and median angle distances between the prediction and the ground truth (the lower the better), and the percentage of pixels whose prediction is within the angles of 1 11.25", 22.5 and 30 to the ground truth as [44 (the higher the better). Depth estimation uses the absolute and relative errors between the prediction and the ground truth are computed (the lower the better). In addition, the percentage of pixels whose prediction within the thresholds of 1.25, 1.25², 1.25 to the ground truth, i.e. å max{ Pureed Put } < thr, is used following
[15] (the higher the better).
Pat Pyrrad
Tiny-Taskonomy is evaluated using the task-specific loss of each task directly, as in 43
Because evaluation metrics from different tasks have different scales, we also use a single relative performance metric (33] with respect to the single-task baseline to compare different approaches. The relative performance At, of a method A on task t; is computed as follows,
At.
((-1)%(MA) MSTL.)/ 100%, 1=0
M
where sy is 1 if the metric M, is the lower the better and 0 otherwise. Ma.J and MSTL.J are the values of the metric M, for the method and the Single-Task baseline respectively. Besides, the overall performance is the average of the above relative values over all tasks, namely At 1 where N is the number of tasks. The model size is evaluated using the number of model parameters.
N Em At,
Baselines for Comparison. We compare with following baselines: the Single- Task baseline where each task has its own model and is trained independently, the vanilla Multi-Task baseline [3] where all tasks share the backbone model but have separate prediction heads, popular MTL methods (e.g.. Cross-Stitch 34), Sluice (38). CNN [18) MTAN (3T), and state-of-the-ar
MTL. 43 and Learn to Branch (9]). We use the same backbone
methods (c.g. DEN MI AdaShare
We implemented its tree-structured multi-task model for Taskonomy based on the architecture reported in the paper by ourselves since there is public code.

---

Table 1: Quantitative Results on CityScapes, (Abs. Prf. & Rel. Prf.)
Params
Semantic Seg.
Depth Estimation
Model
Pixel Acc.
Error +
5, within 1
At
Abs. (M) Rel. (%)
mloU
At1
A2
Abs.
Rel. 1.25
1.25*
1.25
Single-Task
42.569
36.5
73.8
0.026 0.38 57.5
76.9
87.0
Multi-Task
21.285
-50.0
42.7
68.1
+4.6
0.026 0.39 58.8
80.5
89.9
+1.5
+3.1
Cross-Stitch
42,569
+0.0
40.3
74.3
+5.5
0.017 0.34 70.0
86.3
93.1
+17.2
+11.4
Sluice
42.569
+0.0
39.8
74.2
+4.8
0.018 0.35 68.9
85.8
92.8
+15.3
+10.1
NDDR-CNN
44.059
+3.5
41.5
74.2
+7.1
0.018 0.35 69.9
86.3
93.0
+15,9
+11.5
MTAN
51.296
+20.5
40.8
74.3
+6.2
0.017 0.36 71.0
86.3
92.8
+16.4
+11.3
DEN
23,838
-44.0
38.0
74.2
+2.3
0.018
0,41 68.2
84.5
91.6
+11.3
+6.8
AdaShare
21.285
-50.0
40.6
74.7
+6.2
0.018
0.37 71.4
86.8
93.1
+15.5
+10.9
AutoMTL
28.819
-32.3
42.8
74.8
+9.3
00.018 0.33 70.0 86.6
93.4
+17.1
+13.2
Table 3: Results on Taskonomy. (Rel. Prf.)
Table 2: Results on NYUv2. (Rel. Prf.)
Params (%) 4
Models
A1. 12/2 1.t
144 1Ats
Model
Params (5)
A1
At,
1 At
Multi-Task
-80.0
-1.4
-111
Multi-Task
-66.7
-11.4
+2.0
+4.3
-1.7
Cruss Stitch
+0.0
+0.9
-3.5
+0.0
-1.0
2.4
-12
Cross-Stich
Sluice
+0.0
-1.5
+2.4
-2.3
Sluice
+0.0
-6,2
+7.1
+3.3
-1.4
NDDR
+8.2
-42
-0.9
-4.5
+0.5
+4.2
-1.0
NDDR-CNN
+5.0
-12.9
+7.0
-4.4
-3.5
MTAN
-9.8
-8.0
-255
.4.5
+0.0
+2.8
-24
MTAN
+3.7
-188
+11.5
+2.9
+42
DEN
-77.6
-28.2
-26
-22.7
+2.5
+4.2
-9.
DEN
-62.7
-7.7
+5.6
38.9
13.7
AdaShare
-80.0
+2.3
-0.6
-4.5
+3.0
+3.7
+1.2
AdaShare
-66.7
-4.3
+3.8
+5.3
-15
-24
+1.1
ATteMTL
45.11
+0.2
+8.0
+7.8
+5.3
AutoMTL
-50.1
+3.0
+8.2
+0.0
+3.0
+7.1
+4.3
t: Semantic Seg., 12: Surface Normal, ta: Depth Est., 4:: Keypoint Det., is: Edge Det..
model in all baselines and in our approach for fair comparisons. We use Ceeplab-ResNet-te as the backbone model and the Atrous Spatial Pyramid Pooling (ASPP) architecture as the task-specific head (7). Both of them are popular architectures for pixel-wise prediction tasks. We also evaluate the effectiveness of AutoMTL on MobileNetV2 [39] and MNasNet [45).
4.2 Results
Performance Comparison. reppor
report the task performance on each dataset respectively. For CityScapes, both the absolute and the relative performance of metrics are reported (see Table Due to the limited space, only the relative performance is reported for NYUv2 and -in-TTakkyoom (see Table and
According to Table AutoMTL outperforms all the baselines on 4 metrics (bold) and is the second- best on 2 metries (underlined) in terms of task performance. With 17.7% increase in the number of model parameters, the task performance of AutoMTL is far better than the vanilla Multi-Tase baseline. Compared the soft-parameter sharing methods, Stoss- Sluice, and NDDR CNN, which are unable to reduce the memory overhead, AutoMTL could achieve higher task performance with fewer parameters. When comparing with DEN and AdaShare, the most competitive approaches in MTL, AutoMTL is better in terms of the task performance but with more parameters (11.7% 1..7%%%.77/ 17.7%). This is because, unlike DEN and AdaShare which pack tasks into the given backbone model, our search space allows each task to select more task-specie operators to increase the capability of the backbone model. It turns out that a small amount of increase in model parameters could translate to a significant gain in task performance.
The superiority of AutoMTL can be observed more clearly in Tables 2 when more tasks are jointly trained together. AutoMTL outperforms most of the baselines in both task performance and model size. When compared with sfate-of-the-ar NAS b-asedd NAS-based MTL methods, DEN, AdaShare, and Learn to Branch, AutoMTL could achieve substantial increase in task accuracy with only few more parameters. Although DEN and AdaShare need fewer model parameters, the representation power of their multi-task models is limited by the backbone model due to their non-stretchable search space, making essential for users to select a suitable backbone model with sufficient capacity for multiple tasks. This problem goes worse when the number of tasks increases. As shown in Table 3 DEN
7

---

Media
Aprokitieee for AuCuRTie
20.4
0.033
Acc.
Abs
8-
Figure 2: Ablation Study on CityScapes. The figures show the distributions different evaluation metrics for three groups of multi-task models. The orange bar corresponds to the group of models generated from Random policies; the green and blue bars correspond to those sampled from the trained policy with or without the policy regularization (AuloMTL. w/o freg and AutoMTL. w/ Lreg).
Samantic Seg.
shared
specific
skp
10
15
Layer
Depth E-z.
#hared
specificc skp
10
15
Layer
(a) Learnod policy distributions
Depth
(b) Sampled feature sharing pattern
Figure 3: Policy Visualization for CityScapes.
and AdaShare have limited task performance improvement or even suffer from task performance degradation (see columns At2 and Ata) the Taskonomy dataset with five tasks. Similarly, the tree-like multi-task model search space Learn to Branch limits the flexibility of the sharing patterns in its generated multi-task model, causing severe task interference shown in columns Als to Als in 3 In contrast, AutoMTL could generate multi-task model with larger capacity if necessary, leading to higher task performance, 13.6% higher than DEN, 3.15 %% than AdaShare, and 3.2% than Learn to Branch.
Furthermore, for semantic segmentation in NYUv2 (see columns At in Table 2 and surface normal prediction Taskonomy (see columns Afz in Table the performance of almost all the multi-task baselines worse than the Single-Task baseline. It indicates that this particular task is negatively interfered by the other tasks when sharing parameters across them. In contrast, AutoMTL is still able to improve the performance of the two tasks because they tend select more task-specific operators in our search space in order to reduce interference from the other tasks.
We also compare the inference time of different multi-task models in Table AutoMTL could achieve a shorter inference time than indepen- dent models, soft- patt-parmmeter sharing methods, and DEN. It is because AutoMTL allows some computation reuse when consecutive initial lay- shared among tasks and uses skip connec- tions to further reduce the computation overhead. AutoMTL also achieves competitive inference speed compared with AdaShare even though we provide additional task-specific options in the multi-task models.
Table 4: Inference Time (ms).
Model
CityScapes tasks) NYUv2 (3 tasks)
Ind. Models
71.01
107.65
Multi-Tash
29.52
32.43
Cross- Stitch
71.01
107.65
Sluice
71.01
107.65
NDDR
67.41
101.89
MTAN
98.99
133.33
DEN
87.05
127.41
AdaShare
56.79
71.06
AutoMTL
60.84
80.41
The time of each training stage in terms of GPU hours is reported in Table The experiments were conducted on an Nvidia RTX8000. Notice that the time cost of our architecture search process (the policy-train stage) is 12-13 GPU hours CityScapes, 36-37 GPU hours NYUv2, and about 120 GPU hours on Taskonomy, which are 13 ~ 133.X faster than FBNetV5 48 (1600 GPU hours on V V100). state-of-the the-ttherrtt art multi-task architecture search framework.
"Since the work is not open- sperceodrced, we have no task performance comparison with it and the search costs here are extracted from the original paper. RTX8000 and V100 have similar computation capability and are hence comparable.

---

Table 5: Time Cost of Training Stage (GPU hours).
Ablation Studies. We present ablation stud- ies to show the effectiveness of the architecture search process (the policy-train stage) and the proposed policy regularization term (....2). In Figure 2 we use boxplot to show the distri- butions of different evaluation metrics for three groups of multi-task models. The orange group (Random) of models are generated from policies that are randomly initialized without policy-train, while the green (AutoMTL w/o Ereg) and the blue (AutoMTL w/ Creg) groups are sampled from policies after the policy-train stage. The policy of the blue group is trained with the regularization but that of the green group is not. We generate six different models with seed 10 ~ 60 in each group to compare their performance with less bias.
Stage
CityScapes NYUv2 Taskonomy
pre-train 8-9
20-21
~55
policy-train
12-13
36-37
~12
post-train 14-15
44-45
140
We make two main observations from Figure First, both AutoMTI. w/o Greg and AutoMTI. w/ Creg achieve better task performance than Random in terms of the mean and the standard deviation of all the evaluation metries. It indicates that the architecture search process is necessary and effective in predicting good sharing pattern among tasks, Second, the mean of all metrics for AutoMTL w/ Creg is also better than AutoMTL w/o Creg. indicating that the proposed policy regularization term plays an important role in improving task performance. It echos the well-recognized benefits of parameter sharing among tasks in reducing overfitting and improving prediction accuracy.
To further illustrate the benefits of the proposed policy regularization term Lregs we provide more quantitative results on CityScapes with different Xreg in Table 6 We also list the Single-Task model and AdaShare comparison. All the other experiment settings including the training pipeline and the cypper-parameee setting remain the same.
Table 6: Quantitative Results on CityScapes with Different Xreg. (Abs. Prf. & Rel. Prf.)
# Params 4
Semantic Seg.
Depth Estimation
Model
Pixel Acc. f
Error
8, within f
At 1
Abs. (M) Rel. (%)
mloU t
At1
At2 t
Abs.
Rel.
1,25
1.25²
1,25
Single-Task
42.569
36.5
73.8
0.026
0.38
57.5
76.9
87.0
AdaShare
21,285
-50.0
40.6
74.7
+6.2
0.018
0.37
71.4
86.8
93.1
+15.5
+10.9
Areg = 0.01
23.626
-44.5
43.4
74.9
+10.2
00.021
0.36
68.4
85.5
92.7
+12.2
+11.2
Areg 0.001
25.584
-39.9
43.3
74.8
+10.0
0,020
0.34
71.1
87.5
93.7
+15.7
+12.9
Areg 0.0005
28.819
-32.3
42.8
74.8
+9.3
0.018
0.33
70.0
86.6
93.4
+17.1
+13.2
Aveg 0.0001
30.735
-27.8
40.4
74.4
+5.7
0.019
0.37
68.1
84.5
92.0
+12.7
+9.2
As the Areg becomes larger, the probability of parameter sharing, especially those in initial layers, is higher, leading to the fewer number of parameters in the identified multi-task model but relatively lower task performance because of task interference. Users could adjust Xveg to control the tradeoff between resource efficiency and task accuracy. If the computation budget is limited, they could use larger Areg for a more compact model while sacrificing task accuracy. If the users call for the best task performance, they could tune Xreg to find the optimal setting.
Policy Visualization. We further visualize the learned sharing policies to reveal insights on the discovered multi-task architecture. Figure ssoows the visualization of the learned policy distribution and the feature sharing pattern on CityScapes. For each layer in each task, Figure 3 a) illustrates its policy distribution introduced in Section 3.2 A brighter block indicates higher probability of that operator being selected. The figure indicates that tasks tend to share bottom layers. Besides, Figure provides feature sharing pattern sampled from the learned policy distribution. The red arrows connect the operators used by semantic segmentation and the blue ones correspond to depth estimation. Operators that are not selected are grmi-trrnnparent. Overall, semantic segmentation is more likely to share operators with other tasks than depth estimation. Depth estimation has more than 25% of operators are skip connections, implying that this task prefers more compact model than the backbone. Skip connections and operator sharing among the two tasks decrease the number of parameters in the multi-task model.
We also demonstrate the generality of AutoMTL by conducting experiments on CityScapes with two other typical backbone models Mobile 39 and MNasNet 45). Without changing hyperparameters on this dataset, AutoMTI. achieves 7.4% and 9.5% higher relative task performance with 33.5% and 35.9% fewer model parameters than the single-task bascline
Results on Other Backbone Models.

---

and 6.5% and 11.1% higher relative task performance than the Multi- Task baseline for MobileNetV2 and MNasNet respectively.
User Study on Ease-of-Use. There is no re-implementation cost in AutoMTTL when the backbone model changes. The compilation of given backbone specified in prototxt format multi-task supermodel takes only ~0.6s. On the contrary, users with proficient Py PyTorch skills in our user study still expect 20 ~ 40 hours to complete re-implementation of the state-of-the-arl NAS-based MTL approach Adashare
5
Conclusion
In this work, we propose the first programming framework AutoMTL that generates compact multi- task models given an arbitrary input backbone CNN model and set of tasks. AutoMTL features multi-task supermodel compiler that automatically transforms any given backbone CNN into multi-task supermodel that encodes the proposed stretchable architecture search space. Then through policy approximation and regularization, the architecture search component effectively identifies good sharing policies that lead to both high task accuracy and memory efficiency. Experiments on three popular multi-task learning benchmarks demonstrate the superiority of AutoMTL. over state-of-the-art approaches in terms of task accuracy and model size.
Limitations and Broader Impact Statement. Our research facilitates the adoption of multi-task learning techniques to solve many tasks at once in resource-constraint scenarios. Particularly, we offer the first systematic support for automating efficient multi-task model development for vision tasks. The support of other Al tasks (e.g., NLP tasks) is left as future work. It has a positive impact on applications that tackle multiple tasks such as environment perceptions for autonomous vehicles and human-come computer interactions in robotic, mobile, and IoT applications. The negative social impact of our research is difficult predict since it shares the same pitfalls with general deep learning techniques that suffer from dataset bias, adversarial attacks, faimess, etc.
Acknowledgement. This work is supported by UMass Amherst Start-up Funding and Adobe Research Collaboration Grant.
References
[1] Chanho Ahn, Eunwoo Kim, and Songhwai Oh. Deep elastic networks with model selection for multi-task learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6529-6538, 2019.
[2] Jonathan Baxter. A model of inductive hias learning. Journal of Artificial Intelligence Research, 12:149- 198, 2000.
[3] Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cans: Learning specialist and generalist convolution kernels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1385 1394, 2019.
[4] David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Lue Van Gool. Automated search for resource-efficient branched multi-task networks. arXiv preprint Xiv:20008 10292, 2020.
[5] Rich Caruana. Multitask learning. Machine Learning, 28( 1):41-75, 1997.
[6] Richard Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the 10th International Conference on Machine Learning, pages 41-48. Morgan Kaufmann, 1993.
[7] Liang-Chiel Chen, George Papandreou, lasonas Kokkinos. Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-844 848, 2017.
[8] Zhao Chen, Vijay Badrinarayanan, Chen-Ve Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normal- ization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR, 2018
[9] Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stren stream feature aggregation and geometric loss strategy for multi task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
10

---

[10] Marius Cordts. Mohamed Omran. Sebastian Ramos, Timo Rehfeld. Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3213-3223, 2016.
[11] Camille Couprie, Clément Farahet, Laurent Najman, and Yann LeCun. Indoor semantic segmentation using depth information, arXiv preprint arXiv:/ 2013.
[12] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second- order functional knowledge for better option pricing. Advances in Neural Information Processing Systems, pages 472 478, 2001.
[13] Nikita Dvorikk Konstantin Shmelkov, Julien Mairal, and Cordelia Schmid. Blitznet: A real-time deep network for scene understanding. In Proceedings of the IEEE International Conference on Computer Vision, pages 4154 2017.
[14] David Eigen and Rob Fergus, Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In Proceedings of the IEEE International Conference on Computer Vision, pages 2650-2658, 2015.
[15] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. arXiv preprint arXiv: 1406.2240,, 2014.
[16] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503-27516, 2021.
[17] Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architec- ture search towards general- purpose multi-tas learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11543-11552 2020.
[18] Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan Yuille. Nddr-enn: Layerwise feature fusing in multi-task cans by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3205-3214, 2019.
[19] Pengsheng Guo, Chen Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference Machine Learning, pages
PMLR, 2020.
[20] Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image retrieval with dual attribute-aware ranking network. In Proceedings the IEEE International Conference Computer Vision, pages 1062-1070, 2015.
[21] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint 161 144, 2016.
[22] Brendan Jou and Suih-Fu Chang. Deep cross residual learning for multitask visual recognition. In Proceedings of the 24th ACM International Conference on Multimedia, pages 998-1007, 2016.
[23] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty weigh losses for scene geometry and semantics. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
[24] Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6129 6138, 2017.
[25] Isabelle Leang, Ganesh Sistu, Fabian Bürger, Andrei Bursuc, and Senthil Yogamani. Dynamic task weight- ing methods for multi-task networks in autonomous driving systems. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-8. IEEE, 2020.
[26] Wei-Houg Li and Hakan Bilen. Knowledge distillation for multi-tat task learning. In European Conference on Computer Vision Workshop, pages 163-17 176. Springer, 2020.
[27] Jason Liang, Elliot Meyerson, and Risto Müikkulainen. Evolutionary architecture search for deep multitask networks. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 466-473, 2018.
[28] Xi Lin, Hui-Ling Zhen. Zhenhua Li, Qing-Fu Fu Zhang, and Sam Kwong. Pareto multi task learning. Advances in neural information processing systems, 32, 2019.

---

[29] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances Neural Information Processing Systems, 34:18578-18890, 2021.
[30] Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:202..0309, 2022.
2022.
[31] Shikun Liu, Edward Johns, and Andrew J Davison. End to-end multi-task learning with attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1871-1880 2019.
[32] Mingsheng Long, Zhangjic Cao, Jianmin Wang, and S Yu Philip. Learning multiple tasks with multilinear relationship networks. In Advances in Neural Information Processing Systems, pages 1594 1603, 2017.
[33] Kevis-Kokitsi Maninis, Lija Radosavovic, and lasonas Kokkinos. Attentive single-tasking multiple tasks. In Proceedings of the IEEE/CVF Conference on Compuder Vision and Puttern Recognition, pages 1851-1860, 2019.
[34] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994 4003, 2016.
[35] Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, and Ian Reid. Real-time joint semantic segmentation and depth estimation using asymmetric annotations. In 2019 International Conference on Robotics and Automation (ICRA), pages 7101-7107. IEEE, 2019.
[36] Rajeev Ranjan, Vishal M Patel, and Rama Chellappa. Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41()):21--33, 2017.
[37] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint orXiv:1706.000 050988 2017.
[38] Sebastian Ruder. Joachim Bingel. Isabelle Augenstein, and Anders Søgaard. Latent multi-task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4822-4829, 2019.
[39] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4510-4520, 2018
[40] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-ohjective optimization. Advances in neural information processing systems, 31, 2018.
[41] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. In European Conference on Computer Vision, pages 746-760. Springer. 2012.
[42] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pages 913229332 PMLR, 2020.
[43] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-tast learning. arXiv preprint 191. 2019.
[44] Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint 1912 068844, 2019.
[45] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
[46] Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Lue Van Gool. Branched multi-task networks: dcciding what layers to sharc. arXiv preprint 1904::990.0020 02920, 2019.
[47] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans. Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2021.
[48] Bichen Wu, Chaojian Li. Hang Zhang, Xiaoliang Dai, Peizhao Zhang, Matthew Yu. Jialiang Wang, Yingyan Lin, and Peter Vajda. Fbnetv5: Neural architecture search for multiple tasks in one run, arXiv preprint arXiv:2111.10066 2021.
12

---

[49] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman. and Chelsea Finn. Gradient surgery for multi-task learning, Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
[50] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 3712-3722, 2018.
[51] Xinbang Zhang, Zehao Huang, Naiyan Wang. Shiming Xiang, and Chunhong Pan. You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):2891-22 2904, 2020.
[S2] Zhang and Qiang Yang. survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
13


Reviews:
[{"review_id": "mYBcR9Up_RB", "reviewer": "Reviewer_koYc", "summary": "The AutoMTL system integrates the Multi-Task Supermodel Compiler (MTS-Compiler), which transforms a user-provided backbone CNN into a multi-task supermodel that encodes the architecture search space. The authors propose a Stretchable Architecture Search Space that offers flexibility in deriving multi-task models with a wide range of model capacity based on task difficulties and interference, and further propose a novel data structure called Virtual Computation Node to encode the search space and enable compiler-based multi-task supermodel transformation. They propose a policy regularization term on the sharing policy to promote parameter sharing for high memory efficiency.", "questions": "How long does it take to train the model? It seems that the training pipeline is complicated. It would be better to show the comparison of training time among baselines.", "limitations": "The method is interesting and the model can be adapted to other backbone without implementation cost. However, it seems that the training time and inference time is longer than the baseline. The method cannot achieve best on all evaluation metrics.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a  resource-efficient architecture design, which determines what parameters of a backbone model to share across tasks to optimize for both resource efficiency and task accuracy.\n2. AutoMTL takes as inputs an arbitrary backbone CNN and a set of tasks to learn, and then produces a multi-task model that achieves high task accuracy and small memory footprint.\n3. This method considers parameter sharing in MTL at the operator level. The operator-level sharing granularity not only enables the automatic support of arbitrary CNN backbone architectures, but also leads to a stretchable architecture search space that contains multi-task models with a wide range of model capacity.\n\nWeaknesses:\n1. Even though AutoMTL achieves better results, the number of model parameters is larger than Multi-Task baseline.\n2. The inference time of AutoMTL in CityScapes is longer than Multi-Task and AdaShare.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fU1bxtl8533", "reviewer": "Reviewer_yPp6", "summary": "The paper focuses automatically learning to produce efficient neural network architectures for jointly performing multiple tasks given an arbitrary backbone and a set of tasks. The proposed method can efficiently generate efficient network for multi-task learning and achieve good performance on three multi-task learning benchmarks. ", "questions": "1. The idea of learning multi-task architecture is very similar to Adashare while the proposed method has one more operation in each layer (task-specific) such that in a layer each task can keep the layer task-specific (Adashare only have skip or not which make the layer shared across tasks or shared by a subset of tasks or skipped by all tasks). It enhance the power of learning more powerful representations however, it has a larger action space to learn and it can be computationally costly. In this point of view, the novelty of this work is limited.\n\n2. One of the challenge in multi-task learning is the optimization of the multi-task learning network, which is not discussed in this paper and need to be discussed. The optimization of multi-task learning strategies, such as loss weighting, knowledge distillation strategies and gradient-based methods can greatly affects the performance of multi-task learning. Is the optimization of multi-task learning also affects the the network architecture learning and the training of the final network?\n\n3. As mentioned above, the optimization can greatly affects the performance of multi-task learning and it may differs for different architectures. Do the authors tune the hyper-parameters lambda (losses weight) for the proposed method and apply them to other methods? It can be more clear that if all network learned with uniform loss weights or also tune the loss weights for different architectures.\n\n4. The proposed method has more parameters (around +20%) than Adashre while the gain is not significant. The performance gains on NYUv2 dataset and Taskonomy are not significant (Table 6 and Table 7 in appendix). It also looks strange that MTAN uses much more parameters than the vanilla multi-task learning baseline in all datasets and single-task learning in NYUv2. The authors are suggested to check again this.\n\n5. The when applying the proposed method using MobileNetV2 and MNasNet, only Vanilla Multi-task learning baseline and single-task learning baseline are compared, how well are other methods?", "limitations": "The limitation of methods and potential negative societal impact are discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper develops an efficient programming framework that can automatically learn to produce efficient neural network architectures for jointly performing multiple tasks given an arbitrary backbone and a set of tasks. \n\n2. It is shown that the method performs better than related multi-task architecture learning approaches in most metrics in three datasets. \n\n3. And the authors also show that the proposed framework is easier than related literatures e.g. Adashare, for producing multi-task learning network\n\nWeaknesses:\n\n1. The main weaknesses of the paper is its limited novelty as the idea is quite similar to Adashare and its high computational cost for searching the multi-task learning architectures. (Detailed in Questions section.)\n\n2. Another limitation is the effect of the optimization in multi-task learning is not considered while the optimization in multi-task learning can greatly influence the performance of multi-task learning (See Question section for more detailed).\n\n3. The performance gains are not significant while the method requires more parameters than Adashare. (See Questions section for more detailed.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XM_A19g3xCb", "reviewer": "Reviewer_av4d", "summary": "The authors propose a framework for automatically designing and training multi-task models. Given a set of tasks and a particular backbone network, the framework learns when to share layers for the best performance. For each task, the option is given to either use a shared version of a backbone operation, use a task-specific version, or skip the operation altogether. The network is trained with Gumbel-Softmax to learn in a differentiable way which of the three choices to make for every operation. At the end, discrete decisions are assigned and the final network design is trained from scratch. This approach performs well against other multi-task methods on the benchmarks CityScapes, NYUv2, and Taskonomy.\n", "questions": "- Not sure a definition of \"operator\" is ever provided? Is each parameterized operation (conv layer, normalization function) treated separately?\n- Might be nice to see absolute numbers for # params in addition to relative percent drop in Tables 1-3", "limitations": "limitation + social impact discussion is included in the paper", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The authors propose a straightforward and well-motivated search space for multi-task architectures, and show how to produce assignments in this space for an architecture that performs well against other multi-task methods while not requiring much additional parameter overhead.\n- The approach can easily be adopted with new models, and the authors seem to have put a lot of emphasis on coding up a framework that’s easy to use\n\nWeaknesses:\n\nThere are a couple other ablations that would have been nice to see to round out the experiments and get a clearer impression of the benefits of the proposed approach:\n\n- I’m curious about the performance of a baseline like [i] trained with Gumbel-Softmax instead of RL. In contrast to this paper with 1 shared op + independent task-specific ops, the routing can instead be learned through some number of available shared ops.\n- Is the three-stage training process necessary? How important is the initialization with the pre-trained model before beginning to learn the policy? Does the final model need to be trained from scratch or can the discrete subset of layers be sampled from the “policy-train” stage and finetuned from there?\n- The regularization to encourage parameter sharing makes sense and seems to help from the ablations, I wonder about the weighting to encourage sharing particularly in the initial layers, how much does that help?\n\nI don’t necessarily expect that all of those experiments would have been done or need to be done, just that the paper would be stronger with a few extra insights into the final design.\n\n[i] Rosenbaum, Clemens, Tim Klinger, and Matthew Riemer. \"Routing networks: Adaptive selection of non-linear functions for multi-task learning.\" ICLR 2018\n\nOverall:\n\nThe proposed method is a nice contribution in the space of multi-task architecture search, the approach itself is not some huge, novel step away from existing methods, but it is well thought out and well executed with a clear paper and solid results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept