Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal
Authors: Naresh Kumar Gurulingan
Elahe Arani
Bahram Zonooz
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-task Learning
Structural Learning
Brain-inspired Neural Network
Neuron Creation
Neuron Removal

Abstract: Multi-task learning has the potential to improve generalization by maximizing positive transfer between tasks while reducing task interference. Fully achieving this potential is hindered by manually designed architectures that remain static throughout training. In contrast, learning in the brain occurs through structural changes that are in tandem with changes in synaptic strength. Therefore, we propose Multi-Task Structural Learning (MTSL) which simultaneously learns the multi-task architecture and its parameters. MTSL begins with an identical single task network for each task and alternates between a task learning phase and a structural learning phase. In the task learning phase, each network specializes in the corresponding task. In each of the structural learning phases, starting from the earliest layer, locally similar task layers first transfer their knowledge to a newly created group layer after which they become redundant and are removed. Our experimental results show that MTSL achieves competitive generalization with various baselines and improves robustness to out-of-distribution data.

Under review as a conference paper at ICLR 2023
MULTI-TASK STRUCTURAL LEARNING USING LOCAL TASK SIMILARITY INDUCED NEURON CREATION AND REMOVAL
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-task learning has the potential to improve generalization by maximizing positive transfer between tasks while reducing task interference. Fully achieving this potential is hindered by manually designed architectures that remain static throughout training. On the contrary, learning the brain occurs through structural changes that are in tandem with changes in synaptic strength. Thus, we propose Mulki-Task Structural Learning (MTSL) that simultaneously learns the multi-task architecture and its parameters. MTSL begins with identical single task network for each task and alternates between task learning phase and a structural learning phase. In the task learning phase, each network specializes in the corresponding task. In each of the structural learning phases, starting from the earliest layer, locally similar task layers first transfer their knowledge to newly created group layer before they are removed. MTSL then uses the group layer in place the corresponding removed task layers and moves on the next layers. Our empirical results show that MTSL achieves competitive generalization with various baselines and improves robustness to out-of ofdiisribbution data.
INTRODUCTION
Artificial Neural Networks (ANNs) have exhibited strong performance in various tasks essential for scene understanding. Single-Taske Task Learning (STL) YYu et al.) [221]] Wang et al.) [2020;] Orsic et al.) 2019) has been largely at the center of this exhibit driven by custom task-specifie improvements. Despite these improvements, using single task networks for the multiple tasks required for scene understanding comes with notable problems such linear increase computational cost and a lack of inter-task communication.
Multi-Task Learning (MTL), on the other hand, with the aid of shared layers provides favorable benefits over STL such as improved inference efficiency and positive information transfer between tasks. However, a notable drawback of sharing layers task interference. Existing works have attempted to alleviate task interference by modifying the architecture (Kanakis et al. (020 Liu et al. 2019), determining which tasks to group together using a similarity notion (Standley et al. 2020
Fifty et al. 202T Vandenhende et al. 2020). balancing task loss functions (Kendall et al. 2018 Liu
ctt al. 2019: Yu et al. 2020 Lin et al. 2019), or learning the architecture (Guo ct al. 2020 Lu et al. 2017). Although these methods have shown promise, progress can be made by drawing inspiration from the brain, which is the only known intelligent system that excels in multi-task learning. The inner mechanisms of the brain, although not fully understood, can guide research in ANNs through simplified notions. Neuron creation and neuron removal (Maile et al., 2022) are simplified notions that can aid in the automated design of Multi-Task Networks (MTNs).
Neuron removal presents the opportunity to start from a dense set of neurons and move toward a sparse of neurons. In the early stages of brain development, neural circuits consist of excess neurons and connections that provide a rich information pipeline (Maile et al. 2022). This pipeline allows neural circuits to learn specialized functions while undergoing neuron removal and synaptic pruning (Riccomagno & Kolodkin 2015). Thereby, moving from a dense architecture consisting of multiple single-task networks sparse multi- architecture could be beneficial. Code will he made available after acceptance.

---

Under review as a conference paper at ICLR 2023
Neuron creation is open-ended operation due to the difficulty involved in deciding where, how, and when to create neurons (Evci et al. 2022). In the brain, local communication between neurons is an important part of learning. Learning rules that modulate synaptic strength are local in nature (Kudithipudi et al. 2022) and local neural activity could be responsible for the creation of neurons (Luhmann al. 20T6) and also neuron removal (Faust al., 2021). We explore local task similarity to drive neuron creation and removal, which together could improve learning.
Structural learning pertains to the learning of the architecture and its parameters simultaneously (Maile et al., 2022). The neural circuitry in the brain changes even during adulthood, undergoing morphological changes induced by structural plasticity (Kudithipudi et al. 2022). Evidently, learning in the brain does not involve static architecture creation followed by modulatio laaiion of synaptic strengths. Instead, architecture changes occur in tandem with changes in synaptic strength. Thus, utilizing structural learning with strategic neural operations could mitigate the effects of task interference and promote generalization in MTI..
Therefore, we propose Multi-Task Structural Learning (MTSL) to simultaneously learn the multi-task architecture and its parameters. MTSL considers entire layers as computation units (Maile et al.. 2022) and performs neuron creation and neuron removal on them. Inspired by the creation of large number of neurons in the developmental stage of the brain, MTSL begins training by initializing each task with its own network. Similar to the brain, the excess layers of each task network provide a rich information flow to inform grouping decisions. Local task similarity is used to guide task learning through the alignment of task representations, and also make decisions on grouping tasks. A positive decision to group tasks induces the creation of a group layer, and the associated task layers transfer their learned knowledge the group layer before being removed. Finally, few epochs of fine-tuning result in a learned MTN which persists the learned parameters for inference.
Contributions. (i) We propose structural learning algorithm for multi-task learning based on aligning local task representations, grouping similar task layers, transferring information from grouped task layers new group layer, and removing the concerned task layers. (ii) We compare against various of-h-of-th methods and show that MTSL shows improved generalization without the need to retrain. (iii) We show that MTSL improves the robustness to natural corruptions. (iv). We present an ablation on the various components of MTSL and show its utility.
2 RELATED WORKS
Although different lines of work, such as architecture modifications (Liu et al. 2019 Kanakis et al. 2020: Misra et al.) 2016), task grouping (Standley et al.) 2020 Fifty et al. 2021; Vandenhende et al. 2020). or task loss balancing (Kendall et al. 2018 Liu et al. 2019: Yu et al. 2020 Lin et al. 2019) address task interference, they use hand-designed architectures that could be suboptimal. A variety of works in the MTL literature propose methods to learn the architecture, and we categorize these works into two groups. One group of works considers learning architectures that are capable of dynamically changing their structure based on the input (Hazimeh et al.] 2021 Ahhn et al. 2019; |Rosenbamm et al. 2018) while the other group learns the branching structure (Guo et al. [2020] Bruggemann al. 2020
Lu et al. 2017 Zhang et al.) 2022 2021 Raychaudhuri et al. (022))
Input dependent dynamic architectures draw inspiration from the brain and provide many benefits, including improved computational efficiency (Han et al./ [2021). DSelect-k (Hazimeh et al., (2221) mixture of experts model that enables selecting spare
of experts to infer an input sample. (Ahn et al., 2019) learn selector network that learns to pick a subnetwork from a large estimator network based on input. In routing networks (Rosenbaum et al. 2018), uusk-dependent agents are trained using reinforcement learning to pick path within a large network to infer an input. While these approaches aim to optimize networks or subnetworks to specialize for a certain distribution of input samples, MTSL aims to optimize shared network for all tasks.
The branching structure of multi-task networks have been learned using different approaches. (Zhang et al. 2022) propose to estimate the accuracy of a branched multi-task network using two task networks with similar branching. They also suggest data structures and methods to ease branching decisions search in arbitrary network similar to (Zhang et al.) (021). Raychaudhuri et al. 2022) propose two controller networks that predict the branching structure and the weights of the cross-tast task edges based on user preferred task importance and budget constraints. (Guo et al. 2020) start from
2

---

Under review as a conference paper at ICLR 2023
dense search space where child layer is connected to number of parent layers. During learning, a distribution over parent nodes is learned with the aid of path sampling. At the end of training. a valid network path is picked and using neuron removal, the neurons no longer part of the valid path are removed. BMTAS (Bruggemann et al. 2020) takes a similar approach to (Guo et al.) 2020) but additionally use resource loss. On the contrary, MTSL involves progressive neuron removals at different intervals during training. The branching structure learning approaches discussed so far explicitly retrain the learned architecture, while MTSL avoids retraining and confirms with structural learning. Like MTSL, (Lu et al. 2017) also avoid retraining and use neuron creation where tasks are split into different branches starting from the output layer to the input layer using inter-task affinities defined based on task error margins. Unlike (Lu et al. 2017). MTSL starts from dense set of neurons and moves towards a sparse architecture. Also, MTSL designed to leverage both neuron creation and neuron removal. Adashare (Sun et al. 2020) learns task specific policies to determine which residual blocks to execute or skip, leading to residual blocks in the encoder specializing in a subset of tasks. Unlike (Guo et al.) 2020 Zhang al.
2022 202T), BMTAS and MTSL, Adashare does not directly learn a branching structure and is specifical ly designed for ResNet.
3 MULTI-TASK STRUCTURAL LEARNING (MTSL)
The tasks be learned together in an MTN bring in diverse information about the input scene. This diverse information can be leveraged to learn representations with improved generalization on all tasks. Design decisions such as which layers to share and where to branch tasks are complex due to their combinatorial nature. This complexity, along with the crucial role of these decisions in the interplay between positive transfer and task interference are reasons that could render manual architecture design suboptimal. Structural learning, on the other hand, learns the architecture along with its parameters, which is likely in line with how the brain learns.
MTSL uses two neural operators, namely neuron creation and newron removal, to aid in structural learning. In early development, the brain has excess neurons that can provide a rich information pipeline for pruned neural circuit to functionally specialize. Likewise, MTSL creates excess neurons by starting from a disparate network for each task. Through the progress of training, the corresponding task neurons in a layer pave the way for specialized group neuron leading to a structural change. In the next sections, we present the finer details of the MTSL algorithm. In Section 3.1 we formalize the problem setup and establish the terminologies that we use in the rest of the paper. Following this section, we discuss how we align task representations in Section 3.2. how we create neurons in Section 3.3 removal of task neurons in Section 3.4 3.4
and the overall MTSL algorithm in Section 3.5
3.1 PROBLEM SETUP
We consider the problem of structural learning where the MTN architecture and its parameters are learned simultaneously. Given the set of T tasks that each has its own single network with layers, our algorithm results in single MTN capable of inferring all the T tasks accurately without the need for retraining. First, we establish the terminologies that are used the rest of the paper. node is a layer that connects one branch to another branch (or to node), and a branch is sequence of layers that follow a node. Initially, the first layer of each single task network is the task node, while the rest of the task network excluding the task head is called the task branch. Similarly, a group of tasks will have a group node and a group branch. task node of particular significance to our algorithm, as tasks can only be fused at the task node. Also, only task nodes that are connected to the same group branch or group node can be fused. At the start of the training, all task nodes are connected to the input image and can be fused. T and G denote task and group, respectively. Additionally, F and F denote the output features of the task node and the group node, respectively. Figure provides the overall schematic of our approach, where the lefimost column illustrates the initial state of our setup using the terminologies defined so far. In the following sections, we discuss the different components of our approach.
3.2 ALIGNING TASK SPECIFIC REPRESENTATIONS
As is evident from Figure D we begin training from single-task networks. Since the encoder of each task is initialized with ImageNet weights, there exists correspondence between task nodes initially.
3

---

Under review as a conference paper at ICLR 2023
Inpul
Inpul
T Nogl
Tj Node
Neuron Removal
23 Node
T4 Node
Node
Node
Node
Ti Node
T32 Node
Node
Grouping
T3
Node
Node
T7
Fi Fealures
F3 Features
Branch
...
Festures
Tj Branch
Tg Branch
Neuron Creation
T. Head
...
Iy Head
Ft Features
F Fentures
Fr Features
T₁ Head
...
Ty Head
Initiel State
Alignment
Next State
Figure 1: Schematic of the MTSL Algorithm. The grey regions (initial state, alignment, next state) are part of the task learning phase while the white regions (neuron creation, and neuron removal) are part of the structural learning phase. During training, our algorithm loops between alignment in the task learning phase followed by neuron creation and neuron removal in the structural learning phase leading to the next state (last column).
During training, task nodes would learn concepts that minimize particular task loss independent of other tasks. This independence likely breaks any correspondence between parameters mapped one-to-one between any two task nodes. This behavior comes from the permutation invariance of neural networks that leads to no guarantee on the order in which concepts are learned (Wang et al. 2020a). Therefore, MTSL aligns the concepts learned by the task nodes and locally increases their similarity using Centered Kernel Alignment (CKA) (Korbllth et 2019)..
CKA used to measure similarity between two feature representations and has been shown to provide meaningful similarity scores. During training, we introduce CKA-based regularization term between task nodes branching from the same group node/branch (or the input). This regularization term, shown in the alignment part of Figure[] is included between all pairs of task node features (indicated by bi-directional arrows) and enforces the task representations to align by serving as an alignment constraint. We use the unbiased CKA estimator (Nguyen et al. 202 to facilitate reliable estimates of CKA with small batch sizes used during training.
(1) The overall loss that is used for training in the task learning phase of our algorithm is provided in Equation [ where the first term (CMTL) represents the multi-task loss which weighted sum of all individual task losses. The second term represents the CKA regularization term (Cok A) that is included with balancing factor and negative sign to maximize alignment between tasks.
C LMTL + X(1 LCKa)
3.3 CREATING GROUP NODES
The overall loss L used during the task learning phase (discussed in Section 3.2) leads tasks to learn similar features while also minimizing the concerned loss. After the task learning phase, MTSL begins the structural learning phase first leverage neuron creation. In the brain, local neuronal activity can affect the structure of the neural circuitry (Luhmann et al.) 2016) and play a role in learning experiences (Kudithipudi et al., 2022). Taking cues from these notions of locality, CKA to gauge the similarity between task node features that represent the local activity of task neurons. These local task similarities are used to induce the creation of group nodes.
First, CKA between all pairs of task node features is calculated after which all possible groups of task nodes are listed. From these groups, set of groups that maximize the total similarity is chosen and
4

---

Under review as a conference paper at ICLR 2023
the groups that satisfy minimum similarity of induce the creation of group neuron. For instance, in Figure I we see that the groups picked are [Ti, T2] and T3 assuming that the total number of tasks T is three. More details regarding the grouping algorithm are provided in the Appendix.
After the task nodes have been grouped based on local task similarity, for cach group, a group node is created. The learned knowledge in the task neurons is used to initialize the created group node using a two-step process. In the first step, the weights of the group node are obtained by averaging the parameters of the concerned task nodes. This averaging justified by the alignment constraint used in the task learning phase that ensures that the corresponding parameters learn similar concepts. Figure deppicts the averaging initialization using a plus symbol. In the second step, MTSL distills the information learned by multiple task nodes into the group node using an attention-base feature amalgamation method (Ye et al., 2019) referred to as ATT. Figure [ depicts this amalgamation process by using arrows from task node features F1 and F22 to group node feature F.
1 N ATTA
LKA =
The knowledge amalgamation objective LKA A is provided in Equation 2 assuming that there are N tasks grouped together. ATTm denotes the attention network consisting of two linear layers with an intermediate ReLU activation and a final sigmoid activation. ATTmel (F) provides a IxC dimensional attention vector which acts weight for the different channels of J allowing the selective distillation of task features into the group feature.
3.4 REMOVING TASK NEURONS
Starting from a dense set of neurons as in the initial state of MTSL provides the opportunity to leverage a rich information flow originating from diverse task information. Using neuron removal, MTSL moves towards a sparser architecture by removing task nodes that learn similar representations. These locally similar task nodes become redundant once they transfer their knowledge to the group node. The task branch is then disconnected from these redundant task nodes and connected to the group node. As defined in Section [.T] the neurons in the task branch that now connect to the group node become task nodes. These changes are evident in the depicted next state in Figure D
3.5 MTSL ALGORITIIM
Algorithm| T presents the different phases involved in the MTSL algorithm, namely the task learning phase, the structural learning phase, and a fine-tume tee-tnning phase. The task learning phase and the structural learning phase occur alternatively for n number of times, followed by final fine-tuning phase. In the task learning phase, the entire network is trained to minimize multi-task loss and maximize similarity among task nodes described in Section 3.2 The structural learning phase involves neuron creation and neuron removal as discussed in Section 3.3 and in Section 3.4, respectively. F2 determines the number of epochs for which each subsequent task learning phase executed. Similarly, E. determines the epochs for ATT-based knowledge transfer. Considering a total training budget of E' epochs, the task learning phase is executed up to - f epochs where f the minimum epochs allocated for the fine-tuning phase during which the task nodes are no longer forced to align.
EXPERIMENTS
We evaluate the strengths of our approach using two datasets. namely Cityscapes (Cordts al. 2016) and NYUv2 (Silberman et al., 2012). The Cityscapes dataset is an outdoor driving scenes dataset consisting of 2975 training images and 500 validation images. The images are reshaped to a resolution of 256x512 in both training and validation. The NYUv2 dataset consists of indoor scenes with a total 795 training and 654 validation images, respectively. Cityscapes and NYUv2 datasets are also referred to as CS and NYU, respectively. We consider five dense prediction tasks, namely semantic segmentation (S), depth estimation (D), edge detection (E), surface normals (N), and autoencoder (.A). All numbers over three runs. Each of the experiments has been run on one Nvidia Tesla V 100 GPU in a DGX cluster. The ResNet encoder is initialized with the ImageNet pretrained weights and the rest of the
reported on the validation set of each dataset an average
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: MTSL algorithm
Input:
Initial state (as depicted Figure 1. Training budget F,, Minimum fine-tuning budget f:
<- Number of structural learning phases;
Task Learning Epochs E, +- [t', 6² .., 1d), <
Structural Learning Epochs F., +- [s 82 8"1;
while e < E - do Ê - Next value from E;;
1/ Task Learning Phase
for t epochs do Train using loss in Equation
end
e 4 e
Create group nodes using local task similarity; // Structural Learning Phase Average corresponding task nodes to initialize group nodes; Next value from Eg:
for & epochs do
Use ATT to transfer knowledge in corresponding task nodes group nodes;
end
there no more layer in all task branches then
Exit loop;
end
end
for E epochs do
// Fine-Tuning Phase
Fine-tune using multi-task loss LMTLE
end
weights are initialized randomly. The CKA regularization loss weight À is 0.2. Additional training details can be found in Section C in Appendix.
We provide multi-task performance improvement (Vandenhende et al., 2021) using Equation where m and represent task performance multi-task and single task networks, respectively. When higher performance is better is O and when the lower performance better / is 1. We also report multi-task performance improvements in segmentation S and depth tasks as MTL-
SD
1 T
MTTL
(-1) (Mm,i - Mas) /Ma,i
We compare MTSL with state-of-the-atr methods (Section 4.1). evaluate the generalization, inference efficiency (Section 4.2), and robustness MTSL (Section 4.3). We draw insights based on the converged network architecture in Section 4.4 and provide detailed ablation study in Section 4.5
4.1 COMPARISON WITH STATE-OF-TIEE-ART METHODS
We compare MTSL with existing multi-task architectures and methods to learn multi-task architecture branching. Implementations of Cross-stitch (Misra et al. 2016) and MTAN (Liu et al. 2019) have been taken from the repository of /Vandenhende al.) [022] while the implementation of Learning-to-
Branch (LTT; (Guo et al. 2020)) has been taken from LibMTI. (Lin & Zhang, (022). We update L.TB with the resource loss proposed by Branched Multi-Task Architecture Search BMTAS (Bruggemann) al. 2020) from their open source repository obtain the implementation of BMTAS. LTB-R and BMTAS-R refer to the results obtained by retraining the converged architecture from scratch. For all methods we use the same hyperparameter settings and use ResNet18 backbone with DeepLab head.
In Table I we observe that Cross-stitch obtains the best improvement (highlighted bold) over Single Task Networks (STNs). Cross-stitch retains all separate task networks and leams task-specific adapter parameters. These adapter parameters enable the selective transfer of information from other tasks. Additionally, since there is explicit parameter sharing, task interference can be mitigated,

---

Under review as a conference paper at ICLR 2023
Table IID Generalization and inference efficiency comparisons between MTSL and state-of-the-art methods, namely Cross-stitch, MTAN, LTB and BMTAS. LTB-R R and BMTAS-R represent results obtained by retraining the final converged architecture of LTB and BMTAS, respectively. (M) denotes parameter count in millions and GMac denotes Giga multiply-accumulat.
CS
NYU
Network
A MTT.. 1
AMTL
1
# (M)
GMac
A MTL.
AMTL t
# (M)
GMac
STN
79.50
26.73
79.51
29.24
One-Net
-0.841m.s
-1.22.10.3
34.79
7.70
-0.791um1
-4.57
34.80
8.42
Cross-stitch
+2.79
+1.01 10a7
79.50
26.73
+0.731oar
-2.1810.a
79.53
29.24
MTAN
-0.02.0.40
=0.68
36.61
11.13
-0.34
-3.91
36.62
12.18
LTB-R
+0.49
68.15
19.81
-0.13
-0.23 1o.0s
79.17
25.20
BMTAS-R
+0.25
-0.02 20
68.30
21.03
-0. 10
-0.21
79.17
25.20
MTSL-R
+0.17
-0.22
68.37
12.00
-0.39
-3.54
59.98
11.95
LTB
2.95
-1.79
68.15
19.81
-4.7
-5.35
79.17
25.20
BMTAS
20.8
=2.19
68.30
21.03
4.002
-5.07 20.42
79.17
25.20
MTSL
-0.23
-0.34
68.37
12.00
-0.0610.1
-3.21
59.98
11.95
leading increased performance. However, Cross-stitch loses the inference time and memory advantage of using shared parameters. When comparing MTSL with methods that learn branching structures (last group), MTSL outperforms both LTB and BMTAS by a considerable margin in both datasets. For instance, in AMPLL MTSL outperforms LTB by 2.72% and 4.72% and BMTAS by 3.61% 1 and 3.94% in Cityscapes and NYUv2 datasets. Also, with the aid of CKA regularization, MTSL leads to more inference efficient architecture (visualized in Section E. as evidenced from GMac. LTB-R. BMTAS-R and MTSL-R are versions of LTB. BMTAS and MTSL respectively where the final converged architecture is reinitialized and retrained. With additional training, LTBBR R and BMTAS-R can obtain marginal gain over MTSL.. However, they deviate from structural learning the original trained weights no longer relevant and are discarded. MTSL-R is more efficient in terms of inference and shows a performance comparable to that of LTB-R and BMTAS-R, Note that MTSL is designed for structural learning and shows clear performance and inference efficiency improvements over LTB and BMTAS.
The methods discussed far only consider learning branching structure in the encoder. For the upcoming experiments, we extend MTSL to an encoder-decoder architecture with a total of ten locations, six in the encoder and four in the decoder where task nodes participate in neuron creation and removal. The encoder is based on ResNet18 8 and the decoder has ResNet blocks.
4.2 IID GENERALIZATION AND INFERENCE EFFICIENCY
MTSL algorithm begins from STNs and could potentially end in a network with all encoder and decoder layers shared between all tasks (referred to as One-Net). These two possible networks are considered as the baselines for evaluating our approach. Table Table2shows shows the generalization performance of the baselines and MTSL. MTSL achieves better generalization than One-Net in most cases, while being close to One-Net in inference efficiency. as seen in the GMac and parameter count columns. In NYUv2, MTSL even rivals the performance of STN in both S and 'D tasks. We hypothesize that the improved generalization can attributed to brain-inspired aspects of the MTSL algorithm, such as local task similarity and the change of dense to sparse architecture
The training time of MTSL is higher than that of One-Net because the training epochs required for knowledge amalgamation. However, this only adds 34 additional epochs in training where only the network parts up until the task nodes is involved in the computation. For a fair comparison, we train One-Net for 34 more epochs and get the One-Net-L baseline. We note that MTSL also improves generalization over One-Net-L. Overall, we see that with only a fractional increase in training costs, MTSL algorithm provides learned network with better generalization.
7

---

Under review as a conference paper at ICLR 2023
Table 2: IID generalization and inference efficiency comparisons between MTSL and baselines. MTSL performs better than One-Net in most cases and achieves an inference efficiency close to One-Net. # (M) denotes the number of parameters in millions.
Network
S1
D
E
N
A
MTTL
AMTL t
# (M)
GMac
STN
60.87
6.37
2.03
0.61
0.05
107.10
31.03
One-Net
60.34.war
6.76.0m1
0.04.ca
0.5Y
0.06.a.0
-3.47
-9.65 mm.s
21.70
7.06
One-Net-L
60.71 lanm
6.75
0.04
0.59
0.06
-3.15
29.39
21.70
MTSL
60.68.e.m
6.52.am1
0.04.nme
0.60...e
0.06.s.0
-1.35
-7.04.0m
22.86
8.96
NYU
35.63
52.70
0.06
u.80
0.14.aa.0
107.10
33.99
One-Net
34.15
13.44.n30
0.06
0.74
0.17
-2,77
-9.82
21.70
7.77
One-Net-I.
34.42
53.36
0.06game
0,74
0.17
-2.77
21
21.70
7.77
MTSL
35.50 .o2s
52.46.ud 10.2
0.06
0.73
0.16
+0.06
-6.21 10.00
22.81
Table 3: Robustness natural corruptions under four categories. MTSL shows better robustness compared to One-Net in most cases.
SD A MTL f
ATTL f
Network
Noise
Blur
Weather
Digital
Noise
Blur
Weather
Digital
One-Net MTSL
-5.78
-1.32
-3.81
-2.15
+3.02
-7.73
-1.36
-12.22
-3.03
+0.21
+0.49
+1.42
+2.92
-3.72
+2.60
-7.07
NYU
One-Net MTSL
+10 +10.25
-3.71
-3.27
-2.72
+15,72
-8.51
-11.79
-10.35
+1.92
-2.65
-0.84
-1.14
+13.50
-8.03
+3.22
-7.54
4.3 ROBUSTNESS TO NATURAL CORRUPTIONS
MTSL provides an improved generalization, likely due to motivation originating from abstract notions of the brain. Given that the brain is effective in discerning noise from semantics, natural question to ask whether MTSL can lead to improvements robustness to natural corruptions. Table shows the robustness of the baselines and MTSL to various natural corruptions (Hendrycks & Dietterich 2019) categorized into four types, namely noise, blur, weather and digital. Under each corruption category, the average across five different severity levels is taken. We see that the MTSL network exhibits better robustness compared to One-Net in most cases, especially in the weather and digital category of the Cityscapes dataset. These results further demonstrate that MTSL presents compelling case for the utility of drawing inspirations from the brain.
4.4 CONVERGED NETWORK ARCHITECTURE AND TASK GROUPS
Assumptions about the data used for training play a pivotal role determining the learned repre- sentations. In the brain, exposure to the nature of experiences determines the way neural circuitry develops (Kudithipudi et al. 2022). Thus, the resultant architecture obtained with MTSL on two datasets would likely differ. Figure visualizes the One-Ne architecture and the learned architectures in Cityscapes and NYUv2. Evidently, after the first branch from layer six, the task groups and the branching structure emerging the two datasets different. Semantic segmentation (S) and edge detection (E) grouped together in both datasets follows intuition as edge detection requires predictions of semantic edges. However, the emergence of certain groups of tasks such depth and autoencoder in NYUv2 is counter-intuitie similarity and inherent biases in the dataset can also impact the optimal architecture. In Section we analyze the sensitivity of the converged architecture to random seeds and initialization.
In addition to task relationships, these results show that local task
4.5 ABLATION STUDY
To determine the effectiveness of the different components involved in MTSL, we perform systematic evaluations and provide the results in Table 4 When alignment is not used, the task representations diverge largely around layers 3 and 4. As a result, the task nodes no longer become sufficiently similar to be fused. Therefore, the resultant architectures in the first three perform similarly
8

---

Under review as a conference paper at ICLR 2023
Table 4: Effect of alignment (Align), average initialization (Avg), and attention based knowledge amalgamation (ATT). In the first three rows, without alignment, the resultant networks remain close the STN. the last three rows, the alignment results networks closer to One-Net.
CS
NYU
Align
Avg
ATT
A MTL
A
AMTL
# (M)
GMac
SD
MTL
A AMTL
# (M)
GMac
-0.35
-2.25
97.37
17.02
+0.38
-4,24
69.94
17.87
-0.05
-1.92
97.37
17.02
+0.57
-4.23
95.97
17.87
-0.39
-2.12
95.97
16.30
+0.31
-4.25
95.97
17.87
-2.29
-7.67
22.56
8.71
-1.26
-6.68
22.97
10.32
-3.62
-9.66
22.51
8.60
-2.36
-6.71
23.10
10.90
-1.35
-7.04
22.86
8.96
+0.06
-6.21
22.81
9.89
10 network layers
Input
N A
D
A
E
N
S D N A
Cityscapes
NYUv2
One-Net
Figure 2: Visualization of the One-Net architecture and the MTSL resultant architectures. In both datasets, tasks branch out at layer six but differences emerge in the branching structure in the following layers. The oval shapes the end of each path represents the task heads.
and have inference requirements close to STN. On the other hand, in the last three rows, with the help of alignment, the resultant architectures undergo neuron creation and removal in more layers and approach close to the inference efficiency of One-Net. Of these, the last row constitutes all the components used MTSL and results in the best performance. Note that although we use CKA for alignment and attention for knowledge amalgamation, various other methods can also be used.
5 CONCLUSION
Inspired by the notion that both the structure of the neural circuits and the associated synaptic strengths change together in the brain, we proposed Multi-Task Structural Learning (MTSL). MTSL relies on local similarity-induced creation of group neurons and removal of task neurons. We showed that MTSL results in networks with improved generalization and robustness while improving the inference efficiency. We discussed the dependence of converged network architectures on local task similarity and dataset. We studied the role of the different components in MTSL and found that enforcing local task similarities results in architectures with better inference efficiency.
Limitations. A fixed epoch schedule is used transition between different learning phases of MTSL. Instead, it can be explored to automatically determine the periods in training in which structural learning is required. In this regard, the local inter-neuron activity in the brain could provide useful cues to automatically drive structural changes. Further, local similarity should likely be an emergent activity contrary to being explicitly enforced as in MTSL. MTSL does not
result of synaptogenesis and synaptic pruning. thereby limiting its ability learn connections. Despite these limitations, MTSL serves as an indication of the potential of extending simplified notions from neuroscience into multi-task learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Chanho Ahn, Eunwoo Kim, and Songhwai Oh. Deep elastic networks with model selection for multi-task learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool. Automated search for resource-efficient branched multi-task networks. In BMVC, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov. Gradmax: Growing neural networks using gradient information. In International Conference Learning Representations, 2022. URL ottpt:::///pnnneiieeenneettt wwwp:/reeieeeeeeeeeeeeeeeeee
wwUO
Travis E Faust, Georgia Gunner, and Dorothy P Schafer. Mechanisms governing activity-depenndet synaptic pruning in the developing mammalian ens. Nature Reviews Neuroscience, 22 (2)65667766 2021.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. In A. Beygelzimer, Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htttps://openreve lww..wewwww....... net. / forum?
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3854-3863, PMLR, 13-18 Jul 2020. httLhhtts///pooceeeee
ngs. wewgsmmr..reese.. pprsss/vll wwwp////uuuue 9/guo20e. html
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, IEEE transactions on pattern analysis and machine intelligence, PP, 2021.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. DSelect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL www. ew. net forum? didtii==kll> K1GJLYY
Xiaoxi He, Zimu Zhou, and Lothar Thiele. Multi-task zipping via layer-wise neuron sharing. In NeurlPS, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Menelaos Kanakis, David Bruggemann, Suman Saha, Stamatios Georgoulis, Anton Obukhov, and Luc Van Gool. Reparameterizing convolutions for incremental multi-tass learning without task interference. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, pp. 689-707, Cham, 2020. Springer International Publishing. ISBN -78-3-000-58565-5.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3519-3529, PMLR, 09-15 Jun 2019. URL https::/
proceedi ngs ml wwtpss//9/korrro.r//ee 1hh1 9a htm
10

---

Under review as a conference paper at ICLR 2023
Dhireesha Kudithipudi, Mario Aguilar- Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja. Nick Cheney, Jeff Clune, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196-110 2022.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL https: httppatt//pppttttt /ttpa:::/pppeeeeieeeee net / Forum? 1d=
Mikhail lu, Leontev, Viktoriia Islenteva, and Sergey V. Sukhov. Non-iterative knowledge fusion in deep convolutional neural networks. Neural Processing Letters, 51:1-22, 2019.
Wei-Hong Li and Hakan Bilen. Knowledge distillation for multi-task learning. Proceedings of the European Conference on Computer Vision Workshop on Imbalance Problems in Computer Vision, 2020.
Baijiong Lin and Yu Zhang. LibMTL: A python library for multi-task learning, arXiv preprint arXiv:2203. 14338, 2022.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learn- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
http://aaaaanarrrrpoooooooa/a//p wtwp:/owwee.iinn...eeeeeeoeeeeee neurips wctpppppe//pp/////c.ee /21/2//199////
pdf
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1871-1880, 2019.
Yongxi Lu, Abhishek Kumar. Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1131-1140, 2017. doi: 10.1109/CVPR.2017.1222
Heiko J Luhmann, Anne Sinning, Wei Yang, Vicente Reyes-Puerta, Maik C Stüttgen, Sergei Kirischuk, and Werner Kilb. Spontaneous neuronal activity developing neocortical networks: from single cells to large-scale interactions. Frontiers in neural circuits, 10:40, 2016.
Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowledge amalgamation from heterogeneous networks by common feature learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Sihui Luo, Wenwen Pan, Xinchao Wang, Dazhou Wang, Haihong Tang, and Mingli Song. Collabora- tion by competition: Self-coordinated knowledge amalgamation for multi-talent student learning. In ECCV, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. URL
Kaitlin Maile, Luga Hervé, and Dennis George Wilson. Structural learning in artificial neural networks: A neural operator perspective. Transactions Machine Learning Research, 2022. URL Survey Certification.
net Ftwpe///ooooomimimodeddoaahaaaa http://sgssaaaceeooosss/ssssssta
I. Misra, Abhinav Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3994 4003, 2016.
Thao Nguyen, Maithra Raghu, and Simon Korblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021. URL ttttts::/oppnzeeeee nwt/forumt [d=KJNNAkvaata)
11

---

Under review as a conference paper at ICLR 2023
Marin Orsic, Ivan Kreso, Petra Bevandic, and Sinisa Segvic. In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12607-12616, 2019.
Dripta S Raychaudhuri, Yumin Suh, Samuel Schulter, Xiang Yu, Masoud Faraki, Amit K Roy- Chowdhury, and Manmohan Chandraker. Controllable dynamic multi-task architectures. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
11955-10964, 2022.
Martin M Riccomagno and Alex Kolodkin. Sculpting neural circuits by axon and dendrite pruning. Annual review of cell and developmental biology, 31:779-805, 2015.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In International Conference on Learning Representa- tions, 2018. URL
wiw.uummdddiyyddddee R-
Chengchao Shen, Xinchao Wang, Jie Song, Li Sun. and Mingli Song. Amalgamating knowl- edge towards comprehensive classification. Proceedings of the Thirty-Thirl AAAI Confer- ence on Artificial Intelligence and Thirly-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 19/IAAI 19/EAAI 19, AAAI Press, 2019. ISBN 978 17881-57775-800-11 (aai.v33:01.3300 3068. URL https: :ttps://doo 1tt.d//......................... 3301300
doi: 10.1/09/
16w./669966000 aaaii www.aa........11.....
N. Silberman, Derek Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In 2012.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neurul Information Processing Systems, volume 33, pp. 22045-22055. Curran Asso- ciates, Inc., 2020. URL ltttps://proceeddiiii .anggsnnee neurips httpssscppppee/////../a. .dd]]
Trevor Scott Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In JCML, 2020.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what share for efficient deep multi-task learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8728- 8740. Curran Associates, Inc., 2020. URLhttps::/prrceed 2200/F11ee664841e2 0e1.1666866664444122228882881101
neurips wwtpcc/ppp.ppp...e Paper. .
S. Vandenhende, Georgoulis, De Brabandere, and L. Van Gool. Branched Multi-Task Networks: Deciding What Layers To Share. In BMVC, 2020.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, PP. 2021.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- erated learning with matched averaging. In International Conference on Learning Representations, 2020a.
net
Lijun Wang, Jianming Zhang, Oliver Wang, Zhe Lin, and Huchuan Lu. Sde-depth: Semantic divide- and-conquer network for monocular depth estimation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 541-5500 2020b.
Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song. Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2824-2833, 2019.
Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. International Journal of Computer Vision, 129(1 (101111:33113010 -30688 2021.
12

---

Under review as a conference paper at ICLR 2023
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5824 5836. Curran Associates, Inc., 2020. URL htlphtttss://rrceedinng new. cttpppccccccccccc capperrr/ 20laaa000000555300447700-0-00-00
F020/F11e//3ee7aaaaaaaa
Lijun Zhang, Xiao Liu, and Hui Guan. Automtl: A programming framework for automated multi-task learning, 2021.
Lijun Zhang, Xiao Liu, and Hui Guan. A tree-structured multi-task model recommender. In First Conference on Automated Machine Learning (Main Track), 2022. URL URL httttts://ppereeiieeee net /ttr://rrumniddal1Ala.a...
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORKS
MTSL relies on local task similarity to drive group neuron creation and removal of the corresponding task neurons. The learned convolutional filters in different task branches might not align one-on-one due to the permutation invariance of convolutional neural networks (Wang et al. (2020a). Existing works (Wang et al.) 2020a| Leontev et al.) 2019, Singh & Jaggi 2020 He et al.) 2018) use different ways to align the corresponding layers of two models to counteract the permutation invariance. MTSL uses CKA (Kornblith et al.) 2019) to align neurons based on representation similarity. Knowledge amalgamation approaches Li & Bilen 2020 Shen et al., 2019; Luo et al., 2020 2019 [He al. 2018) address distilling the knowledge from multiple learned teachers into a single student. (Ye et al. 2019 create task specific coding at layer the student network using small network for feature distillation. MTSL uses this feature distillation process to exploit the knowledge of the task neurons set to be removed.
B GROUPING ALGORITHM
At the beginning of the structural learning phase, the task nodes are grouped according to the local similarity between task node features. The grouping algorithm used to provide this grouping decision is detailed in Algorithm(2] [ This algorithm provides the best possible task grouping. which subsequently used to determine the creation of group nodes and the removal of task nodes.
Algorithm 2: Grouping
Input:
Task set T and Similarity Sig, i,j E ET,
Group G [tasks];
Grouping 9 + (groups):
Task value in group G is 4 I Sa, where C Git:
Group value Vig 4 1/ E. Vr, where t is the #tasks in group:
Unique Grouping G C G. such that all tasks are present exactly once; for all unique groupings do where g is the #groups in grouping: VB 1 Vg,,,.. end
Vg.
Final grouping + grouping with maximum Vs: Result: Final grouping
C ADDITIONAL TRAINING DETAILS
We provide additional training details to aid with reproducibility. The five tasks used our exper- iments S, D, E, N and A are evaluated with mloU, RMSE, BCE (Binary Cross Entropy Error). Cosine Similarity, and MSE, respectively. The edge detection task concerns the predictions of semantic edges in the scene. We an encoder-decoder architecture where the first four layers of the encoder are initialized with ImageNet pretrained weights, while the last layer of the encoder and the decoder are initialized randomly.
For both baselines and MTSL, we use the same training hyperparameters and evaluation metrics. For training, we the Adam optimizer with learning rate of le-4 for 80 epochs. We use the step-wise learning schedule with steps at epochs 60 and 70. The batch size used is 16, a weight decay of 5e-5 and we equally weigh all task losses (all task losses have a weight of 1). In addition to averaging all parameters to initialize the group node, we also average the optimizer state of the task nodes to get the optimizer state of the group node. All other parameters that are not removed retain their weights, as well as their optimizer state. The grouping threshold used is 0.8 and the number of structural learning phases is set to 10. The task learning epochs for subsequent phases are 2, 2, 2, 2, 4, 4, 8, 8, 8, and 8 and knowledge amalgamation is done for 1, 1, 2, 2, 2, 2, 4, 4, 8, and 8 epochs in the subsequent structural learning phases. The task learning epochs and knowledge amalgamation epochs increased progressively based nature of task representations learned at different stages of training. In early training, all task nodes are in early layers and need to learn low-level features. Our intuition here is that these generic features are likely common across tasks and can be learned fast.
14

---

Under review as a conference paper at ICLR 2023
The förmer suggests use of low epochs in the structural learning phase, and the latter suggests the task learning phase early in the training. As we progress through the training, more epochs would be required to learn task-specific features, suggesting the increase in task- phase epochs. Also, later in the training, features of each task likely diverge becoming more task-specific, suggesting the increase in structural learning phase epochs.
D SENSITIVITY ANALYSIS
D.1 SENSITIVITY OF THE CONVERGED ARCHITECTURES
In the main paper, we showed that the converged architecture differs between the two datasets Cityscapes and NYUv2. This suggests that the algorithm is sensitive to the dataset. In addition, we observe that the converged architectures are also sensitive to the initialization of the encoder and to random seeds. We tabulate different architectures obtained in Table The layer column lists layers in the network, and the corresponding entries in the remaining columns show which tasks have been grouped. For instance, [D, A] means D, and A are grouped together.
Table 5: Sensitivity of the converged architecture to different initialization of the encoder and random seeds in both datasets. The random seeds groupings using ImageNet pretrained weights for the encoder.
Random Seed 2
Layer
Seed 0
Seed [S. N. A] 18. ID, Al (8. (D), [V], [A] [S) [V], [A]
Seed 2
1-6
[S, ES, D. [8, E1, N
ES D. N. A] 18, [D, N. Al [1, N. A] [S, [A]
[S, D.W. A] E, A] D. .]
Cs
7,8
AS, IN] (D)
10
[S,E,
NYU
1-6
ES,
DS.E, [S. [D. A]
[S,E,
A]
7,8.9
[S. [S, N.AI
[S. N.A) (S,E, IS.. A]
D.. A]
10
[8, 181, [D), [A]
D.2 SENSITIVITY GROUPING THRESHOLD
The grouping threshold determines whether or not group of task nodes are similar enough to lead to the creation of a group node and eventual removal the concerned task nodes. The sensitivity of MTSL to different grouping thresholds is shown in Table 6 A high grouping threshold would mean that task nodes are never grouped, and the resultant architecture will be similar to STN terms of GMac and parameters. However, the performance would still be different due to the use of CKA to regularize the task representations. low grouping threshold would mean that task nodes are always grouped leading architectures more similar One-Net. We observe that for threshold of 0.7 or less, MTSL. leads to One-Net architecture, but the performance is not sufficient. A threshold of 0.9 or higher leads slow inference networks. A threshold of 0.8 provides the right trade-off and used for all experiments in the main paper.
D.3 SENSITIVITY TO NUMBER OF TASKS
We evaluate whether or not MTSL leads to improvements over One-Net when only out of the tasks used. To this end, we train multi-task networks for tasks, namely S, E, D and N. The results are tabulated in 7 We observe that even when there are 4 tasks, MTSL improves over One-Net.
E CONVERGED ARCHITECTURES
In Table we provide results for -fate-of-the-art LTB and BMTAS methods. Here, we visualize the converged architectures of LTB and BMTAS along with those of MTSL and provide inferences. FigureBiillssten iillustrates the converged architectures on both the Cityscapes and NYUv2 dataset. We see that LTB and BMTAS generally converged to larger architecture in comparison to MTSL. MTSL is
15

---

Under review as a conference paper at ICLR 2023
Table 6: Sensitivity of MTSL the grouping threshold. For example, MTSL-0.1 denotes MTSL with a grouping threshold of 0.1. # (M) denotes the number of parameters in millions.
Network
S
D
N
A 1L
AMTL 1
ArLL
# (M)
GMac
STN
61.95
6.38
0.0341
0.6108
0.0535
107.10
31.03
One-Net
60.02
6.71
0.0421
0.5941
0.0620
-4.10
-10.06
21.70
7.06
MTSL--0.1
59.98
6.75
0.0439
0.5955
0.0626
-4.48
-11.44
21.70
7.06
MTSL-0.2
59.97
6.68
0.0433
0.5956
0.0629
-3.92
100.98
21.70
7.06
MTSL-0.3
59.97
6.70
0.0437
0.5961
0.0629
-4.06
-1.225
21.70
7.06
MTSL-0.4
60.32
6.67
0.0436
0.5960
0.0626
-3.59
-10.90
21.70
7.06
MTSL-0.5
59.95
6.68
0.0437
0.5963
0.0626
-3.95
-1.009
21.70
7.06
MTSL-0.6
60.31
6.72
0.0431
0.5967
0.0628
-3.98
-1081
21.70
7.06
MTSL-0.7
59.48
6.68
0.0432
0.5965
0.0625
-4.32
-10990
21.70
7.06
MTSL-0.8
60.64
6.52
0.0415
0.6023
0.0565
-2.13
-6.59
22.99
9.53
MTSL-0.9
60.04
6.46
0,0359
0.6074
0.0559
-2.12
2991
95.97
16.30
MTSL-1.0
60.73
6.46
0.0340
0.6105
0.0538
-1.61
-0.71
107.10
31.03
STN
35.36
52.36
0,0566
0.7966
0.1401
107.10
33.99
One-Net
33.99
53.26
0.0634
0.7378
0.1557
-2.80
-7.22
21.70
7.77
MTSL-0.1
34.48
53.29
0.0634
0.7330
0.1859
-2.13
-11.39
21.70
7.77
MTSL-0.2
33.99
52.95
0.0635
0.7328
0.1840
-2.49
-1..30
21.70
7.77
MTSL-0.3
34.69
52.92
0.0638
0.7337
0.1824
-1.48
-10.75
21.70
7.77
NYU
MTSL-0.4
34.07
53.23
0.0634
0.7346
0.1772
-2.65
-0.32
21.70
7.77
MTSL-0.5
34.19
52.94
0.0633
0.7349
0.1861
-2.20
-1.366
21.70
7.77
MTSL-0.6
34.27
53.07
0,0636
0.7360
0.1860
-2.21
-11.43
21.70
7.77
MTSL-0.7
34.53
53.22
0.0638
0.7330
0.1888
-2.00
-11.89
21.70
7.77
MTSL-0.8
34.63
52.81
0.0586
0.7309
0.1649
-1.46
-6.48
22.84
10.11
MTSL-0.9
35.22
52.25
0.0587
0.7149
0.1504
-0.10
-4.30
95.97
17.87
MTSL-1.0
35.71
52.88
0.0565
0.7921
0.1344
0.00
0.74
107.10
33,99
Table 7: Sensitivity of MTSL the number of tasks. (M) denotes the number of parameters in millions.
Network
S t
D +
4
Nt
ASD
MTL 1
AMTL t
# (M)
GMac
STN
59.84
6.43
0.0341
0.6114
85.68
24.80
One-Net
60.38
6.79
0.0415
0.5966
-2.31
-7.18
21.63
6.83
MTSL
60.41
6.43
0.0360
0.6068
0.45
-1.35
77.33
13.75
STN
35.41
53.10
0.0572
0.7881
85.68
27.18
NYU
One-Net
34.37
53.29
0.0623
0.7427
-1.65
-4,49
21.63
7.51
MTSL
35.95
51.69
0.0585
0.7127
2.08
-1.92
77.33
15.09
16

---

Under review as a conference paper at ICLR 2023
Input
N
D
A
8
D
N
A
S
N
A
D
3
LTB
BMTAS
MTSL
(a) Converged architecture on Cityscapes dataset.
Input
S D
3
AW
D S A N
E
S
W A D E
LTB
BMTAS
MTSL
(b) Converged architecture NYUv2 datasct.
Figure 3: Illustration of the converged architecture of LTB, BMTAS, and MTSL. In each figure, each row of circles represents a layer in the encoder. At the last layer of the encoder, each task branch out with their own heads.
able to learn a more efficient architecture, likely because of the use of the CKA regularization term, which forces tasks to learn similar representations, leading to more sharing across tasks.
F CENTERED KERNEL ALIGNMENT (CKA)
Centered Kernel Alignment (CKA) provides the similarity between two layers by computing the similarity between the output representations of the said layers. Let and Y represent the output representations obtained for N images by the two layers be compared. and Y then transformed by taking the mean across the spatial dimension to obtain NxC dimensional representations. We then use the unbiased estimate (Nguyen al. 2021 to obtain the CKA. First, the gram matrices of the two representations G XX" and Gy Y are calculated and centered. CKA is then obtained using Equation
Gx.Gy |xx|| x ||
CKA
In MTSL, to enforce the similarity between tasks, the CKA regularization term is calculated between pairs of task representations by using the current training minibatch. CKA similarities for grouping
17

---

Under review as a conference paper at ICLR 2023
STN
MTSL
STN
MTSL
1.0
1.0
CKA similarity
CKA similarity
0.9
0.9
0.8
0.8
NV
D
.V
(a) In Cityscapes dataset.
(b) In NYUv2 dataset.
Figure 4: Task vs the total similarity of the corresponding task representations with all other task representations the output of the first layer. MTSL leads higher CKA similarities with the aid of CKA regularization, as shown by the taller green bars.
Table 8: Sensitivity of BMTAS to the resource loss weight X. For example, BMTAS-0.1 refers to 0= # (M) denotes the number of parameters in millions.
CS
NYU
Network
A MTL
AMTL f
(M)
GMac
A MTL
A MTL t
(M)
GMac
BMTAS-1.0
-3.07
-1.89
79.69
22.10
-3.84_nat
-5.06
68.15
21.67
B.TTAS-011
-3.45
-2.2++0.0
79.31
24.25
-4.38-0.s
-5.18
79,17
25.20
BMTAS-005
=3.19
-1.95
79.31
24.25
-3.83-1.7
-5.10+06a
79.17
25.20
BMTAS-0005
-3.84
-2.19
68.30
21.03
-4.00
-5,07+0.42
79,17
25.20
BMTAS-00001
-3.55
-2.10.0000
79.17
23.04
-3.58
10.90
78.65
24.02
BMTAS-0.02
-3.48
-2.0610.11
78.64
21.96
-4.40
-5.1610.1
76.69
24.17
decisions during training are calculated using a subset of 800 training images in Cityscapes and all 795 training images in NYUv2. To demonstrate that CKA can be optimized, we look at the sum of CKA values of task layer with all other task layers in STN and MTSL illustrated in Figure 4 The similarities are taken using the output representations the first layer of different single-task networks (STNs). In the case of MTSL, this similarity is taken right before tasks are grouped in the first layer. For example, the gray bar for S is obtained by adding the similarity of the first layer in the single task segmentation network with the first layer of all other task networks. The plot shows that MTSL is able to increase similarity between tasks, as is evident by the taller green bars, by using CKA regularization.
G SENSITIVITY OF BMTAS TO RESOURCE LOSS WEIGHT
We test the sensitivity of BMTAS to the resource loss weight. Both A and AMTT values remain fairly close to each other and remain within standard deviation. Parameter count and GMac change due to the change in the effect of resource loss. We note that in the main paper, use A=0.05 based on values used by the authors.
H ROBUSTNESS NATURAL CORRUPTIONS
We look at the robustness to natural corruptions of the different state-of-the-ar methods and MTSL when using the ResNet encoder and DeepLab head. The results are tabulated in Table g Cross- stitch shows the best results among all methods likely because of the task-specific adapter blocks and task-specific networks. However, discussed in Section 4.1 Cross-stitch has a low inference efficiency contrary to the other methods. When comparing LTE, BMTAS and MTSL, except in noise corruption, MTSL provides the most robustness. The cause of the reduced robustness of MTSL to noise corruption is currently unclear. Task interference could likely play a role, but further investigation can be done in future work.
18

---

Under review as a conference paper at ICLR 2023
Table 9: Robustness to natural corruptions under four categories when using the ResNet encoder and the DeepLab head. MTSL shows better robustness compared to LTB and BMTAS in most cases.
A
SD MTL t
AMTL
Network
Noise
Blur
Weather
Digital
Noise
Blur
Weather
Digital
One-Net
-8.55
-5.48
-9.72
-1.79
-3.26
-3,94
-1.60
-1.38
Cross-stitch
-1.94
-0.17
-3.42
+2.51
+0.14
-1.28
+3.51
+1.04
MTAN
-6.53
-6.03
-6.42
-0.95
-6.23
-7.30
-13.28
-4.03
LTB-R
-2.52
-283
-6.08
-2.23
+1.84
-2.26
-1.24
-1.21
BMTAS-R
-3.48
-2.37
-3.63
-1.63
-1.80
-1.67
+1.01
-0.96
MTSL-R
-5.64
-3.35
-7.13
+0.95
-2887
-2.97
-1.61
+0.03
LTB
-6.50
-3.98
-8.75
-2.35
-0.08
-3,42
-1.68
-1.47
BMTAS
-7.77
-3,49
-11.11
-2.63
-1.60
-2.81
-5.37
-1.55
MTSL
-9.52
-1.68
-4.87
+0.56
-3.21
-2.66
-1.14
+0.16
One-Net
-2.84
2.89
-4.77
-1.25
-1.74
-5.02
-3.00
-4.92
Cross-stitch
+5.82
-1.29
-3.31
+1.23
+5.53
-1,42
+1.12
-1.48
MTAN
+2.42
-4877
-3.78
+0.36
+0.78
-5.57
-4.53
-4.26
NYU
LTB-R
+0.98
-1.74
-1.76
-0.84
+0.19
-0.70
-1.33
-0.92
BMTAS-R
+2.73
-0.20
-0.44
+0.52
-1.16
-0.52
+0.06
-0.14
MTSL-R
-2.13
-3.18
-..4
-0.02
+1.56
-3.11
+0.20
-3.52
LTB BMTAS
+11.21
-2.11
-6.18
-3.76
+6.49
-452
-4.07
-5.47
+11.27
-3.97
-6.82
-3.63
+4.41
-5.18
-6.44
-5.34
MTSL
+1.41
-1.58
-4.11
+1.09
+1.00
-2.67
-0.30
-285
Table 10: Generalization and robustness comparison of MTSI. against a version in which the learned architecture reinitialized and retrained (MTSL-R).
MTL 1 AMTI
SD
AMTL f
Network
Generalization
N
B
W
D
N
B
W
D
One-Net
-3.47
-9.65
-5.78
-1.32
-3.81
-2.15
+3.02
-7.73
-1.36
-12.22
MTSL-R
-1.22
-6.71
-3.49
+0.15
-4.64
+0.51
+4.50
-4.13
+0.23
-8.33
MTSL
-1.35
-7.04
-3.03
+0.21
+0.49
+1.42
+2.92
-3.72
+2.60
-7.07
One-Net
-2.77
-9.82
+10.25
-3.71
-3.27
-2.72
+15.72
-8.51
-11.79
10.35
MTSL-R
-1.34
-6.71
+7.25
-1.84
-2.29
-1.42
+15.92
-7.16
+3.31
-7.72
MTSL
+0.06
-6.21
+1.92
-2.65
-0.84
-1.14
+13.50
-8.03
+3.22
-7.54
I RETRAINING THE CONVERGED ARCHITECTURES
We evaluate the effectiveness of the learned architecture with and without the learned parameters. The learned architecture is reinitialized and retrained under the same training settings as MTSL. The result of this retrained architecture is presented MTSL-R in Table 10 Both MTSL-R and MTSL provide improved results when compared with One-Net, suggesting that both the learned architecture in and of itself and the MTSL training procedure of simultancously learning the architecture and its parameters are beneficial.
Next, we provide results for MTSL-R, where the converged architectures are retrained from scratch. Instead of retraining the architecture converged on dataset to retrain on the same dataset, we use the converged architecture in Cityscapes to train on NYUv2 and vice versa. We refer to this switched evaluation as MTSL-R-Switte The results are provided in Table We observe that switching the converged architectures leads marginal improvements.
19

---

Under review as a conference paper at ICLR 2023
Table 11: Switching of the converged architectures. (M) denotes the number of parameters in millions.
Network
S
D 1.
8
N
A4
ATT f
AMTI. f
# (M)
GMac
STN
61.95
6.38
0.0341
0.6108
0.0535
107.10
31.03
One-Net
60.02
6.71
0.0421
0,5941
0.0620
4.10
-10.06
21.70
7,06
MTSL.-R
60.34
6.46
0.0414
0.6028
0.0567
-1.92
-6.51
22.99
9.53
-TSLt-R-Switch
61,30
6.54
0.0380
0.5987
0.0569
-1.78
-4.67
22.83
9.20
STN
35.36
52.36
0.0566
0.7966
0.1401
107.10
33.99
NYU
One-Net
33.99
53.26
0.0634
0.7378
0.1557
-2.80
-7.22
21.70
7.77
MTSL.-R
34.94
52.89
0.0585
0.7342
0.1524
-1.10
-4.43
22.84
10.11
MTSL-RR-Switch
35.25
52.66
0.0628
0.7308
0.1561
-0.44
-6.30
22.90
10.47
Table 12: MTSL with a different similarity metric called RSA. (M) denotes the number of parameters in millions.
Network
S
8 4
N
A
SSD
AMTL f
(M)
GMac
STN
60.87
6.37
0.03
0.61
0.05
107.10
31.03
MTSI.
60.68
6.52
0.04
0.06
-1.35
-7.04
22.86
8.96
MISL-RSA
60.6liam
6.43
0.04
0.61
0.06
-0.72ioa
-2.36
95.97
16.3
STN
35.6310m
52.70
0.06.ame
0.80.
0.14
107.10
33,99
NYU
MTSL
gm
52.46
0.06
mm73
1.
+0.06-1m
-6.21
22.81
9.89
MTSL-RSA
35.50
52.19
0.06
0.71 2a.m
0.15
+0.31
-4.77
95.97
16.3
J ADDITIONAL SIMILARITY METRIC
Instead of using CKA for alignment and grouping. a variety of other similarity metrics can be used. Here, replace CKA with Representation Similarity Analysis (RSA) and provide the results in Table 12 We observe that RSA is able to improve the results MTSL but at the cost of additional computation.
K EXTENDED NUMBERS FOR COMPARISON WITH SOTA
Table 13 provides extended numbers for all the sota methods against which MTSL is compared.
L ROBUSTNESS TO ADVERSARIAL ATTACK
In addition to the robustness natural corruptions discussed in the main paper, evaluate the robustness to PGD attacks (Madry et al. 2018). We perform the attacks using four epsilon levels (0.25, 0.5, and 2), step size of i and number of iterations determined using min(e 4, [1.25e] (urakin et al.) 2017). For each task S and D, the corresponding task loss is used for attack. MTSL provides improved robustness over One-Net in most cases, as seen in Table 14
M EXTENDED NUMBER FOR GENERALIZATION RESULTS
The generalization performance of the baselines and MTSL have been provided in Table 15 with extended digits after the decimal point where required.
N EXTENDED NUMBERS FOR ABLATION RESULTS
The performance of each task using different components of MTSL is provided in Table 16
20

---

Under review as a conference paper at ICLR 2023
Table 13: Generalization and inference efficiency comparisons between MTSL and state-of-the ofrt---rrt methods, namely cross stitch (Misra et al.) 2016). MTAN (Liu et al. 2019), LTB (Guo et al.] (2020)
and BMTAS (Bruggemann et al. (020). LTB-R and BMTAS-R denote the results obtained by
retraining the converged models of LTB and BMTAS. In MTSL-RSA, the similarity metric CKA is
replaced with RSA. (M) denotes the number of parameters millions.
Network
S
D 14
+
N 1
A 1
STN
50.77 10.18
7.22.0.0:
0.0642
0.5819
0.2246
One-Net
50.9020.13
7.36 oss
0.0650 20.00
0.5783
0.2302
Cross-stitch
52.63+0.34
7.08=0.01
0.0644
0.581 8
0.2250=0.00
MTAN
51.41 10.63
7.3 120000
0.0650 20.00
0.5805 10.00
0.2626 22000
LTB
49.57 4:047
7.47 0.002
0.0649 00000
0.5812
+:0.0 #0000
0.2289 #0000
LTB-R
51.29 #0.28
7.22
0.0643
0.5820
#0.00
0.2253
BMTAS
4
7.50
0.0649=o.oo
0.5809
+00000
0.2290
BMTAS-R
51.1 8 #0.32
7.24
0.0644
0.5820
0.2253
MTSL
50.81 10.50
7.26 20.620003
0.0644 20.00
0.5820
20000
0.2268 20.00
MTSL-RSA
51.10
7.18 10.02
0.0644 10.00
0.5819 19
0.2254
STN
34.56
54.1710.17
0.0764
0.7712 10:00
0.5933
One-Net
33.4910.. 10.38
53.35 1:0.9
0.0776 2.000
0.7041 1.0.00
0.6590 105500.0.00
Cross-stitch
34.25
52.90+a.a.
0.0768 0.000
0.7010 10010am #0.00
0.6096
MTAN
33.7940.5
53.33 20.27
0.0764
0.7070
0.65632z..e
NYU
LTB
31.9210.08
55.2 19.21um 1:0.22
0.0768
0.6964
1696410.00
0.6345
LTB-R
34.04.10447
53.49 $0.40
0.0768 20:00
0.7725 20.00
0.5970
BMTAS
32.09
54.64
0.0768 10.00
0.6978
0.6370
BMTAS-R
34.30+o.26
53.88
0.0764
0.
0.5968=0.0m
MTSL
33.83
53.09
0.0776
0.6993
+0.00
0.6233
MTSL-RSA
33.74 18.7410.s
53.33
0.0768
0.7047 1.000
0.627
10.00
21

---

Under review as a conference paper at ICLR 2023
Table 14: Robustness to PGD attack of the baselines and MTSL.
Network
S
D +
0.25
0.5
1
2
0.25
0.5
STN
45.42
40.76
31.14
24.56
10.80
13.21
19.93
25.86
8
One-Net
45.28
40.97
31.41
24.84
11.63
14.42
22.16
29.04
MTSL.
46.42
41.72
31.94
25.34
11.28
14.02
21.52
28.58
STN
21.58
17.96
11.65
8.24
85.98
103.13
150.55
194.52
NYU
One-Net
20.75
17.31
11.17
7.83
85.40
103.67
149.12
191.13
MTSL
21.88
18.25
11.50
7.90
85.00
102.85
148.80
191.21
Table 15: Generalization comparisons between MTSL and baselines. # (M) denotes the number of parameters in millions.
Network
S
D 4
E 4
N
A &
STN
60.87
6.37
0.034210.00
0.610620.0
0.0535
3
One-Net
60.34_0a7
6.76
0.0423.a.0
0.5943
0.061629.0
One-Net-L.
60.71 Loaa
6.75_occ
0.042 1 10.00
0.5942
0.0615 106615...00
MTSL
60.68=0.10
6.52-0.ca
0.0419 +0.00
0.6029#0.0
0.0583 +
STN
35.63
52.70
0.0570
0.7955
0.1355
NYU
One-Net
34.15 _oas
53.441o.
0.0636
0.7370
1o.00
0.1688
One-Net-L
34.42 .017
53.36
0.06341uum
0.7366....
1o..0
0.1682 L.nux
MTSL
35.50
52.46
0.0595
0.7298
0.1605
Table 16: Ablation results of different tasks.
Align
Avg
ATT
St
D 4
E
N/ t
A 1
60.54
6.382o.22
0.0362
0.6070
0.0558 1158@...o
61.1320.0
6.
0.0359
20.00
0.0557 0.60731mmm
60.95
1.43
0.0358 10.00
0.6075
0.05602o.or
60.2911.a2
Lo60
0.0421 Lo.oz
0.6006
0.05841oa
59.39 11.39
6.67 1o.11
0.0436
Lovoe
0.5974
0.0596 10.00
60.68
6.52+0.05
0.0419
0.6029
0.0583
35.3420.06
51.88
0.0589
0.7145
0.1470
20.54.0.44
51.97
0.0590zo.am
0.7 40..00
0. 0..4669ass 20.00
NYU
35.6420.50
52.40
0.0589
20.90
0.7102
0.1461 1
34.80
52.80
0.0584
0.733 10.000
ND.1634go.og
34. 1 8in.s
53.05
0.0588
0.73441p.0
0.1559
35.50+a.2
52.46+a.25
0.0595=o.cm
0.7298+a.00
0.16055
22


Reviews:
[{"review_id": "fd6LGFrYkL", "reviewer": "Reviewer_mKd4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a structural learning algorithm for concurrently learning the multi-task learning architecture and its parameters. The multitask learning is performed by the creation and removal of neurons based on local similarity. The proposed method is validated on the Cityscapes and NYUv2 datasets for five different dense prediction tasks. Experimental results include SOTA comparison, generalization and inference efficiency, robustness to natural corruptions, and ablation study.", "summary_of_the_review": "I found the idea of multi-task structural learning with task similarity and neuron creation/removal quite interesting. The authors presented results from a number of experiments for validating their proposed method. Although the results are not super convincing, I believe there is the true potential of MTSL in learning similar tasks within a single model.", "strength_and_weaknesses": "Pros:\n+  The paper is well-written and it’s quite well-organized. It was a pleasure reading it.\n+ Task-relatedness is extremely crucial for multi-task learning. Grouping tasks based on local task representations and transferring knowledge to a new group layer is very interesting.\n+  Exposition of the results is good; particularly the ablation study reporting the effect of alignment, average initialization, and attention based knowledge amalgamation.\n\nCons:\n- The generalization experiment reported in 4.2 is not quite clear to me. Did the authors perform cross-dataset evaluation (trained on CS, tested on NYU, and vice-versa) or cross-task evaluation?\n- How would the MTSL work if there are one or multiple tasks requiring varying architectures (e.g., classification from the encoder only and segmentation from the encoder-decoder )?\n- What's the reason behind choosing the particular encoder and decoder networks? Why the decoder is changed from DeepLabv3 to ResNet blocks?   \n-  The authors mentioned that the multitask loss is a weighted sum of all individual task losses. It should be clarified how the task losses are weighted. Which task is to prioritize and when?\n- What value did the authors assign to the balancing factor \\lambda in eq.(1)?\n- Some of the terms are not defined such as, GMac, etc. Table captions could be revised clarifying all the terms. All the performance metrics should be discussed before using them in the tables.\n- it looks like MTSL is consistently poorer than One-Net in the case of noise corruption in Table 3. Why so?\n- Automatic transitioning between the learning phases would be more appropriate as the model is supposed to learn the architecture as well as its parameters automatically.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "bPPwqB3PjY", "reviewer": "Reviewer_wjTd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a network architecture search for a multi-task network, while specifying the search space to tree structures. It jointly trains the model structure as well as the parameters. It starts from the set of single networks and gradually groups layers across tasks if their output features are sufficiently similar. Experiments show that the proposed method outperforms the existing multi-task NAS approaches (BMTAS, LTB) without re-training.", "summary_of_the_review": "In summary, I think this paper proposes a new method for multi-task NAS. However, I could not find an advantage over other existing multi-task NAS approaches in the method and the experiments.", "strength_and_weaknesses": "Strength\n\n- It proposes a novel method for multi-task NAS while constraining the search space to the tree-structured architectures, similarly to BMTAS and LTB. Compared to BMTAS and NAS, it utilized CKA to guide task streams to be similar to each other and compute similarity between layers, which is reasonable.\n- The paper provides an extensive analysis of the sensitivity of several factors, including similarity measures and hyper-parameters. It is also good that the performance under several corruptions is evaluated.\n\nWeakness\n\n- I am not sure about the advantage of this method over other existing methods, BMTAS and LTB.\nGiven the same search space of tree structures, I would like to see the accuracy comparison with BMTAS and LTB after re-initialization and re-training to evaluate the optimality of the found architecture itself. The paper provides accuracy after re-training in Table 8, but a comparison with BMTAS and LTB is unavailable.\n\n- I am not convinced about the value of simultaneous training of the architecture and its parameters, which is claimed as one contribution of this work. Since this method requires training of single-task models for all tasks for initialization, additional one-time re-intialization and re-training of the found architecture will not be a huge burden, roughly spending a similar training cost for one single task network training. For this reason, I also think the comparison with existing methods (BMTAS, LTB) would more reasonable after the re-training.\n\n- The method needs several hyper-parameters, including the task learning epochs, structure learning epochs. How will they affect the performance? Sensitivity analysis seems missing.\n\n- Scalability can be another issue of this method. It needs to initialize the model with N single-task models for N tasks, needs N^2 pairwise similarity, and find optimal groupings during training.\n\n- In Table 1, for fair comparison, BMTAS can be trained with larger lambda (resource regularizer) to match the similar compute cost.\n- The details of the evaluation setting of Table 2 is unclear. Is it a cross-dataset accuracy, e.g., training on NYUv2 and testing on CityScapes?\n- It would be nice to include the comparison with BMTAS and LTB (after re-training) in Table 3, to compare the robustness against the corruptions.\n\n\nRelated work\n\n- [Controllable Dynamic Multi-Task Architectures, CVPR 2022] looks pretty related in the sense that it also uses task similarity to guide branching. It also searches for the tree-structured multi-task architecture but also allows dynamic control of the total compute cost.\n\n\nMinor\n\n- The purpose of explicitly evaluating the multi-task performance for semantic segmentation and depth (SD) is unclear to me.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GP31MinXURz", "reviewer": "Reviewer_hhES", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Multi-Task Structural Learning (MTSL) framework to learn a network architecture and its parameters in a multi-task learning setting. MTSL consists of two phases, a structural learning phase and a fine-tuning phase. The former includes repetition of two alternating steps, network training followed by task grouping. Two datasets in computer vision: Cityscape and NYUv2 were used to evaluate the proposed method. ", "summary_of_the_review": "The method has conceptual merit and is innovative. However, the weaknesses in the implementation and their empirical results dampen my enthusiasm of the work.", "strength_and_weaknesses": "Strength\n+ The development of the method tried to mimic how human brains works. It may be conceptual advantage over existing ones from the artificial intelligence perspective. \n+ Large efforts made to empirical study from multiple aspect, including robustness and contribution of different components of the method.\n\nWeakness\n- The calculation of CKA only directly involves kernel, which is the pairwise similarity between examples based on activation. I do not see this helps align learned filters in two separate networks. \n- The calculation of CKA is quite complicated. It may be challenging to effectively optimize the overall objective in Eq. (1).\n- CKA determines the similarity among tasks. Its calculation should be detailed, for example the (number) examples involved.\n- After the first round of structural learning, not all task nodes are at the same layer of the network. I do not think the way that they described for grouping is appropriate. That way likely leads to a very complicated grouping structure (not like the one that they show in Figure 2, which has nice tree structure) that is difficult to interpret, making little to no sense at all. \n- It would be interesting to report the task grouping/branching learned by other methods and have a comparison with that by the proposed method.\n- The empirical results (Table 1) are weak and not much supportive to the value/advantage of the proposed method.\n- The manuscript is readable on overall, but with large room to improve. There are inappropriate choices of wording in many places.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9jJw1xtuwq", "reviewer": "Reviewer_2dhv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method that \"merges\" N single task networks by grouping similar neurons. The training process\nalternates between \"task learning\" during which the network weights are optimized for task performance, and \"structure\nlearning\", during which neurons are group nodes are optimized with a knowledge distillation-like approach.\nThe method is evaluated on two five-task datasets (cityscapes, NYUv2) and compared with four state-of-the-art methods, with\nfavorable results. Generalization and robustness to corruptions are compared to a baseline single-network.", "summary_of_the_review": "This paper presents a method that performs well, as evidenced by solid experiments, but suffers from an incomplete / unclear description of the methodology. I am unsure regarding the novelty of this work.", "strength_and_weaknesses": "The proposed work describes an effective methodology for combining N single-task networks into a single, branched network\nthat can perform well at multiple tasks. This enables trading-off between task performance and computational efficiency. The\nexperiments appear to be well done, and the authors include an informative ablation study. The results make me confident that\nthe method can perform well and as described.\n\nThe main weakness of the paper is the lack of clarity in parts when describing the method - I give some detailed examples in the\nnext section. \n\nSome minor weaknesses:\nThe paper would be improved if the authors interpret the results for the reader more. For example, the text only very briefly\ndescribes the differences between LTB-R and LTB, which is an important and interesting one. As well - why does cross-stich\nperform so well, and how much has to do with the number of parameters?\n\nI find the analogy with biological neural circuits to be unconvincing, but more concerning was the statement:\n\"The improvement generalization can be attributed to the brain-inspired aspects of the MTSL algorithm.\" I am very\nskeptical of this claim, and couldn't find  evidence in the work about WHY MTSL's performance is good. \nMore care in separating speculation (which is fine when framed as such) from conclusions (and \"can be attributed to\" sounds\nconclusive) would be appropriate.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject