Title: BRANCHED MULTI-TASK NETWORKS: DECIDING WHAT LAYERS TO SHARE

Abstract: In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand. Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers. Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome. Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive. In this paper, we go beyond these limitations and propose a principled approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities. Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific. Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters.

1. INTRODUCTION:

Deep neural networks are usually trained to tackle different tasks in isolation. Humans, in contrast, are remarkably good at solving a multitude of tasks concurrently. Biological data processing appears to follow a multi-tasking strategy too; instead of separating tasks and solving them in isolation, different processes seem to share the same early processing layers in the brain -see e.g. V1 in macaques (Gur & Snodderly, 2007) . Drawing inspiration from such observations, deep learning researchers began to develop multi-task networks with branched architectures.

As a whole, multi-task networks (Caruana, 1997) seek to improve generalization and processing efficiency through the joint learning of related tasks. Compared to the typical learning of separate deep neural networks for each of the individual tasks, multi-task networks come with several advantages. First, due to their inherent layer sharing (Kokkinos, 2017; Lu et al., 2017; Kendall et al., 2018; Guo et al., 2018; Liu et al., 2019) , the resulting memory footprint is typically substantially lower. Second, as features in the shared layers do not need to be calculated repeatedly for the different tasks, the overall inference speed is often higher (Neven et al., 2017; Lu et al., 2017) . Finally, multi-task networks may outperform their single-task counterparts (Kendall et al., 2018; Xu et al., 2018; Sener & Koltun, 2018; Maninis et al., 2019) . Evidently, there is merit in utilizing multi-task networks.

When it comes to designing them, however, a significant challenge is to decide on the layers that need to be shared among tasks. Assuming a hard parameter sharing setting 1 , the number of possible network configurations grows quickly with the number of tasks. As a result, a trial-and-error procedure to define the optimal architecture becomes unwieldy. Resorting to neural architecture search (Elsken et al., 2019) techniques is not a viable option too, as in this case, the layer sharing has to be jointly optimized with the layers types, their connectivity, etc., rendering the problem considerably expensive. Instead, researchers have recently explored more viable alternatives, like routing (Rosenbaum et al., 2018) , stochastic filter grouping (Bragman et al., 2019) , and feature partitioning (Newell et al., 2019) , which are, however, closer to the soft parameter sharing setting. Previous works on hard parameter sharing opted for the simple strategy of sharing the initial layers in the network, after which all tasks branch out simultaneously. The point at which the branching occurs is usually determined ad hoc (Kendall et al., 2018; Guo et al., 2018; Sener & Koltun, 2018) . This situation hurts performance, as a suboptimal grouping of tasks can lead to the sharing of information between unrelated tasks, known as negative transfer (Zhao et al., 2018) .

In this paper, we go beyond the aforementioned limitations and propose a novel approach to decide on the degree of layer sharing between tasks in order to eliminate the need for manual exploration. To this end, we base the layer sharing on measurable levels of task affinity or task relatedness: two tasks are strongly related, if their single task models rely on a similar set of features. Zamir et al. (2018) quantified this property by measuring the performance when solving a task using a variable sets of layers from a model pretrained on a different task. However, their approach is considerably expensive, as it scales quadratically with the number of tasks. Recently, Dwivedi & Roig (2019) proposed a more efficient alternative that uses representation similarity analysis (RSA) to obtain a measure of task affinity, by computing correlations between models pretrained on different tasks. Given a dataset and a number of tasks, our approach uses RSA to assess the task affinity at arbitrary locations in a neural network. The task affinity scores are then used to construct a branched multitask network in a fully automated manner. In particular, our task clustering algorithm groups similar tasks together in common branches, and separates dissimilar tasks by assigning them to different branches, thereby reducing the negative transfer between tasks. Additionally, our method allows to trade network complexity against task similarity. We provide extensive empirical evaluation of our method, showing its superiority in terms of multi-task performance vs computational resources.

2. RELATED WORK:

Multi-task learning. Multi-task learning (MTL) (Caruana, 1997; Ruder, 2017) is associated with the concept of jointly learning multiple tasks under a single model. This comes with several advantages, as described above. Early work on MTL often relied on sparsity constraints (Yuan & Lin, 2006; Argyriou et al., 2007; Lounici et al., 2009; Jalali et al., 2010; Liu et al., 2017) to select a small subset of features that could be shared among all tasks. However, this can lead to negative transfer when not all tasks are related to each other. A general solution to this problem is to cluster tasks based on prior knowledge about their similarity or relatedness (Evgeniou & Pontil, 2004; Abernethy et al., 2009; Agarwal et al., 2010; Zhou et al., 2011; Kumar & Daume III, 2012) .

In the deep learning era, MTL models can typically be classified as utilizing soft or hard parameter sharing. In soft parameter sharing, each task is assigned its own set of parameters and a feature sharing mechanism handles the cross-task talk. Cross-stitch networks (Misra et al., 2016) softly share their features among tasks, by using a linear combination of the activations found in multiple single task networks. Sluice networks (Ruder et al., 2019) extend cross-stitch networks and allow to learn the selective sharing of layers, subspaces and skip connections. In a different vein, multi-task attention networks (Liu et al., 2019) use an attention mechanism to share a general feature pool amongst task-specific networks. In general, MTL networks using soft parameter sharing are limited in terms of scalability, as the size of the network tends to grow linearly with the number of tasks.

In hard parameter sharing, the parameter set is divided into shared and task-specific parameters. MTL models using hard parameter sharing are often based on a generic framework with a shared off-the-shelf encoder, followed by task-specific decoder networks (Neven et al., 2017; Kendall et al., 2018; Chen et al., 2018; Sener & Koltun, 2018) . Multilinear relationship networks (Long et al., 2017) extend this framework by placing tensor normal priors on the parameter set of the fully connected layers. Guo et al. (2018) proposed the construction of a hierarchical network, which predicts increasingly difficult tasks at deeper layers. A limitation of the aforementioned approaches is that the branching points are determined ad hoc, which can easily lead to negative transfer if the predefined task groupings are suboptimal. In contrast, in our branched multi-task networks, the degree of layer sharing is automatically determined in a principled way, based on task affinities.

Our work bears some similarity to fully-adaptive feature sharing (Lu et al., 2017) , which starts from a thin network where tasks initially share all layers, but the final one, and dynamically grows the model in a greedy layer-by-layer fashion. Task groupings, in this case, are decided on the probability of concurrently simple or difficult examples across tasks. Differently, (1) our method clusters tasks based on feature affinity scores, rather than example difficulty, which is arguably a better criterion;

(2) the tree structure is determined offline using the precalculated affinities for the whole network, and not online in a greedy layer-by-layer fashion, which promotes task groupings that are optimal in a global, rather than local, sense; (3) our approach achieves significantly better results, especially on challenging datasets featuring numerous tasks, like Taskonomy.

Neural architecture search. Neural architecture search (NAS) (Elsken et al., 2019) aims to automate the construction of the network architecture. Different algorithms can be characterized based on their search space, search strategy or performance estimation strategy. Most existing works on NAS, however, are limited to task-specific models (Zoph & Le, 2017; Liu et al., 2018b; Pham et al., 2018; Liu et al., 2018a; Real et al., 2019) . This is to be expected as when using NAS for MTL, layer sharing has to be jointly optimized with the layers types, their connectivity, etc., rendering the problem considerably expensive. To alleviate the heavy computation burden, a recent work (Liang et al., 2018) implemented an evolutionary architecture search for multi-task networks, while other researchers explored more viable alternatives, like routing (Rosenbaum et al., 2018) , stochastic filter grouping (Bragman et al., 2019) , and feature partitioning (Newell et al., 2019) . In contrast to traditional NAS, the proposed methods do not build the architecture from scratch, but rather start from a predefined backbone network for which a layer sharing scheme is automatically determined.

Transfer learning. Transfer learning (Pan et al., 2010) makes use of the knowledge obtained when solving one task, and applies it to a different but related task. Our work is loosely related to transfer learning, as we use it to measure levels of task affinity. Zamir et al. ( 2018) provided a taxonomy for task transfer learning to quantify such relationships. However, their approach scales unfavorably w.r.t. the number of tasks, and we opted for a more efficient alternative proposed by Dwivedi & Roig (2019) . The latter uses RSA to obtain a measure of task affinity, by computing correlations between models pretrained on different tasks. In our method, we use the performance metric from their work to compare the usefulness of different feature sets for solving a particular task.

Loss weighting. One of the known challenges of jointly learning multiple tasks is properly weighting the loss functions associated with the individual tasks. Early work (Kendall et al., 2018) used the homoscedastic uncertainty of each task to weigh the losses. Gradient normalization (Chen et al., 2018) balances the learning of tasks by dynamically adapting the gradient magnitudes in the network. Dynamic task prioritization (Guo et al., 2018) prioritizes the learning of difficult tasks. Zhao et al. (2018) observed that two competing tasks can cause the destructive interference of the gradient, and proposed a modulation module to alleviate this problem. Sener & Koltun (2018) cast multi-task learning as a multi-objective optimization problem, with the overall objective of finding a Pareto optimal solution. Note that, addressing the loss weighting issue in MTL is out of the scope of this work. In fact, all our experiments are based on a simple uniform loss weighing scheme.

3. METHOD:

In this paper, we aim to jointly solve N different tasks T = {t 1 , . . . , t N } given a computational budget C, i.e. number of parameters. Consider a backbone architecture: an encoder, consisting of a sequence of shared layers or blocks f l , followed by a decoder with a few task-specific layers. We assume an appropriate structure for layer sharing to take the shape of a tree. In particular, the first layers are shared by all tasks, while later layers gradually split off as they show more task-specific behavior. The proposed method aims to find an effective task grouping for the sharable layers f l of the encoder, i.e. grouping related tasks together in the same branches of the tree. When two tasks are strongly related, we expect their single-task models to rely on a similar feature set (Zamir et al., 2018) . Based on this viewpoint, the proposed method derives a task affinity score at various locations in the sharable encoder. The resulting task affinity scores are used for the automated construction of a branched multi-task network that fits the computational budget C. Fig. 1 illustrates our pipeline, while Algorithm 1 summarizes the whole procedure. for ti, tj in T and d in locations Task affinity 7: D = 1 -A Task dissimilarity 8: Return: Task-grouping with minimal task dissimilarity that fits within C

EQUATION

3.1. CALCULATE TASK AFFINITY SCORES:

As mentioned, we rely on RSA to measure task affinity scores. This technique has been widely adopted in the field of neuroscience to draw comparisons between behavioral models and brain activity. Inspired by how Dwivedi & Roig (2019) applied RSA to select tasks for transfer learning, we use the technique to assess the task affinity at predefined locations in the sharable encoder. Consequently, using the measured levels of task affinity, tasks are assigned in the same or different branches of a branched multi-task network, subject to the computational budget C.

The procedure to calculate the task affinity scores is the following. As a first step, we train a singletask model for each task t i ∈ T . The single-task models use an identical encoder E -made of all sharable layers f l -followed by a task-specific decoder D ti . The decoder contains only taskspecific operations and is assumed to be significantly smaller in size compared to the encoder. As an example, consider jointly solving a classification and a dense prediction task. Some fully connected layers followed by a softmax operation are typically needed for the classification task, while an additional decoding step with some upscaling operations is required for the dense prediction task. Of course, the appropriate loss functions are applied in each case. Such operations are part of the task-specific decoder D ti . The different single-task networks are trained under the same conditions.

At the second step, we choose D locations in the sharable encoder E where we calculate a twodimensional task affinity matrix of size N × N . When concatenated, this results in a threedimensional tensor A of size D × N × N that holds the task affinities at the selected locations. To calculate these task affinities, we have to compare the representation dissimilarity matrices (RDM) of the single-task networks -trained in the previous step -at the specified D locations. To do this, a held-out subset of K images is required. The latter images serve to compare the dissimilarity of their feature representations in the single-task networks for every pair of images. Specifically, for every task t i , we characterize these learned feature representations at the selected locations by filling a tensor of size D × K × K. This tensor contains the dissimilarity scores 1 -ρ between feature representations, with ρ the Pearson correlation coefficient. Specifically, RDM d,i,j is found by calculating the dissimilarity score between the features at location d for image i and j.

For a specific location d in the network, the computed RDMs are symmetrical, with a diagonal of zeros. For every such location, we measure the similarity between the upper or lower triangular part of the RDMs belonging to the different single-task networks. We use the Spearman's correlation coefficient r s to measure similarity. When repeated for every pair of tasks, at a specific location d, the result is a symmetrical matrix of size N × N , with a diagonal of ones. Concatenating over the D locations in the sharable encoder, we end up with the desired task affinity tensor of size D × N × N . Note that, in contrast to prior work (Lu et al., 2017) , the described method focuses on the features used to solve the single tasks, rather than the examples and how easy or hard they are across tasks, which is arguably a better measure of task affinity.

3.2. CONSTRUCT A BRANCHED MULTI-TASK NETWORK:

Given a computational budget C, we need to derive how the layers (or blocks) f l in the sharable encoder E should be shared among the tasks in T . Each layer f l ∈ E is represented as a node in the tree, i.e. the root node contains the first layer f 0 , and nodes at depth l contain layer(s) f l . The granularity of the layers f l corresponds to the intervals at which we measure the task affinity in the sharable part of the model, i.e. the D locations. When the encoder is split into b l branches at depth l, this is equivalent to a node at depth l having b l children. The task-specific decoders D t can be found in the leaves of the tree. Fig. 1b shows an example of such a tree using the aforementioned notation. Each node is responsible for solving a unique subset of tasks.

The branched multi-task network is built with the intention to separate dissimilar tasks by assigning them to separate branches. To this end, we define the dissimilarity score between two tasks t i and t j at location d as 1 -A d,i,j , with A the task affinity tensor2 . The branched multi-task network is found by minimizing the sum of the task dissimilarity scores at every location in the sharable encoder. In contrast to prior work (Lu et al., 2017) , the task affinity (and dissimilarity) scores are calculated a priori. This allows us to determine the task clustering offline. Since the number of tasks is finite, we can enumerate all possible trees that fall within the given computational budget C. Finally, we select the tree that minimizes the task dissimilarity score. The task dissimilarity score of a tree is defined as C cluster = l C l cluster , where C l cluster is found by averaging the maximum distance between the dissimilarity scores of the elements in every cluster. The use of the maximum distance encourages the separation of dissimilar tasks. By taking into account the clustering cost at all depths, the procedure can find a task grouping that is considered optimal in a global sense. This is in contrast to the greedy approach in (Lu et al., 2017) , which only minimizes the task dissimilarity locally, i.e. at isolated locations in the network.

An exhaustive search becomes intractable when the number of tasks is extremely large. For such cases, we propose to derive the tree in a top-down manner, starting at the most outer layer. At every step l, we can perform spectral clustering for each possible number of groups m where 1 ≤ m ≤ b l+1 , with b l+1 the number of branches at layer l + 1. Before proceeding to the next step, we select the top-n task groupings with minimal cost. This constrains the number of possible groupings at the next layer. When we proceed to cluster the tasks at the next layer, we select the top-n groupings from the ones that are still eligible to be constructed. This beam search is used in CelebA experiments.

4. EXPERIMENTS:

In this section, we quantitatively and qualitatively evaluate the proposed method on a number of diverse multi-tasking datasets, that range from real to semi-real data, from few to many tasks, from dense prediction to classification tasks, and so on. 

4.1. CITYSCAPES:

Dataset. The Cityscapes dataset (Cordts et al., 2016) considers the challenging scenario of urban scene understanding. The train, validation and test set contain respectively 2975, 500 and 1525 real images, taken by driving a car in Central European cities. It considers a few dense prediction tasks: semantic segmentation (S), instance segmentation (I) and monocular depth estimation (D). As in prior works (Kendall et al., 2018; Sener & Koltun, 2018) , we use a ResNet-50 encoder with dilated convolutions, followed by a Pyramid Spatial Pooling (PSP) (He et al., 2015) decoder. Every input image is rescaled to 512 x 256 pixels. We reuse the approach from Kendall et al. (2018) for the instance segmentation task, i.e. we consider the proxy task of regressing each pixel to the center of the instance it belongs to. We obtained all results after a grid search on the hyperparameter space, to ensure a fair comparison across the compared approaches. For more details please visit Appendix A.

Results. We measure the task affinity after every block (1 to 4) in the ResNet-50 model (see Fig. 2a ).

The task affinity decreases in the deeper layers of the model, due to the features becoming more taskspecific. We compare the performance of the task groupings generated by our method with those by other approaches. As in (Maninis et al., 2019) , the performance of a multi-task model m is defined as the average per-task performance drop/increase w.r.t. a single-task baseline b.

We trained all possible task groupings that can be derived from branching the model in the last three ResNet blocks. Fig. 2b visualizes performance vs number of parameters for the trained architectures. Depending on the available computational budget C, our method generates a specific task grouping. We visualize these generated groupings as a path in Fig. 2b , when gradually increasing the computational budget C. Similarly, we consider the task groupings when branching the model based on the task affinity measure proposed by Lu et al. (2017) . We find that, in comparison, the task groupings devised by our method achieve higher performance within a given computational budget C. Furthermore, in the majority of cases, for a fixed budget C the proposed method is capable of selecting the best performing task grouping w.r.t. performance vs parameters metric. We also compare our branched multi-task networks with cross-stitch networks (Misra et al., 2016) and NDDR-CNNs (Gao et al., 2019) in Table 1 3 . While the latter give higher multi-task performance, attributed to their computationally expensive soft parameter sharing setting, our branched multi-task networks can strike a better trade-off between the performance and number of parameters. In particular, Fig. 2b shows that we can effectively sample architectures which lie between the extremes of a baseline multi-task model and a cross-stitch or NDDR-CNN architecture. It is worth noting that soft parameter sharing does not scale when the number of tasks increases greatly.

4.2. TASKONOMY:

Dataset. The Taskonomy dataset (Zamir et al., 2018) contains semi-real images of indoor scenes, annotated for 26 (dense preciction, classification, etc.) tasks. Out of the available tasks, we select scene categorization (C), semantic segmentation (S), edge detection (E), monocular depth estimation (D) and keypoint detection (K). The task dictionary was selected to be as diverse as possible, while still keeping the total number of tasks reasonable for all computations. We use the tiny split of the dataset, containing 275k train, 52k validation and 54k test images. We reuse the architecture and training setup from Zamir et al. ( 2018): the encoder is based on ResNet-50; a 15-layer fullyconvolutional decoder is used for the pixel-to-pixel prediction tasks. Appendix B contains a more detailed description on the training setup of the Taskonomy experiments.

Results. The task affinity is again measured after every ResNet block. Since the number of tasks increased to five, it is very expensive to train all task groupings exhaustively, as done above. Instead, we limit ourselves to three architectures that are generated when gradually increasing the parameter budget. As before, we compare our task groupings against the method from Lu et al. ( 2017). The numerical results can be found in Table 2 . The task groupings themselves are shown in Appendix B.

The effect of the employed task grouping technique can be seen from comparing the performance of our models against the corresponding FA models, generated by (Lu et al., 2017) . The latter are consistently outperformed by our models. Compared to the results on Cityscapes (Fig. 2b ), we find that the multi-task performance is much more susceptible to the employed task groupings, possibly due to negative transfer. Furthermore, we observe that cross-stitch networks and NDDR-CNNS can not handle the larger, more diverse task dictionary: the performance decreases when using these models, while the number of parameters increases. This is in contrast to our branched multi-task networks, which seem to handle the diverse set of tasks rather positively. As opposed to (Zamir et al., 2018) , but in accordance with (Maninis et al., 2019) , we show that it is possible to solve many heterogeneous tasks simultaneously when the negative transfer is limited, by separating dissimilar tasks from each other in our case. In fact, our approach is the first to show such consistent performance across different multi-tasking scenarios and datasets. Existing approaches seem to be tailored for particular cases, e.g. few/correlated tasks, synthetic-like data, binary classification only tasks, etc., whereas we show stable performance across the board of different experimental setups.

Table 3 : Quantitative analysis on the CelebA test set. (bold) The Ours-Thin-32 architecture is found by optimizing the task clustering for the parameter budget that is used in the Branch-32-2.0 model.

(italic) The Ours-Thin-64 architecture is found by optimizing the task clustering for the parameter budget that is used in the GNAS-Shallow-Wide model.

Method Accuracy (%) Parameters (Millions) LNet+ANet (Wang et al., 2016) 87 -Walk and Learn (Wang et al., 2016) 88 -MOON (Rudd et al., 2016) 90.94 119.73 Independent Group (Hand & Chellappa, 2017) 91.06 -MCNN (Hand & Chellappa, 2017) 91.26 -MCNN-AUX (Hand & Chellappa, 2017) 91.29 -VGG-16 Baseline (Lu et al., 2017) 91.44 134.41 Branch-32-2.0 (Lu et al., 2017) 90.79 2.09 GNAS-Shallow-Thin (Hand & Chellappa, 2017) 91 images respectively. We treat the prediction of each facial attribute as a single binary classification task, as in (Lu et al., 2017; Sener & Koltun, 2018; Huang et al., 2018) . To ensure a fair comparison: we reuse the thin-ω model from Lu et al. (2017) in our experiments on CelebA; the parameter budget C is set for the model to have the same amount of parameters as prior work. As mentioned in Sec. 3.2, we use the beam search adaptation of our optimization procedure due to the very large number of tasks. We set n = 10, with the top-n being the number of groupings to keep at every layer during the optimization. Note that, the final result remains unchanged when applying small changes to n. Appendix C provides more details on the training setup.

Results. Table 3 shows the results on the CelebA test set. The task groupings themselves are visualized in Appendix C. Our branched multi-task networks outperform earlier works (Lu et al., 2017; Huang et al., 2018) when using a similar amount of parameters. Since our Thin-32 model only differs from the model in (Lu et al., 2017) on the employed task grouping technique, we can conclude that the proposed method devises more effective task groupings for the attribute classification tasks on CelebA. Furthermore, our Thin-32 model performs on par with the VGG-16 baseline, while using 64 times less parameters. We also compare our results with the ResNet-18 model from Sener & Koltun (2018) . Our Thin-64 models performs 1.35% better than the ResNet-18 model when trained with a uniform loss weighing scheme. More noticeably, our Thin-64 model performs on par with the state-of-the-art ResNet-18 model that was trained with the loss weighing scheme from Sener & Koltun (2018), while at the same time using 31% less parameters (11.2 vs 7.7 M).

5. CONCLUSION:

In this paper, we introduced a principled approach to automatically construct branched multi-task networks for a given computational budget. To this end, we leverage the employed tasks' affinities as a quantifiable measure for layer sharing. The proposed approach can be seen as an abstraction of NAS for MTL, where only layer sharing is optimized, without having to jointly optimize the layers types, their connectivity, etc., as done in traditional NAS, which would render the problem considerably expensive. Extensive experimental analysis shows that our method outperforms existing ones w.r.t. the important metric of multi-tasking performance vs number of parameters, while at the same time showing consistent results across a diverse set of multi-tasking scenarios and datasets.

MTAN We tried re-implementing the MTAN model (Liu et al., 2019) using a ResNet-50 backbone. The architecture was based on the Wide-ResNet architecture that is used in the original paper.

After extensive hyperparameter tuning, we were unable to get a meaningful result on the Cityscapes dataset when trying to solve all three tasks jointly. Note that, the authors have only shown results in their paper when training semantic segmentation and monocular depth estimation.

B TASKONOMY:

We reuse the setup from Zamir et al. ( 2018). All input images were rescaled to 256 x 256 pixels. We use a ResNet-50 encoder and replace the last stride 2 convolution by a stride 1 convolution. A 15-layer fully-convolutional decoder is used for the pixel-to-pixel prediction tasks. The decoder is composed of five convolutional layers followed by alternating convolutional and transposed convolutional layers. We use ReLU as non-linearity. Batch normalization is included in every layer except for the output layer. We use Kaiming He's initialization for both encoder and decoder. We use an L1 loss for the depth (D), edge detection (E) and keypoint detection (K) tasks. The scene categorization task is learned with a KL-divergence loss. We report performance on the scene categorization task by measuring the overlap in top-5 classes between the predictions and ground truth.

The multi-task models were optimized with task weights w s = 1, w d = 1, w k = 10, w e = 10 and w c = 1. Notice that the heatmaps were linearly rescaled to lie between 0 and 1. During training we normalize the depth map by the standard deviation.

Single-task models We use an Adam optimizer with initial learning rate 1e-4. The learning rate is decayed by a factor of10 after 80000 iterations. We train the model for 120000 iterations. The batch size is set to 32. No additional data augmentation is applied. The weight decay term is set to 1e-4.

Baseline multi-task model We use the same optimization procedure as for the single-task models.

The multi-task performance is calculated using Eq. 1.

Branched multi-task models We use the same optimization procedure as for the single-task models. The architectures that were generated by our method are shown in Fig. 4 . Fig. 5 shows the architectures that are found when using the task grouping method from Lu et al. (2017) . We show some of the predictions made by our third branched multi-task network in figure 6 for the purpose of qualitative evaluation. Figure 4 : Task groupings generated by our method. The numerical results can be found in Table 2 . 

C CELEBA:

We reuse the thin-ω model from Lu et al. (2017) . The CNN architecture is based on the VGG-16 model (Simonyan & Zisserman, 2015) . The number of convolutional features is set to the minimum between ω and the width of the corresponding layer in the VGG-16 model. The fully connected layers contain 2 • ω features. We train the branched multi-task network using stochastic gradient descent with momentum 0.9 and initial learning rate 0.05. We use batches of size 32 and weight

:

In this setting, the input is first encoded through a stack of shared layers, after which tasks branch out into their own sequence of task-specific layers(Kokkinos,

2017;Lu et al., 2017;Kendall et al., 2018;Guo et al., 2018;Sener & Koltun, 2018). Alternatively, a set of task-specific networks can be used in conjunction with a feature sharing mechanism(Misra et al., 2016;Liu et al., 2019;Ruder et al., 2019). The latter approach is termed soft parameter sharing in the literature.

This is not to be confused with the dissimilarity score used to calculate the RDM elements RDM d,i,j .

We included MTAN(Liu et al., 2019) in our experiments too, but we were unable to solve all three tasks on Cityscapes (S, I, D) simultaneously when using their proposed architecture.

Figures and Tables:

Pipeline overview. (left) We train a single-task model for every task t in T . (middle) We use RSA to measure the task affinity at D predefined locations in the sharable encoder. In particular, we calculate the representation dissimilarity matrices (RDM) for the features at D locations using K images, which gives a D × K × K tensor per task. (right) The affinity tensor A is found by calculating the correlation between the RDM matrices, which results in a three-dimensional tensor of size D ×N ×N , with N the number of tasks.

b) Our pipeline's output is a branched multi-task network, similar to how NAS techniques output sample architectures. An example branched multi-task network is visualized here.

Figure 1: The proposed method: (a) calculate task affinities at various locations in the sharable encoder; (b) construct a branched multi-task network based on the computed affinities.

Task affinity after each ResNet-50 block.

of parameters versus multi-task performance on the Cityscapes validation set.

Figure 2: Qualitative analysis on the Cityscapes dataset.

Figure 5: Task groupings generated using the method from Lu et al. (2017). The numerical results can be found in Table2.

Quantitative analysis on the Cityscapes validation set.

Table content: <table><tr><td>Method</td><td colspan="5">S (mIoU) I (px) D (px) Parameters (M) MTL performance (%)</td></tr><tr><td>Single task</td><td>65.17</td><td>11.70</td><td>2.57</td><td>138</td><td>0.00</td></tr><tr><td>MTL baseline</td><td>61.51</td><td>11.80</td><td>2.66</td><td>92</td><td>-3.28</td></tr><tr><td>Cross-stitch</td><td>65.14</td><td>11.63</td><td>2.55</td><td>140</td><td>0.42</td></tr><tr><td>NDDR-CNN</td><td>65.64</td><td>11.61</td><td>2.54</td><td>190</td><td>0.89</td></tr><tr><td>Ours -1</td><td>62.14</td><td>11.74</td><td>2.66</td><td>107</td><td>-2.81</td></tr><tr><td>Ours -2</td><td>62.67</td><td>11.67</td><td>2.62</td><td>114</td><td>-1.90</td></tr><tr><td>Ours -3</td><td>64.11</td><td>11.62</td><td>2.62</td><td>116</td><td>-1.00</td></tr></table>

Quantitative analysis on the tiny Taskonomy test set. The results for edge (E) and keypoints (K) detection were multiplied by a factor of 100 for better readability. The FA models refer to generating the task groupings with the task affinity technique proposed byLu et al. (2017).

Table content: <table><tr><td>Method</td><td colspan="7">D(L1) S(mIoU) C(top-5) E(L1) K(L1) Params(M) MTL Performance (%)</td></tr><tr><td>Single task</td><td>0.60</td><td>43.45</td><td>66.02</td><td>0.99</td><td>0.23</td><td>224</td><td>0.0</td></tr><tr><td>MTL baseline</td><td>0.75</td><td>47.82</td><td>55.99</td><td>1.37</td><td>0.34</td><td>130</td><td>-22.50</td></tr><tr><td>Cross-stitch</td><td>0.61</td><td>43.99</td><td>58.24</td><td>1.35</td><td>0.50</td><td>224</td><td>-32.29</td></tr><tr><td>NDDR-CNN</td><td>0.66</td><td>45.94</td><td>64.48</td><td>1.05</td><td>0.45</td><td>258</td><td>-21.02</td></tr><tr><td>FA-1</td><td>0.74</td><td>46.14</td><td>62.73</td><td>1.30</td><td>0.39</td><td>174</td><td>-24.5</td></tr><tr><td>FA-2</td><td>0.80</td><td>39.93</td><td>62.36</td><td>1.68</td><td>0.52</td><td>188</td><td>-48.32</td></tr><tr><td>FA-3</td><td>0.74</td><td>46.09</td><td>64.93</td><td>1.05</td><td>0.27</td><td>196</td><td>-8.48</td></tr><tr><td>Ours-1</td><td>0.76</td><td>47.62</td><td>63.28</td><td>1.12</td><td>0.29</td><td>174</td><td>-11.88</td></tr><tr><td>Ours-2</td><td>0.74</td><td>47.96</td><td>63.61</td><td>0.96</td><td>0.35</td><td>188</td><td>-12.66</td></tr><tr><td>Ours-3</td><td>0.74</td><td>47.88</td><td>64.54</td><td>0.94</td><td>0.26</td><td>196</td><td>-4.93</td></tr></table>

The CelebA dataset(Liu et al., 2015) contains over 200k real images of celebrities, labeled with 40 facial attribute categories. The training, validation and test set contain 160k, 20k and 20k

Table content: <table><tr><td/><td>.30</td><td>1.57</td></tr><tr><td>GNAS-Shallow-Wide (Hand &amp; Chellappa, 2017)</td><td>91.63</td><td>7.73</td></tr><tr><td>GNAS-Deep-Thin (Hand &amp; Chellappa, 2017)</td><td>90.90</td><td>1.47</td></tr><tr><td>GNAS-Deep-Wide (Hand &amp; Chellappa, 2017)</td><td>91.36</td><td>6.41</td></tr><tr><td>ResNet-18 (Uniform weighing) (Sener &amp; Koltun, 2018)</td><td>90.38</td><td>11.2</td></tr><tr><td>ResNet-18 (MGDA-UB) (Sener &amp; Koltun, 2018)</td><td>91.75</td><td>11.2</td></tr><tr><td>Ours-Thin-32</td><td>91.46</td><td>2.20</td></tr><tr><td>Ours-Thin-64</td><td>91.73</td><td>7.73</td></tr><tr><td>4.3 CELEBA</td><td/><td/></tr><tr><td>Dataset.</td><td/><td/></tr></table>

Peer review (iclr2020):
This paper presents a method to infer multi-task networks architecture, more specifically to determine which part of the network should be shared among different tasks. The main idea is to first train a specific, standalone encoder/decoder network for each task. Subsequently, a task affinity factor is computed by looking at the similarity (or, more likely the dissimilarity) of an holdout set of images feature representations. Knowing these dissimilarities (based on RDM), one can cluster the tasks and make similar tasks share more of their layers. Computational budget can also be taken into consideration in the form of the number of parameters. Results on Cityscapes, Taskonomy, and CelebA datasets show, to some extent, improvements against the state of the art. The paper is well written and addresses a common problem in multi-task learning. The experiments provided are extensive and cover most of the current multi-task learning methods and interesting problems. I especially like the idea of formalizing the dissimilarity between tasks using RDM. There are, however, a few key points that would need improvement. First, except for CelebA, the experiments provided use ResNet50 with only 4 different *anchor point* in the network. In other words, the task at hand is limited to selecting the best sharing point between 4 choices. This is not wrong per se, but in my opinion, it does not tackle the main problem: what to do when brute force / exhaustive exploration cannot be fulfilled? CelebA provides a more complex case, but it also requires to change the method from an exhaustive search to a beam search (end of Sec. 3.2). Doing so get us back to a kind of greedy approach, precisely what was advocated against the paragraph before (in the discussion about Lu et al. 2017). Second, the fact that task affinities are computed a priori leads to the following conclusion: *this allows us to determine the task clustering offline*. While I agree that this could be useful, one has to keep in mind that compared to other methods, this one has to first train a network for _each_ task independently, which can take a long time. Out of curiosity, did you consider other correlation coefficients than Spearman? Why use a rank-based method? Overall, this is a nice paper, with adequate methodology. It is not groundbreaking, and most of the good results arise when we consider both performance and number of parameters, but it is interesting for the community nonetheless. I am not sure the impact would be very high, since it can probably be replaced by an architecture exhaustive search if the number of tasks and branching points in the network are low, but the formalism of the approach is welcomed.