# Adaptive Dynamic Search for Multi-Task Learning
## 3 The Proposed Method
### Backbone: Modularization for Diverse Models  
We constructed a backbone network based on the popular residual network [4; 36]. The basic residual block (ResBlock) in the residual network has two convolution layers of weight tensors, and we split each convolution layer of the ResBlock into \(g\) groups to \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **MTL** & **CAS** & **IwS** & **Search Space** \\ \hline RNP [27] & ✗ & ✓ & ✓ & \((2^{g})^{l}\) \\ RoutingNet [17] & ✓ & ✗ & ✗ & \(g^{r}\) \\ SkipNet [30] & ✗ & ✗ & ✓ & \(2^{r}\) \\ BlockDrop [28] & ✗ & ✗ & ✓ & \(2^{r}\) \\ VirtualNet [25] & ✓ & ✗ & ✗ & \(g\) \\ DEN [31] & ✓ & ✗ & ✓ & \(g^{r}\) \\ MT-SGF [35] & ✓ & ✗ & ✗ & \((2^{g})^{l}\) \\ TRL [34] & ✓ & ✗ & ✗ & \((2^{g})^{l}\) \\ \hline Ours & ✓ & ✓ & ✓ & \((2^{g})^{r}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Algorithmic comparison of the dynamic search methods with respect to different functionalities: multi-task learning (MTL), cost-adaptive search (CAS), and instance-wise search (IwS). Here, \(g\) and \(r\) are the numbers of groups (or channels) and residual blocks, respectively, and \(l\) and \(e\) are the numbers of layers and experts in each layer, respectively. modularize the weight tensors. Specifically, let \(\mathcal{W}_{1}\in\mathbb{R}^{w\times h\times n_{i}\times n_{o}}\) and \(\mathcal{W}_{2}\in\mathbb{R}^{w\times h\times n_{o}\times n_{o}}\) be the weight tensors corresponding to the two layers in the ResBlock, where \(w\), \(h\), \(n_{i}\), and \(n_{o}\) are the width, height, and the number of input and output channels, respectively. Then, the modularized residual block, which we term _ResBlock-M_, contains \(g\) channel groups in each layer with their corresponding weight tensors \(\mathcal{W}_{11}\), \(\mathcal{W}_{12}\),..., \(\mathcal{W}_{1g}\), \(\mathcal{W}_{21}\), \(\mathcal{W}_{22}\),..., \(\mathcal{W}_{2g}\), where  
\[\mathcal{W}_{1i}\in\mathbb{R}^{w\times h\times n_{i}\times\frac{ n_{o}}{g}},\;\;i=1,2,...,g,\] \[\mathcal{W}_{2i}\in\mathbb{R}^{w\times h\times\frac{n_{o}}{g} \times n_{o}},\;\;i=1,2,...,g.\]  
Note that the \(i\)th weights \(\mathcal{W}_{1i}\) and \(\mathcal{W}_{2i}\) are associated with the \(i\)th groups in two layers in ResBlock-M. Rather than utilizing convolutional channels in each layer as a single group, as deep learning methods generally apply [4, 6], we treated them in a finer way for searches. Based on the modularization, we constructed the \(i\)th building block as two consecutive \(i\)th groups in ResBlock-M (see Figure 3). The building blocks can then be chosen by the search network to construct the final prediction model to perform a task. By using the building blocks, we define the operation in the modularized residual block. In the conventional residual block (ResBlock), we have the following operation:  
\[F(x)+x, \tag{1}\]  
where \(x\in\mathbb{R}^{i_{w}\times i_{h}\times n_{i}}\) is an input for the block and \(F(\cdot)\) is a function consisting of two layers with \(\mathcal{W}_{1}\) and \(\mathcal{W}_{2}\). If \(n_{i}\) does not match with \(n_{o}\), then we bridge the gap by applying zero padding or a \(1\times 1\) convolution. In ResBlock-M, we have \(g\) functions according to the number of groups (i.e., \(F_{1}(x),F_{2}(x),\cdots,F_{g}(x)\)), where \(F_{i}(\cdot)\) contains the \(i\)th building block. The operation of the proposed residual block becomes  
\[\alpha_{1}F_{1}(x)+\alpha_{2}F_{2}(x)+\cdots+\alpha_{g}F_{g}(x)+x, \tag{2}\]  
where \(\alpha_{i}\in\{0,1\}\) determines whether to use the function \(F_{i}(\cdot)\). Notice that the operation in Equation (2) is order-invariant for the functions (or their associated groups). This enriches  
Figure 3: (**a**) The conventional residual block (ResBlock) contains a single building block with its function \(F(\cdot)\), and the output is \(F(x)+x\). (**b**) Our modularized residual block (ResBlock-M) contains \(g\) disjointed building blocks (represented by different colors) with their corresponding functions \(F_{1}(\cdot),F_{2}(\cdot),...,F_{g}(\cdot)\).