# Adaptive Dynamic Search for Multi-Task Learning
## 1 Introduction  
Undoubtedly, deep learning has been a popular choice of learning strategies in many fields of study [1] and has achieved remarkable success in solving challenging problems in computer vision and machine learning [2; 3; 4; 5; 6]. To that end, it became a general trend for practitioners to develop a neural network model specifically tailored for a given task to improve the performance [4; 6; 7]. With high memory requirements for an increasing number of tasks as well as manual efforts in designing the architectures, however, this approach (i.e., one network for a single task) is often found to be non-scalable and presents challenges to be employed for multifunctional devices with limited resources [8]. To address this issue, multiple tasks are learned simultaneously via a shared network architecture in a multi-task learning regime (MTL) [9; 10]. MTL is a learning strategy for solving multiple tasks at the same time while exploiting commonalities and differences across tasks. This learning strategy turned out to be effective for diverse applications [3; 11; 12; 13; 14; 15; 16]. Nonetheless, there remain several challenges that need addressing. First, the performance can degrade when the number of tasks to be learned is high or the tasks are not closely related [17; 18]. Second, manually designing novel architectures for MTL is challenging due to the difficulty of curating optimal prediction models. Last but not least, a naive MTL approach entails highly demanding efforts when cost-adaptivity is required (i.e., performing tasks adaptively for different computational and memory costs). This also requires searching for the corresponding network models manually. Despite the practical benefit of tackling it, this combinatory problem has not been addressed rigorously as of yet. In this work, we propose a novel MTL approach that is cost-adaptive and enables dynamic model search. Given a computation budget, our approach produces a network suited for each task automatically (without the need to design the architecture) on the fly and instance-wise (i.e., a network for an instance). The proposed approach provides awide range of solutions with respect to different computational budgets for each task, as shown in Figure 1. This is achieved by exploring multiple network models automatically (without the need to manually design the architecture) on the fly and instance-wise in a single learning procedure. To this end, we develop a unified framework that consists of two major components, namely backbone and search networks. The backbone network provides a pool of candidate models of different sizes and configurations, from which the search network selects a set of models of different sizes that is most suitable for a given instance with different computational requirements. To offer diverse choices of candidate models, we finely modularize the backbone architecture that constitutes the basic building blocks of disjointed collections of parameters. Then, we construct a new modularized residual block containing the basic building blocks to enrich the search space for competitive dynamic search. The proposed framework is trained for the task performance as well as the computational cost using stochastic gradient descent in an alternating manner. The overall framework of the proposed approach is illustrated in Figure 2. Once the training is complete, the search network can discover different models of choice trading off between the performance and the computational cost for each instance of a task. This can save a significant amount of time and effort in training models of different costs individually. Figure 1: A graphical illustration for a cost-adaptive search of the proposed approach on three tasks (classification datasets).