AdapMTL: Adaptive Pruning Framework for Multitask Learning Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:40:58

Abstract: In the domain of multimedia and multimodal processing, the efficient handling of diverse data streams—such as images, video, and sensor data—is paramount. Model compression and multitask learning (MTL) are crucial in this field, offering the potential to address the resource-intensive demands of processing and interpreting multiple forms of media simultaneously. However, effectively compressing a multitask model presents significant challenges due to the complexities of balancing sparsity allocation and accuracy performance across multiple tasks. To tackle the challenges, we propose AdapMTL, an adaptive pruning framework for MTL models. AdapMTL leverages multiple learnable soft thresholds independently assigned to the shared backbone and the task-specific heads to capture the nuances in different components' sensitivity to pruning. During training, it co-optimizes the soft thresholds and MTL model weights to automatically determine the suitable sparsity level at each component in order to achieve both high task accuracy and high overall sparsity. It further incorporates an adaptive weighting mechanism that dynamically adjusts the importance of task-specific losses based on each task's robustness to pruning. We demonstrate the effectiveness of AdapMTL through comprehensive experiments on popular multitask datasets, namely NYU-v2 and Tiny-Taskonomy, with different architectures, showcasing superior performance compared to state-of-the-art pruning methods.

Main paper: 
 related work: 
 Multitask Learning. Multitask learning (MTL) [1, 4, 12, 64] aims to learn a single model to solve multiple tasks simultaneously by sharing information and computation among them, which is essential for practical deployment. Over the years, various MTL approaches have been proposed, including hard parameter sharing [3] , soft parameter sharing [58] , and task clustering [22] . In hard parameter sharing, a set of parameters in the backbone model are shared among tasks while in soft parameter sharing, each task has its own set of parameters, but the difference between the parameters of different tasks is regularized to encourage them to be similar. MTL has been successfully applied to a wide range of applications, such as natural language processing [8, 18, 29] , computer vision [17, 30, 44, 57] , and reinforcement learning [42, 53] . Subsequently, the integration of neural architecture search (NAS) with MTL has emerged as a promising direction. NAS for MTL, exemplified by works like MTL-NAS [15] , Learning Sparse Sharing Architectures for Multiple Tasks [50] , and Controllable Dynamic Multi-Task Architectures [43] , focuses on discovering optimal architectures that can efficiently learn shared and task-specific features. These approaches, including Adashare [52] and AutoMTL [63] , demonstrate the potential of dynamically adjusting architectures to the requirements of multiple tasks, optimizing both performance and computational efficiency.Pruning. Pruning techniques have been widely studied to reduce the computational complexity of deep neural networks while maintaining their performance. Early works on pruning focused on unstructured weight pruning [20, 25] , where unimportant weights were removed based on a given criterion, and the remaining weights were fine-tuned. There are different kinds of criterion metrics, such as magnitude-based [20, 27] , gradient-based [36, 37] , Hessianbased [21] , connection sensitivity-based [26, 33, 48] , and so on.Other works explored structured pruning [56, 66] , which removes entire filters or channels, leading to more efficient implementations on hardware platforms. Recently, the lottery ticket hypothesis [13] has attracted considerable attention, suggesting that dense, randomly-initialized neural networks contain subnetworks (winning tickets) that can be trained to achieve comparable accuracy with fewer parameters. This has led to follow-up works [13, 32, 38] that provide a better understanding of the properties and initialization of winning tickets. Single-Shot Network Pruning (SNIP) [26] is a data-driven method for pruning neural networks in a one-shot manner. By identifying an initial mask to guide parameter selection, it maintains a static network architecture during training. Some other work, like the layer-wise pruning method [23] , inspiringly attempts to learn a layer-wise sparsity for individual layers rather than considering the network as a whole. This approach allows for fine-grained sparsity allocation across layers. To reduce the total time involved in pruning and training, pruning during training techniques [11, 35, 39] have been proposed to directly learn sparse networks without the need for an iterative pruning and finetuning process. These methods involve training networks with sparse connectivity from scratch, updating both the weights and the sparsity structure during the training process.Pruning for Multitask Learning. Recently, attention has shifted to the intersection of MTL and pruning techniques. A compact multitask model has the potential to deliver high performance across various tasks while minimizing resource requirements, making it well-suited for deployment on resource-constrained devices or in real-time scenarios. For example, MTP [6] focuses on efficient semantic segmentation networks, demonstrating the potential of multitask pruning to enhance performance in specialized domains. Similarly, the work by Cheng et al. [7] introduces a novel approach to multi-task pruning through filter index sharing, optimizing model efficiency through a many-objective optimization framework. Additionally, Ye et al. [59] propose a global channel pruning method tailored for multitask CNNs, highlighting the importance of performance-aware approaches in maintaining accuracy while reducing model size. Disparse [51] proposes joint learning and pruning methods to achieve efficient multitask models. However, these methods often neglect the importance of the shared backbone, leading to equal treatment of each component during pruning, rather than accounting for their varying importance. Our work aims to address this limitation by adaptively allocating sparsity across the shared backbone and task-specific heads based on their importance and sensitivity. 
methodology: 
 In this section, we first present the notations and the definition of the multitask model pruning We then introduce the proposed adaptive multitask model pruning framework in Section 3.2 and describe the adaptive weighting mechanism in Section 3.3.We formulate multitask model pruning as an optimization problem. Given a dataset D = {(𝑥 𝑖 ; 𝑦 1 𝑖 , 𝑦 2 𝑖 , ..., 𝑦 𝑇 𝑖 ), 𝑖 ∈ [1, 𝑁 ]}, a set of T tasks T = {𝑡 1 𝑡 2 ..., 𝑡 𝑇 }, and a desired sparsity level 𝑠 (i.e. the percentage of zero weights), the multitask model pruning aims to find a sparse weight 𝑊 that minimizes the sum of task-specific losses. Mathematically, it is formulated as:EQUATIONwhere the L (•) is the total loss function, L 𝑡 (•) is the task-specific loss for each individual task 𝑡, 𝑊 are the parameters of neural network to be learned, 𝑃 is the total number of parameters and ∥ • ∥ 0 denotes the ℓ 0 -norm, i.e. the number of non-zero weights.The key challenge here is how to enforce sparsity on weight 𝑊 while minimizing the loss. This involves finding an optimal balance between maintaining the performance of each task and pruning the model to achieve the desired sparsity level. We next describe our proposed adaptive pruning algorithm that can effectively handle the unique characteristics of multitask models and efficiently allocate sparsity across different components to preserve the overall model performance.Multitask models typically have a backbone that is shared across tasks and task-specific heads. We observe that these different model components have different sensitivities to pruning and thus should be treated differently. The challenge lies in how to automatically capture the sensitivity of each model component to pruning and leverage the signal to automatically allocate sparsity across components. To address the challenge, we propose a component-wise pruning framework that assigns different learnable soft thresholds to each component to capture its sensitivity to pruning. The framework then co-optimizes the thresholds with model weights to automatically determine the suitable sparsity level for each component during training. Specifically, we introduce a set of learnable soft thresholds 𝛼 = {𝛼 𝐵 , 𝛼 1 , 𝛼 2 , ..., 𝛼 𝑇 } for each component, where 𝛼 𝐵 represents the threshold for the shared backbone and 𝛼 𝑡 represents the threshold for the 𝑡-th task-specific head. The thresholds 𝛼 are determined based on the significance and sensitivity of the respective components and are adaptively updated using gradient descent during the backpropagation process. The soft threshold 𝛼 𝑡 and sparse weight 𝑊 𝑡 for each component can be computed as follows: where 𝜃 init is a learnable parameter that controls the initial pruning threshold 𝛼 𝑡 . We will discuss the choice of 𝜃 init in the supplementary material. The 𝑅𝑒𝐿𝑈 (•) function here is used to set zero weights.EQUATIONIn other words, if some weights |𝑊 𝑡 | are less than the threshold 𝛼 𝑡 , then the sparse version of this weight 𝑆 (𝑤 𝑡 , 𝛼 𝑡 ) is set to 0. Otherwise, we obtain the soft-thresholding version of this weight.The reason why we choose soft thresholding [54] rather than hard thresholding is illustrated in Figure 2 . Soft parameter sharing is the best fit for our approach as it allows us to calculate the gradient and perform the backpropagation process more effectively.AdapMTL reformulates the pruning problem in Equation 1 to find a set of optimal thresholds 𝛼 = {𝛼 𝐵 , 𝛼 1 , 𝛼 2 , ..., 𝛼 𝑇 } across different components as follows:EQUATIONwhere the 𝛽 𝑡 represents the adaptive weighting factor for 𝑡-th task, which will be elaborated in Section 3.3.We next describe how AdapMTL optimizes the problem in Equantion 3. Considering a multitask model with T tasks, we divide the weight parameters into 𝑊 = {𝑊 𝐵 ,𝑊 1 ,𝑊 2 , ...,𝑊 𝑇 }, where 𝑊 𝐵 represents the weight parameters for the shared backbone and 𝑊 𝑡 represents the weight parameters for the 𝑡-th task-specific head. We derive the gradient descent update equation at the 𝑛-th epoch for 𝑊 𝑡 as follows:EQUATIONwhere 𝜂 𝑛 is the learning rate at the 𝑛-th epoch. We use the partial derivative to calculate the gradients. As mentioned earlier, different task heads may have varying sensitivities to pruning and, consequently, may require different levels of sparsity to achieve the best accuracy. By setting a set of learnable parameters for each component and treating them separately during the backpropagation process, our component-wise pruning framework can effectively account for these differences in sensitivity and adaptively adjust the sparsity allocation for each component. Although𝜕𝑆 (𝑊 𝑛 𝑡 ,𝛼 𝑛 𝑡 ) 𝜕𝑊 𝑛 𝑡is non-differentiable, we can approximate the gradients using the sub-gradient method. In this case, we introduce B𝑡 𝑛 , an indicator function that acts like a binary mask. The value of B𝑡 𝑛 should be 0 if the sparse version of the weight 𝑆 (𝑊 𝑛 𝑡 , 𝛼 𝑛 𝑡 ) is equal to 0. This indicator function facilitates the approximation of gradients and the update of the sparse weights and soft thresholds during the backpropagation process. Mathematically, the indicator function is: By updating the sparse weights 𝑊 𝑡 , and similarly the soft thresholds 𝛼 𝑡 , for each component in this manner (the derivation process is provided in the supplementary material), the framework can effectively and discriminatively allocate sparsity across the multitask model. By taking into account the significance and sensitivity of each component, this approach ultimately leads to more efficient and accurate multitask learning.EQUATIONThis subsection introduces the adaptive weighting mechanism that dynamically adjusts the weight of each task loss based on each task's robustness to pruning. The adaptive weighting mechanism determines the 𝛽 𝑡 for the 𝑡-th task in Equation 3 during training.The rationales behind the proposed adaptive weighting mechanism are two folds. First, if the training loss of a specific task 𝑡 tends to be stable, then we can assign a higher weighting factor 𝛽 𝑡 and subsequently prune more aggressively on that component, as the task head is robust to pruning. On the contrary, if the loss of a specific task fluctuates significantly, we should consider pruning less on that component by lowering the weighting factor since the training is less likely to converge at higher sparsity levels. The weighting factor is learned in an adaptive way, eliminating the need for manual effort to elaborately fine-tune the hyper-parameters.Second, the adaptive weighting mechanism should automatically consider different multitask model architectures as well. The ratio of backbone to task head weights, 𝑊 𝑏𝑎𝑐𝑘𝑏𝑜𝑛𝑒 𝑊 ℎ𝑒𝑎𝑑 , matters because it may be beneficial to focus more on pruning the task heads instead if the backbone is already highly compact. For example, in MobileNet-V2, the backbone has only 2.2M parameters, which is 25 times fewer than the task head.We define a set of adaptive weights 𝛽 = {𝛽 𝐵 , 𝛽 1 , 𝛽 2 , ..., 𝛽 𝑇 }, where 𝛽 𝐵 represents the weighting factor for the shared backbone, 𝛽 𝑡 represents the weighting factor for the 𝑡-th task-specific head. The weighting factor can be formulated as follows:EQUATIONHere, ΔL window 𝑡 is the average deviation of the loss within the sliding window for the 𝑡-th task, which is divided by L 𝑡 to normalize the scale. We then divide it by the sum of all tasks to normalize between different tasks. The (•) -1 is a multiplicative inverse. 𝜆 is a scaling factor, and we will discuss the choice of 𝜆 for different architectures in the supplementary material. |𝑊 𝐵 | 𝑏𝑎𝑐𝑘𝑏𝑜𝑛𝑒 , |𝑊 𝑡 | ℎ𝑒𝑎𝑑 represent the weight parameters of shared backbone and 𝑡-th taskspecific head, separately. The right ratio in the equation reveals the importance of each component by considering their relative parameterizing contributions to the overall model structure. The weighting factor 𝛽 𝑡 is used to guide the pruning for the task-specific head, depending on the stability of its loss and its contribution to the model.To make the multitask pruning more robust, we incorporate a sliding window mechanism that tracks the past loss values to calculate the average ΔL window in Equation 6 instead of relying solely on the variance between two adjacent epochs. This approach provides a more stable and reliable estimation of the fluctuations in the task losses, as it accounts for a larger number of samples and reduces the impact of potential outliers or short-term variations. 
experiments: 
 In this section, we first present an overview of the experiment settings, including datasets, tasks, evaluation metrics, loss functions, baselines for comparison, and training details in Section 4.1. Subsequently, we provide comprehensive quantitative experimental results in Section 4.2, comparing our approach with other state-ofthe-art methods to demonstrate the superiority of our proposed method. We also analyze the sensitivity of different components and computational cost in Section 4.3. Finally, we add ablation studies to verify the effectiveness of the proposed method in Section 4.4. Functions. We adopt a range of evaluation metrics for different tasks, evaluating the model performance at different sparsity levels to provide a comprehensive view of the model's effectiveness and robustness across tasks. On the NYUv2 dataset, there are totally three tasks. For Semantic Segmentation, we employ the mean Intersection over Union (mIoU) and Pixel Accuracy (Pixel Acc) as our primary evaluation metrics and use cross-entropy to calculate the loss. Surface normal prediction uses the inverse of cosine similarity between the normalized prediction and ground truth, and is performed using mean and median angle distances between the prediction and the ground truth. We also report the percentage of pixels whose prediction is within the angles of 11.25°, 22.5°, and 30°to the ground truth. Depth estimation utilizes the L1 loss, with the absolute and relative errors between the prediction and ground truth being calculated. Again, We also present the relative difference between the prediction and ground truth by calculating the percentage of 𝛿 = 𝑚𝑎𝑥 ( 𝑦 𝑝𝑟𝑒𝑑 𝑦 𝑔𝑡 , 𝑦 𝑔𝑡 𝑦 𝑝𝑟𝑒𝑑 ) within the thresholds of 1.25, 1.25 2 , and 1.25 3 . On the Taskonomy dataset, there are two more tasks. In the context of both Keypoint and Edge Detection tasks, the mean absolute error compared to the provided ground-truth map serves as the main evaluation metric.In multitask learning scenarios, tasks involve multiple evaluation metrics with values potentially at different scales. To address this, we compute a single relative performance metric following the common practice [34] [49] .△ 𝑇 𝑖 = 1 |𝑀 | |𝑀 | ∑︁ 𝑗=1 (-1) 𝑙 𝑗 • (𝑀 𝑇 𝑖 ,𝑗 -𝑀 𝐷𝑀,𝑗 )/𝑀 𝐷𝑀,𝑗 * 100% (7)where 𝑙 𝑗 = 1 if a lower value shows better performance for the metric 𝑀 𝑗 and 0 otherwise. 𝑀 𝑇 𝑖 ,𝑗 , 𝑀 𝐷𝑀,𝑗 are the sparse and dense model value of metric 𝑗, respectively. The △ 𝑇 𝑖 is defined to compare results with their equivalent dense task values and the overall performance is obtained by averaging the relative performance across all tasks, denoted as △ 𝑇 = 1 𝑇 𝑇 𝑖=1 △ 𝑇 𝑖 , This metric provides a unified measure of relative performance across tasks. Eventually, by employing these diverse evaluation metrics, we can effectively assess the performance of our method as well as the counterparts across various tasks and datasets.We compare our work with LTH [13] , IMP [19] , SNIP [26] , and DiSparse [51] . For LTH, we first train a dense model and subsequently prune it until the desired sparsity level is reached, yielding the winning tickets (sparse network structure). We then reset the model to its initial weights to start the sparse training process. For IMP, we remove the least important weights, determined by their magnitudes, iteratively. For SNIP and IMP, we directly use the official implementation provided by the authors from GitHub. For DiSparse, the latest multitask pruning work and first-of-its-kind, we utilize the official PyTorch implementation and configure the method to use the DiSparse dynamic mechanism, which is claimed as the best-performing approach in the paper. We also train a fully dense multitask model as our baseline, which will be used to calculate a single relative performance metric Norm. Score.We use the same backbone model at the same sparsity level across all methods for a fair comparison. In our work, we define overall sparsity as the percentage of weights pruned from the entire MTL model, which includes both the shared backbone and task-specific heads. We utilize Deeplab-ResNet34 [5] and MobileNetV2 [45] as the backbone models, and the Atrous Spatial Pyramid Pooling (ASPP) architecture [5] as the task-specific head. Both of them are popular architectures for pixel-wise prediction tasks. We share a common backbone for all tasks while each task has an independent taskspecific head branching out from the final layer of the backbone, which is widely used in multitasking scenarios.Table 1 : Comparison with state-of-the-art pruning methods on the NYU-V2 dataset using the Deeplab-ResNet34 backbone. Each pruning method enforces a consistent overall sparsity of 90%, with the △ 𝑇 indicating the normalized performance of all three tasks to the baseline dense model's performance. We also report the evaluation metrics for each task and the sparsity allocation for each component. , LTH [13] , SNIP [26] , and IMP [19] , on the NYUv2 dataset, evaluated with different MTL backbones and under various sparsity settings.We first present the comparison results with state-of-the-art methods on the NYU-V2 dataset in table 1 .Overall, AdapMTL outperforms all other methods by a significant margin across most metrics and achieves the highest △ 𝑇 . Recall that the major difference between our method and the baselines lies in our ability to adaptively learn the sparsity allocation across the components adaptively, maintaining a dense shared backbone (71.74%) while keeping the task-specific heads relatively sparse. Within the scope of our research, we characterize overall sparsity as the percentage of weights pruned from the entire MTL model, which includes both the shared backbone and task-specific heads. SNIP [26] exhibits the lowest performance in the multi-task scenario because its pruning mask is determined from a single batch of data's gradient, which treats all components, including the shared backbone, equally. Since all input information passes through the shared backbone, accuracy loss in the shallow layers is inevitable, regardless of how well the task heads perform with relatively high density. LTH's [13] winning tickets do not sufficiently focus on the backbone, as they intentionally create a dense surface normal prediction task head. Although this approach performs well on this specific task, the bias still causes an imbalance in the metrics across all tasks, resulting in a lower △ 𝑇 score. IMP [19] achieves a good normalized score across all tasks. However, this method is trained in an iterative manner and prunes the model step-by-step, resulting in a significantly longer training time. DiSparse [51] learns an effective dense backbone by adopting a unanimous decision across all tasks. However, it falls short of differentiating the relative sensitivities between specific task heads, leading to an imbalanced normalized score among all tasks. Here, we add an additional row, AdapMTL without adaptive thresholds, to demonstrate the effectiveness of our approach. Rather than using multiple adaptive thresholds, this version utilizes a single shared threshold for all components. As expected, performance significantly deteriorates because a uniform threshold makes it hard to capture the nuances in different components' sensitivity.Table 2 : Comparison with state-of-the-art pruning methods on the NYU-V2 dataset using the MobileNetV2 backbone. Each pruning method enforces a consistent overall sparsity of 90%, with the △ 𝑇 indicating the normalized performance of all three tasks to the baseline dense model's performance. We also report the evaluation metrics for each task and the sparsity allocation for each component. Moreover, we extended our experiments to different model architectures to assess the model-agnostic nature of our method, using MobileNetV2 as an alternative architecture. The results, detailed in Table 2 , show how AdapMTL adeptly manages the dense representation of MobileNetV2's compact backbone, ensuring it remains sufficiently dense (52.74% ) while enforcing higher sparsity in the task-specific heads. This is very important, especially with such backbone compact architectures where over-pruning the backbone can easily lead to significant degradation in accuracy. Our approach ensures that the backbone remains dense enough, thereby preserving overall performance.under various sparsity settings. We show a comparison of results under different sparsity settings using different backbones, namely ResNet34 and MobileNetV2, as illustrated in Figure 4 , where AdapMTL consistently demonstrates superiority over other methods. The normalized test score, following the common practice [34] [49] , is obtained by averaging the relative performance across all tasks with respect to the dense model. We observe a slightly better performance for medium sparsity levels(from 50% to 80% ), which even surpasses dedicated dense multitask learning approaches despite the high sparsity enforced. This observation aligns with our assumptions and motivates the research community to further explore and develop sparse models. The score of SNIP drops significantly as higher sparsity levels (>90%) are enforced. This is because it fails to maintain the density of the shared backbone effectively.Results on Tiny-Taskonomy. On the Tiny-Taskonomy dataset, which encompasses five distinct tasks, AdapMTL exhibits a more consistent performance across all tasks, as detailed in Table 3 . Our method consistently achieved the highest scores in each task, unlike other methods which exhibited noticeable biases. The DiSparse method struggles to achieve unanimous decisions, particularly as the number of tasks increases, highlighting a key limitation in its approach.The consistent superiority of AdapMTL across both NYUv2 and Tiny-Taskonomy datasets, and with different backbone architectures, highlights the effectiveness of our approach in achieving high sparsity with minimal performance degradation for multitask models. More results on the other datasets, using the different architectures, can be found in the supplementary material. 4.3.1 Pruning sensitivity. AdapMTL results in different sparsity for backbone parameters and task-specific parameters, indicating that it captures their different sensitivity to pruning. To compare the sensitivity to pruning between the shared backbone and task heads, we create a 3D plot, as shown in Figure 5 . The x-axis represents the shared backbone sparsity from 50% to 99%, while the y-axis represents the total head sparsity for all three tasks from 90% to 99%. The z-axis represents the normalized score. From the xz-plane, we can observe that the normalized score drops significantly when we prune the backbone at sparsity levels of 90% and higher. In contrast, from the yz-plane, we can see that the task heads are highly robust to pruning, as they maintain a good normalized score even when extreme sparsity levels are reached. This observation highlights the importance of preserving the shared backbone's density and suggests that pruning strategies should prioritize maintaining the backbone's performance while aggressively pruning the task-specific heads to achieve overall model sparsity.4.3.2 Computational cost. The computational cost of the AdapMTL under varying sparsity levels is detailed in Table 4 , which illustrates a significant reduction in both parameters and FLOPs as sparsity increases. These reductions highlight not only the adaptability of AdapMTL across different architectures but also its capability to maintain a balance between performance, measured by △ 𝑇 , and efficiency, evidenced by the substantial decrease in FLOPs. This balance is crucial for deploying high-performance models in resourceconstrained environments. By leveraging specialized hardware and software solutions that can efficiently handle sparse matrix operations, such as sparse matrix-vector multiplication (SpMV), these models can achieve faster inference times [14, 39, 55] .We conducted ablation studies to validate the effectiveness of the proposed adaptive multitask model pruning (Section 3.2), and the adaptive weighting mechanism (Equation 6). We tested variations including models without adaptive thresholds, where all components share a single threshold, and models with only two adaptive thresholds, where the backbone has a unique threshold while other task heads share another. The results, presented in Table 5 , highlight the critical role of adaptive thresholding. Models without adaptive thresholds showed significantly poorer performance, with a drastic decrease in △ 𝑇 , especially affecting tasks with higher sensitivity to pruning, such as Depth Prediction. Conversely, the full AdapMTL configuration, employing independent thresholds for each component, achieved the best △ 𝑇 score. These variations help illustrate the impact and necessity of differentiated thresholding in multitask environments. The results confirm that our full AdapMTL setup, with all components active, performs superiorly across different settings, underscoring the indispensable nature of each proposed component. We have implemented a sliding window mechanism to enhance the robustness and accuracy of our pruning strategy. This mechanism is pivotal in tracking the loss values over a sequence of epochs to compute the average change in loss, ΔL window , as formalized in Equation 6. By integrating this approach, we significantly mitigate the influence of abrupt variations and potential outliers that may occur in task-specific loss calculations. The sliding window, set at a size of 400 as demonstrated in Figure 6 , represents an optimal balance between computational memory demands and the need for a comprehensive data scope. This size ensures that the model captures sufficient temporal loss information without excessive memory consumption, thereby maintaining efficiency. 
conclusion: 
 In this paper, we propose a novel adaptive pruning method designed specifically for multitask learning (MTL) scenarios. Our approach effectively addresses the challenges of balancing overall sparsity and accuracy for all tasks in multitask models. AdapMTL introduces multiple learnable soft thresholds, each independently assigned to the shared backbone and task-specific heads to capture the nuances in different components' sensitivity to pruning. Our method cooptimizes the soft thresholds and model weights during training, enabling automatic determination of the ideal sparsity level for each component to achieve high task accuracy and overall sparsity. Furthermore, AdapMTL incorporates an adaptive weighting mechanism that dynamically adjusts the importance of task-specific losses based on each task's robustness to pruning. The effectiveness of AdapMTL has been extensively validated through comprehensive experiments on the NYU-v2 and Tiny-Taskonomy datasets with different architectures. The results demonstrate that our method outperforms state-of-the-art pruning methods, thereby establishing its suitability for efficient and effective multitask learning.  
: 
 of tasks grows, so does the demand for larger and more powerful models, which in turn require substantial computational resources, memory usage, and longer training times. This trade-off between performance and model complexity has led to a continuous pursuit of more efficient and compact CNN [24] architectures, as well as innovations in pruning techniques that can maintain high performance without compromising the benefits of the model's scale.Pruning techniques [13, 19, 23, 25-27, 36, 48] have emerged as a promising approach to compress large models without significant loss of performance. These techniques aim to reduce the size of a model by eliminating redundant or less important parameters, such as neurons, connections, or even entire layers, depending on the method employed [9, 28, 61] . Parameter-efficient pruned models can provide significant inference time speedups by exploiting the sparsity pattern [14, 31, 56, 60] . These models are designed to have fewer parameters, which translates into reduced memory footprint and lower computational complexity (FLOPs) [31] . By leveraging specialized hardware and software solutions that can efficiently handle sparse matrix operations, such as sparse matrixvector multiplication (SpMV), these models can achieve faster inference times [14, 39, 55] . Additionally, sparse models can benefit from better cache utilization, as they require less memory bandwidth, thereby reducing the overall latency of the computation [41, 60] .Although many techniques have been proposed in the past for pruning a single-task model, there are much fewer works in pruning a multitask model. Multitask models, which are designed to simultaneously handle multiple tasks, have become increasingly popular due to their ability to share representations and learn more effectively from diverse data sources [16, 64, 67] . These models have found wide-ranging applications where tasks are often related and can benefit from shared knowledge [65] . A compact multitask model, which is shown in Figure 1 , has the potential to deliver high performance across various tasks while minimizing resource requirements, making it well-suited for deployment on resourceconstrained devices or in real-time scenarios.Traditional pruning techniques, which are primarily focused on single-task models, may not be directly applicable or sufficient for multitask settings. Recent work has started to explore the intersection of multitask learning and pruning. Disparse [51] considered each task independently by disentangling the importance measurement and taking the unanimous decisions among all tasks when performing parameter pruning and selection. A parameter is removed if and only if it's shown to be not critical for any task. However, as the number of tasks increases, it becomes challenging to achieve unanimous selection agreement among all tasks, which could negatively affect the average performance across tasks. Thus, there is a need for novel compression approaches that are catered to the complexities of multitask models, taking into account the inter-dependencies between tasks, the sharing of representations, and the different sensitivity of task heads. Addressing these challenges is essential for advancing the development and deployment of efficient, compact multitask models.To tackle the challenges, we conduct extensive experiments that reveal two valuable insights on designing an effective multitask model pruning strategy. First, the shared backbone and the taskspecific heads have different sensitivity to pruning and thus should be treated differently. However, current state-of-the-art approaches do not adequately recognize this aspect, leading to equal treatment of each component during pruning, rather than accounting for their varying sensitivities. Second, the change in training loss could serve as a useful guide for allocating sparsity among different components. If the training loss of a specific task tends to be stable, we can prune more aggressively on that component, as the task head is robust to pruning. On the contrary, if the loss of a specific task fluctuates significantly, we should consider pruning less on that component since the training is less likely to converge at higher sparsity levels.Motivated by these observations, we propose AdapMTL, an adaptive pruning framework for MTL models. AdapMTL dynamically adjusts sparsity across different components, such as the shared backbone and task-specific heads based on their sensitivity to pruning, while preserving accuracy for each task. This is achieved through a set of learnable soft thresholds [10, 23] that are independently assigned to different components and co-optimized with model weights to automatically determine the suitable sparsity level for each component during training. Specifically, we maintain a set of soft thresholds 𝛼 = {𝛼 𝐵 , 𝛼 1 , 𝛼 2 , ..., 𝛼 𝑇 } in each component, where 𝛼 𝐵 represents the threshold for the shared backbone and 𝛼 𝑡 represents the threshold for the 𝑡-th task-specific head. In the forward pass, only the weights larger than the threshold 𝛼 will be counted in the model, while others are set to zero. In the backward pass, we automatically update all the component-wise thresholds 𝛼, which will smoothly introduce sparsity. Additionally, AdapMTL employs an adaptive weighting mechanism that dynamically adjusts the importance of task-specific losses based on each task's robustness to pruning. AdapMTL does not require any pre-training or pre-pruned models and can be trained from scratch.We conduct extensive experiments on two popular multitask datasets: NYU-v2 [47] and Tiny-Taskonomy [62  
 
 Appendix:  


Decision: This paper initially received mixed ratings, with critiques primarily focused on its simple experiments limited to unimodal computer vision. However, during the rebuttal process, the authors included new experiments involving Large Language Models (LLMs). These additional experiments successfully addressed the reviewers' concerns, resulting in unanimously positive reviews. Therefore, the Area Chair recommends acceptance.

Reviews: review: paper_summary: paper_summary: The author proposes AdapMTL, an adaptive pruning framework for MTL models that dynamically adjusts sparsity levels across different components to achieve high sparsity and task accuracy. Extensive experiments on popular multitask datasets with varied architectures demonstrate the effectiveness of AdapMTL.
strength_weakness: strength_weakness: (1) The paper is well-presented and easy to follow, featuring delicate images and well-formulated tables. The experiments demonstrate significant effectiveness with notable margins.

(2) The proposed methods make sense to me. The component-wise pruning framework, which assigns different learnable soft thresholds to each component to capture its sensitivity to pruning, is intriguing.
limitations: limitations: (1) The paper only focuses on unimodal computer vision multi-task scenarios, which are somehow not in line with the topic of ACM MM.

(2) The authors apply the proposed methods to relatively small models, including ResNet34 and MobileNet. I am curious whether these methods can scale to larger models, including LLMs.

(3) The paper lacks a deeper analysis of the motivations behind the proposed methods and their operational efficacy. I would appreciate further empirical or mathematical justification during the rebuttal.
paper_summary: paper_summary: The paper introduces a method  AdapMTL to efficiently manage and compress multitask learning (MTL) models through adaptive pruning. The framework utilizes learnable soft thresholds which are independently set for the shared backbone and the task-specific heads of a multitask model. This allows for dynamic adjustment of sparsity levels during training. The method is validated with experiments on two datasets, demonstrating improved performance compared to existing SOTA pruning methods.
strength_weakness: strength_weakness: 1. The approach is novel, including the adaptive weighting and the component-specific soft thresholds. AdapMTL addresses the unique challenges of pruning and keep balance in multitask scenarios

2. The framework is extensively tested on two datasets and two architectures with multiple ablation studies, providing a robust validation of its effectiveness.
limitations: limitations: 1. The paper focuses solely on image-based tasks based on CNNs and does not explore or incorporate other modalities with different models. This unimodal approach may limit the generalizability and applicability of the proposed adaptive pruning framework to true multimedia or multimodal contexts.

2. The experimental results shows that even with the pruning, this method achieves better performance than the original dense models on almost all metrics and tasks. This is a little bit contrary to common sense and it’s not discussed in the paper.
paper_summary: paper_summary: A pruning method designed for multi-task learning scenarios by introducing multiple learnable soft thresholds assigned to a shared backbone and specific task heads, by capturing the sensitivity of different components to pruning.
strength_weakness: strength_weakness: 1.The introduction of learnable soft thresholds for both shared and task-specific components is innovative, enabling the model to adaptively prune each component based on its sensitivity. This allows for a more balanced and efficient pruning process.
limitations: limitations: 1.Comparative Baseline Limitations:
The paper compares AdapMTL with several state-of-the-art methods but does not include comparisons with some newer pruning techniques or other advanced MTL optimization methods.Such as DynaShare: Task and Instance Conditioned Parameter Sharing for Multi-Task Learning.

2.There is very little in the article that deals with multimedia and multimodality.

3.In terms of article writing, does the article have a framework diagram that visually describes the entire algorithm.

4.In Fig. 2, the plot of soft thresholds is continuous, but with folds, which cannot be said to be smooth. Also is the meaning of the vertical coordinate here the number of weight parameters or what?