Task-Interaction-Free Multi-Task Learning with Efficient Hierarchical Feature Representation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 05:42:33

Abstract: Traditional multi-task learning often relies on explicit  task interaction mechanisms to enhance multi-task performance. However, these approaches encounter challenges such as negative transfer when jointly learning multiple weakly correlated tasks. Additionally, these methods  handle encoded features at a large scale, which escalates computational complexity to ensure dense prediction task performance. In this study, we introduce a Task-Interaction-Free Network (TIF) for multi-task learning, which diverges from explicitly designed task interaction mechanisms. Firstly, we present a Scale Attentive-Feature Fusion Module (SAFF) to enhance each scale in the shared encoder to have rich task-agnostic encoded features. Subsequently, our proposed task and scale-specific decoders efficiently decode the enhanced features shared across tasks without necessitating task-interaction modules. Concretely, we utilize a Self-Feature Distillation Module (SFD) to explore task-specific features at lower scales and the Low-To-High Scale Feature Diffusion Module (LTHD) to diffuse global pixel relationships from low-level to high-level scales. Experiments on publicly available multi-task learning datasets validate that our TIF attains state-of-the-art performance.

Main paper: 
 related work: 
 Multi-task learning for dense prediction. Multi-task learning (MTL) architectures are primarily categorized into two paradigms: encoder-focused MTL approaches and decoder-focused MTL approaches [1, 32, 35, 38, 39, 43, 45] . Encoder-focused MTL methods have demonstrated efficacy through the deployment of sophisticated encoder-decoder networks [8, 16, 17, 24] . these methods often struggle to improve performance without incorporating taskspecific feature interactions at higher feature levels. In contrast, decoder-focused MTL methods have distinguished themselves in multi-task dense prediction scenarios by emphasizing task interaction modules, seeking to excel with a shared encoder complemented by task-specific decoders [25] . PAD [32] , a pioneering method in this domain, leverages interactions between auxiliary and target tasks to achieve superior performance on the latter in a localized manner. MTI-Net [26] introduces a novel multi-scale task-specific feature interaction mechanism using a multi-modal distillation module [32] at each scale. ATRC [1] has recently set a benchmark in efficient decoder-focused MTL, harnessing the power of Neural Architecture Search [31] to identify optimal contextual interactions between tasks. InvPT [38] and TaskPrompter [40] achieves remarkable performance by integrating multi-scale techniques [26] with the visual transformer (ViT) architecture [4, 12, 23] , showcasing the synergistic potential of these complementary methodologies. However, the task interaction manner of these methods encounters negative transfer problem. Therefore, we proposed TIF to enhance the task-agnostic features from the encoder and hierarchical task-specific features from decoders, without explicit task interaction.Vision Transformer. The attention mechanism has become increasingly prevalent in computer vision tasks [7] , especially in the wake of the transformative impact of Transformers in natural language processing (NLP). This mechanism [13] has outperformed conventional convolutional neural network (CNN)-based methods in numerous dense prediction tasks by leveraging its strength in capturing pixel-level relational information. For dense prediction tasks, a multitude of Vision Transformer (ViT)-based approaches have been developed, each introducing unique self-attention strategies to encapsulate aggregated features across various scales [34] , global spatial relationships [30] , channel-wise dependencies [6] , local contextual information [12] , learnable attention weights [41] , and focal attention mechanisms [37] . These self-attention methods are primarily designed for individual vision tasks and have not been universally adapted for multi-task learning scenarios. Multi-task . TIF consists of a task-sharing encoder and 𝑁 task-specific decoders for all tasks. In the encoder, Scale Attentive-Feature Fusion Modules (SAFF) generate rich multi-scale task-agnostic features X 0 , X 1 , X 2 , X 3 . In each decoder, Self-Feature Diffusion modules (SFD) model global relationships at the low scales, while Low-To-High Scale Feature Diffusion modules (LTHD) diffuse the global relationships to the high scales. Finally, all low and high scales are fused to generate the final prediction.learning methods [36, 38] also utilize the cross-task attention to model global relationships between tasks. 
overview: 
 The proposed framework is composed of a task-sharing encoder in conjunction with 𝑁 task-specific decoders, as shown in Figure 2 . The task-sharing encoder incorporates two principal modules: a feature extractor and the Scale-Attentive Feature Fusion Module (SAFF). The SAFF is further subdivided into two sub-modules: Scale Alignment (SA) and Scale Enhanced Feature Fusion (SEFF). The feature extractor can be any pre-trained vision model [4, 12, 29] with four blocks. Given an image 𝐼 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 , the feature extractor generates four scale features:EQUATIONwhere 𝑋 𝑖 represents 𝑖th scale level task sharing feature. 𝐹 𝐸𝑋 denotes the feature extractor function. To refine each scale level 𝑋 𝑖 , we process it through the SAFF modules in conjunction with all four scale features. Initially, the SA modules are employed to align all scale features to the target scale, which is equivalent to the scale of 𝑋 𝑖 . Subsequently, SEFF modules are utilized to augment the target scale feature with the aligned multi-scale features, thereby producing a scale-enhanced feature, denoted as X𝑖 . As a result, the encoder component generates a rich set of multi-scale task-sharing features, encompassing X 0 , X 1 , X 2 , X 3 .In contrast to existing decoder-focused MTL methods, our approach is characterized by the design of efficient and effective taskspecific decoders for the target tasks. The decoder component of our framework comprises two innovative modules: the Self-Feature Distillation Module (SFD) and the Low-To High Scale Feature Diffusion Module (LTHD). Within our framework, we assign X 2 and X 3 as the low-scale features and X 1 and X 0 as the high-scale features.The SFD modules are introduced to exploit long-range global relationships among these features in a novel and efficient manner. The SFD modules yield two principal outputs: task-specific low-scale features and intra-task affinities. For the high-scale features, the LTHD module is introduced to facilitate the diffusion of intra-task affinities from the low-scale to the high-scale levels. Ultimately, the task-specific high-scale features and low-scale features are concatenated to produce the final task-specific features, which are then utilized for the respective task predictions.In the realm of multi-task learning, it is a common observation that the parameters of the encoder are significantly larger than those of the decoders, as illustrated in Table 2 . Moreover, encoders are typically pre-trained on extensive image datasets, endowing them with a wealth of image representation information compared to the more lightweight decoders. In light of this, we introduce the Scale Attentive Feature Fusion Module (SAFF) to harness the rich information embedded within the task-sharing feature extractor (Figure 3 ). The SAFF module comprises two key sub-modules: Scale Alignment (SA) and Scale Enhanced Feature Fusion (SEFF). As depicted in our framework (Figure 2 ), all high and low-level features (𝑋 0 , 𝑋 1 , 𝑋 2 , 𝑋 3 ) are refined by the SAFF modules.Scale Alignment. Given a task-sharing feature 𝑋 𝑖 at the target scale level 𝑖, SA modules align all other source scale features to the same scale with the scale 𝑖:EQUATIONwhere 𝑌 𝑖 denotes the 𝑖th scale feature that contains all scale features except scale 𝑖. 𝑿 𝒋 denotes a feature set contains three source scales. P 𝐻 𝑖 ×𝑊 𝑖 denotes that changes input scales to the same scale with 𝑋 𝑖 . 𝑓 𝑐𝑎𝑡 denotes the concatenation on the channel size. 𝑊 𝑖 denotes the reshape dimension operation with learnable weights. To avoid high computational problems, we reduce the dimension of 𝑋 𝑖 to the low dimension 𝑋 𝐿 𝑖 with a 1 × 1 convolution and batch normalization. 𝑊 𝑖 reduces the dimension of the other scale information 𝑌 𝑖 to the same dimension with 𝑋 𝐿 𝑖 , otherwise high dimension of 𝑌 𝑖 lead to model losing the target scale information. Scale Enhanced Feature Fusion. SEFF module incorporates the information from other scales 𝑌 𝑖 to the target scale 𝑋 𝑖 . In this way, target scale features contain rich semantic and structural information from other scales. The architectural design of the SEFF module is inspired by the structure of the Transformer block. The inclusion of convolution to capture more nuanced structural information, as convolution has been observed to excel in this regard compared to attention mechanisms. In our approach, unlike conventional Transformer-based methods, we fusion the benefits of both the Transformer block and convolution. Firstly, given the feature 𝑋 𝐿 𝑖 from scale 𝑖 and 𝑌 𝑖 from other scales, SEFF learns a a unified feature representation for scale 𝑖 with a Scale features fusion module. Subsequently, to preserve the integrity of the information specific to the target scale 𝑖, 𝑋 𝐿 𝑖 are added to the fused feature:EQUATIONwhere 𝑆𝐹 𝐹 () is the Scale features fusion function. In this paper, we utilize two convolution layers and ReLU activation function to achieve 𝑆𝐹 𝐹 (). With this function, we obtain well-learned scale fused feature 𝑆𝐹𝑋 𝑖 with rich information from target scale 𝑖 and other scales. Subsequently, we enhance the feature learning process by introducing a dedicated spatial feature learner and a channel feature mixer, which collectively refine both the spatial and channel dimensions for scale 𝑖. For the spatial feature learner, we have crafted a group-wise 3×3 convolution, which is specifically aimed at capturing local spatial structures. In parallel, for the channel feature mixer, we utilize a 1 × 1 convolution to adeptly blend and refine the channel-wise information derived from the spatial features. This dual approach ensures a comprehensive learning process that attend to both the spatial and channel aspects of the feature representation, thereby enhancing the overall discriminative power of the scalespecific features.EQUATIONEQUATIONwhere 𝑆𝑃 𝑖 denotes 𝑖th scale enhanced spatial features. 𝑓 𝑔𝑟𝑜𝑢𝑝 denotes group-wise convolution with a group equal to the input channel number. 𝑓 1×1 () means 1 × 1 convolution function for mixing channel. X𝑖 represents the 𝑖th scale feature with enhanced by the other scale levels. As a result, X𝑖 is enriched with information not only from scale 𝑖 but also from the other scales, which collectively offer a broad spectrum of semantic and spatial insights from the shared feature extractor. To amplify the characteristics of each target scale feature, we design a SEFF module for each individual scale. Through the application of these specialized modules, we procured four sets of enhanced scale features: X 0 , X 1 , X 2 , X 3 . These comprehensive feature sets lay the foundation for the robust multi-task learning capabilities of our framework.Traditional dense prediction methods [1, 36] process high scale features on the decoder part. The scale of the highest level feature is generally 𝐻 /4 ×𝑊 /4, while the lowest level feature is 𝐻 /32 ×𝑊 /32. The complexity of the highest scale is precisely 64 times that of the lowest when self-attention is applied to them. Additionally, the inter-pixel relationships at the highest scale level often yield global relationships that are predominantly zero. This suggests that high scales may predominantly encode structural information rather than global relationships. Hence, we introduce a Self-Feature Distillation Module (SFD) to extract global relationships from the lowest two scale features, as depicted in Figure 4 .Unlike current decoder-focused MTL methods, the input to each SFD is restricted to the low-scale target task-specific features, excluding any other source task features. Consequently, SFD modules do not involve task interaction processes. We abuse X𝑖 𝑡 to be the task 𝑡 feature at scale 𝑖. To extract the intra-task global relationships from the low-scale features, we incorporate the self-attention mechanism within the SFD modules. Instead of gathering information from other tasks, we focus on extracting diverse information from the target task itself. Consequently, the key and value are derived by self-feature combination, with a set of target task-specific features serving as the input. Given a task-specific feature X𝑖 𝑡 , 𝑖 ∈ {2, 3}:EQUATIONEQUATIONEQUATIONwhere, 𝑄, 𝐾, 𝑉 ∈ 𝑅 𝐻 𝑖 𝑊 𝑖 ×𝐶 denote the query, key, and value of the SFD module. K is equal to V in our setting for the model efficiency. Flat() function denotes the flatten and linear normalization process to the input feature map. 𝑁 is the task number and 𝑊 means the learnable weights. Therefore, K and V are the combination of various learned features from the 𝑡th target task feature itself.𝐴 𝑖 𝑡 ∈ 𝑅 𝐻 𝑖 𝑊 𝑖 ×𝐻 𝑖 𝑊 𝑖 denotes the global relationships of task 𝑡 from the scale 𝑖. In this way, 𝐴 𝑖 𝑡 from the low scales 2, 3 contains rich long-range global relationships, which is designed to diffuse these relationships to the higher scales later, as illustrated by subsection 3.4. Then, we obtain the refined low scale task feature 𝑅𝑋 𝑖 𝑡 by:EQUATION𝑅𝑋 𝑖 𝑡 = 𝐹 𝐹 𝑁 (𝑅𝑋 𝑖 𝑡 ) + 𝑅𝑋 𝑖 𝑡 (10) where FFN() denotes the feed-forward network. With residual process with X𝑖 𝑡 , 𝑅𝑋 𝑖 𝑡 also contains the information from the encoder part. SFD modules save the computational resources with selfattention way on the low task-specific feature scales. Moreover, SFD modules without cross-task information do not encounter the problem mentioned from Figure 1 in section 1 , without using task interaction. Finally, we process SFD modules to the low-scales features to obtain refined task-specific features 𝑅𝑋 2 𝑡 and 𝑅𝑋 3 𝑡 .Typically, high-scale features encapsulate rich structural information, whereas low scales are characterized by high-level semantic relationship information. Current multi-task learning methods [1, 35, 36] typically process high-scale features that integrate information from both high and low levels. This approach embodies a rich mixture of semantic and structural information. However, the model complexity for high-scale features is significantly greater than that of low scales. Therefore, we introduce the Low-To High Scale Feature Diffusion (LTHD) module to propagate the low-scale global relationships to the high scales, as depicted in Figure 5 .Given the global relationships 𝐴 2 𝑡 and 𝐴 3 𝑡 , the objective of LTHD modules is to propagate these relationships to the high-scale features X 0 𝑡 and X 1 𝑡 for task 𝑡. The scale size of the high-level features is substantially larger than that of the lower scales. Therefore, processing the diffusion from 𝐴 2 𝑡 and 𝐴 3 𝑡 to the high scales in the high scale level increases the model complexity. Furthermore, most of the global relationships between pixels on the high scales are close to zero, necessitating a more refined approach to information diffusion. To address these challenges, in LTHD, we first process the pooling operation to the high-scale features X 0 𝑡 and X 1 𝑡 to produce a lower-scale version of them. Since the scale levels of the global relationships 𝐴 2 𝑡 and 𝐴 3 𝑡 are are distinct, we upsample 𝐴 3 𝑡 to match the scale with 𝐴 2 𝑡 . Subsequently, we employ a cross-scale attention mechanism to diffuse the integrated global relationships to the high scales. Specifically, for the scale 0 features of task 𝑡 X 0 𝑡 :EQUATION)EQUATION)EQUATIONwhere, 𝐴 0 𝑡 represents the approximate attention map for scale 0 of task 𝑡. Instead of employing self-attention on high scales to generate 𝐴 0 𝑡 , we design it with low-scale global relationships. 𝑅𝑋 0 𝑡 denotes the refined features from scale 0. 𝐹𝑒𝑎𝑡𝑢𝑟𝑒𝐹𝑢𝑠𝑒 () signifies the fusion process using Convolutional layers followed by Batch Normalization. By processing the LTHD modules on the highest two scales, we obtain refined task-specific features 𝑅𝑋 0 𝑡 and 𝑅𝑋 1 𝑡 . Finally, the ultimate task-specific features 𝐹𝑋 𝑡 are obtained by:EQUATIONThe loss function of the proposed framework is the combination of all task loss functions with different fixed weights. We follow the current methods [1] to design the overall loss function. 
experiments 4.1 dataset: 
 The PASCAL-Context dataset [2] is an extension of the PASCAL VOC dataset [5] . It comprises 10,103 images, with 4,998 designated for training and the remaining 5,105 for testing. This dataset is annotated for five distinct tasks: boundary detection, surface normal estimation, semantic segmentation, human part segmentation, and saliency estimation. The NYUD-v2 dataset [20] focuses on indoor scenes and includes 1,449 labeled RGB images. Of these, 795 are randomly allocated for training, while 654 are reserved for testing. Each image in NYUD-v2 is annotated for four dense prediction tasks: semantic segmentation, depth estimation, boundary detection, and surface normal estimation.Implementation Details. To ensure a fair comparison, we use identical experimental settings as other decoder-focused MTL methods [1] . Our backbone model employs HRNet18 [29] , Swin Transformer [12] and Vit-B. For HRNet18 and Swin Transformer, we set the learning rate to 0.01 and 0.001 for the NYUD-v2 dataset, and 0.0001 for the PASCAL-Context dataset. Across all experiments, we maintain a batch size of 8 and train for 50k iterations with a weight decay rate of 0.0005. All training processes are conducted on NIVDIA RTX3090 GPUs. More details will be provided by the source code.Metrics. We follow the existing MTL methods [1, 35, 39] to evaluate the performance of semantic segmentation (SemSeg) and human parts segmentation (PartSeg) using the mean Intersection over Union (mIoU) metric. For depth estimation (Depth), the root mean square error (RMSE) is utilized, and saliency detection (Sal) is measured by the maximal F-measure (maxF). Surface normal estimation (Normals) is assessed using the mean angular error (mErr), and boundary detection (Bound) employs the optimal-dataset-scale F-measure (odsF). We evaluate the multi-task performance with the per-task performance drop (Δ 𝑚 = 1 𝑁 𝑁 𝑖=1 (𝑃 𝑖 𝑚 -𝑃 𝑖 𝑏𝑎𝑠𝑒 ) 𝑃 𝑖 𝑏𝑎𝑠𝑒 (-1) 𝑤 𝑖 ),where 𝑃 𝑖 𝑚 denotes the performance of MTL method on the 𝑖-th task and 𝑃 𝑖 𝑏𝑎𝑠𝑒 denotes the baseline performance of this task. (-1) 𝑤 𝑖 denotes the sign symbol, (-1) 𝑤 𝑖 = 1 if higher value performance is better, and (-1) 𝑤 𝑖 = -1 if lower value performance is better.We implement the comparison experiments with existing MTL methods [1, 32, 35, 38, 39, 43, 45] on both multi-task datasets NYUD-v2 and PASCAL-Context.NYUD-v2. As shown in Table 3 , the proposed TIF surpasses the current methods both on CNN and Transformer backbones without using any explicit task interaction methods. Meanwhile, the model complexity of the proposed method also achieves better performance with the current efficient single-scale decoder-based MTL methods such as DeMT [36] . It indicates that task-specific decoders with the proposed SFD and LTHD modules can achieve high accuracy on multi-tasks as well as high efficiency. Furthermore, as shown in Table 4 , we compare our method with the large-scale MTL methods (multi-scale decoder-based) such as TaskPrompter [40] , TaskExpert [39] , InvPT [38] , MQT [35] . The results show that the proposed TIF achieves SOTA performance with small parameters and FLOPs. PASCAL-Context. We further evaluate our works with the existing methods on the PASCAL-Context dataset. As shown in 5, our method surpasses the current MTL methods both on the efficient CNN and effective Transformer backbones. Particularly, the performance of our method surpasses DeMT 2.95%, 2.86% on the PartSeg task with Swin-B and Swin-T backbones. It also surpasses DeMT on Bound task 2.42%, 2.17% on Swin-B and Swin-T.The comparison results on both two public datasets and Table 1 denotes that the task interaction technique used by current methods still has negative transfer problems. The proposed TIF with a simple encoder-decoder design achieves state-of-the-art performance without using task interaction modules. Instead, we replace it by strengthening the task interaction by enhancing the task-sharing encoder, which is also demonstrated by Table 1 . By designing efficient decoders without explicit task interaction, the model complexity surpass remarkably compared to multi-scale decoder-based methods [35, [38] [39] [40] and the efficient SOTA methods such as DeMT [36] . 7 demonstrates that applying SFD modules to the two lowest scales outperforms using it on a single scale. This indicates that the global relationships at both scales are advantageous for the performance of most tasks. Furthermore, the model complexity is notably reduced when applying SFD to the two lowest scales compared to using it on the higher scales. Consequently, we exclusively utilize SFD on the two lowest scales. 10 , we compare multi-scale MTL methods, assessing model parameters and performance with HRNet18 and Vit-B backbones.TaskPromoter [40] and MTI-Net [26] are SOTA MTL methods based on multi-scale decoders. Our TIF method also follows this approach. TIF outperforms TaskPromoter in performance with a smaller model size, demonstrating remarkable MTL performance without task interaction modules or complex high-scale global and local dependencies.In this section, we conduct three qualitative studies on the proposed TIF. Firstly, we conduct task similarity experiments, as illustrated in Figure 6 , revealing that task similarity varies across the SFD modules at scales 2 and 3. The lower scale exhibits higher task similarity than the higher scale, validating the introduction of SFD modules at the lowest two scales. Moreover, different tasks exhibit varying task similarities at both scales 2 and 3, with some tasks exhibiting high dissimilarity. This dissimilarity can lead to negative transfer when implementing task interaction. Additionally, we observe that different input images display distinct task similarities, making it challenging to model an explicit and accurate task interaction method for dense prediction multi-tasks. Secondly, we and MQT [35] in complex scenes.conduct qualitative experiments to underscore the significance of LTHD modules. As depicted in Figure 7 , models with LTHD modules achieve more precise predictions for multiple and large objects compared to those without LTHD, indicating that LTHD modules enhance performance by diffusing global relationships in SemSeg tasks. This aligns with the quantitative results presented in Table 6 . Thirdly, we compare our method with the current CNN-based approach ATRC [1] and the transformer-based method MQT [35] . As shown in Figure 8 , our method demonstrates superior performance on most tasks without relying on task interaction. 
conclusion: 
 In this paper, we introduce a Task-Interaction-Free Network (TIF) for multi-task learning. Unlike traditional methods rely on task interaction modules, TIF leverages a novel task-agnostic SAFF to enhance task interaction by task sharing information the encoder part. This approach avoids the negative transfer commonly encountered in task interaction strategies. Additionally, to mitigate the computational complexity of multi-scale decoder-based MTL methods, we propose two complementary modules: the SFD and LTHD modules. The SFD modules extract rich inner-task global dependencies for low-scales, while the LTHD modules propagate these dependencies from the lower scales to the higher scales.  
: 
 , MTL for dense prediction tasks [8, 32, 38, 40] presents more complexity due to the intricacies of pixel-level classification and regression.To address the complexity of learning dense prediction multitasks and effectively modeling and training such challenging tasks, researchers have explored MTL through the development of specialized architectures [14, 17, 22, 32, 35, 38, 40, 42] and optimization strategies [3, 10, 11, 18] . Architectural design of MTL primarily comprised of two directions, (i) encoder-focused MTL [8, 16, 17, 24] and (ii) decoder-focused MTL [32, 35] . Encoder-focused MTL methods [8] typically involve a vast number of parameters due to the absence of a shared encoder, requiring separate encoders for each task. Decoder-focused MTL methods [32] achieve high performance by using a shared encoder and a task interaction mechanism. The shared encoder extracts task-agnostic features from an image, which individual decoders convert into task-specific features. The task interaction module then refines these features using multimodal distillation methods.However, current decoder-focused methods encounter two primary challenges when processing dense prediction tasks.Firstly, many task interaction modules in decoder-based methods [1, 32, 35, 38, 43, 45] encounter the negative transfer [44] problem due to the uncertainty regarding the similarity between each pixel of the target task and the source tasks. Several research efforts [1, 26] aim to address this issue. For instance, PADNet [32] , PSDNet [45] , and MTI-Net [26] have introduced local structures to handle task interaction, ensuring only small spatial relationships between tasks. Leveraging self-attention and Vision Transformers (ViT) [4] , methods like InvPT [38] , DeMT [36] , and TaskPrompter [40] design global relationships between tasks. However, they still struggle to accurately model task interaction and fully resolve negative transfer. We believe one major factor to negative transfer is the potential negative influence of one of the irrelevant source tasks on target tasks through task interaction modules. In Figure 1 , we illustrate two tasks, semantic segmentation (SemSeg) and surface normal estimation (Normals). We analyze scenes with task interaction (Model 1) and without task interaction (Model 2). In Model 1, both SemSeg and Normals features are involved to refine Normals task features. Consequently, Model 1 influences the updating weights of SemSeg task, thereby affecting both SemSeg and Normals features if task interaction modules are not well-designed. Conversely, both tasks remain unaffected if all source tasks are SemSeg, as depicted in Model 2 of Figure 1 . To prove the observation, we implement the comparison experiment between Model 1 and Model 2. As shown in Table 1 , global self attention without other tasks obtains better results compared to global task interaction cross multi-tasks. We compare these models based on different task numbers, which all indicate task interaction encounter the negative transferring.The second challenge faced by current multi-scale decoder-based MTL methods [26, 38, 40] is the significant computational burden arising from the complexity of pixel-level tasks. For example, designing decoders and task interaction modules at a quarter of input image scale results in high computational resource requirements for pixel-level processing.Based on these observations, we propose the Task-Interaction-Free Network (TIF) for multi-task learning.Firstly, instead of designing task interaction modules, we opt to enhance the task-agnostic features from the task-sharing encoder. This decision is motivated by the task sharing encoder typically possesses high number of parameters compared to the decoder. Specifically, the sharing encoder inherently possesses the capability to implicitly achieve task interaction due to its high capacity. Therefore, we propose a novel Scale Attentive-Feature Fusion Module (SAFF) to enhance task-agnostic encoder features at each scale level by utilizing all scale features.Addressing the second observation for multi-scale decoder-based MTL methods, we introduce two novel modules: (1) the Self-Feature Distillation Module (SFD) and the Low-To-High Scale Feature Diffusion Module (LTHD). The SFD module is tasked with extracting self-relative global relationships from task-specific features at the lowest two scales. (2) To mitigate complexity at higher scale levels, the LTHD module is proposed to diffuse global relationships from lower to higher levels. This approach ensures that task-specific features encompass both high semantic information and high structural information while maintaining relatively low computational requirements.In summary, our contributions are as follows:• We propose the TIF for multi-task learning, comprised of three modules: (1) the Scale Attentive-Feature Fusion Module (SAFF); (2) the Self-Feature Distillation Module (SFD);(3) and the Low-To High Scale Feature Diffusion Module (LTHD). Through integrating these modules, TIF facilitates the processing of multi-tasks without the need for explicit task interaction design. • We design the SAFF to augment the task-sharing information from the encoder component. This enables the implicit processing of task interaction and facilitates the extraction of rich information from robust encoders. • SFD and LTHD are proposed to address the computational challenges inherent in current multi-scale based dense prediction multi-tasks.  
 
 Appendix:  


Decision: - Summary of Strengths and Contributions

The paper introduces a multi-task learning framework aimed at mitigating negative transfer between tasks. The framework includes the Scale Attentive-Feature Fusion (SAFF) module, the Self-Feature Distillation (SFD) module, and the Low-To-High Scale Feature Diffusion (LTHD) module to enhance task-agnostic feature modeling and reduce computational complexity. The authors conducted experiments on PASCAL-Context and NYUD-v2 datasets, demonstrating significant performance gains using the proposed method. The paper is well-organized, clearly presented, and includes comprehensive evaluations that highlight the effectiveness of TIF.

- Summary of Weaknesses and Rebuttal Response

Despite the strengths, several concerns were raised by the reviewers:

1. Novelty and Insight (Reviewer 2zx2): Reviewer 2zx2 noted that the overall network pipeline, consisting of a task-sharing encoder and task-specific decoders, is common in addressing multi-task problems. The main technical contribution, the SAFF module, was seen as lacking deep insight. Despite the authors’ rebuttal explaining the design rationale, the reviewer maintained their position that the technical contribution is incremental. I agree with Reviewer 2zx2 the novelty is not significant, as task-specific module is a common way in multi-task learning. However, I acknowledge the techniqual contribution of the proposed framework, which is simple and effective compared with recent methods (e.g., MLoRE).

2. Negative Transfer and Generalizability (Reviewer nnEZ): Reviewer nnEZ appreciated the novelty in addressing negative transfer but emphasized the need for more clarity on when the impact of negative transfer is severe enough to justify a task-interaction-free approach. The reviewer also requested more details on the generalizability of the proposed method with different backbones and larger kernel sizes. The authors addressed these points in the rebuttal, clarifying the scenarios of negative transfer and providing additional results.

3. Comparison with SOTA Methods (Reviewer k6Rn): Reviewer k6Rn highlighted the need for comparisons with more recent state-of-the-art methods and additional analyses on the method’s effectiveness. The reviewer also pointed out minor issues like missing parameter and FLOP comparisons and a typographical error. The authors’ rebuttal included comparisons with recent methods and detailed analyses, addressing most of these concerns.

- Final Recommendation

After considering the detailed feedback from all reviewers and the authors’ responses, I recommend accepting the paper. The authors have provided sufficient evidence of the method’s effectiveness through extensive experiments and addressed most of the concerns raised during the review process. While there are areas for improvement, particularly in providing deeper insights and broader comparisons, the contributions of this paper are significant and relevant to the multimedia and multimodal processing community.

Reviews: review: paper_summary: paper_summary: This paper proposes a Task-Interaction-Free Network (TIF) that uses a designed task-agnostic SAFF to model the task-level relation. The authors that the proposed method can alleviate the negative transfer. Besides, the authors propose the SFD and LTHD modules to reduce the computational complexity. Experiments are conducted on PASCAL-Context and NYUD-v2, and show the performance gain when using their method.
strength_weakness: strength_weakness: This paper conducts extensive experiments to show the effectiveness of the proposed method.

This paper is well-organized and easy to follow
limitations: limitations: - The proposed TIF consisting of a task-sharing encoder and 𝑁 task-specific decoders, which is not novel and a common network pipeline for addressing the multi-task problem. The main technical difference is that the authors design a SAFF to encode the multi-scale features. I think that the SAFF also lacks a certain degree of insight. It is suggested that the authors improve it during the rebuttal period.

- This paper introducs many enchanced modules such as Self-Feature Distillation module, Scale Attentive-Feature Fusion module, Low-To-High Scale Feature Diffusion to boost the multi-task performance, which seems to be an incremental improvement and lacks deep insight for multi-task learning community.
paper_summary: paper_summary: This paper proposed a task-interaction-free multi-task learning framework with efficient hierarchical feature representation, it mainly introduces 3 components: SAFF module providing rich task-agnostic features, SFD module capturing global semantic relationships and LTHD diffusing global pixel relationships.
strength_weakness: strength_weakness: The strengths are: 
(1) They conduct adequate evaluation to show the superiority of the proposed method. 
(2) The novelty is good, especially “the negative transfer when jointly learning multiple weakly correlated tasks” described in Figure 1. 
(3) The clarity is good; the figures of each component and detailed explanation are easy to understand.
limitations: limitations: The limitations are: 
(1) While the existence of negative transfer in multi-task learning is indeed interesting, it is essential to determine when the impact of negative transfer between multiple tasks is so severe that it renders the design of a task-interaction module unnecessary. This question needs to be addressed because it affects the generalizability of the proposed method. 
(2) In Table 2, besides the SwinTransformer backbone, are there other backbones used in the multi-task learning model? If so, please provide more multi-task learning model to illustrate that the encoder is heavier than decoder.
(3) In the spatial feature learner of Figure 3, have you tried using a larger kernel size besides 3 by 3?
(4) In Section 3.4 “most of the global relationships between pixels on the high scales are close to zero, necessitating a more refined approach to information diffusion.” could you provide some proof to support this conclusion?
(5) In Formula 7, X ̂_0^i should be X ̂_t^i .
(6) In the Task-specific modules, there are a total of four scales features (X ̂^0, X ̂^1, X ̂^2, X ̂^3). The authors classify the first two scales as high-scale features with more structural information and the last two scales as low-scale features with more semantic information. My question is whether it would be sufficient to use only the first high-scale feature and only the last low-scale feature in SFD and LTHD, and could you describe the complementary between X ̂^0and X ̂^1 in LTHD, and X ̂^2and X ̂^3 in SFD.
paper_summary: paper_summary: The authors introduce a Task-Interaction-Free Network including three modules to avoid the negative transfer commonly encountered in task interaction strategies.
strength_weakness: strength_weakness: 1. Clear motivation.

2. Detailed illustration of the proposed method.

3. Comprehensive experiments, convincing results, and clear presentation.
limitations: limitations: 1. Need to discuss and compare with more recent SOTA methods

2. The comparison of number of parameters and FLOPs are not presented in Table 5.

3. Missed “Table” in Line 682.

4. The limitations of the work are not discussed.

5. More analyses on the reasons for the method's effectiveness may be better.