Multi-Task Dense Predictions via Unleashing the Power of Diffusion

Authors: Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, Bo Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Diffusion model, Multi-task learning, Dense prediction, Joint denoising, Cross-task encoding

Abstract:
Diffusion models have exhibited extraordinary performance in dense prediction tasks. However, there are few works exploring the diffusion pipeline for multi-task dense predictions. In this paper, we unlock the potential of diffusion models in solving multi-task dense predictions and propose a novel diffusion-based method, called TaskDiffusion, which leverages the conditional diffusion process in the decoder. Instead of denoising the noisy labels for different tasks separately, we propose a novel joint denoising diffusion process to capture the task relations during denoising. To be specific, our method first encodes the task-specific labels into a task-integration feature space to unify the encoding strategy. This allows us to get rid of the cumbersome task-specific encoding process. In addition, we also propose a cross-task diffusion decoder conditioned on task-specific multi-level features, which can model the interactions among different tasks and levels explicitly while preserving efficiency. Experiments show that our TaskDiffusion outperforms previous state-of-the-art methods for all dense prediction tasks on the widely-used PASCAL-Context and NYUD-v2 datasets. Our code is available at https://github.com/YuqiYang213/TaskDiffusion.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730176554711

thread_type:
official_review

<thread_object>
creator: reviewer_qtvq

<content>
summary:
The proposed TaskDiffusion encodes the task-specific labels into a task-integration feature space to unify the encoding strategy.In addition, the proposed cross-task diffusion decoder conditioned on task-specific multilevel features, which can model the interactions among different tasks and levels explicitly while preserving efficiency. The proposed TaskDiffusion’s experiment results also achieve comparable performance.

strengths:
1 This paper is well-organized.

2 This paper leverages diffusion models as an effective solver for multi-task dense prediction.
The core idea of seeking a task-specific proxy to perform cross-task reasoning via diffusion model is intuitive.

3 It outperforms the state-of-the-art in most cases.

weaknesses:
1 The value of \Delta_m (MTL gain metric) requires the value of  the single task (ST) model. It is suggested that the authors release the value of ST in Table 3. Following the calculation criteria of MTL on \Delta_m metric, showing Baseline ST and MT values will have a beneficial impact on the development of MTL. The reviewer saw Table 7, but it appeared in the appendix rather than in the main paper.

2 From the results in Table 3, the core diffusion decoder designed in this paper does not seem to bring much growth to the model. Table 3 also reveals that a good encoder leads to greater performance gains. The relative performance improvement brought by the proposed Cross-Task Diffusion Decoder is not even comparable to ATRC and InvPT.

3 Task-fused features U are generated by A$\cdot$F in line 302. Why the features after A$\cdot$F are fused features? The same question arises on line 308. Why can this matrix multiplication represent feature fusion and what is its mathematical interpretation?
Both task-fused features and task-fusing features mean the same map U.Moreover, the same problem has $p$ in 302 line and $p$ in Eq.2.These inconsistency are confusing to the reader.

4 To ensure comprehensive evaluation, it is imperative that the study provides a comparative analysis of the proposed technique against the most recent methods ("DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction" AAAI 2023; "Multi-Task Learning with Multi-Query Transformer for Dense Prediction" TCSVT 2024; "Multi-Task Dense Prediction via Mixture of Low-Rank Experts" CVPR 2024). In addition, the discussion of the decoder-based MTL approach in the related work section lacks an introduction to recent works, such as DeMT AAAI’23, 3DAwareMTL ICLR’24, MQTransformer TCSVT’24 and MLoRE CVPR’24.

5 Figure 4 does not seem to demonstrate the visualization advantage of the proposed TaskDdiffusion model in the appendix, compared to TaskPrompter.

questions:
1 The authors state: “To our knowledge, we are among the first to leverage diffusion models in …”. The DiffusionMTL (CVPR’24) model already took advantage of the diffusion model.

2 What is Cross-Task Label Encoder based on?  Some tasks are labeled with images or data in matlab format. How do authors encode these different data formats?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730472545895

thread_type:
official_review

<thread_object>
creator: reviewer_t3fk

<content>
summary:
This paper presents a diffusion-based method for multi-task dense prediction, leveraging the conditional diffusion process. The method effectively models task relationships during the denoising process through the cross-task diffusion decoder and cross-task label encoder. The experiments demonstrate its effectiveness on two benchmark datasets.

strengths:
1. The proposed framework facilitates joint diffusion denoising of multiple tasks, allowing the diffusion process to effectively capture cross-task relationships, which is beneficial for multi-task learning.
2. The method demonstrate strong performance on two benchmark datasets, PASCAL-Context and NYUD-v2.

weaknesses:
1. The proposed modules have limited novelty. The cross-task diffusion decoder builds upon existing works, such as DDP [1], while incorporating interaction techniques from multi-task learning. The cross-task label encoder functions as an inverse process of task-specific heads in standard multi-task models. It would be better if the paper could demonstrate more insights in the utility of diffusion for multi-task dense prediction.
2. The authors are recommended to compare their method with more state-of-the-art methods, especially those published in 2024, such as [2, 3].
3. There is room for improvement in presentation, including language and figures. For example, the text in Figure 1 right is too small to be easily readable.

[1] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. arXiv preprint arXiv:2303.17559, 2023.
[2] Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts. In CVPR, 2024.
[3] Shuo Wang, Jing Li, Zibo Zhao, Dongze Lian, Binbin Huang, Xiaomei Wang, Zhengxin Li, and Shenghua Gao. TSP-Transformer: Task-specific prompts boosted transformer for holistic scene understanding. In WACV, 2024.

questions:
1. In Section 3.2, task-specific auxiliary heads are used to generate intermediate predictions that are supervised by ground truth labels. It is interesting to know the contribution of these auxiliary heads to the final results, as they have different training procedure with the diffusion process.
2. In Algorithm 2, what is the relationship between masks_pred and m_preds?
3. In Table 1, the FLOPs of the proposed method with a ViT-large backbone is reported as 610G. However, in Table 6, the ablation study indicates that the model with 3 steps utilizes 667G FLOPs. Why does the model with the ViT-base backbone has a higher computational cost than the ViT-large backbone?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557645713

thread_type:
official_review

<thread_object>
creator: reviewer_cv9p

<content>
summary:
This submission proposes a network named TaskDiffusion based on the recent diffusion work (DDP[1]) for multi-task dense prediction. The TaskDiffusion consists of three proposed modules. Pixel-level encoder encodes an image to feature which is utilized as multi-level conditions for diffusion decoder. Cross-task label encoder encodes the label of different tasks into a shared feature as a denoising input. Cross-task diffusion decoder is used for cross-task feature interaction and prediction decoding. The evaluations on two benchmarks demonstrate the effectiveness of the proposed method.

strengths:
- This submission proposed a new diffusion model for MTL dense prediction, inspired by DDP.
- The empirical evaluations on two datasets (PASCAL-Context and NYUD-v2) suggest that the proposed method could improve the MTL performance with 4 and 5 tasks.

weaknesses:
- The related works of MTL dense prediction are quite old. Some recent research[1-4] related to MTL dense prediction should be considered as references.
- The claim “This strategy utilizes the diffusion model to model the task relations explicitly in a coarse-to-fine process during denoising, …” in lines 79-80 should be clarified more deeply. To be specific, the illustration of $A_i$ in Fig.3 is hard to comprehend. What is the relationship between a pair of task?
- Some experimental details should be corrected and provided. For instance, the original PASCAL-Context does not provide the label for the surface normal task. And what is the resolution of input?
- Some comparisons are not fair. For example, the results of HRNet18 or HRNet48 should not be compared to ViT-large, since the backbone affects the performance remarkably.
- Some magnificent results are missing. In Table.1 and 2, the single task should be considered a baseline to figure out the impact of the negative transfer and the improvement.  And $\Delta_m$ should be provided in Tables 1 and 2.
- The presentation of Table.3 and 4 should be improved. How the $\Delta_m$ is calculated? If it is based on the single task baseline, it should be provided.

[1] Exploiting Diffusion Prior for Generalizable Dense Prediction, CVPR 2024

[2] UNITE: Multitask Learning With Sufficient Feature for Dense Prediction, IEEE Transactions on Systems, Man, and Cybernetics: Systems

[3] Rethinking of Feature Interaction for Multi-task Learning on Dense Prediction, arXiv 2312.13514

[4] MultiMAE: Multi-modal Multi-task Masked Autoencoders, ECCV 2022

questions:
Please refer to weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610882375

thread_type:
official_review

<thread_object>
creator: reviewer_ycpj

<content>
summary:
This paper introduces a diffusion-based model for multi-task dense prediction. During each training step, it first projects multi-task labels into a unified feature space and applies noise within that space. It then implements a cross-task diffusion decoder to iteratively predict the denoised features. Experimental results demonstrate performance improvements over previous methods.

strengths:
- The study of multi-task learning is an important direction.

- We see performance improvement on PASCAL and NYUD compared to previous SOTA methods.

- The algorithm scheme is helpful for understanding.

- Extensive experiments have been conducted to thoroughly evaluate the model performance.

weaknesses:
- The figures lack sufficient detail. The task interaction module and level interaction modules should be included in Figure 2 for better clarity.

- In Algorithm 2, the term "masks_pred" is not mentioned in the context. Should it be "m_preds"?

- The novelty is somewhat limited, as previous research has already demonstrated that diffusion-based multi-task learning is feasible.

questions:
- Can you visualize the cross-task map at different steps to better understand it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732199720463

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all the reviewers for their constructive feedback on our paper. 
Following the reviewers' advice, we revised our paper, and all the revised parts are marked as **blue**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposed a novel framework that benefits from the diffusion model for multi-task dense prediction. The reviewers appreciated the contribution proposed in the method and the superior results achieved by comparing with some existing state-of-the-art performances.  AC agrees with the positive comments from the reviewers and recommends accepting the paper as a poster.