Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: TaskPrompter presents a new framework for multi-task dense scene understanding. The framework assigns each task a  learnable token which allows task-specific and task-generic learning in both the encoder and decoder. The task tokens and image tokens interact through the attention layers in transformers.

strength_and_weaknesses: Strength
- The overall architecture. Developing an encoder and decoder under such settings is challenging and the author’s were able to include task-specific and task-generic representation learning. The architecture can be applied to many different tasks. 
- The idea of using prompts to represent the different tasks 

Weaknesses
- The main weakness is that it is unclear from the experiments how important it is to have task-specific and task-generic features in both the encoder and decoder. This could be addressed through additional experiments where a task-specific only encoder is used. 
- The experimental section could use more details. For example, what is the ‘Task Prompter Baseline?’ Hierarchical Prompting seems to have the largest increase in performance. What happens when different combinations of modules are added? 
clarity,_quality,_novelty_and_reproducibility: The overall architecture and approach presented is novel. Due to the complex nature of the architecture, the description of the method is not always clear and thus reproducibility could be difficult. 
summary_of_the_review: I recommend acceptance because of the novelty of the architecture: the lack of task-specific components and using prompting to represent different tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper address the problem of multi-task dense scene understanding, including semantic segmentation, human parsing etc. A transformer based method is proposed, and task-specific prompt tokens are designed to enable the transformer architecture to utilize its full capacity on all tasks. Authors modified the multi-head attention module by a spatial-channel task prompt learning module, and the decoding part is separately designed to produce predictions. Experiments are carried out on several scene understanding datasets to demonstrate its efficacy.
strength_and_weaknesses: ### Strength
* The proposed method can put most of the network's capacity on learning all tasks simultaneously; the cross-task representation interactions are also modeled.
* The novelty is good. I feel it's quite new to involve the prompting concept into multi-task vision problems.
* Experiments show that the model's performance is good.

### Weaknesses
* It seems that the model and experiments assume that supervision for all tasks are there. It should be easy to extend the work on dataset where each example may have supervision of a subset of the tasks. This may be a more practical and flexible setting, as during the inference time, user may only need a single task for an example.
* The design of "Spatial-Channel Task Prompt Learning Module" is a bit complicated. I am wondering if it makes sense to make the model an encoder-decoder design, where the encoder encodes the images, and the decoder handles the prompts and cross-attend to the image embeddings. Also it would be necessary to analysis the computation overhead of the proposed method.
clarity,_quality,_novelty_and_reproducibility: Clarity: overall good
Quality: good
Novelty: good
Reproducibility: should be possible to reproduce
summary_of_the_review: The paper is interesting and novel, and I feel it may be helpful for future vision model design.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces TaskPrompter - a multi-task transformer network for dense scene understanding. The framework has three parts - (1) Prompt Embedding, (2) Spatial-Channel Task Prompt Learning, and (3) Dense Spatial-Channel Task Prompt Decoding - with a strong focus on the latter two - which constitutes the main contributions of the paper. The effectiveness of the method and its components is thoroughly proven through multiple ablation studies. Experimental analysis is performed on popular benchmarks for multi-task learning - Pascal-Context and NYUD-v2 (the appendix also provides state-of-the-art results on Cityscapes-3D on two of the three tackled tasks).
strength_and_weaknesses: Strengths:
* The topic is of great interest to the research community
* The method has a high degree of novelty
* Good ablation studies
* Method agnostic of the different types of transformer architectures

Weaknesses:
* Minor typos throughout the manuscript - I recommend proof-reading once again
* There is a minor error in the Appendix - Table 4 - the authors should bold the number from SETR-vitB (Zheng et al., 2021) since their performance is better than the one from TaskPrompter.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, well-written, and of high quality. The design idea of using "task prompts", which are task-specific learnable tokens to learn spatial and channel-wise task-specific information for each task, although it has been introduced before, for other NLP tasks, their application for dense prediction in visual tasks is original. 
summary_of_the_review: The paper has no significant flaws, the method is sound, and it has a good amount of novelty. The experimental setup is fair with convincing results over previous state-of-the-art methods. The reviewer has no grounds for rejecting this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a transformer framework with spatial-channel multi-task prompting for both generic and task-specific feature learning. 

The authors observe that both spatial and channel information are important for visual dense prediction tasks. Thus, the authors propose to use spatial-channel task prompts to learn their relationships with image tokens. This design could better train the transformer to capture useful pixel information for better feature learning.

While the prompts are usually task-specific, the authors also found that different prompts can contribute to cross-task performance enhancement.

Experiments on NYUD-V2 and Pascal-Context are strong.
strength_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow. The authors presented details of their module designs. 
2. The proposed idea is interesting and technically valid for dense scene prediction tasks. 
3. The authors conducted several analyses to verify the importance of the proposed components. 
4. The experiments are thorough and clear. 

Weaknesses:
1. The paper only considered multi-task scene-based dense prediction datasets. I wonder whether the authors consider the generic benchmark, such as ADE20K, COCO-stuff and Cityscapes. I understand the authors applied it to multi-task problem to demonstrate the cross-task interaction/contribution, however, since the authors claimed it is able to learn task-specific representations, it will be more convincing to compare with those SOTA on ADE20K and others.
2. In Table 2, adding hierarchical prompting seems the most effective one compared to others. The authors may want to discuss more about the reasons, but not just talk about the empirical observations in appendix.
3. In Table 2, I wonder if the model in each row has the same model size? If the model sizes are different, how to verify the performance improvements are attributed to the proposed components or extra parameters?
4. While the model design is good and interesting, the model seems complex and heavy compared to existing frameworks. Would it be possible to provide some discussions regarding model speed and complexity?

clarity,_quality,_novelty_and_reproducibility: The model design is new and interesting. It seems helpful to improve spatial awareness for the transformer. 
summary_of_the_review: The authors proposed an interesting transformer architecture design that could be useful for dense prediction tasks. However, the architecture design is a lot more complex and not easy to reproduce.

While the authors focused on multi-task learning and shown good signals in cross-task interaction learning, it is unclear how the proposed model performs compared to single-task SOTA on the well-known benchmarks as mentioned in Weaknesses.  Since the authors claimed their model has ability for the learning of task-generic and task-specific representations, additional comparisons might be added. The paper may need some clarifications or adjustments. I will update my rating based on the authors' feedback.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-task architecture based on transformers for dense prediction tasks, named TaskPrompter. The design of TaskPromper was motivated by learning task-general, task-specific and cross-task relationships all into a unified learning module. To achieve this, the paper designs a spatial-channel prompting technique that jointly learns spatial and channel interactions through two self-attention layers. Additionally combined with other design strategies such as hierarchical prompting and cross-task reweigthing, the proposed architecture reaches a great performance on a range of multi-task benchmarks.
strength_and_weaknesses: Strength.

- The experiments were conducted showing consistently improved performance over a range of competitive baselines.
- The visualisation of per-task affinity map looks interesting and intuitive, showing the effectiveness of the proposed design.

Weakness.

- Motivation and Design. The paper mentions in multiple places claiming that prior methods such as Cross-Stitch Netwoks, MTAN, and MTI-Net learn cross-task information through hand-craft structures, with task general, task shared representations were structured in each individual modules. However, from my perspective, the proposed method also learns these representations in different modules. Specifically, we may argue that the TaskPromper even further splits task-specific representation into spatial and channel modules, with cross-task information, are encoded in the decoder layers. As such, the proposed design does not strictly follow the motivation.
- Prompt or Tricks? In Table 1, we can find each design component contributes some sense of performance improvements. Among which, the hierarchical prompting (which is a common design strategy in decoder-based MTL) contributes the most. This also makes me concerned that the overall performance is mainly because of the hierarchical design rather than the complicated prompting module.
- Why channel? I did not follow the motivation of proposing channel prompts in the spatial-channel task prompt learning module design. Why learning interactions among different channels is important for MTL? Sure I can find the improved performance in Table 1, but this does not justify the design. Overall, I found learning channel interactions is very strange and unintuitive. It may be possible that the improved performance is mainly from the additional cross-attention layer.
- Lack of Efficiency measurement. One important aspect of MTL architecture design is efficiency. As highlighted in multiple baseline methods such as MTI-Net, PAD-Net, it's important to show the size of network in terms of number of tasks. This allows us to understand the computation cost of the proposed design with the scaling of different number of tasks.

clarity,_quality,_novelty_and_reproducibility: I have stated the clarity and quality in the previous section. I believe the architecture itself is novel. But the architecture design lacks proper motivation and definitely requires some further clarification. 
summary_of_the_review: The paper proposes a multi-task architecture TaskPrompter for dense pixel-wise scene understanding. The architecture itself is complicated in design, though achieving great performance in standard multi-task benchmarks. The study of the contribution of each design component is limited, and lack of motivation. I am currently holding a borderline score for this work, and will raise/lower the score depending on further rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a transformer-based multi-task architecture called TaskPrompter for dense visual prediction tasks. By assigning each task a learnable prompt token, TaskPrompter enables task-specific and task-agnostic learning in both the encoder and decoder, and also encourages cross-task representation interactions via attention layers with spatial-channel prompting. Hierarchical prompting and cross-task reweighting are further leveraged. Extensive experiments along with ablation studies validate the effectiveness of TaskPrompter and its components.

Generally, the reviewers consider the contribution of introducing the prompting concept (or learnable tokens) that have been widely-used in other fields like NLP into the dense visual prediction tasks as novel and solid. The approach seems to be general, e.g., with respect to different supervision settings and different types of transformer architectures. The performance improvements are noticeable and consistent. In the original manuscript, some implementation and experimental details, design rationale, effect of different components (especially the hierarchical prompting), and analysis on model complexity and efficiency were missing or unclear. The authors’ responses together with additional experimental results have successfully addressed the reviewers’ concerns as well as the public comments.

All reviewers recommend acceptance. I support the reviewers’ recommendation. Authors should attend to main points in the reviews, especially additional analyses and experiments, when preparing a final version. Also, as promised by the authors, the codebase should be open sourced.
justification_for_why_not_higher_score: The current recommendation is based on the reviewers’ scores. A higher score seems to be not well justified.
justification_for_why_not_lower_score: All reviewers recommend acceptance. I agree with this recommendation. Authors should attend to main points in the reviews, especially additional analyses and experiments, when preparing a final version. Also, as promised by the authors, the codebase should be open sourced.
Decision: Accept: poster