TaskPrompter: Spatial-Channel Multi-Task Prompting for Dense Scene Understanding
Authors: Hanrong Ye
Dan Xu
Venue: ICLR 2023 poster (2023)
Keywords: Multi-task Learning
Scene Understanding
Computer Vision

Abstract: Learning effective representations simultaneously from multiple tasks in a unified network framework is a fundamental paradigm for multi-task dense visual scene understanding. This requires joint modeling (i) task-generic and (ii) task-specific representations, and (iii) cross-task representation interactions. Existing works typically model these three perspectives with separately designed structures, using shared network modules for task-generic learning, different modules for task-specific learning, and establishing connections among these components for cross-task interactions. It is barely explored in the literature to model these three perspectives in each network layer in an end-to-end manner, which can not only minimize the effort of carefully designing empirical structures for the three multi-task representation learning objectives, but also greatly improve the representation learning capability of the multi-task network since all the model capacity will be used to optimize the three objectives together. In this paper, we propose TaskPrompter, a novel spatial-channel multi-task prompting transformer framework to achieve this target. Specifically, we design a set of spatial-channel task prompts and learn their spatial- and channel interactions with the shared image tokens in each transformer layer with attention mechanism, as aggregating spatial and channel information is critical for dense prediction tasks. Each task prompt learns task-specific representation for one task, while all the prompts can jointly contribute to the learning of the shared image token representations, and the interactions between different task prompts model the cross-task relationship. To decode dense predictions for multiple tasks with the learned spatial-channel task prompts from transformer, we accordingly design a dense task prompt decoding mechanism, which queries the shared image tokens using task prompts to obtain spatial- and channel-wise task-specific representations. Extensive experiments on two challenging multi-task dense scene understanding benchmarks (i.e. NYUD-V2 and PASCAL-Context) show the superiority of the proposed framework and TaskPrompter establishes significant state-of-the-art performances on multi-task dense predictions. Codes and models are made publicly available at https://github.com/prismformore/Multi-Task-Transformer.

Published as a conference paper at ICLR 2023
TASKPROMPTER: SPATIAL-CHANNEL MULTI-TASK PROMPTING FOR DENSE SCENE UNDERSTANDING
Hanrong Ye and Dan Xu
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology (HKUST)
Clear Water Bay, Kowloon, Hong Kong {hyeae, danxu) ust.hk
ABSTRACT
Learning effective representations simultaneously from multiple tasks in uni- fied network framework is a fundamental paradigm for multi-task dense visual scene understanding. This requires joint modeling (i) task-generic and (ii) task- specific representations, and (iii) cross-task representation interactions. Existing works typically model these three perspectives with separately designed struc- tures, using shared network modules for task-generic learning, different modules for task-specific learning, and establishing connections among these components for cross-task interactions. It is barely explored in the literature to model these three perspectives in each network layer in an end-to-end manner, which not only minimize the effort of carefully designing empirical structures for the three multi-task representation learning objectives, but also greatly improve the repre- sentation learning capability of the multi-task network since all the model capacity will be used to optimize the three objectives together. In this paper, we propose TaskPrompter, novel spatial-channel multi-task prompting transformer frame- work to achieve this target. Specifically, we design a set of spatial-channel task prompts and learn their spatial- and channel interactions with the shared image tokens in each transformer layer with attention mechanism, as aggregating spatial and channel information is critical for dense prediction tasks. Each task prompt learns task-specific representation for one task, while all the prompts can jointly contribute the learning of the shared image token representations, and the in- teractions between different task prompts model the cross-task relationship. To decode dense predictions for multiple tasks with the learned spatial-channel task prompts from transformer, we accordingly design dense task prompt decoding mechanism, which queries the shared image tokens using task prompts obtain spatial- and channel-wist task- task-specific representations. Extensive experiments on two challenging multi-task dense scene understanding benchmarks (i.e. NYUD- V2 and PASCAL-CCntexII) show the superiority of the proposed framework and TaskPrompter establishes significant state-of-the-art performances on multi- task dense predictions. Codes and models publicly available https: / //github. con/ pri wwrismffrrorr Muulti Task Transformes
INTRODUCTION
Dense visual scene understanding is a fundamental research topic in computer vision that involves many dense prediction tasks, including semantic segmentation, depth estimation, surface normal estimation, boundary detection, etc. These distinct tasks share a fundamental understanding of the scene, which motivates researchers design learning systems that model and predict multiple tasks in a unified framework, which is called "multi-tas" learning" (MTL). MTL mainly has two strengths: on one hand, learning unified multi-task model for multiple tasks is typically more parameter- efficient than training several single-task models; on the other hand, different tasks can facilitate each other with good design in MTL (Vandenhende et al., 2021).
With the powerful boost of deep learning, researchers have successfully designed highly promising multi-task learning models by exploiting the commonality and individuality of the tasks (Mani-

---

Published as a conference paper at ICLR 2023
Task Prompt Emberdding
Spatial- Channel Task
Dense Spatial -CChnneel Task
Predictions of T tasks
Prompl Learning
Prompi Decoding
Spatial Task Prompts
Task Prompls
Spatial Task
Patch Tokens
Task Feature
e
Task Prompts
Prompt
Channel Task Prompts
Image
Channel Task
Prompt
Putch Tokens
Patch
Tokens
For each task
Attention Flow
Tearming of tpsk-fieee representations
Leaming of crost-task interactions
Learning task-generic representations
Prompting
Figure 1: Illustration of our Spatial-Chamne Multi-task Prompting framework (TaskPrompter) TromPer unifies the learning of task-specific and task-generic representations as well as cross- task interactions in each layer throughout the whole transformer architecture, with the embedding of task prompts and and patch tokens. The task prompts are projected to spatial task prompts and channel task prompts to learn spatial- and channel-wise interactions, which are critical for dense predictions. The spatial and channel task prompts as well as patch tokens are further used in the pro- posed Dense Channal-Channe Task Prompt Decoding module to prompt dense task-specific features and the final multi-task predictions.
nis et al., 2019; Xu et al., 2018; Kendall et al., 2018; Kokkinos, 2017). Traditionally, researchers manually design different types of modules in the multi-task network architecture to learn useful information for multi-task predictions three aspects: task-generic representations, task-specific representations, and cross-task interactions. For instance, earlier works (Liu et al., 2019; Gao al., 2019; Misra et al., 2016) design dedicated modules to leam task-specific representations and embed cross-task information interactions through hand-crafted structures deployed in the encoder, while several recent works (Ye & Xu, 2022; Li al., 2022b; Vandenhende al., 2020) choose to de- velop task-specific and cross-task modules in the decoder, and share encoder among different tasks. However, all of these methods decouple the learning of task-generic representations, task-specific representations, and cross-task interactions, into different network modules, which not only makes the architecture design more challenging as each module needs to be configured with specific structure and capacity, but also suboptimal as learning effective communication among these three important perspectives of information is critical for multi-task dense prediction.
To tackle the above-mentioned issue, we believe a better MTL framework should be capable of learning task-generic and task-specific representations as well as their interactions jointly in each layer across the whole network architecture. In this paper, we achieve this goal by proposing novel Spatial-Channel Multi-task Prompting framework, coined as TrskPrompter. The core idea of TaskPrompter is to design "spatial-channel task prompts" which task task-specifie learnable tokens to learn spatial- and channel-wise task speciipe information for each task. More specifically, the task prompts embedded together with the task-generic patch tokens computed from the input image as input of a transformer with a specially designed Spatial-Channe module. The task prompts and patch tokens interact with each other and refine themselves by means of attention mechanism in each transformer layer. In this way, TaskPrompter manages to learn task- generic and task-specific representation well as cross-task interaction simultancously and does not require the design of different types of network modules.
Task Prompt Learning
With the learned chatial-channel task prompts and image patch tokens, is a non non-trivial problem how to effectively decode multi-task dense features and predictions from them. To meet this challenge, we further propose novel Dense Spatial-Channel Channel Task Prompt Decoding method, which leverages both the spatial-wise and channel-wise affinities calculated between the task prompts and the patch tokens in attention modules to extract dense task features. The features are further refined by the cross-task affinity obtained from the self-attention weights among task prompts. The final multi-task dense predictions are produced based on the dense task features.
In summary, the contribution of this work consists of three parts:
We propose a novel task dense scene understanding. Our method essentially combines the learning of task-generie and task-specific representations, as well as cross-task interactions in each layer across the whole network architecture by introducing task prompts in our transformer.
Channel Multi-task Prompting framework (TaskPrompter) for multi-
2

---

Published as a conference paper at ICLR 2023
A Spatial-Channel Task Prompt Learning module is designed. It can be flexibly deployed in each transformer layer for learning and refining task prompts and patch tokens along both spatial and channel dimensions.
We further design a novel Dense Spatial-Channe Task Prompt Decoding method based on the learned task-specific task prompts and task lask-generic patch tokens to generate pixel-wise predictions for multiple tasks simultaneously.
Extensive experiments on two challenging multi-task dense prediction benchmarks (i.e. PASCAL- Context and NYUD-v2) clearly verify the effectiveness of the proposed method, which demonstrates superior performance compared with the previous state-of-the-art methods.
RELATED WORK
Multi-task Dense Scene Understanding with Deep Learning Several works have verified that scene understanding tasks can benefit from each other via multi-task learning (MTL) in deep learn- ing (Li et al., 2022a; Maninis ct al., 2019; Kokkinos, 2017; Misra et al., 2016). MTL can also improve the computation efficiency of both training and inference compared with those with single- task models (Vandenhende et al., 2020). On the one hand, some researchers work the optimization process of MTL, including loss design (Yang et al., 2023; Liu et al., 2022; Zamir al., 2020; Kendall al., 2018) and gradient manipulation (Chen et al., 2020; 2018). On the other hand, the architecture design of multi-task deep models is also widely explored (Vandenhende et al., 2021). We can divide most existing works on multi-task architecture design into two categories: encoder-based and decoder- based methods. sacoddr-bassed methods (Gao et al., 2019; Liu et al., 2019; Misra et al., 2016) focus on designing cross-task interaction modules in the encoder, while the -aaeed methods embed these modules in the decoder stage. As the decoder-basee meth- ods can readily leverage the weights of backbone models pre-trained on large-scale image dataset (e.g. ImageNet). they generally yield stronger performance than srcoder-based methods and have become mainstream pipelines for multi-task scene understanding (Bruggemann et al., 2021; Xu et al., 2018; Zhang al., 2019; Zhou et al., 2020; Zhang et al., 2021). Recently, transformer models have been widely explored for multi-task learning (Ye & Xu, 2022; Bhattacharjee et al., 2022; Xu al., 2022b; Liang et al., 2022; Xu al., 2022a). Specifically, Ye & Xu (2022) propose strong multi-task transformer "InvPT", which uses self-attention (Vaswani et al., 2017) to simultaneously learn spatial and cross-task relationships in a global context, establishing strong multi-task perfor- mance. However, these existing methods decouple the learning of three important perspectives in multi-task learning, i.e. task- task-generie, task-specific representations, and cross-task interactions, into separate modules, which requires careful design for each module, making the construction of the overall multi-task architecture challenging and suboptimal. To address this issue, this paper pro- poses a novel and effective multi-task prompting framework based on transformer to simultaneously learn these perspectives in each network layer in an end-to-end manner.
improving
Prompting Prompting is a burgeoning paradigm designed for transformer. It is originally proposed to adapt pre-trained transformer models to downstream tasks by modifying the input of transformer layers without fine-tuning all the model parameters (Schick & Schütze, 2021; Gao et al., 2021; Lester et al., 2021; Brown et al., 2020; Shin et al., 2020). While some works try to explicitly design prompts (Shin et al., 2020; Petroni et al., 2019), more and more researchers turn to back- poc-ppropagation for learning prompts (Lester et al., 2021; Li & Liang, 2021), which is termed as "Prompt Tuning". It typically embeds learnable prompts in the input token sequence of transformer and automatically tunes the prompts in training (Liu et al., 2021a; Hambardzunyan et al., 2021). Recently, the idea of prompting introduced in Vision Vision-language tasks (Zhou al., 2022b; Ju et al., 2022; Yao al., 2021: Ge et al., 2022; Zhou et al., 2022c; Radford et al., 2021), and further explored in visual tasks, such as image classification (Jia et al., 2022; Sandler et al., 2022; Bahng et al., 2022; Wang et al., 2022), open-vocabulary semantic segmentation (Rao et al., 2022; Zhou et al., 2022a), and object detection (Du et al., 2022; Feng et al., 2022). More related to our work, HyperPrompt (He et al., 2022) addresses multiple NLP tasks with prompts generated from task-specific hyper-networks. It is different from our proposal in three major ways: First, HyperPrompt needs to perform separate inferences for different tasks with different prompts; Second. because of the first limit, it has no cross-task interaction mechanism. Third, our special spatial-channe task prompting and decoding mechanisms are designed for dense predictions, which is totally different from NLP tasks. More-
3

---

Published as a conference paper at ICLR 2023
over, end-to-end multi-task prompting framework enables cross-tast interactions throughout the whole network and generates pixel-wise predictions for all tasks with only one inference.
TASKPROMPTER: SPATIAL-CHANNEN MULTI-TASK PROMPTING
The proposed TaskPrompter framework (see Fig. 1) can be divided into three parts, i.e. Prompt Embedding, Spatial-Chamin Task Prompt Learning, and Dense Spatial-Chame Task Prompt De- coding. We now introduce the details of these components one by one in this section.
3.1 PROMPT EMBEDDING
We adopt a classic transformer pipeline to embed an input image into a sequence of patch to- kens (Dosovitskiy al., 2021). should be noted that our method is independent of the selection of different transformer architectures. The input image is first processed by a patch embedding layer, which convolutional layer with feature resolution downsampling. After the patch embedding layer, suppose the output feature map has a shape (H, W, C*) where H and W are the height and width, and C is the number of channels of the feature map. The feature map is first reshaped into N = H x W patch tokens in a C-dimensional latent space and then added by positional encod- ings. To enable multi-task prompting, we propose to embed T learnable task-specific tokens as task prompts in the same C-dimensional latent space that of patch tokens, where T is the number of tasks multi-task learning. Each task prompt corresponds to task. Next, the task prompts are concatenated with the patch tokens and form token sequence matrix Zo:
(1) where [-] indicates concatenation, {p" yT denotes the sequence of task prompts, and {x] de- notes the sequence of patch tokens. In this way, the input image and task prompts are encoded and aggregated into a joint token sequence, which is refined through the proposed spatial-channel task prompt learning process.
Zu= pp: p"; ; x²;x;; :xx] E (T++N)xxC ,N H x W
3.2 SPATIAL-CHANNEL TASK PROMPT LEARNING
We design a spatial-channel task prompt learning module for concurrently learning and refining task prompts and patch tokens along both spatial and channel feature dimensions, the spatial and channel information are both critical for various dense prediction tasks (Fu et al., 2019). The module details depicted in Fig. 2. The goal of our design learn task-generic and task-specific visual representations, as well as cross-task interactions simultaneously, by taking full advantage of the structure of transformer with the embedded task prompts. Specifically, for each transformer layer, we adopt the basic layer structure of the classic ViT (Dosovitskiy et al., 2021) including MLPs, layer normalization layers (LayerNorm), and skip connections, while the core multi-head attention module is replaced by the proposed Spatial-Channe Task Prompt Learning module, in which, we design two types of task prompt learning paradigms: (i) Spatial Task Prompt Learning and (ii) Channel Task Prompt Learning.
Without loss of generality, we use the first transformer layer for illustration. After the first Lay- erNorm. we obtain a token sequence Z LayerNorm (Zo) R(T+N)xC which consists of task prompts P E TxC and patch tokens X E R.NxC. Then, P is projected into spatial task prompts P and channel task prompts Pe where the superscript s denotes 'spatial' and e denotes 'channel', and the projection can be formulated as:
p = fc-c(P) E Txx
p- fc=NN-N(P) E T
where Sc-C denotes an identity mapping with the feature dimension unchanged; SC-NN C TxN is a 2-layer MLP that projects the input task prompts P from C- dimensional latent space to N-dimensional latent space (N H x W). In this way, each channel task prompt aligns the feature dimension to the number patch tokens for performing channel interactions. Then, Pa and X are fed into the proposed spatial task prompt learning module, and concurrently P and X are fed into the channel task prompt learning module.
Spatial Task Prompt Learning This module simultaneously learns spatial task prompts Ps and patch tokens X, where each spatial task prompt interacts with patch tokens along the spatial dimen- sion to model the spatial-wise relationships between task prompts and patch tokens. The spatial
4

---

Published as a conference paper at ICLR 2023
Sparial Task Prompis
Spatial Task Prompt Learning
Mults head Selt- Attention
Updated
Tokens
Idenrity
Concat
LayenNorn
Tisk Prompts
Patch Tokens
Ve
SputlakChanned
Croso Aern-Alleenn Altenium
Prompt Louming Medul
MUP
Transposo
Updated Task Procppts
LayenNorm
Add
Task Promptt& & Patch Tokens
Channel Task Prompts
Channel Task Prompt Learning
Figure 2: An illustration of the proposed Spatial-Channel Channel Task Prompt Learning module in trans- former layer. This module learns the T task prompts by interacting with patch tokens along both the spatial and channel dimensions. The task prompts are projected into T spatial task prompts (each with C-dimensional) for Spatial Task Prompt Learning and into I' channel task prompts (each with N N-dimensional) for Channel Task Prompt Learning.
task prompts Pa and patch tokens X are stacked as a token sequence [P;; X (T++N)xCC which is linearly projected by learnable parameters W Wi, W E RCxC, to respectively produce the query Q", , the key K and the value V as follows:
Q PP: X W q' K = [P W V = pP x W
(3)
As a standard procedure, before we compute the multi-head self-attention (MSA), we need to partition Q" K" V evenly along the last dimension into different groups as input of different heads. Suppose that we utilize heed heads in MSA, after head partition we have Q" K* V K e R Nices X(T+N)x Whoud and spatial self-attention map is computed by A' Q' x K where the symbol 'T' denotes transposing operation that transposes the last two dimensions of a tensor. A is then scaled and normalized by softmax function, and multiplied by V to obtain new token sequence. The token sequence merges different heads and is projected by a linear layer as in stan- dard MSA, which provides ps and patch tokens X'.
output token sequence, consisting of updated spatial task prompts
Channel Task Prompt Learning This module is proposed to model the channel-wise relationships between the channel task prompts P and the patch tokens X along the channel dimension. We perform cross-attention to model the channel-wist relationships. The channel task prompts P are projected by a learnable parameter W; E N to produce the query Q E T and the patch tokens X are separately projected by two learnable parameter matrices Wi E RCxC and W E RCC, and transposed to produce the key K' and the value V
Q = W 9 K (X x W V (X W
(4) along the last
Before computing the multi-head cross-attention, we need to partition Q K K Vo dimension into different groups the input of different heads in multi-head channel-wist cross at- tention. Suppose the number of heads used in Channel Task Prompt Learning is No head the most straightforward way is evenly partitioning the matrices along their last dimension into N head groups However, as K and V are computed from patch tokens X, the last dimension of them contains the spatial relationship of pixels, while a standard partition method disrupts this spatial relationship which is critical for learning features for dense predictions. A more reasonable strategy is reorga- nizing Q K Ve based the spatial adjacency. Specifically, we first reshape these matrices spatial shape HxC as H x W = N, and then partition the spatial planes formed by the last di- mensions evenly into Ni local windows, as shown in Fig. 6. Notably, Whead needs to be properly so that the number of windows along the height dimension No and the width dimension Nw: Wi.. N Whead This process is the proposed "window partition" to maintain the
satisfies N win x N spatial relationship for multi-head channel cross-attention calculation. After the window partition, we have Q E Ni xT N/Ng.ad, KS, V E NecaaeX CNNggea Then, channel attention maps are calculated by A = Q x T A is scaled and normalized by a softmax function and multi- plied by V to obtain the updated channel task prompts Pd after being processed by linear layer. To update the overall task prompts with information learned from both the spatial and channel task prompts, shown in Fig. 2, pe' is projected back to the C-dimensional latent space with
5

---

Published as a conference paper at ICLR 2023
Special Attention
Dot Product
Hadamand Product
Spatial Tisk Prompiing
Cres Tiok
Tranepoce
Affinity
Promppted Task
Purch
Tokets
Speiih Thek Prorqe Affaay
Featuncs
Haal
Yank Prodiction
Fasion
Flod
Fask Prodiction
Chanzel Attention Map
Channel Task Prompting
Reweighting
Cross-task
Faad
Tiok Proliction
Pronpt FxOxili
Dense Spatial-Channel
K
Task Prompt Decoding
Figure 3: A diagram illustration of Dense Spatial-Channel Task Prompt Decoding. The spatial attention map and channel attention map are calculated from the query and key tensors in Spatial Task Prompt Learning and Channel Task Prompt Learning, respectively. They are used to guide the decoding of task-specific features from patch tokens along spatial and channel dimensions.
2-layer MLP IN-C: : RTXN-TXC and then added by pa' to obtain a combined task prompts p as update of P as follows:
P'= + fN-c(P"))
(5)
P' and X' are refined by the typical LayerNorm and MLP, and then stacked new token sequence Z', which is fed into the next transformer layer following the same procedure for further learning.
3.3 DENSE SPATIAL-CHANNER TASK PROMPT DECODING
To decode multiple dense predictions for distinct tasks from the task- task prompts and task- generic patch tokens, we need to design effective decoding method for TaskPrompter. Since the task prompts including the spatial and channel task prompts are task-diseriminative. then the affinities calculated between the spatial/channel task prompts and the shared patch tokens are also distinct. The different task prompts localize different spatial regions or channels on the patch tokens. This can also be confirmed from our visualization of the learned spatial and channel affinities, as shown in Fig. 4. Based on the learned spatial and channel task prompts, we propose a Dense Spatial- Channel Task Prompt Decoding method, which consists of Spatial Task Prompting and Channel Task Prompting strategies, to respectively compute spatial-wise and channel-ve -wine task-specific features, for the final multi-task predictions, as shown in Fig. 3.
Spatial Task Prompting Each spatial task prompt corresponds to spatial affinity map by com- puting the affinity between the spatial task prompt and all the patch tokens. We denote this spatial affinity map Spatial-Task- Task--Topp---P Affinity, the learned spatial attention maps, i.e. A' E R Nicus (T+N)x with N = H x W as the num- ber of patch tokens and T as the number of tasks, in the task-prompt learning stage. For the spatial task prompt of task I, we can extract an attention tensor in space Nood XN from A and then we reshape to be a new tensor in space NNhaaaxxxx H xxW which is the Spatial- Task-ll-aa-- Affin- ity and we denote it as AT On the other hand, given updated token sequence X' E N produced from a transformer layer, we transpose and reshape X' into space Rehhend XC/Nfeud XeN.eeaxHXX XxxW and denote it as X The spatial-wise task task-specific features Fū for task can be decoded by:
shown in Fig. 3. It can be extracted directly from
F fA" & "
(6)
where the symbol O indicates the operation of Hadamard product and far(:) represents the opera- tion of reshaping the tensor into space C H WW
Channel Task Prompting Each channel task prompt corresponds channel affinity vector, which can be computed by measuring the affinity between the channel task prompt and all the channels of the patch tokens. We name it Maannel-Task-Pooonee
Affinity, denoted as Ar- with t indicating the task f, which can be used to decode the channel-wise task-specific representation for task t. We obtain the Shannel-task-prompp affinity A," of task from A Nhena XTxC by slicing one along the second dimension and reshaping it into space RC* N x1x N x1, We also reshape X' as X E RCx Nr... x xhwin No WWain, where htwn H/Nb... and Wwin W/Nw... Then, we can compute the channel-win wise task-specific features Fř for task as follows:
F = Sar (A, X(),

---

Published as a conference paper at ICLR 2023
Image
Semseg
Parsing
Saliency
Normal
Boundary
Tasks
Channels
Figure 4: Visualization examples of the spatial-task-prompl affinity (the first three rows) and the hhannel-task-promete affinity (the last row). It can be observed that different spatial and channel task prompts can both attend to distinct spatial channel locations of the patch tokens, which indicates that the task prompts can effectively learn task-specific representations from the interaction with the image patch tokens.
where (-) denotes the operation of reshaping the tensor into the space Cx
Spatial-Channel Fusion To fuse the task-specific features Fi and Fú from the spatial and channel task prompting, concatenate them along the channel dimension and reduce the channel number by half via using 3 x convolution (CONVaxs) with batch normalization (BN) and GELU to obtain fused task-specific feature F, for task as follows:
F, = GELU o BN CONV3x3 (F;;
(8)
Then, we stack the prompted task task-specific features of the T tasks along the first dimension and obtain an overall task feature map F E R xCxCxXX
Cross-task Reweighting The spatial and channel task prompting decode the relationship between task prompts and patch tokens. The cross-task relationship is not involved, while the cross-task relationship also modeled in the task-promp learning stage in the encoder. To encourage cross- task information exchange in decoding stage, further put forward Cross-task Reweighting as also shown in Fig. 3. First, we extract the affinity tensor among T task prompts from the attention map A The affinity tensor is in space Niead XTxT Then, project the first dimension to using a 2-layer MIP, and obtain the Cross Task Affinity AP-P. The prompted task features IR TxxxxXXM HxW are updated by F + AP F. The prompted task features F contain features for all the 7 tasks. They are split and separately fed into T' task-specific prediction heads for dense predictions. Each prediction head is composed of simple 3 x convolutional block with BN and GELU, and a linear projection layer.
Hierarchical Prompting As discussed by previous multi-task transformer for dense scene under- standing (Ye & Xu, 2022), different levels of transformer features help improve the multi-task per- formance. Therefore, we deploy our prompt decoding method to multiple levels of the transformer, and name it "Hierarchical Prompting (HP)". Specifically, we conduct Dense Spatial-Channel Task Prompt Decoding at multiple levels of transformer for prompting the task features, instead only the last layer. The multi-level task features are later fused one by addition to obtain the final prompted task features, which are fed into the prediction heads as described above.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets We evaluate the proposed TaskPrompter mainly two mostly used multi-task dense visual scene understanding datasets, i.e. NYUD-v2 (Silberman et al., 2012) and PASCAL-Context (Chen al., 2014). Details of the datasets presented in Appendix A.3.
7

---

Published as a conference paper at ICLR 2023
Table I: Effectiveness of different components of Task The performance gains compared
against the baseline are shown in brackets. 'L' means lower better and 'f" means higher better.
Model
Semseg mioU
Parsing mloU 1
Saliency maxF f
Normal mEa 1
Boundary odsF 1
MIL Gain
Am 1
STL Model
78.42
68.36
85.34
13.87
73.90
TaskPrompter Baseline
75.04
64.82
84,59
14.17
68.00
-4.11
SPrompp
75.95 (1 0.91)
0.78)
84.91 (1 0.31)
13.97 (1 0.20)
70.50 (1 2.50)
.259 (1 1.52)
+SPumpp + CProongt
76.46 (1 1.41)
66.25 ( 1.43)
85.00 (1 0.41)
13.97 () 0.20)
71.10 (1 3.10)
2.11 (f 2.00)
SPromppt + CPrompt
76.83 (1 1.79)
66.31
85.00 (1 0.41)
(3 0.23)
71.30
3.30)
11.90 2.21)
+SPrumpi+ + Crronpt RW HIP
79.00 (1 3.96) 67.00 ( 2.18)
85.05 (1 0.46)
13.47 (1 0.70)
73.50
5.50)
0.15 ( 4.26)
Table 2: Comparison between TaskPrompter-Ban and TaskPrompter-Largee
PASCAL- Context
NYUDDv2
Model
Semseg mioL
Farsing mloU
Saliency
Normal
Boundary
Semseg
Depih
Normal
Boundary odsF
maxF" 1
mEi 1
odsF 1
miol
RMSE
mEir 1
TaskPPompter-Baee
79.00
67.00 68.89
85.05 84.83
13.47 13.72
73.50
0,5402
18.91 18.47
77.60
TaskPrompter-Lagge
80.89
73,50
55,30
0,5152
78.20
Evaluation Metries Semantic segmentation (Semseg) and human parsing (Parsing) use the mean Intersection over Union (mloU) metric for evaluation. Monocular depth estimation (Depth) uses Root Mean Square Error (RMSE). Surface normal estimation (Normal) uses mean error (mErr) of predicted angles. Saliency detection (Saliency) adopts maximal F-measure (maxF) for evaluation. Object boundary detection (Boundary) is evaluated with the optimall-datast-scalee F-measure (odsF). Multi-task gain (MTL Gain) is evaluated by metric Am introduced (Maninis et al., 2019).
Models Declaration To verify the model effectiveness, we consider the following model variants in Table 1: (i) "TaskPrompter Baseline" is a very strong baseline with comparable performance to the- otate-of-the-arts for multi-task dense predictions. It is built upon ViT-Base with 12 transformer lay- ers pre-trained on ImageNet-22K, and uses task-specific Conv(3x3)- (B1-333333----L-L---CC ReL--CCnv((X]] blocks as heads for final multi-task predictions. (ii) "STL Model" is a counterpart of (i) under single- task setting with the same network structure. It is used to train a set of single-task models, and each model is only trained for one task at a time. (iii) "TaskPrompter +SPrompt +CPrompt +RW +HP" denotes a full version of TaskPrompter using all the proposed modules. It is built on (i) with the same transformer architecture and parameter initialization. It uses 16 heads for Channel Task Prompt Learning. In Hicrarchical Prompting (HP), it selects every third layer, in total 4 layers, to conduct Dense Spatial-Channe Task Prompt Decoding. (iv) "TaskPrompter +SPrompt +CPrompt +RW" removes Hierarchical Prompting from (iii). (v) "TaskPrompter +SPrompt +CPrompt" re- moves task reweighting from (iv). (vi) "TaskPrompter +SPrompt" further disables Channel Task Prompt Learning from (v).
4.2 EXPERIMENTAL RESULTS
Effectiveness of Spatial and Channel Task Prompt Learning and Decoding We evaluate the pro- posed methods on the PASCAL-Contex dataset and report the results in Table 1. Using Spatial Task Prompt Learning can already bring a clear performance improvement on all the tasks, particularly, with a boost of 0.91, 0.78, and 2.50 points for Semseg, Parsing, and Boundary respectively, com- pared against the baseline. By adding the Channel Task Prompt Learning, the performance gains are further increased to 1.41, 1.48, and 3.10 points for the three tasks, respectively. These experimental results clearly demonstrate the effectiveness of the core design of TaskPrompter.
Effectiveness of Cross-Task Reweighting and Hierarchical Prompting Furthermore, as shown in Table 1, cross-task reweighting can improve the performance of most of the tasks, with a multi-task gain (i.e. A of 0.21 points. Hierarchical Prompting helps increase the performance largely on all the tasks, by deploying task-prompt decoding on multiple levels of the transformer. Hierarchical Prompting is designed to facilitate the decoding of task task-specific features with the learned spatial- channel task prompts at each layer of the transformer encoder. With the embedding and learning of the global task prompts from the beginning of the transformer encoder, our model can naturally per- form spatial-channel hicrarchical prompting at different layers for decoding task-specific features, which is very beneficial for producing more effective multi-task representations.
Scaling TaskPrompter We follow VIT (Dosovitskiy et al., 2021) and build the proposed Spatial-Channel Multi-task Prompting framework on a transformer with 24 layers, denoted as TaskPrompter-Large. We also denote the one with 12 layers as TaskPrompter-Bar Base, and compare their performances on both PASCAL-Contes and NYUD-v2 datasets. The results are reported in Table 2. We can observe that models with bigger capacity generally bring better performance for
8

---

Published as a conference paper at ICLR 2023
Semseg (mloU)
Parsing (mloU)
Saliency (maxF)
186
Normal (mErr)
78.6
Boundary (odsF)
73.5
18.6
73.4
87.0
18.5
73.2
73.1
26 Number HfHaads
54
10
16
64
20 Number Heads
54
Number Heads
Number of Hoa
Number Heads
Figure 5: Influence of the number of heads (windows) in Channel Task Prompt Learning.
Table 3: Comparison with otate-of-the-arts on NYUD-v2 (left) and PASCAL-Context (right). Our
TromPer clearly outperforms the previous state-of-the-are
Modd
Semseg mdoU
RMSE 1
Depth
Normal
miar :
Beundary
odi3"
Model
Semsee stoU
Parsing mioU
Saliency
max!"
Nernal
mBrr _
Boundary
odiE" 1
Choes SSich (Miens et 20163
0.6280
30.86
ASTMT (Muninis al. 2019)
(Zbang at. 2019)
36.72
20.82
76.42
Net (Nu 2013)
53.60
59.60
65.90
72.50
20209
0.6246
20:87
76.42
MII Nec (Vandennende
2020)
61.70
00.18
94.78
70.50
PAD-Nes 2013)
36.61
ATRC (Biurpemennn 2011)
59.02
70.96
MIT-Na (Vandesherle
45.07
0.5365
20:27
77.86
ATRC-ASPP (Boggenann 20213
83.01
14.30
2021)
46.33
0.5363
20.18
77:04
ATRC BMTAS
al.,
62.93
82.25
14.24
72.42
Im/' (1
53.56
0.3185
19.04
7A, 10
InFT
2022)
67.61
54.81
14.15
73.00
TaskPrompper (wars)
58.30
0.3152
18.47
78.20
TiskPrompter (ours)
80.89
68.89
84.83
13.72
73,50
most of the tasks, but as discussed by Ye & Xu (2022); Kendall et al. (2018), the performances of some tasks may be worse because of the multi-task competition issue.
Qualitative Visualization of Spatial-Channel Task Prompt Affinity To investigate whether the task prompts learn task-specific affinity on patch tokens, we visualize the affinity values between task prompts and patch tokens in the Dense Spatial-Channel Task Prompt Decoding module as shown in Fig. 4 and Fig. 8. We can clearly observe that the activated spatial-task poommppromtt-oo affinity values are highly related to the particularity of cach task, which indicates that the spatial task prompts can effectively encode task task-specific representations, and attend to different semantic regions of patch tokens that more beneficial for the prediction of a specific task when performing the decoding. On the other hand, we calculate the average of channel-task tasnnel-task-promptt affinity maps from all the test images of PASCAL-Context dataset and randomly select 60 channels for visualization. We can ob- serve that the different channel task prompts have distinct attention responses to different channels, which verifies that channel-task prompt affinity encodes task specifcc relationships between channel task prompts and patch tokens along the channel dimension.
Study of Number of Heads in Channel Task Prompt Learning As introduced in Section 3.2, we partition query and key tensors in Channel Task Prompt Learning into different groups as input of different heads in aross-attention. We report the performance comparison of using different numbers of heads in Fig. 5. We observe that using more heads brings better performance for most tasks.
Comparison with Previous SOTA on NYUD-v2 and PASCAL-Context Table reports compar- ison of the proposed TaskPrompter against previous sate-of-the-art methods, including InvPT (Ye & Xu,, 2022), ATRC (Bruggemann et al., 2021), ) MTI-Net (Vandenhende et al., 2020) and PAD- Net (Xu et al., 2018), on both NYUD-v2 and PASCAL-Contex, datasets. Notably, the previous best method (i.e. InvPT) and our TrokPrompter are built upon the transformer architecture with the same backbone. Our TaskPrompter establishes new of-te-of-the-arr performances on all 9 metrics these two datasets. On NYUD-v2, the performance of Semseg is clearly boosted from the previous best, i.e. 53.56 to 55.30 (+1.74). On PASCAL-Contex dataset, Semseg is improved from the previous best 79.03 to 80.89 (+1.86) and Parsing is improved from 67.61 to 68.89 (+1.28).
5 CONCLUSION
We have presented a Spatial-Channel Multi-task Prompting (TaskPrompter) framework for simul- taneously learning task-generic and task-specific representations well cross-task interaction in each layer throughout the whole transformer architecture. We first propose to learn task prompts to encode task-specific information, and design a dedicated module to learn the relationship between task prompts and patch tokens along both spatial and channel dimensions. Furthermore, we propose a novel spatial-channel task prompt decoding method to generate dense task-specific features for prediction. The effectiveness of our method is validated by both quantitative and qualitative ex- periments, showing superior performances on different task sets. The performances of our method clearly surpass the existing multi-task dense scene understanding models.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research supported in part by HKUST-SAIL. research funding, the Early Career Scheme of the Research Grants Council (RGC) of the Hong Kong SAR under grant No. 26202321 and HKUST Startup Fund No. R9253.
ETHICS STATEMENT
The proposed method focuses on the improvement of multi-task deep learning algorithms without introducing new tasks or datasets. Thus our work doesn't 't raise any new ethical issues.
REPRODUCIBILITY STATEMENT
The experiments conducted in this paper are based on widely used public datasets. We present the proposed methods in detail and include the implementation particulars in Section 4.1 and Sec- tion A.3. The code will be made publicly available to help further study.
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022.
Deblina Bhattacharjee, Tong Zhang, Sabine Süssttrnkk, and Mathieu Salzmann. Mult: an end-to-end multitask learning transformer. In CVPR, 2022. 3
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurIPS, 2020.
David Bruggemann, Menelaos Kanakis, Anton Obukhov, Stamatios Georgoulis, and Luc Van Gool. Exploring relational context for multi-task dense prediction. In ICCV, 2021. 3, 9, 18, 19
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. De- tect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, 2014. 7, 14
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, 2018. 3
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. In NeurlPS, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 16
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021. 4, 8
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, 2022.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open-vocabulary detection using uncurated images. In ECCV, 2022. 3
Jun Fu, Jing Liu, Haijic Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 4
10

---

Published as a conference paper at ICLR 2023
Nils Gählert, Nicolas Jourdan, Marius Cordts, Uwe Franke, and Joachim Denzler. Cityscapes 3d: Dataset and benchmark for 9 dof vehicle detection. In CVPR Workshop, 2020. 16
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. ACL, 2021.
Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In CVPR, 2019. 2,
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. arXiv preprint arXiv: 06687, 2022.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial re- programming. ACL, 2021.
Muhamad Amirul Haq, Shanq-Jang Ruan, Mei-En Shao, Qazi Mazhar ul Haq, Pei-Jung Liang, and De-Qin Gao. One stage monocular 3d object detection utilizing discrete depth and orientation representation. IEEE Transactions on Intelligent Transportation Systems, 2022. 18
Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, et al. Hyperprompt: Prompt-based task-conditioning of transformers. In ICML, 2022. 3
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022. 3
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. ECCV, 2022.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018. 2,
Iasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In CVPR, 2017. 2,
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP, 2021. 3
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Learning multiple dense prediction tasks from partially annotated data. In CVPR, 2022a. 3
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representations: A unified look multiple task and domain learning. arXiv preprint arXiv:2204.02744, 2022b. 2
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. ACI., 2021. 3
Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, and Zhangyang Wang. M3 vit: -ixeure-of-experts vision transformer for efficient multi-task learning with accel-acoe co-design. In NeurlPS, 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In ICCV, 2017. 18
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586 2021a. 3
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention.
In CVPR, 2019. 2.
Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. TMLR, 2022. 3

---

Published as a conference paper at ICLR 2023
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b. 18
Kevis-Kokitsi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multi- ple tasks. In CVPR, 2019. 1, 3, 8, 9, 14
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016. 3,9
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? EMNLP, 2019. 3
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Andrew Jackson. Fine-tuning image transformers using learnable memory. In CVPR, 2022. 3
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. NAACL, 2021. 3
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. EMNLP, 2020.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup-
port inference from rgbd images. In ECCV, 2012. 7, 14
Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Mti-net: Multi-scale task interaction networks for multi-task learning. ECCV, 2020. 2, 3, 18, 19
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansheke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: survey. TPAMI, 2021. 1,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 3, 14
Lijun Wang, Jianming Zhang, Oliver Wang. Zhe Lin, and Huchuan Lu. Sde-depth: Semantic divide- and-conquer network for monocular depth estimation. In CVPR, 2020. 18
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. FCOS3D: Fully convolutional one-stage monocular 3d object detection. In ICCV Workshop, 2021. 18
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yi Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. ECCV, 2022. 3
Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. In CVPR, 2018. 2, 3. 18, 19
Xiaogang Xu, Hengshuang Zhao, Vibhav Vineet, Ser-Nam Lim, and Antonio Torralba. Mtformer: Multi-task learning via transformer and cross-task reasoning. In ECCV, 2022a. 3
Yangyang Xu, Xiangtai Li, Haobo Yuan, Yibo Yang, Jing Zhang, Yunhai Tong, Lefei Zhang, and Dacheng Tao. Multi-task learning with multi-query transformer for dense prediction. arXiv preprint 2rXiv:2205.14354 2022b. 3
Siwei Yang, Hanrong Ye, and Dan Xu. Contrastive multi-task dense prediction. In AAAI, 2023. 3
12

---

Published as a conference paper at ICLR 2023
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision- Maiione models. arXiv preprint arXiv:2109.1179; 2021. 3
Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. In ECCV, 2022. 2, 3, 7, 9, 14, 15, 18, 19
Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao. Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. In CVPR, 2020. 3
Xiaoya Zhang, Ling Zhou, Yong Li, Zhen Cui, Jin Xie, and Jian Yang. Transfer vision patterns for multi-task pixel learning. In ACMMM, 2021.
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Pattern-ffinitive
propagation across depth, surface normal and semantic segmentation. In CVPR, 2019. 3,
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from sequence-to-sequeece perspective with transformers. In CVPR, 2021. 18
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022b. 3
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 2022c. 3
Ling Zhou, Zhen Cui, Chunyan Xu, Zhenyu Zhang, Chaoqun Wang, Tong Zhang, and Jian Yang. Pattern-structure structure diffusion for multi salti--tskk learning. CVPR, 2020. 3,
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PRELIMINARIES: MULTI-HEAD SELF-ATTENTION AND MULTI-HEAD CROSS-ATTENTION
Multi-head self-attention and multi-head cross-attention are both widely used variants of the atten- tion mechanism proposed by Vaswani al. (2017). As the input of the attention module, the token sequences are projected by individual learnable weights to query Q, key K, and value V matrices. Essentially, attention mechanism a weighted addition operation on the value guided by the affinity between query and key values. The output of attention module is calculated by:
QK
Attention( (Q, Q,X) softmax
X V,
where is the emhedding dimension. the input token sequences are projected into Nhead set of query, key, and value tensors and compute attention separately, it is called "multi-head attention". The difference between multi-head self-attention (MHSA) and multi-head attention cross-attention (MHCA) is that the query, key, and value input of MHSA are projected from the same token se- quence, while the query and key matrices of MHCA are projected from different token sequences.
A.2 WINDOW PARTITION IN CHANNEL TASK PROMPT LEARNING
We show a visual illustration of the window partition technique used Channel Task Prompt Learn- ing in Fig. 6.
Q Kc V
Q K.C
Window Partition
N win
H
W
N wir
Figure 6: We partition the spatial planes formed by the last 2 dimensions of K and Ve evenly into local windows maintain the spatial relationship when fed into different attention heads for cross-attention calculation in the Channel Task Prompt Learning.
A.3 IMPLEMENTATION DETAILS
Datasets We evaluate the proposed TaskPrompter mainly on two mostly used multi-task dense visual scene understanding datasets, i.e. NYUD-v2 (Silberman et al., 2012) and PASCAL-Context (Chen et al., 2014). Specifically, PASCAL-CContet provides 4,998 images in the training set and 5,105 images in the testing set. This dataset offers dense labels for multiple tasks including semantic segmentation, human parsing, and object boundary detection. Additionally, Maninis et al. (2019) provide pseudo ground truth labels for surface normals estimation and saliency detection. On the other hand, NYUD-v2 totally provides 1,449 images, in which 795 are used for training and the rest 654 for testing. It includes dense labels for tasks including semantic segmentation, monocular depth estimation, surface normal estimation, and object boundary detection. In our experimental setup, we include all the tasks in these datasets for comprehensive performance study.
Model Training The models for different experiments are trained for 40,000 iterations on all datasets, with batch size of if not otherwise specified. Adam optimizer is adopted with learn ing rate of 2 x 10-5, and weight decay rate of 1 x 10-6, A polynomial learning rate scheduler used during optimization. For the continuous regression tasks (i.e. Depth and Normal) we use C1 Losses. For the discrete classification tasks (i.e. Semseg, Parsing, Saliency, and Boundary) use cross-entropy losses for them. The learnable task prompts are randomly initialized with normal distribution (mean=1, std=1).
Data Processing, For a fair comparison with Invpt (Ye & Xu, 2022), we follow its data processing pipeline. On PASCAL-Context, we randomly crop the input image to the size of 448 x 576. We use the same data augmentation including random color jittering, random cropping, random scaling, and random horizontal flipping.
pad the image to the size of 512 x 512, while NYUD-v2,
14

---

Published as a conference paper at ICLR 2023
795
Semseg (mloU)
61.0
Parsing (mloU)
Saliency (maxF)
16.0
Normal (mErr)
24.0
Boundary (odsF)
79.
763
67.2
55.1
15.8
73.0
760
67.0
13.7
61.2
55.0
15.5
71.5
61.0
114
75.0
Number
Number Levels
Number Levels
Figure 7: Influence using different numbers of levels in Hierarchical Prompting (HP). The levels are chosen evenly based network depth. "1" means not using HP. Using only two levels in HP can already bring large performance gain.
A.4 ABLATION STUDY OF USING DIFFERENT NUMBERS OF LEVELS IN HIERARCHICAL PROMPTING
We investigate how the number of levels used by Hierarchical Prompting (HP) influences the over- all performance of TaskPrompter. The experimental results with the varying number of levels are can be observed that using only two levels in HP can already largely boost the performances on all the five tasks, demonstrating the effectiveness of the hierarchical prompting scheme. Further increasing the number of levels, it helps further improve some tasks (e.g., Sem- seg, Normal, and Boundary), while for some others (e.g., Parsing and Saliency), it shows saturated performances with small variances.
presented in Fig.
A.5 QUALITATIVE STUDY
More Qualitative Visualization of Learned Spatial-Taske Prompp Affinity Maps We provide more qualitative results by visualizing the learned Spatial-Task-Promp Affinity maps on the PASCAL-Context dataset, as shown in Fig. 8. We can observe that the task prompts has very diverse affinities different positions of the image patch tokens, demonstrating task desd-diirmminatvee charac- teristics of the learned task prompts, and also verifying our motivation that the learned task prompts can directly query the task-generic image patch tokens to decode sask-specific representations.
Image
Semseg
Parsing
Saliency
Normal
Boundary
Figure 8: More visualization results of the spatial-task-promptt affinity maps on the PASCAL- Context dataset. We observe that the task prompts attend different areas of the images based on the characteristics of the tasks.
Qualitative Comparison with the Best Performing Transformer-based Method. We qualita- tively compare the produced predictions of the proposed TaskPrompter with the best performing transformer-fe based method in the literature, i.e. InvPT (Ye & Xu, 2022). As shown in Fig. 9 and Fig. 10, our TaskPrompter model generates dense predictions of multiple distinct tasks with clearly better details than the InvPT as marked in circles.
15

---

Published as a conference paper at ICLR 2023
Input
Semseg
Parsing
Saliency
Normal
Boundary
LINPI
GT
LIET
GT
Figure 9: Qualitative comparison with the best performing transformer-base based method (i.e. InvPT) on PASCAL-Contes TaskPrompter generates predictions with better details.
Input
Semseg
Depth
Normal
Boundary
Ouss
ft Damuda d
GT
Figure 10: Qualitative comparison with the best performing transformer-based method (i.e. InvPT)
on NYUD-v2. TaskPrompter generates better multi-task predictions.
A.6 JOINT 2D-3D MULTI-TASK SCENE UNDERSTANDING ON CITYSCAPES-BD 3D
To further examine the proposed PromPeop adapt to tackle joint 2D-3D multi-task scene understanding problem involving three challenging tasks, i.e. 3D object detection (3Ddet), semantic segmentation (Semseg), and monocular depth estimation (Depth), Cityscapes-31 dataset (Gählert et al., 2020).
Dataset Cityscapes-3D (Gählert et al., 2020) is extension of Cityscapes dataset (Cordts et al., 2016) via providing additional 3D bounding box annotations. Cityscapes-3D dataset consists of 2,975 training images and 500 validation images with fine annotations. The image resolution is
16

---

Published as a conference paper at ICLR 2023
3det
Semseg
Depth
Ours
GT
Ours
GT
GT
Ours
GT
Prediction visualization on Cityscapes-3ID. TaskPrompter can generate competitive re- sults for multiple 2D and 3D scene understanding tasks including 3d detection, semantic segmenta- tion, and depth estimation simultaneously.
Figure 11:
1044x2048. For the evaluation of semantic segmentation, we use the more challenging 19-class labels. The models evaluated on the validation set for all the tasks. 3D detection (3Ddet) uses mean detection score (mDS) as metric, which is the official evaluation index provided by Cityscapes- 3D.
17

---

Published as a conference paper at ICLR 2023
Implementation of TaskPrompter on Cityscapes-3D Since the images in Cityscapes have larger spatial resolution and 3Ddet is sensitive to object sizes, we build our multi-task prompting method on Swin-Base Base model (Liu et al., 2021b), which maintains high-resolution features in the backbone. Notably, as the transformer block in Swin-Base uses local window attention instead of global attention ViT, we clone the spatial task prompts and embed them into each local window to model the interaction with all patch tokens. After computing the window attention, the spatial task prompts of different windows are merged into one by taking their average. In the decoding stage, we affinity tensors from all windows as global affinity tensor to query the patch tokens. The channel task prompt learning and decoding are not affected by the Swin-Base architecture, which can be performed in the same way in ViT backbone. The channel number of task prompts is doubled at each Patch Merging layer of Swin-Base to maintain the same channel number patch tokens. The same as those used in experiments on PASCAL-Context and NYUD- v2, we use task-specific 3x3Conv-BBB B--Coon--B--eLL.. -NN-RLUU blocks as prediction heads for generating final task features for all 3 tasks, and linear layer for generating the final predictions for Semseg and Depth. As for 3Ddet, we adopt the final prediction heads of FCOS-3D (Wang et al., 2021) to predict the location coordinates, rotation angles, sizes, object classes, center-ness, and direction classification. To reduce computation cost, we reduce the resolution of images from 1024x2048 to 768x 1 1536. The batch size is set to 2 and the model is trained for 40k iterations. We also use Adam optimizer with a learning rate 2 x 10-S without weight decay. For the 3D detection task, Non-maximum Suppression is used with a threshold of 0.3.
stitch the Maatial-Task-PPommeee
Loss Functions For 3D detection, similar to FCOS3D (Wang et al., 2021), we use focal loss (Lin al., 2017) for object classification, smooth L1 loss for location coordinates and size regression, and cross- loss for direction classification and center-ness regression.
Table 4: Performance of joint 2D-3D multi-task scene understanding on Cityscapes-3D dataset. TaskPrrmmterr achieves better or comparable results against SOTA methods of multiple tasks. Bold denotes the best.
3Ddet mDS
Semseg mloU f
Depth RMSE +
Model
3Ddet: One-Stage (Haq et al., 2022)
26.90
Single-task Models
Semseg: SETR w/ VTT-B (Zheng et al., 2021)
78.02
Depth: SDC-Depth (Wang et al., 2020)
6.92
Multi-task Models
Our Bascline
29.69
76.90
7.00
TaskPrompter
32.94
77.72
6.78
Experimental Results on Cityscapes-3D The overall performance comparison with state-of-the- art methods (single-task models) and multi-task baseline Cityscapes-3D is shown in Table 4. The multi-task bascline adopts a similar design as the multi-task baseline on PASCAL-Context but is built upon Swin-hase backbone. It should be noted that we the first in the literature to simultaneously perform all three tasks on this dataset. We clearly observe that our Promperr can significantly improve our baseline on all three tasks, further confirming the effectiveness of the method. Moreover, TaskPrompter even yields better performance than several SOTA single-task models, such as on 3Ddet (Haq et al., 2022) and Depth (Wang et al., 2020). It also shows decent performance on the highly competitive task (e.g. Semseg) compared with the SOTA (Zheng et al., 2021). We also visualize our prediction results on Cityscapes-3D and compare them with ground truth labels in Fig. 11. TasskPrompeer can generate competitive results for multiple 2D and 3D scene understanding tasks simultaneously. These experiment results further indicate that the proposed TaskPrompter can be effectively adapted to other transformer models and task sets.
A.7 COMPARISON WITH SOTA METHODS USING VIT-LARGE BACKBONE
As transformer-basee multi-task learning methods only appear recently (Ye & Xu, 2022), we re- implement several CNN-based SOTA methods, including ATRC (Bruggemann et al., 2021), MTI- Net (Vandenhende et al., 2020), and PAD-Net (Xu et al., 2018), on ViT-Large backbone. We compare the performances of TaskPrompter and these methods in Table 5. We observe that our TaskPrompter outperforms all previous SOTA methods with less computation cost. The reason is that Task Prompter avoids using heavy multi-task decoder as previous decoder-focused methods
18

---

Published as a conference paper at ICLR 2023
do and propose novel Dense Spatial-Channe Task Prompt Decoding to decode multi-task dense predictions with the help task prompts.
Table 5: Comparison with the state-of-the-art methods PASCAL-Context. Our TaskPrompter
achieves clearly superior performances on all tasks with the same backbone.
Semseg mloU 1 mloU
Parsing Saliency
Normal
Boundary
Model
FLOPs #Param
maxF f mErr I
odsF 1
PAD-Net (Xu et al., 2018)
124G
8IM
53.60
59.60
65.80
15.30
72.50
MTI-Net (Vandenhende et al., 2020)
161G
128M
61.70
60.18
84.78
14.23
70.80
ATRC (Bruggemann et al., 2021)
216G
96M
67.67
62.93
82,29
14.24
72.42
PAD-Net w/ VIT-L (Xu al. 2018)
773G
330M
78.01
67.12
79.21
14.37
72.60
MTI-Net w/ VIT-L (Vandenhende
2020)
774G
851M
78.31
67.40
84.75
14.67
73.00
ATRC w/ VIT-L (Bruggemann et al., 2021)
871G
340M
77.11
66.84
81.20
14.23
72.10
InvPT (Ye Xu, 2022)
669G
423M
79.03
67.61
84.81
14.15
73.00
TaskPrompter (ours)
497G
401M
80.89
68.89
84.83
13.72
73.50
A.8 ABLATION STUDY OF USING TASK-SPECIFIC ENCODERS
To verify the importance of the joint learning of task task specific, task-generic, and cross-task interaction in TaskPrompter, we design model variant of TaskPrompter that uses task-specific encoder for each task. We name this variant "TaskPrompter prompt for the corresponding task, and thus there is no task-generic feature in the encoder stage. We still use our Dense Spatial-Chane Channnel Task Prompt Decoding to generate prediction for each task from the task-specific feature of the encoder. We adopt VIT-S model backbone and compare the model variant to TrskPrompter with also VIT-S backbone in Table 6. We find that when using task- specific only encoders, the model performance decreases all tasks, despite the model capacity and computation cost being much larger as each task has an independent encoder. The results can verify the importance of jointly modeling both task-specific and task-generic features in TaskPrompter, in terms of both effectiveness and efficiency.
TE". each task-specific encoder, we use one task
Table 6: Ablation study of using task-specific encoders (TE) with ViT-S backbone. means lower
better and 'f' means higher better.
Semseg mloU
Parsing mloU f
Saliency maxF f
Normal
Boundary odsF
Model
FLOPs
#Param
mFr +
TaskPrompter
205G 400G
132M
76.57
63.22 61.99
84.54 84.33
13.93 14.15
70.60 70.40
TaskPrompter w/ TE
495M
74.47
Table 7: More ablation study of the effectiveness of different components of TaskPrompter.
Semseg mloU f
Parsing mloU
Salieney maxF f
Normal mĒrr
Boundary odsF 1
MTL Gain
Model
Am. t
TaskPrompter
79.00
67.00
85.05
13.47
73.50
0.15
CPrompt
78.10
65.99
84.88
13.58
72.10
-0,96
RW 78.55
66.02
84.22
13.60
71.80
-1.10
A.9 MORE ABLATION STUDIES OF DIFFERENT MODULES COMBINATIONS
To further examine the influence of different combinations of modules, we investigate another two combinations of modules for ablation study. In the first variant, we remove CPrompt, i.e., channel task prompt learning, from TaskPrompter with ViT-B backbone. the second variant, we remove RW, i.e., cross-task reweighting. The results results, we disable CPrompt and RW from the TaskPrompter respectively, the model performance shows clear decrease all five tasks in both cases, further confirming the effectiveness of channel task prompt learning and the cross-task reweighting strategies.
shown in Table 7. As
observe from the
A.10 MODEL EFFICIENCY WITH DIFFERENT NUMBERS OF TASKS
TaskPrompter is efficient compared with the previous transformer- based method with heavy decoder design (Ye & Xu, 2022). The reason is that Promper utilizes task prompts to help decode task- specific features by querying the image patch-tokens through tot-prromtttto-pacc affinities without
19

---

Published as a conference paper at ICLR 2023
Table 8: Study of the efficiency of Trompermpter with different numbers of tasks. The last column
reports the increases in the computation costs of the model from one task to five tasks.
Task
2
4
5
Increase
Param FLOPs
364M
373M
383M
392M
401M
+10.16%
389G
416G
443G
470G
497G
+27.76%
using an additional decoder. We show the number of parameters and FLOPs of Promper with different numbers of tasks on PASCAL-Contexti in Table 8. We observe that from 1 task to 5 tasks only increase 10. 16% parameters and 27.76% FLOPs, which demonstrates a strong scaling ability of our multi-task model.
20


Reviews:
[{"review_id": "Uj104oj0jKt", "reviewer": "Reviewer_GABz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-task architecture based on transformers for dense prediction tasks, named TaskPrompter. The design of TaskPromper was motivated by learning task-general, task-specific and cross-task relationships all into a unified learning module. To achieve this, the paper designs a spatial-channel prompting technique that jointly learns spatial and channel interactions through two self-attention layers. Additionally combined with other design strategies such as hierarchical prompting and cross-task reweigthing, the proposed architecture reaches a great performance on a range of multi-task benchmarks.", "summary_of_the_review": "The paper proposes a multi-task architecture TaskPrompter for dense pixel-wise scene understanding. The architecture itself is complicated in design, though achieving great performance in standard multi-task benchmarks. The study of the contribution of each design component is limited, and lack of motivation. I am currently holding a borderline score for this work, and will raise/lower the score depending on further rebuttal. ", "strength_and_weaknesses": "Strength.\n\n- The experiments were conducted showing consistently improved performance over a range of competitive baselines.\n- The visualisation of per-task affinity map looks interesting and intuitive, showing the effectiveness of the proposed design.\n\nWeakness.\n\n- Motivation and Design. The paper mentions in multiple places claiming that prior methods such as Cross-Stitch Netwoks, MTAN, and MTI-Net learn cross-task information through hand-craft structures, with task general, task shared representations were structured in each individual modules. However, from my perspective, the proposed method also learns these representations in different modules. Specifically, we may argue that the TaskPromper even further splits task-specific representation into spatial and channel modules, with cross-task information, are encoded in the decoder layers. As such, the proposed design does not strictly follow the motivation.\n- Prompt or Tricks? In Table 1, we can find each design component contributes some sense of performance improvements. Among which, the hierarchical prompting (which is a common design strategy in decoder-based MTL) contributes the most. This also makes me concerned that the overall performance is mainly because of the hierarchical design rather than the complicated prompting module.\n- Why channel? I did not follow the motivation of proposing channel prompts in the spatial-channel task prompt learning module design. Why learning interactions among different channels is important for MTL? Sure I can find the improved performance in Table 1, but this does not justify the design. Overall, I found learning channel interactions is very strange and unintuitive. It may be possible that the improved performance is mainly from the additional cross-attention layer.\n- Lack of Efficiency measurement. One important aspect of MTL architecture design is efficiency. As highlighted in multiple baseline methods such as MTI-Net, PAD-Net, it's important to show the size of network in terms of number of tasks. This allows us to understand the computation cost of the proposed design with the scaling of different number of tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GcVAoJfCfO", "reviewer": "Reviewer_Xzsy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces TaskPrompter - a multi-task transformer network for dense scene understanding. The framework has three parts - (1) Prompt Embedding, (2) Spatial-Channel Task Prompt Learning, and (3) Dense Spatial-Channel Task Prompt Decoding - with a strong focus on the latter two - which constitutes the main contributions of the paper. The effectiveness of the method and its components is thoroughly proven through multiple ablation studies. Experimental analysis is performed on popular benchmarks for multi-task learning - Pascal-Context and NYUD-v2 (the appendix also provides state-of-the-art results on Cityscapes-3D on two of the three tackled tasks).", "summary_of_the_review": "The paper has no significant flaws, the method is sound, and it has a good amount of novelty. The experimental setup is fair with convincing results over previous state-of-the-art methods. The reviewer has no grounds for rejecting this paper. ", "strength_and_weaknesses": "Strengths:\n* The topic is of great interest to the research community\n* The method has a high degree of novelty\n* Good ablation studies\n* Method agnostic of the different types of transformer architectures\n\nWeaknesses:\n* Minor typos throughout the manuscript - I recommend proof-reading once again\n* There is a minor error in the Appendix - Table 4 - the authors should bold the number from SETR-vitB (Zheng et al., 2021) since their performance is better than the one from TaskPrompter.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FbQSqcUvVcL", "reviewer": "Reviewer_NUwM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a transformer framework with spatial-channel multi-task prompting for both generic and task-specific feature learning. \n\nThe authors observe that both spatial and channel information are important for visual dense prediction tasks. Thus, the authors propose to use spatial-channel task prompts to learn their relationships with image tokens. This design could better train the transformer to capture useful pixel information for better feature learning.\n\nWhile the prompts are usually task-specific, the authors also found that different prompts can contribute to cross-task performance enhancement.\n\nExperiments on NYUD-V2 and Pascal-Context are strong.", "summary_of_the_review": "The authors proposed an interesting transformer architecture design that could be useful for dense prediction tasks. However, the architecture design is a lot more complex and not easy to reproduce.\n\nWhile the authors focused on multi-task learning and shown good signals in cross-task interaction learning, it is unclear how the proposed model performs compared to single-task SOTA on the well-known benchmarks as mentioned in Weaknesses.  Since the authors claimed their model has ability for the learning of task-generic and task-specific representations, additional comparisons might be added. The paper may need some clarifications or adjustments. I will update my rating based on the authors' feedback.", "strength_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow. The authors presented details of their module designs. \n2. The proposed idea is interesting and technically valid for dense scene prediction tasks. \n3. The authors conducted several analyses to verify the importance of the proposed components. \n4. The experiments are thorough and clear. \n\nWeaknesses:\n1. The paper only considered multi-task scene-based dense prediction datasets. I wonder whether the authors consider the generic benchmark, such as ADE20K, COCO-stuff and Cityscapes. I understand the authors applied it to multi-task problem to demonstrate the cross-task interaction/contribution, however, since the authors claimed it is able to learn task-specific representations, it will be more convincing to compare with those SOTA on ADE20K and others.\n2. In Table 2, adding hierarchical prompting seems the most effective one compared to others. The authors may want to discuss more about the reasons, but not just talk about the empirical observations in appendix.\n3. In Table 2, I wonder if the model in each row has the same model size? If the model sizes are different, how to verify the performance improvements are attributed to the proposed components or extra parameters?\n4. While the model design is good and interesting, the model seems complex and heavy compared to existing frameworks. Would it be possible to provide some discussions regarding model speed and complexity?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Da0kbbdqF90", "reviewer": "Reviewer_CiXq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "TaskPrompter presents a new framework for multi-task dense scene understanding. The framework assigns each task a  learnable token which allows task-specific and task-generic learning in both the encoder and decoder. The task tokens and image tokens interact through the attention layers in transformers.\n", "summary_of_the_review": "I recommend acceptance because of the novelty of the architecture: the lack of task-specific components and using prompting to represent different tasks. ", "strength_and_weaknesses": "Strength\n- The overall architecture. Developing an encoder and decoder under such settings is challenging and the author’s were able to include task-specific and task-generic representation learning. The architecture can be applied to many different tasks. \n- The idea of using prompts to represent the different tasks \n\nWeaknesses\n- The main weakness is that it is unclear from the experiments how important it is to have task-specific and task-generic features in both the encoder and decoder. This could be addressed through additional experiments where a task-specific only encoder is used. \n- The experimental section could use more details. For example, what is the ‘Task Prompter Baseline?’ Hierarchical Prompting seems to have the largest increase in performance. What happens when different combinations of modules are added? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "6rLmCUTBbWg", "reviewer": "Reviewer_y2MQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper address the problem of multi-task dense scene understanding, including semantic segmentation, human parsing etc. A transformer based method is proposed, and task-specific prompt tokens are designed to enable the transformer architecture to utilize its full capacity on all tasks. Authors modified the multi-head attention module by a spatial-channel task prompt learning module, and the decoding part is separately designed to produce predictions. Experiments are carried out on several scene understanding datasets to demonstrate its efficacy.", "summary_of_the_review": "The paper is interesting and novel, and I feel it may be helpful for future vision model design.", "strength_and_weaknesses": "### Strength\n* The proposed method can put most of the network's capacity on learning all tasks simultaneously; the cross-task representation interactions are also modeled.\n* The novelty is good. I feel it's quite new to involve the prompting concept into multi-task vision problems.\n* Experiments show that the model's performance is good.\n\n### Weaknesses\n* It seems that the model and experiments assume that supervision for all tasks are there. It should be easy to extend the work on dataset where each example may have supervision of a subset of the tasks. This may be a more practical and flexible setting, as during the inference time, user may only need a single task for an example.\n* The design of \"Spatial-Channel Task Prompt Learning Module\" is a bit complicated. I am wondering if it makes sense to make the model an encoder-decoder design, where the encoder encodes the images, and the decoder handles the prompts and cross-attend to the image embeddings. Also it would be necessary to analysis the computation overhead of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster