Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving
Authors: Xiwen Liang
Yangxin Wu
Jianhua Han
Hang Xu
Chunjing Xu
Xiaodan Liang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Aiming towards a holistic understanding of multiple downstream tasks simultaneously, there is a need for extracting features with better transferability. Though many latest self-supervised pre-training methods have achieved impressive performance on various vision tasks under the prevailing pretrain-finetune paradigm, their generalization capacity to multi-task learning scenarios is yet to be explored. In this paper, we extensively investigate the transfer performance of various types of self-supervised methods, e.g., MoCo and SimCLR, on three downstream tasks, including semantic segmentation, drivable area segmentation, and traffic object detection, on the large-scale driving dataset BDD100K. We surprisingly find that their performances are sub-optimal or even lag far behind the single-task baseline, which may be due to the distinctions of training objectives and architectural design lied in the pretrain-finetune paradigm. To overcome this dilemma as well as avoid redesigning the resource-intensive pre-training stage, we propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training, where the off-the-shelf pretrained models can be effectively adapted without increasing the training overhead. During the adapt stage, we utilize learnable multi-scale adapters to dynamically adjust the pretrained model weights supervised by multi-task objectives while leaving the pretrained knowledge untouched. Furthermore, we regard the vision-language pre-training model CLIP as a strong complement to the pretrain-adapt-finetune paradigm and propose a novel adapter named LV-Adapter, which incorporates language priors in the multi-task model via task-specific prompting and alignment between visual and textual features. Our experiments demonstrate that the adapt stage significantly improves the overall performance of those off-the-shelf pretrained models and the contextual features generated by LV-Adapter are of general benefits for downstream tasks.

Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving
Xiwen Liang", Yangxin Wu' Jianhua Han", Hang Xu², Chunjing Xu², Xiaodan Liang" I
"Shenzhnn Campus of Sun Yat-Sen University, Huawei Noah's Ark Lab {1 11ianxx2200ai11, wuyx290mai12, 1iangxd9@mail)..o ssy.. sysu. eyssu.uu...... edu. cn, (hanjianhua4, xu. xu.hang, xuchunjing)/huarae com com
Abstract
Aiming towards a holistic understanding of multiple downstream tasks simulta- neously, there is need for extracting features with better transferability. Though many latest self- self-super pre-training methods have achieved impressive per- formance on various vision tasks under the prevailing their generalization capacity multi-task tast learning scenarios is yet to be explored. In this paper, we extensively investigate the transfer performance of various types of self-supervised methods, e.g., MoCo and SimCLR, three downstream tasks, including semantic segmentation, drivable area segmentation, and traffic object detection, on the large-scale driving dataset BDD100K. We surprisingly find that their performances are sub-optimal or even lag far behind the single-task baseline, which may be due the distinctions of training objectives and architectural de- sign lied in the pretrain-finetury paradigm. To overcome this dilemma as well as avoid redesigning the inourceeintensive pre-training stage, we propose a sim- cattaainndda--ffin
finettn- paradigm,
ple yet effective ing, where the off-the-shelf pretrained models can be effectively adapted without increasing the training overhead. During the adapt stage, we utilize learnable multi-scale adapters to dynamically adjust the pretrained model weights super- vised by multi-task objectives while leaving the pretrained knowledge untouched. Furthermore, we regard the vision-lange langua-la pre-training model CLIP as a strong complement to the cettraain-ddat-fiieeteee paradigm and propose a novel adapter named LV-Adapter, which incorporates language priors in the multi-task model via task-specific prompting and alignment between visual and textual features. Our experiments demonstrate that the adapt stage significantly improves the overall performance of those off-ther the-sshef pretrained models and the contextual features generated by LV-Adapter are general benefits for downstream tasks.
paradigm for general multi-task train-
Introduction
Multi-task learning is a long-studied problem in computer vision and has become emerging paradigm in autonomous driving [19, 50, 18]. An autonomous vehicle may need to concurrently perform wide range of perception tasks, e.g., locating the pedestrians and cars, deciding road affordability for driving, as well as detecting the lanes to precipitate a driving action. The joint learning of multiple tasks regularizer that enforces the learning of generalizable representations [6, 19]. Apart from efficiency and simplicity, all-in-one architecture is argued to have the potential to enhance the robustness of the driving system by implicitly learning the synergy between heterogeneous tasks [19, 45, 40].
only reduce the training and inference time, but also act
Eqqull contribution.
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

[a] Input
(b) Object detection
(c) Semantic segmentation
(d) Drivable area segmentation
Figure
Our multi-task model takes (a) an RGB image as imput and tackles (b) traffic object
detection, (c) semantic segmentation, and (d) drivable area segmentation simultaneously.
In the pursuit of the merits of multi-tase learning, it is crucial to obtain universal features with great transferability. Recently, many concurrent self-supervised pre-training methods have shown great potential when transferred to different types of vision tasks [47, 49, 44] under the paradigm. Despite the impressive performance, their transferability to multi-task learning scenarios is yet to be explored. We argue that the heterogeneity of tasks will introduce a multitude of challenges to train unified model and it is often not the case that multi-task learning is of universal benefit. Firstly, the widely adopted pretrain- learning due to the misalignment of objectives between pre-training and fine-tuning [12, 16] since most supervised and self-supervised methods are highly specialized for specific type of objective or task [3, 4, 47, 49]. Secondly, the performance of multi-task learning relies on many non-triviae factors including model architectures, data augmentations, hyperparameters. convergence properties, etc [36, 39J. The specialized techniques catered for a specific type of architecture or task are largely not applicable to a universal architecture since they are prone to fail during generalization. Moreover, due to the labor-intensive process of data annotation, it is hard to collect complete annotations of different granularities for all tasks, which further complicates the situation.
paradigm may lead to degraded performance in multi-task
In this paper, we focus heterogeneous multi-task learning on partially labeled data (HMPL) under the realistic scenario of autonomous driving. We first delve into the representation learning stage of HMPL to reveal the performance degradation of different pre-training methods. We thoroughly exam- ine a wide range of pre-training methods including supervised pre-training (pretrained on ImageNet [33] ). Mussification--oientte methods (e.g.. SimCLR [3]. MoCo [15]), detection--rientte meth- ods (c.g., DetCo [47]).
-oriintt methods (e.g., DenseCL [44]), and vision-language pre-training methods (e.g.. CLIP (29]) on three fine- tree-graind tasks on the large-scale driving dataset BDD100K [52], i.e., object detection, semantic segmentation, and drivable area segmentation. Sur- prisingly, the performance of these methods varies greatly under the same training protocol, especially on the dense prediction task. For example, MoCov2 [4] only gets 12.5 mloU in semantic segmentation and regmentation-oriennt
models like DenseCL [44] and DetCo [47] perform unsatisfactorily on dense prediction tasks, which suggests that the misalignment lied in the pretrain-finetume paradigm can lead to prominent performance degradation and there exists much room for improvement.
Given that 'universal' pre-training is still unsolved and redesigning the resource-intensiv pre-training scheme comes with great computation overhead, we aim to develop general approach that can fully hamness the knowledge from the off-the-shelf pretrained models and make them amenable to multi-task scenarios via efficient adaptation. We draw inspiration from the recent progress of prompt- based learning in NLP [24, 55, 29, 10]. where the language model pretrained on massive amounts of raw text can be adapted to new scenarios with high efficiency by introducing hand-crafee or learnable prompts. Following this philosophy, we propose a simple yet effective peetrain-adapl- finetune paradigm for multi-task transfer learning as a substitute for the dominating pretrain- finetune paradigm in computer vision. Concretely, during the adapt stage, learnable multi-scale adapters with a small amount of parameters are tuned with frozen random initialized task-specific heads to dynamically adapt the knowledge from the pretrained models under the multi-task objectives. Albeit simple, we show that the adapt stage mitigates the gap between pre-trainnn pre-training and fine-tuning and significantly improves the overall performance across different pretrained models while being very effective and introducing no extra training overhead.
Apart from the supervised and self-supervised pre-training, Contrastive Language- Image Pre-training (CLIP) [29] manages to learn high-quality visual representation from an enormous amount noisy image-text pairs. CLIP achieves breakthrough performance in zero-shot image recognition, which indicates that it might have great potential for improving the generality and robustness of multi-task learning. To this end, we propose LV-Adapter to further excavate the linguistic knowledge from CLIP

---

Fineture stage
Adapt slage
Frezm paramuters
Pretoii
Self-sipereeed Prefrain
Vilion Language Pretrain
Encodar
Encoder
Image
Image Encoder
Label
Image
Contrstivve
Loss
Contractive
Loss
Impg Encodar
Text
Encader
Text
Finetune
Mutti Finctune
Language Adape Finetune
Head #1
Heoddd
Text
Encoder
Text
Head
Imaaee
Image Fnooder
Adapter
Briied A3
Image
Image
Adritter
Bllled
Billed
FrstrainsFindtnee
Prstrain-daap: Finetune
(c) Language-Gutded Feerain-Adaat--iieeuu
Figure 2: Comparisons of the conventional pretrain-fiettue paradigm and our proposed pretrain- adapt-finetune paradigm. The language- tnguage-guided preirain-culupt-fnee paradigm further incorporates language priors into multiple downstream tasks.
and enhance the visual representations in the multi-task scenarios. To maximize the correspondence between the dense features and the class-specific concepts, we first Iearn task-specific prompts in an end-to-end manner. Then we model the language-to-viion adaptation function based on the cross-attention mechanism [43] to incorporate language priors visual features.
paradigm significantly improves the overall performance of different off-the-shelf pre-training methods. For example, it obtains an absolute improvement by over 40% in semantic segmentation on MoCo-v1/v2, and DenseCL [15, 4, 44], and its superior performance does not rely on extra training costs or sophisticated architectural design. Furthermore, we showcase that our proposed LV-Adapter can serve as an effective complementary approach for multi-task learning. Especially, we experiment on three settings that correspond to different levels of annotation quantity and observe that LV-Adapter brings consistent performance gains on three heterogeneous tasks simultancously. We hope our work can shed light the heterogeneous multi-task learning by showing that effective adaptation is of great essence to mitigate the deficiency in the dominating pretrain- finetune paradigm.
We experimentally verify that our proposed pretrain-
2 Related Work
Multi-Task Learning Multi-task learning aims to improve efficiency and accuracy through shared information among multiple tasks. MultiNet [41] joints classification, detection and semantic segmentation via unified architecture. Eigen al. [8] address depth prediction, surface normal estimation, and semantic labeling. YOLOP [45] leverages CSPDarknet the backbone and adopts three specific decoders to solve object detection, drivable area segmentation, and lane detection respectively. Standley al. [36] and Fifty et al. [9] propose to identify proper task groupings for multi-task training instead of naively training all tasks together. In this paper, we focus on developing general and effective approaches for multi-task transfer learning in autonomous driving scenarios. We provide more discussions with several related works the Appendix.
Pre-traine Pre-training Methods A dominating paradigm in computer vision is pretrain a large scale of data, e.g., ImageNet [33], then finetune on target tasks with usually less training data. Recently, researchers are interested in learning visual representations without human supervision. We roughly divided them into three classes, namely, cuasifiication-oiinnte
dnlecction-orennted and segmentation- oriented methods. Classification-orientedd methods typically rely on contrastive learning and online clustering, e.g., SimCLR [3], MoCo [15, 4], and BYO [14]. Detection-orient methods like DetCo [47] are specially designed for object detection by conducting contrastive learning between the global image and local patches. Segmentation oriented methods like PixPro [49] further work on pixel-level correspondence benefit dense prediction downstream tasks. We show that these methods sub-optimal for multi-task learning, and we focus on improving the performance these off-the-shelf methods instead of redesigning the computation-intensiv pre-training stage.
Prompting I Learning Natural language prompting freezes large-scale pretrained language models and reformulates text input with example learnable prompts to bridge the gap between pre-training

---

Table 1: Comparisons of popular multi-task training methods and our proposed LV-Adapter under the
Disjoint-norme setting.
Method
mIoU (SS)
mIoU (DA)
mAP
AP50
AP75
Zeroing loss [46]
59.4
83.3
23.0
46.0
19.8
Uniform sampler [22]
59.9
83.2
24.3
47.0
21.8
Weighted sampler [22]
59.8
83.2
24.2
46.9
21.4
Round-robin [22]
60.7
83.1
24.2
47.0
21.5
Self-training [12]
60.3
83.1
24.9
48.1
22.2
LV-Adapter (Ours)
62.2
83.7
26.4
50.5
23.7
and model tuning efficiently. GPT-3 [1] first introduces in-context learning, prepending hard text prompts for every task before the input. Then many works based such discrete text prompts have emerged, i.c., designing prompts manually [37, 34] and searching trigger words [35, 11]. Furthermore, recent methods inject learnable soft prompts [21, 26, 53] into the embedding space of the model and achieve impressive performance. In the field of computer vision, CLIP [29] trains on millions of vision-language pairs and demonstrates effective zero-shot classification. CoOp [55], CPT [51], and CLIP-Adapter [10] further tunes fixed CLIP with soft prompts by few-shot supervisions.
3 Empirical Study
3.1 Heterogeneous Multi-Task Setup
We concentrate on three challenging heterogeneous tasks in autonomous driving, i.e., traffic object detection, semantic segmentation, and drivable area segmentation, on large-scale driving dataset BDDI00K [52]. We showcase the input and output of our model in Figure 1. We select the state-of- the-art detector and segmentation decoder head to build up our model. To be specifie, we follow the hard-parameter sharing scheme [6] where each task shares the same backbone and has its task-specific head. We outline our model architecture in Figure 2 and elaborate each component in the following.
Backbone & Neck. We adopt the classic ResNet [17] as backbone and FPN [23] as neck to generate multi-scale features. The FPN neck [23] constructs pyramidal features, namely, {P2, P3, Pa, Ps). which have strides of (4, 8, 16, 32} pixels w.r.t the input image and fixed 256 channels.
Segmentation Head. We implement the decoder for segmentation based MaskFormer [5]. MaskFormer formulates the segmentation task as mask classification problem and its head is composed of fully convolutional pixel decoder and transformer decoder module. The pixel decoder takes { P2, Ps. Pa. Ps} as input and gradually upsamples the features to produce high resolution per- per-pixel embeddings Epice- The transformer decoder module utilizes a fixed set of queries to attend to image features and produces mask embeddings maskk Then Episel and mask is multiplied together to generate prediction masks. We refer the reader to [5] for more details.
Detection Head. We implement our detector based on Sparse R-CNN [38]. The detection pipeline constructs fixed set of learnable proposal boxes (e.g., 300) which serve as region proposals and are fed into series of heads for prediction. The dynamic instance interactive head dynamically generates the weights of convolution filter that attends to the bins in the region proposals and predicts the location and category of object in a cascade manner.
We introduce three settings, i.e., the Disjoint-normal, Disjoint-balance and Full Full-setting, correspond- ing to different levels of annotation quantity in our experiments. For clarity, we defer the detailed descriptions of three settings to Section 5.1. For consistency, we conduct evaluations on the same validation set of BDD100K for different settings above. We mainly work under the Disjoint- setting for comparisons between the performance of different pre-training methods as well as ablation studies. And we further verify the efficacy of our proposed methods on the Disjoint-baland and the Full-setting in Section 5.2.

---

Table 2: Comparisons of different paradigms under the Disjoint-normal setting with ResNet-50 backbone. Orange color indicates the results our proposed sactin--dattt-fieeeeee paradigm, while
others are results of conventional finettmm paradigm.
Semantic Seg.
Drivable Seg.
Object Detection
Type
Model
mloU
pACC
mIoU
pACC
mAP
AP50
AP75
17.8
48.6
70.8
92.0
25.8
49.5
23.1
MoCo-v1 [15]
59.2
93.2
83.6
96.9
25.9
50.0
23.0
10.3
19.8
73.4
93.5
23.2
MoCo-v2 [4]
61.2
93.4
83.8
96.9
26.1
50.4
23.4
Clussfiiation-orientee
60.3
93.3
83.5
96.8
23.4
48.9
22.5
SimCLR [3]
60.1
93.2
83.5
96.8
25.2
48.9
22.3
45.9
71.1
82.0
96.5
25.6
49.1
23.0
SwAV [2]
61.1
93.3
83.1
96.7
25.6
49.3
23.1
59.2
90.2
75.6
93.9
25.9
49.8
23.4
BYOL [14]
61.7
93.4
83.5
96.8
25.7
49.4
23.1
38.1
58.5
83.2
96.7
25.9
49.7
22.9
Detection-oriented
DetCo [47]
61.0
93.4
83.7
96.9
26.2
50.3
23.3
20.0
40.0
73.7
93.7
26.1
50.3
23.5
Segmentation-orientee
DenseCL [44]
60.7
93.3
83.9
96.9
26.3
50.3
23.7
54.5
91.1
74.1
93.1
26.5
50.7
23.8
lasion-language
CLIP [29]
61.0
93.2
83.4
96.8
26.3
50.5
23.5
3.2 Multi-Tass Learning Baselines
Self-training Based on the available annotations for each task, trained three single-task teacher models on the labeled data and use them to generate pseudo labels on unlabeled data following [48, 12]. For single-task teacher models, adopt the same architecture and training schedules of Sparse R-CNN [38] and MaskFormer [5]. For object detection, we use a score threshold of 0.5 to select pseudo box labels. For segmentation,
the score threshold of 0.3 select segmentation masks. Pixels with lower scores are assigned with the ignore label. Thereafter, every image has annotations for all tasks and we train a multi-task student model with weighted sum of all objectives for each task:
Ciotal = Oldet Ceet + asemm Ceem + C'driv Caries
(1)
where Latet. fisem Lariv are losses for object detection, semantic segmentation, and drivable area segmentation, respectively. We include more details in Section 5.1.
Zeroing loss Some works [19, 46] simply zero losses for a particular task the input image does not have the corresponding annotation.
Batch-level Round-Robil
this scheme [52, 22], each batch consists of samples that have the same type of annotations for a task and different tasks are repeated in fixed order during training.
Task Scheduling This is stochastic schedule that the task for each SGD update step is sampled from a distribution [22]. We mainly consider uniform task sampling schedule that sampled from a uniform distribution, and weighted task sampling schedule where the sampling weight for task is proportional to the number of labeled images and training epochs for this task.
Performance Comparison As can be seen in Table I and Table 3, these methods surpass single-task teacher model by a clear margin, which indicates that the model benefits from multi-task learning. Among these baselines, self-training achieves satisfactory performance while being conceptually simple, thus we adopt it for the rest experiments this paper.
We conduct grid search in the range of [0.1, 1.0] with a step size of 0.1 to find the optimal loss weights setting and Oldet, Oteem, Adrii 1.0, 0.7, 0.7, respectively. We fix the loss weights throughout the paper.

---

LVAdapter
g Text Encoder
person car
Pseudo labeling
GT
Teacher
Pseudo labels
Vi
vn
labels
Teacher #2
key & value
road building
Teacher #3
Transformer
direct alternative
Decoder
Vi
Head #1
query
#2
Image Encoder
Head #3
Figure 3: An overview of our proposed model. We first train three specialized teacher on labeled data generate pseudo labels for each task. The multi-task model is then trained on both ground-truth and pseudo labels under the language-guice guided pretrain-adape saapt-finetune paradigm.
3.3 Pretrain-finetune for Multi-Task Learning
In this section, we adopt the identical training protocol and architecture to extensively investigate the performances of different types of pre-training methods, including task-oriented methods and vision-language pre-training, when transferred to the multi-task learning scenario. We revisit the technical details of these different schemes in the Appendix. We report the performance of the aforementioned methods in Table 2, and come to the following observations:
Only SimCLR achieves decent performance on three tasks and many methods encounter substantial degradation pixel-Ievel segmentation tasks. For semantic segmentation, MoCo-v1/v2 and DenseCL achieve the worst mloU (degrade by 66%-833 83% v.s. ImageNet- pretrained). For drivable area segmentation, MoCo-v1/v2, DenseCL, and CLIP have the worst performance (degrade by nearly 12% v.s. ImageNet-pretrained
The pre-training paradigm does not seem to have an explicit correlation with the downstream performance, and even the task-oriented design in pre-training does not guarantee the transfer performance on the corresponding task type. For example, the segmentation- -orienete DenseCL achieves sub-optimal performance
segmentation tasks.
Effective Adaptation for Multi-Task Learning
4.1 Pretrain, Adapt, then Finetune
We attribute the degraded transferring performance of the state-of-the-art pre-training methods to two critical factors, i.e., optimization gap and architectural gap. Firstly, the high specialization of the pre- pre-training methods and the heterogeneity of downstream tasks result in the optimization gap in the classic evetrain-fiettun paradigm. To be specifie, the objective for pre-training type of contrastive loss (e.g., InfoNCE [42]) while the fine-tuning stage is supervised by weighted sum of several task-specific losses. Secondly, we notice that many models in Table 2 are pretrained with convolutional head, e.g., Fast R-CNN detector [13] or convolution- based projection head. In contrast, we adopt the transformer- based MaskFormer head for segmentation tasks. The distinctions between convolution and transformer lie in the inductive bias (local global) and the internal representation structure [30] and we conjecture this plays a non-negligible role in transferring, We further experimentally verify this point in Appendix.
usually
The limited transfer performance presented Table 2 indicates that the 'universal' pre-training remains unsolved. Instead of redesigning another pre-training scheme that brings vast computation overhead, we aim to efficiently reuse the intact knowledge of these off-the-shelf pre-trained models with minimal modifications to the architecture. We draw inspiration from the recent progress of prompt-based learning, where prompt tuning emerges as a new alternative to fine-tuning. For example, P- P-Tuning v2 [25] matches the fine-tuning performance by only tuning learnable prompts

---

while freezing the large-scale language model. Following this philosophy, we propose a simple yet effective pretrain-aday adapt-finetune paradigm to mitigate the gap between the pre-training and fine-tuning stage. During the adaptation stage, we are given the pretrained model weights (e.g., ResNet-50) inherited from the pre-training stage. We aim to transform the model weights via a small amount of learnable parameters to adapt the knowledge of the pretrained weights towards multi-task scenarios. To this end, we freeze the parameters of the random initialized task-specific heads and the backbone, and tune the parameters of Feature Pyramid Network (FPN) supervised by the multi-task loss function in Equation I. During fine-tuning, all parameters are activated and updated via gradient descent. We compare different schemes for multi-task learning in Figure 2. This adaptation stage characterizes several critical design choices: (1) This stage bridges the gap between pre- pre-training and fine-tuning by including the pretrained weights and multi-task objectives simultaneously, and the frozen backbone prevents the pretrained weights from being spoiled before the fine-tuning stage. (2) Compared to the single-layer prompt/adapter in P-Tuning [20] and CLIP- Adapter [10], the FPN acts as a multi-scale adapter that is endowed with greater capacity and provides semantically stronger features for ine-ggained downstream tasks. (3) Taking FPN the adapter paradigm
means that the model architecture for the pretrain-fineture and preirain-addpt-ffiette is identical and effective adaptation can be achieved without any modification the architecture. We have experimented with more sophisticated adapters (e.g., scale-aware adapters, lightweight transformers) and observed marginally better results. Hence, we opt for the original architecture above for simplicity. We experimentally verify that our pvetrain-ddapt adapt-finetune paradigm significantly improves the performance and stability of different kinds of pre-training methods (c.f. Section 5.2) without increasing the total training costs (c.f. Section 5.3).
4.2 Language-to- -oniigg Adapter
From the perspective of transfer learning, CLIP can serve as a complementary scheme enhance the pretrain--alept-ffeetuee paradigm, since it can comprehend the concepts in natural language. Basically, the CLIP model excels in aligning the visual and language embeddings and some works [54, 31] claim that the textual features generated by CLIP have meaningful correspondence to the semantic regions in an image. We note that the result of CLIP in Table 2 only reuses the weights of its image encoder and discards the text encoder. Therefore, we take a step further to explicitly exploit the knowledge in the full CLIP model. We pursue to underpin the compatibility between the semantic concepts of each task and the image features and generate semantically stronger contexts for downstream tasks. The resulting model, named LV-Adapter outlined in Figure 3, and we elaborate on each ingredient in the followings.
The CLIP model adopts ResNet [17] and BERT [7] as the encoder for image and text, respectively. Formally, we denote the last output feature maps from ResNet stages as {x Attentional pooling and L2 normalization are applied to X5 E HCCs to produce the global image feature I, E BlxCs for zero-shot inference, which can be formulated as follows:
X:] = L2_ S22_NOMMMMHSAAGAPP
o xx)),
(2)
where GAP(.) denotes global average pooling, MHSA(.) denotes multi-head self-attention [43], and @ denotes concatenation operation. To extract textual features of each class, class specifcc prompts are constructed via a generator function and are fed into the text encoder. We denote the normalized output features for N classes as T. E RXXC which can be obtained as follow:
T = L2 (ORM
where Gen(-) is the generator function of class class-specific prompts, TE is the text encoder, and (n,).. is the class name embeddings. Though T, and I, are well aligned in the image-text contrastive pre- training, they do not preserve any spatial details and are not readily applicable for downstream vision tasks. In contrast, the multi-level features output by FPN are semantically rich in the spatial dimension but are not directly aligned in the pre-training stage. Therefore, we pursue to learn task-specific prompts and propose a Language-to Vision adapter enhance the pixel-class correspondence.
Learning Task-Specific Prompts. The original generator function of CLIP outputs hand-crafted prompts using the predefined template, e.g., "a photo of [CLS].". However, the performance is highly sensitive to the form of the template [29, 555, which may be sub-optimal when transferring to heterogeneous downstream tasks. Hence, we follow CoOp [55] to use learnable textual contexts in
7

---

Table 3: Results of single-task baselines and multi-task models with ResNet-50 backbone. SS and DA means semantic segmentation and drivable area segmentation. indicates inapplicable.
Setting
Method
mIoU (SS)
mloU (DA)
mAP
AP50
AP75
MaskFormer [
57.1
MaskFormer [5]
83.9
Full
Sparse R-CNN [38]
29.4
53.8
26.4
Self- Self-training [12]
61.8
84.4
30.1
56.6
27.6
LV-Adapter (Ours)
63.1
84.9
31.1
58.2
28.4
MaskFormer [5]
57.1
MaskFormer [5]
78.1
Disjoint-balancs
Sparse R-CNN [38]
18.6
37.8
15.6
Self-training [12]
59.4
80.3
22.4
44.1
19.6
LV-Adapter (Ours)
61.8
80.6
24.6
47.4
21.9
MaskFormer [5]
57.1
MaskFormer [5]
82.0
Disjoint-normal
Sparse R-CNN [38]
20.9
41.9
17.8
Self-training [12]
60.3
83.1
24.9
48.1
22.2
LV-Adapter (Ours)
62.2
83.7
26.4
50.5
23.7
prompting for each task. In spite of the inconsistent output formats of the three tasks (4-D box v.s. dense output), they both need to determine the category of boxes or pixels, and we incorporate the class names of each task in prompting. The task-specific prompting can be formulated as follows:
T L.2 NORM n.l);
where n, refers to the embeddings of the class name belonging to a specific task, and v² is the task-specific learnable contexts and the superscript indicates the task type.
Enhancing Pixel-Class Correspondence. To align the textural features and the dense FPN features, we propose Language-to- Vision Adapter incorporate the language priors into the visual features. Formally, we denote the last feature map of P3 as E IR H,W; XC and we aim to learn an adapter function AL-V generate language- awaree contexts for downstream tasks. We utilize the cross- attention mechanism in Transformer decoder [43] for Language-to to-iiaggoo--Voiioonnoo adaptation:
AL- 25) Trans TransDeecderr(y n5, k Fa, v= Te),
where the g, k stands for query, key, and value. A single linear fully connected layer is used to adjust the channel number of Te and omit in Equation for simplicity. We denote the output of Equation as 2; E IR McWx After adaptation, we simply substitute 2; with žù to inject linguistic knowledge into the high-level features of FPN and leave the task-specific head design unchanged. We follow the same paradigm described Section 4. when incorporating LV-Adapter: only the adapter module (i.e., LV-Adapter and FPN) is tuned during the adaptation stage. Moreover, freeze the weights of the CLIP text encoder throughout the paper to preserve the language priors. We verify the efficacy of our LV-Adapter and disentangle the effect of different modules ablation in Section 5.3.
Experiments
5.1 Experiment Settings
In BDD100K, 70k training images are labeled for both object detection and drivable area segmentation. and only 7k training images are labeled for semantic segmentation. These two sets are not disjoint and about 3k images have complete annotations for all three tasks. As described in [12], data annotation is one of the biggest challenges of training a multi-task model. In real-world scenarios, it is unrealistic all input images, especially the traffic scene where
to obtain complete types of annotations fine-grained annotations are required. In this paper, are interested in the performance of multi-task learning with different quantities of annotations. Thus, we mainly consider three scenarios that correspond three different levels of annotation scarcity, namely. Disjoint-normal Disjoint-balance

---

and Full setting. We mainly work under the Disjoint-normin setting where the annotations of each task are disjoint, and the number of labeled images for each task is in decreasing order as follows: drivable area segmentation (20k), object detection (10k), semantic segmentation (7k). We provide the details of these three settings in the Appendix.
Implementation Details. The training epoch is fixed as 36, syncBN [28] is on, the learning rate at the fine-tuning stage is 2.5 x 10-5, and weight decay is 1 x 10-4, The image scale is 1280 x (720, 600), No other data augmentation is used. We adopt the AdamW [27] optimizer. The warmup iteration is 1000 and the warmup factor is 0.01. For LV-Adapter, the learnable prompts are prepended to the class, and the length of prompts is 16. During the adaptation stage, the learning rate is set to 2.5 10 0- The layer of transformer decoder in Equation 5 is 3. All experiments are conducted on servers with 8 Nvidia V100 GPU (32GB) cards and Intel Xeon Platinum 8 3168 CPU (2.70GHz).
5.2 Main Results
We present the results of our proposed paradigm in Table 2. The adapt and finetune epochs of the proposed paradigm in Table 2 are set as 6 and 30 respectively. As can be seen, with the conventional pretrain-finetune paradigm, the performances of the state-of-the-art self-supervised methods are unstable, especially in semantic segmentation and drivable area segmentation. For example, MoCo-v2 only achieves 10.3 mloU on semantic segmentation. In contrast, under the veerriin-ddapt-imeteee paradigm, MoCo-v2 improves by a large margin in semantic segmentation (+50.9 in mloU). Notably, MoCo-v1/v2 and DenseCL obtain an absolute improvement by over 40% mloU in semantic segmentation. The results demonstrate that the preirain-adapt-frree significantly improves the overall performance across different types of pretrained models. We also implement experiments with popular convolutional heads, i..., Faster R-CNN [32] and Semantic FCN [18], as in Appendix D.2. Results show that architecture is a core reason for the degradation. Moreover, we further showcase the efficiency of this paradigm in Section 5.3. We compare the results of our L.V-Adapter single-task baselines and the self-training baseline using the ImageNet pretrained model under three settings in Table 3. It demonstrates that LV-Adapter performs better on all metrics. Therefore, conclude that the language guided perrrii-addpt-iinttta paradigm can effectively reduce the gap between the pre-training and fine-tuning stage by incorporating the linguistic knowledge visual features.
paradigm
5.3 Ablation Study
Module design. We compare the performance of different module designs in Table 4 using the pre-trained weights from CLIP. Row #1 refers to the performance of self-training with the pretrain- finetune paradigm using the pretrained weights of the CLIP image encoder. Row #2-#5 refer the paradigm and row #3-#5 reuse the pretrained weights serruii--datt-finneeee
performance of the preirain-adapt-Pii of CLIP text encoder. The adapt and finetune epochs for the fixed as 1 and 5. The Prompt column refers to naive prompting without alignment between visual and textual features [31]: we compute class-speeific textual features by prompting the CLIP text encoder (Equation 4) and compute the class activation maps by computing dot product between the textual features and 25 after normalization, and the output maps are concatenated to Z5 and a 1 1x 1 convolution layer is used to reduce the number of channel. The V2L column refers to the model module in [31], where the image context used to prompt the language model and class activation maps are concatenated back to 25. In comparison, our Language-to- Vision Adapter (L2V column) directly incorporates linguistic knowledge into visual features and does not rely on dense activation maps and is thus more lightweight. As can be seen, when retrieving knowledge from pretrained text encoder via continuous prompt, the model improves in semantic segmentation Adapter to update the text features,
paradigm are
by 0.3 mloU (#2 vs #3). When adopt the the performance degrades slightly in semantic segmentation (#3 vs #4). In contrast, our proposed LV-Adapter can better extract prior knowledge from CLIP and achieve the best results (+0.4 and +0.2 in mloU in semantic segmentation and drivable area segmentation respectively).
Number of epochs of the adapt and finetune stage. We compare different configurations of adapt and fineuune epochs in Table When adapting with fewer epochs and fixing full epochs as 36, the performance improves gradually and the best configuration is epoch for adapp and 35 epochs for finetune. Since we freeze the backbone and randomly initialized heads during the adapt stage, the number of trainable parameters is moderate. We can also see that with fixed finetune epochs, more
9

---

Table 4: Ablation study of the components of our proposed LV-Adapter.
Table 5: Comparison of different configurations of adapt and finetume epochs.
#
Prompt
V2L
L2V
mloU (SS)
mloU (DA)
Adapt
Finetune
mloU (SS)
mloU (DA)
mAP
x
x
x
54.5
74.1
12
24
59.5
82.9
25.8
2
X
x
X
61.5
83.5
6
30
61.0
83.4
26.3
X
6188
83.5
1
35
61.5
83.5
26.4
4
/
X
61.3
83.6
6
35
61.3
83.5
26.5
12
35
61.9
83.5
26.6
5
/
V
62.2
83.7
adapt epochs have little impact on the results. We hypothesize that fewer adapt stage is enough for the training paradigm since there are few training parameters (only the adapter) during the adapt stage. For fair comparison with the baseline whose full epochs are 36, we fix the adapt and finetune epochs and 35 respectively.
Conclusion
In this paper, present the first effort revealing the degradation of state-of-the-art self-supervised models under multi-task learning scenario autonomous driving. reduce the gap between the pre-training and fine-tuning stage, we propose a simple yet highly efficient pattrrain-alat-ffneeteee paradigm, which boosts the performances of different self-supervised pretrained models by large margin without increasing the overall training overhead. We further utilize the vision-language pre-training model CLIP as a complement to our proposed paradigm and propose LV-Adapter, which incorporates linguistic knowledge into visual features via learning task-specific prompts in novel fashion. Extensive experiments showcase that (1) the adapt stage is the key to mitigate the deficiency of the dominating evetrain-finetuee paradigm in multi-task learning, and (2) the language priors encoded by CLIP are of general benefits for multiple downstream tasks.
Acknowledgements
We gratefully acknowledge the support of MindSpore', CANN (Computer Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry,
A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler. J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, Chess, J. Clark, C. Berner,
S. McCandlish, Radford. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances Neural Information Processing Systems (NeurlPS), volume 33, pages 1877-1901, 2020.
M. Caron, Misra, Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural Information Processing Systems (NeurIPS), 33:9912-9924, 2020.
[3] T. Chen, Kornblith, Norouzi, and G. Hinton. simple framework contrastive learning visual representations. In International Conference on Machine Learning (ICML), pages 1597-1607, 2020.
X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv:2003.0429) 2020.
[5] B. Cheng. A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances Neural Information Processing Systems (NeurIPS). 34, 2021.
[6] M. Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint 27X99:000.
2020.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 1810v:1810.04800. 2018.
Phttps:/ ://ww www.w.mineeeeee mindspore.com cn/
10

---

[8] D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2650-2658, 2015.
[9] C. Fifty, F. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems (NewrlPS), 34, 2021.
[10] P. Gao, S. Geng, R. Zhang, T. Ma. R. Fang. Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision- language models with feature adapters. arXiv preprint 1rXiv:2110004544 2021.
[11] T. Gao, A. Fisch, and D. Chen. Making pre-trained language models better few-shot leamners. In Association for Computational Linguistics (ACL), 2021.
[12] G. Ghiasi, B. Zoph, E. D. Cubuk, Q. V. Le, and T.-Y. Lin. Multi-task self-training for learning general representations, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8856 8865, 2021.
[13] R. B. Girshick. Fast r-cnn. pages 1440 14400-148,,00 2015.
[14] J.-B. Grill, Strub, F. Altché, C. Talle, P. Richemond, E. Buchatskaya, C. Doersch, Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurlPS), 13:21271-21281 2020.
[15] K. Hc, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729 7729-9738, 2020.
[16] K. He, R. B. Girshick, and P. Dollár. Rethinking imagenet pre-training, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4917-4926. 2019.
[17] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[18] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár. Panoptic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9404 9413, 2019.
[19] I. Kokkinos. Ubernet: Training a 'universal' convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6129-6138, 2017.
[20] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045-3059, 2021.
[21] X. L. and P. Liang. Prefix teeix-tuning: Optimizing continuous prompts for generation. arXiv preprint M0Xiv:2001.001900 2021.
[22] V. Likhosherstov, A. Amab, Choromanski, M. Lucie, Y. Tay, A. Weller, and M. Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint arXiv:2111.129) 2021.
[23] T.-Y. Lin, P. Dollár, R. Girshick, K. He. B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 936-944,72 2017.
[24] P. Liu, W. Yuan, J. Fu, 7. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: systematic survey prompting methods in natural language processing, arXiv preprint Ni.::2117.11586 2021.
[25] X. Liu, K. Ji, Y. Fu, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2 2021.
[26] X. Liu, Y. Zheng, Z. Du. M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too. arXiv preprint arXiv:2103. 10385, 2021.
[27] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 171.0511,,, 2017.
[28] C. Peng. T. Xiao, Z. Li, Y. Jiang. X. Zhang, K. Jia, G. Yu, and Sun. Megdet: A large mini- object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6181-6189, 2018

---

[29] A. Radford. J. W. Kim, C. Hallacy. A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 9748-8763, 2021.
[30] M. Raghu, T. Unterthiner, S. Korbllith, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems (NeurlPS), 34, 2021.
[31] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang. J. Zhou, and J. Lu, Denseclip: Language-guided dense prediction with context-aware prompting. arXiv preprint arXiv:2112.01518, 2021.
[32] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-can: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems (NeurlPS), 28, 2015.
[33] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang. A. Karpathy, A. Khosla, M. S. Bersstein, A. C. Berg, and L. Fei-Fci. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):21 (1:3))211-2222 2015.
[34] T. Schick and H. Schlitze. It's not just size that matters: Small language models are also few-shot learerr.. In Proceedings of NAACL, 2021.
[35] T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
[36] T. S. Standley, A. R. Zamir, D. Chen, L. J. Guibas. J. Malik, and S. Savarese. Which tasks should be Icarned together in multi-task learning? In International Conference on Machine Learning (ICML), pages 9120-9132, 2020.
[37] K.KK. Sun and C.-I. Lai. Conditioned natural language generation using only unconditioned language model: An exploration. arXiv preprint arXiv:2011.07347, 2020.
[38] P. Sun, R. Zhang, Y. Jiang. T. Kong, C. Xu, Zhan, M. Tomizuka. Li, Z. Yuan, Wang, and Luo. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IFFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14454-14463, 2021.
[39] X. Sun, R. Panda, and R. S. Feris. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 33:8728- 8740, 2020.
[40] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties neural networks. arXiv preprint arXiv: 219.,6999..... 2013.
[41] M. Teichmann, M. Weber, M. Zoellner, R. Cipolla, and R. Urtasun. Multinet: Real-time joint semantic reasoning for autonomous driving. In IEEE Intelligent Vehicles Symposiun (IV), pages 1013-1020, 2018.
[42] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: arXiv:1807.033.. 807.03748, 2018.
[43] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkorcit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[44] X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li. Dense contrastive learning for self-supervised visual pre-training. Proceedingy of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 3023-3032, 2021.
[45] D. Wu, M. Liao, W. Zhang, and X. Wang. Yolop: You only look once for panoptic driving perception. arXiv preprint arXiv: 108. /250, 2021.
[46] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for seene understanding. In European Conference on Computer Vision (ECCV), pages 418 434, 2018.
[47] E. Xie, Ding, W. Wang, X. Zhan, H. Xu, Li, and P. Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8392-8401, 2021.
[48] Q. Xie, F. H. Hovy, M.-T. Luong, and Q. V. Le. Self-training with noisy student improves imagenet classification. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2020.
[49] Z. Xie, Y. Lin, Z. Zhang. Y. Cao, S. Lin, and H. Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16679- 16688, 2021.
12

---

[50] Z. Yang. Y. Zhang. J. Yu. J. Cai, and J. Luo. End-to-end multi-modal multi-task vehicle control for self-driving with visual perceptions. 24th International Conference Pattern Recognition (ICPR), pages 2289-2294, 2018.
[51] Y. Yao, A. Zhang, Zhang, Z. Liu, T.-S. Chua, and M. Sun. Cpt: Colortul prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.1177 2021.
[52] F. Yu, H. Chen, X. Wang. Xian, Chen, F. Liu, V. Madhavan, and T. Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (336-2645, 2020.
[53] Z. Zhong, Friedman, and D. Chen, Factual probing [mask]] Learning vs. learning recall. arXiv preprint arXiv:2104.05224 2021.
[54] Zhou. Loy, Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXiv:2112.0107, 2021.
[S5] Zhou, Yang, C. Loy, and Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:219.01I 133,0119 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section Abstract and Introduction.
(b) Did you descrihe the limitations of your work? [Yes] See Limitations in Appendix G.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Potential Negative Social Impact in Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] Code will be released.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section Experiment Settings and Appendix A.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] Our experiments are quite stable with multiple runs. All results are observed via fixed seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Implementation Details in Section Experiment Setting.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section Reference.
(b) Did you mention the license of the assets? [No] The dataset used in the paper publicly released.
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes] The dataset used in the paper is publicly released.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The dataset used in the paper does not contain personally identifiable information offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "oy92vu6mdaV", "reviewer": "Reviewer_72BH", "summary": "This paper is the first to study the use of self-supervised pretrained models in multi-task learning scenarios under autonomous driving settings. It points out that the conventional pretrian-finetune scheme is not effective, and instead, it proposes the pretrain-adapt-finetune paradigm. The adaption is conducted on the FPN, and language-to-vision adaptation with pretrained language model further improve the performance.", "questions": "Please see the Weaknesses section.", "limitations": "No additional limitations or potential negative societal impact.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Multi-task training and inference is a critical research direction for the autonomous driving industry due to its multi-task nature of daily operations. I am pleased to see a pioneering effort on this topic.\n2. The proposed pretrain-adapt-finetune scheme is effective. The discussion on using FPN as the adaption module is especially convincing.\n3. The inclusion of a language model in assisting learning of downstream tasks is interesting.\n\nWeaknesses:\n1. Line 173-174: it is stated “these methods surpass single-task teacher model by a clear margin”, which rows are for single-task teacher models?\n2. Although FPN seems to be a good choice for adaptation, it may be more convincing to try other parts and combinations involving the backbone and the head.\n3. It is interesting to see that L2V works, and it is helpful to discuss more in depth that why L2V works better than V2L, probably from the angle that what role the language model plays.\n4. On learning task-specific prompts, what are the typical learned prompts for each of the tasks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O5i0RMbvHPY", "reviewer": "Reviewer_kNyY", "summary": "With the success of self-supervised learning, this paper attempts to make better use of the pre-trained model under the multi-task learning scenario in autonomous driving. \nThis paper proposed a pretrain-adapt-finetune paradigm, which boosts and performance of different self-supervised pre-trained models.\nIt further utilizes the vision-language pre-training model CLIP and propose LV-Adapter. The experiments show that the language priors encoded by CLIP can have some benefits for multiple downstream tasks.\n\n", "questions": "1. It would be better to compare your methods with previous multi-task learning methods, maybe Table 1 should have your approach for comparison.\n2. It would be better for Table 2 to have a supervised baseline. As you mention in Appendix D.2, architecture is one of the core reasons for the degradation. From the results, we can see the degradation usually happens in semantic seg, the reason is the transformer-based segmentation head. Therefore, the effectiveness of ‘adapt’ is not well reflected. \n3. Table 5 shows that fewer adapt stage (1) can have the best performance. I wonder the results of longer adapt stage with fixed fine-tune stage.\n4. If the annotations of different tasks are not aligned, I wonder the realization of co-training process. This means using the pre-trained model to make pseudo labels and then training together? Does the gain come from training together or from more data utilization?\n5. Why the Language prior can help the model? Is it useful only under the condition of multi task learning, or can it also assist the learning of each sub task alone?\n", "limitations": "The author discuss them in the appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well-organized and the studied problems have practical value.\nThe author tried many pre-trained methods, the analysis and experiment are sufficient to verify the proposed viewpoint.\n\nWeaknesses:\nThe novelty is moderate, both the pretrain-adapt-finetune paradigm and using language priors is not new in this field, but it is worthy to explore them under the multi-task learning scenario in autonomous driving.\nThere are some confusing points in the experiments, I listed them in the question part.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4haaf0OHQYw", "reviewer": "Reviewer_5cHS", "summary": "This paper proposes a pretrain-adapt-finetune paradigm to train a multi-task network from a well-pretrained network which improves the overall performance compared to the conventional pretrain-finetune paradigm. During the adaption, authors propose to learn a multi-scale adapter. To better use textual modeling in the recent CLIP model, authors further proposes a LV-adapter via prompting learning to incorporate the language priors. The paper experiments on 8 different pretrained models to show their proposed diagram learns a better MTL model from the given pretrained network.", "questions": "1. Could you elaborate more on the experiment settings of Table 4 and Table 5? How do the experiments in Table 4 and 5 relate to those in Table 2?\n\n2. Have you tried with other datasets or other types of tasks? In this paper, experiments mainly focus on the segmentation and detection tasks. What the performance would be when we have other types of tasks, e.g. classification?\n\n3. In Table 3, authors show LV-Adapter is better than self-training. In my understanding, self-training is somehow orthogonal to LV-adapter. Specifically, we can use self-training to generate pseudo labels and train LV-adapter with the available gt labels and pseudo-labels. In this case, what is performance of the LV-Adapter with self-training? Will LV-Adapter keep improving with high-quality pseudo labels?", "limitations": "They have included in the supplementary", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The pretrain-adapt-finetune diagram is shown effective with 8 different SoTA pretrained networks with various pretraining goals.\n\n2. It is novel to learn a multi-task model for vision tasks from CLIP via prompt learning.\n\n3. The adaption stage is cheap based on the ablation studies.\n\nWeaknesses:\n\n1. The experiment settings for Table 4 and Table 5 are not clearly stated in the paper. For instance, which pretrained model is used in for Table 5? What tasks are considered in Table 4? Why are the best results of Table 4 and 5 not matching the results in Table 2? \n\n2. Experiments are only limited to BDD100k dataset\n\n3. There is no ablation studies related to the different designs of multi-scale adapter. Can they be replaced with adapters, such as residual adapters?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept