Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is the first to study the use of self-supervised pretrained models in multi-task learning scenarios under autonomous driving settings. It points out that the conventional pretrian-finetune scheme is not effective, and instead, it proposes the pretrain-adapt-finetune paradigm. The adaption is conducted on the FPN, and language-to-vision adaptation with pretrained language model further improve the performance.
strengths_and_weaknesses: Strengths:
1. Multi-task training and inference is a critical research direction for the autonomous driving industry due to its multi-task nature of daily operations. I am pleased to see a pioneering effort on this topic.
2. The proposed pretrain-adapt-finetune scheme is effective. The discussion on using FPN as the adaption module is especially convincing.
3. The inclusion of a language model in assisting learning of downstream tasks is interesting.

Weaknesses:
1. Line 173-174: it is stated “these methods surpass single-task teacher model by a clear margin”, which rows are for single-task teacher models?
2. Although FPN seems to be a good choice for adaptation, it may be more convincing to try other parts and combinations involving the backbone and the head.
3. It is interesting to see that L2V works, and it is helpful to discuss more in depth that why L2V works better than V2L, probably from the angle that what role the language model plays.
4. On learning task-specific prompts, what are the typical learned prompts for each of the tasks?
questions: Please see the Weaknesses section.
limitations: No additional limitations or potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: With the success of self-supervised learning, this paper attempts to make better use of the pre-trained model under the multi-task learning scenario in autonomous driving. 
This paper proposed a pretrain-adapt-finetune paradigm, which boosts and performance of different self-supervised pre-trained models.
It further utilizes the vision-language pre-training model CLIP and propose LV-Adapter. The experiments show that the language priors encoded by CLIP can have some benefits for multiple downstream tasks.


strengths_and_weaknesses: Strengths:
The paper is well-organized and the studied problems have practical value.
The author tried many pre-trained methods, the analysis and experiment are sufficient to verify the proposed viewpoint.

Weaknesses:
The novelty is moderate, both the pretrain-adapt-finetune paradigm and using language priors is not new in this field, but it is worthy to explore them under the multi-task learning scenario in autonomous driving.
There are some confusing points in the experiments, I listed them in the question part.

questions: 1. It would be better to compare your methods with previous multi-task learning methods, maybe Table 1 should have your approach for comparison.
2. It would be better for Table 2 to have a supervised baseline. As you mention in Appendix D.2, architecture is one of the core reasons for the degradation. From the results, we can see the degradation usually happens in semantic seg, the reason is the transformer-based segmentation head. Therefore, the effectiveness of ‘adapt’ is not well reflected. 
3. Table 5 shows that fewer adapt stage (1) can have the best performance. I wonder the results of longer adapt stage with fixed fine-tune stage.
4. If the annotations of different tasks are not aligned, I wonder the realization of co-training process. This means using the pre-trained model to make pseudo labels and then training together? Does the gain come from training together or from more data utilization?
5. Why the Language prior can help the model? Is it useful only under the condition of multi task learning, or can it also assist the learning of each sub task alone?

limitations: The author discuss them in the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a pretrain-adapt-finetune paradigm to train a multi-task network from a well-pretrained network which improves the overall performance compared to the conventional pretrain-finetune paradigm. During the adaption, authors propose to learn a multi-scale adapter. To better use textual modeling in the recent CLIP model, authors further proposes a LV-adapter via prompting learning to incorporate the language priors. The paper experiments on 8 different pretrained models to show their proposed diagram learns a better MTL model from the given pretrained network.
strengths_and_weaknesses: Strengths

1. The pretrain-adapt-finetune diagram is shown effective with 8 different SoTA pretrained networks with various pretraining goals.

2. It is novel to learn a multi-task model for vision tasks from CLIP via prompt learning.

3. The adaption stage is cheap based on the ablation studies.

Weaknesses:

1. The experiment settings for Table 4 and Table 5 are not clearly stated in the paper. For instance, which pretrained model is used in for Table 5? What tasks are considered in Table 4? Why are the best results of Table 4 and 5 not matching the results in Table 2? 

2. Experiments are only limited to BDD100k dataset

3. There is no ablation studies related to the different designs of multi-scale adapter. Can they be replaced with adapters, such as residual adapters?
questions: 1. Could you elaborate more on the experiment settings of Table 4 and Table 5? How do the experiments in Table 4 and 5 relate to those in Table 2?

2. Have you tried with other datasets or other types of tasks? In this paper, experiments mainly focus on the segmentation and detection tasks. What the performance would be when we have other types of tasks, e.g. classification?

3. In Table 3, authors show LV-Adapter is better than self-training. In my understanding, self-training is somehow orthogonal to LV-adapter. Specifically, we can use self-training to generate pseudo labels and train LV-adapter with the available gt labels and pseudo-labels. In this case, what is performance of the LV-Adapter with self-training? Will LV-Adapter keep improving with high-quality pseudo labels?
limitations: They have included in the supplementary
review_title: null

Meta-review: metareview: 
 This paper provides an empirical analysis of the effectiveness of self-supervised learning-based pre-training on multi-task learning, specifically for tasks within autonomous driving. After showing that the standard fine-tuning procedure does not work well in this context, the authors propose a pretrain-adapt-finetune procedure, involving multi-scale adapters and a language-to-vision adapter via task-specific prompt learning. 

  The reviewers appreciated the focus on multi-task learning, especially in a domain where it is highly relevant and standard self-supervised learning methods are under-explored. The experiments were noted to be thorough, with results across many different tasks. However, several concerns were raised including a better set of ablations and adapter design exploration, and performance on another dataset. The author provided a thorough rebuttal, including significant new ablations/experiments and results on NuImages. All reviewers subsequently recommended acceptance, including the reviewer who had a borderline acceptance but mentioned the new results are convincing. 

 While many of the elements of the proposed approach are not new, after the new experiments and ablations especially, this paper provides a nice contribution exploring an under-appreciated setting of multi-task learning and the use of pre-trained models for them. The experiments are thoroughly done and would be of value to the community. As a result, I recommend acceptance. 
confidence: Certain
Decision: Accept