Progressive Parameter Efficient Transfer Learning for Semantic Segmentation

Authors: Nan Zhou, Huiqun Wang, Yaoyan Zheng, Di Huang
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Parameter Efficient Transfer Learning, Semantic Segmentation

Abstract:
Parameter Efficient Transfer Learning (PETL) excels in downstream classification fine-tuning with minimal computational overhead, demonstrating its potential within the pre-train and fine-tune paradigm. However, recent PETL methods consistently struggle when fine-tuning for semantic segmentation tasks, limiting their broader applicability. In this paper, we identify that fine-tuning for semantic segmentation requires larger parameter adjustments due to shifts in semantic perception granularity. Current PETL approaches are unable to effectively accommodate these shifts, leading to significant performance degradation. To address this, we introduce ProPETL, a novel approach that incorporates an additional midstream adaptation to progressively align pre-trained models for segmentation tasks. Through this process, ProPETL achieves state-of-the-art performance on most segmentation benchmarks and, for the first time, surpasses full fine-tuning on the challenging COCO-Stuff10k dataset. Furthermore, ProPETL demonstrates strong generalization across various pre-trained models and scenarios, highlighting its effectiveness and versatility for broader adoption in segmentation tasks. Code is available at: https://github.com/weeknan/ProPETL.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730001246226

thread_type:
official_review

<thread_object>
creator: reviewer_fur2

<content>
summary:
This paper proposes a framework for progressively adapting pre-trained models to downstream semantic segmentation tasks through intermediate tasks, where perception granularity and supervision diversity are interesting and novel. Although this work has some weaknesses in descriptions and experiments, its overall contribution benefits the PETL community, especially in fine-tuning VFMs from classification tasks to semantic segmentation tasks. Thus, I recommend the weak acceptance of this paper and hope the authors' responses.

strengths:
1. The motivations of the proposed approach are clear.
2. The intermediate task designation, including perception granularity and supervision diversity, is interesting and novel.
3. The proposed approach achieves better performance on multiple benchmarks.

weaknesses:
1. The evaluations of the generalization ability are insufficient. ProPETL demonstrates strong generalization across various pre-trained models and scenarios. However, the experimental evaluation does not show the proposed ProPETL's generalization ability under cross-scenario tasks, such as the domain generalization setting. I am interested in the proposed ProPETL's generalization ability under cross-scenario tasks.
2. The descriptions of the proposed approach are unclear. For example, the summary of the overall training and inference process and the design of the GPA's architecture in intuition or theory.
3. Do α of supervision diversity and pooling window of perception granularity correspond one to one?
4. The computational cost of the proposed approach during the training and inference stages should be introduced and compared with other fine-tuning strategies in detail, such as the total training time, inference time, FLOPs, GPU Memory, Storage, etc.
5. More backbones, such as DINOv2 and EVA02, and more segmentation heads, such as Mask2Former and SemFPN, should be included to verify the effectiveness of the proposed approach.
6. How to evaluate and compare to the Rein [1] which also introduces the PETL method for semantic segmentation tasks.
[1] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation. CVPR 2024.
7. The fail examples, limitations, and future works should be provided and discussed.
8. Figure 1 (a) and (b) should be more elegant. Labels of the x-axis are too narrow.
9. The code should be released.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730299651040

thread_type:
official_review

<thread_object>
creator: reviewer_fzgq

<content>
summary:
This paper presents ProPETL framework, which applies progressive adaptation to Parameter-Efficient Transfer Learning (PETL) and introduces a novel midstream adaptation stage to address perception granularity differences and supervision diversity in semantic segmentation tasks.

strengths:
This paper analyzes two candidate progressive adaptation strategies: Generalized Parametric Adaptation (GPA) and Decoupled Structured Adaptation (DSA), and employs DSA to effectively mitigate the "forgetting" issue observed in GPA. 

Extensive comparisons demonstrate that ProPETL delivers superior performance across benchmarks, underscoring its effectiveness and superiority.

weaknesses:
1. Both Generalized Parametric Adaptation (GPA) and Decoupled Structured Adaptation (DSA) implement dimensionality reduction and expansion through bypass branches to reduce the number of parameters. However, the fixed reduction ratio $r$ may lead to the loss of critical feature information, especially in tasks with high perceptual demands (such as semantic segmentation). The fixed ratio of the reduction operation lacks adaptability to the importance of features and the complexity of input data, which may result in the bypass branch being unable to effectively retain fine-grained information, thus affecting the overall performance of the model.

2. This paper suggests that Perception Granularity and Supervision Diversity are key factors in the design of intermediate tasks, but it lacks theoretical support for these design choices. For example, it does not explain why image-level perception granularity and multi-label supervision can enhance the fine-grained performance of downstream tasks.

questions:
1. DSA divides the adapter parameters into two independent parts: the parameters for the midstream adaptation stage( $\phi$) and the parameters for the downstream fine-tuning stage \($ \psi$ \). During the intermediate adaptation phase, DSA only optimizes $\phi$, while in the downstream fine-tuning phase, $\phi$ remains frozen, and only $ \psi$ is optimized. However, the strategy of freezing  $\phi$ during the downstream task, which reduces the parameter space that can be optimized and increases the burden on optimizing  $ \psi$  , leading the model to seek the best solution in a smaller space. For complex downstream tasks, this limitation may affect the model's  performance, as the optimization of  $ \psi$ cannot fully replace the tuning of  $\phi$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730424010888

thread_type:
official_review

<thread_object>
creator: reviewer_gsci

<content>
summary:
The paper presents ProPETL, a new method to improve Parameter Efficient Transfer Learning (PETL) for semantic segmentation tasks, which usually have more difficulties in fine-tuning compared to classification tasks. Current PETL methods find it hard because they need to adjust more parameters to handle changes in the details of semantic understanding. ProPETL solves this problem by adding a midstream adaptation step, which gradually aligns the pre-trained models for segmentation tasks. This proposed method achieves the state-of-the-art performance, even better than full fine-tuning, on various evaluation benchmarks.

strengths:
1. The authors provide a detailed analysis of why existing PETL methods do not perform effectively for semantic segmentation tasks. This paper not only introduces ProPETL but also thoroughly tests its effectiveness compared to state-of-the-art methods across multiple benchmark datasets. The authors conduct extensive ablation studies to show the effect of each component of the method, which supports the strength of the proposed contributions.

2. The paper is well-written and organized clearly, making the paper easy to understand. The authors include background information and visualize to illustrate the weaknesses of current PETL methods and the strength of ProPETL. 

3. ProPETL outperforms full fine-tuning on various benchmarks, showing that parameter-efficient methods can match or even exceed traditional methods.

weaknesses:
1. The proposed midstream adaptation phase makes the whole framework more complex than previous methods. Selecting intermediate tasks, such as perception granularity and supervision diversity, looks like it’s based on trial and error without a clear, systematic way for users to apply these choices to new datasets or tasks.

2. While the progressive adaptation process improves segmentation performance, it may need more computational time because of the extra midstream phase, especially for large datasets. The paper claims that this extra cost is worth it for better performance, but having more exact comparisons of training time or resource usage with other PETL methods would help show the practical trade-offs, especially when computational efficiency is very important.

questions:
1. The paper shows that ProPETL works well with a Swin-L model pre-trained on ImageNet-21K, but how would the performance change if we use smaller or less powerful pre-trained models? Is there any guideline or expectation for using ProPETL in situations with limited resources? Would ProPETL still be helpful in transfer learning cases that use self-supervised models with lower representational capacity? It would be good to see more discussion or results about how much ProPETL depends on the quality of pre-trained models.

2. The paper says that the midstream adaptation improves segmentation performance, but how does the computational cost compare with traditional PETL methods, in terms of training time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630730327

thread_type:
official_review

<thread_object>
creator: reviewer_4nue

<content>
summary:
The paper finds that there is a perceptual granularity gap between pre-trained models and downstream segmentation tasks, necessitating significant parameter adjustments. However, existing PETL approaches fail to effectively address this gap due to the constraint of limited trainable parameters. To tackle this issue, the paper introduces ProPETL, which incorporates an additional midstream adaptation phase to progressively align pre-trained models with downstream segmentation tasks. Specifically, the paper conducts an in-depth analysis of candidate progressive adaptation strategies, such as Generalized Parametric Adaptation and Decoupled Structured Adaptation, as well as intermediate tasks designed to enhance the perception of finer granularity and diverse classes. Notably, ProPETL achieves state-of-the-art performance on most segmentation benchmarks, surpassing even full fine-tuning.

strengths:
1. The paper addresses the gap between pre-trained models and downstream segmentation tasks by introducing an additional midstream adaptation process, while maintaining parameter efficiency.
2. The method demonstrates its superiority by outperforming other parameter-efficient approaches and even surpassing full fine-tuning.

weaknesses:
1. The paper is motivated by the observation that the perceptual granularity gap between pre-trained models and downstream segmentation tasks necessitates significant parameter adjustments. However, for models like SAM[1], which are pre-trained on segmentation tasks, there is a lack of evidence to support that these observations still hold. 
2. The necessity of additional midstream adaptation is not entirely clear, as the proposed intermediate tasks resemble data augmentation, which could potentially be integrated into the fine-tuning phase.

[1] Kirillov A, Mintun E, Ravi N, et al. Segment anything[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4015-4026

questions:
Could the intermediate tasks be considered a form of data augmentation? Would performance improve if these intermediate tasks were integrated into the fine-tuning phase? For example, scaling the images to different sizes and generating varying numbers of categories at random ratios during fine-tuning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732164960176

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response

comment:
We sincerely appreciate the detailed and constructive feedback from all the reviewers. In particular, we are grateful for their recognition of the work's contribution to the PETL community (Fur2) and their acknowledgment of the idea as novel (fzgQ, Fur2), interesting (Fur2), and supported by clear motivation (Fur2) and strong writing (GSCi). Moreover, we are pleased that the method has been recognized as well-analyzed (4Nue, GSCi) and well-performed (4Nue, GSCi, fzgQ, Fur2).

Before addressing the individual reviews, we briefly outline the manuscript changes and key points highlighted in the feedback:
1. **Additional Analysis**:
   - Added analysis on computational overhead. 
   - Updated sections on failure cases, limitations, and future work. 
2. **Extended Evaluation**:
   - **Pre-trained Models**: Additional evaluations using various sizes of Swin-Transformers, MoCo-v3, SAM, and DINOv2 as encoders. 
   - **Segmentation Heads**: Mask2Former has been added as a segmentation head for evaluation.
   - **Cross-Domain Evaluation**: The effectiveness of ProPETL has been assessed under the *GTAV → CityScapes* domain generalization setting.
3. **Minor Edits**:
   - Refined descriptions of the training and inference processes.
   - Optimized Figure 1 for improved clarity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a framework named ProPETL, which can adapt pre-trained models to downstream semantic segmentation tasks through intermediate tasks progressively, together with a new midstream adaptation stage to address perception granularity differences and supervision diversity. The manuscript was reviewed by four experts in the field. The recommendations are (4 x "6: marginally above the acceptance threshold"). Based on the reviewers' feedback, the decision is to recommend the acceptance of the paper. The reviewers did raise some valuable concerns like framework and computational complexity, unconvincing experimental evaluations, unclear descriptions and statements that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.