Progressive Parameter Efficient Transfer Learning for Semantic Segmentation
Authors: Nan Zhou
Huiqun Wang
Yaoyan Zheng
Di Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Parameter Efficient Transfer Learning
Semantic Segmentation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Parameter Efficient Transfer Learning (PETL) excels in downstream classification fine-tuning with minimal computational overhead, demonstrating its potential within the pre-train and fine-tune paradigm. However, recent PETL methods consistently struggle when fine-tuning for semantic segmentation tasks, limiting their broader applicability. In this paper, we identify that fine-tuning for semantic segmentation requires larger parameter adjustments due to shifts in semantic perception granularity. Current PETL approaches are unable to effectively accommodate these shifts, leading to significant performance degradation. To address this, we introduce ProPETL, a novel approach that incorporates an additional midstream adaptation to progressively align pre-trained models for segmentation tasks. Through this process, ProPETL achieves state-of-the-art performance on most segmentation benchmarks and, for the first time, surpasses full fine-tuning on the challenging COCO-Stuff10k dataset. Furthermore, ProPETL demonstrates strong generalization across various pre-trained models and scenarios, highlighting its effectiveness and versatility for broader adoption in segmentation tasks. Code is available at: https://github.com/weeknan/ProPETL.

Published as a conference paper at ICLR 2025
PROGRESSIVE PARAMETER EFFICIENT TRANSFER LEARNING FOR SEMANTIC SEGMENTATION
Nan Zhou Huiqun Wangh.2, Yaoyan Zheng Di Huang!.31
1 State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China Schooo of Computer Science and Engineering, Beihang University, Beijing. China [zhounanC43: hqwangscse, yaoyanzheng, dhuang) @buaa edu.cn
ABSTRACT
Parameter Efficient Transfer Learning (PETL) excels in downstream classifica- tion fine-tuning with minimal computational overhead, demonstrating its poten- tial within the pre-train and fine-tune paradigm. However, recent PETL methods consistently struggle when fine-tuning for semantic segmentation tasks, limiting their broader applicability. In this paper, we identify that fine-tune tin-ttnnig for seman- tic segmentation requires larger parameter adjustments due to shifts in semantic perception granularity. Current PETL approaches are unable to effectively ac- commodate these shifts, leading significant performance degradation. To ad- dress this, we introduce ProPETL, a novel approach that incorporates an additional midstream adaptation to progressively align pre-trained models for segmentation tasks. Through this process, ProPETL achieves state-of-the-arr performance on most segmentation benchmarks and, for the first time, surpasses full fine-tuning on the challenging COCO-Stuff10 dataset. Furthermore, ProPETL demonstrates strong generalization across various pre-trained models and scenarios, highlight- ing its effectiveness and versatility for broader adoption in segmentation tasks. Code is available at: bbttps://itthiii www.///weeea///rPP......
INTRODUCTION
Parameter Efficient Transfer Learning (PETL) aims to leverage the representational knowledge from large-scale pre-trained models (e.g., MAE (He et al. 2022) and DINOv2 (Oquab et al.] (023)) for downstream tasks while minimizing computational costs. Recent studies (Jia et al., 2022] Chen full
al. 2022 Hu al. 2022) show that PETL can match or even surpass the performance fine-tuning for downstream classification tasks, achieving this with less than one percent of the parameters needing adjustment. This efficiency has motivated researchers to explore its application
across various computer vision scenarios (Han et al. 2024
However, existing PETL methods face performance bottlenecks when applied to semantic segmenta- tion (Jia et al.] 2022) Hu et al. 2022). Unlike classification tasks, semantic segmentation requires the model to develop fine-grained perceptual capability to predict semantic labels for each pixel. This shift in perceptual granularity complicates the fine-tuning process for segmentation. The discrepancy further illustrated in Fig. [a). Despite their superior results in classification tasks, PETL meth- ods consistently exhibit performance gap compared to full fine-tuning across most segmentation benchmarks. Consequently, full fine-tuning remains the preferred approach in current segmentation efforts, limiting PETL's broader adoption.
Upon examining the fine-tuning process for classification and segmentation, we identify two key phenomena. First, transitioning from a classification pre-trained model to a downstream segmen- tation task requires extensive parameter adjustments during full fine-tuning, as indicated by the increased mean Euclidean distance in Fig. [b)). Second, the limited tunable parameters in PETL hinder its ability to capture semantically-awant aware changes when fine-tuning directly for segmentation. As illustrated in Fig. [c). the PETL method, AdaptFormer (Chen al. 2022). exhibits small adjust-
Equal contribution Corresponding author.

---

Published as a conference paper at ICLR 2025
(97idwp
4000
in Iiteraaie
LifS
KIAA
hultt Chand
PureTT (Oum)
3000
Ad Alattforme
Pigft Tound
2000
a0s fo Fowrrty
(a0a) cocOland vocet
neti Ackcts
Fiat ot to AR to clta vocast RIC coo
3000
Normakiool Eudioeen Datarce
Figure I: (a) Relative performance of PETL methods compared to full fine-tuning on image classifi- cation [CLS) and semantic segmentation [SEG] tasks. (b) Euclidean distance between the parameter vectors of the classification pre-trained model before and after full fine-tuning across different down- stream tasks. The optimizer settings and the number of training iterations are aligned to minimize the influence of training conditions and data volume. Hc))osgggram showing the normalized Euclidean distance between the feature map of the pre- pre-trained model and the feature map after adaptation by different methods. (d) A visualization from ADE20K, illustrating the normalized Fuclidean distance map after applying various adaptation methods.
ment magnitudes across most regions of the feature maps, unlike the uniformly distributed adjust- ments observed in full fine-tuning, resulting in statistically right-skewed distribution. This trend is further evident in the samples visualized in Fig. dd). These observations indicate that fine-tuning for segmentation tasks demands both larger parameter adjustments and broader semantic changes to bridge the perceptual granularity gap between pre-trained models and downstream tasks. However, existing PETL methods struggle with these shifts, leading to suboptimal segmentation performance.
These insights motivate us to develop solution that retains PETL's computational efficiency while adapting to the significant changes in semantic perceptual granularity during fine-tuning. Inspired by previous works (Hsu et al. 2020; Dong et al. 2024b) that improve domain generalization through a progressive paradigm, we introduce an additional phase, midstream adaptation, into the PETI. framework to progressively address semantic granularity differences. Achieving this involves two major challenges. First, developing a strategy to bridge the semantic perception gap remains un- derexplored. Unlike prior adaptations that address data domain changes (Dong al. 2024b), the progressive adaptation for segmentation must effectively bridge the differences in semantic percep- tual granularity between upstream and downstream tasks, requiring detailed analysis and modeling. Second, it remains uncertain which level of supervision and specific perceptual granularity for in- termediate tasks would most benefit progressive fine-tuning for segmentation. Identifying the most effective intermediate task for enhancing segmentation perception remains an open question.
To address these challenges, we begin with analyzing two candidate progressive adaptation strate- gies: Generalized Parametric Adaptation (GPA) and Decoupled Structured Adaptation (DSA). We then conduct a comprehensive investigation into the impact of different intermediate tasks and em- pirically determine the optimal selection. By integrating both approaches, we propose ProPETL, a novel progressive PETI framework for segmentation adaptation. By effectively bridging the per- ception gap between pre-trained representational knowledge and downstream segmentation tasks, ProPETL significantly enhances performance while maintaining computational efficiency. Com- pared to its counterparts, ProPETL substantially narrows the performance gap with full fine-tuning on most benchmarks and, for the first time, surpasses full fine-tuning on the challenging COCO- Stuff10k Ok dataset, demonstrating its effectiveness.
Our contributions are summarized as follows:
1) We introduce ProPETL, innovative framework for progressively adapting pre- pre-trained models to downstream semantic segmentation tasks through intermediate tasks.
2) We conduct in-depth analysis and comprehensive comparison of candidate progressive adap- tation strategies and intermediate tasks to facilitate the continuous migration of representational knowledge from pre- rreitrined models to segmentation tasks.
3) We achieve state-of-te the-art segmentation fine-tume tuning performance across diverse benchmarks and demonstrate robust generalizability across various pre-trained models and segmentation tasks.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 PARAMETER EFFICIENT TRANSFER LEARNING
PETL aims to effectively fine-tune pre-trained models for downstream tasks with minimal parameter updates. Recent approaches can be categorized into three main types: partial tuning, prompt-based. and adapter-based methods. Partial tuning methods strategically select a subset of the pre-trained model for fine-tuning. For example, BitFit (Zaken al.] 2022) updates only the bias terms of the network while freezing other parameters, and SPT (He et al. (023) calculates sensitivity metric based on gradients to fine-tune the high-sensitivity parameters. Prompt-base methods incorporate visual prompt tokens (Jia et al. (2022) into the downstream inputs, which then undergo parameter updates. Various prompt structures (Das et al. 2023; Zhou et al.) 2024) and parameter-efficient
strategies (Han al. 2023) are explored to further enhance performance. Adapter-based approaches introduce additional lightweight adaptation networks rather than modifying inputs. AdaptFormer
(Chen al. (2022) and LoRand (Yin et al. 2023) incorporate adaptation modules with bottleneck
structures and residual connections, while LoRA (Hu et al., (2022) and RLRR (Dong et al.
2024a)
employ learnable low-rank parameter bypasses to facilitate adaptation.
More recently, some efforts apply PETI. methods to more complex tasks like segmentation Yin et
leverage stronger pre-trained models such as SAM (Kirillov (2023) However, despite achieving promising results in fields like medical image segmentation (Wu et al. 2023) and
(023) 2023)
camouflaged object detection (Chen al. 2023), , a notable performance gap persists between PETL and full fine-tuning in semantic segmentation tasks, particularly when compared to the successes realized image classification.
2.2 PROGRESSIVE DESIGN IN TRANSFER LEARNING
Progressive paradigm is a prevalent concept in transfer learning (Rusu et al. 2016 Weinshall et al., 2018). aimed at enhancing model performance by gradually adapting the target domain. This approach is particularly advantageous when there is substantial gap between the source and target domains (Hsu al., 2020), facilitating smoother transitions and more effective knowledge transfer. For instance, Progressive Neural Networks (Rusu et al. (016) introduce new columns of neurons for each new reinforcement learning task while keeping the parameters of the previous columns fixed. In the field of domain adaptation, Hsu et al. (2020) generate intermediate domain data situated between the source and target domains using CycleGAN (Zhu al. 2017), progressively addressing domain adaptation challenges. Similarly, PPEA (Dong et al.] (024b) designs progressive training framework tailored for depth estimation that utilizes external datasets to construct an easy-to-hard pipeline, iteratively updating the adapter various training stages.
Despite the success of these approaches in mitigating data-domain shifts, extending the progres- sive paradigm from adaptation across similar tasks in different data domains to addressing the fine- grained perceptual requirements of downstream segmentation remains underexplored.
METHOD
3.1 VANILLA PETL
Given a pre-trained model fo(-) parameterized by 0, the goal of PETL is efficiently adapt this pre-trained model to downstream task, yielding a model fe (-), where & denotes the weights of the adaptation module. During the adaptation process, with the training set D = {(>;; y:) ) from the downstream task, PETL freezes 0 and only updates &f by optimizing the following objective:
Φ² arg min C (:) D D.
where L()) denotes the loss function for the downstream task and o² indicates random initialization of the parameters in the adaptation module.
By fully leveraging the pre-training weights of the pre- pre-trained model, PETL achieves the same or even better performance than full fine-tuning by updating only &b. Since the number of parameters in
3

---

Published as a conference paper at ICLR 2025
Adaptation Module
Generalized Parametric Adaptation (GPA)
Decoupled Structured Adaptation (DSA)
Add
FFN
FFN
FFN
+
FFN
Concat
up
ReLD
Φ
Trained
down
Initialize
Initializze
Figure 2: Illustration of two progressive adaptation strategies. Left: Generalized Parametric Adap- tation. Right: Decoupled Structured Adaptation. "FFN" indicates the Feed-Forward Network.
& only small fraction of the total model parameters 0, PETL can significantly enhance computa- tional efficiency and reduce the cost of the fine-tuning process while improving accuracy compared to full fine-tuning. However, PETL encounters bottlenecks when applied to downstream semantic segmentation tasks due changes in perceptual granularity.
3.2 MIDSTREAM PROGRESSIVE ADAPTATION
To mitigate semantic granularity shifts, we divide the adaptation into two phases: midstream adap- tation and downstream fine-tuning. By introducing these additional training phases, the adaptation can be enhanced with supervision from intermediate tasks, thereby facilitating a better transition of the pre-trained model into downstream tasks. For this purpose. we investigate two candidate pro- gressive adaptation strategies: Generalized Parametric Adaptation (GPA) and Decoupled Structured Adaptation (DSA), which are graphically illustrated in Fig. 2
Generalized Parametric Adaptation is straightforward progressive adaptation strategy that uti- lizes enhanced supervision during the midstream phase to bridge perception gaps between the pre- trained model and downstream tasks. Formally, during the midstream adaptation phase, the pre- trained weights 0 are frozen and only Φ updated by optimizing the following objective:
arg min L (f): D imss)
where L and Dd denote the loss function and training data of the intermediate task, & génit denotes the initial weights and d=th indicates the optimized weights of $. In the second training phase, GPA begins to fit & starting from 6= using the downstream data and objective, formalized as:
arg min L (f) D.
(3)
With the improved initialization from the midstream adaptation, & tends to generalize better in the downstream fine-tuning phase compared to vanilla PETL methods.
For the adaptation module, GPA introduces a bypass branch to the Feed-Forward Network (FFN) layer. This bypass branch uses bottleneck structure with down-projection layer and up- projection layer, parameterized by Wdown Dx and Wup R respectively. D denotes the channel dimension and is the reduction factor for parameter efficiency. For the input feature of the FFN layer, the adapted feature obtained by adding the output of the bypass branch to the output of the FFN layer, formalized as:
= FFN(2)) + ReLU(z Wdown) Wup- up-
(4)
During midstream adaptation, GPA optimizes the parameters within the bypass branch according to Eq. 2 and keeps the optimized parameters (Wie down: W ]- In downstream fine-tuning, GPA uses the
same structure as in the midstream adaptation phase (refer to Eq. 4) and initializes (Waowny Wap3
with {Wm
down: Wm) and further optimizes them based on the downstream task loss.
Decoupled Structured Adaptation is another progressive adaptation strategy for PETL. Unlike GPA, which updates the same adapter for both intermediate and downstream tasks, DSA splits the adaptation parameters into two parts {Φ, v) During the midstream adaptation phase, only & will be optimized. In the downstream fine-tuning phase, Φ is frozen, and optimization is focused solely on zb. By successively optimizing the following objective:
Φ arg min L (fe.o)); ( timtt,
(5)
2/ arg min L (+): simi))
(6)
4

---

Published as a conference paper at ICLR 2025
DSA thus obtains the optimized parameters 6t and 2 during the two-phase training process. The key difference between DSA and GPA lies in the decoupled weights. By freezing part of the adaptation parameters & after midstream adaptation, DSA is able to avoid the forgetting problem that can occur during downstream fine-tuning in GPA. Refer to Section 4.3 for further analysis.
Specifically, during the midstream adaptation, DSA introduces bypass branch to the FFN layer, which shares the same structure as in Eq. and obtains the optimized parameters (Wawwn: down> Wim)
according to Eq In the downstream fine-tuning stage, DSA first recovers the feature 2 after midstream adaptation by applying:
(=) + ReL.U( Wdown)
Waps
where the projection layer parameters [Wdownx Wap} are initialized with { (W down' W and kept
frozen. To address potential task bias introduced during midstream adaptation, DSA concatenates z with the original FFN output along the channel dimension and feeds them into randomly ini- tialized bypass branch parameterized by [Woown g2Dx 22 Wup E IR 2P xD). This allows
{ down. Wup to selectively integrate midstream information and progressively adapt intermediate features for downstream tasks. DSA obtains the final adapted feature by adding the bypass branch output to the FFN output and optimizes [Wdown: W.p} according Eq. 6 The above process is formalized as:
せ = FFN(z) + ReLU ([=, Wdowa)
W up-
(8)
where [. indicates the channel concatenation operator.
3.3 INTERMEDIATE TASK DESIGNATION
In progressive transfer learning for semantic segmentation, the intermediate task expected to en- hance the model's fine-grained perception to produce dense predictions for diverse classes, re- quired by downstream applications. Two key factors should be considered this context, i.e, perception granularity and supervision diversity.
Perception Granularity. Unlike pre-trained tasks focused on global perception for object-level classification, segmentation tasks require much finer perception granularity to capture pixel-level se- manties. The intermediate task should help the model bridge this shift in granularity. To achieve this, we design a transfer strategy that converts the downstream training data with dense annotations to support training at varying levels of perception granularity.
W 32 32
32
H W 64"64
1 x Building
Specifically, for the ground truth dense annotation map y E HW from the downstream dataset D, where H and W indicate the image height and width, we obtain its one-hot label vector ý e k W xC, where C' is the total number of categories in D. The per- ception granularity transformation F-G(-) is performed by down- sampling ý using pooling windows of size x 8 with a stride of different levels of granularity by controlling s.
64 x 64
H x W
Figure 3: Illustration of FPG.
thereby generating annotations
For generating level labels, FPa(.) applies an H x W pooling window, producing a single label vector FPG (9) E IRC for the entire input image. For patch-level labels, FPG(.) sets s to match the patch size of the pre-trained transformer, resulting in FPG (t) E TR x xC which yields и x label vectors, each corresponding to a distinct patch region. Fig, visualizes the output of iPa (y) with varying pooling window size. Intuitively, after applying [PG( ), the intermediate task requires the model to perform coarse-grained region classification rather than classifying each input pixel individually.
Supervision Diversity. In addition to shifts in perception granularity, downstream segmentation tasks require classification pre-trained models develop a broader semantic awareness beyond rec- ognizing single dominant class. To address this gap, we incorporate semantic diversity modeling into the intermediate task design by introducing supervision diversity transformation (.D
5

---

Published as a conference paper at ICLR 2025
Midstream Adaptation
Intermediate Task
Downstream Task
Pre-training
predict
Perception Granularity
Pre-trained Model
Supervision Diversity
Midstream
Adaptation
Downstream Fine-tuning
Decoupled Structured Adaptation
Downstream Task
Trained
predict
Downstream
Pre-trained Model
Sepmentation Head
Fine-tuning
Frazen
Figure 5: Illustration of the ProPETL framework. In the midstream phase, we generate the mid- stream dataset via perception granularity and supervision diversity transformations and update the adaptation module. In the downstream fine-tuning phase, we apply the decoupled structured adapta- tion strategy and train both the adaptation module and segmentation head the downstream dataset.
To be specific, F-p(-) refines the pooling method utilized (-G by employing max mooxiin to generate one-hot labels Y. and mean- pooling to create many-hot labels ÿm. which reflect the semantic density within given region. To finely control supervision diver- sity and inspired by label smoothing (Szegedy et al. 2016). (-D(1) interpolates between Yo and Ym using the equation [1 yo + 9m. where a E [0, 1] serves as a smoothing factor. Fig. illustrates output of (1) with varying a when
0.5% +0.5%
9m
By vary- ing F-D(.) generates labels with different numbers of categories, thus modulating the supervision diversity of the intermediate tasks.
0.5
=
Figure 4: Illustration of FSD-
By combining these two factors, we formulate intermediate tasks that require predicting the generated labels FSD FrG((:) from input 2;; This approach enhances the pre-trained model's fine- fine-grained perception ability, helping it better adapt to downstream seg- mentation applications.
3.4 FRAMEWORK AND COMPLEXITY
Framework. The entire framework is illustrated in Fig. 5 The training process of ProPETLLccosi of two phases: midstream adaptation and downstream fine-tuning.
In the midstream adaptation stage, transform the downstream dataset D with respect to per- ception granularity and supervision diversity to obtain an intermediate dataset mm To prevent overfitting the intermediate task, we a two-layer MLP the intermediate task head, fol- lowing standard practices (Chen et al. 2020a b). We evaluate different combinations of perception granularity and supervision diversity, and based on these empirical studies, we employ image-level granularity (s = H x W) and multi-label diversity (a 1) for midstream adaptation in our frame- work. The standard cross-entropy loss, denoted as Lm (-), used optimize both the bypass branch and the intermediate task head. In the downstream fine-tuning stage, ProPETL first processes adap- tation module parameters according to the decoupled structured adaptation strategy. Subsequently, the downstream segmentation head is randomly initialized, and the learnable parameters are fine- tuned with the downstream dataset. For inference, the trained adaptation module, combined with the pre-trained backbone, extracts feature maps from input images, which are then fed to the seg- mentation head to generate the final segmentation results.
Parametric Complexity. When employing GPA the progressive adaptation strategy, the num- ber of learnable parameters in the backbone network is O(2LD") where denotes the number of backbone layers. In contrast, DSA increases the number of learnable parameters to O ( SLDD due to the incorporation of additional structures. To maintain parameter consistency, we empirically set the reduction factor in DSA be four times that used in GPA.
y

---

Published as a conference paper at ICLR 2025
Table 1: Comparative results on four semantic segmentation datasets. The experiments utilize an ImageNet-21k supervised pre-trained Swin-L as the backbone and UperNet as the segmentation head. The best and second-best results of the PETL method are highlighted in bold and underline, respectively. The symbol u*" denotes the learnable parameters in the backbone.
VOC2012
ADE20k
COCO-Stuf710k
CityScapes
Mean
Learnable
Param. (M)
Method
mloU
mAce
mloU mAce
mloU
mAce
mloU
mAce
mloU
mAcc
Full fine-tuning
84.38
89.82
51.71
63.13
46.30
58.98
82.36
88.53
66.19
75.12
195.00
Freeze
83.32
89.16
47.51
59.74
42.36
54.39
75.54
82.91
62.18
71.55
BitFit
84.25
90.25
48.00
60.81
43.75
56.37
77.73
84.89
63.43
73.08
0.30
LoRand
84.09
90.32
48.08
59.31
43.96
56.41
76.88
83.89
63.25
72.48
3.59
RLRR
84.77
90.72
48.72
60.53
44.51
57.03
78.43
85.35
64.11
73.41
0.46
VPT
85.69
91.12
50.35
61.37
45.04
57.58
78.95
86.56
65.01
74.16
3,61
AdaptFormer
85.41
90.93
50.38
62.55
45.45
57.83
79.41
86.03
65.16
74.34
2.64
LoRA
85.36
91.34
50.45
62.91
45.03
57.81
80.54
87.23
65.35
74.82
4.55
ProPETL
86.11
92.08
51.05
63.61
46.48
58.69
81.67
88.23
66.33
75.65
3,30
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We conduct comprehensive evaluation of the proposed ProPETL. for segmentation adap- tation benchmarks, including PASCAL VOC2012 (Everingham et al. 2015). ADE20k (Zhou ctt al., 2019), COCO-Stuff10K (Caesar et al. 2018), and CityScapes (Cordts et al. 2016). 1) PAS- CAL VOC2012 consists of 1,464 training images and 1,449 validation images spanning 21 cate- gories. is widely adopted in object detection and semantic segmentation, making it key bench- mark for tasks involving limited-scas scale data. 2) ADE20k includes 20,210 training images and 2,000 validation images, making it one of the most comprehensive datasets for semantic segmentation with wide range of data variances. 3) COCO-Stuff10k extends the COCO dataset (Lin et al., 2014) with dense annotations for semantic segmentation, including 9,000 training images and 1,000 validation images across 172 categories. 4) CityScapes includes finely annotated images across 19 categories, with 2,975 for training and 500 for validation. It is widely used for evaluating urban scene understanding.
Implementation details. We employ AdamW (Loshchilov & Hutter 2018) for optimization, with 31 and 32 set to 0.9 and 0.999, respectively. PolyLR scheduler is used to dynamically adjust the learning rate during the training process. The batch size is set to 16 across all benchmarks, and the iterations for downstream fine-tuning range from 40k to 160k, consistent with those in previous works (Xiao et al.) 2018 Liu et al. [222]) for fair comparison. For midstream adaptation, the iterations are set to half of the corresponding downstream settings.
Following the counterpart (Ynn al. 2023), we utilize Swin-L (Liu et al. 2021). pre-trained on ImageNet-21K (Deng et al. 2009). as the backbone, and UperNet (Xiao et 20188) as the seg- mentation framework. The mean intersection over union (mloU) and mean accuracy (mAcc) on the validation set across all benchmarks are used as metrics to provide comprehensive comparison with state-of-the-art methods. All experiments implemented using MMSSggennattin (Contributors, 2020) on NVIDIA A800 GPU.
4.2 COMPARISON WITH THE STATE-OF-THE-
We compare the proposed ProPETL with its counterparts on segmentation benchmarks, including full fine-tuning, freeze (a.k.. linear probing), and several typical PETL methods. The PETL coun- terparts include partial tuning methods like Bias (Zaken et al. 2022), prompt-based methods like
VPT (Jia al. 2022), and adapter- adapter-based methods such as AdaptFormer (Chen et al. 2022 Lo-
Rand Yin et al. 2023). LoRA (Hu et al. 2022). and RLRR (Dong et al. 2024a).
As illustrated in Table
ProPETL delivers highly competitive performance across all settings, sig-
nificantly surpassing PETL counterparts by a large margin both metrics. Notably, on the chal-
lenging COCO-Stuffi benchmark, it outperforms the previous state-of-the-art, AdaptFormer, by

---

Published as a conference paper at ICLR 2025
VOC2012
CitScapess
OOCO-Stuff1O
87
GPA DSA
46.5
47.0
GPA
82
GPA
DSA
mlou (%)
81
46.0
miou
80
45.5
85
45.0
78
44.5
I-M
PM
145
1-M
P-S
P-M
I-M
P-S
P-M
Intermediate Task (a)
Intermediate Task
Intermediate Task (c)
(b)
Figure 6: Ablation results of two progressive adaptation strategies. The dashed line represents the result of the baseline approach, i.e. AdaptFormer. "T" and "P" denote that the perception granularity of the intermediate task is image-level and patch-level, respectively. "S" and "M" indicate that the supervision diversity of intermediate task single-label and multi-labed respectively.
1.03% mloU. Additionally, ProPETI. exceeds full fine-tuning while using only 1.7% the param- eters. To the best of our knowledge, this is the first time PETI. method has outperformed full fine-tuning on this benchmark, demonstrating its superiority. On the most challenging benchmark, ADE20K, the proposed method also delivers performance comparable to full fine-tuning and even surpasses it in the mAce metric, indicating its generalizability across various segmentation scenarios.
When considering parameter efficiency, it is worth noting that although LoRand, VPT, and LoRA utilize a comparable amount of learnable parameters as ProPETL, their performances are signifi- cantly inferior. This phenomenon demonstrates that the introduction of the progressive paradigm effectively enhances the adaptation ability to manage the semantic perception granularity shift in downstream segmentation tasks within PETL, while also maintaining computational efficiency.
4.3 ABLATION STUDY
Progressive Adaption Strategy. We conduct detailed ablation studies on two typical progressive adaptation strategies with representative intermediate tasks the VOC2012, COCO- Stuff10k and CityScapes benchmarks. The results are illus- trated in Fig. 6 From Fig. 6. observe that, compared to the AdaptFormer baseline, the introduction of the progres- sive paradigm significantly boosts performance in most set- tings, demonstrating its effectiveness for semantic segmen- tation adaptation. Moreover, DSA consistently outperforms GPA in most settings, highlighting its superiority.
COCO-Stuff10k
Intermediate Task Loss
4.0
Midstream Adaptation
Downstream Fine tuning
2.5
2.0
100
120
Training Iterations (1e3)
We further investigate this phenomenon and visualize the loss
Figure 7: Visualization of the loss curves for the intermediate task.
function curve during training in Fig. 7 As shown in the fig- ure, when training with downstream tasks, GPA tends to forget portion of the knowledge learned during the intermediate tasks, as indicated by the increasing loss value of the intermediate task. contrast, since DSA freezes parts the adapter after midstream adaptation, it continues to transfer effectively to the downstream task without degradation, thereby delivering better performance. Based on these analysis, we employ DSA in our framework.
Intermediate Task Design. We conduct ablation studies to investigate the impact of perception granularity and supervision diversity. To mitigate perturbations, we average results from experi- ments with varying supervision diversities at each perception granularity level, and vice versa As shown in Table the VOC2012 and COCO-Stuff10k datasets exhibit a preference for intermediate task with image-level perception, while the City CityScapes dataset favors more fine-grained intermedi- ate tasks. This divergence can be attributed to the fact that VOC2012 and POCO-Stuffil contain a greater number of object-centric images; the image-level intermediate task necessitates perceiving the objects existing in the image, thereby providing basis for the downstream segmentation tasks.
Please refer Table 13 the Appendix for full results.
8

---

Published as a conference paper at ICLR 2025
Table Ablation study on the perception granularity of intermediate tasks using DSA during down- stream fine-tuning. "s" denotes the pooling window size. The first line is the result of AdaptFormer.
Patch-level
Image-level
VOC2012
COCO-Stuff10k
CityScapes
* 32
128
640
mloU
mAce
mloU
mAce
mloU
mAce
85.41
90.93
45.45
57.83
79.41
86.03
85.75
91.30
45.60
57.80
80.84
87.26
85.62
92.04
45,99
58.04
81.03
87.58
85.89
91.68
46.32
58.33
80.89
87.54
Table Ablation study on the supervision diversity of intermediate tasks using DSA during down- stream fine-tuning. "a" represents the smoothing factor. The first line is the result of AdaptFormer.
Single label
Multi label
VOC2012
COCO-Stuff10k
CityScapes
a a 0.5
a ==
mloU
mAce
mloU
mAcc
mloU
mAce
85.41
90.93
45.45
57.83
79.41
86.03
85.65
91.59
45.86
58.08
80.56
87.06
85.6
91.39
46.00
57.75
80.89
87.40
85.94
92.05
46.05
58.33
81.31
87.92
Conversely, the images in CityScapes typically feature complete traffic scenes with relatively fixed semantic categories, limiting the diversity of image-level labels. Hence, fine- graingr intermediate tasks are preferred. When examining supervision diversity in Table 3 observe that the multi- label intermediate task consistently outperforms the single-label task. This advantage may stem from the single-label task's focus perceiving only the most frequent semantic categories within a region, therehy neglecting other important semantic information and complicating adaptation to downstream pixel-level segmentation tasks.
Overall, the optimal performance is achieved when using a combination of image-level perception, where the pooling window size is equal the input image size (640 640) and multi-label supervision, with the smoothing factor set to 1. Based on this empirical observation, we adopt this combination our framework.
4.4 GENERALIZATION EVALUATION.
To evaluate the generalization ability of the proposed ProPETL, we evaluate with different pre- trained models and downstream tasks. In these comparisons, consider AdaptFormer, one of the state-of-the-art methods, as a counterpart since shares similar framework with ProPETL, excluding the progressive paradigm.
Pre-trained models. To assess ProPETL's versatility across diverse pre-training paradigms, we ex- tend the evaluation models with self-supervised pre-training. Specifically, Tablen results for MAE, a self-supervised model based on Masked Image Modeling (MIM) pre-training, while Table 5 shows results for MoCo-v3, self-supervised model that utilizes contrastive learning. As shown in Tables 4 and 5 even when using a less powerful backbone such as ViT-B/16 (Dosovitskiy al. 2020). ProPETL still significantly outperforms its counterparts. Notably, on the challenging CityScapes dataset, ProPETL. surpasses AdaptFormer by a large margin of 7% mloU based on MAE, demonstrating its robust generalization ability across different pre-trained models. Additionally, we evaluate SAM (Kirillov al. 2023), model specifically pre-trained for mask segmentation. As shown in Table 6, ProPETL consistently outperforms AdaptFormer and the Freeze baseline when using the SAM backbone, although it slightly lags behind full fine-tuning in certain cases.
Downstream tasks. When addressing a more complex segmentation task, i.e. instance segmenta- tion. we further caleulate the average precision on CityScapes in Table The improvements even more pronounced in this context, with ProPETL substantially outperforming the counterparts, highlighting its potential for broader application in other challenging computer vision tasks.

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of results using ViT-B/16 pre-trained by MAE in a supervspe manner as the backbone and UperNet the segmenta- tion head.
Table 5: Comparison of results using ViT-B/16 pre-trained by MoCo-v3 in self-supervised manner as the backbone and UperNet the seg- mentation head.
COCO-Stuff10k CityScapes Param.
COCO-Stuff10k CityScapes Param. (M)
mloU mAcc (M)
Method
mloU
mAce
Method
mloU
mAcc
mloU mAcc
Full ft.
39.64
51.64
80.87 87.79 87.02
Full ft.
18,00
26.05
64.58 73.26 85.84
Freeze
28.94
39.58
62.22 69.96
Freeze
10.10
15.60
42.02 48.47
AdaptFormer
34.42
45.60
70.99 78.75
1.19
AdaptFormer 14.69
22.33
54.93 62.86
1.19
ProPETL
37.03
49.19
77.75 85.25
1.19
ProPETL 15.48
23.04
58.83 67.00
1.19
Table 6: Comparison of results using SAM-
Table 7: Instance segmentation results using super- vised pre-trained Swin-L. as the backbone and Mask R-CNN the segmentation head.
Large as the backbone and Mask2Former as the segmentation head.
CityScapes
Param. (M)
CityScapes
Param. (M)
Method
mloU
mAce
Method
Apbox
AP
Apmark
AP
Full ft.
82.38
89.96
305.58
Full ft.
41.60
69.40
38.20
64,50
195.00
Freeze
67.04
79.31
Freeze
9.60
21.70
8.80
18.80
0
AdaptFormer
79.02
88,05
6.32
AdaptFormer
23.10
45.80
21.50
42.00
2.65
ProPETL
80.24
89.10
6.32
ProPETL
30.00
58.50
29.30
54.00
2.65
CONCLUSION
In this paper, we propose ProPETL, a progressive PETL paradigm designed to adapt pre-trained models to downstream segmentation tasks. By introducing an additional training phase in the fine- tuning process, ProPETL successfully enhances fine-grained perception ability and improves per- formance on downstream segmentation tasks. Various progressive adaptation strategies and inter- mediate task designs are comprehensively explored to achieve optimal transfer effectiveness. Fur- thermore, extensive comparisons demonstrate that ProPETL delivers superior performance across benchmarks, underscoring its effectiveness and superiority.
Limitations and Future Works. While ProPETL demonstrates significant improvements perfor- mance, it has several limitations. On the one hand, while the midstream adaptation phase enhances fine-tuning performance, it does increase the overall computational cost of training Exploring methods to shorten the global fine-tuning process and further improve fine-tuning efficiency remains an open challenge. On the other hand, the current design of intermediate tasks primarily relies on empirical studies. Developing dynamic approaches for designing intermediate tasks and generaliz- ing ProPETL to accommodate a broader range pre-trained models and downstream scenarios are promising directions for future research.
Acknowledgments. The authors sincerely thank the anonymous reviewers for their insightful com- ments and constructive suggestions. This work is partly supported by the National Natural Science Foundation of China (82441024), the Research Program of State Key Laboratory of Critical Soft- ware Environment, and the Fundamental Research Funds for the Central Universities.
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
Shoufa Chen, GE Chongjian, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022.
Please refer Table 8 and Table
9 in the Appendix for more analysis.
10

---

Published as a conference paper at ICLR 2025
Tianrun Chen, Lanyun Zhu, Chaotao Deng, Runlong Cao, Yan Wang, Shangzhan Zhang, Zejian Li, Lingyun Sun, Ying Zang, and Papa Mao. Sam-adapter: Adapting segment anything in underper- formed scenes. In ICCV, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint (xXi:20003042297 2020b.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR. 2022.
MMSegmentation Contributors, Mmsegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Rajshekhar Das, Yonatan Dukler, Avinash Ravichandran, and Ashwin Swaminathan. Learning ex- pressive prompting with residuals for vision transformers. In CVPR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Wei Dong, Xing Zhang, Bihui Chen, Dawei Yan, Zhijun Lin, Qingsen Yan, Peng Wang, and Yang Yang. Low-rank rescaled vision transformer fine-tuning: A residual design approach. In CVPR, 2024a.
Yue-Jiang Dong. Yuan-Chen Guo, Ying-Tian Liu, Fang-Lue Zhang, and Song-Hai Zhang. Ppea- depth: Progressive parameter-efficien adaptation for self-supervise monocular depth estimation. In AAAI, 2024b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: transformers for image recognition at scale. In ICLR, 2020.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang. Siyuan Qi. and Dongfang Liu. E. 2vpt: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity- visual parameter-efficient fine-tuning. In ICCV, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Han-Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung-Yu Yu Tseng, Maneesh Singh, and Ming-Hsuan Yang. Progressive domain adaptation for object detection. In WACV, 2020.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In ICCV, 2023.
11

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719,, 2023.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1X06::600.04000 2016.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Zhixiang Wei, Lin Chen, Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In CVPR, 2024.
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In ICML, 2018.
Junde Wu, Wei Ji. Yuanpei Liu, Huazhu Fu, Min Xu, Yanwu Xu, and Yueming Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304. / /2620, 2023.
Tete Xiao, Yingeheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Dongshuo Yin, Yiran Yang, Zhechao Wang, Hongfeng Yu, Kaiwen Wei, and Xian Sun. 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. In CVPR, 2023.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: simple parameter-efficient fine-tuning for transformer-based masked language-modeh. In ACL, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
Nan Zhou, Jiaxin Chen, and Di Huang. ivpt: Improving telk-relevant information sharing in visual prompt uning by cross-layer dynamic connection. arXiv preprint (00.::22404.022007,. 2024.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
In this section, we present an analysis of computational overhead, visualization results, and complete ablation study results on intermediate task design. Additionally, we provide extended evaluations using a broader range of pre-trained models and segmentation heads.
Computational Overhead. We compare the computational overhead including training time, in- ference time, inference workload, and GPU memory footprint- of the proposed ProPETI. against its counterparts Table To ensure a fair comparison, we use Swin-L. the backbone and train all methods for 80,000 iterations with a batch size of 16 on two NVIDIA A800 GPU. ProPETL con- sumes only 55GB GPU memory during training and 627 GFLOPs during inference, significantly less than both VPT and LoRA. While its GPU memory usage and parameter count slightly higher than AdaptFormer's s, ProPETL outperforms substantially, as shown in Table [ To further reduce computational costs, introduce a lighter version of ProPETL that minimizes the fine-tuning pro- cess to achieve the shortest training time. The comparisons are detailed in Table
Table 8: Comparison of the computational overhead. indicates the short training version. Note: reparameterization for LoRA was not implemented.
Training time. (Hour)
Inference time. (Millisecond)
Workload + (GFLOPs)
GPU Memory, (MB)
Param...
Method
(M)
Full fine-tuning
41
182
623,310
94,198
195.00
VPT
96
210
872.222
135,498
3.61
LoRA
30
208
718.097
94,564
4.55
AdaptFormer
185
627.085
51,433
2.64
ProPETL
47
191
627.111
55,613
3.30
ProPETL'
29
191
627.111
55,613
3.30
As shown in Table 9 although AdaptFormer uses the same total number of iterations as ProPETI. the latter significantly outperforms AdaptFormer on both the COCO-Stuff1Ok and CitySeapes extend AdaptFormer's
benchmarks, thanks to its progressive adaptation strategy. Additionally, iterations to 120k for a more comprehensive comparison with ProPETL. However, this extension only slightly improves AdaptFormer's performance, it has already saturated in mitigating the perceptual granularity shifts toward downstream tasks. In contrast, ProPETL shows substantial per- formance improvements, its midstream task convergence enables pre-trained models better adapt to downstream segmentation scenarios, further demonstrating the effectiveness of the pro- posed strategy.
Table 9: Ablation study on the training iterations. "!" indicates the short training version.
COCO-StuffiOk
CityScapes
Midstream Adaptation
Downstream Fine-tuning
Total
Method
Iterations
mloU
mAce
mloU
mAcc
AdaptFormer
45.45
57.83
79.41
86.03
N/A
80,000
80,000
ProPETL" 45.98
58.20
80.93
87.51
40,000
40,000
80.000
AdaptFormer 45.81 46.48
58.13
79.78
86.42
N/A 40,000
120,000
120,000
ProPETL
58.69
81.67
88.23
80,000
120,000
More Generalization Evaluation. We evaluate the effectiveness of ProPETL across more pre- trained models and segmentation heads. In the main body, we report results using pre-trained Swin- transformer-large as the backbone. Additionally, Table 10 presents an analysis of ProPETL on the VOC2012 dataset using different sizes of Swin Transformers. These results consistently show that ProPETL outperforms both AdaptFormer and full fine-tuning, underscoring its effectiveness even with smaller pre-trained models. To further assess ProPETL's versatility across diverse pre- training paradigms, we extend the evaluation to models with self-supervised pre-training. Table TT] reports results using DINOv2, self-supervised model built upon an extended contrastive learning framework. For segmentation heads, we evaluate the performance of Fask2Former (Cheng et al. 2022), which employs mask-based prediction strategies. Notably, Table 1T shows that when paired
13

---

Published as a conference paper at ICLR 2025
with DINOv2 as the backbone and Mask2Former as the segmentation haed, ProPETL surpasses both full fine-tuning line-tuning and AdaptFormer, further highlighting its potential as a flexible and high- performing PETL framework.
Table 10: Comparative results on VOC2012 using different size tranfrrassfsrerss with UperNet.
Swin-small
Swin-base
Swin-large
Method
mIoU
mAcc
mIoU
mAcc
mloU
mAcc
Full fine-tuning
80,83
86.90
82.07
87.63
84.38
89.82
AdaptFormer
80,94
87.26
84.29
90.31
85.41
90.83
ProPETL
81.26
87.82
85.03
90.96
86.11
92.08
Table 11: Comparative results of using DINOv2 with Mask2Former.
VOC2012
CityScapes
Param.
(M)
Method
mIoU
mAce
mloU
mAcc
Full fine-tuning 86.81
93.35
84.10
90.57
304.19
AdaptFormer
88.93
94.34
83.83
90.87
4.74
ProPETL 89.53
94.79
84.29
91.37
4.74
Cross-scenario Evaluation. We conducted cross-scenario evaluation of ProPETL under the domain generalization setting and compared against Full fine-tuning and AdaptFormer. We additionally compare Rein(Wei al. (2024). which is recent parameter efficient method for domain general- ization semantic segmentation. For a fair comparation, we utilized the DINOv2 et al. 2023) MMask2Formerrhheee Chhnn et al. (022) framework and adhered to the training and evaluation proto- cols outlined in Wei et al. (2024). The evaluation was performed on the GTAV - CityScapes setup (Richter et al., 2016] Cordts et al. 2016). As shown Table 12 ProPETL significantly outperforms its counterparts, demonstrating its generalization ability under cross-scenario task.
Table 12: Domain generalization semantic segmentation results. * results from Wei et al. 2024).
Method
CityScapes (mloU)
Param. (M)
Full fine-tuning"
63.7
304.20
Freeze
63.3
0
AdaptFormer"
64.9
3.17
Rein"
66.4
2.99
ProPETL
67.9
3.17
Intermediate Task. In the main body of the paper, we present the ablation results of intermediate tasks from the perspectives of perception granularity and supervision diversity. To mitigate pertur- bations, we average results from experiments with varying supervision diversities at each perception granularity level, and vice versa. Here, we provide the full results of intermediate tasks comprising various combinations of perception granularity and supervision diversity on the VOC2012, COCO- Stuff10k, and CityScapes datasets. The results illustrated in Table 13 As shown in this table, the combination of image-level perception and multi-label diversity yields the optimal performance, leading us adopt this configuration within our framework.
Visualizations. To further investigate the impact of the proposed ProPETL framework, we visu- alize the comparisons of normalized Euclidean distance between the feature maps before and after adaptation in Fig. 8 As illustrated in the figure, representative methods such as BitFit and Adapt- Former tend to make localized adjustments, often overlooking broader perceptual regions. Con- versely, LoRA generally implements globally consistent adjustments but may neglect challenging areas, such as edges. The feature adjustment pattern observed in VPT resembles that of full fine- tuning: however, it exhibits messy adjustment, particularly in the last row of examples. ProPETL demonstrates an adaptation pattern akin to full fine- tuning, effectively capturing semantic adjust- ments with fine granularity while also focusing on critical edge regions. This highlights the efficacy
14

---

Published as a conference paper at ICLR 2025
Table 13: Ablation study on the intermediate task using the DSA during downstream fine-tuning. "s" indicates the size of the pooling window. By adjusting the perception granularity of the in- termediate tasks ranges from patch-level (s 32) to image-level (s 640). "a" indicates the smoothing factor, respectively. By adjusting the supervision diversity of intermediate tasks tran- sitions from single label (a 0) to multi label (ar 1). The first line the result AdaptFormer.
Perception Granularity
Supervision Diversity
VOC2012
COCO-Stufri0k
CityScapes
32
128
640
0.5 a=1
mloU
mAce
mloU
mAcc
mioU
mAce
85.41
90.93
45.45
57.83
79.41
86.03
85.72
91.27
45.74
57.90
80.73
86.94
85.63
90.96
45.50
57.19
80.87
87.24
85.91
91.67
45.57
58.31
80.92
87.61
85,53
91.78
45.76
58.15
80.85
87.26
85.54
91.96
46.11
$7.96
80.89
87.56
85.79
92.39
46.09
58.00
81.35
87.92
85.70
91.71
46.09
58.20
80.09
86.99
85.87
91.26
46.40
58.11
80.91
87.39
86.11
92.08
46.48
58.69
81.67
88.23
of the proposed progressive paradigm. In Fig. 9 visualize several images and corresponding segmentation results in the VOC2012 and ADE20k dataset. Compared to AdaptFormer, ProPETL produces smoother segmentation contours (first row) and more complete object masks (second and third rows). The last two rows illustrate failure cases where ProPETL struggles with segmentation redundancies in challenging scenarios, such object occlusion (fifth row) and regions with high gnreground-backgroung similarity (last row).
Input Image
Full fine-tuning
BitFit
VPT
LoRA
AdaptFormer
ProPETL (Ours)
Figure 8: Visualization of the normalized Euclidean distance between the feature map before and after model adaptation on the ADE20k dataset.
15

---

Published as a conference paper at ICLR 2025
Input Image
Ground Truth
Full fine-tuning
AdaptFormer
ProPETL (Ours)
Figure 9: Visualization of the segmentation results on the VOC2012 and ADE20k dataset.
16


Reviews:
[{"review_id": "oNfksEtYfN", "reviewer": "Reviewer_4Nue", "summary": "The paper finds that there is a perceptual granularity gap between pre-trained models and downstream segmentation tasks, necessitating significant parameter adjustments. However, existing PETL approaches fail to effectively address this gap due to the constraint of limited trainable parameters. To tackle this issue, the paper introduces ProPETL, which incorporates an additional midstream adaptation phase to progressively align pre-trained models with downstream segmentation tasks. Specifically, the paper conducts an in-depth analysis of candidate progressive adaptation strategies, such as Generalized Parametric Adaptation and Decoupled Structured Adaptation, as well as intermediate tasks designed to enhance the perception of finer granularity and diverse classes. Notably, ProPETL achieves state-of-the-art performance on most segmentation benchmarks, surpassing even full fine-tuning.", "questions": "Could the intermediate tasks be considered a form of data augmentation? Would performance improve if these intermediate tasks were integrated into the fine-tuning phase? For example, scaling the images to different sizes and generating varying numbers of categories at random ratios during fine-tuning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the gap between pre-trained models and downstream segmentation tasks by introducing an additional midstream adaptation process, while maintaining parameter efficiency.\n2. The method demonstrates its superiority by outperforming other parameter-efficient approaches and even surpassing full fine-tuning.", "weaknesses": "1. The paper is motivated by the observation that the perceptual granularity gap between pre-trained models and downstream segmentation tasks necessitates significant parameter adjustments. However, for models like SAM[1], which are pre-trained on segmentation tasks, there is a lack of evidence to support that these observations still hold. \n2. The necessity of additional midstream adaptation is not entirely clear, as the proposed intermediate tasks resemble data augmentation, which could potentially be integrated into the fine-tuning phase.\n\n[1] Kirillov A, Mintun E, Ravi N, et al. Segment anything[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4015-4026", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SkV7pq5FbW", "reviewer": "Reviewer_GSCi", "summary": "The paper presents ProPETL, a new method to improve Parameter Efficient Transfer Learning (PETL) for semantic segmentation tasks, which usually have more difficulties in fine-tuning compared to classification tasks. Current PETL methods find it hard because they need to adjust more parameters to handle changes in the details of semantic understanding. ProPETL solves this problem by adding a midstream adaptation step, which gradually aligns the pre-trained models for segmentation tasks. This proposed method achieves the state-of-the-art performance, even better than full fine-tuning, on various evaluation benchmarks.", "questions": "1. The paper shows that ProPETL works well with a Swin-L model pre-trained on ImageNet-21K, but how would the performance change if we use smaller or less powerful pre-trained models? Is there any guideline or expectation for using ProPETL in situations with limited resources? Would ProPETL still be helpful in transfer learning cases that use self-supervised models with lower representational capacity? It would be good to see more discussion or results about how much ProPETL depends on the quality of pre-trained models.\n\n2. The paper says that the midstream adaptation improves segmentation performance, but how does the computational cost compare with traditional PETL methods, in terms of training time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors provide a detailed analysis of why existing PETL methods do not perform effectively for semantic segmentation tasks. This paper not only introduces ProPETL but also thoroughly tests its effectiveness compared to state-of-the-art methods across multiple benchmark datasets. The authors conduct extensive ablation studies to show the effect of each component of the method, which supports the strength of the proposed contributions.\n\n2. The paper is well-written and organized clearly, making the paper easy to understand. The authors include background information and visualize to illustrate the weaknesses of current PETL methods and the strength of ProPETL. \n\n3. ProPETL outperforms full fine-tuning on various benchmarks, showing that parameter-efficient methods can match or even exceed traditional methods.", "weaknesses": "1. The proposed midstream adaptation phase makes the whole framework more complex than previous methods. Selecting intermediate tasks, such as perception granularity and supervision diversity, looks like it’s based on trial and error without a clear, systematic way for users to apply these choices to new datasets or tasks.\n\n2. While the progressive adaptation process improves segmentation performance, it may need more computational time because of the extra midstream phase, especially for large datasets. The paper claims that this extra cost is worth it for better performance, but having more exact comparisons of training time or resource usage with other PETL methods would help show the practical trade-offs, especially when computational efficiency is very important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sJLM1x7o4O", "reviewer": "Reviewer_fzgQ", "summary": "This paper presents ProPETL framework, which applies progressive adaptation to Parameter-Efficient Transfer Learning (PETL) and introduces a novel midstream adaptation stage to address perception granularity differences and supervision diversity in semantic segmentation tasks.", "questions": "1. DSA divides the adapter parameters into two independent parts: the parameters for the midstream adaptation stage( $\\phi$) and the parameters for the downstream fine-tuning stage \\($ \\psi$ \\). During the intermediate adaptation phase, DSA only optimizes $\\phi$, while in the downstream fine-tuning phase, $\\phi$ remains frozen, and only $ \\psi$ is optimized. However, the strategy of freezing  $\\phi$ during the downstream task, which reduces the parameter space that can be optimized and increases the burden on optimizing  $ \\psi$  , leading the model to seek the best solution in a smaller space. For complex downstream tasks, this limitation may affect the model's  performance, as the optimization of  $ \\psi$ cannot fully replace the tuning of  $\\phi$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper analyzes two candidate progressive adaptation strategies: Generalized Parametric Adaptation (GPA) and Decoupled Structured Adaptation (DSA), and employs DSA to effectively mitigate the \"forgetting\" issue observed in GPA. \n\nExtensive comparisons demonstrate that ProPETL delivers superior performance across benchmarks, underscoring its effectiveness and superiority.", "weaknesses": "1. Both Generalized Parametric Adaptation (GPA) and Decoupled Structured Adaptation (DSA) implement dimensionality reduction and expansion through bypass branches to reduce the number of parameters. However, the fixed reduction ratio $r$ may lead to the loss of critical feature information, especially in tasks with high perceptual demands (such as semantic segmentation). The fixed ratio of the reduction operation lacks adaptability to the importance of features and the complexity of input data, which may result in the bypass branch being unable to effectively retain fine-grained information, thus affecting the overall performance of the model.\n\n2. This paper suggests that Perception Granularity and Supervision Diversity are key factors in the design of intermediate tasks, but it lacks theoretical support for these design choices. For example, it does not explain why image-level perception granularity and multi-label supervision can enhance the fine-grained performance of downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DgaJIB7cYf", "reviewer": "Reviewer_Fur2", "summary": "This paper proposes a framework for progressively adapting pre-trained models to downstream semantic segmentation tasks through intermediate tasks, where perception granularity and supervision diversity are interesting and novel. Although this work has some weaknesses in descriptions and experiments, its overall contribution benefits the PETL community, especially in fine-tuning VFMs from classification tasks to semantic segmentation tasks. Thus, I recommend the weak acceptance of this paper and hope the authors' responses.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivations of the proposed approach are clear.\n2. The intermediate task designation, including perception granularity and supervision diversity, is interesting and novel.\n3. The proposed approach achieves better performance on multiple benchmarks.", "weaknesses": "1. The evaluations of the generalization ability are insufficient. ProPETL demonstrates strong generalization across various pre-trained models and scenarios. However, the experimental evaluation does not show the proposed ProPETL's generalization ability under cross-scenario tasks, such as the domain generalization setting. I am interested in the proposed ProPETL's generalization ability under cross-scenario tasks.\n2. The descriptions of the proposed approach are unclear. For example, the summary of the overall training and inference process and the design of the GPA's architecture in intuition or theory.\n3. Do α of supervision diversity and pooling window of perception granularity correspond one to one?\n4. The computational cost of the proposed approach during the training and inference stages should be introduced and compared with other fine-tuning strategies in detail, such as the total training time, inference time, FLOPs, GPU Memory, Storage, etc.\n5. More backbones, such as DINOv2 and EVA02, and more segmentation heads, such as Mask2Former and SemFPN, should be included to verify the effectiveness of the proposed approach.\n6. How to evaluate and compare to the Rein [1] which also introduces the PETL method for semantic segmentation tasks.\n[1] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation. CVPR 2024.\n7. The fail examples, limitations, and future works should be provided and discussed.\n8. Figure 1 (a) and (b) should be more elegant. Labels of the x-axis are too narrow.\n9. The code should be released.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)