Decoupled Finetuning for Domain Generalizable Semantic Segmentation
Authors: Jaehyun Pahk
Donghyeon Kwon
Seong Joon Oh
Suha Kwak
Venue: ICLR 2025 Poster (2025)
Keywords: Domain Generalization
Decoupled Optimization
Domain Generalizable Semantic Segmentation
Robustness
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Joint finetuning of a pretrained encoder and a randomly initialized decoder has been the de facto standard in semantic segmentation, but the vulnerability of this approach to domain shift has not been studied. We investigate the vulnerability issue of joint finetuning, and propose a novel finetuning framework called Decoupled FineTuning (DeFT) for domain generalization as a solution. DeFT operates in two stages. Its first stage warms up the decoder with the frozen, pretrained encoder so that the decoder learns task-relevant knowledge while the encoder preserves its generalizable features. In the second stage, it decouples finetuning of the encoder and decoder into two pathways, each of which concatenates an adaptive component (AC) and retentive component (RC); the encoder and decoder play
 different roles between AC and RC in different pathways. ACs are updated by gradients of the loss on the source domain, while RCs are updated by exponential moving average biased toward their initialization to retain their generalization capability. By the two separate optimization pathways with opposite AC-RC configurations, DeFT reduces the number of learnable parameters virtually, and decreases the distance between learned parameters and their initialization, leading to improved generalization capability. DeFT significantly outperformed existing methods in various domain shift scenarios, and its performance was further boosted by incorporating a simple distance regularization.

Published as a conference paper at ICLR 2025
DECOUPLED FINETUNING FOR DOMAIN GENERALIZ- ABLE SEMANTIC SEGMENTATION
Jaehyun Pahk' Dept. of CSE, POSTECH
Donghyeon Kwon'
Seong Joon Oh
Suha Kwak 1.2
GSAI, POSTECH²
Tübingen Al Center, Universität Tübingen*
ABSTRACT
Joint finetuning of pretrained encoder and a randomly initialized decoder has been the de facto standard in semantic segmentation, but the vulnerability of this approach to domain shift has not been studied. We investigate the vulnerability issue of joint finetuning, and propose a novel finctuning framework called Decou- pled Fine'Tuning (DeFT) for domain generalization in two stages. Its first stage warms up the decoder with the frozen, pretrained encoder that the decoder learns task-relevant knowledge while the encoder pre- serves its generalizable features. In the second stage, it decouples finetuning of the encoder and decoder into two pathways, each of which concatenates component (AC) and retentive component (RC); the encoder and decoder play different roles between AC and RC in different pathways. ACs are updated by gradients of the loss on the source domain, while RCs are updated by exponen- tial moving average biased toward their initialization retain their generaliza- tion capability. By the two separate optimization pathways with opposite AC-RC configurations, DeFT reduces the number of learnable parameters virtually, and decreases the distance between learned parameters and their initialization, lead- ing to improved generalization capability. DeFT significantly outperformed ex- isting methods in various domain shift scenarios, and its performance was further boosted by incorporating a simple distance regularization.
solution. DeFT operates
adaptive
INTRODUCTION
The current de facto standard for learning semantic segmentation is to jointly finetune a pretrained encoder and a segmentation decoder on training data with segmentation labels (Logg et al.) 2015)
Noh et al., 2015; Ronneberger et al. 2015 Chen et al. 2017 Zhao et al. 2017 Xie et al. 2021
Yu et al. 2022 Chen et al. (023). This practice allows significant performance improvement, but also often leads to models vulnerable to domain shift in testing caused by, for example, weather conditions and geolocations they do not experience in training (Ganin al. [2016] Pan al. [018]] Saito et al.] 2018 Yue et al. 2019a] Choi et al. (021). A straightforward solution to this issue is collect vast amount of training data from diverse domains. However, this does not guarantee
that the collected data cover any potential test domains, and more importantly, pixel-wise class annotation for such data will be prohibitively expensive.
To resolve this issue, we study domain generalization for semantic segmentation, i.e., learning a model on a single source domain that generalizes well to unseen, arbitrary target domains that may arise in testing (uue et al.) 2019b Lee et al., 2022 Zhao et al. 2022] Chattopadhyay et al,, 2023 Kim et al. 2023a). large body of domain generalization research has focused on simulating
diverse target domains by data or feature augmentation during training (Lee et al. 2022: Zhao et al.
2022 Chattopadhyay et al. (023),, or removing domain-specific information from features (Choil al. 2021; Peng al. 2022 Pan et al. 2018: 2019). Although these approaches have driven remarkable success, there is still large room for further improvement in that they do not take into ac- count potential negative impact of the joint finetuning of encoder and decoder, the common practice in semantic segmentation, on domain generalization.
We argue that the joint finetuning of encoder and decoder degrade the model's generalization capability. First, the joint finetuning causes the pretrained encoder to overfit to the source domain and thus corrupts its generalization capability (Kumar et al. 2022 Saito et al. 2023). Also, the

---

Published as a conference paper at ICLR 2025
Joint finetuning
Freezing encoder
Freezing decoder
DeFT
Source Loss (GTAV)
Target Loss (Cityscapes)
loss (log scale)
-0.5
-1.0
loss
-1.5
-2.0
10K
20K
30K
10K
30K
iteration
iteration
(a)
We empirically verify that the joint finetuning causes overfitting to the source domain and degrades the generalization capability by comparing (a) loss on the source domain-GTAY (Richter) ct al. 2016) and (b) that on an unseen target domain-Cityscapss (Cordis et al. 2016) during fine- tuning. Freezing either the encoder or decoder before being overfitted mitigates the issue to some extent, which suggests that preventing one them from being trained with the other overfitted the source domain may improve the entire model's generalization. Of course, this approach is far from the optimal solution due to the lack of task-relevant knowledge of the frozen module, as demon- strated in Table TT Meanwhile, DeFT demonstrates significantly better generalization capability: its loss on the unseen target domain decreases more quickly and reliably during finetuning.
Figure 1:
Table I: Comparison between the proposed framework, DeFT, and freezing either an encoder or decoder. All the methods were trained on GTAV using ResNet-50 backbone. After the warm-up stage, we selectively froze either the encoder or decoder while continuing to update the other. The results indicate that the model cannot learn sufficient task-relevant knowledge when one module is not updated at all, especially the encoder, which contains most of the parameters of the model.
Method
Cityscapes
BDD 100K
Mapillary
Avg.
Freezing Encoder
41.88
36.85
44.59
41.11
Freezing Decoder
42.52
38.75
45.03
42.10
DeFT
50.06
43.17
50.51
47.91
decoder relying on the encoder's output inevitably draws distorted decision boundaries, producing gradients that cause the encoder be more overfitted. Figure empirically verifies this argument.
Building on this insight, we propose a new, simple yet effective finetuning framework dubbed Decoupled FineTTuning for domain generalization (DeFT). DeFT comprises two stages. In the first stage, the decoder is warmed up with a frozen pretrained encoder, following Kumar et al. (2022). By warming up the source domain, the decoder learns the target task (i.e., semantic segmentation in this paper) without distorting the generalizable knowledge of the pretrained encoder.
The main contribution of DeFT lies its second stage, which finetunes both the encoder and de- coder in a decoupled manner. Motivated by the observation in Figure I we propose decoupling the finetuning of the two trainable modules in the model, the encoder and decoder. To this end, we employ two parallel encoder-decodee pathways for finetuning, each combining an adaptive compo- nent (AC) and retentive component (RC). AC is updated using the standard error backpropagation based on the training loss from the source domain, while RC is not updated using the gradients that might be overfitted to the source domain. As result, RC maintains its generalization capability during finetuning, guiding the coupled AC's updates using its generalizable knowledge. Note that the encoder and decoder play different roles between AC and RC in different pathways.
A naive strategy for managing RCs is not updating them at all, which however leads to suboptimal solution due to the lack of task-relevant knowledge in RCs as demonstrated Table [ To allow RCs to learn task-relevant knowledge while preserving their generalization capability, variant of exponential moving average (EMA) (Tarvainen & Valpolal (017)) that is biased towards the model's initial parameters, as update scheme for RCs. This EMA method assigns higher
adopt
2

---

Published as a conference paper at ICLR 2025
Wp
Finetuning
Inference
piu
e
Ca Y1
f
d
C.
Initialize
62
d2
f2
é
d
pi)
f2
f2
AC
RC
Forward Pass
Backward Pass
Inference Pass
Frozen Weights
Figure 2: An overview of DeFT. The first step of DeFT is to warm up the decoder with the frozen,
pretrained encoder. After warming up the decoder, the decoupled finetuning is conducted through two parallel encoder-decoder pathways. In the pathways, the parameters of encoders and decoders initialized with those from the warmed-up model. In the second step, DeFT finetunes both the encoder and decoder in decoupled manner: the RCs (d, and êτ) are updated by the exponential moving average of their counterpart ACs (d2 and e1), ) while the ACs are updated by gradients of the loss. Our final model for inference is configured as the combination of the well-generalized RCs (di and ē2), i.e., final = d1 ē2.
weights the early parameters during the finetuning, performing gradual temporal ensemble in the parameter space.
At the end of finetuning, DeFT produces RC-AC pairs. Since RCs better preserve the rich and generalizable knowledge from pretraining and thus have better generalization capability than AC, we set our final model as the combination two RCs, i.e. EMA encoder and EMA decoder. The overall pipeline of DeFT, including each stage, is illustrated in Figure 2
Our method was evaluated on five different datasets, Cityscapes (Cordts al. 2016). BDD- 100K Yu et al. 2020), Mapillary Neuhold et al. (2017), GTAV (Richter et al.) 2016) and SYN- THIA (Ros et al. 2016), and it demonstrated superior performance to previous work every exper- iment. In summary,
contribution is three-fold:
We empirically demonstrate that joint finetuning of the encoder and decoder degrades gener- alization performance, and simply decoupling them in the finetuning process can substantially improve the performance.
We propose novel training framework for domain generalizable semantic segmentation, dubbed as DeFT, which finetunes the encoder and decoder in decoupled manner. We also provide detailed analysis of our method through extensive experiments.
DeFT was evaluated on various domain shift scenarios using multiple semantic segmentation datasets, where it outperformed previous work by large margins in all evaluations.
2 RELATED WORK
2.1 DOMAIN GENERALIZABLE SEMANTIC SEGMENTATION
The objective of domain generalization is to develop models that generalize well to unseen do- mains (Muandet et al. 2013 Li al. 2018b). Early methods (Lietal. 2017: 2018 8a Pan et al. 2018 Nam et al.) 2021; Zhou et al. (021)) primarily focused on classification tasks. Recently, sigmificatt
2017:2018a
progress has been made in semantic segmentation (Yue al. 2019h Lee al. 2022 Zhao al.

---

Published as a conference paper at ICLR 2025
2022
Chattopadhyay al., 2023 Kim et al. 2023a). Yue et al. (2019b) suggest learning features
invariant to random style variations in the input. Methods like Lee et al. (0022: Zhao et al. (2022) simulate diverse style spaces by manipulating the channel-wise means and standard deviations of features. Additionally, (Chattopadhyay et al.] (2023) introduced a frequency-domate randomization technique, particularly for strong augmentation (2023a) collected variety of images with different styles from web repositories to enhance gener- alization performance. Also, some recent work explores alternative model architectures for the pur- pose. For instance, Ding et al. (2023) propose HGFormer, hierarchical grouping transformer that integrates both local and global feature interactions to improve generalization in unseen domains. Luo et al. 2024) demonstrate that network pruning can enhance domain generalization by reducing model complexity and increasing robustness. DAFormer (Hoyer al. 2022) introduces domain- adaptive semantic segmentation by incorporating architectural refinements and training strategies that enhance robustness across diverse domains. Meanwhile, DGInStyle (Jia et al., (024) employs image diffusion models to generate diverse stylized versions of training images, simulating various domain shifts; this approach enables models to learn domain-invariant features, effectively improv- ing their robustness to diverse input distributions during inference.
high-frequency regions. Furthermore, Kim et al.
However, the listed methods still adopt joint finetuning, despite its negative impact on generalization ability, which remains common practice in existing frameworks. To address this generalization issue in previous work, we propose a novel decoupled finetuning strategy called DeFT, and present dedicated training algorithm based on EMA.
2.2 WEIGHT AVERAGING FOR MODEL ENSEMBLING
Weight averaging has been explored as an effective method for leveraging the historical training trajectories of deep neural networks to improve generalization performance. Snapshot ensem- bling (SSE) (Huang et al. 2017) and fast geometric ensembling (FGE) (Garipov et al.) 2018) were early attempts to utilize weight trajectories from historical training by employing cyclic learning rates guide the learning process through multiple local minima, which are then saved as ensemble members. Building on FGE, stochastic weight averaging (SWA) (Izmailov et al. 2018) updates pretrained model using a cyelical or high constant learning rate, gathers model parameters during training, and averages them to form model ensemble. Extending from SWA, trainable weight averaging (TWA) (Li et al. 2023) introduced a technique that allows for weight averaging with adjustable coefficients. Additionally, model soups (Wortsman et al.] 2022a) demonstrated that av- eraging the weights of multiple models finetuned with different hyperparameters can improve both accuracy and robustness.
We adopt a variant of EMA as temporal ensembling in the model parameter space for DeFT, allowing the model to learn task-relevant knowledge while minimizing overfitting to the source domain.
2.3 ROBUST FINETUNING FOR OUT-OF-DOMAID GENERALIZATION
Robust finetuning using pretrained weights enhances out-of-dommin (OOD) performance. Recent studies (Wen et al. 2021 Gouk et al., (202)) demonstrate that leveraging pretrained models can sig- nificantly booster robustness on OOD datasets. Moreover, the finetuning process plays critical role in improving OOD generalization capability. Research (Nagarajan & Kolter 2019 Lin & Zhang 2019 Gouk et al. 202T Li & Zhang 202T) indicates that generalization performance is affected by the distance between the initial and finetuned models: as this distance increases, generaliza- tion tends decline. WiSE-FT (Wortsman al., 20226) shows significant improvements in OOD generalization by linearly interpolating pretrained weights with finetuned ones during inference. LP- FT (Kumar et al.) (022)) demonstrates that simultaneously finetuning both the final linear layer and the feature backbone can distort pretrained features, and proposes two-stage training strategy: first, warming up the decoder while freezing the encoder, then finetuning the entire network. Lastly, Tian et al. (2023) introduces per-layer regularization, which automatically learns constraints for more accurate finetuning.
Our method adopts two-step training strategy motivated by LP-FT, but it is clearly different from LP-FT: after warming up the decoder, we finetune encoder and decoder in a disjoint manner with the proposed DeFT' framework, rather than jointly finetuning them as in I.P-FT.
4

---

Published as a conference paper at ICLR 2025
3 METHOD
We consider training a domain--gnneralzze segmentation model, f = e, with an encoder and decoder d using labeled images from a single source domain, where the image height, width, and the number of semantic classes are denoted by h, w, and c, respectively. Our framework, dubbed DeFT, consists of two stages: warming up the decoder while freezing the pretrained encoder (Section 3.1 followed by decoupled finetuning of the encoder and decoder (Section 3.2
3.1 WARMING UP DECODER WITH PRETRAINED ENCODER
The first step of DeFT is to warm up the decoder on the source domain dataset using the pretrained encoder that remains frozen. For an input image and its ground truth Y, let s(X) = (X)) be the class probability map, where P E Rxxwxx Let Lee (P, Y) denote a standard pixel-wise cross- entropy loss, which is given by:
h.w
(o.) h Σ Y, log(P.).
(1)
where i is the pixel index and Y, is the one-hot vector of the ground truth for pixel i. Then, the weights of the randomly initialized decoder are updated using gradients of the cross-entropy loss fee (P. Y), while the pretrained encoder is frozen. This warming up stage enables the decoder to learn the target task without distorting the generalizable knowledge of the pretrained encoder.
3.2 DECOUPLED FINETUNING OF ENCODER AND DECODER
After warming up the decoder, the encoder and decoder are fine-tuned a decoupled manner. Dur- ing decoupled finetuning, they are assigned one of two components, the retentive component (RC) and the adaptive component (AC), but are different from each other. Then AC updated using gra- dients of the training loss from the source domain, while RC is updated by exponential moving average (EMA) scheme to retain its generalization capability.
To decouple the encoder and decoder then finetune both, define two distinct pathways, which can be represented as two segmentation models fi and f2 that share the same architecture but have opposing configurations for AC and RC. Let f1 = d1 61 and fz = d2 o ê2 be the segmentation models, where dj and &2 are the RCs, and d2 and e are the ACs. The weights of ACs, d2 and e1, are updated using the cross-entropy loss Lee Y) and (c(2),,
where p(1) = fX) and p(2) = f2(X) are the predictions of fi and f2, respectively. On the other hand, RC one model is not updated by the cross entropy loss but by the exponential moving average of AC in the other, with an update ratio 3:
Øt 30 + 3)-3) 0ct 30 + -))0
where 0% and 0r the weights of d, and ée at the t-th iteration, respectively. Updates of ACs and RCs are conducted simultancously during the second stage.
3.3 OUR FINAL MODEL FOR INFERENCE
DeFT produces two encoders, E and ε2, and two decoders, di and d2. As the two RCs better preserve their initial states which are more generalizable and thus have superior generalization pability compared to the ACs, we set our final model as the combination the two RCs from fi and f2: finat = dj ě2. Since each RC from different pathways is updated based on its counterpart AC in Eq. (2). their feature distributions are implicitly aligned, properly adapting each other.
3.4 EMPIRICAL JUSTIFICATION FOR DEFT
The decoupled finetuning improves the model's generalization capability by enabling the encoder and decoder to trained independently, each benefiting from less-overfitted decision boundaries features derived from their respective generalized counterparts. Moreover, tighten the
5

---

Published as a conference paper at ICLR 2025
Joint finetuning
DeFT
Distance (L.2)
Distance (MARS)
Distanee (OP)
distance
distance
distance
400
400
4K
200
2k
20k iteration
20k iteration
iteration
(a)
(b)
(c)
Figure 3: We empirically demonstrate that the model jointly finetuned tends to move further away
from its initial parameters than the model finetuned with DeFT. We measured the distance between the current and initial parameters of each model using three metries: (a) L2 norm (Nagarajan & Kolter, 2019), (b) MARS norm (Gouk et al. 2021), and (c) operator norm (Long & Sedghil 2020). The final model ffnnal = d, 02 was used to measure the distance for DeFT. The results show that the model finetuned with DeFT exhibits a shorter distance from its initial parameters than the jointly finetuned one, resulting in better generalization performance.
generalization bounds the model, as it reduces both the number of parameters to be optimized each pathway and the distance from initial parameters, i.e., the distance between the learned parameters and their initial values.
DeFT divides single optimization objective of joint finetuning, which handles all the parameters in the model simultancously, into two separate optimization objectives: one for the encoder and one for the decoder. As result, each of the networks f1 and f2 is trained on a separate objective with fewer parameters to optimize compared to the original model. This leads to tighter generalization bound for each module than the joint finetuning (Du al. 2018b Long & Sedghi 2020).
We also empirically demonstrate that DeFT reduces the distance from initial parameters, which is one of the critical factors for the model's generalization bounds (Nagarajan Kolter 2019 Long & (edghii 2020) Gouk et al. 2021 Li & Zhang 2021). Thanks to the EMA update scheme in DeFT, RCs are updated to maintain their initial states, partially retaining their initial weights. RCs, ég and d,, which comprise our final model, exhibit a shorter distance from their initialization as shown in Figure resulting in better domain generalization capability.
Moreover, we compare DeFT to other methods which explicitly regularize the distance from initial parameters: using low learning rates and adding distance regularization into optimization objec- tive. The results in Table 2 show that simply reducing the learning rate does not always lead to performance improvement, rather degrading its generalization performance due to the increased the risk of falling in local minima. For the distance regularization, we add a regularization term to the optimization objective regularize the sum of the Euclidean distances from initial parameters. Let be the number of layers in the model, and, W₁ and w( be weights of the model and its initial parameters of the 1-th layer. Then the revised training loss after warming up the decoder is given by:
Lirining
Lee + Ot Σ Ilw, -W:
where || denotes the Frobenius norm for matrices and the Euclidean norm for vectors, respectively. The results reported in Table 3 demonstrate that DeFT outperforms joint finetuning, even the dis- tance regularization is applied. The result also suggests that distance regularization can be applied orthogonally to DeFT. In addition to the empirical justification, we provide theoretical foundation of DeFT Section
EXPERIMENTS
In this section, first describe the experimental settings and implementation details, followed by a demonstration of the effectiveness of our DeFT framework through a series of extensive experi- ments, including various ablation studies.

---

Published as a conference paper at ICLR 2025
Table 2: Analysis of the impact of learning rate. We investigate whether reducing the learning rate after the warm-up stage can lead better generalization. All the methods were trained GTAV using ResNet-50 backbone. The results demonstrate that a small learning rate increases the risk of falling in local minima, resulting in suboptimal performance.
Method
Learning rate
Cityscapes
BDD100K
Mapillary
Avg.
Joint Finetuning
le-2
42.32
40.33
44.88
42.51
3e-3
44.87
42.10
49.38
45.45
le-3
44.01
39.47
47.45
43.64
le-4
40.84
37.38
44.83
41.02
DeFT
le-2
50.06
43.17
50.51
47.91
Table 3: Impact of distance-based regularization. All the methods were trained on GTAV using ResNet-50 backbone. DeFT outperformed joint finetuning in both settings. The result also suggests that the distance regularization is orthogonal to DeFT and further improve its performance.
Method
Cityscapes
BDD100K
Mapillary
Avg.
Joint Finetuning
42.32
40.33
44.88
42.51
Joint Finetuning Distance Regularization
46.06
40.80
47.93
44.93
DeFT
50.06
43.17
50.51
47.91
DeFT + Distance Regularization
51.09
43.46
51,58
48.71
4.1 EXPERIMENTAL. SETUP
Datasets. We used three real-world datasets, Cityscapes (Cordts et al. 2016). BDD-100K (Yu al. 2020), and Mapillary (Neuhold et al. 2017), and two synthetic datasets, GTAV (Richter et al. 2016) and SYNTHIA (Ros et 2016) for the experiment. Cityscapes is real- world urban driving scene dataset, comprising 2,985 images for training and 500 for validation. BDD-1001 is another real- world urban driving scene dataset, and we used the 1,000 validation images for evaluation. Mapillary consists of 25,000 images collected from various worldwide locations, and we used 2,000 valida- tion images for evaluation. GTAV contains 24,966 images generated from the Grand Theft Auto V (GTAV) game engine, split into 12,403 images for training and 6,382 for validation. SYNTHIA is a photo-realistic synthetic urban scene dataset, consisting 9,400 images. We used 6,382 validation images for evaluation.
Network architecture. We utilized DeepLab v3+ (Chen al. 2018) as the segmentation model with ImageNet (Deng al.) 2009) pretrained ResNet-(50/101) (He et al. (2016) backbone networks. During training, we introduced two segmentation models that share the same architecture.
Implementation details. The model was trained with batch size of 4 through SGD with mo- mentum of 0.9. For the warm-up stage, model was trained for iterations for Cityscapes and 8K iterations for GTAV, with learning rate of le-2 and weight decay of 5c-3. During the decou- pled finetuning, the model was trained for 40K iterations with learning rate of 1c-2 and a weight decay of 5e-4. We employed polynomial learning rate decay schedule with power of 0.9. For data augmentation, we adopted color jittering, Gaussian blurring, random horizontal flipping with probability of 0.5, random scaling in the range [0.5, 2.0], and random cropping with size of 768 x 768. The weight update ratio 3 was set to 0.9999. We used the mean Intersection-over- (mloU) as the evaluation metric. We excluded the auxiliary cross-entropy loss applied to the en- coder, which has been widely adopted in previous work (Zhao et al., 2017 Pan et al. 2018 Lee
al. 2022 Zhao et al. 2022 Chattopadhyay et 2023
Ahn et 2024 as it degrades O0D
generalization capability.
4.2 COMPARISON WITH STATE OF THE ART
We conducted series of experiments to evaluate the effectiveness of DeFT. DeFT was compared with existing domain generalization methods, including IBN-Net (Pan et al. 2018). DRPC (Yue

---

Published as a conference paper at ICLR 2025
Table 4: Quantitative result comparison mloU (%) using ResNet-50 and ResNet-101 backbones.
The model trained on GTAV and evaluated Cityscapes (C), BDDI00K (B), and Mapil- lary (M).
ResNet-50
ResNet- 101
Methods
M
Avg.
C
B
M
Avg.
Baseline
35.16
29.71
31.29
32.05
35.73
34.06
33,42
34.40
IBN-Net Pan al
2018
33.85
32.30
37.75
34.63
37.37
34.21
36.81
36.13
DRPC
37.42
32.14
34.12
34.56
42.53
38.72
38.05
39.77
ISW Chol ei 2021
36.58
35.20
40.33
37.37
37.20
33.36
35.57
35.38
WildNet
44.62
38.42
46.09
43.04
45.79
41.73
47.08
44.87
SAN-SAW Peng et
2022
39.75
37.34
41.86
39.65
45.33
41.18
40.77
42.43
DIRL
41.04
39.15
41.60
40.60
SHADE Zhao
2022
44.65
39.28
43.34
42,42
46.66
43.66
45,50
45.27
PASTA Chatiopulleay et
44.12
40.19
47.11
43.81
45.33
42.32
48.60
45.42
TLDR Kim al. 20236
46.51
42.58
46.18
45.09
47.58
44.88
48.80
47.09
Blind Ahn 2024
45.72
41.32
47.08
44.71
DeFT (Ours)
50.06
43.17
50.51
47.91
52.14
45.16
53.15
50.15
Table 5: Quantitative result comparison in mloU (%) using ResNet-50 backbone. The model was
trained on Cityscapes and evaluated on BDD-100K (B), SYNTHIA (S), and GTAV (G).
Methods
S
G
Avg.
Baseline
44.96
23.29
42.55
36.93
IBN-Net Pas
2018
48.56
26.14
45.06
39.92
DRPC Yue
49.86
26.58
45.62
40.69
ISW Choi cl
2021
50.74
26.20
45.00
40.64
WildNet et
50.94
27.95
47.01
41.97
SAN-SAW Peng ctal.
2022
52.95
28.32
47.28
42.85
DIRL
51.80
26.50
46.52
41.61
SHADE Zhao et 2022
50.95
27.62
48.61
42.39
BlindNet
51.84
28.51
47.97
42.77
DeFT (Ours)
53.12
28.87
48.72
43.57
al. 2019a). ISW (Cho et al. 2021 WildNet Lee et al. (002). SAN-SAW (Peng al. 2022
DIRL Xu et al. (2022). SHADE Zhao et al. (2022). PASTA (Chattopadhyay et al. 2003)), TI.DR Kim et al. 2023b). and BlindNet (Ahn al.) 2024), using five datasets datasets- (C)ttscaaps,
(B)DD-1OOK, (M)apillary, YS)TTA,, and (GyTAV, and two different backbone networks- ResNet-50 and ResNet- 101. To evaluate the generalization ability of our method on various un- seen domains, conducted experiments in two scenarios: 1) the model was trained on GTAV and evaluated on Cityscapes, BDD-I00K, and Mapillary, or 2) the model was trained on Cityscapes and evaluated BDD-100K, SYNTHIA, and GTAV. For the first case, the results in Table ssho that our method outperforms all other methods by a large margin when trained on GTAV, using either ResNet-50 ResNet-101 our method also outperforms all other methods in the second case, where the model was trained on Cityscapes with ResNet-50 backbone.
backbone. Similarly, the results in Table 5 demonstrate that
4.3 ABLATION STUDIES
In this subsection, we study the individual contribution and effectiveness of each component within method. For the all experiments, the model was trained GTAV and evaluted on Cityscapes, BDD100K and Mapillary with ResNet-50 backbone.
Ablation study of the impact of individual component for training. To investigate the tribution of individual component during training, we investigated the impact various training components and measured its performance. We considered four different components for the
8

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study of the impact of individual component for training. Aux. and Aug. denote the auxiliary cross-entropy loss attached to the encoder and data augmentation, respectively. All the methods were trained on GTAV using ResNet-50 backhone. Warm-Up represent warming up the decoder while freezing the encoder before finetuning, and DeFT the proposed decoupled finetuning.
w/o Aux.
Aug.
Warm-Up
DeFT
Cityscapes
BDD100K
Mapillary
Avg.
35.16
29.71
31.29
32.05
36.58
34.49
39.08
36.72
40.77
37.87
43.39
40.66
42.32
40.33
44.88
42.51
50.06
43.17
50.51
47.91
Table 7: Ablation study of the impact of the decoupled finetuning strategy.
Finetuning strategy
Cityscapes
BDD100K
Mapillary
Avg.
Joint finetuning Joint finetuning + EMA
42.32
40.33
44.88
42.51
48.30
42.29
49.02
46.54
DcFT
50.06
43.17
50.51
47.91
Table 8: Analysis on the impact of final model configuration.
ID
e (AC)
82 (RC) dj (RC) d2 (AC)
Cityscapes
BDD100K
Mapillary
Avg.
39.30
37.41
43.14
39.95
V
43.15
39.82
45.55
42.84
47.29
41.84
49.33
46.15
IV
50.06
43.17
50.51
47.91
removing the auxiliary cross-entropy loss attached to the encoder (w/o Aux.), data aug- mentation (Aug-), decoder warming up (Warm-up) and DeFT framework (DeFT). Note that all the settings except the last row (DeFT) conducted joint finetuning, instead of decoupled finetuning. The results in Table 6 show that each component contributes to the performance, and applying all of them improves the most.
periments:
Ablation study the decoupled finetuning strategy. conducted experiment investigate the effect of decoupled finetuning and that of weight ensemble separately. To be specific, we jointly finetuned the encoder and decoder, and considered their EMA versions as the final model for eval- uation. The EMA update ratio 3 was set to 0.9999, the same as DeFT. The results in the Table 7 show that the proposed decoupled finetuning strategy better preserves generalizable knowledge of the pretrained encoder and decoder than joint finetuning.
Ablation study on final model configuration. We our final model the combination of two RCs, i.e., EMA encoder èq and EMA decoder d1, as RCs preserve the rich and generalizable knowl- edge. To investigate this, we conducted additional ablation study on various combination of RCs and ACs. We measured the performance of each combination when the whole training ended, and the results are listed in Table g The experiments show that superior generalization capability of RC (Exp. II, III and IV) than AC (Exp. I), where using both of RCs outperformed all other settings by large margin.
Alation study on the impact of the update ratio B. We employed the exponential moving average update scheme for RC with the update ratio B in Eq. ( To investigate the impact of 8, conducted additional experiments by varying the values 3. The results in Table d demonstrate that assigning higher weight to the model's initial parameters yields better generalization performance.

---

Published as a conference paper at ICLR 2025
Table 9: Ablation study the impact of the update ratio 3 in Eq. 22)
EMA update ratio (3)
Cityscapes
BDD100K
Mapillary
Avg.
0.99
44.37
40.79
46.81
43.99
0.999
46.19
42.14
48.81
45.71
0.9999
50.06
43.17
50.51
47.91
Image
Baseline
DeFT
GT
Cityscapes
BDD100K
Mapillary
Figure 4: Qualitative results of DeFT and its baseline. The model was trained on the GTAV dataset using ResNet-50 and tested on the Cityscapes, BDD100K, and Mapillary datasets.
CONCLUSION
In this paper, we have demonstrated the detrimental effects of jointly finetuning the encoder and de- coder in semantic segmentation models on domain generalization. Our empirical analysis revealed that this common practice leads to overfitting on the source domain, thereby degrading the model's generalization capability. To address this issue, we introduced DeFT, novel and effective training framework that decouples the finetuning of the encoder and decoder. This decoupled finetuning pre- vents them from being trained based on their counterpart, which might be overfitted to the source domain, resulting in improved generalization capability. DeFT operates two stages. In the first stage, warm up the decoder while keeping the pretrained encoder frozen. In the second stage, decouple the finetuning process by employing two parallel encoder-decoder pathways, each consist- ing of adaptive components (ACs) and retentive components (RCs). The ACs updated through standard backpropagation on the source domain, while the RCs are updated using an exponential moving average of the ACs' parameters. This approach enables the RCs learn task-relevant infor- mation while maintaining their generalization ability. Our extensive experiments demonstrate that DeFT consistently outperforms existing methods in domain generalizable semantic segmentation.
Limitation and discussion. Although DeFT has demonstrated impressive performance improve- ment through a large number of experiments, there is still room for further theoretical analysis. Also, there might be better design choices for RCs, although the EMA models showed their gener- alization capability. We believe that building a more concrete theoretical foundation and exploring a better alternative configurations for the RCs will be promising future research directions.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS AND DISCLOSURE OF FUNDING
This work was supported by Samsung Research Funding & Incubation Center of Samsung Elec- tronics under Project Number SRFC-IT1801- and the Institute of Information & Communica- tions Technology Planning & Evaluation (IITP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research, (No. RS-2024-00469482 RS-2024-00509258)
REFERENCES
Woo-Jin Ahn, Geun- Yeong Yang, Hyun-Duck Choi, and Myo-Taeg Lim. Style blind domain gen- eralized semantic segmentation via covariance alignment and semantic consistence contrastive learning. arXiv preprint arXiv:2403.06122, 2024.
Prithvijit Chattopadhyay, Kartik Sarangmath, Vivek Vijaykumar, and Judy Hoffman. Pasta: Pro- portional amplitude spectrum training augmentation for syn-to-real domain generalization. In IEEE/CVF International Conference in Computer Vision (ICCV), 2023.
Liang-Chie Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy. and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017.
Liang-Chie Chang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proc. European Conference on Computer Vision (ECCV), 2018.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. The Eleventh International Conference on Learning Representations, 2023.
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: a large-scale hier- archical image database. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2009.
Jian Ding, Nan Xue, Gui-Song Xia, Bernt Schiele, and Dengxin Dai. Hgformer: Hierarchical group- ing transformer for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15413-15423, June 2023.
Simon S Du. Wei Hu, and Jason D Lee. Algorithmic regularization in learn- ing deep homogeneous models: Layers are automatically balanced. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018a. URL https::/proceeddeee wwwp:/iiiiinnnnnnnnoerem www.wne...ic...mo ccwppper......pe.ee.eeee http//ff///sppe//11/..... File/ Lee/fel fell 31 www.7755663333ccc [tt.://cFFaooo.222cc7731111. 3dae2 pdf
Simon S Du, Yining Wang. Xiyu Zhai, Sivaraman Balakrishnan, Russ R Salakhutdinov, and Aarti Singh. How many samples are needed to estimate convolutional neural network? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018b. 8b..

---

Published as a conference paper at ICLR 2025
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversane training of neural net- works. Journal of Machine Learning Research (JMLR), 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, 2018
Henry Gouk, Timothy Hospedales, and massimiliano pontil. Distance- based regularisation of deep networks for fine-tuning, In International Conference Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2016.
Lukas Hoyer, Dengxin Dai, and Lue Van Gool. DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9924-9935. 2022.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snap- shot ensembles: Train 1, get for free. In International Conference on Learning Representations (ICLR), 2017.
Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018.
Yuru Jia, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Lue Van Gool, Konrad Schindler, and Anton Obukhov. Dginstyle: samain-generalizable semantic segmentation with image diffusion models and stylized semantic control. European Conference on Computer Vision, 2024.
Namyup Kim, Taeyoung Son, Jaehyun Pahk, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Wedge: Web-image assisted domain generalization for semantic segmentation. In 2023 International Conference on Robotics and Automation (ICRA), 2023a.
Sunghwan Kim, Dae-hwan Kim, and Hoseong Kim. Texture learning domain randomization for domain generalized segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 677-687, October 2023b.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distributio.. In International Con- ference on Learning Representations, 2022.
Suhyeon Lee, Hongje Seong, Seongwon Lee, and Euntai Kim. Wildnet: Learning domain gener- alized semantic segmentation from the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5543-5551, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: meta- learning for domain generalization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. AAAI Press, 2018a.
Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, volume 34, pp. 27249-27262 Curran Associates, Inc., 2021.
Haoliang Li, Sinno Jialin Pan, Shigi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 201 8b.
12

---

Published as a conference paper at ICLR 2025
Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable weight averaging: Efficient training by optimizing historical solutions. In The Eleventh International Conference on Learning Representations, 2023.
Shan Lin and Jingwei Zhang. Generalization bounds for convolutional neural networks. ArXiv, Abs/1910.01449 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Philip M. Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks In International Conference on Learning Representations, 2020.
Yawei Luo, Ping Liu, and Yi Yang. Kill birds with one stone: Domain generalization for semantic segmentation via network pruning, International Journal of Computer Vision, pp. 1-18, 07 2024. doi: 10.1007/s1 1261-12662222 12-22223--0000 021-422...5.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 Proceedings of Machine Learning Research, pp. 10-18, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Vaishnavh Nagarajan and Zico Kolter. Generalization deep networks: The role of distance from initialization, 2019.
Hyconseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proc. IEEE International Conference Computer Vision (ICCV), 2017.
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for seman- tic segmentation. In Proc. IEEE International Conference on Computer Vision (ICCV), 2015.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In Proc. European Conference Computer Vision (ECCV), 2018.
Xingang Pan. Xiaohang Zhan, Jianping Shi, Xiaoou Tang, and Ping Luo. Switchable whitening for deep representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Duo Peng, Yinjie Lei, Munawar Hayat, Yulan Guo, and Wen Li. Semantic-aware domain generalized segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern June 2022.
Recognition (CVPR), pp.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Proc. European Conference Computer Vision (ECCV), 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In Proc. Medical Image Computing and Computer-Assistel Intervention (MICCAI), 2015.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
13

---

Published as a conference paper at ICLR 2025
Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Rogerio Feris, and Kate Saenko. Mind the backbone: Minimizing backbone distortion for robust object detection, 2023.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Proc. Neural Information Pro- cessing Systems (NeurIPS), 2017.
Junjiao Tian, Xiaoliang Dai, Chih- Cao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Train- able projected gradient method for robust fine-tuning, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7836-7845, 2023.
Yang Wen, Leiting Chen, Yu Deng, and Chuan Zhou. Rethinking pre-training medical imaging. J. Vis. Comun. Image Represent, 78(C), jul 2021. ISSN 1047-3203.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre. Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965- 23998. PMLR, 17-23 Jul 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7959-7971, June 2022b.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS), 2021.
Qi Xu, Liang Yao. Zhengkai Jiang, Guannan Jiang, Wenqing Chu, Wenhui Han, Wei Zhang, Chengjie Wang, and Ying Tai. Dirl: semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3): 2884-2892, Jun. 2022.
representation learning for generalizable
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng. and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10819-10829, 2022.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Sangiovanni-Vinceeelli Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In Proc. IEEE International Conference on Computer Vi- sion (ICCV), 2019a.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto I.. Sangiovanni-Vincentie Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-real generaliza- tion without accessing target domain data. 2019 IEEE/CVF International Conference Com-
puter Vision (ICCV), pp. 2100-2110, 2019b.
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Yuyang Zhao, Zhun Zhong, Na Zhao, Nicu Sebe, and Gim Hee Lee. Style-hallucinated dual con- sistency learning for domain generalized semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In Proc. International Conference Learning Representations (ICLR), 2021.
14

---

Published as a conference paper at ICLR 2025
A ALGORITHMS FOR EACH STAGE OF DEFT
describes In this sections, we present Py Porcch-likee pseudocodes for each stage of DeFT. A desorribs the training procedure for decoder warm-up, and Algorithm 2 describes the training procedure for decoupled finetuning and the configuration of the final model for inference.
Algorithm Pseudocode for Decoder Warm-Up, PyTorch-like
pretrained encoder
ranaomly initialized decoder d
CE: standard pixel-wise cross entropy loss Lco
freezing_weights (e)
for x, in scurce_loader:
lcad a minibatch x with samples
p de )
# prediction P P= f(X) = d(e(X)) calculate loss
L CE y)
10
b.bawwaard
12
update (d)
only the decoder d is updated
Algorithm 2 Pseudocode for Decoupled Finetuning and Inference, Py TyToch-like
e: warmed-up encoder e, the same as the pretrained encoder
d: warmed-up decoder d
CE: standard pixel-wise cross entropy loss L.. beta: EMA update ratio B
e1 copy (e)
# AC encoder e1
e2 copy (e)
A RC encoder é2
dl copy (d)
# RC decoder di
10 d2 copy (d)
# AC decoder da
12 freezing freezing_weights (e2)
13 froozing_weights (d1)
14
for x, in scurce_loader:
lead a minibatch x with samples
16
pl dl (el (x))
prediction pt) from Ji == di oe # calculate loss
17
L1 (E y)
19
p2 d2 (e2 (x))
prediction pl) from f2 = dg 02 calculate loss
20
L2 (E2, y)
11 backward
23
L2 backward
24
25
update (el) update (d2)
# update AC encoder 61
$ update AC decoder d2
27
28
update_EMM. (e2, el, beta) @po (dl, d2, peta)
update encoder E2 update RC decoder di
30
31
def Docatt_EMA-A (rc, ac, beta):
32
for param_re, param_ac in zip par.meeeete (), ac parameterss ()
param_rc = beta param + (1 beta) paran_ac
35
def inference (x):
return dl (e2 (x))
A final model frina d, 0
15

---

Published as a conference paper at ICLR 2025
B THEORETICAL FOUNDATION OF DEFT
This section theoretically justifies the motivation and design choices of DeFT to validate its efficacy. To be specific, we discuss (1) the issues of joint finetuning, and (2) how DeFT resolves them.
Kumar al. 2022) showed that jointly finetuning well-generalizable encoder with randomly initialized decoder distorts the representation of the encoder, and proved this theoretically for the case of two-layer linear neural network. Let us suppose finetuning a two-layer linear network JB. consisting of an encoder B E R&xd and a decoder head v E RR For in-distribution (ID) data X E Raxd and its corresponding label E I,", the loss to be optimized is L(ū. 4) ||@ yll.. where 5 = [ = X XBBT is the prediction of the network f for X. Then, the gradient of B for the loss L is as follows:
V VBL(Ú. 2u(XB' v) -)
Considering an out- -ffto-f--itttbutiinn (OOD) data point E Rd which is orthogonal to the ID subspace = rowspace( X). Then the feature representation of the updated encoder for u, denoted by B' u, where B' = B- AV T. v), is as follows:
Bdu = (B AV ( u))u = Bu AV AV = Bu 20 v - Xu.
Note that is orthogonal the rowspace of X, and thus 2v 2v(XBT v) Xu goes O. Therefore, in such case, changes in B with respect to the ID data X cannot affect the feature representation of the OOD data u. However, the decoder head also changes since loss gradients for updating the encoder and decoder are coupled, which is referred to as "balancedness" in Du et al. (201 Sa). Then the change of the decoder head affects the predictions for OOD data which is given by
UB o' UB (u ( V UB XUB V.L
Since the representations for are same during the update, i.e. UB = UB,, the predictions for U can be distorted as XU B" V,L, where V,L 2BX u y). Such distortion in prediction can impair the generalization capability on OOD data as V. is calculated solely based on ID data X. As result, the entire model becomes less capable of handling OOD data by the joint finetuning. This suggests the need to decouple one module from being affected by the distortion of another module.
v
On the other hand, consider a linear probing case where the encoder is fixed and no feature distortion occurs. According to Lemma A.14 in Kumar al. (2022), the upper bound of OOD error of the linear probing is:
2
V Troaa (v Bo) <
C5
(Bo B,)lwu,l|2,
cos Omnxx (S, R)
where (up Bb) is OOD error with the frozen initial encoder Bo and linearly probed decoder head uig cos maxx ( S, R) is the cosine of the largest angle between = rowspace( () and R = (owspace(Bo), w. = B.v. with the optimal encoder B. and decoder v., and d(Bo, B, is the distance between Bo and B. A rigorous proof of the above upper bound can be found in Kumar et al. (2022). This demonstrates that the upper bound of OOD error of the linear probing is inversely proportional to the difference between the pretrained encoder and the "optimal" encoder, which shows the lowest errors for both ID and OOD data. In other words, in the linear probing case, decoder coupled with more generalizable encoder results a more generalizable final model.
Based on this, we first decouple the encoder and decoder to prevent each module from being dis- torted by their jointly finetuned counterparts, coupling them with counterparts which are not affected by distribution (i.c., domain) shift thus well generalizable. Through the empirical analysis of the distance from initialization in Figure 3 it can be inferred that DeFT suppresses such "distortion" from the initial states during training, resulting in better generalization capability.
The optimization behavior from the perspective of individual network modules, such as encoders and decoders, well as the interactions between these modules during the optimization process, seems to remain relatively underexplored. We expect that further foundational study in this direction might pave the way for more rigorous theoretical analysis of DeFT.
16

---

Published as a conference paper at ICLR 2025
C VERSATILITY OF DEFT AT A TRANSFORMER BACKBONE
We evaluated DeFT using the MiT-B5 (Xie et al. 2021 transformer backbone. All models were trained on GTAV and evaluated Cityscapes, B BDD100K, and Mapillary. As shown in Table [O] DeFT outperformed existing methods, such as DAFormer (Hoyer et al. (022) and the combination of DAFormer and DGinStyle (Jia et al., 2024), using the same backbone. Notably, DeFT achieved this outstanding performance without additional modifications, unlike DAFormer, which adapts its model architecture for domain generalization, or DGinStyle, which relies heavily on extreme data augmentation. We believe that incorporating such strategies could further enhance DeFT's s perfor- mance. These results demonstrate that DeFT' a versatile training strategy applicable across various backbones.
Table 10: Comparison between DeFT and other methods incorporating MiT-B5 backbone.
Segformer MiT-B5
Cityscapes
BDD100K
Mapillary
Avg.
DAFormer (Hoyer et al. 2022)
52.65
47.89
54.66
51.73
DAFormer + DGInStyle (Jia et al. 2024
55.31
50.82
56.62
54.25
DeFT (Ours)
57.16
49.32
59.99
55.49
D OTHER UPDATE SCHEMES FOR RETENTIVE COMPONENTS (RCs)
We conducted additional experiments by replacing the update scheme for RC in DeFT (i.e., EMA) with different weight ensemble methods.
(A) Exponentially decreasing the later ensemble coefficient:
* ARC
BAC
1-1
Σ 0²,
a>
(4)
T
f=0
Simply averaging the latest AC weights with the initial weights, similar to WiSE-FT (Wortsman
et al. 2022b):
@RC 0.5 * 0.C 0.5 0g-
(5)
All models trained GTAV and evaluated Cityscapes, BDD100K, and Mapillary. As shown in Table[[]] the EMA biased towards initialization employed in DcFT clearly outperformed the other weight ensemble methods.
Table 11: Comparison between the EMA and other RC update schemes.
Update scheme
Cityscapes
BDD100K
Mapillary
Avg.
(A) Exponentially decreasing
44.46
40.86
46.72
44.01
(B) WiSE-FT
45.03
40.47
47.04
44.18
(C) EMA (DeFT)
50.06
43.17
50.51
47.91
E ADDITIONAL. QUALITATIVE RESULTS
In this section, provide additional qualitative results on Cityscapes (Cordts et al. 2016) in Fig- ure BDD100K Yu et al. 2020) in Figure 6 and Mapillary (Neuhold et al.] (017) in Figure 7 respectively, which are presented in the main sections due the space limit. We used the model trained on the GTAV (Richter et al., (016)) using ResNet-50 (He et al. 2016) backbone. We also provide the color code of 19 classes Figure
17

---

Published as a conference paper at ICLR 2025
Image
Baseline
DeFT
GT
Figure 5: Qualitative results of DeFT and its baseline on Cityscapes dataset.
Image
Baseline
DeFT
GT
Figure 6: Qualitative results of DeFT and its baseline on BDD100K dataset.
18

---

Published as a conference paper at ICLR 2025
Image
Baseline
DeFT
GT
Figure 7: Qualitative results of DeFT and its baseline on Mapillary dataset.
road
sidewalk traffic light
building traffic sign
wall vegetation
fence
pole
terrain
sky
person
rider motorbike
car bicycle
truck
bus
train
Figure 8: The color code of 19 classes on the training and test datasets.
19


Reviews:
[{"review_id": "yOPiPWbAd1", "reviewer": "Reviewer_exsS", "summary": "This paper investigates the vulnerability issue of joint finetuning, and proposes a novel finetuning framework called Decoupled FineTuning for domain generalization (DeFT) as a solution. DeFT operates in two stages. Its first stage warms up the decoder with the frozen, pretrained encoder so that the decoder learns task-relevant knowledge while the encoder preserves its generalizable features. In the second stage, it decouples finetuning of the encoder and decoder into two pathways, each of which concatenates a usual component (UC) and generalized component (GC); each of the encoder and decoder plays a different role between UC and GC in different pathways. DeFT significantly outperformed existing methods in various domain shift scenarios, and its performance could be further boosted by incorporating a simple distance regularization.", "questions": "What are the performance comparisons on Transformer-based backbones?\n\nA more rigorous theoretical analysis is essential.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A  new finetuning framework dubbed Decoupled FineTuning (DeFT) is proposed for domain generalization. Overall, it is a simple and universal solution for DG tasks.\n\n2. The experimental details are sufficient, the reproducibility is good, and the experimental results are convincing.", "weaknesses": "1. While the approach presented in this paper is quite straightforward, my primary concern is that its framework demonstrates effectiveness solely in practical applications. The authors provide an analysis based on parameter distance, yet they lack a more profound theoretical exploration. Given the standards of an ICLR paper, I believe a rigorous theoretical analysis is essential.\n\n2. The paper exclusively utilizes convolutional backbones, such as ResNet-50, without any experiments on more advanced transformer-based backbones [1-2]. The absence of results with these state-of-the-art encoder-decoder frameworks limits the generalizability and relevance of the proposed method. \n\n[1] HGFormer: Hierarchical Grouping Transformer for Domain Generalized Semantic Segmentation\n[2] Kill Two Birds with One Stone: Domain Generalization for Semantic Segmentation via Network Pruning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJEvGlePhT", "reviewer": "Reviewer_sHdo", "summary": "In this work the authors propose a fine tuning strategy for encoder-decoder networks to improve domain generalization. The idea is to decouple the fine tuning of encoder and decoder by keeping a frozen copy of the decoder (encoder) when fine tuning the encoder (decoder), and then combine the encoder and decoder with their frozen copies via exponential moving averages. This seems to result in more robustness to overfitting and better generalization, showing improved performance in semantic segmentation scenarios.", "questions": "Please address the comments in the weakenesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is sound and well motivated.\n- Comprehensive experiments that show a notable improvement in the generalization capabilities of the model", "weaknesses": "- The contribution of decoupling encoder and decoder learning, and the contribution of averaging with earlier frozen models, in my view, should be differentiated and ablated properly. In particular, I suggest to evaluate some baseline without decoupling but that averages the joint encoder-decoder model with earlier clones.\n- The method is presented in an overcomplicated and confusing way, and be described in a simpler way. In particular the terminology usual component (UC) and generalized component (GC) to refer to the model and its frozen earlier clone is more confusing than clarifying. \n- There is no theoretical analysis (even preliminary) of why the proposed approach improves generalization (the authors leave it to future work).\n- There is no analysis of the additional computational and memory costs of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6AfKd4QcwC", "reviewer": "Reviewer_qfZ6", "summary": "This paper introduces Decoupled FineTuning (DeFT), a novel framework designed to improve domain generalization in semantic segmentation tasks, where models often perform poorly when exposed to domain shifts. The traditional joint fine-tuning of encoder and decoder networks can lead to overfitting on source domains, reducing generalizability to new, unseen domains.\n\nThe paper have three contributions:(1)By separating the fine-tuning of encoder and decoder components, DeFT avoids overfitting and improves the model's ability to generalize across domains.(2)Using EMA updates for GCs allows the model to retain initialization-based generalization properties, enhancing the robustness of learned representations.(3)DeFT is validated on five datasets (e.g., Cityscapes, BDD-100K, Mapillary), showing substantial improvements over state-of-the-art methods across diverse domain shift scenarios.\n\nThese contributions position DeFT as an effective framework for domain-generalizable semantic segmentation.", "questions": "1.Can you clarify why you chose specific datasets and domain shifts for evaluating DeFT?\n2.Have you conducted ablation experiments on the updated method? For example, if you use EMA-based updates only for the encoder or only for the decoder, how will DeFT behave?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Originality:Rather than jointly fine-tuning encoder and decoder components, the paper innovates with a decoupled strategy. This approach uniquely assigns the encoder and decoder to two different components—Usual Components (UC) and Generalized Components (GC)—which are updated in parallel but through distinct pathways. This structure is particularly inventive in preventing overfitting to the source domain while preserving generalizable features,I think this is a very clever and useful method,At the same time, the author uses EMA to update GC and maintain its generalization ability in the fine-tuning process,This method is also very unique, different from the traditional method.\n\nQuality:In this paper, five different data sets (such as Cityscapes, BDD-100K, GTAV, etc.) and a variety of domain offsets are comprehensively tested, and other methods (such as WildNet, SHADE, BlindNet, etc.) are compared.\n\nClarity:This paper clearly expounds each stage from the initialization of the decoder to the decoupling fine tuning, provides the pseudo-code of the algorithm, and effectively conveys the architecture and ablation experimental results by using diagrams and tables. I think this makes me clearly understand the whole algorithm process and the advantages of the algorithm.\n\nSignificance:This paper solves a key problem of domain generalization of semantic segmentation, and I think it plays a important role.", "weaknesses": "limitation:While the empirical results support the DeFT framework, the theoretical grounding behind the decoupling strategy (specifically the benefits of separating encoder and decoder updates into Usual and Generalized Components) is limited. Strengthening this theoretical component would help clarify why the decoupling approach reduces overfitting and improves generalization.\n\nSuggestion:Consider providing a more detailed theoretical analysis or explanation, possibly referencing or building upon works in model fine-tuning and parameter space regularization.Your overview map is very brief, please enrich it if you can.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7uWgY7sebo", "reviewer": "Reviewer_wUh8", "summary": "The paper, titled Decoupled Finetuning for Domain Generalizable Semantic Segmentation, introduces a novel approach called DeFT. Traditional joint finetuning of an encoder and decoder often leads to overfitting to the source domain, which degrades generalization capabilities on unseen domains. To address this, DeFT proposes a two-stage framework: (1) warming up the decoder while keeping the pretrained encoder frozen to preserve its generalizable features, and (2) decoupling the finetuning of the encoder and decoder into two pathways, each involving a usual component (UC) and a generalized component (GC) updated by an exponential moving average. This decoupling helps retain initial generalizable knowledge while improving task-specific learning. The method significantly outperformed existing state-of-the-art domain generalization techniques across various domain shift scenarios.", "questions": "1. How does the complexity of implementing DeFT, with its two separate pathways, impact training time and computational resources compared to standard joint finetuning methods?\n2. Have the authors explored other update schemes for the GCs besides the exponential moving average? If so, what were the results, and how did they compare?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Originality: DeFT introduces a unique decoupling strategy for finetuning the encoder and decoder, setting it apart from prior methods that only optimize them jointly. The method's use of an exponential moving average for GC updates further enhances its originality.\n2. Quality: The paper is well-structured, providing comprehensive empirical evidence through experiments across five datasets. Ablation studies and comparisons with state-of-the-art methods reinforce the robustness of DeFT.\n3. Clarity: The description of DeFT, supported by visual diagrams, pseudocode, and detailed explanations, is clear and facilitates understanding of the training process.", "weaknesses": "1. Complexity of Implementation: The decoupled pathway design and maintaining separate UC and GC pathways may introduce additional implementation complexity that is not addressed in terms of potential computational overhead. That is, there is no relevant model complexity analysis.\n2. Sensitivity analysis experiments for the main parameters are not available.\n3.The experimental equipment is not described in sec.4 EXPERIMENTS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFxYmr6ZaW", "reviewer": "Reviewer_ck24", "summary": "**Motivation**\nThe paper identifies a critical gap in the semantic segmentation field, where the standard practice of joint fine-tuning pre-trained encoders with randomly initialized decoders has not been adequately examined regarding its vulnerability to domain shift. This lack of investigation highlights a potential risk to domain generalization, motivating the need for a new approach to fine-tuning that can better address these challenges.\n\n**Method**\nThe authors propose the Decoupled Fine-Tuning (DeFT) framework, which consists of two key stages:\n\nStage One: A frozen pre-trained encoder is used to preheat the decoder. This allows the decoder to learn task-relevant knowledge while keeping the encoder's generalization ability intact.\nStage Two: The fine-tuning process for the encoder and decoder is decoupled into two separate paths. Each path includes:\nA Conventional Component (UC) that updates based on the loss gradient from the source domain.\nA Generalization Component (GC) that updates through an exponential moving average biased towards its initialization, thus retaining its generalization capability.\n\n**Contribution**\nThe introduction of the **DeFT** framework, which explicitly addresses the negative impact of joint fine-tuning on domain generalization in semantic segmentation tasks.\nThe provision of a novel approach to decouple the fine-tuning of encoders and decoders, leading to improved performance and robustness against domain shifts compared to existing methods that primarily focus on data or feature augmentation.", "questions": "1. I would like to know the impact of the two EMA updates in DeFT on training speed. Specifically, how much slower is it compared to the baseline? Could you provide detailed data on this?\n\n2. In Table 8, I noticed that the EMA update ratio (β) has a significant impact on the experimental results, even with slight variations in this parameter. Could you provide a further analysis of the reasons behind this?\n\n3. In Figure 1, the issue of target loss fluctuations is raised. Does DeFT address this problem? Could further experiments be conducted to investigate this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. A novel training method, DeFT has been proposed, abandoning the previous joint fine-tuning approach and demonstrating its effectiveness through experiments. \n2. Extensive experimental analyses have been conducted to validate the effectiveness of the experiments and assess the sensitivity to parameters, proving the efficacy of DeFT. \n3. This method can be applied to multiple domains and provides ideas for future research.", "weaknesses": "1. I am worried that this method may significantly reduce computational speed because the introduction of two EMAs could have a considerable impact on training speed, even if it does not significantly affect inference speed. \n2. There is a lack of insightful analysis regarding the performance loss associated with joint fine-tuning, making the motivation less compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)