Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jaehyun Pahk
authors: Donghyeon Kwon
authors: Seong Joon Oh
authors: Suha Kwak
reviews: 
Comments: **Motivation**
The paper identifies a critical gap in the semantic segmentation field, where the standard practice of joint fine-tuning pre-trained encoders with randomly initialized decoders has not been adequately examined regarding its vulnerability to domain shift. This lack of investigation highlights a potential risk to domain generalization, motivating the need for a new approach to fine-tuning that can better address these challenges.

**Method**
The authors propose the Decoupled Fine-Tuning (DeFT) framework, which consists of two key stages:

Stage One: A frozen pre-trained encoder is used to preheat the decoder. This allows the decoder to learn task-relevant knowledge while keeping the encoder's generalization ability intact.
Stage Two: The fine-tuning process for the encoder and decoder is decoupled into two separate paths. Each path includes:
A Conventional Component (UC) that updates based on the loss gradient from the source domain.
A Generalization Component (GC) that updates through an exponential moving average biased towards its initialization, thus retaining its generalization capability.

**Contribution**
The introduction of the **DeFT** framework, which explicitly addresses the negative impact of joint fine-tuning on domain generalization in semantic segmentation tasks.
The provision of a novel approach to decouple the fine-tuning of encoders and decoders, leading to improved performance and robustness against domain shifts compared to existing methods that primarily focus on data or feature augmentation.
Strengths: 1. A novel training method, DeFT has been proposed, abandoning the previous joint fine-tuning approach and demonstrating its effectiveness through experiments. 
2. Extensive experimental analyses have been conducted to validate the effectiveness of the experiments and assess the sensitivity to parameters, proving the efficacy of DeFT. 
3. This method can be applied to multiple domains and provides ideas for future research.
Weaknesses: 1. I am worried that this method may significantly reduce computational speed because the introduction of two EMAs could have a considerable impact on training speed, even if it does not significantly affect inference speed. 
2. There is a lack of insightful analysis regarding the performance loss associated with joint fine-tuning, making the motivation less compelling.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: The paper, titled Decoupled Finetuning for Domain Generalizable Semantic Segmentation, introduces a novel approach called DeFT. Traditional joint finetuning of an encoder and decoder often leads to overfitting to the source domain, which degrades generalization capabilities on unseen domains. To address this, DeFT proposes a two-stage framework: (1) warming up the decoder while keeping the pretrained encoder frozen to preserve its generalizable features, and (2) decoupling the finetuning of the encoder and decoder into two pathways, each involving a usual component (UC) and a generalized component (GC) updated by an exponential moving average. This decoupling helps retain initial generalizable knowledge while improving task-specific learning. The method significantly outperformed existing state-of-the-art domain generalization techniques across various domain shift scenarios.
Strengths: 1. Originality: DeFT introduces a unique decoupling strategy for finetuning the encoder and decoder, setting it apart from prior methods that only optimize them jointly. The method's use of an exponential moving average for GC updates further enhances its originality.
2. Quality: The paper is well-structured, providing comprehensive empirical evidence through experiments across five datasets. Ablation studies and comparisons with state-of-the-art methods reinforce the robustness of DeFT.
3. Clarity: The description of DeFT, supported by visual diagrams, pseudocode, and detailed explanations, is clear and facilitates understanding of the training process.
Weaknesses: 1. Complexity of Implementation: The decoupled pathway design and maintaining separate UC and GC pathways may introduce additional implementation complexity that is not addressed in terms of potential computational overhead. That is, there is no relevant model complexity analysis.
2. Sensitivity analysis experiments for the main parameters are not available.
3.The experimental equipment is not described in sec.4 EXPERIMENTS.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper introduces Decoupled FineTuning (DeFT), a novel framework designed to improve domain generalization in semantic segmentation tasks, where models often perform poorly when exposed to domain shifts. The traditional joint fine-tuning of encoder and decoder networks can lead to overfitting on source domains, reducing generalizability to new, unseen domains.

The paper have three contributions:(1)By separating the fine-tuning of encoder and decoder components, DeFT avoids overfitting and improves the model's ability to generalize across domains.(2)Using EMA updates for GCs allows the model to retain initialization-based generalization properties, enhancing the robustness of learned representations.(3)DeFT is validated on five datasets (e.g., Cityscapes, BDD-100K, Mapillary), showing substantial improvements over state-of-the-art methods across diverse domain shift scenarios.

These contributions position DeFT as an effective framework for domain-generalizable semantic segmentation.
Strengths: Originality:Rather than jointly fine-tuning encoder and decoder components, the paper innovates with a decoupled strategy. This approach uniquely assigns the encoder and decoder to two different components—Usual Components (UC) and Generalized Components (GC)—which are updated in parallel but through distinct pathways. This structure is particularly inventive in preventing overfitting to the source domain while preserving generalizable features,I think this is a very clever and useful method,At the same time, the author uses EMA to update GC and maintain its generalization ability in the fine-tuning process,This method is also very unique, different from the traditional method.

Quality:In this paper, five different data sets (such as Cityscapes, BDD-100K, GTAV, etc.) and a variety of domain offsets are comprehensively tested, and other methods (such as WildNet, SHADE, BlindNet, etc.) are compared.

Clarity:This paper clearly expounds each stage from the initialization of the decoder to the decoupling fine tuning, provides the pseudo-code of the algorithm, and effectively conveys the architecture and ablation experimental results by using diagrams and tables. I think this makes me clearly understand the whole algorithm process and the advantages of the algorithm.

Significance:This paper solves a key problem of domain generalization of semantic segmentation, and I think it plays a important role.
Weaknesses: limitation:While the empirical results support the DeFT framework, the theoretical grounding behind the decoupling strategy (specifically the benefits of separating encoder and decoder updates into Usual and Generalized Components) is limited. Strengthening this theoretical component would help clarify why the decoupling approach reduces overfitting and improves generalization.

Suggestion:Consider providing a more detailed theoretical analysis or explanation, possibly referencing or building upon works in model fine-tuning and parameter space regularization.Your overview map is very brief, please enrich it if you can.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 2