Learning Frequency-Adapted Vision Foundation Model for Domain Generalized Semantic Segmentation

Authors: qi  bi
jingjun  yi
hao  zheng
haolan  zhan
yawen  huang
wei  ji
yuexiang  li
yefeng  zheng
bdd  cityscapes
gtav  mapillary
  synthia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-26 10:32:12

Abstract: The emerging vision foundation model (VFM) has inherited the ability to generalize to unseen images.
Nevertheless, the key challenge of domain-generalized semantic segmentation (DGSS) lies in the domain gap attributed to the cross-domain styles, i.e., the variance of urban landscape and environment dependencies.
Hence, maintaining the style-invariant property with varying domain styles becomes the key bottleneck in harnessing VFM for DGSS. 
The frequency space after Haar wavelet transformation provides a feasible way to decouple the style information from the domain-invariant content, since the content and style information are retained in the low- and high- frequency components of the space, respectively. 
To this end, we propose a novel Frequency-Adapted (FADA) learning scheme to advance the frontier.
Its overall idea is to separately tackle the content and style information by frequency tokens throughout the learning process.
Particularly, the proposed FADA consists of two branches, i.e., low- and high- frequency branches. The former one is able to stabilize the scene content, while the latter one learns the scene styles and eliminates its impact to DGSS. 
Experiments conducted on various DGSS settings show the state-of-the-art performance of our FADA and its versatility to a variety of VFMs.
Source code is available at \url{https://github.com/BiQiWHU/FADA}.

Main paper: 
 introduction: 
 Most existing semantic segmentation tasks assume that the training and inference images follow the independent and identical distribution (i.i.d.) [12, 13, 36, 35, 34, 81, 54, 44, 74] , which is far from reality. Domain-generalized semantic segmentation (DGSS) aims to infer robust pixel-wise semantic predictions on arbitrary unseen target domains when a segmentation model is trained on the source domain (as illustrated in Fig. 1a ). Compared with general domain generalization tasks, the feature distribution discrepancy between the source domain and unseen target domains in the context of DGSS holds some unique factors. Specifically, the cross-domain images in DGSS usually share the same content information (i.e., common semantic categories in driving scenes), while the cross-domain styles (i.e., urban landscape, weather, lighting conditions, etc.) mainly account for the feature distribution difference [65, 49, 18, 11, 72, 6, 23, 43, 24] .Existing DGSS methods can be summarized into three categories. The first category intends to decouple the style information from the scene representation [55, 30, 14, 56, 78, 71, 58] , but does not ensure a strong content representation ability. The second category, on the contrary, directly focuses on the content representation ability regardless of the cross-domain style variations [19, 4, 7] . Such Figure 1: (a) The key challenge of domain generalized semantic segmentation (DGSS) lies in the stability of the scene content, while the domain gap is caused by the style variation. (b) Analysis of frozen VFM features after Haar wavelet transform. We compute the correlation coefficient between the CityScapes source domain (C) and BDD, Mapillary, SYHTHIA, GTA5 unseen target domains (B, M, G, S). The low-frequency component exhibits a higher correlation and smaller domain gap. In contrast, the high-frequency components exhibit a lower correlation and a larger domain gap. methods were recently developed much owing to the stronger pixel-wise representation ability of mask attention [12, 13] . The third category focuses on enriching the styles as much as possible during training [82, 41, 83] . However, the content representation ability is rarely taken into consideration.The emerging vision foundation model (VFM) [62, 39, 20, 37] , with strong generalization ability inherited from a large quantity pre-trained images, provides a new possible paradigm for DGSS. Under the realm of parameter-efficient fine-tuning, the hypothesis, i.e., a foundation model relies on the low intrinsic dimension [42, 1] to adapt to the downstream tasks, exemplified by low-rank adaptation (LoRA) [28] , has shown great success, where the key idea is to inject trainable rank decomposition matrices into each layer. More recently, the low-rank adaptation paradigm has demonstrated to be feasible to fine-tune the VFM for DGSS [69] . Unfortunately, the style-invariant properties of VFM, which is a fundamental problem for the extraction of domain generalized semantics, remain unexplored.An ideal style invariant low-rank adaptation is supposed to discern the subtle low intrinsic dimension while does not pose shift on the fragile and frozen VFM features. The prior DGSS methods learned style invariance from the fully trained image features (e.g., instance normalization [55, 30] and instance whitening [14, 56, 71, 58] ), which are more likely to collapse and less feasible. Therefore, we adapt the low-rank adaptation to the frequency space, where the style and content have been separated to high-frequency and low-frequency components [29, 46, 40, 77, 67] , respectively.In this paper, we present a novel Frequency-Adapted learning scheme, dubbed FADA, to push this frontier. Its conceptual idea is to adapt the style and content representation from the VFM features separately in the frequency space. After transforming the frozen VFM features to the frequency space by the Haar wavelet transform [60] , the low-frequency branch exploits the scene content from frozen VFM features by the learnable low-frequency tokens. In contrast, in the high-frequency branch, the high-frequency token features are implemented using the instance normalization operation, so that the representation becomes invariant to the scene style.Notably, the proposed FADA introduces two new research lines. Firstly, the possibility of learning lowrank adaptation in the frequency space is explored, which can further benefit other VFM downstream tasks strongly related to the style and content representation. Secondly, it demonstrates the potential of harnessing the Haar wavelet transform for DGSS, which can also inspire advancements in general visual domain generalization.Concretely, our contributions can be summarized as follows:• We propose a Frequency-Adapted learning scheme, dubbed FADA, to fine-tune VFMs for domain-generalized semantic segmentation.• The proposed FADA, aided by the Haar wavelet guidance to mine the style-invariant property of VFM, is versatile to a variety of VFMs.• Experimentally, the proposed FADA significantly outperforms the state-of-the-art DGSS methods, and yields an improvement up to 2.9% mIoU over the contemporary REIN [69] .The Haar transform [60] cross-multiplies a function with various shifts and stretches, which has been demonstrated to be effective in applications, such as signal and image processing. In other words, it is able to analyze the local aspects of a signal.Definition 1. Haar Scaling Function. Given an input signal x, the Haar scaling function is mathematically defined asEQUATION)Given the space of all functions of the form k∈Z a k ϕ(x -k) as V 0 , where k ∈ Z is an arbitrary integer, and a k ∈ R. As each element of V 0 is zero outside a bounded set, such a function a k ϕ(x -k) has finite or compact support.Definition 2. Basis of the Step Function Space. Given an arbitrary nonnegative integer j ∈ Z + 0 , Let V j denote the step function space at the level j, which is spanned by the setEQUATION)Theorem 1. (1) A function f (x) belongs to V 0 ⇐⇒ f (2 j x) belongs to V j . (2) A function f (x) belongs to V j ⇐⇒ f (2 -j x) belongs to V 0 . Proof. (1) If f (x) ∈ V 0 , then we have f (x) = k∈Z a k ϕ(x -k), where a k ∈ R. Then, f (2 j x) = k∈Z a k ϕ(2 j x -k). It means f (2 j x) ∈ V j . (2)The proof of ( 2) is similar. Theorem 2. The set of functions {2 j/2 ϕ(2 j x -k), k ∈ Z} is an orthonormal basis of V j .Proof. The norm of a certain basis2 j/2 ϕ(2 j x-k) is |2 j/2 ϕ(2 j x-k)| = 2 j/2 |ϕ(2 j x-k)| = 2 j/2 • 1 2 j/2=1. ( 2) For any two basis m and n (m̸ = n), < 2 m/2 ϕ(2 m x -k), 2 n/2 ϕ(2 n x -k) > = 2 m/2 • 2 n/2 • < ϕ(2 m x -k), ϕ(2 n x -k) > = 0.Thus, the set of functions {2 j/2 ϕ(2 j x -k), k ∈ Z} is an orthonormal basis of V j . Definition 3. Haar Wavelet Function. The Haar wavelet is the function ψ(x) = ϕ(2x) -ϕ(2x -1).Theorem 3. The Haar wavelet function ψ(x) ∈ V 1 , and is orthogonal toV 0 . Lemma 1. Any function f 1 (x) = k∈Z a k ϕ(2x -k) ∈ V 1 , i.e., orthogonal to each ϕ(x -l), l ∈ Z if and only if a 1 = -a 0 , a 3 = -a 2 , • • • Proof. Given ϕ(x) ∈ V 0 , if f 1 (x) = k∈Z a k ϕ(2x -k) ⊥ V 0 , then k∈Z a k ϕ(2x -k) ⊥ ϕ(x). Therefore, f 1 (x) ⊥ V 0 if and only if < k∈Z a k ϕ(2x -k), ϕ(x) >= 0. As ϕ(x) = ϕ(2x) -ϕ(2x - 1), we have < k∈Z a k ϕ(2x-k), ϕ(2x)-ϕ(2x-1) >= a 0 < ϕ(2x), ϕ(2x) > +a 1 < ϕ(2x-1), ϕ(2x-1) >= 0. (15) Thus, a 0 = -a 1 . Similarly, by inspecting < k∈Z a k ϕ(2x -k), ϕ(x -1) >= 0, we have a 2 + a 3 = 0, • • • .Proof of Theorem 3. Based on Lemma 1, we haveEQUATIONA.2 Impact on Token Length m By default we set the token length m in the proposed FADA as 100. To study its impact on the unseen target domain, the scenarios when m are set 25, 50, 75, 125, 150 and 175 are tested. CityScapes is used as the source domain. The results on B, M, G and S unseen target domains are displayed in Fig. 8 a, b, c and d, respectively. It is observed that, the generalization performance when m ranges from 75 to 125 is relatively stable, while the performance when m is too small (i.e., 25, 50) or too large (i.e., 150, 175) shows a slight decline. 7RNHQ6HTXHQFHm The high-level idea of our intuition focuses on the frequency space. As existing LoRA based and side-adapter based methods implement the adaptation on each of the transformer layer, therefore we treat the frequency space as a whole, and embed it into each transformer layer.P,R8 )$'$2XUV (a) C to B 7RNHQ6HTXHQFHm P,R8 )$'$2XUV (b) C to M 7RNHQ6HTXHQFHm P,R8 )$'$2XUV (c) C to G 7RNHQ6HTXHQFHm P,R8 )$'$2XUV (d) C to SNevertheless, it would be meaningful to have an in-depth analysis on the learning behaviour from shallow to deep. To this end, apart from the REIN baseline [69] and the proposed FADA, we further provide two experiments, where the frequency adapter is attached in the first half seven layers (denoted as shallow) and the second half seven layers (denoted as deep), respectively. Results in Table 6 show that:• Using the frequency adapter on the first half layers (shallow) shows a slightly better performance than on the second half layers (deep). It may be explained that the shallower features contain more cross-domain styles, such as illumination, landscape and etc. • Using the frequency adapter on all layers (ours) achieves the best performance, indicating its effectiveness on all layers.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: This paper proposes to learn style-invariant property from vision foundation model from the frequency space by Haar wavelet transformation. 
related work: 
 Domain Generalization handles the challenging setting where the feature distribution of an arbitrary target domain is not identical to that of the source domain. It has been extensively studied in the past few years. Multiple techniques, to namely a few, optimal transport [22, 79] , batch normalization [66] , causal inference [48, 47, 25] , discrepancy regularization [80, 68, 17, 5] , and uncertainty modeling [59, 70] , have been proposed. Furthermore, domain generalization via unsupervised learning [26, 27] or from a single source domain [61, 59, 84, 75] has also been recently studied.Domain Generalization by Frequency Decoupling has drawn increasing attention. Its general idea rests in that the style and content have been demonstrated on high-frequency and low-frequency components [29, 46, 40, 77, 67, 7, 6] , respectively. Most of these methods implement Fast Fourier Transform (FFT) to transfer the image to the frequency space, and then represent the style and content by the amplitude (high-frequency) and phase (low-frequency) components, respectively. However, to the best of our knowledge, 1) leveraging Haar wavelet for domain generalization; and 2) enhancing the generalization ability of VFM features via frequency space have been rarely explored. Compared with other frequency analysis methods such as FFT, the orthogonal property of Haar wavelet basis leads to a stronger decorrelation [50] . In the context of domain generalization, it indicates a better separation between low-and high-frequency components and deserves exploration.Domain Generalized Semantic Segmentation (DGSS) in the CNN era either decouple the style information [55, 30, 56, 14, 58, 71, 78] or enrich the style diversity [41, 82, 83, 45, 52] . With the rapid development of Vision Transformer (ViT), recent DGSS methods usually leverage the masked attention mechanism [12, 13] to enhance the content representation [19, 8, 7] . Later, the masked attention is used to decode the frozen contrastive image-text pre-trained features [33] , and REIN [69] fine-tunes the VFM under the low-rank adaptation paradigm. However, the style invariant properties of VFM, which are the key of the DGSS representation, remain unexplored.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: At the end of the conclusion section, the limitation of the proposed method has been discussed.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
low-rank adapted vfm: 
 To fine-tune a VFM with parameter efficiency, the low intrinsic dimension [42, 1] assumes that a VFM relies on the intrinsic low-dimension in the frozen VFM features to adapt to the downstream tasks. Inspired by this, the low-rank adaptation (LoRA) paradigm [28] is devised to inject trainable rank decomposition matrices into each layer. Given a VFM with N sequential layers (denoted asL 1 , L 2 , • • • , L N ), each layer corresponds to a pre-trained weight matrix of W 1 , W 2 , • • • , W N , i.e., W i ∈ R c×c .The frozen features from layer L i are denoted asf i ∈ R c×n . Particularly, for the first layer L 1 , f 1 is generated by f 1 = W 1 x.Here x denotes the image embedding, n denotes the patch number, and c denotes the channel size.Denoting the learnable weight matrix as ∆W i ∈ R c×c and the input of layer L i as f i , the feature propagation from the layer L i to L i+1 can be formulated as f i+1 = W i f i + ∆W i f i . Then, we assume that the learnable weight matrix ∆W i can be formulated as a low-rank decomposition, i.e., ∆W i = BA, where B ∈ R c×r and B ∈ R r×c (r ≪ c). More recently, REIN [69] specifies this paradigm in DGSS by transferring the learnable matrix term ∆W i into a learnable token T i followed by a MLP M i (•), denoted asf i+1 = W i f i + M i (T i (W i f i )), where T i ∈ R m×c and m is the token length. This modification allows a significance reduction of the token length (from a thousand magnitude c to a hundred or even ten magnitude m), which can alleviate Curse of Dimensionality and allow each token to be better connected to the instances in an image [69] . Our low-rank adaptation is implemented on T i , i.e., T i = A i B i , where A i ∈ R m×r and B i ∈ R r×c (r ≪ min(m, c)). The Haar transform [60] cross-multiplies a function with various shifts and stretches, which has been demonstrated to be effective in applications, such as signal and image processing.Definition 1. Haar Scaling Function. Given an input signal x, the Haar scaling function is mathematically defined asEQUATIONLet V 0 denote the space of all functions of the form k∈Z a k ϕ(x -k), where k ∈ Z is an arbitrary integer, and a k ∈ R. As each element of V 0 is zero outside a bounded set, such a function a k ϕ(x -k) has finite or compact support.Definition 2. Basis of the Step Function Space. Given an arbitrary nonnegative integer j ∈ Z + 0 , Let V j denote the step function space at the level j, which is spanned by the setEQUATIONDefinition 3. Haar Wavelet Function. The Haar wavelet is the function ψ(x) = ϕ(2x) -ϕ(2x -1).For more details of the properties of the Haar transform, please refer to the supplementary material.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: The computation resources and details are discussed in experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
methodology: 
 Fig. 2 gives an overview of the proposed FADA. After each frozen VFM layer, it consists of three key steps, namely, low-/high-frequency decomposition (in Sec. 4.1), low-frequency adaptation (in Sec. 4.2), and high-frequency adaptation (in Sec. 4.3). Finally, the frequency components are fused together and transferred from the frequency space back to the spatial space, and then fed to the next VFM layer.Orthogonal property (in Sec. 3.2) leads to the result of a strong decorrelation [50] . This property after the Haar wavelet function allows a better separation between low-and high-frequency components of input signal than other frequency analysis methods such as Fourier transform. In the context of DGSS, the domain gap is caused by the cross-domain style variation, while the cross-domain content is stable [14, 78, 71, 58] . As it has been well documented that the style and content are predominate in the high-frequency and low-frequency components, respectively [29, 46, 40, 77, 67] , a feasible way to explore the style-invariant properties is to mitigate the variation of high-frequency components in the VFM features. Therefore, the first step is to decompose the frozen VFM feature f i into the lowand high-frequency components, respectively.In our work, we exploit the Haar wavelet transform to decouple the low-and high-frequency components, where four kernels, namely, LL T , LH T , HL T , HH T , are given byEQUATIONAs discussed in prior works [60, 3] , the LL T kernel captures the average of the pixel responses, which is more robust to the scene content and therefore preserves more low-frequency components. In contrast, by taking the differences between adjacent pixels into account, the LH T , HL T and HH T kernels tend to preserve the details from the horizontal, vertical and diagonal directions, respectively. These details are more related to the structures, edges, etc, which attribute to the style.For a layer L i , we implement the Haar wavelet transform on the frozen VFM feature W i f i by the above four kernels LL T , LH T , HL T and HH T , respectively. The f LL i component filtered by LL T captures more scene content, and in contrast the f LH i , f HL i and f HH i components filtered by LH T , HL T and HH T capture more style information. This decomposition is computed asf LL i = (Wifi) ⊗ LL T , f LH i = (Wifi) ⊗ LH T , f HL i = (Wifi) ⊗ HL T , f HH i = (Wifi) ⊗ HH T . (4)In the context of DGSS, a stable scene content representation despite the style variance is important to predict the scene semantics. For each layer L i , the scene content from the frozen VFM features rests more in the low-frequency component f LL i (in Eq. 4), which is learned in the low-frequency adaptation branch.Assume we have a low-frequency token T L i ∈ R m×c , where m is the sequence length of T L i , and c is the dimension of the frozen VFM feature defined in Sec. 3.1. The low-frequency token T L i is used to exploit the scene content from the low-frequency component f LL i , while at the same time following the low-rank adaptation paradigm (in Sec. 3.1).First, we compute a similarity map S i ∈ R n×m between the token T L i and low-frequency component f LL i from the frozen VFM feature, which measures the correlation between each element in T L i and each patch embedding represented in f LL i , given byEQUATIONwhere Softmax denotes the softmax activation function.Then, we project the token feature T L i into the feature space of f LL i by a multilayer perceptron (MLP) parameterized by weight parameters W 1 i and bias parameters b 1 i , followed by the point-wise product with the similarity map S L i . The product with S L i allows the token features T i to better align to f LL i , where the scene content is highlighted. Assume the output is denoted as f LL i . Briefly, this process can be mathematically expressed asEQUATIONThen, we fuse the projected token features f LL iwith the low-frequency features f LL i by another MLP parameterized by weight parameters W 2 i and bias parameters b 2 i , followed by the skip connection. Assuming the output is f LL i , this process can be mathematically computed asEQUATIONFor DGSS, the robustness to the cross-domain style variance is particularly important. Such style difference is usually reflected on the high-frequency components. The Haar wavelet transform enables the separation of these high-frequency components f LH i , f HL i and f HH i (in Eq. 4). Directly eliminating all the high-frequency components seems to be a simple and straight-forward solution, but it also leads to the loss of other information such as structure and object boundary. It may degrade a scene representation and decline the segmentation performance. Therefore, the objective of the high-frequency adaptation branch is to mitigate the impact of cross-domain style variation, not directly removing all the high-frequency components.As the decoupling of styles does not differentiate whether the high-frequency components are from the horizontal, vertical or diagonal directions, for simplicity, we concatenate them together for processing in this branch. Specifically, still in layer L i , assume we have a high-frequency tokenT H i ∈ R 3m×c .The token size of T H i is tripled compared with the token size of T L i , as three highfrequency components are involved. Similar to the low-frequency branch, the high-frequency token T H i isEQUATIONwhere [•, •] denotes the concatenation operation.Then, the highlighted positions in S H i reveal the predominant style responses from the source domain images. The high responses, which reflect more domain-specific styles, are supposed to be suppressed during training. It allows the fine-tuned VFM features to be less impacted by the domain-specific styles. Instance normalization [55, 31] , which computes the channel-wise mean and standard deviation, is effective to eliminate the styles. To this end, an instance normalization is implemented on the feature-token similarity map S H i , given byEQUATIONThen, we project the token feature T H i into the high-frequency feature space by a multilayer perceptron (MLP) parameterized by weight parameters W 3 i and bias parameters b 3 i , followed by the point-wise product with the similarity map S H i . The product with S H i allows the token features T H i to better align to the decoupled high-frequency features, which is less relevant to the source domain. Assume the output is denoted as f H i . Briefly, this process can be mathematically expressed asEQUATIONAfterwards, we fuse the projected token features f H i with the high-frequency features f H i by another MLP parameterized by weight parameters W 4 i and bias parameters b 4 i , followed by the skip connection. Assuming the outputs of these three components are f LH i , f HL i and f HH i , this process can be mathematically computed asEQUATIONFinally, the low-frequency component f LL Same as the REIN [69] baseline, the loss function L of FADA directly inherits the losses from the Mask2Former decoder [12] , given byEQUATIONwhere L ce , L dice and L cls denote the cross-entropy loss, dice loss and classification loss. Here the hyper-parameters λ ce , λ dice and L cls are 5.0, 5.0 and 2.0, respectively.By default we use DINO-V2 [53] as the frozen VFM, but the proposed FADA is also feasible to other VFMs. For fair evaluation, the Mask2Former segmentation decoder [13] is used to generate the pixel-wise prediction as REIN does. Same as the existing paradigm [69] , the images are re-sized to 512 × 512 pixels before input to the models. The Adam optimizer with an initial learning rate of 1 × 10 -4 is used to train the model. The training process terminates after 20 epochs. 
datasets & evaluation protocols: 
 Five driving-scene semantic segmentation datasets that share 19 common scene categories are used for validation. Specifically, CityScapes (C) [16] consists of 2,975 and 500 images for training and validation, respectively. The images are captured under the clear conditions in tens of Germany cities. BDD-100K (B) [76] has 7,000 and 1,000 images for training and validation, respectively. The images are captured under diverse conditions from a variety of global cities. Mapillary (M) [51] is another large-scale semantic segmentation dataset, which consists of 25,000 images from diverse conditions. SYNTHIA (S) [64] is a synthetic driving-scene segmentation dataset, which has 9,400 images. GTA5 (G) [63] is another synthetic dataset, which has 24,966 simulated images from the American street landscape.Following the evaluation protocol of existing DGSS methods [55, 56, 14, 58] , a certain dataset is used as the source domain for training and the rest four are used as unseen target domains for validation. Three commonly-used evaluation settings are: 1) G → C, B, M, S; 2) S → C, B, M, G; and 3) C → B, M, G, S. The evaluation metric is mean Intersection of Union (mIoU, in percentage %). All of our experiments are implemented and averaged by three independent repetitions, starting from different random seeds.Existing DGSS methods are involved for comparison: 1) ResNet based methods, namely, IBN [55] , IW [56] , Iternorm [31] , DRPC [78] , ISW [15] , GTR [57] , DIRL [71] , SHADE [82] , SAW [58] , WildNet [41] , AdvStyle [83] and SPC [32] ; 2) Mask2Former based methods, namely, HGFormer [19] and CMFormer [8] ; 3) VFM based methods, namely, DIDEX [52] and REIN [69] . By default, the performance is directly cited from prior works [55, 56, 14, 58] . '-' denotes that the authors did not reported the results nor provided source code. '*' denotes re-implementation with official source code under all default settings.GTA5 Source Domain. From left to right, the third column of Table 1 reports the performance. Compared with the VFM based REIN [69] , the proposed FADA shows an mIoU improvement of 1.83%, 1.54%, 1.99% and 1.50% on the C, B, M and S target domains, respectively. In addition, the proposed FADA shows an average mIoU improvement of 20% and 10% when compared with ResNet and Mask2Former based DGSS methods, respectively.SYNTHIA Source Domain. The fourth column of Table 1 shows that the proposed FADA achieves the state-of-the-art performance, outperforming the REIN by 1.45%, 1.41%, 1.22% and 1.29% mIoU on the C, B, M and G unseen target domains, respectively. In addition, the proposed FADA shows an average mIoU improvement of 15% and 6% over ResNet and Mask2Former based DGSS methods.CityScapes Source Domain. The last column of Table 1 shows that the proposed FADA shows an mIoU improvement of 1.58%, 1.83%, 1.37% and 1.19% on the B, M, G and S unseen target domains, respectively. In addition, the proposed FADA shows an average mIoU improvement of 15% and 5% over existing ResNet and Mask2Former based methods, respectively.On Each Haar Component. Five settings are involved for experiments: (1) No wavelet components are used. The model fine-tunes on the frozen VFM, which is a simplified version of REIN [69] removing the instance link module; (2) Only fine-tuning on the low-frequency component f LL i , and (5) fine-tuning on all the four frequency components (Ours). Table 2 reports the results. Harnessing the low-frequency features leads to an average of 1% mIoU improvement on most experimental settings. In addition, fine-tuning on each high-frequency component leads to a slight improvement on unseen target domains. It demonstrates the necessity to decouple the impact of cross-domain styles.→ C → B → M → S → C → B → M → G → B → M → G → SOn Low-Rank Dimension r. By default we set the low-rank dimension r as 16. We further test the generalization performance when r is 4, 8, 32 and 64, respectively. Table 3 reports the performance when CityScapes is used as the source domain. When r is 16 or 32, the performance on unseen target domains shows the most stable performance. However, when r is too small (e.g., 4 or 8) or too large We extract the three high-frequency components from the last VFM layer, and display them by t-SNE visualization. The feature space without (denoted as w.o.) and with (denoted as with) implementing the instance normalization function is visualized in the first and second row of Fig. 5 , respectively. The experiments are conducted under the C → B, M, G, S setting. It is observed that the implementation of instance normalization allows the samples from different unseen target domains to be more uniformly distributed, indicating its effectiveness to mitigate the domain-specific information containing in the high-frequency components.To Different VFMs. We test the translation ability of the proposed FADA to other VFMs, namely, CLIP [62] , SAM [39] and EVA02 [20] . For comprehensive evaluation, each VFM is validated under full-training, fine-tuning (REIN), or frozen scheme [69] , respectively. The reported one decimal results are directly cited from [69] . Experiments are conducted under the G → {C, B, M} setting. Table 4 shows the superiority of FADA when embedded into these VFMs. For comparison with parameter-efficient fine-tuning (PEFT) methods, please refer to Table 7 in the supplementary material.To Adverse Domains Adverse Conditions Dataset with Correspondence (ACDC) [65] is a semantic segmentation dataset that consists of samples from four types of adverse conditions, namely, rain, fog, night and snow. Table 5 shows that the proposed FADA outperforms existing DGSS methods by up to 0.7%, 1.5%, 2.5% and 2.9% on the fog, night, rain and snow domains, respectively. , SAW [58] , WildNet [41] , SPC [32] , CMFormer [8] , and REIN [69] ) and FADA under the C →{B, G, M, S} setting.Unseen images ISW SAW REIN Ours WildNet SPCFigure 7 : Exemplar segmentation results of existing DGSS methods (ISW [14] , SAW [58] , WildNet [41] , SPC [32] , CMFormer [8] , and REIN [69] ) and FADA under the C → four ACDC setting.Some exemplar segmentation results are compared under the C → B, M, G, S and C → adverse domains are provided in Fig. 6 and Fig. 7 , respectively. The proposed FADA shows better pixel-wise prediction than not only ResNet based methods (i.e., ISW [14] , SAW [58] , WildNet [41] , and SPC [32] ) and Mask2Former based methods (i.e., CMFormer [8] ), but also VFM based REIN [69] . 
conclusion: 
 In this paper, we focused on adapting VFM for DGSS by exploiting the style-invariant properties from the VFMs, and presented a novel Frequency-ADApted learning scheme to push this frontier. Concisely, Haar wavelet transform was introduced to decouple the frozen VFM features into low-and high-frequency components, which contain more scene content and style information, respectively. We innovatively modified the low-rank adaptation paradigm to both frequency features, and alleviated the impact of cross-domain variation on high-frequency features. Consequently, the model achieved a better generalization on unseen target domains. Extensive experiments and ablation studies on a variety of settings showed the effectiveness of the proposed FADA.Limitation Discussion & Broader Societal Impact. The proposed FADA handles the low-and highfrequency features separately, which increases the trainable parameters compared with prior work (Table 4 ). However, the increase of about 6M parameters is acceptable. The proposed FADA advances the reliability and safety of autonomous driving and alleviates human involvement, benefiting the human well-being. We do not envision any negative social impact. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: At the end of the conclusion section, the broader impacts have been discussed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This work does not have such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The asserts used in this paper are all public available for academic researches. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.We further compare the proposed method with some existing parameter-efficient fine-tuning (PEFT) methods, namely, AdvStyle [83] , PASTA [9] , GTR-LTR [57] , LoRA [28] , AdaptFormer [10] , VPT [38] and REIN [69] . Following the setting in REIN [69] , GTAV is used as the source domain. BDD, CityScapes and Map are used as unseen target domains. Results in Table 7 show that the proposed FADA outperforms these methods on all unseen target domains.Fig. 9 and Fig. 10 show more results under C → B, M, G, S and C → ACDC setting. On both settings, the segmentation results show that the proposed FADA shows better pixel-wise prediction than the compared DGSS methods, especially in terms of the completeness of objects. The proposed FADA is compared with ISW [15] , SAW [58] , WildNet [41] , SPC [32] and Rein [? ]. [15] , SAW [58] , WildNet [41] , SPC [32] and Rein [? ].• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The authors have read the code of ethics. The experiments are all on public datasets without obeying the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: Dear authors, 
Overall this draft got positive reviews (R:5 / C:5, R:4 / C:4,  R:6 / C:4, R: 7 / C:3,  R:5 / C:2,  R:5 / C:1 ). Two reviewers updated their rating to positive side. 
After careful review of the reviewer comments, answers by the authors, and draft itself, we are accepting the draft. However, authors are strongly encouraged to improve the readability of the paper, incorporate suggestions by the reviewers and handle the comments. 
For example,  
1) "advantages of Haar wavelet over other methods or provide comparative analysis in this context."  --- Needs to be  added with clarity and given due importance in the. text.
2) Role of Instance Normalization for style invariance needs more explanation and study (even if added to supplementary material and referred in the text). 
3) Fig.1's. caption should include basic information to read the figure. LL, HH, etc.. are defined much below in the text rather than before. Therefore, add this information in the caption.  At the same time  statements like "Domain Generalized Semantic Segmentation (DGSS) learns to generalize to arbitrary unseen target domains when only trained on the source domain; The key challenge lies in the stability
of the scene content, while the domain gap is caused by the style ..."  should be part of the text rather than caption. 
4) Table-1 either explain what is C, B and M are or explicitly refer to the text where these are properly defined. 
5) incorporate experiments using to reply to the author's questions.
etc..

Reviews: review: paper_summary: paper_summary: This paper leverages the vision foundation model (VFM) for domain-generalized semantic segmentation (DGSS). It proposes to adapt the VFM to the downstream task in the frequency domain. The proposed method decouples the low-frequency and high-frequency components of the VFM features and separately applies low-rank adaptation to them. This aims to stabilize the domain-invariant content and, on the other hand, eliminate the impact of domain-specific styles on DGSS.
strength_weakness: strength_weakness: * **Insufficient justification of key concepts:** The paper's claim that domain-specific style and domain-invariant content can be demonstrated by high-frequency and low-frequency components is not well-justified in this context. And, the explanation of how the proposed method stabilizes domain-invariant content while mitigating domain-specific style is insufficiently supported.

* **Lack of detailed analysis on style elimination:** Given the stated importance of style-invariant properties in VFM for DGSS representations, eliminating the impact of domain-specific styles is crucial. While the paper uses instance normalization to address this, it lacks a detailed analysis of the effectiveness of this approach.

* **Unclear representation of Haar components:** Although the paper suggests that the three high-frequency components represent domain-specific style information that should be eliminated for better DGSS, Table 2 indicates that the best results are achieved by preserving all components. The discussion around this in Line 250 is not sufficiently informative.

* **Inadequate comparison with previous methods:** The paper also mentions previous works that have utilized FFT for frequency analysis and claims that it may be the first to use Haar wavelet for domain generalization. However, it does not adequately discuss the advantages of Haar wavelet over other methods or provide comparative analysis in this context.
questions: questions: VFMs are known for their strong generalization ability across different domains. What specific style-invariant properties of VFM are the authors attempting to exploit in this paper?
limitations: limitations: Please provide the necessary evidence and adequate clarification of the problems in the weaknesses.
paper_summary: paper_summary: This paper focuses on adapting the vision foundation model for domain-generalized segmentation via frequency-aware adaptation. Specifically, the intermediate features are decomposed into low-frequency (content) and high-frequency (style) components, which are then being processed with separate low- and high- frequency branches. To this end, sufficient experiments on extensive datasets validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. When conducting the adaptation, it is intuitive that the low-frequency/ high-frequency components correspond to the content/style. However, can the author comment on the intuition on adapting in all the transformer laters? It may not be that clear whether the low-frequency and high-frequency features in the later layers have such correspondence anymore. From that sense, it would be nice to ablate the layer location where the frequency adapter is added (as opposed to being added to all the layers) 


2. When comparing with the REIN, REIN only has a trainable parameter of 2.99M (compared to FADA 11.65), the author also shows in Table 3 that FADA performance is influenced by the number of Rank and tunable parameters. It would be nice to add an experiment that compares these two methods with similar tunable parameters. 


3. The core intuition behind the decomposition is to prioritize the domain-invariant low-frequency features (content features) over the high-frequency features. However, as indicated by the ablation results in Table 2, better outcomes were achieved by adapting to all features. The current model design does not differentiate between high-frequency and low-frequency features, raising questions about the alignment between the model design and the initial intuition. A potential experimental setup that might clarify this involves training the model on multiple domain datasets with a shared low-frequency branch and distinct high-frequency branches. However, this would alter the experimental conditions. I would appreciate the authors' comments on this matter.

4. Conducing Haar transform can cause additional inference time, it would be nice to include the inference time comparision.
questions: questions: See the weakness section.
limitations: limitations: Limitations have been discussed.
paper_summary: paper_summary: This paper introduces a novel FADA learning scheme to improve domain-generalized semantic segmentation. The proposed method leverages the Haar wavelet transformation to separate style and content information into high- and low-frequency components, respectively, allowing for better handling of domain variations. Experimental results demonstrate that FADA outperforms existing DGSS methods, showcasing its effectiveness and versatility across various vision foundation models (VFMs).
strength_weakness: strength_weakness: 1. Lack of comparison with other recent low rank adaptation methods during the experiments, e.g., the works as follows,

[1] Gao, Z., Wang, Q., Chen, A., Liu, Z., Wu, B., Chen, L., & Li, J. (2024). Parameter-Efficient Fine-Tuning with Discrete Fourier Transform. arXiv preprint arXiv:2405.03003.
[2] Yang, Y., Chiang, H. Y., Li, G., Marculescu, D., & Marculescu, R. (2024). Efficient low-rank backpropagation for vision transformer adaptation. Advances in Neural Information Processing Systems, 36.


2. Lack of insights in the experimental analysis, e.g., in section 5.2, the authors simply listed out the performance gains brought by the proposed method compared with the state-of-art methods without mentioning why it can achieve these superior performances.

3. The ground truth images are not listed on Figure 5 and Figure 6, which hinders the comparison regarding the qualitative performances.


4.Lack of description of the implementation details. What losses are used during the experiments? Which lr scheduler is used?
questions: questions: 1. Can the proposed low rank adaptation method outperforms the other recent low rank adaptation methods as mentioned in weakness 1?


2. The authors are encouraged to add more insights towards the proposed method in the experiment analysis section.

3. The ground truth figures are suggested to be added in Figure 5 and Figure 6-

4. The implementation details are encouraged to be enriched.
limitations: limitations: yes it is mentioned be the authors.
paper_summary: paper_summary: This paper proposes a Frequency Adaptive (FADA) learning approach. Its core idea is to process content and style information separately, by using frequency tokens. Specifically, the FADA comprises two branches for low-frequency and high-frequency components. The high-frequency components learn scene styles and eliminate their impact on DGSS, while the lower-frequency components contribute to stabilizing scene contents. Experiments in various DGSS settings demonstrate FADA's performance and versatility across different VFMs.
strength_weakness: strength_weakness: Honestly, I do not know well about this topic.
So I would like to ask some questions from the general perspective.

1. Is the assumption of "similar content yet varied style" practical? It seems that the main target of this paper is autonomous driving. Then it is rational to assume almost the same categorical distribution would be observed over various data, including the unseen test one. However, how about the joint distribution of content and style? Can we assume that they are independent across the dataset?

2. Can I see the results on the other VFMs? I understand that DINO-V2 is one of the most popular choices, but I am personally interested in whether the proposed method functions well on a more segmentation-oriented model, such as SAM. I think replacing DINO-V2 with the SAM image encoder can be an interesting experiment.
questions: questions: See the weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes a Frequency-Adapted (FADA) learning scheme, where Haar wavelet transformation is introduced to decouple the frozen VFM features into low- and high-frequency components. Experiments demonstrate the proposed method achieves a better generalization on unseen target domains.
strength_weakness: strength_weakness: 1. The writing and logic are poor. For example, the relationship among the existing three DGSS category methods is missing. Why explore the style-invariant properties of VFM is important and urgent? The introduction is not accompanied by an explanation of Figure 1.
2. What are the advantages of 1) leveraging Haar wavelet for domain generalization and 2) enhancing the generalization ability of VFM features via frequency space in Domain Generalization by Frequency Decoupling over existing related works?
3. [1] also focuses on mitigating the effects of style variations on the DGSS. What is the difference between the paper with [1]?
[1] Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning. CVPR 2024
questions: questions: I'm more concerned about [1] also focusing on mitigating the effects of style variations on the DGSS. What is the difference between the paper with [1]? Besides, why is exploring the style-invariant properties of VFM important and urgent? What are the advantages of 1) leveraging Haar wavelet for domain generalization and 2) enhancing the generalization ability of VFM features via frequency space in Domain Generalization by Frequency Decoupling over existing related works?
[1] Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning. CVPR 2024
limitations: limitations: The proposed method heavily relies on Low-/High-Frequency Decomposition. However, the manner is fixed and not adjusted dynamically.
paper_summary: paper_summary: This paper deals with adapting a computer vision foundational model to a new domain.  They propose to use the frequency based on Haar wavelets to  to decouple the style and content information and then to address separately  content and style  domain adaptation. Experiments show state of the art results.
strength_weakness: strength_weakness: The adaptation for low-frequency and high-frequency components are different. The paper could provide more intuition to explain the particular adaptations chosen for low and high frequency components.
questions: questions: see above
limitations: limitations: Yes

Comments: title: Re: Official Comment by Reviewer eWsS
Re: Official Comment by Reviewer 4xQk
Re: Official Comment by Reviewer nB2o
Re: Response to Reviewer 3Mxh
Response to AC
Response to the author
comment: We would like to express our gratitude to be recognized by the reviewer, and are glad that your concerns have been resolved. 

We will improve our work carefully per your suggestions.
Thank the author for the detailed feedback, this solves most of my concerns. I will keep the original rating to reflect my confidence of this paper
We are glad to see your questions resolved, and appreciate your positive feedback after the rebttual.
We will improve our work carefully per your suggestion.
Thanks to the efforts put in by the author for rebuttal. These answers have solved my concerns and I am raising my score to 5: Borderline accept.
We are glad that your concerns have been clearly addressed. We will improve our work carefully per your suggestions. Thanks again for your time and effort.
Thank you for the rebuttal.
As my concerns are clearly addressed, I would like to increase my rating for now. 
However, as I do not know much about this topic, I think I should keep my rating in the range of borderline.
Probably AC will consider my opinion properly.
We would like to express our gratitude to be recognized by the reviewer, and are glad to see your questions resolved. We will improve our work carefully per your suggestions.
Dear AC,

my concerns are solved by the authors, thereby I would like to change my rating to 7.
Thank you for your effort during the rebuttal. My concerns are addressed by the authors, thereby I would like to improve my rating to 7.