Learning Spectral-decomposited Tokens for Domain Generalized Semantic Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-23 22:36:07

Abstract: The rapid development of Vision Foundation Model (VFM) brings superior out-domain generalization for a variety of down-stream tasks.
Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information (i.e., semantics) but vary greatly in terms of the style variation (e.g., urban landscape, environment dependencies).  
How to effectively fine-tune VLM for DGSS has recently become an open research topic for the vision community.
In this paper, we present a novel Spectral-decomposited Tokens (SET) learning framework to push the frontier.
Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on how to learn style-invariant features from these learnable tokens. 
Specifically, the frozen VLM features are first decomposited into the phase and amplitude component respectively in the frequency space, where the phase / amplitude component reflects more on the content / style, respectively. 
Then, learnable tokens are adapted to learn the content and style, respectively.
As the cross-domain differences mainly rest in the style from the amplitude component, such information is decoupled from the tokens.
Consequently, the refined feature maps are more stable to represent the pixel-wise content despite the style variation. 
Extensive cross-domain experiments under a variety of backbones and VFMs show the state-of-the-art performance.
We will make the source code publicly available.

Main paper: 
 introduction: 
 The emergence of various Vision Foundational Models (VFMs) has started a new era for semantic segmentation. Most of these methods assume that the unseen target domains in the reference stage share the independent and identical distribution (i.i.d.) with the accessed source domain in the training stage. In the real-world applications like autonomous driving, this assumption usually does not necessarily hold. As a matter of fact, large style variations between the source and unseen target domains can be witnessed due to many shifting factors such as urban landscape, weather, and lighting conditions [10, 28, 45] .Domain generalized semantic segmentation (DGSS) is the task to address this challenge, in which the segmentation models are trained only on the source domain but inferred on arbitrary unseen target domains [1, 2, 16] . Before the VFM era, extensive efforts have been made on DGSS, which focus on either style decoupling [1, 2, 12, 16, 36, 37] or style augmentation [21, 26, 60, 62] . However, the representation ability of the scene content (e.g., semantics) itself, which is relatively stable between different domains, remains less explored. In recent years, VFMs (e.g., CLIP [41] , DALL-E [42] , DINOv2 [32] , and SAM [24] ) have significantly advanced a variety of vision tasks. The inherited generalization ability of VFM from large-scale image pre-training [25] has great potential to be harnessed for DGSS.Figure 2: A toy example on how different styles of a driving scene (left row) impact the amplitude (second row) and the phase component (third row). As the styles change, the phase component remains stable, while the amplitude adjusts correspondingly. Thus, the amplitude component provides a feasible path to inspect the cross-domain style variation.Recent works show that, in the context of DGSS, fine-tuning VFM with learnable tokens yields better generalization than the one with full parameter tuning [51] . However, the key challenge of DGSS, i.e., the domain gap caused by the style variation, remains unaddressed. Hence, we naturally raise an open research question: how to learn style-invariant representation by fine-tuning VFMs? Spectral decomposition has been long acknowledged effective to handle the style and content information separately in the context of domain generalization [53, 55] , where the style/content information highly rests in the high-/low-frequency component [49] . This further leads to our second research question: how to design a frequency-space-based method to effectively composite the style and content information from the VFM features?In this paper, we propose a novel Spectral-dEcomposited Token (SET) learning scheme to address this challenge. Fast fourier transform (FFT) [3] is implemented to decompose the image feature from the previous frozen VFM layer to the amplitude and phase components. The phase component, which contains more low-frequency component on the scene content, is relatively stable despite the cross-style variation [38, 54] . In contrast, the amplitude component, which has more high-frequency component on the styles [6, 55] , can be rather unstable when handling images from different domains. Therefore, the token-based fine-tuning in the amplitude branch deserves more attentions in domain generalization.To this end, we design a token-based feature enhancement pipeline which extracts task-specific information and adjusts the spectraldecomposed features in amplitude and phase branches separately.Learnable tokens capture task-relevant features within each branch, and enhance the original feature representations based on their similarities. During inference, these tokens are fixed. However, since the amplitude component is susceptible to style variation, the enhancement process in the amplitude branch can become unstable due to the fluctuating representation and static tokens. Therefore, we further propose an attention optimization method to make the token-based enhancement more robust to style changes. Finally, the enhanced amplitude and phase components are projected back to the spatial space and input to the subsequent layer.We conduct extensive experiments in various domain-generalized semantic segmentation scenarios, where models are trained on one dataset from [14, 30, 43, 44, 56] as the source domain and validated on the remaining four datasets as unseen target domains. Although all datasets share the same 19 semantic categories, they differ in scene styles. Results demonstrate that our proposed SET achieves up to an improvement of: 1) 20% on mIoU over state-of-theart CNN-based methods [26, 36] ; 2) 10% mIoU over Mask2Former based methods [1, 16] ; 3) up to 3.12% mIoU over the VFM based Rein [51] on unseen ACDC-snow domain. Additionally, the proposed SET can be seamlessly transferred to a variety of VFMs with good generalization capability.Our contributions are summarized as follows:• We propose a novel spectral-decomposed token learning (SET) scheme for DGSS. It harnesses the style-invariant properties for VFM. • The proposed SET, consisting of three key steps, namely, spectral decomposition, learning spectral tokens, and attention optimization in the amplitude branch, can be seamlessly integrated into existing VFMs in a trainable manner. [15] , domain alignment [9, 47] , meta-learning [7, 40] , data augmentation [48, 58, 61] , selfsupervised learning [4] , and regularization techniques [23, 50] , have been adapted to learn the domain-invariant representation. However, the above works usually focus on the non task-specific settings, not especially devised for DGSS. In DGSS, the key challenge lies in that the cross-domain images share common content information (i.e., semantics) but vary greatly in terms of the style variation (e.g., urban landscape, environment dependencies). 
vision foundation models: 
 Foundation model is a new paradigm for deep learning. Its key idea is to pre-train a deep network on a board set of unlabeled images, which has a strong representation ability to be fine-tuned on a variety of down-stream tasks. This paradigm firstly emerges in the field of Natural Language Processing (NLP), and later also draws increasingly attention in the community of computer vision. For simplicity, in the following text, the foundation models in the computer vision field is termed as Vision Foundation Model (VFM).Here, we review several typical VFMs in the past few years: Contrastive Language-Image Pre-Training (CLIP) [41] acquires highfidelity visual representations via contrastive learning with largescale image-text pairs. Masked Auto-encoder (MAE) [18] employs a masked image modeling framework to derive latent image representations. Segment Anything Model (SAM) [24] pioneers a promptable model pre-trained on a diverse dataset for segmentation tasks. Explore the limits of Visual representation at scAle (EVA) [17] merges Masked Image Modeling pre-training with CLIP's vision features as the target of pretext tasks. Self-DIstillation with NO labels (DINO) [32] pre-trains the deep model on extensive, meticulously curated datasets without explicit supervision.To summarize, these VFMs have shown great successes on improving a variety of downstream tasks, underscoring their remarkable generalization capabilities. Nevertheless, an in-depth exploration of their effectiveness in the specialized realm of DGSS tasks remains rarely explored. The recent-developed Rein shows that fine-tuning VFM [8, 22] with learnable tokens yields the better generalization than the one with full parameter tuning [51] .Semantic segmentation in driving scenes can encounter great domain shift, caused by factors such as adverse weather, diverse illumination, and urban landscape diversity. In the past decade, domain adaptation methods [39, 59] have shown great successes, but they can only generalize to the target domain that has learned in the training stage. To generalize to arbitrary unseen target domains, DGSS has drawn increasing attentions in the past few years.In the convolutional neural network (CNN) era, DGSS methods can be categorized into two types, namely style decoupling and style augmentation. For style-decoupling-based methods, instance normalization (e.g., IBN [33] , Iternorm [20] ) and instance whitening (e.g., IW [34] , SAW [36] , ISW [12] , DIRL [52] ) operations are commonly used. For style-augmentation-based methods, external training images from other sources are usually used to enrich the domain diversity [21, 26, 37, 57, 60, 62] . Later in the Vision Transformer (ViT) era, DGSS methods usually leverage the strong content representation ability of mask attention mechanism [1, 2, 16] , which learns a more global-wise representation than convolution and is more robust to the cross-domain style variation.Nevertheless, the exploration of DGSS methods based on VFMs remains relatively limited. A recent work Rein shows that, finetuning VFM with learnable tokens yields better generalization for DGSS than full parameter tuning [51] . 
problem definition: 
 In DGSS, there are source domain 𝑆 = (𝑥 𝑠 , 𝑦 𝑠 ) and a number of target domains 𝑈 1 , 𝑈 2 , • • • , 𝑈 𝑛 , which have rather different feature distributions. Here 𝑥 𝑠 represents an image and 𝑦 𝑠 denotes its corresponding pixel-wise label. The target domain 𝑈 is inaccessible during training. Our objective is to train a VFM-based semantic segmentation model which is capable of achieving superior generalization performance when inferring on these target domains.The overview of the proposed Spectral-dEcomposed Tokens (SET) framework is shown in Fig. 3 . To efficiently fine-tune VFMs, spectral-decomposed tokens are applied to each frozen layer. Within each layer, the frozen features are decomposed into amplitude and phase components by Fast Fourier Transform (FFT) (in Sec. 3.2). Spectral tokens are linked to amplitude and phase branches to extract the task-specific information and enhance the original features (in Sec. 3.3). Additionally, since the style variation mainly affects the amplitude component, we further design an attention optimization method to improve the generalization ability of the learned tokens (in Sec. 3.4). Finally, two enhanced components are projected back to the spatial space by Inverse Fast Fourier Transform (IFFT) and fed into the subsequent layer.As discussed above, the low-frequency and high-frequency components in the frequency space provide a feasible solution to separate the content and style information. Inspired by this, we turn to the spectral decomposition [5, 35] via Fast Fourier Transform (FFT) to realize this objective. Specifically, given the intermediate image features 𝑋 𝑘 ∈ R 𝑑 ×𝐻 ×𝑊 of layer 𝑘, it is fed to a 2D Fast Fourier Transform, independently to each channel to obtain the corresponding projected representations 𝑋 𝑓 ∈ R 𝐻 ×𝑊 in frequency space. This process on 𝑋 𝑖 within each channel can be computed asEQUATIONwhere 𝐻 and 𝑊 are the height and the width of the image features.In addition, 𝑑 denotes the feature dimension.On the other hand, the inverse transformation from the frequency space to the spatial space is usually implemented by the Inverse Fast Fourier Transform (IFFT), which is mathematically computed asEQUATIONThe frequency representation 𝑋 𝑓 can be divided into two parts, namely the real part 𝑋 𝑟𝑒𝑎𝑙 𝑓 and the imagery part 𝑋 𝑖𝑚𝑔 𝑓 respectively, defined asEQUATIONDecomposing the image feature 𝑋 𝑖 into its amplitude 𝛼 and phase 𝜌 is known as spectral decomposition, given byEQUATIONThrough this series of mathematical transformations, the frequency representation 𝑋 𝑓 can be presented as𝑋 𝑓 = 𝛼𝑐𝑜𝑠 (𝜌) + 𝑖𝛼𝑠𝑖𝑛(𝜌).(5)Source Domain Define the function to decompose the frequency feature into its amplitude and phase components as 𝑑𝑒𝑐𝑜𝑚𝑝𝑜𝑠𝑒 (•), and the opposite process as 𝑐𝑜𝑚𝑝𝑜𝑠𝑒 (•), this process can be presented as 𝛼, 𝜌 = 𝑑𝑒𝑐𝑜𝑚𝑝𝑜𝑠𝑒 (𝑋 𝑘 ), 𝑋 𝑘 = 𝑐𝑜𝑚𝑝𝑜𝑠𝑒 (𝛼, 𝜌).Layer Layer 1(6)Leveraging learnable tokens with only a few number of parameters [19] has been turned out to be an effective path to fine-tune the VFMs. Following this simple yet effective paradigm, in our pipeline, learnable tokens are utilized to refine the spectral components 𝛼 𝑘 and 𝜌 𝑘 of features at each layer within the frozen VFM backbone. Since these tokens refine the original features in the frequency domain, we name them as spectral tokens in this paper. Specifically, for the features 𝑋 𝑘 generated by the 𝑘-th layer 𝑉 𝑘 , spectral components 𝛼 𝑘 , 𝜌 𝑘 is obtained by spectral decomposition. The enhanced spectral features α𝑘 , ρ𝑘 are produced by token-based adjustment 𝛽 (•) and composed to output feature as the input for the subsequent layer, given by To achieve this, each spectral token includes a bag of learnable features to capture the task-specific knowledge in frequency domain. Within the token-based adjustment process 𝛽 (•), these bag features are used to enhance the spectral-decomposed features based on their similarity. Through this enhancement, task-relevant representations in the original features are further highlighted, while task-irrelevant category information is partially filtered out.𝛼 𝑘 , 𝜌 𝑘 = 𝑑𝑒𝑐𝑜𝑚𝑝𝑜𝑠𝑒 (𝑋 𝑘 ), α𝑘 = 𝛼 𝑘 + 𝛽 (𝛼 𝑘 ), ρ𝑘 = 𝜌 𝑘 + 𝛽 (𝜌 𝑘 ), 𝑋 𝑘+1 = 𝑉 𝑘+1 (𝑐𝑜𝑚𝑝𝑜𝑠𝑒 ( α𝑘 , ρ𝑘 )).Specifically, inner-product is utilized to measure the similarity between token features and original features from frozen layers. Similarity map 𝑀 𝛼 𝑘 ∈ R 𝐻𝑊 ×𝑙 is built to capture the association between spectral-decomposed features (𝛼 𝑘 as example) and spectral token (amplitude token 𝑇 𝛼 𝑘 as example). Following the widely-used attention mechanism, a softmax function is applied to each line of 𝑀 𝛼 𝑘 to normalize the weights of token features. To summarize, the feature-token similarity can be mathematically computed asEQUATIONThrough the feature-token similarity map 𝑀 𝛼 𝑘 , we can select the relevant token features for each position of the spectral-decomposed features. These relevant token features include learned task-specific knowledge which is used to enhance the original features. Before added into the frozen spectral-decomposed features, the token features are further processed by a Multi-Layer Perceptron (MLP) layer to generate a more suitable representation for enhancement. This intermediate process 𝛽 (𝛼 𝑘 ) can be presented asEQUATIONFinally, another MLP layer is adapted to the enhanced features. This layer is designed to extract task-related information and filter out the irrelevant representations. The overall process of tokenbased adjustment 𝛽 (•) can be written as𝛽 (𝛼 𝑘 ) = 𝑀𝐿𝑃 (𝛼 𝑘 + 𝛽 (𝛼 𝑘 )), 𝛽 (𝜌 𝑘 ) = 𝑀𝐿𝑃 (𝜌 𝑘 + 𝛽 (𝜌 𝑘 )).(10)During inference, the parameters of the learnable tokens are fixed. As shown in Fig. 2 , the style variations can significantly reflect in the amplitude component. As a result, when dealing with images from unseen target domains, the weights of the feature-token similarity map in the amplitude branch may be severely affected by the style change. Since the task-related category information is stored in the tokens features, the perturbation of weights leads to the injection of incorrect category information during the enhancement process, thereby weakening the domain generalization ability. To address this problem, we further proposed an attention optimization method to adjust the similarity map in amplitude branch.Specifically, we perform a further normalization on the featuretoken similarity map,EQUATIONwhere 𝜇 ans 𝜎 denote the mean and standard deviation of 𝑀 𝛼 𝑘 . Let 𝐻,𝑊 denote the size of similarity map, 𝜇 ans 𝜎 are computed byEQUATION)Due to the softmax operation when acquiring 𝑀 𝛼 𝑘 , the original mean value of 𝑀 𝛼 𝑘 is 1 𝑙 and the max standard deviation is√︃ 1 𝑙 (1 -1 𝑙 ), where 𝑙 is the number of features in each token. After normalization, the standard deviation increases to 1, which means higher weights are assigned to the relevant token features and the weight distribution becomes more uneven. When amplitude feature representation affected by style variation, its similarity with fixed tokens trained on source dataset decreases, leading to more uniform weight distribution within the feature-token similarity map. By the proposed attention optimization, the weight distribution is refined and the relevant token features receive more attention, alleviating the impacts of style change for token-based fine-tuning. After that, the enhancement process in amplitude branch can be written asEQUATIONFinally, amplitude component is combined with phase component, projected back to the spatial space and fed into the next layer in VFM, given byEQUATIONFollowing prior work [51] , the model is trained 40000 iterations with a batch size of 4 and an initial learning rate of 1e-4 for DGSS tasks. The resolution of input images is 512 × 512. DINOv2 is chosen as the default VFM, and the segmentation head of Mask2Former [11] is utilized to produce pixel-level prediction. CityScapes [14] is constructed based on the driving-scenes in Germany cities, which includes 2,975 and 500 well-annotated samples for training and validation, respectively. The resolution of CityScapes is 2,048×1,024. BDD-100K [56] provides diverse scenes of driving videos under various weather conditions. It contains 7,000 and 1,000 fine-annotated samples for training and validation of semantic segmentation, respectively. The resolution of BDD-100K is of 1,280×720. SYNTHIA [44] provides a large-scale synthetic dataset, and provides 9,400 images with a resolution of 1,280×760. Mapillary [30] provides a large-scale semantic segmentation dataset based on street scenes with 25,000 samples.GTAV [43] is a synthetic semantic segmentation dataset rendered by the GTAV game engine. It provides 24,966 simulated urban-street samples with a resolution of 1,914×1,052. 
evaluation settings: 
 We illustrate our domain generalization settings as follows. Firstly, we use C, B, S, M and G to denote the above five datasets respectively. Following prior DGSS works [12, 33, 34, 36] , the segmentation model is trained on one dataset as the source domain, and is validated on the rest of the four datasets as the target domains. Three settings include: 1) G → {C, B, M, S}; 2) S → {C, B, M, G}; and 3) C to → {B, M, G, S}. We employ the mIoU (%) metric for the evaluation. All the reported performance is directly cited from prior works [12, 33, 34, 36] .Existing DGSS methods are included for comparison, namely, IBN [33] , IW [34] , Iternorm [20] , DRPC [57] , ISW [12] , GTR [37] , DIRL [52] , SHADE [60] , SAW [36] , WildNet [26] , AdvStyle [62] , SPC [21] , HGFormer [16] , CMFormer [1] , DIDEX [31] and Rein [51] . 3 compares the performance of the proposed SET and existing state-of-the-art DGSS methods under the C → {B, M, G, S} setting. The proposed SET also shows a clear performance improvement than the second-best Rein [51] . Specifically, the mIoU improvements on the B, M, G and S unseen target domains are 1.53%, 1.64%, 1.39% and 1.05%, respectively. In addition, the mIoU improvements on ResNet based and Mask2Former based DGSS methods are more than 15% and 8% in average. To better understand how the proposed SET improves the feature generalization when compared with Rein [51] , Fig. 4 visualizes the feature space of the Rein baseline (left) and the proposed SET (right). The proposed SET allows the samples from different unseen target domains to be more uniformly distributed. 4 studies the impact of each component in the proposed SET. The experiments are conducted under the G → {C, B, M, S} setting. When only the frozen VFM (DINOv2) is used for DGSS (in the first row), the mIoU is relatively low. The spectral decomposition operation (in the second row) contributes to 1.93%, 3.24%, 0.42% and 0.60% mIou gain on the C, B, M and S target domains by separately learning the amplitude and phase information. Furthermore, when spectral tokens are implemented (in the third row), the performance gain are 3.10%, 4.30%, 3.10% and 2.36% mIou on the C, B, M and S target domains, respectively. Finally, based on the spectral tokens, attention optimization contributes to an additional improvement of 1.28%, 1.05%, 1.60% and 1.09%. 5 further studies the attention optimization operation on each component. The outcomes show that, attention optimization in phase branch can potentially harm the model's representational capability. In contrast, the optimal choice is to only used for amplitude component, which shows the most predominate improvement on the generalization ability. MAE [18] , SAM [24] , EVA02 [17] and DINOV2 [32] under fulltraining, full-freezing and fine-tuning scheme, respectively. One decimal result is reported and compared following prior references. Besides, Rein [51] , as the fine-tuning baseline, is also involved for comparison. Table 6 reports the outcomes under the G → {C, B, M} setting. Our SET shows a significant performance improvement than Rein [51] and other methods with all listed VFMs.We further inspect the generalization ability of the proposed SET to a variety of adverse conditions. Adverse Conditions Dataset with Correspondences (ACDC) (ACDC) [45] is the largest semantic segmentation dataset under adverse conditions. Following prior works, fog, night, rain and snow are set as four different unseen domains. CityScapes is used as the source Unseen images ISW SAW Rein Ours WildNet SPC , SAW [36] , WildNet [26] , SPC [21] and Rein [51] .Unseen images ISW SAW Rein Ours WildNet SPC , SAW [36] , WildNet [26] , SPC [21] and Rein [51] .domain. Fig. 5 demonstrates some visual segmentation results on unseen target domains under the C → B, M, G, S setting. Fig. 6 illustrates the visual segmentation results under the C → ACDC setting. On both settings, the segmentation results show that the proposed SET shows better pixel-wise prediction than the compared DGSS methods, especially in terms of the completeness of objects. 
conclusion: 
 In this paper, we aim to fine-tune VFM for the down-stream task DGSS. While the VFMs have inherent generalization to out-ofdistribution, how to exploit the style-invariant property of a VFM remains to be the bottleneck. We propose a Spectral-Decomposed Token (SET) learning scheme. As the content and style information in a scene reflects more from the low-and high-frequency components in the frequency space, we transform the frozen VFM features into the phase and amplitude components respectively. Then, spectral tokens are adapted to enhance the learning of task-specific knowledge within each branch. As the cross-domain differences mainly affects the amplitude branch, an attention optimization method is further proposed to mitigate the impacts of style variation. Extensive experiments under multiple cross-domain settings show the state-of-the-art performance of the proposed SET and its flexibility on a variety of VFMs. 
: 
 Figure 1 : Segmentation performance on unseen target domain (in mIoU, %) v.s. trainable parameter number (in million, M). GTAV [43] is used as the source domain and CityScapes [14] is used as unseen target domain. The proposed SET shows state-of-the-art performance with little trainable parameters. 
 
 Appendix:  


Decision: This paper received three weak accept, one weak reject and one reject final ratings from the reviewers. The main concern of some reviewers is that this work lacks discussion and explanation about why the proposed approach works well for semantic segmentation. But the experimental results can sufficiently confirm the effectiveness of the proposed approach. AC agrees that this paper benefits from good writing and interesting idea. However, the authors are encouraged to make the necessary changes to the best of their ability.

Reviews: review: paper_summary: paper_summary: The paper proposed a novel method based on spectral-decomposed for domain generalisation. It can be integrated into existing VFMs and bring improvement by comparing with different methods on many datasets.
strength_weakness: strength_weakness: 1. Decomposed in frequency level is good for separating the structural and style components in the image features.
2. The proposed method has good generalisation ability among different datasets. The feature after processing is more uniform as seen in Figure 4, which removes the domain gap between different datasets.
limitations: limitations: 1.	
```
The phase component, which contains more low-frequency components on the scene content, is relatively stable despite the cross-style variation.
```
This claim is not always correct. Low-frequency components contain more structural information, which includes low-frequency and high-frequency information. Although low-frequency components represent mainly the large-scale structures in the image, high-frequency components also contain structures as well. In some cases, the phase may contain more high-frequency components than low-frequency components. The distribution of phase information is not directly related to the amount of the frequency component. In fact, since the style transfer doesn't change the structural information, the phase wouldn't have changed much in the first place.

2.
```
Figure 3:
```
The amplitude and phase are two interrelated components. But in the framework, they are processed independently, and they will be composed in the end, which raise my concern. 

3. Lack of ablation study. The contribution of each section is not clear.

4. 
```
Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
```
This sentence at the end of References does not make sense.
paper_summary: paper_summary: Effectively fine-tuning VFM for DGSS has recently become an open research topic for the vision community. In this paper, the authors present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Delving into further than the existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on the way learning style-invariant features from
these learnable tokens.  After the decomposition, style variation primarily impacts the token-based feature enhancement within the amplitude branch. To address this issue, the authors further develop an attention optimization method to bridge the
gap between style-affected representation and static tokens during inference. Extensive cross-domain experiments under a variety of
backbones and VFMs show state-of-the-art performance.
strength_weakness: strength_weakness: --- The organization of this paper is clear and easy to follow.

--- The proposed Spectral-dEcomposed Token (SET) is novel and effective. 

--- Extensive cross-domain experiments under a variety of backbones and VFMs show state-of-the-art performance.
limitations: limitations: --- In FSDR [A], the authors first converted the latent embedding into frequency space (64FCs) with Discrete Cosine Transform and found that only low-frequency and high-frequency components capture more domain variant features as compared with middle-frequency components. While in this manuscript, the authors used ONE global Amplitude. This needs to be explained in detail.

--- The article primarily addresses the factor of style. While it's important to acknowledge that style is a significant factor in domain drift, particularly in semantic segmentation tasks, it's also crucial to recognize a limitation in this approach: the predominant focus on style by many works, including this paper, limits a comprehensive understanding of the issue.

--- Lacks of some closely related work to be discussed[B-C].

[A] FSDR: Frequency Space Domain Randomization for Domain Generalization.
[B] Test-time Fourier Style Calibration for Domain Generalization.
[C] Adversarial Style Mining for One-Shot Unsupervised Domain Adaptation.
paper_summary: paper_summary: In this paper, the authors present a framework named SET for domain-generalized semantic segmentation. In this framework, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space and then refined by learned phase tokens and amplitude tokens, respectively. The SET gets SOTA in several datasets.
strength_weakness: strength_weakness: 1. Based on VFM, SET achieves remarkable performance in domain-generalized semantic segmentation with very few learnable parameters.
2. The article presents exhaustive and convincing experiments, complemented by comprehensive and well-executed visualizations.
limitations: limitations: The efficacy of "Spectral Decomposition" in domain-generalized semantic segmentation has been validated in [A]; your fine-tuned network architecture resembles to [B], which also employs VFM for the same task. Your technical innovation lies in the "attention optimization." Thus, I view your work's contribution as a successful exploration of Spectral Decomposition in VFM-based domain-generalized semantic segmentation, accompanied by adaptive improvements. So my main concern lies in :
1. In-depth Analysis of Attention Optimization: Explain why this technique works in your framework, perhaps with examples or visualizations of the attention matrix M.
2. Practicality of SET: As I know, FFT and IFFT can be computationally intensive, please analyze the computational complexity of SET or provide inference time costs to gauge its practical feasibility.

Additionally, there are minor inaccuracies in your manuscript:

3. Figure 3: The figure should indicate that it's the phase (not phase token) that is added to β(ρk).
4. Shouldn't the dimensions of Xf be d×H×W instead of H×W?

[A] FDA: Fourier Domain Adaptation for Semantic Segmentation

[B] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation
paper_summary: paper_summary: The paper proposes a novel Spectral-dEcomposed Token (SET) learning framework that uses spectral decomposition and learnable tokens to improve the domain generalization capabilities of Vision Foundation Models (VFMs) for semantic segmentation, achieving state-of-the-art performance across various unseen target domains.
strength_weakness: strength_weakness: The paper is well written, the overall structure is clear, and the experiments are comprehensive.
limitations: limitations: The core idea of this paper is to perform domain transformation on the side features of CNNs, followed by multiple feature fusion methods. This idea is not novel; I have seen it frequently in related tasks. I think the major flaw of this work is that the authors list many experiments to demonstrate that the method is good and better than its competitors, but they do not discuss and reveal why adopting this method can achieve such good results. I want the authors to explain this point very clearly in the rebuttal and include it in the final text. Since the idea is not novel, I hope the authors can delve deeper into the fundamental nature of the work.

Other flaws include the t-SNE visualization in Figure 4. The new cluster results seem much worse than the baseline.
paper_summary: paper_summary: For the goal of DGSS, this paper presents a frequency-based fine-tuning method for the large CV models. By adding learnable amplitude and phase tokens, the proposed method achieve better performance than previous DGSS methods.
strength_weakness: strength_weakness: + As illustrated in Fig.2, the motivation of this paper is clear.
+ The presentation of the proposed method is clear and good. 
+ The idea to fine-tune features from large models from the perspective of spectrum is novel for the reviewer.
+ The experimental results show the superiority of the proposed method.
limitations: limitations: - This paper may need proof-reading. The "reference stage" in 3-rd line of introduction may be "inference stage". Fig.1 and Fig2 are not mentioned in the introduction.
- Why the attention optimization is not used in the phase branch?