Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning

Authors: fei  zhou
peng  wang
lei  zhang
zhenghua  chen
wei  wei
chen  ding
guosheng  lin
yanning  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:32:53

Abstract: Meta-learning offers a promising avenue for few-shot learning (FSL), enabling models to glean a generalizable feature embedding through episodic training on synthetic FSL tasks in a source domain. Yet, in practical scenarios where the target task diverges from that in the source domain, meta-learning based method is susceptible to over-fitting. To overcome this, we introduce a novel framework, Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning, which is crafted to comprehensively exploit the cross-domain transferable image prior that each image can be decomposed into complementary low-frequency content details and high-frequency robust structural characteristics. Motivated by this insight, we propose to decompose each query image into its high-frequency and low-frequency components, and parallel incorporate them into the feature embedding network to enhance the final category prediction. More importantly, we introduce a feature reconstruction prior and a prediction consistency prior to separately encourage the consistency of the intermediate feature as well as the final category prediction between the original query image and its decomposed frequency components. This allows for collectively guiding the network's meta-learning process with the aim of learning generalizable image feature embeddings, while not introducing any extra computational cost in the inference phase. Our framework establishes new state-of-the-art results on multiple cross-domain few-shot learning benchmarks.

Main paper: 
 introduction: 
 Meta-learning Finn et al. [2017] , Lee et al. [2019] , Rusu et al. [2019] , Zhmoginov et al. [2022] , Baik et al. [2020] represents a potent paradigm within the domain of FSL Vinyals et al. [2016] , Snell et al. [2017] , Huang et al. [2022] , Zhang and Huang [2022] , Chen et al. [2021] . This paradigm harnesses a feature embedding network to capture task-agnostic meta-knowledge, facilitating generalization to novel tasks. To this end, meta-learning systematically samples a sequence of FSL episodes in the source domain to supervisedly enforcing learn an effective feature embedding network that assimilating cross-task transferable essentials and generalize well to novel target tasks. Due to its exceptional learning-to-learn capabilities, meta-learning has established itself as the de facto approach for the development of effective few-shot solvers Vinyals et al. [2016] , Snell et al. [2017] , Huang et al. [2022] , Zhang and Huang [2022] , Finn et al. [2017] , Lee et al. [2019] , Rusu et al. [2019] , Zhmoginov et al. [2022] , Baik et al. [2020] .However, in practical cross-domain scenarios where the target task exhibits a noticeable distribution discrepancy from that in the source domain, meta-learning based methods are susceptible to overfitting. This phenomenon can be attributed to two main reasons. Firstly, tasks randomly sampled in source domain often come from one or several fixed patterns, and thus the continual switching of episodes training may cause the model to over-fit on some task-specific priors. For instance, in tasks involving the discrimination between tigers and giraffes, meta-learning methods may compel the model to emphasize appearance outlines, while in tasks focused on fine-grained bird identification, models tend to prioritize local discriminative textures. Yet, these task-specific priors prove challenging to transfer across different tasks Lyu et al. [2021] , Zhou et al. [2023] . Secondly, the iterative episodic training in the source domain can result in the model over-fitting to semantic prior properties specific to that domain. For example, source domains comprised of natural scene images often exhibit obvious semantic priors, whereas specialized target domains like medical image analysis or remote sensing may lack clear semantic concepts. This overall domain bias also hampers the model's cross-domain generalization. For all these challenges, the underlying evil lies in the absence of cross-domain invariant priors to guide meta-learning in the source domain.To address this challenge, we introduce a novel framework, Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning. Inspired by classical image transform theories (Fourier Nussbaumer and Nussbaumer [1982] or wavelet Zhang and Zhang [2019] ), where each image can be decomposed into low-frequency content details and high-frequency structural characteristics, despite which domain it belongs to, we attempt to cast such a cross-domain invariant image property into appropriate frequency priors and utilize them to guide the meta-learning in source domain. Following this idea, we decompose each query image into a high-frequency and a low-frequency parts, and feed each into the feature embedding network for final category prediction, mirroring the process applied to the original query image. These allows for the independent feature learning in both spatial and frequency domains. In addition, the low-frequency and high-frequency branch will separately exploit the complementary image content and structures for feature enhancement, which are often concealed in the spatial domain of original query image. More importantly, we further develop two frequency priors, namely a feature reconstruction prior and a prediction consistency prior, which separately forces the original query image and its decomposed frequency components to produce the consistent intermediate feature representation as well as the final category prediction. In a specific, the feature reconstruction prior requires to reconstruct the feature of original image through fusing the features of both decomposed frequency parts using a deep projection network. The prediction consistency prior aims to minimize the separate Kullback-Leibler divergence between the prediction scores produced by the original query image and its each frequency component. By doing these, meta-learning in the source domain can be appropriately regularized and produce the exceptional cross-domain generalizable feature embeddings. Moreover, such frequency priors only perform in the meta-learning phase without introducing any extra computational cost in inference. Through a series of rigorous experiments, our framework establishes itself as a front-runner, achieving state-of-the-art results across multiple cross domain FSL benchmarks. Additionally, our method exhibits significant efficiency advantages.The primary contributions of this study can be summarized as follows:• We present a novel insightful meta-learning framework that exploits cross-domain invariant frequency priors to alleviate the over-fitting problems of classic meta-learning in crossdomain FSL tasks. • We propose two frequency prior, namely a prediction consistency prior and a feature reconstruction prior, to collectively guide the meta-learning procedure. • We achieve state-of-the-art results on multiple cross-domain FSL benchmarks. 
methodology: 
 Problem formulation. Cross-Domain Few-Shot Learning (CD-FSL) aims to transfer the knowledge acquired by a model in the source domain D s to perform few-shot tasks in the target domain D t . It is noteworthy that the categories in D t differ from those in the source domain. Each task T involves the random sampling of N categories, with K samples and M samples randomly selected Figure 1 : Framework of the proposed method. In this work, we present an insightful meta-learning framework that exploits cross-domain invariant frequency priors to alleviate the over-fitting problems of classic meta-learning in cross-domain FSL tasks. Our method consists of an Image Decomposition Module (IDM) and a Prior Regularization Meta-Network (PRM-Net). Among them, IDM aim at explicitly decomposing every image in few-shot task into low-and high-frequency components. PRM-Net develops a prediction consistency prior and a feature reconstruction prior to jointly regularize the feature embedding network during meta-learning, aiming to learn generalizable image feature embeddings. Once the model is trained, only the main branch is retained for meta-testing on target domains.from each category to constitute the support set T S and the query set T Q , respectively. The support set T S is employed for constructing a task-specific classifier, while the query set T Q is used to assess the classification accuracy for that specific task. To emulate the meta-testing process, methods based on meta-learning typically sample a series of few-shot tasks from the source domain for training.Overview. In this study, we introduce a sophisticated meta-learning framework that leverages cross-domain invariant frequency priors to mitigate the over-fitting problems of classic meta-learning in cross-domain FSL tasks. As illustrated in Fig. 1 , our method comprises two key components: the Image Decomposition Module (IDM) and the Prior Regularization Meta-Network (PRM-Net). The IDM is designed to explicitly decompose each image within a few-shot task into its low-and high-frequency components using Fast Fourier Transform (FFT) Nussbaumer and Nussbaumer [1982] . PRM-Net is a key component responsible for introducing a prediction consistency prior and a feature reconstruction prior. PRM-Net is organized into three branches: the main branch, the low-frequency content branch, and the high-frequency structure branch. In each branch, all images undergo feature extraction through the embedding network. Subsequently, a task-specific classifier is constructed based on the support set to predict the query set. Two frequency priors, namely the prediction consistency prior and the feature reconstruction prior, are proposed to collectively guide the network's meta-learning process with the aim of learning generalizable image feature embeddings. The IDM and PRM-Net work collaboratively to provide a robust meta-learning framework, aiming to enhance cross-domain generalization by explicitly considering image decomposition and introducing effective regularization during the meta-learning process. The subsequent sections will provide a detailed description of each module.In the realm of signal processing, classical image transform theory Nussbaumer and Nussbaumer [1982] , Zhang and Zhang [2019] posits that every image can be decomposed into low-frequency content and high-frequency structure, irrespective of its domain. Therefore, within the Image Decom-position Module, we adhere to this theory and employ Fast Fourier Transform (FFT) Nussbaumer and Nussbaumer [1982] to explicitly decompose each image from the few-shot task T into a lowfrequency content image and a high-frequency structure image. Specifically, for an image x in T , the initial step involves decomposing it into the frequency domain:EQUATIONwhere F represents FFT, f x low and f x high represent the low-frequency and high-frequency components of x in the frequency domain respectively. Following this decomposition, we transform these components back into the image space using the inverse FFT:EQUATIONwhere F -1 represents inverse transform of FFT, x low and x high represent the decomposed lowfrequency content image and high-frequency structure image respectively. Similarly, we apply the same decomposition process to each image in T to obtain the corresponding low-frequency content task T low and high-frequency structure task T high .The proposed Prior Regularization Meta-Network is designed to leverage cross-domain invariant frequency priors, addressing meta-learning over-fitting in the source domain. To achieve this objective, we introduce a three-branch meta-learning network, dedicated to processing the raw few-shot task T , the low-frequency task T low , and the high-frequency task T high , respectively. Significantly, we propose a prediction consistency prior and a feature reconstruction prior to jointly regularize the feature embedding network during meta-learning. This approach empowers the learning process, facilitating the acquisition of a cross-domain generalizable feature embedding. Upon completing the meta-training on the source domain, we discard the high-frequency and low-frequency branches, retaining only the main branch for cross-domain validation.The main branch. As depicted in Fig. 1 , the main branch includes a feature embedding network and a task-specific classifier. For a few-shot task T , the main branch first feeds each image into the feature embedding network to obtain features, and then utilizes the support set T S to build a prototype classifier Snell et al. [2017] :EQUATIONwhere c n represents the prototype of the n-th category, x n,k represents the k-th support sample of the n-th category, f θ represents the feature embedding network. Finally, we utilize the prototype classifier to make prediction on the query set:EQUATIONwhere x j ∈ T Q , P xj represents the prediction scores of x j , d (•) represents the Euclidean distance.For the query image x j , the category corresponding to the highest score in P xj is used as the predicted label ŷxj . Subsequently, we calculate the cross-entropy loss between the predicted label ŷxj and the ground truth y xj as:EQUATIONwhere H (•) denotes the cross-entropy loss function.The high-or low-frequency branch. As illustrated in Fig. 1 , both the high-frequency branch and the low-frequency branch maintain consistency with the architecture of the main branch. In practice, we input the decomposed high-frequency task T high and low-frequency task T low into these two branches, respectively, to obtain the corresponding features and prediction scores for the query set. Mathematically, the prediction scores for the query image x j in the high-frequency branch and the low-frequency branch are denoted as P low xj and P high xj , respectively.Frequency prior regularization. In this work, we posit that the over-fitting problem is the core crux that limits the cross-domain generalization of meta-learning model. To this end, we resort to cross-domain invariant priors to regularize meta-learning in the source domain. Motivated by this perspective, we propose a prediction consistency prior and a feature reconstruction prior to jointly regularize the feature embedding network during meta-learning using high-low frequency information obtained from image decomposition. Specifically, the prediction consistency prior aims to minimize the separate Kullback-Leibler divergence between the prediction scores produced by the original query image and its each frequency component. Formally, for a query image x j , we align its high-frequency prediction distribution P high xj and low-frequency prediction distribution P low xj with the original distribution P xj respectively:EQUATIONwhere D KL (•) is the Kullback-Leibler divergence loss function. The rationale behind this approach is twofold. Firstly, through explicit decomposition-alignment, we compel the model to attend to both low-frequency content and high-frequency structure. Despite their distinct nature, these two types of features synergistically contribute and complement each other in the challenge of cross-domain generalization. Secondly, establishing prediction consistency between high-low frequency and the original one is domain-invariant. This consistency aids the model in generalizing effectively across different domains.The feature reconstruction prior aims at reconstructing the original features utilizing low-frequency and high-frequency information in the latent space, which promotes the model to learn comprehensive representations. Specifically, we first project embedding features into the low-dimensional latent space, and then utilize the information retained by high-and low-frequency to reconstruct the original features:EQUATIONEQUATIONwhere f φ and f ϕ are the feature embedding network of the low-frequency branch and the highfrequency branch respectively, g η is a projector composed of one layer full connected neural network (512×256), ẑxi is the reconstructed feature. Then, the reconstruction loss is calculated as:EQUATIONwhere M SE (•) represents the mean square error loss function.Meta-training. Based on the description provided above, for a few-shot task T , the total loss can be formulated as:EQUATION)Following this, we compute the gradient based on the total loss L to update both the main branch θ and the projector η. While one straightforward approach is to share parameters between the high-low frequency branches and the main branch, this might lead the feature embedding network to primarily focus on common features among the three, potentially causing distinctive features in the high-frequency or low-frequency branches to be overlooked. To address this concern and extract more distinctive features, we opt for an explicit design where three separate feature embedding networks are employed without parameter sharing. However, updating the high-frequency and low-frequency branches through gradient back-propagation can introduce additional computational overhead. As a solution, we update the high-frequency branch ϕ and low-frequency branch φ as the Exponential Moving Average (EMA) of the main branch θ during meta-training:EQUATIONwhere m 1 and m 2 are momentum hyper-parameters. We describe the entire meta-training process in detail in Algorithm 1. Cross-domain evaluation. Once the model is trained, only the main branch f θ is retained for meta-testing on target domains. The proposed method is designed to enable the model to learn cross-domain transferable knowledge during the training phase, achieving effective generalization without relying on task-level feature extractor fine-tuning during meta-testing. Specifically, for each meta-testing task in the target domain, the main branch is utilized to extract features. Subsequently, the support set T S is employed to construct a task-specific classifier for inference on the query set T Q . It's important to note that the proposed method does not require image decomposition during the meta-testing phase, thereby avoiding additional computational overhead. 
experimental analysis: 
 In this section, we begin by providing a detailed description of the experimental configuration, encompassing pre-training, meta-training, and meta-testing. Following that, we analyze the advantages of the proposed method in comparison with state-of-the-art methods. Lastly, we delve into a comprehensive ablation study to further investigate the effectiveness of our approach. Due to space limitations, we put more experiments and analyses in the appendix.Source domain and target domains. We focus on the most challenging scenario of single-source domain Cross-Domain Few-Shot Learning (CD-FSL). Following the established setup Guo et al. [2020] , Li et al. [2022] , Zhou et al. [2023] , we employ the base classes of the mini-ImageNet Vinyals et al. [2016] as the source domain dataset. Our model is evaluated across multiple target domains, encompassing natural image domains (CUB, Cars, Places, Plantae), remote sensing domain (Eu-roSAT), agricultural domain (CropDisease), and medical domains (ChestX, ISIC). These datasets are widely recognized in the field of cross-domain few-shot learning. Additional details about each dataset can be found in Guo et al. [2020] , Tseng et al. [2019] .Pre-training and meta-training. In the context of CD-FSL, pre-training is a common technique Li et al. [2022] , Zhou et al. [2023] , Hu and Ma [2022] , Wang and Deng [2021] , aiming to provide feature initialization for meta-training. Specifically, it involves supervised classification on the source domain through batch training. Following Li et al. [2022] , Zhou et al. [2023] , Hu and Ma [2022] , we utilize ResNet-10 as the feature embedding network and a one-layer fully connected neural network as the classifier. The total number of pre-training epochs is set to 400. After pre-training, only the feature embedding network is retained as the feature extractor for meta-training. During the meta-training phase, we employ Adam as the optimizer and conduct meta-training for 50 epochs with a learning rate of 0.001. In each epoch, we randomly sample 100 meta-tasks, where each meta-task consists of 5-way 5-shot 15-query. Data augmentation techniques such as "Resize," "ImageJitter," and "RandomHorizontalFlip" are applied during meta-training. We set hyper-parameters m 1 =0.997 and m 2 =0.999. All experiments were performed on a 4090 GPU. Our experimental platform is a 4090 GPU. Further details and verification of hyper-parameters can be found in the supplementary material.Meta-testing. Upon completion of meta-training, we directly employ the learned model for metatesting across all target domains. Specifically, for each target domain, we randomly sample 600 meta-tasks for testing. We consider two challenging meta-tasks: a 5-way 1-shot 15-query task and a 5-way 5-shot 15-query task. In each meta-task, we learn a Logistic Regression classifier using the support set and then conduct inference on the query set.Methods. In the realm of single-source domain CD-FSL, the state-of-the-art methods primarily include LDP-net Zhou et al. [2023] , StyleAdv Fu et al. [2023] , GNN+wave- SAN Fu et al. [2022] , RDC Li et al. [2022] , AFA Hu and Ma [2022] , ATA Wang and Deng [2021] , LRP Sun et al. [2021] , FWT Tseng et al. [2019] , and Fine-tuning Guo et al. [2020] . These methods can be categorized into three types: direct inference, using query samples to assist inference (marked with †), and fine-tuning-based inference (marked with * ).Among these methods, direct inference (e.g., LDP-net, MatchingNet, AFA, ATA) is the most straightforward manifestation of model generalization. It handles each test task without fine-tuning the feature embedding network, meeting practical application requirements. Using query samples to assist inference (e.g., RDC † , LDP-net † , AFA † , ATA † ) is also a common experimental setting. It is noteworthy that GNN+wave-SAN † and StyleAdv † also leverage query samples in an unsupervised manner. The main reason is that both GNN+wave-SAN and StyleAdv use Graph Neural Network (GNN) Garcia and Bruna [2018] as a classifier. GNN treats each sample in the few-shot task as a node of the graph, and the associations between different samples as edges for reasoning, akin to label propagationLiu et al. [2019] . This approach implicitly leverages unsupervised query samples when the number of query samples in the few-shot task exceeds one. The original GNN paper Garcia and Bruna [2018] tested a single query image for each few-shot task, avoiding this issue. For a fair comparison, we also implement a variant that uses query samples to assist inference. Specifically, we train a classifier based on the support set to generate pseudo-labels for the query set, then filter samples from the query set based on these pseudo-labels to expand the support set, and finally retrain the classifier based on the expanded support set for the ultimate prediction on the query set.Results. Tables 1 and 2 present the experimental results under 5-way 1-shot and 5-way 5-shot settings, respectively. For an easy comparison, the average performance across eight target domains is calculated as the metric. Our method achieves 46.85% (1-shot) and 63.77% (5-shot) under the direct inference setting. Compared to the second-highest method LDP-net Zhou et al. [2023] , the proposed method improved by 0.51% and 1.17% on the 1-shot and 5-shot tasks, respectively. In comparison to other methods like AFA Hu and Ma [2022] , ATA Wang and Deng [2021] , LRP Sun et al. [2021] , and FWT Tseng et al. [2019] , the proposed method demonstrates greater performance advantages. Moreover, the proposed method achieves the best results on five target domains, showcasing its robust generalization ability across diverse domains. When the proposed method further utilizes query samples to assist inference, the performance is further improved. Under the same comparison, the proposed method ( † ) improved by 1.10% (1-shot) and 1.40% (5-shot) compared to the secondbest method LDP-net † Zhou et al. [2023] . In contrast to methods based on fine-tuning (e.g., Finetuning * Guo et al.[2020], RDC * † Li et al. [2022] ), the proposed method still achieves certain performance advantages without requiring additional fine-tuning. In summary, the proposed method has demonstrated the best cross-domain few-shot learning performance, indicating its ability to learn generalizable features in the source domain. Additionally, the method's independence from task-level embedding network fine-tuning makes it suitable for potential industrial applications.Comparison with baselines. We design two baselines: the "Pre-training baseline" and the "Metabaseline." For the "Pre-training baseline", we directly use the pre-trained model for meta-testing.The proposed method performs meta-training on the basis of pre-training. When all components are removed from the proposed method, it is equivalent to the "Meta-baseline" Chen et al. [2021] .We take the "Meta-baseline" as the baseline of the proposed method. For a fair comparison, during the meta-testing stage, these two baselines also use the same classifier as the proposed method. The comparison results are shown in Table3. Overall, compared with these two baselines, the proposed method has achieved greater performance advantages. Specifically, compared with the "Pre-training baseline", the performance of the proposed method is improved by 2.79% (1-shot) and 3.28% (5-shot) on average. Compared with the "Meta-baseline", the performance of the proposed method is improved by 2.26% (1-shot) and 3.10% (5-shot) on average. These results show that the proposed method can improve the baselines and provide a novel meta-learning framework for CD-FSL. Effectiveness of the proposed frequency prior. In this work, we propose a prediction consistency prior and a feature reconstruction prior to jointly regularize the embedding network during metalearning. Among them, the prediction consistency prior encourages to align the predictions produced by the original query image and its each frequency component. The feature reconstruction prior aims at reconstructing the original features utilizing low-frequency and high-frequency information in the latent space, which promotes the model to learn comprehensive representations. We conduct ablation studies to illustrate the contribution of these two components. The results are shown in Table 3 . As can be seen, compared to the "Meta-baseline" (meaning not using any components), the proposed method achieves average gains of 1.61% (1-shot) and 2.45% (5-shot). The above experimental results show the proposed prediction consistency prior is effective. We can draw similar conclusions for feature reconstruction prior. In addition, the proposed method improves by nearly 0.65% under both 1-shot and 5-shot tasks when the feature reconstruction module is added. In particular, for the CUB dataset, the proposed method can achieve nearly 1% improvement on the 5-shot task. This indicates that the proposed feature reconstruction prior is beneficial to the entire method.Feature highlight. we adhere to established practices Zhou et al. [2023] , utilizing the model trained on the source domain to extract features from target domain images. Subsequently, these features serve as attention scores to activate the original images. The results are presented in Fig. 2 . Overall, our proposed method exhibits the capability to capture more nuanced representations compared to the baseline, a critical aspect for effective cross-domain generalization. As an illustrative example, consider image (d) in Fig. 2 . The baseline tends to concentrate solely on the neck of the bird, neglecting the broader characteristics of its entire shape. In contrast, our method not only hones in on local texture details, such as the head, wings, and claws, but also encapsulates the entirety of the contour shape. This underscores the capacity of our method to learn comprehensive features, avoiding undue emphasis on local textures alone.Domain gap. The t-SNE Van der Maaten and Hinton [2008] visual results are shown in Fig. 3 (a-b ).The blue cluster represents the source domain distribution, while the other four colors denote distinct target domain distributions. Notably, the baseline exhibits a substantial gap between target domains and the source domain. In contrast, our method effectively mitigates this domain gap. In addition, we conduct a quantitative assessment of the distribution distance between different target domains and the source domain. Specifically, we compute the first-order statistics based on the sampled samples in each domain, treating it as the statistical characteristic of that domain. Subsequently, we measure the Euclidean distance between the first-order statistics of different target domains and the source domain.The resulting quantitative metrics, comparing the proposed method and the baseline, are visualized in Fig. 3 (c ). Evidently, the proposed method exhibits a smaller distribution distance between the source domain and the target domains, with particularly notable improvements in the medical domain (ISIC) and the remote sensing domain (EuroSAT). This underscores the capability of our method to learn robust representations in the source domain, effectively mitigating domain shifts.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The data and code will be released. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusions: 
 In this work, we propose an insightful meta-learning framework inspired by the cross-domain invariant frequency priors. Furthermore, we present a prediction consistency prior and a feature reconstruction prior to jointly regularize meta-learning on source domain, enabling learning crossdomain transferable features. This simple yet effective work achieves state-of-the-art experimental results as well as excellent inference efficiency.Limitations. The limitation of the proposed method lies on its robustness in some extremely challenging cross-domain tasks. For example, on the Chest dataset, the proposed method fails to outperforms all competitors. This indicates that the fixed image decomposition (e.g., Fast Fourier Transform or Wavelet Transform) strategy may be not the optimal solution for all unknown cases in terms of exploit frequency priors. In the future, we will attempt to exploit the learnable image decomposition strategy. In addition, the proposed method requires to decompose the query image before being fed into the network. In contrast to methods relying on texture or shape priors, the proposed method starts from the principles of image transformation theory, focusing on cross-domain invariant frequency priors to enhance the robustness of model. Additionally, our approach compels the model to simultaneously attend to both high-frequency structure and low-frequency content. This strategy enables our method to strike a balance between the contributions of texture and shape to model generalization. Furthermore, unlike other methods grounded in frequency priors, our work's primary innovation lies in the seamless integration of low-frequency and high-frequency features within a meta-learning framework. This integration provides an elegant solution to the persistent challenges of cross-domain few-shot learning, allowing for the independent learning of features from these distinct segments, each excelling in capturing unique aspects of visual information. [2022] , Cheng et al. [2023] have explored the integration of various priors, including texture, shape, and frequency, among others. To highlight the advantages of our proposed method, we conducted comparative experiments with different variants. To ensure a fair comparison, we replaced different priors in our framework while maintaining other settings constant. Specifically, we adopted the approach of Jain et al. [2022] to model the texture prior. For the shape prior, similar to Stojanov et al. [2021] , Padmanabhan et al. [2023] , we employed the Canny operator Ding and Goshtasby [2001] to extract the shape prior. For high-frequency or low-frequency priors, we retained only the high-frequency branch or the low-frequency branch in our framework. The results are presented in Table 4 . Overall, our method outperforms the variants with different priors. The reasons behind this superiority are as follows. Firstly, the texture prior compels the model to excessively focus on local discriminative regions, leading to texture bias and impairing generalization. Secondly, the shape prior directs the model's attention to global shapes, which may cause the model to exhibit shape bias and overlook semantic information. Thirdly, compared to texture or shape priors, the frequency prior provides more original information, expanding the model's search space and enabling a higher generalization upper bound. Fourthly, our method couples high-frequency and low-frequency information within a unified framework, presenting an elegant solution to the persistent challenges of cross-domain few-shot learning. This approach allows for the simultaneous consideration of both types of information, harnessing their complementary aspects for improved generalization.Why can our work alleviate overfitting? In this study, we subscribe the limited generalization capacity of exiting methods in cross-domain few-shot learning (CD-FSL) to their over-fitting onto source domain. Since during the meta-training procedure, only tasks randomly sampled from source domain are utilized for model update, when the target domain shows obvious distribution discrepancy from the source domain, these existing methods are prone to over-fitting, in other words, fail to generalize well in the new target domain. Why can our work alleviate overfitting? To solve overfitting, a direct solution is to introduce appropriate prior (e.g., regularization) during training on source domain. Inspired by this, we attempt to comprehensively exploit the cross-domain transferable image frequency prior that each image can be decomposed into complementary low-frequency content details and high-frequency robust structural characteristics. Following this idea, we first utilizes Fast Fourier Transform to explicitly decouple the high-frequency and low-frequency components of the image. Then, we feed each component and the query image into a three-branch feature embedding network for category prediction. More importantly, we further establish a feature reconstruction prior and a prediction consistency prior to collectively guiding the network's meta-learning process. The feature reconstruction prior requires to reconstruct the feature of original image through fusing the features of both decomposed frequency parts using a deep projection network, while the prediction consistency prior aims to minimize the separate Kullback-Leibler divergence between the prediction scores produced by the original query image and its each frequency component. By doing these, both priors encourage to exploit a deep feature space where no matter full-frequency band (original image), high-frequency component or low-frequency component can lead to the unique and correct classification prediction, i.e., the idea semantic feature space which is transferable cross-domain. Therefore, the proposed method is able to mitigate the over-fitting problems in CD-FSL. Our stateof-the-art performance on eight benchmark datasets as well as the ablation study also support this conclusion.Whether this work is equivalent to data augmentation or self-supervised learning? In this work, our core idea is to utilize cross-domain invariant frequency priors to alleviate the over-fitting problem of classical meta-learning in cross-domain few-shot learning tasks. To this end, we propose two key components: the Image Decomposition Module (IDM) and the Prior Regularization Meta-Network (PRM-Net). Among them, IDM aims to use Fast Fourier Transform (FFT) to explicitly decompose each image from few-shot task into its low-and high-frequency components Nussbaumer and Nussbaumer [1982] . PRM-Net is a key component responsible for introducing a prediction consistency prior and a feature reconstruction prior. It is important to note that our method is fundamentally different from simple data augmentation methods and self-supervised learning methods. The insight behind our method is divide and conquer, that is, explicit decomposition and implicit coupling. First of all, the IDM is to explicitly obtain the frequency priors of the image rather than to simply perform data augmentation. IDM provides frequency prior for the subsequent PRM-Net, and its role is "divide". In addition, PRM-Net designs regular terms with the help of frequency priors, rather than simple self-supervised learning, and its role is "conquer". More importantly, this divide-and-conquer strategy enables IDM and PRM-Net to collaborate to provide a powerful meta-learning framework, aiming to enhance cross-domain generalization by explicitly considering image decomposition and introducing effective regularization during the meta-learning process. We design some experiments to compare the proposed method with other data augmentation methods and self-supervised learning methods. For the data augmentation method, we use random rotation to augment the image, and design the angular self-supervised loss as the regularization term Gidaris et al. [2018] . For self-supervised learning methods, we choose the most representative SimCLR Chen et al. [2020] and BYOL Grill et al. [2020] . All compared methods keep the same backbone network and training data as the proposed method. The experimental results are shown in Table 5 . Overall, the proposed method can achieve better results compared to using simple data augmentation and self-supervised learning methods. Whether it is applicable to any image decomposition method? To answer this question, we conducted comparative experiments using FFT-based Nussbaumer and Nussbaumer [1982] decomposition and Wavelet-based Zhang and Zhang [2019] decomposition. The results are presented in Table 6 . In comparison to the baseline, our method consistently demonstrates significant performance advantages regardless of the decomposition method employed. This indicates the scalability and effectiveness of our method across different decomposition techniques. Additionally, the model trained with Wavelet decomposition exhibits further performance improvement on the Places dataset. Does performance benefit from additional parameters? First of all, we clarify that our method does not introduce additional learnable parameters and the parameter amount is same as our baseline. This is because only the parameters in the main branch (e.g., query image branch) are learnable, and the parameters of the high-frequency branch and low-frequency branch are updated through the exponential moving average of the main branch parameters. Moreover, after training, we only keep the main branch for prediction in the test phase, since the prediction consistency prior have forced these three branches to produce the same prediction results when the training procedure converged. Therefore, our method does not introduce any additional inference costs compared with our baseline. We also supplemented experiments to answer whether the performance gain comes from additional parameters. Specifically, we triple the learnable parameters of the baseline method and then compare it with our method. As shown in Table 7 , our method still outperforms the baseline when the parameters of the baseline are increased three times.Efficiency. As mentioned previously, the proposed method focuses on obtaining a generalizable model through meta-learning without relying on fine-tuning the embedding network on the target domain. This makes the proposed method very practical and efficient in handling target domain tasks.To verify the efficiency of the proposed method, we chose a classic fine-tuning based method Guo et al. [2020] for comparison. For the fine-tuning based method, we follow its original settings for experiments. We report the average classification accuracy across different target domains as well as the required inference time for each target domain task. Our experimental platform is a single 3090 GPU. The results are shown in Table 8 . It can be seen that the proposed method only requires about 0.06 seconds to effectively handle a few-shot task in the target domain. Compared with the fine-tuning based method, the proposed method is nearly 100 times faster on the 5-shot task. At the same time, the proposed method also has great advantages in performance. The above experiments show that the proposed method is efficient and has the potential for practical applications. Hyper-parameters validation. The hyper-parameters of the proposed method encompass the momentum parameters m 1 and m 2 , which are integral to the Exponential Moving Average (EMA) update strategy. In our approach, we employ the loss for calculating gradients and subsequently updating the network parameters of the main branch through back-propagation. Conversely, for the parameters in the low-frequency and high-frequency branches, we update them using EMA. To validate the advantages of EMA, we conducted a parameter sharing experiment for comparison, denoted as "none" in Table 9 . The results indicate a substantial performance decrease when using parameter sharing as opposed to EMA, underscoring the necessity of employing EMA. Additionally, we explored different values for the momentum parameters and observed that the model's performance is not highly sensitive to the specific values of m 1 and m 2 . We have set m 1 to 0.997 and m 2 to 0.999 based on these findings.Additional visualization. More feature highlight results are provided in Fig. 4 and Fig. 5 . In summary, the baseline model tends to focus narrowly on specific local regions of the object. Conversely, our method exhibits a more extensive focus on the object, indicating an ability to capture a more comprehensive semantic understanding and, consequently, achieve superior generalization. Justification: The authors performed discuss the limitations of the work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This work does not contain any negative social impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The authors have cited the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: In this paper, the authors propose to improve cross-domain few-shot learning by decomposing images into high- and low-frequency components. Extensive experimental results are provided to show the effectiveness of the proposed method. After the rebuttal period, this paper receives consistent positive ratings. Therefore, it is a clear acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel framework called Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning, which aims to improve meta-learning's generalization by decomposing images into high- and low-frequency components. This method leverages these components to guide the feature embedding network, enhancing category prediction consistency and feature reconstruction. The framework achieves state-of-the-art results on various cross-domain few-shot learning benchmarks, demonstrating its effectiveness and efficiency.
strength_weakness: strength_weakness: 1. The method relies heavily on the strong prior assumption that using low and high-frequency components can effectively mimic the distribution shift encountered during testing. However, this assumption may not hold true when the test and training datasets are entirely unrelated, as observed in the EuroSAT dataset where the performance of the method is not as impressive. The authors need to clarify and justify this assumption.

2. The paper does not provide an evaluation of the method's performance on within-domain few-shot learning tasks. Understanding how the proposed framework performs in a scenario where the training and testing data come from the same domain could provide a more comprehensive view of its robustness and generalizability.

3. The computational complexity of the proposed framework during the training phase is not thoroughly discussed. While the paper claims no additional inference cost, the added steps of image decomposition and feature reconstruction during training could introduce significant computational overhead. Detailed analysis and discussion on the computational requirements and efficiency would be beneficial.

4. The impact of the choice of image decomposition method (e.g., Fast Fourier Transform) on the overall performance of the framework is not thoroughly explored. Alternative decomposition techniques may yield better results or be more computationally efficient. A comparison of different decomposition methods and their influence on the performance and efficiency of the framework would provide valuable insights and strengthen the paper's contributions.
questions: questions: See weakness
limitations: limitations: yes
paper_summary: paper_summary: The paper introduces an innovative framework that leverages the concept of frequency priors for cross-domain few-shot learning. The novel idea of decomposing images into high and low-frequency components and integrating these into the meta-learning process is a creative advancement in the field.
Built upon established image transformation theories such as the Fourier Transform, the proposed method is grounded in a solid theoretical foundation, which enhances its credibility and applicability across various domains. The empirical validation through extensive experiments on multiple cross-domain benchmarks further substantiates the effectiveness of the framework, showcasing its superiority over state-of-the-art methods. And the experiment, along with the supplementary materials, has been very comprehensive, addressing most of my questions.
strength_weakness: strength_weakness: (1) While the paper highlights the efficiency advantages of the proposed method, it falls short of offering a detailed analysis of computational complexity. A critical evaluation extending beyond inference times to include the total floating-point operations (FLOPs) during both training and inference phases is essential. Understanding the complete computational costs provides insights into the method's practicality, scalability, and suitability for deployment across varying computational environments.
(2) The paper could be improved by including more extensive ablation studies on the choice of backbone network and loss functions. The current presentation lacks a thorough exploration of why specific architectural decisions and loss formulations were made, which is crucial for understanding the contribution of these choices to the overall performance.
(2) Although the paper demonstrates strong empirical results, there is a need for a more profound theoretical analysis underpinning the effectiveness of frequency priors in cross-domain generalization. A deeper theoretical understanding would bolster the paper's claims and offer insights into the broader applicability of the proposed method.
questions: questions: (1) Is there a hyperparameter used during the FFT process to determine the high and low-frequency information? If so, how was the threshold for decoupling these frequencies determined, and how might it influence the experimental results?
(2) How does the model perform over a longer period of training, and are there any signs of degradation or improvement in performance? Can the loss optimization curve and loss landscape be provided?
(3) The content task and structure task are measured in a decoupled manner. Are these tasks entirely independent, and theoretically, could the low-frequency information be used as the query set to predict within the raw image support set, and vice versa for high-frequency information? And so on.
(4) Besides the EMA (Exponential Moving Average) update method, what other update mechanisms have been considered or could be applicable to the model's training process?
(5) There seems to be a limited ablation study regarding the choice of loss functions. Why was MSE chosen for the Feature Reconstruction Loss, and could cosine similarity loss be a viable alternative? Similarly, why was KL divergence chosen for the Prediction Loss, and was this decision based on empirical results or theoretical analysis?
(6)The experiments were conducted using ResNet-10. Could the authors provide insights into whether the proposed method could be applied to larger models such as ResNet-101? Additionally, could the method be beneficial when applied to models like CLIP from OpenAI or the DINO model, and are there any explorations in this direction?
(7) Some of the experimental content in the supplementary materials could be incorporated into the main text, such as the sections on 'low' and 'high' in Table 4. This is just a suggestion.
limitations: limitations: The authors have transparently addressed the limitations of their method, particularly its performance on the Chest dataset, and have responsibly considered the broader societal impacts, confirming no negative social effects. They also acknowledge the increased training time associated with their approach. Proactively, they suggest future improvements and responsibly guide how their work can contribute to the field's progression, aligning well with the checklist for addressing limitations and societal impacts.
paper_summary: paper_summary: This paper introduces a novel framework for cross-domain few-shot learning. The input images are first decomposed into low-frequency content and high-frequency structure using FFT. Then, the PRM-Net includes three branches, low-frequency, high-frequency, and main branch. The PRM-Net includes two priors to regularize the feature embedding network. The approach shows state-of-the-art results on multiple benchmarks.
strength_weakness: strength_weakness: 1. The method avoids additional inference costs, however, the decomposition and additional branches (high-low frequency branches) could introduce significant computational overhead. Moreover, the improvement by incorporating the reconstruction prior seems marginal according to Table 3.
2. Lack of proper citation for the datasets mentioned in section 3.1, e.g., CropDisease, CUB, ISIC.
3. The reliance on fixed decomposition strategies like FFT might not be optimal for some scenarios, e.g., the foreground objects have similar colors to the background, or the background is complex. The whole model cannot be trained end-to-end due to FFT.
questions: questions: 1. How do you see the value of research in the (cross-domain) few-shot learning while the foundation models can perform well in zero-shot tasks?
2. Is it possible to combine this approach with fine-tuning strategies to further enhance performance in certain domains?
limitations: limitations: n/a