Visual Fourier Prompt Tuning

Authors: runjia  zeng
cheng  han
qifan  wang
chunshu  wu
tong  geng
lifu  huang
ying nian wu
dongfang  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-28 17:01:48

Abstract: With the scale of vision Transformer-based models continuing to grow, finetuning these large-scale pretrained models for new tasks has become increasingly parameter-intensive. Visual prompt tuning is introduced as a parameter-efficient finetuning (PEFT) method to this trend. Despite its successes, a notable research challenge persists within almost all PEFT approaches: significant performance degradation is observed when there is a substantial disparity between the datasets applied in pretraining and finetuning phases. To address this challenge, we draw inspiration from human visual cognition, and propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach innovatively incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Apart from its inherent simplicity and intuitiveness, VFPT exhibits superior performance across all datasets, offering a general solution to dataset challenges, irrespective of data disparities. Empirical results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks, with low parameter usage (e.g., 0.57% of model parameters on VTAB-1k) and notable performance enhancements (e.g., 73.20% of mean accuracy on VTAB-1k). Our code is avaliable at https://github.com/runtsang/VFPT.

Main paper: 
 visual parameter-efficient finetuning: 
 With the significant growth in the scale of vision models, especially following the emergence of Vision Transformers [21, 22, 23, 24, 25] , the development of PEFT methods under "pretrain-thenfinetune" paradigm becomes increasingly critical. Current methods under this paradigm can be generally categorized into partial tuning [26, 27, 28] , extra module (i.e., including reparameterization approaches such as Low-Rank Adaptation (LoRA) [29] ) [30, 31, 32, 33, 34, 10, 35, 36] , and prompt tuning [4, 37, 38, 39, 40, 41] . Partial tuning and extra module face several limitations that hinder their application. ① Unsatisfactory performance: they generally cannot reach competitive performance with regard to full finetuning [4, 26, 27, 28, 33, 10] ; ② Model-oriented design: most research requires to insert specific architecture/block design [31, 30, 32] during tuning, rendering them non-universal solutions when considering different backbones. In contrast, prompt tuning [2] , originally proposed for language-domain [42, 43, 44, 45] , provides a general and straightforward solution in vision with powerful performance gains. It signals a new paradigm in PEFT in the field of computer vision.Generally, prompt tuning introduces a sets of learnable parameters to the input sequence of backbone models, updating only these parameters during the finetuning. Despite its apparent simplicity, the paradigm of visual prompt tuning has demonstrated notable performance enhancements. Current developments on visual prompt tuning primarily concentrate on engineering optimizations, such as reducing parameter usage [5] and expanding applicability across diverse tasks [39, 46, 47, 48] . These approaches often involve introducing additional constraints and functionalities to the foundational design, which deviate from the principles of simplicity and elegance to the original concept of visual prompt tuning. Our approach, in sharp contrast, endeavors to explore visual prompt tuning from the perspective of human visual intelligence, while diligently maintaining the simplicity of prompt tuning. It is also essential to emphasize that visual prompt tuning diverges markedly from visual instruction tuning [49] (i.e., aiming at improving the model's instruction following abilities).FFT is a powerful mathematical algorithm used to compute the Discrete Fourier Transform (DFT) and its inverse [50, 51] . It is pivotal in information processing, allowing the detailed analysis of various signals (e.g., image [52, 53, 54] , radar [55, 56, 57] ) for frequency determinations. In vision, FFT's ability to transform complex data in spatial domain into frequency domain makes it an invaluable tool for abstracting critical features from noisy or high-dimensional datasets [58, 59] . This abstraction is particularly beneficial as the identification of salient features are shown to have better generalization ability across domains [60, 61, 62, 63] , directly influences the performance [64, 65, 66, 67] of image analysis and processing tasks. Current research on FFT in vision predominantly explores areas such as conventional image processing [52, 68, 69, 70] , image pre-processing for deep neural networks (DNNs) [71, 72] and DNN architectural design [20, 66, 65, 73, 74, 75, 76] .Despite its profound utility and effectiveness, the integration of FFT within the paradigm of visual prompt tuning remains largely underexplored. Recent work [77] adapts the pretrained multi-modal network to the tasks under modality-incomplete segmentation scenarios via FFT prompt tuning. This approach demonstrates the potential of FFT operations to handle missing modalities (i.e., substantial disparity) effectively. However, it primarily focuses on task-specific optimization and design. The extensive applicability and generality of FFT, especially in cross-dataset analysis, have yet to be recognized or exploited. Another work [36] incorporates Fourier transform into the LoRA-based approach. While the expressive Fourier basis facilitates the recovery of weight changes, it does not fully integrate frequency domain information during finetuning, which remains orthogonal to our approach. In this paper, we aim to broaden the scope of exploration and contribute to advancing the field of Fourier-based research in vision. By studying the integration of FFT with visual prompt tuning, we fully explore how to improve both the efficacy (see §3) and the adaptability of learning models to diverse and challenging datasets (see §4). Furthermore, we present novel evidence indicating that VFPT establishes strong correlations within the Transformer's input space, aligning with the performance enhancements (see §4.4). Overall, the generality of VFPT suggests a novel understanding of the Fourier-based method in current machine learning applications. 
methodology: 
 In this section, we introduce VFPT, a novel visual prompt tuning approach for effective and general large-scale transformer-based model finetuning. We first define the problem and notations of visual prompt tuning and FFT in §3.1. The integration of Fourier-based visual prompt tuning is presented in §3.2. The overall framework is shown in Fig. 1(c ), where we compare our model with original VPT.Visual Prompt Tuning. Given a pretrained Transformer model T with N layers, the objective of prompt tuning in vision is to finetune a model T into a new task with only a few set of d-dimensional embedding vectors, i.e., prompts, in the input space after patch Emb layer. These learnable prompts are defined as P = {P 1 , P 2 , . . . , P N }, where P i represents the learnable visual prompts in the i th encoder layer. Formally, the encoder layers with prompts are defined as:EQUATIONwhere the embeddings of the input image patches E are initialized with frozen Emb projection, and Z i is the contextual embeddings computed by the i th encoder layer. The colors and indicate trainable and frozen parameters, respectively. Here, trainable prompts only accounts for a small proportion of the total parameters (e.g., 1.14% on VTAB-1k [78] in VPT [4] ). Fast Fourier Transform. The FFT is a powerful algorithm for computing the Discrete Fourier Transform (DFT), which transforms a finite sequence of equally-spaced function samples into a same-length discrete-time Fourier transform sequence. Specifically, given a sequence {x n } where n is a member of the interval n ∈ [0, N -1], the DFT is defined as:EQUATIONFor a finite sequence of equally-spaced samples {x n }, the DFT generates a same-length sequence of equally-spaced samples {X k }. This transform is denoted as F. The initial DFT is in complexity O(n 2 ). For acceleration, we use Cooley-Tukey FFT algorithm [79] following common practice [80] (i.e., complexity O(n log n)). FFT serves as a powerful tool for domain transition. Consequently, we explore the integration of the FFT operation within PEFT methods, particularly in prompt tuning.Visual prompt tuning is particularly useful under the pretrain-then-finetune paradigm. However, it suffers a significant performance reduction when substantial disparities exist between pretrain and finetune datasets. The reason is that during finetuning on new data, the image distribution may deviate markedly from the examples used in pretraining the backbone model [9] . Existing prompt tuning [4, 5] , focusing predominantly on spatial information, can only harness the shared information embedded within the pretrained backbone, limiting their capacity to adapt effectively to novel tasks. Thus, it is crucial to strengthen the ability to capture distinguishing feature from finetuning data.To this end, we introduce VFPT, an intuitive yet powerful method with advanced performance and generality. Compared to VPT (see Fig. 1(a )), our model (see Fig. 1(c )) transforms partial prompts from spatial domain to frequency domain via 2D FFT (see §3.1) to consider both the spatial and frequency domain information. Formally, for each learnable visual prompts in the i th encoder layer P i ∈ P = {P 1 , P 2 , . . . , P N }, we haveP i = {p i 1 , p i 2 , . . . , p i M }.We select m partial prompts as visual Fourier prompts at each layer, where 0 ≤ m ≤ M . Further, α = m/M represents the fraction of Fourier participation, where zero indicates all prompts are original visual prompts, and one implies all prompts are given after FFT. We apply a 2D FFT on α visual prompt embedding input with respect to both sequence (i.e., F seq ) and hidden dimensions (i.e., F h ). Note that the operations F seq (F h (x)) and F h (F seq (x)) are mathematically equivalent due to the commutative property of the two one-dimensional FFTs [80] . Here, indicates Fourier operations.EQUATIONTo maintain the pretrained structure's consistency, we only alter the prompt embeddings, and thus retain only the real component (i.e., ℜ) from the output. This design does not require any adjustments to accommodate complex numbers in the self-attention module, ensuring that the remaining elements of the model remain unchanged. Consequently, the overall integrated prompts P i in the i th encoder layer are formed by the concatenation between the visual Fourier prompts and visual prompts as:EQUATIONOur elegant design of VFPT enjoys a few appealing characteristics:• Simplicity: VFPT only requires several lines of code based on the implementation of the visual prompt tuning. Its intuitive integration of information between spatial and frequency domains brings nearly free performance efficacy. The low complexity of FFT (i.e., O(n log n)) leads to an overall marginal reduction during the training schedule.(i.e., 2.8% on VTAB-1k [78] ). In sharp contrast, current endeavors in visual prompt tuning mainly emphasize augmenting architectural complexity for superior performance [5, 81, 42] , undermining the inherent simplicity of prompt tuning and introducing significant training overhead (e.g., [81] learns 2D prompt token map for densely image relationship construction, [5] incorporates additional self-attention K-V prompts). • Generality: The frequency and spatial analysis of imagery inputs can be mutually complementary, leading to a more comprehensive feature understanding from distinct perspectives (e.g., the frequency domain allows for the distraction and decomposition of luminance and noise to a considerable degree [82] , while the spatial domain excels in capturing intricate object details Baselines. For consistency, we follow [4, 5] and compare VFPT with other widely applied parameterefficient fine-tuning methods. Results of two vision transformer architectures, Vision transformer [23] (ViT) and Swin transformer [24] (Swin), on image classification are discussed in §4.2. We also apply VFPT on two self-supervised objectives: MAE [90] and MoCo v3 [26] .Training. Following [4, 5] , we conduct grid search to find the best tuning hyperparameters, learning rate (i.e., [50, 25, 10, 5, 2.5 , 1, 0.5, 0.25, 0.1, 0.05]), and weight decay (i.e., [0.01, 0.001, 0.0001, 0.0]) on val set. Notably, VFPT does not require specific-designed large learning rate in [4] . The learning rate is scheduled by a cosine decay policy and trained for 100 epochs.Reproducibility. VFPT is implemented in Pytorch [91] . Experiments are conducted on NVIDIA A100-40GB GPUs. To guarantee reproducibility, our full implementation will be publicly released. 
main results: 
 In this section, we demonstrate the effectiveness of VFPT from two key perspectives: ♠ Superior Performance: Our model demonstrates significant performance improvements across diverse datasets, including challenging tasks with large disparities in data, thus showcasing its generalizability. ♡ Fourier Contribution: We observe that Fourier components play a critical role in VFPT, where tasks with larger data disparities tend to favor higher percentages of Fourier components. Definition of disparity. Following [9] , we use the Fréchet Inception Distance (FID) [99, 100] to measure the disparity between the datasets used in pretraining (i.e., ImageNet) and funetuning (i.e., downstream tasks). Average FID scores of each group are reported in Fig. 2 , where the Natural group has low disparities due to its close relationship to ImageNet21K [84] and the Specialized and Structured groups (i.e., orientation prediction task) are considered distinct from image classification. The dataset description of VTAB-1k is covered in §4.1 (FGVC is excluded due to lack of categorization).♠ Superior Performance. In order to have a comprehensive understanding on generality, we examine VFPT on ViT-Base/16 [23] , Swin-Base [24] , and two self-supervised objectives, following common practice [4, 5] . We also report the individual per-task results for Table 1 , 2 and 3 in Appendix.VFPT on ViT. We report the average accuracy score on VTAB-1k and FGVC benchmarks across four diverse task groups for three runs in Table 1 , where fifteen protocols under pretrain-then-finetune paradigm are considered. Specifically, Full [92] updates both backbone and classification head; Linear [92] , Parital-1 [93] (top layer), and MLP-3 [94] (3 MLP layers) are partial tuning approaches; Sidetune [31] , Bias [30] , Adapter [32] , LoRA [35] , AdaptFormer [95] and ARC att [96] are extra module methods which add new trainable parameters to backbone for adaptation; VPT-S [4] , VPT-D [4] , EXPRES [97] and E 2 VPT [5] are concurrent visual prompt tuning approaches. Consequently, we have several key observations. First, VFPT is able to outperform the full fine-tuning method in 22 out of 24 tasks. For example, our model achieves 0.13% improvement on FGVC and 5.21% improvements on VTAB-1k Structured, respectively. The empirical results show the effectiveness of VFPT. Second, VFPT tunes only 0.66% of the overall parameters in the backbone, establishing it as a competitive method within the PEFT approaches. Third, while VPT struggles to capture the image information when having significant dataset disparity, VFPT achieves notable performance improvements by integrating both spatial and frequency information (see §3.2) without additional architectural modifications. (i.e., 60.19% vs. 54.98% on VTAB-1k Structured).VFPT on Hierarchical Transformer. We further extend VFPT to a hierarchical transformer -Swin-Base [24] for architectural generalization. The MSA layer of Swin is employed in local shifted windows, and patch embeddings are merged at deeper layers. For consistency, we follow the same settings from ViT to apply and prepend Fourier prompts ahead of the visual prompts. The results on Fourier Percentage (%)Figure 2 : Image classification accuracy of various Fourier percentages of VTAB-1k [78] for ViT-Base/16 [23] . For better illustration, we randomly select 3 datasets in each group of VTAB-1k.The "Average FID Score of Each Group" is reported in <•>. Our conclusion aligns with 16 of 19 cases. The cross framed by the square indicates the best percentage for each downstream task. Those datasets with only three Fourier percentage reports are due to the prompt length limits.the ImageNet-21k supervised pretrained Swin-Base [24] are reported in Table 2 . It can be seen that VFPT consistently outperforms all the other parameter-efficient methods on three VTAB-1k groups.VFPT on Different Pretraining Objectives. In Table 3 , we report the experimental results on two self-supervised objectives: MAE [90] and MoCo v3 [26] . ♡ Fourier Contribution. We conducted experiments to understand the impact of Fourier components by varying the percentages of Fourier prompts in VFPT. As shown in Fig. 2 , we observed distinct preferences across the VTAB-1k benchmark, which comprises three groups with varying data disparities (see §4.1). Specifically, the Natural group, which has a data distribution similar to the pretrained task (low disparity), shows peak performance when half of the visual prompts are transformed into Fourier prompts, as indicated by the accuracy curves in Fig. 2(a) . This suggests that transfer learning is less challenging in this group. Conversely, for the Specialized and Structured groups, which have data distributions significantly different from the pretrained task (high disparity), the accuracy curves in Fig. 2(b-c ) demonstrate that higher classification performance is achieved with an increased percentage of Fourier components. These observations are consistent with our expectations, demonstrating the effectiveness of Fourier prompts in VFPT, especially for tasks with large data disparities. In other words, our approach can be viewed as a generalization of VPT, where the Fourier components learn effective representations from the frequency domain that complement the knowledge from the spatial domain.In this section, we investigate why VFPT achieves better performance and generalization across various tasks from an optimization perspective. Previous works [102] demonstrate that landscape geometry significantly impacts model generalization, so we visualize the loss landscape to understand the enhanced generality of VFPT. Specifically, in Fig. 3 (a), we randomly select two parameter directions for the study, as randomness in directions does not significantly affect the results [102] . There are two key observations supporting the enhanced generality of VFPT. i) Flatness: VFPT provides a larger connected region around the local minimum [103] (e.g., ⋆ in the yellow square, where the larger blue area in VFPT offers more optimization choices) and a smoother edge of the loss landscape for mitigating chaotic landscapes (e.g., • in the green square, where the bumpy contour in VPT is sensitive to loss variations, resulting in worse generality). This indicates that VFPT achieves a flatter minimizer, which consistently correlates with lower test error [102] . ii) Convexity: As eigenvalues of the Hessian directly assess the convexity of a loss function [102] , we compute both the maximum and minimum eigenvalues of the Hessian and map their ratios [102] . As shown in Fig. 3 (b), a higher prevalence of near-zero negative eigenvalues (in deep blue) in VFPT suggests the presence of more convex regions (25.0% vs. 20.0%) for model optimization. This finding indicates that the incorporation of the Fourier transform in visual prompt tuning effectively mitigates the sharpness of the loss landscape. To the best of our knowledge, research on the understanding of prompt tuning remains rare [9, 5] . Consequently, our research seeks to both quantitatively and qualitatively examine the impact of Fourier components on the enhancement of visual prompt tuning. For fairness, instead of using enhanced visualization methods [105, 106, 104] that may alter the original expression of the learnable prompts, we visualise and examine the raw average attention head on the last layer of VPT and VFPT.Significant attention distribution in learnable prompts. Observations from both VPT and VFPT in Fig. 4 (a) reveal a common phenomenon: there exists a pronounced accumulation of attention scores at learnable prompt locations (i.e., narrow color area on the left side of 2D attention map), indicating that these prompts have a substantial impact on the frozen embeddings during the finetuning stage.Global attention scores pattern in Fourier prompts. We further observe a notably higher concentration in global attention scores when integrating visual Fourier prompts. Specifically, the global attention scores indicate that VFPT also establishes robust correlations within the Transformer's input space [4] (see Fig. 4 (a)). In contrast, VPT lacks this correlation, suggesting that it does not adequately consider or integrate extensive information from the frozen backbone. Moreover, we find a positive relationship between strong associations and performance gains quantitatively (see §4. In summary, our findings provide significant insights into the interpretability of prompt tuning, revealing that for both VPT and VFPT, a considerable portion of attention is directed towards the learnable prompts. Further, VFPT exhibit enhanced global feature learning capabilities compared to VPT by interfacing effectively with frozen embeddings, thereby enabling precise capture of distinctive features across diverse downstream tasks. This observation corroborates our findings in §4.2. We ablate VFPT's key components on VTAB-1k [78] Natural and Specialized.More studies are provided in §S2.5.Transform Type. We ablate on other transform method instead to certify the impact of Fourier transform in Table 4 , where the Fixed Linear Layer (i.e., FLL) and the Learnable Linear Layer (i.e., LLL) are considered. Compared with FFT, a fixed non-parameter Fourier domain transform in sequence and hidden dimension, the FLL operation considers only a fixed spatial domain transform in hidden dimension; the LLL further unfixes the transformation to enable gradient updates. As seen, both FLL and LLL show inferior performance to FFT. We further consider the impact of current Fourier domain adaption approach [71] , which maps a source image to a target "style" without altering semantic content. However, no significant improvement can be observed. Fourier Prompt Dimension. A fundamental distinction between VFPT and other methods is the incorporation of FFT into visual prompts. In our standard implementation, we utilize 2D FFTs across both sequence length and hidden dimensions. Here, we explore the impact of each dimension's transformation individually. As shown in Table 5 (a), the separate Fourier transformations along each dimension appear to have similar contributions (i.e., 80.88% vs. 80.74% in Natural). However, the combined application of transformations across both dimensions (i.e., 2D FFTs) demonstrates a synergistic effect, yielding significant improvement in performance. Fourier Prompt Location. In Table 5 (b), three prompt locations are considered for VFPT, which are "Prepend" (i.e., P), "Append" (i.e., A), and "Random" (i.e., R). Specifically, P and A prepend visual Fourier prompts before or after visual prompts, and R randomly selects the position for visual Fourier prompts in each layer. As seen, both P and A show competitive results, validating the robustness of VFPT w.r.t. prompt locations. In alignment with the findings in [5, 4] , we choose P as our baseline method in all experiments since it reaches superior results (i.e., 81.35% vs 81.02% in Natural). Fourier Prompt Depth. Table 5 (c) presents the performance of VFPT based on the specific layer at which visual Fourier prompts are employed. The results suggest that employment on separate layers also yields a accuracy improvement compared with VPT. Further application of visual Fourier prompts across all layers fosters the best overall performance. 
conclusion: 
 We present Visual Fourier Prompt Tuning (VFPT), a simple yet powerful parameter-efficient visual prompt tuning approach that draws insights from human visual cognition. It has merits in: i) integrating spatial and frequency domain information through an intuitive yet effective design; ii) demonstrating generality across datasets with varying disparities while ensuring powerful performance; and iii) thoroughly investigating the associations between learnable prompts and frozen embeddings to elucidate this generality. As a whole, we conclude that the outcomes elucidated in this paper impart essential understandings and necessitate further exploration within this realm. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: §S10 discusses the social impact of our research.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We show related asset license and consent to our work in §S8.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not release new assets. In §S8, we include asset license and consent.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 1 Introduction "Fourier's theorem is not only one of the most beautiful results of modern analysis, but it may be said to furnish an indispensable instrument in the treatment of nearly every recondite question in modern physics." -Lord William Thomson Kelvin [1] Prompt tuning [2, 3] is initially introduced for parameter-efficient adaptation of large foundation models in natural language processing (NLP). As vision models continue to scale for enhanced performance, visual prompt tuning [4] has been applied to various vision domains (e.g., image classification [5] , segmentation [6, 7] , detection [8] ), demonstrating superior performance and lower parameter usage compared to other parameter-efficient fine-tuning (PEFT) methods. However, a common challenge within the research community remains unaddressed: significant performance degradation occurs when there is a substantial disparity between the data used in pretraining and finetuning [9, 10] . This issue hinders the broader application of visual prompt tuning. Consequently, a natural question arises: ① Can prompt tuning generalize across datasets with varying disparities?As researchers commonly draw insights from human to replicate the principles in intelligent machines [11, 12, 13, 14] , we consider to answer this question from the human visual cognition's perspective. While humans comprehend the world through past experiences/knowledge, it is essential to generalize and adapt this understanding to new tasks efficiently and effectively. The robust and rapid adaptability of human visual cognition thus arises from various domain analysis, capturing the new patterns from different channels and perspectives [15, 16, 17] .Interestingly, we find that the paradigm of visual prompt tuning is conceptually analogous to human visual cognition. While the frozen large-scale vision model functions as accumulated knowledge, the fast adaptation mechanism resembles visual prompt tuning, requiring the incorporation of diverse domains of information (e.g., time, frequency) to achieve comprehensive understandings [18, 19, 20] . The Fast Fourier Transform (FFT) [18, 19, 20] , renowned for its ability to convert signals from their original domain (e.g., time or spatial) to the frequency domain and vice versa, serves as an ideal tool for contributing informative insights in the frequency domain. By leveraging the capabilities of FFT, visual prompts can naturally integrate both spatial and frequency domain information during finetuning, thereby enabling the frozen vision model to achieve consistent and robust performance across datasets with varying disparities. Consequently, our research question evolves into: ② How can FFT be integrated into visual prompt tuning to emulate the human visual mechanism?To this end, we employ a simple yet effective strategy that utilizes the Fourier operations to facilitate visual prompt tuning (see Fig. 1(c )). By integrating frequency domain information into learnable prompt embeddings, our approach elegantly assimilates data from both spatial and frequency domains, simulating the human visual cognition. We name our approach Visual Fourier Prompt Tuning (VFPT), which exhibits several compelling advantages: ❶ Simplicity. The intuitive application of FFT in prompt tuning emulates the rapid processing capabilities of the human visual system, making VFPT both elegant and straightforward to implement (see §2.1). ❷ Generality. By incorporating frequency domain information, the search space for latent embeddings of prompts is naturally expanded, resulting in advanced enhancement in performance across different datasets and tasks with varying data disparities (see §4.2). The generality of our model is further illustrated through our analysis of the optimization process, which enables smoother navigation towards local minima, increasing flatness around them and exhibiting apparent convexity. ❸ Interpretability. To intuitively demonstrate the advantages of Fourier components, we visually illustrate that the introduction of Fourier transform in visual prompt tuning results in a markedly higher concentration of attention scores within the Transformer's input space, which correlates positively with enhancements in performance (see §4.4 ). This observation, in turn, explains the effectiveness of our approach.Comprehensive experiments are conducted to evaluate the performance of VFPT. In §2, we conduct a literature review and discuss relevant works. Our approach is presented in §3, where we describe how we simple yet effectively integrate FFT into visual prompt tuning. In §4.2, we present compelling experimental results on various benchmarks, backbones, and different pretraining objectives, achieving superior performance without complex engineering design. Specifically, our approach achieves an average improvement of 7.63% in accuracy on VTAB-1k compared to full finetuning, and 3.77% compared to VPT [4] . In §4. 4 , we demonstrate that the FFT prompts significantly enhance the activation of the frozen vision model. Additionally, we study the optimization process of prompt tuning approaches, indicating that VFPT provides a more favorable optimization process. Finally, we demonstrate the strong algorithmic generalization of our approach to the language domain, and show additional visual explanations in the Appendix. We trust that this work provides valuable insights.2 Related Work 
 
 Appendix: This appendix contains additional experimental results and discussions of our NeurIPS 2024 submission: Visual Fourier Prompt Tuning, organized as follows:• §S1 provides per-task results on VTAB-1k and FGVC image classification benchmarks with confidence analysis, where the overall results have been provided in the main paper. • §S2 provides per-task results on ablation study, where the overall results have been provided in the main paper. Further study of sensitivity of Fourier prompt percentages and prompt lengths is included in §S2.5. • §S3 provides per-task results on Fourier percentage, where partial results have been provided in the main paper. • §S4 presents more details and results of visualization of attention maps.• §S4 presents more details and results of visualization of loss landscapes.• §S6 discusses our potential extension to language tasks. • §S7 further analyze the complexity of our approach.• §S8 shows related asset license and consent to our work.• §S9 claims reproducibility of our approach. • §S10 discusses the social impact of our research.• §S11 adds more discussions, and points out potential directions of our future work.To provide comprehensive results from the paper, we report the average per-task test accuracy (i.e., 3 runs, 24 tasks) on VTAB-1k [78] Natural, Specialized and Structured, respectively (see Table S1 , S2 and S3). We also report per-task FGVC [4] results (5 tasks) in Table S4 . VPT-SHALLOW [4] is also included for completeness (i.e., VPT-SHALLOW only introduces 1-st layer visual prompts). In conclusion, VFPT shows consistently better performance in various downstream tasks. Table S1 : VTAB-1k [78] Natural per-task results for ViT-Base/16 [23] pretrained on supervised ImageNet-21k. Consistent to our paper, "Number of Wins" in [•] compared to full fine-tuning [92] . "Tuned/Total" is the percentage of tuned parameters in each task, along with the average results of those percentages in each group. The highest accuracy among all approaches except FULL are shown in bold. † denotes method using soft filtered prompts to reduce the parameter usage in learnable visual prompts, requiring specialized devices to facilitate acceleration. All results are averaged in three runs with different initialization seeds. Same for Table S2 -S21. We also report standard deviation error bars for our main results (Table S1 , S2, S3 and S4) by calculating each task respectively and averaging across them. Other tables show similar trends on standard deviation error bars.ViT-Base/16 [23] VTAB-1k [78] Table S8 : VTAB-1k [78] Natural per-task results for ViT-Base/16 [23] pretrained on MAE [90] .Since VPT [4] have considerably lower performance, we do not list the per-task results for simplicity. We instead compare our method to full fine-tuning, and the highest accuracy is shown in bold. We post the "Number of Wins" in [•] to full fine-tuning (FULL) [92] . Same for Table S9 In this section, we present more details and results of visualization of attention maps to support our findings in §4.4. All samples selected from VTAB-1k [78] have the same prompt length (i.e., 10 prompts) with one class token and 196 input patches.In Fig. S2 (a), we can first observe a significant attention distribution in learnable prompts and then a notably higher concentration in global attention scores when integrating visual Fourier prompts, showing consistency with our paper.In Fig. S2 (b), we present more visualization inspection results for VPT and VFPT using Grad-CAM [104] . Overall, we present additional visual evidence to support the notion that the integration of visual Fourier prompts encourage clear foreground-background separation. While ViT-Base/16 [23] is structurally similar to BERT [107] , we follow [5, 108] and naturally test the efficiency of the VFPT on natural language understanding (NLU) tasks. Specifically, we include BERT-Large [107] for evaluation, and compare full fine-tuning (FULL) [2] , Prompt Tuning [2] , P-Tuning v2 [108] and E 2 VPT [5] on SuperGlue [107] dataset: a collection of text classification tasks to test the general language understanding ability. The tasks include natural language inference (RTE and CB), coreference resolution (WSC), sentence completion (COPA), word sense disambiguation (WiC), and question answering (MultiRC (Fla), ReCoRD (F1) and BoolQ). In Table S25 , we show that VFPT outperforms FULL and Prompt Tuning and show competitive results to P-Tuning v2 [108] .Considering VFPT is designed for visual-related tasks, and text understanding tasks might not need fruitful frequency domain information, these results are impressive and suggest future work for a general solution across modalities under the pretrain-then-finetune paradigm. Table S25 : Per-task results for SuperGLUE development set [109] with a pretrained BERT-Large [107] . See §S6. We have provided a detailed comparison of our computational results in this section. More specifically, we experimented with different Fourier percentage settings (i.e.., the alpha rate) on the CIFAR-100 benchmark and reported their maximum memory consumption, average training batch time, and average inference batch time. All settings were tested with the same batch size and prompt length. The experiments were conducted on NVIDIA A100-40GB GPUs.As illustrated in Table S26 , no significant increase in maximum memory consumption at the MB level is observed across different Fourier percentage settings. However, we do observe a slight increase in average batch time during both training and inference, on the order of 10 -3 and 10 -4 , respectively. This suggests that a lower Fourier percentage incurs a higher computational burden. This effect is likely attributable to suboptimal parallel acceleration and the implementation inefficiencies associated with prompts that have partial Fourier transformation. We will investigate this further in future research.The majority of VPT [4] is licensed under CC-BY-NC 4.0. Portions of [4] are available under separate licenses: google-research/task_adaptation and huggingface/transformers are licensed under Apache-2.0; Swin-Transformer [24] and ViT-pytorch [23] are licensed under MIT; and MoCo-v3 [26] and MAE [90] are licensed under CC BY 4.0.All the datasets included in our study are publicly available (VTAB-1K, FGVC), and all the models are publicly available. We would like to state that the contents in the dataset do NOT represent our views or opinions.VFPT is implemented in Pytorch [91] . Experiments are conducted on NVIDIA A100-40GB GPUs. To guarantee reproducibility, our full implementation shall be publicly released upon paper acceptance.For training schedule, the superior low-complexity of FFT (i.e., O(n log n)) allows for efficient training of visual Fourier prompts with only a slight decrease in training speed (i.e., 2.8% on VTAB-1k [78] compared to VPT).This study presents VFPT, demonstrating significant and generalizable performance enhancements over state-of-the-art baselines across two benchmarks. The incorporation of the FFT contributes these advantages without necessitating architecture-specific designs or incurring substantial computational overhead under pretrain-then-finetune paradigm for large-scale models (see §3). Our approach enjoys advanced model accuracy, and is valuable in real-world computational-sensitive applications, e.g., training machine learning models on edge devices. Moreover, VFPT advances significantly towards achieving generality across datasets, demonstrating substantial performance improvements even when faced with large dataset disparities (see §4). This progress is crucial for the continuous development of PEFT across a wider spectrum of applications.For potential limitations, drawing inspirations from human visual cognition, our method incorporates spatial and frequency information, which brings an additional hyper-parameter -Fourier percentage (i.e., α in §3.2). However, in practical applications, we observe in §4.2 that dataset disparity (i.e., low disparity tasks prefer small α value, and vice versa) serves as a guideline for selecting an appropriate Fourier percentage. Nonetheless, we argue that the implementation of an automatic Fourier percentage search can further augment efficiency.In §2, we review PEFT methods and the application of the fast Fourier transform in vision. Notably, a recent study [36] in NLP incorporates Fourier transform as a viable PEFT approach, which warrants discussion. Specifically, it learns a set of spectral coefficients of Fourier basis using a LoRA-based approach and then applies the inverse discrete Fourier transform to the spectral matrix, yielding its spatial-domain counterpart as the updated weight change. Although the Fourier basis's orthogonal and expressive advantages reduce the need for extensive parameter fine-tuning, the inverse transform applied to the spectral matrix discards frequency information, ultimately considering only traditional spatial domain features. The parameter-efficient use of the Fourier transform in this study is orthogonal to our method, where both spatial and frequency domain information are integrated (see §3) for enhanced generality (see §4.2) and interpretability (see §4.4).Despite VFPT systemic effectiveness and simplicity, it also comes with new challenges and unveils some intriguing questions. For example, the balance between spatial and frequency information is presently dictated by task-specific, manually set percentages (see §4.2). Introducing a small network within the VFPT framework to autonomously search for optimal combinations might enhance training efficiency and facilitate additional performance improvements. Another essential future direction deserving of further investigation is the integration of visual information from both the spatial and frequency domains. In §4.5, we demonstrate through ablation studies that integration at the preprocessing stage may not yield satisfactory performance. Consequently, we outline several alternative integration approaches in Table 4 , demonstrating that VFPT holds the most advantageous position under the prompt tuning paradigm. Nonetheless, the applicability of this integration to other PEFT methods requires further investigation.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes a novel parameter-efficient finetuning method for vision models that relies on the fast Fourier transform. This allows for strong performance while saving in number of adapted parameters.

This paper has received A,A,WA ratings and the rebuttal has addressed the concerns and questions raised by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper proposes Visual Fourier Prompt Tuning (VFPT) to address performance degradation in parameter-efficient finetuning (PEFT) methods caused by dataset disparities. VFPT integrates Fast Fourier Transform (FFT) into prompt embeddings, enhancing performance with minimal parameter usage. This work is built upon the VPT method. The key innovation is to integrate frequency domain information using FFT to VPT. The performance is evaluated on VTAB-1k and FGVC.
strength_weakness: strength_weakness: Overall, this is a well-organized paper.  I have several questions regarding the methods and results;
1. What is the larger model’s performance when applying the VFPT? Whether the VFPT can work well with large models is an important question that hasn't been answered in the current version. As the model size grows, efficient, prompt learning can be more critical. However, current results are mainly on a relatively small size of models.
2. Whether VFPT can be integrated into the CLIP model to enhance zero-shot performance is also an interesting question. I am very interested in vision and text encoders and whether VFPT can be applied to CLIP models. 
3. Current results do not answer the first question in the introduction: “Can prompt tuning generalize across datasets with varying disparities?”. It confuses me because I think of the transferable prompt design, which is trained on one dataset and then transferred to another. This motivation, to me, needs a further rephrase or validation.
questions: questions: I have several expectations for seeing the potential of VFPT. If the author can address these questions, I will change my scores.
limitations: limitations: Yes, the authors discussed the limitation of their methods.
paper_summary: paper_summary: This paper introduces Visual Fourier Prompt Tuning (VFPT), an approach for parameter-efficient fine-tuning of large vision models. VFPT integrates Fast Fourier Transform (FFT) operations into visual prompt tuning, allowing it to incorporate both spatial and frequency domain information. The method demonstrates superior performance and generalizability across various tasks and datasets, particularly when there are significant disparities between pretraining and fine-tuning data. VFPT outperforms several state-of-the-art baselines on benchmarks like VTAB-1k and FGVC, using only a small fraction of trainable parameters. The authors provide extensive empirical results, ablation studies, and visualizations to demonstrate the effectiveness of their approach. They also explore the optimization landscape and interpretability aspects of VFPT.
strength_weakness: strength_weakness: 1. The paper lacks a rigorous theoretical analysis of why integrating Fourier components improves generalization. While empirical results are strong, a deeper theoretical investigation could provide insights into:

- Why frequency domain information helps with dataset disparities

- How the balance between spatial and frequency information affects model behavior

- Potential limitations or failure cases of the approach

Suggestion: Develop a theoretical framework explaining the interplay between spatial and frequency domain information in prompt tuning. This could involve analyzing the properties of the loss landscape or examining how Fourier components affect the model's representation space.

2. The Fourier percentage (α) is a crucial hyperparameter, but its selection process is not fully automated. The paper suggests using dataset disparity as a guideline, but this may not always be easily quantifiable.

Suggestion: Develop an adaptive method to automatically determine the optimal Fourier percentage based on task characteristics. This could involve a small meta-network that predicts the optimal α, or a dynamic adjustment mechanism during training.

3. While the paper mentions a slight decrease in training speed (2.8% on VTAB-1k), a more comprehensive analysis of computational trade-offs is missing. This is important for practitioners considering adopting the method.

Suggestion: Provide a detailed analysis of computational overhead across different settings, including:

- Training time comparisons with baselines

- Memory usage

- Inference time implications

- Potential optimizations to reduce overhead

4. The paper primarily uses FFT as a black-box operation. A deeper exploration of how different frequency components contribute to performance could yield additional insights.

Suggestion: Conduct experiments analyzing the impact of different frequency bands on model performance. This could involve selectively filtering certain frequency components or visualizing which frequencies are most important for different tasks.

5. The generalization to other architectures is not clear. While the method is tested on ViT and Swin Transformer, its applicability to other architecture types (e.g., convolutional networks, MLP-Mixers) is not explored.

Suggestion: Extend the experimentation to a broader range of architecture types to demonstrate the method's generality. If challenges arise, analyze and discuss the limitations of applying VFPT to different model families.

6. The paper doesn't deeply explore how Fourier prompts interact with regular prompts or how they influence different layers of the network.

Suggestion: Conduct a layer-wise analysis of how Fourier prompts affect feature representations throughout the network. Visualize and quantify the interactions between Fourier and regular prompts to provide insights into their complementary roles.

7. The paper doesn't thoroughly address the robustness of VFPT to different initializations or potential instabilities during training.

Suggestions: Perform a comprehensive stability analysis, including:

- Sensitivity to different random seeds

- Learning rate schedules that work well with VFPT

- Potential gradient issues or training instabilities

- Robustness to different prompt lengths and configurations

8. While benchmark performance is strong, the paper lacks discussion on how VFPT performs on real-world, large-scale applications outside standard benchmarks.

Suggestion: Include case studies or experiments on practical, large-scale vision tasks (e.g., autonomous driving, medical imaging) to demonstrate the method's effectiveness in real-world scenarios. Discuss any challenges or modifications needed for such applications.
questions: questions: Please refer to the weaknesses section.
limitations: limitations: Limitations:

The authors do discuss limitations in Section S9 of the appendix, which is a positive point. They specifically mention the introduction of an additional hyperparameter (Fourier percentage) and acknowledge that while dataset disparity can guide its selection, an automatic search method could further improve efficiency. This shows awareness of a key limitation in their approach.

However, the limitations discussion could be more comprehensive. For example:

1. They could elaborate on potential scenarios where VFPT might not perform well.

2. Discussion of scalability limitations or computational constraints for very large models is not clearly addressed.

3. Potential limitations in applying VFPT to other domains or model architectures could be explored further.

Societal Impact:

The authors do discuss social impact in Section S9 of the appendix, which is commendable. They highlight positive aspects such as:

1. Potential for improving model accuracy without substantial computational overhead.

2. Applicability in computationally-sensitive real-world applications.

3. Advancements towards achieving generality across datasets.

However, the discussion of potential negative societal impacts is limited. The authors could have explored:

1. Potential misuse of more efficient and generalizable models in privacy-invading or surveillance applications.

2. Environmental impacts of encouraging more fine-tuning experiments.

3. Potential biases that might be introduced or amplified through this method.
paper_summary: paper_summary: This paper introduces Visual Fourier Prompt Tuning (VFPT), a novel approach to parameter-efficient fine-tuning (PEFT) for large-scale Transformer-based vision models. VFPT integrates Fast Fourier Transform (FFT) with prompt tuning, allowing the model to adapt to new tasks effectively by utilizing both spatial and frequency domain information, especially when there is a significant disparity between the datasets used in pre-training and fine-tuning phases. This method not only retains the simplicity and intuitiveness of standard visual prompt tuning but also enhances model performance across various tasks without substantially increasing the parameter count. Empirical results show that VFPT outperforms several state-of-the-art PEFT methods on benchmark datasets like VTAB-1k and FGVC, achieving higher accuracy with fewer parameters. For instance, VFPT uses only 0.66% of the total model parameters while achieving 73.20% mean accuracy on VTAB-1k, surpassing both full fine-tuning and conventional visual prompt tuning methods.
strength_weakness: strength_weakness: 1.	The abstract states that the proposed method, VFPT, utilizes only 0.57% of the model parameters to achieve a performance of 73.20% mean accuracy on VTAB-1k dataset. However, Table 1 contradicts this by listing the parameter usage as 0.66%. This discrepancy raises concerns about the accuracy of the reported statistics.
2.	The paper does not fully elucidate the complexity introduced by integrating FFT into the prompt tuning process. While the method is praised for its parameter efficiency, the computational overhead, especially in terms of runtime and memory consumption during the FFT operations, is not thoroughly discussed.
3.	Table 3 reveals that VFPT appears to underperform relative to some established partial tuning and extra module methods when applied to models pretrained with MAE self-supervised objective. This might indicate that VFPT's reliance on Fourier transforms does not synergize as effectively with the feature representations or data distributions typical of self-supervised learning models. Understanding why VFPT performs less effectively in this context is crucial for refining the approach or identifying its optimal application scenarios.
questions: questions: Table 5(b) explores the effects of different Fourier Prompt Locations (Prepend, Append, and Random) on the performance of Visual Fourier Prompt Tuning (VFPT). The results suggest variations in performance based on the placement of these prompts within the input sequence. Since VPT mentions that different prompt locations are mathematically equivalent, I wonder why the location of Fourier prompts impacts model performance? The following sentence is from section 3.2 of the original VPT paper: “Notably for ViT, $x_N$ is invariant to the location of prompts since they are inserted after positional encoding, e.g., $[\mathbf{x_0}, \mathbf{P}, \mathbf{E_0}]$ and $[\mathbf{x_0}, \mathbf{E_0}, \mathbf{P}]$ are mathematically equivalent. This also applies to VPT-Deep.”
limitations: limitations: The limitations have been discussed in Appendix S9.

Comments: title: Thank you for your valuable suggestions
Thanks for the rebuttal
Thank you for your positive response
Acknowledging response
Looking forward to the discussion
Thank you for the prompt reply
comment: Thank you for your valuable suggestions. We are immensely appreciative of the discussions on the performance of large models and vision-language models, as they clearly illuminate the future direction of our work.
The improvement over scaling up of models and CLIP is not marginal; hence, it enhances the authors' claim.
We are delighted to hear that we have successfully addressed all your concerns and received your approval of the work.
The authors have addressed my comments. Considering the feedback from the other reviewers and the authors' responses, I raise my rating to a weak accept for this submission.
Dear Reviewers,

We sincerely appreciate the time and effort you've devoted to reviewing our work. We understand that your schedule may be quite busy, especially during the weekend. As the authors-reviewer discussion phase draws to a close, we kindly request your attention to our responses (special thanks to Reviewer TrZ7 for acknowledging our response). Our aim is to gain insights into whether our responses effectively address your concerns, and to ascertain if there are any additional questions or points you would like to discuss.

We look forward to the opportunity for further discussion with you. Thank you for your thoughtful consideration.

Best regards,\
The Authors
Thank you for your prompt response and the constructive feedback, which is crucial for enhancing our work.
I appreciate your detailed response. The authors have addressed all the concerns I raised in my initial review, and after considering the other feedback, I have raised my score to 7.
(Continue from where rebuttal ends.)

**Q5: The generalization to other model families.**

**A5:** Thank you for the great suggestion. Following VPT, we further conduct experiments on ConvNeXt-Base pretrained on ImageNet21K. ConvNeXt-Base is a convolutional neural network with size 87.6M. We follow the padding strategy to incorporate learnable prompts for the input images for an intuitive setup. The results presented below show a noticeable improvement, indicating the generalization of VFPT to other architectures.

|  ConvNeXt-Base    | VTAB-1k Specialized (4) |
| ---- | ------------------- |
| FULL FT | 83.73%              |
| VPT  | 83.00%              |
| VFPT | 83.69%              |

In addition, we conducted a preliminary study on a vision-language model: CLIP. Specifically, we applied VFPT to the vision encoder of CLIP using the optimal settings from this work. The results on ImageNet are shown in the table below. As seen, there is a noticeable improvement compared to the standard CLIP, indicating the generalization of VFPT to different models. We’ll provide these additional results and discussions in the revision.

|    ImageNet  | Tuned / Total (%) | Accuracy (%) |
| -------------- | ----------------- | ------------ |
| CLIP (zero-shot) | 0.00              | 66.73       |
| CLIP + VPT     | 0.22              | 70.30        |
| CLIP + VFPT    | 0.21              | 71.49        |

**Q6: Regarding the interaction of Fourier prompts and regular prompts.**

**A6:** We conduct experiments analyzing the impact of prompts, as detailed in Section 4.4. We observe from the 3D attention map (Figure 4(a)) that there is a pronounced accumulation of attention scores at both Fourier and regular prompt locations, indicating that these prompts substantially impact the feature representation learning. These observations are from the raw average attention head in the last layer of VFPT, and we plan to investigate how they influence different layers of the network as suggested.

**Q7: Regarding model robustness and stability.**

**A7:** Thank you for the great suggestion. In our experiments, all results are averaged over three runs with different random seeds to mitigate potential sensitivity issues related to random seed variation. Additionally, we conduct the same grid search using the same learning rate and weight decay as VPT and E2VPT, detailed in Section 4.1. We further include an ablation study on different prompt lengths and configurations in Appendix 2.5 to evaluate the robustness across varying prompt lengths and Fourier percentages. 

Following the suggestion, we conduct an additional ablation study with different initialization techniques (i.e., xavier uniform [4] and He. normal [5] initialization scheme). As seen, both He and xavier initialization show competitive results, validating the robustness of VFPT w.r.t. prompt initialization. We’ll supplement the results with further discussion in the revision.



|    Initialization     | VTAB-1k Specialized (4) |
| --------------------- | ----------------------- |
| VFPT-He.              | 84.88%                  |
| VFPT-Xavier (default) | 84.93%                  |

[4] Understanding the difficulty of training deep feedforward neural networks. AISTATS 2010.

[5] Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. ICCV 2015.


**Q8: How does VFPT perform on real-world, large-scale applications.**

**A8:** Thank you for the question. We want to highlight that VTAB-1k includes several insightful scenes valuable for real-world and diverse applications. For instance, the **specialized group** contains two sub-groups: remote sensing and medical. These are particularly useful for applications like satellite monitoring and medical diagnosis. Many tasks from the **structured group** are generated from simulated environments, contributing to object detection in virtual video game contexts. Our results on VTAB-1k demonstrate the potential of VFPT for various applications. In the future, we plan to employ VFPT to tackle more challenging real-world applications, such as out-of-distribution detection.

**Q9: Regarding Limitations and Social Impact.**

**A9:** Thank you for all the excellent suggestions. We’ll provide more discussion according to the suggestion in our revision.

We appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.