Visual Fourier Prompt Tuning
Authors: Runjia Zeng
Cheng Han
Qifan Wang
Chunshu Wu
Tong Geng
Lifu Huang
Ying Nian Wu
Dongfang Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-efficient finetuning
Primary area: machine_vision
TL;DR: Visual Fourier Prompt Tuning (VFPT) is a simple yet effective parameter-efficient finetuning approach for large-scale vision Transformer models, addressing the challenge of dataset disparity between pretraining and finetuning phases.

Abstract: With the scale of vision Transformer-based models continuing to grow, finetuning these large-scale pretrained models for new tasks has become increasingly parameter-intensive. Visual prompt tuning is introduced as a parameter-efficient finetuning (PEFT) method to this trend. Despite its successes, a notable research challenge persists within almost all PEFT approaches: significant performance degradation is observed when there is a substantial disparity between the datasets applied in pretraining and finetuning phases. To address this challenge, we draw inspiration from human visual cognition, and propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach innovatively incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Apart from its inherent simplicity and intuitiveness, VFPT exhibits superior performance across all datasets, offering a general solution to dataset challenges, irrespective of data disparities. Empirical results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks, with low parameter usage (e.g., 0.57% of model parameters on VTAB-1k) and notable performance enhancements (e.g., 73.20% of mean accuracy on VTAB-1k). Our code is avaliable at https://github.com/runtsang/VFPT.

Visual Fourier Prompt Tuning
Runjia Zeng",
Cheng Han"", Qifan Wang', Chunshu Wu', Tong Geng', Ying Nian Wu and Dongfang Liu
Lifu Huang',
'Rochester Institute of Technology University of Missouri Kansas City *University of Rochester
Meta AI
SCC Davis University of California, Los Angeles
Abstract
With the scale of Transformer-based vision models continuing to grow, finetun- ing these large-scale pretrained models for new tasks has become increasingly parameter-intensive. Visual prompt tuning is introduced as parameter-efficient finetuning (PEFT) method this trend. Despite its successes, a notable research challenge persists within almost all PEFT approaches: significant performance degradation is observed when there a substantial disparity between the datasets used in pretraining and finetuning phases. To address this challenge, we draw inspiration from human visual cognition, and propose the Visual Fourier Prompt Tuning (VFPT) method an effective and efficient solution for adapting large- scale Transformer-basee models. Our approach innovatively incorporates the Fast Fourier Transform into prompt embeddings, seamlessly integrating both spatial and frequency domain information. Apart from inherent simplicity and intuitiveness, VFPT exhibits superior performance across various tasks, offering a general solu- tion to address the data disparity challenge. Empirical results demonstrate that our approach outperforms several state-of-the-ee baselines two benchmarks, with low parameter usage (e.g., 0,57% of model parameters on VTAB-IK Ikk and notable performance enhancements (e.g., 73.20% of mean accuracy VTAB-1k). Our code is avaliable at / com http/runtsang//V// /VFPT
Introduction
" "Forriers theorem not only one of the most beautiful results of modern analysis, but it may be said to furnish indispensable instrument in the treatment of nearly every recondite question modern physics."
Lord William Thomson Kelvin
Prompt tuning 2 23 initially introduced for parameter-efficient adaptation of large foundation models in natural language processing (NLP), As vision models continue to scale for enhanced performance, visual prompt tuning [4] has been applied to various vision domains (e.g., image classification (5), segmentation 66] 77 detection (8), demonstrating superior performance and lower parameter usage compared to other parameter-efficient fine-tuning (PEFT) methods. However, a common challenge within the research community remains unaddressed: significant performance degradation occurs when there is substantial disparity between the data used in pretraining and finetuning 2 10 This issue hinders the broader application of visual prompt tuning. Consequently. natural question arises: @ Can prompt tuning generalize across datasets with varying disparities?
As researchers commonly draw insights from human to replicate the principles in intelligent [3] [4), we consider to answer this question from the human visual cognition's perspective, While humans comprehend the world through past experiences/knowlddee it essential to generalize and adapt this understanding to new tasks efficiently and effectively. The robust and rapid adaptability of human visual cognition thus arises from various domain analysis, capturing the new patterns from different channels and perspectives [5] 16
chines 11 12
Equal contribution.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Interestingly, we find that the paradigm of visual prompt tuning is conceptually analogous to human visual cognition. While the frozen large-scale vision model functions as accumulated knowledge, the
fast adaptation mechanism resembles visual prompt tuning, requiring the incorporation of diverse
domains of information (e.g., time, frequency) to achieve comprehensive understandings [18 19] 20).
The Fast Fourier Transform (FFT) [8] 19 (0], renowned for its ability to convert signals from their original domain (e.g.. time or spatial) to the frequency domain and vice versa, serves as an ideal tool for contributing informative insights in the frequency domain. By leveraging the capabilities of FFT, visual prompts can naturally integrate both spatial and frequency domain information during finetuning, thereby enabling the frozen vision model to achieve consistent and robust performance across datasets with varying disparities. Consequently, our research question evolves into: @ How can FFT be integrated into visual prompt tuning to emulate the human visual mechanism?
To this end, we employ simple yet effective strategy that utilizes the Fourier operations to facilitate visual prompt tuning (see Fig. [c). By integrating frequency domain information into learnable prompt embeddings, our approach elegantly assimilates data from both spatial and frequency domains, simulating the human visual cognition. We name our approach Visual Fourier Prompt Tuning (VFPT), which exhibits several compelling advantages: Simplicity. The intuitive application of FFT in prompt tuning emulates the rapid processing capabilities of the human visual system, making VFPT both elegant and straightforward to implement (see $2.1 @ Generality. By incorporating frequency domain information, the search space for latent embeddings of prompts is naturally expanded, resulting in advanced enhancement in performance across different datasets and tasks with varying data disparities (see our analysis of the optimization process, which enables smoother navigation towards local minima, increasing flatness around them and exhibiting apparent convexity. 2 Interpretability. To intuitively demonstrate the advantages of Fourier components, we visually illustrate that the introduction of Fourier transform in visual prompt tuning results in a markedly higher concentration of attention scores within the Transformer's input space, which correlates positively with enhancements in performance (see $4.4). This observation, in turn, explains the effectiveness of our approach.
The generality of our model is further illustrated through
Comprehensive experiments are conducted to evaluate the performance of VFPT. In 2 we conduct a literature review and discuss relevant works. Our approach is presented in 3 where we describe how we simple yet effectively integrate FFT into visual prompt tuning. In 44.2 we preseentcommplll experimental results on various benchmarks, backbones, and different pretraining objectives, achieving superior performance without complex engineering design. Specifically, our approach achieves an average improvement of 7.63 % in accuracy VTAB- 1k compared to full finetuning, and 3.77% % compared to VPT [4. In $4.4 we demonstrate that the FFT prompts significantly enhance the activation of the frozen vision model. Additionally, we study the optimization process of prompt tuning approaches, indicating that VFPT provides a more favorable optimization process. Finally, we demonstrate the strong algorithmic generalization of our approach to the language domain, and show additional visual explanations in the Appendix. We trust that this work provides valuable insights.
2
Related Work
2.1 Visual Parameter-efficieet Finetuning
With the significant growth in the scale of vision models, especially following the emergence of
Vision Transformers 21 22 23 24 25}, the development of PEFT methods under "pretrain-the -thnn-
finetune" paradigm becomes increasingly critical. Current methods under this paradigm can be generally categorized into partial tuning (26] 27 28 extra module (i.e., including reparameterization approaches such as Low-Rank Adaptation (LoRA) (29) 30 31 32]
10] 35 36]. and prompt tuning 37 38 39 40 41 Partial tuning and extra module face several limitations that hinder their application. ① Unsatisfactory performance: they generally cannot reach competitive performance
33
34
@ Model-oriented design: most research requires 32 during tuning, rendering them non-universal solutions when considering different backbones. In contrast, prompt tuning [2]. originally proposed for srgguugge-dmaiin 42 44 45). provides a general and straightforward solution in vision with
with regard to full finetuning 44] 26 27
28
33
10
to insert specific architecture/llock design B1
30
powerful performance gains. signals a new paradigm in PEFT in the field of computer vision.
Generally, prompt tuning introduces a sets of learnable parameters to the input sequence of backbone models, updating only these parameters during the finetuning. Despite its apparent simplicity, the paradigm of visual prompt tuning has demonstrated notable performance enhancements. Current

---

developments on visual prompt tuning primarily concentrate on engineering optimizations, such as reducing parameter usage [5] and expanding upplicability across diverse tasks [39 46
47 48). These
approaches often involve introducing additional constraints and functionalities to the foundational design, which deviate from the principles of simplicity and elegance to the original concept of visual prompt tuning. Our approach, in sharp contrast, endeavors to explore visual prompt tuning from the perspective of human visual intelligence, while diligently maintaining the simplicity of prompt tuning. It is also essential to emphasize that visual prompt tuning diverges markedly from visual instruction tuning [49] (i.e., aiming at improving the model's ' instruction following abilities).
2.2 Fast Fourier Transform in Vision
FFT is a powerful mathematical algorithm used to compute the Discrete Fourier Transform (DFT) and its inverse [50]51]. is pivotal in information processing, allowing the detailed analysis of various signals (e.g., image [52 (3 54, radar [55 56 57) for frequency determinations. In vision, FFT's ability to transform complex data in spatial domain into frequency domain makes an invaluable tool for abstracting critical features from noisy or nigh-dimensional datasets [58
59] This abstraction is
particularly beneficial as the identification of salient features are shown to have better generalization
ability across domains [60] 61 62 63], directly influences the performance 64
65
66
67 of image
analysis and processing tasks. Current research on FFT in vision predominantly explores areas such conventional image processing [52 68 69 [70,, image pre- preppeccessing for deep neural networks
(DNNs) 72 and DNN architectural design 220 66 65 73 74 75 76].
Despite its profound utility and effectiveness, the integration of FFT within the paradigm of visual prompt tuning remains largely underexplored. Recent work [77 adapts the pretrained multi-modal network to the tasks under modality-incompler approach demonstrates the potential of FFT operations to handle missing modalities (i.e., substantial disparity) effectively. However, it primarily focuses task-specific optimization and design. The extensive applicability and generality of FFT, especially in cross-dataset analysis, have yet to be recognized or exploited. Another work [3)] incorporates Fourier transform into the LoRA-based approach. While the expressive Fourier basis facilitates the recovery of weight changes, it does not fully integrate frequency domain information during finetuning, which remains orthogonal to approach. In this paper, we aim to broaden the scope of exploration and contribute to advancing the field of Fourier-based research in vision. By studying the integration of FFT with visual prompt tuning, we fully explore how to improve both the efficacy (see $3) and the adaptability of learning models diverse and challenging datasets (see $ Furthermore, we present novel evidence indicating that VFPT establishes strong correlations within the Transformer's 's input space, aligning with the performance enhancements (see $4.4). Overall, the generality of VIPT suggests novel understanding of the Fourier-based method in current machine learning applications.
segmentation scenarios via FFT prompt tuning. This
3
Methodology
In this section, we introduce VFPT, novel visual prompt tuning approach for effective and general large-scale transformer-bases model finetuning. We first define the problem and notations of visual prompt tuning and FFT in $3.1 The integration of Fourier-based visual prompt tuning is presented in The overall framework is shown in Fig. [c), where we compare our model with original VPT.
$32
3.1 Preliminary
Visual Prompt Tuning. Given a pretrained Transformer model T with N layers, the objective of prompt tuning vision is to finetune a model T into new task with only few set of d-dimensional embedding vectors, i.e,, prompts, in the input space after patch Emb layer. These learnable prompts are defined as P= (PI, p2...., pNJ, where Ps represents the learnable visual prompts in the inh encoder layer. Formally, the encoder layers with prompts are defined as:
z = L E) z = L( zi-1) i 2.3..... N
where the embeddings of the input image patches E are initialized with frozen Emb projection, and Z' is the contextual embeddings computed by the in encoder layer. The colors and m indicate trainable and frozen parameters, respectively. Here, trainable prompts only accounts for small proportion of the total parameters (e.g.. 1.14% on VTAB-IK 1k (78) in VPT
3

---

Frozen
Head
Visual Fourier Proingis
Head
Tuned
Transformer Encoder Lay er Lx
2D EDFFT in Partial Prompte
WIl
Transformer Encoder Layer Ly
Sequence-wise FFT
Transfomerr Encoder Layer 12
Transformer Encoder Layer L
Transformer Encoder Layer L,
Hidden-wise FFT
Trapsformer Encoder Layer L;
Visual
Prompts
(a) Visual Prompt Tuning
(b) Fast Fourier Transforn in Prompts
(c) Visual Fourier Prompt Tuning
Figure 1: Overview VPT us. VFPT (ours) frameworks. (a) Original Visual Prompt Tuning. (b) 2D Fast Fourier Transform operations in partial visual prompts along hidden and sequence length dimensions. (c) The overall architecture of our proposed VFPT (see $3.2).
Fast Fourier Transform. The FFT is powerful algorithm for computing the Discrete Fourier Transform (DFT), which transforms a finite sequence of equally-spaced function samples into a same-length discrete-time Fourier transform sequence. Specifically, given sequence (zn) where a member of the interval n E [0, 1], the DFT is defined as:
N-1
F(x)
Σ Tne m=0
12 & DSKSN-1.
For finite sequence of equally-spaced samples { the DFT generates a same-length sequence of equally-spaced samples {. This transform denoted as F. The initial DFT is in complexity Onn)). For acceleration, we use Cooley-Tukey FFT algorithm [79] following common practice (80) (i.e., complexity (n log n)). FFT serves as a powerful tool for domain transition. Consequently, we explore the integration of the FFT operation within PEFT methods, particularly in prompt tuning.
3.2 Visual Fourier Prompt Tuning
Visual prompt tuning is particularly useful under the meetrain-then-finetie paradigm. However, suffers a significant performance reduction when substantial disparities exist between pretrain and finetune datasets. The reason is that during finetuning on new data, the image distribution may deviate markedly from the examples used in pretraining the backbone model [9. Existing prompt tuning 4 5), focusing predominantly on spatial information, can only harness the shared information embedded within the pretrained backbone, limiting their capacity to adapt effectively to novel tasks. Thus, it is crucial to strengthen the ability to capture distinguishing feature from finetuning data.
To this end, we introduce VFPT, an intuitive yet powerful method with advanced performance and generality. Compared to VPT (see Fig. a our model (see Fig. [(c) transforms partial prompts from spatial domain to frequency domain via 2D FFT (see $3.1) to consider both the spatial and frequency domain information. Formally, for each learnable visual prompts in the ém encoder layer pi P= (P1, p2...., PN J. we have p (pi, .h.,p,,.., PPr)). We select m partial prompts as visual Fourier prompts at each layer, where 0 < < M. Further, Fourier participation, where zero indicates all prompts are original visual prompts, and one implies visual prompt embedding input with respect to both sequence (i.e., Fsgq) and hidden dimensions (i.e., F.D. Note that the operations Fseq (z)) and F (F.FFso (x)) are mathematically equivalent due to the commutative property of the two one- mie-dmmesioonl FFTs 80 Here, indicates Fourier operations.
m// represents the fraction of
all prompts given after FFT. We apply 2D FFT
P R (Fesq (Fo P ( .))
To maintain the pretrained structure's consistency, we only alter the prompt embeddings, and thus retain only the real component (i.e., R) from the output. This design does not require any adjustments to accommodate complex numbers in the self-attention module, ensuring that the remaining clements

---

of the model remain unchanged. Consequently, the overall integrated prompts pi in the it encoder layer are formed by the concatenation between the visual Fourier prompts and visual prompts as:
p Pm PM
(4)
Our elegant design of VFPT enjoys a few appealing characteristics: Simplicity: VFPT only requires several lines of code based on the implementation of the visual prompt tuning. Its intuitive integration of information between spatial and frequency domains brings nearly free performance efficacy. The low complexity of FFT (i.e.. (n log n)) leads to an overall marginal reduction during the training schedule.(i.e., 2.8% on VTAB- Ik 78). In sharp contrast, current endeavors in visual prompt tuning mainly emphasize augmenting architectural complexity for superior performance 55 81 42 undermining the inherent simplicity of prompt tuning and introducing significant training overhead (e.g., [8]] learns 2D prompt token map for densely image relationship construction, [5] incorporates additional self-attention K-V prompts). Generality: The frequency and spatial analysis of imagery inputs can be mutually complemen- tary, leading more comprehensive feature understanding from distinct perspectives (e.g., the frequency domain allows for the distraction and decomposition of luminance and noise to a con- siderable degree (82), while the spatial domain excels in capturing intricate object details). By incorporating learnable prompts from both domains, VFPT demonstrates enhanced prompt learning capabilities, which makes it superior to finetune across diverse tasks (see $4.2). The empirical findings of flatness and convexity of VFPT further strength our claim.
Interpretability: In visual prompt tuning, notable challenge arises concerning the interpretability of learnable prompts. Unlike in NI., where tokens explicitly represent these prompts, visual prompts have historically lacked a clear and explainable representation. In order to intuitively perceive the function of visual prompts, we offer a possible way to understand why prompts play an important role in fine-tuning a new task through the visualization of attention maps. Moreover, we can also observe a better and stronger global feature learning pattern through introducing visual Fourier prompts, showing how Fourier prompts work. More discussion will be elaborated $4.4
Experiment
4.1 Experiment Setup
Datasets. Following common practice [5 4 81 83 our experiments are carried out on two image classification benchmarks. VTAB-IK 78 collects 19 benchmarked Visual Task Adaptation, separated into three groups: (1) Natural includes natural images captured by standard cameras, (2) Specialized consists of images taken by specialized equipment, and (3) Structured considers tasks considering geometric comprehension (i.e., counting, distance), which has substantial dataset disparities (i.e., tasks in Natural and Specialized are closely related to image classification and thus have low disparities, while tasks in Structured are regarded as distinct from image classification) when comparing to
the pretrained dataset (9] (i.e., ImageNet21K (44). ). Each task of VTAB-IE kk contains 1000 training
examples with the 800/200 split for train/val set. FGVC contains 5 benchmarked Fine-Grained
Visual Classification, including CUB-200-201 2011 85), NABirds [6) Oxford Flowers [87], Stanford
Dogs (8) and Stanford Cars (9). The training set is split into 90% train and 10% val.
Baselines. For consistency, we follow 54 and compare VFPT with other widely applied parameter- efficient fine-tuning methods. Results of two vision transformer architectures, Vision transformer 223
(ViT) and Swin transformer (24] (Swin), on image classification are discussed in $4.2 We also apply VFPT two eeff-supervised objectives: MAE [90] and MoCo v3 (26).
Training. Following [4,5], [5, we conduct grid search to find the best tuning hyperparameters, learning
rate (i.e., [50. 25, 10. 5, 2.5, 1, 0.5, 0.25, 0.1, 0.05)), and weight decay (i.e., [0.01, 0.001, 0.0001, 0.0)) on val set. Notably, VFPT does not require specific-designed large learning rate in [4]. The learning rate is scheduled by a cosine decay policy and trained for 100 epochs.
Reproducibility. VFPT is implemented in Pytorch [91]. Experiments are conducted on NVIDIA A100-40GB GPUs. To guarantee reproducibility, our full implementation will be publicly released.
4.2 Main Results
In this section, we demonstrate the effectiveness of VFPT from two key perspectives: Supe- rior Performance: Our model demonstrates significant performance improvements across diverse datasets, including challenging tasks with large disparities in data, thus showcasing its generalizability.
5

---

Table I: Image classification accuracy for ViT-Base/Ik [3] pretrained supervised ImageNet-211 Following [4] 51, we report the average test accuracy (three runs) on FGVC [4] and VTAB-1k [78] benchmarks, and "Number of Wins" in [-] compared to full fine-tuning (Full) [92] denotes the method with highest "Number of Wins" compared to Full. We further report "Number of Wins to VPT in {.). "Tuned/Total" is the average percentage of tuned parameters required by 24 tasks. ""Sopee" indicates the tuning scope of each method. "Additional parameters" is the existence of parameters in addition to the pretrained backbone and linear head. Bold and Underline indicate the best and the second best results. VFPT outperforms full fine-tuning in 22 of 24 instances with fewer trainable parameters and beats VPT in 23 of 24 cases with lower parameters. denotes methods using soft filtered prompts to reduce the parameter usage in learnable visual prompts, requiring specialized devices to facilitate acceleration. task results are available in Appendix. Same for Table 2 and
ViT-Base/To
(85.8M)
Total
Tnput
Scape
Backbome
parmmns
PGVC 4 [5]
Natuoal
VIAB-IE Specialized [4]
Strectured
Mean Total
Fall Firee
TUR10055
54.54%
75. 18.38%
53.36%
47.64%
65.37%
Lineur
22
01.08%
79.32% [0]
68 93%
77.10%
26,845
52.94%
Pantil-1
(0)
60, 44%
14.17%
525
MLP-3 covina
[0)
(0)
30.62%
53.21%
Sidelume (1Sc
K1
1008%
75. (0)
58.21%
1274 (0)
23:415
45 45.65%
Bias (iaama
30
0.80%
88.41% [8]
73.30% 1.5]
78.25 25%
10
44.09% 121
02 05%
Adupteer Neatin
12
1.025
85 46% [1]
70.67% [4]
77 8179
(0)
13.09'% (0]
62-4 41.4
LoRA nitu022
331
89.16% [3]
78, 266 [5]
83, 13.78%%
(2)
56.20% [7]
72 25.25%
AdapdFormer
95
50.56%
84 58/8 (4)
58. $8.635
22 37333
ARCat
19.
85.55%
58,38%
72
[1]
76.81%
(0)
46.95%
64.35%
VPT-D
0.73%
[4]
78, 48%
[2]
54.91%
43%
EXPRES IovIE
79.69%
0300 [3]
54, 54.99%
70 20/3
1E2PP
S
0.39%
19.2216 [4]
30.011% [6]
84.43% [3]
57.39% [8]
71 42%
Durs
0.6665
89.34% [4] [4]
81.35% [4] (7)
94.939 [4]
[4]
60.19% [8] (8)
73.20P
4 Fourier Contribution: We observe that Fourier components play critical role in VFPT, where tasks with larger data disparities tend to favor higher percentages of Fourier components.
Definition of disparity. Following 99. Table 2: Image classification accuracy for Swin-
we use the Fréchet Inception Distance (FID) 199, 100] to measure the disparity between the datasets used in pretraining (i.e., ImageNet) and funetuning (i.e., down- stream tasks). Average FID scores of each group are reported in Fig- 2 where the Natural group has low disparities due its close relationship to ImageNet2IK [84] and the Specialized and Structured groups (ie.. orientation prediction task) are considered distinct from image classification. The dataset description of VTAB- 1k is covered in $4.1 (FGVC is excluded due to lack of categorization).
Base [24] pretrained on supervised ImageNet-21k.
Swin
Tuned/ Total
VIAB-IK (28 [D]
(86.7M)
Natural [7]
Specialzed [4]
Scruetuedd [8]
79.100 19.10%
86.21%
59.65%
Lincar 10
08
71.325 [5]
80.77% [1]
14.32% [0]
Partial-1 piadtrn
9
14.58%
73.11%
[4]
81.70% [0]
34.96% [0]
MLP-3 s
[4
2.42%
73.56% [5]
75.21% 101
35.69% [0]
74.19%
80.145 [0]
42.42% [0]
VPT movoan
0.25%
76,78%
[6]
83.33%
[0]
$1.85% [0]
EZVPT pncei
0.21%
83.311
[6]
84.95% [2]
57.35% [3]
0.27%5
84.33%
(7)
86.15%
(4)
58.21%
Superior Performance. In order to have a comprehensive understanding on generality, we examine VPPT ViT-Base/16 (23. Swin-Base [24], and two self-supervised objectives, following common practice [ ( We also report the individual per-task results for Table [ 2 and 3 in Appendix.
VFPT on VIT. We report the average accuracy score on VTAB-1k and FGVC benchmarks across four diverse task groups for three runs in Table [ where fifteen protocols under
paradigm are considered. Specifically, Full [92] updates both backbone and classification head;
Linear 192]. Parital-1 [93] (top layer), and MLP-3 [94] (3 MLP layers) are partial tuning approaches; Sidetune 311, Bias [30,. Adapter (32), LORA [35), AdaptFormer (95) and ARCat.. (96) are extra module methods which add new trainable parameters to backbone for adaptation; VPT-S 4 VPT- 4. EXPRES (97] and ETVPT (5) are concurrent visual prompt tuning approaches. Consequently, we have several key observations. First, VFPT is able outperform the full fine-tuning method in 22 out of 24 tasks. For example, our model achieves 0.13% % improvement on FGVC and 5.21 % improvements on VTAB-Ik Structured, respectively. The empirical results show the effectiveness of VFPT. Second, VFPT tunes only 0.66% of the overall parameters in the backbone, establishing it competitive method within the PEFT approaches. Third, while VPT struggles to capture the image information when having significant dataset disparity, VFPT achieves notable performance improvements by integrating both spatial and frequency information (see $3.2) without additional architectural modifications. (i.e.. 60.19% % us. 54.98% on "TAB-Ik Structured).
VFPT on Hierarchical Transformer. We further extend VFPT to a hierarchical transformer Swin-Base (24 for architectural generalization. The MSA layer of Swin is employed in local shifted windows, and patch embeddings are merged at deeper layers. For consistency, we follow the same settings from ViT to apply and prepend Fourier prompts ahead of the visual prompts. The results on

---

Table 3: Image classification accuracy for different pretrained objectives MAE 90 and MoCo
v3 26 with ViT-Base [23 as backbone. denotes the rerun results that calibrate the VPT
Pretrained
Methods
Theal
Natural
VTAB-IK
Speciallon [4]
[19
Siructaree [5]
Tunedi
Natural [7]
VIAB-IK [19]
Specialize
100.00%
59.31%
79.88
53.82%
T000005
71.55%
84.72%
51.98%
18.87%
[0]
0,44
(0)
Partial-1 (hoatts
8.50%
58.44%
78.28% [1]
47,648111
8.30%
72.31%
51.58%
47.89% [1]
Blas (Nsttt5
10
0.16%
54.35% [0
75.65% (1)
47.70% [0]
0.16%
72.89% [3]
81.14% [0]
53.43% [4]
Adapter (heviiso
+
0.87%
54 54.90%
75. 19% [1]
38.98%
1.12%
74 19%
K2.66% (1
47.69% [2]
VPTS
60335 [1]
27.50%
17.349%
0..22%
GPT
0,055
47.61%
[1]
36
0.069
74.84%
33.388
[1]
10%
Owes
0.38%
53.591 [6] [6]
20.2516
[1]
36.15%
(6)
0.22%
77.47% [5] [2]
85.76% [31 (4)
58.74% [6]
80
X Plovest02
X Pas
$3 SVHN
X BunSAT
Cametron
Pacc
* Retinopaity
88
dSptiia
ocatzation
88 clevazoz
SmaiNORBI
83
SmalNoOs
(a) Nahural 4FID: 156:39>
(b) Sperialized 4FID: 245
Structured -ID: 231.6
99.3-
60.0
97.9-
90.3
96.3
80
88
91.3-
Aocurayy (%)
87.2
83.3
83
83
02.0
8
75.4
30,0 83
76.4
68.0
0.5
1.0
0.0
0.7
1.0
Fourier Percentage (%)
Figure 2: Image classification accuracy of various Fourier percentages of VTAB-1k [78] for ViT-Base/16 [23]. For better illustration, we randomly select 3 datasets each group of VTAB-IK
The "Average FID Score of Each Group" is reported in <>>. Our conclusion aligns with 16 of 19 cases. The cross framed by the square indicates the best percentage for each downstream task. Those datasets with only three Fourier percentage reports due to the prompt length limits.
the ImageNet-21k supervised pretrained Swin-Base 24 reported in Table 2 It can be seen that VFPT consistently outperforms all the other parameter-effie efficient methods on three VTAB-IK 1k groups VFPT on Different Pretraining Objectives. In Table 3, report the experimental results two self-supervised objectives: MAE 90) and MoCo v3 [26]. While VPT yields inconclusive results, VFPT has the highest "Number of Wins" compared to full fine-tuning among PEFT methods (i.e., of 19 instances under MAE, and 14 of 19 instances under MoCo v3, respectively). Our method also outperforms VPT by large margin (e-g., 53.59 %
36.02% under MAE on VTAB-1K Natural).
Fourier Contribution. We conducted experiments to understand the impact of Fourier components by varying the percentages of Fourier prompts in VFPT. As shown in Fig. 2 we observed distinct preferences across the VTAB-IK benchmark, which comprises three groups with varying data disparities (see $4.1). Specifically, the Natural group, which has data distribution similar to the pretrained task (low disparity), shows peak performance when half the visual prompts are transformed into Fourier prompts, indicated by the accuracy curves in Fig. 2[a). This suggests that transfer learning is less challenging in this group. Conversely, for the Specialized and Structured groups, which have data distributions significantly different from the pretrained task (high disparity), the accuracy curves Fig. (b-c) demonstrate that higher classification performance is achieved with increased percentage of Fourier components. These observations consistent with our expectations, demonstrating the effectiveness of Fourier prompts in VFPT, especially for tasks with large data disparities. In other words, our approach be viewed generalization of VPT, where the Fourier components learn effective representations from the frequency domain that complement the knowledge from the spatial domain.
4.3 Study of Optimization
In this section, we investigate why VFPT achieves better performance and generalization across various tasks from an optimization perspective. Previous works [102) demonstrate that land- scape geometry significantly impacts model generalization, visualize the loss landscape to
7

---

understand the enhanced generality of VFPT. Specifically, in Fig. aa), we randomly select two parameter directions for the study, as ran- domness in directions does not significantly af- fect the results [102]. There are two key obser- vations supporting the enhanced generality of VFPT. i) Flatness: VFPT provides larger con- nected region around the local minimum [033 (e.g., in the yellow square, where the larger blue area VFPT offers more optimization choices) and smoother edge of the loss land- scape for mitigating chaotic landscapes (e.g., in the green square, where the humpy contour in
66
54
VPPT
Ba
52
(4) Loes Lambbogpe
(h) Ratio Mep of Hessia
Figure 3:
Visualization of loss landscape ratio map of Hessian [102].
102 and the
VPT sensitive to loss variations, resulting in worse generality). This indicates that VFPT achieves flatter minimizer, which consistently correlates with lower test еrrоr [102]. ii) Convexity: As eigenvalues of the Hessian directly assess the convexity of loss function [102), we compute both the maximum and minimum eigenvalues of the Hessian and map their ratios 102 As shown in Fig. b), higher prevalence of near-zero negative eigenvalues (in deep blue) in VFPT suggests the presence of more convex regions (25.0% vs. 20.0%) for model optimization. This finding indicates that the incorporation of the Fourier transform in visual prompt tuning effectively mitigates the sharpness of the loss landscape.
4.4 Study of Interpretability
To best our knowledge, research on the understanding of prompt tuning remains rare [. Consequently, our research seeks to both quantitatively and qualitatively exam- ine the impact Fourier components on the enhancement of visual prompt tuning. For fair- ness, instead of using enhanced visualization methods [OS 106 104 that may alter the orig- inal expression of the learnable prompts, we visualise and examine the average attention head on the last layer of VPT and VFPT.
VPT
8.20
100
286
8.12
206
7:36
Significant attention distribution in learn- able prompts. Observations from both VPT and VFPT in Fig. 4a) reveal common phe- nomenon: there exists a pronounced accumula- tion of attention scores at learnable prompt loca- tions (i.e., narrow color area on the left side of 2D attention map), indicating that these prompts have a substantial impact on the frozen embed- dings during the finetuning stage.
160
150
3D Attention Map
2D Attention Map
Attention
Input
VPT (b) GraCCAM Heatmap
VEPT
Global attention scores pattern in Fourier prompts. We further observe notably higher concentration in global attention scores when integrating visual Fourier prompts. Specifically, the global attention scores indicate that VFPT also establishes robust correlations within the Transformer's input space [4] (see Fig. 4Ta) (a)). In contrast, VPT lacks this correlation, suggesting that it does not adequately consider or integrate extensive information from the frozen backbone. Moreover, we find positive relationship between strong associations and performance gains quantitatively (see $4.2) and qualitatively (see Fig. 4b)) in VFPT, suggesting that the integration of visual Fourier prompts encourage clear foreground (i.e., tree with high frequency component) background (i.e., sky with low frequency component) separation.
Figure 4: Study of interpretability. (a) The 3D and 2D attention map in VPT and VFPT on ran- domly selected sample. The colors M. and indi- cate class, prompt and patch tokens, respectively. (b) Corresponding GradCAM [044 maps. Note that red regions correspond to high score for the class. We present more visualization results $54

---

Table 5: set of ablative studies on VTAB-Ik [78] Natural and Specialized benchmarks in three runs. "Prompt Location" is the placement of the visual Fourier prompts relative to original visual prompts. "Prompt Depth" indicates the layer we use visual Fourier prompts. "Transform Type" is the method we use to transform prompts and input images. FFourierTransfonn Dimension" indicates the dimension we apply using specific transform method. Per task results are available Appendix. Same for Table
Fourier Dimension
VTAB-IK 78 [19]
Prompt Location
VTAB-IK 778 [19]
VIAB [8)
Sequence Hidden Natural [7] Specialied [4]
Natural [7] Specialized [4]
Depth
[7] Specialized
80.88%
83.57%
A
81.02%
83,80%
135799 79
80.48%
83.73%
80.7996
54.34%
80.74%
83.87%
R
78.62%
82.47%
7-12
80.835
83.93%
81.35%
84,93%
P
81.35%
84.93%
1:12
$1.35%
84.93%
(a) Fourier Prompt Dimension
(b) Fourier Prompt Location
(c) Fourier Prompt Depth
In summary, our findings provide sigmificant insights into the interpretability of prompt tuning, revealing that for both VPT and VFPT, considerable portion of attention is directed towards the learnable prompts. Further, VFPT exhibit enhanced global feature learning capabilities compared to VPT by interfacing effectively with frozen embeddings, thereby enabling precise capture of distinctive features across diverse downstream tasks. This observation corroborates our findings in $4.2
4.5
Ablation Study
We ablate VFPT's key components on VTAB-IK 788 Natural and Specialized.
Table 4: Ablative studies of transform type on VTAB- 1K (78) Natural and Specialized benchmarks in three runs. Per-task results are available in Appendix.
More studies are provided in $S2.5]
Transform Type. We ablate other trans- form method instead to certify the impact of Fourier transform in Table 4 where the Fixed Linear Layer (i.e., FLL) and the Learnable Linear Layer (i.e.. L.L.) are con-
Transform Type
Transform Dimension
VIAB.IK [88 (19)
(Domain)
Sequence
Hidden
Natural (7l Specialized [4]
(L)
80.98%
84.02%
LLL(S)
80.54%
82.64%
(7) + (J)
00
80,909
84.0%%
(F)
81.35%
84.93%
sidered. Compared with FFT, a fixed non non-parameter Fourier domain transform in sequence and hidden dimension, the FLL operation considers only fixed spatial domain transform in hidden dimension; the LLL further unfixes the transformation to enable gradient updates. As seen, both FLL and LLL show inferior performance to FFT. We further consider the impact of current Fourier domain adaption approach 771, which maps source image to a target "style" without altering semantic content. However, no significant improvement can be observed.
Fourier Prompt Dimension. A fundamental distinction between VFPT and other methods is the incorporation FFT into visual prompts. In our standard implementation, we utilize 2D FFTs across both sequence length and hidden dimensions. Here, we explore the impact of each dimension's s transformation individually. As shown in Table 5ia), the separate Fourier transformations along each dimension appear have similar contributions (i.e., 80.88% vs. 80.74% in Natural). However, the combined application of transformations across both dimensions (i.e., 2D FFTs) demonstrates a synergistic effect, yielding significant improvement in performance.
Fourier Prompt Location. In 5(ble three prompt locations are considered for VFPT, which are "Prepend" (i.e., P). "Append" (i.e., A), and "Random" (i.e., R). Specifically, P and A prepend visual Fourier prompts before or after visual prompts, and R. randomly selects the position for visual Fourier prompts in each layer. As seen, both P and A show competitive results, validating the robustness of VFPT w.r.f. prompt locations. In alignment with the findings in [5] 4. we choose P as our baseline method in all experiments since it reaches superior results (i.e., 81.35% us 81.02% in Natural).
Fourier Prompt Depth. Table 5(c) presents the performance of VFPT based on the specific layer at which visual Fourier prompts are employed. The results suggest that employment on separate layers also yields a accuracy improvement compared with VPT. Further application of visual Fourier prompts across all layers fosters the best overall performance.
Conclusion
We present Visual Fourier Prompt Tuning (VFPT), a simple yet powerful parameter-efficient vi- sual prompt tuning approach that draws insights from human visual cognition. It has merits in: i) integrating spatial and frequency domain information through an intuitive yet effective design; ii) demonstrating generality across datasets with varying disparities while ensuring powerful perfor- mance; and iii) thoroughly investigating the associations between learnable prompts and frozen embeddings to elucidate this generality. As whole, we conclude that the outcomes elucidated in this paper impart essential understandings and necessitate further exploration within this realm.

---

Acknowledgements
This research was supported by the National Science Foundation under Grant No. 2242243.
References
[]] William Thomson and Peter Guthrie Tait. Treatise on Natural Philosophy. Claredon Press, 1867.
[2] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[3] Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang. Xiaojun Quan, Zenglin Xu, and Dongfang Liu. Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models. In EMNLP. 2023.
[4] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[5] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E2vpt: An effective and efficient approach for visual prompt tuning. ICCV, 2023.
[6] Wenhao Xu, Rongtao Xu, Changwei Wang, Shibiao Xu, Li Guo, Man Zhang, and Xiaopeng Zhang. Spectral prompt tuning: Unveiling unseen classes for zero-shot semantic segmentation. In AAAI, 2024.
[7] Muzhi Zhu, Hengtao Li, Hao Chen, Chengxiang Fan, Weian Mao, Chenchen Jing, Yifan Liu, and Chunhua Shen. Segprompt: Boosting open-world segmentation via category-level prompt learning. In ICCV, 2023.
Xing Nie, Bolin Ni, Jianlong Chang, Gaofeng Meng, Chunlei Huo, Shiming Xiang, and Qi Tian. Pro-tuning: Unified prompt tuning for vision tasks. IEEE TCSVT, 2023.
[9] Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang. Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? In ICLR, 2024.
[10] Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parfmeter-efficient fine-tuning. arXiv preprint arXiv:2306.07966 2023.
[11] Lin Zhao, Lu Zhang, Zihao Wu, Yuzhong Chen, Haixing Dai, Xiaowei Yu, Zhengliang Liu, Tuo Zhang, Xintao Hu, Xi Jiang, al. When brain-inspired ai meets agi. Radiolaa page 100005, 2023.
[12] Demis Hassabis. Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245-258, 2017.
[13] Hadi Salehi and Rigoberto Burgueño. Emerging artificial intelligence methods in structural engineering. Engineering Structures, 171: 171:170-189, 2018.
[14] Ben Shneiderman. Human-centered Al. Oxford University Press, 2022.
[15] Aaron Quigley and Peter Eades. Fade: Graph drawing, clustering, and visual abstraction. In ISGD, 2000.
[16] Ivan Viola, Min Chen, and Tobias Isenberg. Visual abstraction. Foundations of Data Visualiza- tion, pages 15-37, 2020.
[17] Christopher P Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv:1901.11332 2019.
[18] Ronald I. Allen and Dunean Mills, Signal analysis: time, frequency, scale, and structure. John Wiley & Sons, 2004.
10

---

[19] VU Reddy. On fast fourier transform: a popular tool for spectrum analysis. Resonance, 3(10):79-5) 88, 1998.
[20] Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution, In NeurlPS, 2020.
[21] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
[22] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi- scale vision transformer for image classification. In ICCV,, 2021.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[25] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[26] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
[27] Menglin Jia, Zuxuan Wu, Austin Reiter, Claire Cardie, Serge Belongie, and Ser-Nam Lim. Exploring visual engagement signals for representation learning. In ICCV, 2021.
[28] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. ECCV, 2018
[29] Shibo Jie, Haoqing Wang, and Zhi- Zhi-Hong Deng. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. In ICCV, 2023.
[30] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. NewrIPS, 2017.
[31] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side- tuning: baseline for network adaptation via additive side networks. In ECCV, 2020.
[32] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In NeurIPS, 2020.
[33] Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter- efficient model adaptation for vision transformers. In AAAI, 2023.
[34] Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation vision transformer. In AAAI, 2023.
[35] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0904
2021.
[36] Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. In ICML, 2024.
[37] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting vasual-language models for efficient video understanding. In ECCV, 2022.
[38] Shaohua Dong, Yunhe Feng, Qing Yang. Yan Huang. Dongfang Liu, and Heng Fan. Efficient multimodal semantic segmentation via dual-prompt learning. arXiv preprint 122.002312.000600
2023.
11

---

[39] Liqi Yan, Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. Prompt learns prompt: exploring knowledge-award aware generative prompt collaboration for video captioning. In IJCAI, 2023.
[40] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225,, 2022.
[41] Taowen Wang, Yiyang Liu, James Chenhao Liang, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng. Zenglin Xu, Cheng Han, et al. Mmpt: Multimodal prompt tuning for zero-shot instruction learning. In EMNLP, 2024.
[42] Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. Xprompt: Exploring the extreme of prompt tuning. In EMNLP, 2022.
[43] Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, et al. Hyperprompt: Prompt-based cuskicondiiininn of transformers. ICML, 2022.
[44] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 55(9):1-35, 2023.
[45] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre- trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):187--1897 2020.
[46] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In CVPR, 2023.
[47] Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, Han Zhang, Yuan Hao, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. In CVPR, 2023.
[48] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. Al Open, 5:30-38, 2024.
[49] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2024.
[50] Henri Nussbaumer and Henri Nusshaumer. The fast Fourier transform. Springer, 1982.
[51] Luis Almeida. The fractional fourier transform and frequeeyy representations. IEEE TSP, 1994.
[52] Isa Servan Uzun, Abbes Amira, and Ahmed Bouridane. Fpga implementations of fast fourier transforms for real-time signal and image processing. IEE Proceedings- Vision, Image and Signal Processing, 152(3):283-296 2005.
[53] Todd A Ell and Stephen Sangwine. Hypercomplex fourier transforms of color images. JEEE TIP, 2006.
[54] Pierre Duhamel and Martin Vetterli. Fast fourier transforms; tutorial review and state of the art. Signal Processing, 1990.
[55] David Brandwood. Fourier transforms in radar and signal processing. Artech House, 2012.
[56] M Sifuzzaman, M Rafiq Islam, and Mostafa Z Ali. Application of wavelet transform and its advantages compared to Fourier transform. Vidyasagar University, Midnapore, West-Bengal, India, 2009.
[57] Jia Xu, Ji Yu. Ying-Ning Peng, and Xiang- Gen Xia. Radon-fourier transform for radar target detection, i: Generalized doppler filter bank. IEEE TAES, 2011.
[58] Tim OF Conrad, Martin Genzel, Nada Cvetkovie, Niklas Wulkow, Alexander Leichtle, Jan Vybiral, Gitta Kutyniok, and Christof Schütte. Sparse proteomics analysis-a compressed sensing-based approach for feature selection and classification of high-dimensional proteomics mass spectrometry data. BMC Bioinformatics, 18:1-20, 2017.
12

---

[59] Niklas Mevenkamp and Benjamin Berkels. Variational multi- segmentation using high- dimensional local features. In WACV, 2016.
[60] A Oppenheim, Jae Lim, Gary Kopec, and SC Pohlig. Phase in speech and pictures. In ICASSP, 1979.
[61] Alan V Oppenheim and Jae S Lim. The importance of phase in signals. Proceedings of the IEEE, 6955),5,-,51, 1981.
[62] Leon N Piotrowski and Fergus W Campbell. demonstration of the visual importance and flexibility of ppatial-frequenge amplitude and phase. Perception, 11(3):337-345 1982.
[63] Bruce C Hansen and Robert Hess. Structural sparseness and spatial phase alignment in natural scenes. JOSA A, 24(7):1873-1885, 2007.
[64] Nikunj Raghuvanshi, Rahul Narain, and Ming C Lin. Efficient and accurate sound propagation using adaptive rectangular decomposition. IEEE TVCG, 15(5):789-8000 -011 2009.
[65]
Tim-Oliver Buchholz and Florian Jug. Fourier image transformer. CVPR, 2022.
[66] Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourier- former: Transformer meets generalized fourier integral theorem. In NeurIPS, 2022.
[67] Dario Fuoli, Luc Van Gool, and Radu Timofte. Fourier space losses for efficient perceptual image super-resolution. In ICCV, 2021.
[68] B Hinman, Jared Bernstein, and D) Staelin. Short-space fourier transform image processing. In ICASSP, 1984.
[69] Normand Beaudoin and Steven S Beauchemin. An accurate discrete fourier transform for image processing. In ICPR, 2002.
[70] Rama Chellappa and Bagdazian. Fourier coding of image boundaries. (1):102-105, 1984.
IEEE TPAMI,
[71] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, 2020.
[72] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based frame- work for domain generalization. In CVPR, 2021.
[73] Harry Pratt, Bryan Williams, Frans Coenen, and Yalin Zheng. Fenn: Fourier convolutional neural networks. In ECML PKDD, 2017.
[74] Tianyi Chu, Jiafu Chen, Jiakai Sun, Shuobin Lian, Zhizhong Wang, Zhiwen Zuo, Lei Zhao, Wei Xing, and Dongming Lu. Rethinking fast fourier convolution in image inpainting. In ICCV, 2023.
[75] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. In NeurlPS, 2021.
[76] John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv22111.13887 2021.
[77] Ruiping Liu, Jiaming Zhang, Kunyu Peng. Yufan Chen, Ke Cao, Junwei Zheng, M Saquib Sarfraz, Kailun Yang, and Rainer Stiefelhagen. Fourier prompt tuning for modality-incomplet scene segmentation. arXiv preprint arXiv:2401.16923 2024.
[78] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04866 2019.
13

---

[79] James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of Computation. 19(90):297330, 1965.
[80] James Lee-Thorp. Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprini arXiv:2105.03522 2021.
[81] Wenjie Pei, Tongqi Xia, Fanglin Chen, Jinsong Li, Jiandong Tian, and Guangming Lu. Sa²vp: Spatially aligned-and aad---ddddddpdd visual prompt. In AAAI, 2024.
[82] Chongyi Li, Chun-Le Guo, Man Zhou, Zhexin Liang, Shangchen Zhou, Ruicheng Feng, and Chen Change Loy. Embedding fourier for ultra-high-defiiiioon low-light image enhancement. arXiv preprint arXiv:2302.11831 2023.
[83] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey on vision transformer. IEEE TPAMI, 2022.
[84] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[85] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. California Institute of Technology, 2011.
[86] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grate -ra-irai dataset collection. In CVPR, 2015.
[87] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
[88] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In CVPR Workshop. 2011.
[89] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- grained car detection for visual census estimation. In AAAI, 2017.
[90] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[91] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[92] Eugenia Iofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well do sparse imagenet models transfer? In CVPR, 2022.
[93] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurIPS, 2014.
[94] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. In CVPR, 2020.
[95] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022.
[96] Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. In NeurlPS, 2024.
[97] Rajshekhar Das, Yonatan Dukler, Avinash Ravichandran, and Ashwin Swaminathan. Learning expressive prompting with residuals for vision transformers, In CVPR, 2023.
14

---

[98] Yi Ren, Shangmin Guo, Wonho Bae, and Danica J Sutherland. How to prepare your task head for finetuning. In ICLR, 2023.
[99] Min Jin Chong and David Forsyth. Effectively unbiased and inception score and where to find them. In CVPR, 2020.
[100] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr', echet inception distance. In ICLR, 2023.
[101] Seungryong Yoo, Funji Kim, Dahuin Jung, Jungheom Lee, and Sungroh Yoon. Improving visual prompt tuning for self-supervised vision transformers. ICML, 2023.
[102] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NeurlPS, 2018.
[103] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Compulation, 9(1):1-42, 1997.
[104] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. ICCV, 2017.
[105] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.0009 2020.
[106] Cheonbok Park, Inyoup Na, Yongjang Jo, Sungbok Shin, Jachyo Yoo, Bum Chul Kwon, Jian Zhao, Hyungjong Noh, Yeonsoo Lee, and Jaegul Choo. Sanvis: Visual analytics for understanding self-attention networks. In IEEE VIS, 2019.
[107] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2018.
[108] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 1 2021.
[109] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, In NeurIPS, 2019.
15

---

SUMMARY OF THE APPENDIX
This appendix contains additional experimental results and discussions of our NeurIPS 2024 submis- sion: Visual Fourier Prompt Tuning, organized as follows:
S provides per-task results on VTAB-1k and FGVC image classification benchmarks with
confidence analysis, where the overall results have been provided in the main paper.
S2
provides per-task results on ablation study, where the overall results have been provided in
the main paper. Further study of sensitivity of Fourier prompt percentages and prompt lengths is
included in $52.5)
provides per-task results on Fourier percentage, where partial results have been provided in the main paper.
S4
presents more details and results of visualization of attention maps.
S4
presents more details and results of visualization of loss landscapes.
discusses our potential extension to language fasks.
S7
further analyze the complexity of our approach.
S8
shows related asset license and consent to our work.
$9
claims reproducibility of our approach.
$10
discusses the social impact of our research.
S11
adds more discussions, and points out potential directions of our future work.
S1
Per-task Results on VTAB-1k and FGVC
S1.1
Per-task Results on ViT-Base
To provide comprehensive results from the paper, we report the average per-task test accuracy (i.e., runs, 24 tasks) on VTAB- 1k (78) Natural, Specialized and Structured, respectively (see Table S1 S2 and S3 We also report per-task FGVC [4] results (5 tasks) in Table S4 VPT-SHALLOW 4 is
also included for completeness (i.e., VPT-SHALLOW only introduces 1-st layer visual prompts). In
conclusion, VFPT shows consistently better performance in various downstream tasks.
Table $1: VTAB-1k [78] Natural per-task results for ViT-Base/16 (3 pretrained on supervised Imtte-Net-21k. Consistent to our paper, "Number of Wins" in [-] compared to full fine-tuning [92] "Tuned/Total" the percentage of tuned parameters in each task, along with the average results of those percentages in each group. The highest accuracy among all approaches except FULL are shown bold. i denotes method using soft filtered prompts to reduce the parameter usage learnable visual prompts, requiring specialized devices to facilitate acceleration. All results are averaged in three runs with different initialization seeds. Same for Table S2T We also report standard deviation error bars for our main results (Table S1
S2 S3 and S4) by calculating each task respectively and
averaging across them. Other tables show similar trends on standard deviation error bars.
WIT-Base766
VIAB-L Ik 78 Natural [7]
(85.8M)
CIFAR-100
Caltech101
DTD
Flowers 102
Pats
SVHN
Sun397
Mean
FULL
68.9
$7.7
61.3
97.2
56.9
$7.1
38.8
75.38
LINEAR [
63.4
85.0
63.2
97.0
86.3
36.6
51.0
68.93 [1]
PARTIAL
105
66.8
85.9
62.5
97.3
83.5
37.6
50.6
69.44 [2]
04
97.6
85.9
34.1
47.8
67.70
97.4
49.2
[2]
MLP.5
04)
50.3
84.4
50.9
96.1
$4,4
30.9
46.8
65.98 [1]
MLP-9
51
53.1
80.5
53.9
95.1
82.6
21.1
43.7
61.90 [1]
SIDETUNE 31
61.7
34.9
33.3
58.21
30
ADAPTER-250
74,1
86.1
63.2
97.7
87.0
34.6
50.8
70.50 [4]
ADAPTER-61 [X]
74,2
85.8
62.7
97.6
87.2
36,1
$0.9
70.65 [4]
ADAPTER-K 1:
74.2
85.7
62.7
97.8
87.2
36.4
50.7
70 [4]
VPEEHALLKIW (%)
0.08
0.19
0.36
VPT-DEEP
78.8
90.8
65.8
98.0
$8.3
78.1
49.6
78.48 (6]
Tumed/ Total (%)
0.20
0.20
0.15
0.10
0.04
0.54
0.41
0.23
E2VPT
76.6
$9.1
67.8
98.2
88.5
85.3
$2,3
80.01 [6]
Tomal (8)
OURS
80.7
(0.13)
91.4
1: (0.11) 69.4
(0.27)
99.3
1 (0,05)
90.3 1: (0.29)
$5.6
(0.95)
$2.7
(0.47) 81.35
(0.33)
Tumed/ / Total (9)
0.20
0.31
0.20
0.11
0.06
0.12
0.41
0.21
Fourier Percentage
70.0
50.0
30.0
50.0
50.0
20.0
50.0
45.7
16

---

Table S2: VTAB-IK 78 Specialized per-task results for ViT-Base/16 1 23 ImageNet-2IK.
pretrained on supervised
Specialized (4)
HM)
Camelyon
EurSATT
Resiso45
Retinoputity
81.2
LINEAR
78.5
87.5
74.0
77.16 [1]
PARTIAL-1 01
78.6
89.8
72.5
73.3
78.55 (0]
MLP-2 04
74.3
58.8
62.1
73.2
75.86 [0]
MLP33
77.0
38.0
70.2
56.1
72.83
MLP-9
78.5
33.0
60.2
72.3
73,49 [0]
SIDEUNE 31
58.5
$7.7
65.2
61.3
68.12 [0]
BIAS 20
78.7
91.6
72.9
69.8
78:25
ADAPTER-256
ADAPTER-64
76.3
87.5
73.7
70.9
10 [0]
ADAPTER-8
76.9
89.2
73.5
71.5
77 80 (0]
VPESHALLOW
75.2
9210
75.6
72.9
79.66
VPT-DERP
[3]
Taned Total (96
1.06
1.07
0.15
0,02
0.57
i E2YPT 5
82.5
96.8
84.8
73.6
84.43 [31
Tuned/ Total (%)
0,20
0.29
0,12
0.07
0,17
OURS
(0.09)
(0.06)
(0.14)44
Tined Total (R)
1.06
0.12
0.11
0.33
Fourier Percentage (%)
100.0
30.0
100.0
100.0
82.5
Table S3: VTAB-1k 778 Structured per-task results for ViT-Base/16 23 pretrained supervised ImageNet-21k.
VIE-Base/IG
VTAE-IK 28 KITT//
Sienttored [)
Clevel count
Cleval distance
dSprites/ location
dSprites/ orientation
SmalNNOR: SmalINORR/
Micin
HM)
DMLab
distance
aziouth
elevation
FULL 1
56.3
38.5
41.7
63.3
57.5
26.7
25.7
25.1
47.64
LINFAR [3
34.3
30.6
33.2
55.4
12.5
20.0
9.6
19.2
26.84 (0)
PARITAL-1
05
41.5
34.3
33,9
61.0
51.3
32.8
16.3
22.4
34.17
MLP-2 2
M1
45.2
31,6
31,8
55.7
30.9
24.6
23.3
32.47
MLP3
47.8
30.62
MI
SIDETUNE
31
27.6
22.5
21,3
$1.7
11A
9X
21,8
25.41
BIAS
61.5
55.6
32.4
$5.0
56.6
40.0
15.7
25.1
44099 [2]
ADAPTER-155 2
12]
45.7
37.4
31.2
53.2
30.3
25.4
13.8
22.1
32.39 [0]
ADAPTER-64
53
4299
39.9
30.4
54.5
31.9
25.6
13.5
21.4
32.51
ADAPTER& 52
VITSSALLOOW
0.09
VPT-DEEP
68.5
60.0
46.5
72.8
73,6
47.9
$4.98
Duncd Total (%)
0.54
2.11
1.07
0.54
0.32
01.55
2.12
2.11
1.14
E2VPT [S]
71.7
61.2
47.9
75.8
80.8
48.1
31.7
41.9
$7.39 (8)
Tuned Total (5))
01.34
0.65
0.44
0.36
0.10
11.38
1.14
0.66
0.51
OURS
Tined/ Total
0.54
2.11
0.11
0.71
Percentage
Table S4: FGVC 4 per-task results for ViT-Base/16 23
pretrained on supervised ImageNet-21k.
ViT-Baee11b 23
FOVC [|1
CUB-200-2011
NAbirds
Oxford Flowers
Stanford Dogs
Stanford
Mean
LINEAR 9X
85.3
75.9
97.9
86.2
51.3
79.32 [0]
PARTIAL-1 93
85.6
77.8
98.2
85.5
66.2
82.63
MLP33
30
85.1
77.3
97.9
$4,9
53.8
79.80 [0]
MI.S.
84.2
84.8
50.2
78.71
MLP-9
83.2
76.0
96.2
83.7
47.6
77.31 [0]
SIDETUNE 31
84.7
75.8
96.9
85.8
48.6
78.35
ADAPTER 256
52
87.2
84,3
98.5
89,9
68.6
85.70 121
ADAPTER. 64
32
87.1
84.3
98.5
89.8
68.6
85.67
ADAPTER-S 32
87.3
84.3
98.4
88.8
68.4
85.46
VPT-SHALLOW 4
86.7
78.8
98.4
90.7
68.7
84.62
Tuned/ Total (16)
VPT-DEEP 4
88.5
84,2
99.0
90.2
83.6
89.11 [4]
Tuned/ Total
1.17
0.98
E2VPT S
89.1
84.6
99.1
90.5
82.8
89.22 (4)
Tuned/ Total (%)
0.32
0.65
0.15
0.88
1.27
0.65
Ours
(0.02)
(0.01)
(0.01)
90.4 (0.13)
(0.04)
(0.04)
- Tuned/ / Total (%)
0.29
1.02
0.15
1.17
2,27
0.98
Foarier Percentage (%)
30.0
$0.0
36.0
17

---

S1.2 Per-fask Results on Swin-Base
Table S5: VTAB-1k (78) Natural per-task results for Swin-Base [24
pretrained supervised
ImageNet-21k. Specially, the highest accuracy is shown bold. Same for Table SS6 and S7
Swin Base 24
VTAB-I Ik 788 Natural (?)
(86.7M)
CIFAR-I01 100
Caltech101
DTD
Flowers102
Pets
SVHN
Sun397
FULL 992
72.2
58.0
71.2
98.3
89.5
89.4
45.0
79.10
VPT-SHALLOW
77.7
86.9
62.6
97.5
87.3
74.5
51.2
76.81 [4]
0.27
0.17
VPT-DEEP 4
79.6
90.8
78.0
99.5
91.4
46.4
51.7
78.78 [6]
Tuned/ Total (%)
0.13
0.13
0.07
0.13
0.06
0.70
0.48
0.28
122PT 5
82.9
92.4
78.5
99.6
82.2
83.31 [6]
Tuned/ / Total (%)
0.27
0.15
0.08
0.15
0.07
0.44
0.49
0.24
OURS
83.9
93.0
77.9
99.6
91.4
89.5
56.4
84.53 [7]
TTnedd/ Total
0.15
0.13
0.15
0.07
0,70
0.26
Fourier Percentage (%)
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Table S6: VTAB-1k (78) Specialized per-task results for Swin-Base [24 pretrained supervised ImageNet-21k.
Swin- Base 24
VTAB-IK 78
Spectalized [4]
(86.7M)
Patch Camelyon EuroSAT
Resise45 Retinopathy
FULL 192
86.6
96.9
87.7
73.6
86.21
VPT-SHALLOW 4
78.2
92.0
75.6
72.9
79.66 [0]
Tuned Total (%)
0.01
0.05
0.01
0.04
VPT-DEEP 4
80.1
96.2
85.0
72.0
83.33
Tuned/ / Total (%)
0.07
0.13
0.19
0.02
0.10
E2VPT 5
83.8
97.2
84.95 [2]
Tuned Total
0.09
0.04
0.20
0.03
0.09
OURS
86.3
97.3
86.9
74.1
86,15 [2]
Tuned/ Total (%)
0.07
0.15
0.19
0.03
0.11
Fourier Percentage (%)
100.0
100.0
100.0
87.5
Table S7: VTAB-Ik 78 Structured per-task results for Swin-Base
24 pretrained on supervised
ImageNet-21k.
Swin-Base 24
VTAB-Ik (78) Structured
Clevr/ count
Cleve! distance
KITTI/ dSprites/ location
dSprites/ orientation
SmalINORB/
SmalINORB/ elevation
Mean
(86.7M)
DMLab
distance
azimuth
FULL 92
75.7
59.8
54.6
78.6
79.4
53.6
34.6
40.9
59.65
VPT-SHALLOW 4
50.5
58.6
40.3
67.1
68.7
36.1
20.2
34.1
46. [4]
Tuned Total (%)
0.10
0.18
0.09
0.09
0.10
0.10
0.19
0.19
0.13
VPT-DEEP
67.6
59.4
74.4
50.6
25.7
25.7
51.85 [0]
Tuned/ Total (%)
0.70
0.70
0.14
0.69
0.15
0.09
0.16
0.02
0.38
E2VPT
74.0
61.2
49.5
80.3
50.7
27.9
57.35 [3]
Tuned/ Total
0.70
0.43
0.14
0.51
0.17
0.17
0.16
0.04
0.29
OURS
74.9
61.5
50.0
80.5
82.7
50.6
29.9
35.6
58.21 [3]
Tuned/ Total (9)
0.70
0.70
0.15
0.92
0.16
0.09
0.16
0.04
0.36
Fourier Percentuge (%)
100.0
50.0
100.0
100.0
100.0
100.0
87.5
S1.3 Per-task Results on MAE and MoCo v3
Table S8: VTAB-Ik (78 Natural per-task results for ViT-Base/16 (23 pretrained on MAE [90]
Since VPT [4] have considerably lower performance, do not list the per-task results for simplicity. We instead compare our method to full fine-tuning, and the highest accuracy is shown in bold. We
post the "Number of Wins" in [ to full fine-tuning (FULL) (92. Same for Table S9 S13
ViT-Buse/16 23
VTAB-IK [8] Natural [7]
(85.8M)
CIFAR-100
Caltech101
DTT Flowers 102
Pets
SVHN
Sun397
Mean
FULL 92
24.6
84.2
56.9
72.7
74.4
86.6
15.8
59.31
OURS
36.0
58.0
74.3
76.3
19.6
23.3
53.59 [G]
Tuned/ / Total (%)
0.13
0.11
0.06
0.11
0.06
1.07
0.38
0.28
Fourier Percentage (%-)
100.0
50.0
50.0
100.0
50.0
100.0
100.0
75.6
18

---

Table S9: VTAB-Ik 78 Specialized per-task results for ViT-Base/16 1 23
pretrained on MAE 900
ViT Base/16 23
VTAB-Ik 28 Specialized
(85 8M)
Patch Camelyon
EuroSAT
Resisc45
Retinopathy
FULL 22
81.8
94,0
72.3
70.6
79.68
OURS
91.3
69.2
73.6
77.75 [1]
Tuned/ / Total (5)
0.06
0.03
0.13
0.54
0.17
Fourier Percentage (5i)
50.0
100.0
50.0
50.0
62.5
Table S10: VTAB-1k 78
Streutured per-task results for ViT-Base/16 2
pretrained on MAE 990
ViT-Base/ 16 23
VIAB-IK Ik (8) Structured [8]
Cleve! count
Cleve' distance
KITTV
dSprites/
dSprites/ orientation
SmalINORB/
SmalINORB/ elevation
Mean
(85.8M)
DMLab
distance
location
azimuth
FULL 92
67.0
59.8
45.2
75.3
72.5
47.5
30.2
53.82
OURS
47.6
45.3
40.7
80.7
13.7
34.6
9.3
17.3
36.15 [1]
Tuned/ Total (%)
0.03
2.11
0.03
0.20
2.12
0.04
0.04
0.12
0.58
Fourier Percentage (%)
50.0
Table S11: VTAB-Ik 78 Natural per-task results for ViT-Base/16 23
pretrained MOCO
26
ViT Base/16 (85.8M)
VTAB-I 78 Natural [7]
Mcan
CIFAR-100 Caltech101
DTD
Flowers102
SVHN
Sun397
FULII 92 OURS
57.6
91.0
64.6
91.6
79.9
89.8
29.1
71.95
90.5
70.5
92.4
88.3
84.7
42.3
77.47 [S]
Tuned/ / Total (%)
0.20
1.15
0.06
0.11
0.14
0.06
0.46
0.31
Fourier Percentage (%)
50.0
100.0
50.0
50.0
100.0
100.0
50.0
71.4
Table S12: VTAB-1k 78 Specialized per-task results for ViT-Base/16 23
pretrained on
MOCO [26].
ViT Base/16 23
VTAB Ik [8] Specialized
(85.8M)
Patch Camelyon
EuroSAT Resisc45 Retinopathy
PULL 92
85.1
96.4
83.1
14.2
84.72
OURS
86.7
95.7
85.2
75.5
85.76 [3]
Tuned Total (S)
0.11
0.03
0.15
0.06
0.09
Fourier Percentage (5)
100.0
100.0
50.0
50.0
75.0
Table $13:
VTAB-1k
Structured per-task results for ViT-Base/16 23
pretrained on
MOCO 26
(88 Structured dSprites/ location
(85.8M)
Cleve! count
Clevel distance
DMLab
KITTI/
dSprites/
SmalINORB/
SmalINORB/ elevation
Mean
distance
orientation
azimuth
FULL 92
55.2
56.9
44.6
77.9
63.8
49.0
31.5
36.9
$1.98
OURS
76.3
63.0
46.1
82.2
85.3
47.4
23.8
45.8
58.74 [6]
Tuned/ Total
1.07
0.12
0.07
0.07
0.06
0.22
Fourier Percentage (%)
50.0
50.0
50.0
50.03
50.0
50.0
100.0
50.0
56.3
19

---

S2 Per-task Results on Ablation Study
S2.1
Per-task Results of Transform Type on VTAB-IK Natural and Specialized
Table S14: Transform type per-task results VTAB-Ik 78 pretrained on supervised ImageNet-2Ikk.
Natural for ViT-Base/16 23
ViT-Base/16 (85.8M)
VTAB-IK
Natural
Mean
CIFAR- 100 Caltech101 DTD Howers102 Pets
SVHN
Sun397
FULL
79.9
89.8
VPT-SHALLOW
77.7
86.9
62.6
97.5
87.3
74.5
51.2
76.31 [4]
Tuned / Total (%)
0.18
0.10
0.04
0.27
0.08
0.19
0.36
0.17
VPI-DEEP A
78.8
90.8(3)
65.8
98.0
88.3
78.1
49.6
78.48 [6]
Tuned/ Total
0.20
0.15
0.04
0.54
0.41
0.23
OURS-FLL
80.8
91.7
70.5
98.5
89.4
83.3
52.7
80.98
Tuned / Total (%)
0.20
0.31
0.20
0.11
0.06
0.12
0.41
0.21
OURS-LLL
79.5
91.5
70.1
98.5
89.6
82.0
52.6
80.54 [6]
Tuned
/ Total
0.31
0.20
0.06
0.12
0.41
0.21
OURS-FFT FDA 71
80.7
91,4
69.4
98.5
89.9
83.6
52.7
80,90 [6]
Tuned / Total
(%)
0.20
0.31
0.20
0.11
0.06
0.12
0.41
0.21
OURS-FFT (default)
81.35 [6]
Tuned / Total (%)
0.20
0.31
0.20
0.11
0.06
0.12
0.41
0.21
Table S15:
Transform type per-task results on VTAB-1k pretrained on supervised ImageNet-21k.
Specialized for VYT-Base/16
ViT-Base/16 23
VTAB- Ik Specialized
85)
Patch Camelyon
EuroSAT
Resisc45 Retinopathy
Mean
FULL
85.1
96.4
83.1
74.3
84.72
VPT-SHALLOW 4
78.2
92.0
75.6
72.9
79.66
Tuned/ / Total (%)
0.01
0.05
0.09
0.01
0.04
VPT-DEEP 4
81.8
96.1
68.4
82.43 [2]
Tuned/ Total
1.06
1.07
0.15
0.02
0.57
OURS-FLL
83.3
95.2
83,5
74.1
84,02 [3]
Tuned/ / Total (%)
1.06
0.12
0.11
0.03
0.33
OURS-LLL
77.3
95.5
82.64 [3]
Tuned/ / Total (%)
1.06
0.12
0.11
0.03
0.33
OURS-FFT + FDA [7]
83.2
95.1
82,4
75.4
84.03 [3]
Tuned Total (%)
1.06
0.12
OURS-FFT (default)
83.5
96.5
84.4
75.4
84,93
Tuned// / Total (%)
1.06
0.12
0.11
0.03
0.33
S2.2 Per-task Results of Fourier Prompt Depth VTAB-1k Natural and Specialized
Table $16: Fourier prompt depth per-task results VTAB-1k 78 Natural for ViT-Base/16 pretrained on supervised ImageNet-21k..
ViT-Base/16 23
VTAB-IK Natural [7]
(85.8M)
CIFAR-100 100
Caltech101
DTT Flowers102 Pets
SVHN
Sun397
Mean
FULL
91.0
91.6
79.9
89.8
VPT-SHALLOW 4
62.6
97.5
87.3
74.5
51.2
76.81
Tuned / Total (%)
0.18
0.10
0.04
0.27
0.08
0.19
0.36
0.17
VPT-DEEP 4
78.8
90.8(3)
65.8
98.0
88.3
78.1
49.6
78.48 (6]
Tuned Total (%)
0.15
0.10
0.04
0.54
OURS 55
68.4
98.5
89.5
82.7
52.7
80.48
/ Total Total (%)
0.20
0.31
0.20
0.11
0.06
0.12
0.41
0.21
OURS (1-6) Tuned Total (%)
80.8
91.8
69.5
98.5
89.4
83.5
52.0
80.79 [6]
0.20
0.11
0.06
0.12
OURS (7-12)
98.6
89.4
83.6
52.7
80.83
Tuned Total (%)
0.20
0.31
0.20
0.11
0.06
0.12
0.41
0.21
OURS (1-12 (default)
80.7
91.4
69.4
99.3
90.3
85.6
52.7
81,35 (6]
Tuned Total (%)
0.20
0.20
0.11
0.06
0.12
0.41
0.21
20

---

Table S17: Fourier prompt depth per-task results VTAB-1k
78
Specialized for ViT-
Base/16 23
pretrained on supervised ImageNet-2Ik.
VrT-Basc/I6 23
VTAB-IE IK 78 Specialized [4]
(85.8M)
Patch Camelyon
EurSAT
Resise45 Retinopathy
FULL 192)
85.1
96.4
83.1
74.3
84.72
VPI-SHALLLOW
78.2
92.01
75.6
72.9
79.66
-Tuned Total (%)
0.01
0.05
0.09
0.01
0.04
VPT-DEEP
81.8
96.1
83.4
68.4
82.43 [2]
Tuned Total (5))
1.06
0.15
0.02
0.57
OURS (1 3 5 9 11)
82.9
95.2
81.8
75.1
83.73 [3]
Tuned/ / Total (%)
1.06
0.12
0.11
0.03
0.33
OURS (1-6)
84.0
83,6
74.7
84.34 [3]
Tuned/ Total (%)
1.06
0.12
0.11
0.03
0.33
OURS (7-12)
83.3
95.4
82,4
74.7
83,93 [3]
Tumed/ Total (%)
1.06
0.12
0.11
0.03
0.33
OURS (1-12 (default)
84.4
75.4
[4]
Tuned/ Total (%)
1.06
0.12
0.11
0.03
0.33
S2.3
Per-fask Results of Fourier Prompt Location on VTAB-1k Natural and Specialized
Table S18: Fourier prompt location per-task results VTAB-1k 78
Natural for ViT-
Base/16 23
pretrained on supervised ImageNet-21k.
Vil-Base/16
VTAB- 78
Natural [7]
(85.8M)
CIFAR- 100
Caltech101
DTD Flowers 102
Pets
SVHN
Sun397
Mean
FULL
57.6
91.0
64.6
91.6
79.9
89.8
29.1
71.95
VPT-SHALLOW 4
77.7
86.9
62.6
97.3
87.3
74.5
51.2
76.81 [4]
Tuned / Total (%)
0.18
0.10
0.04
0.27
0.08
0.19
0.36
0.17
VPT-DEEP 4
78.8
90.8(3)
98.0
78.1
49.6
78.48 [6]
"Tuned / Total (%)
0.20
0.20
0.15
0.10
0.04
0.54
0.41
0.23
OURS-Append
81.0
92.4
72.2
98.4
86.7
85.6
50.8
81.02 [6]
Tuned / Total (%)
0.20
0.11
0.06
0.12
0.41
OURS-Rundom
81.9
91.8
66.0
98.3
71.7
51.5
78.62
Tuned / Total (%)
0.20
0.31
0.20
0,11
0.06
0.12
0.41
0.21
DreS-Prepend (default)
80.7
91.4
69.4
99.3
90.3
85.6
$2.7
81,35 [6]
Tuned Total (%)
Table S19: Fourier prompt location per-task results on VTAB-1k 778) Specialized for ViT-
Base/16 23
pretrained supervised Ima
ViT-Base/16
VIAB-IK 78 Specialized [4]
(85.8M)
Patch Camelyon
FaroSAT
Resisc45 Retinopathy
Mean
FULL 992
85.1
96.4
74.3
84.72
VPT-SHALLOW
78.2
92.0
75.6
72.9
79.66
Tuned / Total (%)
0.01
0.05
0.09
0.01
0.04
VPI-DEEP [
81.8
96.1
83.4
68.4
82.43 [2]
Tuned Total
0.15
0.57
OURS-Append
83.2
95.1
81.5
75.4
83.80
Tuned Total (9E
1.06
0.12
0.11
0.03
0.33
AURS-Randon
83.2
82.47
Tuned/ Total (%)
1.06
0.12
0.11
0.03
0.33
OURS-Prepend (default)
83.5
96.5
84,4
75.4
84.93 [4]
TTuned Total
21

---

S2.4
Per-fask Results of Fourier Prompt Dimension on VTAB-1k Natural and Specialized
Table S20: Fourier prompt dimension per-task results on VTAB-1k 788 Natural for ViT-
Base/16 23
pretrained on supervised ImageNet-2Ik.
ViT-Base/16 23
VTAB- IK [78 Natural [7]
Mean
(85.8M)
CIFAR-100
Caltech101
DTD
Flowers102 Pets
SVHN
Sun397
FULL 522
57.6
91.0
64,6
91.6
79.9
89.8
29.1
71.95
VPI-SHALLOW 4
77.7
86.9
62.6
97.5
87.3
74.5
51.2
76.81 [4]
Tuned Total (%)
0.18
0.10
0.04
0.27
0.08
0.19
0.36
VPT-DEEP 4
78.8
90.8(3)
65.8
98.0
88.3
78.1
49.6
78.48
Tuned Total (%)
0.20
0.20
0.15
0.10
0.04
0.54
0.41
0.23
OURS-Sequence length
79.8
91.6
70.3
98.5
89.6
84.0
$2.3
80.88 [6]
Tuned/ Total (%)
0.31
0.06
OURS-Hidden
80.5
91.5
69.9
98.5
89.5
83.5
51.9
80.74
Tuned/ / Total (%)
0,20
0.31
0,20
0.11
0,06
0.12
0.41
0.21
OURS-Both (default)
80.7
91.4
69.4
99.3
90.3
85.6
52.7
81.35 [6]
Tunndd/ Total
0.20
0.31
0.20
0.06
0.12
0.41
0.21
Table S21: Fourier prompt dimension per-task results on VTAB-1k [88] Specialized for ViT- pretrained supervised ImageNet-2Ik.
Base/16 23
ViT-Base/16 23
VTAB-Ik 781 Specialized [4]
Mean
(85.8M)
Patch Camelyon
EuroSAT
Resise45 Retinopathy
FULL 192
85.1
96.4
83.1
74.3
84.72
VPT-SHALLOW H
78.2
92.0
75.6
72.9
79.66
Tuned/ / Total (%)
0.01
0.15
0.09
0.01
0.04
VPT-DEEP AI
81.8
96.1
83.4
68.4
82.43 [2]
Tuned/ Total (9)
1.06
1.07
0.02
OURS-Seqqence length
81.5
95.3
82.5
75.0
83.57
Tuned / Total (%)
1.06
0.12
0.11
0.03
0.33
OURS-Hidden Hidden
83.3
94.7
82.8
74.6
83.87 [3]
Tuned Total (%)
1.06
0.12
0.11
0.03
0.33
OURS-Both (default)
83,5
96,5
84,4
75.4
84,93 [4]
Tuned Total (%)
1.06
0.12
0.11
0.03
0.33
S2.5 Sensitivity of Fourier Prompt Percentages and Prompt Lengths
Fourier Percentage of Visual Fourler Prompt (%)
30
50
100
Length of Visudl Fourker Prong
10-
65.4
68.5
68.2
68.4
20-
69.4
65.2
66.6
67.55
50-
63.8
68.4
67.5
100-
61.0
65.0
64.2
63.2
-61
Figure $1: Sensitivity visual Fourier prompt percentages and its prompt lengths on VTAB-Ik
DTD.
S3
Per-task Results on Fourier Percentage
22

---

Table S22: Fourier percentage per-task results VTAB-Ik 78
Natural for ViT-Base/16 23
pretrained on supervised ImageNet-22 2 lk. The highest accuracy among all Fourier percentages are
shown in bold. for Table S23 and S24
Fourier
VTAB-IK [78
Natural [7]
Percentage (%)
CIFAR- 100 Caltech101
DTD Flowers102
Pets
SVHN
Sun397
0
78.8
90.8
65.8
97.9
88.4
76.4
49.6
30
79.7
91.4
69.4
83.1
51.3
50
80.3
91.4
68.5
99.3
90.3
84.3
52.7
80.7
91.3
66.6
84.0
52.1
100
80.6
91.0
67.8
98.3
87.2
78.5
52.3
Table S23: Fourier percentage per-task results on VTAB-1k (78) pretrained on supervised ImageNet-21k,
Specialized for ViT- WTT-Base/16 23
Fourier
VTAB-IK [78
Specialized (4)
Percentage (%)
Patch Camelyon EuroSAT Resisc45 Retinopathy
82.0
96.1
83.4
68.0
30
82.6
95.3
84.3
82.4
96.1
83.6
74.6
83.2
83.2
100
83.3
83.1
75.4
Table S24: Fourier percentage per-task results VTAB-1k 78 Structured for ViT- ViT-Base/16
pretrained on supervised ImageNet-21k..
Fourier
VTAB-Ik Ik 78 Structured [8]
Clevr/ Clevrl count distance
KITTV/ dSprites/ distance location orientation
dSprites/
SmalINORB/ SmalINORR/
Percentage (%)
DMLab
uzimuth
clevation
0
46.5
72.8
73.6
29.3
40.2
30
73.7
61.2
46.7
76.8
74.7
46.1
24.6
42.0
50
73.5
62.1
47.1
79.3
74,5
47.9
30.6
41.9
70
74,3
62.7
48.3
79.0
79.7
56.0
30.8
43.4
100
75.8
63.2
47.5
77.1
47.9
34.1
42.0
23

---

S4 Visualization of Attention Map
Sample A
Sample B
Sample C
Sample D
Sample
Sample F
BL
VPT
VFPT
VPT (b) 3D Attention Map
VFPT
Input
VPT
VEPT
(a) 21) Attention Map
(c) GraCCAM M Heatmap
Figure S2: (a) More visualization results of 2D attention map on VTAB-IK 78
(b) Correspond-
ing 3D attention maps. Figures best viewed by zooming in. (c) More visual inspection of VPT and VFPT using GradCAM [04]. Consistent to our paper, the red regions correspond to high score for class. From left to right are input image after standard data augmentation, GradCAM results for VPT and GradCAM results for VFPT. Figure best viewed in color.
In this section, we present findings in $4.4 All samples selected from VTAB-1k (78 have the same prompt length (i.e., 10 prompts) with one class token and 196 input patches.
details and results of visualization of attention maps support our
In Fig a), we can first observe significant attention distribution in learnable prompts and then notably higher concentration in global attention scores when integrating visual Fourier prompts, showing consistency with our paper.
In (2, we present more visualization inspection results for VPT and VFPT using Grad- CAM [104] Overall, we present additional visual evidence to support the notion that the integration of visual Fourier prompts encourage clear segeround-backgrouun separation.
24

---

S5 Visualization of Loss Landscape
10
as
055
-10
-1.0
1.0
10
1.0
00
00
405
Powesstt
as
00
-10
-0.
60
0.5
1.0
-10
435
03
10
2D
Cootour
2D Heat Map
3D Surface
2D Contour
2D Heat Map
3D Sufface
VPT Loes Less Lambare
(b) VEPT Les Landaapee
Figure S3: Loss landscape on VTAB-Ik
78
Natural.
10
405
-05
-1.0
80
06
455
-05
-44
01
0.5
0.0
-1.
.LD
60
-1.
405
100
05
2D Contour
2D Heat Map VPT Loss Landscape
3D Surtace
2D Coatour
2D Heat Map
3D Surface
VPPT Loss Landscaee
Figure S4: Loss landscape on VTAB-IK 78 Specialized.
10
1.0
4.5
415
-10
80
0.
403
415
1.0
310
43
00
10
0U
10
-1.
405
00
1.0
-10
465
00
05
1.0
20
Conlour
2D Heat Map
3D Surface
2D Contour
2D Heat Map
3D Surface
VPT Lass Landscape
VFPT Loss Lanksapp
Figure S5: Loss landscape on VTAB-Ik Structured.
25

---

S6 Extension to Language Tasks
While ViT-Base/16 23 is structurally similar to BERT [07], we follow [5] 108 and naturally test
the efficiency of the VFPT on natural language understanding (NLU) tasks. Specifically, include
BERT-Large [07 for evaluation, and compare full fine-tuning (FULL) (2. Prompt Tuning [2].
P-Tuning v2 108 and V2 [5] on SuperGlue [07] dataset: collection of text classification tasks to test the general language understanding ability. The tasks include natural language inference (RTE and CB), coreference resolution (WSC), sentence completion (COPA), word sense disambiguation (WiC), and question answering (MultiRC (Fla), ReCoRD (F1) and BoolQ). In Table [S25) we show that VFPT outperforms FULL and Prompt Tuning and show competitive results to P-Tuning v2 [[88] Considering VFPT is designed for visual-related tasks, and text understanding tasks might not need fruitful frequency domain information, these results are impressive and suggest future work for general solution across modalities under the pretrain- teeeainni-eeeeeeee fiten-finetune paradigm.
Table S25: Per-task results for SuperGLUE development set [109] with a pretrained BERT-
Large [107. See $6
BERT-Large 107
SuperGLUE 107 [8]
Mean
(335M)
BoolQ
CB
COPA
MultiRC (Fla) ReCoRD (FI)
RTE
WIC
WSC
FULL 2
77.7
94.6
69.0
70.3
70.6
70.4
74.9
68.3
74.30
Prompt Tuning 2
67.2
80.4
55.0
59.6
44.2
53.5
63.0
64.4
60.91
P.Tuning v2
73.1
94.6
73.0
70.6
72.8
68.3
75.73
C2 [
74.4
80.4
77.0
65.8
71.9
78.7
74.3
67.3
73.73
OURS
74,8
81.2
78,1
67.8
72.9
77.2
75.3
68.4
74.46
S7 Extension of Complexity Analysis
Table S26: Complexity analysis of fourier percentage settings on CIFAR-100 benchmark. The percentages in the results indicate the rate of improvement compared to VPT.
Fourier Percentage
Maximum Memory Consumption (GB)
Training Average Batch Time (s) Inference Average Batch Time
VPT (0S)
[,8100
0.
0.0499
VFPT (30%)
1.8210 (0%)
0,1169 54%)
0.0505 (+1 (+1.20%% 20%))
VEPT (50%)
1.8210 (0%)
0.1155 (2%)
0,0502 (40.60%)
VFPT (70%)
1.8210 (0%)
0,1150 (+0.88%)
0.0500 (+0.20%)
VFPT (100%)
1.8210 (0%)
0.1150 (+0.88%)
0.0501 (40.40%)
We have provided detailed comparison of our computational results in this section. More specifically, we experimented with different Fourier percentage settings (i.e.., the alpha rate) on the CIFAR-100 benchmark and reported their maximum memory consumption, average training batch time, and average inference batch time. All settings were tested with the same batch size and prompt length. The experiments were conducted on NVIDIA A A100-40GB GPUs.
As illustrated in Table S26 no significant increase in maximum memory consumption at the MB level observed across different Fourier percentage settings. However, do observe slight increase in average batch time during both training and inference, on the order 10 and 10 10-4 respectively. This suggests that lower Fourier percentage incurs higher computational burden. This effect is likely attributable to suboptimal parallel acceleration and the implementation inefficiencies associated with prompts that have partial Fourier transformation. We will investigate this further in future research.
S8 Asset License and Consent
The majority of VPT 4 is licensed under CCC-BY-CC 4.0, Portions of 4 are available under separate licenses: gowgle--eseerrh/eeeeeeeeooo
adaptation and huggingfaeerrrasfforerrs are licensed under Apache 2.0, Swin- Transformer [24) and ViT-pytorch 23 are licensed under MIT; and MoCo-v3 26
and MAE 90 are licensed under CC BY 4.0.
All the datasets included in our study are publicly available (VTAB-IK, FGVC), and all the models are publicly available. We would like to state that the contents in the dataset do NOT represent our views or opinions.
26

---

S9
Reproducibility
VFPT is implemented in Pytorch [9]]. Experiments are conducted NVIDIA A100-40GB GPUs. To guarantee reproducibility, our full implementation shall be publicly released upon paper acceptance. For training schedule, the superior low-complexity of FFT (i.e. O(n log n)) allows for efficient training of visual Fourier prompts with only a slight decrease in training speed (i.e., 2.8% on VTAB-IK 78 compared to VPT).
S10
Social Impact and Limitations
This study presents VFPT, demonstrating significant and generalizable performance enhancements over state-of-the tort---rrtttttttt baselines across two benchmarks. The incorporation the FFT contributes these advantages without necessitating cchitecture-specitti designs incurring substantial computational overhead under pretrain-then--mmetuue paradigm for large-scale models (see $ Our approach enjoys advanced model accuracy, and is valuable in real-world computational--ennitti applications, e-g-, training machine learning models on edge devices. Moreover, VFPT advances significantly towards achieving generality across datasets, demonstrating substantial performance improvements even when faced with large dataset disparities (see $4). This progress is crucial for the continuous development of PEFT across a wider spectrum applications.
For potential limitations, drawing inspirations from human visual cognition, our method incorporates spatial and frequency information, which brings additional hyper-parameter Fourier percentage (i.e., a in (3.2). However, in practical applications, we observe in $4.2 that dataset disparity (i.e., low disparity tasks prefer small value, and vice versa) serves guideline for selecting appropriate Fourier percentage. Nonetheless, we argue that the implementation of an automatic Fourier percentage search further augment efficiency.
S11 Discussion and Future Work
In $ we review PEFT methods and the application of the fast Fourier transform in vision. Notably, recent study 36 in NLP incorporates Fourier transform a viable PEFT approach, which warrants discussion. Specifically, it learns a set of spectral coefficients of Fourier basis using LoRA-based approach and then applies the inverse discrete Fourier transform to the spectral matrix, yielding its ppatial-domain counterpart as the updated weight change. Although the Fourier basis's orthogonal and expressive advantages reduce the need for extensive parameter fine-tuning, the inverse transform applied to the spectral matrix discards frequency information, ultimately considering only traditional spatial domain features. The parameter-efficient use of the Fourier transform in this study orthogonal to method, where both spatial and frequency domain information integrated (see }3)
for enhanced generality (see (4.2) and interpretability (see $4.4).
Despite VFPT systemic effectiveness and simplicity, it also comes with new challenges and unveils some intriguing questions. For example, the balance between spatial and frequency information is presently dictated by task-specific, manually set percentages (see $4.2). Introducing a small network within the VFPT framework autonomously search for optimal combinations might enhance training efficiency and facilitate additional performance improvements. Another essential future direction deserving of further investigation is the integration of visual information from both the spatial and frequency domains. In $4.5) we demonstrate through ablation studies that integration at the pre- processing stage may not yield satisfactory performance. Consequently, we outline several alternative integration approaches in Table demonstrating that VFPT holds the most ad advantageous position under the prompt tuning paradigm. Nonetheless, the applicability this integration to other PEFT methods requires further investigation.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose VFPT, an intuitive yet effective strategy that utilizes the Fourier- based operations to facilitate visual prompt tuning. The main contributions of VFPT (i.e., simplicity, generality, and interpretability) are claimed both the abstract and introduction accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in$ in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
28

---

Answer: [Yes]
Justification:
All theorems used in the paper are properly referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We claim reproducibility in both 4 and $99 available after acceptance.
Our code will be publicly
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We claim reproducibility in both $4 and $59 All the datasets included in our study are publicly available (VTAB-1K, FGVC). Our code will be publicly available after acceptance. The publicly available code should be adequate to replicate the primary experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
httpe//guuieee/oeeSuuuuin www.sssoooiiioooo.oo olicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips /twps.cccppblll.... httppppullllippeeeeeppppppppppee htte:/guddeeeeeeeSubuimiieeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in $4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report standard deviation error bars for our main result in SI
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify all the training and test computing resources in and and $9
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::///// http::::///e.i....eeee http://rrripsscccclllel//////eee http:///bb.ceeeiicccccceeeelllll
Answer: [Yes]
Justification: We conform to the NeurIPS Code of Ethics and show related asset license and consent to our work in $S8
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
$10 discusses the social impact of our research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We show related asset license and consent to our work in $8
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets. In $88 we include asset license and consent.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "EH2wm97ZPj", "reviewer": "Reviewer_PyKu", "summary": "This paper proposes Visual Fourier Prompt Tuning (VFPT) to address performance degradation in parameter-efficient finetuning (PEFT) methods caused by dataset disparities. VFPT integrates Fast Fourier Transform (FFT) into prompt embeddings, enhancing performance with minimal parameter usage. This work is built upon the VPT method. The key innovation is to integrate frequency domain information using FFT to VPT. The performance is evaluated on VTAB-1k and FGVC.", "questions": "I have several expectations for seeing the potential of VFPT. If the author can address these questions, I will change my scores.", "limitations": "Yes, the authors discussed the limitation of their methods.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments are comprehensive and performance on several benchmark are impressive as well. Study of Interpretability shows some interesting findings can be helpful for future research. The ablation study is completed. We can see compared with VPT baseline, FFT’s effectiveness has been well demonstrated.\nThe demonstration including figure/tables of this paper is very well.", "weaknesses": "Overall, this is a well-organized paper.  I have several questions regarding the methods and results;\n1. What is the larger model’s performance when applying the VFPT? Whether the VFPT can work well with large models is an important question that hasn't been answered in the current version. As the model size grows, efficient, prompt learning can be more critical. However, current results are mainly on a relatively small size of models.\n2. Whether VFPT can be integrated into the CLIP model to enhance zero-shot performance is also an interesting question. I am very interested in vision and text encoders and whether VFPT can be applied to CLIP models. \n3. Current results do not answer the first question in the introduction: “Can prompt tuning generalize across datasets with varying disparities?”. It confuses me because I think of the transferable prompt design, which is trained on one dataset and then transferred to another. This motivation, to me, needs a further rephrase or validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PyKu,\n\nWe sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We provide explanations to your questions point-by-point in the following.\n\n**Q1: Regarding the VFPT performance with larger models.**\n\n**A1:** Thank you for the great suggestion. We use ViT-Base and Swin-Base for comparison as they are the most commonly adopted models in previous works. We completely agree that prompt learning is also important for large models. Based on your suggestion, we have conducted additional experiments by applying VFPT to the ViT-Huge (632M), which is significantly larger than the ViT-Base (86M) and Swin-Base (86.7M). The preliminary results in the table below show that VFPT consistently achieves better performance with larger models. We will include the full results with further discussion in the revision. Thank you again for your suggestion.\n\n| ViT-Huge  | Natural (7) | Specialized (4) | Structured (8) |\n| ---- | ----------- | --------------- | -------------- |\n| VPT  | 77.9        | 83.3            | 52.2           |\n| VFPT | 78.9        | 84.4            | 57.0           |\n\n**Q2: Regarding the integration of VFPT into the CLIP model.**\n\n**A2:** Thank you for the insightful question. The proposed VFPT is a general and robust solution that can be applied to different models. Based on your suggestion, we conducted a preliminary study on CLIP. Specifically, we applied VFPT to the vision encoder of CLIP using the optimal settings from this work. The results on ImageNet are shown in the table below. As seen, there is a noticeable improvement compared to the standard CLIP, indicating the flexibility of adapting VFPT to different models. We appreciate your suggestion and plan to investigate further in the direction of prompt tuning on vision-language models.\n\n|    ImageNet  | Tuned / Total (%) | Accuracy (%) |\n| -------------- | ----------------- | ------------ |\n| CLIP (zero-shot) | 0.00              | 66.73       |\n| CLIP + VPT     | 0.22              | 70.30        |\n| CLIP + VFPT    | 0.21              | 71.49        |\n\n**Q3: Regarding the motivation of ‘Can prompt tuning generalize across datasets with varying disparities?**\n\n**A3:** Sorry for the confusion. We would like to provide some clarification. **First**, this question is motivated by the observation that although original prompt tuning methods (e.g., VPT) have achieved promising results, significant performance degradation occurs when there is a substantial disparity between the datasets used in the pretraining and finetuning phases. The purpose of designing VFPT is to bridge this gap by incorporating frequency domain information. **Second**, we try to answer this question in Section 4.2 (lines 240-268). The empirical results indicate that VFPT achieves noticeably better performance on datasets with large disparities compared to other baselines. We’ll rephrase the question during revision to enhance the clarity.\n\nWe appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q4OqGfKQnJ", "reviewer": "Reviewer_9y4B", "summary": "This paper introduces Visual Fourier Prompt Tuning (VFPT), an approach for parameter-efficient fine-tuning of large vision models. VFPT integrates Fast Fourier Transform (FFT) operations into visual prompt tuning, allowing it to incorporate both spatial and frequency domain information. The method demonstrates superior performance and generalizability across various tasks and datasets, particularly when there are significant disparities between pretraining and fine-tuning data. VFPT outperforms several state-of-the-art baselines on benchmarks like VTAB-1k and FGVC, using only a small fraction of trainable parameters. The authors provide extensive empirical results, ablation studies, and visualizations to demonstrate the effectiveness of their approach. They also explore the optimization landscape and interpretability aspects of VFPT.", "questions": "Please refer to the weaknesses section.", "limitations": "Limitations:\n\nThe authors do discuss limitations in Section S9 of the appendix, which is a positive point. They specifically mention the introduction of an additional hyperparameter (Fourier percentage) and acknowledge that while dataset disparity can guide its selection, an automatic search method could further improve efficiency. This shows awareness of a key limitation in their approach.\n\nHowever, the limitations discussion could be more comprehensive. For example:\n\n1. They could elaborate on potential scenarios where VFPT might not perform well.\n\n2. Discussion of scalability limitations or computational constraints for very large models is not clearly addressed.\n\n3. Potential limitations in applying VFPT to other domains or model architectures could be explored further.\n\nSocietal Impact:\n\nThe authors do discuss social impact in Section S9 of the appendix, which is commendable. They highlight positive aspects such as:\n\n1. Potential for improving model accuracy without substantial computational overhead.\n\n2. Applicability in computationally-sensitive real-world applications.\n\n3. Advancements towards achieving generality across datasets.\n\nHowever, the discussion of potential negative societal impacts is limited. The authors could have explored:\n\n1. Potential misuse of more efficient and generalizable models in privacy-invading or surveillance applications.\n\n2. Environmental impacts of encouraging more fine-tuning experiments.\n\n3. Potential biases that might be introduced or amplified through this method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n\nThe paper presents an approach by integrating Fast Fourier Transform (FFT) operations into visual prompt tuning. This is an original idea that bridges frequency domain analysis with parameter-efficient fine-tuning of large vision models. The authors draw inspiration from human visual cognition to incorporate both spatial and frequency domain information, which is a creative angle not explored in previous visual prompt tuning methods.\n\nQuality:\n\n- Comprehensive experiments are conducted across multiple benchmarks (VTAB-1k, FGVC) and model architectures (ViT, Swin Transformer).\n\n- Rigorous ablation studies examine various components of the proposed method.\n\n- The authors provide in-depth analysis of the optimization landscape and interpretability aspects.\n\n- Error bars and statistical significance are reported for main results.\n\n- The paper includes thorough comparisons with state-of-the-art baselines.\n\nClarity:\n\nThe paper is well-structured and clearly written.\n\nSignificance:\n\n- It addresses a key challenge in parameter-efficient fine-tuning - performance degradation when there's a large disparity between pretraining and fine-tuning datasets.\n\n- The method achieves strong performance while using only a small fraction of trainable parameters, which is crucial for adapting large models efficiently.\n\n- The approach is general and can potentially be applied to various vision transformer architectures.", "weaknesses": "1. The paper lacks a rigorous theoretical analysis of why integrating Fourier components improves generalization. While empirical results are strong, a deeper theoretical investigation could provide insights into:\n\n- Why frequency domain information helps with dataset disparities\n\n- How the balance between spatial and frequency information affects model behavior\n\n- Potential limitations or failure cases of the approach\n\nSuggestion: Develop a theoretical framework explaining the interplay between spatial and frequency domain information in prompt tuning. This could involve analyzing the properties of the loss landscape or examining how Fourier components affect the model's representation space.\n\n2. The Fourier percentage (α) is a crucial hyperparameter, but its selection process is not fully automated. The paper suggests using dataset disparity as a guideline, but this may not always be easily quantifiable.\n\nSuggestion: Develop an adaptive method to automatically determine the optimal Fourier percentage based on task characteristics. This could involve a small meta-network that predicts the optimal α, or a dynamic adjustment mechanism during training.\n\n3. While the paper mentions a slight decrease in training speed (2.8% on VTAB-1k), a more comprehensive analysis of computational trade-offs is missing. This is important for practitioners considering adopting the method.\n\nSuggestion: Provide a detailed analysis of computational overhead across different settings, including:\n\n- Training time comparisons with baselines\n\n- Memory usage\n\n- Inference time implications\n\n- Potential optimizations to reduce overhead\n\n4. The paper primarily uses FFT as a black-box operation. A deeper exploration of how different frequency components contribute to performance could yield additional insights.\n\nSuggestion: Conduct experiments analyzing the impact of different frequency bands on model performance. This could involve selectively filtering certain frequency components or visualizing which frequencies are most important for different tasks.\n\n5. The generalization to other architectures is not clear. While the method is tested on ViT and Swin Transformer, its applicability to other architecture types (e.g., convolutional networks, MLP-Mixers) is not explored.\n\nSuggestion: Extend the experimentation to a broader range of architecture types to demonstrate the method's generality. If challenges arise, analyze and discuss the limitations of applying VFPT to different model families.\n\n6. The paper doesn't deeply explore how Fourier prompts interact with regular prompts or how they influence different layers of the network.\n\nSuggestion: Conduct a layer-wise analysis of how Fourier prompts affect feature representations throughout the network. Visualize and quantify the interactions between Fourier and regular prompts to provide insights into their complementary roles.\n\n7. The paper doesn't thoroughly address the robustness of VFPT to different initializations or potential instabilities during training.\n\nSuggestions: Perform a comprehensive stability analysis, including:\n\n- Sensitivity to different random seeds\n\n- Learning rate schedules that work well with VFPT\n\n- Potential gradient issues or training instabilities\n\n- Robustness to different prompt lengths and configurations\n\n8. While benchmark performance is strong, the paper lacks discussion on how VFPT performs on real-world, large-scale applications outside standard benchmarks.\n\nSuggestion: Include case studies or experiments on practical, large-scale vision tasks (e.g., autonomous driving, medical imaging) to demonstrate the method's effectiveness in real-world scenarios. Discuss any challenges or modifications needed for such applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9y4B,\n\nWe sincerely appreciate your time and effort in reviewing our paper and providing detailed comments and suggestions, which are crucial for improving our work. We provide explanations to your questions point-by-point in the following.\n\n**Q1: Regarding theoretical analysis.**\n\n**A1:** This is indeed an excellent suggestion. While we empirically investigate why VFPT achieves better performance and generalizes well across various tasks from an optimization perspective, we also want to highlight several potential directions for future theoretical analysis:\n- The influence of the self-attention mechanism: the self-attention mechanism is the key component of transformers, we plan to conduct a deeper exploration into how the incorporation of frequency information influences the attention module.\n- The potential mitigation of low-pass filter phenomena [1] in transformers. The work in [1] indicates that as ViT scales up, excessive low-pass filtering can cause attention collapse. In future research, we aim to theoretically analyze whether the Fourier transform can reduce low-frequency passing and introduce high-frequency components to mitigate this issue.\n\n[1] Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice. ICLR 2022.\n\n**Q2: Automatically determine the optimal Fourier percentage.**\n\n**A2:** We are exploring introducing a small network within the VFPT framework to autonomously search for optimal combinations as mentioned in Appendix S10 (line 733-735). This approach might enhance training efficiency and facilitate additional performance improvements. We plan to conduct more experiments in this direction. Thank you for the suggestion.\n\n**Q3: A more comprehensive analysis of computational trade-offs.**\n\n**A3:** Following the suggestion, we provide a detailed computational analysis with different settings below. Specifically, we experiment with different Fourier percentages on the CIFAR-100 benchmark and report their maximum memory consumption, average training batch time, and average inference batch time. All settings are tested with the same batch size and prompt length. The experiments are conducted on NVIDIA A100-40GB GPUs.\n\n\n|   Settings    | Maximum Memory Consumption (GB) | Training Average Batch Time (s) | Inference Average Batch Time (s) |\n| ---------------- | ------------------------------ | ------------------------------- | -------------------------------- |\n| VPT (alpha=0.0)  | 1.8210                     | 0.1140                          | 0.0499                           |\n| VFPT (alpha=0.3) | 1.8210 (0%)                | 0.1169 (+2.54%)          | 0.0505 (+1.20%)       |\n| VFPT (alpha=0.5) | 1.8210 (0%)                | 0.1155 (+1.32%)         | 0.0502 (+0.60%)        |\n| VFPT (alpha=0.7) | 1.8210 (0%)                | 0.1150 (+0.88%)       | 0.0500 (+0.20%)          |\n| VFPT (alpha=1.0) | 1.8210 (0%)                | 0.1150 (+0.88%)    | 0.0501 (+0.40%)             |\n\n\n**Q4: Analyzing the impact of different frequency bands.**\n\n**A4:** Thank you for the insightful suggestion. Following the suggestion, we further visualize the spectral response of an attention map in the last layer (**Figure 1 in the rebuttal PDF**) on the KITTI/distance benchmark following Anti-Oversmoothing [1]. The preliminary results show that VFPT effectively filters out specific low-frequency components while introducing high-frequency components when compared to VPT. This observation suggests that VFPT may effectively mitigate the attention collapse phenomenon (i.e., as the transformer goes deeper, the attention maps gradually become similar and even much the same after certain layers), as introduced in [1-3]. This observation is consistent with VFPT’s superior performance when compared to VPT. We’ll supplement the analysis with further discussion in the revision.\n\n[2] Improving vision transformers by revisiting high-frequency components. ECCV 2022.\n\n[3] Deepvit: Towards deeper vision transformer. arXiv 2021\n\n(Due to character limitations, we will continue to answer in another comment.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Bu5uq6rWE", "reviewer": "Reviewer_TrZ7", "summary": "This paper introduces Visual Fourier Prompt Tuning (VFPT), a novel approach to parameter-efficient fine-tuning (PEFT) for large-scale Transformer-based vision models. VFPT integrates Fast Fourier Transform (FFT) with prompt tuning, allowing the model to adapt to new tasks effectively by utilizing both spatial and frequency domain information, especially when there is a significant disparity between the datasets used in pre-training and fine-tuning phases. This method not only retains the simplicity and intuitiveness of standard visual prompt tuning but also enhances model performance across various tasks without substantially increasing the parameter count. Empirical results show that VFPT outperforms several state-of-the-art PEFT methods on benchmark datasets like VTAB-1k and FGVC, achieving higher accuracy with fewer parameters. For instance, VFPT uses only 0.66% of the total model parameters while achieving 73.20% mean accuracy on VTAB-1k, surpassing both full fine-tuning and conventional visual prompt tuning methods.", "questions": "Table 5(b) explores the effects of different Fourier Prompt Locations (Prepend, Append, and Random) on the performance of Visual Fourier Prompt Tuning (VFPT). The results suggest variations in performance based on the placement of these prompts within the input sequence. Since VPT mentions that different prompt locations are mathematically equivalent, I wonder why the location of Fourier prompts impacts model performance? The following sentence is from section 3.2 of the original VPT paper: “Notably for ViT, $x_N$ is invariant to the location of prompts since they are inserted after positional encoding, e.g., $[\\mathbf{x_0}, \\mathbf{P}, \\mathbf{E_0}]$ and $[\\mathbf{x_0}, \\mathbf{E_0}, \\mathbf{P}]$ are mathematically equivalent. This also applies to VPT-Deep.”", "limitations": "The limitations have been discussed in Appendix S9.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-structured and clearly written, with each section logically flowing into the next.\n2.\tVFPT introduces a unique combination of Fast Fourier Transform (FFT) with visual prompt tuning in Transformer-based models, which leverages both spatial and frequency domain information.\n3.\tThe empirical results are robust, demonstrating VFPT's superiority over existing state-of-the-art parameter-efficient fine-tuning methods. The paper includes detailed comparative analyses, showing VFPT's performance advantages across multiple benchmark datasets.\n4.\tVFPT has significant implications for the scalability and efficiency of Transformer-based models in vision tasks. The ability to maintain high performance with fewer parameters makes large-scale models more accessible and practical for a wider range of applications.\n5.\tThe paper discusses the theoretical implications of integrating FFT within visual prompt tuning, noting improvements in the optimization landscape and model interpretability. VFPT promotes a smoother loss landscape, which correlates with better generalization and lower test error rates.", "weaknesses": "1.\tThe abstract states that the proposed method, VFPT, utilizes only 0.57% of the model parameters to achieve a performance of 73.20% mean accuracy on VTAB-1k dataset. However, Table 1 contradicts this by listing the parameter usage as 0.66%. This discrepancy raises concerns about the accuracy of the reported statistics.\n2.\tThe paper does not fully elucidate the complexity introduced by integrating FFT into the prompt tuning process. While the method is praised for its parameter efficiency, the computational overhead, especially in terms of runtime and memory consumption during the FFT operations, is not thoroughly discussed.\n3.\tTable 3 reveals that VFPT appears to underperform relative to some established partial tuning and extra module methods when applied to models pretrained with MAE self-supervised objective. This might indicate that VFPT's reliance on Fourier transforms does not synergize as effectively with the feature representations or data distributions typical of self-supervised learning models. Understanding why VFPT performs less effectively in this context is crucial for refining the approach or identifying its optimal application scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TrZ7,\n\nWe sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We provide explanations to your questions point-by-point in the following.\n\n**Q1: Regarding the discrepancy of tuned parameter rate.**\n\n**A1:** Sorry for the confusion. We want to clarify that the average percentage 0.57% mentioned in the abstract is exclusive to the results obtained on the VTAB-1k benchmark, which comprises 19 subset datasets. While 0.66 % in Table 1 is the average percentage of tuned parameters required by the overall 24 tasks. i.e., encompasses both the VTAB-1k benchmark and the FGVC datasets (i.e., 19 from VTAB-1k and 5 from FGVC). We will revise accordingly to make it more clear.\n\n**Q2: Regarding complexity analysis.**\n\n**A2:** Thank you for the excellent suggestion. We have provided a detailed comparison of our computational results below. More specifically, we experimented with different Fourier percentage settings (i.e., the alpha rate) on the CIFAR-100 benchmark and reported their maximum memory consumption, average training batch time, and average inference batch time. All settings were tested with the same batch size and prompt length. The experiments were conducted on NVIDIA A100-40GB GPUs. We’ll supplement these results in the revision.\n\n|    Configurations   | Maximum Memory Consumption (GB) | Training Average Batch Time (s) | Inference Average Batch Time (s) |\n| ---------------- | ------------------------------ | ------------------------------- | -------------------------------- |\n| VPT (alpha=0.0)  | 1.8210                     | 0.1140                          | 0.0499                           |\n| VFPT (alpha=0.3) | 1.8210 (0%)                | 0.1169 (+2.54%)         | 0.0505 (+1.20%)        |\n| VFPT (alpha=0.5) | 1.8210 (0%)                | 0.1155 (+1.32%)         | 0.0502 (+0.60%)        |\n| VFPT (alpha=0.7) | 1.8210 (0%)                | 0.1150 (+0.88%)    | 0.0500 (+0.20%)             |\n| VFPT (alpha=1.0) | 1.8210 (0%)                | 0.1150 (+0.88%)    | 0.0501 (+0.40%)             |\n\n\n\n**Q3: Regarding the performance in MAE tasks.**\n\n**A3:** Thank you for the insightful question. There are two main reasons why VFPT performs less effectively in MAE objectives.\n\n**First**, our work is built on top of VPT, which has significantly low performance on different pretrained objectives. VPT claims this may be due to the training strategies being fundamentally different between the two self-supervised ViTs and the supervised ones. Though in this work, we manage to significantly narrow this gap (e.g., 53.59% vs. 36.02% under MAE on VTAB-1k Natural) by introducing frequency domain information, the discrepancy in the training objectives might still cause lower performance. \n\n**Second**, we observe an interesting phenomenon that during prompt tuning, the evaluation performance in self-supervised objective tasks is less stable compared to supervised tasks (e.g., 0.33 vs. 1.30 average standard deviation on VTAB-1k Natural). This observation further supports our claim above. In the future, we plan to explore and further mitigate the gap between different pretrained objectives.\n\n**Q4: Regarding Fourier Prompt Locations.**\n\n**A4:** Thank you for the question. We want to clarify that the claim in VPT holds when switching between $E_{0}$ and $P$, as it only changes the position considering **all prompts as an entire block**. In contrast, in VFPT, the Fourier operation is introduced **within prompts**, indicating that it still exists in different location settings when the Fourier percentage is determined. Our ablative study on Fourier Prompt Location investigates how we incorporate Visual Fourier Prompts within the prompt representation. Different locations result in varied prompt representations, thereby distinctly impacting model performance.\n\nWe appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To All Reviewers:

We sincerely thank all reviewers for your valuable suggestions and constructive feedback. We have revised our paper accordingly. The major changes are as follows:
- We’ve conducted additional experiments on the large model (i.e., ViT-Huge), vision-language model (i.e., CLIP), and other architectures (i.e., ConvNeXt-Base), following Reviewer PyKu and 9y4B's suggestion.
- We’ve added additional discussions with real-world, large-scale applications, as suggested by Reviewer 9y4B.
- We’ve rephrased our exploration and theoretical analysis of VFPT, as suggested by Reviewer 9y4B.
- We’ve tried to visualize the spectral response of an attention map in the last layer in Figure 1 of the rebuttal PDF, as suggested by Reviewer 9y4B.
- We’ve provided details about the robustness of VFPT and discussions of determining Fourier percentage automatically, as suggested by Reviewer 9y4B.
- We’ve supplemented runtime analysis and comparison, as suggested by Reviewer 9y4B and TrZ7.
- We’ve provided additional discussion on the less effective performance in self-supervised tasks, as pointed out by Reviewer TrZ7.

We hope our response addresses the concerns from the reviewers. Please let us know if there are any additional questions, and we will be happy to discuss further.

Best Regards,\
Authors

Meta-review: The paper proposes a novel parameter-efficient finetuning method for vision models that relies on the fast Fourier transform. This allows for strong performance while saving in number of adapted parameters.

This paper has received A,A,WA ratings and the rebuttal has addressed the concerns and questions raised by the reviewers.
Decision: Accept (poster)