Training Spatial-Frequency Visual Prompts and Probabilistic Clusters for Accurate Black-Box Transfer Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 14:35:48

Abstract: Despite the growing prevalence of black-box pre-trained models (PTMs) such as prediction API services and proprietary software, there remains a significant challenge in directly applying general models to real-world scenarios due to the data distribution gap. Considering a data deficiency and constrained computational resource scenario, this paper proposes a novel parameter-efficient transfer learning framework for vision recognition models in the black-box setting. Our framework incorporates two novel training techniques. First, we align the input space (i.e., image) of PTMs to the target data distribution by generating visual prompts of spatial and frequency domain. Along with the novel spatial-frequency hybrid visual prompter, we design a novel training technique based on probabilistic clusters, which can enhance class separation in the output space (i.e., prediction probabilities). In experiments, our model demonstrates superior performance in a few-shot transfer learning setting across extensive visual recognition datasets, surpassing state-of-the-art baselines. Additionally, the proposed method efficiently reduces computational costs for training and inference phases.

Main paper: 
 related work: 
 Visual prompting. The recent advent of foundation PTMs, as highlighted in key studies [3, 21, 22] , has significantly increased interest in deploying these models for real-world applications. Despite their potential, a notable performance drop is often observed when these general-purpose models are applied directly to specialized domains. To bridge this gap, the concept of PETL [12, 24, 36, 38] has emerged as a prominent research direction. PETL focuses on the efficient fine-tuning of large models to tailor their capabilities more closely to specific domain requirements.Visual prompting, one of promising approaches of PETL, aims to adapt the input data distribution for a specific target model by introducing learnable, dataset-specific perturbations at the pixel level [2, 5] . Following this initial approach, subsequent studies have concentrated on refining the design of VPs. For example, Enhanced Visual Prompting [30] introduces a novel approach for integrating an image with a VP, creating a synergistic blend that enhances model performance. Additionally, Diversity-Aware Meta Visual Prompting [10] advocates for the development of multiple group-specific VPs. This method is designed to address the complex issue of distribution shift, aiming to more closely align with the original data distribution of the target model, thereby improving effectiveness across varied datasets.Recently, a visual prompting in a black-box setting has been studied [20, 25] with an increasing attention towards privacy models. To the best of our knowledge, black-box adversarial reprogramming (BAR) [25] represents the initial effort to incorporate a VP into a black-box setting. This approach aims to effectively repurpose models pre-trained on natural images for applications in the medical domain. BlackVIP [20] propose an input-dependant VP design and the corresponding optimization method for black-box setting, and show a promising results on various benchmark datasets.Unlike the previous methods, several studies [9, 18, 29] have ventured into examining the impact of VPs within the frequency domain. Initial investigations [18, 29] have shown that applying visual prompting techniques to the frequency domain can significantly enhance performance across a broad spectrum of downstream tasks. These tasks include unsupervised domain adaptation for segmentation, presenting the versatility and effectiveness of frequency-based visual prompting. Furthermore, Tsao et al. [26] have found the effect of harnessing low-frequency features to boost classification performance. This approach points towards the potential strategic importance of frequency components in visual prompts.Output space alignment. In the context of general principles for black-box settings [20] , the only resources available for training are each input sample along with its corresponding prediction probabilities. Consequently, there have been few attempts to tune the output space in such environments. In contrast, in white-box scenarios, a variety of techniques can be employed, such as adding a linear layer at the end of the network (i.e., linear probing). Specifically, in the field of vision-language models [14] [15] [16] 21] , there are numerous efforts to align the output space by tuning text embeddings.Observations from research on text prompt tuning [35, 37, 38 ] serve as foundational inspirations for our cluster-based prediction refinement approach. Both CoOp [38] and CoCoOp [37] introduce methods for adjusting text prompts with learnable components, showcasing their effectiveness in achieving a more harmonious alignment between images and text. Moreover, earlier studies [35] highlight the critical role of text prompt tuning in alleviating low inter-class variance of text features, further underscoring the potential impact of such adjustments. is the corresponding class label. Given D, this paper aims to construct a visual prompter 𝑉 𝜙 to conduct PETL of a pre-trained visual recognition model 𝑃 𝜃 , as in previous works [2, 10, 30] .For our black-box problem formulation, we follow the black-box adversarial reprogramming (BAR) setting [25] , which first defines the problem of black-box visual prompting. Specifically, it is assumed that 𝑃 𝜃 is frozen and one can only access 1) each input sample (X 𝑖 , 𝑦 𝑖 ) and 2) its prediction probabilities 𝑃 𝜃 (𝑉 𝜙 (X)). As it is intractable to compute gradient through 𝑉 𝜙 , the objective𝜙 * = arg min 𝜙 E D∼(X,𝑦) L (𝑃 𝜃 (𝑉 𝜙 (X)), 𝑦)can be optimized by zeroth-order optimization [17] to find an optimal 𝜙, where L is a task-specific loss function.Overview. To further improve effectiveness when tuning a PTM in a black-box manner, we propose a novel visual prompter and first introduce an output-tuning method, where our contribution can be summarized as follows:• For effective tuning of the input space of 𝑃 𝜃 by using fewshot target data samples, we design a novel visual prompter 𝑉 𝜙 , which simultaneously provides prompts in the spatial and frequency domains. (Section 3.  
preliminaries: 
 In this subsection, we introduce preliminary concepts to facilitate a better understanding of our method.Visual prompting. In BAR [25] , its prompting method is defined by 𝑉 𝜙 (X) = Pad(X 𝑟 ) + V 𝜙 , where Pad(X 𝑟 ) is a resized & zero-padded version of X and V 𝜙 ∈ R ℎ×𝑤×𝑐 is a VP (zero-masked at center) directly added to X 𝑟 , respectively. These frame-like VPs apply trainable pixel-level perturbations to the periphery of images. Various designs of frame VPs are possible; for instance, one might downsample an image and pad the VPs outside [25, 30] , or overlap the VPs directly onto the original image [2] . Further techniques, such as those considering multiple subdomain-specific visual prompts (VPs) [10] and selecting VP sizes that vary according to dataset characteristics [27] , have also been explored. BlackVIP [20] , a recently proposed black-box visual prompting method, addresses the challenges associated with the high parameter requirements and limited performance of BAR [25] . The approach effectively resolves these issues through the implementation of an encoder-decoder architecture in the visual prompter, which can be defined by In Eq. ( 1), 𝐸 is a self-supervised feature extractor, 𝐷 𝜙 𝑑 is a decoder generating a VP of size (ℎ × 𝑤 × 𝑐), 𝜙 𝑣 is a trainable trigger vector, and 𝜖 ∈ (0, 1] is a hyperparameter.EQUATIONZeroth-order optimization. In the white-box scenario, where there is complete access to PTMs, optimizing the loss function and retrieving its gradient through back-propagation are straightforward. However, in the case of a black-box model, where only the model outputs are available, back-propagation is infeasible due to the unavailability of the gradient. In such situations, zeroth-order optimization emerges as a viable solution to this challenge. The two principal components of this method are 1) gradient estimation and 2) gradient descent using the estimated gradient.In BlackVIP [20] , the training parameters 𝜙 = {𝜙 𝑑 , 𝜙 𝑣 } are optimized by the proposed SPSA-GC algorithm, which is a modified version of simultaneous perturbation stochastic approximation (SPSA) [23] based on the principle of Nesterov's accelerated gradient [19] . By using the conventional cross-entropy loss as L, the authors conduct multi-point gradient estimation for SPSA, i.e., the 𝑡-th iteration estimated gradient for the current parameters 𝜙 𝑡 is ĝ𝑡EQUATIONwhere 𝑐 𝑖 ∈ [0, 1] is a decaying parameter and Δ 𝑠 is a perturbation vector sampled from mean-zero distributions that satisfy finite inverse momentum condition such as Rademacher and segmented uniform distribution. For more in-depth description of the SPSA-GC optimization, readers are referred to the BlackVIP method [20] .Discrete cosine transform. The discrete cosine transform (DCT) plays a crucial role in converting spatial domain data-specifically, the actual pixels of an image-into the frequency domain. Each component of the DCT output represents a specific frequency present in the original image. The organization of these frequencies is such that low-frequency components are located in the top-left corner of the DCT matrix, while high-frequency components are positioned progressively towards the bottom-right. This structured arrangement enables a clear differentiation between varying types of image information, ranging from general trends to finer details. The ability to isolate and modify these frequencies is particularly advantageous, as it allows for targeted analysis and manipulation, thus providing insights into how different image characteristics influence the behavior of pre-trained models. When performing 2D DCT on an 𝑁 × 𝑁 -sized image, the operation can be defined by the following formula:EQUATIONwhere 𝐷 (𝑢, 𝑣) is the DCT coefficient at frequencies 𝑢 and 𝑣, 𝑓 (𝑥, 𝑦) is the pixel intensity at coordinates (𝑥, 𝑦), and 𝛼 (𝑢) and 𝛼 (𝑣) are normalization factors.Recall the visual prompter design as described in Eq. ( 1). This setup shows that an encoder-decoder-based VP generation can be a more effective solution than directly tuning pixel-level perturbations in resource-constrained black-box scenarios. However, as shown in its experiments, an inappropriate choice of encoder can lead to unintended outcomes and significant variability in effectiveness across different datasets. This is especially problematic given that the architecture of 𝑃 𝜃 is not directly accessible. Moreover, 𝐸 incurs additional computational costs during training.To alleviate the potential aforementioned issue and save additional computational costs (e.g., memory and time) at training and inference phases, we design an encoder-free visual prompter. By keeping the decoder-based VP generation framework for efficiency, we aim to design a novel visual prompter capable of effectively adapting the input space of PTMs across various datasets. Inspired by the recent study [26] , which investigated that manipulation in the low-frequency domain can facilitate efficient learning of VPs in certain datasets, we consider the following aspects:• To enhance effectiveness, we incorporate a frequency-domain prompting strategy into our visual prompter. • The impact of frequency-domain VPs can vary across datasets; therefore, we make their influence adjustable for each dataset and compensate with spatial-domain VPs. • Considering a resource-constrained setting, we generate VPs in both the frequency and spatial domains in a memoryefficient manner, by using a single decoder structure.As illustrated in Fig. 1 (a). our visual prompter consists of a single decoder taking a learnable trigger vector 𝜙 𝑣 1 as input. We use additional one 𝜙 𝑣 2 to enable flexible generation of a spatial-domain VP. Denoting spatial and frequency VPs as V 𝑠 ∈ R ℎ×𝑤×𝑐 and V 𝑓 ∈ R ℎ 𝑓 ×𝑤 𝑓 ×𝑐 , respectively, our decoder 𝐷 𝜙 𝑑 isEQUATIONwhere ℎ ≫ ℎ 𝑓 and 𝑤 ≫ 𝑤 𝑓 . Based on Eq. ( 4), our visual prompting process using 𝑉 𝜙 can be described as follows:Frequency-domain prompting. Image manipulation in the lowfrequency band has become a popular strategy in adversarial attacks and adaptations [7, 13] to minimize redundant noise in the spatial domain and preserve the perceptual similarity of images. The low-frequency region, which accounts for most of an image's energy, is associated with content, whereas the high-frequency region pertains to edge and texture information.Among the frequency analysis techniques such as fast Fourier transform and discrete wavelet transform, we employ discrete cosine transform (dct) and its inversion (idct) in our VPs. Given X, downsample the image to X 𝑟 ∈ R ℎ 𝑑 ×𝑤 𝑑 ×𝑐 , where ℎ 𝑑 > ℎ 𝑓 and 𝑤 𝑑 > 𝑤 𝑓 . Then, frequency-domain prompted X 𝑟 isEQUATIONwhere 𝜎 and 𝜙 𝑠 denote the sigmoid operation and a learnable scalar parameter for scaling the strength of frequency-domain prompting, respectively. In Eq. ( 5), we pad the bottom and right sides of the dct(V 𝑓 ) with zeros to match it to the size of dct(X 𝑟 ). EQUATIONwhere Mask(V 𝑠 , X 𝑟 ) is an operation masking the center portion of V 𝑠 with zeros, matching the size of X 𝑟 . In summary, our visual prompter 𝑉 𝜙 applies both spatial-and frequency-domain VPs to each input image based on a single encoder 𝐷 𝑑 𝜙 , two trigger vectors (𝜙 𝑣 1 , 𝜙 𝑣 2 ), and a scaling parameter 𝜙 𝑠 , i.e., 𝜙 = {𝜙 𝑑 , 𝜙 𝑣 1 , 𝜙 𝑣 2 , 𝜙 𝑠 }.As mentioned, in the output space of a black-box visual recognition model, we can only obtain prediction probabilities. These probabilities tend to be computed using class-specific parameters that have been trained with source-domain data. For example, if logits are computed through the distances between class-specific prototypes and output features, each class prototype is dependent on the source-domain classes. Consequently, these prototypes may be inadequate for accurately representing target-domain classes. This inherent limitation persists even if the input space is manipulated to influence the stages just prior to the output space.Furthermore, if the class prototypes remain fixed and are not adjusted to suit different domains, they maynot provide a strong learning signal to the input space. In situations where only probability vectors are accessible, we aim to enhance the flexibility of learning by redefining class-specific prototypes. We first explore the potential for prediction refinement through probabilistic clustering, and then define auxiliary simplex prototypes, Whose training techniques will be introduced in the next subsection.Probabilistic clustering for prediction refinement. Assuming that the largest index of 𝑃 𝜃 (𝑉 𝜙 (X)) can be a sub-optimal class prediction in some cases, one can use clustering methods for prediction refinement. Since only prediction probabilities are accessible, we apply KL k-means clustering [4] with {𝑃 𝜃 (𝑉 𝜙 (X 𝑖 ))} 𝑛 𝑖=1 , which is a probabilistic clustering method for simplex data.By clustering {𝑃 𝜃 (𝑉 𝜙 (X 𝑖 ))} 𝑛 𝑖=1 into 𝐾 clusters based on the KL divergence metric, where 𝐾 is the number of target classes, we obtain class-wise simplex means m = {𝑚 𝑘 } 𝐾 𝑘=1 . Then, a refined version of 𝑃 𝜃 (𝑉 𝜙 (X)) can be formulated byEQUATIONwhere -KL(𝑃 𝜃 (𝑉 𝜙 (X))||𝑚 𝑘 ) works as a class-wise logit.Auxiliary simplex prototypes for training. The refinement strategy of Eq. ( 7) is a post-hoc method, i.e., one can obtain informative class-wise simplex means {𝑚 𝑘 } 𝐾 𝑘=1 after finishing the training process of 𝑉 𝜙 . To incorporate the principle into the training phase for a further enhancement, we introduce auxiliary simplex prototypes a = {𝑎 𝑘 } 𝐾 𝑘=1 . In other words, one can useEQUATIONat training time to improve the class prediction accuracy of Eq. ( 7).To obtain initial prototypes a 0 = {𝑎 0,𝑘 } 𝐾 𝑘=1 , we conduct clustering with initial classification prediction probabilities {𝑃 𝜃 (X 𝑖 )} 𝑛 𝑖=1 . In the following section, we introduce our training strategies including the updating rules of the auxiliary simplex prototypes.In the previous subsection, we introduced the concept of auxiliary simplex prototypes. These prototypes enable us to fine-tune the output space of PTMs in conjunction with visual prompting at the input space. We anticipate that introducing flexibility in the output space may yield synergistic effects during VP training. Consequently, we have carefully designed training strategies that minimize conflicts between tuning in the input and output spaces. This consideration is crucial because we cannot process through PTMs in a straightforward manner; alternating updates between the input and output spaces separately may lead to unstable training.Objective function L. When tuning a pre-trained visual recognition model, one can obtain {𝑃 𝜃 (𝑉 𝜙 (X 𝑖 ))} 𝑛 𝑖=1 and {𝑃 a (𝑉 𝜙 (X 𝑖 ))} 𝑛with the training data {(X 𝑖 , 𝑦 𝑖 )} 𝑛 𝑖=1 , prediction probabilities of the black-box vision-language model and their refined version based on Eq. ( 8), respectively. Using the conventional cross-entropy loss function, we define L cls = L CE ({𝑃 𝜃 (𝑉 𝜙 (X 𝑖 )), 𝑦 𝑖 } 𝑛 𝑖=1 ) and L aux = L CE ({𝑃 a (𝑉 𝜙 (X))} 𝑛 𝑖=1 ) for classification. In addition to L cls and L aux , we employ an additional loss function to enhance the alignment between the sets {𝑃 𝜃 (𝑉 𝜙 (X 𝑖 ))} 𝑛 𝑖=1 and {𝑃 a (𝑉 𝜙 (X 𝑖 ))} 𝑛 𝑖=1 . For this purpose, we adapt the intra-class relation loss [11] , which is proposed to share the relational dynamics within each class in knowledge distillation. In our scenario, the relation loss L intra focuses on preserving the relative rankings or preferences of instances within a class, rather than replicating exact probability values from {𝑃 𝜃 (𝑉 𝜙 (X 𝑖 ))} 𝑛 𝑖=1 to {𝑃 a (𝑉 𝜙 (X 𝑖 ))} 𝑛 𝑖=1 . Unlike traditional methods that require strict output matching, L intra adopts a more relaxed approach, prioritizing the correlation of predictions. In summary, our overall loss function is defined byEQUATION)Updating rules of a. Following the zeroth-order optimization method of the recently proposed black-box VP [20] , we use the principle of the SPSA-GC algorithm to train our visual prompter 𝑉 𝜙 . Instead of updating a by computing gradients, which may not align with the gradients computed in the SPSA-GC algorithm for input-space VP training, we continuously update auxiliary simplex prototypes following the visual prompters' subsequent updates.Starting from the initial prototypes a 0 = {𝑎 0,𝑘 } 𝐾 𝑘=1 , we update the prototypes by selectively obtaining the probabilistic distributions that result in a diminished loss direction.To be specific, we discern between the outcomes of bidirectional perturbations (𝜙 𝑡 ±𝑐 𝑡 Δ 𝑠 in Eq. ( 2)), and incorporate the probabilistic distribution that have lesser loss, thus having superior and accurate probabilistic representation. Averaging class-wise prediction probabilities, we refine the prototype vector in a weighted sum manner. Denoting 𝑝 𝑡,𝑘 as the average of the probability predictions for the 𝑘-th class training samples at the 𝑡-th iteration, we computeEQUATIONwhere L1 is 𝑙 1 -normalization to maintain the probabilistic integrity of auxiliary simplex prototypes.Gradient surgery. Although SPSA-GC has achieved faster convergence during training, the black-box optimization algorithm requires careful hyperparameter tuning to acquire desired performance. Noting that the multi-point gradient estimation of Eq. ( 2) simply averages 𝑆 estimated gradients to build a more stable gradient, we incorporate the principle of gradient surgery for robustness.Inspired by a multi-task learning baseline [34] , which project a conflicting gradient onto the normal plane of a gradient of other task, we seek to refine the estimation of a conflicting gradient.Formally, let ĝ𝑡 1 , ĝ𝑡 2 be a pair of sub-gradients among the the 𝑆 gradient estimations obtained concurrently at the same iteration step in Eq. (2). Then, the gradient projection process between two sub-gradients can be formulated byEQUATIONWe update each gradient estimation relative to other estimations to ensure that the revisions encompass all possible interactions between the gradients under consideration. 
experiments: 
 In this section, we describe the experimental settings (Section 4.1) including implementation details, datasets, and baselines. Before we present our main experimental results, we first introduce exploratory results related to our visual prompter (Section 4.2). Afterwards, we compare the performance of our model with baseline methods on benchmark and synthetic datasets (Section. 4.3), and then show performance improvement by each contribution (Section. 4.4).PTMs and Baselines. As a representative pre-trained visual recognition model, we employ the CLIP zero-shot classifier [21] , which can be extended across various numbers of classes and multimodal scenarios, throughout our experiments. Specifically, along with each image X, CLIP typically takes a set of text prompts based on a hand-crafted template (e.g., "a photo of a [CLASS]."). The classifier then processes these prompts to extract text features, utilizing its tokenizer and text encoder. These extracted features are instrumental in generating class prediction probabilities.For the baseline methods, we consider CLIP zero-shot classification (ZS), black-box adversarial reprogramming (BAR) [25] , a black-box version of VP [1] , and BlackVIP [20] .Implementation details. As aformentioned, we utilize CLIP VIT-B/16 [21] for a frozen PTM 𝑃 𝜃 , which is capable of strong zeroshot generalization. As the default input size of 𝑃 𝜃 is (224 × 224 × 3), we resize each image into (112 × 112 × 3) or (192 × 192 × 3) according to its original resolution.We adopt the decoder design from BlackVIP [20] , in which lowfrequency visual prompts (VPs) of size (56 × 56 × 3) are generated by applying a 1 × 1 convolution operation to its intermediate layer.For the scaling parameter 𝜙 𝑠 , we initialize it at 𝜙 𝑠 = -5 so that 𝜎 (𝜙 𝑠 ) ≈ 0, thereby minimizing the impact of frequency-domain In this subsection, we outline two foundational observations that underpin the development of our visual prompter design. These observations are essential for guiding the architectural decisions and functional strategies we implement. Our aim is to craft a visual prompter that is not only effective but also aligns with our goals of enhancing model performance.Frequency-domain prompting. Recall the framework of Black-VIP [20] , which makes spatial-domain VPs of the same size as each image based on an encoder-decoder structure. To design an effective visual prompter without encoder, we first construct three variants of BlackVIP (spatial-domain VP with an encoder) and test them with a number of datasets, SVHN, Oxford-Pets, EuroSAT, and RESISC.Here, the variants are 1) spatial-domain VP without encoder, 2) lowfrequency VP with an encoder, and 3) low-frequency VP without encoder, where we follow the experimantal settings of BlackVIP. In Fig. 2 , we illustrate various visual prompting methods for a better understanding of spatial-and frequency-domain prompting. Table 1 illustrates that frequency-domain prompting can successfully adapt to certain datasets, such as SVHN, even without a self-supervised encoder, although it may yield suboptimal results in other contexts. To leverage the benefits of both spatial and frequency domain prompting without the drawbacks of each interfering, we have developed our visual prompter as shown in Fig. 1 . In this design, we introduce a learnable scaling parameter, 𝜙 𝑠 , which enables 𝑉 𝜙 to dynamically adjust the impact of low-frequency prompting. This allows for customized application, tailored to maximize effectiveness in various contexts.Trigger vectors. To enable effective tuning, we introduce a visual prompter designed to produce VPs in both the spatial and frequency domains. Our approach involves training a singular decoder that is proficient in generating VPs across these two distinct domains simultaneously. This is achieved by employing two trigger vectors, where an additional trigger vector is incorporated into the decoder after extracting features for frequency-domain prompting. Such strategy of Eq. ( 4) is visually represented in Fig. 1(a) .We evaluated the impact of incorporating an additional trigger vector in dual-domain visual prompt generation by comparing training outcomes with and without it. Anticipating that the trigger vector would provide increased learning flexibility and enhance adaptability across different domains, our findings confirmed that its inclusion significantly enhanced both performance and the overall learning process compared to the version without it.Transfer learning on benchmark datasets. Utilizing 14 benchmark datasets, we implement a few-shot approach as described in previous studies [2, 37, 38] to conduct our evaluations, with the results detailed in Table 2 . Our findings suggest that our method consistently outperforms existing black-box visual prompting techniques in terms of overall performance. Additionally, in certain cases, it achieves results that are comparable to those of white-box methods, thereby demonstrating the effectiveness and adaptability of our approach across diverse testing environments.Our method has demonstrated considerable superiority across multiple datasets by substantial margins, underscoring the effectiveness of our visual prompter, even in the absence of an encoder integration. This enhanced performance is particularly evident when compared to various baseline variants, as detailed in Table 1 . These comparisons effectively highlight the benefits of our hybrid approach, which combines spatial and frequency-domain prompting, thereby validating the design's efficacy in boosting model performance across diverse testing environments.Transfer learning on synthetic datasets. To evaluate the robustness of visual prompting methods against distribution shifts, we follow the experimental protocol outlined by Oh et al. [20] , using datasets such as Biased MNIST [1] and Loc-MNIST. This approach allows us to systematically assess the effectiveness of these methods in varying data environments.Biased MNIST is designed to examine generalization ability under color bias shift. At training, every digit is assigned a distinct background color that is closely linked to its class label, where the correlation strength is set by the value of 𝜌 ∈ [0, 1]. In test phase, the correlation ratio is inverted to 1 -𝜌. Table 3 shows that ourTo assess the individual contributions of the components in our black-box visual prompting method, we conducted an ablation study. Beginning with the basic spatial-frequency visual prompting, we gradually incorporated the proposed techniques: prediction refinement, auxiliary simplex training, intra-class relation loss, and gradient surgery. This systematic approach helped us to meticulously analyze and comprehend the influence of each component on the overall performance.Referring to Table 2 , it is evident from Table 4 that our hybrid visual prompter achieves higher classification accuracy compared to existing state-of-the-art black-box tuning methods. Additionally, it is clear that employing prediction refinement through KL k-means clustering consistently enhances recognition accuracy. The auxiliary simplex training strategy further improves accuracy, particularly in certain datasets (e.g., DTD). Furthermore, the intraclass relation loss and gradient surgery strategies have proven effective in stabilizing the training process.Computational costs. In contrast to BlackVIP [20] , a state-ofthe-art black-box visual prompting technique that employs an encoder-decoder architecture, our approach achieves greater efficiency during training, due to the omission of the self-supervised encoder. Through experimental evaluation on the SVHN and Eu-roSAT datasets, utilizing an NVIDIA RTX A5000 GPU, our method showed a more resource-efficient footprint, consuming 2970MB of GPU memory and requiring only 0.225 seconds per iteration. This is in comparison to the previous method, which necessitated 4223MB and 0.374 seconds for the same metrics, respectively. This efficiency gain underscores the practical advantages of our methodology. 
conclusion: 
 In conclusion, our work has successfully implemented a novel transfer learning framework for large PTMs that effectively bridges the data distribution gap in resource-constrained black-box API environments. By integrating spatial-frequency hybrid visual prompts with cluster-based prediction refinement, we have achieved significant improvements in few-shot transfer learning performance across a variety of datasets. This approach also results in reduced computational costs. Our methods enhance the adaptability and efficiency of PTMs in practical applications, while also fostering new research opportunities in transfer learning and model adaptation. The progress documented in this paper sets the stage for further investigation and advancement in the field, promising to enhance the utility and reach of machine learning across multiple scenarios.  
: 
 to downstream tasks by tuning only a small number of additional parameters, can be an economic and effective solution.In addition to the challenge of dealing with data deficiency, the implementation of transfer learning methods in real-world industrial scenarios poses significant challenges. After great success of ChatGPT 1 , there has been a noticeable trend towards the release of access-restricted PTMs in the form of APIs or software packages. Essentially, the intricate details of recent enterprise-level ML models, including their architecture, weights, training data, and techniques, are often kept confidential. While such API services offer efficient solutions for users with limited resources-such as small companies or individuals who face computational limitations when computing gradients through large-scale ML models-it remains challenging to adapt these opaque, black-box PTMs to domain-specific tasks that require tailored applications.To address this challenge, we propose a novel tuning methodology for visual recognition models that adheres to the fundamental principles of operating within a black-box environment. In this setting, the method utilizes only the available input-output responses (given input data samples and their corresponding prediction probabilities) for tuning. Given that the input-output response represents the only accessible information, one may posit that strategically optimizing the input space can be an effective manner for enhancing model performance within such constraints. Although it may be feasible to tune the output space in white-box scenarios (e.g., by applying a linear layer to the penultimate layer), these methods can become intractable in black-box settings. In this context, one can primarily focus on the concept of visual prompting [1, 30] , which aims to learn pixel-level perturbation with the goal of adjusting the input space to better match the target distribution.To date, studies focusing on visual prompting techniques in the black-box setting, where all network parameters and gradients are not directly accessible, are comparatively limited when contrasted with the white-box setting. Notably, only a few investigations, such as those documented in [20, 25] , have delved into this area. The black-box adversarial reprogramming (BAR) method, introduced in [25] , marks a pioneering effort in emphasizing the importance of input-space adaptation for black-box PTMs (PTMs). This method employs frame-shaped visual prompts (VPs) to effectively facilitate the adaptation process. In a subsequent development, BlackVIP [20] proposes the generation of input-dependent VPs through the use of an auxiliary encoder, which is specifically designed to extract features from images. This approach underscores the versatility and adaptability of visual prompting, significantly enhancing compatibility of models within specialized domains.Building upon existing frameworks, our research aims to achieve two challenging objectives, 1) effective visual prompting and 2) novel output-space tuning strategies, respectively. For the first goal, we design a spatial-frequency hybrid visual prompter, where its primary motivation is to effectively manipulate images within the frequency domain. Techniques such as low-frequency perturbations, which are known to maintain the structural integrity of images, have demonstrated promising results in various studies related to domain adaptation [28, [31] [32] [33] and adversarial attacks [7, 13] .The efficacy of such frequency-domain manipulations, however, may vary across different datasets. To enhance the performance of frequency-domain VPs, our approach also integrates spatial-domain VPs. This integration is carefully managed to avoid conflicts between the two domains, thereby resulting in an effective adaptation strategy for PTMs within the input space across various target datasets. Our novel method of merging visual prompts from both spatial and frequency domains is accomplished through the creation of a spatial-frequency hybrid visual prompter. Notably, this integration is achieved without the reliance on an auxiliary encoder, illustrating the potential for improved adaptation by synergistically utilizing both spatial and frequency information.The motivation of our second goal, output-space tuning, stems from the observation that prediction probabilities generated by PTMs frequently show minimal differences across classes when applied to a new domain. This similarity complicates the extraction of a distinct and robust learning signal, presenting a challenge for effective domain adaptation. In white-box PETL for PTMs, two strategies can be baseline approaches to address such problems: linear probing, which involves applying a learnable linear layer to the logits from the penultimate layer, and text prompt tuning for vision-language models. Nevertheless, the restrictive nature of the black-box setting makes it challenging to modify the output space directly. As a workaround, we propose trainable cluster-based prediction refinement as a strategy to enhance the output space that might otherwise be less distinguishable. This method specifically involves the utilization of auxiliary simplex prototypes to disentangle and reorganize the output space, making it more structured and easier to differentiate among classes.In our experiments, we rigorously assess the performance of our model through few-shot transfer learning across a wide range of benchmark datasets, thereby showcasing the robustness and effectiveness of our proposed methodologies. Additionally, we conduct a thorough analysis of each component of our method to provide evidence supporting its architectural design. Our experiments further include an examination of computational overheads, specifically targeting memory efficiency and the reduction in training time, to highlight the practical advantages of our approach. 
 
 Appendix:  


Decision: this paper addresses the problem of parameter efficient transfer learning. The 4 reviewers appreciate the originality of the paper as well as the quality of the answers in the rebuttal material. Authors made there several promises which should be implemented in order to strongly enhance the quality of the manuscript -- the questions of reviewers were instrumental here. All reviewers judge that this contribution is too unimodal in nature, but it is understood that the contribution can be profitable to the MM  community.

the quality of the work, the settings and the originality of the approach push the recommendation to ACCEPT, poster.

Reviews: review: paper_summary: paper_summary: This paper proposes a parameter-efficient transfer learning method, containing two training strategies: 1) aligning input space with target distribution by both spatial and frequency domains; 2) enhancing the class separation in the output space by probabilistic clustering. Experiments on standard benchmark show the performance of the proposed method.
strength_weakness: strength_weakness: 1. The proposed method shows good average performance against other considered methods.

2. The paper is well-structured and easy to follow.
limitations: limitations: 1. In Section 3.2, the paper argues the drawbacks of including encoders, thus choosing to only include decoders in the model. More justifications are needed for such a choice. 

2. For 'updating rule of a', the choice on normalisation method and coefficient for weighted sum should also be analysed. 

3. The setup solely considers the 16-shot learning framework and ViT-B/16 as the backbone. Additional experiments are required to show the validity and effectiveness of the method.

4. The novelty is somehow limited. The method is proposed by modifying existing methods with techniques that are used in other areas.
paper_summary: paper_summary: The paper introduces a new transfer learning framework for black-box pre-trained models (PTMs), aimed at addressing data distribution gaps and computational limitations. It uses spatial-frequency hybrid visual prompts to align PTMs' input space with the target data distribution and employs a probabilistic clustering method to enhance class separation in the output space. Evaluations on various visual recognition datasets demonstrate improved performance in few-shot transfer learning while reducing computational costs.
strength_weakness: strength_weakness: 1. Enhancing black-box PTMs is a topic worthy of research.

2. The framework is computationally efficient, a critical factor for applications with constrained resources.

3. The paper includes extensive experiments on multiple datasets, demonstrating the robustness and effectiveness of the proposed method.
limitations: limitations: 1. The spatial prompt and the frequency prompt are both generated from a single decoder. What is the impact on the final results when using two decoders instead? Will generating them from a single decoder cause any conflicts?

2. In line 572, "Updating rules of a," please add the definition of 'a.' This will enhance the readability of the paper.

3. In Eq. (10), is the selection of the hyper-parameters 0.9 and 0.1 sensitive?

4. Both [a] and [b] use visual prompts for transfer learning. Can the authors compare the impact of different visual prompt placements on the model's performance?

[a] Decorate the newcomers: Visual domain prompt for continual test time adaptation

[b] Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction
paper_summary: paper_summary: The paper introduces a novel parameter-efficient transfer learning framework for vision recognition models in a black-box setting. By aligning input space and generating visual prompts, along with utilizing probabilistic clusters, the model demonstrates superior performance in few-shot transfer learning scenarios. Experimental validation on extensive visual recognition datasets showcases the effectiveness of the proposed method.
strength_weakness: strength_weakness: (1) The paper introduces a novel transfer learning framework for large pre-trained models (PTMs) that addresses the data distribution gap in resource-constrained black-box API environments.
(2) Experimental results on synthetic datasets like Biased MNIST and Loc-MNIST, as well as benchmark datasets like SVHN, EuroSAT, and DTD, demonstrate the effectiveness of the hybrid visual prompting method and its enhancements.
limitations: limitations: (1)While the paper briefly mentions related work on visual prompting and transfer learning, a more comprehensive review and analysis of existing literature in the field would provide better context for the proposed framework. A deeper discussion of how the current work builds upon or differs from prior research would strengthen the paper's contribution.
(2)The experimental details are incomplete, with a lack of specific implementation details for training, making it difficult for other researchers to reproduce the findings presented in the paper.
(3)In addition to providing VIT-B/16 as the baseline performance, the paper should also include the performance of baseline models such as RN101 and ViT-B/32 to adequately demonstrate the effectiveness of the proposed method.
(4)Although the paper shows some comparative experiments comparing performance with and without encoders and in different domains, it lacks comparison with other state-of-the-art black-box transfer learning methods.
(5)The method's complexity and computational requirements are not thoroughly discussed, which could impact scalability. Furthermore, the authors should provide computational cost comparisons with other methods.
paper_summary: paper_summary: This manuscript introduces a novel transfer learning framework tailored for vision recognition models, particularly effective in black-box scenarios. The authors adeptly tackle the challenge of deploying generalized models into real-world contexts, where discrepancies in data distribution are prevalent. The proposed framework is distinguished by its integration of two innovative training methodologies: spatial-frequency hybrid visual prompting and probabilistic cluster-based prediction refinement. The empirical findings underscore the framework's exceptional performance in few-shot transfer learning environments and across a diverse array of visual recognition datasets. Furthermore, the authors highlight the framework's commendable efficiency, significantly reducing computational expenses during both the training and inference phases.
strength_weakness: strength_weakness: 1. The paper presents an innovative, parameter-efficient transfer learning framework that leverages spatial-frequency hybrid visual prompts and probabilistic clusters, effectively addressing a critical issue in the field of visual recognition.

2. The methodology's incorporation of zeroth-order optimization and discrete cosine transform (DCT) showcases a significant technical advancement in the domain.
limitations: limitations: 1. The Introduction section lacks the precision and clarity necessary to effectively convey the paper's research objectives and contributions, which may impede the reader's understanding and appreciation of the work's significance.

2. While the paper introduces a transfer learning framework that aims to be parameter-efficient, it falls short of clearly delineating how its spatial-frequency hybrid visual prompts and probabilistic clusters offer a significant advancement over existing methodologies in visual recognition.

3. The manuscript's failure to extend beyond the current paradigms of visual prompting and cluster-based prediction refinement is a notable weakness. The paper does not clearly articulate how the new training techniques contribute to the evolution of these frameworks.

4. A significant shortcoming of the manuscript is the absence of visual analysis techniques, such as Grad-CAM analysis and prompt visualization. These methods are indispensable for offering insights into the model's decision-making mechanisms and for understanding the importance of the framework's components. The lack of such visualizations detracts from the paper's ability to foster interpretability and transparency, which are crucial for establishing credibility and understanding among the research community and potential end-users.

5. A potential limitation of the proposed methods could be the complexity of implementation, especially for users without a deep understanding of transfer learning or visual recognition models. Ensuring clear documentation and support for code replication would be beneficial.

I will revise my ratings based on other comments and the author's rebuttals.