Bayesian-guided Label Mapping for Visual Reprogramming

Authors: chengyi  cai
zesheng  ye
lei  feng
jianzhong  qi
feng  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:23:28

Abstract: *Visual reprogramming* (VR) leverages the intrinsic capabilities of pretrained vision models by adapting their input or output interfaces to solve downstream tasks whose labels (i.e., downstream labels) might be totally different from the labels associated with the pretrained models (i.e., pretrained labels). 
When adapting the output interface, label mapping methods transform the pretrained labels to downstream labels by establishing a gradient-free one-to-one correspondence between the two sets of labels.
However, in this paper, we reveal that one-to-one mappings may overlook the complex relationship between pretrained and downstream labels. Motivated by this observation, we propose a ***B**ayesian-guided **L**abel **M**apping* (BLM) method. 
BLM constructs an iteratively-updated probabilistic label mapping matrix, with each element quantifying a pairwise relationship between pretrained and downstream labels.
The assignment of values to the constructed matrix is guided by Bayesian conditional probability, considering the joint distribution of the downstream labels and the labels predicted by the pretrained model on downstream samples. Experiments conducted on both pretrained vision models (e.g., ResNeXt) and vision-language models (e.g., CLIP) demonstrate the superior performance of BLM over existing label mapping methods. The success of BLM also offers a probabilistic lens through which to understand and analyze the effectiveness of VR.
Our code is available at https://github.com/tmlr-group/BayesianLM.

Main paper: 
 introduction: 
 Repurposing pretrained models from data-rich domains [6, 28, 58] has emerged as an effective strategy to address downstream tasks without re-training a task-specific model. For visual tasks, visual reprogramming (VR) [3, 4, 48, 51] -also called adversarial reprogramming [12, 38, 47] repurposes a pretrained model for downstream tasks without changing the model. In particular, VR (full task setup detailed in Appendix A) modifies the model's input interface by adding trainable noise patterns to the images of downstream tasks. Since pretrained and downstream tasks typically have distinct label spaces, a label mapping (LM) function is needed to map outputs of the pretrained models to downstream labels. Often, existing VR methods adopt a gradient-free one-to-one LM [4, 12, 47] , avoiding the computational cost of training fully-connected output layers through backpropagation.However, we find that a one-to-one LM overlooks the complex many-to-many relationship between pretrained and downstream labels, which may limit the performance of VR. In Figure 1 , we repurpose a model pretrained on ImageNet [45] for downstream classification tasks using a one-to-one LM strategy [4] , and present statistical results. The two subfigures Figure 1a and Figure 1b , illustrate drawbacks from the perspectives of individual images and the entire dataset, respectively. Figure 1a shows the distribution of logits (i.e., model output before the softmax layer) for the most likely An ImageNet-pretrained classifier is reused in downstream tasks. In (a), images 'Dog' and 'Osteospermum' from downstream tasks are mapped into only one pretrained label, respectively, ignoring other probabilities. In (b), the distribution of [predicted pretrained label y S , ground-truth downstream label y T ] pairs reveals the existence of suboptimal solutions, where 'Automobile' cannot be paired with the optimal pretrained label 'Moving Van', which has already been mapped to 'Truck'.predicted pretrained labels of two images from downstream tasks: a 'Dog' image from CIFAR10 [30] and an 'Osteospermum' image from Flowers102 [40] . For the 'Dog' image, multiple pretrained labels like 'Chihuahua', 'Basenji'-subclasses of dogs-receive high logits. Similarly, for the 'Osteospermum' image, pretrained labels such as 'Sea Urchin', 'Daisy', which share similar features, also score high. Despite these connections, the one-to-one LM retains only the label with the highest logit, suggesting the probabilities of other related labels are ignored. Figure 1b shows the frequency distribution of the predicted pretrained labels and the ground-truth downstream labels of downstream samples, with the diagonal representing the results derived from one-to-one LM. The 'Automobile' class from CIFAR10, for example, can no longer be paired with the optimal pretrained label 'Moving Van', which has already been greedily mapped to the label 'Truck', implying suboptimal label assignments.The above observation motivates us to go beyond these binary mappings. In Section 3, we replace the one-to-one LM function with a probabilistic LM matrix. Each matrix element is a real number that quantifies the relationship between a pretrained label and a downstream label, updated iteratively during VR optimization. This allows predictions for each downstream sample to consider diverse contributions from all pretrained labels, enabling a flexible many-to-many mapping strategy.Specifically, we present Bayesian-guided label mapping (BLM) in Section 4, which assigns values to elements in the probabilistic LM matrix based on Bayesian conditional probabilities, derived from the joint distribution of the predicted pretrained labels on downstream tasks and the groundtruth downstream labels. We further extend BLM to BLM+, which aggregates top-K predicted probabilities instead of using a single predicted label when estimating the joint distribution, accounting for uncertainty in the predictions. We also provide a theoretical analysis that justifies the potential of probabilistic many-to-many LM to outperform deterministic one-to-one LM.To show the effectiveness of BLM, experiments are conducted on 12 widely used datasets, with BLM and BLM+ being applied to different input VR methods-padding and watermarking-on pretrained ResNet and ResNeXt (see Section 5) . The ablation study and parameter analysis are also included, along with visualization results and discussions of why VR is effective. BLM and BLM+ are also applied to vision-language models (see Appendix L) to demonstrate their general applicability.In summary, both theoretical analysis and empirical findings (Tables 1 2 ) provide compelling evidence that BLM and BLM+, grounded in Bayesian principles, facilitate VR to leverage pretrained knowledge for diverse downstream tasks. Beyond performance improvement, BLM and BLM+ offer insights into understanding the effectiveness of VR (Figures 3 4 ): revealing the relations between pretrained and downstream label spaces may guide future studies into more interpretable VR methods. 
related works: 
 Model Reprogramming. Among cutting-edge transfer learning methods (see Appendix B), model reprogramming introduces an efficient learning framework for adapting models pretrained on large-scale data to downstream tasks constrained by limited resources [7] . By changing the input or output interfaces (i.e., input or output space) purposefully, while preserving the integrity of the pretrained model, knowledge can be reused on new tasks, sidestepping exhaustive finetuning of the model.Many recent studies focus on repurposing diverse pretrained models for downstream tasks, including pretrained vision models [1, 4, 38, 47, 48, 51] such as ResNet [17] and ViT [11] , language models [15, 49] such as BERT [24] , acoustic [21, 59, 60] and graph models [23] . Such repurposing encompasses several types: cross-modal (e.g., from voice to time-series [60] , or vision to text [38] ), different tasks within the same modality (e.g., from image classification to out-of-distribution detection [51] ), and different domains within the same task (e.g., from ImageNet to medical images [47] ).Prompting and Input VR. Prompting incorporates meticulously designed prompts (additional parameters) into pretrained models with specific architectures to utilize pretrained models in downstream tasks. Leveraging ViT, VPT [22] integrates prompts alongside image embeddings, while EEVPT [16] further enhances VPT by embedding parameters within self-attention layers. TransHP [52] additionally learns prompt tokens to encode coarse image categories. In vision-language models such as CLIP [44] , besides text-prompting methods such as CoOP [67] and CoCoOP [66] , models like MaPLe [25] also learn layer-specific mapping functions that bridge vision and text.Slightly different from prompt tuning, input VR offers a model-agnostic approach by introducing trainable noise to images in the input space before feeding those images into pretrained models. This process does not impact the visual effect of the images. Two prevalent techniques are paddingbased VR and watermarking-based VR. Padding-based models [4, 12, 47, 48] preserve the integrity of images while introducing trainable noise patterns to the outer frames around images, whereas watermarking-based models [1, 3, 41, 51] train noise patterns that overlay the images.Output Mapping for VR. Because pretrained labels and downstream labels are often different, relying solely on input VR may be insufficient for downstream tasks. To bridge this gap, output mapping methods are introduced to facilitate alignment between different label spaces. Mainstream approaches include deep learning-based and statistical inference-based (i.e., gradient-free) LM methods. Deep learning-based methods insert a learnable fully connected layer to connect pretrained and downstream labels [27, 48] . However, for tasks with large label spaces, the additional model layers would result in extra training costs, potentially canceling the efficiency advantages of VR.As for gradient-free LM methods, random label mapping (RLM) [12] establishes mappings between an equal number of randomly selected pretrained labels and downstream labels, masking out other unused ones. Frequent label mapping (FLM) [47] selects optimal one-to-one mappings using a greedy approach based on the number of pairs between pretrained and downstream labels. Iterative label mapping (ILM) [4] extends FLM by updating mappings at each epoch, refining the output label mapping as input VR patterns evolve. As depicted in Figure 1 , these one-to-one mappings overlook potential probabilities and lead to suboptimal solutions. We propose BLM to address these issues.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations are discussed in Appendix H.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 Problem Setup. Consider a pretrained task with input and output variables X S and Y S , jointly defined over X S × Y S , where X S ⊆ R dS has the input dimensionality d S and Y S = {1, . . . , k S }.We have a pretrained classifier f pre : X S → R kS producing a logits vector f pre (x S ) ∈ R kS for each x S ∈ X S . For a downstream task with input and output variables X T and Y T defined over X T × Y T , where X T ⊆ R dT has the input dimensionality d T and Y T = {1, . . . , k T }, VR seeks to adapt f pre to the downstream task without modifying its parameters. To achieve this, VR introduces two functions: 1) input VR function f in (•|θ) : X T → X S with learnable parameters θ that converts downstream inputs for compatibility with f pre ; and 2) output LM function f ω out (•) : R kS → R kT that aligns the output logits of f pre with the downstream label space by a transformation ω. Concretely, given a training dataset D T = {(x T i , y T i )} n i=1 with n training samples drawn from X T × Y T for the downstream task, the training objective of VR can be formulated as:EQUATIONwhere ℓ is a loss function, and f ω out • f pre • f in denotes the composition of input VR, pretrained model and output LM. In this study, we focus on gradient-free LM, where f ω out does not introduce additional trainable parameters but strategically leverages f in and f pre to determine ω.Modeling Existing LM. As mentioned, f ω out serves to find a mapping between each y S ∈ Y S and y T ∈ Y T . This can be achieved by constructing an output label transformation ω such that for each downstream sample x T i , its label ŷT i is predicted by arg max softmax(ỹ T i ), with:EQUATIONwheref (x T i ) is shorthand for (f pre • f in )(x T i ; θ). ω can be updated iteratively [4] with input VR. A deterministic one-to-one relation between Y S and Y T implies only a single "correct" y S ∈ Y S exists for each y T ∈ Y T . Formally, ω in Eq. ( 2) is a binary matrix, where just a single element ω j,k is set to 1 in each column of ω (i.e., ω ∈ {0, 1} kS×kT satisfying kS j=1 ω j,• = 1). Our Probabilistic LM. Considering aforementioned drawbacks of one-to-one mappings, we propose a probabilistic LM for VR, assigning real values to all elements in ω (i.e., ω ∈ [0, 1] kS×kT satisfying kS j=1 ω j,• = 1). Each element ω y S ,y T quantifies the relationship between y S ∈ Y S and y T ∈ Y T . This acknowledges contributions from all pretrained labels for the prediction of downstream samples. The flexible many-to-many LM implies the inherent complexity in label correspondence. In Section 4, we investigate how to assign values to our probabilistic LM based on Bayes' theorem.4 Bayesian-guided Probabilistic Label Mapping (BLM)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
method demonstration: 
 Interpreting p(Y T |X T ). The objective of VR is to maximize p(Y T |X T ) defined over the downstream task space. By using the law of total probability, we can express p(Y T |X T ) as:EQUATIONMirroring the structure of Eq. ( 2), Eq. ( 3) enables us to estimate p(Y T |X T ) with the i.i.d observationsEQUATIONwhere ➀ denotes the predicted probability of pretrained label y S for input x T i , obtained from f pre •f in . Essentially, ➀ can be viewed as the standard input VR and is orthogonal to the LM methods employed; ➁ represents the probability that the true downstream label y T i is mapped from the predicted y S and input x T i , which amounts to estimating the output label transformation ω ∈ [0, 1] kS×kT . Since ➀ is independent of output LM, the focus now shifts to estimating ➁.Estimating ω y S ,y T Using Conditional Probability. Since ω y S ,y T is used to quantify the contributions from pretrained label y S to downstream label y T , we can associate it with the conditional probability:EQUATIONBy applyingf pre • f in to D T , we can empirically estimate the joint distribution of p(Y T = y T , Y S = y S |X T ), then obtain p(Y S = y S |X T ) = y T ∈Y T p(Y T = y T , Y S = y S |X T), and substitute them into Eq. (5) . Two strategies, BLM and BLM+, are presented for these estimations in this paper. To help understanding, we include a simple example to illustrate the estimation of p(YT = y T , Y S = y S |X T ) and p(Y S = y S |X T ) in Appendix C. BLM. Let f (x T i ) ≡ (f pre • f in )(x T i ; θ)denote the predicted logits obtained from the pretrained model for a given input x T i . We define ŷS i = arg max y ′ ∈Y S f (x T i ) y ′ to be the predicted pretrained label for x T i and 1{•} to be the indicator function. Starting with the joint distribution p(Y T = y T , Y S = y S |X T ), we could intuitively count the frequency of (ŷ S i = y S ∧ y T i = y T ) to estimate: EQUATIONStep 3 PredictedStep 4Step 2 Then, the true labels (of y T ) and predicted labels (of y S ) are used to estimate ω BLM or ω BLM+ . Next, using ω BLM or ω BLM+ that reweights output logits of pretrained models for the downstream labels, the predicted results can be derived. Finally, backpropagation is performed to update the input VR.For p(Y S = y S |X T ), in addition to summing up Eq. ( 6) for y T ∈ Y T , we add Laplace smoothing coefficient λ to ensure the denominator of Eq. ( 5) being non-zero, with k S being the size of Y S :EQUATIONSubstituting Eq. ( 7) and Eq. ( 6) back to Eq. ( 5) yields the estimation of ωy S ,y T to be pBLM (Y T = y T |Y S = y S , X T ). After column-wise sum normalization of ωy S ,y T to satisfy kS j=1 ω j,• = 1 (as formulated in Section 3), we obtain the final probabilistic LM, denoted as ω BLM .BLM+. Recall that BLM estimates p(Y T = y T , Y S = y S |X T ) by frequency-counting based on a single most likely predicted label. However, this strategy disregards other high-ranking predictions that could offer valuable information. Thus, we introduce BLM+, an extension of BLM that considers top-K predicted probabilities of the pretrained model for the estimation of p(Y T = y T , Y S = y S |X T ). Rather than relying solely on the tally, BLM+ aggregates probabilities for samples where y S ranks among the top-K predictions. In this way, BLM+ acknowledges the uncertainty in f (x T i ) and exploits other potential predictions, providing more robust estimations.K,i ≡ {y ′ | arg max y1,...y K f (x T i ) y ′ } denote the set of the top-K predicted pretrained labels for input x T i , and p(y S |x T i ) ≡ (softmax • f )(x T i ) y S denote the predicted probability for any y S ∈ Y S given x T i . Then, within the BLM+ strategy, the joint density is approximated2 as:EQUATIONSimilar to BLM, with the Laplace smoothing coefficient being λ and the size of Y S being k S , p(Y S = y S |X T ) can be expressed by applying BLM+ as:EQUATIONCombining Eq. ( 9) and Eq. ( 8) with Eq. ( 5), and going through all y T ∈ Y T and y S ∈ Y S , we obtain the full BLM+ estimation as ω BLM+ after column-wise sum normalization of ωy S ,y T , similar to BLM. In practice, we set K = ⌊α • k T ⌋, with ratio α being a hyper-parameter that decides K based on the size of downstream label space k T .Pipeline θ ← θ -a ▽ θ n i=1 ℓ(y T i , f ω out (f (x T i ; θ)))16: end for 17: return ωBLM+The iterative process of learning ω BLM , ω BLM+ comprises these four steps: 1) Input images, with VR patterns, are fed into the fixed pretrained model to obtain output logits and predicted pretrained labels.2) BLM and BLM+ replace previous LM (e.g., RLM, FLM or ILM) to estimate ω.3) The initial logits are reweighted using ω BLM or ω BLM+ , yielding refined predictions for downstream labels. 4) Loss functions (e.g., cross-entropy) and backpropagation are employed to update the input VR.Furthermore, we include a justification of why probabilistic many-to-many LM (e.g., BLM and BLM+) should be favored over deterministic one-to-one LM (e.g., RLM, FLM and ILM). Define the label spaces Y S = {0, 1} and Y T = {0, 1} as binary sets 3 . Consider the set of potential LM functions F lm = {f lm : Y S → Y T }, including each function f lm (y S ) ∈ {y T , 1 -y T }. For any f lm ∈ F lm , the expected accuracy of f lm regarding the entire downstream label space is defined as 4 :EQUATIONwhere p(y S ) is the marginal distribution of the pretrained labels and p f lm (y S ) = y T |y S is the conditional probability that f lm correctly predicts a downstream label y T from a pretrained label y S . Let f plm and f dlm denote the probabilistic LM (Definition E.1) and deterministic LM (Definition E.2), respectively. We finally prove that Acc(f plm ) ≥ Acc(f dlm ) (Corollary E.5) in Appendix E, which further verifies the effectiveness of our methods in the view of theoretical understanding. 
experiments: 
 Tasks and Baselines. Following ILM [4] , we employ ResNet-18 [17] pretrained on ImageNet-1K [45] and ResNeXt pretrained on Instagram [37] to test the performance of VR. The results are evaluated on twelve downstream datasets: Flowers102 [40] , DTD [9] , UCF101 [46] , Food101 [2] , GTSRB [19] , EuroSAT [18] , OxfordPets [43] , StanfordCars [29] , SUN397 [57] , CIFAR10/100 [30] and SVHN [39] . Previous gradient-free LM methods RLM [12] , FLM [47] and ILM [4] are used as the baselines. The results of deep learning-based LM will also be included for reference, where LM is treated as a single-layer linear neural network connected to the output of the pretrained model for training alongside VR. More dataset and implementation details are in Appendix F. Regarding hyper-parameters of BLM, λ is set as 1, and the top-K ratio α is 0.15 (analyzed in Appendix G).Results for Padding-based VR. Padding-based input VR adds trainable noise to the outer frames of centered images. Table 1 shows the performance of BLM and BLM+ applied with padding-based input VR. BLM and BLM+ yield the highest accuracy across all datasets except for SVHN. On ResNet-18, compared to the SOTA (i.e., ILM), BLM achieves an average improvement of 4.7% across the 12 datasets, whereas BLM+ achieves a 6.1% enhancement. On ResNeXt-101, BLM and BLM+ achieve accuracy improvements of 3.2% and 3.8% on average, respectively. The elevation in accuracy is particularly pronounced in tasks with a higher number of classes (e.g., UCF101, CIFAR100). On SVHN, ILM performs slightly better, which could be attributed to the minimal inter-class variation and the smaller number of classes (which is 10) in SVHN, resulting in similar mapping values for different downstream labels and thus reducing our method's advantage (discussed in Appendix H). However, compared to current gradient-free LM methods, the deep learning-based LM may still have an advantage in the performance of downstream tasks due to the learning capacity of the linear layer neural network. Our proposed BLM and BLM+ aim to bridge the gap between gradient-free LM and deep learning-based LM. Additionally, BLM and BLM+ have been observed to possess greater interpretability (see Appendix I for more experiments) and fewer parameters (see Appendix J for details) compared to deep learning-based LM. Results for Vision-Language Models. The application of our BLM and BLM+ on visionlanguage models (i.e., CLIP), along with the performance, and visualization results are dis- Ablation Study. Table 3 presents the ablation study results of BLM and BLM+. For BLM, we list the results of replacing probabilistic LM with a one-to-one LM, denoted as '-Bayes', and the results of calculating ω BLM only once in the first epoch without subsequent iterations, denoted as '-Iter'.For BLM+, the removal of aggregating probabilities results in BLM; hence, we report the results of aggregating all probabilities instead of top-K predicted probabilities, denoted as '-Top-K'. Like that for BLM, '-Iter' shows the results of calculating ω BLM+ only once without subsequent iterations.Besides, when both '-Top-K' and '-Bayes' are applied to BLM+, BLM+ degenerates into the same results as '-Bayes' of BLM, which is displayed in the previous column of Table 3 . For BLM, employing 'Bayes' to compute ω improves accuracy across most datasets, with slightly smaller gains observed for datasets with fewer classes (EuroSAT, CIFAR10, and SVHN). For BLM+, applying 'Top-K' assists in filtering out redundant information, yielding positive impacts across most datasets. In particular, on the OxfordPets dataset, to classify cat and dog breeds, using top-K predicted probability effectively filters out numerous irrelevant categories in the pretrained label space, which leads to significant improvements. Furthermore, for both BLM and BLM+, iterative updates are crucial as the initial input VR may deviate considerably from the final iteration. The greater the disparity between the domains of downstream and pretrained tasks (GTSRB, SVHN), the more pronounced the impact of the input VR, thereby emphasizing the necessity of iteration updates.Visualization Results. The probabilistic LM obtained by BLM or BLM+ can elucidate the connection between pretrained and downstream label spaces. Figure 3 shows the visualization results for three Pretrained ResNet-18 is used, and the downstream label 'Marigold' is selected as an example. labels in downstream tasks, taking ResNet-18 pretrained on ImageNet-1K as an example. Each column of ω computed using BLM or BLM+ is a vector with length k S = 1000, representing the weights assigned to the 1,000 outputs-one for each y S -of the pretrained model corresponding to a downstream label y T . The top-weighted labels (i.e., y S where ω y S ,y T is larger) for 'Edamame' correspond to organisms such as snakes and artichokes, which share similarities in color and shape. Similarly, the predominant labels associated with 'Fibrous' from the texture dataset include roughtextured items like 'Hay' and 'Komondor'. 'Dog' encompasses various sub-breed canines. These findings suggest that BLM and BLM+ establish an optimal probabilistic LM between label spaces, and handle similarity or inclusion relationship, addressing the drawbacks in Figure 1 .Discussion of Why VR Is Effective. From a visualization perspective, Figure 4 shows the topweighted pretrained labels and input VR patterns θ at different iteration stages using BLM+. The training loss for each iteration and changes in ω, measured by the Euclidean norm, are also plotted.During the update of ω and θ, the pretrained labels with top ω y S ,y T for y T being 'Marigold' transition gradually from dissimilar labels such as 'Reef' and 'Teddy' to 'Cauliflower' and 'Pineapple' which share more similarities in color, shape and texture. Meanwhile, the training loss diminishes gradually, and ω converges, demonstrating the effectiveness of VR and BLM+.Impact of Label Space Sizes k T . Figure 5 shows the relationship between different sizes of the downstream label space and the accuracy improvement achieved by BLM and BLM+. Tasks with larger label spaces report more pronounced performance improvements. While simpler tasks with smaller label spaces might not fully showcase the power of our approach, the strength of BLM and BLM+ lies in unraveling the complex many-to-many relationship that often arises in tasks with more numerous classes. In such scenarios, our probabilistic LM methods demonstrate their full potential. Lastly, the application of BLM and BLM+ for vision-language models is explored in Appendix L.  
conclusion: 
 We focus on output LM methods for VR and reveal the drawbacks in current gradient-free LM methods, which use one-to-one mappings that overly simplify the relationship between the pretrained and downstream label spaces. To address this issue, we propose BLM, which calculates probabilistic LM matrices guided by Bayes' theorem. Additionally, we aggregate the probability of top-K predicted pretrained labels instead of counting a single label during the estimation of probabilistic LM matrices, yielding an improved method BLM+. Both theoretical analysis and experimental results validate the effectiveness of BLM and BLM+ while offering insights into understanding the effectiveness of VR through a probabilistic lens.A The Problem Setting of VR The task of VR reuses fixed pretrained models for downstream tasks. As illustrated in Figure 7 , an input VR module operates before pretrained models, directly altering the input space of downstream tasks. Concurrently, an output LM function acts after pretrained models, taking the predicted pretrained labels as input and outputting those for downstream tasks. Hence, VR achieves the reusability of pretrained models for downstream tasks without adapting the model parameters, primarily through modifications to the input and output spaces. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] . Justification: The paper does not release data or models that have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Creators or original owners of code or data have been cited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Visual reprogramming is one type of methods that aim to obtain models on downstream tasks with the help of pretrained models. This process is similar to the aim of transfer learning which is used to leverage knowledge from a data-rich domain [13] or a pretrained model [50] to address tasks on other domains. The former is known as domain adaptation, and the latter is known as finetuning.Finetuning. Given a pretrained model, finetuning uses trainable parameters to accommodate new taskspecific information of the downstream tasks. As pretrained models grow in size, recent progresses in transfer learning have prioritized parameter-efficient finetuning (PEFT) [20] to support resourcefriendly adaptation. Regarding PEFT, the prevailing methods can be categorized as follows. The most widely adopted approach is selective finetuning [45, 62] , which adjusts a subset of parameters from the pretrained model while keeping the remaining components fixed, thereby reducing the total number of trainable parameters for downstream tasks. Other methods may involve adding adapters [5, 31, 42] , which introduce extra trainable layers or parameters to the pretrained model and finetune only these adapters during training. Moreover, low-rank adaptation methods [20, 63, 68] have also been proposed for pretrained Vision Transformers. They apply low-rank decomposition to the parameters during training, achieving remarkable performance on downstream tasks with a significantly reduced number of parameters. Additionally, Prompt Tuning methods [16, 22, 52] , similarly directed at pretrained Vision Transformers, integrate trainable parameters parallel to the features at the input and intermediate layers. The primary distinction of these methods from VR [1, 3, 4, 47, 48, 64] lies in their necessity to be designed according to different pretrained model architectures and may also involve modifying the model weights. In contrast, VR is model-agnostic and does not require alterations to pretrained model parameters.Domain Adaptation. Domain adaptation (DA) aims to bridge distributional gap by aligning feature spaces of the source task to the target domain with different data distributions [14, 36, 53] . Often, DA is achieved by learning invariant representations or transforming parameters to manage domainspecific shifts of the source and target data. CDAN [33] addresses this by introducing a conditional discriminator for class label-conditioned feature adaptation, while UDA [34] leverages residual layers to capture both domain-specific and domain-shared representations. More recently, source-free DA [32, 61, 65] , which seeks adaptation without access to the source data, has gained popularity due to growing concerns over data privacy and storage limitations, as well as the need for adaptation in scenarios where source data is inaccessible [8, 10] . p(Y T = Dog, Y S = CockerSpaniel | X T ) = 1 2 , p(Y T = Dog, Y S = EnglishSpringer | X T ) = 1 4 , p(Y T = Cat, Y S = EgyptianCat | X T ) = 1 4 ,as is shown in Figure 8 . Similarly, p(Y S = y S | X T ) can also be estimated.This section provides a detailed exposition of gradient-free LM methods. Such methods, derived from data distributions, obviate the need for gradients in the output mapping phase. The pseudocode is presented below. Similar to Section 3, ω represents the one-to-one LM or probabilistic LM. T ← T ∪ {y S } 9: end for 10: return ωThe process of random label mapping (RLM) is outlined in Algorithm 3. , where the computation of ω does not involve the downstream training set. The algorithm establishes a random one-to-one mapping between the pretrained and the downstream labels, ensuring that each y T corresponds to a unique y S . RLM is computed once before learning the input VR f (•; θ). T ← T ∪ {y S } 13: end while 14: return ωThe procedure of frequent label mapping (FLM) is outlined in Algorithm 5. Initially, it utilizes the pretrained model to obtain predicted pretrained labels for samples of the downstream task.Subsequently, it computes a joint distribution matrix between the predicted pretrained labels and the ground-truth downstream labels. Finally, employing a greedy algorithm, it iteratively identifies the maximum value in the rows and columns corresponding to unmatched label pairs in the matrix to determine the one-to-one mappings. FLM is also computed prior to the training of f in (•; θ). θ ← θ -a • ∇ θ n i=1 ℓ(y T i , f ω out (f pre (f in (x T i ; θ)))) 17: end for 18: return ωAs an enhanced version of FLM, iterative label mapping (ILM) employs interleaved updates with θ at each epoch, as outlined in Algorithm 6. Such interleaved updates take into consideration the variations in the output space induced by updates to the input VR during the training process, thereby ensuring that the output LM will be matched with the updated VR.The detailed procedure of Bayesian-guided label mapping (BLM) proposed in this paper is shown in Algorithm 7. Compared to ILM, BLM replaces the previous one-to-one mapping ω ∈ {0, 1} kS×kT with probabilistic LM ω ∈ [0, 1] kS×kT , both satisfying kS j=1 ω j,• = 1 as stated in Section 3. Meanwhile, the process of matrix computation for BLM is based on the Bayes' theorem (detailed in Section 4) to reflect the complex relationship among label spaces, rather than determining the optimal match through the oversimplified greedy algorithm.As mentioned in Section 4, BLM+ extends BLM by incorporating the aggregation of top-K predicted probabilities, shown in Algorithm 9. This divergence manifests in the computation process of the joint distribution matrix between predicted pretrained labels and ground-truth downstream labels. Previous methods (i.e., RLM, ILM, BLM) computed a non-negative integer matrix d ∈ Z kS×kT based on the frequency of occurrence of samples (Algorithm 4).In BLM+, the calculation entails replacing the deterministic frequencies with predicted probabilities from the top K pretrained labels to estimate the joint distribution matrix d ∈ R kS×kT , as is shown in Algorithm 8. In the procedure, the probability aggregation substitutes the binary frequency distribution {0, 1} with a probability distribution within the range of [0, 1], while the top-K technique # Training f in (•; θ) 13: θ ← θ -a • ∇ θ n i=1 ℓ(y T i , f ω out (f pre (f in (x T i ; θ))))′ ∈ R kS×kT 3: Initialize d ′ ← {0} kS×kT , temp matrix Q = [Q 1 , ..., Q k ] ⊤ ∈ R k , K = [K 1 , ..., K k ] ⊤ ∈ N + k 4: # Computing aggregation distribution matrix d ′ 5: for i = 1...n do 6: Q ← TopK j (softmax(f pre (f in (x T i ; θ)) j ), k) # top-K 7: K ← TopKIndices j (f pre (f in (x T i ; θ)) j , k) 8:d ′ Ks,yi T ← d ′ Ks,yi T + Q s for s = 1...k # Probability Aggregation 9: end for 10: return d ′ serves to retain the most probable k predicted labels rather than selecting only one (i.e., BLM) or all labels (denoted as '-Top-K' in ablation studies in Section 5).The baseline method FLM calculates the mapping ω once and keeps it fixed, while ILM and our methods update ω at each step. However, updating ω does not require running the model twice to obtain current predictions for each epoch. Instead, predictions from the most recent epoch can be reused. Therefore, only in the first epoch is it necessary to run the pretrained model an additional time to initialize the weights of LM, which is the same as FLM. In subsequent epochs, these methods do not require any extra runs. More details can be found in the quick version of our released code.In this section, we investigate why probabilistic LM should be favored over deterministic one-toone mapping. This analysis assumes the existence of true correspondences between labels in the pretrained and downstream domains. We establish that, under certain conditions, probabilistic LM (Definition. E.1) outperforms deterministic LM (Definition E.2) in estimating the distribution of true label correspondences, quantified by the expected accuracy of the LM function (Eq. ( 10)). # Training f in (•; θ)13: θ ← θ -a • ∇ θ n i=1 ℓ(y T i , f ω out (f pre (f in (x T i ; θ)))) 14: end for 15: return ω This analysis focuses on the comparisons of LM. Given that the pretrained model f pre , input x, and input transformations f in are the same across different LM methods, we will omit these notations below unless explicitly needed. We begin by introducing key definitions. Definition E.1 (probabilistic label mapping (PLM)). Let F plm ⊂ F lm be a set of mapping functions such that for all f plm ∈ F plm , we haveEQUATIONHere, p(f plm (y S ) = y T |y S ) is the conditional probability that a pretrained label y S is mapped to a downstream label y T . Definition E.2 (deterministic label mapping (DLM)). Let F dlm ⊂ F lm be a set of mapping functions, defined by f dlm (y S ) = g(y S ) for all y S ∈ Y S , where g(y S ) specifies a deterministic rule, either g(y S ) = y S for identity mapping; or g(y S ) = 1 -y S for flip mapping, respectively. Then, deterministic label mapping is defined as: ∀f dlm ∈ F dlm , p(f dlm (y S ) = y T |y S ) = δ y S ,g(y S ) , with δ y S ,g(yEQUATION)where δ is the Kronecker delta function, ensuring y T is uniquely mapped from a pretrained label y S .Then, we demonstrate the conditions where Acc(f plm ) ≥ Acc(f dlm ). Since DLM is defined by g, following either identity mapping or flip mapping exclusively, each case will be discussed separately. Lemma E.3. Given a collection of paired labels {(y S , y T )} n i=1 . If the aggregate conditional probabilities p(y S = 1|y T = 0) ≥ p(y S = 0|y T = 0) and p(y S = 0|y T = 1) ≥ p(y S = 1|y T = 1) hold true, and considering f dlm is defined by identity mapping as outlined in Definition E.2, then it follows that Acc(f plm ) ≥ Acc(f dlm ).Lemma E.3 (proof in Appendix E.2) implies that PLM achieves at least as high expected accuracy as DLM defined by identity mapping, under the following conditions: for downstream samples with y T = 0, the inequality is satisfied when they are more likely to correspond to pretrained samples with y S = 1 than those with y S = 0; for downstream samples with y T = 1, the inequality is satisfied when the corresponding pretrained samples are more likely to have y S = 0 than y S = 1.Uncertainty in Label Inter-Dependencies. Essentially, the conditions above reflect potential complex patterns of label correspondence that arise when inter-dependencies between the labels exist across domains. While this "label mismatch" problem has been discussed in binary settings, it can be generalized to multi-class settings without loss of generality. Unlike DLM, which merely relies on a static mapping rule and hence may fail when true label correspondence conflicts with this predefined rule, PLM captures the conditional probabilities of y T given y S . By harnessing the inherent uncertainty encoded in the probabilistic form of ω, PLM is expected to achieve more robust label mapping predictions.Next, we compare PLM with DLM using the flip mapping rule.Lemma E.4. Given a collection of paired labels {(y S , y T )} n i=1 . If the aggregate conditional probabilities p(y S = 0|y T = 0) ≤ p(y S = 1|y T = 0) and p(y S = 0|y T = 1) ≤ p(y S = 1|y T = 1), and f dlm is defined by flip mapping as outlined in Definition E.2, then Acc(f plm ) ≥ Acc(f dlm ).Lemma E.4 (proof in Appendix E.2) establishes another sufficient condition under which PLM could achieve an expected accuracy at least as high as DLM defined by flip mapping. The condition applies to all downstream samples, regardless of their labels (both y T = 0 or y T = 1), stating that it is more likely that their corresponding pretrained label being y S = 1 rather than y S = 0.Bias in Label Correspondences. The bias in Label correspondence refers to a phenomenon where a disproportionate number of downstream samples correspond to pretrained samples with a specific label. For example, consider a medical diagnosis task where both pretrained and downstream data come from populations with low disease prevalence, the label correspondences may exhibit this bias [54] . While this bias may be overlooked by DLM, it could be captured and even exploited by PLM, which flexibly adjusts the weighting schemes, e.g., assigning higher value to ω 1,0 than ω 0,0 for samples where y T = 0, and to ω 1,1 over ω 0,1 for samples where y T = 1.Corollary E.5. Let f plm and f dlm denote the label mapping functions defined in Definition E.1 and Definition E.2, respectively. Given pretrained and downstream label spaces Y S = {0, 1} andEQUATION)where ā is the opposite label of a, then we have Acc(f plm ) ≥ Acc(f dlm ).Remark E.6. Corollary E.5 implies a theoretical foundation for preferring PLM over DLM in scenarios where the label mapping relationship between two domains is uncertain, biased and potentially deviates from a deterministic one-to-one mapping assumption. This finding holds importance in label mappings for VR, as the label spaces may encompass multi-class settings. Furthermore, in VR settings, the pretrained labels derived from f pre predictions, are subject to increased uncertainties and biases influenced by the quality and distribution of the pretrained model and dataset 5 .Lemma E.7 (cf. Lemma E.3). Given a collection of paired labels {(y S , y T )} n i=1 . If the aggregate conditional probabilities p(y S = 1|y T = 0) ≥ p(y S = 0|y T = 0) and p(y S = 0|y T = 1) ≥ p(y S = 1|y T = 1) hold true, and f dlm is defined by identity mapping as outlined in Definition E.2, then Acc(f plm ) ≥ Acc(f dlm ).Proof. Expand Eq. ( 10) by taking all possibilities of y T , we have:EQUATION)Note that the conditional independence holds since the output of f lm relies solely on the input y S . For DLM defined by identity mapping, p(f dlm (y S ) = y T |y S ) = 1 if y S = y T , and 0 otherwise. Taking into account all the samples, the expected accuracy Acc(f dlm ) can then be expressed by Acc(f dlm ) = y T ∈Y T p(y T ) p(y S = y T |y T ) = p(y T = 0)p(y S = 0|y T = 0) + p(y T = 1)p(y S = 1|y T = 1).As for PLM, the expected accuracy can be rewritten asEQUATION)where ω 0,0 stands for ω y S =0,y T =0 , and similarly for the remaining ω 0,1 , ω 1,0 , ω 1,1 .To evaluate the expected accuracy of f plm and f dlm , we look into the comparison separately for each y T . Specifically, for the samples with y T = 0, we aim to show thatEQUATION)Given the constraints ω 0,0 + ω 1,0 = 1 and p(y S = 0|y T = 0) + p(y S = 1|y T = 0) = 1, the LHS of Eq. ( 17) becomesEQUATIONThe inequality we need to show is then simplified to p(y T = 0)[(ω 0,0 -ω 1,0 ) • p(y S = 0|y T = 0) + ω 1,0 ] ≥ p(y T = 0)p(y S = 0|y T = 0). This inequality holds if p(y S = 0|y T = 0) ≤ p(y S = 1|y T = 0).Similarly, for samples with y T = 1, the inequality of interest isp(y T = 1) ω 1,0 • p(y S = 1|y T = 0) + ω 1,1 • p(y S = 1|y T = 1)≥ p(y T = 1)p(y S = 1|y T = 1).This holds if p(y S = 1|y T = 1) ≤ p(y S = 0|y T = 1).Both conditions can be satisfied without conflict. Thus, we can confirm Lemma E.3 by evaluating these conditions jointly.Lemma E.8 (cf. Lemma E.4). Given a collection of paired labels {(y S , y T )} n i=1 . If the aggregate conditional probabilities p(y S = 0|y T = 0) ≤ p(y S = 1|y T = 0) and p(y S = 0|y T = 1) ≤ p(y S = 1|y T = 1), and f dlm is defined by flip mapping as outlined in Definition E.2, then Acc(f plm ) ≥ Acc(f dlm ).Proof. When defined deterministically by flip mapping, DLM can be equivalently expressed as p(f dlm (y S ) = y T |y S ) = 1 if y S ̸ = y T , and 0 otherwise. This allows the expected accuracy of DLM to be expanded as:Acc(f dlm ) = y T ∈Y T p(y T ) p(y S ̸ = y T |y T ) = y T ∈Y T p(y T ) 1 -p(y S = y T |y T ) = p(y T = 0) 1 -p(y S = 1|y T = 0) + p(y T = 1) 1 -p(y S = 0|y T = 1) . (20)Meanwhile, the expected accuracy of PLM remains consistent as in Eq. ( 16). Again, to show that Acc(f plm ) ≥ Acc(f dlm ) holds, we compare the expected accuracy with respect to different y T samples separately.For samples with y T = 0, we need to show p(y T = 0) ω 0,0 • p(y S = 0|y T = 0) + ω 1,0 • p(y S = 1|y T = 0)EQUATION)Given the constraints that ω 0,0 + ω 1,0 = 1 and p(y S = 1|y T = 0) = 1 -p(y S = 0|y T = 0), the LHS of Eq. ( 21) can be expressed by p(y T = 0) ω 0,0 • p(y S = 0|yEQUATIONWe rearrange the terms:EQUATIONIt is then concluded that Eq. ( 23) holds if p(y S = 0|y T = 0) ≤ p(y S = 1|y T = 0).As with y T = 1 samples, a similar derivation is performed to satisfy the inequality p(yT = 1) ω 0,1 • p(y S = 0|y T = 1) + ω 1,1 • p(y S = 1|y T = 1)≥ p(y T = 0)p(y T = 0) 1 -p(y S = 1|y T = 0) .Resembling y T = 0 samples, the derivation yields the condition p(y S = 0|y T =) ≤ p(y S = 1|y T = 1).Notably, the condition p(y S = 0|y T = 0) ≤ p(y S = 1|y T = 0) does not conflict with p(y S = 0|y T =) ≤ p(y S = 1|y T = 1), and both conditions can be jointly satisfied.Additional dataset information is presented in Table 4 . For a fair comparison, we adhere to the data partitioning scheme employed by ILM [4] through all datasets. The batch size for Oxfordpets and DTD is set to be 64 while 256 for the remaining datasets.Consistent training settings are maintained to ensure a fair comparison. For training input VR patterns, we apply the Adam optimizer [26] with an initial learning rate of 0.01. The number of epochs is 200, with the learning rate decay being 0.1, scheduled at epochs 100 and 145. All experiments are conducted on a single A100 GPU and the average accuracy of three distinct random seeds are reported.As described in Section 4, the ratio α is used in calculating k = ⌊α • k T ⌋. The experimental results to tune hyper-parameters α and λ are reported in Table 5 . α is chosen among [0.01, 0.05, 0.15, 0.5, 1], The optimized λ is determined first to be 1 by the average accuracy of different α values, followed by deriving an optimal α = 0.15.While the same hyper-parameters may not necessarily be optimal across different datasets, for the sake of consistency and fairness, this paper employs identical hyper-parameters for all datasets.Figures 9 and Figure 10 illustrate the impact of λ and α on accuracy. It is observed that the optimal hyper-parameters vary across different datasets.In general, as λ increases, the test accuracy initially rises and then declines. This parameter is used to balance the contributions of individual pretrained labels. An over-small λ might overly rely on the distribution of pretrained labels obtained from pretrained models, while a too-large one might overlook differences among pretrained labels. Meanwhile, with an increase in α, accuracy first increases, then plateaus or slightly decreases. This is because excessively small or large α values may lead to the neglect of certain crucial labels or the emphasis on redundant ones during the estimation of the probability aggregation matrix. Therefore, choosing moderate values for λ and α appears to be more appropriate.We used universal hyper-parameters to show that BLM and BLM+'s performance gains over baselines are not sensitive to hyper-parameters. However, we assume that the dataset-specific tuning for hyper-parameters could yield more optimized results.Additional experiments are conducted using a validation set and training set split of 30% and 70% to find optimal hyper-parameters for each dataset. Results are shown in Table 6 . We observe that optimal hyper-parameters tailored for each dataset achieve better performance compared to using shared hyper-parameters, which matches our assumption. Less effective for tasks with very few classes. As shown in Table 1 , when the number of classes (i.e., size of the label space) in downstream tasks is smaller (10 classes in SVHN and 10 classes in EuroSAT) and the original task is relatively simple, the advantage of BLM and BLM+ is not very pronounced. This is because BLM and BLM+ replace the one-to-one mapping with a pairwiseconnected probabilistic LM. While this optimization yields positive results in most tasks, for a small subset of simple tasks, the one-to-one mapping may better reflect the relationship between the pretrained label space and the downstream label space. For such tasks, BLM and BLM+ no longer exhibit significant effects.Not solving the cases where VR is not applicable for downstream tasks. For example, in the case of the StanfordCars dataset in vision models, as shown in Table 1 and Table 2 , the accuracy of the downstream task remains consistently low (<10%) through learning using input VR. While applying BLM and BLM+ in such scenarios yields better results compared to using one-to-one mapping, it still cannot significantly enhance VR performance to the extent of being comparable to finetuning the entire model.Based on the example of ResNet-18 pretrained on ImageNet-1K applying to the downstream task CIFAR10, Figure 11 shows the results of gradient-free methods BLM and BLM+, while the second row shows deep learning-based methods which learn a linear neural network for f ω out . 'Deep Learning' refers to a single-layer neural network without constraints (i.e., ω ∈ R kS×kT ), while 'Deep Learning + Sigmoid' refers to applying the Sigmoid function to restrict ω ∈ [0, 1] kS×kT aligning with the range of ω BLM and ω BLM+ . The right part of Figure 11 depicts the specific pretrained and downstream labels corresponding to these matrices.It is observed that BLM and BLM+ are good at revealing similarities between pretrained and downstream labels. For example, for the downstream label 'Airplane', which visually resembles 'Great White Shark', 'Hammerhead' and 'Stingray', the weights in ω BLM or ω BLM+ tend to be higher. Conversely, for dissimilar labels like 'Truck' and 'Ostrich', the weights will be approaching 0. However, the weight matrices obtained from deep learning-based methods fail to capture such clear-label relationship. The results demonstrate the advantages of BLM and BLM+ in terms of interpretability.The Required Number of Epochs. Different label mapping methods require varying numbers of epochs to converge. We initially used 200 epochs as with [4] to ensure a fair comparison with the baseline methods. Additional experiments are conducted to assess the impact of different epoch numbers from [60, 100, 150] on our BLM and BLM+ model, using ResNet-18 as the pretrained model. The results are shown in Table 7 .We found that running 100 epochs yields results comparable to those achieved with 200 epochs. This demonstrates that BLM and BLM+ require less convergence time, highlighting their efficiency. pretrained model complexity) along with the necessity of backpropagation for gradient computation. Conversely, the gradient-free LM methods along with VR emphasized in this study do not encounter these challenges. Results when the pretrained and downstream labels exhibit appearance resemblance. Figures 12 and 15 respectively depict the outcomes on Flowers102 and Food101 datasets, each about classification tasks of various flowers and food. BLM+ is adept at assigning higher weights to pretrained labels with a greater resemblance to the downstream label in terms of color, shape, and intricate features.In terms of color, as evidenced in Figure 15 , the top-weighted labels for 'Edamame' comprise 'Green Snake', 'Artichoke', and 'Green Mamba', all sharing a green hue. Regarding shape, in Figure 12 , the 'Gazania' with petal stripes corresponds to top weighted labels such as 'Banana' and 'Zucchini', which exhibit similar striping patterns. As for intricate features, in Figure 12 , the 'Globe Thistle' with needle-like appearance aligns with top weighted labels including 'Sea Urchin', 'Porcupine', and 'Cardoon', which possess akin prickly characteristics. Results when the pretrained and downstream labels exhibit similarities in texture. Figure 13 presents the results on the DTD dataset, which pertains to the classification of various textures. Both BLM+ and BLM assign higher weights to labels sharing akin textures. For example, 'Spiralled' corresponds to top-weighted labels embodying spiral-shaped entities such as 'Snail', 'Centipede', and 'Coil', while 'Fibrous' aligns with entities possessing a rough and fibrous texture, including 'Hay', 'Komondor', and 'Porcupine'.Results when the pretrained and downstream labels exhibit similarities in backgrounds. Figure 14 illustrates the results on the UCF101 dataset, a dataset for action classification. In this task, both BLM and BLM+ tend to assign higher weights to pretrained labels with backgrounds or environments akin to the downstream labels. For example, the action 'Apply Lipstick' often involves the presence of a human face; hence, pretrained labels such as applying 'Lipstick', eating 'Ice Lolly', and spraying 'Hair Spray' contribute significantly. Likewise, labels closely associated with 'Baseball pitch' include 'Ballplayer' and 'Scoreboard', featuring backgrounds of vast grass fields.Results when pretrained and downstream labels exhibit inclusion relationship. Figure 16 illustrates the results on the CIFAR10 dataset, which comprises images broadly categorized into ten main classes, with each category corresponding to several subcategories within the pretrained domain. It is noted that unlike the singular class selection of ILM, both BLM and BLM+ allocate similar weights to multiple subcategories. For example, 'Dog' corresponds to different breeds such as 'Cocker Spaniel', 'English Springer', and 'English Setter', while 'Bird' encompasses subcategories Justification: The claims made in the abstract are detailed in the introduction part, and further discussed in each section. Each paragraph in the introduction is provided with a corresponding link to certain sections.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Note that this approximation is not normalized, and thus, is not strictly equivalent to the true probability.This analysis focuses on the binary setting for simplicity.Input x is intentionally omitted as all LM methods operate on the same inputs.For the analysis purpose, in this section we simplify the setting and operate with ground-truth Y S . In practice, VR does not have access to true y S but must rely on the predicted y S from the well-trained fpre instead. 


Decision: This paper presents work on visual reprogramming -- leveraging existing vision models and adapting them to solve tasks involving different labels.  The core contribution is a Bayesian approach to label mapping that alleviates shortcomings of previous (one-to-one mapping) approaches, and better utilizes cross-category information in conducting label mapping.

The reviewers noted the clear strengths of the paper in this regard, with a clear motivation, solid execution, and impressive empirical results.  A few minor issues were raised in the reviews, which were largely addressed in the author responses.

Visual reprogramming, and more broadly adapting pre-trained models to solve new tasks, is an active, important area of research.  This paper makes solid contributions to this literature, and as such is recommended for acceptance to NeurIPS.

Reviews: review: paper_summary: paper_summary: Pretrained models play a crucial role in current machine learning and computer vision tasks, and effectively leveraging them in downstream tasks has become increasingly important. This paper explores the research area of visual reprogramming (VR), which diverges from traditional fine-tuning by adjusting the input space rather than the parameter space of pretrained models. Additionally, VR necessitates establishing a mapping from pretrained labels to downstream labels, which is the primary focus of this paper. The proposed method for mapping pretrained labels to downstream labels is well-motivated and convincing. The experimental results are robust and demonstrate the efficacy of the approach.
strength_weakness: strength_weakness: 1. Although the motivation is clear, it feels somewhat dry. More detailed explanations are needed for Section 4.1. Specifically, some of the conditional probabilities require further clarification (e.g., Eq. (6)).

2. It is unclear how Section 4.2 can be extended to the multi-class case. The accuracy metric (Acc) used here appears to be suited only for binary cases.

3. While it is commendable that a universal hyperparameter performs well in your experiments, it raises the question of whether these hyperparameters can be tuned using a validation set. How do all methods perform when evaluated on the same validation set?

4. I am not entirely convinced by Figure 4. Why is pineapple selected as well? This needs further explanation.

5. The absence of algorithm tables in the main content is a significant omission and should be addressed. Similar to the first weakness, the current method presentation feels somewhat dry.

6. Is BLM+ without Top-K and Bayes the second column in Table 3? If so, additional descriptions should be included for clarity.

7. What are the fine-tuning results in this context? It is essential to establish a standard to determine when VR is needed. Additionally, what is the total time cost compared to fine-tuning?
questions: questions: See Weaknesses.
limitations: limitations: The authors have adequately discussed the limitations of the work.
paper_summary: paper_summary: Another type of transfer learning approach is considered in this paper: model reprogramming. Different from classical transfer learning approaches, model reprogramming only changes models via changing the input space and output space, which is more efficient to fit a pretrained model to some downstream tasks. Specifically, this paper focuses on the output space, label mapping. The motivation is strong as it is clear that single map does not work well. Then, new methods are proposed based on Bayes' theorem. Experiments are enough and solid, in terms of model size or dataset size.
strength_weakness: strength_weakness: 1. Transfer learning literature is missing in this paper. Although model reprogramming is not considered as a transfer learning method in the literature, it is one of transfer learning techniques. This review can help position this paper better.

2. Can we transfer a pretrained model to any possible task? Are there standards to help choose which model should be used for a specific downstream task?

3. For example, if we can use a model trained with cars to help recognize images of animals? It looks impossible. How to tell when we can use a pretrained model?

4. Computing w in BLM also needs time. Is there efficient advantage of VR anymore? What is the performance of finetuning? The gap should be reported in this paper.

5. Main algorithm table should be moved to the main paper. It is not very easy to implement BLM or BLM+ based on the formula, but the algorithms are very helpful.

6. It would be better to explain meaning of some conditional probabilities.
questions: questions: 1. Transfer learning literature is missing in this paper. Although model reprogramming is not considered as a transfer learning method in the literature, it is one of transfer learning techniques. This review can help position this paper better.

2. Can we transfer a pretrained model to any possible task? Are there standards to help choose which model should be used for a specific downstream task?

3. For example, if we can use a model trained with cars to help recognize images of animals? It looks impossible. How to tell when we can use a pretrained model?

4. Computing w in BLM also needs time. Is there efficient advantage of VR anymore? What is the performance of finetuning? The gap should be reported in this paper.
limitations: limitations: No concerns for this paper.
paper_summary: paper_summary: Visual reprogramming is an interesting way to reuse a pre-trained classifier or an VLM. In previous methods, the way to change the output interface is basically gradient-free and one-on-one mapping. In this paper, the authors found that the previous way is suboptimal and ignores  information. Then, from a theoretical perspective, a new objective is derived using Bayes' theorem. By optimizing this objective, the performance of VR methods is significantly improved, which is demonstrated via extensive experiments. A theoretical study is provided as well.
strength_weakness: strength_weakness: 1. In Eq. (1), what is the real gap between right and left terms? It seems not easy to get this gap. I suggest using right one directly, which is easy to be acceptable.

2. Above the Eq. (6), one might misunderstand the way to calculate the frequency. Using () might be better.

3. Figure 2 is confusing. It looks like step 3 follows step 1, rather than step 2 following step 1. What is the actual sequence between step 1~3?

4. In section 4.2, the calculation of accuracy is confusing. $x$ is missing.

5. What does $\bar{r}$ mean in l.262?

6. Training time is potentially a concern. How many training epochs are needed for BLM?
questions: questions: Please refer to "Weaknesses".
limitations: limitations: No further concerns regarding limitations.
paper_summary: paper_summary: This paper focuses on a technique called label mapping (LM) which is used in visual reprogramming. It finds the relation between pre-trained labels and downstream labels. In conventional fine-tuning, LM is just the backpropogation using the loss function. However, this paper and previous similar papers aim to develop a gradient-free way to find the weights in the last layer in the conventional fine-tuning. The whole paper is based on Bayes' rules (using it on the output of a pre-trained model, which is interesting), which is solid and theoretically justified.
strength_weakness: strength_weakness: 1. Benchmark datasets follow the previous papers, which is good. However, these tasks seem not the most relevant to what you propose in this paper. Are there some scenarios where the fine-tuning the last layer is not feasible? 

2. The paragraph between 253-266 is confusing, which can be moved to other section. Appendix perhaps.

3. One major drawback of the presentation is the algorithm tables. The equations are still not the best way for practitioners. Main algorithm table should be moved in Section 4. After looking at the algorithms, it is easy to find that the proposed method is easy to be implemented. However, I cannot feel that after I read the section 4.

4. How to choose Padding or Watermarking in practice?

5. Experiments to increase n are not necessary. Could the authors explain why it matters? As long as the method is valid, the accuracy will be improved when n increase.
questions: questions: See the weaknesses above.
limitations: limitations: No concerns for this paper.

Comments: title: Many thanks for your support!
Thanks for your support!
Response to Rebuttal
comment: Dear Reviewer 885t,

We are glad to hear that your concerns are addressed. Thanks for your support.

Best,

Authors of Submission7993
My concerns have been addressed. I will support this paper with the positive score.
Dear Reviewer zxuh,

Many thanks for your support and increasing your score to 8! We will merge all comments into the updated version.

Best regards,

Authors of Submission7993
Thank you very much for your review. My concerns are addressed, and I have raised my score.