Title: VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Parameter-efficient fine-tuning

Prompt tuning

Transfer learning
Abstract: In the era of large-scale foundation models, fully fine-tuning pretrained networks for each downstream task is often prohibitively resource-intensive. Prompt tuning offers a lightweight alternative by introducing tunable prompts while keeping the backbone frozen. However, existing visual prompt tuning methods often fail to specialize the prompts or enrich the representation space--especially when applied to self-supervised backbones. We show that these limitations become especially pronounced in challenging tasks and data-scarce settings, where effective adaptation is most critical. In this work, we introduce VIPAMIN, a visual prompt initialization strategy that enhances adaptation of self-supervised models by (1) aligning prompts with semantically informative regions in the embedding space, and (2) injecting novel representational directions beyond the pretrained subspace. Despite its simplicity--requiring only a single forward pass and lightweight operations--VIPAMIN consistently improves performance across diverse tasks and dataset sizes, setting a new state of the art in visual prompt tuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIdentifying the uniform attention and subspace collapse limitation, this paper introduces VIPAMIN, a initialization technique that addresses these issues via attention-guided matching and orthogonal subspace injection. Specifically, VIPAMIN aligns prompts with semantically informative regions in the embedding space, and injects representational directions beyond the pretrained subspace. VIPAMIN are validated under few-shot settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper introduces several novel technical modules—specifically, “Selecting Tokens for Prompt Propagation” and “Injecting Semantic Bias into Attention Outputs”—both of which represent innovative contributions.\n\n**Weaknesses**\n\n1. **Lack of Detailed Visualization for Semantic Token Selection and Bias Injection**\n   Although Figure 1 suggests the importance of “semantic token selection” and “bias injection,” the paper does not provide explicit visualizations that identify which tokens are being selected or which biases are being injected. Without such detailed illustrations, it is difficult to evaluate whether the selected tokens indeed capture the intended semantics or whether the injected biases align with the theoretical motivation.\n\n2. **Unclear Definition of “Failure Modes” in VPT**\n   The term “mode” in Figure 2’s “Failure modes of VPT” is not defined, leaving the reader uncertain about the specific failure cases being depicted. This ambiguity prevents a clear understanding of the limitations of VPT and hinders the ability to assess whether the proposed method effectively addresses those failure modes.\n\n3. **Ambiguity Surrounding the “Spec” Metric in Table 3**\n   Table 3 reports a metric labeled “Spec,” but the paper does not define this term. Furthermore, “Spec” underperforms relative to baseline methods, yet no detailed analysis is provided to explain why. The absence of a clear definition and an in-depth investigation into the poorer performance undermines the reader’s ability to interpret the results and understand the method’s limitations.\n\nQUESTIONS:\n1. In Figure 1, the importance of “semantic token selection” and “bias injection,” are observed, but is there a more explicit visualization? For example, which specific semantic tokens are selected, and what precise biases are being injected?\n2. In Figure 2, the “Failure modes of VPT” are mentioned. To which modes does “mode” refer? Please clarify the specific failure modes being illustrated.\n3. In Table 3, what does the metric labeled “Spec” denote? Its performance is inferior to several baselines—can the authors provide a detailed analysis explaining this discrepancy?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VIPAMIN, a visual prompt initialization strategy addressing uniform attention and subspace collapse in self-supervised Visual Prompt Tuning (VPT).\n\nSTRENGTHS AND WEAKNESSES:\nIts strengths include a simple design requiring minimal computation and competitive results on VTAB-1k and few-shot benchmarks. However, significant weaknesses exist. The method fails to consistently outperform top baselines like VFPT in key experiments, lacks robustness validation under distribution shifts, and its novelty is limited by superficial comparisons to prior work. The orthogonalization mechanism lacks theoretical grounding, and critical implementation details (e.g., hyperparameter sensitivity) are underexplored.\n\nQUESTIONS:\n1. VIPAMIN does not surpass state-of-the-art methods (e.g., VFPT achieves 71.33% vs. VIPAMIN-Deep’s 71.23% on VTAB-1k MoCo-v3), weakening claims of superiority.\n2. Experiments omit validation under distribution shifts (e.g., corrupted data or domain gaps), despite identifying this as a core VPT failure mode.\n3. The orthogonalizing module lacks theoretical motivation (e.g., no formal proof of subspace collapse mitigation), and comparisons to SPT/rand overlook deeper methodological distinctions.\n4. Critical parameters k \\lamda are tuned empirically without analysis of their impact on stability or generalization.\n5. Figure 7 shows subspace collapse only in the first transformer block, yet VIPAMIN-Deep applies orthogonalization to all blocks without justification. This inflates computation without empirical benefit.\n\nLIMITATIONS:\n1. The method is tested only on ViT-B/16 backbones. Results may not hold for larger models or non-ViT architectures, limiting generalizability claims.\n2. The paper states VIPAMIN is \"lightweight\" but provides no latency/memory measurements versus baselines. Real-world overhead for orthogonalization (Eq. 9) and large-batch processing (B=256) remains unquantified.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a visual prompt initialization strategy, termed VIPAMIN, for self-supervised Vision Transformers that aims to tackle near-uniform prompt attention and collapse into the frozen embedding subspace of the VPT. VIPAMIN aligns each prompt with semantically coherent image tokens and injects an orthogonal component that expands representation space in single forward pass. Empirical evaluations on VTAB-1k and FGVC demonstrate improvement over VPT on few-shot classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This paper has solid motivation with both empirical and theoretical analysis on why VPT failed under distribution shift, attended every patch equally, and contributed little to diverse representational directions. VIPAMIN is proposed to mitigate the gap based on these observations.\n2. The proposed method relies on two matrix operation, which is comparatively efficient. \n3. VIPAMIN shows consistent better performance in few-shot benchmarks against VPT.\n\nWeakness:\n1. Currently experiments are exclusively on image classification, and the author should also consider few-shot settings in other tasks like segmentation and object detection to demonstrate this method's generalization beyond classification problems.\n2. VIPAMIN falls short on Specialized group, which requires novel semantic representations, even with the orthogonalizing module comparing to SPT. The author should include more discussion to this performance gap.\n3. The author claimed that vanilla VPT failed to attend to meaningful token. The author should compare with, or at least discuss, semantic driven VPT (e.g. DA-VPT[1]). \n4. The orthogonalizing module creates $p^{orth}$ by projecting a random vector prompt outside the frozen subspace. Although this expands rank, there's nothing ensures the new direction is meaningful for the task. \n\n[1] DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers, CVPR 2025\n\nQUESTIONS:\nRefer to weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}