Title: An Effective Levelling Paradigm for Unlabeled Scenarios
Venue: NeurIPS 2025
Accepted: yes
Keywords: Unlabeled Scenarios

deep learning

generalization

few-shot
Abstract: Advancements in direct-integration fine-tuning frameworks have underscored their potential to enhance the performance of labeled scenarios and tasks. To enhance the generalization of different categories in the same dataset, some methods have added visual loss to these frameworks for unlabeled scenarios. However, the performance of these methods through visual loss does not improve significantly in domain generalization and cross-dataset generalization tasks. This may be attributed to the uncoordinated learning of the two-modalities alignment and visual loss. To mitigate this issue of uncoordinated learning, we propose a novel method called Levelling Paradigm (LePa) to improve performance for unlabeled tasks or scenarios. The proposed LePa, designed as a plug-in module, dynamically constrains and coordinates multiple objective functions, thereby improving the generalization of these baseline methods. Comprehensive experiments have shown that our design can effectively address generalized scenarios and tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a prompt tuning for VLM on unlabeled tasks or scenarios. PromptSrc or FAP has improved performance on labeled tasks using VLM like CLIP, but they often underperform on unlabeled tasks, particularly in domain generalization and cross-dataset generalization. This shortfall is attributed to uncoordinated learning of textual and visual embeddings in these methods. This paper proposes a plug-in method (LePa) that coordinates the optimization of multiple objectives (in text and visual modalities). LePa shapes the loss landscape to improve robustness during fine-tuning, especially in unlabeled or domain-shifted scenarios. Existing VLP designs align text and vision modalities separately, which might cause visual feature tuning (learned from image encoders) to fail or diverge, leading to unbalanced or degraded generalization, especially in domain generalization and cross-dataset tasks. It outperforms several baselines on PromptSRC/FAP/TAP/MaPLe on 11 datasets.\n\nSTRENGTHS AND WEAKNESSES:\nLePa introduces a lightweight method to regularize and balance learning in vision-language prompting by aligning the objectives of text-to-image alignment and visual tuning. It is compatible with 4 existing methods on vlm prompting and effective in 11 datasets (mostly) but may suffer from limitations in scalability, hyperparameter tuning and manual design dependecies. In terms of scalability, LePA requires manual prompts and handcrafted feature vectors to initialize score lists. This adds preprocessing overhead and may not scale to large prompt spaces or tasks without well-defined templates. Moreover, ablation on each term $\\gamma_{1}$ and $\\gamma_{2}$ are important (Table 5a) only presents $\\gamma_{1}$ and $\\gamma_{2}$ are positive scenarios. It has several interpretability issues as the calibration logic is somewhat ad hoc; it relies on choosing \"worst\" prompts and averaging their performance, which may be sensitive to prompt variability or batch statistics.Assumes that minimizing the alignment gap between $L_{HCE}$ and $L_{visual}$ improves generalization. This may not hold if both branches are misaligned for fundamental reasons (e.g., semantic mismatch). Unlabeled tasks might contain semantic mismatch and did the authors perform experiments on samples in unlabeled tasks that contain semantic mismatch data?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-02"}

comments: SUMMARY:
This paper addresses the suboptimal parameter configuration issue in unlabeled tasks. The authors propose a plug-in method named Levelling Paradigm (LePa) to alleviate the problem of uncoordinated learning in such scenarios. The proposed method is compatible with vision-language prompting frameworks and dynamically constrains the objective functions of two-modal alignment and visual alignment. The method is evaluated on 11 real-world unlabeled tasks, and demonstrates superiority over existing approaches in multiple aspects.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper is clearly written and logically organized, with no significant grammatical issues.

2. The proposed method shows strong generality and can be integrated as a plug-in into various VLP frameworks.

3. The experimental results are generally strong across most tasks, although there are a few cases where the performance does not meet expectations.

Weeknesses:

1. The method shows limited performance on the source dataset (labeled ImageNet) used for training.

2. The approach relies on several hyperparameters, which may affect reproducibility and tuning difficulty.

3. The proposed method heavily depends on manually crafted prompts. It remains unclear how well it would perform under fully automatic prompt generation.

QUESTIONS:
1. Why might the two objectives,  $L_{CE}$  and $L_{Visual}$ exhibit uncoordinated learning? Is there an intuitive explanation for this phenomenon?

2. Since the method relies on manually crafted prompts, what would happen if fully automatic prompts (e.g., generated by LLMs) were used instead? Would the performance improve or degrade?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a plug-in module, Levelling Paradigm (LePa), to enhance the generalization of directly-integrated Vision-Language Prompting (VLP) frameworks, particularly in domain and cross-dataset generalization under unlabeled settings. LePa addresses the issue of uncoordinated optimization between vision-language alignment and visual feature preservation by introducing a dynamic constraint objective. The method is compatible with multiple VLP baselines and shows consistent gains across 11 datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tThe proposed LePa is designed as a plug-in module, making it compatible with a variety of existing VLP methods, which improves its applicability and practical value.\n\n2.\tThe method is evaluated across 11 diverse datasets covering multiple generalization scenarios (base-to-novel, domain generalization, cross-dataset), demonstrating consistent performance improvements.\n\n3.\tThe writing of this paper is good.\n\n\nWeaknesses\n\n1.\tWhy does LePa not consider $L_{textual}$? Is there any experimental evidence supporting the claim that uncoordinated learning occurs between $L_{CE}$ and $L_{visual}$? A clear justification is crucial to support the motivation of this work.\n\n2.\tThe authors claim that uncoordinated learning occurs between $L_{CE}$ and $L_{visual}$. However, it is unclear why LePa does not directly minimize the difference between those two losses. Also, this paper lacks experimental analysis or ablation studies to verify the effectiveness of this direct minimization approach.\n\n3.\tTo alleviate the uncoordinated learning between two losses, a vanilla approach is to assign different hyper-parameters to control their contributions, or, more advanced methods could treat those hyper-parameters as learnable parameters. However, this paper lacks a discussion or experimental evaluation regarding such strategies, which are crucial for multi-losses optimization.\n\n4.\tWhy does the proposed method ignore $L_{textual}$ in Eq. (6)? If so, it would be better to remove  “Textual” from Figure 2.\n\n5.\tTo alleviate the uncoordinated learning between $L_{CE}$ and $L_{visual}$, this work introduces a new loss. However, there remains a risk of uncoordinated learning between the new loss and the original losses.\n\n6.\tSince DePT is also a plug-in method, it would be better to include a discussion in the main text comparing DePT and LePa. For example, what are the disadvantages of DePT, and how does LePa address or avoid them?\n\n7.\tIn Section 4.3, what does the term embedding length refer to? \n\n8.\tIn the Related Work section, it would be beneficial to provide more details about the VLP methods used in the comparison experiments, i.e., MaPLe, PromptSRC, FAP, and TAP.\n\nSmall issues:\n\n1.\tIn line 87, “base-to novel” should be “base-to-novel”\n\n2.\tIn line 89, “several representative VLP works incorporates” should be “several representative VLP works incorporate”\n\n3.\tIn line 90, “task-specific feature” should be “task-specific features”\n\nQUESTIONS:\nPlease refer to the weaknesses outlined above.\n\nLIMITATIONS:\nThe limitations are included in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Leveling Paradigm (LePa) loss, an additional regularization term that can be plugged into the prompt-tuning of CLIP models for image classification. Prompt-tuning with CLIP has been shown to result in overfitting to the target-task making the learned model worse at image-classification than the base CLIP model; various methods have attempted to mitigate this by adding additional regularization terms to keep the representations generated from the learnable prompts close to the representations generated without (e.g. keeping the learned representations close to the pre-trained representations). However, balancing these two objectives can be difficult, leading to what the authors term unbalanced learning. LePa is designed to address unbalanced learning by adding an additional constraint to ensure that the visual regularization term and the task-specific cross-entropy loss decrease at similar rates.\n\nMore specifically, LePa operates as a plugin to the traditional prompt-tuning CLIP paradigm; it computes an additional cross-entropy loss, termed the \"Robust Cross-Entropy Loss\", which is constructed using the textual features of the class prompts (without the prompt-tuning parameters) and the image representation from the image encoder with the learnable image prompt. The HCE is computed as the cross-entropy between the ground truth label and the cosine-similarity between the image representation and the k-worst class prompts (assuming a set of class prompts are being used for diversity as in PromptSRC). The LePa loss is the constructed by taking the L1 norm of the difference of the HCE loss and the visual regularization loss; in other words, the LePa loss is designed to ensure that the visual regularization term and the HCE loss decrease at similar rates.\n\nThe authors validate LePA as a plugin for various CLIP-tuning techniques, including PromptSRC, FAP, TAP, and MaPLe. They show that, in all cases, the LePa loss results in the best generalization to out-of-domain task sets and novel label sets, despite not achieving the best generalization on the source distribution. These results suggest that LePa can help to learn prompt parameters that are more generalizable for image classification. The authors also perform some analysis of their method, including ablating over the hyperparameters of each loss term, over the length and depth of the learnable embeddings, the number of worst-case class prompts used, and the underlying ViT instances.\n\nSTRENGTHS AND WEAKNESSES:\nSummary of Strengths\n\n\nThe empirical results of the proposed LePa loss are quite extensive and all show pretty convincingly that LePa is more effective than the base tuning methods, as well as the DePT plugin, at preserving model generalization to unseen datasets and classes during image classification prompt-tuning. Moreover, the method is extremely simple to implement and plugin, and it is highly modular, as it appears to be applicable to a number of popular CLIP-tuning methods.\n\nSummary of Weaknesses\n\n\nAlthough the method's results are strong, the motivation for various components of the method are not well-explained or motivated in the paper. The method makes several key decisions in computing the LePa loss that are not empirically validated or even, as far as I can tell, well motivated in the paper. For instance, LePa uses the HCE loss which is computed using text representations from the class prompts without input from the tuned parameters. It is not clear to me why this decision is made, or is necessary. The LePa loss is constructed using the HCE loss and the Visual regularization loss but the textual regularization loss is ignored; this is fine if the problem in CLIP tuning is always that there is an imbalance between the CE loss and the Visual loss, and the Textual loss is optimized well each time but there is not empirical evidence given to suggest this nor is it shown that the method actually balances these losses as expected during training. Finally, while the decision to only consider the top-k empirically worse class prompts is validated in the ablations, the motivation for doing this to begin with is, in my opinion, not stated clearly during the introduction of the method.\n\nThe point about comparing the rate at which losses are optimized is especially significant to me. L1 and Cross-Entropy losses are not necessarily on the same scale, and it is not clear to me that taking the L1 loss between them is a principled way to ensure that the losses get optimized in a balanced manner. I think the paper would benefit significantly from showing the results of how the trajectory Visual and CE loss imbalance is changed by adding the LePa loss to the models.\n\nFinally, while the ablation over the number of worst-cases considered is useful, as it helps us to understand the role of aggregating a few worst-case prompts, the ablations over e.g. model size, embedding size, etc. don't make much sense as stand-alone results. Suggesting that the method is effective on different architectures doesn't make sense without the context of the performance of those architectures without the LePa loss applied. Similarly, it is not clear whether the ablations over hyper-parameters will transfer to other tasks, e.g. if I was going to use the LePa method for my own task I would likely still need to tune these parameters myself. So their purpose in the paper isn't clear to me.\n\nQUESTIONS:\nCould you clarify what this sentence means? I have trouble following it (line 95-97): \"performance growth is not significant through the visual embeddings learning directly based on textual embeddings learning.\"\n\nAdditionally, if you could address my issues around the motivation and clarity of the decisions made in constructing LePa that would be great!\n\nFinally, is there any empirical evidence from your experiments that the LePa loss actually serves to improve the balance between the Visual and CE losses throughout training?\n\nLIMITATIONS:\nI think they discuss the limitations of their proposed method well, but there is not discussion on the limitations of how they analyze the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-12"}