Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model

Authors: Yushu Li, Yongyi Su, Adam Goodge, Kui Jia, Xun Xu
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Vision-Language Model, Label Propagation, Training-Free

Abstract:
Vision-language models (VLMs) have revolutionized machine learning by leveraging large pre-trained models to tackle various downstream tasks. Although label, training, and data efficiency have improved, many state-of-the-art VLMs still require task-specific hyperparameter tuning and fail to fully exploit test samples. To overcome these challenges, we propose a graph-based approach for label-efficient adaptation and inference. Our method dynamically constructs a graph over text prompts, few-shot examples, and test samples, using label propagation for inference without task-specific tuning. Unlike existing zero-shot label propagation techniques, our approach requires no additional unlabeled support set and effectively leverages the test sample manifold through dynamic graph expansion. We further introduce a context-aware feature re-weighting mechanism to improve task adaptation accuracy. Additionally, our method supports efficient graph expansion, enabling real-time inductive inference. Extensive evaluations on downstream tasks, such as fine-grained categorization and out-of-distribution generalization, demonstrate the effectiveness of our approach. The source code is available at https://github.com/Yushu-Li/ECALP.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730010769722

thread_type:
official_review

<thread_object>
creator: reviewer_9agk

<content>
summary:
This paper proposes graph-based efficient adaption and inference for vision-language models. Specifically, the paper maximizes the use of test samples through iterative label propagation without the need for task-specific tuning. Additionally, the paper introduces a novel method to mitigate biases in VLMs through context-aware re-weighting and a interesting approach for identifying KNNs within each modality. Extensive experiments on various downstream tasks demonstrate the effectiveness of the proposed graph-based method.

strengths:
1. The proposed use of K-nearest neighbors within each modality and context-aware edge re-weighting are novel approaches that align well with the property of pre-trained VLMs.
2. The paper is well-written and easy to understand.
3. Extensive experiments were fairly conducted with recent baselines across a range of datasets and architectures. 
4. The paper improves the feasibility of the proposed method by using dynamic graph expansion, supported by a practical analysis of wall-clock time in Table 6 and time complexity in Appendix A.1.

weaknesses:
I think the paper has no significant weaknesses.
1. Evaluating the performance degradation when using one-step label propagation instead of iterative label propagation could provide valuable insights for practitioners. This analysis would shed light on the trade-off between reduced complexity and potential accuracy decline, assisting in making informed decisions for applications where computational efficiency is crucial.
2. When utilizing an additional 16-shot training samples in Table 2, it appears that ECALP uses the training data as a component of the graph rather than for training. Wouldn’t it be possible to apply ECALP in addition to traditional prompt learning methods such as CoOp and CoCoOp?

questions:
1. In Figure 1, it seems that the second image encoder and text encoder should be switched.
2. In Line 161, how can we obtain multiple textual prompts $z_{cm}$ for the $c$-th class? It seems that the value of $M$ can vary from class to class.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484549210

thread_type:
official_review

<thread_object>
creator: reviewer_uwxi

<content>
summary:
The paper proposes a approach for adapting vision-language models (VLMs) using efficient, context-aware label propagation. It introduces dynamic graph construction to improve inference accuracy without task-specific tuning, and employs context-aware feature re-weighting for better task adaptation. The method demonstrates superior performance and efficiency across various downstream tasks, highlighting its potential in zero-/few-shot scenarios.

strengths:
1. The introduction of dynamic graph expansion is innovative, allowing for real-time adaptation and efficient use of test samples. 
2. The method enhances feature relevance by re-weighting based on context, which is a novel approach to improving model adaptability to downstream tasks. This can potentially lead to better performance in diverse scenarios.
3. Employing an iterative solution rather than a closed-form one reduces computational costs and enhances scalability. This is particularly beneficial for handling large datasets.

weaknesses:
1. More detailed motivation behind the model design is preferred. It is important to explain why the authors propose the method in this work.
2. The method may be complex to implement, particularly for large-scale or varied tasks. Detailed guidelines for implementation would be beneficial.
3. The approach assumes that the context-aware feature re-weighting will generalize well across different tasks, but this might not hold true for all types of data or in cases with significant domain shifts.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607288877

thread_type:
official_review

<thread_object>
creator: reviewer_qcpb

<content>
summary:
This paper proposes a graph-based approach for label-efficient adaptation and inference. The method dynamically constructs a graph based on available samples to enable real-time inductive inference.

strengths:
1. The method requires no additional unlabeled support set and effectively leverages the test sample manifold through dynamic graph expansion.
2. This paper is clear and easy to follow.
3. This paper conducts extensive evaluations on diverse downstream tasks.

weaknesses:
1. Compared to the SOTA methods, the performance improvement seems limited. Additionally, the second-highest performance can be highlighted with an underline for clarity.
2. In the computational efficiency section, does the testing time for ECALP include the dynamic graph construction process, or does it only account for the label propagation time?

questions:
1. It appears that ECALP requires additional storage. How does its CUDA memory compare to that of previous methods?
2. If the initial testing samples are used to create an incorrect adjacency graph, will this lead to a cumulative error?
3. In corrupted downstream tasks, is ECALP still effective when the severity is low (e.g., at level 1)?
4. Is the method sensitive to the label propagation iterations T?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610927787

thread_type:
official_review

<thread_object>
creator: reviewer_9d8m

<content>
summary:
The paper proposes ECALP, a novel graph-based framework for training-free adaptation of vision-language models. The proposed method constructs a graph over text prototypes, optional training samples and testing samples in a simple, dynamic fashion, and adopts label propagation for inference. Furthermore, edge re-weighting is carefully designed to take into account the semantic context of different classes. Throughout training and inference, the framework introduces little augmentation and hyper-parameter search, showing its efficiency and practicality. Finally, comprehensive experimental results over a wide range of datasets on zero-shot/few-shot fine-grained classification, style-transfer and out-of-distribution detection tasks show significant improvement in performance of the proposed approach over previous methods.

strengths:
1. Comprehensive experiments and great results:

The paper provides a comprehensive comparison of ECALP against a number of highly competitive prior works, and carried out experiments on a variety of datasets as well as visual tasks, which is sufficient to validate the approach. The empirical results clearly demonstrate that ECALP achieves significantly better performance than previous methods. The experimental design, including the main setting, comparison baselines as well as ablation studies, are well justified. Moreover, the method proves to be computationally efficient.

2. Nice presentation and clear structure

The figures in this paper are well-designed and effectively highlight the pipeline as well as advantages of the proposed framework. The formulation, presentation of the results as well as the visuals are aesthetically pleasing and easy to follow.

3. Clear motivation and good writing

The introduction and abstract are well written and lays a good foundation for the paper. It is easy to see the motivation behind the work.

weaknesses:
1. In Figure 1, the inputs to the second image encoder and the text encoder seem to be mixed up. 

2. As is discussed in the paper, ECALP builds upon similar ideas with ZLaP (i.e. label propagation), and although I acknowledge that the proposed method achieves improvement in performance and eliminates the need for additional unlabelled datasets, the contribution seems somewhat limited.

3. In the detailed few-shot results provided in the appendix, it seems that ECALP does not always achieve the best performance. Especially on datasets like OxfordPets and Food101, the accuracy of ECALP even goes down as the number of samples per class increases. This weakens the claim of ECALP’s robustness in low-shot scenarios.

questions:
1. As is mentioned above, I wonder if the authors could offer some explanation on ECALP’s behavior on certain datasets in low-shot settings.

2. In the ablation study, in Table 4, it’s interesting to see that without Text Reweight, the model achieves slightly better performance under the 16-shot setting on ImageNet. It would be nice if the authors could shed some light on this phenomenon.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732206911998

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers' Comments on Manuscript Revision

comment:
We appreciate the constructive feedback provided by the reviewers, which has significantly helped us enhance the clarity and depth of our manuscript. Below, we outline the specific revisions made in response to each comment:

1. **Combination with CoOp**

    In **Appendix A.2.4**, we explored integrating our method, ECALP, with the traditional prompt learning approach CoOp. This combination was tested with varying numbers of training samples from 1 to 16 shots. Our results showed that CoOp combined with ECALP further enhances CoOp's performance significantly. This demonstrates that ECALP can be seamlessly integrated with traditional prompt learning methods to boost their performance, further validating our applications.

2. **Impact of Label Propagation Iterations**

    In **Appendix A.2.5**, we investigated how varying the number of label propagation iterations affects performance. Increasing iterations generally improved accuracy by better utilizing the data manifold but also increased computational demands. We found an optimal balance by selecting a moderate number of iterations to maintain efficiency without sacrificing much accuracy. This illustrates the trade-off between computational cost and performance enhancement. Moreover, multiple iterations of label propagation show significant advantages over a single iteration, indicating the effectiveness of our propagation approach.

3. **Performance on Different Corruption Severities**

    In **Appendix A.2.6**, we tested our method on tasks with various levels of data corruption. ECALP consistently outperformed the CLIP baseline by 3% to 5%, demonstrating robust handling of corrupted data. This robustness is crucial for real-world applications where data quality may vary significantly.

4. **Robustness to Bad Initialization**

    In **Appendix A.2.7**, we evaluated how well our approach handles poor initial conditions during testing. By dynamically updating relationships between data points, our method shows minimal performance drop even when starting with challenging data samples. This robustness is vital for deploying the method across diverse and unpredictable real-world tasks.

5. **Visualization Samples for Corruption Task**

    In **Appendix A.2.8**, we provide visualizations of sample images subjected to various types of corruption from the ImageNet-C dataset. These visualizations are intended to illustrate the types of distortions and challenges posed by severe corruption conditions.

6. **Correction in Figure 1**

    In **Fig. 1**, we corrected a mistake by swapping the positions of the second image encoder and text encoder. We thank the reviewers for pointing this out.

7. **Highlighting Second-highest Performance**

    In **Tab. 1, 2, and 3**, we have underlined the second-highest performance for clarity. Thanks for this suggestion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel framework, called ECALP, an approach designed to efficiently adapt vision-language models (VLMs) through the construction of a graph over text prototypes. ECALP can dynamically construct a graph over text prompts, few-shot examples, and test samples, using label propagation for inference without task-specific tuning. The experiments of this work is extensive and solid, with remarkable performance on several downstream tasks. Overall, I found the methodological design novel and the experimental results solid, and therefore would like to suggest accept this work.