HyperCLIP: Adapting Vision-Language models with Hypernetworks

Authors: Victor Akinwande, Mohammad Sadegh Norouzzadeh, Devin Willmott, Anna Bair, Madan Ravi Ganesh, J Zico Kolter
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: vision-language models, edge computing, CLIP, hyper-networks

Abstract:
Self-supervised vision-language models trained with contrastive objectives form the basis of current state-of-the-art methods in AI vision tasks. The success of these models is a direct consequence of the huge web-scale datasets used to train them, but they require correspondingly large vision components to properly learn powerful and general representations from such a broad data domain. This poses a challenge for deploying large vision-language models, especially in resource-constrained environments. To address this, we propose an alternate vision-language architecture, called HyperCLIP, that uses a small image encoder along with a hypernetwork that dynamically adapts image encoder weights to each new set of text inputs.  All three components of the model (hypernetwork, image encoder, and text encoder) are pre-trained jointly end-to-end, and with a trained HyperCLIP model, we can generate new zero-shot deployment-friendly image classifiers for any task with a single forward pass through the text encoder and hypernetwork. HyperCLIP increases the zero-shot accuracy of SigLIP trained models with small image encoders by up to 3% on ImageNet and 5% on CIFAR-100 with minimal training throughput overhead.

Decision: reject
Reviewer scores: 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730283873604

thread_type:
official_review

<thread_object>
creator: reviewer_qh13

<content>
summary:
This paper proposes HyperCLIP, a vision-language architecture, as an alternative efficient solution for large vision-language models. The approach leverages hypernetworks to dynamically adjust a smaller image encoder based on input text embeddings. This approach allows large vision-language models to operate effectively with reduced resource requirements, making them suitable for edge devices and resource-constrained environments. Experiments in the paper show that the proposed approach improves SigLIP zero-shot accuracy by 3% on ImageNet and 5% on CIFAR-100.

strengths:
1. The approach presents an efficient way of using smaller models to deploy vision-langauge models for resource-constrained real-world applications.
2. The adaptive approach that modifies weights at test-time for VLMs/CLIP is novel.
3. The datasets used in experiments align well with prior works.

weaknesses:
1. The approach doesn't generalize to broader VLMs especially the larger models. CLIP models are generally the smaller models among recent VLMs. I don't think the proposed approach generalizes to the larger VLM models such as LLaVa [1] that uses very large decoder-only transformer LLM as component.
2. CLIP models have wide range of applications, among which an important one is to use the visual features produced by the image encoder as inputs to downstream models. HyperCLIP reduces CLIP applications to only image classification.
3. The experiments are not solid. The paper claims that HyperCLIP performs well on ImageNet in zero-shot setting. However, In Appendix A.5, the paper writes 'selects samples containing text that overlaps with ImageNet class names', which shows that the ImageNet class names are used in data collection which is used in training.  - so it's not truly zero-shot.


[1] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

questions:
Can you explain why do you think your experiments are conducted in zero-shot setting? 

My biggest concern is weakness #3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730368116893

thread_type:
official_review

<thread_object>
creator: reviewer_scqd

<content>
summary:
The paper introduces HyperCLIP, a novel architecture that adapts vision-language models by using a hypernetwork to dynamically adjust the weights of a small image encoder for each new set of text inputs. 
This approach enables the creation of zero-shot deployment-friendly image classifiers with a single forward pass through the text encoder and hypernetwork. 
HyperCLIP is designed to overcome the challenges of deploying large models in constrained environments by offering a smaller, yet powerful alternative.
The model increases zero-shot accuracy on ImageNet and CIFAR-100 with minimal training throughput overhead.

strengths:
1. The paper proposes using a hypernetwork to generate weights for a smaller image encoder within the SigLIP contrastive pre-training framework, allowing for task-specific specialization without extensive retraining.
2. HyperCLIP achieves significant improvements in zero-shot accuracy on ImageNet and CIFAR-100, with minimal overhead, making it suitable for resource-constrained environments.
3. The method is compatible with any type of contrastive pre-training, enhancing its versatility.
4. The paper demonstrates that HyperCLIP can improve the performance of small vision models on standard benchmarks by adapting only the normalization layers.
5. HyperCLIP has the potential to democratize computer vision by enabling the deployment of high-performing models on devices with limited resources.

weaknesses:
1. By focusing on adapting only normalization parameters, the paper may not fully leverage the potential of hypernetworks to modify other model parameters.

questions:
1. Will you plan to release the code ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704729120

thread_type:
official_review

<thread_object>
creator: reviewer_grgs

<content>
summary:
This paper proposed a hyper-network to generate parameters for normalization layers in the vision model of CLIP. The paper is aimed at solving the efficiency problem of CLIP on edge computing scenarios. The experiment results show that the proposed method can get better zero-shot accuracy over the CLIP baseline.

strengths:
1. As shown in Table 2, the proposed hyper-network can improve the zero-shot accuracy on ImageNet by up to 3%.

weaknesses:
**Unclear Motivation and Unsupported Claims by Experiment Results**

The motivation for using a hyper-network to address CLIP's efficiency challenges in edge computing applications is unclear. The introduction does not clearly explain this choice. The experiments show performance improvement over a baseline without a hyper-network but fail to address the efficiency problem. In other words, the hyper-network enhances performance only in small-scale models rather than improving CLIP's efficiency directly.

**Poor Writing and Difficult to Follow**

* Several English expressions are incorrect, likely due to machine translation. For instance, in lines 041-044: "These methods often include first training a large model, and then applying the chosen technique in a post-hoc fashion. Additionally, many of these methods can require specialized hardware support for actual memory and latency reduction." This section is hard to understand.

* The text includes many excessively long sentences. For example, in lines 045-050: "We propose a method of pre-training vision-language models (VLMs) that allows us to derive small vision models appropriate for deployment on edge devices without requiring multi-step training procedures or any specialized hardware. We suggest a new contrastive learning architectural design based on hypernetworks that improves performance over current state-of-the-art baselines and can additionally be used in conjunction with a variety of model compression methods for further memory or latency improvements."

**Limited References and Literature Review**

* The introduction has limited references, mentioning only three prior works (Sun et al., 2023a; Dettmers et al., 2022; Frantar & Alistarh, 2023). Including more background citations would strengthen the paper’s credibility.
* The related work section is limited to a single paragraph (L488-503). A more thorough literature review, including discussions on established methods like LoRA [a] and adaLN [b], would enhance this section and contextualize the paper’s contributions.

[a] Edward et al., "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022.

[b] William et al., "Scalable Diffusion Models with Transformers." ICCV 2023.

questions:
1. Why the text transformer (a width of 768, 8 heads, feed-forward dimension of 2560) is not a small model, since the proposed HyperCLIP is targeted for compute-restricted scenarios. 
2. Is the HyperCLIP trained from scratch or initialized from SigLIP?
3. Which part of the experiments are zero-shot results, and which part are linear probing results?
</content>
</thread_object>

</thread>


Metareview:
metareview:
### **Summary**
The paper proposes **HyperCLIP**, a vision-language architecture designed for efficient deployment of CLIP-based models in resource-constrained environments. The key innovation lies in using a hypernetwork to dynamically adapt the weights of a smaller image encoder based on text inputs. HyperCLIP achieves competitive improvements in zero-shot image classification accuracy on ImageNet and CIFAR-100, demonstrating its potential for edge computing scenarios. The paper also provides evaluations on retrieval tasks, highlighting broader applicability beyond image classification.

### **Strengths**
1. **Innovative Use of Hypernetworks**: 
   - The approach demonstrates a novel application of hypernetworks for adapting normalization layers in CLIP’s image encoder, enabling task-specific specialization.
2. **Practical Relevance**:
   - HyperCLIP addresses a pressing need to deploy high-performing models on resource-constrained devices while maintaining competitive accuracy.
3. **Empirical Performance**:
   - HyperCLIP achieves up to a 3% improvement in zero-shot accuracy on ImageNet and 5% on CIFAR-100 compared to SigLIP.
4. **Compatibility and Versatility**:
   - The method integrates seamlessly with existing contrastive learning frameworks, offering a practical pathway to improve edge deployment without significant additional overhead.

### **Weaknesses**
1. **Evaluation Concerns**:
   - The paper’s claim of a zero-shot setting is undermined by the use of ImageNet-21k class names for filtering training data. While common in CLIP-based benchmarks, this setup introduces potential bias and reduces the perceived novelty of the approach.
   - Generalization to larger vision-language models (e.g., LLaVA) or tasks beyond image classification is not explored, limiting broader applicability.
2. **Presentation and Writing**:
   - Reviewers noted unclear motivations, overly complex sentences, and insufficient clarity in experimental setups. The paper would benefit from significant editing for readability.
3. **Limited Experimental Scope**:
   - Ablation studies on the choice of components (e.g., hypernetwork architecture or the focus on normalization layers) are absent.
   - The related work section is sparse, missing connections to well-established methods like LoRA and deeper discussions on compression techniques.
4. **Narrow Applications**:
   - The method primarily focuses on zero-shot image classification and retrieval tasks, narrowing its potential impact within the broader domain of vision-language applications.

### **The reason and decision**
While the proposed approach is novel and promising, it falls short in terms of evaluation rigor, clarity, and generalizability. The issues surrounding the zero-shot claims, combined with limited exploration of broader applications, prevent it from meeting the standard for acceptance at ICLR.