Title: un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
Venue: NeurIPS 2025
Accepted: yes
Keywords: CLIP model

unCLIP generative model

model inversion

visual detail capture
Abstract: Contrastive Language-Image Pre-training (CLIP) has become a foundation model and has been applied to various vision and multimodal tasks. However, recent works indicate that CLIP falls short in distinguishing detailed differences in images and shows suboptimal performance on dense-prediction and vision-centric multimodal tasks. Therefore, this work focuses on improving existing CLIP models, aiming to capture as many visual details in images as possible. We find that a specific type of generative models, unCLIP, provides a suitable framework for achieving our goal. Specifically, unCLIP trains an image generator conditioned on the CLIP image embedding. In other words, it inverts the CLIP image encoder. Compared to discriminative models like CLIP, generative models are better at capturing image details because they are trained to learn the data distribution of images. Additionally, the conditional input space of unCLIP aligns with CLIP's original image-text embedding space. Therefore, we propose to invert unCLIP (dubbed un$^2$CLIP) to improve the CLIP model. In this way, the improved image encoder can gain unCLIP's visual detail capturing ability while preserving its alignment with the original text encoder simultaneously. We evaluate our improved CLIP across various tasks to which CLIP has been applied, including the challenging MMVP-VLM benchmark, the dense-prediction open-vocabulary segmentation task, and multimodal large language model tasks. Experiments show that un$^2$CLIP significantly improves the original CLIP and previous CLIP improvement methods. Code and models are available at https://github.com/LiYinqi/un2CLIP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to improve the fine-grained detail representation capability of a CLIP with a reconstruction loss. The proposed method, un^2CLIP, achieves this by introducing the image generator of unCLIP (a previous work that learns a reconstruction model upon the clip feature). The original image encoder is fine-tuned with image data only, using the reconstruction loss as guidance.  The authors conducted evaluation on MMVP-VLM, open-vocabulary segmentation and MLLMs. The evaluations and ablation studies justified the design of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* The problem studied in this work is interesting. The poor fine-grained visual representation capability of CLIP's visual encoder has long been a problem in the MLLM community.\n\n* The proposed method is straightforward, easy to understand and implement. \n\n\n**Weaknesses:**\n\n* The proposed method relies on a pretrained image generator from unCLIP, which limits its generalization to other CLIP encoders since not every CLIP model has a corresponding image generator.\n\n* Some evaluations are missing, such as training with image-text pair, evaluation on CLIP related benchmarks (see questions below). \n\n* Utilizing generation models to improve the representation capability of an image encoder is not a very new paradigm, the novelty of the proposed method is relatively limited.\n\n\nOverall, I think this paper is good and interesting, but the novelty seems to be relatively limited. Adding more experiments can be helpful to justify the claims in the paper and help the readers better understand the property of the proposed method.\n\nQUESTIONS:\n* The authors freeze the image generator and only fine-tune the image encoder, so as to reduce language-shift when training the model with only image data. How about jointly tuning the encoder and generator with text-image data (since the training was conducted on CC-\n3M with image-text pairs)? I see the ablation in Table 4, but it seems the ablation was still conducted with image data only?\n\n* This work claims the freeze-generator-finetune-encoder training method reduces the language-shift problem, making the embedding space of the finetuned model still lie in the original space. How is the performance of the the fine-tuned model on zero-shot recognition/text-image retrieval? More evaluation on this would be helpful to justify this claim. \n\n* Some properties of the un^2CLIP are not evaluated. For example, in the concurrent work GenHancer, the performance of generation and understanding with respect to training iterations are different: better reconstruction does not always lead to better understanding. Does this also apply to the proposed method? More experiments are expected to better showcase the property of the proposed method.\n\n\n\n**Minor:**\n\nIt is not very clear to me how the diffusion loss is used as the indicator of the training difficulty. It would be better if the authors could elaborate on this at somewhere in the paper or the appendix.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an enhancement to CLIP's ability to capture visual details by utilizing gradient feedback from unCLIP. Unlike its closely related work, DIVA, this approach specifically addresses the issue of feature space misalignment between the input of the generator and the output of CLIP's image encoder. To implement this, the main technical modifications include replacing the text-to-image generator with unCLIP and removing the MLP projector. The effectiveness of the proposed method is demonstrated through various benchmarks that assess visual detail accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n- The proposed method demonstrates strong effectiveness, which is thoroughly validated through comprehensive experimental evaluations.\n\n- The paper's focus on addressing the misalignment between the input space of generators and the output space of the CLIP image encoder is reasonable.\n\n- The qualitative improvements in visual detail capture, as clearly illustrated in Figure 4, provide compelling evidence for the benefits of the fine-tuned CLIP approach.\n\n### Limitations:\n- Novelty:\nThe technical contribution in comparison to DIVA is somewhat limited.\n\n- Performance Trade-off:\nThe reported degradation in zero-shot classification performance (Tab.8, Appendix C) raises concerns. While the authors attribute this to the model's preference for fine-grained details, the superior performance of DIVA (a comparable approach) undermines this explanation. Further investigation into this performance discrepancy would strengthen the paper.\n\n- Implementation:\nTraining a separate unCLIP generator for each CLIP image encoder creates significant computational overhead and limits implementation flexibility. Moreover, the effect of the scale of the training dataset on the generalizability of the CLIP image encoder during the unCLIP training process remains an unresolved question that deserves further discussion.\n\n- Training Procedure:\nA two-stage ablation—first training the MLP projector and then fine-tuning the image encoder while keeping the projector's weights fixed—could provide clearer insights into the role of the projector head. The current joint training strategies in the ablation may introduce unnecessary instability to the optimization process.\n\nQUESTIONS:\nplease refer to the weakness part.\n\nLIMITATIONS:\nThe limitations are properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes un²CLIP, a method to enhance CLIP’s ability to capture fine-grained visual details without requiring additional text-annotated data or retraining from scratch. By leveraging the unCLIP generative framework—where a diffusion-based image generator is conditioned on CLIP image embeddings—the authors invert the process: they finetune the CLIP image encoder using the fixed unCLIP generator so that the encoder’s embeddings retain more image information while remaining aligned with the original text encoder. The proposed approach is shown to yield significant improvements over both baseline CLIP and prior generative model-based enhancements (notably DIVA) on benchmarks such as MMVP-VLM, open-vocabulary semantic segmentation (dense prediction), and vision-centric multimodal LLM tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Though the idea of inverting unclip is simple, the method is well-motivated, and the experimental results are quite good. It can improve the performance of CLIP on multiple tasks and benchmarks. Compared to other competitors, the improvement is significant.\n- The paper provides comprehensive experiments, including multiple baselines, detailed ablation studies, and both quantitative and qualitative results that strongly support its claims.\n- The paper is generally well-structured and readable, with logical development from motivation, through theoretical grounding, to empirical validation.\n\nWeaknesses:\n- My main concern is that the improvement of un^2CLIP on MLLM evaluations is quite marginal. As shown in table 3, it performs similar to DIVA on MMVP and general benchmarks, which indicates the general image understanding ability of un^2CLIP remains the same. While MLLM related tasks are important, this is one of the most important metrics to evaluate the image encoder. Also, on traditional tasks like zero-shot classification and retrieval, its performance drops compared to original CLIP, showing that generative training focuses more on details while ignoring the global semantics.\n\nQUESTIONS:\nSee the weaknesses. I generally think the method is simple and elegant, but the performance on some benchmarks hinders me from giving high scores.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies the unCLIP-inversion framework, which finetunes the image encoder of CLIP with a frozen generative module of unCLIP. Benefiting from the generator conditioned on the CLIP image embedding, the image encoder is able to capturing more rich visual details and improve the performance of MMVP-VLM, open-vocabulary segmentation and vision-centric MLLM tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. This work makes full use of the pre-alignment of CLIP image features and generative models in unCLIP, allowing the image encoder to benefit from the generative model in a relatively unified feature space.\n2. The method in this paper only relies on pure image data and has a relatively low fine-tuning cost.\n\nWeakness\n1. Although this method freezes the text encoder of CLIP and claims to retain the initial image-text alignment capability of CLIP, the lack of comparison with the base CLIP model (such as SigLIP ViT-SO-14@384) in the experiments raises doubts about whether generative fine-tuning would undermine the initial discriminative ability of CLIP.\n2. The comparative methods of the experiment only selected models that introduced the generator. Although this method outperforms them, to prove that the generator indeed brings more abundant visual details and knowledge to the image encoder, comparisons with other CLIP methods that do not use the generator are necessary.\n\nQUESTIONS:\nplease see weaknesses.\nThe method in this paper is reasonably designed and has superior performance compared with other CLIP methods that introduce generators. However, it is limited to comparisons with such methods and cannot strongly prove the promotion of the generative model to the image encoder of CLIP. Moreover, this paper also needs to prove that the generator will not undermine the initial image-text alignment ability of the CLIP model.\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-05"}