Title: Advancing Interpretability of CLIP Representations with Concept Surrogate Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Surrogate Model

Concept-based explanations

Representation Learning

Interpretability

CLIP
Abstract: Contrastive Language-Image Pre-training (CLIP) generates versatile multimodal embeddings for diverse applications, yet the specific information captured within these representations is not fully understood. Current explainability techniques often target specific tasks, overlooking the rich, general semantics inherent in the representations. Our objective is to reveal the concepts encoded in CLIP embeddings by learning a surrogate representation, which is expressed as a linear combination of human-understandable concepts evident in the image. Our method, which we term EXPLAIN-R, introduces a novel approach that leverages CLIP's learned instance-instance similarity to train a surrogate model that faithfully mimics CLIP's behavior. From the trained surrogate, we derive concept scores for each input image; these scores quantify the contribution of each concept and act as the explanation for the representation. Quantitative evaluations on multiple datasets demonstrate our method's superior faithfulness over the baseline. Moreover, a user study confirms that our explanations are perceived as more relevant, complete, and useful. Our work provides a novel approach for interpreting CLIP image representations, enhancing the user interpretability of representations and fostering more trustworthy AI systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EXPLAIN-R, a novel method for generating concept-based explanations for general-purpose CLIP image representations. The core idea is to train a simple, interpretable surrogate model to mimic CLIP's behavior. Instead of traditional reconstruction loss, the surrogate is trained to match the instance-instance similarity distributions of the original CLIP model, measured by KL divergence. While the paper presents an interesting new learning objective and shows promising results in a user study, I recommend rejection due to its insufficient experimental validation, strong underlying assumptions, and failure to acknowledge closely related prior work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe paper addresses the important and challenging problem of explaining task-agnostic representations. The core idea of training a surrogate model by matching similarity distributions, rather than by direct embedding reconstruction, is novel and theoretically well-motivated. It aligns more closely with the contrastive nature of CLIP itself.\n\nWeaknesses\n\nDespite these strengths, the paper has several major weaknesses that prevent me from recommending acceptance.\n\n1. Insufficient experimental validation: This is the most significant flaw. The empirical evidence provided is not comprehensive enough to fully support the claims of a new, general-purpose explanation method: \n\na. Limited datasets: The experiments are confined to general-domain datasets (COCO, ImageNet, etc.). The method's effectiveness is not tested on more challenging fine-grained recognition tasks (e.g., CUB-200 for birds, Oxford Flowers), where concept-based explanations are arguably most needed and most difficult to generate. The proposed concept discovery method, which relies on statistical extraction from general captions, is unlikely to capture the subtle, domain-specific concepts required for these tasks.\n\nb. Single baseline comparison: The quantitative evaluation relies on a comparison against a single baseline, SpLiCE. To robustly demonstrate superiority, a new method should be compared against a wider range of relevant approaches, such as prototype-based models (e.g., ProtoPNet) or concept bottleneck models (CBMs), even if they require adaptation.\n\nc. Lack of ablation studies: The method's performance is heavily dependent on the quality of its concept vocabulary, which in turn depends on an external captioning model (BLIP-2) and heuristic filtering. The paper lacks a rigorous ablation study on how the quality of captions (e.g., human-authored vs. machine-generated) or the choice of filtering heuristics impacts the final explanation quality.\n\nd. Failure to cite and differentiate from highly related work: The paper fails to acknowledge or discuss highly relevant and concurrent work, which significantly undermines its claim to originality.\nThe work Align2Concept (ACM MM 2024)[1] shares a remarkably similar goal: providing language-guided, concept-level explanations for visual recognition by bridging visual and textual modalities through CLIP's multimodal space. Both methods aim to align visual parts with textual concepts to create interpretable models.\nWhile the technical approaches differ (surrogate learning vs. manifold alignment with prototypes), the high-level motivation and conceptual framework are nearly identical. The omission of such a closely related work is a major oversight and prevents a fair assessment of this paper's true contribution to the field.\n\ne. Unjustified strong assumption of linearity: The entire framework is built on the strong assumption that concepts in CLIP's embedding space interact linearly. This is a significant simplification of a complex, high-dimensional space where non-linear interactions are highly probable. The paper acknowledges this as a limitation but fails to discuss its potential impact, which could lead to misleading or overly simplistic explanations that do not capture the true reasoning of the model.\n\n[1] Align2Concept: Language Guided Interpretable Image Recognition by Visual Prototype and Textual Concept Alignment, ACM MM2024\n\nQUESTIONS:\nsee Strengths And Weaknesses\n\nLIMITATIONS:\nsee Strengths And Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose EXPLAIN-R, a new method to tackle interpretability when using CLIP representations, independantly of the downstream taskl. The method consists in extracting concepts from captions, generated by a captioning model, that are used as input of a surrogate model that aims at reproducing the distribution of image-text similarities. They conducted extensive experiments on several datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** The method, EXPLAIN-R, introduced in this work to tackle the interpretability of CLIP representations, which is a key challenge given the widespread use of the CLIP model. The authors provide a clear and intuitive explanation of the methodology, making it easy to follow. Regarding the experiment section, the inclusion of a user study strengthens the experimental evaluation. \n\n**Weaknesses:** As the authors themselves acknowledge, relying on captions generated by the same captioning model for concept extraction can be a limitation (see Question 2). Furthermore, I am uncertain about how different captions were produced using a single model and whether these captions were truly distinct from one another (see Question 1). Additionally, I remain unconvinced about the necessity of task-independent interpretability. I would think that the relevance of explanatory concepts likely depends on the specific task. For instance, distinguishing birds from planes may require different concepts than differentiating between two bird species (see Question 3).\n\nQUESTIONS:\n1. Could you please clarify how different captions were generated using a single captioning model? Specifically, was the goal of generating 10 captions to eliminate noise, or perhaps to capture a wider range of concepts? Additionally, were these captions distinct from one another in terms of content and phrasing?\n\n2. Did you consider using multiple models to generate diverse captions?\n\n3. Could you please give more motivation behind the task independence?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose EXPLAIN-R, an interpretability for CLIP that attempts to faithfully reconstruct the behavior of\nthe vision encoder while operating instead over concepts. In doing so, they provide a concept-based explanation of the vision\nencoder while also remaining faithful to the downstream performance of the CLIP model.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n * Training a model to be faithful to the vision encoder is a novel idea and helps develop off of previous concept-based\nexplanations of CLIP . The ability to get cross-modal concept scores and use them in class-specific concept decompositions for\nmisprediction analysis was interesting as well.\n * The experiments validate the faithfulness of the method and the qualitative examples also seem to validate the method.\n\n## Weaknesses\n * Your description of how to compute concept vectors in 3.1 is very clear but I would appreciate an equation, figure, or\nalgorithm to diagram out the computation of each concept. In figure 1 the specific computation of the weights of the\nconcept vector are abstracted away .\n * To some extent this is a concept bottleneck model - you extract concepts and then train a model on these concepts for a\ntask (in this case, replicating CLIP’s image embedding). You may want to address this connection in more detail.\n\nQUESTIONS:\n* Do you think image captions contain all the relevant concepts in an image? For example, in the image example in Fig. 1,\nyour captions cover “A tall bird with a visible bill” and “dry vegetation”. However, I could imagine a relevant concept being “sunset” or “sunny weather” despite annotators generally describing the foreground and background over other visually-represented concepts such as the time of day or the weather.\n * I would be curious to also compare performance with the ground truth labels instead of CLIP — you optimize for faithfulness so it is expected that you do well on this metric.\n * It appears to me that this paper is more focused on explaining the behavior of CLIP’s image encoder over concepts than the concept extraction from images itself — your method for concept extraction is only a small section in the paper. I think this might be important to explain in the framing of your work.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}