Restyling Unsupervised Concept Based Interpretable Networks with Generative Models
Authors: Jayneel Parekh
Quentin Bouniot
Pavlo Mozharovskyi
Alasdair Newson
Florence d'Alché-Buc
Venue: ICLR 2025 Poster (2025)
Keywords: explainability
generative models
concepts
Primary area: interpretability and explainable AI

Abstract: Developing inherently interpretable models for prediction has gained prominence in recent years. A subclass of these models, wherein the interpretable network relies on learning high-level concepts, are valued because of closeness of concept representations to human communication. However, the visualization and understanding of the learnt unsupervised dictionary of concepts encounters major limitations, especially for large-scale images. We propose here a novel method that relies on mapping the concept features to the latent space of a pretrained generative model. The use of a generative model enables high quality visualization, and lays out an intuitive and interactive procedure for better interpretation of the learnt concepts by imputing concept activations and visualizing generated modifications. Furthermore, leveraging pretrained generative models has the additional advantage of making the training of the system more efficient. We quantitatively ascertain the efficacy of our method in terms of accuracy of the interpretable prediction network, fidelity of reconstruction, as well as faithfulness and consistency of learnt concepts. The experiments are conducted on multiple image recognition benchmarks for large-scale images. Project page available at https://jayneelparekh.github.io/VisCoIN_project_page/

Published as a conference paper at ICLR 2025
RESTYLING UNSUPERVISED CONCEPT BASED INTER- PRETABLE NETWORKS WITH GENERATIVE MODELS
Jayneel Parekh Quentin Bouniot 2,3,4455 Pavlo Mozharovskyl
Alasdair Newson' Florence d' d'Alché-Bue"
ISIR, Sorbonne Université, 2LTCI, Télécom Paris, Institut Polytechnique de Paris, France, Technical University of Munich, "Helmholtz Munich, 6 Munich Center for Machine Learning (MCML)
ABSTRACT
Developing inherently interpretable models for prediction has gained prominence in recent years. subclass of these models, wherein the interpretable network relies on learning high-level concepts, are valued because of closeness of con- cept representations to human communication. However, the visualization and understanding of the learnt unsupervised dictionary of concepts encounters major limitations, especially for large-scale images. We propose here a novel method that relies mapping the concept features to the latent space of pretrained generative model. The use of a generative model enables high quality visual- ization, and lays out intuitive and interactive procedure for better interpreta- tion the learnt concepts by imputing concept activations and visualizing gen- erated modifications. Furthermore, leveraging pretrained generative models has the additional advantage of making the training of the system more efficient. We quantitatively ascertain the efficacy of our method in terms of accuracy of the in- terpretable prediction network, fidelity of reconstruction, as well as faithfulness and consistency of learnt concepts. The experiments are conducted on multiple image recognition benchmarks for large-scale images. Project page available at https:/ / jaynee parekh. g thub. i SCassCCoNPPPPP pwwoooNo_oooeeppaaeooeeee
INTRODUCTION
Deep neural networks (DNNs) learn complex patterns from data to make predictions or decisions without being explicitly programmed how to perform the task. Interpreting decisions of DNNs, iee being able to obtain human-understandabh (Beaudouin et al., 2020; Arrieta et al., 2020; Montavon et al., 2019). This lack of transparency impacts their trustworthiness (Rudin et al., 2022) and hinders their democratization for critical ap- plications such as assisting medical diagnosis or autonomous driving. Two different paths have been explored in order to interpret DNNs outputs. The simplest approach for practitioners is to provide interpretations post-hoc, i.e. by analysing the so-called black-box model after training (Bachrens al., 2010; Ribeiro et al., 2016; Lundberg & Lee, 2017; Selvaraju et al., 2017). However, post-hoc methods have been criticized for their high computational costs and a lack of robustness and faith- fulness of interpretations (Yosinski et al., 2015; Kindermans al., 2019; Alvarez-Melis Jaakkola, 2018b). On the other hand, one preferred way to obtain more meaningful interpretations is to use interpretable by-design approaches (Al-Shedivat al., 2017; Adel et al., 2018; Böhle et al., 2022; Gautam et al., 2022), that aim to integrate the interpretability constraint into the learning process, while maintaining state-of-the performance.
insights about their decisions, is a difficult task
Concept- based Interpretable Networks (CoINs) are a recent subcategory of these inherently inter- pretable prediction models, that learn dictionary of high-level concepts for prediction. The concept representation is either learnt in supervised way using ground-truth concepts annotations (Koh et al., 2020; Sarkar et al., 2022), or in an unsupervised fashion by enforcing properties through care- fully designed loss functions (Alvarez-Melis & Jaakkola, 2018a; Parekh et al., 2021; Sarkar et al., 2022). The output of the model can be interpreted by looking at the activations of each concept
Equull contribution

---

Published as a conference paper at ICLR 2025
and how they are combined to obtain the final prediction. When working in the unsupervised set- ting, learnt concepts have to additionally be interpreted, usually through visualization (Parekh et al,, 2021: Sarkar et al., 2022). concept-based interpretations have gained prominence as alternative to popular feature-wise saliency maps (Springenberg et al., 2014; Ribeiro et al., 2016; Lundberg & Lee, 2017; Selvaraju al., 2017) for two main reasons: (1) their ability to provide interpretations closer to human reasoning and communication (Ych et al., 2019), and (2) in specific case of vi- sual modalities, their ability to more effectively highlight which features are important for model and not just where in the input image they focus on (Colin et al., 2022). However, the underlying concepts in current unsupervised CoINs are understood through a separate visualization pipeline, by finding inputs that highly activate given concept, either from natural images in the available dataset (Alvarez-Melis & Jaakkola, 10; Sarkar et al., 2022), or from virtual images by solving an opti- mization problem in the input space that maximally activates the concept (Mahendran & Vedaldi, 2016; Parekh ct al., 2021). For large-scale images, concepts generally activate for local pattern in- formation (color, texture, shape etc.) and these visualization approaches face major limitations in highlighting this information to a user. Simply visualizing the most activating samples does not highlight the specific feature concept activates for. Visualizing using activation maximization procedure leads to the generation of repeated patterns linked to the underlying concept in the image, but are hard for user discern any einar-iiteerretable signal. For example, in Fig. 1, it can be hard to identify that the concept activates for "Yellow-cole colored head" from the activation maximiza- tion ("FLINT visualization"). Furthermore, previous CoIN systems fail to include the visualization process in their quantitative evaluation of concepts and their use-case for interpretation.
We thus propose a novel set of specifications for the concepts to be learnt: additionally to fidelity to outpur (predictive capability from the concepts), fidelity to input (encoding input relevant infor- mation in concepts) and sparsity (a few concepts activated simultaneously), we also promote the viewability of concepts during training. This viewability is now defined as the ability of the system to reconstruct high-quality images from the learnt concepts, by leveraging a preirained generative model. In order to obtain this viewability property. we propose to learn concept translator, i.e., mapping from the concept representation space to the latent space of the generative model. Learning the concept translator along with the other parameters of this novel CoIN system helps improve the quality of the concepts. Finally, interpretation of concepts is obtained through translation to the generative model, allowing for more granular and interactive process. Our contributions are:
(i) We propose Visualizable CoIN (VisColN), novel architecture for unsupervised training of CoINs relying on a concept translator module that maps concept vectors to the latent space of pretrained generative model.
(ii) We introduce a new property for unsupervised CoIN systems, related to viewability. This property is imposed during the training of the system by enforcing perceptual similarity of the reconstruction, in addition to other constraints, and made possible by the use of a generative model.
(iii) We define a novel concept interpretation pipeline based on the concept translator and the associated generative model that allows to both obtain high-quality and more comprehen- sive visualization of each concept.
(iv) We introduce new metries in the context of unsupervised CoINs to evaluate the quality of concepts learnt, from the point of view of visualization and its usage for interpretation. We then quantitatively and qualitatively evaluate our proposed method on three different large-scale image datasets, spanning multiple settings.
2 RELATED WORKS
Interpretable predictive models In the context of deep learning architectures, host of early approaches studying interpretability tackled the post-hoc interpretation problem (Simonyan et al., 2013; Springenberg et al., 2014; Ribeiro et al., 2016; Lundberg & Lee, 2017; Sundararajan et al., 2017; Chen et al., 2018). However, previous works such as those of Al-Shedivat et al. (2017); Li et al. (2018); Alvarez-Melis & Jaakkola (2018a) have contributed to surge of developing predictive models that are also interpretable by-design (Yoon et al., 2018; Agarwal et al., 2020; Zhang et al., 201 Sa; Lee et al., 2019; Gautam et al., 2022). The earlier systems, however, trained the complete
2

---

Published as a conference paper at ICLR 2025
Original Image
FLINT Visualization
VisColN Visualization
Increasing concept activation
Figure I: Comparison of the generated images obtain for the same learnt concept ( ("Yellow-colored head") using FLINT visualization (Parekh et al., 2021) and our proposed VisCoIN visualization (in red boxes). Using our concept translator, that maps concept representation space to the latent space of generative model, more granular and interactive interpretation. Visual modifications manually indicated by red boxes.
visualize each concept at different activation values, allowing for
model from scratch. Recent approaches reflect a growing interest in building interpretable models on top of pretrained models (Koh et al., 2020; Angelov et al., 2023). Our approach falls in the latter category, wherein we learn an interpretable predictive model on top of a pretrained backbone.
Generative models for interpretations One of the earliest applications of generative models for interpretability was by Nguyen et al. (2016) to synthesize image for visualizing neurons in network using GANs. More recently, a variety of methods have employed generative models for post-hoc counterfactual interpretations (Zemni et al., 2022; Lang et al., 2021; Farid et al., 2023; Ghandehar- ioun et al., 2021). Their central theme revolves around the idea of embedding any given input to the latent space of generative model and finding meaningful perturbations in the latent space that affect the given predictor's 's output the most. One recent work (Ismail et al., 2023) also included the task of learning supervised concepts within generative models, to be able interpret and steer their latent spaces. Our aimed use-case of generative models differs in a major way from these methods, because wish to it in order learn and visualize explicit dictionary of interpretable cept representation, simultaneously used in predictive model.
Concept- based interpretability Providing interpretations via representations of high-level con- cepts has gained significant prominence recently. Similar to the overall literature, one set of concept- based methods have focused on post-hoe interpretation (Ghorbani et al., 2019; Ych al., 2019; Lang et al., 2021; Achtibat et al., 2022; Fel et al., 2023), with most based on the notion of concept activation vectors (Kim al., 2017). The other type of methods tackle the by-design/ante-hoo inter- pretation problem by learning concepts (Alvarez-Melis & Jaakkola, 201 8a; Koh et al., 2020; Parekh et al., 2021; Sarkar al., 2022: Sawada & Nakamura, 2022; Sheth & Ebrahimi Kahou, 2024) ab- breviated CoIN systems in Section 1. We cover these methods in more detail in Section 3.1 with particular focus on networks based on learning completely unsupervised concepts (Alvarez-Melis & Jaakkola, 2018a; Parekh et al., 2021; Sarkar al., 2022; Garg et al., 2024), a key starting point of our approach. Recent variants of concept bottleneck models using language models (Oikarinen al., 2023; Panousis et al., 2024) and why concept visualization is still useful despite their ability to obtain automated text descriptions, is discussed in detail in Appendix A.
3 APPROACH
3.1 BACKGROUND
In this part, we provide an overview of a concept based interpretable network (CoIN). Our focus in this paper is CoIN systems that learn unsupervised dictionary of concepts.
3

---

Published as a conference paper at ICLR 2025
G
Wx
(T)
adivations
Leee
((z)
f(x)
Let
g(x)
Overview of a standard CoIN system that makes prediction g(x) from extracted concepts (xx). Right: Design of our unsupervised concept-based basedd interpretable network VisCoIN leveraging pretrained generative model G for visualization, and pretrained classifier f. Purple blocks denote trainable subnetworks.
Figure 2: Left:
Concept-based interpretable networks We denote training set for supervised image classifi- cation task as S = ((z,, yt) H Each input image x E X' C. R" associated with a class label y V. one-hot vector of size number of classes C. The by-design interpretable classification network based on learning concept representation is denoted by g + y. In the standard setup for concept-based prediction models (supervised or unsupervised), given input s. the computa- tion of g(x) broken down into two parts. There is first concept extraction representation क and then subnetwork ④ that computes the final prediction using concept activations ((x), such that g(x) = θ ((x) (Fig. 2, left). Supervised concept-based networks use sample-wise ground-truth concept annotations train 4. For instance, concept bottleneck models (CBM) (Koh et al., 2020) use human -anonattnnntt binary concept vector denoting presence/absence of each concept. Although language model based CBMs do not require manual annotation of concepts, they also follow a very similar paradigm, e.g. by using CLIP "image-concepl description" similarities
proxy annota- tions (Oikarinen et al., 2023; Yang et al., 2023a; Panousis et al., 2024). The core of unsupervised concept-based methods instead lies in learning & by imposing loss functions. These loss functions are typically selected to encourage a certain set of properties that shape simultaneously for both interpretation and prediction. We list the properties below:
1. Fidelity to output:
This requires (xx) to model the output space via the θ function, either by predicting ground-truth label y (Alvarez-Mehle & Jaakkola, 2018a) the classification output f(x: (Parekh et al., 2021; Sarkar et al., 2022). trains ((x) for the prediction task and, during the interpretation phase, helps in identifying important concepts for prediction.
2. Fidelity to input: This requires (xx) to reconstruct the input via decoder function. This property considered important to encode semantically meaningful input features in the concept representation. All the previous methods (Alvarez-Melis & Jaakkola, 201 Sa; Parekh et al., 2021; Sarkar al., 2022) rely on this loss and employ standard non-generative decoders for pixel-wise reconstruction to learn the concept dictionary ④. Note that CBMs don't 't adhere to this property, Icading to major design and training differences, as they don't include decoder compared unsupervised CoINs, that remain the focus of this work.
3. Sparsity of activations: This requires concept activations ((x) to be sparse for any æ. It reinforces the high-level nature of & and limits the number of important concepts for pre- diction, thus enhancing interpretability and reducing the visualization overhead for users.
Limitations with concept visualization in previous unsupervised CoINs A common trait among prior CoINs learning unsupervised concepts (Alvarez-Melis & Jaakkola, 2018a; Parekh et al., 2021; Sarkar et al., 2022; Garg et al., 2024) is the deployment of a decoder to reconstruct input x from ((z). Unlike supervised methods, they do not have access to any concept labels and thus need an additional visualization pipeline to understand the information encoded by each concept. However, their visualization pipeline does not utilize the decoder, but instead relies proxy methods to probe the concept activation. Typically, it consists of finding an input that highly activates a concept, either by selecting from the training data (Alvarez-Melis & Jaakkola, 2018a; Sarkar al., 2022)
4

---

Published as a conference paper at ICLR 2025
via imput optimization (Parekh et al., 2021). In the former case, simply visualizing the set of most activating training samples lacks granularity to highlight the features encoded by the concept. Using input optimization, while relatively more insightful, is still difficult for user understand as the optimized images are often unnatural. Moreover, these issues exacerbate for large-scale images, as seen in Fig. 1. A natural strategy overcome these limitations is to enable direct control of concept's 's activation and visualizing its effect on the input. Since the decoder defines the relationship between concept activations and input samples, a generative model is a perfect candidate for decoder to unlock this ability, in contrast to standard decoders used previously. While Garg et al. (2024) includes a GAN as a decoder model, it is learned simultaneously. This makes the overall training challenging, since GANs are notoriously difficult to train. Furthermore, they do not leverage the GAN for visualizing the concepts, only relying on maximum activating samples (MAS) for visualization. The GAN is used as a decoder with higher expressivity, to improve accuracy.
In the next part, we describe the architecture behind our by-design interpretable network 9, that additionally includes concept translator module Ω to map concept features ((x) to the latent space of a pretrained generative model G. This collectively defines our VisCoIN method.
3.2 INTERPRETABLE PREDICTION NETWORK DESIGN
The complete design of VisCoIN is illustrated in Fig. 2 (right). We assume a fixed pretrained net- work for classification f and a fixed pretrained generator G, for generation on the input dataset respectively. We use these two networks to guide our design and learning of g and its concept ex- traction function o. We first discuss modelling of g by describing its constituents, & and θ. Interpretable network design The dictionary
consists of K concept functions ø1,..., ΦK- Given an input x, each concept activation dhx is represented by a small convolutional feature map with non-negative activation. Thus (xx) (x)) ..., (x(x)] E R where b is the total number of elements in each feature map. We model computation of concept activations (xx) using the pretrained classification network f and learn relatively lightweight network T top of its selected hidden layers denoted as fz (z), i.e. feature maps to obtain single concept activation of size K and make the final prediction by passing it through a linear layer followed by softmax, i.e. g(x) = ((((z)) = softmax (% (=))( where Ow E T KxC are the weights in the linear layer. The simplified design of O makes estimat- ing importance of each concept function Φx for any prediction straightforward.
((x) = V a fz)).. θ is designed to simply pool the
Viewability property In order to improve visualization for unsupervised CoINs, and thus inter- pretation of learnt concepts, we propose add the requirement for viewability property. Given an input image x, this property requires to be able to reconstruct high--uualit images from ((x). Specifically, reconstructions should have high enough quality to "view" input samples through gen- erated outputs and thus ground modifications to ((x) back to x. We propose to achieve this by using a pretrained generative model G and learning an additional concept translator module Q to map (xx) to the latent space of G, such that high-quality reconstructed images can be obtained from (xx) through Q and G. This also retains flexibility to design & for instance in choosing number of concepts K according to the problem, regardless of the choice of pretrained G.
Pretrained generative model G as decoder In practice, we want our generative model to (i) have a low dimensional latent space, (ii) have a structured latent space that admits meaningful latent traver- sals, (iii) be able to generate high-quality images for the underlying data distribution. The choice of using pretrained generator instead of simultaneously training is because it significantly lowers training costs, reduces training complexity and improves reusability. We discuss the significance of these properties in relation to various generative architectures in Appendix B. We also experimen- tally demonstrate the versatility of our method to different generative architectures in Appendix D. Concept translator ? Concept representations in CoINs are typically smaller dimensional than latent spaces of generative models as latent spaces encode lot more information about input than needed for classification. We thus learn the concept representation ((x) separately from the latent space of pretrained G' and instead learn a concept translator Ω to map (xx) to latent space of G. The design of S2 depends on the architecture of underlying generative model. In general, it consists of a fully-comm callecconee (FC) layer that predicts for each input image x, the latent vector wa from (zx), that will then be used as input for G. The computation of the reconstructed input 7 is given by:
ã G(w,,, where w ())()))
We discuss in more technical details, the architectures of each network in Appendix C.
5

---

Published as a conference paper at ICLR 2025
3.3 TRAINING LOSSES
Based on the previous discussion about concept-based networks and our proposed reconstruction pipeline design, define here our training loss Lerain and each of its constitutive terms.
For the fidelity to outpul property, we define an output fidelity loss Lof, that grants predic- tive capabilities to g. It's defined generalized cross entropy (CB) between g and ʃ:
Cof (z; V. θ) aCE(g(z),
The most critical part of our training loss the reconstruction loss E computed through the pretrained generative model G, that gathers all constraints between inputs z and their reconstruction = G(S)((r))) It combines & and l2 penalties, enforcing pixel-wise reconstruction for fidelity to input, with perceptual similarity LPIPS (Zhang et al., 2018b) and final reconstruction classification term, both linked to viewability. The reconstruc- tion classification term, defined as (z) fx)), encourages the generative model to reconstruct ă with more classification specific features pertaining to input æ. Similar losses have been introduced for inversion in generative model and its training for post-hoc inter- pretation (Lang et al., 2021). Our reconstruction loss is thus defined as follows:
Cren V .2) || =112 || x + SLPIPS x) + f(:) (3)
We impose the sparsity property along with two other regularizations, combined under the term Creg. More specifically. v is regularized to encourage sparsity of activations (xx) through an & penalty, and diversity while reducing redundancy in learnt dictionary & with a kernel orthogonality loss Corth, applied on weights of final convolution layer of (Xie et al., 2017; Wang et al., 2020). Then, Ω is encouraged to predict latent vectors close to average latent vector W, a common practice inversion systems of generative models (Tov et al., 2021). The regularization terms are written as follows: Greg(1: (z; V, Q2 Lreg- v (2;; V + (2;
(4)
freg 2 (1; I) ||,,- w1l. freg v (2; V) 01]0())]1 Lorth (V).
Finally, the training loss and the optimization can be summarized as: Lof(x: V. 0) + Eec V. (2) + Greg (2; 12
Ctrain (2; L O,
1 v = arg min v.n
1
Σ Larain (2, V, Q)
(5)
N
RES
In the above equations, the loss hyperparameters denoted by a, B, During training, Ltrain is simultaneously optimized w.r.t parameters of V θ and Ω while keeping f and G fixed.
3.4 INTERPRETATION PHASE
The interpretation generation process can be divided in two parts. (1) Concept relevance estima- tion, that requires estimating the importance of any given concept function dR in prediction for particular sample (local interpretation) or class c general (global interpretation). and (2) Con- cept visualization, which pertains to visualizing the concept encoded by any given concept function
dk. We describe each of them in greater detail below:
(1) Concept relevance: Since our g(x) adheres to structure of CoINs and among them closest to Parekh et al. (2021), the first step of relevance estimation almost follows as is. The estimation is based concept activations ((r), and how the pooled version of ((x) is combined by the fully connected layer in θ (with weights θw) to obtain the output logits. Note that this step does not rely on using the decoder/generatoo G. Specifically, the local relevance (z: of concept function dx for a given sample x is computed as the normalized version (between [-1, I)) of its contribution to logit of the predicted class é g(x). The global relevance of concept function dh for given class denoted Tlkc, is computed as the average of local relevance rx(x)) for samples from class c. The above description is summarized in equation below wherein e/f denotes the weight on concept for predicted class è in weight matrix θw:
0xk (x)
Tk (x) maxi (ox(x))
ax = (ox The = E(ra(x)lo(z) = c)

---

Published as a conference paper at ICLR 2025
Interaction concept in (2()
Fixed Concept Translator Generator
Interpretation of concept function dhu
Input image
(x(x))
(2
Q
G
Figure 3: Visualization for a given image and concept function Φt- By imputing a higher acti- vation for (x)) in ((z) (by a factor À = 4 in the figure), and comparing the obtained visualization to the original reconstruction (obtained with the untouched ((x)), we interpret information en- coded by du about image æ.
(2) Concept visualization: Once the importance concept function is estimated, one can extract the most important concepts for sample or class c by thresholding r2x(z) Tk.c respectively, For visualizing any concept Φk. following previous CoINs, one can start by selecting most activating training samples for (xxxx) over the whole training set separately for each class it is highly relevant for. However, the core of our concept visualization process, is to utilize generator G to visualize the impact of du on any input æ it is relevant for. We do by (1) directly modifying activation of dx (x)) by a factor À x Φn (z),, 2 0 while keeping all other activations in (xx) intact, and (2) Visualizing generated output for increasing value of X. The case of 1 corresponds
R. the reconstructed version of r. This process is summarized in Fig. 3. Extremely high À can push the predicted latent vectors far from the average latent vector in which case the generated output is less reliable. For our experiments, qualitatively we found À upto 3 or 4 reliable with consistent modifications.
4 EXPERIMENTS AND RESULTS
Datasets We experiment on image recognition tasks for large-scale images in three different domains with a greater focus on multi-class classification tasks: (1) Binary age classification (young/old) on CelebA-HQ (Karras et al., 2018), (2) fine fine-grained bird classification for 200 classes on Caltech-UCSII UCSDD-irds-2200 (CUB-200) (Wah et al., 2011) and (3) fine-grained car model classifi-
cation of 196 classes on Stanford Cars (Krause et al., 2013).
Implementation details Experiments in main text use ResNet50 (He al., 2016) as our ar- chitecture for f. and StyleGAN2-ADA (Karras et al., 2020) for G. Experiments with different architectures for G (ProgressiveGAN (Karras et al., 2018), B-VAB (Higgins et al., 2017) on Fash- ionMNIST (Xiao et al., 2017)) and ff (ResNet101) can be found in Appendix D. All images processed at resolution 256 x 256. We use a dictionary size K = 64 on CelebA-HQ and K 256 conducted
on CUB-200 and Stanford-Cars, All experiments Complete details about network architectures, obtaining pretrained checkpoints for f, G, VisCoIN training and evaluation metric implementations are provided in Appendix C.
a single V100-32GB GPU.
4.1 EVALUATION STRATEGY
One major goal of by-design interpretable architectures is to obtain high prediction performance. Prediction accuracy of g is thus the first metric we evaluate. We next discuss multiple functionally- grounded metrics (Doshi-Velez & Kim, 2017) that evaluate the learnt concept dictionary & from an interpretability perspective and its use visualization, including two novel metrics in the context of evaluating CoINs ("faithfulness" and "consistency").
Fidelity of reconstruction Since reconstructed output plays a crucial role in our visualization pipeline, evaluate how well does G reconstruct the input. We compute averaged per-sample mean squared error (MSE), perceptual distance (LPIPS) (Zhang el., 2018b) and distance of overall distributions (FID) (Heusel et al., 2017) of reconstructed images Ø and original input images
Faithfulness of concept dictionary + The aspect of faithfulness for a generic interpretation method asks the question "are the features identified in the interpretation traly relevant for the

---

Published as a conference paper at ICLR 2025
prediction process?" (Alvarez-Melis & Jaakkola, 2018a; Parekh et al., 2022). This is generally computed via simulating "feature removal" from the input and observing the change in predictor's s output (Hedström et al., 2023). Simulating feature removal from input is relatively straightforward for saliency methods compared to concept-hased methods, for example by setting the pixel value to O. For CoIN systems, this is significantly more tricky, concept activations (x(xx) don't represent the input I exactly. However, through the decoder, we can evaluate if the concepts identified as relevant for an input encode information that is important for prediction. We adopt an approach sim- ilar to previous proposal of faithfulness evaluation for audio interpretation systems (Parekh et al., 2022). Concretely, for a given sample x with activation $
(xx), predicted class and a threshold T, we first manipulate (xx) such that (x) is set to O if (x) > T, T. E [I...., K}. That is, we "remove" all concepts with relevance greater than some threshold. This modified version of (xx)) is referred to as Frem (z)). To compute faithfulness for given x, denoted by FF z. we compute the change in probability of the predicted class from original reconstructed sample ī = G(L(((z))) to new sample Trem G(S2( reem (z)), that is, FF, = g( g(Zrem)e. Ideally, we expect to see a drop in probability (FF,z > 0) if the set of relevant concepts "truly" encode information relevant for classification. Following Parekh et al. (2022) we report the median of FF over the test data for different thresholds 0 < r < 1.
Consistency of concept visualization We expect during visualization of given concept Φu that a user observes similar semantic modifications across different images. Thus, we hypothesize that if modifying any specific concept activation du leads to consistent changes for different samples x, then generated output for two versions of ((x). one with (x) set to large value and one with dhe (x) = O, should be separable the embedding space of f (all other concept activations unchanged). In other words, embeddings for images with high Φx (z) and low (x()) should be well separated. To compute this metric, we first create a dataset of generated images with two different sets of activations. For each of training and test set, this is done by first selecting set of Noe sam- ples for which Φ% is highly activating and relevant for. Then we find its maximum activation 0% among these samples, and create two generated outputs for each of Nee samples, one zi such that óx( is set to A0x with À 2 1, and the other 2yz such that Φx (xr) = O. The two sets of gener- ated images are then gathered into a single dataset S₁- = {xxt, 1), (Izz 0)} such that [S& = 2No.. and we learn a binary classifier 4p : X -> {0, 1). from pooled feature maps of intermediate em- bedding of f. We train for binary classification for sets created from training data Siaan and test on sets created from test data Shest Our concept consistency metric CC2s for a given concept & is thus obtained as the accuracy of the binary classifier on Ste::
CC&(S :px) = 2NNc
1
Σ
pa(xT) + (1 ypx (zzz)
(of FH DEST
This performance is tabulated for each concept for a fixed X, and mean and standard deviation across all concepts is reported.
Baselines The primary comparison methods for us are CoINs that learn unsupervised concepts ef- ficiently, even for large-scale images, FLINT (Parekh et al., 2021) and FLAEM (Sarkar et al., 2022). SENN (Alvarez-Melis & Jaakkola, 2018a) suffers from computational issues for large-scale images as it requires to compute jacobian of concept dictionary w.r.t input pixels for its loss computation. Thus, for all the metrics we compare with FLINT and FLAEM as our primary baselines. Addition- ally, for accuracy evaluation, we track the performance of our pretrained classifier f. Note that (ResNet50) is not an interpretable model and trained entirely for accuracy. Lastly, for faithfulness we compare with a "random"" baseline that randomly selects concepts for whom activation is set to O. Since there is no notion of threshold in selection, in order to make it comparable for a given thresh- old, we select the same number of concepts randomly as we would for our method. Our proposed system for all evaluations is abbreviated as VisColN (Visualizable CoIN).
4.2 RESULTS AND DISCUSSION
Quantitative results Table Ia reports the test accuracy of all the evaluated systems. Our pro- posed system, VisCoIN performs competitively with the pretrained f considered uninterpretable and purely trained for performance. It also performs better than the other recent CoIN systems for more complex classification tasks (CUB- 200 and Stanford Cars) with large number of classes and diverse images. Metrics quantifying the fidelity of reconstruction on test data are in Table 2a. The other baselines only optimize for wixel-wise reconstruction and FL.NT achieves lower MSE than
8

---

Published as a conference paper at ICLR 2025
Table 1: (a) Accuracy (in %) of interpreter g of CoIN systems, and of the baseline pretrained classi- fier f- (b) Mean and standard deviation for consistency CC& over all concept functions Φ (binary accuracy in %). Higher is better, the best performance is reported in bold, second best in underline.
(a) Accuracy of interpreter g
(b) Consistency of changes
Dataset
Original-f
FLINT
FLAEM
VisCoIN (Ours)
Datasct
FLINT
FLAEM
VisCoIN (Ours)
Calebr-HQ
82.71
87,25
88.18
87.71
CelehA-HQ
82,6 22.7
57 + 17.3
85,5 13.9
CUB-200
80,56
77.2
51.76
29.44
CUB-200
72.6 18
55.6 13.6
8,4
Stanford
82.28
75.95
50.02
29.89
Stanford Cars
70 16.3
$4.9 13.3
82.7 = 8.3
Table 2: (a) Reconstruction quality (MSE, LPIPS and FID) of CoIN systems. Lower is better. (b)
Faithfulness (median F2) of CoIN systems and random baseline, for different threshold. Higher is better. Best performance is in bold, second best in underline.
(a) Reconstruction quality
(b) Faithfulness
Dataset
Metric
FLINT
FLAEM
VisColN (Ours)
Dutset
Taresh.
Randon
FLINT
FLAEM
VisCoIN (Dus))
CulebA-1QQ
LPIPS
MSE
0,533
0
30.45
0.119
0.688
0,405
0.034
CdleM-O
0.088
0.005
0.151
0.307
MSB
0.161
CUR--D0
0.2
0.1
0.634
0.897
0100
0.034
< 11003
<.10"
0.251
BL146
CUR-300
1IPS
0.75
0.4
0.01
< 10r9
< 1000
RJ144
RID
53.16
51.15
15.85
0.35
0090
<1003
IL161
MSE
0.121
0.278
0,172
Sunford Cars
0.06
IL1IR
Suaforr Cars
LPIPS
0.697
0.734
0,488
0,022
FID
64
69.44
6.77
VisCoIN. However, crucially, reconstruction from method approximates the input data consid- erably better, in terms of perceptual similarity (LPIPS) and overall distribution (FID), which highly contributes to better viewability. Table 2b tabulates the median faithfulness FF, for the evaluated systems on 1000 random test samples for different thresholds. The performance of Random base- line being close to O even for small thresholds indicates that a random selection of concepts often contains little information relevant for classification of the predicted class. In contrast, concepts identified relevant as part of g in VisCoIN tend encode information about input that noticeably affects classification. In regard to other CoINs, while the faithfulness results are competitive on CelebA-HQ. for more complex datasets, concept dictionary in VisCoIN is significantly more faith- ful than FLINT or FLAEM, which do not demonstrate more faithfulness than the Random baseline. Finally, the mean and standard deviation for visualization consistency of all concepts is reported in Table 1b with x~ = 2. Concepts learnt with VisColN demonstrate a higher mean consistency of visualization compared to baselines. The deviation across concepts is also lower for our method. We also evaluate concept consistency with higher values of À and observe increased separation with better classification performance (Appendix E).
Qualitative results Fig. 4 shows visualization for different class-concept pairs across the three datasets that are determined to have high global relevance Tk through predictive structure of g(x), as described in Section 3.4. For each class-concept pair, we show two maximum activating training samples for the concept from the corresponding class, the reconstructed input from ((x) (A 1) and the generated output with modified concept activation (x(()) by a factor trations, increasing the activation of the concept, i.e. moving from À = 1 to 4, strongly emphasizes some specific concept in the generated output that can be clearly grounded to the input, and the name of the concept is then manually inferred. For instance, increasing the activation of concept for "Red-eye" in Fig. 4a increases the size of red eye of the bird, a key feature of samples from class 25 enough quality that allows us to "view" the input sample through the generated output and ground the modifications in generated output the input. However, we also observed that learnt concept functions can be prone to modifying more than one high-level feature in the image. For, e.g., in Fig. 4d, increasing the concept activation increases both "eye-squint" and "beard" in the generated output. A longer discussion about limitations is available in Appendix H.
4. In all the illus-
We can also qualitatively verify that the reconstruction has high
Ablation study We ablate multiple aspects of our system, with detailed results in Appendix F. No- tably, we observed a tradeoff' induced by strength of reconstruction caannsttuctioclaassfiiiiio classification loss (weight 7). A high y positively impacts faithfulness, but negatively impacts perceptual similarity of reconstruction.

---

Published as a conference paper at ICLR 2025
Original image
Reconstruction
Interpretation À
Original image a
Reconstruction
Interpretation ==
À
A =
(a) Concept "Red-eye" in class 25
(b) Concept "Blue upperparts in class 15
AUG
(c) Concept "Makeup" in class Young
(d) Concept "Eye squint" class Old
(c) Concept "Silver front" in class
(f) Concept "Radiator grille" in class 60
Figure 4: Qualitative examples obtained for different concepts, classes (a)-(b) CUB-200, (c)- (d) CelebA-HQ, (e)-(f) Stanford-Cars datasets. On each subfigure, first column corresponds to maximum activated samples for class-concept pairs with high relevance (rk.c > 0.5), second column to reconstructed image obtained with original (xx), and third column to the image obtained by imputing (x)) in & (x). Red boxes manually added to indicate key regions of modifications in generated images.
CONCLUSION
We introduced novel architecture for Visualizable CoIN (VisColN), that addresses major limita- tions to visualize unsupervised concept dictionaries learnt in ColN systems for large-scale images. Our architecture integrates the visualization process the pipeline of the model training, by lever- aging pretrained generative model using concept translator module. This module maps concept representation to the latent space of the fixed generative model. During training, we additionally enforce viewability property that promotes reconstruction of high-quality images through the gen- erative model. Finally, we defined new evaluation metrics for this novel interpretation pipeline, to better align evaluation of concept dictionaries and interpretations provided to user. Future works include adapting the design of this system for supervised CoINs, multimodal generative models, or extending its application to different data modalities.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Throughout the paper, we made sure that all our experiments were fully reproducible, describing in details all datasets and architectures considered in Section 4. We then explain the design of and Ω training settings, hyperparameters and computation of evaluation metrics in Appendix C.
ACKNOWLEDGEMENTS
This work was supported by the LIMPID (ANR-20-CE23-0028) and FAR-SEE (ANR-24-CE23- 0921) projects of the French National Research Ageney (ANR). The authors also thank Thibaut de Saivre and Hugo Aoyagi for restructuring the codebase.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019. 19
Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wie- gand, Wojeiech Samek, and Sebastian Lapuschkin. From" where" to" what": Towards human-understandable explanations through concept relevance propagation. arXiv preprint arXiv:2206.033208 2022. 3
Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable representations for both deep generative and discriminative models. In International Conference on Machine Learning, pp. 50-59. PMLR, 2018. 1
Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey Hinton. Neural ad- ditive models: Interpretable machine learning with neural nets. arXiv preprint arXiv:2004.13912, 2020. 2
Maruan Al-Shedivat, Avinava Dubey, and Eric Xing. Contextual explanation networks. arXiv preprint arXiv: 10301, 2017. 1,2
David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances Neural Information Processing Systems (NeurIPS), pp. 7775- 7784, 2018a. 2, 3, 8
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint /806 2018b.
Plamen Angelov, Dmitry Kangin, and Ziyang Zhang. Towards interpretable-by-dee byy-igg deep learning algorithms. arXiv preprint arXiv:2311.113% 2023. 3
Alejandro Barredo Arrieta, Natalia Dodz-Rodr,, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-I fóllz Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunitics and challenges to- ward responsible ai. Information fusionn 58:82-115,203
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. The Journal of Machine Learn- ing Research, 11: 803-1831, 2010.
Valérie Beaudouin, Isabelle Bloch, David Bounie, Stéphan Clémençon, Florence d' A'ché Buc, James Fagan, Winston Maxwell, Pavlo Mozharovskyi, and Jayneel Parekh. Flexible and context- specific explainability: a multidisciplinary approach. arXiv preprint arXiv:2003. 2020.
Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mvtec anomaly detection dataset: a comprehensive real-world dataset for unsupervised anomaly detec- tion. International Journal of Computer Vision, 1934))1038-10055 2021. 17

---

Published as a conference paper at ICLR 2025
Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos networks: Aligmment is all we need for in- terpretability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022. 1
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information-theoreti perspective model interpretation. In International Conference Ma- chine Learning, pp. 883-892. PMLR, 2018. 2
Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What i cannot predict, do not under- stand: A human-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems, 35:2832-28455 2022. 2
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition,
pp. 248-255. Icee, 2009. 24
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 1XXi::7022.8608, 2017. 7
Karim Farid, Simon Schrodi, Max Argus, and Thomas Brox. Latent diffusion counterfactual expla- nations. arviv preprint arXiv:2310.06668 2023.
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2711-2721, 2023. 3
Tanmay Garg, Deepika Vemuri, and Vineeth N Balasubramanian. Advancing ante-hoc explainable models through generative adversarial networks. arXiv preprint arXiv:2401.044: 00644, 2024. 3, 4,
Srishti Gautam, Ahcene Boubekki, Stine Hansen, Suaiba Amina Salahuddin, Robert Jenssen, Ma- rina Höhne, and Michael Kampffmeyer. Protovae: A trustworthy self-explainable prototypical variational model. arXiv preprint arXiv:2210.08155, 2022. 1,2
Asma Ghandeharioun, Been Kim, Chun-Liang Li, Brendan Jou, Brian Eoff, and Rosalind Picard. Dissect: Disentangled simultaneous ex explanttions via concept traversals. International Confer- ence on Learning Repressenaations, 2021. 3
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9277-9286,
2019. 3
Chenhui Gou, Abdulwahab Felemban, Faizan Farooq Khan, Deyao Zhu, Jianfci Cai, Hamid Rezatofighi, and Mohamed Elhoseiny. How well can vision language models see image details? arXiv preprint arXiv:2408.0300 2024. 18
Timofey Grigoryev, Andrey Voynov, and Artem Babenko. When, why, and which pretrained gans are useful? In International Conference on Learning Representations, 2022. 20
Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23386-23397 2022. 18
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual
networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Anna Hedström, Leander Weber, Daniel Krakowezyk, Dilyara Bareeva, Franz Motzkus, Wojciech Samek, Sebastian Lapuschkin, and Marina M-C Höhne. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research. 24(34):1-11, 202.. 8
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017."
12

---

Published as a conference paper at ICLR 2025
Irina Higgins, Loic Matthey, Arka Pal, Christopher! P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2017. 7, 18, 20, 23
Aya Abdelsalam Ismail, Julius Adebayo, Hector Corrada Bravo, Stephen Ra, and Kyunghyun Cho. Concept bottleneck generative models. In The Twelfth International Conference on Learning Representations, 2023. 3
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. 7, 18, 20, 22
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 13:12104-12114, 2020. 7, 18, 20
Oren Katzir, Vicky Perepelook, Dani Lischinski, and Daniel Cohen-Or. Multi-level latent space structuring for generative control. arXiv preprint arXiv:2202.05910, 2022. 19
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation 1279, 2017.
vectors (teav). arXiv preprint
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting. Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019. 1
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
1412,6990 2014. 20
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Machine Learning, pp. 5538-5345 PMLR, 2020. 3,
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013. 7, 17, 20
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In The Eleventh International Conference on Learning Representations, 2023. 18
Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T Freeman, Phillip Isola, Amir Globerson, Michal Irani, et al. Explaining in style: Training a gan to explain a classifier in stylespace. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 693-702, 2021. 3,6
Yann LeCun. Lenet-5, convolutional neural networks. 2015. 23
URL: http://yann. lecun. comlexdb/lenet,
Guang-He Lee, Wengong Jin, David Alvarez-Melis, and Tommi S Jaakkola. Functional transparency for structured data: a game- theertte approach. arXiv preprint arXiv:1902.000 2019. 2
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 32, 2018.
Zachary C. Lipton. The mythos of model interpretability. Commun. ACM, 61(10):36-43 2018. 26
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017. 1.2
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal Computer Vision, 120(3):233-255 2016.
13

---

Published as a conference paper at ICLR 2025
Grégoire Montavon, A Binder, S Lapuschkin, W Samek, and KR Müller. Explainable ai: interpret- ing, explaining and visualizing deep learning. Spring er LNCS, 11700, 2019.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016. 3
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Wei Weng. Label-free concept bottle- neck models. arXiv preprint arXiv:2304.060 06129, 2023. 3, 4, 17
Konstantinos Panousis, Dino Ienco, and Diego Marcos. Coarse-to- fine concept bottleneck models. In Conference on Neural Information Processing Systems, 2024. 3, 4, 17
Jayneel Parekh, Pavlo Mozharovskyi, and Florence d' d'Alché Buc. A framework to learn with inter- pretation. Advances in Neural Information Processing Systems (NeurIPS), 2021. 1, 3, 4, 5, 6. 8, 19
Jayneel Parekh, Sanjeel Parekh, Pavlo Mozharovskyi, Florence d' A'Ach Buc, and Gaël Richard. Listen to interpret:
Post-hoc interpretability for audio networks with nmf. Advances in Neural Information Processing Systems, 35:35270-35288 2022. 8
Yong-Hyun Park, Mingi Kwon, Junghyo Jo, and Youngjung Uh. Unsupervised discovery of seman- tic latent directions in diffusion models. arXiv preprint arXiv:2302. 12469, 2023. 18
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR, 2021. 18
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, ACM, 2016. 1,
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022. 18
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. In- terpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian. A frame- work for learning ante-hoc explainable models via concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10286-10295, 2022. 1, 2, 3, 4, 8, 31
Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 10:41758-41765, 2022. 3
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626 2017. 1,2
Ivaxi Sheth and Samira Ebrahimi Kahou. Auxiliary losses for learning generalizable concept-based models. Advances in Neural Information Processing Systems, 36, 2024. 3
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
2
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015. 18
14

---

Published as a conference paper at ICLR 2025
Yue Song, Andy Keller, Nicu Sebe, and Max Welling. Latent traversals in generative models as potential flows. arXiv preprint arXiv:2304.12944 2023. 35
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint urXiv::412.68000 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Learning-Volume Volume 70, pp. 3319-
Proceedings of the 34th International Conference on Machine Learning- 3328. JMLR. org, 2017.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions Graphics (TOG), 40(4):1-14, 2021. 6
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-201 dataset. 2011. 7, 20
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogni- tion, pp. 11505-115155 2020. 6
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020. 24
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 21
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017. 7,23
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6176 6185, 2017. 6
Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19187-19197, 2023a.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023b. 17
Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier. Feature-style encoder for style-based gan inversion. arXiv preprint arXivv2202.02218 0833, 2022. 19, 21
Chih-Kuai Kuan Yeh, Been Kim, Sercan O Arik. Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfister. On concept- based explanations in deep neural networks. arXiv preprint arXiv: 2019. 2,
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance- variable selection using neural networks. In International Conference on Learning Representations, 2018. 2
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv: | 00009, 2015. 1
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1706v:1506.03365. 2015. 20
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv: /5480, 2022. 17
15

---

Published as a conference paper at ICLR 2025
Mehdi Zemni, Mickaël Chen, Éloi Zablocki, Hédi Ben- Younes, Patrick Pérez, and Matthieu Cord. Octet: Object-aware counterfactual explanations. arXiv preprint arXiv:2211.12380, 2022.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8827- 8836, 2018a. 2
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 201 8b. 6, 7
16

---

Published as a conference paper at ICLR 2025
The Appendix is organized follow:
We discuss relation with CBMs (supervised CoINs) using language models to extract con-
cepts annotations, and the general usefulness of concept visualization in Appendix A.
We discuss the desidered properties for the generative model G Appendix
We describe in details the architectures of the different network, the training procedures of the system and the evaluations in Appendix C.
We present additional experiments with different architectures for f and G, to showcase the flexibility of the system in Appendix D.
We present additional evaluation and comparisons with other unsupervised CoINs in Ap- pendix E.
We present ablation studies and discussions about design selection of the different compo- nents of the system in Appendix F.
We show additional visualizations for qualitative analysis in Appendix G.
We discuss overall limitations of the system in Appendix H and potential negative impacts in Appendix I.
A LANGUAGE-BASED CONCEPTS
A.1 SUPERVISED COINS AND LANGUAGE MODEL BASED CBMS
Recent variants of concept bottleneck models (CBMs) are trained with concept sets generated from language models (Yuksekgonul et al., 2022; Oikarinen et al., 2023; Yang al., 2023b; Panousis et al., 2024). These models are also CoINs that do not require human annotations for training. They obtain sample-wise concept annotations using concept set constructed from language models. Specifically, for each class, they prompt a language model to generate set of text descriptions describing the class. This set descriptions then used as the concept bank. For each sample, the concept annotation during training is obtained by CLIP similarity between the image and text descriptions of its ground-trui -ruth class. Note that all of these steps are executed prior to training the concept bottleneck model.
Interestingly, while these recent CBMs provide text descriptions for concepts thanks to automatic extraction from LLMs, they even worse offenders of the visualization limitations highlighted in Section 1. This is because they currently do not have any reliable visualization pipeline to confirm if the visual detection of given concept by the underlying CBM corresponds to its text descrip- tion or not. Moreover, in many cases the concepts as part of concept annotations are not even visually grounded (e.g. "Loyal/honest" for class "Dog") which case it impossible to visualize the concept in the first place. However, they are methodologically very close to original CBMs or supervised CoINs as they still use sample- wise annotations train (xx).
Besides the use of concept labels, CBMs do not have decoder model. There is no way to conduct any of the main evaluation (reconstruction, faithfulness, consistency) without decoder, since we need to approximate the input from concept activations to do any of these. All unsupervised CoINs have a decoder and none of the CBMs do, as a result of their training methodological differences. Thus, they currently out of scope for VisCoIN which is focused more unsupervised CoINs.
A.2 ON THE USEFCLNESS OF VISUALIZATION
We present below arguments why visualizing concepts is still important:
When considering expert or domain specific datasets like Stanford Cars (Krause et al., 2013) (for car models classification) or the MVTec Anomaly Detection (Bergmann et al., 2021) (for anomaly detection of object in production lines) for instance, visualizing cepts directly on the objects is simpler and faster to understand for human operators, rather than reading text description.
17

---

Published as a conference paper at ICLR 2025
For certain computer vision applications (eg. self-driving cars, medical imaging tasks), visualization provides spatially localized interpretations, which is more difficult and cum- bersome with text. For instance, if a concept relating "red light" is activated for an image, to get thorough understanding of the model's 's decision, which regions and what content in the image activates the concept. Ideally, this would be best proved by human evaluation comparing language descriptions and visualizations for real-world applications, to evaluate if/how much advantage visualizations add. However, for fair comparison language description and visualization should be of the same concept dictionary for the same model. To design such a study and system remains a challenging problem, which we will explore as a future direction.
crucial to identify
The LLMs/VLMs which the recent CBMs are based on (particularly CLIP (Radford et al., 2021)) )) limited when detecting concepts and image details at finer spatial scale (Gou et al., 2024).
As discussed above, the current methods are prone to generating concept descriptions not grounded in any visual information, which also harms their interpretability.
In the case of LLM/VLM based CBMs, there are also concerns about faithfulness of con- cept detection to the text description. This is a similar issue to concept leakage (Havasi et al., 2022). We believe that the ideas presented in work, such viewability, can help in identifying such issues in LLM/VLM based CBMs.
B DESIDERATA FOR GENERATIVE MODEL
We discuss below the desired properties that can influence the choice of an appropriate generative model G, previously mentioned in Section 3.2.
(i) The generative model should have the ability model the input with a low dimensional latent space (compared to input dimensions), since a concept representation is typically much lower dimensional than input.
(ii) It possesses structured latent space that admits convenient mechanisms for meaningful la- tent traversal. This directly helps in designing a Ω such that modifying a concept activation can enable latent traversal for visualization.
(iii) The generative model G is able to unconditionally generate high quality samples, close to the underlying dataset/data distribution. This is essential not only for better reconstruction of images, but also crucial ground any visual modifications in the generated images back to the original input.
The desiderata discussed above enables our approach to be compatible with a variety of generative models. Notably, most GANs and VAEs satisfy the first two requirements. Provided they are capa- ble to approximate well the underlying data distribution, our approach fits well with them as choice of G. The experiments in main text focus on StyleGAN2-ADA (Karras et al., 2020) as our G archi- tecture given its high-quality generation for various large-scale image domains. We also illustrate applicability of our method of other architectures such as ProgressiveGAN (Karras et al., 2018) and 8- VAE (Higgins et al., 2017) in Appendix D.
For diffusion model (Sohl-Dickstein et al., 2015) architectures, despite the recent positive steps towards understanding their latent space (Kwon et al., 2023; Park et al., 2023), it is currently difficult to design an that allows a straightforward and meaningful latent traversal. Nevertheless, is worth mentioning that latent diffusion models (Rombach et al., 2022) do satisfy (i) and (iii) in many cases. With further research in understanding their latent spaces, we believe it could be possible to satisfy (ii) and extend VisColN to them.
18

---

Published as a conference paper at ICLR 2025
C FURTHER SYSTEM DETAILS
C.I NETWORK ARCHITECTURES
The networks f and G are pretrained and fixed during training of VisCoIN. As part of our training, train three subnetworks V, S2 θ. We already described é in the main text, as consisting of pooling (maxpool), linear and softmax layers in the respective order.
General Architecture of y Our architecture of y mostly follows proposed architecture of y for FLINT (Parekh et al., 2021) which accesses output of two layers for ResNet18 close to the output layer (output of block 3 and penultimate layer of block 4). The ResNet50 also follows a similar structure with 4 blocks. Each block however contains 3, 4, and 3 sub-blocks termed "bottteneck"" respectively. In terms of the set of layers accessed by Vo for VisCoIN, in addition to the corresponding two layers in ResNet50 (output of block 3 of shape 1024 16 x 16, output of penultimate bottleneck layer in block 4 of shape 2048 8 x 8), we also access a third layer for improved reconstruction (output of block 2 of shape 512 x 32 x 32). Each layer output is passed through a convolutional layer and brought to common shape of 512 x 8 8, the lowest resolution and feature maps. We then concatenate all the feature maps to output D (x). We apply two convolutional and pooling layer yielding an output shape of K x 3, where K is the number of concepts, and each dx (2:) is a convolutional map of size 3 x 3. Thus, the total number of elements in each dx (I) is b 9.
General Architecture Q The typical theme we use to implement Q is as a single linear layer that takes as input (xx) and outputs a vector in the latent space of the generative model. This directly associates each ΦR with vector in the latent space of G. specified by the k-th column vector of weight matrix in Ωt While this exactly corresponds to our implementation for ProgressiveGAN, 3- VAE, our network designs for StyleGAN2 have slight modifications to V, V even though we still follow the same themes. We discuss next the precise architectures with StyleGAN next.
C.1.1 RECONSTRUCTION WITH STYLEGAN
The reconstruction architecture with StyleGAN is similar to encoder based GAN-inversion archi- tectures used for StyleGAN. Following previous works in this regard (Abdal et al., 2019; Yao et al., 2022), we use the extended latent space W for inversion, which corresponds to different latent vectors for different resolutions. We learn the concept translator Ω to map the concept activations to W and bias its computation with average latent vector w of G.
Since (xx) is a much lower dimensional representation compared to elements in W and con- strained by losses unrelated to reconstruction, we found it challenging to achieve reconstruction quality close to GAN inversion methods. This issue in principle cannot be completely eliminated without compromising the interpretable predictive structure. However, we alleviate by learning an unconstrained "supporting" representation
secondary output from V termed ('(z). The only goal for l'(x) is to assist Q in embedding the input in W To predict f'(x) the concatenated feature maps in Vi are sent in a second parallel branch, that applies two fully connected layers to output same number of elements as in & (x)) The computation for reconstruction å is then given as:
f = =Gww)) ), where w = 22 (( (x), o (x)) E W+
In this case, the concept translator Ω consists of a set of single fully-connected (FC) layers, one for predicting each latent vector. Each FC layer either takes (xx) or d'(x) as input depending upon the latent vector it predicts. To determine, which latent vectors should be controlled by 4', we rely on findings from the work in Katzir et al. (2022) which roughly divides the different style vectors in W+ as controlling the coarse, mid and fine level features of the generated image with increasing resolution. For 14 latent vectors in case of 256 x 256 output resolution, it corresponds to 4, 4, vectors respectively. We expect the relevant features for classification to be controlled mostly by mid-level and fine-level style vectors, except possibly for the highest resolution where very fine- controlled. Thus, predict the first three and last two style vectors using F'(x), The rest of the style vectors (9 out of 14 for resolution 256) are predicted from & (z)) The choice of using 'p' is analyzed with ablation study in Table 15.
scaled details
19

---

Published as a conference paper at ICLR 2025
C.2 TRAINING DETAILS
The steps to train our system on a given dataset can be divided into three modular parts: (1) Ob- taining pretrained classifier with "strong" performance that can provide high-quality source representations to learn from, (2) Obtaining pretrained generator G that can approximate well the distribution of the given dataset, and (3) Training of g with VisCoIN using the pretrained f and G. When pretrained f or G is not easily available, we train them on their respective tasks on the given dataset. Among our 3 datasets, CelebA-HQ (Karras al., 2018), CUB-200 (Wah et al., 2011) and Stanford Cars (Krause et al., 2013), we easily found pretrained G for CelebA-HQ. All other combinations of f and were pretrained. We describe the training details f. G and VisCoIN below.
C.2.1 PRETRAINING f
We pretrain f for classification on each of our datasets before using it for training VIsCoIN. We use Adam optimizer (Kingma& & Ba, 2014) with fixed learning rate 0.0001 on CUB-200 and 0.001 on CelebA-HQ to train f. On Stanford-Cars we use SGD optimizer with starting learning rate of 0.1,, decayed by factor 0.1 after 30 and 60 epochs. The training initialized with pretrained weights from ImageNet in each case, and fine-tuned for 10, 30 and 90 epochs on CelebA-HQ, CUB- 200 and Stanford-Cars respectively. In all cases, during pretraining, the images are resized to size 256 x 256. The accuracy of f is already reported in the main paper. All of these experiments have been conducted on a single A 1100 GPU, with a batch size of 64 for CelebA-HO and 128 for Stanford- Htanford-Cars dataset and on V 100 GPU with a batch size of 32 for CUB-200
C.2.2 PRETRAINING G
We use a pretrained StyleGAN2-ADA (Karras et al., 2020) for experiments in the main text. On CelebA-HQ, we used a pretrained checkpoint available from NVIDIA. For CUB-200 and Stanford- Cars, we pretrain G ourselves. Note that since we want G to generate images entirely from in- formation provided by ((x), we do not use any class labels when training G. We use the official StyleGAN2-ADA Pytorch repository to train our models. A challenge that can arise is from limi- tations to training resources since these models might require to be trained with tens of millions of real/dataset images ("shown" to the discriminator) in order to reach high quality generation. This could potentially require training with multiple GPUs for multiple days. We address this issue to reasonable extent by fine-tuning pretrained checkpoints. We utilize the insights from (Grigoryev et al., 2022) and fine-tune a checkpoint from ImageNet for CUB-200, and LSUN Cars (Yu et al., 2015) for Stanford Cars. The choices of these specific models was specifically based on the idea that these datasets were the closest domains we had access of pretrained checkpoints to. We also present experiments with pretrained ProgressiveGANN (Karras et al., 2018) and B-VAE (Higgins al., 2017) in Appendix D. For ProgressiveGAN, we use pretrained checkpoint on CelebA-HQ provided by NVIDIA. The 3-VAE trained by us with B = 2.
We train the G on a single Tesla V100-32GB GPU with mostly default parameters from the official repository. We only differ in (1) Learning a mapping function (that learns to predict latent vectors from gaussian noise vector) with 2 FC layers and (2) For Stanford cars, observed a collapse in generation of viewpoints with default training after 600k images shown, thus we reduced the strength of horizontal flip augmentation to 0.1 instead of default 1.
We use a batch size of 16 for training. The GANs are trained only on the training data. The final pretrained model for CUB-200 is obtained after training the discriminator with 2 million real images (21 hours). The final model for Stanford-Cars was obtained after training with 1.8 million dataset images (18.5 hours). The pretrained models achieve FID of around 9.4 and 8.3 on CUB-200 and Stanford Cars, respectively.
C.2.3 TRAINING VISCOIN
We train for 50K iterations on CelebA-HQ and 100K iterations on CUB-200 and Stanford-Cars We use Adam optimizer with learning rate 0.0001 for all subnetworks and on all datasets. During training, each batch consists of 8 samples from the training data and 8 synthetic samples randomly
https://g/ www.. www.s//wwb//ae/aa /tty aattoo pytorch
20

---

Published as a conference paper at ICLR 2025
generated using G. This practice of utilizing the synthetic samples from G' is fairly common for mucoder-based GAN inversion systems (Yao et al., 2022; Xia et al., 2022), and an additional ad- vantage for our system to pretrained G. Note that the use of fidelity loss with pretrained instead of a classification loss on g(x) fits neatly with this, as one cannot obtain any ground-truth annotations for the synthetic samples. The training data samples
a random cropping and random horizontal flip augmentation in all cases, All images are normalized to the range [-1, 1] and have resolution 256 x 256 for processing. This the default range and resolution we use for pretraining for f and G too. We have already described the architectures of all our components, pretrained or trained as part of training VisCoIN. We tabulate below in Table 3 the hyperparameter values for all our datasets. To limit the amount of hyperparameters to tune, we used fixed a 0.5 (weight for output fidelity loss) and 3 = 3 (weight for LPIPS loss) for all datasets. The rationale behind choice of all hyperparameters is discussed in Appendix F, wherein we also present the ablation studies w.r.t to multiple components.
Table 3: Hyperparameters values for VisColN
Parameter
CelebA-HQ
CUB-200
Stanford Cars
K- Size of concept dictionary 4
64
256
256
a Weight output fidelity
0.5
0,5
0.5
3 Weight for LPIPS
3.0
3.0
3.0
Weight for eccostractionnclaasficaal
0.2
0.1
0.05
5 Weight for sparsity
2
0.2
0.2
C.3 EVALUATION DETAILS
C.3.1 METRIC COMPUTATION
The median faithfulness is computed over 1000 random samples from the test data. For consistency, we use N₂. = 100, À= 2, i.e., given any concept ds. we extract its 100 most activating samples over samples of classes its most relevant for. The constant high activation is twice (A = 2) the maximum activation of Φ& (z) over the pool of 100 samples. Thus the binary training dataset created via samples from training data consists of 200 samples, 100 "positive" samples with high activation of Φx (I) and 100 "negative" samples with zero activation Φz (I). The binary testing dataset also contains the same number of samples of each type but is created via samples from test data. The feature maps we extract are the output of the second block of the pretrained f (ResNet50), the 22nd convolutional layer. The shape for each feature map is 512 32 x 32. We pool them across the spatial axis to obtain an embedding of size 512 for any input sample. The linear classifier we train linear SVM. We select its inverse regularization strength C from the set {0.01, 0.1, 1.0, 5.0) (lower value is stronger regularization). The parameter is selected using 5-fold cross validation on the created training data.
C.3.2 BASELINE IMPLEMENTATIONS
We utilize the official codebase available for FLINT and FLAEM for our baseline implementation. For fairness, we use the same number of concepts for both of them. Since our architecture is closer FLINT, we update and adapt it to implement in similar settings ours. We use the same f archi- tecture for both the systems and keep it pretrained and fixed. The V architecture is also similar in that it accesses the same set of hidden layers and has the same structure and depth. For other hyper- parameters we use their default settings applied earlier for CUB-200. Implementing FLAEM with same network architecture is more complicated as deviates considerably from the proposed archi- tecture, thus mostly use their default settings. In their code, they use base classifier architecture similar to ResNet101 and use the output final conv layer as the concept representation. In both cases, we do not modify the decoders. FLAEM uses simpler decoder that learns deconvolution layers, while FLINT learns a deeper decoder consisting of transposed convolution layers.
httpp://ggttuub..mm//aeeete https://githube ani cat. honnte-hoo_Kppll1 a nabi iity_Concepps
21

---

Published as a conference paper at ICLR 2025
Table Accuracy of interpreter (in %), reconstruction quality (MSE, LPIPS and FID), faithfulness (median F,F, for threshold 7 0.2) and consistency CC2, (mean and standard deviation of binary accuracy in %), on CelebA-HQ dataset, when using different generative models as decoder.
Method
Acc. (1)
MSE (L)
LPIPS (L)
FID (L)
FF. (1)
CC2 (1)
VisCoIN ProgressiveGAN
87.82
0.095
0,453
6.98
0.37
93.8 + 6.8
VisCoIN StyleGAN2-ADA
87.71
0.094
0.405
8.55
0.171
85.5 + 13.9
Original image
== 0
λ == 4
Concept "Smile long hair" in class Young
Figure 5: Qualitative results of VisCoIN with ProgressiveGAN (Karras et al., 2018) on CelebA-HQ dataset.
D EXPERIMENTS WITH OTHER ARCHITECTURES FOR f AND G
In this section, we demonstrate that our model can generalize to variety of network architectures for f and G. In particular, show applicability of VisCoIN with LeNet, ResNet101 and ViT-B/16 architectures for f. and B-VAE and ProgressiveGANs architectures for G.
D.1 EXPERIMENT WITH PROGRESSIVEGAN
We present Table results of experiments using a pretrained ProgressiveGAN (Karras et al., 2018) as our generative model G, on CelebA-HQ dataset, all other hyperparameters being identical as experiments presented the main text on this dataset. As can seen in the table, we obtain simi- lar accuracy and MSE, but FID, faithfulness and consistency are better using ProgressiveGAN, while LPIPS is better using StyleGAN2-ADA, We show an example concept visualization Fig. 5. The visualization clearly reveal two features ('Smile", 'Lonn Hair") both controlled by single concept
Table Accuracy of interpreter (in %), reconstruction quality (MSE), faithfulness (median FF, different threshold) and consistency CC2 (mean and standard deviation of binary accuracy in %), on FashionMNIST dataset.
FF, (t)
Method
Acc. (t)
MSE (L)
CC2 (1)
0.4
0.2
0.1
VisColN 8- V-V
88.06
0.029
0.576
0.693
0.762
94.3 + 7.4
FLINT
86.41
0.031
0.564
0.688
0.728
96.2 + 4.8
22

---

Published as a conference paper at ICLR 2025
Original image Reconstruction
Interpretation À =
Original image
Reconstruction
Interpretation
À
A= =
X 4
(a) Concept "Heel" in class Ankle boot
(b) Concept "Cut between legs" in class Trousers
Figure 6: Qualitative results for VisCoIN B-VAE FashionMNIST.
Table 6: Accuracy of interpreter (in %), reconstruction quality (MSE, LPIPS and FID), faithfulness
(median F F2 for threshold r 0.2) and consistency CC2, (mean and standard deviation of binary accuracy in %), CUB dataset, when using different architectures for f.
Method
Acc. (f)
MSE (L)
LPIPS (L)
FID (L)
FF, (t)
CC2 (1)
VisCoIN RN50
79.44
0.16
0.545
15.85
0.146
85.0 1 8.4
VisCoIN RN101
79.44
0.16
0.542
11.24
0.202
94.0 = 6.2
function. The high consistency and faithfulness is also possibly the result strong modifications in generated images when traversing latent space in ProgressiveGAN, However, most importantly, reconstruction quality worsens compared to results on StyleGAN and affects visualization. This is major factor in our preference of the StyleGAN version of VisCoIN over ProgressiveGAN ver- sion of VisCoIN. This is also a good example to illustrate the important role viewability plays in visualization.
D.2 EXPERIMENT WITH BETA-VAE
We also experiment with applying VisCoIN using 3-VAE (Higgins al., 2017) generative model, on FashionMNIST dataset (Xiao et al., 2017). We use = 25 as the number concepts. The VAE trained with 3 and latent embedding size of 48. Given that the dataset consists of grayscale images of smaller resolution, we do not use LPIPS loss during training. We use f architecture similar LeNet (LeCun, 2015).
Table 5 reports the result of our VisColN system against FLINT as baseline. For reconstruction, we only report the MSE since both LPIPS and FID rely on fixed networks pretrained on large scale color images and are thus not suitable for this dataset. We can see that our VisCoIN achieves better accuracy, reconstruction quality and faithfulness than FLINT. For small and grayscale images, even though we observe an advantage in using generative model over standard decoder, the effects are less pronounced compared to more complex images and classification tasks (as main text). The main reason for this is that in the current scenario, the standard decoders are also quite capable at generating viewable reconstructions. This gap with generative models grows larger as the underlying complex. Nonetheless. crucially, these experiments show that
data distributions grows method is effective even with a completely different generative model. Qualitative results of two concepts and their visualizations be found in Fig. 6.
23

---

Published as a conference paper at ICLR 2025
Original Input
Difference Image
=
x=4
Figure 7: Example concept visualization (ø36, class 15) for ViT-B/16 CUB-200 ("Red front/belly")) ") We show the original input, generated/reconstrrueed images with intervened acti- vations (A 0, 4) and the difference between generated images to localize the modifications.
D.3 EXPERIMENT WITH RESNETI01
present in Table additional experiments on CUB dataset, when using a ResNet101 network as our classification network f. all other hyperparameters being identical. We can see that VisCoIN achieves similar accuracy but better perceptual reconstruction quality (LPIPS and FID), faithfulness and consistency. While relatively nominal, the consistent improvement in all interpretability evalu- ation metrics compared to ResNet-50 is likely the result of better quality of hidden layers supplied by larger backbone.
D.4 EXPERIMENT WITH VTT-B/16
We report in Table an additional experiment using ViT-B/16 for ff on CUB dataset, while keeping other architectures almost identical to experiments main text. We started from a ViT-B/16, pre- trained on ImageNet-21 (Wu et al., 2020; Deng et al., 2009) and only finetuned the classification head on CUB, to use as f. We take the patch embeddings of final layer as input to v. We keep identical Ω θ hyperparameters, and slightly modify V/ for reduced number of feature maps. As can be seen from the numerical results below, we achieve better accuracy thanks to a better pretrained f. but reconstruction (LPIPS) and faithfulness worse. The results could be improved by better designing V and accessing more internals embeddings. However, they certainly show that VisCoIN can generalize to other backbone architectures. An example concept visualization support this is shown in Fig. 7.
24

---

Published as a conference paper at ICLR 2025
Table 7: Accuracy of interpreter (in %6), reconstruction quality (LPIPS), faithfulness (median FF2 for threshold = 0.2), on CUB-200 dataset, when using ViT-B/16 architecture for f.
Model
Acc. f
Acc. g
LPIPS ())
FID (J)
FF2 (T = 0.2) (1)
VisCoIN ResNet50
80.56
79,44
0.545
15.85
0.146
VisCoIN ViT-B/16
86.66
85.86
0.582
14.88
0.081
Table 8: Mean and standard deviation for consistency CC2 over all concept functions Øy (binary accuracy in %), using À 3. Higher is better. Consistency increases when compared to À (results in main text) and the increase is greatest for VisCoIN.
Dataset
FLINT
FLAEM
VisCoIN (Ours)
CelebA-HQ
83.4 + 22.9
59.5 + 19.2
90.9 + 13,8
CUB-200
76.4 + 20.8
56.7 + 14.1
93.4 + 6.2
Stanford Cars
77.3 = 19.6
55.2 + 13.5
91.6 + 6.3
E ADDITIONAL EVALUATIONS
E.1 CONSISTENCY WITH HIGHER À
We present the results here for evaluating consistency with a higher value of À 3, compared to the main text where À 2. Thus, the constant high activation of (x(xx) used to generate "posi- tive" samples of the dataset is increased further. We thus expect the "separation" in the embedding space to increase and consequently a higher performance CC2 of the binary classifier μp for any k. The results for both are presented in Table 8. The results confirm that emphasizing the concept indeed makes the visual modifications more stronger and consistent. Moreover, we also observe that increase in consistency of VisCoIN tends to be larger than increase for other CoIN systems.
E.2 QUALITATIVE VISUALIZATION FID
clearly observe the difficulty understand activation maximization based visualization in FLINT. We further support our claim about the unnaturalness of these vi- sualizations compared to visualization in VisCoIN by computing the distance of distributions of visualizations in FLINT, visualizations in VisCoIN and original data distribution, reported in Table Note that we can't any reconstruction metrics FLINT visualizations don't reconstruct given input. Instead they initialize using given maximum activating sample and execute the op- timization procedure of activation maximization maximally activate a Φu (x). We thus compute the FID distance between the visualizations and the data distribution. For VisCoIN visualization we select our most extreme value of X. For FLINT visualization we follow their implementation and run the input optimization procedure for 1000 iterations. Since the FLINT visualizations relatively lot more expensive to compute (1000 backward passes vs forward pass for VisCoIN), we compute the visualizations for 3 maximum activating samples for random 400 relevant class pairs (with Fk.c > 0.5). Thus the FIDs are computed between 1200 data samples and corresponding visualizations.
While qualitatively,
Table 9: Quantitative evaluation (FID) of the visualization obtained for interpretation with the orig- inal data distribution (1200 samples). Lower is better. For FLINT, visualization activation maxi- mization output. For VisCoIN, visualization is generated image with À 4.
Dataset
FLINT
VisCoIN (Ours)
CelebA-HQ
21.12
9.83
CUB-200
26.55
11.71
Stanford Cars
45.72
8
25

---

Published as a conference paper at ICLR 2025
Table 10: Impact of selecting only the Top- activated concepts in ((x) before for prediction, on accuracy of g (in %), for different values of N.
N
Dataset
4
8
16
32
64
128
256
CUB
23.75
42.25
59.28
70.07
76.25
78.97
79.44
Stanford Cars CelebA-HQ
13.38
26.43
45.75
62.76
72.43
77.20
79.89
79.92
80.63
84.00
86.90
87.71
Table 11: AUC of accuracy
when gradually adding most activated top-N (N {2, 4, 8, 16, 32, 64, 128, 200, 256} concepts for each test sample to (xx), along with final accuracy of y (in %) on reconstructed images, on CUB-200 dataset. Higher is better.
Method
AUC-FF metric
Acc. 9 on reconstructions
VisCoIN FLINT
0.407
58%
0.042
4.5%
E.3 TOP-N CONCEPT ACTIVATION FILTER
To preserve maximal amount of interpretable by-design structure, we design the θ function as single linear layer with softmax, similar to other unsupervised CoINs. Nevertheless, this by-design interpretable structure also "erodes" as the size of concept dictionary K increases. For large dic- tionary sizes, the activations are not extremely sparse, interpreting the prediction even through linear θ can become tedious and less interpretable (Lipton, 2018).
One way to preserve the interpretable structure even with large K by controlling the number of non-zero concept activations. We experimented applying "Top-N" function on Φ(x) before θ, to keep only the most activated concept for prediction, for different values of N, and report results in Table 10. Although it improves interpretability and conciseness of interesting concepts, it comes at the cost of accuracy of the overall system. However, we can see that using about 25% of the most activated concepts still preserves good accuracy in general.
E.4 AUC-FAITHFULNESS METRIC
We performed preliminary experiments to compare faithfulness of VisCoIN and FLINT on CUB- 200 by computing the area under the curve (AUC) of accuracy w.r.t number of "added" concepts. Specifically, for each test sample we initialize (xx) (r) to 0, increasingly add the most activated top- N concepts for N E {2, 4, 8, 12, 64, 128, 200, 256} to ((x), plot the accuracy of g(x'). x' ((((((Z)))) and compute its AUC.
We report the AUC Table 11. Note that since the accuracy is on generated images, the accuracy of g is lower than its accuracy on the same dataset. The results are strongly in favor of VisCoIN. We expect VisColN to generally outperform other unsupervised CoINs on this metric as it's capable to generate high-quality reconstructions.
E.5 ADDITIONAL CONSISTENCY EVALUATION
For datasets with annotations for presence/absence of different object parts, consistency for any given concept function Φb can also be evaluated by analyzing the "consisteney" of their impact on the "presence" of object parts in the generated images. Among the three tasks we evaluate VIsCoIN on, the CUB-200 dataset consists of binary annotations for 312 bird-parts for each image. Since such annotations are not available for generated images, prior to the evaluation, we first train a ResNet-50 based predictor to predict probabilities for each bird part. Let this predictor denoted as fpart- For given concept function 9u the goal of this evaluation is to ascertain if there is bird part which "consistently affected" when activation Φr (x) is modified. To quantify this notion, we first select test images of the classes for which dy is highly relevant, i.e. Tk,e 0.5. This set of images is denoted as X, For each x E Xt we compute ((x) and generate two images with intervened acti- vations, with (x) and other with (x) À x Φmaa done for consistency evaluation
26

---

Published as a conference paper at ICLR 2025
Table 12: Average part consistency over all concepts for FLINT, VisCoIN on CUB-200 (Scale: 0-1, Higher is better).
Part probability change threshold
Method
0.05
0.1
7 7= 0.2
VisCoIN
77.7 + 13.2
64.4 = 16.1
41.7 = 15.7
FLINT
56.6 27.5
39.6 + 27.1
18.8 = 20.8
Number of consistent concepts (%)
VisColN
T
FLINT
0.1 Part probability change threshold
0.05
0.2
Figure 8: Fraction of concepts which affect some bird part significantly for more than 50% of their relevant images (CC (Xe; r) > 0.5). Results reported for different levels of significant part probability change threshold T for VisColN, FLINT on CUB-200.
in main paper. We À for this experiment. The generated images are denoted 5y , respectively. Note that this dataset preparation is very similar to one used for consistency evaluation in main paper.
We compute part probabilities for all 312 parts (xe ), (eare(z7) We define ΦR to signifi- cantly affect part for image if the probability prediction part increases when dR activates by more than threshold т, i.e. fpart (x j (Txrt( ) > т. For each bird part j. its part consistency computed as fraction of x X s.t. 6% significantly affects part j in x. Consistency of Φ is calculated maximum part consistency over all the parts
Hx Sport (xt), ) -part(2x)) TH
CC (Ne;T) max
We report results for mean of (aa with 7 E {0.05, 0.1 for FLINT and VisCoIN in Table 12. We also report the fraction concepts that significantly affect some bird part for more than 50% of the images in Fig. 8. Both results indicate that on average concepts in VisCoIN affect some bird part, more consistently, compared FLINT.
E.6 SPARSITY EVALUATION
We report in Table 13 sparsity results of VisCoIN and FLINT for CUB-200 at different relevance thresholds. The sparsity is calculated as the average number of relevant concepts per class such that global relevance Tk.c threshold.
FLINT achieves better sparsity because of its use of entropy based losses to compress ((x). While sparsity of VisCoIN could be improved by increasing the L₁ regularization weight, prioritized optimizing for eeconstruction/vieeabbii major limitation. (ii) The current levels of sparsity seemed reasonable (total number of concepts K = 256 is much higher than relevant for any class), (iii) Excessive compression of information can make concepts less interpretable and similar class logits.
because (i) For previous unsupervised CoINs this
27

---

Published as a conference paper at ICLR 2025
Table 13: Sparsity, measured as average number of concepts per class with global relevance higher than threshold, of VisColN and FLINT on CUB-200 dataset. Lower is better.
Threshold on Tk,c
Method
0.7
0.5
0.2
VisCoIN
2.3
6.1
27.1
FLINT
1.5
3.4
10.2
loss in the total training loss, measured by Faithfulness, LPIPS and FID, on CUB-200. Faithfulness computed with threshold of 0.2. Bold indicates setting selected for our experiments. Experimentally, 7 causes tradeoff between faithfulness and quality of reconstruction.
Table 14: Effect of the weight of the
7
Faithfulness (t)
MSE (L)
LPIPS ())
FID (L)
0
0.001
0.142
0.52
13.11
0.1
0.146
0.161
0.545
15.85
0.2
0,236
0.192
0.607
8.84
0.5
0.24
0.209
0.634
11.54
F ABLATION STUDIES
We present ablation studies for various components of our system and simultaneously discuss our rationale behind the design selection of these components. Specifically, we study (a) effect of loss with weight ? in Appendix F.1, (b) role of using a supporting representation &' (x) to assist in reconstruction in Appendix F.2, (c) selection of number of concepts K in Appendix F.3, (d) effect of orthogonality loss in Appendix F.4, (c) effect of fidelity and sparsity loss weights in Appendix F.5, and (f) usefulness of concept translator in Appendix R.6.
cogonstructioncllassfccaac
We highlight at this point to the reader that there is an overarching theme that governed many of our design choices. In particular, most of them are based on shaping the systems suitability for better optimization of perceptual similarity for reconstruction. We constantly aim to achieve better reconstruction without major negative impacts for any other properties. This is because for complex datasets (CUB-200) Stanford Cars), the key bottleneck in the design is to achieve the high-quality reconstruction for viewability.
F.1 RECONSTTRUCTOONCCLSSSPCCCC.
LOSS
Table 14 reports the perceptual similarity and faithfulness with threshold 7 = 0.2 on test data of CUB-200 for different strength 7y Interestingly, it indicates tradeoff between faithfulness and perceptual similarity. One possible reason for this tradeoff is that higher y can push the model to generate spurious features captured by the classifier at the expense of input quality. Thus, even though the reconstructed images remain relatively far from input this can still lead to "accurate" predictions from the classifier i.e. predicts the same class input. Completely removing this loss heavily impacts the faithfulness. However, among the positive y, our choice was driven strongly by achieving reconstruction with high-enough quality and perceptual similarity to enable effective visualization. Thus, we chose 0.1. The key reason for this is that perceptual similarity is
Table 15: Effect of using the support representation Φ', measured by MSE, LPIPS and FID, on CUB-200. Bold indicates setting selected for experiments.
K
MSE (L)
LPIPS (L)
FID (L)
Yes
256
0.161
0.545
15.85
No
512
0.178
0.568
13.52
No
256
0.187
0.584
9.55
28

---

Published as a conference paper at ICLR 2025
Kocer
(a) Original batch of inputs
(b) Corresponding reconstructed images
Figure 9: (a) Final training batch of images shown the model. (b) Reconstruction obtained using the model trained with = 0.2. Even though FID is better, reconstruction quality is noticeably worse.
Table 16: Impact of the number concepts K used 4 on accuracy of g (in %),, LPIPS and FID, for CUB-200. Bold indicates setting selected for our experiments.
K
Accuracy (t)
MSE (4)
LPIPS (L)
FID (4)
512
79.78
0.156
0.537
17.27
256
79.44
0.161
0.545
15.85
128
79.03
0.183
0.578
8.64
64
78.91
0.203
0.624
6.3
much better indicator for viewability. For instance, for the 0.2, even though the FID is better, the reconstruction (LPIPS) is noticeably worse. Reconstruction of the final training batch is indicated in 9, to highlight this issue with high y. As is apparent from the figure, the perceptual similarity and consequently the viewability of the model is poor with high y. We thus kept smaller y for CUB-200 and Stanford Cars where the reconstruction is more challenging and slightly higher value for CelebA-HQ.
F.2 USE OF SUPPORT REPRESENTATION 4
The construct support representation and our use of it is limited to StyleGAN as architecture of G. We describe its precise design leveraging the StyleGAN architecture in Appendix C.1.1. While is not essential to learning and operation of VisCoIN, its inclusion offers lever to achieve better reconstruction without having increase the concept dictionary size K. Table 15 presents the reconstruction metrics for different concept dictionary sizes and use of &' in reconstruction. Using K 256 with the support representation offers slightly better reconstruction than even using K = 512 but no support representation. As before, we prioritized optimization of LPIPS and using Ø' assists in achieving better reconstruction whilst allowing to employ a smaller dictionary.
F.3 SELECTING NUMBER OF CONCEPTS
The ablation with different number of concepts is given in Table 16 and is an important hyperparam- eter of the system. While choosing K, it is easy to filter out smaller K values as they clearly lead worse reconstruction. However, hypothetically a higher should improve for all the metrics since it allows for more expressivity in the concept representation. Thus finding upper bound for K is more subjective. Our choice was mainly influenced by (1) the observation that increasing from 256 to 512 offered relatively minimal advantage in reconstruction, and (2) previous methods that used supervised concepts train with 312 concepts. Hence we intended to use similar dictionary size. Since the Stanford Cars dataset had comparable number of samples and classes, used the same number of concepts. For CelebA-HQ, we experimented with reduced K as there are only two classes and the images are less diverse compared the other two datasets.
29

---

Published as a conference paper at ICLR 2025
Table 17: Impact of orthogonality loss Lorth on accuracy of g (in %), MSE, LPIPS and FID, for CUB-200. Bold indicates setting selected for our experiments.
Cotth
Accuracy (1)
MSE (4)
LPIPS (L)
FID (1)
Yes
79.44
0.161
0.545
15.85
No
79.25
0.171
0.556
9.43
Table 18: Effect of weight on output fidelity loss Lof. measured accuracy of (in %), LPIPS and FID, for CUB-200. Bold indicates setting selected for our experiments. Low a affects perfor- mance and high affects reconstruction quality without much gain in performance
Accuracy (t)
MSE (L)
LPIPS (4)
FID (L)
0.1
76.9
0.162
0.545
9.37
0.5
79.44
0.161
0.545
15.85
79.63
0.187
0.586
7.58
F.4 ORTHOGONALITY LOSS
For the final layer of V choose layer can be represented as single matrix of size number of input feature maps times number of concepts K. We encourage the normalized columns be orthogonal which turn encourages each Φx (x: to be predicted using different feature maps. We report the quantitative effect of this loss in Table 17. Incorporating this loss offers slight advantage improved perceptual similarity. However, another key reason we incorporated this loss in our experiments is that qualitatively observed greater propensity of multiple concepts highly relevant for class to capture common concept about that class. This loss thus offered a way to encourage different concepts to rely on different feature maps. Note that it only affects parameters final layer ! that outputs (zz), and do not use any additional hyperparameter for it.
convolutional layer. Thus, the weights/kernels for this
F.5 OTHER LOSS WEIGTHS
We report the accuracy and reconstruction metrics for different a (weight for output fidelity loss) and 5 (weight for sparsity of activations). A small weight output fidelity degrades the performance of the system and a high weight affects the reconstruction without benefiting the performance much. We found balance with the performance and reconstruction since it encourages activation of smaller number of concepts for any input more strongly. However, eliminating 5 can result in poor sparsity and consequently interpretability of the concept activations for prediction. Our overall strategy was thus to use a high enough 8 that it does not significantly affect reconstruction quality. For CUB-200 and Stanford Cars, due to greater need of prioritizing reconstruction we opted for smaller 5 = 0.2, while for -Q, since obtaining a good reconstruction was relatively easier, we opted for higher 8 = 2.
0.5 which we employed for all datasets. high impacts both
We keep fixed B weight for LPIPS reconstruction loss throughout, for our datasets and ablations. Even though the system still provides meaningful results for 3 < 3, the lower values were ruled out mainly because of the importance
lower perceptual similarity loss, mentioned earlier. The higher values were ruled out because in our initial experiments we observed some instability with high? 4. Thus we fixed 3 for all datasets which provided a good balance.
F.6 BASELINE FOR CONCEPT TRANSLATOR Ω
We compare system with variant where directly use ((x) as latent vector wy for G, elim- inating Ω. While our model allows this design, it comes with certain limitations: (i) The user can't 't control the number of concepts. They are forced to employ concept dictionary of same size as dimension of the latent space. (ii) Since the generator is pretrained and fixed, the resulting ((x) learnt is not sparse. (iii) Finally, in particular for GANs, it foreibly associates concept functions
30

---

Published as a conference paper at ICLR 2025
Table 19: Effect of weight on sparsity, measured on accuracy of (in %),, MSE, LPIPS and FID, for CUB-200. Bold indicates setting selected for our experiments. A high å can affect both reconstruction and performance.
Accuracy (1)
MSE (U)
LPIPS ())
FID (L)
0.2
79.44
0.161
0.545
15.85
2
79.54
0.174
0.562
11.44
20
76
0.201
0.629
9.83
Table 20: Impact of concept translator Ω on accuracy of g (in %), MSE, LPIPS and FID, for CUB- 200. Bold indicates setting selected for our experiments.
Ω
Accuracy (f)
MSE (1)
LPIPS (L)
FID (U)
Yes No
79.44
0.161
0.545
15.85
79.24
0.182
0.572
15.96
with columns of identity matrix as directions in latent space. Using an Ω (for instance linear layer) allows the model to learn general directions in the latent space to associate to each concept func- tion, which aligns with the conventional strategy for latent traversal inside GAN, As can be seen in Table 20, removing Ω leads to poorer reconstruction.
F.7 OUTPUT FIDELITY LOSS
One can consider using cross-entropy loss with ground truth labels instead of "output fidelity loss". We specify the possibility use both when describing the general architecture of unsupervised CoINs (in Section 3.1). We decided to use the output fidelity loss following Sarkar et al. (2022). The current design also draws inspiration from knowledge distillation setting, in which student model is trained to reproduce output of teacher model. One additional perk of using this "output fidelity loss" during VisCoIN training is that it can also applied for images without annotations, such images sampled from G (further details about VisCoIN training in Appendix C). This provides additional guidance and stability to train g. For completeness, we include in Table 21 an experiment of VisCoIN trained using standard cross-entropy loss with ground truth labels CUB-20 B-0 dataset.
Table 21: Comparison between the "Output fidelity loss" and "Cross-entropy loss". measured by ac- curacy of interpreter g (in %), reconstruction quality (LPIPS), faithfulness (median FF, for thresh- old r = 0.2), on CUB dataset.
Model
Acc. g (1)
LPIPS (L)
FF, (T 0.2) (t)
VisCoIN Output fidelity loss VisCoIN Cross-entropy loss
79.44
0.545
0.146
78.89
0.559
0.076
31

---

Published as a conference paper at ICLR 2025
Original image
Removed concept
Reconstruction
Interpretation A= 4
Original image
Removed concepp A= 0
Reconstruction À A=
Interpretation
À
A= 4
(a) Concept "Red neck" in class 15
(b) Concept "Yellow front" in class 175
(c) Concept "Paleness" in class Old
(d) Concept "Smooth skin" in class Young
(e) Concept "Big headlights grille bars" class 194
(f) Concept "Logo" in
Figure 10: Additional qualitative examples obtained for different concepts, classes on (a)-(b) CUB- 200, (c)-(d) CelebA-HQ. (e)-(f) Stanford-Cars datasets. On each subfigure, first column corresponds to maximum activated samples for class-concept pairs with high relevance (Fk.e > 0.5), third column to reconstructed image obtained with original (xx), while second and fourth columns the images obtained by imputing respectively (x) 0 and 4 (x(x) in ((x).
G ADDITIONAL ANALYSIS
We show additional visualizations for different highly relevant class-concept pairs (rk,c > 0.5) in Fig. 10. For each class-concept pair, we show the effect of modifying the concept activation on the generated output for three maximum activating training samples. For each sample (on the far-left), we show the corresponding generated outputs for 0 (center-left), À = 1 (center-right) and (far-right).
32

---

Published as a conference paper at ICLR 2025
Original image
Reconstructed image
Φ 110: Blue upperparts
144 Black wing T144 = 0.774
Φ37: Red Neck
P110
T37 0.541
Figure 11: Local interpretation CUB dataset (sample from class 15) for different concepts along with their relevance scores.
G.1 LOCAL INTERPRETATIONS
We present in Fig. 11 local interpretations test samples of CUB dataset, for different concepts relevant to that class, For a single test sample, user obtains multiple concepts that are relevant for the prediction of the test sample. We extract the concepts with high relevances for the given input. For each concept one can use the visualization pipeline introduced to visualize what part of the input activates the respective concept. Note that the visualizations for 10 637 remain consistent with their global visualizations.
G.2 USE OF DIFFERENCE IMAGES CAN BE USEFUL
To better highlight regions in the image impacted by modifying concept activations, one can addi- tionally visualize the differences between two generated outputs. We show visualizations with the difference in the generated outputs in Fig. 12. Again, we selected highly relevant class-concept pairs (Tk,e 0.5), and show the effect of modifying the concept activation on the generated out- put for three maximum activating training samples. For each sample (on the far-left), we show the corresponding generated outputs for compute and show the difference between the two generated outputs X' È (far-right). It is worth noting that this exact strategy might work effectively for all types of concepts and can require modifications. For example, if "black feathers" are emphasized by a concept, the increasing "black" color won't be visible in the difference between 3' Instead, one could either visualize the reverse difference between ž, Y' to identify color being "removed" or visualize the energy of difference for each pixel to identify which regions are modified the most.
(center-left), and X 4, f! (center-right). We then
G.3 AVERAGE LATENT VECTOR
We illustrate through example CUB in Fig. 13 that the average latent vector of pretrained G' is typically representative the dataset.
33

---

Published as a conference paper at ICLR 2025
Original image
Reconstruction
Interpretation
Difference of images
(a) Concept "Orange eye" in class
(b) Concept "Headlight shine" in class 18
Figure 12: More qualitative examples obtained for different concepts, classes on (a) CUB-200, (b) Stanford-Cars datasets. On each subfigure, first column corresponds to maximum activated samples for caas-ct pairs with high relevance (rk,e 0.5), second column to reconstructed image obtained with original (xx), third column to the image obtained by imputing dx(x)) in ((r), and fourth column shows the differennc between third and second images.
34

---

Published as a conference paper at ICLR 2025
Figure 13: Generated image for average latent vector ("center" of latent space) in CUB (temporary figure).
H LIMITATIONS OF APPROACH
As is the case for other CoINs learning unsupervised concepts, the proposed system cannot guarantee that concepts precisely correspond human concepts and not encode other addi- tional information. However, one interesting aspect is that visualization process in VisCoIN gives better handle at identifying any deviations as visualizations other unsupervised CoINs can be much harder to understand with granularity for large-scale images.
The choice of using a pretrained G improves training time, complexity and reusability, but also implies that the system's quality limited by the quality of the pretrained G. For instance, for visualization, if G can't generate some specific feature, it can be difficult to visualize a concept Φb that encodes that feature.
For the case of single FC layers in Ω visualization process follows linear trajectories in the latent space of G when modifying an activation. Recent work has shown that linear trajectories are not necessarily optimal for latent traversals (Song et al., 2023).
POTENTIAL NEGATIVE IMPACTS
Given that the understanding of neural network decisions is considered vital feature for many applications employing these models, specially in critical decision making domains, we expect our method to have an overall positive societal impact. However, the wrong hands almost any technol- ogy can be misused. In the context of VisCoIN, it can be used to provide deceiving interpretations by corrupting training mechanisms (for example by training on misleading annotated samples, using deliberately altered pretrained models etc.). Thus, expect responsible use of the proposed methodology to realize its positive impact.
35


Reviews:
[{"review_id": "aBa6AhRhcJ", "reviewer": "Reviewer_BbA8", "summary": "This paper proposes a new approach for inherently interpretable models by mapping the concept space to the latent space of a pre-trained generative model. In particular, this approach focuses on using this mapping to associate semantics with the concepts. To this end, the paper proposes the use of three adapter modules along with a pre-trained classifier and generator to achieve the goal. Appropriate losses and metrics are defined to train and evaluate the method. The experiments on standard large-scale benchmark datasets shows promise in the approach.", "questions": "Please see weaknesses above. Below are some additional questions:\n* Are there any restrictions of what f and G should be pre-trained on? How close should those datasets be to the one being studied? Since interpretability is the focus of this work, it would be useful to know how semantically related these must be.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The idea of using the latent space of a generative model to map implicitly learned concepts of a neural network model is interesting.\n+ The methodology is simple and effective.\n+ The paper is well-written with a good treatment of relevant literature.\n+ The method and experiments are well-documented for reproducibility.", "weaknesses": "- One fundamental concern (a part of which is briefly discussed in Appendix A): considering recent concept-based models that use LLMs for semantics, it can be argued that concepts can directly be interpreted through human-understandable language semantics. How important is visualization in such a scenario? Is it possible to show through some user studies that a user necessarily requires visualization beyond just language semantics in real-world applications? Without this, the premise of this work may be weak.\n\n- A second major concern is the limited baselines used for experimental comparison. Many baselines seem missing: Label-free CBMs, LaBo, Posthoc CBMs, Sparse CBMs. It is not clear why some of these were not considered -- it may not be difficult to adapt some of them for comparison. Further, while some of them lack \"visualizability\", comparing the proposed method w.r.t. these baselines on other metrics is important for completeness of understanding.\n\n- While the appendix reports many ablation studies, I found in general a lack of depth of analysis of the results, with a propensity of very brief analysis of multiple factors. I would have preferred seeing at least some of the important analysis being carried out in depth. In fact, I found the appendix hard to parse since there were too many studies, but with too little discussion on inferences and take-aways.\n\n- One reference that is close to this work is: \"Garg et al, Advancing Ante-Hoc Explainable Models through Generative Adversarial Networks, arXiv:2401.04647, AAAI-W'24\". It may be good to compare the proposed work against this paper, since they have similar objectives esp the viewability property.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ySam4GFU9K", "reviewer": "Reviewer_gc79", "summary": "A primary limitation in human-understandable concept learning within intrinsic XAI approaches is the effective visualization of learned, unsupervised concept dictionaries, particularly for large-scale images. To address this challenge, the authors introduce VisCoIN, a novel, concept-based interpretable network that includes a concept translator, which maps concept vectors into the learned space within a generative model. Experimental results visualizing the learned concepts highlight the efficacy and practical value of the proposed approach.", "questions": "Most of my major questions/concerns are listed in the Weakness sections.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- S1: The definition of viewability and the proposed method are clear, well-founded, and intuitive. Experimental visualizations of the learned concept effectively demonstrate its amplified appearance as the lambda value varies within the concept function, providing compelling evidence of the approach's effectiveness.", "weaknesses": "- W1: The authors introduced an additional pretrained classifier f, and incorporated its output along with g(x) into the output fidelity loss, \\( L_{of} \\). The rationale behind this approach, however, requires clarification. It would be beneficial for the authors to validate the use of output fidelity loss \\( L_{of} \\), particularly in comparison to employing cross-entropy loss with ground-truth labels.\n\n- W2: The auto-encoding objective is inherently insufficient for the acquisition of compositional representations, as the optimization of reconstruction quality does not necessarily entail the disentanglement of features at the object or concept level [1]. The integration of functions \\(\\Psi\\) and \\(\\Omega\\) can certainly be characterized as an auto-encoder architecture. In the main text and appendix, the authors present further applications that employ various functions \\( f \\) and \\( G \\). Nevertheless, it would be advantageous to explore modern architectures, such as transformers for \\( f \\) and diffusion models for \\( G \\).\n\n- Reference\n- [1] Jung, Whie, et al. \"Learning to Compose: Improving Object Centric Learning by Injecting Compositionality.\" arXiv preprint arXiv:2405.00646 (2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OxePCjt3AT", "reviewer": "Reviewer_9qiE", "summary": "This paper proposes a new system for concept-based interpretable networks, where the learned concepts are transformed to the latent space of a generative model. This transformation is learned, enforcing consistency by minimising the distance loss between the original image, and the image produced by the generative model using the concepts of the original image. This allows users to easily visualise the learned concepts, by generating images in a range for a particular concept.", "questions": "- The authors state that the maximum $\\lambda$ that is reliable is 3 or 4. Is this due to the magnitude of the concept seen by the translator network during training? And if so, couldn't think potentially be an issue when setting $\\phi_k(x)$ to 0 when measuring faithfulness, as this might be outside of the range seen during training for a particular concept.\n- $\\Omega$ used is a very simple architecture, did the authors ever experiment using multiple layers to allow a more complex transformation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I agree with the idea that using examples from your training set that maximally activate a certain concept is not ideal, and the authors address this nicely. The concept visualisations generated are much simpler to understand than concept attribution maps and visualising image patches that maximise the concept.\n- Leverages pretrained generative models rather than training an additional decoder, instead using a less complex concept translator, making the method more efficient. \n- The measure of consistency of concept visualisation is novel and seems appropriate as a proxy for human experiments.", "weaknesses": "- Some of the reconstructions shown throughout the paper are quite far from the original image, and sometimes changing the image quite drastically, such as Figure 4b and more examples in the appendix. This is one benefit existing methods like concept attribution maps have, the concepts are highlighted on the original images rather than relying on a reconstruction. \n- Some details in C1.1 on how reconstruction is performed with StyleGAN are important, for example the need for using a secondary, unconstrained representation $\\Phi'(x)$, should be in the main text, as this suggests that reconstruction purely based off the interpretable concepts is not possible. This could be added in Section 3.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sIlveZcdn0", "reviewer": "Reviewer_2hPr", "summary": "The paper proposes a new method of interpreting latent features which builds upon concept-interpretable neural networks (CoINs) and adds the new means of interpretation through mapping the latent features into the space of generative models. It helps address the known problem of interpretations of concepts-based models in a way that is complementary to the existing explanations, which often relates the concepts to the real data.", "questions": "1. It would be important to clarify upon the limitations regarding to the relation of the proposed method to the inherently-interpretable class of methods. Such methods are supposed to provide by-design explanation, where the output is causally related to the explanation. For example, concept bottleneck models (Koh et al, 2020),  ensure such by-design claim by making the prediction directly rely upon the intermediate concepts (i.e., first, we predict a  number of properties, and then infer the class solely on these properties).  In this work, I can identify two points where this by-design property could be broken: (1) I understand that the output depends upon the whole set of features, which means that the less-contributing features can still influence the prediction enough to change the label. Alternatively, one can select only a part of features, perhaps at a cost of accuracy (2) the matching between the latent space and the generative model is performed using a concept translator; it means that in some scenarios the mapping between the latent space and the generative model can be imperfect. One might find it useful, perhaps even at a cost of accuracy, to bridge these gaps and make the classification inherently-interpretable. It may be achieved, to address the by-design limitation (1), by only performing the prediction from the features which contribute the most and discard the rest, and provide by-feature explanations for these features. To address the limitation (2), one may think of learning the latent space in a way that it coincides with the generative model’s one (i.e., through distillation). I wonder if this model allows for this?\n2. I see that the model is evaluated using ResNet-50 backbone. I wonder if it can generalise to the transformer-based architectures? In relation to this, would the authors clarify upon the computational overhead of the proposed method in comparison with the CoIN and the ResNet-based models?\n3. Figure 4 states “Visual modifications of more local concepts indicated by red boxes” I am not sure I could  understand what it would mean and entirely follow how the red box visualisation works .", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Correctness: I checked the notation and did’t see any errors. \n- Reproducibility: the paper looks reproducible to me (see Question 3 below)\n- Novelty: the method builds upon the existing CoINs methods but provides the new means of interpretation comparing to the original CoIN methods, and therefore is novel in this way. \n- Clarity: the outline of the paper is clear, however with some suggestions on the presentation\n- Significance: this work complements previous work on concept-based interpretation. The approach is well-motivated by the need in providing more detailed interpretations of the vision recognition models and is, as the authors discuss in the introduction, well-grounded in the state-of-the-art. The significance of the work is mostly empirical, with the authors presenting both the advantages of the method and the evaluation protocol.", "weaknesses": "- Clarity: There are a few questions below which could help clarify upon the relation between the method and the by-design approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdGWy8K7Xm", "reviewer": "Reviewer_Lumc", "summary": "The authors propose a method to map concept features to a generative model latent space to have a better interpretation of the learned concepts. They focus on concept-based interpretable networks (COIN) to build the visualization system. They evaluate the interpretation in terms of the accuracy of the prediction network (fidelity to output), the fidelity of the reconstruction (fidelity to input), and the fidelity and consistency of the learned concepts. They also propose the analysis of sparsity and viewability (capacity of reconstruction over concepts).", "questions": "Some questions are already presented in the weakness section.\n\nOther questions:\n\n→ Why does the bird head of the first row in Figure 4 also increase in size with the red eye? Is this related to another concept?\n\n→ How did you determine the \"name\" of the concept? Is it a deduction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Writing and Implementation Details: I appreciate the authors' attention to motivating and well-describing the methodology, including detailed implementation parameters in the main and supplementary texts.\n\n- Improvement of general interpretability: the authors present an interesting approach to improve the interpretability of visual networks by learning sparse concepts that can be quantified and visualized.\n\n- Numerous evaluation methods: an important aspect of this work is the aspect in which the methodology is evaluated, which configures an important validation of the proposition: prediction accuracy, fidelity to reconstruction, faithfulness, consistency, sparsity, and viewability.", "weaknesses": "- Combination of two complex models to interpret and explain: Have you analyzed the problems to include the pre-trained generative network? Is there a bias that could change the visualizations even with the same feature extractor? It seems to me that you are including a new possible bias in the pipeline.\n    \n    As I understand it, the concept functions are learned along with the concept translator, but in doing so, the latent space of the generator affects the learning concept functions. Can this mean that the bias from both networks is present in these concepts? Are we explaining the feature extractor or the generative model?\n\n- The number of concepts and their semantics: How was the size of the concept dictionary chosen? What happens with smaller K? You show some ablation studies, and then you decide on 64. What about smaller numbers of concepts? It would be interesting to see how the visualizations change depending on that number.\n\n- Faithfulness and consistency: In the paper, the authors mention these evaluations as new, but faithfulness is commonly used as a technique of evaluation [1]. I also understand the approach used as feature (concept) removal. A suggestion to the authors is to also evaluate the effect of inserting only the top concepts into the generated image. Moreover, an interesting evaluation would be the one proposed in the paper of XRAI [2], to iteratively insert the top concepts and analyze the AUC of the accuracy curve. Regarding the consistency, I like this idea, I just have questions about the classifier used to separate the representations with and without the concepts, what is its architecture? Is it linear before decision? How many layers? Also, I would suggest an additional experiment: use a part label dataset like CUB-200 with bird parts and check if the most changed parts when removing a concept are the same for birds in the same class.\n\n- Sparsity and Viewability: I don't see much discussion of these metrics other than the loss used during training. I would like to see at least some viewability analysis (qualitative) and possibly a human evaluation compared to the baseline approaches.\n\n[1] Covert, I., Lundberg, S., & Lee, S. I. (2021). Explaining by  removing: A unified framework for model explanation. Journal of Machine  Learning Research, 22(209), 1-90.\n\n[2] Kapishnikov A., Bolukbasi T., Vi'egas F., Terry M. XRAI: Better Attributions Through Regions. ICCV. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)