DISCOVER: Making Vision Networks Interpretable via Competition and Dissection
Authors: Konstantinos P. Panousis
Sotirios Chatzis
Venue: NeurIPS 2023 poster (2023)
Keywords: Interpretability
Explainability
Network Dissection
Competitive Networks
Sparsity
Multimodal Models
TL;DR: We propose a principled design framework for automatic description of neurons of Vision Networks leveraging competition-based activations with high sparsity, multimodal models and Network Dissection.

Abstract: Modern deep networks are highly complex and their inferential outcome very hard to interpret. This is a serious obstacle to their transparent deployment in safety-critical or bias-aware applications. This work contributes to *post-hoc* interpretability, and specifically Network Dissection. Our goal is to present a framework that makes it easier to *discover* the individual functionality of each neuron in a network trained on a vision task; discovery is performed in terms of textual description generation. To achieve this objective, we leverage: (i) recent advances in multimodal vision-text models and (ii) network layers founded upon the novel concept of stochastic local competition between linear units. In this setting, only a *small subset* of layer neurons are activated *for a given input*, leading to extremely high activation sparsity (as low as only $\approx 4\%$). Crucially, our proposed method infers (sparse) neuron activation patterns that enables the neurons to activate/specialize to inputs with specific characteristics, diversifying their individual functionality. This capacity of our method supercharges the potential of dissection processes: human understandable descriptions are generated only for the very few active neurons, thus facilitating the direct investigation of the network's decision process. As we experimentally show, our approach: (i) yields Vision Networks that retain or improve classification performance, and (ii) realizes a principled framework for text-based description and examination of the generated neuronal representations.

DISCOVER:
Making Vision Networks Interpretable via Competition and Dissection
Konstantinos Panousis
Department of Electrical Eng.. Computer Eng., and Informatics
Cyprus University of Technology
Limassol 3036, Cyprus
k. pwwossis@uut.ccmmmmmmm www.soouttac.....
Sotirios Chatzis
Department of Electrical Eng.. Computer Eng., and Informatics
Cyprus University of Technology
Limassol 3036, Cyprus
sotirios. ctt.llosschttzisscccecccc ac.
Abstract
Modern deep networks are highly complex and their inferential outcome very hard to interpret. This is serious obstacle to their transparent deployment in safety- critical or bias-aware applications. This work contributes to post-hoc interpretabil- ity, and specifically Network Dissection. Our goal is to present framework that makes it easier discover the individual functionality of each neuron in a network trained on vision task; discovery is performed in terms of textual description generation. To achieve this objective, we leverage: (i) recent advances in multi- modal vision-text models and (ii) network layers founded upon the novel concept of stochastic local competition between linear units. this setting, only a small subset of layer neurons are activated for given input, leading to extremely high activation sparsity (as low as only x 4%). Crucially, our proposed method infers (sparse) neuron activation patterns that enables the neurons to activate/speciaalie to inputs with specific characteristics, diversifying their individual functionality. This capacity of our method supercharges the potential of dissection processes: human understandable descriptions are generated only for the very few active neu- rons, thus facilitating the direct investigation of the network's decision process. As we experimentally show, our approach: (i) yields Vision Networks that retain improve classification performance, and (ii) realizes principled framework for text-based description and examination of the generated neuronal representations.
Introduction
In recent years, Deep Neural Networks (DNNs) have exhibited an overwhelming success in a variety of tasks and applications. achieving state-of-the-a thett-rrrrt results in Machine Vision. Automatic Speech Recognition, and NLP. This unprecedented success is however accompanied with limited capacity to audit them in view of reliability standards. Indeed, due to the immense complexity of their architectures, DNNs are usually employed as off-the-shelf solutions for a variety of applications. However, their vast and highly accelerated adoption rate emphasizes the importance of being able to explain how DNNs function and interpret how predictions are made. This process will allow for
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

uncovering their inherent biases and even for correcting the failures in their decision making process, facilitating a safe, trustworthy and robust real-world deployment in safety-critical applications.
Fortunately, this limitation has recently received a lot of attention in the Deep Learning (DL. re- search community, leading to the development of ways forward towards Interpretable Deep Net- works. In this context, we can identify two major approaches: ante- and post-hoc methods. In the former, the main rationale is to build networks from scratch, that integrate intepretability components into the network itself; Concept Bottleneck Models (CBMs) are a characteristic case [II 17,, 19, 25J. In post-hoc approaches. backbone architectures are augmented with novel frameworks aiming to provide explanations of the prediction process, e.g. saliency maps [055, [r] and Network Dissection [2, 9). Each approach exhibits its own advantages and drawbacks. For example, ante-hoc models usually suffer from significant performance drop, often requiring additional data to train. On the other hand, even though post-hoc models avoid the complexity of re-training, the results may lend themselves to subjective interpretations that depend on the method's assumptions.
This work takes novel route: aims improving the efficacy of post-hoc methods by addressing some fundamental building blocks of Deep Networks. Specifically, we depart from the commonly used forms of non-linearities in Deep Networks and propose the use of competition-baset activa- tions imposed upon otherwise linear projection units; these form the so-called Stochastic Local Winner-Takes-
(LWTA) layers [22]] Our inspiration stems from biological arguments: it has been shown that brain neurons with similar functions aggregate together in blocks and locally com- pete for their activation; the winner gets to pass its response other neurons outside the block while the rest are inhibited to silence [30, III 14]. Stochastic LWTA activations have recently been em- ployed on DNNs with striking success, exhibiting: (i) significant compression capabilities [22], (ii) unique representation diversification ability [23]], (iii) strong adversarial robustness against powerful adversarial attacks [241], (iv) the best ever reported performance in model-agnostic meta-learning (MAML), in terms of both accuracy and network footprint [[]]. Other significant properties of LWTA-based networks include noise suppression, specialization, automatic gain control and robust- ness to catastrophic forgetting [22, R. G]. while their prowess has also been shown in Transformer networks dealing with end-to-end to-end translation of sign-language video into text [32]
In building interpretable networks, the proposed competition- based rationale comes with two impor- tant benefits: (i) naturally arising activation sparsity: only the winner unit each block passes its computed output to the next layer, while the rest pass zero values; and (ii) a data-driven pattern of neuron functionality, based on the probability of a unit being the active winner for given input.
This mode of operation naturally lends itself to the post-hoc paradigm of unraveling and inspect- ing individual neuron functionality, which this work addresses. In this setting, recent works on automated methods aim to provide high quality descriptions of neurons by leveraging multi-modal models to match neuron functionality to text-based unrestricted concepts.
We leverage these ideas to address the interpretation limitations of conventional architectures, by introducing the Dissection of Competitive Vision Networks (DISCOVER) approach, a novel frame- work towards interpretable Vision Networks. Our contributions can be summarized as follows:
We construct, implement and train variety of Competitive Vision Networks (CVNs) uti- lizing novel competition arguments, including both CNN and Transformer- based architec- tures. This is the first time in the literature that CVNs trained on large scale and their behavior is investigated with respect to both: (i) performance, and (ii) interpretation properties.
We perform post-hoc network dissection using multimodal models to identify the under- lying functionality and contribution of each individual neuron; to this end, leverage the unique characteristics of competition between neurons and novel Network Dissection approaches.
We examine the specialization properties of CVNs in the context of interpretability by investigating the generated per-example neuronal representations.
2 Background
Image-Text Models. Contrastive Language-Image Pre-training (CLIP) [26] constitutes one of the best-known methods for multimodal deep learning, combining visual representations with natural
2

---

language supervision. The main operating principle of CLIP relies on the usage of image-caption pairs in order obtain representations. Specifically., CLIP comprises an image encoder, denoted by (-) and a text encoder (-) that are trained simultaneously on the grounds of contrastive learning objective HJ. Specifically, assuming batch of N (image, text) pairs, i.e. (.).,4 CLIP learns a multi-modal embedding space by maximizing the cosine similarity of the embeddings 1, = E (x²) and T; = ET (tz), between the N correct pairs, while minimizing the cosine similarity between the other N2 - N combinations. During inference, we can use natural language supervision to perform zero-shot classification of any given image and labels. Thus, given a test image x=Sa and some labels ft, j=1 we can use the learned encoders and select the image-labe pair that has the highest similarity of all the possible combinations.
Network Dissection. Network Dissection constitutes one of the most popular approaches for un- derstanding and describing the contribution of individual deep network units in the produced infer- ential outcomes. In [Z], method that yields descriptive outcomes in terms of concepts is presented. To this end, the authors introduce a newly developed densely-labeled dataset, named Broden; this contains pre-determined concepts, images and their associated pixel-level labels. The main principle is to use the Intersection over Union (IoU) score as measure of relevance between each concept and neuron in the set. However, the proposed method requires the use of dense labels for training, while at the same time the concepts are restricted to the pre-defined label set.
This limitation inspired the development of MILAN [У], automated method that addresses the notion of pre-defined concepts through the generation of unrestricted description via generative image-to-text models. Nevertheless, the requirement for human annotation is still present, simi- lar to Network Dissection [2]] CLIP-Dissect (20] bypasses this restriction by leveraging multimodal image-text models, and specifically CLIP; this allows for decoupling the dependence between the concept set and the probing dataset. Thus, any network can be dissected, using any text corpus to form the concept set, any image probing dataset, and any appropriate similarity measure for match- ing concepts to neurons.
In this work, we exploit the flexibility of CLIP-Dissect and extend it in the context of Competitive Vision Networks. Differently than CLIP- Dissect, we mainly focus our analysis on the properties of Vision Transformer architectures with respect neuron identification.
Competitive Vision Networks. Recent works have explored a fundamentally different paradigm for the latent unit operation of DNNs, based on the concept of local competition among units (LWTA) [29]] In this setting, latent units compute their response and compete for their activation. The winner passes its (linear) response the next layer, while the rest output zero values. In the following, we will be referring to these networks as Competitive Vision Networks (CVNs).
Let us consider input E R presented to a hidden layer of a DNN comprising K hidden units and a weight matrix W E J- To compute a typical non- non-linear activation, each hidden unit in the layer performs inner product computation with its respective weights w. E IR yielding the intermediate response hgg = w æ E R; this usually passes through a non-linear activation (-)), such that the final layer output reads: y [y1,..., yk], where yv= a (he)) Vk.
In a corresponding LWTA- -bsed layer. U singular units are grouped together forming the so-called LWTA blocks. Hereinafter, we denote by B the number of LWTA blocks in the layer, and with U the number of linear competing units therein. The aggregation operation is manifested via the definition of the layer's weights implies that each input dimension of æ is now presented to each block b and each unit u therein; in turn, all units compute their response via the standard inner-product computation ha w.,, x e R, Vb, and competition takes place among them. The fundamental principle of competition is that out of the U competing units, only one can be the winner. This unit gets to convey its activation outside the block, i.c. the next layer, while the rest output zero values. This process is instantiated via an appropriate competition function that encodes the outcome of the competition in each block.
matrix W J this structural modification
Contrary to the conventional definition, where the layer's output arises as the concatenation of the individual response of each unit, in the LWTA-based framework, the final output y BB is now con- structed from B sub-vectors, one for each LWTA block that contains single non-zero entry; this corresponds to the response of the winner unit in said block. Thus, an inherent property of the LWTA mechanism is the naturally emerging sparse representations. Indeed, considering fixed number of
3

---

Layer Exput
8
Block Ouipui
lnner Product
Mil Bu Iug
Diccess
Bup Aast
Categerical Canngerioo/Soottteeeo (Softeran (.
Figure I: Detailed bisection of the 7th Stochastic LWTA block. Presented with an input æ E R each unit u computes its activation ha. via different weights Wh,u E R. i.e., ha, w T. The linear responses are concatenated, s.t., h2 = [ ...,, h. and transformed into probabilities via the softmax operation. Then, a Discrete sample E, = . f6,u vector with a single non-zero entry at position u', denoting the winner unit in the block; unit u', passes its linear response to the next layer, while the rest pass zero values. Image from [23]]
is drawn, which is a one-hot
units per layer, we can observe that the higher the number of competitors U, the sparser the layer output: when U 2, only 50% of the units will be active per example, when U only 25% and so on. This gives rise to sparse activation mode of operation that has been empirically shown to en- dow deep architectures with significant properties including strong adversarial robustness [24] and substantial representation diversífication [23]. One design choice that we need to address concerns the nature of the competition: deterministic or stochastic. The former is the most typical approach in the LWTA literature; in this case, the unit with the highest activation is deemed the winner each time. The latter is founded upon novel stochastic arguments proposed in [22]]. This formulation has
been shown to consistently outperform its deterministic counterpart [2I], 24 12 [T]. Thus, we adopt stochastic competition our approach.
The stochastic variant of the competition function entails the introduction of an appropriate set of discrete latent variables for each LWTA block & E one hot(U), that encode the (stochastic) outcome of the competition therein. The latent indicators constitute one-hot vectors with U compo- nents; that is, vectors containing single non-zero entry the index position corresponding to the winner unit in the block. The output of stochastic LWTA layer's (b, u)th component yb.u yields:
Yb.u = Eo.ux
Wj.b.a F3 E R
j=1
We postulate that these latent indicators &,, Vb, are drawn from a Categorical distribution driven from the intermediate linear mnner-produs computations that each unit performs. Evidently, the higher the response of a particular unit in a block (relative to the others), the higher its probability of being the winner; however, the final decision remains stochastic. We can formulate this rationale as:
a(E,)) Categorical
(x Softnax E, Mo((r)
Σ EU
vb
(2)
where [x) is vector comprising the activation probability of each neuron in the block, and 10 [w),,,))= denotes the vector concatenation of the set {w Each unit in each block com- putes its intermediate linear inner-produst these are then passed softmax transformation and used to draw samples from a categorical distribution. The resulting discrete vector contains a single non-zero entry denoting the winner of the block. The intermediate computation of the winner unit passes out as its activation; the rest units pass zero-valued activations. A graphical illustration of the stochastic LWTT structure is provided in Fig. II
In the context of attention-based architectures, the encoder of Vision Transformers (ViTs) [ com- prises alternating layers of Multi-head Self-Attentin (MSA), MLP blocks, Layer Normalization

---

(LN) and residual connections. Each MLP block comprises two layers with GELU -on-linear in between. Within the CVN framework, we replace GELU layers with stochastic LWTA layers. This modeling decision is motivated by recent works that have shown that transformer MLPs en- code knowledge attributes [5L IS]. We posit that this formulation facilitates neuron identification and interpretation via analysis of the arising competition patterns; we explore its potency in the following sections.
Further, convolutional operations also constitute an integral part of various architectures. To account for this fact, and for completeness, we also consider the convolutional variant of the stochastic LWTA layer proposed in [[2]]. Due to space constraints, and since we mainly focus our analysis on attention-basee architectures, we provide the formulation in the Supplemental Material.
3 DISCOVER
3.1 Training and Inference Algorithms for CVNs
CVNs comprise additional auxiliary variables denoting the winner each LWTA block; thus, we need to devise appropriate training regime that takes into consideration the stochastic nature of said variables. To this end, we turn to Stochastic Gradient Variational Bayes (SGVB) [02], and construct an Evidence Lower Bound (ELBO) objective. Assuming dataset D (x,, , Yi J== and denoting as f(X;; E) the cross-entropy between the target labels yı and the target probabilities emerging from a CVN, the objective reads:
CVN
Σ ((X:E.(X;;E)) (L(g(E)))p(E)) X..IEED
(3)
where & denotes samples from the (posterior) distributions of all latent variables, E. in all lay- ers, and KL(g(E) [p(E)] denotes the Kullback-Leibler divergence between the posterior and the prior. We assume symmetric Categorical prior for the latent variable indicators &: hence, p(E,)) = Categorical /U), Yb. For all the computations we draw just one sample, but we consider it differentiable expression; this allows for very low variance gradients during training, which ensures convergence (reparameterization trick). Since the Categorical distribution is not amenable differentiation, we enable reparameterization by resorting
continuous relaxation, namely Gumbel-Softmnn (GS) [ IOO.. In contrast to previous works [22 23), we use the Straight-Through variant of GS, which we have seen offer better convergence properties. We present the exact sam- pling procedure in the Supplemental Material. During inference, we directly draw samples from the trained posteriors and determine the winning units in each layer along with their responses (Eq. D)
3.2 Dissection
For dissecting the considered CVNs, we draw inspiration from CLIP-Dissect [20]. We employ a CLIP model identify the functionality of each neuron in stochastic LWTA block in terms of given concept set; this will be matched with a textual description. There are three key components that define the approach: (i) concept set S, where |S| M, (ii) dataset that is used
probe to identify the individual functionality of neurons denoted by Proobe. where [Dprotel = N, and (iii) the network to be investigated, denoted by fz). The main principle is to match neuronal activations with concepts using: (i) the activation matrix P that measures the similarity between images and concepts, and (ii) summary of the individual neuronal activations of the network being probed to the given probe dataset.
Concept Activation Matrix. We use the image and text encoder of the CLIP model, denoted by F, and Fr respectively, and compute the respective embeddings J; and T; for each image 2; and concept f,. This results a concept activation matrix P E XMM where each entry (i,j) denotes the cosine similarity between image 2; and concept ty in the CLIP embedding space.
Record Activations for each Competitive Neuron. Given a neuron and for each image in the probe dataset (X, J. CLIP-Dissect computes the neuron's 's non-linear responses ( (Ax(X ) J and records them into a single vector Qs E R.N. s.t., d = (X), A& (x( N)]- Each acti- vation vector gs essentially encodes the activation of neuron & across the whole probing dataset.
5

---

Concopt
(Ti Embeehing Onlla Imappe Emhhnhiing sim: Similarity Mensime
Gusppt Sct
Wans,
Probe
Aew Aov if Gu olberwion
Inaage Eiciader
S7
ST
Competitive Network Dissection
Tusa Layor
Record Activations
Matching ang mane, sim/ 23,3 sim/
Aud th Aa
Fox Water
A
mone, simites On
Cat Bird
m000, cim/ on
Figure 2: The DISCOVER framework. Given a probe dataset D Proote (Xn)== a concept set S, S| M, and layer to be dissected: compute the concept activation matrix P E R.N using CLIP; then, for each LWTA block b and each neuron therein, compute the responses Ab,a (Xn), Vn, s.t. qb = [Ao (X1)...... (X Each neuron b, u is then matched to the concept whose activation vector P. has the highest similarity (sim) with the neuron's 's activation vector q6.w-
Subsequently, it can be exploited as a data-wide neuron representation to match neurons to textual descriptions describe next. When the response A& (X,,) ,.) is not scalar, summary function that maps it to a real number is employed. For inner-product neurons, the response is already a real-number. For convolutional kernels, we consider mean over the spatial dimensions.
Our approach. Building on these outcomes, CVNs adopt novel regard toward interpretation, which leverages the sparse nature of the representations obtained from stochastic LWTA layers; here, we focus on attention-based architectures. In this setting, the MI.P blocks contain patch- specific information relating to the tokens of each image; principle, we could introduce a summary function to aggregate the watch-wise information and output real number. However, we have access to another representation: the feature embedding pertaining to the class token; this comprises a set of units dedicated to capturing both global and local contextual information in each layer.
Within the framework of CVNs, this is treated in local competition fashion: CVN units on each MLP are grouped together in blocks of competitors and compete for the output of the block. Contrary the singular recording of activations in the base formulation for each neuron k. { (A&(X (Ax,),)) xx)] the activations are computed for each LWTA block b and unit therein, yielding {A (X,) ] These activations can be negative, positive or exactly zero, depending not only on the linear compu- tation of the unit in consideration, but also the responses of the remaining units in its LWTA block, following the form of El We concatenate them to construct a summary for each competitive unit, (X))
such that 4. (A, (X i)...... Ag.u(
Due to the nature of the proposed stochastic local competition, the emerging summary vectors will be sparse; only the winner of each LWTA block will retain its linear computation and pass it as a non- zero activation for each example X,; the rest units in the block pass zero activations. This mode of operation not only encourages specialization, but can also diversify individual neuron identification; we explore its potency in the next section.
Matching Neurons to Concepts. Having recorded the activations vectors for each LWTA block b and each neuron u therein, Io.. E N we aim to discover the most similar concept {4m m=1 to describe each neuron. To achieve this matching, similarity function sim(Im Io.u P) is defined; this is used to quantify the relation (in terms of similarity) between the neuron's 8 activation vector

---

4b.u E R and the concept's 's m activation vector, P... E ER., For example, considering the cosine similarity, we compute:
sim(tm, Lb,ui P) P. P. Lb,
(4)
Thus, for assigning concept to neuron b, u, we compute its similarity to each concept in the set and select the one that exhibits the highest value, s.t., t = arg max, sim Ib.u P). Characteristic similarity functions include, Rank Reorder, WPMI and SoftWPMI [20].
At this point, it is important to highlight principal benefit of CVNs: After neuron identification, i.e., matching neurons concepts, and since only small subset of neurons is active for each example, it becomes practically tractable to perform per-example analysis on that particular small subset of "winner" (active) neurons; this greatly facilitates practical concept interrogation.
4 Experimental Analysis
For evaluating and dissecting the proposed CVNs, we train two sets of models: (i) Transfomer- and (ii) Convolutional architectures. We consider stochastic LWTA layers with different numbers of competitors, ranging from U = 2 to & = 24. In every architecture, we retain the total number of parameters of the conventional model by splitting a layer comprising K singular neurons B blocks of U competing neurons, such that B U = K. This choice facilitates fair -iizewsse- comparison between an original network and its CVN counterpart. The number of competitors has a direct effect on the per example neuron activation in the respective layers. For example, when U = 2, only 50% of neurons are activated for given input, when U only 12.5% and so on.
For the Transformer architecture, we select the DeiT model, specifically DeiT-Tiny (DeiT-T, 5M pa- rameters) and DeiT-Small (DeiT-S, 22M parameters), which we train from scratch on ImageNet- Ik. For the convolutional paradigm, we chose the same network as in CLIP-Dissect [20] for compara- bility; that is ResNet-18 18 trained on Places365. In all cases, we follow the original training schemes concerning the hyperparameters, and remove the excessive augmentations of DeiT, i.c., DropPath, Color Jittering, Random Erase, CutMix and MixUp; we have found that these do not improve ac- curacy of CVNs. For completeness, all hyperparameter settings for each model are provided in the Supplementary Material. All models were trained on a single NVIDIA A6000 GPU. Our code implementation is available at: https: // thub. comm/ooppnousis//Dnaaaaaa DISCOVER
Accuracy. We begin our evaluation with classification results pertaining CVNs and how block size (number of competitors in each block) affects performance. This is essential, since CVNs have not been explored in the literature, especially when dealing with larger datasets and architectures, and no baselines exist for the considered setup. However, it is highly important to note that this work is not focused on achieving state-of-the of-te-ot- performance for vision tasks using CVNs. Instead, our focus is on providing fundamental component towards a novel Network Dissection paradigm. Thus, we focus on Network Dissection efficacy, as opposed to optimally tuning hyperparameters and augmentation to increase accuracy by few points.
The obtained comparative results are presented in Table II Therein, we observe that despite the fact that we did not aim for performance improvements, CVNs exhibit near or even superior performance compared to their conventional counterparts. This finding persists even when using a large number of competitors, which directly translates to high activation sparsity. Let us consider for example DeiT-T/16; this model comprises LWTA blocks with 16 competing units each, leading to 1/16 6.25% active (winning) neurons in each layer. We observe that this network exhibits only 1% drop in prediction accuracy (again without any tuning). On the other hand, when using a smaller number of competitors, e.g., DeiT-T/2 or DeiT-T/8, we obtain better and on par performance to the conventional architecture, respectively. Also note that, even though there is a minor computational overhead during training (up to 10%), the computational costs of inference accuracy-wise CVNs provide a realistic alternative to conventional Vision Networks that can even yield accuracy improvements when trained with the same (potentially suboptimal) setup.
comparable. Thus,
Quantitative Analysis. We follow the novel proposal of [20]] to compare our CVN-based approach to conventional non-competitive networks. In this context, we can compare the generated neuron labels, for a specific layer of a network, with the ground truth descriptions, i.e., the class labels. Specifically, we measure the cosine similarity in a text embedding space between the ground truth
7

---

Table 1:
Classification results for ImageNet and Places365. /None denotes a non-competitive archi-
tecture, e.g., using ReLU/GELU, while numbers denote the number of competitors U.
(**)] Looally reproduced results due to unavailability of pretrained models/resource limitations.
ImageNet-Ik
Model/Competitors
Accuracy + Std (%)
Active Neurons Proportion
DeiT-T/None
72.2 + N/A
Data Specific
DeīT-T/2
72.5 0.10
0.500
DeïT-T/8
71.7 + 0.15
0.125
DeîT-T/16
71.1 + 0.25
0.062
DeiT-T/24
70.5 0.45
0.041
DeiT-S/None
T..+=N/A"
Data Specific
DeřT-S/2
77.3 + 0.30
0.500
DeîT-S/12
77.0 + 0.25
0.083
DeiT-S/16
76.7 + 0.50
0.062
Places365
DesNet-18/Non
52.25 N/A"
Data Specific
ResNet-188 18/2
53.9 + 0.20
0,500
ResNet-
51.0 + 0.50
0.250
ResNet- 18/8
49.5 + 0.75
0.125
class name and the description of the neuron. We use two different text encoders: (i) CLIP ViT-B/16, denoted as CLIP cos and (ii) all-mpnet-base-w2 denoted mpnet cos. Here, we focus on the last layer of the MLP block of the DeïT-T model. The obtained comparative results are depicted in Table 2 Therein, we observe that the CVN consistently outperforms its non-competitive counterpart when using both the best cest-performing softWPMI and the Cubed Cosine Similarity function proposed in [20]
Table 2: Cosine similarity between the last layer neuron descriptions and ground truth labels on DeiT- T/None and DeiT-T/16 trained on ImageNet. For the former we SoftWPMI for identification, while for CVNs, PM and the cubed cosine similairity measure [20]. We use two text encoders: (i) CLIP ViT-B/16 and (ii) all-mpnet-hase- denoted mpnet cos.
Vision Network
peobe
Concept Set
DeiT-T/NNone (SPI
DeiT- (Sol
DeiT- T/16 (CosSinCubeed)
CLJP cns
mpnet
CLIIP cos
mpnet
CLJIP cos
mpnnet
Net Val
Broden
0.6250
0,1800
0.6460
0.1829
0.6543
0,1918
3%
0.6226
0.1688
0.6650
0.1895
0.6699
0,1842
10k
0.6187
D.1715
0.6553
0.1878
0.6616
0.1831
0.6128
0.1758
0.6455
0.1929
0.6519
0. 1.18
ImageNet
0.5850
0.1782
0.5854
0.1842
0.5840
0,1835
CIFARI00 Train
20k
0,6392
0.1765
0 05633
0.1969
0.6530
0,1876
Broden
20k
0.6338
0,1780
0.6470
0.1815
U. 6666
0,1546
ImageNet Val & Broden
20k
0.6299
0.1821
0.6333
0.1876
0.6729
0.1433
To further assess the quality of the obtained neuronal deseriptions, we turn to the neuron identifica- tion accuracy metric proposed in [20]. In this context, and in situations where the considered concept set contains exact class labels, we compute the percentage of neurons that have been assigned the exact correct label. Evidently, this metric pertains to the classification layer of considered network. We consider four datasets: CIFAR-100 Train, Broden, ImageNet Val and ImageNet Val & Broden; the ImageNet classes as the concept set S. For matching neurons concepts, WPMI, since it outperformed all other similarity functions. Further results with the other similarity metrics are provided in the Supplemental Material.
Soft-
The comparative results for all conventional and competitive configurations are presented in Table E We observe that on average, CVNs consistently outperform their conventional ReLU/GELU based counterparts, with up to 2% improvement. It is striking, that DeiT-S/12 improves accuracy on CIFAR-100 Train by a staggering 10% compared to the conventional ResNet-50 that has the same number of parameters; at the same time, DeiT-T/8, also provided significant improvement of s 7%.
8

---

Table Accuracy as the percentage of neurons assigned the correct label, i.e., class name. Concept
Set S: ImageNet, Similarity: SoILWPMI, ResNet- results from [20].
Vision Network
D probe
ResNet-50/None
DerT-T/None
DeiT-T78
Derl-T716
DeiT-S/2
DeiT-S/12
CIFAR-IO0 Train
46.20%
53.00%
53.10%
50.80%
55.20%
10%
Broden
70.50%
68.80%
67.40%
67.40%
68.30%
68. 98.10%
ImageNet Val
95.00%
95.00%
96.00%
95.30%
95. 95.60%
95.60%
ImageNet Val Broden
95.40%
95.20%
95.70%
95.50%
95.00%
94.80%
Average
76.78%
78.00%
78.05%
78.30%
78.70%
78.65%
Netzon Chee-D (Bett WPXC:
mortatnng
Nenzon 41:
CGIP Drseett (Soft WPKT)
(Sutt WPMI;
groen
Netzon CL.Pt.Neect
add
Nosson CL.IP-DKecst WPPAI))
Nouron 512:
CT.PP.Neeccl (Sun WPLT):
Nauron 312: CL.PPc (Sull WPLIT)
Figure 3:
Neuron Identification for the first and last DeiT-T/16 MLP blocks using: SoftWPMI, Drrtee ImageNet & Broden, 20K most common English words [20]]
Qualitative Analysis. We now turn qualitative analysis of the obtained neuronal representa- tion. In this context, we visualize several neuron identification results in terms of generated descrip- tions in Fig. 3. To this end, we select random neurons of the first and last MLP blocks of DeiT-T/16 and use combination of the ImageNet and Broden datasets as a probe; the concept set S comprises the 20, 000 most common English words similar to [20]]. Since we do not use the same pretrained backbone, and the considered CVNs were trained from scratch, there isn't 't an exact matching be- Thus, we can not make direct comparisons pertaining to the exact same neuron an illustration basis. Nev- ertheless, we observe that in the context of CVNs we obtain highly accurate descriptions of functionality of the networks' neurons. We observe that the randomly selected neurons activated by semantically similar inputs that may contain only part of the matching concept.
tween the neurons presented therein and the neurons of our networks, enumeration-we
A principal property of CVNs is the per-example sparsity that naturally arises due to the competition mechanism; this can facilitate specialization and accelerate the process of examining the active con- cepts for each test point, greatly enhancing the interpretability of the network. Indeed, we perform this analysis for the last MLP block the GELU-based DeiT-T/None, and its DeiT-T/8 and DeiT- T/16 competitive counterparts. This block comprises 768 neurons; for each example. DeiT-T/8 will activate only 768/8 / 96 neurons and DeiT-T/16 only 48 neurons. The number of activated neurons for the conventional DeiT-T/None will vary according to the probing dataset. For ImageNet Val, we found that on average, 98% of neurons are activated for each example when using DeiT-T/None.
Thus, in Table M. and for each architecture, we present the 7 most relevant concepts tied to neurons with the highest magnitude of activation the considered example, along with 6 concepts tied to neurons with the lowest magnitude. Therein, we observe that the most contributing concepts in the conventional setting contain descriptions that are highly irrelevant or even contradicting the presented input as "red/orange beak & legs", "black coat/tan markings" and "reddish brown fur"; the same time 757/768 neurons are active for this example, rendering the concept investigation a strenuous process. In stark contrast, the considered CVNs exhibit substantially more relevant concepts, while allowing for a straightforward examination of the emerging per-example concepts. Similar qualitative visualizations for different settings can be found in the Supplemental Material.

---

Table Concepts tied activated neurons for random image from the ImageNet validation set for both DeiT-T/None and DeiT-T/16. For the former, /57/ 768 98.56% neurons are activated leading to an arduous interpretation process; DeiT-T/16 only 48/768 6.25% of neurons are active.
DeiT-T-None
DeiT-T18
DeiT-T/16
Concepts
Act
Concepts
Act
Art
Concepts
white under tail
+3.00
white under tail
+10.5
Buffy/whitc oppearnne
+11.6
rediorange beak & legs
+2.53
cucumber plant
+9.32
largentutty white dog
+9.99
white plumage black coatiun markings
+2.11
fox
+8.00
langellutly white dog
+7.71
+22.4
palsigray or cream fur
+8.36
egret
+7.37
largoiflufty white dog
1-1.1
white plamage
+7.35
whildleasm colourad cost
+7.07
molliish//lowwn for
+.1.0
whitelgray plumage
17:27
bluclrra body
+6.85
sbagzy/gna cont
cabbage like leaves
+7.04
white
+5.06
smallblack
largeffulty white
shargy/blick
wolf-like appearance
displays
handage
hugelllegnn
ogg-layin
shorp talond/beok
lanooiutiy
Limitations & Conclusions
Limitations. Considering the of the CLIP-Dissect approach for identifying neuron functional- ity in CVNs, this comes with its respective limitations, such as disregarding the spatial information; this, could allow for identifying lower level concepts and patterns. However, the structure of CVNs could potentially accommodate such a ffunctionality via block (high) level and unit (low) level de- scriptions; we leave these explorations for future work. Finally, even though incorporating the com- petition mechanism to Vision Networks is quite straightforward task, they aren't 't still implemented an optimized way popular frameworks, due to their limited adoption.
Conclusions. In this work, introduced DISCOVER, novel framework towards Interpretable vision architectures through Competitive Vision Networks and Network Dissection. For the first time in the literature, we trained LWTA networks using sufficiently large datasets and architectures. Despite the fact that we did not perform any hyperparameter tuning, the resulting networks yielded on par
improved performance, even when using up to 4% of active neurons per example. Our qualitative and quantitative results vouch for the efficacy and interpretation capabilities of our approach. We yielded highly interpretable neuron functionality, while using a small and explorable subset of all the available neurons per datapoint.
Acknowledgements
This work has received funding from the European Union's 's Horizon 2020 research and innovation program under grant agreement No 872139, project ail).
References
[1] Andersen, Gary N Gross, T Lomo, and Ola Sveen. Participation of inhibitory and excita- tory interneurones in the control of hippocampal cortical output. In UCLA forum in medical sciences, 1969.
[2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proc. CVPR, 2017.
[3] Gail A. Carpenter and Stephen Grossberg. The art of adaptive pattern recognition by self- organizing neural network. Computer, 1988.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple frame- work for contrastive learning of visual representations. In Proc. ICML, 2020.
[5] Damai Dai, Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proc. ACL, 2022.
10

---

[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. ICLR, 2021.
[7] Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy, Transformer feed-forward layers are key-value memories. In EMNLP. 2021.
[8] Stephen Grossberg. Contour enhancement, short term memory, and constancies in reverberat- ing neural networks. In Studies of mind and brain. Springer, 1982.
(9] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In Proc. ICLR, 2022.
[10] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumbel-softmant In Proc. ICLR, 2017.
[1I] Konstantinos Kalais and Sotirios Chatzis. Stochastic deep networks with linear competing units for model-agnostic meta-learning. In Proc. ICML, 2022.
[12] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Proc. ICLR, 2014.
[13] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Proc. ICML, 2020.
[14] Anders Lansner. Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations. Trends in neurosciences, 2009.
[15] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proc. NIPS, 2017.
[16] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proc. ICLR, 2017.
[17] Diego Marcos, Ruth Fong. Sylvain Lobry, Rémi Flamary, Nicolas Courty, and Devis Tuia. Contextual semantic interpretability. In Proc. ACCV, 2020.
[18] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. In Proc. ICLR, 2023.
[19] Tuomas Oikarinen, Subhro Das. Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In Proc. ICLR, 2023.
[20] Tuomas Oikarinen and Tsui-Wei Wei Weng. CLIP-dissect: Automatic description of neuron repre- sentations in deep vision networks. In Proc. ICLR, 2023.
[21] Konstantinos Panousis, Sotirios Chatzis, Antonios Alexos, and Sergios Theodoridis. Local competition and stochasticity for adversarial robustness in deep learning. In Proc. AISTATS, 2021.
[22] Konstantinos Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric Bayesian deep networks with local competition. In Proc. ICML., 2019.
[23] Konstantinos Panousis, Anastasios Antoniadis, and Sotirios Chatzis. Competing mutual information constraints with stochastic competition-based activations for learning diversified representations. In Proc. AAAI, 2022.
[24] Konstantinos P Panousis and Sotirios Chatzis. Stochastic local winner-takes-all networks en- able profound adversarial robustness. In NIPS Bayesian Deep Learning (BDL) Workshop, 2022.
[25] Konstantinos Panousis, Dino Ienco, and Diego Marcos. Sparse linear concept discovery models. In Proc. ICCCW CLVL, 2023.

---

[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proc. ICMI., 2021.
[27] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explain- ing the predictions of any classifier. In Proc. SIGKDD, 2016.
[28] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proc. NIPS, 2016.
[29] Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Jürgen Schmidhuber. Compete to compute. Proc. NIPS, 2013.
[30] Costas Stefanis. Interneuronal mechanisms in the cortex. In UCLA forum in medical sciences, 1969.
[31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou, Training data-efficient image transformers & distillation through attention. In Proc. ICML, 2021.
[32] Andreas Voskou, Konstantinos P. Panousis, Dimitrios Kosmopoulos, Dimitris N. Metaxas, and Sotirios Chatzis. Stochastic transformer networks with linear competing units: Application to end-to-end si translation. In Proc. ICCV, 2021.
12

---

A Supplemental Material
A.1 Experimental Details
Integrating the competition mechanism in modern architectures constitutes a pretty straightforward task. One just needs replace the existing calls to the conventional non-linear activations with a call to a module/function implementing the LWTA rationale. This can be performed in-place without any other changes necessary. Thus, in the accompanied code, we can directly change the definitions of the Transformer or ResNet based models and train the model in the same manner as in the conventional Vision Networks. In this context, for the DeiT architectures, we alter the definitions in the timm library, while for ResNet-based architectures, we slightly alter the example implementation found in the official Pytorch repository For dissecting CVNs, we use the official repository of ELIP-Dissed [20]]. We describe all changes in the respective README file. Our code and trained models will be public after publication.
Transformers. For training the CVN counterparts of the DeîT-T and DeïT-S models, we use the of- ficial implementation. P We train both architectures from scratch using ImageNet- Ik for 300 epochs with the default parameters found therein. Specifically, we use a 5-epoch warm-up period, starting with an initial learning rate of 10-6 following a cosine annealing schedule up 5- 5-- We use the same Adam W optimizer and changed the used weight decay from 0.05 to 0.02 since we found that hurt performance. We re-run the conventional GELU-based architectures with this selection and observed no change in the obtained accuracy. In contrast, we turned off the excessive augmentation setup, since it hurt the performance of CVNs. this context, and in line with the initial ablation study presented in [3I]] we found that performance deteriorates when augmentations are removed in GELU-based networks. For training, we use single sample for the Monte Carlo sampling es- timation for the Evidence Lower Bound loss, described in the main text. For this, we turn to the continuous relaxation of the categorical distribution that allows for reparameterized samples and we describe in the next section. During inference, we draw 4 samples,
low variance estimation, average the logits in the Bayesian Averaging sense; we then compute the predicted loss and accuracy. We did not observe any improvement when drawing more samples.
ResNet. For training the ResNet- 18 model, we used the ResNet ImageNet example PyTorch script and adapted the data loading for Places 365. We train the model for 90 epochs, using SGD with an initial learning rate of 0.1 that is reduced by factor of 0.1 every 30 epochs, a weight decay of 10-4 and 0.9 momentum. The batch size was set to 256. For the Gumbel-Softmax trick, we set the temperature to 0.67 and used the Straight-Througe
estimator. During training, we only draw one sample for the reparameterization trick, while during inference we draw 4 samples from the trained posterior. We trained both conventional and CVN architectures since the official pretrained models were not available online. For both methods, we used the standard RandomResizeddrop and RandomHorizontal Flip augmentations.
A.2 The Gumbel-Softmax Trick
In our work, we perform Monte-Carlo sampling using a single reparameterized sample for each of the corresponding latent variables. These are obtained via the reparameterization trick of the continuous relaxation of the Categorical distribution [[ [[]] as described next. We focus on the reparameterization trick for the dense case; the convolutional case is analogous.
As a reminder, the latent indicators Eo. Yb, are drawn from a Categorical distribution driven from the intermediate linear inner-produce computations that each unit performs g(E) (Eq. (2) in the main text):
((E)) = Categorical
Eb (b(ar)))) Ab, I&(x)) = Softmax
Σ [w.. 2z
(5)
where (x) is a vector comprising the activation probability of each neuron in the block, and (. | denotes the vector concatenation of the set [W),,,. /==1-
10
13

---

Then, the samples & are expressed as:
Eo.. Softmax (log [o]))] + gb,u)/7), 4b 1,..., B, -u......!
(6)
where Ib.u log(- log Vo,a), Vo,u Uniform 1). and (0, o0) is temperature factor, controlling how "closely" the continuous relaxation approximates the Categorical distribution. In this work, we use temperature of 0.67 as suggested in [[ and used in several other works [22, 21I
A.3
Convolutional Formulation
In this setting, local competition is performed among feature maps on position-wise basis. Each kernel treated an LWTA block with competing feature maps; each layer comprises B kernels. Specifically, each feature map u = 1,..., U in the 8th LWTA block of convolutional LWTA layer computes:
H Wa X E HXL
Competition remains stochastic, and is now implemented on position-wise basis as follows:
(( Categorical (Ea.c.r ( 4h',"' 1
(8)
where T ( (X) = Softmax (HH I. Ho.u,.r)) comprises the position-wise activation probubilities for all neurons in the block.
For each position in kernel, only the feature map that wins the said position contains non-zero entry. This yields sparse feature maps with mutually exclusive activated positions. Now, the output Y g obtained via concatenation of the sub-tensors Y b, that read:
W b,u * x): vb, u
H.LL where Eb,u b/ P- The corresponding detailed bisection of convolutional stochastic LWTA block can be found in the Appendix.
In the following, we use these definitions to construct Competitive Vision Networks by replacing the usually employed non-lincarities with the competition mechanism in each hidden layer of the considered architecture.
g LWTA Convolational Block
Xi
Camperer Comgattnl Jobne JM MM
Figure 4: A detailed bisection of the 6th convolutional stochastic LWTA block. Presented with input X l LxCC competition takes place among feature maps on a position-specific basis. Only the winner feature map contains a non-zero entry in a specific position. This leads to sparse feature maps, each comprising uniquely position-wis activated pixels. Image from [23]]
B Further Qualitative Analysis
In this section, provide further qualitative neuron identification results for various architectures and similarity functions.
14

---

Netzon UG. CLIP Stacet (Sut WPSIE
Nereon
CLIP Stneet (Su WPAT
Neuscu 365. Clece (Sul WPAL::
gaan
Nescon 32
CIuIP: (Son WPPLT))
Nowci (Sill пPКT)
pallen
CTTIP- WPAI))
Nearon
Figure 5: Neuron Identification for the first and last DeiT-T/8 MLP blocks using: SoftWPMI, D probo: ImageNet & Broden, S: 20K most common English words|220].
CLer-Duent Smileryy's strpe
Stnclaans
Netzon CLIP Diseet (Cos Stncluszy"
Nonton
CLIP Dinect (Cos Stndlezie *x
Neuson 753. CLaP-Itecct (Coi Stullusky"s yellow
Nenson 753. CLIP:-Xeect (Coa Simtlacke Ps sudl
Neusen 1081 P.ccc (Cloe Sinilarks" " dick
Neuron 1084: CI.IP:Dbeect (Cls Similackss"
lise
Figure 6: Neuron Identification for the first and last DeiT-S/12 MLP block using the Cosine Sim- ilarity Cubed similarity function [20]. For the former we D pmoee: (i) ImageNet Val & Broden (Left), and for the latter: (ii) ImageNet Val. We use the S: 20 most common English words for both settings.
Necson 227 CLIP Discet (Pank: Recodert: yellor
Neszon 1591 CLIP Dinect (Pank Recedes]; fect
(Pank Recodae):
CLiP--Xuest
Roceda];
Jeatco
Nauson CC.P-:ckeeci
Nousan Ct.IP-XXeect
linling
Nacron 100: CT.PP: Teecct (Fank Recebb)::
Noaron 500) CT.IP- (Pank Rececbe]:
Anbandic
Figure 7: Neuron Identification for the first and last DeiT-S/12 MLP block using the Rank Reorder similarity function [20]] We use D p ImageNet Val and S: 20K most common English words for both settings.
15

---

Table Concepts tied activated neurons for random image from the ImageNet validation set for DeiT-T/8 using various similarity functions.
Cos Similarity
WPMI
SoftWPMI
Cancejes
Concepes
Belgiaa
greestycllow eyes
white fur underside
+8.90
white for under tail
+3.300
White for under tail
+8.90
hard Algine thex
+5.51
black stripes on legs
+-8.51
Mack/whitt stripes
+ +.551
lengisharp quills
+5.02
longisharp quills
+-8.01
long/shapp quills
+ +8.01
fun/whhte tips
+7.55
for
+7.55
+7.55
white'sray
Whitelgray head
white/gray head
stock black maakkings
block/tnn coloration
larpe/flufty white dog
+0.35
organ
40.37
organ
+1.37
gills
-00.18
salamander
+0.15
calamnder
+-0.88
cur lowing RV
+-0.12
headsail
+0.112
headsail
+0.12
other croodiiles
+00.12
loying mammal
+0.12
leving manmnal
+0.12
Table 6: Concepts tied to activated neurons for a random image from the ImageNet validation set for DeiT-T/None and DeiT-S/12 using various similarity functions.
(SeiWPMI)
DeIT-S/12
Concejts
tartle
turtle
hardkiiiny exndseleron
+-117
etbe hind feet
turtle
+0.55
brpplelephant animal
dark coleede
largnheed crab
+0.55
hard shell covered
+9
brrplllesh
largefted
cating dinosaur
+0.51
+5.32
toughiscaly
shonbnisly fur
doterman
+9.08
blockhan coloraiion
+9.00
sluped
10.63
SuttyWhite
largentutly
+0.06
other crocodies
10.67
metal plates/trays
-0.16
10.22
opponent
whise markings
-0.16
-40.00
altnched
smallloon like body
esoskiletos
Sshing exeeklatoon
16


Reviews:
[{"review_id": "HiOKuVYPi5", "reviewer": "Reviewer_BNcZ", "summary": "The work proposes a interpreatable network that uses concepts from competition networks specifically there are for a specific output multple copies of the neuron and the neurons compete to provide the winning activation for that output. The method proposes MLP and  convolutional versions of their network. The work also provides evidence of interpreatability by using CLIP-DISSECT and shows that their method is more interpreatable while using same number of parameters as DiET.", "questions": "See the weakness.\n\n1. Gumbel softmax generally requires a bit of finetuning. How stable is the gumbel softmax training on different datasets?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is quite simple and elegant in its motivation\n\n2. Existing modules can be swapped with the proposed block without any changes", "weaknesses": "1. The method is essentially a stochastic version of maxpooling where the deterministic version is exactly maxpooling. Is there an ablation to show how the stochastic version is more appropriate than a deterministic one?\n\n2. Having 2 competetors seems to provide the best performance whereas using more decreases the performance of the network. Is this because of using less \"effective neurons\" (as the model size kept same) or with more competition the network inherently starts degrading? If it's the second case than it does not put competition networks in a good light. \n\n3. It seems wasteful that with more competition exponentially lower number of neurons in the model are activated. \n\n5. Can it be used in networks for other tasks? Segmentation, object detection etc.?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his insightful analysis and crucial comments and questions.\n\n> 1. The method is essentially a stochastic version of maxpooling where the deterministic version is exactly maxpooling. Is there an ablation to show how the stochastic version is more appropriate than a deterministic one?\n\nThere is one pivotal difference between maxpooling and the LWTA mechanism: in the LWTA context we do not just output the max value but instead, and since we do not perform any reduction operation, we retain the positional information of both the active and inactive neurons, yielding *sparse* representations for each layer; this in turn allows for highly different patterns for each example, facilitating learning through specialization. See also the discussion in [1]. \n    \nWe did not perform an ablation study to assess the relative performance between the deterministic and stochastic variants considering the significant computational complexity. However, several prior works (line 137, main text) have consistently exhibited that the stochastic LWTA formulation outperforms its deterministic counterpart in various settings. Moreover, only the stochastic variant allows for  introducing an additional metric for matching neurons to labels, i.e., the proposed Jensen-Shannon divergence. \n\n> 2. Having 2 competitors seems to provide the best performance whereas using more decreases the performance of the network. Is this because of using less \"effective neurons\" (as the model size kept same) or with more competition the network inherently starts degrading? If it's the second case than it does not put competition networks in a good light.\n\nWe thank the reviewer for this question, as it will help clarifying some key points.  The goal of this work was not to achieve state-of-the-art performance for Vision tasks (main text, line 283), but instead provide empirical evidence towards a viable and radically different LWTA-based mode of operation for both Vision Networks and interpetability overall. To this end, we retained the training procedures for conventional ReLU/GeLU-based networks that have been rigorously optimized. We strongly believe that the observed performance difference is not a matter of using ``less effective'' neurons, but that that these procedures are highly sub-optimal for the mode of operation of LWTA; thus, extensive ablation studies and optimization will alleviate this issue (if one aims for classification accuracy as was the case with the baselines). However, this is not feasible in the scope of a single publication. \n\nMoreover, prior works in pruning and sparsity mechanisms have provided substantial empirical evidence of the significant redundancy of components in modern architectures. This fact further supports our claim that is not a problem of using less effective neurons but rather that CVNs have yet to reach their full potential. This work constitutes a first, and in our view very attentive, investigation of the behavior of CVNs on large scale. \n\n> 3. It seems wasteful that with more competition exponentially lower number of neurons in the model are activated.\n\nSimilarly to the previous points, using a lower number of neurons may seem wasteful from a first glance but it also comes with significant benefits in various settings from meta learning to adversarial robustness (see also [1,2]). In the context of interpretability, activating, e.g., $1/16$ of the original neurons per each layer, potentially facilitates the process of neuron specialization: since not all neurons are activated for each example, neurons can specialize to only the features of the examples for which they are active. After Neuron Identification is performed, we can even narrow down the neurons that were activated by a particular test input and try to reason using only these for a downstream task. For example, if we consider a hidden layer with 800 components and $U=16$ competitors, it is easier to reason and investigate the 50 activated components instead of investigating the effect all components in conventional architectures. This can be applied in a variety of settings such as Concept Bottleneck Models and other approaches that we aim to explore in the future.\n\n> 4. Can it be used in networks for other tasks? Segmentation, object detection etc.?\n\nWTA-based activations can in principle be used in any kind of network replacing any of the conventional nonlinearities, in any kind of setting and task. For example, it has recently been explored in the context of end-to-end sign language translation [3]. Intepretability in the context of classification is a significant research and societal challenge, thus we focused on this setting. We hope that this work can facilitate the application of this underexplored but important mechanism to every ML task. \n\n> Q1: Gumbel softmax generally requires a bit of finetuning. How stable is the gumbel softmax training on different datasets?\n\nFor training all our networks we set the temperature of the Gumbel-Softmax relaxation to a fixed $\\tau = 0.67$ value as suggested in [4]. We found this choice to be extremely stable without requiring any additional finetuning. It is however possible that finetuning this value may provide further improvements accuracy-wise, especially when altering the number of competitors (essentially the number of categories of the distribution). We aim to explore training improvements in the near future.\n\n\n[1] Compete to Compute, R. Srivastava, J. Masci, S. Kazerounian, F. Gomez, J. Schmidhuber, NIPS, 2013\n   \n[2] Konstantinos Kalais and Sotirios Chatzis. Stochastic deep networks with linear competing units for model-agnostic meta-learning, ICML, 2022\n\n[3]  Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation, Voskou et al., ICCV, 2021\n\n[4] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proc. ICLR, 2017."}, {"review_id": "I6AhVxZ0lJ", "reviewer": "Reviewer_kXaB", "summary": "The paper proposes DISCOVER, a novel framework for creating interpretable vision networks. It utilizes stochastic Winner-Takes-All layers and multimodal vision-text models to uncover the specialization of each neuron. The framework achieves high activation sparsity, allowing for direct examination of concept representations. It also introduces a new similarity metric for network dissection based on Jensen Shannon Divergence. Experimental results demonstrate improved classification performance and provide a principled framework for text-based descriptions of neuron representations. The paper pioneers the training of LWTA networks on large datasets, yielding interpretable functionality with a small subset of neurons per datapoint.", "questions": "As I mentioned earlier, there is a concern regarding the lack of clear evidence establishing a direct relationship between sparsity and interpretability. While the paper assumes that sparse representations lead to increased interpretability, it would be beneficial to provide results that demonstrate this relationship across different levels of sparsity, utilizing various interpretability measures. By systematically examining the impact of different sparsity levels on interpretability using diverse evaluation metrics, the paper could provide stronger evidence for the claim that sparsity enhances the interpretability of individual units. This analysis would contribute to a deeper understanding of the relationship between sparsity and interpretability and provide more robust support for the proposed framework.", "limitations": "I believe the limitation section could take into account the points I raised in weaknesses, as they were not mentioned.", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The strengths of this paper include:\n\n1. Novel Framework for Network Dissection: The paper introduces a novel framework, DISCOVER, for creating interpretable vision networks. It combines stochastic Winner-Takes-All layers and multimodal vision-text models, providing a unique approach to uncovering neuron specialization.\n\n2. Improved Classification Performance: Despite not performing hyperparameter tuning, the DISCOVER framework achieves comparable classification performance. \n\n3. Interpretable Similarity Metric: The paper introduces a new similarity metric based on Jensen Shannon Divergence to match neuron representations to concepts.\n\n4. Pioneering LWTA Networks: The paper contributes to the literature by being one of the first to train large-scale LWTA networks using substantial datasets and architectures.", "weaknesses": "1. The WTA approach in this neural network framework draws inspiration from previous methods employed before the advent of deep learning. For instance, techniques like bag of words, Locality-constrained Linear Coding (LLC) (Wang et al., \"Locality-constrained linear coding for image classification,\" CVPR, 2010) and a long etc, also involved neuron competition for activation. \n\n2. It would be valuable to have a clearer understanding of the level of sparsity in the network without WTA. While the paper mentions that in a network without WTA sparseness is \"input specific,\" providing some statistical analysis or comparisons of the sparseness level with non-WTA networks could provide insights into the necessity and effectiveness of the WTA mechanism.\n\n3. The paper assumes that sparse representations lead to more interpretable units, but the connection between sparsity across neurons and human interpretable concepts needs further clarification. It would be beneficial to elaborate on the theoretical arguments or underlying assumptions supporting this claim. Additionally, providing empirical evidence or previous research that supports the notion that sparse representations indeed enhance interpretability would strengthen the argument.\n\n4. The qualitative results provided in the paper are limited, particularly regarding the comparison between applying the WTA mechanism and not applying it. It would be insightful to include examples of units that are not interpretable when the WTA mechanism is not utilized. This comparison would help illustrate the added interpretability that the WTA approach brings to the network and provide a more comprehensive understanding of its impact on the model's performance.\n\n5. The dependence on the interpretability similarity metric raises questions about the generalizability of the results. While the paper introduces a new similarity metric based on Jensen Shannon Divergence, it would be valuable to investigate the robustness of the findings using alternative metrics and visualization techniques. Demonstrating the consistency of the results across different evaluation methods would strengthen the validity and generalizability of the proposed framework for increased interpretability. Exploring the impact of alternative metrics and visualization techniques could provide a more comprehensive understanding of the interpretability achieved by the network. \n\nBy addressing these concerns, the paper can further enhance its contributions by providing more insights into the necessity and effectiveness of the WTA mechanism, establishing theoretical justifications, and expanding the qualitative analysis to better understand the interpretability aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his constructive comments on improving the presentation of the strengths of our work.\n\n> Similarity of WTA to Linear Coding (LLC). \n\nThe LWTA approach does indeed bear some similarity with Locality-constrained linear coding relating to the grouping of features and selecting one when using hard VQ. However the construction, learning of the features and applicability of the approaches are highly different. \n\n>  It would be valuable to have a clearer understanding of the level of sparsity in the network without WTA. While the paper mentions that in a network without WTA sparseness is \"input specific,\" providing some statistical analysis or comparisons of the sparseness level with non-WTA networks could provide insights into the necessity and effectiveness of the WTA mechanism. \n\nThis is a very valuable suggestion from the reviewer and will be included in the main text. \n\nWe consider the official pretrained conventional DeiT-T checkpoint with the standard GeLU activation and report the sparsity level: this is obtained by averaging the number of activated neurons after the application of the GeLU nonlinearity in the MLP block of DeiT and considering each neuron as active if its absolute value is greater than a small constant, i.e., $10^{-6}$ to account for rounding errors. For this test, we average over all the test examples of the ImageNet-1k validation set in all layers of DeiT. We observed that $98$% of the neurons are active in this case. Using the exact same implementation for our LWTA-based DeiT-tiny implementation with 16 competitors, yields the expected $6.25$% sparsity.\n\n> The paper assumes that sparse representations lead to more interpretable units, but the connection between sparsity across neurons and human interpretable concepts needs further clarification. \n\nWe thank the reviewer for raising this point as it will help improve on the phrasing of the main text and avoid potential misunderstandings for our claims about the relation between sparsity and interpretability. For this point, we re-iterate the response to Question 3 of reviewer BNcZ:\n\n    In the context of interpretability, activating, e.g., 1/16 of the original neurons per each layer, surely facilitates the process of neuron specialization: since not all neurons are activated for each example, neurons can specialize to only the features of the examples for which they are active. After Neuron Identification is performed, we can even narrow down the neurons that were activated by a particular test input and try to reason using only these for a downstream task. For example, if we consider a hidden layer with 800 components and U=16 competitors, it is easier to reason and investigate the 50 activated components instead of investigating the effect all components in conventional architectures. \n\nWe believe that this argument clarifies our sparsity-interpretability claims and resolves the worries of the Reviewer. In the context of our future work, we aim to extensively analyse and evaluate how this data-driven sparsity mechanism can further facilitate interpretability of modern architectures while considering a variety of different ante and post-hoc methods starting from linear probe methods up to and further than Concept Bottleneck Models.\n\n>  The qualitative results provided in the paper are limited, particularly regarding the comparison between applying the WTA mechanism and not applying it.\n\nConsidering the radically different mode of operation of Competitive Vision Networks and conventional architectures, it is not easy to draw a distinction between the methods on a qualitative basis since there isn't a one-to-one matching. This point is emphasized in the main text in line 305. \n    \nThere are instances where both the conventional networks and the CVNs do not provide the optimal results, e.g., in Fig. 3 Neuron 242; this result was purposely included in the Figure to highlight this fact. This is also case with the original CLIP-DISSECT work: therein, in Fig. 1 Neuron 683, some of the examples that highly activate the neuron have no relationship with the Neuron Identification label. In turn, this is why we considered the quantitative analysis in Table 2 using the metrics introduced in the CLIP-Dissect paper. Using the introduced metrics, it is  quite clear that using the LWTA mechanism yields a higher cosine similarity between the intermediate neuron descriptions and the ground truth labels compared to the conventional method that does not employ the LWTA mechanism; this illustrates the improvement in performance. Devising other quantitative methods for comparing conventional and LWTA-based Vision Networks, especially in the context of Network Dissection where one-to-one direct comparison is rendered difficult, is a substantial research challenge that we aim to nevertheless address in the future. \n\n> The dependence on the interpretability similarity metric raises questions about the generalizability of the results. \n\nThe reviewer is concerned about the generalizability of the results.  However, as mentioned in the main text (and we will further expand in the camera ready), one of the main ``limitations'' of the proposed framework is the dependence on and the comparison restriction using only the CLIP-DISSECT approach. Concerning the visualization techniques issue that the reviewer raises, we believe this was addressed in the previous question. For the evaluation metrics of the results we considered both the best performing metric that the original publication uses (Soft WPMI), that substantially improved upon their other considered metrics, as well as the introduced Jensen Shannon divergence. We validated the improved performance using the cosine similarity that the original publication uses but apart from that, we believe that considering and introducing even more measures and techniques, goes beyond the rational scope of this work."}, {"review_id": "HdsqegBrmZ", "reviewer": "Reviewer_crQf", "summary": "This paper proposes a novel method for interpreting vision networks by combining an architecture that incorporates stochastic local-winner-takes-all activations with the CLIP-DISSECT framework. The author makes use of the probabilistic interpretation of LWTA as a categorical distribution to derive a similarity measure for LWTA activations based on the Jenson Shannon Divergence. They show the benefit of their approach by comparing the discovered interpretations to ground truth labels for conventional and competitive (LWTA) networks.", "questions": "How exactly is LWTA incorporated into the transformer?", "limitations": "The paper does discuss two relevant limitations. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The studied problem of interpretability is very important, the idea is well motivated, the background and related work is informative, and the writing is clear.\nThe accuracy of the LWTA adapted transformer is very encouraging and suggests the usefulness of competition independently of interpretability. \nThe improvements in automated neuron description, are encouraging.", "weaknesses": "\nClarity could be improved: The figures are not very helpful. Especially fig 1 could be improved by simplifying it  (e.g. without showing individual neurons and weights). In my opinion Fig 2 is not necessary and could instead make space for a much needed figure to illustrate the DISCOVER framework.\nThe math feels a bit convoluted with too many new symbols introduced in each paragraph. Again, simplification could help understandability.\nThe paper spends quite time explaining how LWTA and DISCOVER can be used for convolutional networks. However, the main results of the evaluation focus on transformer architectures for which it is less clear how LWTA is incorporated (only replacing the ReLU inside the MLP block or also in the attention and residual branch?). I suggest switching focus and making conv architectures much less prominent.\n\nThe evaluation feels incomplete. The main evidence for the efficacy of DISCOVER are:\n1. the qualitative results in Figure 3, which are only partially convincing. Eg. the \"aquaculture\" neuron seems to react to a highway, and the \"dashboard\" neuron mostly picks up on hifi equipment.\n2. Table 2, which shows that, compared to non-LWTA-networks) the textual embedding of the discovered neuron descriptions is more similar to the corresponding ground truth label. However the CLIP cosine similarity score is hard to interpret and the reported gains seem mild.\nFor a paper that focuses on interpretability, I find the results hard to interpret. As it stands, the paper has me convinced that stochastic LWTA is interesting and should be studied further, but fails to convince me of the added value of DISCOVER over e.g. CLIP-dissect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his kind comments on the usefulness of our approach.\n\n> Clarity could be improved: The figures are not very helpful. Especially fig 1 could be improved by simplifying it (e.g. without showing individual neurons and weights). In my opinion Fig 2 is not necessary and could instead make space for a much needed figure to illustrate the DISCOVER framework. The math feels a bit convoluted with too many new symbols introduced in each paragraph. Again, simplification could help understandability. The paper spends quite time explaining how LWTA and DISCOVER can be used for convolutional networks. However, the main results of the evaluation focus on transformer architectures for which it is less clear how LWTA is incorporated (only replacing the ReLU inside the MLP block or also in the attention and residual branch?). I suggest switching focus and making conv architectures much less prominent.\n\nWe thank the reviewer for his suggestions. Since LWTA-based networks are quite underexplored, we focused on the most used operations, i.e., inner-product and convolutions. Per the reviewer's suggestion we will simplify the figures and notation in the camera-ready and shift the focus to the Transformer architectures. As the reviewer correctly notes, in this work, we replaced the GeLU nonlinearity in the MLP layer of the transformer: this decision was motivated by recent works that have shown that transformer MLPs encode knowledge attributes [1,2,3]. We aim to explore other modifications in transformer architectures in the near future. \n\n[1] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers, ACL, 2022\n    \n[2] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, EMNLP, 2021\n\n[3] Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer, ICLR, 2023\n\n> The qualitative results in Figure 3, which are only partially convincing. Eg. the \"aquaculture\" neuron seems to react to a highway, and the \"dashboard\" neuron mostly picks up on hifi equipment. \n\nPlease see the response to Q4 of Rev. kXaB:\n\n    There are instances where both the conventional networks and the competitive vision networks do not provide the optimal results, e.g., in Fig. 3 Neuron 242; this result was purposely included in the Figure to highlight this fact. This is also case with the original CLIP-DISSECT work: therein, in Fig. 1 Neuron 683, some of the examples that highly activate the neuron have no relationship with the Neuron Identification label. In turn, this is why we considered the quantitative analysis in Table 2 using the metrics introduced in the CLIP-Dissect paper. Using the introduced metrics, it is  quite clear that using the LWTA mechanism yields a higher cosine similarity between the intermediate neuron descriptions and the ground truth labels compared to the conventional method that does not employ the LWTA mechanism; this illustrates the improvement in performance. Devising other quantitative methods for comparing conventional and LWTA-based Vision Networks, especially in the context of Network Dissection where direct comparison is rendered difficult, is a substantial research challenge that we aim to nevertheless address in the future.\n\n In our view, it is not an easy task to construct a best-performing method in all settings and tasks. However, we believe that the LWTA mechanism and its unique mode of operation provide substantial benefits for constructing more interpretable networks.\n\n> Table 2, which shows that, compared to non-LWTA-networks) the textual embedding of the discovered neuron descriptions is more similar to the corresponding ground truth label. However the CLIP cosine similarity score is hard to interpret and the reported gains seem mild. For a paper that focuses on interpretability, I find the results hard to interpret. As it stands, the paper has me convinced that stochastic LWTA is interesting and should be studied further, but fails to convince me of the added value of DISCOVER over e.g. CLIP-dissect.\n\nWe believe that the previous response answers this question. We want to emphasize that CLIP-Dissect is a post-hoc intepretability method that was applied on top of conventional vision architectures. In this work, we explored the potency of Competitive Vision networks by training such networks from scratch in a variety of settings and applying CLIP-Dissect as a post-hoc method. Thus, in our view, it is not a matter of if DISCOVER is better than CLIP-Dissect (since it uses CLIP-Dissect itself) but if the inherent competition mechanism facilitates improvement in post hoc analysis and interepretability of the results. Thus, and given that we are \"limited'\" by the CLIP-Dissect framework, we believe that our investigation has provided strong empirical evidence for the efficacy of CVNs compared to conventional architectures in this setting.\n\n>  How exactly is LWTA incorporated into the transformer?\n\nWe believe the answer in Question 1 of the reviewer answers this question. We focused on altering the GeLU activation in the MLP block of the transformer with the LWTA mechanism."}, {"review_id": "1BdgOxCFKj", "reviewer": "Reviewer_CPpE", "summary": "This paper proposes to use stochastic local winner-takes-all (LWTA) layers in vision networks combined with CLIP-Dissect to improve the ability of a neuron to capture a concept. A stochastic LWTA layer groups pre-activation neurons into $B$ blocks of $U$ features each and within each block, chooses one feature with probability according to the softmax of the pre-activation neurons in the block. The chosen feature is preserved while the rest are zeroed out.\n\nThe paper tests DeiT-T, DeiT-S, and ResNet-18 architectures on ImageNet and Places365 replacing original activation functions with stochastic LWTA. The classification performance slightly improves when each block is of size 2, or $U=2$, and degrades as $U$ increases.\n\nThe stochastic nature of LWTA allows the authors to use CLIP-Dissect with the Jensen-Shannon Divergence (JSD), which they do. The learned concepts for the different investigated neurons, such as “green,” “red,” “trains,” and “snowmobiles” match the images those neurons are activated for. The text embedding similarity of found concepts to ground truth ImageNet label is higher with LWTA neurons and JSD than normal neurons and WPMI.", "questions": "I put the most relevant questions in the “Weaknesses” section.\n- In L228, it is claimed “In the context of inner-product neurons, we directly obtain the presence probability via Eq. (2).” But aren’t vision transformer neurons also spatially arranged, like convolutional neurons? I.e. for a given feature there are different instances of it which corresponding to a spatial map, corresponding to the different embedded patches. So do you also average over spatial positions for transformer?", "limitations": "Limitations were adequately addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The presentation is clear and the motivation in the opening sections is well-expressed. Also, extensive code is provided to support the approach.\n\nThe idea of introducing competition between neurons and having one win out seems intuitive and suited for interpretability, the stated goal of this paper.\n\nThe results are promising (the introduced module does not harm performance too much, and potentially increases performance if a very mild form of it with $U=2$ is used, and some quantitative evidence for superior interpretability is provided).", "weaknesses": "It’s not fully clear whether the approach is being presented as an ante-hoc method or a post-hoc method. It seems one could argue that needing to train a network from scratch with the LWTA nonlinearity makes it an ante-hoc method. It also seems one could argue that since LWTA is a drop-in replacement for any non-linearity, it functions more as an enhancement to any ante-hoc or post-hoc methods applied on top of the architecture rather than an ante-hoc/post-hoc method itself. Could the authors clarify this?\n\nThe evaluation could be more thorough and more convincing. Here are some suggestions, not all of which might be necessary:\n- Significant decline in performance is stated as a drawback of other ante-hoc methods, which this method is arguably an instance of; is it possible to compare classification performance with one or more baseline ante-hoc approaches? \n- Can you provide some sort of comparison between $U$ and the interpretability of the proposed approach? Such as, a plot of $U$ vs. cosine similarity between intermediate neuron description, or comparing concepts found with $U=2$ to $U=16$. Since $U=2$ achieves the best performance, it’s not clear why we would prefer $U=16$. The argument that $U=16$ achieves higher sparsity and thus higher interpretability intuitively makes sense, but is this supported empirically?\n- In my opinion, an additional interpretability metric to evaluate the potential of CVNs for interpretability purposes would improve the paper. Would it make sense to use the IOU-based interpretability metric from the Network Dissection paper? If that would require too much additional experimentation, is some sort of more lightweight added experiment/comparison possible?\n\nMiscellaneous typos/comments:\n- L177: Doesn’t $\\mathrm{CE}(y_i, f(X_i, \\hat\\xi))$ denote the cross entropy, not $f(X_i, \\hat\\xi)$\n- L340: “the resulting networks yielded on par or even improved performance, even when using up to 4% of active neurons per example” seems a bit of a stretch; going from original network to most sparse competition-based network for DeiT-T on ImageNet-1k, the accuracy dropped from 72.2 to 70.5, for ResNet18 on Places365, from 53.5 to 49.5. I would call these drops significant. I would call the drop of DeiT-S on ImageNet-1k from 77.0 to 76.7 “on par”, but not improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his insightful analysis and positive comments for our work. We will correct and update the phrasing on the noted typos/comments for the camera-ready.\n\n> It’s not fully clear whether the approach is being presented as an ante-hoc method or a post-hoc method. \n\nThis constitutes a very interesting point and we'll clarify in the camera ready. Post-hoc interpretability methods are considered for already trained backbone architectures, where a post-hoc analysis method like Neuron Dissection is applied. In the case of LWTA-based networks, there weren't any pretrained models available to apply Network Dissection on. That is why we need it to train them from scratch; after publication, these will be made available for further research investigations. Thus, the proposed framework constitutes an application of a post-hoc interpretability method to CVNs since it is based on the CLIP-Dissect approach that is itself post-hoc; nevertheless it required training since pretrained models were not available.\n\nThe LWTA mechanism is easily applicable to any model architecture, however it can't just be used as drop-in replacement to existing ReLU-trained backbones. For example, replacing the activations in a pretrained ReLU network and directly performing inference will lead to performance degradation. This is natural considering the LWTA unique mode of operation. The model needs to be trained with the competition mechanism to learn the different neuronal activation patterns and how the grouped components compete for their output. In this work, we argue and experimentally validate that this mechanism is indeed a viable alternative to ReLU-based activations, that is accompanied by some significant benefits in term of per-example activation sparsity, and subsequently, intepretability.\n\n> Significant decline in performance is stated as a drawback of other ante-hoc methods, which this method is arguably an instance of; is it possible to compare classification performance with one or more baseline ante-hoc approaches?\n\nWe believe that the answer to the previous question clarifies that our approach is not an ante-hoc method like Concept Bottleneck Models. We are simply training Vision Networks from scratch with the LWTA competition mechanism instead of the conventional non-linearities. Moreover, the obtained performance is comparable or even improves the classification results compared to the baseline conventional networks in some settings. \n\n> Can you provide some sort of comparison between $U$ and the interpretability of the proposed approach? The argument that achieves higher sparsity and thus higher interpretability intuitively makes sense, but is this supported empirically?  \n\n It is true that $U=2$ offers the best-performance accuracy-wise under the same training regime but in the scope of per-example sparsity $U=16$ offers a better alternative. However, this question is related to Question 3 of Rev. BNcZ, where we noted that:\n\n    In the context of interpretability, activating, e.g., 1/16 of the original neurons per each layer, surely facilitates the process of neuron specialization: since not all neurons are activated for each example, neurons can specialize to only the features of the examples for which they are active. After Neuron Identification is performed, we can even narrow down the neurons that were activated by a particular test input and try to reason using only these for a downstream task. For example, if we consider a hidden layer with 800 components and U=16 competitors, it is easier to reason and investigate the 50 activated components instead of investigating the effect all components in conventional architectures.\n\nThe per-example sparsity will facilitate the examination and potentially correction of the networks decisions. In the context of the used similarity metrics, we obtain similar cosine similarity values for different $U$ values. Our next steps comprise the exploration of the activation patterns and interpretation results in various settings. This nevertheless requires extensive investigations that go beyond the scope of this work.\n\n>  In my opinion, an additional interpretability metric to evaluate the potential of CVNs for interpretability purposes would improve the paper. Would it make sense to use the IOU-based interpretability metric from the Network Dissection paper? If that would require too much additional experimentation, is some sort of more lightweight added experiment/comparison possible?\n\nThis is an interesting suggestion from the reviewer. However, considering other interpretability metrics as the IOU one proposed in the Network Dissection paper 1) requires a totally different treatment, and (2) defeats the purpose of using the post-hoc CLIP-DISSECT method. In this work, we selected the CLIP-DISSECT method over the conventional NetworkDissection. That is why even in the original CLIP-DISSECT publication there isn't any comparison on a metric basis using the IOU or other interpretability measures proposed in related works.\n\n> In L228, it is claimed \"In the context of inner-product neurons, we directly obtain the presence probability via Eq. (2).\" But aren’t vision transformer neurons also spatially arranged, like convolutional neurons? I.e. for a given feature there are different instances of it which corresponding to a spatial map, corresponding to the different embedded patches. So do you also average over spatial positions for transformer?\n\nWe thank the reviewer for raising this point. It is something that we wanted to clarify in the main text. For the transformer architecture in particular, we alter the MLP block of the transformer architecture and specifically we replace the used GeLU architecture with the LWTA mechanism. Indeed in this case, we average over the spatial positions in order to obtain the probability. \n"}, {"review_id": "mpSLzHo2ag", "reviewer": "Reviewer_PAgG", "summary": "This paper proposes a Dissection of Competitive Vision Networks (DISCOVER) to build interpretable neural networks. It aims to understand the neuron functionality for image inference. Moreover, this paper introduces the Jensen-Shannon divergence between the probability of concept presence and the probability of neuron activation for network dissection. Experimental results evaluate the effectiveness of the proposed CVNs.", "questions": "Refer to the Weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Relatively interesting idea of the proposed CVNs for network dissection. \n2. high interpretability of the proposed framework evaluated in experiments.\n", "weaknesses": "1. Limited novelty. DISCOVER method is similar to DMoE [ref-1] combined with the work [ref-2]. Please clarify their difference.\n2. Lack of more visualization results to evaluate the interpretability over other methods.\n\n[Ref-1]. Deep mixture of experts via shallow embedding. Uncertainty in artificial intelligence. PMLR, 2020.\n[Ref-2] Interpretable Neural Network Decoupling. In ECCV, 2020.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for kind comments concerning our approach and for pointing out potentially related work to ours. We believe however that the reviewer is a bit harsh concerning the novelty of the proposed framework. We support this claim by highlighting the pivotal differences between our work and the suggested works.\n\n> Limited novelty. DISCOVER method is similar to DMoE [ref-1] combined with the work [ref-2]. Please\nclarify their difference.\n\nBoth DMoE [ref-1] and the proposed Competitive Vision networks aim to sparsify the networks on a per example basis and do so in a way that renders the networks more expressive. However, the setup of the networks, the rationale, the training procedures and overall the way these two  methods achieve this sparsification is radically different. On the one hand, in DMoE, softmax-ed embeddings of the *input image* are obtained using a \"shallow\" network; these are then passed through a RELU activated multi-headed sparse gating networks to sparsify the results in each layer. On the other hand, CVNs group feature maps into blocks and a competitive random sampling procedure is employed in order to select the winner in each block. This is instantiated by using the response of each feature map to the **layer input** (and not the original image) and by introducing appropriate discrete latent indicators to select the winners that are modeled via solid Bayesian arguments.\n    \nMoreover: (i) the \"shallow\" embedding network that DMoE introduces comprises 4-5 layers; in contrast we do not introduce any additional parameters and use the layer embedding of its input to drive the sparsification mechanism. At the same time the embeddings of DMoE not only arise from the original raw image, but a different weight matrix is introduced for each layer via the gating network that multiply the latent mixture weights, while relying on the ReLU operation to \"sparsify\" the results; instead, we rely on foundational Bayesian arguments and procedures to learn categorical variables that explicitly indicate unit activation in a competitive manner.\n\nWith respect to [ref-2], their method uses a Mutual Information based approach between the inputs and an encoding vector in each layer, in order to discover a unique calculation path for each input. Similarly to the previously discussed work, the only similarity this method bears with our CVN models is the per-example sparsification of the network. We do not consider any mutual information constraints, we do not use any Architecture Controlling Modules, i.e., additional ad-hoc embedding networks to drive the sparsification and more importantly we rely on the concept of local competition between channels, instantiated via a principled formulation via appropriate discrete latent variables and not discretization approaches for training binary (and not categorical) latent variables.\n\n\n> Lack of more visualization results to evaluate the interpretability over other methods.\n\nAs mentioned in the main text, line 305, and our responses to the concerns of other reviewers, since we trained the novel CVN models from scratch, and considering the highly different mode of operation, a direct comparison with other inteprertable methods is not feasible at this point. In the context of this work, and since interpretability in this case is based on the novel CLIP-Dissect framework, most visualization pertain to the standalone Neuron Identification results and the relative to conventional networks quantitative analysis provided in the main text (with some additional qualitative figures in the Supplementary); we nevertheless showed that CVNs improve performance compared to the their conventional counterparts. Since this is the first work that trained CVNs on a large scale, we aim to expand the interpetability investigation to other interpretability-focused methods.\n\n"}]

Author rebuttal:

Meta-review: This paper proposed a framework for creating interpretable vision networks based on stochastic winner-take-all layers. Key results show that the framework leads to a small and interpretable subset of neurons per data point.

Strengths
- The use of stochastic winner-take-all layers in the context of network dissection is novel and elegant.
- The use of LWTA does not impact the accuracy of the underlying model making this a useful technique for interpretability

Weaknesses
- Qualitative results in the paper can be improved. I suggest the authors add more such results in the appendix.
- Paper writing is confusing overall and should be simplified. RcrQf & RCppE made good suggestions on how to do this.

Reviews

This paper received mixed feedback from the reviewers. However, after thoroughly reading the reviewer responses and the author responses, the AC decided to accept the paper since the authors did provide satisfactory responses to each of the reviewers.

Justification

While the reviewers raised objections about the claims of sparsity in the representation, the paper does not suggest that this property should always hold. All the reviewers recognized the simplicity of the proposed approach. The AC also believes this work is more practical since it doesn’t trade off accuracy for intrepretability.
Overall, this paper proposes an interesting way to construct neural networks and should be valuable to the community.
Decision: Accept (poster)