SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DISCOVER:
Making Vision Networks Interpretable via
Competition and Dissection
Konstantinos Panousis
Department of Electrical Eng., Computer Eng., and Informatics
Cyprus University of Technology
Limassol 3036, Cyprus
k.panousis@cut.ac.cy
Sotirios Chatzis
sotirios chatzis@cut.ac.cy
Modern deep networks are highly complex and their inferential outcome very hard to interpret. This is a serious obstacle to their transparent deployment in safety-critical or bias-aware applications. This work contributes to post-hoc interpretability, and specifically Network Dissection. Our goal is to present a framework that makes it easier to discover the individual functionality of each neuron in a network trained on a vision task; discovery is performed in terms of textual description generation. To achieve this objective, we leverage: (i) recent advances in multimodal vision-text models and (ii) network layers founded upon the novel concept of stochastic local competition between linear units. In this setting, only a small subset of layer neurons are activated for a given input, leading to extremely high activation sparsity (as low as only  $\approx 4\%$ ). Crucially, our proposed method infers (sparse) neuron activation patterns that enables the neurons to activate/specialize to inputs with specific characteristics, diversifying their individual functionality. This capacity of our method supercharges the potential of dissection processes: human understandable descriptions are generated only for the very few active neurons, thus facilitating the direct investigation of the network's decision process. As we experimentally show, our approach: (i) yields Vision Networks that retain or improve classification performance, and (ii) realizes a principled framework for text-based description and examination of the generated neuronal representations.
In recent years, Deep Neural Networks (DNNs) have exhibited an overwhelming success in a variety of tasks and applications, achieving state-of-the-art results in Machine Vision, Automatic Speech Recognition, and NLP. This unprecedented success is however accompanied with a limited capacity to audit them in view of reliability standards. Indeed, due to the immense complexity of their architectures, DNNs are usually employed as off-the-shelf solutions for a variety of applications. However, their vast and highly accelerated adoption rate emphasizes the importance of being able to explain how DNNs function and interpret how predictions are made. This process will allow for
uncovering their inherent biases and even for correcting the failures in their decision making process, facilitating a safe, trustworthy and robust real-world deployment in safety-critical applications.
Fortunately, this limitation has recently received a lot of attention in the Deep Learning (DL) research community, leading to the development of ways forward towards Interpretable Deep Networks. In this context, we can identify two major approaches: ante- and post-hoc methods. In the former, the main rationale is to build networks from scratch, that integrate interpretability components into the network itself; Concept Bottleneck Models (CBMs) are a characteristic case [13, 17, 19, 25]. In post-hoc approaches, backbone architectures are augmented with novel frameworks aiming to provide explanations of the prediction process, e.g. saliency maps [15, 27] and Network Dissection [2, 9]. Each approach exhibits its own advantages and drawbacks. For example, ante-hoc models usually suffer from significant performance drop, often requiring additional data to train. On the other hand, even though post-hoc models avoid the complexity of re-training, the results may lend themselves to subjective interpretations that depend on the method's assumptions.
This work takes a novel route: it aims at improving the efficacy of post-hoc methods by addressing some fundamental building blocks of Deep Networks. Specifically, we depart from the commonly used forms of non-linearities in Deep Networks and propose the use of competition-based activations imposed upon otherwise linear projection units; these form the so-called Stochastic Local Winner-Takes-All (LWTA) layers [22]. Our inspiration stems from biological arguments: it has been shown that brain neurons with similar functions aggregate together in blocks and locally compete for their activation; the winner gets to pass its response to other neurons outside the block while the rest are inhibited to silence [30, 1, 14]. Stochastic LWTA activations have recently been employed on DNNs with striking success, exhibiting: (i) significant compression capabilities [22], (ii) unique representation diversification ability [23], (iii) strong adversarial robustness against powerful adversarial attacks [24], (iv) the best ever reported performance in model-agnostic meta-learning (MAML), in terms of both accuracy and network footprint [11]. Other significant properties of LWTA-based networks include noise suppression, specialization, automatic gain control and robustness to catastrophic forgetting[29, 8, 3], while their prowess has also been shown in Transformer networks dealing with end-to-end translation of sign-language video into text [32].
In building interpretable networks, the proposed competition-based rationale comes with two important benefits: (i) a naturally arising activation sparsity: only the winner unit in each block passes its computed output to the next layer, while the rest pass zero values; and (ii) a data-driven pattern of neuron functionality, based on the probability of a unit being the active winner for a given input.
This mode of operation naturally lends itself to the post-hoc paradigm of unraveling and inspecting individual neuron functionality, which this work addresses. In this setting, recent works on automated methods aim to provide high quality descriptions of neurons by leveraging multi-modal models to match neuron functionality to text-based unrestricted concepts.
We leverage these ideas to address the interpretation limitations of conventional architectures, by introducing the Dissection of Competitive Vision Networks (DISCOVER) approach, a novel framework towards interpretable Vision Networks. Our contributions can be summarized as follows:
- We construct, implement and train a variety of Competitive Vision Networks (CVNs) utilizing novel competition arguments, including both CNN and Transformer-based architectures. This is the first time in the literature that CVNs are trained on a large scale and their behavior is investigated with respect to both: (i) performance, and (ii) interpretation properties.
- We perform post-hoc network dissection using multimodal models to identify the underlying functionality and contribution of each individual neuron; to this end, we leverage the unique characteristics of competition between neurons and novel Network Dissection approaches.
- We examine the specialization properties of CVNs in the context of interpretability by investigating the generated per-example neuronal representations.
Image-Text Models. Contrastive Language-Image Pre-training (CLIP) [26] constitutes one of the best-known methods for multimodal deep learning, combining visual representations with natural
language supervision. The main operating principle of CLIP relies on the usage of image-caption pairs in order to obtain representations. Specifically, CLIP comprises an image encoder, denoted by  $E_{T}(\cdot)$  and a text encoder  $E_{I}(\cdot)$  that are trained simultaneously on the grounds of a contrastive learning objective[28, 4]. Specifically, assuming a batch of  $N$  (image, text) pairs, i.e.  $\{(x_{i}, t_{i})\}_{i=1}^{N}$ , CLIP learns a multi-modal embedding space by maximizing the cosine similarity of the embeddings  $I_{i} = E_{I}(x_{i})$  and  $T_{i} = E_{T}(t_{i})$ , between the  $N$  correct pairs, while minimizing the cosine similarity between the other  $N^2 - N$  combinations. During inference, we can use natural language supervision to perform zero-shot classification of any given image and labels. Thus, given a test image  $x^{\mathrm{test}}$  and some labels  $\{t_{j}\}_{j=1}^{M}$ , we can use the learned encoders and select the image-label pair that has the highest similarity of all the possible combinations.
Network Dissection. Network Dissection constitutes one of the most popular approaches for understanding and describing the contribution of individual deep network units in the produced inferential outcomes. In [2], a method that yields descriptive outcomes in terms of concepts is presented. To this end, the authors introduce a newly developed densely-labeled dataset, named Broden; this contains pre-determined concepts, images and their associated pixel-level labels. The main principle is to use the Intersection over Union (IoU) score as a measure of relevance between each concept and neuron in the set. However, the proposed method requires the use of dense labels for training, while at the same time the concepts are restricted to the pre-defined label set.
This limitation inspired the development of MILAN [9], an automated method that addresses the notion of pre-defined concepts through the generation of unrestricted description via generative image-to-text models. Nevertheless, the requirement for human annotation is still present, similar to Network Dissection [2]. CLIP-Dissect [20] bypasses this restriction by leveraging multimodal image-text models, and specifically CLIP; this allows for decoupling the dependence between the concept set and the probing dataset. Thus, any network can be dissected, using any text corpus to form the concept set, any image probing dataset, and any appropriate similarity measure for matching concepts to neurons.
In this work, we exploit the flexibility of CLIP-Dissect and extend it in the context of Competitive Vision Networks. Differently than CLIP-Dissect, we mainly focus our analysis on the properties of Vision Transformer architectures with respect to neuron identification.
Competitive Vision Networks. Recent works have explored a fundamentally different paradigm for the latent unit operation of DNNs, based on the concept of local competition among units (LwTA) [29]. In this setting, latent units compute their response and compete for their activation. The winner passes its (linear) response to the next layer, while the rest output zero values. In the following, we will be referring to these networks as Competitive Vision Networks (CVNs).
Let us consider an input  $\pmb{x} \in \mathbb{R}^J$  presented to a hidden layer of a DNN comprising  $K$  hidden units and a weight matrix  $\pmb{W} \in \mathbb{R}^{J \times K}$ . To compute a typical non-linear activation, each hidden unit in the layer performs an inner product computation with its respective weights  $\pmb{w}_k \in \mathbb{R}^J$  yielding the intermediate response  $h_k = \pmb{w}_k^T \pmb{x} \in \mathbb{R}$ ; this usually passes through a non-linear activation  $\sigma(\cdot)$ , such that the final layer output reads:  $\pmb{y} = [y_1, \dots, y_K]$ , where  $y_k = \sigma(h_k), \forall k$ .
In a corresponding LWTA-based layer,  $U$  singular units are grouped together forming the so-called LWTA blocks. Hereinafter, we denote by  $B$  the number of LWTA blocks in the layer, and with  $U$  the number of linear competing units therein. The aggregation operation is manifested via the definition of the layer's weights as a three-dimensional matrix  $\mathbf{W} \in \mathbb{R}^{J \times B \times U}$ ; this structural modification implies that each input dimension of  $\mathbf{x}$  is now presented to each block  $b$  and each unit  $u$  therein; in turn, all units compute their response via the standard inner-product computation  $h_{b,u} = \mathbf{w}_{b,u}^T \mathbf{x} \in \mathbb{R}, \forall b, u$  and competition takes place among them. The fundamental principle of competition is that out of the  $U$  competing units, only one can be the winner. This unit gets to convey its activation outside the block, i.e. the next layer, while the rest output zero values. This process is instantiated via an appropriate competition function that encodes the outcome of the competition in each block.
Contrary to the conventional definition, where the layer's output arises as the concatenation of the individual response of each unit, in the LWTA-based framework, the final output  $\pmb{y}^{B\cdot U}$  is now constructed from  $B$  sub-vectors, one for each LWTA block that contains a single non-zero entry; this corresponds to the response of the winner unit in said block. Thus, an inherent property of the LWTA mechanism is the naturally emerging sparse representations. Indeed, considering a fixed number of
Figure 1: Detailed bisection of the  $b^{\mathrm{th}}$  Stochastic LWTA block. Presented with an input  $\pmb{x} \in \mathbb{R}^{J}$ , each unit  $u$  computes its activation  $h_{b,u}$  via different weights  $\pmb{w}_{b,u} \in \mathbb{R}^{J}$ , i.e.,  $h_{b,u} = \pmb{w}_{b,u}^{T}\pmb{x}$ . The linear responses are concatenated, s.t.,  $\pmb{h}_b = [h_{b,1},\dots,h_{b,U}]$ , and transformed into probabilities via the softmax operation. Then, a Discrete sample  $\pmb{\xi}_b = [\xi_{b,1},\dots,\xi_{b,U}]$  is drawn, which is a one-hot vector with a single non-zero entry at position  $u'$ , denoting the winner unit in the block; unit  $u'$ , passes its linear response to the next layer, while the rest pass zero values. Image from [23].
units per layer, we can observe that the higher the number of competitors  $U$ , the sparser the layer output: when  $U = 2$ , only  $50\%$  of the units will be active per example, when  $U = 4$  only  $25\%$  and so on. This gives rise to a sparse activation mode of operation that has been empirically shown to endow deep architectures with significant properties including strong adversarial robustness [24] and substantial representation diversification [23]. One design choice that we need to address concerns the nature of the competition: deterministic or stochastic. The former is the most typical approach in the LWTA literature; in this case, the unit with the highest activation is deemed the winner each time. The latter is founded upon novel stochastic arguments proposed in [22]. This formulation has been shown to consistently outperform its deterministic counterpart [21, 24, 32, 11]. Thus, we adopt stochastic competition in our approach.
The stochastic variant of the competition function entails the introduction of an appropriate set of discrete latent variables for each LWTA block  $b$ ,  $\pmb{\xi}_b \in \mathrm{one\_hot}(U)$ , that encode the (stochastic) outcome of the competition therein. The latent indicators constitute one-hot vectors with  $U$  components; that is, vectors containing a single non-zero entry at the index position corresponding to the winner unit in the block. The output  $\pmb{y}$  of a stochastic LWTA layer's  $(b, u)^{th}$  component  $y_{b,u}$  yields:
$$
y _ {b, u} = \xi_ {b, u} \sum_ {j = 1} ^ {J} w _ {j, b, u} \cdot x _ {j} \in \mathbb {R} \tag {1}
$$
We postulate that these latent indicators  $\xi_{b},\forall b,$  are drawn from a Categorical distribution driven from the intermediate linear inner-product computations that each unit performs. Evidently, the higher the response of a particular unit in a block (relative to the others), the higher its probability of being the winner; however, the final decision remains stochastic. We can formulate this rationale as:
$$
q \left(\boldsymbol {\xi} _ {b}\right) = \text {C a t e g o r i c a l} \left(\boldsymbol {\xi} _ {b} \mid \Pi_ {b} (\boldsymbol {x})\right), \Pi_ {b} (\boldsymbol {x}) = \operatorname {S o f t m a x} \left(\sum_ {j = 1} ^ {J} \left[ w _ {j, b, u} \right] _ {u = 1} ^ {U} \cdot x _ {j}\right), \quad \forall b \tag {2}
$$
where  $\Pi_b(x)$  is a vector comprising the activation probability of each neuron in the block, and  $[w_{j,b,u}]_u^U$  denotes the vector concatenation of the set  $\{w_{j,b,u}\}_{u = 1}^U$ . Each unit in each block computes its intermediate linear inner-product; these are then passed to a softmax transformation and used to draw samples from a categorical distribution. The resulting discrete vector contains a single non-zero entry denoting the winner of the block. The intermediate computation of the winner unit passes out as its activation; the rest units pass zero-valued activations. A graphical illustration of the stochastic LWTA structure is provided in Fig. 1.
In the context of attention-based architectures, the encoder of Vision Transformers (ViTs) [6] comprises alternating layers of Multi-head Self-Attention (MSA), MLP blocks, Layer Normalization
(LN) and residual connections. Each MLP block comprises two layers with a GELU non-linearity in between. Within the CVN framework, we replace GELU layers with stochastic LWTA layers. This modeling decision is motivated by recent works that have shown that transformer MLPs encode knowledge attributes [5, 7, 18]. We posit that this formulation facilitates neuron identification and interpretation via analysis of the arising competition patterns; we explore its potency in the following sections.
Further, convolutional operations also constitute an integral part of various architectures. To account for this fact, and for completeness, we also consider the convolutional variant of the stochastic LWTA layer proposed in [22]. Due to space constraints, and since we mainly focus our analysis on attention-based architectures, we provide the formulation in the Supplemental Material.
3 DISCOVER
3.1 Training and Inference Algorithms for CVNs
CVNs comprise additional auxiliary variables denoting the winner in each LWTA block; thus, we need to devise an appropriate training regime that takes into consideration the stochastic nature of said variables. To this end, we turn to Stochastic Gradient Variational Bayes (SGVB) [12], and construct an Evidence Lower Bound (ELBO) objective. Assuming a dataset  $\mathcal{D} = \{X_i,y_i\}_{i = 1}^N$  and denoting as  $f(\pmb {X}_i;\hat{\pmb{\xi}})$  the cross-entropy between the target labels  $y_{i}$  and the target probabilities emerging from a CVN, the objective reads:
$$
\mathcal {L} ^ {\mathrm {C V N}} = - \sum_ {\boldsymbol {X} _ {i}, y _ {i} \in \mathcal {D}} \operatorname {C E} \left(y _ {i}, f \left(\boldsymbol {X} _ {i}; \hat {\boldsymbol {\xi}}\right)\right) - \mathrm {K L} [ q (\boldsymbol {\xi}) | | p (\boldsymbol {\xi}) ] \tag {3}
$$
where  $\hat{\xi}$  denotes samples from the (posterior) distributions of all latent variables,  $\xi$ , in all layers, and  $\mathrm{KL}[q(\pmb{\xi})||p(\pmb{\xi})]$  denotes the Kullback-Leibler divergence between the posterior and the prior. We assume a symmetric Categorical prior for the latent variable indicators  $\xi$ ; hence,  $p(\pmb{\xi}_b) = \mathrm{Categorical}(1 / U)$ ,  $\forall b$ . For all the computations we draw just one sample, but we consider it as a differentiable expression; this allows for very low variance gradients during training, which ensures convergence (reparameterization trick). Since the Categorical distribution is not amenable to differentiation, we enable reparameterization by resorting to a continuous relaxation, namely Gumbel-Softmax (GS) [16, 10]. In contrast to previous works [22, 23], we use the Straight-Through variant of GS, which we have seen to offer better convergence properties. We present the exact sampling procedure in the Supplemental Material. During inference, we directly draw samples from the trained posteriors and determine the winning units in each layer along with their responses (Eq. 2).
3.2 Dissection
For dissecting the considered CVNs, we draw inspiration from CLIP-Dissect [20]. We employ a CLIP model to identify the functionality of each neuron in a stochastic LWTA block in terms of a given concept set; this will be matched with a textual description. There are three key components that define the approach: (i) a concept set  $S$ , where  $|S| = M$ , (ii) a dataset that is used as a probe to identify the individual functionality of neurons denoted by  $\mathcal{D}_{\mathrm{probe}}$ , where  $|\mathcal{D}_{\mathrm{probe}}| = N$ , and (iii) the network to be investigated, denoted by  $f(x)$ . The main principle is to match neuronal activations with concepts using: (i) the activation matrix  $P$  that measures the similarity between images and concepts, and (ii) a summary of the individual neuronal activations of the network being probed to the given probe dataset.
Concept Activation Matrix. We use the image and text encoder of the CLIP model, denoted by  $E_{I}$  and  $E_{T}$  respectively, and compute the respective embeddings  $I_{i}$  and  $T_{i}$  for each image  $x_{i}$  and concept  $t_{i}$ . This results in a concept activation matrix  $P \in \mathbb{R}^{N \times M}$ , where each entry  $(i,j)$  denotes the cosine similarity between image  $x_{i}$  and concept  $t_{j}$  in the CLIP embedding space.
Record Activations for each Competitive Neuron. Given a neuron  $k$  and for each image in the probe dataset  $\{X_{n}\}_{n = 1}^{N}$ , CLIP-Dissect computes the neuron's non-linear responses  $\{A_k(\pmb {X}_n)\}_{n = 1}^N$  and records them into a single vector  $\pmb {q}_k\in \mathbb{R}^N$ , s.t.,  $\pmb {q}_k = [A_k(\pmb {X}_1),\dots ,A_k(\pmb {X}_N)]$ . Each activation vector  $\pmb {q}_k$  essentially encodes the activation of neuron  $k$  across the whole probing dataset.
Figure 2: The DISCOVER framework. Given a probe dataset  $\mathcal{D}_{\mathrm{probe}}:\{X_n\}_{n = 1}^N$ , a concept set  $S,|S| = M$ , and a layer to be dissected: compute the concept activation matrix  $P\in \mathbb{R}^{N\times M}$  using CLIP; then, for each LWTA block  $b$  and each neuron  $u$  therein, compute the responses  $A_{b,u}(X_n),\forall n$ , s.t.  $q_{b,u} = [A_{b,u}(X_1),\dots ,A_{b,u}(X_N)]$ . Each neuron  $b,u$  is then matched to the concept  $t_l$  whose activation vector  $P_{:,l}$  has the highest similarity (sim) with the neuron's activation vector  $q_{b,u}$ .
Subsequently, it can be exploited as a data-wide neuron representation to match neurons to textual descriptions as we describe next. When the response  $A_{k}(\mathbf{X}_{n})$  is not a scalar, a summary function that maps it to a real number is employed. For inner-product neurons, the response is already a real-number. For convolutional kernels, we consider a mean over the spatial dimensions.
Our approach. Building on these outcomes, CVNs adopt a novel regard toward interpretation, which leverages the sparse nature of the representations obtained from stochastic LWTA layers; here, we focus on attention-based architectures. In this setting, the MLP blocks contain patch-specific information relating to the tokens of each image; in principle, we could introduce a summary function to aggregate the patch-wise information and output a real-number. However, we have access to another representation: the feature embedding pertaining to the class token; this comprises a set of units dedicated to capturing both global and local contextual information in each layer.
Within the framework of CVNs, this is treated in a local competition fashion: CVN units on each MLP are grouped together in blocks of competitors and compete for the output of the block. Contrary to the singular recording of activations in the base formulation for each neuron  $k$ ,  $\{A_k(\mathbf{X}_n)\}_{n=1}^N$ , the activations are computed for each LWTA block  $b$  and unit  $u$  therein, yielding  $\{A_{b,u}(\mathbf{X}_n)\}_{n=1}^N$ . These activations can be negative, positive or exactly zero, depending not only on the linear computation of the unit in consideration, but also the responses of the remaining units in its LWTA block, following the form of Eq.1. We concatenate them to construct a summary for each competitive unit, such that  $q_{b,u} = [A_{b,u}(\mathbf{X}_1),\dots,A_{b,u}(\mathbf{X}_N)]$ .