SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Due to the nature of the proposed stochastic local competition, the emerging summary vectors will be sparse; only the winner of each LWTA block will retain its linear computation and pass it as a nonzero activation for each example  $X_{n}$ ; the rest units in the block pass zero activations. This mode of operation not only encourages specialization, but can also diversify individual neuron identification; we explore its potency in the next section.
Matching Neurons to Concepts. Having recorded the activations vectors for each LWTA block  $b$  and each neuron  $u$  therein,  $\pmb{q}_{b,u} \in \mathbb{R}^N$ , we aim to discover the most similar concept  $\{t_m\}_{m=1}^M$  to describe each neuron. To achieve this matching, a similarity function  $\mathrm{sim}(t_m, \pmb{q}_{b,u}; \pmb{P})$  is defined; this is used to quantify the relation (in terms of similarity) between the neuron's activation vector
$\pmb{q}_{b,u} \in \mathbb{R}^N$  and the concept's  $m$  activation vector,  $\pmb{P}_{:,m} \in \mathbb{R}^N$ . For example, considering the cosine similarity, we compute:
$$
\sin \left(t _ {m}, \boldsymbol {q} _ {b, u}; \boldsymbol {P}\right) \propto \boldsymbol {P} _ {:, m} \boldsymbol {q} _ {b, u} ^ {T} \tag {4}
$$
Thus, for assigning a concept to neuron  $b$ ,  $u$ , we compute its similarity to each concept in the set and select the one that exhibits the highest value, s.t.,  $t_{l} = \arg \max_{m}\sin (t_{m},\pmb{q}_{b,u};\pmb{P})$ . Characteristic similarity functions include, Rank Reorder, WPMI and SoftWPMI [20].
At this point, it is important to highlight a principal benefit of CVNs: After neuron identification, i.e., matching neurons to concepts, and since only a small subset of neurons is active for each example, it becomes practically tractable to perform a per-example analysis on that particular small subset of "winner" (active) neurons; this greatly facilitates practical concept interrogation.
4 Experimental Analysis
For evaluating and dissecting the proposed CVNs, we train two sets of models: (i) Transfomer-based, and (ii) Convolutional architectures. We consider stochastic LWTA layers with different numbers of competitors, ranging from  $U = 2$  to  $U = 24$ . In every architecture, we retain the total number of parameters of the conventional model by splitting a layer comprising  $K$  singular neurons to  $B$  blocks of  $U$  competing neurons, such that  $B \cdot U = K$ . This choice facilitates a fair -sizewise-comparison between an original network and its CVN counterpart. The number of competitors has a direct effect on the per example neuron activation in the respective layers. For example, when  $U = 2$ , only  $50\%$  of neurons are activated for a given input, when  $U = 8$  only  $12.5\%$  and so on.
For the Transformer architecture, we select the DeiT model, specifically DeiT-Tiny (DeiT-T, 5M parameters) and DeiT-Small (DeiT-S, 22M parameters), which we train from scratch on ImageNet-1k. For the convolutional paradigm, we chose the same network as in CLIP-Dissect [20] for comparability; that is ResNet-18 trained on Places365. In all cases, we follow the original training schemes concerning the hyperparameters, and remove the excessive augmentations of DeiT, i.e., DropPath, Color Jittering, Random Erase, CutMix and MixUp; we have found that these do not improve accuracy of CVNs. For completeness, all hyperparameter settings for each model are provided in the Supplementary Material. All models were trained on a single NVIDIA A6000 GPU. Our code implementation is available at: https://github.com/konpanousis/DISCOVER.
Accuracy. We begin our evaluation with classification results pertaining to CVNs and how block size (number of competitors in each block) affects performance. This is essential, since CVNs have not been explored in the literature, especially when dealing with larger datasets and architectures, and no baselines exist for the considered setup. However, it is highly important to note that this work is not focused on achieving state-of-the-art performance for vision tasks using CVNs. Instead, our focus is on providing a fundamental component towards a novel Network Dissection paradigm. Thus, we focus on Network Dissection efficacy, as opposed to optimally tuning hyperparameters and augmentation to increase accuracy by few points.
The obtained comparative results are presented in Table 1. Therein, we observe that despite the fact that we did not aim for performance improvements, CVNs exhibit near or even superior performance compared to their conventional counterparts. This finding persists even when using a large number of competitors, which directly translates to high activation sparsity. Let us consider for example DeiT-T/16; this model comprises LWTA blocks with 16 competing units each, leading to  $1 / 16 = 6.25\%$  active (winning) neurons in each layer. We observe that this network exhibits only a  $1\%$  drop in prediction accuracy (again without any tuning). On the other hand, when using a smaller number of competitors, e.g., DeiT-T/2 or DeiT-T/8, we obtain better and on par performance to the conventional architecture, respectively. Also note that, even though there is a minor computational overhead during training (up to  $10\%$ ), the computational costs of inference are comparable. Thus, accuracy-wise, CVNs provide a realistic alternative to conventional Vision Networks that can even yield accuracy improvements when trained with the same (potentially suboptimal) setup.
Quantitative Analysis. We follow the novel proposal of [20] to compare our CVN-based approach to conventional non-competitive networks. In this context, we can compare the generated neuron labels, for a specific layer of a network, with the ground truth descriptions, i.e., the class labels. Specifically, we measure the cosine similarity in a text embedding space between the ground truth
Table 1: Classification results for ImageNet and Places365. /None denotes a non-competitive architecture, e.g., using ReLU/GELU, while numbers denote the number of competitors  $U$ . {*/ *}Locally reproduced results due to unavailability of pretrained models/resource limitations.
ImageNet-1k
Model/Competitors	Accuracy + Std (%)	Active Neurons Proportion
DeiT-T/None	72.2 ± N/A	Data Specific
DeiT-T/2	72.5 ± 0.10	0.500
DeiT-T/8	71.7 ± 0.15	0.125
DeiT-T/16	71.1 ± 0.25	0.062
DeiT-T/24	70.5 ± 0.45	0.041
DeiT-S/None	77.0 ± N/A*	Data Specific
DeiT-S/2	77.3 ± 0.30	0.500
DeiT-S/12	77.0 ± 0.25	0.083
DeiT-S/16	76.7 ± 0.50	0.062
Places365
ResNet-18/None	52.25 ± N/A*	Data Specific
ResNet-18/2	53.9 ± 0.20	0.500
ResNet-18/4	51.0 ± 0.50	0.250
ResNet-18/8	49.5 ± 0.75	0.125
class name and the description of the neuron. We use two different text encoders: (i) CLIP ViT-B/16, denoted as CLIP cos and (ii) all-mpnet-base-v2, denoted as mpnet cos. Here, we focus on the last layer of the MLP block of the DeiT-T model. The obtained comparative results are depicted in Table 2. Therein, we observe that the CVN consistently outperforms its non-competitive counterpart when using both the best-performing softWPMI and the Cubed Cosine Similarity function proposed in [20].
Table 2: Cosine similarity between the last layer neuron descriptions and ground truth labels on DeiT/None and DeiT-T/16 trained on ImageNet. For the former we use softWPMI for identification, while for CVNs, softWPMI and the cubed cosine similarity measure [20]. We use two text encoders: (i) CLIP ViT-B/16 and (ii) all-mpnet-base-v2, denoted as mpnet cos.
Dprobe	Concept Set S	Vision Network
DeiT-T/None (SoftWMI)	DeiT-T/16 (SoftWPMI)	DeiT-T/16 (CosSimCubed)
CLIP cos	mpnet cos	CLIP cos	mpnet cos	CLIP cos	mpnet cos
ImageNet Val	Broden	0.6250	0.1800	0.6460	0.1829	0.6543	0.1918
—"—	3k	0.6226	0.1688	0.6650	0.1895	0.6699	0.1842
—"—	10k	0.6187	0.1715	0.6553	0.1878	0.6616	0.1831
—"—	20k	0.6128	0.1758	0.6455	0.1929	0.6519	0.1899
—"—	ImageNet	0.5850	0.1782	0.5854	0.1842	0.5840	0.1835
CIFAR100 Train	20k	0.6392	0.1765	0.6533	0.1969	0.6530	0.1876
Broden	20k	0.6338	0.1780	0.6470	0.1815	0.6606	0.1546
ImageNet Val & Broden	20k	0.6299	0.1821	0.6333	0.1876	0.6729	0.1433
To further assess the quality of the obtained neuronal descriptions, we turn to the neuron identification accuracy metric proposed in [20]. In this context, and in situations where the considered concept set contains exact class labels, we compute the percentage of neurons that have been assigned the exact correct label. Evidently, this metric pertains to the classification layer of a considered network. We consider four datasets: CIFAR-100 Train, Broden, ImageNet Val and ImageNet Val & Broden; we use the ImageNet classes as the concept set  $S$ . For matching neurons to concepts, we use SoftWPMI, since it outperformed all other similarity functions. Further results with the other similarity metrics are provided in the Supplemental Material.
The comparative results for all conventional and competitive configurations are presented in Table 3. We observe that on average, CVNs consistently outperform their conventional ReLU/GELU based counterparts, with up to  $2\%$  improvement. It is striking, that DeiT-S/12 improves accuracy on CIFAR-100 Train by a staggering  $10\%$  compared to the conventional ResNet-50 that has the same number of parameters; at the same time, DeiT-T/8, also provided a significant improvement of  $\approx 7\%$ .
Table 3: Accuracy as the percentage of neurons assigned the correct label, i.e., class name. Concept Set  $S$  : ImageNet,Similarity: SoftWPMI. ResNet-50 results from [20].
Dprobe	Vision Network
ResNet-50/None	DeiT-T/None	DeiT-T/8	DeiT-T/16	DeiT-S/2	DeiT-S/12
CIFAR-100 Train	46.20%	53.00%	53.10%	50.80%	55.20%	56.10%
Broden	70.50%	68.80%	67.40%	67.40%	68.30%	68.10%
ImageNet Val	95.00%	95.00%	96.00%	95.30%	95.60%	95.60%
ImageNet Val & Broden	95.40%	95.20%	95.70%	95.50%	95.00%	94.80%
Average	76.78%	78.00%	78.05%	78.30%	78.70%	78.65%
Neuron 10: CLIP-Dissect (Soft WPMI): snorkeling
Neuron 20: CLIP-Dissect (Soft WPMI): green
Neuron 70: CLIP-Dissect (Soft WPMI): red
Neuron 302:CLIP-Dissect (Soft WPMI): maroon
Figure 3: Neuron Identification for the first and last DeiT-T/16 MLP blocks using: SoftWPMI,  $\mathcal{D}_{\mathrm{probe}}$ : ImageNet & Broden, S:  $20K$  most common English words [20].
Neuron 41: CLIP-Dissect (Soft WPMI): dog
Neuron 218: CLIP-Dissect (Soft WPMI): wine
Neuron 357:CLIP-Dissect (Soft WPMI): snow
Neuron 512: CLIP-Dissect (Soft WPMI): coyote
Qualitative Analysis. We now turn to a qualitative analysis of the obtained neuronal representation. In this context, we visualize several neuron identification results in terms of generated descriptions in Fig. 3. To this end, we select random neurons of the first and last MLP blocks of DeiT-T/16 and use a combination of the ImageNet and Broden datasets as a probe; the concept set  $S$  comprises the 20,000 most common English words similar to [20]. Since we do not use the same pretrained backbone, and the considered CVNs were trained from scratch, there isn't an exact matching between the neurons presented therein and the neurons of our networks, enumeration-wise. Thus, we can not make direct comparisons pertaining to the exact same neuron on an illustration basis. Nevertheless, we observe that in the context of CVNs we obtain highly accurate descriptions of the functionality of the networks' neurons. We observe that the randomly selected neurons are activated by semantically similar inputs that may contain only a part of the matching concept.
A principal property of CVNs is the per-example sparsity that naturally arises due to the competition mechanism; this can facilitate specialization and accelerate the process of examining the active concepts for each test point, greatly enhancing the interpretability of the network. Indeed, we perform this analysis for the last MLP block of the GELU-based DeiT-T/None, and its DeiT-T/8 and DeiT-T/16 competitive counterparts. This block comprises 768 neurons; for each example, DeiT-T/8 will activate only  $768 / 8 = 96$  neurons and DeiT-T/16 only 48 neurons. The number of activated neurons for the conventional DeiT-T/None will vary according to the probing dataset. For ImageNet Val, we found that on average,  $98\%$  of neurons are activated for each example when using DeiT-T/None.
Thus, in Table 4, and for each architecture, we present the 7 most relevant concepts tied to neurons with the highest magnitude of activation to the considered example, along with 6 concepts tied to neurons with the lowest magnitude. Therein, we observe that the most contributing concepts in the conventional setting contain descriptions that are highly irrelevant or even contradicting to the presented input as "red/orange beak & legs", "black coat/tan markings" and "reddish brown fur"; at the same time 757/768 neurons are active for this example, rendering the concept investigation a strenuous process. In stark contrast, the considered CVNs exhibit substantially more relevant concepts, while allowing for a straightforward examination of the emerging per-example concepts. Similar qualitative visualizations for different settings can be found in the Supplemental Material.
Table 4: Concepts tied to activated neurons for a random image from the ImageNet validation set for both DeiT-T/None and DeiT-T/16. For the former,  $757 / 768 = 98.56\%$  neurons are activated leading to an arduous interpretation process; for DeiT-T/16 only  $48 / 768 = 6.25\%$  of neurons are active.
DeiT-T/None	DeiT-T/8	DeiT-T/16
Concepts	Act	Concepts	Act	Act	Concepts
white fur under tail	+3.06	white fur under tail	+10.5	fluffy/white appearance	+11.6
red/orange beak & legs	+2.53	a cucumber plant	+9.32	large/fluffy white dog	+9.89
white plumage	+2.19	a fox	+8.90	large/fluffy white dog	+7.71
black coat/tan markings	+2.14	pale/gray or cream fur	+8.36	egret	+7.37
large/fluffy white dog	+1.46	white plumage	+7.33	white/cream coloured coat	+7.07
reddish/brown fur	+1.03	white/gray plumage	+7.27	blue/gray body	+6.87
shaggy/gray coat	+0.71	cabbage-like leaves	+7.04	white plumage	+5.96
a fox	-0.17	small/black dog	+1.57	a cub	+1.29
large/fluffy white dog	-0.17	shaggy/black-brown coat	+0.56	wolf-like appearance	+0.96
chicken	-0.17	product displays	+0.46	bandage	+0.68
klin	-0.17	a large/elegant dog	+0.34	egg-laying mammal	+0.37
sharp talons/beak	-0.17	a bus driver	+0.18	large/fluffy white dog	+0.35
camel rider	-0.17	an insect	+0.08	black/tan coat	+0.08
5 Limitations & Conclusions
Limitations. Considering the use of the CLIP-Dissect approach for identifying neuron functionality in CVNs, this comes with its respective limitations, such as disregarding the spatial information; this, could allow for identifying lower level concepts and patterns. However, the structure of CVNs could potentially accommodate such a functionality via block (high) level and unit (low) level descriptions; we leave these explorations for future work. Finally, even though incorporating the competition mechanism to Vision Networks is a quite straightforward task, they aren't still implemented in an optimized way in popular frameworks, due to their limited adoption.
Conclusions. In this work, we introduced DISCOVER, a novel framework towards Interpretable vision architectures through Competitive Vision Networks and Network Dissection. For the first time in the literature, we trained LWTA networks using sufficiently large datasets and architectures. Despite the fact that we did not perform any hyperparameter tuning, the resulting networks yielded on par or even improved performance, even when using up to  $4\%$  of active neurons per example. Our qualitative and quantitative results vouch for the efficacy and interpretation capabilities of our approach. We yielded highly interpretable neuron functionality, while using a small and explorable subset of all the available neurons per datapoint.
This work has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No 872139, project aiD.
[1] P Andersen, Gary N Gross, T Lomo, and Ola Sveen. Participation of inhibitory and excitatory interneurones in the control of hippocampal cortical output. In UCLA forum in medical sciences, 1969.
[2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proc. CVPR, 2017.
[3] Gail A. Carpenter and Stephen Grossberg. The art of adaptive pattern recognition by a self-organizing neural network. Computer, 1988.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proc. ICML, 2020.
[5] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proc. ACL, 2022.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. ICLR, 2021.
[7] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In EMNLP, 2021.
[8] Stephen Grossberg. Contour enhancement, short term memory, and constancies in reverberating neural networks. In Studies of mind and brain. Springer, 1982.
[9] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In Proc. ICLR, 2022.
[10] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumbel-softmax. In Proc. ICLR, 2017.
[11] Konstantinos Kalais and Sotirios Chatzis. Stochastic deep networks with linear competing units for model-agnostic meta-learning. In Proc. ICML, 2022.
[12] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proc. ICLR, 2014.
[13] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Proc. ICML, 2020.
[14] Anders Lansner. Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations. Trends in neurosciences, 2009.
[15] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proc. NIPS, 2017.
[16] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proc. ICLR, 2017.
[17] Diego Marcos, Ruth Fong, Sylvain Lobry, Rémi Flamary, Nicolas Courty, and Devis Tuia. Contextual semantic interpretability. In Proc. ACCV, 2020.
[18] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In Proc. ICLR, 2023.
[19] Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In Proc. ICLR, 2023.
[20] Tuomas Oikarinen and Tsui-Wei Weng. CLIP-dissect: Automatic description of neuron representations in deep vision networks. In Proc. ICLR, 2023.
[21] Konstantinos Panousis, Sotirios Chatzis, Antonios Alexos, and Sergios Theodoridis. Local competition and stochasticity for adversarial robustness in deep learning. In Proc. AISTATS, 2021.
[22] Konstantinos Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric Bayesian deep networks with local competition. In Proc. ICML, 2019.
[23] Konstantinos P Panousis, Anastasios Antoniadis, and Sotirios Chatzis. Competing mutual information constraints with stochastic competition-based activations for learning diversified representations. In Proc. AAAI, 2022.
[24] Konstantinos P Panousis and Sotirios Chatzis. Stochastic local winner-takes-all networks enable profound adversarial robustness. In NIPS Bayesian Deep Learning (BDL) Workshop, 2022.
[25] Konstantinos P. Panousis, Dino Ienco, and Diego Marcos. Sparse linear concept discovery models. In Proc. ICCCW CLVL, 2023.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021.
[27] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proc. SIGKDD, 2016.
[28] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proc. NIPS, 2016.
[29] Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Jürgen Schmidhuber. Compete to compute. In Proc. NIPS, 2013.
[30] Costas Stefanis. Interneuronal mechanisms in the cortex. In UCLA forum in medical sciences, 1969.
[31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Proc. ICML, 2021.
[32] Andreas Voskou, Konstantinos P. Panousis, Dimitrios Kosmopoulos, Dimitris N. Metaxas, and Sotirios Chatzis. Stochastic transformer networks with linear competing units: Application to end-to-end sl translation. In Proc. ICCV, 2021.
A Supplemental Material
A.1 Experimental Details
Integrating the competition mechanism in modern architectures constitutes a pretty straightforward task. One just needs to replace the existing calls to the conventional non-linear activations with a call to a module/function implementing the LWTA rationale. This can be performed in-place without any other changes necessary. Thus, in the accompanied code, we can directly change the definitions of the Transformer or ResNet based models and train the model in the same manner as in the conventional Vision Networks. In this context, for the DeiT architectures, we alter the definitions in the timm library, while for ResNet-based architectures, we slightly alter the example implementation found in the official Pytorch repository<sup>1</sup>. For dissecting CVNs, we use the official repository of CLIP-Dissect [20]. We describe all changes in the respective README file. Our code and trained models will be public after publication.
Transformers. For training the CVN counterparts of the DeiT-T and DeiT-S models, we use the official implementation. We train both architectures from scratch using ImageNet-1k for 300 epochs with the default parameters found therein. Specifically, we use a 5-epoch warm-up period, starting with an initial learning rate of  $10^{-6}$ , following a cosine annealing schedule up to  $5 \cdot 10^{-4}$ . We use the same AdamW optimizer and changed the used weight decay from 0.05 to 0.02 since we found that it hurt performance. We re-run the conventional GELU-based architectures with this selection and observed no change in the obtained accuracy. In contrast, we turned off the excessive augmentation setup, since it hurt the performance of CVNs. In this context, and in line with the initial ablation study presented in [31], we found that performance deteriorates when augmentations are removed in GELU-based networks. For training, we use a single sample for the Monte Carlo sampling estimation for the Evidence Lower Bound loss, described in the main text. For this, we turn to the continuous relaxation of the categorical distribution that allows for reparameterized samples and low variance estimation, as we describe in the next section. During inference, we draw 4 samples, average the logits in the Bayesian Averaging sense; we then compute the predicted loss and accuracy. We did not observe any improvement when drawing more samples.