SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ResNet. For training the ResNet-18 model, we used the ResNet ImageNet example PyTorch script and adapted the data loading for Places 365. We train the model for 90 epochs, using SGD with an initial learning rate of 0.1 that is reduced by a factor of 0.1 every 30 epochs, a weight decay of  $10^{-4}$  and 0.9 momentum. The batch size was set to 256. For the Gumbel-Softmax trick, we set the temperature to 0.67 and used the Straight-Through estimator. During training, we only draw one sample for the reparameterization trick, while during inference we draw 4 samples from the trained posterior. We trained both conventional and CVN architectures since the official pretrained models were not available online. For both methods, we used the standard RandomResizedCrop and RandomHorizontal Flip augmentations.
A.2 The Gumbel-Softmax Trick
In our work, we perform Monte-Carlo sampling using a single reparameterized sample for each of the corresponding latent variables. These are obtained via the reparameterization trick of the continuous relaxation of the Categorical distribution [16, 10] as described next. We focus on the reparameterization trick for the dense case; the convolutional case is analogous.
As a reminder, the latent indicators  $\xi_{b},\forall b,$  are drawn from a Categorical distribution driven from the intermediate linear inner-product computations that each unit performs.of  $q(\pmb {\xi})$  (Eq. (2) in the main text):
$$
q \left(\boldsymbol {\xi} _ {b}\right) = \text {C a t e g o r i c a l} \left(\boldsymbol {\xi} _ {b} \mid \Pi_ {b} (\boldsymbol {x})\right), \forall b, \quad \boldsymbol {\Pi} _ {b} (\boldsymbol {x}) = \operatorname {S o f t m a x} \left(\sum_ {j = 1} ^ {J} \left[ w _ {j, b, u} \right] _ {u = 1} ^ {U} \cdot x _ {j}\right) \tag {5}
$$
where  $\Pi_b(\pmb{x})$  is a vector comprising the activation probability of each neuron in the block, and  $[w_{j,b,u}]_{u=1}^U$  denotes the vector concatenation of the set  $\{w_{j,b,u}\}_{u=1}^U$ .
$^{1}$ https://github.com/pytorch/examples/tree/main/imagenet $^{2}$ https://github.com/facebookresearch/deit.
Then, the samples  $\hat{\pmb{\xi}}$  are expressed as:
$$
\hat {\xi} _ {b, u} = \operatorname {S o f t m a x} \left(\log \left[ \Pi_ {b} (\boldsymbol {x}) \right] _ {u} + g _ {b, u}\right) / \tau), \forall b = 1, \dots , B, u = 1, \dots , U \tag {6}
$$
where  $g_{b,u} = -\log (-\log V_{b,u})$ ,  $V_{b,u} \sim \mathrm{Uniform}(0,1)$ , and  $\tau \in (0,\infty)$  is a temperature factor, controlling how "closely" the continuous relaxation approximates the Categorical distribution. In this work, we use a temperature of  $\tau = 0.67$  as suggested in [16], and used in several other works [22, 21].
A.3 Convolutional Formulation
In this setting, local competition is performed among feature maps on a position-wise basis. Each kernel is treated as an LWTA block with competing feature maps; each layer comprises  $B$  kernels. Specifically, each feature map  $u = 1,\dots ,U$  in the  $b^{\mathrm{th}}$  LWTA block of a convolutional LWTA layer computes:
$$
\boldsymbol {H} _ {b, u} = \boldsymbol {W} _ {b, u} \star \boldsymbol {X} \in \mathbb {R} ^ {H \times L} \tag {7}
$$
Competition remains stochastic, and is now implemented on a position-wise basis as follows:
$$
q \left(\boldsymbol {\xi} _ {b, h ^ {\prime}, l ^ {\prime}}\right) = \text {C a t e g o r i c a l} \left(\boldsymbol {\xi} _ {b, h ^ {\prime}, l ^ {\prime}} \mid \boldsymbol {\Pi} _ {b, h ^ {\prime}, l ^ {\prime}} (\boldsymbol {X})\right), \forall h ^ {\prime}, l ^ {\prime} \tag {8}
$$
where  $\Pi_{b,h',l'}(\mathbf{X}) = \mathrm{Softmax}\left([H_{b,1,h',l'},\dots ,H_{b,U,h',l'}]\right)$  comprises the position-wise activation probabilities for all neurons in the block.
For each position in a kernel, only the feature map that wins the said position contains a non-zero entry. This yields sparse feature maps with mutually exclusive activated positions. Now, the output  $\mathbf{Y} \in \mathbb{R}^{H \times L \times B \cdot U}$  is obtained via concatenation of the sub-tensors  $\mathbf{Y}_{b,u}$  that read:
$$
\boldsymbol {Y} _ {b, u} = \boldsymbol {\Xi} _ {b, u} \left(\boldsymbol {W} _ {b, u} \star \boldsymbol {X}\right), \forall b, u \tag {9}
$$
where  $\Xi_{b,u} = [\xi_{b,u,h',l'}]_{h',l' = 1}^{H,L}$ . The corresponding detailed bisection of a convolutional stochastic LWTA block can be found in the Appendix.
In the following, we use these definitions to construct Competitive Vision Networks by replacing the usually employed non-linearities with the competition mechanism in each hidden layer of the considered architecture.
Figure 4: A detailed bisection of the  $b^{\mathrm{th}}$  convolutional stochastic LWTA block. Presented with input  $\pmb{X} \in \mathbb{R}^{H \times L \times C}$ , competition takes place among feature maps on a position-specific basis. Only the winner feature map contains a non-zero entry in a specific position. This leads to sparse feature maps, each comprising uniquely position-wise activated pixels. Image from [23].
B Further Qualitative Analysis
In this section, we provide further qualitative neuron identification results for various architectures and similarity functions.
Neuron 146: CLIP-Dissect (Soft WPMI): motif
Neuron 365: CLIP-Dissect (Soft WPMI): green
Neuron 461: CLIP-Dissect (Soft WPMI): pattern
Neuron 662:CLIP-Dissect (Soft WPMI): blue
Figure 5: Neuron Identification for the first and last DeiT-T/8 MLP blocks using: SoftWPMI,  $\mathcal{D}_{\mathrm{probe}}$ : ImageNet & Broden, S:  $20K$  most common English words[20].
Neuron 6: CLIP-Dissect (Soft WPMI): wildlife
Neuron 32: CLIP-Dissect (Soft WPMI): labradon
Neuron 357: CLIP-Dissect (Soft WPMI):presso
Neuron 450: CLIP-Dissect (Soft WPMI): compass
Neuron 21: CLIP-Dissect (Cos Similarity $^3$ ): stripe
Neuron 672: CLIP-Dissect (Cos Similarity $^3$ ): crimson
Neuron 783: CLIP-Dissect (Cos Similarity<sup>3</sup>) yellow
Neuron 1084: CLIP-Dissect (Cos Similarity $^3$ ): dark
Figure 6: Neuron Identification for the first and last DeiT-S/12 MLP block using the Cosine Similarity Cubed similarity function [20]. For the former we use  $\mathcal{D}_{\mathrm{probe}}$ : (i) ImageNet Val & Broden (Left), and for the latter: (ii) ImageNet Val. We use the  $S$ :  $20K$  most common English words for both settings.
Neuron 21: CLIP-Dissect (Cos Similarity $^3$ ): lobster
Neuron 672:CLIP-Dissect (Cos Similarity):guinea
Neuron 783:CLIP-Dissect (Cos Similarity):eagle
Neuron 227: CLIP-Dissect (Rank Reorder): yellow
Neuron 726: CLIP-Dissect (Rank Reorder): blue
Neuron 1104: CLIP-Dissect (Rank Reorder): mammals
Neuron 1206:CLIP-Dissect (Rank Reorder): blue
Figure 7: Neuron Identification for the first and last DeiT-S/12 MLP block using the Rank Reorder similarity function [20]. We use  $\mathcal{D}_{\mathrm{probe}}$ : ImageNet Val and  $S$ :  $20K$  most common English words for both settings.
Neuron 193: CLIP-Dissect (Rank Reorder): tort
Neuron 224: CLIP-Dissect (Rank Reorder): gentoo
Neuron 726: CLIP-Dissect (Rank Reorder): birding
Neuron 820: CLIP-Dissect (Rank Reorder): antarctic
Table 5: Concepts tied to activated neurons for a random image from the ImageNet validation set for DeiT-T/8 using various similarity functions.
Cos Similarity	WPMI	SoftWPMI
Concepts	Act	Concepts	Act	Act	Concepts
Belgian Malinois	+10.5	ferret	+10.5	ferret	+10.5
green/yellow eyes	+9.55	ferret	+9.55	ferret	+9.55
white fur underside	+8.90	white fur under tail	+8.90	white fur under tail	+8.90
herd Alpine ibex	+8.51	black stripes on legs	+8.51	black/white stripes	+8.51
long/sharp quills	+8.02	long/sharp quills	+8.01	long/sharp quills	+8.01
gray fur/white tips	+7.53	fox	+7.53	fox	+7.53
white/gray head	+7.52	white/gray head	+7.52	white/gray head	+7.52
baby stork	+0.96	black ruff around neck	+1.09	black ruff around neck	+1.09
black fur/white markings	+0.44	black/tan coloration	+0.44	black/tan coloration	+0.44
large/fluffy white dog	+0.35	organ	+0.37	organ	+0.37
gills	+0.18	salamander	+0.18	salamander	+0.18
car towing RV	+0.12	headsail	+0.12	headsail	+0.12
other crocodiles	+0.12	egg-laying mammal	+0.12	egg-laying mammal	+0.12
Table 6: Concepts tied to activated neurons for a random image from the ImageNet validation set for DeiT-T/None and DeiT-S/12 using various similarity functions.
DeiT-T/None (SoftWPMI)	DeiT-S/12 (SoftWPMI)	DeiT-S/12 (Rank Reorder)
Concepts	Act	Concepts	Act	Act	Concepts
a coast	+1.18	sea turtle	+15.0	sea turtle	+15.2
hard/shiny exoskeleton	+1.06	cygnet	+11.7	large/webbed hind feet	+11.7
sea turtle	+0.54	large/elephant animal	+10.9	dark-colored carapace	+10.9
large/red crab	+0.54	hard shell covered in spines	+9.96	large/fleshy cap	+9.96
large/red crustacean	+0.53	sea turtle	+9.52	marine ecosystem	+9.52
large/plant eating dinosaur	+0.51	python	+9.32	tough/scaly exterior	+9.32
short/bristly fur	+0.46	doberman	+9.08	black/tan coloration	+9.08
white bill	+0.20	dome-shaped cap	+1.14	round/spouted body	+1.41
a race	+0.17	iguana	+0.63	fluffy/white appearance	+0.33
large/fluffy white dog	+0.06	other crocodiles	+0.67	operating system	+0.22
two-metal plates/trays	-0.16	case	+0.22	opponent	+0.10
white markings wings	-0.16	players	+0.10	wings attached	+0.10
small/crab like body	-0.17	exoskeleton	+0.01	hard/shiny exoskeleton	+0.10