Title: DISCOVER: Making Vision Networks Interpretable via Competition and Dissection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpretability

Explainability

Network Dissection

Competitive Networks

Sparsity

Multimodal Models
Abstract: Modern deep networks are highly complex and their inferential outcome very hard to interpret. This is a serious obstacle to their transparent deployment in safety-critical or bias-aware applications. This work contributes to *post-hoc* interpretability, and specifically Network Dissection. Our goal is to present a framework that makes it easier to *discover* the individual functionality of each neuron in a network trained on a vision task; discovery is performed in terms of textual description generation. To achieve this objective, we leverage: (i) recent advances in multimodal vision-text models and (ii) network layers founded upon the novel concept of stochastic local competition between linear units. In this setting, only a *small subset* of layer neurons are activated *for a given input*, leading to extremely high activation sparsity (as low as only $\approx 4\%$). Crucially, our proposed method infers (sparse) neuron activation patterns that enables the neurons to activate/specialize to inputs with specific characteristics, diversifying their individual functionality. This capacity of our method supercharges the potential of dissection processes: human understandable descriptions are generated only for the very few active neurons, thus facilitating the direct investigation of the network's decision process. As we experimentally show, our approach: (i) yields Vision Networks that retain or improve classification performance, and (ii) realizes a principled framework for text-based description and examination of the generated neuronal representations.


Peer reviews:
comments: SUMMARY:
This paper proposes a Dissection of Competitive Vision Networks (DISCOVER) to build interpretable neural networks. It aims to understand the neuron functionality for image inference. Moreover, this paper introduces the Jensen-Shannon divergence between the probability of concept presence and the probability of neuron activation for network dissection. Experimental results evaluate the effectiveness of the proposed CVNs.

STRENGTHS:
1. Relatively interesting idea of the proposed CVNs for network dissection. 
2. high interpretability of the proposed framework evaluated in experiments.

WEAKNESSES:
1. Limited novelty. DISCOVER method is similar to DMoE [ref-1] combined with the work [ref-2]. Please clarify their difference.
2. Lack of more visualization results to evaluate the interpretability over other methods.

[Ref-1]. Deep mixture of experts via shallow embedding. Uncertainty in artificial intelligence. PMLR, 2020.
[Ref-2] Interpretable Neural Network Decoupling. In ECCV, 2020.

QUESTIONS:
Refer to the Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use stochastic local winner-takes-all (LWTA) layers in vision networks combined with CLIP-Dissect to improve the ability of a neuron to capture a concept. A stochastic LWTA layer groups pre-activation neurons into $B$ blocks of $U$ features each and within each block, chooses one feature with probability according to the softmax of the pre-activation neurons in the block. The chosen feature is preserved while the rest are zeroed out.\n\nThe paper tests DeiT-T, DeiT-S, and ResNet-18 architectures on ImageNet and Places365 replacing original activation functions with stochastic LWTA. The classification performance slightly improves when each block is of size 2, or $U=2$, and degrades as $U$ increases.\n\nThe stochastic nature of LWTA allows the authors to use CLIP-Dissect with the Jensen-Shannon Divergence (JSD), which they do. The learned concepts for the different investigated neurons, such as “green,” “red,” “trains,” and “snowmobiles” match the images those neurons are activated for. The text embedding similarity of found concepts to ground truth ImageNet label is higher with LWTA neurons and JSD than normal neurons and WPMI.\n\nSTRENGTHS:\nThe presentation is clear and the motivation in the opening sections is well-expressed. Also, extensive code is provided to support the approach.\n\nThe idea of introducing competition between neurons and having one win out seems intuitive and suited for interpretability, the stated goal of this paper.\n\nThe results are promising (the introduced module does not harm performance too much, and potentially increases performance if a very mild form of it with $U=2$ is used, and some quantitative evidence for superior interpretability is provided).\n\nWEAKNESSES:\nIt’s not fully clear whether the approach is being presented as an ante-hoc method or a post-hoc method. It seems one could argue that needing to train a network from scratch with the LWTA nonlinearity makes it an ante-hoc method. It also seems one could argue that since LWTA is a drop-in replacement for any non-linearity, it functions more as an enhancement to any ante-hoc or post-hoc methods applied on top of the architecture rather than an ante-hoc/post-hoc method itself. Could the authors clarify this?\n\nThe evaluation could be more thorough and more convincing. Here are some suggestions, not all of which might be necessary:\n- Significant decline in performance is stated as a drawback of other ante-hoc methods, which this method is arguably an instance of; is it possible to compare classification performance with one or more baseline ante-hoc approaches? \n- Can you provide some sort of comparison between $U$ and the interpretability of the proposed approach? Such as, a plot of $U$ vs. cosine similarity between intermediate neuron description, or comparing concepts found with $U=2$ to $U=16$. Since $U=2$ achieves the best performance, it’s not clear why we would prefer $U=16$. The argument that $U=16$ achieves higher sparsity and thus higher interpretability intuitively makes sense, but is this supported empirically?\n- In my opinion, an additional interpretability metric to evaluate the potential of CVNs for interpretability purposes would improve the paper. Would it make sense to use the IOU-based interpretability metric from the Network Dissection paper? If that would require too much additional experimentation, is some sort of more lightweight added experiment/comparison possible?\n\nMiscellaneous typos/comments:\n- L177: Doesn’t $\\mathrm{CE}(y_i, f(X_i, \\hat\\xi))$ denote the cross entropy, not $f(X_i, \\hat\\xi)$\n- L340: “the resulting networks yielded on par or even improved performance, even when using up to 4% of active neurons per example” seems a bit of a stretch; going from original network to most sparse competition-based network for DeiT-T on ImageNet-1k, the accuracy dropped from 72.2 to 70.5, for ResNet18 on Places365, from 53.5 to 49.5. I would call these drops significant. I would call the drop of DeiT-S on ImageNet-1k from 77.0 to 76.7 “on par”, but not improved.\n\nQUESTIONS:\nI put the most relevant questions in the “Weaknesses” section.\n- In L228, it is claimed “In the context of inner-product neurons, we directly obtain the presence probability via Eq. (2).” But aren’t vision transformer neurons also spatially arranged, like convolutional neurons? I.e. for a given feature there are different instances of it which corresponding to a spatial map, corresponding to the different embedded patches. So do you also average over spatial positions for transformer?\n\nLIMITATIONS:\nLimitations were adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for interpreting vision networks by combining an architecture that incorporates stochastic local-winner-takes-all activations with the CLIP-DISSECT framework. The author makes use of the probabilistic interpretation of LWTA as a categorical distribution to derive a similarity measure for LWTA activations based on the Jenson Shannon Divergence. They show the benefit of their approach by comparing the discovered interpretations to ground truth labels for conventional and competitive (LWTA) networks.\n\nSTRENGTHS:\nThe studied problem of interpretability is very important, the idea is well motivated, the background and related work is informative, and the writing is clear.\nThe accuracy of the LWTA adapted transformer is very encouraging and suggests the usefulness of competition independently of interpretability. \nThe improvements in automated neuron description, are encouraging.\n\nWEAKNESSES:\nClarity could be improved: The figures are not very helpful. Especially fig 1 could be improved by simplifying it  (e.g. without showing individual neurons and weights). In my opinion Fig 2 is not necessary and could instead make space for a much needed figure to illustrate the DISCOVER framework.\nThe math feels a bit convoluted with too many new symbols introduced in each paragraph. Again, simplification could help understandability.\nThe paper spends quite time explaining how LWTA and DISCOVER can be used for convolutional networks. However, the main results of the evaluation focus on transformer architectures for which it is less clear how LWTA is incorporated (only replacing the ReLU inside the MLP block or also in the attention and residual branch?). I suggest switching focus and making conv architectures much less prominent.\n\nThe evaluation feels incomplete. The main evidence for the efficacy of DISCOVER are:\n1. the qualitative results in Figure 3, which are only partially convincing. Eg. the \"aquaculture\" neuron seems to react to a highway, and the \"dashboard\" neuron mostly picks up on hifi equipment.\n2. Table 2, which shows that, compared to non-LWTA-networks) the textual embedding of the discovered neuron descriptions is more similar to the corresponding ground truth label. However the CLIP cosine similarity score is hard to interpret and the reported gains seem mild.\nFor a paper that focuses on interpretability, I find the results hard to interpret. As it stands, the paper has me convinced that stochastic LWTA is interesting and should be studied further, but fails to convince me of the added value of DISCOVER over e.g. CLIP-dissect.\n\nQUESTIONS:\nHow exactly is LWTA incorporated into the transformer?\n\nLIMITATIONS:\nThe paper does discuss two relevant limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DISCOVER, a novel framework for creating interpretable vision networks. It utilizes stochastic Winner-Takes-All layers and multimodal vision-text models to uncover the specialization of each neuron. The framework achieves high activation sparsity, allowing for direct examination of concept representations. It also introduces a new similarity metric for network dissection based on Jensen Shannon Divergence. Experimental results demonstrate improved classification performance and provide a principled framework for text-based descriptions of neuron representations. The paper pioneers the training of LWTA networks on large datasets, yielding interpretable functionality with a small subset of neurons per datapoint.\n\nSTRENGTHS:\nThe strengths of this paper include:\n\n1. Novel Framework for Network Dissection: The paper introduces a novel framework, DISCOVER, for creating interpretable vision networks. It combines stochastic Winner-Takes-All layers and multimodal vision-text models, providing a unique approach to uncovering neuron specialization.\n\n2. Improved Classification Performance: Despite not performing hyperparameter tuning, the DISCOVER framework achieves comparable classification performance. \n\n3. Interpretable Similarity Metric: The paper introduces a new similarity metric based on Jensen Shannon Divergence to match neuron representations to concepts.\n\n4. Pioneering LWTA Networks: The paper contributes to the literature by being one of the first to train large-scale LWTA networks using substantial datasets and architectures.\n\nWEAKNESSES:\n1. The WTA approach in this neural network framework draws inspiration from previous methods employed before the advent of deep learning. For instance, techniques like bag of words, Locality-constrained Linear Coding (LLC) (Wang et al., \"Locality-constrained linear coding for image classification,\" CVPR, 2010) and a long etc, also involved neuron competition for activation. \n\n2. It would be valuable to have a clearer understanding of the level of sparsity in the network without WTA. While the paper mentions that in a network without WTA sparseness is \"input specific,\" providing some statistical analysis or comparisons of the sparseness level with non-WTA networks could provide insights into the necessity and effectiveness of the WTA mechanism.\n\n3. The paper assumes that sparse representations lead to more interpretable units, but the connection between sparsity across neurons and human interpretable concepts needs further clarification. It would be beneficial to elaborate on the theoretical arguments or underlying assumptions supporting this claim. Additionally, providing empirical evidence or previous research that supports the notion that sparse representations indeed enhance interpretability would strengthen the argument.\n\n4. The qualitative results provided in the paper are limited, particularly regarding the comparison between applying the WTA mechanism and not applying it. It would be insightful to include examples of units that are not interpretable when the WTA mechanism is not utilized. This comparison would help illustrate the added interpretability that the WTA approach brings to the network and provide a more comprehensive understanding of its impact on the model's performance.\n\n5. The dependence on the interpretability similarity metric raises questions about the generalizability of the results. While the paper introduces a new similarity metric based on Jensen Shannon Divergence, it would be valuable to investigate the robustness of the findings using alternative metrics and visualization techniques. Demonstrating the consistency of the results across different evaluation methods would strengthen the validity and generalizability of the proposed framework for increased interpretability. Exploring the impact of alternative metrics and visualization techniques could provide a more comprehensive understanding of the interpretability achieved by the network. \n\nBy addressing these concerns, the paper can further enhance its contributions by providing more insights into the necessity and effectiveness of the WTA mechanism, establishing theoretical justifications, and expanding the qualitative analysis to better understand the interpretability aspect.\n\nQUESTIONS:\nAs I mentioned earlier, there is a concern regarding the lack of clear evidence establishing a direct relationship between sparsity and interpretability. While the paper assumes that sparse representations lead to increased interpretability, it would be beneficial to provide results that demonstrate this relationship across different levels of sparsity, utilizing various interpretability measures. By systematically examining the impact of different sparsity levels on interpretability using diverse evaluation metrics, the paper could provide stronger evidence for the claim that sparsity enhances the interpretability of individual units. This analysis would contribute to a deeper understanding of the relationship between sparsity and interpretability and provide more robust support for the proposed framework.\n\nLIMITATIONS:\nI believe the limitation section could take into account the points I raised in weaknesses, as they were not mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}

comments: SUMMARY:
The work proposes a interpreatable network that uses concepts from competition networks specifically there are for a specific output multple copies of the neuron and the neurons compete to provide the winning activation for that output. The method proposes MLP and  convolutional versions of their network. The work also provides evidence of interpreatability by using CLIP-DISSECT and shows that their method is more interpreatable while using same number of parameters as DiET.

STRENGTHS:
1. The method is quite simple and elegant in its motivation

2. Existing modules can be swapped with the proposed block without any changes

WEAKNESSES:
1. The method is essentially a stochastic version of maxpooling where the deterministic version is exactly maxpooling. Is there an ablation to show how the stochastic version is more appropriate than a deterministic one?

2. Having 2 competetors seems to provide the best performance whereas using more decreases the performance of the network. Is this because of using less "effective neurons" (as the model size kept same) or with more competition the network inherently starts degrading? If it's the second case than it does not put competition networks in a good light. 

3. It seems wasteful that with more competition exponentially lower number of neurons in the model are activated. 

5. Can it be used in networks for other tasks? Segmentation, object detection etc.?

QUESTIONS:
See the weakness.

1. Gumbel softmax generally requires a bit of finetuning. How stable is the gumbel softmax training on different datasets?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27