Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to train a sparse neural network that only utilizes a small portion of all possible input features per class. The paper argues that due to a small number of input features, they are more interpretable. Another main highlight is that despite using only 10% of the features, the accuracy of the tested network remains relatively stable (97-100%). The paper shows that the small number of features that the model uses to make decisions can also be easily aligned with human-nameable concepts for additional explainability. 
strength_and_weaknesses: **Strengths:**

*[S1]:* Surprisingly effective results: The main highlight of the paper for me is the fact that despite using very few features, the proposed method can achieve such good accuracy results. 

*[S2]:* Clearly presented: The paper is well-written and easy to follow. The experiments are described clearly and the main techniques used are adequately clear.


**Weaknesses:**

*[W1]:* Unclear why the proposed method is better for anything: The paper does a poor job of motivating exactly why a model using few features is more interpretable. I attempt to understand it from several angles, but do not find a satisfactory answer. 

- Possibility 1: Since there are few features, they can be visualized easier: I'm not satisfied because now we are talking about the efficacy of visualization techniques, many of which can map complex decisions to simple-to-understand visualizations (Grad-CAM, etc..)
- Possibility 2: Fewer features can be easily aligned with the human-nameable concept: I do not understand why the analysis in Fig 3b and Fig 4 can also not be done for traditional networks. In the end, we would only be left with features that correlated with the concepts anyways; Why does it matter whether we run them on 5 or 50 features to start with? If more than one feature correlates with a single concept, they can then simply be visualized as a "group". 
- Possibility 3: Fewer features can show us what was really important: There are a lot of ways to rank the "importance" of features in neural networks. Why isn't ranking the features by feature importance enough? Why do we need to train a network to only make decisions using these features?

For all of those, there is a direct actionable item for the authors to show; i.e., show us the results of the alternatives and how the proposed method fares better. 

*[W2]:* Limited Novelty: The proposed work is a fairly straightforward adoption of glm-saga; While this would absolutely not be a problem on its own, the fact that there is also not sufficient analysis (W1 above) makes it harder to overlook the fact that the paper does not present "previously unknown" technique that others might use differently. 
clarity,_quality,_novelty_and_reproducibility: **Clarity:** To me, it is quite clear, See S2 above.

**Quality:** See strengths and weaknesses above.

**Novelty:** There is a limited novelty in both the techniques used and the empirical evidence (and discussion).

**Reproducibility:** The method is built on existing models and a code is also promised, I think reproducibility is excellent. 
summary_of_the_review: While it was interesting to read, I found myself having critical questions about the main motivations of the paper by the end. The paper hinges on two main things: A network that is "inherently interpretable" is better than interpretability tools built on top of traditional networks. And the second is that fewer features lead to better interpretability. As mentioned above in W1, I am not sure about either of those claims and the paper does not show enough proof that it is true. So, despite interesting results (only using a few features gives good accuracy), this is a sub-par paper for me looking at it primarily from an interpretability angle. The paper isn't really written from a feature-compressive perspective, So, I am also unable to comment on that.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed model that makes predictions from a sparse set of features. To do so, it employs a novel diversity-based loss and a feature selection process. The model is shown to achieve competitive performance from only very few features (e.g. 5). Empirical analysis on the learned features is able to map them to human-understandable semantic concepts. 
strength_and_weaknesses: Strengths: 

1. The paper differs from most vision interpretability papers in proposing a self-explaining model rather than focusing on post hoc explanations. 
2. This paper proposed a very sparse model that achieves very good performance, using a novel loss formulation. 

Weaknesses: 

1. My key concern is the analysis of the learned features. First, the global analysis seems to require dense attribute-level annotations, such as what's found in the CUB dataset. However, these attributes are expensive to collect and not often provided.
2. A hidden assumption behind the above analysis is that the model does use these semantic attributes for decision making. This seems to be a strong assumption, as various works [e.g. 1, 2] found that neural networks can use various spurious correlations and shortcuts, and furthermore many popular explanation methods are not well-equipped to detect such usage. I would appreciate any discussion on this aspect. 
3. The manual feature alignment on the four-engine attribute seems a bit too ad hoc. There could be many different explanations, such as the color or style of the plane, that result in the heatmap focus. Such statements should be made more quantitative, and even better, falsifiable (e.g. does removing two engines affect the heatmap?). 
4. The description on the diversity loss function moves quite fast, especially for readers not familiar with MCL and FRL. The same goes with the feature selection, which assumes prior knowledge of glm-saga. 

[1] https://arxiv.org/abs/2004.07780

[2] https://arxiv.org/abs/2104.14403
clarity,_quality,_novelty_and_reproducibility: See above. 
summary_of_the_review: Overall, I think this paper is on track for making a good contribution, but the interpretability analyses feel a bit thin. While I would empathize if the authors feel that they may not be necessary because the model is "self-explaining", I would push back with [3], which argues to the contrary. 

[3] https://aclanthology.org/2020.acl-main.386/
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides a training strategy by applying sparsification to obtain a more interpretable network. The training loss combines the feature diversity loss and sparsification loss used in Glm-saga. The measure of feature diversity is proposed and the paper shows that classes with the diverse feature can improve the accuracy. 
strength_and_weaknesses: 
Strength:

Regularizing the sparse feature for a more interpretable network is a doable method. 

Weakness:

1. The paper is not easy to follow with missing details and an unorganized structure. For example, $\lambda_1, \lambda_2$ is not explained in Section 3.2.2. In Section 3.2.1, $L_{final}$  is defined as diversity loss plus cross-entropy, while in Section 3.3 $\mathcal{L}_{target}$ is cross-entropy alone. 

2. The main contribution of the paper is not clear. For example, the claimed contribution includes:

a). Feature diversity loss: Such loss is widely explored in the existing literature like MCL, FRL, CCMP. It is unclear why the proposed $\mathcal{L}_{div}$ is better. 

b). Feature sparsification loss: It is a direct follow-up of glm-saga. And the sparsification in the final layer leading to better interpretability was also investigated. 

c). Measure of interpretability: The measurement in this paper is coarse by counting the dimension of the feature. A low dimensionality does not necessarily lead to better interpretability. For example, one can inject an identity matrix after the final FC layer. The identity matrix is fully sparsed, but meaningless. Therefore, a better measurement would be the one-to-one correspondence between a concept and a single feature vector. Such measurement is already well-explored in Net-dissection (Bau et al., 2017).
clarity,_quality,_novelty_and_reproducibility: See Strength And Weaknesses. 
summary_of_the_review: This paper targets an interesting problem. However, the contribution of the paper is not clear. Before having a better understanding of the challenges that this paper resolves, I tend to vote for a reject recommendation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new model intepretability technique. The main idea is creating a final sparse layer composed of interpretable features using a loss function formulation that promotes diversity and sparsity. The technique is tested on different architectures and datasets, with a particular focus on fine grained classification.

The reviewers remarked the good accuracies obtained with the sparse features. The reviewers also pointed out some aspects on the paper presentation that needed improvement (e.g. order of the sections was confusing; the descriptions of the diversity loss function and the Glm-saga need more details), which were addressed by the authors in their feedback and in the reviewed version of the paper. However, there are concerns about this paper that still hold after the authors' feedback: (1) limited novelty with respect Glm-saga; (2) limited interpretability capactiy of the proposed approach; (3) need for a deeper understanding on why the proposed method is better with respect to previous approaches. Overall, the reviewers think the work has an interesting direction but it's not mature enough for being published yet. We recommend the authors to follow the recommendations to extend the analysis of the results.

As a minor comment, the current version of the paper is more than 9 pages in length, which exceeds the page limit for ICLR papers (this has not been taken into account for the final decision).
justification_for_why_not_higher_score: The direction of the paper is interesting. However, all the concerns raised by the reviewers suggest that the paper is not mature enough to be published.
Decision: Reject