Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: The authors propose an explainable deep learning model that uses a hierarchical voting strategy to perform hierarchical feature selection.
main_review: raises predictive performance, decreases feature combinations, supports efficient pruning and feature analysis.

"humans usually aggregate information to infer intermediate concepts step-by-step." This should have a citation.  Really a lot of the information in the first paragraph of 2.1 needs justification through citation.  

"That is, while deep neural networks make the decision according to a unified, inseparable complicated function, humans usually aggregate information to infer intermediate concepts step-by-step"  Is this not what many convolutional networks have been shown to do in recent work on interpretation and explainability?

How does VOTEN constrain the model to capture indidual concepts in each module?

In equation 1, why replace C with x?  Is C_d+1 a weighted sum of the output of f(C_d)?  It is not clear at this point in the paper what the function "f" does.  By theorem 1 this is the key difference from a regular MLP so this should be defined sooner.

In general I am not sure what function "f" is in equation 1 or how voting occurs.  Is voting done by outputing a weight for each concept that are then summed to form a new concept?

The approach described in section 3.3 is not very clear and seems difficult to generalize.  Can the authors provide a clear high level description of this process as it would generalize to other settings?

I am not convinced at using correlating decision paths and model activations for explainability as there are many variables to consider.  For the subway example, are thos paths possibly attributed to some other co-ocurring variable with subway in the data?

A related work section should be added to address how this approach relates to other work in network explainability.
limitations_and_societal_impact: Ye
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a novel type of model, Voting Transformation-based Explainable Neural Network (VOTEN), which represents the transformation from the input to the output to be a hierarchical voting process. This simple structure makes the model interpretable for a human. The papers systematically analyses the advantages of VOTEN, in particular the model's explainability (global and local) and reduced complexity (pruning).
main_review: I am very much at odds with myself when it comes to this paper. On the one hand it aims to solve a very relevant problem. Through it specific structure VOTEN is relatively easy to interpret while obtaining quite good accuracies, significantly better than e.g. simple decision trees. The experiments presented in the paper are very nice and cover different aspects, e.g., performance, local and global explainability and complexity.
On the other hand I am not fully convinced about the proposed model. First, I find all the discussions claiming that VOTEN is very similar to human reasoning very strange and not appropriate. The authors do not present any results, which support this claim. Furthermore, in the experimental section VOTEN is only applied to relatively simple problems. The big strength of deep neural networks (this to some degree also hold for models such as random forests, SVMs etc.) is that they also perform well on complex, high-dimensional problem. I am not sure that this also holds for VOTEN. The true practical abilities of VOTEN are hard to assess, also because the selected datasets are not the typical benchmark datasets for e.g. GMBs or RandomForest classifiers. Furthermore, there is no comparison with post-hoc explanation methods such as LRP or SHAP applied to a deep neural network or e.g. the work of Lundberg computing Shapley values on tree-based models. Thus, in practice it is unclear how much insights the explanations provided by VOTEN really bring. Also there is no comparison or extensive discussion of the proposed model to the work of Agarwal et al. 2020. Thus, the advantages VOTEN over sota remain unclear.
Overall, because of these limitations I am not fully convinced about VOTEN and therefore see the paper lying below the acceptance threshold.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a novel network structure for improving the explainability of deep learning models. The authors theoretically analyzed their model and proved with abundant experiments that the proposed model is effective both in terms of explainability and model performance. 
main_review: Generally speaking, this is a well-written paper, and I enjoyed reading it. The proposed model is introduced well in a convincing and insightful way. 

The authors give straightforward analysis to show the priority of their model in terms of explainability. Then, they conducted extensive experiments to show the effectiveness of their model. I can see the authors' efforts to improve the usability of their model by providing supportive algorithms and tools. The case studies and visualizations are impressive and show human-understandable inherent classification process of the model. Therefore, I think this work can be a powerful and explainable off-the-shelf solution for classification tasks in many real-world fields. 

It is impressive that, while there is usually a trade-off between model explainability and accuracy, the authors have innovatively proposed a new solution to achieve higher performance while assuring high explainability. The authors claimed their model to be generic and can be easily integrated with existing deep learning structures and showed its priority over MLP. Considering more and more MLP-based deep learning models have been achieving SOTA performance in many fields these years, it is exciting to see a powerful substitution with high explainability, which potentially benefits a wide range of deep learning applications.

Still, just like the authors have mentioned in the limitation section, I think the significance of this work can be further enlarged if the authors improve existing task-specific MLP-based models with the proposed structure. I expect the authors in their future work to explore more potentials applications of their model. In addition to models for recommender systems the authors have mentioned (i.e., deepFM and MMoE), models for Computational Vision (e.g., MLP-Mixer) may also be considered. 

Besides, the authors have submitted the code for review as supplementary information. Considering its potential influence, I urge the authors to release their code to public platforms as soon as they publish this paper.

The advantage and disadvantage of this paper are summarized as follows:

Advantage:
1. The authors provide a powerful and explainable network structure. The proposed model is generic and can be easily integrated with existing deep learning structures. 
2. Impressive experimental results are given to prove the model's effectiveness and explainability. 
3. The authors provide a relatively complete workflow for their model's practical use, with carefully designed algorithms and tools for quantifications and visualizations.

Disadvantage:
1. The presentation of this paper needs to be improved. There are typos in the paper, and texts in several figures are small. 
2. The significance of this work can be further enlarged if the authors improve existing task-specific MLP-based models with the proposed structure. 

limitations_and_societal_impact: The authors have properly discussed the limitations of their work and discussed the future works for improvement. I look forward to their future works.
I cannot see potential negative societal impact of this work.
review_title: null

Meta-review: This work proposes a novel network structure for better explainability. While there are many aspects of explainability that are not assessed in this paper, the local and global prediction processing cases are provided beside the accuracy. The term decision-making is a bit confusing as it often refers to decision-making tasks, while this paper handles prediction tasks. Overall, this paper is significantly novel, while its applicability is not quite clear. 
Decision: Accept (Poster)