Discerning Decision-Making Process of Deep Neural Networks with Hierarchical Voting Transformation
Authors: Ying Sun
Hengshu Zhu
Chuan Qin
Fuzhen Zhuang
Qing He
Hui Xiong
Venue: NeurIPS 2021 Poster (2021)
Keywords: Explainable Neural Networks

Abstract: Neural network based deep learning techniques have shown great success for numerous applications. While it is expected to understand their intrinsic decision-making processes, these deep neural networks often work in a black-box way. To this end, in this paper, we aim to discern the decision-making processes of neural networks through a hierarchical voting strategy by developing an explainable deep learning model, namely Voting Transformation-based Explainable Neural Network (VOTEN). Specifically, instead of relying on massive feature combinations, VOTEN creatively models expressive single-valued voting functions between explicitly modeled latent concepts to achieve high fitting ability. Along this line, we first theoretically analyze the major components of VOTEN and prove the relationship and advantages of VOTEN compared with Multi-Layer Perceptron (MLP), the basic structure of deep neural networks. Moreover, we design efficient algorithms to improve the model usability by explicitly showing the decision processes of VOTEN. Finally, extensive experiments on multiple real-world datasets clearly validate the performances and explainability of VOTEN.

Discerning Decision-Making Process of Deep Neural Networks with Hierarchical Voting Transformation
Ying Sun 1,2,5,t Hengshu Zhu Chuan Qin?, Fuzhen Zhuang Qing Hel.5, Hui Xiong'
IIP, Institute of Computing Technology, Chinese Academy of Sciences Baidu Talent Intelligence Center, Baidu Inc.
3 Institute of Artificial Intelligence, Beihang University
Artificial Intelligence Thrust, The Hong Kong University of Science and Technology
University of Chinese Academy of Sciences
SKLSDE, School of Computer Science, Beihang University
(sunying174 heqing) @ ict.ac.cn, (zhuhengshu, chuanqin0426 @ @gmail.com, zhuangfuzhen buaa.edu.cn, xionghui @st..
Abstract
Neural network based deep learning techniques have shown great success for numerous applications. While is expected understand their intrinsic decision- making processes, these deep neural networks often work in a black-box way. To this end, in this paper, we aim to discern the decision-mant makinn processes of neural networks through a hierarchical voting strategy by developing an explainable deep learning model, namely Voting Transformation-basee work (VOTEN), Specifically, instead of relying massive feature combinations, VOTEN creatively models expressive single-valued voting functions between ex- plicitly modeled latent concepts to achieve high fitting ability. Along this line, we first theoretically analyze the major components of VOTEN and prove the relation- ship and advantages of VOTEN compared with Multi-Layer Perceptron (MLP), the basic structure of deep neural networks. Moreover, we design efficient algorithms improve the model usability by explicitly showing the decision processes of VOTEN. Finally, extensive experiments on multiple real-world datasets clearly validate the performances and explainability of VOTEN.
Explainable Neural Net-
1
Introduction
Neural network based deep learning techniques have attracted great attention from both academia and industry in the past decade. Compared with classic machine learning models, deep neural networks have much higher expressiveness and adaptability for complicated data input, and thus have made tremendous success in various application domains, such as Computational Vision 22 (6], Natural Language Processing [54] [20], and Recommender Systems 332 244. Nevertheless, since deep neural networks usually have complicated connections of hidden units, a mongsstanding challenge is how to decipher what's inside the black box of models for understanding their intrinsic decision -acciig
processes. Indeed, in many real-world scenarios, such as business analysis 33 57 49 and human
resource management [50 46 44,, the lack of model explainability makes people less likely to be convinced when the decision-making process of the model is not understandable. This prevents a broader application of deep neural networks.
While many research efforts have been made developing explainable deep learning models 45
most of existing works focus post-hoc explanation [5] [8], i.e., designing metrics to measure the feature relevance/contrriutioo the outputs of a trained model. Although these methods have
Corresponning authors.
Thhis work was accomplished when Ying Sun was intern in Baidu supervised by Hengshu Zhu.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Voling Layer
Voting Layer 2
Voting Layer
Relevance Analysis
Decision Path Recognition
Model Pruning
Decision Score
Concept
Observation
Voting Channel
Counting Layer
Figure
Structure overview of VOTEN.
made progress finding important features, the secision-making process of deep learning models still not available for users. For example, users often need to guess the reason why prediction is made from the relevant features instead of understanding the decision-making process. Indeed, understanding the intrinsic decision-making process of deep neural networks is a non-trivial task. A major reason is that deep neural networks usually involve massive feature combinations to gain expressiveness on fitting complicated functions. During this process, the effect of features and hidden units may be largely coupled with each other. This indicates that the decision logic of models is inherently buried in the massive feature combinations. Meanwhile, it is difficult for human to understand the intrinsic modeling process of deep learning in a natural manner. Therefore, a key point on improving the explainability of deep neural networks is to decouple the feature combinations and make the modeling process consistent with human decision process. In this way, the model will have an explicit decision-making process and become human -umnan-uddertaadaable....
To this end, in this paper, we propose explainable deep learning model, namely Voting Transformation- based Explainable Neural Network (VOTEN). Specifically, VOTEN assumes the transformation from the input to the output to be hierarchical voting process. During this process, lower-level concepts vote for higher-level concepts layer-by-layer in an expressive but explainable way. Instead of relying massive feature combinations, VOTEN creatively models expressive single-valued voting functions between explicitly modeled hidden concepts to gain expressiveness on fitting complicated functions. This process is explainable for its consistency with human decision- making process. We first theoretically analyze the major components of VOTEN and prove the relationship and advantages of VOTEN compared with Multi- Lulti-Layer Perceptron (MLP), which the basic structure of deep neural networks. The results show that MLP can be derived from VOTEN by using the inexpressive voting functions. Accordingly, we further analyze the effect of inherent votings and design efficient algorithms for pruning and explaining VOTEN. Finally, we evaluated VOTEN on multiple real-world datasets with comprehensive experiments. The experimental results demonstrate that VOTEN generally promotes powerful and explainable deep learning. Specifically, VOTEN (1) significantly raises prediction performance; (2) exponentially decreases feature combinations; and (3) supports efficient pruning and effective feature analysis. Mcanwhile, we also visualize the intrinsic decision decision-making process of VOTEN through case studies, which show VOTEN is explainable and can discover meaningful latent concepts.
VOTEN
In this section, we introduce the technical details of VOTEN.
2.1 Structure
When dealing with complicated information, we usually aggregate them step-by-step to form compli- cated inference. Although deep neural networks also extract abstract features aayer-by-layee they still difficult to be explained. A major difference is that makes us able to explain our inference in a straightforward way.
induce explicit concepts, which
For example, we can predict that "This student may get high GPA because he/she works very hard". If the more explicit explanation needed, we can further explain that observed that he/she stay long in the library". In this process, with the observation that "stay long in the library", first

---

infer Thard-working" then infer "high GPA". Moreover, when trying to make more comprehensive inference, we may get more observations for existing concepts (e.g., "little missing of classes" may also imply "hard-workingg), or induce more concepts (e.g.. "learnin-eefficinnyy process straightforward and conforms with our way of understanding. However, deep neural networks cannot model concepts in an explicit way. Instead, concept information are embedded in massive hidden units. As we cannot understand these induced concepts, neural networks are like unified, inseparable complicated functions, even though they are actually performing information aggregation. Along this line, we believe a natural way to understand how a model decides the output is to make the intrinsic concepts explicit for human.
This kind of
Therefore, we propose VOTEN, whose structure overview is shown in Figure I VOTEN hierar- chically models a small number of explicit concepts. During inference, the higher-level concepts will be induced via information aggregation from the lower-level concepts. To achieve meaningful information aggregation, VOTEN focuses on quantifying relationship between individual concepts of different levels. For better understanding. in VOTEN, observations and concepts can be regarded as voters. Based their own value, each voter independently votes for the higher-level concepts. The votes are aggregated to get the value estimation of each higher-level concept. These concepts will further vote for the next level. In the training process, the model learns to builds intermediate concepts and their quantitative voting functions.
Formally, for VOTEN model with D levels of concepts, we use Cd to denote the i-th concept in the d-th layer, where C. denotes an imput feature. We refer to transformations between adjacent levels of concepts voting layer. In the d-th voting layer, each concept C₁ votes for each higher-level concept Cd+1 with an independent voting channel Vig. Vi takes the value of CØ as the input and votes with single-valued nonlinear function fi, IR + R. Then, a counting layer gets weighted average of the votes and estimates the value of e as
s.t. Vd.
d a
=1
(1)
R=1
where nı denotes the number of concepts in the d-th layer, á denotes the value of C₁ o denotes the weight of V The concepts of the last layer is regarded the decision score, which generates the model output with o = pVOTEN (aP). In particular, to assure the ability of the voting functions on effective concept transformation, VOTEN models each voting function with voting network. The voting network can be designed in complicated ways without influencing model explainability, as long the function is still single-valued. For example, we can adopt weight-sharing structures to reduce model complexity.
2.2 Why is VOTEN more explainable than MLP?
In this part, theoretically discuss VOTEN's advantages MLP. Specifically, claim that voting expressiveness is the core proposition of VOTEN that raises explainability.
Theorem MLP can be derived from subset of degenerated VOTEN models whose voting functions in the form of fi (x²) w (x+) + bi. where a is a predefined activation function, the scalars w b9, IR are trainable parameters.
Proof. Please refer to Appendix A.
From this point of view, MLP also conforms to human inference. However, MLP is still difficult to explain. Indeed, we can more easily understand an inference process with (1) fewer concepts, (2) shorter concept transformations, and (3) fewer reasoning patterns. Under VOTEN schema, we show how inexpressive voting channels make MLP disobey these three traits.
Corollary In MLP, votings from the same concept are linearly correlated.
This means individual voters MLP are weak in distinguishing different concepts in the next level. Therefore, MLP relies on highly complicated feature combinations of large number of deeply tiled voters to achieve high fitting ability. During this process, necessary intermediate information is inherently modeled through combinatorial effects of hidden concepts with inexplicit meanings. Indeed, previous works have proved that human-understandable concepts are inherently mounted in the hidden units of neural network models (28).
3

---

Corollary 2 In MLP, the hypothesis space for voting distribution is limited to scaling and shifting an input distribution.
This means that MLP voters cannot always induce complicated concepts that have different distri- bution from their value. As result, the input needs to go through long path of transformations between similar concepts until contributes to the output. Moreover, the deeply tiled massive concepts make each input feature has massive paths to the output, which brings a large number of possible decision- maaking patterns of the model. In addition, the effect of votings can easily couple and cancel each other out in the downstream calculations. As a result, separately analyzing the role of individual concepts or decision paths becomes meaningless.
Different from MIP, VOTEN has far more expressive voting functions. It directly models nonlinear transformations between essential intermediate concepts without relying on massive feature combina- tions and naive transformations. As result, VOTEN's decision-makinn process is explicit with only a small number meaningful concepts, thus is explainable.
2.3 Explaining the effect of votings
In VOTEN, individual voting channels play explicit roles in influencing the model decision. In this part, we analyze the effect of votings. First, we use a concept function g. IR. -+ TR to represent the transformation from model inputs x. TR to C, which decides the meaning of the concept.
Definition 1 In VOTEN, we refer to two concept functions g and g' as equivalent iff. there exists an invertible function 4 so that Va. E TR""o $(g(x")) = g(xx²).
Since & is invertible, the outputs of two equivalent concept functions have one-to tn-one correspondence over all the possible inputs. Then, they can effect equally in the decision-makee process.
Theorem In VOTEN, if replacing a concept function with an equivalent form, there exists a way to replace its voting functions that all the downstream concept functions stay unchanged. Proof. Please refer to Appendix
To analyze how voting channels effect on concept functions, we can reformulate Equation T as
23 d+1
9, 2+1 (2")
a
(f (o +b b7+1
where Ex, [ (xx)) denotes the expectation of fig over all the instances, but1 sample- independent bias. In particular,
Th Σ
8
of Fe (o (x))) = E
dd+1 For simplicity, we use R2 to denote E (g (x)).. Notably, by adding an arbitrary bias 9j we obtain an equivalence of the original concept. According to Theorem 2 we can construct a model with exactly the same expression (i.e., equivalent concepts and the same predictions) as the previous one. This implies that VOTEN may converge to models with differed internal bias but exactly the same decision-making making process, indicating VOTEN explanation should be independent of concept bias. According to Equation 2 and 3 voting channels' average only influence concept bias while the
voting deviation SLj (x²)) Ne reveals the effect of Vi for the decision. This can be intuitively explained as each voter can vote with different basic scores and only the deviation from the basic score reflects their judgement for a specific instance. Similarly, from the global view, we reformulate fil (ax)) R..
the concept function as 25t1 = E; of Kd (x") + 6571, where K (x²))
of
Notably, K (-) generates distribution with mean 0 and variance 1 over all the instances. Therefore, at and of jointly decide the overall effect of votings. Specifically, the counting layer explicitly adjusts ad so that reliable voters have stronger influences. Meanwhile, the voter implicitly adjusts of. When the concept is less related to the target concept and cannot support proper votes, they decrease oft and tend to always vote the basic score to avoid disturbing the model. Otherwise, they increase of and vote confidently to lead the model to correct estimation.
Based on the above analysis, we can easily design algorithms to ease both VOTEN local and global explanation, such as recognizing decision paths and quantifying the concept/feature relevance to the prediction, which can be found in Appendix B and Appendix C.

---

Table 1: Model Performance. We conducted 10 independent runs on each dataset and show the average standard deviation of AUC and AP, particular, for multi-classfiiationn, we estimated the macro average of each metric. We also did significance test, where and denote significantly (i.e., p-value I 0.05) and very significantly (i.e., p-value v 0.01) worse than VOTEN.
DT 45
RF 48
LGB 22
MLP [2
NAM
VOTEN
PR
0.2557*
0.4624 **
0.4817
0.4870
0.4412
0.5007
+
0.0007
0.0010
0.0007
0.0040
0.0014
0.0041
MR
AUC
0.6762
0.9007
0.9209
0.9199
0.9031*
0.9237
0.0096 -
0.0006
0.0001
0.0008
0.0014
0.0005
PR
0.0177
0.0349
0.0525"
0.0516
0.0485
0.0550
+
0.0001
0.0011
0.0017
0.0014
0.0014
0.0001
RP
AUC
0.5134
0.6566**
0.7305
0.7250
0.7102**
0.7322
+
0.0014
0.0052
0.0016
0.0017
0.0082
0,0003
PR
0.8119
0.9763
0.9753
0.9662
0.7000
0.9783
+
0.0010
0.0003
0.0003
0.0018
0.0008
0.0013
CT
AUC
0.9396
*
0.9979
0.9965
0.9965 **
0.9497
0.9983
+ 0.0005
0.0001
0.0001
0.0002
0.0001
0,0001
PR
0.2514
0.6348
0.6972
0.6215
0.6567**
0.6522
+ 0.0022
0.0014
0.0001
0.0030
0.0043
0.0019
CI
AUC
0.7250
0.9388
0.9566
0.9454
0.9506*
0.9508
0.0018 +
0.0005
0.0001
0.0004
0.0006
0.0002
PR
0.6408
0.8505
0.8590"*
0.8297**
0.7897**
0.8612
+
0.0002
0.0001
0.0001
0.0007
0.0003
0.0005
HG
AUC
0.6705
0.8429
0.8459
0.8157**
0.7751
0.8481
+
0.0002
0.0001
0.0001
0.0007
0.0001
0.0005
PR
0.0081
0.0085
0.0120
0.0113
0.0108
0.0120
+ 0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
AS
AUC + 0.0002
0.4999
0.5139
0.6026
0.5917
0.5742
0.5975
0.0009
0.0001
0.0005
0.0018
0.0021
2.4 VOTEN supports effective model pruning
In VOTEN, concepts are estimated by averaging the votes. This means we can delete voting channel while keeping the physical meaning of the target concept unchanged. This supports effective link pruning. Specifically, since only the voting deviation from the average decides the effect, we can assume the absent channel votes the basic score regardless of the input. Formally, the value of cd+1
estimated as 2 = E Id (. ( (x/)((")) h + 55+1, where Isd E {0, 1} indicates V achieve
absent. Furthermore, since only involving small number of voting channels, network pruning on VOTEN by exhaustively exploring how the performance will get influenced some voting channels absent, which is infeasible for MIP. In MIP, usually prune unimportant hidden units to reduce model complexity. However, usually not all the voting channels from an important concept necessary. In VOTEN, these unnecessary concept transformations can be further eliminated to not only reduce model complexity but also raises the explainability of the model. In Appendix D, give efficient VOTEN pruning algorithm with a lazy updating strategy.
Experiment
To evaluate the effectiveness and explainability of VOTEN for seizing comprehensive decision- making patterns. We conducted experiments with 6 large public datasets, including Context-aware
Multi-Modal Transportation Recommendation (MR) 2 (58, IJCAI-18 Search Conversion Rate Prediction (RP) [3), Forest Cover Type Prediction (CT) 11).
Prediction (CI) (38).
Allstate Claim Prediction (AS) []. and Higgs boson dataset (HG) 11. The detailed descriptions of experimental setup can be found in Appendix E.
3.1 Performance Evaluation: Can VOTEN achieve higher performance than MIP?
We used two widely adopted metrics for imbalanced classification performance evaluation, including Area Under ROC Curve (AUC) [14) and average precision (AP) [19], whose higher value means
Our code is available hutp://ggthbbbcom//uuugggglyy//n

---

higher performance. In Table 1 we compare the performance of VOTEN with several baselines, including Decision Tree (DT) 19. Random Forest (RF) 48), LightGBM (LGB) [77] MLP [22]. and Neural Additive Model (NAM) 7 (see Appendix F.2 for detail). For each dataset, we have carefully tuned the parameters of the baselines to achieve their best performance. Especially, detailed analysis on MLP parameters can be found in Appendix E.3. It can be observed that, while deep neural networks are powerful when incorporated with purposely designed modules or prior knowledge for specific tasks, its standard form (i.e., MLP) without task-specific structures may perform worse than LightGBM, which have also been shown by many previous studies [47
Indeed, LightGBM is believed to be powerful in handling structured data and often appears the major model of the top solutions in Mata- competitions [58] 35 In contrast, VOTEN, also in its standard form, significantly outperforms MLP for all these tasks and comparable with LightGBM. Indeed, on many datasets, it outperforms L LightGBM if complicated feature engineering is not performed. This shows the effectiveness of VOTEN in terms of handling real-world problems. Indeed, VOTEN is more suitable for handling data-mining tasks since it has more reasonable decision-making process. It should be noticed that, similar to MLP, VOTEN is standard and generic model that can be easily expanded to task-specific networks to raise model performance (for example, we can replace MLP with VOTEN in DeepFM (24)). While this paper focuses on the generic performance of standard VOTEN, it shows the possibility of building more powerful deep learning solutions for a wide range of applications.
3.2 Explanation Complexity: Is the decision-making process of VOTEN recognizable?
In Table 2 we compare the explanation com- plexity of MLP and VOTEN. As we have dis- cussed in Section 2.2, we use the number of feature combinations, the length of decision paths, and the number of possible decision- making patterns to show the explanation com- plexity of a model. In particular, we trained two VOTEN models with different settings for each dataset. Specifically, "VOTEN-" -" is com- parable to the best performance of MLP, with the least feature combinations. "VOTEN" is the one with the best performance. It can be observed that VOTEN greatly reduces feature combinations and decision paths. For exam- ple, on the MR dataset, MLP needs 7 voting layers that each contain 128 concepts. This brings an exponentially large number of long decision paths. In contrast, VOTEN achieves comparable performance with 16 hidden con- cepts in total. Then, each feature only has 16 possible paths with a length of 2 to reach the output. This significantly eases the understanding of the decision-making process. For some datasets, VOTEN with no intermediate concepts (i.c., features directly vote for the prediction) can achieve comparable performance to MLP. Moreover, even when reaching its best performance, VOTEN still has a much smaller explanation complexity than MLP. should be noticed that we have listed the maximum possible number of decision paths for convincing illustration. In practice, we can easily distinguish a fewer number of important decision paths, which will be shown in the following experiments.
Table 2: Explanation complexity. "#C/L" counts hid- den concepts in each layer. "#P/F" counts possible decision paths from each feature.
Dala
Model
AP
Performance
AUC
#CIL
Depth
AP/I
MLP
0.487
0.920
128
7
2
MR
VOTEN
0.497
0.922
16
1
16
VOTEN
0,501
0.924
16
256
MLP
0.052
0.725
12
1,728
RP
VOTEN
0.055
0.732
0
0
1
VOTEN
0.055
0.732
8
2
64
MLP
0,966
0.996
128
249
VOTEN
0.969
0.997
32
1,024
VOTEN
0.978
0,998
64
2
4,096
MLP
0.622
0.945
32
1,024
VOTEN
0.950
VOTEN
0.652
0.951
4
2
16
MLP
0.830
0.816
64
4
2
HG
VOTEN
0,831
0.816
g
1
8
VOTEN
0.862
0.848
64
4,096
MLP
0.011
0.592
64
22
AS
VOTEN
0.012
0.594
16
16
VOTEN
0.012
0.598
64
2
4,096
3.3 Case Study: How to explain VOTEN model?
We show how to explain a VOTEN model with an example in the MR dataset. Specifically, we first analyze the global decision- making process and discover the meaning of concepts by observing voting functions on important decision paths. Then, we locally explain how the inputs of an instance hierarchically vote the final prediction.
Task Description. The task is to recommend the transport mode for online map app users, given a user and an Origin-Destination (OD) pair. The features mainly contain user portraits and an ordered list of recommended plans of the map app. Each plan consists of transport mode, time, distance, and price. We deleted the first recommended plan's mode information since it is too strongly correlated

---

R5-modo
R4-mode
CTNB
Class
mo-
R2-price
Ri-distance
Ri-Emm
Ai-prine
L1H12 LIHTS
2-5
2-7.
LIH15
L1H14
L2H7
11-12
1-15
1,14
RI-distance RI-enoe
R5 mode
R2-mode
K4-mode
L2H7
L2H5
L1H12
L1H8
Class 1
Figure Case study of VOTEN global Recision-making process in MR dataset. We show the important paths from inputs to one output, where wide lines indicate important voting channels. The concept LrHy indicates the y-th concept in the x-th layer. We visualize the important voting functions for each concept along the paths relevant to L2H7, where blue lines show the function while green lines show the average vote.
with the label (many users choose the first recommendation as default). In this way, we can better observe how the model incorporates complicated information for meaningful decisions.
Global Decision-Maken Making Process. We focus on the decision making process for the class "Subway".. First, we discover the important transformations from observations to the prediction with our decision path recognition algorithm. The results are shown in Figure 2 Next, we analyze the meaning of concepts from the bottom to the top. L1H8 gets larger when modes of more recommended plans are "Subway", which we regard as "OD pair with flexible subway-based plans". LIHI2 decreases with higher prices and lower distances, which we regard as "OD pair's distance-cost performance". LIHI4 observes the OD pair distant but still available with inexpensive and fast transportation. Besides, it also observes if modes of public transportation ever appear in the plan list. Therefore, we regard LIH14 as "distant OD pair with convenient and economical public transportation" LIHI5 is sensitive to a time-consuming top-1 1 plan. It also observes the other plans' modes and prices to estimate if trading money for efficiency is infcasible. Therefore, we regard LIHI5 as "no choice but time-consuminng transportation". In the second layer, 1.277 is estimated based L1H8, L1H12, L1H14, and LIH15. It gets higher if many subway-based plans available (higher L1H8), transportation with balanced distance-com is recommended (has a peak for LIH12), distant OD pair but still has convenient public transportation (higher LIH14), or costly time-saving transportation is infeasible (higher LIH15). Comprehensively considering these reasons, it implies "OD pair suitable for subway transportation, which votes for the score of "Subway" with a monotonic transformation. Along the other path, LIH4 is concept "OD pair with inexpensive transportation". Then, L2H5 is also about the price since it is mainly based on L1H4. It should be noticed that its estimation may still be enhanced by other lower-level concepts when dealing with specific instances, although they are less important from the global view. Finally, L2H5 and L2H7 vote nonlincarly to the score so that the model can make accurate quantitative predictions. With the above analysis, we can qualitatively understand the logic of VOTEN on recommending "Subway"". Actually, the eecision-making process quantitatively more complicated and can handle more special cases. In practice, domain experts can thoroughly analyze the shape and gradients of the voting functions get more insights into the concepts. This may help researchers to find new concepts and develop new theories, especially in fields such psychology and management, where scientists work on finding mechanisms linking observations to outcomes. Extra visualizations on global decision path and voting functions can be found in Appendix F and Appendix G.
Then, we analyze the decision paths for specific instances. In Figure 3. we visualize the most important paths for sample with high score for "Subway"" which are filtered with a small threshold in our local decision path recognition algorithm. Based on the global analysis, we can easily tell how the observations gradually transformed to higher-level concepts. Specifically, for LIH12, the vote from price is filtered out, showing the OD pair's iir's price is normal over the dataset. However, it finds the OD pair to be distant, which contributes to a relatively larger distance-cost performance than average, given the normal price. This indicates the plan to be relatively more economical. L2H7 observes the value of LIH12, and finds that the OD pair can
Local Decision-Making Process.
7

---

Subway
L1H8
L1H12
Distance
Figure 3: Case study of VOTEN local decision decision-making process in MR dataset, where wider and darker lines indicate stronger influence (negatively in blue and positively in red). The bottom left shows filtered paths when we gradually bring up the thresholds. We also show the important voting functions, where the green lines show the average and red points show vote for the current instance.
61.2-
25.2
M.3
40
2.2
40
S (a) MR
1s
14
(b) CT
Figure 4: Heatmap for the propogation-baser relevance in VOTEN, The x-axis represents features and the y-axis represents the instance-output pairs, where # OO indicates the feature relevance of the I-th instance class O. Red means positive relevance while blue means negative relevance.
choose transportation whose distance-cosi performance more balanced than average cases (reaches the peak of voting). In this case, 1.2H7 votes high for subway, which is economical and balanced transportation mode. On the other decision path, L1H8 thinks "Subway" may not be suitable since subway-hor -hased plans seldom appear in the list, thus votes negatively for 1.2H7. But L.H12 makes very confident judgment based on the balance of distance-price performance, which dominates L2H7 and makes the model predict correctly. Interestingly, in this task is to guess the transportation mode most recommended by the app (the information that we hide in prior), which is reasonable. On the one hand, the app's recommender system trained with abundant information can naturally achieve high performance. On the other hand, many users will click the first recommendation as default. addition this strategy, the model will use more important decision paths to achieve more accurate predictions. As gradually increase the threshold, more decision-mail -aaiin patterns appeared. Extra visualizations on local explanation of other datasets can be found in Appendix H.
find of the most important strategies
3.4 Relevance Analysis: Can VOTEN help quantify feature relevance?
Propagation-Based Relevance. Motivated by relevance propagation [10] we propose algorithm (see Appendix D) to quantify the relevance of features and concepts, based on the important decision paths. The short decision paths of VOTEN decreases error accumulation during the propagation and enables more accurate relevance estimation. In Figure 4 we visualize the propagation- based feature relevance with heatmaps. Obviously, features from #69 to #86 are generally important in the MR dataset, among which the first several features (information about the top-1 1 plan in the list) are the most relevant. Furthermore, the relevant features vary for different samples in terms of different classes, which indicates VOTEN to predict in multiple patterns. For example, 4bg shows that VOTEN assigns high score to class 2 for sample 4 and sample 5 with different reasons. Specifically, feature #3 and #53 contribute positively for sample 5 while negatively for sample 4. Instead, #50 more positively relevant for sample 4. Extra visualizations
be found in Appendix
Single-Sighted Prediction Strength. We can also estimate the feature relevance from the view of model performance when the prediction is supported by single voter in some layer. Specifically.

---

Table 3: Features with top-5 single-sighted prediction strength for class 0 and class of MR dataset in VOTEN. "Input" indicates the AUC of ranking the samples according to the feature's value.
Class 0
Class
Model
Top-1
Top-2
Top-3
Top-4
Top-5
Top-1
Top-2
Top-3
Top-4
Top-5
VOTEN
0.675
0.653
0.647
0.606
0.592
0.747
0.632
0.625
0.587
0.566
Input
0.596
0,533
0.572
0.600
0,586
0,535
0.527
0.534
0.555
0.525
0.
2.80
0.65
-05
4
(a) MR (Class 0)
(b) RP
Figure 5: Single-sighted prediction strength in VOTEN. We also show the AUC of linearly ranking the samples comparisons. Darker color means higher AUC.
we disable the other voters in a similar way as we do in model pruning. Table 3 shows MR features achieving the highest AUC when conducting single-sighted prediction for class 0 and class in VOTEN. As a comparison, we also show the AUC of directly ranking with the feature, which reveals the feature's 's linear correlation with the prediction. In particular, since there can be negative correlations, we evaluate AUC for the rank in increasing and decreasing order, and use the performance. Moreover, Figure shows the results on all the features. It can be observed that, even if approximated into single-sighted, VOTEN significantly raises AUC, showing VOTEN to recognize important features and strengthen their effectiveness with nonlinear transformations. Especially, while feature #85 originally seems not correlated with the output, VOTEN finds in practice nonlinearly very relevant the output. This proves the effectiveness of VOTEN on quantifying the inherent nonlinear relationships between the observations and the prediction. Interestingly, VOTEN weakens the effect of some features (e.g., feature #3 for RP) to prevent them from disturbing the prediction. Extra visualizations can be found in Appendix J.
the larger
3.5 Pruning Experiment
We conducted pruning experiments on VOTEN for MR and CT datasets with our pruning algorithm. As we gradually deleted voting channels, we monitored the change AUC during this process, which shown in Figure 6 For MR, AUC is still near 0.924 after pruning nearly half of the voting channels. For CT, AUC is still near 0.998 after pruning quarter of the channels. Interestingly, proper prun- ing may slightly raise model performance, which is reasonable a simple model has less chance of over-fitting. In practice, op- erations like fine-tuning can be adopted
0.0%
4.0%
4
CM
0.00
0.3
(a) MR
(b) CT
Figure 6: VOTEN pruning experiments. The x-axis
shows the ratio of pruned channels while the y-axis shows AUC. The green line shows MI.Ps performance.
to further raise the performance of the pruned VOTEN model. Then, the model can be further compressed without affecting the prediction much. These results prove VOTEN to support effective pruning, which is helpful. We can use complicated information for training and prune the model to decrease the complexity for storage, calculation and explanation.
4 Related Work
Post-Hoc Deep Learning Explanation. Post-Hoc explaining algorithms analyze the relevance of features in model-free way, mainly including propagation-be based methods [30 6 and perturbation- based methods (59 [56). Propagation-base
methods propagate the relevance score backward to the inputs. For example, Simonyan et al. (45) generates saliency maps with the gradients of the output category with respect to the inputs. Bach el al. [10] proposed Layer-wise Relevance Propagation
(LRP), which designed effective rules for the propagation. Perturbation-banl methods explain

---

model behavior by observing how the output reacts to purposely perturbed or constructed inputs. For example, Local Interpretable -ggost-A Explanations (LIME) 42 trains a local explainable approximation model around the prediction with randomly perturbed features and the corresponding outputs. SHapley Additive exPlanations (SHAP) [4] estimates the Shapley value of features to measure their contribution to model performance. In addition to features, some works estimate concept importance for a model [23] 53, 28). For example, Kim et al. (28) learn the representation of human- understandable concepts with labeled concept-relevant examples and estimate concept sensitivity according to the directional derivative towards the concepts. Along this line, abundant works have been proposed to further raise the effectiveness of post-hoc interpretation algorithms [3 55| [5]. However, these algorithms regard models as blackboxes and heuristically explain with their own metries, which cannot give explicit understandings of the actual prcision-making process. Different from existing works, we proposed a naturally understandable neural network model.
Intrinsically Explainable Machine Learning Techniques. Intrinsically explainable models can be explained without relying on post-hoc algorithms (9 S) mainly including classic models such as logistic regression (37], linear support vector machine (26). decision trees [43, generalized additive models (25) and Bayesian models [1 [52. Recently, Agarwal et al. 7 proposed Neural Additive Model that predicts with a linear combination of neural networks. However, all these models usually have tight restrictions on the hypothesis space, which limits their fitting ability on complicated real- world problems. Based on these methods, complicated models are developed for higher performance. However, even with intrinsically explainable base models, these complicated models still need post- hoc algorithms for explanation [9]. For example, ensemble tree models [T6] 27 predict with large number of weak learners. However, the joint decision- making process of massive decision trees is difficult to understand. Kernel functions (39 are incorporated in support vector machines to seize high dimensional feature interactions. However, the dimension transformation is implicit and not understandable. In recent years, researchers also try design explainable neural network models by incorporating purposely designed task-specific constraints or structures [47] 50 17 40). However, these models cannot be adopted by the general tasks. Besides, they only provide heuristic and Nomain-speciinn intermediate information instead of telling the complete decision -aaonng process. Different from these works, we aim at general explainable neural network model, which has an intrinsically explainable decision- decision-making process while retaining the high fitting ability.
5 Concluding Remarks
In this paper, we have proposed an explainable deep learning model, VOTEN. Specifically, we theoretically analyzed the major components of VOTEN and discussed its priority over MLP. and accordingly proposed some efficient algorithms to raise the model usability. Experimental results on multiple real-world datasets clearly demonstrated that VOTEN can significantly improve the explainability and performance of deep learning.
Limitations. In this paper, we focused on comparing VOTEN with MLP, which is the generic and basic structure of deep learning models. Many powerful problem-specific structures can be derived from MLP by adding operations such as weight sharing (e.g.. CNN). Similar to MLP, VOTEN is a basic and generic structure. It can be adopted to problem-specific models (e.g., we can simply use VOTEN to replace MLPs in deepFM [24] or MMoE (6)). Indeed, recent studies show that if properly designed, simple MLP-basee structure achieves comparable performance to complicated SOTA models (2 [T). VOTEN's advantages over MLP provides great possibility on further improving wide range of deep learning applications. In the future, we will also explore building VOTEN-based task- sseccffc structures. In addition, since VOTEN automatically extracts concepts during training, human effort is needed to observe the voting functions for understanding the concepts, which is a common issue in unsupervised concept modeling, such as Latent Dirichlet Allocation 13. In the future, we will work on easing the concept understanding of VOTEN, such as recognizing concept-related samples or aligning VOTEN with numan-unnderstnddable concepts.
Acknowledgments and Disclosure of Funding
The research work supported by the National Key Research and Development Program of China the National Natural Science Foundation of China under Grant
under Grant No. No. U1836206, U1811461, 62176014 91746301, 61836013, 61773361.
10

---

References
[1]
prediction Cl ClaimPraediciioncaalleenng
Allstate claim
challenge.
c/
[2] Context-aware multi-modal transportation recommendation. httpps://dianshi,t hcw.::aaanniii.....b.........e baidu
com/ http://cppeiitiaoo//////ssstttot
[3] Ijcai-18 alimama sponsored search conversion rate(cvr) prediction contest, https://tianchh
[4] Lightgbm vs neural network.
lightgbm-ve vs- neural network/
[5] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: a survey explainable artificial intelligence (xai). IEEE access, 1.52118-52160, 2018.
[6] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03292. 2018.
[7] Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. arXiv preprint arXiv:2004. 13912, 2020.
[8] Aneseh Alvanpour, Sumit Kumar Das, Christopher Kevin Robinson, Olfa Nasraoui, and Dan Popa. Robot failure mode prediction with explainable machine learning, In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 61-66. IEEE, 2020.
[9] Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Ben- jamins, al. Explainable artificial intelligence (xai): Concepts, taxonomies. opportunities and challenges toward responsible ai. Information Fusion, 58:82-115, 2020.
[10] Sehastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
[11] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
[12] Jock Blackard and Denis Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture. 24(3):131-15) 1999.
[13] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
[14] Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Patlern recognition, 30(7):1 145-1159, 1997.
[15] Steven Bramhall, Hayley Hor, Michael Tieu, and Nibhrat Lohia. Qlime-a quadratic local interpretable model-agnostic explanation approach. SMU Data Science Review, 3(1):4, 2020.
[16] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
[17] Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In Proceedings the
of the 42nd International ACM SIGIR Conference on Research and Development Information Retrieval, pages 765-774, 2019.

---

[18] Arun Das and Paul Rad. Opportunities and challenges in explainable artificial intelligence (xai): survey, arXiv preprint arXiv:2006.1137 2020.
[19] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233-240, 2006.
[20] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 11171-416,, 2019.
[21] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmlp: Re-parameterizing convolutions into fully-connected layers for image recognition. arXiv e-prints, pages arXiv- 2105, 2021.
[22] Matt W Gardner and SR Dorling. Artificial neural networks (the multilayer perceptron) review of applications the atmospheric sciences. Atmospheric environment, 32(14-15):2627- 2636, 1998.
[23] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. arXiv preprint arXiv:1902.03/29, 2019.
[24] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfmm a factorization-machine based neural network for ctr prediction. arXiv preprini arXiv: 1703.0477 2017.
[25] Trevor Hastie and Robert Tibshirani. Generalized additive models, volume 43. CRC press, 1990.
[26] Shujun Huang, Nianguang Cai, Pedro Penzuti Pacheco, Shavira Narrandes, Yang Wang, and Wayne Xu. Applications of support vector machine (svm) learning in cancer genomics. Cancer Genomics- Proteomicc, 15(1):41-55 51,1151,77 2018.
[27] Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3 10:3146-31540 2017.
[28] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International conference on machine learning, pages 1666-26677. PMIR,, 2018.
[29] Alex Krizhevsky, Ily Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1 105, 2012.
[30] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Robertt Müller. Unmasking clever hans predictors and assessing what machines really learn Nature communications, 10(1):1-8, 2019.
[31] Heyi Li, Yunke Tian, Klaus Mueller, and Xin Chen. Beyond saliency: understanding convolu- tional neural networks from saliency prediction on layer-wise relevance propagation. Image and Vision Computing, 83:70-86, 2019.
[32] Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1754-1763, 2018
[33] Hao Lin, Hengshu Zhu, Yuan Zuo, Chen Zhu, Junjie Wu, and Hui Xiong. Collaborative company profiling: Insights from an employee's perspective. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[34] Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions. arXiv preprint 17050777, 2017.
12

---

[35] Zhipeng Luo, Jianqiang Huang, Ke Hu, Xue Li, and Peng Zhang. Accuair: Winning solution to air quality prediction for kdd cup 2018. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1842-1850, 2019.
[36] Jiaqi Ma. Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate -fxxture-f--eperrs.... In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD *18 page 1930-1939, 2018
[37] Edward C Norton, Bryan E Dowd, and Matthew L Maciejewski. Marginal srfects--quantfyiing the effect of changes in risk factors in logistic regression models. Jama, 321(13):1304 1305, 2019.
[38] Nikunj Oza and Stuart Russell, Experimental comparisons of online and batch versions of bagging and boosting. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 359-364, 2001.
[39] Luis Carlos Padierna, Martin Carpio, Alfonso Dojas-Dominguez, Hector Puga, and Hector Fraire. A novel formulation of orthogonal polynomial kernel functions for svm classifiers: the gegenbauer family. Pattern Recognition, 84:211 1-225, 2018.
[40] Chuan Qin, Hengshu Zhu, Tong Xu, Chen Zhu, Liang Jiang, Enhong Chen, and Hui Xiong. Enhaneing person-job fit for talent recruitment: An ability-aware neural network approach. In The 41st international ACM SIGIR conference research & development in information retrieval, pages 25-34, 2018
[41] Adrian E Raftery. Bayesian model selection in social research. Sociological methodology, pages 111-163, 1995.
[42] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[43] S Rasoul Safavian and David Landgrebe. A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics, 21(3):660 674, 1991.
[44] Dazhong Shen, Chuan Qin, Hengshu Zhu, Tong Xu, Enhong Chen, and Hui Xiong. Joint repre- sentation learning with relation- enhanced topic models for intelligent job interview assessment. ACM Transactions on Information Systems (TOIS), 40(1):1-36, 2021.
[45] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
[46] Ying Sun, Fuzhen Zhuang, Hengshu Zhu, Xin Song, Qing He, and Hui Xiong. The impact of person-organization fit on talent management: structure-aware convolutional neural network approach. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1625-1633, 2019.
[47] Ying Sun, Fuzhen Zhuang, Hengshu Zhu, Qi Zhang, Qing He, and Hui Xiong. Market-oriented job skill valuation with cooperative composition neural network. Nature communications, 12(1):1-12, 2021.
[48] Vladimir Svetnik, Andy Liaw, Christopher Tong, J Christopher Culberson, Robert P Sheridan, and Bradley P Feuston. Random forest: classification and regression tool for compound classification and qsar modeling. Journal of chemical information and computer sciences, 4376)1194 2003.
[49] Fangshuang Tang, Qi Liu, Hengshu Zhu, Enhong Chen, and Feida Zhu. Diversified social influence maximization. In 2014 IEEF/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014), pages 455-459, IEEE, 2014.
13

---

[50] Mingfei Teng. Hengshu Zhu, Chuanren Liu, and Hui Xiong. Exploiting network fusion for organizational turnover prediction. ACM Transactions on Management Information Systems (TMIS), 12(2):1-18, 2021.
[51] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mip-mixer: An all- all-mlp architecture for vision. arXiv preprint arXhv:2105.01600 2021.
[52] Chao Wang, Hengshu Zhu, Chen Zhu, Xi Zhang, Enhong Chen, and Hui Xiong. Personalized employee training course recommendation with career development awareness. In Proceedings of The Web Conference 2020, pages 1648-16599 2020.
[53] Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R Lyu, and Yu-Wing Tai. Towards global explanations of convolutional neural networks with concept attribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8652-8661, 2020.
[54] Zichao Yang. Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages
2016.
[55] Muhammad Rehman Zafar and Naimul Mefraz Khan. Dlime: a deterministic local interpretable model-agnostic explanations approach for computer-aided diagnosis systems. arXiv preprint arXiv: 10263, 2019.
[56] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
[57] Hongke Zhao, Qi Liu, Hengshu Zhu, Yong Ge, Enhong Chen, Yan Zhu, and Junping Du.
A sequential approach to market state modeling and analysis in online p2p lending. IEEE Transactions on Systems, Man, and Cyhernetics: Systems, 48(1):21-33, 2017.
[58] Wenjun Zhou, Taposh Dutta Roy, and Iryna Skrypnyk. The kdd cup 2019 report. ACM SICKDD Explorations Newsletter, 22(1):8-17, 2020.
[59] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04599 2017.
14


Reviews:
[{"review_id": "ZySzfFqPJDj", "reviewer": "Reviewer_v1Ts", "summary": "The authors propose an explainable deep learning model that uses a hierarchical voting strategy to perform hierarchical feature selection.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "raises predictive performance, decreases feature combinations, supports efficient pruning and feature analysis.\n\n\"humans usually aggregate information to infer intermediate concepts step-by-step.\" This should have a citation.  Really a lot of the information in the first paragraph of 2.1 needs justification through citation.  \n\n\"That is, while deep neural networks make the decision according to a unified, inseparable complicated function, humans usually aggregate information to infer intermediate concepts step-by-step\"  Is this not what many convolutional networks have been shown to do in recent work on interpretation and explainability?\n\nHow does VOTEN constrain the model to capture indidual concepts in each module?\n\nIn equation 1, why replace C with x?  Is C_d+1 a weighted sum of the output of f(C_d)?  It is not clear at this point in the paper what the function \"f\" does.  By theorem 1 this is the key difference from a regular MLP so this should be defined sooner.\n\nIn general I am not sure what function \"f\" is in equation 1 or how voting occurs.  Is voting done by outputing a weight for each concept that are then summed to form a new concept?\n\nThe approach described in section 3.3 is not very clear and seems difficult to generalize.  Can the authors provide a clear high level description of this process as it would generalize to other settings?\n\nI am not convinced at using correlating decision paths and model activations for explainability as there are many variables to consider.  For the subway example, are thos paths possibly attributed to some other co-ocurring variable with subway in the data?\n\nA related work section should be added to address how this approach relates to other work in network explainability.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Ye", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "AenARmaqQFs", "reviewer": "Reviewer_1htR", "summary": "This paper proposes a novel network structure for improving the explainability of deep learning models. The authors theoretically analyzed their model and proved with abundant experiments that the proposed model is effective both in terms of explainability and model performance. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally speaking, this is a well-written paper, and I enjoyed reading it. The proposed model is introduced well in a convincing and insightful way. \n\nThe authors give straightforward analysis to show the priority of their model in terms of explainability. Then, they conducted extensive experiments to show the effectiveness of their model. I can see the authors' efforts to improve the usability of their model by providing supportive algorithms and tools. The case studies and visualizations are impressive and show human-understandable inherent classification process of the model. Therefore, I think this work can be a powerful and explainable off-the-shelf solution for classification tasks in many real-world fields. \n\nIt is impressive that, while there is usually a trade-off between model explainability and accuracy, the authors have innovatively proposed a new solution to achieve higher performance while assuring high explainability. The authors claimed their model to be generic and can be easily integrated with existing deep learning structures and showed its priority over MLP. Considering more and more MLP-based deep learning models have been achieving SOTA performance in many fields these years, it is exciting to see a powerful substitution with high explainability, which potentially benefits a wide range of deep learning applications.\n\nStill, just like the authors have mentioned in the limitation section, I think the significance of this work can be further enlarged if the authors improve existing task-specific MLP-based models with the proposed structure. I expect the authors in their future work to explore more potentials applications of their model. In addition to models for recommender systems the authors have mentioned (i.e., deepFM and MMoE), models for Computational Vision (e.g., MLP-Mixer) may also be considered. \n\nBesides, the authors have submitted the code for review as supplementary information. Considering its potential influence, I urge the authors to release their code to public platforms as soon as they publish this paper.\n\nThe advantage and disadvantage of this paper are summarized as follows:\n\nAdvantage:\n1. The authors provide a powerful and explainable network structure. The proposed model is generic and can be easily integrated with existing deep learning structures. \n2. Impressive experimental results are given to prove the model's effectiveness and explainability. \n3. The authors provide a relatively complete workflow for their model's practical use, with carefully designed algorithms and tools for quantifications and visualizations.\n\nDisadvantage:\n1. The presentation of this paper needs to be improved. There are typos in the paper, and texts in several figures are small. \n2. The significance of this work can be further enlarged if the authors improve existing task-specific MLP-based models with the proposed structure. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have properly discussed the limitations of their work and discussed the future works for improvement. I look forward to their future works.\nI cannot see potential negative societal impact of this work. \n", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "7kWl5sEQH2", "reviewer": "Reviewer_4PGX", "summary": "This paper proposes a novel type of model, Voting Transformation-based Explainable Neural Network (VOTEN), which represents the transformation from the input to the output to be a hierarchical voting process. This simple structure makes the model interpretable for a human. The papers systematically analyses the advantages of VOTEN, in particular the model's explainability (global and local) and reduced complexity (pruning).", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am very much at odds with myself when it comes to this paper. On the one hand it aims to solve a very relevant problem. Through it specific structure VOTEN is relatively easy to interpret while obtaining quite good accuracies, significantly better than e.g. simple decision trees. The experiments presented in the paper are very nice and cover different aspects, e.g., performance, local and global explainability and complexity.\nOn the other hand I am not fully convinced about the proposed model. First, I find all the discussions claiming that VOTEN is very similar to human reasoning very strange and not appropriate. The authors do not present any results, which support this claim. Furthermore, in the experimental section VOTEN is only applied to relatively simple problems. The big strength of deep neural networks (this to some degree also hold for models such as random forests, SVMs etc.) is that they also perform well on complex, high-dimensional problem. I am not sure that this also holds for VOTEN. The true practical abilities of VOTEN are hard to assess, also because the selected datasets are not the typical benchmark datasets for e.g. GMBs or RandomForest classifiers. Furthermore, there is no comparison with post-hoc explanation methods such as LRP or SHAP applied to a deep neural network or e.g. the work of Lundberg computing Shapley values on tree-based models. Thus, in practice it is unclear how much insights the explanations provided by VOTEN really bring. Also there is no comparison or extensive discussion of the proposed model to the work of Agarwal et al. 2020. Thus, the advantages VOTEN over sota remain unclear.\nOverall, because of these limitations I am not fully convinced about VOTEN and therefore see the paper lying below the acceptance threshold.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a novel network structure for better explainability. While there are many aspects of explainability that are not assessed in this paper, the local and global prediction processing cases are provided beside the accuracy. The term decision-making is a bit confusing as it often refers to decision-making tasks, while this paper handles prediction tasks. Overall, this paper is significantly novel, while its applicability is not quite clear. 
Decision: Accept (Poster)