Discerning Decision-Making Process of Deep Neural Networks with Hierarchical Voting Transformation

1 Introduction . Neural network based deep learning techniques have attracted great attention from both academia and industry in the past decade . Compared with classic machine learning models , deep neural networks have much higher expressiveness and adaptability for complicated data input , and thus have made tremendous success in various application domains , such as Computational Vision [ 29 , 56 ] , Natural Language Processing [ 54 , 20 ] , and Recommender Systems [ 32 , 24 ] . Nevertheless , since deep neural networks usually have complicated connections of hidden units , a long-standing challenge is how to decipher what ’ s inside the black box of models for understanding their intrinsic decision-making processes . Indeed , in many real-world scenarios , such as business analysis [ 33 , 57 , 49 ] and human resource management [ 50 , 44 , 46 , 44 ] , the lack of model explainability makes people less likely to be convinced when the decision-making process of the model is not understandable . This prevents a broader application of deep neural networks . While many research efforts have been made in developing explainable deep learning models [ 45 , 10 ] , most of existing works focus on post-hoc explanation [ 5 , 18 ] , i.e. , designing metrics to measure the feature relevance/contribution to the outputs of a trained model . Although these methods have * Corresponding authors . †This work was accomplished when Ying Sun was an intern in Baidu supervised by Hengshu Zhu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . made progress on finding important features , the decision-making process of deep learning models is still not available for users . For example , users often need to guess the reason why a prediction is made from the relevant features instead of understanding the decision-making process . Indeed , understanding the intrinsic decision-making process of deep neural networks is a non-trivial task . A major reason is that deep neural networks usually involve massive feature combinations to gain expressiveness on fitting complicated functions . During this process , the effect of features and hidden units may be largely coupled with each other . This indicates that the decision logic of models is inherently buried in the massive feature combinations . Meanwhile , it is difficult for human to understand the intrinsic modeling process of deep learning in a natural manner . Therefore , a key point on improving the explainability of deep neural networks is to decouple the feature combinations and make the modeling process consistent with human decision process . In this way , the model will have an explicit decision-making process and become human-understandable . To this end , in this paper , we propose an explainable deep learning model , namely Voting Transformation-based Explainable Neural Network ( VOTEN ) . Specifically , VOTEN assumes the transformation from the input to the output to be a hierarchical voting process . During this process , lower-level concepts vote for higher-level concepts layer-by-layer in an expressive but explainable way . Instead of relying on massive feature combinations , VOTEN creatively models expressive single-valued voting functions between explicitly modeled hidden concepts to gain expressiveness on fitting complicated functions . This process is explainable for its consistency with human decisionmaking process . We first theoretically analyze the major components of VOTEN and prove the relationship and advantages of VOTEN compared with Multi-Layer Perceptron ( MLP ) , which is the basic structure of deep neural networks . The results show that MLP can be derived from VOTEN by using the inexpressive voting functions . Accordingly , we further analyze the effect of inherent votings and design efficient algorithms for pruning and explaining VOTEN . Finally , we evaluated VOTEN on multiple real-world datasets with comprehensive experiments . The experimental results demonstrate that VOTEN generally promotes powerful and explainable deep learning . Specifically , VOTEN ( 1 ) significantly raises prediction performance ; ( 2 ) exponentially decreases feature combinations ; and ( 3 ) supports efficient pruning and effective feature analysis . Meanwhile , we also visualize the intrinsic decision-making process of VOTEN through case studies , which show VOTEN is explainable and can discover meaningful latent concepts . 2 VOTEN . In this section , we introduce the technical details of VOTEN . 2.1 Structure . When dealing with complicated information , we usually aggregate them step-by-step to form complicated inference . Although deep neural networks also extract abstract features layer-by-layer , they are still difficult to be explained . A major difference is that we can induce explicit concepts , which makes us able to explain our inference in a straightforward way . For example , we can predict that “ This student may get high GPA because he/she works very hard ” . If the more explicit explanation is needed , we can further explain that “ I observed that he/she stay long in the library ” . In this process , with the observation that “ stay long in the library ” , we first infer “ hard-working ” , then infer “ high GPA ” . Moreover , when trying to make more comprehensive inference , we may get more observations for existing concepts ( e.g. , “ little missing of classes ” may also imply “ hard-working ” ) , or induce more concepts ( e.g. , “ learning-efficiency ” ) . This kind of process is straightforward and conforms with our way of understanding . However , deep neural networks can not model concepts in an explicit way . Instead , concept information are embedded in massive hidden units . As we can not understand these induced concepts , neural networks are like unified , inseparable complicated functions , even though they are actually performing information aggregation . Along this line , we believe a natural way to understand how a model decides the output is to make the intrinsic concepts explicit for human . Therefore , we propose VOTEN , whose structure overview is shown in Figure 1 . VOTEN hierarchically models a small number of explicit concepts . During inference , the higher-level concepts will be induced via information aggregation from the lower-level concepts . To achieve meaningful information aggregation , VOTEN focuses on quantifying relationship between individual concepts of different levels . For better understanding , in VOTEN , observations and concepts can be regarded as voters . Based on their own value , each voter independently votes for the higher-level concepts . The votes are aggregated to get the value estimation of each higher-level concept . These concepts will further vote for the next level . In the training process , the model learns to builds intermediate concepts and their quantitative voting functions . Formally , for a VOTEN model with D levels of concepts , we use Cdi to denote the i-th concept in the d-th layer , where C0i denotes an input feature . We refer to transformations between adjacent levels of concepts as a voting layer . In the d-th voting layer , each concept Cdi votes for each higher-level concept Cd+1j with an independent voting channel Vdi , j . Vdi , j takes the value of Cdi as the input and votes with a single-valued nonlinear function fdi , j : R → R. Then , a counting layer gets weighted average of the votes and estimates the value of Cd+1j as xd+1j = nd∑ k=1 adk , jf d k , j ( x d k ) s.t . ∀d , j , nd∑ k=1 adk , j = 1 , ( 1 ) where nd denotes the number of concepts in the d-th layer , xdk denotes the value of Cdk , adk , j denotes the weight of Vdi , j . The concepts of the last layer is regarded as the decision score , which generates the model output with o = FV OTEN ( xD ) . In particular , to assure the ability of the voting functions on effective concept transformation , VOTEN models each voting function with a voting network . The voting network can be designed in complicated ways without influencing model explainability , as long as the function is still single-valued . For example , we can adopt weight-sharing structures to reduce model complexity . 2.2 Why is VOTEN more explainable than MLP ? . In this part , we theoretically discuss VOTEN ’ s advantages over MLP . Specifically , we claim that voting expressiveness is the core proposition of VOTEN that raises explainability . Theorem 1 MLP can be derived from a subset of degenerated VOTEN models whose voting functions in the form of fdi , j ( x d i ) = w d i , jσ ( x d i ) + b d i , j , where σ is a predefined activation function , the scalars wdi , j , b d i , j ∈ R are trainable parameters . Proof . Please refer to Appendix A . From this point of view , MLP also conforms to human inference . However , MLP is still difficult to explain . Indeed , we can more easily understand an inference process with ( 1 ) fewer concepts , ( 2 ) shorter concept transformations , and ( 3 ) fewer reasoning patterns . Under VOTEN schema , we show how inexpressive voting channels make MLP disobey these three traits . Corollary 1 In MLP , votings from the same concept are linearly correlated . This means individual voters in MLP are weak in distinguishing different concepts in the next level . Therefore , MLP relies on highly complicated feature combinations of a large number of deeply tiled voters to achieve high fitting ability . During this process , necessary intermediate information is inherently modeled through combinatorial effects of hidden concepts with inexplicit meanings . Indeed , previous works have proved that human-understandable concepts are inherently mounted in the hidden units of neural network models [ 28 ] . Corollary 2 In MLP , the hypothesis space for voting distribution is limited to scaling and shifting an input distribution . This means that MLP voters can not always induce complicated concepts that have different distribution from their value . As a result , the input needs to go through a long path of transformations between similar concepts until it contributes to the output . Moreover , the deeply tiled massive concepts make each input feature has massive paths to the output , which brings a large number of possible decision-making patterns of the model . In addition , the effect of votings can easily couple and cancel each other out in the downstream calculations . As a result , separately analyzing the role of individual concepts or decision paths becomes meaningless . Different from MLP , VOTEN has far more expressive voting functions . It directly models nonlinear transformations between essential intermediate concepts without relying on massive feature combinations and naive transformations . As a result , VOTEN ’ s decision-making process is explicit with only a small number of meaningful concepts , thus is explainable . 2.3 Explaining the effect of votings . In VOTEN , individual voting channels play explicit roles in influencing the model decision . In this part , we analyze the effect of votings . First , we use a concept function gdi : Rn0 → R to represent the transformation from model inputs x0 ∈ Rn0 to Cdi , which decides the meaning of the concept . Definition 1 In VOTEN , we refer to two concept functions g and g′ as equivalent iff . there exists an invertible function Φ , so that ∀x0 ∈ Rn0 , Φ ( g ( x0 ) ) = g′ ( x0 ) . Since Φ is invertible , the outputs of two equivalent concept functions have one-to-one correspondence over all the possible inputs . Then , they can effect equally in the decision-making process . Theorem 2 In VOTEN , if replacing a concept function with an equivalent form , there exists a way to replace its voting functions so that all the downstream concept functions stay unchanged . Proof . Please refer to Appendix A . To analyze how voting channels effect on concept functions , we can reformulate Equation 1 as xd+1j = g d+1 j ( x 0 ) = nd∑ i=1 adi , j ( f d i , j ( g d i ( x 0 ) ) − Ex [ fdi , j ( gdi ( x ) ) ] ) + bd+1j , ( 2 ) where Ex [ fdi , j ( gdi ( x ) ) ] denotes the expectation of fdi , j over all the instances , b d+1 j is a sampleindependent bias . In particular , bd+1j = nd∑ i=1 adi , jEx [ fdi , j ( gdi ( x ) ) ] = Ex [ g d+1 j ( x ) ] . ( 3 ) For simplicity , we use hdi , j to denote Ex [ fdi , j ( gdi ( x ) ) ] . Notably , by adding an arbitrary bias to g d+1 j , we obtain an equivalence of the original concept . According to Theorem 2 , we can construct a model with exactly the same expression ( i.e. , equivalent concepts and the same predictions ) as the previous one . This implies that VOTEN may converge to models with differed internal bias but exactly the same decision-making process , indicating VOTEN explanation should be independent of concept bias . According to Equation 2 and 3 , voting channels ’ average only influence concept bias while the voting deviation fdi , j ( g d i ( x 0 ) ) − hdi , j reveals the effect of Vdi , j for the decision . This can be intuitively explained as each voter can vote with different basic scores and only the deviation from the basic score reflects their judgement for a specific instance . Similarly , from the global view , we reformulate the concept function as xd+1j = ∑ i a d i , jσ d i , jK d i , j ( x 0 ) + bd+1j , where K d i , j ( x 0 ) = fdi , j ( g d i ( x 0 ) ) −hdi , j σdi , j . Notably , Kdi , j ( · ) generates a distribution with mean 0 and variance 1 over all the instances . Therefore , adi , j and σ d i , j jointly decide the overall effect of votings . Specifically , the counting layer explicitly adjusts adi , j so that reliable voters have stronger influences . Meanwhile , the voter implicitly adjusts σdi , j . When the concept is less related to the target concept and can not support proper votes , they decrease σdi , j and tend to always vote the basic score to avoid disturbing the model . Otherwise , they increase σdi , j and vote confidently to lead the model to correct estimation . Based on the above analysis , we can easily design algorithms to ease both VOTEN local and global explanation , such as recognizing decision paths and quantifying the concept/feature relevance to the prediction , which can be found in Appendix B and Appendix C .

Summary: This paper proposes a novel type of model, Voting Transformation-based Explainable Neural Network (VOTEN), which represents the transformation from the input to the output to be a hierarchical voting process. This simple structure makes the model interpretable for a human. The papers systematically analyses the advantages of VOTEN, in particular the model's explainability (global and local) and reduced complexity (pruning).