Interpreting Layered Neural Networks via Hierarchical Modular Representation

1 INTRODUCTION To construct a method for interpreting the prediction mechanism of complex statistical models is currently one of the most important tasks in the machine learning field , especially with layered neural networks ( or LNNs ) , which have achieved high predictive performance in various practical tasks . Due to their complex hierarchical structure and the nonlinear parameters that they use to process the input data , we can not understand the function of a trained LNN as it is , and we need some kind of approximation method to convert the original function of an LNN into a simpler interpretable representation . Recently , various methods have been proposed for interpreting the function of an LNN , and they can be roughly classified into ( 1 ) the approximation of an LNN with an interpretable model , and ( 2 ) the investigation of the roles of the partial structures constituting an LNN ( e.g . units or layers ) . As for approach ( 1 ) , various methods have been investigated for approximating an LNN with a linear model ( Lundberg & Lee , 2017 ; Nagamine & Mesgarani , 2017 ; Ribeiro et al. , 2016 ) or a decision tree ( Craven & Shavlik , 1996 ; Johansson & Niklasson , 2009 ; Krishnan et al. , 1999 ; Thiagarajan et al. , 2016 ) . For image classification tasks in particular , methods for visualizing an LNN function have been extensively studied in terms of which part of an input image affects the prediction result ( Ancona et al. , 2018 ; Bach et al. , 2015 ; Shrikumar et al. , 2017 ; Simonyan et al. , 2014 ; Smilkov et al. , 2017 ; Springenberg et al. , 2015 ; Sundararajan et al. , 2017 ) . Approach ( 2 ) has been studied by several authors who examined the function of a given part of an LNN ( Alain & Bengio , 2017 ; Luo et al. , 2016 ; Raghu et al. , 2017 ; Zahavy et al. , 2016 ) . There has also been an approach designed to automatically extract the cluster structure of a trained LNN ( Watanabe et al. , 2017b ; a ; 2018a ) based on network analysis . Although the above studies have made it possible to provide us with an interpretable representation of an LNN function with a fixed resolution ( or number of clusters ) , there is a problem in that we do not know in advance the optimal resolution for interpreting the original network . In the methods described in the previous studies ( Watanabe et al. , 2017a ; b ; 2018a ; b ; c ) , the unit clustering results may change greatly with the cluster size setting , and there is no criterion for determining the optimal cluster size . Another problem is that the previous studies could only provide us with information about the magnitude of the relationship between a cluster and each input or output dimension value , and we could not determine whether this relationship was positive or negative . In this paper , we propose a method for extracting a hierarchical modular representation from a trained LNN , which provides us with both hierarchical clustering results with every possible number of clusters and the function of each cluster . Our proposed method mainly consists of three parts : ( a ) training an LNN for a given data set based on error back propagation , ( b ) determining the feature vectors of each hidden layer unit based on its correlation with the input and output dimension values , and ( c ) the hierarchical clustering of the feature vectors . Unlike the clustering methods in the previous studies , the role of each cluster is computed as a centroid of the feature vectors defined by the correlations in step ( b ) , which enables us to know the representative mapping performed by the cluster in terms of both sign and magnitude for each input or output dimension . We show experimentally the effectiveness of our proposed method in interpreting the internal mechanism of a trained LNN , by applying it to two kinds of data sets : the MNIST data set that contains digit image data and a sequential data set of food consumer price indices . Based on the experimental results for the extracted hierarchical cluster structure and the role of each cluster , we discuss how the overall LNN function is structured as a collection of individual units . 2 TRAINING A LAYERED NEURAL NETWORK An LNN can be trained to approximate the input-output relationship of an arbitrary data set ( x , y ) that consists of input data x ∈ RM and output data y ∈ RN , by using a function f ( x , w ) from x ∈ RM and a parameter w ∈ RL to RN . An LNN parameter is defined by w = { ωdij , θdi } , where ωdij is the connection weight between the i-th unit in a depth d layer and the j-th unit in a depth d + 1 layer , and θdi is the bias of the i-th unit in the depth d layer . Here , d = 1 and d = d0 , respectively , correspond to the input and output layers . The LNN function f ( x , w ) is a set of functions { fj ( x , w ) } for all output dimensions j , each of which is defined by fj ( x , w ) = σ ( ∑ i ω d0−1 ij o d0−1 i + θ d0−1 j ) . Here , σ ( x ) = 1/ ( 1 + exp ( −x ) ) , and odi is the output value of the i-th unit in the depth d layer and o1i = xi holds in the input layer . Such output values in each layer are given by odj = σ ( ∑ i ω d−1 ij o d−1 i + θ d−1 j ) . The purpose of training an LNN is to find an optimal parameter w to approximate the true inputoutput relationship with a finite size training data set { ( Xn , Yn ) } n1n=1 , where n1 is the sample size . The training error E ( w ) of an LNN is given by E ( w ) = 1n1 ∑n1 n=1 ‖Yn − f ( Xn , w ) ‖2 , where ‖ · ‖ is the Euclidean norm of RN . Since the minimization of the training error E ( w ) leads to overfitting to a training data set , we adopt the L1 regularization method ( Ishikawa , 1990 ; Tibshirani , 1996 ) to delete redundant connection weights and obtain a sparse solution . Here , the objective function to be minimized is given by H ( w ) = n12 E ( w ) + λ ∑ d , i , j |ωdij | , where λ is a hyperparameter used to determine the strength of regularization . The minimization of such a function H ( w ) with the stochastic steepest descent method can be executed by an iterative update of the parameters from the output layer to the input layer , which is called error back propagation ( Rumelhart et al. , 1986 ; Werbos , 1974 ) . The parameter update is given by ∆ωd−1ij = −η ( δ d j o d−1 i + λ sgn ( ω d−1 ij ) ) , ∆θ d j = −ηδdj , where δd0j = ( o d0 j − yj ) ( o d0 j ( 1 − o d0 j ) + 1 ) , and δ d j = ∑ld+1 k=1 δ d+1 k ω d jk ( o d j ( 1 − odj ) + 1 ) for d = d0 − 1 , · · · , 2 . Here , yj is the j-th output dimension value of a randomly chosen nth sample ( Xn , Yn ) , 1 is a hyperparameter for the LNN convergence , and η is the step size for training time t that is determined such that η ( t ) ∝ 1/t . In the experiments , we adopt 1 = 0.001 and η = 0.7× a1n1/ ( a1n1 + 5t ) , where a1 is the mean iteration number for LNN training per dataset . 3 HIERARCHICAL MODULAR REPRESENTATION OF LNNS 3.1 DETERMINING FEATURE VECTORS OF HIDDEN LAYER UNITS To apply hierarchical clustering to a trained LNN , we define a feature vector for each hidden layer unit . Let vk be the feature vector of the k-th hidden layer unit in a hidden layer . Such a feature vector should reflect the role of its corresponding unit in LNN inference . Here , we propose defining such a feature vector vk of the k-th hidden layer unit based on its correlations between each input or output dimension . In previous studies ( Watanabe et al. , 2018b ; c ) , methods have been proposed for determining the role of a unit or a unit cluster based on the square root error . However , these methods can only provide us with knowledge about the magnitude of the effect of each input dimension on a unit and the effect of a unit on each output dimension , not information about how a hidden layer unit is affected by each input dimension and how each output dimension is affected by a hidden layer unit . In other words , there is no method that can reveal whether an increase in the input dimension value has a positive or negative effect on the output value of a hidden layer unit , or whether an increase in the output value of a hidden layer unit has a positive or negative effect on the output dimension value . To obtain such sign information regarding the roles of each hidden layer unit , we use the following definition based on the correlation . Definition 1 ( Effect of i-th input dimension on k-th hidden layer unit ) . We define the effect of the i-th input dimension on the k-th hidden layer unit as vinik , where vinik = E [ ( X ( n ) i − E [ X ( n ) i ] ) ( o ( n ) k − E [ o ( n ) k ] ) ] √ E [ ( X ( n ) i − E [ X ( n ) i ] ) 2 ] E [ ( o ( n ) k − E [ o ( n ) k ] ) 2 ] . Here , E [ · ] represents the mean for all the data samples , X ( n ) i is the i-th input dimension value of the n-th data sample , and o ( n ) k is the output of the k-th hidden layer unit for the n-th input data sample . Definition 2 ( Effect of k-th hidden layer unit on j-th output dimension ) . We define the effect of the k-th hidden layer unit on the j-th output dimension as voutkj , where voutkj = E [ ( o ( n ) k − E [ o ( n ) k ] ) ( y ( n ) j − E [ y ( n ) j ] ) ] √ E [ ( o ( n ) k − E [ o ( n ) k ] ) 2 ] E [ ( y ( n ) j − E [ y ( n ) j ] ) 2 ] . Here , y ( n ) j is the value of the j-th output layer unit for the n-th input data sample . We define a feature vector of each hidden layer unit based on the above definitions . Definition 3 ( Feature vector of k-th hidden layer unit ) . We define the feature vector of the k-th hidden layer unit as vk ≡ [ vin1k , · · · , vini0k , v out k1 , · · · , voutkj0 ] . Here , i0 and j0 , respectively , represent the dimensions of the input and output data . Alignment of signs of feature vectors based on cosine similarity The feature vectors of Definition 3 represent the roles of the hidden layer units in terms of input-output mapping . When interpreting such roles of hidden layer units , it is natural to regard the roles of any pair of units ( k1 , k2 ) as being the same iff they satisfy vk1 = vk2 or vk1 = −vk2 . The latter condition corresponds to the case where the k1-th and k2-th units have the same correlations with input and output dimensions except that their signs are the opposite , as depicted in Figure 1 . To regard the roles of unit pairs that satisfy one of the above conditions as the same , we propose an algorithm for aligning the signs of the feature vectors based on cosine similarity ( Algorithm 1 ) . By randomly selecting a feature vector and aligning its sign according to the sum of the cosine similarities with all the other feature vectors , the sum of the cosine similarities of all the pairs of feature vectors increases monotonically . We show experimentally the effect of this sign alignment algorithm in Appendix 2 . 3.2 HIERARCHICAL CLUSTERING OF UNITS IN A TRAINED LNN Once we have obtained the feature vectors of all the hidden layer units as described in section 3.1 , we can extract a hierarchical modular representation of an LNN by applying hierarchical clustering Algorithm 1 Alignment of signs of feature vectors based on cosine similarity 1 : Let vk and a0 respectively be the feature vector for the k-th hidden layer unit and the number of iterations . 2 : for a = 1 to a0 do 3 : Randomly choose the k-th hidden layer unit according to the uniform distribution . 4 : if ∑ l 6=k vk·vl√ vk·vk √ vl·vl < 0 then 5 : vk ← −vk . 6 : end if 7 : end for to the feature vectors . Among the several existing methods for such hierarchical clustering including single-link and complete-link , Ward ’ s method ( Ward , 1963 ) has been shown experimentally to be effective in terms of its classification sensitivity , so we employ this method in our experiments . We start with k0 individual hidden layer units , and sequentially combine clusters with the minimum error sum of squares ( ESS ) , which is given by ESS ≡ ∑ m ( ∑ k : uk∈Cm ‖vk‖2 − 1 |Cm| ∥∥∥ ∑ k : uk∈Cm vk ∥∥∥2 ) , ( 1 ) where uk and vk , respectively , are the k-th hidden layer unit ( k = 1 , · · · , k0 ) and its corresponding feature vector , Cm is the unit set assigned to the m-th cluster , and | · | represents the cluster size . From Equation ( 1 ) , the ESS is the value given by first computing the cluster size ( |Cm| ) times the variance of the feature vectors in each cluster , and then by taking the sum of all these values for all the clusters . When combining a pair of clusters ( Cm1 , Cm2 ) into one cluster , the ESS increases by ∆ESS = |Cm1 ||Cm2 | |Cm1 |+ |Cm2 | ∥∥∥ 1|Cm1 | ∑ k : uk∈Cm1 vk − 1 |Cm2 | ∑ k : uk∈Cm2 vk ∥∥∥2 . ( 2 ) Therefore , in each iteration , we do not have to compute the error sum of squares for all the clusters , instead we simply have to compute the error increase ∆ESS given by Equation ( 2 ) for all the pairs of current clusters ( Cm1 , Cm2 ) , find the optimal pair of clusters that achieves the minimum error increase , and combine them . We describe the whole procedure of Ward ’ s method in Algorithm 2 . This procedure to combine a pair of clusters is repeated until all the hidden layer units are assigned to one cluster , and from the clustering result { C ( t ) m } in each iteration t = 1 , · · · , k0 − 1 , we can obtain a hierarchical modular representation of an LNN , which connects the two extreme resolutions given by “ all units are in a single cluster ” and “ all clusters consist of a single unit. ” The role of each extracted cluster can be determined from the centroid of the feature vectors of the units assigned to the cluster , which can be interpreted as a representative input-output mapping of the cluster . 4 EXPERIMENTS We apply our proposed method to two kinds of data sets to show its effectiveness in interpreting the mechanism of trained LNNs . The experimental settings are detailed in the Appendix 3 . In Appendix 1 , we provide a qualitative comparison with the previous method ( Watanabe et al. , 2018c ) . 4.1 EXPERIMENT USING THE MNIST DATA SET First , we applied our proposed method to an LNN trained with the MNIST data set ( LeCun et al. , 1998 ) to recognize 10 types of digits from input images . Before the LNN training , we sharpened the top , bottom , left and right margins and then resized the images to 14 × 14 pixels . Figure 2 shows sample images for each class of digits . Although our proposed method provided us with a clustering Algorithm 2 Ward ’ s hierarchical clustering method ( Ward , 1963 ) 1 : Let uk and vk , respectively , be the k-th hidden layer unit ( k = 1 , · · · , k0 ) and its corresponding feature vector , and let { C ( t ) m } be the unit set assigned to the m-th cluster in the t-th iteration ( m = 1 , · · · , k0 − t+ 1 ) . Initially , we set t← 1 and C ( 1 ) m ← { um } . 2 : for t = 2 to k0 − 1 do 3 : ( C ( t−1 ) m1 , C ( t−1 ) m2 ) ← arg min ( C ( t−1 ) i , C ( t−1 ) j ) ∆ESS ( C ( t−1 ) i , C ( t−1 ) j ) , where ∆ESS ( C , C ′ ) ≡ |C||C ′| |C|+ |C ′| ∥∥∥ 1|C| ∑ k : uk∈C vk − 1 |C ′| ∑ k : uk∈C′ vk ∥∥∥2 . Here , we assume m1 < m2 . 4 : Update the clusters as follows : C ( t ) m ← C ( t−1 ) m1 ∪ C ( t−1 ) m2 ( m = m1 ) C ( t−1 ) m ( 1 ≤ m ≤ m2 − 1 , m 6= m1 ) C ( t−1 ) m+1 ( m2 ≤ m ≤ k0 − t+ 1 ) . 5 : end for result for all the possible resolutions or the numbers of clusters c , we have only plotted the results for c = 4 , 8 , 16 , for ease of visibility . Figures 3 and 4 , respectively , show the hierarchical cluster structure extracted from the trained LNN and the roles or representative input-output mappings of the extracted clusters . From these figures , we can gain knowledge about the LNN structure as follows . - At the coarsest resolution , the main function of the trained LNN is decomposed into Clusters 1 , 2 , 3 and 4 . Cluster 1 captures the input information about black pixels in the shape of a 6 and white pixels in the shape of a 7 , and it has a positive and negative correlation with the output dimensions corresponding to “ 6 ” and “ 7 ” , respectively . Cluster 2 correlates negatively with the region in the shape of a 9 , and positively with the other areas . It has a positive correlation with the recognition of “ 2 ” and “ 6 , ” and it has a negative one with “ 0 , ” “ 4 ” and “ 9. ” Cluster 3 correlates positively with the black pixels in the left part of an image , and it has a positive correlation with “ 0 , ” “ 4 ” and “ 6 , ” and a negative correlation with “ 3 ” and “ 7. ” Cluster 4 captures the 0-shaped region , and it has a larger correlation with the output of “ 0 ” compared with the other digits . - Cluster 2 is decomposed into three smaller clusters , 7 , 8 and 9 . Cluster 7 captures similar input information to Cluster 2 , and it also correlates strongly with the lower area of an image . This cluster mainly affects the recognition result for “ 5 ” and “ 6. ” Cluster 8 uses the input information of the area with the shape of a 9 , however , its main recognition target is “ 2. ” Cluster 9 correlates positively with the area extending from the upper right to the lower left of an image , and it correlates negatively with the digits “ 4 ” and “ 9. ” - Cluster 8 consists of two smaller clusters , 17 and 18 . Cluster 17 is mainly affected by the upper part and lower right part of an image , and the absolute value of its correlations with output dimensions are all less than 0.2 , while the role of Cluster 18 is almost the same as that of Cluster 8 . 4.2 EXPERIMENT USING THE CONSUMER PRICE INDEX DATA SET We also applied the proposed method to an LNN trained with a data set of a consumer price index ( e Stat , 2018 ) to predict the consumer price indices of taro , radish and carrot for a month from 36 months ’ input data . With this data set , we plotted the results for c = 3 , 6 , 12 , where c is the number of clusters . Figures 5 and 6 , respectively , show the hierarchical cluster structure extracted from the trained LNN and the roles or representative input-output mappings of the extracted clusters . From these figures , we can gain knowledge about the LNN structure as follows . - Clusters 1 , 2 and 3 represent the main input-output function of the hidden layer units . Interestingly , all of these clusters have similar correlations with the output dimensions ( 0 < radish < taro < carrot ) . However , these three clusters use different input information : Cluster 1 strongly reflects seasonal information , and its correlation is especially high with the consumer price indices of the three vegetables one month before and one , two and three years earlier . Cluster 3 also reflects seasonal information , however , the absolute values of the correlations are less than 0.3 and it correlates strongly with the input information of eight , 20 and 32 months before . On the other hand , Cluster 2 does not use such a seasonal effect very much , and it is affected almost equally by the information of all months , except the recent information of radish from nine months before . - Cluster 1 is composed of smaller clusters of 16 and 17 . Cluster 16 is mainly used to predict the consumer price index of taro and it strongly correlates with the input information for taro from one month before and one , two and three years before . Compared with Cluster 16 , Cluster 17 affects the three output dimensions more equally . - Cluster 7 is a part of Cluster 3 , and consists of smaller clusters of 11 , 12 and 13 . These clusters have mutually different relationships with the output dimension values : Cluster 11 correlates positively with consumer price indices of taro and carrot , and negatively with that of radish . It mainly uses recent information about carrot ( within a year ) and the values of taro of five , 17 and 29 months before . Cluster 13 is mainly used to predict the radish output value . It has a positive correlation with the input information for taro , radish and carrot of about six , 18 and 30 months earlier , and it has a negative correlation with values for one month before and one , two and three years before . The absolute values of the correlations between Cluster 12 and the output dimension values are less than 0.2 , so , unlike with Clusters 11 and 13 , it does not significantly affect the prediction result . 5 DISCUSSION Here , we discuss our proposed method for obtaining a hierarchical modular representation from the perspectives of statistical evaluation and visualization . Our proposed method provides us with a series of clustering results for an arbitrary cluster size , and the resulting structure does not change if we use the same criterion ( e.g . error sum of squares for Ward ’ s method ) for evaluating the similarity of the feature vectors . However , there is no way to determine which criterion yields the optimal clustering result to represent a trained LNN , due to the fact that interpretability of acquired knowledge can not be formulated mathematically ( although there has been an attempt to quantify the interpretability for a specific task , especially image recognition ( Bau et al. , 2017 ) ) . This problem makes it impossible to compare different methods for interpreting LNNs quantitatively , as pointed out in the previous studies ( Lipton , 2016 ; DoshiVelez & Kim , 2017 ) . Therefore , the provision of a statistical evaluation method as regards both interpretability and accuracy for the resulting cluster structure constitutes important future work . Although we can apply our proposed method to an arbitrary network structure , as long as it contains a set of units that outputs some value for a given input data sample , the visualization of the resulting hierarchical modular representations becomes more difficult with a deeper and a larger scale network structure , since a cluster may contain units in mutually distant layers . Additionally , the number of possible cluster sizes increases with the scale ( or the number of units ) of a network , and so it is necessary to construct a method for automatically selecting a set of representative resolutions , instead of visualizing the entire hierarchical cluster structure . 6 CONCLUSION Finding a way to unravel the function of a trained LNN is an important issue in the machine learning field . While LNNs have achieved high prediction accuracy with various data sets , their highly complex and nonlinear parameters have made it difficult to interpret their internal inference mechanism . Recent studies have enabled us to decompose a trained LNN into simpler cluster structure , however , there is no method for ( 1 ) determining the optimal number of clusters , or ( 2 ) knowing whether the outputs of each cluster have a positive or negative correlation with the input and output dimension values . In this paper , we proposed a method for extracting the hierarchical modular representation of a trained LNN , which consists of sequential clustering results with every possible number of clusters . By determining the feature vectors of the hidden layer units based on their correlations with input and output dimension values , it also enabled us to know what range of input each cluster maps to what range of output . We showed the effectiveness of our proposed method experimentally by applying it to two kinds of practical data sets and by interpreting the resulting cluster structure . REFERENCES G. Alain and Y. Bengio . Understanding intermediate layers using linear classifier probes . In ICLR 2017 Workshop , 2017 . M. Ancona , E. Ceolini , A. C. Öztireli , and M. Gross . Towards better understanding of gradientbased attribution methods for deep neural networks . In International Conference on Learning Representations , 2018 . S. Bach , A. Binder , G. Montavon , F. Klauschen , K-R. Müller , and W. Samek . On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation . PLOS ONE , 10 : 1–46 , 2015 . D. Bau , B. Zhou , A. Khosla , A. Oliva , and A. Torralba . Network dissection : Quantifying interpretability of deep visual representations . In Computer Vision and Pattern Recognition , 2017 . M. Craven and J. W. Shavlik . Extracting tree-structured representations of trained networks . In Advances in Neural Information Processing Systems 8 , pp . 24–30 , 1996 . F. Doshi-Velez and B. Kim . Towards a rigorous science of interpretable machine learning . arXiv:1702.08608 , 2017. e Stat . Consumer price index of food nationwide from January 1970 to January 2018. https : //www.e-stat.go.jp/dbview ? sid=0003143513 , 2018 . M. Ishikawa . A structural connectionist learning algorithm with forgetting . Journal of Japanese Society for Artificial Intelligence , 5:595–603 , 1990 . U. Johansson and L. Niklasson . Evolving decision trees using oracle guides . In 2009 IEEE Symposium on Computational Intelligence and Data Mining , pp . 238–244 , 2009 . R. Krishnan , G. Sivakumar , and P. Bhattacharya . Extracting decision trees from trained neural networks . Pattern Recognition , 32 ( 12 ) :1999–2009 , 1999 . Y. LeCun , L. Bottou , Y. Bengio , and P. Haffner . Gradient-based learning applied to document recognition . In Proceedings of the IEEE , volume 86 , pp . 2278–2324 , 1998 . Z. C. Lipton . The mythos of model interpretability . In Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning , 2016 . S. M. Lundberg and S. Lee . A unified approach to interpreting model predictions . In Advances in Neural Information Processing Systems 30 , pp . 4765–4774 , 2017 . W. Luo , Y. Li , R. Urtasun , and R. Zemel . Understanding the effective receptive field in deep convolutional neural networks . In Advances in Neural Information Processing Systems 29 , pp . 4898– 4906 , 2016 . T. Nagamine and N. Mesgarani . Understanding the representation and computation of multilayer perceptrons : A case study in speech recognition . In Proceedings of the 34th International Conference on Machine Learning , pp . 2564–2573 , 2017 . M. Raghu , J. Gilmer , J. Yosinski , and J. Sohl-Dickstein . SVCCA : Singular vector canonical correlation analysis for deep learning dynamics and interpretability . In Advances in Neural Information Processing Systems 30 , pp . 6076–6085 , 2017 . M. T. Ribeiro , S. Singh , and C. Guestrin . “ Why should I trust you ? ” : Explaining the predictions of any classifier . In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pp . 1135–1144 , 2016 . D. Rumelhart , G. Hinton , and R. Williams . Learning representations by back-propagating errors . Nature , 323:533–536 , 1986 . A. Shrikumar , P. Greenside , and A. Kundaje . Learning important features through propagating activation differences . In Proceedings of the 34th International Conference on Machine Learning , pp . 3145–3153 , 2017 . K. Simonyan , A. Vedaldi , and A. Zisserman . Deep inside convolutional networks : Visualising image classification models and saliency maps . In ICLR 2014 Workshop , 2014 . D. Smilkov , N. Thorat , B. Kim , F. Viégas , and M. Wattenberg . Smoothgrad : removing noise by adding noise . arXiv:1706.03825 , 2017 . J. T. Springenberg , A. Dosovitskiy , T. Brox , and M. Riedmiller . Striving for simplicity : The all convolutional net . In ICLR 2015 Workshop , 2015 . M. Sundararajan , A. Taly , and Q. Yan . Axiomatic attribution for deep networks . In Proceedings of the 34th International Conference on Machine Learning , pp . 3319–3328 , 2017 . J. J. Thiagarajan , B. Kailkhura , P. Sattigeri , and K. N. Ramamurthy . Treeview : Peeking into deep neural networks via feature-space partitioning . In NIPS 2016 Workshop on Interpretable Machine Learning in Complex Systems , 2016 . R. Tibshirani . Regression shrinkage and selection via the lasso . Journal of the Royal Statistical Society , Series B , 58 ( 1 ) :267–288 , 1996 . J. H. Ward . Hierarchical grouping to optimize an objective function . Journal of the American Statistical Association , 58 ( 301 ) :236–244 , 1963 . C. Watanabe , K. Hiramatsu , and K. Kashino . Recursive extraction of modular structure from layered neural networks using variational Bayes method . In Proceedings of Discovery Science 2017 , Lecture Notes in Computer Science , volume 10558 , pp . 207–222 , 2017a . C. Watanabe , K. Hiramatsu , and K. Kashino . Modular representation of autoencoder networks . In Proceedings of 2017 IEEE Symposium on Deep Learning , 2017 IEEE Symposium Series on Computational Intelligence , 2017b . C. Watanabe , K. Hiramatsu , and K. Kashino . Modular representation of layered neural networks . Neural Networks , 97:62–73 , 2018a . C. Watanabe , K. Hiramatsu , and K. Kashino . Understanding community structure in layered neural networks . arXiv:1804.04778 , 2018b . C. Watanabe , K. Hiramatsu , and K. Kashino . Knowledge discovery from layered neural networks based on non-negative task decomposition . arXiv:1805.07137v2 , 2018c . P. Werbos . Beyond regression : new tools for prediction and analysis in the behavioral sciences . PhD thesis , Harvard University , 1974 . T. Zahavy , N. Ben-Zrihem , and S. Mannor . Graying the black box : Understanding DQNs . In Proceedings of the 33rd International Conference on Machine Learning , pp . 1899–1908 , 2016 . APPENDIX 1 : COMPARISON WITH CLUSTERING METHOD BASED ON NON-NEGATIVE MATRIX FACTORIZATION Here , we show the effectiveness of our proposed method by comparing it with the clustering method based on non-negative matrix factorization ( or NNMF ) , which was proposed in a previous study ( Watanabe et al. , 2018c ) . We applied this NNMF-based clustering method to the same data sets that we used in the experiments described in section 4 . In the previous study ( Watanabe et al. , 2018c ) , the feature vectors of the hidden layer units are defined by the magnitude of the effect of each input dimension value on a cluster and the effect of a cluster on each output dimension value , computed by the square root error of the unit output values . By definition , the elements of such feature vectors are all non-negative , which is a necessary condition for applying NNMF to the feature vectors . We applied the NNMF-based clustering method to the trained network with exactly the same parameter as the network shown in Figures 3 and 5 . With the MNIST data set ( LeCun et al. , 1998 ) and the data set of a consumer price index ( e Stat , 2018 ) , respectively , we decomposed the trained networks into 16 and 12 clusters . With both data sets , we set the number of iterations of the NNMF algorithm at 1000 . We applied the NNMF algorithm for 10000 times , and used the best result in terms of the approximation error . Initial values of the two low-dimensional matrices were randomly chosen according to the normal distribution N ( 0.5 , 0.5 ) . Figures 7 , 8 , 9 , and 10 show the resulting cluster structures and the representative roles of the clusters . Comparing these figures with the results in Figures 3 , 4 , 5 , and 6 , we can observe that the previous NNMF-based method could not capture the structures of the input and output dimension values in as much detail as our proposed method , since it does not take the sign information into account . Furthermore , with the NNMF-based method , we should define the number of clusters in advance , and we can not observe the hierarchical structure of clusters to find the optimal resolution for interpreting the roles of partial structures of an LNN . APPENDIX 2 : EFFECT OF SIGN ALIGNMENT OF FEATURE VECTORS Here , we discuss the effect of the sign alignment of the feature vectors based on cosine similarity ( Algorithm 1 ) . Figures 11 shows the effect of the sign alignment of the feature vectors extracted from an LNN trained with the MNIST data set ( LeCun et al. , 1998 ) . The left and center figures , respectively , show the feature vectors before and after the alignment of the signs . The right figure shows the monotonic increase of the sum of the cosine similarities through the alignment algorithm . Figure 12 shows the dendrograms of the hierarchical clustering results with the original feature vectors of Definition 3 and with the feature vectors after the alignment of the signs . From this figure , we can observe that the height of the dendrogram , which shows the similarity of all the hidden layer units , is higher with the original feature vectors than with the feature vectors after the sign alignment . In other words , it was shown that the algorithm successfully aligned the feature vectors so that they became similar to each other . Figures 13 and 14 show the effect of the sign alignment of the feature vectors extracted from an LNN trained with the food consumer price index data set ( e Stat , 2018 ) . These figures show similar results to those of the MNIST data set . APPENDIX 3 : EXPERIMENTAL SETTINGS Here , we detail the experimental settings . E1 and E2 , respectively , represent the settings of the experiments described in sections 4.1 and 4.2 . - The training sample size n1 was : 500 per class ( E1 ) , and 270 ( E2 ) . - We normalized the input data so that the minimum and maximum values of an element , respectively , were −1 and 1 . Similarly , we normalized the output data so that the minimum and maximum values of an element , respectively , were 0.01 and 0.99 . - The mean iteration number for LNN training per dataset a1 was : 100 per class ( E1 ) , and 500 ( E2 ) . - We generated the initial connection weights and biases of a layered neural network as follows : ωdij i.i.d.∼ N ( 0 , 0.5 ) , θdi i.i.d.∼ N ( 0 , 0.5 ) . - The hyperparameter of the L1 regularization λ was : 1.1× 10−5 ( E1 ) , and 2× 10−5 ( E2 ) . - As regards the LNN training with the MNIST data set , we chose training data with the following deterministic procedure to stabilize the training . Let Z ( k ) n ≡ { X ( k ) n , Y ( k ) n } be the n-th training data sample in class k. The training data were chosen in the following order : Z ( 1 ) 1 , · · · , Z ( 10 ) 1 , Z ( 1 ) 2 , · · · , Z ( 10 ) 2 , · · · , Z ( 1 ) n1 , · · · , Z ( 10 ) n1 , Z ( 1 ) 1 , · · · , Z ( 10 ) 1 , · · · - The iteration number for the alignment of the signs of the feature vectors a0 was : 5000 ( E1 and E2 ) . - The weight removing hyperparameter ξ was : 0.6 ( E1 ) , and 0.001 ( E2 ) In Figures 3 and 5 , we only draw connections where the absolute values of weights were ξ or more .

Summary: In this paper, the authors try to interpret the prediction mechanism of Layered Neural Networks (LNNs). The authors proposed to first define a feature vector that represents the roles of each hidden layer unit, via computing Pearson correlation coefficient. Then a hierarchical clustering method is applied to the generated feature vectors, such that tree-structured relationships among hidden layer units are revealed.