Extract Local Inference Chains of Deep Neural Nets

1 INTRODUCTION . Deep neural networks ( DNNs ) greatly reshape a variety of tasks — object classification , semantic segmentation , natural language processing , speech recognition , robotics , etc . Despite its success on a vast majority of clean data , DNNs are also well-known to be sensitive to small amounts of adversarial noises . The lack of sufficient interpretability about their success or failure is one major bottleneck of applying DNNs to important areas such as medical diagnosis , public health , transportation systems , financial analysis , etc . Interpretable machine learning has attracted growing interest in a variety of areas . The forms of interpretation vary across different methods . For example , attribution methods ( Bach et al. , 2015 ; Sundararajan et al. , 2017 ; Shrikumar et al. , 2017 ; Montavon et al. , 2017 ; Kindermans et al. , 2017 ; Smilkov et al. , 2017 ) produce the importance score of each input feature to the output prediction for an given sample , while some other methods ( Zeiler & Fergus , 2014 ; Simonyan et al. , 2013 ; Erhan et al. , 2009 ) aim to explain the general functionality of each neuron/filter or an individual layer regardless of the input sample . Another line of works ( Ribeiro et al. , 2016 ; Wu et al. , 2018 ; Hou & Zhou , 2018 ) explain DNNs in a local region of data space by training a shallow ( e.g. , linear ) and easily interpretable model to approximate the original DNN on some locally similar samples . Thereby , they reduces the problem to explaining the shallow model . These methods essentially reveal the neuron to neuron correlations ( e.g. , input to output , intermediate layer/neuron to output , etc ) , but they can not provide an overview of the whole inference process occurring inside the complicated structure of DNNs . In this paper , we study a more challenging problem : Can we unveil the major hidden steps of inference in DNNs and present them in a succinct and human-readable form ? Solving this problem helps to answer many significant questions , e.g. , which layer ( s ) /neuron ( s ) plays the most/least im- portant role in the inference process ? Do two similar samples really share most inference steps ? Do all samples need the same number of neurons/layers to locate the key information leading to their correct predictions ? How/when/where does a failure happen during the inference on a DNN ? Which neuron ( s ) /layer ( s ) /feature ( s ) are shared by different samples from the same region even when their labels differ ? Are DNNs using entirely different parts for data from different local regions ? Some of them are related to other problems such as network pruning ( Han et al. , 2015 ; Li et al. , 2016 ) and neural architecture search ( NAS ) ( Zoph & Le , 2017 ) . For example , are winning tickets ( Frankle & Carbin , 2018 ; Liu et al. , 2018 ) universal over different local regions and classes ? Does the weight sharing scheme in recent NAS methods ( Pham et al. , 2018 ; Liu et al. , 2019 ; Ying et al. , 2019 ) limit the searching space or quality ? We develop an efficient tool called NeuroChains to extract the underlying inference chains of a DNN for a given local data region . Specifically , we aim to extract a much smaller sub-network composed of a subset of neurons/filters exactly copied from the original DNN and whose output for data from the local region stays consistent with that of the original DNN . In experiments , we assume the data from the same classes reside in a local region . While the selected filters explain the key information captured by the original DNN when applied to data from the local region , the architecture of the sub-network stitches these information sequentially , i.e. , step by step and layer by layer , and thus recover the major steps of inference that lead to the final outputs . Despite its combinatorial nature , we parameterize the sub-network as the original DNN with an additional score multiplied to each filter/layer ’ s output featuremap . Thereby , we formulate the above problem of sub-network extraction as optimizing a differentiable sparse scoring of all the filters and layers in order to preserve the outputs on all given samples . The above problem can be solved by an efficient back-propagation that only updates the scores with fixed filter parameters . The objective is built upon the Kullback–Leibler ( KL ) divergence between the sub-network ’ s output distribution and that of the original DNN , along with an ` 1 regularization for sparse scores over filters . We further use a sigmoid gate per layer to choose whether removing the entire layer . The gate plays an important role in reducing the sub-network size since most local regions do not rely on all the layers . In practice , we further apply a thresholding to the sparse scores to obtain an even smaller sub-network and employ an additional fine-tuning to the filter scores on the sub-network . We illustrate the subnetwork ’ s architecture and visualize its filters and intermediate-layers ’ featuremaps by exist- ing methods ( Mundhenk et al. , 2019 ; Erhan et al. , 2009 ) . NeuroChains is a novel pruning technique specifically designed for interpreting the local inference chains of DNNs . As aforementioned , it potentially provides an efficient tool to study other problems in related tasks . However , it has several fundamental differences to network pruning and exist- ing interpretation tasks , which deter methods developed for these two problems from addressing our problem . Comparing to network pruning : ( 1 ) fine-tuning is not allowed in NeuroChains ; ( 2 ) it targets a much larger pruning rate for succinct visualization , e.g. , ≥ 95 % for VGG-19 and ≥ 99 % for ResNet-50 on ImageNet with ≤ 200 filters remained ; ( 3 ) it is for data from a local region instead of the whole data distribution . Comparing to mainstream interpretation tasks : ( 1 ) NeuroChains produces an interpretation of the entire inference process in a local region ( which may contain different classes ) rather than of one neuron/filter or output on/around a single sample ; ( 2 ) the sub-network ’ s architecture provides orthogonal information to the importance of individual neurons/filters . 2 RELATED WORKS . Interpretable machine learning methods can be mainly categorized into the ones aiming to evaluate the importance of each input feature of a single sample and the ones explaining individual neurons/filters . Approaches in the first category usually rely on certain back-propagation from the DNN ’ s output to derive an importance score for each input feature or hidden node . Earlier works are based on the back-propagated gradients , e.g. , deconvolution ( Zeiler & Fergus , 2014 ) , backpropagation ( Simonyan et al. , 2013 ) and guided back-propagation ( Springenberg et al. , 2014 ) . Sundararajan et al . ( 2017 ) proposed to ( approximately ) calculate the integral of the gradients along a path between a baseline point and the input sample , which ensures the sensitivity and implementation invariance lacking in some previous methods . More recent methods propose novel backpropagation rules to directly derive the attribution scores of neurons from output to input , e.g. , DeepLIFT ( Shrikumar et al. , 2017 ) , deep Taylor decomposition ( Montavon et al. , 2017 ) , and layerwise relevance propagation ( LRP ) ( Bach et al. , 2015 ) . Methods in the second class treat DNNs as black boxes and seek a simple model to explain how the DNN ’ s output changes in a local region . For example , they add perturbations to different parts of the input to evaluate how the perturbations change the output ( Ancona et al. , 2017 ) , which reflect the importance of different parts . Zeiler & Fergus ( 2014 ) covered different parts of the input image with a gray square , which led to different prediction probabilities on the true class . Instead , Zintgraf et al . ( 2017 ) replaced each patch in the input image with the surrounding patch and tracked the induced changes in the output . In LIME ( Ribeiro et al. , 2016 ) , Ribeiro et al . trained a sparse linear model on noisy input-output pairs as a local surrogate approximating the original DNNs , where the sparse weights are used to explain the importance of input features . As mentioned before , our main difference to the above methods are we explain DNNs for a local region of multiple samples and we further explain how DNNs step by step integrate the information of important filters/neurons . Network pruning ( Han et al. , 2015 ; Li et al. , 2016 ) remove redundant neurons/nodes or connections/weights from a pre-trained DNN and fine-tune the sub-network . Structural pruning removes whole layers/channels/filters/neurons according to a certain norm of the associated weights ( Li et al. , 2016 ) or sparsity ( Hu et al. , 2016 ) . In contrast , Frankle & Carbin ( 2018 ) ; Liu et al . ( 2018 ) prune a DNN during its training . Luo et al . ( 2017 ) apply pruning to two adjacent convolution layers at each time to take the dependency between the two layers into account . Several recent works empirically verify “ lottery ticket hypothesis ” , i.e. , there exists sub-networks ( i.e. , winning tickets ) that can reach comparable generalization performance as the original DNN if re-trained . In contrast , the sub-network extracted by NeuroChains can not be fully re-trained since it has to preserve the original DNN ’ s filters , and our goal is to retain the generalization performance only for a local region . 3 NEUROCHAINS . 3.1 PROBLEM : EXTRACT LOCAL INFERENCE CHAINS AS SUB-NETWORKS . Although the DNNs widely used nowadays are usually composed of hundreds of layers and millions to billions of hidden nodes . When applied to samples from a local region in data space , it is plausible that its inference process mainly relies on a small subset of layers/neurons/filters . In this paper , we verify this conjecture by developing an efficient and practical algorithm , i.e. , NeuroChains , to extract the subset and its underlying architecture as a sub-network whose weights/filters are selected and exactly copied from the original DNN while its outputs in a given local region retain the ones produced by the original DNN . Although DNNs are usually non-smooth in definition if using a nonsmooth piece-wise activation such as ReLU , when trained with the commonly used techniques , e.g. , data augmentation , mix-up , dropout , the resulted DNNs are relatively smooth in a sufficiently small local region . In order to preserve the original inference chains , we do not allow any fine-tuning or re-training on any filter or the the weight vector corresponding to any neuron : they can only be exactly copied from the original DNN . Let F ( · ; { W ` } ` =1 : L ) ( a mapping from input to output ) denote the original DNN , W ` represents the set of filters/weight vectors in layer- ` , and W ` [ i ] represents the ith filter/weight vector in layer- ` . Any sub-network fulfilling our above requirement can be defined and parameterized by an indicator vector M ` per layer , whose each entry is a { 0 , 1 } value indicating whether retaining the associated filter/neuron in W ` . We further define operator ◦ as ( W ` ◦M ` ) [ i ] , { W ` [ i ] , M ` [ i ] = 1 ; 0 , M ` [ i ] = 0 . ( 1 ) Thereby , { M ` } ` =1 : L defines a qualified sub-network for inference chain and its weights are { W ` ◦ M ` } ` =1 : L , where we extend the operator ◦ to make W ◦M = { W ` ◦M ` } ` =1 : L given the original DNN ’ s weights W = { W ` } ` =1 : L. Given a set of samples X drawn from a local region of data space , we can formulate the problem of finding an inference chain as the following combinatorial optimization , which aims to find the most sparse indicator M ( i.e. , the sub-network with the fewest filters retained ) that does not change the outputs of the original DNN for ∀x ∈ X , i.e. , min { M ` } ` =1 : L L∑ ` =1 ‖M ` ‖1 s.t . F ( x ; W ) = F ( x ; W ◦M ) , ∀x ∈ X . ( 2 ) However , directly solving this combinatorial optimization is impractical since the possible choices forM ` is of exponential number . In this paper , we relax the 0-1 indicator vectorM ` to a real-valued score vector S ` of the same size . We define an operator applied to W ` and its associated scores S ` as ( W ` S ` ) [ i ] , S ` [ i ] ·W ` [ i ] . ( 3 ) Note we do not limit entries in S ` within [ 0 , 1 ] due to the possible redundancy among filters in the original DNN , i.e. , there might be filters of similar functionality for the given samples and a preferred pruning should be able to only preserve one of them and multiply it by the number of those redundant filters in the sub-network . In addition , less constrains are easier to handle in optimization and helpful to find sub-network whose outputs are closer to that of the original DNN , since the class of sub-networks with parameters W S includes all the sub-networks with parameters W ◦M . Hence , we relax the challenging combinatorial optimization to the following unconstrained continuous optimization , i.e. , min { S ` } ` =1 : L 1 |X | ∑ x∈X l ( F ( x ; W ) , F ( x ; W S ) ) + λ L∑ ` =1 ‖S ` ‖1 , ( 4 ) where l ( · , · ) is a loss function aiming to minimize the distance between the original DNN ’ s output F ( x ; W ) and the sub-network ’ s output F ( x ; W S ) . In our experiments , for classification , we use KL-divergence between the output distributions over classes , where the two output distributions are computed by applying softmax to F ( x ; W ) and F ( x ; W S ) respectively , i.e. , l ( F ( x ; W ) , F ( x ; W S ) ) = DKL ( softmax ( F ( x ; W ) ) ||softmax ( F ( x ; W S ) ) ) . ( 5 ) In addition , empirical evidences ( Krueger et al. , 2017 ; Singh et al. , 2016 ) show that for most samples there exist some layers that can be entirely removed without changing the final prediction . Hence , only a few hard and confusing samples need more delicate features , while most other samples can be correctly classified based on simple patterns from shallower layers . Therefore , in NeuroChains , we apply a sigmoid function with input score α ` as a gate G ` determining whether removing the entire layer- ` during pruning , i.e. , G ` = 1/ [ 1 + exp ( −α ` /T ) ] , ( 6 ) where T is a temperature parameter . With a gate G ` applied after each layer- ` whose input and output has the same size ( which is common in many DNNs ) , we can recursively define the input H ` +1 ( · ) to the next layer- ( ` + 1 ) , i.e. , H ` +1 ( x ; { W ` ′ S ` ′ , α ` ′ } ` ′=1 : ` ) = { G ` · F ` ( H ` ; W ` S ` ) + ( 1−G ` ) ·H ` ( x ; { W ` ′ S ` ′ , α ` ′ } ` ′=1 : ` −1 ) , if input size = output size F ` ( H ` ; W ` S ` ) otherwise ( 7 ) where F ` ( H ` ; W ` S ` ) denotes the output of layer- ` . The reason to use a gate here is that we expect to either remove the whole layer or retain it without adding an extra shortcut ( which will change the original DNN ’ s architecture ) . We apply a T < 1 to sharpen the gate values . Since we prefer to remove non-informative layers , in the objective , we add another regularization α ` to encourage the removal of entire layers ( because decreasing α reduces G ` and thus increase the chance of layer removal ) . Therefore , the final optimization for NeuroChains is min { S ` , α ` } ` =1 : L 1 |X | ∑ x∈X l ( F ( x ; W ) , HL+1 ( x ; { W ` ′ S ` ′ , α ` ′ } ` ′=1 : L ) ) +λ L∑ ` =1 ‖S ` ‖1+λg L∑ ` =1 α ` , ( 8 ) Our objective above is similar to the one used in Network Slimming ( Liu et al. , 2017 ) but we optimize it for a local region ( so we can consider to remove layers ) and we do not allow fine tuning on weights W . 3.2 ALGORITHM Our algorithm is simply a standard back-propagation for the optimization problem in Eq . ( 8 ) , which produces sparse scores for filters and gate values for layers . Note the weights in W are fixed and the backpropagation only updates S. We initialize the filter scores S = 1 so W S = W at the beginning of optimization . We initialize the gate score α ` = 0 for all ` = 1 : L so G ` = 0.5 at the beginning , i.e. , the probabilities to remove or to retain a layer is equal . For classification , we set loss l ( · , · ) to be the KL-divergence between the output distributions of the original DNN and the sub-network . After convergence of the optimization , we then apply a simple thresholding to these scores to further remove more filters and layers : ( 1 ) we remove the filters with score under a threshold τ ; ( 2 ) we remove layer- ` if G ` < 0.5 . This yields a sufficiently small sub-network architecture but the scores might be sub-optimal in preserving the original DNN ’ s outputs within the local region . Therefore , we further fine-tune the nonzero scores in S ( note filters are always fixed ) , i.e , the scores for the retained filters/layers , by minimizing Eq . ( 8 ) without the two regularizations , if the first term in Eq . ( 8 ) exceeds a threshold t. Given a sub-network produced by NeuroChains , we then visualize its architecture and scores as the structure of the inference chains . Moreover , we visualize the retained filters and layers on the chains by their activation patterns and featuremaps , respectively , using existing interpretation methods ( Mundhenk et al. , 2019 ; Erhan et al. , 2009 ) .

Summary: The submission considers the task of extracting a small sub-network of a pre-trained neural network that can explain a local region of the data space (a small number of the same classes). The resulting sub-network can be interpreted as the inference path moving from the input to the prediction and the filters and associated weights along this path can be visualised. The key of the proposed approach is to apply a multiplicative weight to each filter and layer in the network and enforce these weights to be sparse. The subnetwork is then selected by thresholding the weights. Some quantitative and qualitative analyses were provided.