Extract Local Inference Chains of Deep Neural Nets

1 INTRODUCTION . Deep neural networks ( DNNs ) greatly reshape a variety of tasks — object classification , semantic segmentation , natural language processing , speech recognition , robotics , etc . Despite its success on a vast majority of clean data , DNNs are also well-known to be sensitive to small amounts of adversarial noises . The lack of sufficient interpretability about their success or failure is one major bottleneck of applying DNNs to important areas such as medical diagnosis , public health , transportation systems , financial analysis , etc . Interpretable machine learning has attracted growing interest in a variety of areas . The forms of interpretation vary across different methods . For example , attribution methods ( Bach et al. , 2015 ; Sundararajan et al. , 2017 ; Shrikumar et al. , 2017 ; Montavon et al. , 2017 ; Kindermans et al. , 2017 ; Smilkov et al. , 2017 ) produce the importance score of each input feature to the output prediction for an given sample , while some other methods ( Zeiler & Fergus , 2014 ; Simonyan et al. , 2013 ; Erhan et al. , 2009 ) aim to explain the general functionality of each neuron/filter or an individual layer regardless of the input sample . Another line of works ( Ribeiro et al. , 2016 ; Wu et al. , 2018 ; Hou & Zhou , 2018 ) explain DNNs in a local region of data space by training a shallow ( e.g. , linear ) and easily interpretable model to approximate the original DNN on some locally similar samples . Thereby , they reduces the problem to explaining the shallow model . These methods essentially reveal the neuron to neuron correlations ( e.g. , input to output , intermediate layer/neuron to output , etc ) , but they can not provide an overview of the whole inference process occurring inside the complicated structure of DNNs . In this paper , we study a more challenging problem : Can we unveil the major hidden steps of inference in DNNs and present them in a succinct and human-readable form ? Solving this problem helps to answer many significant questions , e.g. , which layer ( s ) /neuron ( s ) plays the most/least im- portant role in the inference process ? Do two similar samples really share most inference steps ? Do all samples need the same number of neurons/layers to locate the key information leading to their correct predictions ? How/when/where does a failure happen during the inference on a DNN ? Which neuron ( s ) /layer ( s ) /feature ( s ) are shared by different samples from the same region even when their labels differ ? Are DNNs using entirely different parts for data from different local regions ? Some of them are related to other problems such as network pruning ( Han et al. , 2015 ; Li et al. , 2016 ) and neural architecture search ( NAS ) ( Zoph & Le , 2017 ) . For example , are winning tickets ( Frankle & Carbin , 2018 ; Liu et al. , 2018 ) universal over different local regions and classes ? Does the weight sharing scheme in recent NAS methods ( Pham et al. , 2018 ; Liu et al. , 2019 ; Ying et al. , 2019 ) limit the searching space or quality ? We develop an efficient tool called NeuroChains to extract the underlying inference chains of a DNN for a given local data region . Specifically , we aim to extract a much smaller sub-network composed of a subset of neurons/filters exactly copied from the original DNN and whose output for data from the local region stays consistent with that of the original DNN . In experiments , we assume the data from the same classes reside in a local region . While the selected filters explain the key information captured by the original DNN when applied to data from the local region , the architecture of the sub-network stitches these information sequentially , i.e. , step by step and layer by layer , and thus recover the major steps of inference that lead to the final outputs . Despite its combinatorial nature , we parameterize the sub-network as the original DNN with an additional score multiplied to each filter/layer ’ s output featuremap . Thereby , we formulate the above problem of sub-network extraction as optimizing a differentiable sparse scoring of all the filters and layers in order to preserve the outputs on all given samples . The above problem can be solved by an efficient back-propagation that only updates the scores with fixed filter parameters . The objective is built upon the Kullback–Leibler ( KL ) divergence between the sub-network ’ s output distribution and that of the original DNN , along with an ` 1 regularization for sparse scores over filters . We further use a sigmoid gate per layer to choose whether removing the entire layer . The gate plays an important role in reducing the sub-network size since most local regions do not rely on all the layers . In practice , we further apply a thresholding to the sparse scores to obtain an even smaller sub-network and employ an additional fine-tuning to the filter scores on the sub-network . We illustrate the subnetwork ’ s architecture and visualize its filters and intermediate-layers ’ featuremaps by exist- ing methods ( Mundhenk et al. , 2019 ; Erhan et al. , 2009 ) . NeuroChains is a novel pruning technique specifically designed for interpreting the local inference chains of DNNs . As aforementioned , it potentially provides an efficient tool to study other problems in related tasks . However , it has several fundamental differences to network pruning and exist- ing interpretation tasks , which deter methods developed for these two problems from addressing our problem . Comparing to network pruning : ( 1 ) fine-tuning is not allowed in NeuroChains ; ( 2 ) it targets a much larger pruning rate for succinct visualization , e.g. , ≥ 95 % for VGG-19 and ≥ 99 % for ResNet-50 on ImageNet with ≤ 200 filters remained ; ( 3 ) it is for data from a local region instead of the whole data distribution . Comparing to mainstream interpretation tasks : ( 1 ) NeuroChains produces an interpretation of the entire inference process in a local region ( which may contain different classes ) rather than of one neuron/filter or output on/around a single sample ; ( 2 ) the sub-network ’ s architecture provides orthogonal information to the importance of individual neurons/filters . 2 RELATED WORKS . Interpretable machine learning methods can be mainly categorized into the ones aiming to evaluate the importance of each input feature of a single sample and the ones explaining individual neurons/filters . Approaches in the first category usually rely on certain back-propagation from the DNN ’ s output to derive an importance score for each input feature or hidden node . Earlier works are based on the back-propagated gradients , e.g. , deconvolution ( Zeiler & Fergus , 2014 ) , backpropagation ( Simonyan et al. , 2013 ) and guided back-propagation ( Springenberg et al. , 2014 ) . Sundararajan et al . ( 2017 ) proposed to ( approximately ) calculate the integral of the gradients along a path between a baseline point and the input sample , which ensures the sensitivity and implementation invariance lacking in some previous methods . More recent methods propose novel backpropagation rules to directly derive the attribution scores of neurons from output to input , e.g. , DeepLIFT ( Shrikumar et al. , 2017 ) , deep Taylor decomposition ( Montavon et al. , 2017 ) , and layerwise relevance propagation ( LRP ) ( Bach et al. , 2015 ) . Methods in the second class treat DNNs as black boxes and seek a simple model to explain how the DNN ’ s output changes in a local region . For example , they add perturbations to different parts of the input to evaluate how the perturbations change the output ( Ancona et al. , 2017 ) , which reflect the importance of different parts . Zeiler & Fergus ( 2014 ) covered different parts of the input image with a gray square , which led to different prediction probabilities on the true class . Instead , Zintgraf et al . ( 2017 ) replaced each patch in the input image with the surrounding patch and tracked the induced changes in the output . In LIME ( Ribeiro et al. , 2016 ) , Ribeiro et al . trained a sparse linear model on noisy input-output pairs as a local surrogate approximating the original DNNs , where the sparse weights are used to explain the importance of input features . As mentioned before , our main difference to the above methods are we explain DNNs for a local region of multiple samples and we further explain how DNNs step by step integrate the information of important filters/neurons . Network pruning ( Han et al. , 2015 ; Li et al. , 2016 ) remove redundant neurons/nodes or connections/weights from a pre-trained DNN and fine-tune the sub-network . Structural pruning removes whole layers/channels/filters/neurons according to a certain norm of the associated weights ( Li et al. , 2016 ) or sparsity ( Hu et al. , 2016 ) . In contrast , Frankle & Carbin ( 2018 ) ; Liu et al . ( 2018 ) prune a DNN during its training . Luo et al . ( 2017 ) apply pruning to two adjacent convolution layers at each time to take the dependency between the two layers into account . Several recent works empirically verify “ lottery ticket hypothesis ” , i.e. , there exists sub-networks ( i.e. , winning tickets ) that can reach comparable generalization performance as the original DNN if re-trained . In contrast , the sub-network extracted by NeuroChains can not be fully re-trained since it has to preserve the original DNN ’ s filters , and our goal is to retain the generalization performance only for a local region . 3 NEUROCHAINS . 3.1 PROBLEM : EXTRACT LOCAL INFERENCE CHAINS AS SUB-NETWORKS . Although the DNNs widely used nowadays are usually composed of hundreds of layers and millions to billions of hidden nodes . When applied to samples from a local region in data space , it is plausible that its inference process mainly relies on a small subset of layers/neurons/filters . In this paper , we verify this conjecture by developing an efficient and practical algorithm , i.e. , NeuroChains , to extract the subset and its underlying architecture as a sub-network whose weights/filters are selected and exactly copied from the original DNN while its outputs in a given local region retain the ones produced by the original DNN . Although DNNs are usually non-smooth in definition if using a nonsmooth piece-wise activation such as ReLU , when trained with the commonly used techniques , e.g. , data augmentation , mix-up , dropout , the resulted DNNs are relatively smooth in a sufficiently small local region . In order to preserve the original inference chains , we do not allow any fine-tuning or re-training on any filter or the the weight vector corresponding to any neuron : they can only be exactly copied from the original DNN . Let F ( · ; { W ` } ` =1 : L ) ( a mapping from input to output ) denote the original DNN , W ` represents the set of filters/weight vectors in layer- ` , and W ` [ i ] represents the ith filter/weight vector in layer- ` . Any sub-network fulfilling our above requirement can be defined and parameterized by an indicator vector M ` per layer , whose each entry is a { 0 , 1 } value indicating whether retaining the associated filter/neuron in W ` . We further define operator ◦ as ( W ` ◦M ` ) [ i ] , { W ` [ i ] , M ` [ i ] = 1 ; 0 , M ` [ i ] = 0 . ( 1 ) Thereby , { M ` } ` =1 : L defines a qualified sub-network for inference chain and its weights are { W ` ◦ M ` } ` =1 : L , where we extend the operator ◦ to make W ◦M = { W ` ◦M ` } ` =1 : L given the original DNN ’ s weights W = { W ` } ` =1 : L. Given a set of samples X drawn from a local region of data space , we can formulate the problem of finding an inference chain as the following combinatorial optimization , which aims to find the most sparse indicator M ( i.e. , the sub-network with the fewest filters retained ) that does not change the outputs of the original DNN for ∀x ∈ X , i.e. , min { M ` } ` =1 : L L∑ ` =1 ‖M ` ‖1 s.t . F ( x ; W ) = F ( x ; W ◦M ) , ∀x ∈ X . ( 2 ) However , directly solving this combinatorial optimization is impractical since the possible choices forM ` is of exponential number . In this paper , we relax the 0-1 indicator vectorM ` to a real-valued score vector S ` of the same size . We define an operator applied to W ` and its associated scores S ` as ( W ` S ` ) [ i ] , S ` [ i ] ·W ` [ i ] . ( 3 ) Note we do not limit entries in S ` within [ 0 , 1 ] due to the possible redundancy among filters in the original DNN , i.e. , there might be filters of similar functionality for the given samples and a preferred pruning should be able to only preserve one of them and multiply it by the number of those redundant filters in the sub-network . In addition , less constrains are easier to handle in optimization and helpful to find sub-network whose outputs are closer to that of the original DNN , since the class of sub-networks with parameters W S includes all the sub-networks with parameters W ◦M . Hence , we relax the challenging combinatorial optimization to the following unconstrained continuous optimization , i.e. , min { S ` } ` =1 : L 1 |X | ∑ x∈X l ( F ( x ; W ) , F ( x ; W S ) ) + λ L∑ ` =1 ‖S ` ‖1 , ( 4 ) where l ( · , · ) is a loss function aiming to minimize the distance between the original DNN ’ s output F ( x ; W ) and the sub-network ’ s output F ( x ; W S ) . In our experiments , for classification , we use KL-divergence between the output distributions over classes , where the two output distributions are computed by applying softmax to F ( x ; W ) and F ( x ; W S ) respectively , i.e. , l ( F ( x ; W ) , F ( x ; W S ) ) = DKL ( softmax ( F ( x ; W ) ) ||softmax ( F ( x ; W S ) ) ) . ( 5 ) In addition , empirical evidences ( Krueger et al. , 2017 ; Singh et al. , 2016 ) show that for most samples there exist some layers that can be entirely removed without changing the final prediction . Hence , only a few hard and confusing samples need more delicate features , while most other samples can be correctly classified based on simple patterns from shallower layers . Therefore , in NeuroChains , we apply a sigmoid function with input score α ` as a gate G ` determining whether removing the entire layer- ` during pruning , i.e. , G ` = 1/ [ 1 + exp ( −α ` /T ) ] , ( 6 ) where T is a temperature parameter . With a gate G ` applied after each layer- ` whose input and output has the same size ( which is common in many DNNs ) , we can recursively define the input H ` +1 ( · ) to the next layer- ( ` + 1 ) , i.e. , H ` +1 ( x ; { W ` ′ S ` ′ , α ` ′ } ` ′=1 : ` ) = { G ` · F ` ( H ` ; W ` S ` ) + ( 1−G ` ) ·H ` ( x ; { W ` ′ S ` ′ , α ` ′ } ` ′=1 : ` −1 ) , if input size = output size F ` ( H ` ; W ` S ` ) otherwise ( 7 ) where F ` ( H ` ; W ` S ` ) denotes the output of layer- ` . The reason to use a gate here is that we expect to either remove the whole layer or retain it without adding an extra shortcut ( which will change the original DNN ’ s architecture ) . We apply a T < 1 to sharpen the gate values . Since we prefer to remove non-informative layers , in the objective , we add another regularization α ` to encourage the removal of entire layers ( because decreasing α reduces G ` and thus increase the chance of layer removal ) . Therefore , the final optimization for NeuroChains is min { S ` , α ` } ` =1 : L 1 |X | ∑ x∈X l ( F ( x ; W ) , HL+1 ( x ; { W ` ′ S ` ′ , α ` ′ } ` ′=1 : L ) ) +λ L∑ ` =1 ‖S ` ‖1+λg L∑ ` =1 α ` , ( 8 ) Our objective above is similar to the one used in Network Slimming ( Liu et al. , 2017 ) but we optimize it for a local region ( so we can consider to remove layers ) and we do not allow fine tuning on weights W . 3.2 ALGORITHM Our algorithm is simply a standard back-propagation for the optimization problem in Eq . ( 8 ) , which produces sparse scores for filters and gate values for layers . Note the weights in W are fixed and the backpropagation only updates S. We initialize the filter scores S = 1 so W S = W at the beginning of optimization . We initialize the gate score α ` = 0 for all ` = 1 : L so G ` = 0.5 at the beginning , i.e. , the probabilities to remove or to retain a layer is equal . For classification , we set loss l ( · , · ) to be the KL-divergence between the output distributions of the original DNN and the sub-network . After convergence of the optimization , we then apply a simple thresholding to these scores to further remove more filters and layers : ( 1 ) we remove the filters with score under a threshold τ ; ( 2 ) we remove layer- ` if G ` < 0.5 . This yields a sufficiently small sub-network architecture but the scores might be sub-optimal in preserving the original DNN ’ s outputs within the local region . Therefore , we further fine-tune the nonzero scores in S ( note filters are always fixed ) , i.e , the scores for the retained filters/layers , by minimizing Eq . ( 8 ) without the two regularizations , if the first term in Eq . ( 8 ) exceeds a threshold t. Given a sub-network produced by NeuroChains , we then visualize its architecture and scores as the structure of the inference chains . Moreover , we visualize the retained filters and layers on the chains by their activation patterns and featuremaps , respectively , using existing interpretation methods ( Mundhenk et al. , 2019 ; Erhan et al. , 2009 ) .

Summary: The paper proposes a method called NeuroChains for extracting a sub-network from a deep neural network (DNN) that can accurately match the outputs of the full network for inputs in a small region of the input space. The goal is to be able to explain the important steps that a DNN takes to get from inputs in a small region of the input space, to its predictions for those inputs. NeuroChains initialises the sub-network as the original DNN except each weight/filter is multiplied with a real-valued score. The L1 norm of these scores is minimised while penalising any deviation of the output of the sub-network from the output of the original network. After the minimisation, weights/filters with a score below some threshold value are removed, leaving a sub-network. In the paper there are experiments that aim to verify three claims: (1) NeuroChains can find sub-networks containing less than 5% of the filters in a deep convolutional neural network while preserving its outputs in some small region of the input space; (2) every filter selected by NeuroChains is important for preserving the outputs and removing one of them leads to considerable drop in performance; (3) the sub-networks extracted for small regions of the input space can be generalized to unseen samples in nearby regions.