Knowledge Consistency between Neural Networks and Beyond

1 INTRODUCTION . Deep neural networks ( DNNs ) have shown promise in many tasks of artificial intelligence . However , there is still lack of mathematical tools to diagnose representations in intermediate layers of a DNN , e.g . discovering flaws in representations or identifying reliable and unreliable features . Traditional evaluation of DNNs based on the testing accuracy can not insightfully examine the correctness of representations of a DNN due to leaked data or shifted datasets ( Ribeiro et al. , 2016 ) . Thus , in this paper , we propose a method to diagnose representations of intermediate layers of a DNN from the perspective of knowledge consistency . I.e . given two DNNs pre-trained for the same task , no matter whether the DNNs have the same or different architectures , we aim to examine whether intermediate layers of the two DNNs encode similar visual concepts . Here , we define the knowledge of an intermediate layer of the DNN as the set of visual concepts that are encoded by features of an intermediate layer . This research focuses on the consistency of “ knowledge ” between two DNNs , instead of comparing the similarity of “ features. ” In comparison , the feature is referred to as the explicit output of a layer . For example , two DNNs extract totally different features , but these features may be computed using similar sets of visual concepts , i.e . encoding consistent knowledge ( a toy example of knowledge consistency is shown in the footnote1 ) . Given the same training data , DNNs with different starting conditions usually converge to different knowledge representations , which sometimes leads to the over-fitting problem ( Bengio et al. , 2014 ) . However , ideally , well learned DNNs with high robustness usually comprehensively encode various types of discriminative features and keep a good balance between the completeness and the discrimination power of features ( Wolchover , 2017 ) . Thus , the well learned DNNs are supposed to converge to similar knowledge representations . In general , we can understand knowledge consistency as follows . Let A and B denote two DNNs learned for the same task . xA and xB denote two intermediate-layer features of A and B , respec- ∗Ruofan Liang and Tianlin Li contribute equally to this research . Quanshi Zhang is the corresponding author with the John Hopcroft Center and MoE Key Lab of Artificial Intelligence AI Institute , Shanghai Jiao Tong University , China . zqs1022 @ sjtu.edu.cn 1As a toy example , we show how to revise a pre-trained DNN to generate different features but represent consistent knowledge . The revised DNN shuffles feature elements in a layer xrev = Ax and shuffles the feature back in the next convolutional layer x̂ = Wrev · xrev , where Wrev = WA−1 ; A is a permutation matrix . The knowledge encoded in the shuffled feature is consistent with the knowledge in the original feature . tively . Features xA and xB can be decomposed as xA = x̂A + A and xB = x̂B + B , where neural activations in feature components x̂A and x̂B are triggered by same image regions , thereby representing consistent knowledge . Accordingly , feature components x̂A and x̂B are termed consistent features between A and B . Then , feature components A and B are independent with each other , and they are termed inconsistent features . We assume that consistent components x̂A , x̂B can reconstruct each other , i.e . x̂A can be reconstructed from x̂B ; vice versa . In terms of applications , knowledge consistency between DNNs can be used to diagnose feature reliability of DNNs . Usually , consistent components ( x̂A , x̂B ) represent common and reliable knowledge . Whereas , inconsistent components ( A , B ) mainly represent unreliable knowledge or noises . Therefore , in this paper , we propose a generic definition for knowledge consistency between two pre-trained DNNs , and we develop a method to disentangle consistent feature components from features of intermediate layers in the DNNs . Our method is task-agnostic , i.e . ( 1 ) our method does not require any annotations w.r.t . the task for evaluation ; ( 2 ) our method can be broadly applied to different DNNs as a supplement evaluation of DNNs besides the testing accuracy . Experimental results supported our assumption , i.e . the disentangled consistent feature components are usually more reliable for the task . Thus , our method of disentangling consistent features can be used to boost performance . Furthermore , to enable a solid research on knowledge consistency , we consider the following issues . • Fuzzy consistency at different levels ( orders ) : As shown in Fig . 1 , the knowledge consistency between DNNs needs to be defined at different fuzziness levels , because there is no strict knowledge consistency between two DNNs . The level of fuzziness in knowledge consistency measures the difficulty of transforming features of a DNN to features of another DNN . A low-level fuzziness indicates that a DNN ’ s feature can directly reconstruct another DNN ’ s feature without complex transformations . • Disentanglement & quantification : We need to disentangle and quantify feature components , which correspond to the consistent knowledge at different fuzziness levels , away from the chaotic feature . Similarly , we also disentangle and quantify feature components that are inconsistent . There does not exist a standard method to quantify the fuzziness of knowledge consistency ( i.e . the difficulty of feature transformation ) . For simplification , we use non-linear transformations during feature reconstruction to approximate the fuzziness . To this end , we propose a model gk for feature reconstruction . The subscript k indicates that gk contains a total of k cascaded non-linear activation layers . x̂A = gk ( xB ) represents components , which are disentangled from the feature xA of the DNN A and can be reconstructed by the DNN B ’ s feature xB . Then , we consider x̂A = gk ( xB ) to represent consistent knowledge w.r.t . the DNN B at the k-th fuzziness level ( or the k-th order ) . x̂A = gk ( xB ) is also termed the k-order consistent feature of xA w.r.t . xB . In this way , the most strict consistency is the 0-order consistency , i.e . x̂A = g0 ( xB ) can be reconstructed from xB via a linear transformation . In comparison , some neural activations in the 1-order consistent feature x̂A = g1 ( xB ) are not directly represented by xB and need to be predicted via a non-linear transformation . The smaller k indicates the less prediction involved in the reconstruction and the stricter consistency . Note that the number of non-linear operations k is just a rough approximation of the difficulty of prediction , since there are no standard methods to quantify prediction difficulties . net.pdf More crucially , we implement the model g as a neural network , where k is set as the number of nonlinear layers in g. As shown in Fig . 2 , g is designed to disentangle and quantify consistent feature components of different orders between DNNs . Our method can be applied to different types of DNNs and explain the essence of various deep-learning techniques . 1 . Our method provides a new perspective for explaining the effectiveness of knowledge distillation . I.e . we explore the essential reason why the born-again network ( Furlanello et al. , 2018 ) exhibits superior performance . 2 . Our method gives insightful analysis of network compression . 3 . Our method can be used to diagnose and refine knowledge representations of pre-trained DNNs and boost the performance without any additional annotations for supervision . Contributions of this study can be summarized as follows . ( 1 ) In this study , we focus on a new problem , i.e . the knowledge consistency between DNNs . ( 2 ) We define the knowledge consistency and propose a task-agnostic method to disentangle and quantify consistent features of different orders . ( 3 ) Our method can be considered as a mathematical tool to analyze feature reliability of different DNNs . ( 4 ) Our method provides a new perspective on explaining existing deep-learning techniques , such as knowledge distillation and network compression . 2 RELATED WORK . In spite of the significant discrimination power of DNNs , black-box representations of DNNs have been considered an Achilles ’ heel for decades . In this section , we will limit our discussion to the literature of explaining or analyzing knowledge representations of DNNs . In general , previous studies can be roughly classified into the following three types . Explaining DNNs visually or semantically : Visualization of DNNs is the most direct way of explaining knowledge hidden inside a DNN , which include gradient-based visualization ( Zeiler & Fergus , 2014 ; Mahendran & Vedaldi , 2015 ) and inversion-based visualization ( Dosovitskiy & Brox , 2016 ) . ( Zhou et al. , 2015 ) developed a method to compute the actual image-resolution receptive field of neural activations in a feature map of a convolutional neural network ( CNN ) , which is smaller than the theoretical receptive field based on the filter size . Based on the receptive field , six types of semantics were defined to explain intermediate-layer features of CNNs , including objects , parts , scenes , textures , materials , and colors ( Bau et al. , 2017 ; Zhou et al. , 2018 ) . Beyond visualization , some methods diagnose a pre-trained CNN to obtain insight understanding of CNN representations . Fong and Vedaldi ( Fong & Vedaldi , 2018 ) analyzed how multiple filters jointly represented a specific semantic concept . ( Selvaraju et al. , 2017 ) , ( Fong & Vedaldi , 2017 ) , and ( Kindermans et al. , 2018 ) estimated image regions that directly contributed the network output . The LIME ( Ribeiro et al. , 2016 ) and SHAP ( Lundberg & Lee , 2017 ) assumed a linear relationship between the input and output of a DNN to extract important input units . ( Zhang et al. , 2019 ) proposed a number of metrics to evaluate the objectiveness of explanation results of different explanation methods , even when people could not get ground-truth explanations for the DNN . Unlike previous studies visualizing visual appearance encoded in a DNN or extracting important pixels , our method disentangles and quantifies the consistent components of features between two DNNs . Consistent feature components of different orders can be explicitly visualized . Learning explainable deep models : Compared to the post-hoc explanations of DNNs , some studies directly learn more meaningful CNN representations . Previous studies extracted scene semantics ( Zhou et al. , 2015 ) and mined objects ( Simon & Rodner , 2015 ) from intermediate layers . In the capsule net ( Sabour et al. , 2017 ) , each output dimension of a capsule usually encoded a specific meaning . ( Zhang et al. , 2018b ) proposed to learn CNNs with disentangled intermediate-layer representations . The infoGAN ( Chen et al. , 2016 ) and β-VAE ( Higgins et al. , 2017 ) learned interpretable input codes for generative models . Mathematical evaluation of the representation capacity : Formulating and evaluating the representation capacity of DNNs is another emerging direction . ( Novak et al. , 2018 ) proposed generic metrics for the sensitivity of network outputs with respect to parameters of neural networks . ( Zhang et al. , 2017 ) discussed the relationship between the parameter number and the generalization capacity of deep neural networks . ( Arpit et al. , 2017 ) discussed the representation capacity of neural networks , considering real training data and noises . ( Yosinski et al. , 2014 ) evaluated the transferability of filters in intermediate layers . Network-attack methods ( Koh & Liang , 2017 ; Szegedy et al. , 2014 ; Koh & Liang , 2017 ) can also be used to evaluate representation robustness by computing adversarial samples . ( Lakkaraju et al. , 2017 ) discovered knowledge blind spots of the knowledge encoded by a DNN via human-computer interaction . ( Zhang et al. , 2018a ) discovered potential , biased representations of a CNN due to the dataset bias . ( Deutsch , 2018 ) learned the manifold of network parameters to diagnose DNNs . ( Huang et al. , 2019 ) quantified the representation utility of different layerwise network architectures in the scenario of 3D point cloud processing . Recently , the stiffness ( Fort et al. , 2019 ) was proposed to evaluate the generalization of DNNs . The information-bottleneck theory ( Wolchover , 2017 ; Schwartz-Ziv & Tishby , 2017 ) provides a generic metric to quantify the information contained in DNNs . The information-bottleneck theory can be extended to evaluate the representation capacity of DNNs ( Xu & Raginsky , 2017 ; Cheng et al. , 2018 ) . ( Achille & Soatto , 2018 ) further used the information-bottleneck theory to improve feature representations of a DNN . ( Guan et al. , 2019 ; Ma et al. , 2019 ) quantified the word-wise/pixelwise information discarding during the layerwise feature propagation of a DNN , and used the information discarding as a tool to diagnose feature representations . The analysis of representation similarity between DNNs is most highly related to our research , which has been investigated in recent years ( Montavon et al. , 2011 ) . Most previous studies ( Wang et al. , 2018 ; Kornblith et al. , 2019 ; Raghu et al. , 2017 ; Morcos et al. , 2018 ) compared representations via linear transformations , which can be considered as 0-order knowledge consistency . In comparison , we focus on high-order knowledge consistency . More crucially , our method can be used to refine network features and explain the success of existing deep-learning techniques .

Summary: The goal of this paper is to analyze knowledge consistency between pretrained deep neural nets. In order to do so the paper trains neural networks to predict a hidden layer of one DNN using a hidden layer of another DNN. The model is interesting in that it is multi layer but it also allows decomposing its prediction as the sum of outputs of neural nets with different numbers of hidden layers. The prediction is dubbed "consistent features", which are further decomposed in consistent features of different complexity levels, while the error is dubbed "inconsistent features".