Knowledge Consistency between Neural Networks and Beyond
Authors: Ruofan Liang
Tianlin Li
Longfei Li
Jing Wang
Quanshi Zhang
Venue:  (2020)
Keywords: Deep Learning
Interpretability
Convolutional Neural Networks

Abstract: This paper aims to analyze knowledge consistency between pre-trained deep neural networks. We propose a generic definition for knowledge consistency between neural networks at different fuzziness levels. A task-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As a generic tool, our method can be broadly used for different applications. In preliminary experiments, we have used knowledge consistency as a tool to diagnose representations of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance.

Published as a conference paper at ICLR 2020
KNOWLEDGE CONSISTENCY BETWEEN NEURAL NET- WORKS AND BEYOND
Ruofan Liang" Tianlin Li Longfei , Jing Wang", and Quanshi Zhang "Shanghai Jiao Tong University, "Huawei Technologies
ABSTRACT
This paper aims to analyze knowledge consistency between pre-trained deep neu- ral networks. We propose generic definition for knowledge consistency between neural networks at different fuzziness levels. A tusk-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As generic tool, our method can be broadly used for different applications. In preliminary experi- tool to diagnose representations
ments, we have used knowledge consistency of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance.
INTRODUCTION
Deep neural networks (DNNs) have shown promise in many tasks of artificial intelligence. However, there is still lack of mathematical tools to diagnose representations in intermediate layers of a DNN, e.g. discovering flaws in representations or identifying reliable and unreliable features. Traditional evaluation of DNNs based on the testing accuracy cannot insightfully examine the correctness of representations of a DNN due to leaked data or shifted datasets (Ribeiro et al. 20166).
Thus, in this paper, we propose method to diagnose representations of intermediate layers of DNN from the perspective of knowledge consistency. I.e. given two DNNs pre-trained for the same task, matter whether the DNNs have the same different architectures, aim to examine whether intermediate layers of the two DNNs encode similar visual concepts.
Here, define the knowledge of an intermediate layer of the DNN as the set of visual concepts that are encoded by features of an intermediate layer. This research focuses on the consistency of "knowledge" between the feature is referred to as the explicit output of a layer. For example, two DNNs extract totally different features, but these features may be computed using similar sets of visual concepts, i.e. encoding consistent knowledge (a toy example of knowledge consistency is shown in the footnote
DNNs, instead of comparing the similarity of "features." In comparison,
Given the same training data, DNNs with different starting conditions usually converge to differ- ent knowledge representations, which sometimes leads to the over-fitting problem (Bengio et al.) 2014]. However, ideally, well learned DNNs with high robustness usually comprehensively encode various types of discriminative features and keep a good balance between the completeness and the discrimination power of features (Wolchover) 2017). Thus, the well learned DNNs are supposed to converge to similar knowledge representations.
In general, we can understand knowledge consistency as follows. Let A and B denote two DNNs learned for the same task. ZA and æn denote two intermediate-layee features of A and B, respec-
'Ruofan Liang and Tianlin Li contribute equally to this research. Quanshi Zhang is the corresponding author with the John Hopcroft Center and MoE Key Lab of Artificial Intelligence Al Institute, Shanghai Jiao Tong University. China. 12qs10228e jtu. u..uuee edu.cn
As toy example, we show how to revise a pre-trained DNN to generate differrntt features but represent consistent knowledge. The revised DNN shuffles feature elements in layer Zrev = Ax and shuffles the feature back in the next convolutional layer é W Jaa. where W W WA'" A1; A is permutation matrix. The knowledge encoded in the shuffled feature is consistent with the knowledge in the original feature.

---

Published as a conference paper at ICLR 2020
Disentanged into
Consistent with
Net A
9o (x)
Net B
0-order
L-odder
2-order
Targel
Target
0-order
I-order
2-order Inconsistent
O-order
1-order
2-order
Inconsistent
feature
feature
Figure I: Knowledge consistency. We define, disentangle, and quantify consistent features between two DNNs. Consistent features disentangled from a filter are visualized at different orders (levels) of fuzziness.
tively. Features zд and æp can be decomposed as =,A = &A + CA and zв = åD + cB, where neural activations in feature components f and žø triggered by same image regions, thereby repre- senting consistent knowledge. Accordingly, feature components & and &g are termed consistent features between and B. Then, feature components e and eg independent with each oth- er, and they are termed inconsistent features. We assume that consistent components A &B can reconstruct each other, i.e. a can be reconstructed from &g; vice versa.
In terms of applications, knowledge consistency between DNNs can be used to diagnose feature re- liability of DNNs. Usually, consistent components & represent common and reliable knowl- edge. Whereas, inconsistent components (cA, cB) mainly represent unreliable knowledge or noises.
Therefore, in this paper, we propose a generic definition for knowledge consistency between two pre-trained DNNs, and we develop a method to disentangle consistent feature components from features of intermediate layers in the DNNs. Our method task- aask-agnostic, i.e. (1) our method does not require any annotations w.r.f the task for evaluation; (2) our method can be broadly applied to different DNNs as a supplement evaluation of DNNs besides the testing accuracy. Experimental results supported our assumption, iee. the disentangled consistent feature components are usually more reliable for the task. Thus, our method of disentangling consistent features can be used to boost performance.
Furthermore, to enable a solid research on knowledge consistency, we consider the following issues.
Fuzzy consistency at different levels (orders): As shown in Fig. | the knowledge consistency between DNNs needs to defined at different fuzziness levels, because there no strict knowledge consistency between two DNNs. The level of fuzziness in knowledge consistency measures the difficulty of transforming features of a DNN to features another DNN. A low-level fuzziness indicates that DNN's feature directly reconstruct another DNN's feature without complex transformations.
Disentanglement & quantification: We need to disentangle and quantify feature components, which correspond to the consistent knowledge at different fuzziness levels, away from the chaotic feature. Similarly, also disentangle and quantify feature components that inconsistent.
There does not exist a standard method to quantify the fuzziness of knowledge consistency (i.e. the difficulty of feature transformation). For simplification, we use non-linear transformations during feature reconstruction to approximate the fuzziness. To this end, we propose model for feature reconstruction. The subscript & indicates that şu contains total of k cascaded non-linear activation layers. A gl (Ir) represents components, which are disentangled from the feature æ of the DNN and can be reconstructed by the DNN B's feature TR. Then, we consider à = gu (IR) to represent consistent knowledge w.r.f. the DNN B at the k-th fuzziness level (or the k-th order). a (* also termed the k-order consistent feature of A w.r.f. æp-
In this way, the most strict consistency is the 0-order consistency, iee. 2 (0( can be recon- structed from øB via a linear transformation. In comparison, some neural activations in the l-order consistent feature A (1(2)) are not directly represented by and need to be predicted via non-linear transformation. The smaller & indicates the less prediction involved in the reconstruction and the stricter consistency. Note that the number of non-linear operations & is just rough approx- imation of the difficulty of prediction, since there are no standard methods to quantify prediction difficulties.
2

---

Published as a conference paper at ICLR 2020
p
(
X
Conv
Norm
ReLUU
Aahiion
Conv
Norm
ReLU
Branch 3
Addition
Conv
Norm
ReLU
Addittion
Conv
Brunch 2
(x))
Branch
Branch 0
Figure 2: Neural network for disentanglement of consistent features (K 3).
More crucially, implement the model g as neural network, where is set the number of non- linear layers in g. As shown in Fig. 2 g is designed to disentangle and quantify consistent feature components of different orders between DNNs. Our method can be applied different types of DNNs and explain the essence of various deep-learning techniques.
1. Our method provides new perspective for explaining the effectiveness of knowledge distillation.
Lee we explore the essential reason why the born-again network (Furlanello et al. 2018 exhibits superior performance.
2. Our method gives insightful analysis of network compression.
Our method can be used to diagnose and refine knowledge representations of pre-trained DNNs and boost the performance without any additional annotations for supervision.
Contributions of this study can be summarized as follows. (1) In this study, we focus on a new prob- lem, i.e. the knowledge consistency between DNNs. (2) We define the knowledge consistency and propose a task-agnostic method to disentangle and quantify consistent features of different orders. (3) Our method can be considered as mathematical tool to analyze feature reliability of different DNNs. (4) Our method provides new perspective on explaining existing deep-learning techniques, such knowledge distillation and network compression.
2 RELATED WORK
In spite of the significant discrimination power DNNs, black-box representations of DNNs have been considered Achilles' heel for decades. In this section, will limit discussion to the lit- crature explaining or analyzing knowledge representations of DNNs. general, previous studies be roughly classified into the following three types.
Explaining DNNs visually or semantically: Visualization of DNNs is the most direct way of explaining knowledge hidden inside DNN, which include gradient-basee visualization (Zeiler & Fergus 2014 Mahendran & Vedaldi 2015) and inversion-basse visualization (Dosovitskiy & Brox] 2016). Zhou |l.]||111, 2015) developed method compute the actual image-resolution receptive field of neural activations in feature map of a convolutional neural network (CNN), which is smaller than the thcoretical receptive field based on the filter size. Based on the receptive field, six types of semantics were defined to explain intermediate-lan layer features of CNNs, including objects, parts, scenes, textures, materials, and colors (Bau et al.. 2017 Zhou al. 2018).
Beyond visualization, some methods diagnose a pre-trained CNN to obtain insight understanding of CNN representations. Fong and Vedaldi (Fong & Vedaldi 2018) analyzed how multiple filters jointly represented specific semantic concept. (Selvaraju al. 2017]. (Fong & Vedaldi 2017). and Kindermans et al. 2018) estimated image regions that directly contributed the network output. The LIME (Ribeiro et al. 2016) and SHAP (Lundberg & Lee] (017)) assumed linear relationship be- tween the input and output of DNN to extract important input units. (Zhang et al. (2019) proposed number of metries to evaluate the objectiveness of explanation results of different explanation methods, even when people could not get ground-trun -truth explanations for the DNN.
Unlike previous studies visualizing visual appearance encoded in DNN extracting important pixels, our method disentangles and quantifies the consistent components features between two DNNs. Consistent feature components of different orders can be explicitly visualized.
3

---

Published as a conference paper at ICLR 2020
Learning explainable deep models: Compared to the post-hoc explanations of DNNs, some stud- ies directly learn more meaningful CNN representations. Previous studies extracted scene seman- ties (Zhou et al.] 2015) and mined objects (Simon & Rodner 2013) from intermediate layers. In the capsule net Sabour et al. 2017), each output dimension of a capsule usually encoded a specific meaning. (Zhang al. 20T8b) proposed to learn CNNs with disentangled intermediate-laye repre-
sentations. The infoGAN (Chen et al. 2016) and 3-VAE (Higgins et al. (017)) learned interpretable input codes for generative models.
Mathematical evaluation of the representation capacity: Formulating and evaluating the repre- sentation capacity of DNNs is another emerging direction. (Novak et al.) 2018) proposed generic metrics for the sensitivity of network outputs with respect to parameters of neural networks. (Zhang et al., 2017) discussed the relationship between the parameter number and the generalization ca- pacity of deep neural networks. (Arpit et al. (017) discussed the representation capacity of neural networks, considering real training data and noises. (Yosinski et al., (2044) evaluated the transferabil- ity of filters in intermediate layers. Network-attack methods (Koh & Liang 2017: Szegedy et al., 2014 Koh & Liang 2017) can also be used to evaluate representation robustness by computing adversarial samples. (Lakkaraju et al., 2017) discovered knowledge blind spots of the knowledge encoded by a DNN via human-computer interaction. (Zhang et al.] (2018a) discovered potential, biased representations of CNN due to the dataset bias. (Deutsch, 2018) learned the manifold of network parameters to diagnose DNNs. (Huang et al.) 2019) quantified the representation utility of different layerwise network architectures in the scenario of 3D point cloud processing. Recently, the stiffness (Fort et al. (019) was proposed to evaluate the generalization of DNNs.
theory (Wolchover, 2017; Schwartz-Ziv & Tishby 2017 provides generic metric to quantify the information contained in DNNs. The information
The information-bottier
informatiio--botlenedd bottleneck theory
can be extended to evaluate the representation capacity of DNNs (Xu & Raginsky, 2017; Cheng et al. 2018). (Achille & Soatto 2018) further used the information conmaaioon-botteneckk. bottleneck theory to improve feature representations a DNN. (Guan et al., 2019 Ma et al. (019) quantified the word-wise/pixel- wise information discarding during the layerwise feature propagation of a DNN, and used the infor- mation discarding as a tool to diagnose feature representations.
The analysis of representation similarity between DNNs is most highly related to research, which has been investigated in recent years (Montavon et al. 2011). Most previous studies (Wang et al.) 2018 Korbblith et al. 2019 Raghu al. [017]] Morcos al. 2018) compared representations via linear transformations, which can be considered as O-order knowledge consistency. In comparison, focus on high-order knowledge consistency. More crucially, our method be used to refine network features and explain the success of existing deep-learning techniques.
ALGORITHM
In this section, we will introduce the network architecture to disentangle feature components of consistent knowledge at a certain fuzziness level, when we use the intermediate-layer feature æ of DNN to reconstruct intermediate-lay
feature x' of another DNN As shown in Fig 2 the network g with parameters 0 has a recursive architecture with K + blocks. The function of the k-th block is given as follows.
h (k) = (K)) [r + x higher]
x higher = R (Sutt)aatti) = K-1 (1)
The output feature is computed using both the raw input and the feature of the higher order (++1)
((*) denotes a linear operation without bias term. The last block is given as (R) = (x), This linear operation can be implemented as either a layer in an MLP or a convolutional layer in a CNN. ((k-1)
of............................ o is referred to diagonal matrix for element-wiss variance of (0+)) where o2 the variance of the m-th element of & (1++) through various input images. Σ{++1y is used to normalize the magnitude of neural activations. Because the normalization, the sealar value (*++1) roughly controls the activation magnitude of A (l+1) W.E.t (k), (0) go (x(x) corresponds to final output of the network.
Re-scaling the magnitude of all neural activations does not affect the knowledge encoded in the DNN. Therefore, we normalize and x² to zero mean and unit variance remove the magnitude effect.
4

---

Published as a conference paper at ICLR 2020
In this way, the entire network can be separated into (K + 1) branches (see Fig. where the k-th branch (k I X) contains a total of k non-linear layers. Note that the order consistent knowledge can also be represented by the network with 12-th branch, if ln < k2.
In order to disentangle consistent features of different orders, the k-th branch is supposed to exclu- sively encode the k-order consistent features without representing lower-order consistent features. Thus, we propose the following loss to guide the learning process.
(210)2
Loss (0) = (1)0(x)) x=
where and x' denote intermediate-laye features of two pre-trained DNNs. The second term in this loss penalizes neural activations from high-order branches, thereby forcing as much low-order consistent knowledge possible to be represented by low-order branches.
Furthermore, based on (K + 1) branches of the network, we can disentangle features of xt into (K + 2) additive components.
(0(x))
go(x) (0) + +.)
x (K)
where _² = (z) indicates feature components that cannot be represented by x. (+) denotes feature components that are exclusively represented by the k-order branch.
Based on Equation ( the signal-processing procedure for the k-th feature component can be rep- resented Conv, Norma + ReLU& + kk) Conuk-1 -... ReLUi + D0 Convo (see Fig. 2) Therefore, we can disentangle from Equation (1) all linear and non-linear operations along
the entire k-th branch follows, and in this way, (1()) can be considered as the k-order consistent feature component.
A
(k) = w(O)
II
p
D Wr(d)) x
(4)
where A()) diag(a1, ......, as) ) is diagonal matrix. Each diagonal element am E {0, 1} rep- resents the binary information-passing state of 2m through an ReLU layer (1 S m M). Each
element given as am 1wmm > 0), where LeLU(W*(()) + ReLU (zzhihhrr)r) (z but A A(0 + =dightrr) ((OW>* 2 Thus, we record such information-pasee states to decompose feature components.
T Note that ReLU + xlishher) #
4 COMPARATIVE STUDIES
As a generic tool, knowledge consistency based on g can be used for different applications. In order to demonstrate utilities of knowledge consistency, we designed various comparative studies, including (1) diagnosing and debugging pre-trained DNNs, (2) evaluating the instability of learning DNNs, (3) feature refinement of DNNs, (4) analyzing information discarding during the compres- sion of DNNs, and (5) explaining effects of knowledge distillation in knowledge representations.
A total of five typical DNNs for image classification were used, i.e. the AlexNet (Krizhevsky jt al. 2012), the VGG-16 (Simonyan & Zisserman 2015). and the 18,. ResNet-34, ResNet- 50 (He et al. 2016). These DNNs were learned using three benchmark datasets, which included the CUB200-201 dataset (Wah et al.) 2011). the Stanford Dogs dataset (Khosla et al. 2011). and the Pascal VOC 2012 dataset (Everingham al. 2013). Note that both training images and testing im- ages were cropped using bounding boxes of objects. We set X 0.1 for all experiments, except for feature reconstruction of AlexNet (we set 8.0 for AlexNet features). It was because the shallow model of the AlexNet usually had significant noisy features, which caused considerable inconsistent components.
4.1 NETWORK DIAGNOSIS BASED ON KNOWLEDGE CONSISTENCY
The most direct application of knowledge consistency is to strong (well learned) DNN to diag- nose representation flaws hidden in a weak DNN. This is of special values in real applications, e.g. shallow (usually weak) DNNs more suitable to be adopted mobile devices than deep DNNs.
5

---

Published as a conference paper at ICLR 2020
features
Peatumos the week
by
Blind spets
features
Featumes the weak LNN
Blind spots
features
Festures the strong DNK
Unneliable features
features
the
moumemucted
Featumes
DNN
Unreliable
(a) Blind spots the
(a) Unreliable/noisy features the DNN
Figure 3: Unreliable components and blind spots of weak DNN (AlexNet) w.r.f strong DNN (ResNet-34). Please see Section [.1 for definitions of "unreliable components" and "blind spots." (left) When we used features of the weak DNN reconstruct features of the strong DNN, we visualize raw features of the strong DNN. feature components that can be reconstructed, and blind spots (x*) disentangled from features of the strong DNN. The weak DNN mainly encoded the head appearance and ignored others. We can find that some patterns in the torso are blind spots of the weak DNN. (right) When we used features of the strong DNN to reconstruct features of the weak DNN, we visualize raw features of the weak DNN, feature components that can be reconstructed, and unreliable features (x²) disentangled from features of the weak DNN. Based on visualization results, unreliable features of the weak DNN usually repesent noisy activations.
Let two DNNs be trained for the same task, and one DNN significantly outperforms the other DNN. We assume that the strong DNN has encoded ideal knowledge representations of the target task. The weak DNN may have the following two types of representation flaws.
Unreliable features in the weak DNN are defined as feature components, which cannot be recon-
structed by features of the strong DNN. (see Appendix for detailed discussions).
Blind spots of the weak DNN are defined as feature components in the strong DNN, which are inconsistent with features of the weak DNN. These feature components usually reflect blind spots of the knowledge of the weak DNN (see Appendix B for detailed discussions).
For implementation, we trained DNNs for fine-grained classification using the CUB200-201 dataset Wah et al., (201T) (without data augmentation). We considered the AlexNet (Krizhevsky cet al.) 2012) as the weak DNN (56.97% top-1 accuracy), and took the ResNet-34 (He et al. 2016) as the strong DNN (73.09% top-1 accuracy).
Please see Fig. We diagnosed the output feature of the last convolutional layer in the AlexNet, which is termed AAA Accordingly, we selected the last 14 x 14 x 256 feature map of the ResNet-34 (denoted by #x) for the computation of knowledge consistency, because FA and za had similar map sizes. We disentangled and visualized unreliable components from v. (i.e. inconsistent components in T.A). We also visualized components disentangled from xp (i.e. inconsistent components in zg), which corresponded to blind spots of the weak DNN's knowledge.
Furthermore, we conducted two experiments to further verify the claim of blind spots and unreliable features in a DNN. The first experiment aimed to verify blind spots. The basic idea of this experiment was to examine the increase of the classification accuracy, when we added information of blind spots to the raw feature. We followed above experimental settings, which used the intermediate- layer feature A of the AlexNet (with 56.97% top-1 accuracy) to reconstruct the intermediate-layer feature z of Resnet-34 (with 73.09% top-1 accuracy). Then, inconsistent feature components were termed blind spots. In this way, we added feature components of blind spots back to the AlexNet's feature (adding these features back to go (z A)), and then learned a classifier upon the new feature. To enable a fair comparison, the classifier had the same architecture as the AlexNet's 's modules above AA and during the learning process, we fixed parameters in DNN A and to avoid the revision of such parameters affecting the performance. We found that compared to the raw feature AA of the AlexNet, the new feature boosted the classification accuracy by 16.1%.
The second experiment was conducted to verify unreliable features. The basic idea of this experi- ment was to examine the increase of the classification accuracy, when we removed information of unreliable features from the raw feature. We designed two classifiers, which had the same architec- ture as the AlexNet's modules above AA We fed the raw feature I'A to the first classifier. Then, we removed unreliable feature components from A (i.e. obtaining go (xB)), and fed the revised feature to the second classifier. We learned these two classifiers, and classifiers with and without unreliable features exhibited classification accuracy of 60.3% and 65.6%, respectively.
Therefore, above two experiments successfully demonstrated that both the insertion of blind spots and the removal of unreliable features boosted the classification accuracy.

---

Published as a conference paper at ICLR 2020
Learning DNNs from different initializations
conv4 @ AlexNet conv5 @ AlexNet conv4-3 VGG-16 conv5-3 @ VGG-16
last @ ResNet-34
0.086
0.116
0.124
0.196
0.776
Learning DNNs using different training data
conv4 @ AlexNet conv5 @ AlexNet conv4-3 VGG-16 conv5-3 VGG-16
last conv @ ResNet-34 0.275
0.089
0.155
0.121
0.198
Table I: Instability of learning DNNs from different initializations and instability of learning DNNs using different training data. Without huge training set, networks with more layers (e.g. ResNet- 34) usually suffered more from the over-fitting problem.
AlexNet
conv3 AlexNet conv4-3 VGG-I6
conv5-3 VGG-I6
last ResNet-34
)
105.
424.67
1.06
0.88
Var(z(i))
10.51
73.73
0.07
0.03
Var(z(81)
43.69
0.02
0.004
Var(za)
11.14
71.37
0.16
0.22
2.75
Table 2: Magnitudes consistent features of different orders, when we train DNNs from different initializations. Features in different layers have significantly different variance magnitudes. Most neural activations of the feature belong to low-order consistent components.
4.2 STARILITY OF LEARNING
The stability of learning DNNS is of considerable values in deep learning, i.e. examining whether not all DNÑs represent the same knowledge, when people repeatedly learn multiple DNNs for the same task. High knowledge consistency between DNNs usually indicates high learning stability.
More specifically, we trained two DNNs (A, B) with the same architecture for the same task. Then, disentangled inconsistent feature components अं = - (o(()) and 2 =zg go(za) from their features and 2B of a specific layer, respectively. Accordingly, ge(ze) and go ge(xa) corresponded consistent feature components in IA and zB, respectively, whose knowledge was shared by two networks. The inconsistent feature x was quantified by the variance of feature element through different units of z and through different input images Var(=²) def Br,.((=1111 By e ()))))))) where z denotes the i-th element of z given the image T. We can Var (var)/Vr((()) to measure the instability of learning DNNs A and B.
In experiments, evaluated the instability of learning the AlexNet (Krizhevsky et al. 2012). the VGG-16 (Sioonyan & Zisserman 2015), and the ResNet-34 (He et al. (2016). We considered the following cases.
Case 1, learning DNNs from different initializations using the same training data: For each net- work architecture, trained multiple networks using the CUB200-2011 dataset (Wah al. [00IT) The instability of learning DNNs was reported as the average of instability over all pairs of neural networks.
Case 2, learning DNNs using different sets of training data: We randomly divided all training sam- ples in the CUB200-2011 dataset (Wah al.] (0IIT) into two subsets, each containing 50% sam- ples. For each network architecture, we trained two DNNs (A, B) for fine-grained classification (without pre-training). The instability of learning DNNs was reported as (Var(=A)/Vartt + Var(e (Vr(==))2.
Table compares the instability of learning different DNNs. Table 2 reports the variance of con- sistent components of different orders. We found that the learning of shallow layers in DNNs was usually more stable than the learning of deep layers. The reason may as follows. DNN with more layers usually samples. Without a huge training set (e.g. the ImageNet dataset (Krizhevsky et al. (012)), a deep network may be more likely to suffer from the over-fitting problem, which causes high variances in Table iee. DNNs with different initial parameters may learn different knowledge representations.
represent more complex visual patterns, thereby needing more training
4.3 FEATURE REFINEMENT BASED ON KNOWLEDGE CONSISTENCY
Knowledge consistency can also be used to refine Intermediate-layee features of pre- pre-trained DNNs. Given multiple DNNs pre- treitrained for the same task, feature components, which are consistent with various DNNs, usually represent common knowledge and are reliable. Whereas, inconsistent feature components w.r.f. other DNNs usually correspond to unreliable knowledge noises. In this way,

---

Published as a conference paper at ICLR 2020
VGG-I6 conv4-3
VGG-I6 conv5-2
ResNet-18
ResNet-34
ResNet-50
Network A Network
43.13
34.74
31.05
29,98
42.89
35.00
30.46
31.15
z00)
45.15
44.48
38.16
31.49
30,40
x00) + +a(l))
44.98
44.22
38.45
31.76
31.77
x00) + +x(1))
+ 2
45.06
44.32
38.23
31.96
31.84
Table 3: Classification accuracy by using the original and the refined features. Features of DNN A were used to reconstruct features of DNN B. For residual networks, we selected the last feature map with size of 14 14 as the target for refinement. All DNNs were learned without data augmentation or pre-training. The removal of effects of additional parameters in ge is discussed in Section 443 fairness of comparisons and Appendix A Consistent components slightly boosted the performance.
intermediate-layer features can be refined by removing inconsistent components and exclusively using consistent components to accomplish the task.
More specifically, given two pre-trained DNNs, we use the feature of a certain layer in the first DNN to reconstruct the corresponding feature of the second DNN. The reconstructed feature is given as a. In this way, we can replace the feature of the second DNN with the reconstructed feature 2, and then use & to simultaneously learn all subsequent layers in the second DNN to boost performance. Note that for clear and rigorous comparisons, we only disentangled consistent feature components from the feature of single layer and refined the feature. It was because the simultancous refinement of features of multiple layers would increase it difficult to clarify the refinement of which feature made the major contribution.
In experiments, we trained DNNs with various architectures for image classification, including the VGG-16 (Simonyan & Zisserman 2015), the ResNet-18, the ResNet-34, and the ResNet-50 (He) ct al. 2016). We conducted the following two experiments, in which we used knowledge consistency to refine DNN features.
Exp. 1, removing unreliable features (noises): For each specific network architecture, we trained two DNNs using the CUB200-2011 dataset (Wah al., (01I]) with different parameter initializa- tions. Consistent components were disentangled from the original feature of a DNN and then used for image classification. As discussed in Section 4.1 consistent components can be considered as refined features without noises. We used the refined features as input to finetune the pre-trained up- per layers in the DNN B for classification. Table reports the increase of the classification accuracy by using the refined feature. The refined features slightly boosted the performance.
Fairness of comparisons: (1) To enable fair comparisons, we first trained go and then kept go un- changed during the finetuning of classifiers, thereby fixing the refined features. Otherwise, allowing ge to change would be equivalent to adding more layers/parameters to DNN B for classification. We needed to eliminate such effects for fair comparisons, which avoided the network from benefitting from additional layers/parameters. (2) As baselines, we also further finetuned the corresponding upper layers of DNNs A and B to evaluate their performance. Please see Appendix A for more discussions about how we ensured the fairness.
Exp. 2, removing redundant features from pre-trained DNNs: A typical deep-learning method- ology is to finetune DNN for a specific task, where the DNN is pre-trained for multiple tasks (including both the target and other tasks). This quite common in deep learning, egg. DNNs pre-trained using the ImageNet dataset (Deng et al.) 2009) are usually finetuned for various tasks. However, in this case, feature components pre-trained for other tasks are redundant for the target task and will affect the further finetuning process.
Therefore, we conducted three new experiments, in which our method removed redundant features w.r.l the target task from the pre-trained DNN. In Experiment 2.1 (namely VOC-animal), let two DNNs and B be learned to classify 20 object classes in the Pascal VOC 2012 dataset (Everingham al. 2015). We were also given object images of six animal categories (bird, cat, cow, dog, horse, sheep). We fixed parameters in DNNs A and B, and our goal was to use fixed DNNs A and B to generate clean features for animal categories without redundant features.
Let FΛ and ug denote intermediate- features of DNNs A and B. Our method used æA to reconstruct æa. Then, the reconstructed result (Ix)) corresponded reliable features for animals, while inconsistent components A indicated features of other categories. We used clean features (o())) learn the animal classifier. go were fixed during the learning of the animal classifier to
8

---

Published as a conference paper at ICLR 2020
VGG-16 conv4-3
VGG-IC 16 conv5-2
VOC-animal
Mix-CUB
Mix-Dogs
VOC-animal
Mix-CUB
Mix- Mix-Dogs
Features from the network A Features from the network 200) 20)) z(2)
$1.55
44,44
15.15
51.35
44.44
15.15
50.80
45.93
15.19
50.80
45.93
15.19
59.38
47.50
16.53
60.18
46.65
16.70
ResNet-18
ResNet-34
VOC-animal
Mix-CUB
Mix-Dogs
VOC-animal
Mix-CUB
Mix-Dogs
Features from the network A Features from the network z10) 2(1)) y(2)
37.65
31,93
14.20
39,42
30.91
12.96
37.22
32.02
14.28
35,95
27.74
12.46
53.52
38.02
16.17
49.98
33.98
14.21
Table 4: Top-1 classification accuracy before and after removing redundant features from DNNs. Original DNNs were learned from scratch without data augmentation. For residual networks, selected the last feature map with a size of 14 x 14 as the target for feature refinement. The removal of effects of additional parameters in go is discussed in Section 4.3 fairness of comparisons and Appendix|AL [A] Consistent features significantly alleviated the over-fitting problem, thereby exhibiting superior performance.
Var(e"") discarded Rratones during comproadion
Var Var(x
geraratuu
Gener
2001522
Dicandal fratum compereenns during comprooiion
Concbazee compererms in the dicillornd DAN
Pormagii neturdis
Figure 4: Effects of network compression and knowledge distillation. (left) We visualized the dis- carded feature components when 93.3% parameters of the DNN were eliminated. We found that the discarded components looked like trivial features and noises. In the curve figure, a low value of Var(=*) indicates lower information discarding during the compression, thereby exhibiting a higher accuracy. (right) We visualized and quantified knowledge blind spots of born-again networks in different generations. Nets in new generations usually had fewer blind spots than old nets.
enable a fair comparison. In comparison, the baseline method directly used either x or æB to finetune the pre-trained DNN to classify the six animal categories.
In Experiment 2.2 (termed Mix- Mix-CUB), two original DNNs were learned using both the CUB200- 2011 1 dataset (Wah et al.) 2011) and the Stanford Dogs dataset (Khosla et al.) (011) to classify both 200 bird species and 120 dog species. Then, our method used bird images to disentangle feature components for birds and then learned a new fine-grained classifier for birds. The baseline method was implemented following the same setting as in VOC-animal. Experiment 2.3 (namely Mix-Dogs) was similar Mix- Mix-CUB. Our method disentangled dog features away from bird features learn new dog classifier. In all above experiments, original DNNs were learned from scratch without data augmentation. Table compares the classification accuracy of different methods. It shows that method significantly alleviated the over-fitting problem and outperformed the baseline.
4.4 ANALYZING INFORMATION DISCARDING OF NETWORK COMPRESSION
Network compression is an emerging research direction in recent years. Knowledge consistency between the compressed network and the original network can evaluate the discarding of knowledge during the compression process. L.e. people may visualize analyze feature components in the original network, which are not consistent with features in the compressed network, to represent the discarded knowledge in the compressed network.
In experiments, we trained the VGG-16 using the CUB200-2011 dataset (Wah et al., al,, (011)) for fine- grained classification. Then, we compressed the VGG-16 using the method of (Han al.) 2016) with different pruning thresholds. We used features of the compressed DNN to reconstruct Teatures of the original DNN. Then, inconsistent components disentangled from the original DNN usually corresponded to the knowledge discarding during the compression process. Fig. 1eft) visualizes the discarded feature components. We used Var(= (defined in Section to quantify the information discarding. Fig. 4 compares the decrease of accuracy with the discarding of feature information.
4.5 EXPLAINING KNOWLEDGE DISTILLATION VIA KNOWLEDGE CONSISTENCY
As generic tool, method also explain the success of knowledge distillation. In particu- lar, (Furlanello et al. 2018) proposed a method to gradually refine a neural network via recursive knowledge distillation. Lee. this method recursively distills the knowledge of the current net

---

Published as a conference paper at ICLR 2020
Using Ty to recunsiruct X
Using x, to recomsiruct Xy
Turget feature O-oeder
1-order
2 order
Inconsistent
Target feature O O-order
l-onder
2-onder Inconsistent
Figure 5: Consistent and inconsistent feature components disentangled between DNN B learned for binary classification and DNN A learned for fine fineggrained classification.
new net with the same architecture and distilling the new net to an even newer net. The new(er) is termed a born-again neural network and learned using both the task loss and the distillation loss. Surprisingly, such a recursive distillation process can substantially boost the performance of the neural network various experiments.
In general. the net in a new generation both inherits knowledge from the old net and learns new knowledge from the data. The success of the born-again neural network can be explained as that knowledge representations of networks are gradually enriched during the recursive distillation pro- cess. To verify this assertion, in experiments, we trained the VGG-16 using the CUB200-200 2011 dataset (Wah et al. 201T for fine-grained classification. We trained born-again neural networks of another four generations We disentangled feature components in the newest DNN, which were not consistent with intermediate DNN. Inconsistent components
considered as blind spots of knowledge representations of the intermediate DNN and were quantified by V Var(z*), Fig. rright) shows V ))r(x*) of DNNs in the 1st, 2nd, 3rd, and 4th generations. Inconsistent components were gradually reduced after several generations.
4.6 CONSISTENT AND INCONSISTENT FEATURES BETWEEN DIFFERENT TASKS
In order to visualize consistent and inconsistent features between different tasks (Lee. the fine- ene-grained classification and the binary classification), trained DNN A (a VGG-16 network) simultane- ously classify 320 species, including 200 bird species in the CUB200-201 dataset (Wah et al. 2011T) and 120 dog species in the Stanford Dogs dataset (Khosla et al. 20TT), in fine-grained manner. On the other hand, we learned DNN B (another VGG-16 network) for the binary classification of the bird and the dog based on these datasets. We visualized the feature maps of consistent and inconsistent feature components between the two DNNs in Fig. Obviously, DNN A encoded more knowledge than DNN B, thereby A reconstructing more features than ZB.
5 CONCLUSION
In this paper, have proposed generic definition of knowledge consistency between intermediate- layers two DNNs. task-agnostic method developed to disentangle and quantify consistent features of different orders from intermediate- features. Consistent feature components usually more reliable than inconsistent components for the task, so our method can be used to further refine the pre-trained DNN without need for additional supervision. As a mathematical tool, knowledge consistency can also help explain existing deep-learning techniques, and experiments have demonstrated the effectiveness of our method.
ACKNOWLEDGEMENTS
This work partially supported by National Natural Science Foundation of China (U19B2043 and 61906120) and Huawei Technologies.
Because Furlanello et al. 2018 did not clarify the distillation loss, we applied the distillation loss
in (Hinton et al. 2014) following parameter settings
1.0 in Mishra & Marr) 2018) i.c. Loss
Loss + 0.5 distill
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: learning optimal representations through noise. Transactions on PAMI, 40(12))2897-2995, 2018.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look at memorization deep networks. In ICLR, 2017.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection:
Quantifying interpretability of deep visual representations. In CVPR, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. In 0rXiv:1206.553888 2014.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, 2016.
Hao Cheng, Dongze Lian, Shenghua Gao, and Yanlin Geng. Evaluating capability of deep neural networks for image classification via information plane. In ECCV, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Lior Deutsch. Generating neural networkswith neural networks. ar..iv:180000019222 2018.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In CVPR, 2016.
M. Everingham, M. A. Eslami, L. Van Gool, C. K. I. Williams, Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. In International Journal of Computer Vision (IJC), 111(1))98-136, 2015.
Ruth Fong and Andrea Vedaldi. Net2vec:
Quantifying and explaining how concepts are encoded by filters in deep neural networks. In CVPR, 2018.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In ICCV, 2017.
Stanislav Fort, Pawel Krzysztof Nowak, and Srini Narayanan. Stiffness: new perspective on generalization in neural networks. In ArXiv:1901.0900 2019.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, 2018.
Chaoyu Guan, Xiting Wang, Quanshi Zhang, Runjin Chen, Di He, and Xing Xie. Towards deep and unified understanding of deep neural models in nlp. In JCML, 2019.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. β-vae: learning basic visual concepts with con- strained variational framework. ICLR, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Workshop, 2014.
Shikun Huang, Binbin Zhang, Wen Shen, Zhihua Wei, and Quanshi Zhang. Utility analysis of network architectures for 3d point cloud processing. In arXiv: 1.00053, 2019.
11

---

Published as a conference paper at ICLR 2020
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First CVPR Workshop on Fine-Gree Grained Visual Categorization (FGVC), 2011.
Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternat- tribution. In ICLR, 2018.
PangWei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. arXiv:/905.004 2019.
A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In AAAI, 2017.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NIPS, 2017.
Haotian Ma, Yinqing Zhang, Fan Zhou, and Quanshi Zhang. Quantifying layerwise information discarding of neural networks. In 496iv: 2019.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. CVPR, 2015.
Asit K. Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. 2018.
Grégoire Montavon, Mikio L. Braun, and Klaus-Rober Müller. Kernel analysis of deep networks. In Journal of Machine Learning Research, 22:55332581. 2011.
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In NIPS, 2018.
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl- Dock--iccstein. Sensitivity and generalization in neural networks: An empirical study. In ICLR, 2018
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca:
Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NIPS, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In KDD, 2016.
Sara Sabour, Nicholas Frosst. and Geoffrey E. Hinton. Dynamic routing between capsules. In NIPS, 2017.
Ravid Schwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via infor- mation. In arXiv:/703.008/ 2017.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-base based local- ization. In ICCV, 2017.
Marcel Simon and Erik Rodner. Neural activation constellations: Unsupervised part model discov- ery with convolutional networks. In ICCV, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In arXiv: 1312.6199 2014.
12

---

Published as a conference paper at ICLR 2020
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds 2irds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In NIPS, 2018.
Natalie Wolchover. New theory cracks open the black box of deep learning. In Quanta Magazine, 2017.
Aolin Xu and Maxim Raginsky. cntemattio-thhoreete analysis of generalization capability of learn- ing algorithms. In NIPS, 2017.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Undersantding deep learning requires rethinking generalization. In ICLR, 2017.
Hao Zhang, Jiayi Chen, Haotian Xue, and Quanshi Zhang. Towards unified evaluation of explana- tion methods without ground truth. In arXiv:191 190o:1911.9917, 2019.
Quanshi Zhang, Wenguan Wang, and Song-Chus Zhu. Examining enn representations with respect to dataset bias. In AAAI, 2018a.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In CVPR, 2018b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. In ICLR, 2015.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. ECCV, 2018
13

---

Published as a conference paper at ICLR 2020
A FAIRNESS OF EXPERIMENTS IN SECTION 4.3
In Section used knowledge consistency refine intermediate-laye features of pre-trained DNNs. Given multiple DNNs pre-trained for the same task, feature components, which are con- sistent with various DNNs, usually represent common knowledge and are reliable. In this way, intermediate-layer features can be refined by removing inconsistent components and exclusively using consistent components accomplish the task.
Both Experiment and Experiment 2 followed the same procedure. Le. we trained two DNNs A and B for the same task. Then, extracted consistent feature components g(z,) when we used the DNN A's feature to reconstruct the DNN B's feature. We compared the classification performance of the DNN A, the DNN B, and the classifier learned based on consistent features g(
However, an issues of fairness may be raised, i.e. when we added the network upon the DNN A, this operation increased the depth the network. Thus, the comparison between the revised DNN and the original DNN A may be unfair.
In order to ensure fair comparison, we applied following experimental settings.
For the evaluation of DNNs A and B, both the DNNs were further refined using target training samples in Experiments and 2. For the evaluation our method, without using any annotations of the target task, we first trained g(Ea) to A to reconstruct Eg and disentangle consistent feature components. Then, we fixed all parameters DNNs A, B, and g. and only used output features of g( train classifier, order test the discrimination power of output features of g(
100
Reconstraction
g(X ,)
Image
X,
Image
Classifier
Classifier
DNN A
DNN B
As shown in the above figure, for the evaluation of our method, first, we annotations for the target task. Without any prior knowledge of the target task, we used the pre- trained DNNs (blue area in the above figure) train the network g (red area in the above figure). Then, we fixed parameters of DNNs in both red area and blue area in the above figure. Finally, we received training samples of the target task, and used them exclusively train the classifier (the blue area in the above figure).
given any training
short, the learning of the network, was independent with the target task, and parameters of A, B, and g were not refine-tuned during the learning of the new classifier.
Therefore, comparisons in Tables and fairly reflects the discrimination power of raw features of DNNs A and B and consistent features produced by g(æ
B BLIND SPOTS AND UNRELIABLE FEATURES
By our assumption, strong DNNs can encode true knowledge, while there are more representation flaws in weak DNNs. we try to use the intermediate features of a weak DNN to reconstruct intermediate features of a strong DNN, the strong DNN features cannot be perfectly reconstructed due to the inherent blind spots of the weak DNN. On the other hand, when we use features a strong DNN to reconstruct weak DNN's features, the unreconstructed feature components in these
14

---

Published as a conference paper at ICLR 2020
reconstructed features also exist. These unreconstructed parts not modeled in the strong DNN, and they are termed unreliable features by us.
15


Reviews:
[{"review_id": "r1xeVM5WiB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to disentangle intermediate features between two different deep neural networks. More specifically, given two networks, the proposed approach aims to find consistent and inconsistent feature components for a certain layer in each network. If one network is more powerful to the other (e.g., ResNet and AlexNet), the method can figure out which components are weak or strong (i.e., helpful to the performance) for the given task. The authors design a simple yet effective algorithm for extracting knowledge consistency. In addition, they provide a variety of practical experiments including network diagnosis, feature refinement, and network compression. Most of the experimental results support that the proposed method can practically extract consistent feature components with promising improvements.\n\nMain concerns:\n\n1. In section 4.1, the authors provide both the unreliable features and blind spots with their visualization (Figure 3). It is expected that blind spots are more informative than the unreliable features since they are part of a more powerful DNN. However, in Figure 3, the Blind spots do not seem to be important features. This makes the lack of the motivation of visualizing the effectiveness of features. Could you provide more promising visualizing results?\n\n2. In equation (1), it seems that the scaling factor p^(k+1) can be unnecessary because ReLU is scale-invariant and \\Sigma_(k+1) can scale the feature h^(k+1) with p_(k+1). Does p_(k+1) significantly improve the performance or negligible?\n\n3. In Table 3, using x^{(0)} to the refined feature shows the best result under VGG-16. Are the result in Table 4 improved when changing the refined features? E.g., only x^{(0)} or x^{(0)} + x^{(1)}?\n\nIn short, the proposed disentangle method is simple, novel and very useful in many practical applications as provided in the experimental section. But, some experiments and intuitions of network construction are not fully promising. Thus, I vote for weak acceptance.\n\nMinor concerns:\n\n1. Does the proposed method show meaningful results when the tasks are different but the datasets are the same? Or the same task but different datasets.\n\n2. The disentanglement network can be constructed with other operations, e.g., h^{(k)} = W^{(k)} concat(x , x^{higher}). Did the author try to design the network with other operations?\n\n3. In the fourth paragraph on the first page, the reference of “Wolchover (2017)” need to change “(Wolchover, 2017)”.\n\n4. The proposed method can be applied in other tasks, not with image datasets. Is the algorithm promising for other tasks?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BylN-zSe5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a method for extracting \"knowledge consistency\" in neural networks and using that toward analyzing different aspects of them, eg understanding the representations, explaining knowledge distillation, and analyzing network compression. What's defined as consistent knowledge is essentially the stable parts of representations of different networks trained for the same task (stable-->consistent). I found the submission insightful, well executed, and backed up by many experimental results. \n\nStrengths: \nA) the proposed concept, extracting the *consistency* among representation of different networks, is interested and using that towards understanding what's going on under-the-hood of neural networks makes sense. I'm not aware of a similar existing method and didn't see one among the citations, so my current assumption is that this proposal is pretty novel. \n\nB) the extracted representational consistency quantity appears to be meaningful and strong, as authors were able to use it toward explaining several existing phenomena (e.g. knowledge distillation, network compression, etc). \n\nC) Authors perform extensive experimental studies on various aspects of neural networks in relation to the proposed representational consistency quantity. I applaud the efforts made by authors.  \n\nImprovements/questions:\nD) The submission uses a few different phrases in close connection or interchangeably, eg \"fuzziness\", \"order\", \"linear/nonlinear transformation\", \"easy/hard to guess/estimation\" (see the last 2 paragraphs of page 2). While I understand what the authors are trying to convey, those concepts are not in principle necessarily the same, so some clarification/unification would make the presentation more solid. For instance: nonlinear<--> higher order<-->fuzzy<-->hard to guess. Also guess<-->estimate. \n\nE) Sec 3 is the most important part of the paper and the technical meat. However, I found it harder to follow compared to the rest of the paper. It probably deserves more than 1 page. Authors could consider smoothing the presentation and adding details even at the cost of slightly extending the length. \n\nF) I think section 4.2 could walk the reader through more details to make sure the basics are understood, as this is the first time results of the method are being presented. E.g. fig 3 could be analyzed further and the color maps should be defined. \n\nG) I found the post hoc explanation in the last paragraph of page 6 somewhat dubious. \n\nH) In the feature refinement experiment (paragraph 2 of sec 4.3), is the process done in a recursive manner (first layer 1, then layer 2, and so on)? Or as a one time process? The former seems stronger. \n\nI) The consistency quantity is based on comparing different networks trained for the *same task*. Have you considered doing the same among different networks trained for *different tasks*? Would that say something about similarities among tasks and multi-task learning in a fashion similar to the analysis of Taskonomy 2018? \n\nJ) I did not understand \"and Beyond\" in the title. I'd consider a more directly descriptive title. \n\nK) Seems like authors define \"knowledge\" and \"visual concept\" to be the invariant part of a feature (see paragraph 3 of intro). I don't see a particular problem with that, but a direct statement like 'invariant features' would have resonated with me just fine, while whether we can call that \"knowledge\" is a matter of (unnecessary) semantics in my opinion. \n\n\n=====\nPost rebuttal comments: \n\nThanks to authors for the detailed response and additions. I read through the comments and skimmed the revised PDF. Overall I preserve my positive rating toward this paper. The updates did improve the paper, but I would recommend the authors to use the camera ready opportunity to further address the original comments especially with respect to the delivery. \n\nI find the added experiments in multi task learning interesting especially given the time constraints of rebuttal. However the adopted setting is not the strongest case as what is defined to be \"two tasks\" is basically a superclass classification of a fine grained classifier. More concrete datasets and tasks specifically targeting challenging multi task learning are available now (see the full discussion in the original review). I would recommend taking those into consideration either for more experimentation or calibrating the conclusion made out of the current experiment. \n\nI recommend acceptance.  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJefPfcRYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of this paper is to analyze knowledge consistency between pretrained deep neural nets. In order to do so the paper trains neural networks to predict a hidden layer of one DNN using a hidden layer of another DNN. The model is interesting in that it is multi layer but it also allows decomposing its prediction as the sum of outputs of neural nets with different numbers of hidden layers. The prediction is dubbed \"consistent features\", which are further decomposed in consistent features of different complexity levels, while the error is dubbed \"inconsistent features\".\n\nThe paper then attempts to use this decomposition in \"consistent\" and \"inconsistent\" features in a number of ways. Assuming hidden layers of DNN A are being used to predict layer of DNN B, if A is stronger than B then the inconsistent features of B are claimed to be \"unreliable features\". If A is weaker, then the inconsistent features of B are claimed to be \"blind spots\". A figure is given to support this intuition, but I think some real evidence would have to be collected to support a claim like this.\n\nSimilarly the authors analyze model compression and distillation with their technique. They are able to show a decrease in the variance of inconsistent features as a models get progressively compressed or as a models get better during generations of distillation. Again the evidence that of the usefulness of this analysis seems very limited. The visualizations of consistent and inconsistent components don't seem to give any clear evidence.\n\nThe most interesting part of the paper shows experiments from boosting performance of a model by finetuning on features consistent with another model. None of the improvements seem close to state of the art though, and might just be some byproduct of model ensembling.\n\nOverall the paper is very interesting, but more convincing experimental results would have to be collected to prove that the method is actually useful.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for extracting "knowledge consistency" between neural networks and  understanding their representations. 

Reviewers and AC are positive on the paper, in terms of insightful findings and practical usages, and also gave constructive suggestions to improve the paper. In particular, I think the paper can gain much attention for ICLR audience.  

Hence, I recommend acceptance.
Decision: Accept (Poster)