Title: KNOWLEDGE CONSISTENCY BETWEEN NEURAL NET-WORKS AND BEYOND

Abstract: This paper aims to analyze knowledge consistency between pre-trained deep neural networks. We propose a generic definition for knowledge consistency between neural networks at different fuzziness levels. A task-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As a generic tool, our method can be broadly used for different applications. In preliminary experiments, we have used knowledge consistency as a tool to diagnose representations of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance.

1. INTRODUCTION:

Deep neural networks (DNNs) have shown promise in many tasks of artificial intelligence. However, there is still lack of mathematical tools to diagnose representations in intermediate layers of a DNN, e.g. discovering flaws in representations or identifying reliable and unreliable features. Traditional evaluation of DNNs based on the testing accuracy cannot insightfully examine the correctness of representations of a DNN due to leaked data or shifted datasets (Ribeiro et al., 2016) . Thus, in this paper, we propose a method to diagnose representations of intermediate layers of a DNN from the perspective of knowledge consistency. I.e. given two DNNs pre-trained for the same task, no matter whether the DNNs have the same or different architectures, we aim to examine whether intermediate layers of the two DNNs encode similar visual concepts.

Here, we define the knowledge of an intermediate layer of the DNN as the set of visual concepts that are encoded by features of an intermediate layer. This research focuses on the consistency of "knowledge" between two DNNs, instead of comparing the similarity of "features." In comparison, the feature is referred to as the explicit output of a layer. For example, two DNNs extract totally different features, but these features may be computed using similar sets of visual concepts, i.e. encoding consistent knowledge (a toy example of knowledge consistency is shown in the footnote 1 ).

Given the same training data, DNNs with different starting conditions usually converge to different knowledge representations, which sometimes leads to the over-fitting problem (Bengio et al., 2014) . However, ideally, well learned DNNs with high robustness usually comprehensively encode various types of discriminative features and keep a good balance between the completeness and the discrimination power of features (Wolchover, 2017) . Thus, the well learned DNNs are supposed to converge to similar knowledge representations.

In general, we can understand knowledge consistency as follows. Let A and B denote two DNNs learned for the same task. In terms of applications, knowledge consistency between DNNs can be used to diagnose feature reliability of DNNs. Usually, consistent components (xA, xB) represent common and reliable knowledge. Whereas, inconsistent components ( A, B ) mainly represent unreliable knowledge or noises. Therefore, in this paper, we propose a generic definition for knowledge consistency between two pre-trained DNNs, and we develop a method to disentangle consistent feature components from features of intermediate layers in the DNNs. Our method is task-agnostic, i.e. (1) our method does not require any annotations w.r.t. the task for evaluation; (2) our method can be broadly applied to different DNNs as a supplement evaluation of DNNs besides the testing accuracy. Experimental results supported our assumption, i.e. the disentangled consistent feature components are usually more reliable for the task. Thus, our method of disentangling consistent features can be used to boost performance. Furthermore, to enable a solid research on knowledge consistency, we consider the following issues.

• Fuzzy consistency at different levels (orders): As shown in Fig. 1 , the knowledge consistency between DNNs needs to be defined at different fuzziness levels, because there is no strict knowledge consistency between two DNNs. The level of fuzziness in knowledge consistency measures the difficulty of transforming features of a DNN to features of another DNN. A low-level fuzziness indicates that a DNN's feature can directly reconstruct another DNN's feature without complex transformations.

• Disentanglement & quantification: We need to disentangle and quantify feature components, which correspond to the consistent knowledge at different fuzziness levels, away from the chaotic feature. Similarly, we also disentangle and quantify feature components that are inconsistent.

There does not exist a standard method to quantify the fuzziness of knowledge consistency (i.e. the difficulty of feature transformation). For simplification, we use non-linear transformations during feature reconstruction to approximate the fuzziness. To this end, we propose a model g k for feature reconstruction. The subscript k indicates that g k contains a total of k cascaded non-linear activation layers. In this way, the most strict consistency is the 0-order consistency, i.e. xA = g0(xB) can be reconstructed from xB via a linear transformation. In comparison, some neural activations in the 1-order consistent feature xA = g1(xB) are not directly represented by xB and need to be predicted via a non-linear transformation. The smaller k indicates the less prediction involved in the reconstruction and the stricter consistency. Note that the number of non-linear operations k is just a rough approximation of the difficulty of prediction, since there are no standard methods to quantify prediction difficulties. More crucially, we implement the model g as a neural network, where k is set as the number of nonlinear layers in g. As shown in Fig. 2 , g is designed to disentangle and quantify consistent feature components of different orders between DNNs. Our method can be applied to different types of DNNs and explain the essence of various deep-learning techniques.

1. Our method provides a new perspective for explaining the effectiveness of knowledge distillation.

I.e. we explore the essential reason why the born-again network (Furlanello et al., 2018 ) exhibits superior performance.

2. Our method gives insightful analysis of network compression.

3.:

Our method can be used to diagnose and refine knowledge representations of pre-trained DNNs and boost the performance without any additional annotations for supervision.

Contributions of this study can be summarized as follows.

(1) In this study, we focus on a new problem, i.e. the knowledge consistency between DNNs.

(2) We define the knowledge consistency and propose a task-agnostic method to disentangle and quantify consistent features of different orders.

(3) Our method can be considered as a mathematical tool to analyze feature reliability of different DNNs. (4) Our method provides a new perspective on explaining existing deep-learning techniques, such as knowledge distillation and network compression.

2. RELATED WORK:

In spite of the significant discrimination power of DNNs, black-box representations of DNNs have been considered an Achilles' heel for decades. In this section, we will limit our discussion to the literature of explaining or analyzing knowledge representations of DNNs. In general, previous studies can be roughly classified into the following three types.

Explaining DNNs visually or semantically: Visualization of DNNs is the most direct way of explaining knowledge hidden inside a DNN, which include gradient-based visualization (Zeiler & Fergus, 2014; Mahendran & Vedaldi, 2015) and inversion-based visualization (Dosovitskiy & Brox, 2016) . (Zhou et al., 2015) developed a method to compute the actual image-resolution receptive field of neural activations in a feature map of a convolutional neural network (CNN), which is smaller than the theoretical receptive field based on the filter size. Based on the receptive field, six types of semantics were defined to explain intermediate-layer features of CNNs, including objects, parts, scenes, textures, materials, and colors (Bau et al., 2017; Zhou et al., 2018) .

Beyond visualization, some methods diagnose a pre-trained CNN to obtain insight understanding of CNN representations. Fong and Vedaldi (Fong & Vedaldi, 2018) analyzed how multiple filters jointly represented a specific semantic concept. (Selvaraju et al., 2017) , (Fong & Vedaldi, 2017) , and (Kindermans et al., 2018) estimated image regions that directly contributed the network output. The LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017) assumed a linear relationship between the input and output of a DNN to extract important input units. (Zhang et al., 2019) proposed a number of metrics to evaluate the objectiveness of explanation results of different explanation methods, even when people could not get ground-truth explanations for the DNN.

Unlike previous studies visualizing visual appearance encoded in a DNN or extracting important pixels, our method disentangles and quantifies the consistent components of features between two DNNs. Consistent feature components of different orders can be explicitly visualized.

Learning explainable deep models: Compared to the post-hoc explanations of DNNs, some studies directly learn more meaningful CNN representations. Previous studies extracted scene semantics (Zhou et al., 2015) and mined objects (Simon & Rodner, 2015) from intermediate layers. In the capsule net (Sabour et al., 2017) , each output dimension of a capsule usually encoded a specific meaning. (Zhang et al., 2018b) proposed to learn CNNs with disentangled intermediate-layer representations. The infoGAN (Chen et al., 2016) and β-VAE (Higgins et al., 2017) learned interpretable input codes for generative models.

Mathematical evaluation of the representation capacity: Formulating and evaluating the representation capacity of DNNs is another emerging direction. (Novak et al., 2018) proposed generic metrics for the sensitivity of network outputs with respect to parameters of neural networks. (Zhang et al., 2017) discussed the relationship between the parameter number and the generalization capacity of deep neural networks. (Arpit et al., 2017) discussed the representation capacity of neural networks, considering real training data and noises. (Yosinski et al., 2014) evaluated the transferability of filters in intermediate layers. Network-attack methods (Koh & Liang, 2017; Szegedy et al., 2014; Koh & Liang, 2017) can also be used to evaluate representation robustness by computing adversarial samples. (Lakkaraju et al., 2017) discovered knowledge blind spots of the knowledge encoded by a DNN via human-computer interaction. (Zhang et al., 2018a) discovered potential, biased representations of a CNN due to the dataset bias. (Deutsch, 2018) learned the manifold of network parameters to diagnose DNNs. (Huang et al., 2019) quantified the representation utility of different layerwise network architectures in the scenario of 3D point cloud processing. Recently, the stiffness (Fort et al., 2019) was proposed to evaluate the generalization of DNNs.

The information-bottleneck theory (Wolchover, 2017; Schwartz-Ziv & Tishby, 2017) provides a generic metric to quantify the information contained in DNNs. The information-bottleneck theory can be extended to evaluate the representation capacity of DNNs (Xu & Raginsky, 2017; Cheng et al., 2018) . (Achille & Soatto, 2018 ) further used the information-bottleneck theory to improve feature representations of a DNN. (Guan et al., 2019; Ma et al., 2019) quantified the word-wise/pixelwise information discarding during the layerwise feature propagation of a DNN, and used the information discarding as a tool to diagnose feature representations.

The analysis of representation similarity between DNNs is most highly related to our research, which has been investigated in recent years (Montavon et al., 2011) . Most previous studies (Wang et al., 2018; Kornblith et al., 2019; Raghu et al., 2017; Morcos et al., 2018) compared representations via linear transformations, which can be considered as 0-order knowledge consistency. In comparison, we focus on high-order knowledge consistency. More crucially, our method can be used to refine network features and explain the success of existing deep-learning techniques.

3. ALGORITHM:

In this section, we will introduce the network architecture to disentangle feature components of consistent knowledge at a certain fuzziness level, when we use the intermediate-layer feature x of a DNN to reconstruct intermediate-layer feature x * of another DNN2 . As shown in Fig. 2 , the network g with parameters θ has a recursive architecture with K + 1 blocks. The function of the k-th block is given as follows.

h (k) = W (k) x + x higher , x higher = p (k+1) ReLU Σ -1 2 (k+1) h (k+1) , k = 0, 1, . . . , K -1 (1)

The output feature is computed using both the raw input and the feature of the higher order h (k+1) . W (k) denotes a linear operation without a bias term. The last block is given as K) x. This linear operation can be implemented as either a layer in an MLP or a convolutional layer in a CNN.

h (K) = W (

EQUATION

n ) is referred to a diagonal matrix for element-wise variance of h (k+1) , where σ 2 m the variance of the m-th element of h (k+1) through various input images. Σ (k+1) is used to normalize the magnitude of neural activations. Because of the normalization, the scalar value p (k+1) roughly controls the activation magnitude of h (k+1) w.r.t. h (k) . h (0) = g θ (x) corresponds to final output of the network.

In this way, the entire network can be separated into (K + 1) branches (see Fig. 2 ), where the k-th branch (k ≤ K) contains a total of k non-linear layers. Note that the k1-order consistent knowledge can also be represented by the network with k2-th branch, if k1 < k2.

In order to disentangle consistent features of different orders, the k-th branch is supposed to exclusively encode the k-order consistent features without representing lower-order consistent features. Thus, we propose the following loss to guide the learning process.

Loss(θ) = g θ (x) -x * 2 + λ K k=1 p (k) 2

(2)

where x and x * denote intermediate-layer features of two pre-trained DNNs. The second term in this loss penalizes neural activations from high-order branches, thereby forcing as much low-order consistent knowledge as possible to be represented by low-order branches.

Furthermore, based on (K + 1) branches of the network, we can disentangle features of x * into (K + 2) additive components.

EQUATION

where

x ∆ = x * -g θ (x)

indicates feature components that cannot be represented by x. x (k) denotes feature components that are exclusively represented by the k-order branch.

Based on Equation ( 1), the signal-processing procedure for the k-th feature component can be represented as 2 ). Therefore, we can disentangle from Equation (1) all linear and non-linear operations along the entire k-th branch as follows, and in this way, x (k) can be considered as the k-order consistent feature component.

Conv k → N orm k → ReLU k → p (k) → Conv k-1 → • • • → ReLU1 → p (1) → Conv0 (see Fig.

EQUATION

where A (k ) = diag(a1, a2, . . . , aM ) is a diagonal matrix. Each diagonal element a m ∈ {0, 1} represents the binary information-passing state of x m through an ReLU layer (1 ≤ m ≤ M ). Each element is given as

am = 1(vm > 0), where v = Σ -1 2 (k ) h (k ) . Note that ReLU (W k (x + x higher )) = ReLU (W k (x)) + ReLU (W k (x higher )), but A (k ) (W k (x + x higher )) = A (k ) W k x + A (k ) W k x higher .

Thus, we record such information-passing states to decompose feature components.

4. COMPARATIVE STUDIES:

As a generic tool, knowledge consistency based on g can be used for different applications. In order to demonstrate utilities of knowledge consistency, we designed various comparative studies, including (1) diagnosing and debugging pre-trained DNNs, (2) evaluating the instability of learning DNNs, (3) feature refinement of DNNs, (4) analyzing information discarding during the compression of DNNs, and (5) explaining effects of knowledge distillation in knowledge representations.

A total of five typical DNNs for image classification were used, i.e. the AlexNet (Krizhevsky et al., 2012) , the VGG-16 (Simonyan & Zisserman, 2015) , and the ResNet-18, ResNet-34, ResNet-50 (He et al., 2016) . These DNNs were learned using three benchmark datasets, which included the CUB200-2011 dataset (Wah et al., 2011) , the Stanford Dogs dataset (Khosla et al., 2011) , and the Pascal VOC 2012 dataset (Everingham et al., 2015) . Note that both training images and testing images were cropped using bounding boxes of objects. We set λ = 0.1 for all experiments, except for feature reconstruction of AlexNet (we set λ = 8.0 for AlexNet features). It was because the shallow model of the AlexNet usually had significant noisy features, which caused considerable inconsistent components.

4.1. NETWORK DIAGNOSIS BASED ON KNOWLEDGE CONSISTENCY:

The most direct application of knowledge consistency is to use a strong (well learned) DNN to diagnose representation flaws hidden in a weak DNN. This is of special values in real applications, e.g. shallow (usually weak) DNNs are more suitable to be adopted to mobile devices than deep DNNs. For implementation, we trained DNNs for fine-grained classification using the CUB200-2011 dataset (Wah et al., 2011) (without data augmentation). We considered the AlexNet (Krizhevsky et al., 2012) as the weak DNN (56.97% top-1 accuracy), and took the ResNet-34 (He et al., 2016) as the strong DNN (73.09% top-1 accuracy).

Please see Fig. 3 . We diagnosed the output feature of the last convolutional layer in the AlexNet, which is termed xA. Accordingly, we selected the last 14 × 14 × 256 feature map of the ResNet-34 (denoted by xB) for the computation of knowledge consistency, because xA and xB had similar map sizes. We disentangled and visualized unreliable components from xA (i.e. inconsistent components in xA). We also visualized components disentangled from xB (i.e. inconsistent components in xB), which corresponded to blind spots of the weak DNN's knowledge. Furthermore, we conducted two experiments to further verify the claim of blind spots and unreliable features in a DNN. The first experiment aimed to verify blind spots. The basic idea of this experiment was to examine the increase of the classification accuracy, when we added information of blind spots to the raw feature. We followed above experimental settings, which used the intermediatelayer feature x A of the AlexNet (with 56.97% top-1 accuracy) to reconstruct the intermediate-layer feature x B of Resnet-34 (with 73.09% top-1 accuracy). Then, inconsistent feature components were termed blind spots. In this way, we added feature components of blind spots back to the AlexNet's feature (adding these features back to g θ (x A )), and then learned a classifier upon the new feature. To enable a fair comparison, the classifier had the same architecture as the AlexNet's modules above x A , and during the learning process, we fixed parameters in DNN A and θ to avoid the revision of such parameters affecting the performance. We found that compared to the raw feature x A of the AlexNet, the new feature boosted the classification accuracy by 16.1%.

The second experiment was conducted to verify unreliable features. The basic idea of this experiment was to examine the increase of the classification accuracy, when we removed information of unreliable features from the raw feature. We designed two classifiers, which had the same architecture as the AlexNet's modules above x A . We fed the raw feature x A to the first classifier. Then, we removed unreliable feature components from x A (i.e. obtaining g θ (x B )), and fed the revised feature to the second classifier. We learned these two classifiers, and classifiers with and without unreliable features exhibited classification accuracy of 60.3% and 65.6%, respectively. Therefore, above two experiments successfully demonstrated that both the insertion of blind spots and the removal of unreliable features boosted the classification accuracy.

Learning DNNs from different initializations conv4 @ AlexNet conv5 @ AlexNet conv4-3 @ VGG-16 conv5-3 @ VGG-16 last conv @ ResNet-34 0.086 0.116 0.124 0.196 0.776 Learning DNNs using different training data conv4 @ AlexNet conv5 @ AlexNet conv4-3 @ VGG-16 conv5-3 @ VGG-16 last conv @ ResNet-34 0. conv4 @ AlexNet conv5 @ AlexNet conv4-3 @ VGG-16 conv5-3 @ VGG-16 last conv @ ResNet-34 V ar(x More specifically, we trained two DNNs (A, B) with the same architecture for the same task. Then, we disentangled inconsistent feature components

x ∆ A = xA -g θ (xB) and x ∆ B = xB -g θ (xA)

from their features x A and x B of a specific layer, respectively. Accordingly, g θ (xB) and g θ (xA) corresponded to consistent feature components in x A and x B , respectively, whose knowledge was shared by two networks. The inconsistent feature x ∆ A was quantified by the variance of feature element through different units of x ∆ A and through different input images

V ar(x ∆ A ) def = EI,i[(x ∆ A,I,i -E I ,i [x ∆ A,I ,i ]) 2 ],

where x ∆ A,I,i denotes the i-th element of x ∆ A given the image I. We can use V ar(x ∆ A )/V ar(xA) to measure the instability of learning DNNs A and B.

In experiments, we evaluated the instability of learning the AlexNet (Krizhevsky et al., 2012) , the VGG-16 (Simonyan & Zisserman, 2015) , and the ResNet-34 (He et al., 2016) . We considered the following cases.

Case 1, learning DNNs from different initializations using the same training data: For each network architecture, we trained multiple networks using the CUB200-2011 dataset (Wah et al., 2011) . The instability of learning DNNs was reported as the average of instability over all pairs of neural networks.

Case 2, learning DNNs using different sets of training data: We randomly divided all training samples in the CUB200-2011 dataset (Wah et al., 2011) into two subsets, each containing 50% samples. For each network architecture, we trained two DNNs (A, B) for fine-grained classification (without pre-training). The instability of learning DNNs was reported as Table 3 : Classification accuracy by using the original and the refined features. Features of DNN A were used to reconstruct features of DNN B. For residual networks, we selected the last feature map with a size of 14×14 as the target for refinement. All DNNs were learned without data augmentation or pre-training. The removal of effects of additional parameters in g θ is discussed in Section 4.3fairness of comparisons and Appendix A. Consistent components slightly boosted the performance.

[V ar(x ∆ A )/V ar(xA) + V ar(x ∆ B )/V ar(xB)]/2.

intermediate-layer features can be refined by removing inconsistent components and exclusively using consistent components to accomplish the task.

More specifically, given two pre-trained DNNs, we use the feature of a certain layer in the first DNN to reconstruct the corresponding feature of the second DNN. The reconstructed feature is given as x.

In this way, we can replace the feature of the second DNN with the reconstructed feature x, and then use x to simultaneously learn all subsequent layers in the second DNN to boost performance. Note that for clear and rigorous comparisons, we only disentangled consistent feature components from the feature of a single layer and refined the feature. It was because the simultaneous refinement of features of multiple layers would increase it difficult to clarify the refinement of which feature made the major contribution.

In experiments, we trained DNNs with various architectures for image classification, including the VGG-16 (Simonyan & Zisserman, 2015) , the ResNet-18, the ResNet-34, and the ResNet-50 (He et al., 2016) . We conducted the following two experiments, in which we used knowledge consistency to refine DNN features.

Exp. 1, removing unreliable features (noises): For each specific network architecture, we trained two DNNs using the CUB200-2011 dataset (Wah et al., 2011) with different parameter initializations. Consistent components were disentangled from the original feature of a DNN and then used for image classification. As discussed in Section 4.1, consistent components can be considered as refined features without noises. We used the refined features as input to finetune the pre-trained upper layers in the DNN B for classification. Table 3 reports the increase of the classification accuracy by using the refined feature. The refined features slightly boosted the performance.

Fairness of comparisons::

(1) To enable fair comparisons, we first trained g θ and then kept g θ unchanged during the finetuning of classifiers, thereby fixing the refined features. Otherwise, allowing g θ to change would be equivalent to adding more layers/parameters to DNN B for classification. We needed to eliminate such effects for fair comparisons, which avoided the network from benefitting from additional layers/parameters. (2) As baselines, we also further finetuned the corresponding upper layers of DNNs A and B to evaluate their performance. Please see Appendix A for more discussions about how we ensured the fairness.

Exp. 2, removing redundant features from pre-trained DNNs: A typical deep-learning methodology is to finetune a DNN for a specific task, where the DNN is pre-trained for multiple tasks (including both the target and other tasks). This is quite common in deep learning, e.g. DNNs pre-trained using the ImageNet dataset (Deng et al., 2009) are usually finetuned for various tasks. However, in this case, feature components pre-trained for other tasks are redundant for the target task and will affect the further finetuning process.

Therefore, we conducted three new experiments, in which our method removed redundant features w.r.t. the target task from the pre-trained DNN. In Experiment 2.1 (namely VOC-animal), let two DNNs A and B be learned to classify 20 object classes in the Pascal VOC 2012 dataset (Everingham et al., 2015) . We were also given object images of six animal categories (bird, cat, cow, dog, horse, sheep) . We fixed parameters in DNNs A and Table 4 : Top-1 classification accuracy before and after removing redundant features from DNNs. Original DNNs were learned from scratch without data augmentation. For residual networks, we selected the last feature map with a size of 14 × 14 as the target for feature refinement. The removal of effects of additional parameters in g θ is discussed in Section 4.3-fairness of comparisons and Appendix A. Consistent features significantly alleviated the over-fitting problem, thereby exhibiting superior performance. In Experiment 2.2 (termed Mix-CUB), two original DNNs were learned using both the CUB200-2011 dataset (Wah et al., 2011) and the Stanford Dogs dataset (Khosla et al., 2011) to classify both 200 bird species and 120 dog species. Then, our method used bird images to disentangle feature components for birds and then learned a new fine-grained classifier for birds. The baseline method was implemented following the same setting as in VOC-animal. Experiment 2.3 (namely Mix-Dogs) was similar to Mix-CUB. Our method disentangled dog features away from bird features to learn a new dog classifier. In all above experiments, original DNNs were learned from scratch without data augmentation. Table 4 compares the classification accuracy of different methods. It shows that our method significantly alleviated the over-fitting problem and outperformed the baseline.

4.4. ANALYZING INFORMATION DISCARDING OF NETWORK COMPRESSION:

Network compression is an emerging research direction in recent years. Knowledge consistency between the compressed network and the original network can evaluate the discarding of knowledge during the compression process. I.e. people may visualize or analyze feature components in the original network, which are not consistent with features in the compressed network, to represent the discarded knowledge in the compressed network.

In experiments, we trained the VGG-16 using the CUB200-2011 dataset (Wah et al., 2011) for finegrained classification. Then, we compressed the VGG-16 using the method of (Han et al., 2016) with different pruning thresholds. We used features of the compressed DNN to reconstruct features of the original DNN. Then, inconsistent components disentangled from the original DNN usually corresponded to the knowledge discarding during the compression process. Fig. 4 (left) visualizes the discarded feature components. We used V ar(x ∆ ) (defined in Section 4.2) to quantify the information discarding. Fig. 4 compares the decrease of accuracy with the discarding of feature information.

4.5. EXPLAINING KNOWLEDGE DISTILLATION VIA KNOWLEDGE CONSISTENCY:

As a generic tool, our method can also explain the success of knowledge distillation. In particular, (Furlanello et al., 2018) proposed a method to gradually refine a neural network via recursive knowledge distillation. I.e. this method recursively distills the knowledge of the current net to a new net with the same architecture and distilling the new net to an even newer net. The new(er) net is termed a born-again neural network and learned using both the task loss and the distillation loss. Surprisingly, such a recursive distillation process can substantially boost the performance of the neural network in various experiments.

In general, the net in a new generation both inherits knowledge from the old net and learns new knowledge from the data. The success of the born-again neural network can be explained as that knowledge representations of networks are gradually enriched during the recursive distillation process. To verify this assertion, in experiments, we trained the VGG-16 using the CUB200-2011 dataset (Wah et al., 2011) for fine-grained classification. We trained born-again neural networks of another four generations3 . We disentangled feature components in the newest DNN, which were not consistent with an intermediate DNN. Inconsistent components were considered as blind spots of knowledge representations of the intermediate DNN and were quantified by V ar(x ∆ ). Fig. 4 (right) shows V ar(x ∆ ) of DNNs in the 1st, 2nd, 3rd, and 4th generations. Inconsistent components were gradually reduced after several generations.

4.6. CONSISTENT AND INCONSISTENT FEATURES BETWEEN DIFFERENT TASKS:

In order to visualize consistent and inconsistent features between different tasks (i.e. the fine-grained classification and the binary classification), we trained DNN A (a VGG-16 network) to simultaneously classify 320 species, including 200 bird species in the CUB200-2011 dataset (Wah et al., 2011) and 120 dog species in the Stanford Dogs dataset (Khosla et al., 2011) , in a fine-grained manner.

On the other hand, we learned DNN B (another VGG-16 network) for the binary classification of the bird and the dog based on these two datasets. We visualized the feature maps of consistent and inconsistent feature components between the two DNNs in Fig. 5 . Obviously, DNN A encoded more knowledge than DNN B, thereby x A reconstructing more features than x B .

5. CONCLUSION:

In this paper, we have proposed a generic definition of knowledge consistency between intermediatelayers of two DNNs. A task-agnostic method is developed to disentangle and quantify consistent features of different orders from intermediate-layer features. Consistent feature components are usually more reliable than inconsistent components for the task, so our method can be used to further refine the pre-trained DNN without a need for additional supervision. As a mathematical tool, knowledge consistency can also help explain existing deep-learning techniques, and experiments have demonstrated the effectiveness of our method.

:

Re-scaling the magnitude of all neural activations does not affect the knowledge encoded in the DNN. Therefore, we normalize x and x * to zero mean and unit variance to remove the magnitude effect.

Because(Furlanello et al., 2018) did not clarify the distillation loss, we applied the distillation loss in(Hinton et al., 2014) following parameter settings τ = 1.0 in(Mishra & Marr, 2018), i.e. Loss = Loss classify + 0.5Loss distill .

Figures and Tables:

Figure 1: Knowledge consistency. We define, disentangle, and quantify consistent features between two DNNs. Consistent features disentangled from a filter are visualized at different orders (levels) of fuzziness. tively. Features xA and xB can be decomposed as xA = xA + A and xB = xB + B , where neural activations in feature components xA and xB are triggered by same image regions, thereby representing consistent knowledge. Accordingly, feature components xA and xB are termed consistent features between A and B. Then, feature components A and B are independent with each other, and they are termed inconsistent features. We assume that consistent components xA, xB can reconstruct each other, i.e. xA can be reconstructed from xB; vice versa.

xA = g k (xB) represents components, which are disentangled from the feature xA of the DNN A and can be reconstructed by the DNN B's feature xB. Then, we consider xA = g k (xB) to represent consistent knowledge w.r.t. the DNN B at the k-th fuzziness level (or the k-th order). xA = g k (xB) is also termed the k-order consistent feature of xA w.r.t. xB.

Figure 2: Neural network for disentanglement of consistent features (K = 3).

Figure 3: Unreliable components and blind spots of a weak DNN (AlexNet) w.r.t. a strong DNN (ResNet-34). Please see Section 4.1 for definitions of "unreliable components" and "blind spots." (left) When we used features of the weak DNN to reconstruct features of the strong DNN, we visualize raw features of the strong DNN, feature components that can be reconstructed, and blind spots (x ∆ ) disentangled from features of the strong DNN. The weak DNN mainly encoded the head appearance and ignored others. We can find that some patterns in the torso are blind spots of the weak DNN. (right) When we used features of the strong DNN to reconstruct features of the weak DNN, we visualize raw features of the weak DNN, feature components that can be reconstructed, and unreliable features (x ∆ ) disentangled from features of the weak DNN. Based on visualization results, unreliable features of the weak DNN usually repesent noisy activations. Let two DNNs be trained for the same task, and one DNN significantly outperforms the other DNN. We assume that the strong DNN has encoded ideal knowledge representations of the target task. The weak DNN may have the following two types of representation flaws. • Unreliable features in the weak DNN are defined as feature components, which cannot be reconstructed by features of the strong DNN. (see Appendix B for detailed discussions). • Blind spots of the weak DNN are defined as feature components in the strong DNN, which are inconsistent with features of the weak DNN. These feature components usually reflect blind spots of the knowledge of the weak DNN (see Appendix B for detailed discussions).

Figure 4: Effects of network compression and knowledge distillation. (left) We visualized the discarded feature components when 93.3% parameters of the DNN were eliminated. We found that the discarded components looked like trivial features and noises. In the curve figure, a low value of V ar(x ∆ ) indicates a lower information discarding during the compression, thereby exhibiting a higher accuracy. (right) We visualized and quantified knowledge blind spots of born-again networks in different generations. Nets in new generations usually had fewer blind spots than old nets. enable a fair comparison. In comparison, the baseline method directly used either xA or xB to finetune the pre-trained DNN to classify the six animal categories.

Figure 5: Consistent and inconsistent feature components disentangled between DNN B learned for binary classification and DNN A learned for fine-grained classification.

Instability of learning DNNs from different initializations and instability of learning DNNs using different training data. Without a huge training set, networks with more layers (e.g. ResNet-34) usually suffered more from the over-fitting problem.

Table content: <table><tr><td>089</td><td>0.155</td><td>0.121</td><td>0.198</td><td>0.275</td></tr></table>

Magnitudes of consistent features of different orders, when we train DNNs from different initializations. Features in different layers have significantly different variance magnitudes. Most neural activations of the feature belong to low-order consistent components.4.2 STABILITY OF LEARNINGThe stability of learning DNNs is of considerable values in deep learning, i.e. examining whether or not all DNNs represent the same knowledge, when people repeatedly learn multiple DNNs for the same task. High knowledge consistency between DNNs usually indicates high learning stability.

Table content: <table><tr><td>(0) )</td><td>105.80</td><td>424.67</td><td>1.06</td><td>0.88</td><td>0.66</td></tr><tr><td>V ar(x (1) )</td><td>10.51</td><td>73.73</td><td>0.07</td><td>0.03</td><td>0.10</td></tr><tr><td>V ar(x (2) )</td><td>1.92</td><td>43.69</td><td>0.02</td><td>0.004</td><td>0.03</td></tr><tr><td>V ar(x ∆ )</td><td>11.14</td><td>71.37</td><td>0.16</td><td>0.22</td><td>2.75</td></tr></table>

Table1compares the instability of learning different DNNs. Table2reports the variance of consistent components of different orders. We found that the learning of shallow layers in DNNs was usually more stable than the learning of deep layers. The reason may be as follows. A DNN with more layers usually can represent more complex visual patterns, thereby needing more training samples. Without a huge training set (e.g. the ImageNet dataset(Krizhevsky et al., 2012)), a deep network may be more likely to suffer from the over-fitting problem, which causes high variances in Table1, i.e. DNNs with different initial parameters may learn different knowledge representations.

Table content: <table><tr><td/><td colspan="5">VGG-16 conv4-3 VGG-16 conv5-2 ResNet-18 ResNet-34 ResNet-50</td></tr><tr><td>Network A</td><td>43.15</td><td/><td>34.74</td><td>31.05</td><td>29.98</td></tr><tr><td>Network B</td><td>42.89</td><td/><td>35.00</td><td>30.46</td><td>31.15</td></tr><tr><td>x (0)</td><td>45.15</td><td>44.48</td><td>38.16</td><td>31.49</td><td>30.40</td></tr><tr><td>x (0) + x (1)</td><td>44.98</td><td>44.22</td><td>38.45</td><td>31.76</td><td>31.77</td></tr><tr><td>x (0) + x (1) + x (2)</td><td>45.06</td><td>44.32</td><td>38.23</td><td>31.96</td><td>31.84</td></tr></table>

B, and our goal was to use fixed DNNs A and B to generate clean features for animal categories without redundant features.Let xA and xB denote intermediate-layer features of DNNs A and B. Our method used xA to reconstruct xB. Then, the reconstructed result g θ (xA) corresponded to reliable features for animals, while inconsistent components x ∆ indicated features of other categories. We used clean features g θ(xA) to learn the animal classifier. g θ were fixed during the learning of the animal classifier to VGG-16 conv4-3 VGG-16 conv5-2 VOC-animal Mix-CUB Mix-Dogs VOC-animal Mix-CUB Mix-Dogs Features from the network A

Table content: <table><tr><td/><td>51.55</td><td>44.44</td><td>15.15</td><td>51.55</td><td>44.44</td><td>15.15</td></tr><tr><td>Features from the network B</td><td>50.80</td><td>45.93</td><td>15.19</td><td>50.80</td><td>45.93</td><td>15.19</td></tr><tr><td>x (0) + x (1) + x (2)</td><td>59.38</td><td>47.50</td><td>16.53</td><td>60.18</td><td>46.65</td><td>16.70</td></tr><tr><td/><td/><td>ResNet-18</td><td/><td/><td>ResNet-34</td><td/></tr><tr><td/><td colspan="6">VOC-animal Mix-CUB Mix-Dogs VOC-animal Mix-CUB Mix-Dogs</td></tr><tr><td>Features from the network A</td><td>37.65</td><td>31.93</td><td>14.20</td><td>39.42</td><td>30.91</td><td>12.96</td></tr><tr><td>Features from the network B</td><td>37.22</td><td>32.02</td><td>14.28</td><td>35.95</td><td>27.74</td><td>12.46</td></tr><tr><td>x (0) + x (1) + x (2)</td><td>53.52</td><td>38.02</td><td>16.17</td><td>49.98</td><td>33.98</td><td>14.21</td></tr></table>

Peer review (iclr2020):
The submission proposes a method for extracting *knowledge consistency* in neural networks and using that toward analyzing different aspects of them, eg understanding the representations, explaining knowledge distillation, and analyzing network compression. What*s defined as consistent knowledge is essentially the stable parts of representations of different networks trained for the same task (stable-->consistent). I found the submission insightful, well executed, and backed up by many experimental results. Strengths: A) the proposed concept, extracting the *consistency* among representation of different networks, is interested and using that towards understanding what*s going on under-the-hood of neural networks makes sense. I*m not aware of a similar existing method and didn*t see one among the citations, so my current assumption is that this proposal is pretty novel. B) the extracted representational consistency quantity appears to be meaningful and strong, as authors were able to use it toward explaining several existing phenomena (e.g. knowledge distillation, network compression, etc). C) Authors perform extensive experimental studies on various aspects of neural networks in relation to the proposed representational consistency quantity. I applaud the efforts made by authors. Improvements/questions: D) The submission uses a few different phrases in close connection or interchangeably, eg *fuzziness*, *order*, *linear/nonlinear transformation*, *easy/hard to guess/estimation* (see the last 2 paragraphs of page 2). While I understand what the authors are trying to convey, those concepts are not in principle necessarily the same, so some clarification/unification would make the presentation more solid. For instance: nonlinear<--> higher order<-->fuzzy<-->hard to guess. Also guess<-->estimate. E) Sec 3 is the most important part of the paper and the technical meat. However, I found it harder to follow compared to the rest of the paper. It probably deserves more than 1 page. Authors could consider smoothing the presentation and adding details even at the cost of slightly extending the length. F) I think section 4.2 could walk the reader through more details to make sure the basics are understood, as this is the first time results of the method are being presented. E.g. fig 3 could be analyzed further and the color maps should be defined. G) I found the post hoc explanation in the last paragraph of page 6 somewhat dubious. H) In the feature refinement experiment (paragraph 2 of sec 4.3), is the process done in a recursive manner (first layer 1, then layer 2, and so on)? Or as a one time process? The former seems stronger. I) The consistency quantity is based on comparing different networks trained for the *same task*. Have you considered doing the same among different networks trained for *different tasks*? Would that say something about similarities among tasks and multi-task learning in a fashion similar to the analysis of Taskonomy 2018? J) I did not understand *and Beyond* in the title. I*d consider a more directly descriptive title. K) Seems like authors define *knowledge* and *visual concept* to be the invariant part of a feature (see paragraph 3 of intro). I don*t see a particular problem with that, but a direct statement like *invariant features* would have resonated with me just fine, while whether we can call that *knowledge* is a matter of (unnecessary) semantics in my opinion. ===== Post rebuttal comments: Thanks to authors for the detailed response and additions. I read through the comments and skimmed the revised PDF. Overall I preserve my positive rating toward this paper. The updates did improve the paper, but I would recommend the authors to use the camera ready opportunity to further address the original comments especially with respect to the delivery. I find the added experiments in multi task learning interesting especially given the time constraints of rebuttal. However the adopted setting is not the strongest case as what is defined to be *two tasks* is basically a superclass classification of a fine grained classifier. More concrete datasets and tasks specifically targeting challenging multi task learning are available now (see the full discussion in the original review). I would recommend taking those into consideration either for more experimentation or calibrating the conclusion made out of the current experiment. I recommend acceptance.