Title: ON THE DISTRIBUTION OF PENULTIMATE ACTIVATIONS OF CLASSIFICATION NETWORKS

Abstract: This paper considers probability distributions of penultimate activations in deep classification networks. We first identify a dual relation between the activations and the weights of the final fully connected layer: learning the networks with the cross-entropy loss makes their (normalized) penultimate activations follow a von Mises-Fisher distribution for each class, which is parameterized by the weights of the final fully-connected layer. Through this analysis, we derive a probability density function of penultimate activations per class. This generative model allows us to synthesize activations of classification networks without feeding images forward through them. We also demonstrate through experiments that our generative model of penultimate activations can be applied to real-world applications such as knowledge distillation and class-conditional image generation.

1. Introduction:

Deep neural networks have achieved remarkable success in image classification (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016; Huang et al., 2017; Hu et al., 2018) . In most of these networks, an input image is first processed by multiple layers of neurons, whose final output, called penultimate activations, is in turn fed to the last fully connected layer that conducts classification; these networks are typically trained in an end-to-end manner by minimizing the crossentropy loss. The penultimate activations are the deepest image representation of the networks and have proven to be useful for various purposes besides classification such as image retrieval (Zhai & Wu, 2019) , semantic segmentation (Noh et al., 2015) , and general image description of unseen classes (Simonyan & Zisserman, 2014) .

This paper studies the nature of penultimate activations in classification networks. We first identify a dual relationship between these activations and the weights of the final classification layer. Specifically, we show that minimizing the cross-entropy loss implicitly performs inference with respect to a specific generative model of the penultimate activations, which is parameterized by the final classification layer. Through this analysis, a probability density function of the penultimate activations of a class is derived; this function can be regarded as an approximate inverse of the final classification layer of the networks, and used to sample penultimate activations of a certain class.

We demonstrate by experiments that our generative model of penultimate activations can be applied to real world applications such as Knowledge Distillation (KD) (Hinton et al., 2015; Ahn et al., 2019) and class-conditional image generation (Kingma et al., 2014; Davidson et al., 2018; Miyato & Koyama, 2018) . For KD, our model enables distilling knowledge from a teacher network without feeding images forward through the teacher by generating its activations directly; this new approach to KD is complementary to the standard one (Hinton et al., 2015) and more robust against domain shift. We also show that our model can be naturally integrated with a class-conditional image generation model and enhance the quality of synthesized images. This paper is organized as follows. We present an analysis of the penultimate activations of a classification network in Section 2, which yields a probabilistic model of these activations. After reviewing our model's relation to previous work in Section 3, we apply our model of penultimate activations to the two real-world applications in Section 4. In Section 5, we conclude the paper with a discussion about limitations and future directions of our approach.

2. Analysis of the Penultimate Activations:

In this section, we examine how the penultimate activations are affected by the process of minimizing the cross-entropy loss of a classification network. Our analysis shows that the statistics of the penultimate activations have a very close relation to the weights of the final classification layer, and yields a scheme to approximately invert the classification layer to infer the distribution of penultimate activations only from the layer's weights. We use this scheme in later sections to generate the penultimate activations of a classification network without feeding any data forward.

2.1. Notation:

We begin with the notation we will use throughout our analysis of classification networks. Logits l are computed by the product of penultimate activations a and the weight matrix W of the final fully connected layer. Denote the

EQUATION

)

We use y to denote the categorical random variable whose value is determined by the softmax values of l, and i to denote the corresponding ground-truth label. We make the standard assumption that the network is trained by minimizing the cross-entropy between the distributions of y and i.

Let ā, w1 , • • • , wc be unit vector normalizations of a, w 1 , • • • , w c respectively:

EQUATION

)

Note that the normalized vectors ā and wi lie on the unit hypersphere S d-1 .

2.2. Cross-Entropy Minimization as Inference:

The cross-entropy loss can be written as an expectation over a and i:

-E a,i∼p(a|i)p(i) log exp(w i a)

j exp(w j a)

= -E a,i∼p(a|i)p(i) log exp( a w i ā)

EQUATION

where a acts as a datapoint-specific temperature term. Note that the joint distribution of a and i is factorized as p(a, i) = p(a|i)p(i) since data depends on labels and activations are determined by data. We assume that directional statistics of a contain sufficient information for classification by themselves; see Section 2.3 for empirical justification of our assumption. Thus, disregarding the temperature term, Eq. ( 3) is simplified as

EQUATION

where q(ā|i) = vMF(ā; wi , w i ) is a von Mises-Fisher (vMF) distribution with mean direction wi and concentration w i . The vMF distribution, an analogue of the Gaussian distribution on the unit hypersphere, is a well-known distribution in directional statistics whose density function is

EQUATION

where µ ∈ S d-1 is mean direction, κ ∈ [0, ∞) is concentration, and C(κ) is a normalizing constant that depends only on κ. In Eq. ( 4), w i is assumed to be constant for all i since we empirically observed that the effect of w i is marginal in terms of classification performance; verification of this assumption is also given in Section 2.3. (He et al., 2016) , D: DenseNet (Huang et al., 2017) , S: ShuffleNet (Ma et al., 2018) , RX: ResNeXt (Xie et al., 2017) .

Note that Eq. ( 4) is the expected negative log posterior probability of i assuming the generative model of normalized activations q(ā, i) = p(i)q(ā|i) with uniform prior p(i). Therefore, minimizing cross-entropy loss can be seen as posterior inference with respect to our specific generative model q(ā|i) for directional statistics of penultimate activations.

Minimizing the cross-entropy loss in Eq. ( 4) aims to maximize q(ā|i) for the ground-truth class i while minimizing all other q(ā|j). The maximization of q(ā|i) is expanded into

EQUATION

where D KL (• || •) is the Kullback-Leibler (KL) divergence. Therefore, the cross-entropy loss directly encourages normalized penultimate activations ā for images of class i to follow our generative model q(ā|i) by minimizing the KL divergence between p(ā|i) and q(ā|i). We additionally have an entropy term H(p(ā|i)) which encourages ā to be more concentrated, but it does not cause p(ā|i) to collapse to a point mass because of the KL divergence term.

2.3. Empirical Verification:

We provide empirical support for our analysis in Section 2.2 by verifying

• Our core assumption that directional statistics of penultimate activations contain sufficient information for classification by themselves,

• The conclusion of Eq. ( 4) and Eq. ( 6) that, for a trained network, p(ā|i) follows a von Mises-Fisher distribution for all i.

Directional statistics are sufficient for classification. When deriving the generative model of ā in a form of vMF distribution, the magnitudes of a and w i are ignored as we assume that their directional statistics are already sufficient for classification. To verify our assumption, we examine how much the performance of pretrained classification networks drop by normalizing their a and w i . Specifically, we choose 9 networks pretrained for the ImageNet classification task (Russakovsky et al., 2015) , and apply them to the ImageNet validation set to measure their performance. As summarized in Table 1 , the performance drop by the normalization is marginal, especially when the network is deeper and more powerful. This result supports our assumption that directional statistics of a is sufficient for classification and the effect of w i is marginal.

Penultimate activations follow vMF distributions. For qualitative verification of our assertion, we visualize in Figure 1 penultimate activations of a classification network trained on the MNIST dataset (LeCun & Cortes, 2010) and the vMF distributions derived from its final classification layer. The network consists of 4 convolution layers followed by the final fully connected layer, and is designed to produce 2-dimensional penultimate activations for ease of visualization. As shown in Figure 1 , normalized penultimate activations are not well aligned with vMF distributions in early stages of training, but become grouped for each class and following the corresponding distributions tightly as training progresses. Additionally, from the density of original penultimate activations in Figure 1 , one can observe that their directions are sufficient for classification as demonstrated also in the previous paragraph. 

3. Related Work:

Understanding deep neural networks. Understanding what a neural network learns about data is a fundamental problem in deep learning. Previous approaches have analyzed classification networks by optimizing an image to maximally activate a specific neuron (Erhan et al., 2009; Yosinski et al., 2015) or to maximize the predicted probability of a specific class (Simonyan et al., 2013) . Mahendran & Vedaldi (2015) uses a similar technique to visualize the entire feature map of an image. Instead of generating representative images, Zhou et al. (2016) ; Selvaraju et al. ( 2017) locates the most salient region within an image for each class by computing a weighted average of each activation channel. While such techniques offer high-level insights into the characteristics of learned activations, there is no obvious way to use these insights to facilitate the learning of other models. In contrast, our generative model allows us to sample activations from, and quantify the relationship between, different classes. Our experiments demonstrate that these features of our model can be applied to real-world problems such as knowledge distillation and accelerating the training of a generative model. Besides the tasks we considered in this paper, our approach of modeling the behavior of a trained network has many other potential applications such as domain adaptation, anomaly detection, and uncertainty calibration.

vMF distributions in deep learning. The von Mises-Fisher (vMF) distribution is a common component of models of directional data. Mixtures of vMFs have been studied for the task of clustering such directional data (Banerjee et al., 2005; Gopal & Yang, 2014) . For Bayesian inference of neural network weights, Oh et al. (2019) consider a decomposition of model weights into radial and directional components, and use vMF distributions to model the directional component. Hasnat et al. (2017) learn a vMF embedding space for deep metric learning. Such hyperspherical embedding spaces have the desirable property that the total surface area decreases as dimension increases beyond 8 (see Figure 2 of Hasnat et al. (2017) ), unlike Euclidean embedding spaces for which volume increases exponentially with dimension. Kumar & Tsvetkov (2019) parameterize word embeddings as vectors on a unit hypersphere and uses the negative log likelihood of a vMF distribution as an objective, reducing the large computations involved in normalizing the softmax in sequence-tosequence models. Our use of directional statistics differs from these previous methods in that we use it as a tool for explaining the behavior of standard classification models rather than for specialized purposes such as constructing a compact embedding space or reducing computation.

4. Applications:

In this section, we demonstrate that our generative model of penultimate activations can be applied to two practical applications, knowledge distillation (Hinton et al., 2015; Ahn et al., 2019; Romero et al., 2014) and class-conditional image generation (Davidson et al., 2018) .

4.1. Class-wise Knowledge Distillation:



4.1.1. Algorithm Details:

Our generative model is first applied to Knowledge Distillation (KD), the task of distilling knowledge from a teacher network T to a student network S (Hinton et al., 2015) . Unlike most of the existing approaches, our model enables KD without feeding images forward through T by directly generating activations of a certain class. Specifically, our model is used to approximate the average prediction of T per class, which is represented as the probability of T 's prediction y given class i and estimated by

EQUATION

where we employ Monte Carlo integration since the exact integral is intractable. Also, each āj is an i.i.d. sample from vMF( wi , κ), where κ is set to 85 for all experiments on KD.

The estimated p T (y|i) in Eq. ( 7) quantifies the relationship between two classes y and i that is captured by T , and is employed as a target for KD in our approach. Recall that the standard KD loss (Hinton et al., 2015) is

EQUATION

where y denotes prediction and x and i are data and label, respectively. The loss in Eq. ( 8) is designed to minimize the KL divergence between p T (y|x) and p S (y|x) for each data x. Unlike this data-wise KD, our approach is a Class-wise KD (CKD) whose objective is

EQUATION

where the categorical distribution p T (y|i) is given by Eq. ( 7). Note again that while the standard KD objective in Eq. ( 8) requires a forward pass through the teacher network T to compute p T (y|x), ours in Eq. ( 9) utilizes the pre-computed distribution p T (y|i) without exploiting T during training of S. This property of CKD is useful especially when it is hard to conduct forward propagation through T (e.g., online learning of S with limited memory and computation power) or if there is domain shift between training datasets for T and S as demonstrated by experiments in Section 4.1.4.

The overall procedures of the standard KD and our CKD are described below in Algorithm 1 and 2, respectively, where the main differences between the two methods are colored in red.

Algorithm 1 KD (Hinton et al., 2015) Require: teacher network x → p T (y|x) Require: student network x → p S (y|x) 1: while not converged do 2:

x, i ∼ p(x, i)

3:

p T ⇐ p T (y|x) 4:

p S ⇐ p S (y|x)

5:

L KD ⇐ -p T • log p S 6: end while Algorithm 2 Class-wise KD (ours)

Require: teacher network x → p T (y|x) Require: student network x → p S (y|x) 1: p T (y|i) ⇐ 1 N N j=1 p T (y|ā j ) 2: while not converged do 3: x, i ∼ p(x, i) 4: p T ⇐ p T (y|i) 5: p S ⇐ p S (y|x) 6:

L CKD ⇐ -p T • log p S 7: end while

4.1.2. Qualitative Analysis on the Effect of CKD:

To investigate which kind of information of T is transfered to S through CKD, we qualitatively examine penultimate activations and their generative models of the two networks on the MNIST dataset. In this experiment, T consists of 4 convolution layers followed by the final fully connected layer, and produces 2-dimensional penultimate activations. S has the same architecture but with half the number of convolution kernels. From the visualization results in Figure 2 , one can observe that T and S have the same cyclic order of classes in the space of their penultimate activations. This demonstrates that Eq. ( 9) encourages S to follow the inter-class relationships captured by T .

Note also that the activation distributions of S are substantially more concentrated compared to those of T . This is similar with the observation of Müller et al. (2019) that embedding vectors of neural networks are more tightly clustered when they are trained with label smoothing. However, instead of the uniform label smoothing of Müller et al. (2019) , our CKD objective in Eq. ( 9) smoothes labels by considering the inter-class relationships in T .

To further investigate whether our CKD objective in Eq. ( 9) can capture inter-class relations learned by large-scale neural networks, we compare those relations extracted from various Imagenet pretrained networks. Specifically, we compute log p T (j|i) as in Eq. ( 7) for every pair of classes using 5 different ImageNet pretrained networks. The results are represented as 1000 × 1000 matrices as shown in Figure 3 , where one can find that all the 5 matrices exhibit similar inter-class relations although their corresponding network architectures vary widely. This implies that the inter-class relations extracted from Eq. ( 7) and Eq. ( 9) reflect the similarity among the groups of data belonging to different classes, to which we attribute the capability of CKD to transfer knowledge from T to S.

4.1.3. Network Compression and Self Distillation:

The effectiveness of CKD is first evaluated on the CIFAR-100 dataset (Krizhevsky et al., 2009) in the scenario of network compression. Following the protocol of Ahn et al. ( 2019), we use WRN-40-2 as T and WRN-16-2 as S, both presented in Zagoruyko & Komodakis (2016) . As summarized in Table 2 , CKD outperforms the baseline "Label" that does not utilize T , and is as competitive as previous distillation techniques like "FitNet" (Romero et al., 2014) and "VID-I" (Ahn et al., 2019) . This result demonstrates that CKD is capable of extracting useful knowledge from T . On the other hand, the performance of CKD is worse than that of the standard "KD" (Hinton et al., 2015) since the data-wise approach can extract a larger amount of knowledge than CKD by feeding individual datapoints forward through T . However, CKD and the standard KD are complementary to each other and the performance is further enhanced by integrating them. We further demonstrate the efficacy of CKD in the self distillation scenario where S has the same architecture with T (i.e., . In this setting, the same tendency has been observed.

4.1.4. KD in the Presence of Domain Shift:

Most KD techniques assume that T and S are trained with the same dataset or, at least, on the same domain. However, this assumption does not hold always in real world settings, e.g., when the dataset used to train T is not available due to privacy issues or S is trained with streaming data that can be We argue that our CKD is more robust against domain shift since it can perform KD without taking input data explicitly. To validate the advantage of CKD, it is evaluated and compared to the standard KD (Hinton et al., 2015) on the CIFAR-100 dataset (Krizhevsky et al., 2009) while simulating domain shift. Specificlly, we consider two different types of domain shift: photometric transform and downsampling. For the photometric transform, we randomly alter brightness, contrast, and saturation of input image with 5 different degrees of alteration; degree 0 means no alteration. Also, for the image downsampling, we reduce the resolution of input image with 3 different rates (×0.75, ×0.5, ×0.25) using nearest neighbor interpolation. Furthermore, as in the setting of Section 4.1.3, we employ WRN-40-2 as T and WRN-16-2 as S.

In this experiment, CKD consistently enhances the performance of the baseline using only ground truth labels ("Label") while the standard KD ("KD") even deteriorates when the domain shift is significant, as summarized in Table 3 . We believe this result is mainly due to the fact that the standard KD strongly depends on the data distribution. On the other hand, the knowledge captured by CKD can still be useful in the presence of domain shift since it extracts inter-class relationships directly from the weights of the final classification layer rather than relying on the data.

4.2. Class-Conditional Image Generation:

We apply our generative model of penultimate activations to class-conditional image generation by modifying the Hyperspherical Variational Auto Encoder (HVAE) of Davidson et al. (2018) , a latent variable model with a hyperspherical latent space. This section first describes our approach along with two baselines, then demonstrates its efficacy by experiments on the MNIST dataset.

4.2.1. Baselines and Our Approach:

Baseline 1 -Hyperspherical VAE (HVAE): HVAE is a latent variable model, which first maps the given data x to a latent variable z ∈ S d with a prior p(z) by an encoder q(z|x), then reconstructs x from z by a stochastic decoder p(x|z). Especially, HVAE assumes that p(z) is a uniform distribution on the unit hypersphere S d . Specifically, the encoder of HVAE is trained by maximizing the following lower bound of the evidence: Table 4 : Comparison on the MNIST generative modeling task.

EQUATION

HVAE:

the latent vector z. Specifically, whereas HVAE assumes that x is generated from z alone (i.e., p(x, z) = p(z)p(x|z)), HVAE-C assumes that x is generated from both z and i (i.e., p(x, z, i) = p(i)p(z)p(x|z, i)). We train HVAE-C by maximizing the following lower bound of the evidence considering i:

EQUATION

Ours -HVAE Conditioned by Learned Prior (HVAE-L): Recall from Section 2.2 that we can utilize the weights of the final fully-connected layer of a classifier network to model a distribution of penultimate activations per class i. We employ this activation distribution conditioned on i as a learned prior for z of HVAE. This approach is similar to HVAE-C in that i is involved in the process of generating x, but the two models differ in where i is integrated; in our HVAE-L, x is generated from z alone, where the distribution of z is determined by i (i.e., p(x, z, i) = p(i)p(z|i)p(x|z)). HVAE-L is trained by optimizing the following objective: E q(z|x,i) log p(x|z) + log p(i) -D KL (q(z|x) || p(z|i)).

(12) Also, the above objective differs from that of HVAE in Eq. ( 10) since the two models assume different generation procedures.

4.2.2. Experiments:

Our model (HVAE-L) is compared against the two baseline models (HVAE, HVAE-C) on the MNIST generative modeling task. The experimental setup including network architecture and hyperparameter setting follows directly that of Davidson et al. (2018) . In addition, we ensure that the dimensionality of the latent vector z is the same for all the models. Also, our prior distribution for HVAE-L is learned by a classification network with the same architecture as the encoder, and its hyperparameter κ is set to 5. Quantitative results of the three models are summarized in Table 4 , where our HVAE-L outperforms both of the baselines, demonstrating that our model of class-conditional activations is a useful prior for class-conditional image generation.

5 Discussion and Future Work

Our analysis focuses on standard classification networks that are trained to convergence, and our modelling assumptions do not necessarily hold in settings that don't satisfy those conditions. An obvious counterexample is a randomly initialized network: the classifier holds no information about the dataset at initialization. As another example, Hoffer et al. (2018) uses a fixed projection matrix as the final layer with no impairment on accuracy at convergence. The final layer of such a network obviously cannot reflect class relations since the weights were determined independently of data.

Next on our agenda includes (1) more precise modeling of penultimate activations of classification network than vMF distributions, and (2) investigating and implementing more applications of our activation generation model such as domain adaptation, data augmentation, and uncertainty calibration. We also plan to develop a new architecture for image classification networks based on the observations in this work.

:



Figures and Tables:

Figure 1: Visualization of penultimate activations and vMF distributions derived from the classification layer on the MNIST dataset. (top) Kernel density estimates of a. (bottom) Distributions of ā represented by dots and vMF distributions q(ā|i) drawn by solid lines.

Figure 2: Visualization of penultimate activations and their vMF distributions on the MNIST dataset. For both of the teacher and student, (left) kernel density estimates of a, (right) distributions of ā represented by dots and vMF distributions q(ā|i) drawn by solid lines.

Figure 3: Visualization of log p T (j|i) of all pairs of classes calculated as Eq. (7) using 5 different networks pretrained on the ImageNet dataset. Best viewed zoomed in.

Baseline 2 -HVAE Conditioned by Concatenation (HVAE-C): For HVAE, a straightforward way to model the distribution of x while taking label i into account is to attach i to the end of

Label Label † KD KD † FitNet † VID-I † CKD CKD+KD Top-1 test accuracy of the student networks on the CIFAR-100 dataset when using WRN-40-2 as the teacher. The scores of the approaches with † are taken directly fromAhn et al. (2019).

Table content: <table><tr><td>WRN-16-2 73.3</td><td>70.4</td><td colspan="2">74.1 72.9</td><td>70.9</td><td>73.3</td><td>73.7</td><td>74.0</td></tr><tr><td>WRN-40-2 76.2</td><td>74.2</td><td colspan="2">77.7 75.8</td><td>74.3</td><td>75.3</td><td>76.6</td><td>77.7</td></tr><tr><td/><td colspan="4">Photometric Transform</td><td colspan="2">Downsampling</td></tr><tr><td/><td>0.2</td><td>0.4</td><td>0.6</td><td>0.8</td><td colspan="3">×0.75 ×0.5 ×0.25</td></tr><tr><td>Label</td><td colspan="7">73.53 73.37 72.01 71.24 69.46 63.29 49.69</td></tr><tr><td>KD</td><td colspan="7">74.05 73.05 69.96 66.14 65.63 57.53 44.96</td></tr><tr><td colspan="8">CKD (ours) 73.86 73.92 73.55 71.86 70.32 63.53 50.02</td></tr></table>

Top-1 test accuracy of the student networks on the CIFAR-100 dataset with various degrees of photometric transform and image downsampling. corrupted by sensor noises. In those cases, the quality of knowledge extracted from T in a data-wise manner could be degraded since T assumes a different data distribution from what S observes.

Table content: <table/>

Peer review (iclr2020):
This paper studies the distribution of the second-to-last layers of neural networks. The paper argues that, under certain assumptions, minimizing the standard cross-entropy loss of a classification task corresponds to minimizing the KL divergence between the distribution of the penultimate activations and a von Mises-Fisher (vMF) distribution. They assess their assumptions through experiments, and use this interpretation of the penultimate activations as vMF distributions on two tasks: knowledge distillation and class-conditional image generation. They argue that this interpretation outperforms other knowledge distillation methods under domain shift, and report a similar success for class conditional image generation. This is an important problem, especially since these penultimate activations have become more frequently used as object representations in downstream tasks. Their result could also help us understand better interpretations of neural networks. The writing is clear throughout, and it*s straightforward to read. The results are based on assumptions that I will try to list below: * normalizing the penultimate activations does not affect classification performance * the class-specific vectors in the final weight matrix have constant normalizations * minimizing cross-entropy loss is equivalent to minimizing the KL divergence between the penultimate activation distribution and a vMF distribution (i.e. the entropy term in equation 6 is negligible) In light of these assumptions, theoretical claims like *learning the networks with the cross-entropy loss makes their (normalized) penultimate activations follow a von Mises-Fisher distribution for each class* in the abstract are false; even if the normalization assumptions are true, this is an approximate result, due to 1) no guarantee that the KL-divergence will be zero and 2) the entropy term in Equation 6. Additionally, Equation 6 is only applied to the numerator of the objective in Equation 4, so the objectives aren*t equivalent. These results are thus necessarily based on approximations and some heuristics. This is not inherently a problem, but I would argue that it necessitates that experiments prove these assumptions to be useful. I find Table 1 to be convincing of the assumption that the weight and activation normalizations are negligible. Figure 1 is also a nice visualization for arguing that penultimate activations are approximately vMF. The MNIST experiment in Table 4 is a red flag, since the presented results do not match the results in the HVAE paper. The paper claims the experimental setup is identical to the HVAE paper, so it is concerning that the results do not match, especially since the results in the HVAE paper appear to outperform the proposed method. It is possible that the results in the original paper are not reproducible or that the experiments differ, but this should be stated in the paper, since the natural assumption would be that the baseline was not adequately reproduced. Small comment: In Table 2 it is not clear from the figure or the caption that CKD is the method the paper is proposing. It would also make sense to bold the best performing method since you do that for the other tables in the paper. I like the paper and think the results are interesting, but because they are based on heuristics and assumptions, the paper currently promises too much. This puts the burden on presenting experiments that are convincing of improvement, which is not currently the case with the HVAE experiment. I would be open to changing my score if these reviews are addressed.