Title: SHADE: SHANNON DECAY INFORMATION-BASED REGULARIZATION FOR DEEP LEARNING

Abstract: Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. We explain why this quantity makes our model able to achieve invariance with respect to input variations. We empirically validate the efficiency of our approach to improve classification performances compared to standard regularization schemes on several standard architectures.

1. INTRODUCTION:

Deep neural networks (DNNs) have shown impressive state-of-the-art results in the last years on numerous tasks and especially for image classification (Krizhevsky et al., 2012; He et al., 2016) . One key element is the use of very deep models with a huge number of parameters. Accordingly, DNNs need to be trained on a lot of data (e.g. ImageNet). Regularization methods such as weight decay (Krogh & Hertz, 1992) , dropout (Srivastava et al., 2014) or batch normalization (Ioffe & Szegedy, 2016) are common practice to mitigate the ratio between the numbers of training samples and model parameters. Despite constant progress of DNNs performance, their generalization ability is still largely misunderstood and theoretical tools such as PAC-based analysis seem limited to explain it. Meanwhile, the question of DNN regularization remains open as demonstrated by Zhang et al. (2017) .

Information Bottleneck (IB) is an interesting framework to address regularization (Tishby et al., 1999) . It suggests that it is possible to compare the generalization ability of different models by studying the compression rate of their representations: a model that is able to remove more information from the input while providing enough information about the label is more likely to generalize well. Considering an input variable X, label C and its deep representation Y = h(X), IB regularizes the training by minimizing the mutual information I(X, Y ) at constant mutual information I(C, Y ). Along with IB also come some theoretical investigations, with the definition of generalization bounds in Shamir et al. (2010) .

A challenge for applying IB principles to DNNs is the definition of information-based regularizers that are differentiable and easily computable in order to be efficiently trained using back-propagation (LeCun et al., 1998) . There are recent attempts in this direction. For example, Alemi et al. (2017) proposes a DNN that predicts the parameters of the distribution of the random representation variables conditionally to the input and use it to compute IB measures with variational bounds. Achille & Soatto (2016) model the information about the data in the learned weights and propose to minimize it.

In this paper, we propose a new regularization loss minimizing H(Y | C), the entropy of the representation variable conditionally to the label variable, instead of the mutual information I(X, Y ). For a deterministic model, meaning that the entropy H(Y | X) is null, we have I(X, Y ) = I(C, Y ) + H(Y | C)1 . By minimizing directly H(Y | C) we explicitly decouple the data fitting term I(C, Y ), which attempts at finding correlations between inputs and labels, and a term which represents the compression rate of the representation H(Y | C).

Our contributions are three-fold. First, we derive in Section 2 a new regularization scheme, called SHADE for SHAnnon DEcay, to tackle the difficulty of getting an accurate, differentiable and scalable estimate of H(Y | C). It is based on a latent code Z representing the activation state of a neuron Y , allowing us to express a complete tractable loss with few additional computation. Second, we discuss the interest of our regularization framework based on conditional entropy in Section 3. In particular, we argue that this measure H(Y | C) is well designed to compare representations in terms of invariance while still taking the target task into account. Finally, we extensively experiment our SHADE regularizer on different deep learning models and several challenging datasets (CIFAR-10, ImageNet, MNIST-M) in Section 4.

2. SHADE: A NEW REGULARIZATION METHOD BASED ON H(Y | C):

Notations and definitions. X is the input from input space X with high dimensionality and C is the class label from finite class space C = {1, 2, .., |C|}. A hypothesis space is designed with a family of parametric (continuous) functions H = {h(w, .) : X → Y; w ∈ W = R m }. A model h ∈ H outputs a representation of the input from which a class is predicted. In many tasks like image classification, the representation Y = h(X) of an input X usually defines a distribution, noted q Y |X , on the class space (e.g. obtained by applying a softmax function). The training in supervised learning consists in finding the parameters w of h minimizing the expectancy of a loss function L cls (Y, C) that quantifies the difference between the prediction and the label. It is usually the cross-entropy between the obtained distribution q Y |X and the one-hot distribution with unit probability for the class

C noted p C : cls (Y, C) = -c∈C p C (c) log q Y |X (c) . A regularization term Ω(•)

with a coefficient β is often added to the classification loss, aiming at influencing the learning toward models with lower complexity2 . In image classification, the most commonly used regularization criterion is the weight decay: Ω(w) = ||w|| 2 2 . In the IB framework, the regularization term would be Ω(w) = I h(w, X), X . The final loss for supervised learning is:

EQUATION

In this section, we will further describe SHADE, a new regularization term based on the conditional entropy H(Y | C) designed to drive the optimization toward more invariant representation.

2.1. ENTROPY-BASED REGULARIZATION FOR DEEP NEURAL NETWORKS:

Layer-wise regularization. A DNN is composed of a number L of layers that transform sequentially the input. Each one can be seen as an intermediate representation variable, noted Y for layer , that is determined by the output of the previous layer and a set of parameters w . Each layer filters out a certain part of the information from the initial input. Thus, from the data processing inequality in Cover & Thomas (1991) can be derived the following inequalities for any :

EQUATION

Similarly to the recommendation of Tishby & Zaslavsky (2015) , we apply regularization to all the layers, using a layer-wise criterion H(Y | C), and producing a global criterion: Since

EQUATION

H(Y | C) = c∈C p(c)H(Y | c) it is necessary to compute |C| different entropies H(Y | c).

This means that, given a batch, the number of samples used to estimate one of these entropies is divided by |C| on average which becomes particularly problematic when dealing with a large number of classes such as the 1,000 classes of ImageNet. Furthermore, entropy estimators are extremely inaccurate considering the number of samples in a batch. For example, LME estimators of entropy in Paninski (2003) converge in O( (log K) 2 /K) for K samples. Finally, most estimators such as LME require discretizing the space in order to approximate the distribution via a histogram. This raises issues on the bins definition considering that the variable distribution is unknown and varies during the training in addition to the fact that having a histogram for each neuron is computationally and memory consuming.

To tackle these drawbacks we investigate the two following tricks: the introduction of a latent variable that enables to use more examples to estimate the entropy; and a bound on the entropy of the variable by an increasing function of its variance to avoid the issue of entropy estimation with a histogram and make the computation tractable and scalable.

Latent code. First, inspecting a neuron Y (before ReLU), the ReLU activation makes it act as a detector, returning a signal when a certain pattern is present on the input. If the pattern is absent the signal is zero, otherwise, it quantifies the resemblance with it. We therefore propose to associate a binomial variable Z with each unit variable Y (before ReLU). This variable Z indicates if a particular pattern is present on the input (Z = 1 when Y 0) or not (Z = 0 when Y 0). It acts like a latent code in variational models (e.g. Kingma & Welling, 2014) or in generative models (e.g. Chen et al., 2016) . In our implementation, we chose a binomial distribution p(Z = 1 | Y ) = sigmoid(Y ) that matches this intuition, as detailed in Sec. 2.3. Furtherfore, it is very likely that most intermediate features of a DNN can take similar values for inputs of different classes -this is especially true for low-level features. The semantic information provided by a feature is thus more about a particular pattern than about the class itself. Only the association of features allows determining the class. So Z represents a semantically meaningful factor about the class C and from which the input X is generated. The feature value Y is then a quantification of the possibility for this semantic attribute Z to be present in the input or not.

Then, we assume the Markov chain C → Z → X → Y . Indeed, during the training, the distribution of Y varies in order to get as close as possible to a sufficient statistic of X for C (see definition in Cover & Thomas, 1991) . Therefore, we expect Z to be such that Y draws near a sufficient statistic of Z for C as well. By assuming the sufficient statistic relation

I(Y, C) = I(Y, Z) we get the equivalent equality H(Y | C) = H(Y | Z)

, and finally obtain:

ω unit (Y | C) = H(Y | C) = H(Y | Z) = z∈{0,1} p(z)H(Y | Z = z).

(5)

This modeling of Z as a binomial variable (one for each unit) has the advantage of enabling good estimators of conditional entropy since we only divide the batch into two sets for the estimation (z = 0 and 1) regardless of the number of classes.

Algorithm 1 Moving average updates: for z ∈ {0, 1}, p z estimates p(Z = z) and µ z estimates

E(Y | Z = z) 1: Initialize: µ 0 = -1, µ 1 = 1, p 0 = p 1 = 0.5, λ = 0.8 2: for each mini-batch {y (k) , k ∈ 1..K} do 3:

for z ∈ {0, 1} do 4:

p z ← λp z + (1 -λ) 1 K K k=1 p(z | y (k) )

5:

µ z ← λµ z + (1 -λ) 1 K K k=1 p(z | y (k) ) p z y (k) 6:

end for 7: end for Variance bound. The previous trick allows computing fewer entropy estimates to obtain the global conditional entropy, thus increasing the sample size used for each entropy estimation. Unfortunately, it does not solve the bin definition issue. To address this, we propose to use the following bound on H(Y | Z), that does not require the definition of bins:

EQUATION

This bound holds for any continuous distributions Y and there is equality if the distribution is Gaussian. For many other distributions such as the exponential one, the entropy is also equal to an increasing function of the variance. In addition, one main advantage is that variance estimators are much more robust than entropy estimators, converging in

O( 1 /K) for K samples instead of O( log(K) 2 /K).

Finally, the ln function being one-to-one and increasing, we only keep the simpler term Var(Y | Z) to design our final loss:

EQUATION

In next section, we detail the definition of the differential loss using Var(Y | Z) as a criterion computed on a mini-batch.

2.3. INSTANTIATING SHADE:

For one unit of one layer, the previous criterion writes:

EQUATION

)

The quantity Var(Y | Z) can be estimated with Monte-Carlo sampling on a mini-batch of input-target pairs (x (k) , c (k) ) 1≤k≤K of intermediate representations y (k) 1≤k≤K as in Eq. ( 9). p(Z | y) interpreted as the probability of presence of attribute Z on the input, it should clearly be modeled such that p(Z = 1 | y) increases with y. The more similarities between the input and the pattern represented by y, the higher the probability of presence for Z. We suggest using:

p(Z = 1 | y) = σ(y) p(Z = 0 | y) = 1 -σ(y) with sigmoid function σ(y) = 1 1 + e -y .

For the expected values µ z = E(Y | z) we use a classic moving average that is updated after each batch as described in Algorithm 1. Note that the expected values are not changed by the optimization since they have no influence on the entropy H(Y | Z).

The concrete behavior of SHADE can be interpreted by analyzing its gradient as described in Appendix C.

For this proposed instantiation, our SHADE regularization penalty takes the form:

EQUATION

)

We have presented a regularizer that is applied layer-wise and that can be integrated into the usual optimization process of a DNN. The additional computation and memory usage induced by SHADE is almost negligible (computation and storage of two moving averages per neuron). Namely, SHADE adds half as many parameters as batch normalization does.

3. CONDITIONAL ENTROPY AS INVARIANCE MEASURE FOR CLASSIFICATION:

Inspired by the IB framework, we have derived a criterion that SHADE loss aims at minimizing. In this section, we further explore the relevance of our conditional entropy term for regularization. We first establish a connection between the entropy H(Y ) of a representation and its invariance. Then we explain why H(Y | C) seems to be a better fitting criterion for a classification task.

H(Y ) to measure invariance. For an input X and a deterministic mapping f (X) = Y , we have H(Y | X) = 0 and therefore:

EQUATION

H(X) being fixed, H(Y ) is inversely related to H(X | Y ). Besides, we assume that H(X | Y )

is a good measure to quantify how invariant a representation is. Indeed, if a representation is invariant to many transformations, many inputs have the same representation. Consequently, given a representation sample, it is difficult to guess from which input it has been computed. These properties are perfectly captured by H(X | Y ), representing the uncertainty in X knowing Y . The bigger the uncertainty, the harder it is to predict X precisely. Concretely, when trying to guess X knowing Y , we can lower bound the error made in the best case, with an increasing function of the conditional entropy as developed in Appendix A. Therefore, it seems that H(Y ) is a good measure of the invariance of the model.

In the particular case of deep learning, He et al. (2016) explain that the stacking of multiple layers is responsible for improving the generalization of DNN. This fact can be explained by the data processing inequality (Cover & Thomas, 1991) . In the case of finite input space, each layer is responsible for filtering a certain amount of information. As clearly illustrated in Tishby & Zaslavsky (2015) , for each stage, the representation has a lower entropy than the representation of the preceding layer. Increasing the depth increases the capacity of the network to reduce the overall entropy of the DNN representation thus increasing their invariance.

H(Y ) as a regularizer. From this intuition, one can use a variance bound similar to Eq. ( 6):

H(Y ) ≤ 1 /2 ln(2πeVar(Y ))

to derive a loss in order to minimize the representations' entropy H(Y ).

For instance, we can show that the weight decay reduces this variance under some approximations.

In fact when Y = w X + b, by estimating Λ = Cov(X), the variance takes the immediate form Var(Y ) = w Λw. If Λ = Id, meaning that the input coordinates are considered independent with unit variance, then Var(Y ) = ||w|| 2 2 . It corresponds to the weight decay regularization or L 2 penalty. Even if within a DNN layer, the batch normalization tends to enforce this unit variance hypothesis and the depth of DNN tends to ensure the independence hypothesis the weight decay remains poor at improving generalization as illustrated in Zhang et al. (2017) . Considering fewer assumptions, it is possible to reduce the variance of an output coordinate by penalizing its empirical variance on a batch during the training. We introduce an alternative loss on the variance called VarEntropy, constructed the same way SHADE has been derived, but avoiding the introduction of a latent variable Z:

EQUATION

)

Even if experimentally, VarEntropy regularization seems to behave better than simple weight decay, it is not as efficient as SHADE and we try to explain why below.

Importance of conditional entropy. We have seen that lowering the entropy of the representation enables to make it more invariant. However, another fact reported by He et al. (2016) is that stacking layers increases the difficulty to train the network. Indeed, when reducing too much the entropy, there is a risk that the information about the label is filtered as well. The representation is so invariant that it is no longer possible to distinguish between the classes. In He et al. (2016) , they solve this issue by forcing the transmission of additional information through skip-connections while IB prescribes to maximize the compression rate at constant information about the label. All this highlights the fact that having invariant representations is interesting if it is intra-class invariant. Since we do not want two inputs from different classes to have the same representation, we prefer to focus on a criterion quantifying the intra-class compression rate in order to maximize intra-class invariance:

H(Y | C).

As We perform image classification on the CIFAR-10 dataset, which contains 50k training images and 10k test images of 32×32 RGB pixels, fairly distributed within 10 classes (see Krizhevsky, 2009, for details) . Following the architectures used in Zhang et al. (2017) , we use a small Inception model, a three-layer MLP, and an AlexNet-like model with 3 convolutional and 2 fully connected layers. We also use a ResNet architecture from Zagoruyko & Komodakis (2016) . Those architectures represent a large family of DNN and some have been well studied in Zhang et al. (2017) within the generalization scope. For training, we use randomly cropped images of size 28×28 with random horizontal flips.

For testing, we simply center-crop 28×28 images. We use momentum SGD for optimization (same protocol as Zhang et al., 2017) .

We compare SHADE with three regularization methods, namely weight decay, dropout and Var-Entropy presented in Eq. ( 12). For all architectures, the regularization parameters have been crossvalidated to find the best ones for each method and the obtained accuracies on the test set are reported in Table 1 .

We obtain the same trends as Zhang et al. (2017) , which get a small improvement of 0.31% with weight decay on AlexNet. The improvement with weight decay is slightly more important with ResNet and Inception (0.87% and 0.90%) probably thanks to the use of batch normalization. In our experiments dropout improves generalization performances only for AlexNet and MLP. It is known that the use of batch normalization lowers the benefit of dropout, which is in fact not used in He et al. (2016) .

We first notice that for all kind of architectures the use of SHADE significantly improves the generalization performance. Finally, SHADE shows better performances than dropout on all architecture except on AlexNet, for which they seem to be complementary, probably because of the very large number of parameters in the fully-connected layers, with best performances obtained with SHADE coupled with dropout (named SHADE+D). This association is also beneficial for MLP. On Inception and ResNet, even if dropout and SHADE independently improve generalization performances, their association is not as good as SHADE alone, probably because it enforces too much regularization.

4.2. LARGE SCALE CLASSIFICATION ON IMAGENET:

In order to experiment SHADE regularization on very large scale dataset, we train on ImageNet (Russakovsky et al., 2015) a WELDON network from Durand et al. (2016) adapted from ResNet-101. This architecture changes the forward and pooling strategy by using the network in a fully-convolutional way and adding a max+min pooling, thus improving the performance of the baseline network. We used the pre-trained weights of ResNet-101 (from the torchvision package of PyTorch) giving performances on the test set of 77.56% for top-1 accuracy and 93.89% for top-5 accuracy. Provided with the pre-trained weights, the WELDON architecture obtains 78.51% for top-1 accuracy and 94.65% for top-5 accuracy. After fine tuning the network using SHADE for regularization we finally obtained 80.14% for top-1 accuracy and 95.35% for top-5 accuracy for a concrete improvement. This demonstrates the ability to apply SHADE on very large scale image classification successfully.

4.3. TRAINING WITH A LIMITED NUMBER OF SAMPLES:

When datasets are small, DNNs tend to overfit quickly and regularization becomes essential. Because it tends to filter out information and make the network more invariant, SHADE seems to be well fitted for this task. To investigate this, we propose to train DNNs with and without SHADE on CIFAR-10 and MNIST-M with different numbers of samples in the training set.

First, we tested this approach on the digits dataset MNIST-M (Ganin & Lempitsky, 2015) . This dataset consists of the MNIST digits where the background and digit have been replaced by colored and textured information (see Fig. 1c for examples). The interest of this dataset is that it contains lots of unnecessary information that should be filtered out, and is therefore well adapted to measure the effect of SHADE. A simple convolutional network has been trained with different numbers of samples of MNIST-M and the optimal regularization weight for SHADE have been determined on the validation set (see training details in Appendix D). The results can be seen on Figure 1a . We can see that especially for small numbers of training samples (< 1000), SHADE provides an important gain of 10 to 15% points over the baseline. This shows that SHADE helped the model in finding invariant and discriminative patterns using less data samples.

Additionally, Figure 1c shows samples that are misclassified by the baseline model but correctly classified when using SHADE. These images contain a large amount of intra-class variance (color, texture, etc.) that is not useful for the classification tasks, explaining why adding SHADE, that encourages the model to discard information, allows important performance gains on this dataset and especially when only few training samples are given.

Finally, to confirm this behavior, we also applied the same procedure in a more conventional setting by training an Inception model on CIFAR-10. Figure 1b shows the results in that case. We can see that once again SHADE helps the model gain in performance and that this behavior is more noticeable when the number of samples is limited, allowing a gain of 6% when using 4000 samples.

4.4. CLASS-INFORMATION PRESERVATION BY CONDITIONAL ENTROPY:

The main difference between SHADE and VarEntropy is the introduction of the latent variable Z, supposed to contain the information about the label C. The motivation of this extension is that minimizing H(Y | C) instead of H(Y ) enables to save the class information during the optimization of the loss, as explained in Section 3. To illustrate this benefit, we compare the impact of the two regularization losses on the classification performances of a pre-trained model. To do so, we fine tune a pre-trained Inception model only with a regularization loss, either based on Var(Y ) or Var(Y | Z); without any label data or classification loss. The network performance obviously declines for both regularizers as we can see on Figure 2 . However, this decline is slower with SHADE than VarEntropy. This confirms the intuition that Z contains class-information and that SHADE produces less class-information filtering. This is explained by the optimization of the metric Var(Y | Z) that uses implicitly-learned information encoded in the network.

5. CONCLUSION:

In this paper, we introduced a new regularization method for DNNs training, SHADE, which focuses on minimizing the entropy of the representation conditionally to the labels. This regularization aims at increasing the intra-class invariance of the model while keeping class information. SHADE is tractable, adding only a small computational overhead when included into an efficient SGD training. We show that our SHADE regularization method significantly outperforms standard approaches such as weight decay or dropout with various DNN architectures on CIFAR-10. We also validate the scalability of SHADE by applying it on ImageNet. The invariance potential brought out by SHADE is further illustrated by its ability to ignore irrelevant visual information (texture, color) on MNIST-M. Finally, we also highlight the increasing benefit of our regularizer when the number of training examples becomes small.

A ENTROPY BOUNDING THE RECONSTRUCTION ERROR:

In section 3 we highlight a link between the entropy H(X | Y ) and the difficulty to recover the input X from its representation Y . Here we exhibit a concrete relation between the reconstruction error, that quantifies the error made by a strategy that predicts X from Y , and the conditional entropy. This relation takes the form of an inequality, bounding the error measure in the best case (with the reconstruction strategy that minimizes the error) by an increasing function of the entropy. We note x(Y ) ∈ X the reconstruction model that tries to guess X from Y .

The discrete case In case the input space is discrete, we consider the zero-one reconstruction error for one representation point

Y : ε(Y ) = E X [1(x(Y ) = X)]

. This is the probability of error when predicting x(Y ) from Y . The function that minimizes the expected error

E = E Y (ε(Y )) is x(Y ) = arg max x∈X p(x | Y ) as shown in Proof 1.

We derive the following inequality:

EQUATION

The left side of the inequality uses Fano's inequality in Cover & Thomas (1991) , the right one is developed in proof 2. This first bound shows how the reconstruction error and the entropy are related. For very invariant representations, it is hard to recover the input from Y and the entropy of H(X | Y ) is high.

Besides, there can be an underlying continuity in the input space and it could be unfair to penalize predictions close to the input as much as predictions far from it. We expose another case below that takes this proximity into account.

The continuous case In the case of convex input space and input variable with continuous density, we consider the 2-norm distance as reconstruction error: (24) The Eq. ( 21) is obtained using Jensen inequality and Eq. ( 23) is obtained using the result of Proof 1. The gradient of Ω SHADE with respect to w is:

ε(Y ) = E X|Y ||X -X(Y )||

∇ w Ω SHADE = (δ 1 + δ 2 )x

with δ 1 = σ(y)(1 -σ(y)) (y -µ 1 ) 2 -(y -µ 0 ) 2 and δ 2 = 2σ(y)(y -µ 1 ) + 2(1 -σ(y))(y -µ 0 ).

We can interpret the direction of this gradient by analyzing the two terms δ 1 and δ 2 as follows:

• δ 1 : If (y -µ 0 ) 2 is bigger than (y -µ 1 ) 2 that means that y is closer to µ 1 than it is to µ 0 . Then δ 1 is positive and it contributes to increasing y meaning that it increases the probability of Z being from mode 1. In a way it increases the average margin between positive and negative detections. Note that if there is no ambiguity about the mode of Z meaning that σ(y) or 1 -σ(y) is very small then this term has negligible effect. • δ 2 : This term moves y toward the µ z of the mode that presents the bigger probability. This has the effect of concentrating the outputs around their expectancy depending on their mode to get sparser activation.

:

For any two variables U , V we haveI(U, V ) = H(U ) -H(U | V ) = H(V ) -H(V | U ).

There are other methods of regularization such as dropout(Srivastava et al., 2014) or data augmentation(van Dyk & Meng, 2001) that do not apply directly to the loss.

This upper bound is well justified in deep learning as the neurons of a layer tend to be more and more independent of each other as we go deeper within the network.

Figures and Tables:

It demonstrates the ability of SHADE to regularize the training of deep architectures. Moreover, SHADE systematically outperforms other regularizations of the same type Examples of MNIST-M images misclassified by the baseline and correctly classified using SHADE, both trained with 250 samples.

Figure 1: Results when training with a limited number of samples in the training set for MNIST-M and CIFAR-10 with and without SHADE.

Figure 2: Evolution of the accuracy of a pre-trained Inception model on CIFAR-10 when only applying regularization.

2 2 . This error penalizes the average distance of the input and its reconstruction from Y . The function that minimizes the expected error E = E Y [ε(Y )] is the conditional expected value: x(y) = E[X | Y = y]. Then E = Var(X | Y ). Helped by the well-known inequality H(X | Y ) ≤ 1 2 ln(2πeVar(X | Y )) we obtain: notice that a high entropy H(X | Y ) implies a high reconstruction error in the best case. (y) | y) ≤ p(arg max x∈X p(x | y) | y), (18) the reconstruction that minimizes the error is x(y) = arg max x∈X p(x | y). However, this is theoretical because in most cases p(x | Y ) is unknown. | y) log(p(x | y)) dx dy (23) = -H(X | Y ).

| y)(y -E(Y | z)) 2 dy dz. (30)C SHADE GRADIENTSHere is studied the influence of SHADE on a gradient descent step for a single neuron Y of a single layer and for one training sample X. The considered case of a linear layer, we have: Y = w X + b.

)

Table content: <table><tr><td colspan="5">unit i of every layer we design a loss ω unit (Y ,i | C) = H(Y ,i | C) that will be part of the global</td></tr><tr><td>regularization loss:</td><td/><td/><td/></tr><tr><td>L</td><td>D</td><td/><td/></tr><tr><td>Ω layers ≤ Ω units =</td><td/><td>H(Y ,i | C)</td><td>.</td><td>(4)</td></tr><tr><td>l=1</td><td>i=1</td><td>ωunit(Y ,i |C)</td><td/></tr><tr><td colspan="5">Later in the paper, we use the notation Y instead of Y ,i for simplicity since the coordinates are all</td></tr><tr><td>considered independently to define ω unit (Y ,i | C).</td><td/><td/><td/></tr><tr><td>2.2 ESTIMATING ENTROPY</td><td/><td/><td/></tr><tr><td colspan="5">In this section, we describe how to define a loss based on the measure H(Y | C) with Y being one</td></tr><tr><td colspan="5">coordinate variable of one layer. Defining this loss is not obvious as the gradient of H(Y | C) with</td></tr><tr><td colspan="5">respect to the layer's parameters may be computationally intractable. Y has an unknown distribution</td></tr><tr><td colspan="5">and without modeling it properly it is not possible to compute H(Y | C) precisely for the following</td></tr><tr><td>reasons.</td><td/><td/><td/></tr><tr><td colspan="5">Unit-wise regularization. Examining one layer , its representation variable is a random vector</td></tr><tr><td colspan="5">of coordinates Y ,i and of dimension D : Y = (Y ,1 , ..., Y ,D ). The upper bound 3 H(Y | C) ≤</td></tr><tr><td colspan="5">D i=1 H(Y ,i | C) enables to define a unit-wise criterion that SHADE seeks to minimize. For each</td></tr></table>

mentioned in the introduction, H(Y | C) thus differs from standard IB frameworks based on I(X, Y )(Achille & Soatto, 2016;Alemi et al., 2017). Contrarily to the mutual informationI(X, Y ) = H(Y | C) + I(C, Y ), our regularizer applied to H(Y | C) ignores I(C, Y ). When minimizing I(X, Y ), there is no control on how both terms H(Y | C) and I(C, Y ) are modified. Our regularizer is therefore beneficial in the sense that minimizing H(Y | C) does not conflict with the mutual information I(C, Y ) between the representation and the label, information that is therefore for classification and should not be penalized. Classification accuracy (%) on CIFAR-10 test set.

Table content: <table><tr><td>4 EXPERIMENTS</td><td/><td/><td/><td/></tr><tr><td colspan="5">4.1 IMAGE CLASSIFICATION WITH VARIOUS ARCHITECTURES ON CIFAR-10</td></tr><tr><td/><td colspan="4">MLP AlexNet ResNet Inception</td></tr><tr><td>No regul.</td><td>62.38</td><td>83.25</td><td>89.84</td><td>90.97</td></tr><tr><td colspan="2">Weight decay 62.69</td><td>83.54</td><td>91.71</td><td>91.87</td></tr><tr><td>Dropout</td><td>65.37</td><td>85.95</td><td>89.94</td><td>91.11</td></tr><tr><td>VarEntropy</td><td>63.70</td><td>83.61</td><td>91.72</td><td>91.83</td></tr><tr><td>SHADE</td><td>66.05</td><td>85.45</td><td>92.15</td><td>93.28</td></tr><tr><td>SHADE+D</td><td>66.12</td><td>86.71</td><td>92.03</td><td>92.51</td></tr></table>

Peer review (iclr2018):
 Summary: The paper presents an information theoretic regularizer for deep learning algorithms. The regularizer aims to enforce compression of the learned representation while conditioning upon the class label so preventing the learned code from being constant across classes. The presentation of the Z latent variable used to simplify the calculation of the entropy H(Y|C) is confusing and needs revision, but otherwise the paper is interesting. Major Comments: - The statement that I(X;Y) = I(C;Y) + H(Y|C) relies upon several properties of Y which are not apparent in the text (namely that Y is a function of X, so I(X;Y) should be maximal, and Y is a smaller code space than X so it should be H(Y)). If Y is a larger code space than X then it should still be true, but the logic is more complicated. - The latent code for Z is unclear. Given the use of ReLUs it seems like Y will be O or +ve, and Z will be 0 when Y is 0 and 1 otherwise, so I*m unclear as to when the value H(Y|Z) will be non-zero. The data is then partitioned within a batch based on this Z value, and monte carlo sampling is used to estimate the variance of Y conditioned on Z, but it*s really unclear as to how this behaves as a regularizer, how the z is sampled for each monte carlo run, and how this influences the gradient. The discussion in Appendix C doesn*t mention how the Z values are generated. - The discussion on how this method differs from the information bottleneck is odd, as the bottleneck is usually minimising the encoding mutual information I(X;Y) minus the decoding mutual information I(Y;C). So directly minimising H(Y|C) is similar to the IB, and also minimising H(Y|C) will affect I(C;Y) as I(C;Y) = H(Y) - H(Y|C). - The fine tuning experiments (Section 4.2) contain no details on the parameters of that tuning (e.g. gradient optimiser, number of epochs, batch size, learning rates etc). - Section 4.4 is obvious, and I*d consider it a bug if regularising with label information performed worse than regularising without label information. Essentially it*s still adding supervision after you*ve removed the classification loss, so it*s natural that it would perform better. This experiment could be moved to the appendix without hurting the paper. - In appendix A an upper bound is given for the reconstruction error in terms of the conditional entropy. This bound should be related to one of the many upper bounds (e.g. Hellman & Raviv) for the Bayes rate of a predictor, as there is a fairly wide literature in this area. Minor Comments: - The authors do not state what kind of input variations they are trying to make the model invariant to, and as it applies to CNNs there are multiple different kinds, many of which are not amenable to a regularization based system for inducing invariance. - The authors should remind the reader once that I(X;Y) = H(Y) - H(Y|X) = H(X) - H(X|Y), as this fact is used multiple times throughout the paper, and it may not necessarily be known by readers in the deep learning community. - Computing H(Y|C) does not necessarily require computing c separate entropies, there are multiple different approaches for computing this entropy. - The exposition in section 3 could be improved by saying that H(X|Y) measures how much the representation compresses the input, with high values meaning large amounts of compression, as much of X is thrown away when generating Y. - The figures are difficult to read when printed in grayscale, the graphs should be made more readable when printed this way (e.g. different symbols, dashed lines etc). - There are several typos (e.g. pg 5 "staking" -> "stacking").