Title: INFORMATION REGULARIZED NEURAL NETWORKS

Abstract: We formulate an information-based optimization problem for supervised classification. For invertible neural networks, the control of these information terms is passed down to the latent features and parameter matrix in the last fully connected layer, given that mutual information is invariant under invertible map. We propose an objective function and prove that it solves the optimization problem. Our framework allows us to learn latent features in a more interpretable form while improving the classification performance. We perform extensive quantitative and qualitative experiments in comparison with the existing state-of-the-art classification models.

1. INTRODUCTION:

Quantities of information are nonlinear measures capable of describing complex relationship between unstructured data and they form the basis of the probabilistic algorithms in the literature of machine learning. Information theoretic methods are also reported to be effective on improving deep generative models (Chen et al. (2016) ; Kim & Mnih (2018) ) and deep learning models for classification (Grandvalet & Bengio (2004) ; Pereyra et al. (2017) ). Information Bottleneck (IB) problem (Tishby et al., 1999) is formulated as: minimize I(X; T ) -λI(Y ; T ) ,

where the solution random variable T is interpreted as a minimal sufficient representation of signal X for label Y and the mutual information is defined as

I(X; Y ) = X Y

p(x, y) log p(x, y) p(x)p(y) dydx .

(2)

The term I(X; T ) has its origins in Lossy Compression and Rate-Distortion Theory (Cover & Thomas, 2006) , conveying an simple idea of "keep only what is relevant".

However, Saxe et al. (2018) argued that the mutual information I(X; T ) between signal X and feature T in intermediate layer is infinite, as the transformation from X to continuous random variable T is deterministic. In addition they showed experimentally that layers equipped with ReLU actually do not compress too much information, which is supported by many recent work on the invertibility of the neural network (Dosovitskiy & Brox (2015) ; Jacobsen et al. (2018) ). This motivates us to consider a different problem with similar principle idea: we would like to establish a theoretically valid objective that allows the neural network to extract only the relevant information for classification from the data.

We focus on the discrete prediction random variable Y inferred by the probabilistic model P( Y |X) and introduce the following information optimization problem for supervised classification: maximize I(Y ; Y ) subject to I(X; Y ) -I(Y ; Y ) < τ , for some τ > 0 .

(3)

The intuition behind this objective lies in twofold:

Information perspective: A good classification model should be robust against irrelevant features of X, and prevent over-fitting in the learning process. In optimization problem (3) we maximize the relevant information I(Y ; Y ), while constraining the irrelevant information I(X; Y ) -I(Y ; Y )

that X has about Y . Although I(X; Y ) -I(Y ; Y ) converges to zero as I(Y ; Y ) approaching its maximum (see Figure 1L ), in practice it's never attained due to the limited capacity of the models or over-fitting. Our proposed constrain addresses the problem of over-fitting: if two models achieve the similar classification accuracy, this constraint prefers the one that does not overfit to spurious factors of variation in X (e.g., pixel-level artifact/noise in the image that accidentally correlates to the labels in the training data).

Prediction confidence perspective: A good classification model should not be certain about its decision which is in fact wrong. However, modern neural networks are too confident in their predictions (Guo et al., 2017; Szegedy et al., 2015; Pereyra et al., 2017) . To be more precise, high capacity neural networks mostly assign labels of data with prediction confidence near 0 or 1. In particular, they assign 0 probability to some correct labels and therefore do not have enough flexibility to correct themselves from making the wrong prediction. We propose to compress the irrelevant information I(X; Y ) -I(Y ; Y ), where minimizing I(X; Y ) decreases the confidence on all predictions but maximizing I(Y ; Y ) increases the confidence on the correct predictions. Therefore the overall effect reduces the certainty on the false prediction of Y (see Figure 1L ).

To solve this optimization problem, we first present some insights on the dynamics of deep neural network, which can be decomposed into two stages: (i) Transformation stage: samples {X k } k=1:n of the high dimensional unstructured signal X are transformed under the deep invertible (information preserving) feature map F to become (almost) linearly separable; (ii) Classification stage: the weight matrix w in the last fully connected layer together with the Softmax function, takes structured features {F (X k )} k=1:n as input and gives predictions.

Invertibility of F allows neural networks to pass the control of I(X; Y ) = I(F (X); Y ) towards F (X) and w in the last layer, where F (X) can be interpreted as transformed signal that preserves information about the original signal X and the inference model becomes conceptually linear with classifier w (see Figure 1R ). In Section 2 we derive objective function ( 7) and prove that it solves the optimization problem (3). We show the classification performance is improved in Section 4.1 and the features F (X) are sculpted into a form with more interpretability entry-wise in Section 4.2. The optimal solution is obtained when the smaller disks coincide, which is typically not achieved in practice. In particular, the trained model may be extremely confident in its prediction (when H( Y ) lies inside of H(X)), but predicts the wrong label (having large grey area).

Our optimization problem explicitly prohibits the growth of grey area throughout the training. (R) Logic chart of our formulation: our proposed optimization problem only involves F (X) and w, allowing us to have control over the latent feature F (X) directly.

The invertibility property has been empirically demonstrated for complex non-linear deep neural networks that are widely used in practice. We will discuss the literature in Section 3. In addition, we prove in Proposition C.1 that a lower bound of classification error is minimized if neural network is invertible.

Our contribution: Our contribution lies in the following: (i) we formulated a novel information optimization problem for supervised classification; (ii) we propose a simple objective function that improves supervised deep learning with better performance and interpretability; (iii) we formally justify the use of 1 , 2 regularization from an information perspective. Different from the naive regularization on w, our regularization on w T F (X) is novel and effective.

2. MAIN RESULTS:

Consider a classification problem where the training data D = {(x k , y k )} k=1:n are sampled from random variable pair (X, Y ) with unknown joint distribution. Each x k is fed into a deep probabilistic model, which outputs probability densities and predicts y k , a realization of the prediction random variable Y . Let C denote the label class and X denote the signal space, then the mutual information between random variables, e.g., continuous X and discrete Y , is defined as I(X; Y ) = y∈C X p(x, y) log p(x, y) p(x)p( y) dx, and the entropy of Y is defined as H(Y ) = -y∈C p(y) log p(y).

We first call out our assumptions used throughout our analysis. (I): we assume the marginal densities of Y, Y are uniform over C; (II): there exists a unique true label for every sample of X.

Mutual information is bounded and its gradient with respect to logits is approximately zero over a large domain. In particular if the logits are initially small for true labels, gradient updates cannot effectively correct them. Therefore it's not practical to train mutual information terms directly. In this section we introduce alternative terms and prove that they are feasible for our purpose. We show in Proposition 2.2 that I(Y ; Y ) is maximized if the classical cross entropy objective is minimized. On the other hand, we show in Proposition 2.1 that for invertible F , I(X; Y ) = I(F (X); Y ) is minimized if w T F (X) is minimized. We derive our objective function (7) in Section 2.2. Our experimental result in Section 4.1 verifies that the proposed objective function does compress the irrelevant information I(X; Y ) -I(Y ; Y ).

2.1. SOLVING OPTIMIZATION PROBLEM WITH FEASIBLE TERMS:

Without loss of generality, we consider the binary classification problem, i.e. the label class C = {±1}. To tract the population quantities I(F (X); Y ) and I(Y ; Y ), we decompose each of them into an empirical part and a probabilistic bound, which is negligible if sample size n is large. In Proposition 2.1, we show that in order to compress I(F (X); Y ), we need to compress the norm of classifier w and feature F (X). In particular, smaller |w T F (X)| represents lower confidence of the model on its predictions Y , indicating a smaller amount of mutual information I(F (X); Y ). The proof is provided in Appendix A. 

k=1 log σ(y k w T F (x k )) given that y k w T F (x k ) > 1 2 , for all k ∈ {1, ..., n}. Here π y y = 1 2ny ny i=1 σ( yw T F (x i ))

is an unbiased estimate of p(y, y) and π y+ = y∈C π y y , π + y = y∈C π y y .

2.2. DERIVATION OF OBJECTIVE FUNCTION:

In Lagrangian form of optimization problem (3), the constant τ can be dropped and the objective becomes

maximize (1 + λ)I(Y ; Y ) -λI(F (X); Y ) ⇐⇒ maximize I(Y ; Y ) - λ 1 + λ I(F (X); Y ) . (4)

Consider a single signal x k and its true label y k , we propose the following objective function for binary supervised classification problem:

EQUATION

where R is some regularizer. According to results in Section 2.1, minimizing (5) allows us to maximize I(Y ; Y ) while constraining I(X; Y ). We typically choose the hyper-parameter α > 0 to be a reasonably small number. The intuition comes from the observation that λ/(1 + λ) is upper bounded by one. If we compress I(X; Y ) harshly, then neural networks may choose to minimize I(F (X); Y ) at a cost of minimizing I(Y ; Y ).

Recall from the information theoretic perspective of our proposed optimization problem, our regularizer should prefer a model that does not overfit among all the ones with high training accuracy.

In this case neural networks assign only large logits w T y k F (x k ) to true label y k for each signal x k , and generalization of (5) to multi-class case for I(F (X); Y ) can be simplified to constraining

w T y k F (x k ) -w T j F (x k )

, where w j is the jth column of w, assigning feature F (x k ) a probability to label j. We propose to simply constrain w T y k F (x k ) and does not encourage increasing w T j F (x k ):

EQUATION

In our experiment, we take the Elastic Net approach by Zou & Hastie (2005) using a combination of 2 and 1 regularizers: we use Holder's inequality to bound

|w T y k F (x k )| with both sup F (X) • w y k 1 and w y k 2 • F (x k ) 2 .

In practice we assume sup F (X) to be a constant and is absorbed into the hyper-parameter. Our proposed objective function is of the form: 

EQUATION

3. RELATED WORK:

Recent experimental work reported that neural networks with invertible structure have better performance. Dosovitskiy & Brox (2015) showed that images can be resconstructed from the latent features in AlexNet through an inverting process; this reconstruction is further improved by Zhang et al. (2016) , where they built an encoder-decoder structure to encourage invertibility and showed reconstructive objective is beneficial to the performance of the neural network (e.g., VGGNet). Information Maximization(RIM) approach for classification problems; Chen et al. (2016) proposed to add information ingredients to the objective of GANs, encourage to learn disentangled representations.

Our framework decomposes deep neural network into a composition of nonlinear transformation map and a linear probablistic model; this idea was originated in Bell & Sejnowski (1995) where they considered the blind separation problem and decompose the prediction Y to be the sum of an invertible deterministic part and a stochastic part. Amjad & Geiger ( 2018 

4. EXPERIMENTS:

In our experiments we build the feature map F with ResNet or InvNet (introduced in Section 4.2).

In Appendix D we prove that ResNet by He et al. (2015) is invertible under mild assumptions. We prefix the name of the model trained under our objective with "Reg", i.e. RegResNet/RegInvNet.

4.1. PERFORMANCE:

We report the accuracy of ResNet on test data of CIFAR100 in Table 1 . 1We compare the performance between our proposed regularization on w T y k F (x k ) and the naive regularization on w. In both form of regularization we take α 1 = 0 and test over different choices of α 2 . We pick smaller α 2 for naive regularization because it's applied to the full matrix. We observe the performance of ResNet-32 under naive regularization drops monotonically as α 2 increases.

Under a suitable choice of hyperparameters, RegResNet outperforms ResNet by a noticeable margin. It implies that our proposed constrain on the irrelevant information I(X; Y ) -I(Y ; Y ) is beneficial to the classification performance. However, if the hyperparameters are too large, the performance drops, i.e. α 2 = 0.05 for ResNet-32 and α 2 = 0.15 for ResNet-Wide; this matches the discussion in Section 2.2 that the model may try to reduce the relevant information I(Y ; Y ). Our approach addresses the problem of over-fitting; ResNet-Wide is improved by a larger margin compared to ResNet-32 because ResNet-Wide has higher capacity and is therefore over-fits more to the training data.

4.2. ANALYSIS ON FEATURES:

We introduce another invertible structured neural network on MNIST dataset and analyze the feature F (X) learned in the last layer qualitatively. The feature map F is built to be LeNet-300-100 and the decoder D has the opposite structure. At each step during the training process, we update the autoencoder F + D and the InvNet F + w alternatively; our regularization is applied to w as usual.

In this section we report our result with α 1 = α 2 = 0.002.

4.2.1. LEARNING TO OVERLOOK IRRELEVANT INFORMATION:

We feed 1k testing samples of digit 9 into the neural network to get 1k samples of features F (X).

Recall that the features F (X) are vectors of dimension 100 and w is a matrix of size 100 × 10. We calculate the mean and standard deviation of each entry of the features from these 1k samples. From Figure 2 we see that under our regularization, the entry-wise products w 10 T i F (X) i becomes sparse as only a few entries have large values for both feature and weight. This implies that the information needed to compute the logits for classification is encoded into only a few entries of the feature, which we regard as relevant entries. Our regularization forces small w 10 on irrelevant entries, so the logits it outputs are not sensitive to variations in these entries; on the other hand, we do encourage large w 10 on the relevant entries. This matches our motivation that a neural network should be robust against irrelevant information I(F (X); Y ) -I(Y ; Y ) and focus on the relevant information I(Y ; Y ). Recall that the features F (X) are vectors of dimension 100 and w is a matrix of size 100 × 10. The horizontal coordinate represents the index of entries of F (X) ranging from 0 to 99, the vertical coordinate represents entry values of the 10th column of w, the values of mean and standard deviation for each entry of features from samples of digit 9. For RegInvNet, w 10 only assigns large value to representative entries and is more robust against perturbations in other irrelevant entries.

4.2.2. LEARNING TO EXTRACT RELEVANT INFORMATION:

It had been argued by Szegedy et al. (2013) that it is the space but not individual units in high level features that encodes interpretable information. Under our regularization, a meaningful basis of the informative space is found; in particular, the features of 9 are encoded into 10 entries (see Figure 2 ). On the other hand, features that have high values in these entries are expected to be the features of digit 9.

To validate this conjecture, for each digit, we find the entry of its feature mean with the highest value. We use the micro and macro average ROC metric on these feature entries and compare the results from InvNet and RegInvNet in Figure 3 . The curve with larger area underneath indicates higher representative power of individual entries learned in the features. We conclude that under our regularization, relevant information for classification is encoded into only a few key entries of the features, and these entries are highly indicative and interpretable. 

4.3. RELATIONSHIP TO LINEAR REGRESSION:

We trained ResNet-32 on CIFAR10, the feature F (X) is a vector of size 64, the classifier w is a matrix of size 64 × 10. Note that the product w T F (X) is a vector of size 10 representing the probability assigned by the model for each class.

Under our framework, deep learning can be conceptually simplified to regularized linear regression if we regard F (X) as input data. However, F (X) depends on the model parameters in the previous layers so it's not fixed like real data. Moreover we observe in our experiments that naive regularization on w alone will upscale the norm of F (X), which neutralizes the effect of regularization.

In Figure 4 we show that as under our regularization, the 2 -norm of w is suppressed while the 2norms of feature F (X) remain similar. In addition, several rows of w are trained to be zero, which implies that many entries of the feature F (X) are regarded by the network as irrelevant information for classification, since any variance in the entries of F (X) where the corresponding rows of w are zero has no influence on the probability assigned to each label class by the model.

Figure 4 : Compression of the RegResNet-32 (Blue) and the original ResNet-32 (Orange) on CIFAR-10 over the training process: the first plot records the average 2 norm of the last layer features F (X) in a batch; the second plot records the average 2 norm of the columns of w; the third plot records the ratio of zero entries among all entries of w; the plots for trained w with/without regularization after 84000 steps are listed on the right. Best test accuracy are 92.86% and 92.64% for regularized and original ResNet-32 respectively. Under our regularization, the norm of the feature learned remains similar and the norm of classifier w is smaller. Therefore w is less sensitive to "support" and "outlier" features.

5. THE ROLE OF INVERTIBILITY:

Invertibility allows us to treat F (X) as transformed data that preserves all the information from X, and therefore work on the information regularization problem under a linear scheme.

In Appendix D we prove that ResNet is fairly invertible due to the intrinsic invertibility of the operator I + L given L < 1. In this section we build a PlainNet by using only L as the operator for each building block, so the theoretical guarantee for invertibility is not present for PlainNet. In Table 2 , we see that PlainNet-32 can still benefit from our regularization, however, it's performance is less stable compared to ResNet-32 if the hyper-parameters are too large. The reason is for PlainNet, the feature in the last layer F (X) does not preserve information about X very well, so it has a higher demand on the capacity of the classifer w and is therefore more sensitive to our regularization. The means and standard deviations reported are based on 5 samples. It can be observed that although PlainNet-32 obtains marginal improvement for small α 2 , the performance drops dramatically and becomes unstable as α 2 increases.

6. CONCLUSION:

We give an interpretation of the deep learning dynamics by decomposing it into an signal transformation stage and feature classification stage, where we emphasis importance of the classifier w in the last fully connected layer given that the feature map F is invertible. Then we take the advantage of the fact that mutual information quantities are invariance under invertible mapping to attack our proposed information optimization problem for supervised classification in deep learning. Our theory justifies the use of direct regularization terms on w, F (X) for neural networks with invertibility property. Our regularization improves the performance of neural networks by a noticeable margin and is capable of encouraging the interpretability of the entries of features learned in the last layer.

A PROOF OF PROPOSITION 2.1

Proposition 2.1 establishes a connection between I(X, Y ) and the absolute value of the logits |w T F (X)| for the binary case. Intuitively, decreasing the confidence of the model on its predictions will decrease the mutual information I(X; Y ).

Proposition 2.1. I(X; Y ) = I(F (X); Y ) is well estimated by its empirical version (Montecarlo approximation) with high probability, which shares the same unique (global) minimum with

n k=1 |w T F (x k )| at w T F (x k ) = 0, for all k ∈ {1, ..., n}.

The mutual information I(X; Y ) is given as

EQUATION

Apply the assumption (II), the marginal distribution of Y is uniformly distributed:

EQUATION

Substituting ( 9) into (8) yields

EQUATION

According to the Hoeffding's inequality for bounded random variables [Proposition 2.2.6, Vershynin

], let M, m denote upper and lower bounds of the integrand of (10) correspondingly, we have

EQUATION

Equivalently, with probability at least 1 -δ,

EQUATION

Here n k=1 y∈C p( y|x k ) log(2p( y|x k ))/n is a Monte carlo estimation of RHS of I(X; Y ). Recall that, for the binary case p( y|x) = p( y|F (x)) can expressed as

EQUATION

Then we have

EQUATION

which is bounded by [0, log(2)].

Take M = log(2), m = 0, we have

EQUATION

hold with probability at least 1 -δ.

The conclusion follows from the fact that n k=1 y∈C p( y|x k ) log(2p( y|x k ))/n has a unique global minimum at w T F (x k ) = 0 for each x k .

B PROOF OF PROPOSITION 2.2:

Consider the training samples {(x k , y k )} k=1:n , each x k is fed into a deep probabilistic model which outputs probability densities and predicts y k , a realization of the prediction Y . Let C = {±1} be the binary class and n y be the counts of observed occurrences of k satisfying y k = y ∈ C, then n = y∈C n y = y n y , where we omit the range over C for convenience.

We denote the true joint probability with π y y = p(y, y), the marginal probabilities with π y+ = y π y y and π + y = y π y y . The mutual information I(Y ; Y ) can be expressed as

EQUATION

)

Our empirical mutual information I( π) is defined as

EQUATION

where π y y = 1 2ny ny i=1 σ( yw T F (x i )).

Proposition 2.2 establishes a connection between I(Y ; Y ) and the cross-entropy objective for the binary case. Intuitively, increasing the confidence of the model on its correct predictions will establish a more deterministic relationship between Y and Y and thus increase the mutual information I(Y ; Y ).

Proposition 2.2. I(Y ; Y ) is well estimated by I( π) with high probability, which shares the same unique (global) maximum with ) is an unbiased estimate of π y y . Here σ to represent the sigmoid function defined by σ(x) = e x / (e x + 1). By leveraging the concentration property of bounded variables, i.e., σ( yw T F (x i )), the estimation error can be bounded with high probability (Lemma B.2). Lemma B.1. The empirical joint probability, defined as

n k=1 log σ(y k w T F (x k )) at y k w T F (x k ) → ∞ given that y k w T F (x k ) > 1 2 ,

EQUATION

is an unbiased estimate of the true joint distribution π y y . Lemma B.2. With probability at least 1 -δ, we have

EQUATION

)

Proposition B.1. With probability at least 1 -δ,

EQUATION

)

Proof. To estimate the empirical mutual information given fixed samples, we use the approach by Hutter & Zaffalon (2005) . In particular, taylor expansion gives

EQUATION

where ∆ y y = π y y -π y y . Hence, Eq (21) together with Lemma B.2 yield, with probability exceeding 1 -|C| 2 δ, 

EQUATION

Given assumption (I) which states that the marginal density of Y is uniform over C, for every true label y ∈ C we have p(y) = 1 2 . We can therefore rewrite (23) as

EQUATION

According to assumption (II), p(x|y) is a probability density over space of signal x with true label y.

The Monte Carlo estimation of (24) gives the empirical joint probability which is unbiased:

EQUATION

Proof of Lemma B.2. Again, by leveraging the Hoeffding's inequality for bounded random variables [Proposition 2.2.6 of Vershynin ( 2018)], we have

EQUATION

)

where X y is the data random variable whose true label is y.

Equivalently, with probability at least 1 -δ,

EQUATION

where M, m are upper and lower bounds of random variable σ( yw T F (X y )), respectively. Substitute ( 24) and ( 25) into ( 27), with probability at least 1 -δ,

EQUATION

To estimate the upper and lower bounds M, m for σ( yw T F (X)), we use the Taylor's theorem:

EQUATION

and

EQUATION

given that the derivative of sigmoid function is bounded by 1 4 . It follows that

EQUATION

Also notice that M, m are the bounds for sigmoid function, so their difference is at most 1.

From derivations above, we can rewrite (28) as

EQUATION

and the lemma follows.

Proposition B.2. The empirical mutual information I( π) shares the same unique (global) maximum with n k=1 log σ(y k w T F (x k )) as y k w T F (x k ) → ∞ given that y k w T F (x k ) > 1 2 , for all k ∈ {1, ..., n}.

Proof. The empirical information I( π) is defined by

EQUATION

where the empirical joint probability is given by

EQUATION

It then follows that for any y ∈ C,

EQUATION

)

In binary case it means that

EQUATION

Proof. We will only prove the second inequality and the first inequality follows by a similar argument. Consider the decomposition I(Y ; X, Z) = 

Next we present a lower bound for the classification error. This lower bound is negatively related to the mutual information I(Y ; F (X)), and it attains its minimum if F is invertible. Although the performance also depends on the classifier w, Proposition C.1 implies that an invertible feature map F allows more chances for the classifier w to perform well.

Proof of Proposition C.1. Consider the random variable E defined as: 

EQUATION

E IMPLEMENTATION DETAILS:

We implement all models using Tensorflow. We modify the ResNet based on the code provided at https://github.com/tensorflow/models/tree/master/research/resnet. We follow the same learning rate scheme proposed in the original code. For the InvNet on MNIST, we train the network with initial learning rate 0.1, and decay it by 0.7 every 10 epochs. For both InvNet and ResNet, we apply the 1 norm regularization every 30 iterations. We observe that each digits have their specific entries with high value assigned to both weights and feature means. An additional observation is that typically the feature entry with low mean also has low standard deviation, such entry rarely contributes to the logits for classification if the corresponding value of weight is also small.

G RESULTS ON I-REVNET:

We reproduce the results in Section 4.2 on i-RevNet (Jacobsen et al. (2018) ). The statistics and ROC curves of the features produced by i-RevNet and Reg-i-RevNet on CIFAR10 is similar to those presented in Section 4.2. 

:

We use the open source code implemented by Xin Pan at https://github.com/tensorflow/ models/tree/master/research/resnet and build our objective function based upon it. Our results for the baseline models match the reported ones on the website.

Figures and Tables:

Figure 1: (L) An information Venn diagram: three disks represent the entropy of X, Y, Y respectively, the area in red is the relevant information I(Y ; Y ), the area in grey is the irrelevant information I(X; Y ) -I(Y ; Y ).The optimal solution is obtained when the smaller disks coincide, which is typically not achieved in practice. In particular, the trained model may be extremely confident in its prediction (when H( Y ) lies inside of H(X)), but predicts the wrong label (having large grey area). Our optimization problem explicitly prohibits the growth of grey area throughout the training. (R) Logic chart of our formulation: our proposed optimization problem only involves F (X) and w, allowing us to have control over the latent feature F (X) directly.

Proposition 2.1. I(X; Y ) = I(F (X); Y ) is well estimated by its empirical version n k=1 y∈C p( y|x k ) log(2p( y|x k ))/n with high probability, which shares the same unique (global) minimum with n k=1 |w T F (x k )| at w T F (x k ) = 0, for all k ∈ {1, ..., n}. Denote the sigmoid function with σ(a) = 1/(1 + e -a ). Proposition 2.2 establishes the relationship between maximization over mutual information I(Y ; Y ) and minimization over cross entropy n k=1 log σ(y k w T F (x k )); higher confidence of the model on its correct predictions over the samples indicates a larger value of I(Y ; Y ). The proof is provided in Appendix B. Proposition 2.2. I(Y ; Y ) is well estimated by its empirical version y y π y y log π y y πy+ π + y with high probability, which shares the same unique (global) maximum with

n

Notice that classical training methods maximize cross entropy and therefore I(Y ; Y ), but do not compress I(X; Y ) explicitly. In Equation (7), we explicitly encourage the compression of irrelevant information I(X; Y ) -I(Y ; Y ). As we demonstrate in Section 4.3, the proposed objective function does encourage a smaller amount of I(X; Y ) -I(Y ; Y ) throughout the training process.

) andKolchinsky et al. (2017) also studied the IB problem in a stochastic setting. Our idea of explicit regularization on w and F (X) is related to the margin based and stability based interpretations of generalization in deep learning respectively, studied byArora et al. (2018),Bartlett & Mendelson (2003), Neyshabur et al. (2017),Sun et al. (2015).

Figure2: Statistics plots for feature entries of digit 9. Recall that the features F (X) are vectors of dimension 100 and w is a matrix of size 100 × 10. The horizontal coordinate represents the index of entries of F (X) ranging from 0 to 99, the vertical coordinate represents entry values of the 10th column of w, the values of mean and standard deviation for each entry of features from samples of digit 9. For RegInvNet, w 10 only assigns large value to representative entries and is more robust against perturbations in other irrelevant entries.

Figure 3: The macro/micro average ROC curves for representative entries of features genearated by original and regularized model. The entries of features learned under our regularization strongly indicate the categories of the digits.

for all k ∈ {1, ..., n}. Proposition 2.2 follows from Proposition B.1 and Proposition B.2, where Proposition B.1 shows that I(Y ; Y ) is well approximated by I( π) with high probability and Proposition B.2 shows the remaining claims in Proposition 2.2. As shown in Lemma B.1, π y y = 1 2ny ny i=1 σ( yw T F (x i )

Notice that, in the binary case the cardinality |C| = 2. Next we prove the intermediate results, Lemmas B.1 and B.2. Proof of Lemma B.1. Direct derivation on the true joint distribution π y y gives π y y = p(y, y) = X p(y, y, x)dx = X p(y| y, x)p( y|x)p(x)dx = X p(y|x)p( y|x)p(x)dx = X p(x|y)σ( yw T F (x))p(y)dx .

; X, Z) = I(X; Y ) + I(Y ; Z|X) . (45) equation 44 together with equation 45 yields I(Y ; Z) + I(X; Y |Z) = I(X; Y ) + I(Y ; Z|X) . (46) According to the Markov Chain setting, Y and Z are conditionally independent given X, hence I(Y ; Z|X) = 0; in addition, the mutual information I(X; Y |Z) is It follows from (46) that I(Y ; Z) ≤ I(Y ; X) .

Figure 5: We measure the operator norm of L in each building block of ResNet-32 over 80k training steps. It can be observed that, the operator norms are all bounded by 1 throughout the training process, which verifies the hypothesis made in Appendix D. We conlude that ResNet is invertible.

Figure 6: A reproduction of the feature statistical results of InvNet on MNIST for all digits

Figure 7: Reproduction of Experimental Results in Section 4.2 on i-RevNet. The plots in the first row are for i-RevNet and the plots in the second row are for Reg-i-RevNet.

Shang et al. (2016) proposed an invertible activation scheme named CReLU to preserve information;Gilbert et al. (2017) analyzed theoretically the invertibility of CNN;Jacobsen et al. (2018) built a theoretic invertible structure whose performance is comparable to ResNetHe et al. (2015). Invertibility seems to be an intriguing property or design principle that often emerges in the recent state-of-the-art deep architectures. Performance comparison on CIFAR100, Best Accuracy (%, test set) and the average values of the constrain I(X; Y ) -I(Y ; Y ) throughout the training process are provided in the tables. The first table gives the results of naive regularization on w for ResNet-32, with α 1 = 0, α 2 = [0.0005, 0.001, 0.002, 0.003, 0.004], note that the choices of hyper-parameters are small because it regularize the whole matrix w but not to specific columns as proposed; the second table is for ResNet-32, comparing original ResNet and RegResNets with α 1 = 0, α 2 = [0.01, 0.03, 0.05]; the third table is for ResNet-28-10-Wide, comparing original ResNet and RegResNets with α 1 = 5e -6 , α 2 = [0.01, 0.05, 0.09, 0.15]. All results are calculated from 10 samples. The constrain decreases as α 2 increases; the performance can be effectively improved under proper choice of α 2 .

Table content: <table><tr><td>Information theoretic methods are reported to be effective attacking machine learning problems. In</td></tr><tr><td>deep learning, IB was first introduced in Tishby &amp; Zaslavsky (2015) and the follow-up experimen-</td></tr><tr><td>tal work Shwartz-Ziv &amp; Tishby (2017). They argued that DNN structure forms a markov chain</td></tr><tr><td>and information is compressed layer by layer. The theoretical breakthrough by Achille &amp; Soatto</td></tr></table>

The performance statistics for ResNet-32 and PlainNet-32 without or under various regularizations. For regularized models we hold α 1 = 1e -5 and take α 2 = [0.01, 0.05, 0.09, 0.15, 0.3].

Table content: <table><tr><td>Performance %</td><td>Original</td><td>α2 = 0.01</td><td>α2 = 0.05</td><td>α2 = 0.09</td><td>α2 = 0.15</td><td>α2 = 0.3</td></tr><tr><td>ResNet-32</td><td>92.49 ± 0.14</td><td>92.52 ± 0.30</td><td>92.76 ± 0.33</td><td>92.45 ± 0.37</td><td>88.36 ± 3.12</td><td>78.95 ± 4.33</td></tr><tr><td>PlainNet-32</td><td>90.06 ± 0.21</td><td>90.33 ± 0.24</td><td>90.25 ± 0.24</td><td>90.06 ± 0.31</td><td>85.97 ± 3.40</td><td>62.76 ± 16.22</td></tr></table>

Peer review (iclr2019):
 In this paper, the authors propose to train a model from the point of view of maximizing mutual information between the predictions and the true outputs, with a regularization term that minimizes irrelevant information while learning. They show that the objective can be minimized by looking to make the final layer vectors be as uncorrelated as possible to the final layer representations, and simplify the same by applying Holder’s inequality to make the optimization tractable. They also apply an L1 penalty on the final layer. Experiments on CIFAR and MNIST show that using their regularizer to train DNN models yield gains in performance. The presence of the L1 penalty also makes the results more interpretable (to the extend possible by looking at a subset of features in the last layer of a DNN). COMMENTS: - Meta Point: To really see that the regularization framework you’re proposing is good, why not just pick a simple, feedforward model or convNet, see the performance and then compare it with the regularizer you’re proposing? That will help hit the point home. - Page 1: before jumping to equations (1) and (2), please formally define Mutual Information. The actual definition is much later in the text, but it’s better to define first. - Beyond referring the user to section 3 on Page 1, please also mention a couple of key references in the appropriate locations. - Page 3 paragraph 2: “Mutual information is bounded … correct them” : Can you provide some formulas for this and make this concrete? Or perhaps provide some references? This line is vague. - prop 2.1 and 2.2: can you define what you mean by “empirical version”? Again, it’s probably good to have these terms crisply defined before using them. - eqn (6) is interesting. Holder’s inequality gives you the product terms. Then you can also apply the AM-GM inequality, and get a sum. So then at the end of it all, you’re left with the standard elastic net penalty and not the product form. In that case, aren’t we back to just the usual regularization strategy? And in which case, should I interpret the results you have in sec 4 as “using L1 penalties with L2 is good” ? - To the point above, I guess one difference after the AM-GM step is that you will not have a squared L2 norm, but just L2. This is reminiscent of linear models where they use L2 loss instead of squared L2 loss. But on the penalty, squaring just adds smoothness. Can you comment on this? - sec 4.2.1: I don’t see how fig (2) (L) is “roughly Gaussian”. Can you explain? Maybe plot the histogram? Also for fig (2, R): the coefficients are approximately sparse. It’s not sparse as you claim since there are almost no zeros in the coefficients. - I don’t get the point of sec 4.3: How does this claim not apply to all deep learning models, regardless of the penalizations you propose? edit: I have read the responses and the other reviews. The authors have addressed the few major points I had. I still think there are a few gaps that need to be addressed (as pointed by the other reviewers)