Information Regularized Neural Networks

1 INTRODUCTION . Quantities of information are nonlinear measures capable of describing complex relationship between unstructured data and they form the basis of the probabilistic algorithms in the literature of machine learning . Information theoretic methods are also reported to be effective on improving deep generative models ( Chen et al . ( 2016 ) ; Kim & Mnih ( 2018 ) ) and deep learning models for classification ( Grandvalet & Bengio ( 2004 ) ; Pereyra et al . ( 2017 ) ) . Information Bottleneck ( IB ) problem ( Tishby et al. , 1999 ) is formulated as : minimize I ( X ; T ) − λI ( Y ; T ) , ( 1 ) where the solution random variable T is interpreted as a minimal sufficient representation of signal X for label Y and the mutual information is defined as I ( X ; Y ) = ∫ X ∫ Y p ( x , y ) log ( p ( x , y ) p ( x ) p ( y ) ) dydx . ( 2 ) The term I ( X ; T ) has its origins in Lossy Compression and Rate-Distortion Theory ( Cover & Thomas , 2006 ) , conveying an simple idea of ” keep only what is relevant ” . However , Saxe et al . ( 2018 ) argued that the mutual information I ( X ; T ) between signal X and feature T in intermediate layer is infinite , as the transformation from X to continuous random variable T is deterministic . In addition they showed experimentally that layers equipped with ReLU actually do not compress too much information , which is supported by many recent work on the invertibility of the neural network ( Dosovitskiy & Brox ( 2015 ) ; Jacobsen et al . ( 2018 ) ) . This motivates us to consider a different problem with similar principle idea : we would like to establish a theoretically valid objective that allows the neural network to extract only the relevant information for classification from the data . We focus on the discrete prediction random variable Ŷ inferred by the probabilistic model P ( Ŷ |X ) and introduce the following information optimization problem for supervised classification : maximize I ( Y ; Ŷ ) subject to I ( X ; Ŷ ) − I ( Y ; Ŷ ) < τ , for some τ > 0 . ( 3 ) The intuition behind this objective lies in twofold : Information perspective : A good classification model should be robust against irrelevant features of X , and prevent over-fitting in the learning process . In optimization problem ( 3 ) we maximize the relevant information I ( Y ; Ŷ ) , while constraining the irrelevant information I ( X ; Ŷ ) − I ( Y ; Ŷ ) that X has about Ŷ . Although I ( X ; Ŷ ) − I ( Y ; Ŷ ) converges to zero as I ( Y ; Ŷ ) approaching its maximum ( see Figure 1L ) , in practice it ’ s never attained due to the limited capacity of the models or over-fitting . Our proposed constrain addresses the problem of over-fitting : if two models achieve the similar classification accuracy , this constraint prefers the one that does not overfit to spurious factors of variation in X ( e.g. , pixel-level artifact/noise in the image that accidentally correlates to the labels in the training data ) . Prediction confidence perspective : A good classification model should not be certain about its decision which is in fact wrong . However , modern neural networks are too confident in their predictions ( Guo et al. , 2017 ; Szegedy et al. , 2015 ; Pereyra et al. , 2017 ) . To be more precise , high capacity neural networks mostly assign labels of data with prediction confidence near 0 or 1 . In particular , they assign 0 probability to some correct labels and therefore do not have enough flexibility to correct themselves from making the wrong prediction . We propose to compress the irrelevant information I ( X ; Ŷ ) − I ( Y ; Ŷ ) , where minimizing I ( X ; Ŷ ) decreases the confidence on all predictions but maximizing I ( Y ; Ŷ ) increases the confidence on the correct predictions . Therefore the overall effect reduces the certainty on the false prediction of Ŷ ( see Figure 1L ) . To solve this optimization problem , we first present some insights on the dynamics of deep neural network , which can be decomposed into two stages : ( i ) Transformation stage : samples { Xk } k=1 : n of the high dimensional unstructured signal X are transformed under the deep invertible ( information preserving ) feature map F to become ( almost ) linearly separable ; ( ii ) Classification stage : the weight matrixw in the last fully connected layer together with the Softmax function , takes structured features { F ( Xk ) } k=1 : n as input and gives predictions . Invertibility of F allows neural networks to pass the control of I ( X ; Ŷ ) = I ( F ( X ) ; Ŷ ) towards F ( X ) and w in the last layer , where F ( X ) can be interpreted as transformed signal that preserves information about the original signal X and the inference model becomes conceptually linear with classifier w ( see Figure 1R ) . In Section 2 we derive objective function ( 7 ) and prove that it solves the optimization problem ( 3 ) . We show the classification performance is improved in Section 4.1 and the features F ( X ) are sculpted into a form with more interpretability entry-wise in Section 4.2 . The invertibility property has been empirically demonstrated for complex non-linear deep neural networks that are widely used in practice . We will discuss the literature in Section 3 . In addition , we prove in Proposition C.1 that a lower bound of classification error is minimized if neural network is invertible . Our contribution : Our contribution lies in the following : ( i ) we formulated a novel information optimization problem for supervised classification ; ( ii ) we propose a simple objective function that improves supervised deep learning with better performance and interpretability ; ( iii ) we formally justify the use of ` 1 , ` 2 regularization from an information perspective . Different from the naive regularization on w , our regularization on wTF ( X ) is novel and effective . 2 MAIN RESULTS . Consider a classification problem where the training data D = { ( xk , yk ) } k=1 : n are sampled from random variable pair ( X , Y ) with unknown joint distribution . Each xk is fed into a deep probabilistic model , which outputs probability densities and predicts ŷk , a realization of the prediction random variable Ŷ . Let C denote the label class and X denote the signal space , then the mutual information between random variables , e.g. , continuous X and discrete Ŷ , is defined as I ( X ; Ŷ ) = ∑ ŷ∈C ∫ X p ( x , ŷ ) log ( p ( x , ŷ ) p ( x ) p ( ŷ ) ) dx , and the entropy of Y is defined as H ( Y ) = − ∑ y∈C p ( y ) log p ( y ) . We first call out our assumptions used throughout our analysis . ( I ) : we assume the marginal densities of Y , Ŷ are uniform over C ; ( II ) : there exists a unique true label for every sample of X . Mutual information is bounded and its gradient with respect to logits is approximately zero over a large domain . In particular if the logits are initially small for true labels , gradient updates can not effectively correct them . Therefore it ’ s not practical to train mutual information terms directly . In this section we introduce alternative terms and prove that they are feasible for our purpose . We show in Proposition 2.2 that I ( Y ; Ŷ ) is maximized if the classical cross entropy objective is minimized . On the other hand , we show in Proposition 2.1 that for invertible F , I ( X ; Ŷ ) = I ( F ( X ) ; Ŷ ) is minimized if ‖wTF ( X ) ‖ is minimized . We derive our objective function ( 7 ) in Section 2.2 . Our experimental result in Section 4.1 verifies that the proposed objective function does compress the irrelevant information I ( X ; Ŷ ) − I ( Y ; Ŷ ) . 2.1 SOLVING OPTIMIZATION PROBLEM WITH FEASIBLE TERMS . Without loss of generality , we consider the binary classification problem , i.e . the label class C = { ±1 } . To tract the population quantities I ( F ( X ) ; Y ) and I ( Y ; Ŷ ) , we decompose each of them into an empirical part and a probabilistic bound , which is negligible if sample size n is large . In Proposition 2.1 , we show that in order to compress I ( F ( X ) ; Y ) , we need to compress the norm of classifier w and feature F ( X ) . In particular , smaller |wTF ( X ) | represents lower confidence of the model on its predictions Ŷ , indicating a smaller amount of mutual information I ( F ( X ) ; Ŷ ) . The proof is provided in Appendix A . Proposition 2.1 . I ( X ; Ŷ ) = I ( F ( X ) ; Y ) is well estimated by its empirical version∑n k=1 ∑ ŷ∈C p ( ŷ|xk ) log ( 2p ( ŷ|xk ) ) /n with high probability , which shares the same unique ( global ) minimum with ∑n k=1 |wTF ( xk ) | at wTF ( xk ) = 0 , for all k ∈ { 1 , ... , n } . Denote the sigmoid function with σ ( a ) = 1/ ( 1 + e−a ) . Proposition 2.2 establishes the relationship between maximization over mutual information I ( Y ; Ŷ ) and minimization over cross entropy − ∑n k=1 log σ ( ykw TF ( xk ) ) ; higher confidence of the model on its correct predictions over the samples indicates a larger value of I ( Y ; Ŷ ) . The proof is provided in Appendix B . Proposition 2.2 . I ( Y ; Ŷ ) is well estimated by its empirical version ∑ yŷ π̂yŷ log ( π̂yŷ π̂y+π̂+ŷ ) with high probability , which shares the same unique ( global ) maximum with ∑n k=1 log σ ( ykw TF ( xk ) ) given that ykwTF ( xk ) > 12 , for all k ∈ { 1 , ... , n } . Here π̂yŷ = 1 2ny ∑ny i=1 σ ( ŷw TF ( xi ) ) is an unbiased estimate of p ( y , ŷ ) and π̂y+ = ∑ ŷ∈C π̂yŷ , π̂+ŷ = ∑ y∈C π̂yŷ . 2.2 DERIVATION OF OBJECTIVE FUNCTION . In Lagrangian form of optimization problem ( 3 ) , the constant τ can be dropped and the objective becomes maximize ( 1 + λ ) I ( Y ; Ŷ ) − λI ( F ( X ) ; Ŷ ) ⇐⇒ maximize I ( Y ; Ŷ ) − λ 1 + λ I ( F ( X ) ; Ŷ ) . ( 4 ) Consider a single signal xk and its true label yk , we propose the following objective function for binary supervised classification problem : Lk = αR ( |wTF ( xk ) | ) − log σ ( ykwTF ( xk ) ) , ( 5 ) where R is some regularizer . According to results in Section 2.1 , minimizing ( 5 ) allows us to maximize I ( Y ; Ŷ ) while constraining I ( X ; Ŷ ) . We typically choose the hyper-parameter α > 0 to be a reasonably small number . The intuition comes from the observation that λ/ ( 1 + λ ) is upper bounded by one . If we compress I ( X ; Ŷ ) harshly , then neural networks may choose to minimize I ( F ( X ) ; Ŷ ) at a cost of minimizing I ( Y ; Ŷ ) . Recall from the information theoretic perspective of our proposed optimization problem , our regularizer should prefer a model that does not overfit among all the ones with high training accuracy . In this case neural networks assign only large logits wTykF ( xk ) to true label yk for each signal xk , and generalization of ( 5 ) to multi-class case for I ( F ( X ) ; Ŷ ) can be simplified to constraining wTykF ( xk ) − w T j F ( xk ) , where wj is the jth column of w , assigning feature F ( xk ) a probability to label j . We propose to simply constrain wTykF ( xk ) and does not encourage increasing w T j F ( xk ) : Lk = αR ( |wTykF ( xk ) | ) − log ( ew T yk F ( xk ) ew T yk F ( xk ) + ∑ j 6=yk e wTj F ( xk ) ) . ( 6 ) In our experiment , we take the Elastic Net approach by Zou & Hastie ( 2005 ) using a combination of ` 2 and ` 1 regularizers : we use Holder ’ s inequality to bound |wTykF ( xk ) | with both supF ( X ) · ‖wyk‖1 and ‖wyk‖2 · ‖F ( xk ) ‖2 . In practice we assume supF ( X ) to be a constant and is absorbed into the hyper-parameter . Our proposed objective function is of the form : L = α1||w||1 + 1 n n∑ k=1 { α2‖wyk‖2 · ‖F ( xk ) ‖2 − log ( ew T yk F ( xk ) ew T yk F ( xk ) + ∑ j 6=yk e wTj F ( xk ) ) } . ( 7 ) Notice that classical training methods maximize cross entropy and therefore I ( Y ; Ŷ ) , but do not compress I ( X ; Ŷ ) explicitly . In Equation ( 7 ) , we explicitly encourage the compression of irrelevant information I ( X ; Ŷ ) − I ( Y ; Ŷ ) . As we demonstrate in Section 4.3 , the proposed objective function does encourage a smaller amount of I ( X ; Ŷ ) − I ( Y ; Ŷ ) throughout the training process .

Summary: In this paper, the authors propose to train a model from the point of view of maximizing mutual information between the predictions and the true outputs, with a regularization term that minimizes irrelevant information while learning. They show that the objective can be minimized by looking to make the final layer vectors be as uncorrelated as possible to the final layer representations, and simplify the same by applying Holder’s inequality to make the optimization tractable. They also apply an L1 penalty on the final layer. Experiments on CIFAR and MNIST show that using their regularizer to train DNN models yield  gains in performance. The presence of the L1 penalty also makes the results more interpretable (to the extend possible by looking at a subset of features in the last layer of a DNN).