Adaptive Estimators Show Information Compression in Deep Neural Networks

1 INTRODUCTION . Although deep learning ( reviewed by Schmidhuber ( 2015 ) ) has produced astonishing advances in machine learning ( Silver et al. , 2017 ) , a rigorous statistical explanation for the outstanding performance of deep neural networks ( DNNs ) is still to be found . According to the information bottleneck ( IB ) theory of deep learning ( Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ) the ability of DNNs to generalize can be seen as a type of representation compression . The theory proposes that DNNs use compression to eliminate noisy and task-irrelevant information from the input , while retaining information about the relevant segments ( Chechik & Tishby , 2003 ) . The information bottleneck method ( Tishby et al. , 2000 ) quantifies the relevance of information by considering an intermediate representation T between the original signal X and the salient data Y . T is the most relevant representation of X , and is said to be an information bottleneck , when it maximally compresses the input , retaining only the most relevant information , while maximizing the information it shares with the target variable Y . Formally , the information bottleneck minimizes the Lagrangian : L [ p ( t̃|x ) ] = I ( T , X ) − βI ( T , Y ) ( 1 ) where I ( · ) is mutual information . In this Lagrangian β is the Lagrange multiplier , determining the trade-off between compression and retention of information about the target . In the context of deep learning , T is a layer ’ s hidden activity represented as a single variable , X is a data set and Y is the set ∗With gratitude to Tilo Burghardt , Rui Ponte Costa , Carl Henrik Ek , Galina Malisheva and Daniil Malkin for their valuable contributions to this research . of labels . Compression for a given layer is signified by a decrease in I ( T , X ) value , while I ( T , Y ) is increasing during training . Fitting behaviour refers to both values increasing . Shwartz-Ziv & Tishby ( 2017 ) visualized the dynamic of training a neural network by plotting the values of I ( T , X ) and I ( T , Y ) against each other . This mapping was named the information plane . According to IB theory the learning trajectory should move the layer values to the top left of this plane . In fact what was observed was that a network with tanh activation function had two distinct phases : fitting and compression . The paper and the associated talks1 show that the compression phase leads to layers stabilizing on the IB bound . When this study was replicated by Saxe et al . ( 2018 ) with networks using ReLU ( Nair & Hinton , 2010 ) activation function instead of tanh , the compression phase did not happen , and the information planes only showed fitting throughout the whole training process . This behaviour required more detailed study , as a constant increase in mutual information between the network and its input implies increasing memorization , an undesired trait that is linked to overfitting and poor generalization ( Morcos et al. , 2018 ) . Measuring differential mutual information in DNNs is an ill-defined task , as the training process is deterministic ( Saxe et al. , 2018 ) . Mutual information of hidden activity T with input X is : I ( T , X ) = H ( T ) −H ( T |X ) ( 2 ) If we consider the hidden activity variable T to be deterministic then entropy is : H ( T ) = − N∑ i=1 pi ( t ) log pi ( t ) ( 3 ) However , if T is continuous then the entropy formula is : H ( T ) = − ∫ pt ( t ) log pt ( t ) dt ( 4 ) In the case of deterministic DNNs , hidden activity T is a continuous variable and p ( T |X ) is distributed as the delta function . For the delta function : H ( T |X ) = − ∫ pt ( t ) log pt ( t ) dt = −∞ ( 5 ) Thus , the true mutual information value I ( T , X ) is in fact infinite . However , to observe the dynamics of training in terms of mutual information , finite values are needed . The simplest way to avoid trivial infinite mutual information values , is to add noise to hidden activity . Two ways of adding noise have been explored previously by Shwartz-Ziv & Tishby ( 2017 ) and Saxe et al . ( 2018 ) . One way is to add noise Z directly to T and get a noisy variable T̂ = T + Z . Then H ( T |X ) = H ( Z ) and mutual information is I ( T̂ , X ) = H ( T̂ ) +H ( Z ) . When the additive noise is Gaussian , the mutual information can be approximated using kernel density estimation ( KDE ) , with an assumption that the noisy variable is distributed as a Gaussian mixture ( Kolchinsky & Tracey , 2017 ) . The second way to add noise is to discretize the continuous variables into bins . To estimate mutual information , Shwartz-Ziv & Tishby ( 2017 ) and Saxe et al . ( 2018 ) primarily relied on binning hidden activity . The noise comes embedded with the discretization that approximates the probability density function of a random variable . In context of neural networks , adding noise can be done by binning hidden activity and approximating H ( T ) as a discrete variable . In this case H ( T |X ) = 0 since the mapping is deterministic and I ( T , X ) = H ( T ) . Generally , when considering mutual information in DNNs , the analyzed values are technically the result of the estimation process and , therefore , are highly sensitive to it . For this reason it is vital to maintain consistency when estimating mutual information . The problem is not as acute when working with DNNs implemented with saturating activation functions , since all hidden activity is bounded . However , with non-saturating functions , and the resulting unbounded hidden activity , the level of noise brought by the estimation procedure has to be proportional and consistent , adapting to the state of every layer of the network at a particular epoch . In the next section adaptive estimation schemes are presented , both for the binning and KDE estimators . It is shown that for networks with unbounded activation functions in their hidden layers , 1Available at : https : //youtu.be/FSfN2K3tnJU and https : //youtu.be/bLqJHjXihK8 the estimates of information change drastically . Moreover , the adaptive estimators are better able to evaluate different activation functions in a way that allows them to be compared . This approach shows considerable variation in compression for different activation functions . It also shows that L2 regularization leads to more compression and clusters all layers to the same value of mutual information . When compression in hidden layers is quantified with a compression metric and compared with generalization , no significant correlation is observed . However , compression of the last softmax layer is correlated with generalization . 1.1 METHODS . In this paper we used a fully connected network consisting of 5 hidden layers with 10-7-5-4-3 units , similar to Shwartz-Ziv & Tishby ( 2017 ) and some of the networks described in Saxe et al . ( 2018 ) . ADAM ( Kingma & Ba , 2014 ) optimizer was used with cross-entropy error function . We trained the network with a binary classification task produced by Shwartz-Ziv & Tishby ( 2017 ) for consistency with previous papers . Inputs were 12-bit binary vectors mapped deterministically to the 1-bit binary output , with the categories equally balanced ( see Shwartz-Ziv & Tishby ( 2017 ) for details ) . Weight initialization was done using random truncated Gaussian initialization from Glorot & Bengio ( 2010 ) , with 50 instances of this initialization procedure for every network configuration used . Based on the observed data , even 10 initializations provide a clear picture of the average network behaviour , but for the purpose of consistency with previous studies , we used 50 initializations . 80 % of the dataset was used for training , using batches of 512 samples . During the training process , hidden activity for different epochs was saved . The calculation of mutual information was done using the saved hidden activity , after training has been completed ; the two processes did not interfere with one another . 2 MUTUAL INFORMATION ESTIMATION . The observed dynamics of a neural network ’ s training on the information plane depends strongly on the estimation procedure . Estimating mutual information at different epochs and layers presents a problem as these variables are inherently different from one another . Using more adaptive frameworks makes it possible to choose estimator parameters that would produce comparable estimate values . 2.1 ENTROPY-BASED BINNING . Binning networks with saturating activation functions is straightforward , since the saturation points define the range of the binning . With non-saturating activation functions a range of activation values must be specified . In Saxe et al . ( 2018 ) for networks with ReLU units in their hidden layer , all hidden activity was binned using a single range , going from zero to m , where m is the maximum value any ReLU unit has achieved throughout the training . The argument behind this choice of binning range is that the whole space explored by the ReLU function must be included in the binning process . This approach is inherently limited , since a network at every epoch is different . If the binning range is determined by picking a maximum value across all epochs and all layers , then the information plane values estimated for a given epoch and a given layer depend on factors that are unrelated to their own properties . In fact , the behaviour of the activation values is markedly different in different layers ; the maximum values can differ by as much as two orders of magnitude in different layers and epochs ( Figure 1 ) . Using a single maximum value when binning the data usually means that some layers during some epochs will have all of their values binned together in the lowest bin . To avoid this we use an entropy-based binning strategy ( Kohavi & Sahami , 1996 ) to choose bin boundaries : the bin boundaries are chosen so that each bin contains the same number of unique observed activation levels in a given layer and epoch . When there are repeated activation levels , as can happen for any activation function that is not unbounded , these are ignored when calculating the boundaries . This often occurs near the saturation regions where the activation levels are indistinguishable even as float64 values . However , we calculate the entropy for the whole layer rather than for individual units ; in other words , the vector of activation levels of the units in the layer is converted to a vector of bin indices and the entropy is calculated for these vectors . This binning procedure will be referred to as entropy-based adaptive binning ( EBAB ) . In this approach there is no need to define a range as part of the binning procedure . It has other advantages . Firstly , the definition of the bins depends on the distribution of the observed activation values , not on the shape of the activation functions . This means this binning strategy applies in the same way to all activation functions , allowing different networks with different activation functions to be compared . Secondly , a fixed width binning approach can be insensitive to the behaviour of the layer because much of the activity occurs in a single bin . An example of binning where a very small subset of hidden activity achieves high values is shown in Figure 2 , as such , most of the hidden activity falls into the first bin . With entropy-based binning , bin edges are more densely distributed closer to the lower bound of binning range , providing a better spread . The difference in mutual information estimates produced with an adaptive binning procedure is significant when visualized on the information plane . Non-adaptive binning consistently underestimates compression . Information planes of the same network in Figure 3 using ReLU analyzed with different estimators show how non-adaptive binning consistently underestimates compression .

Summary: This paper proposes a method for the estimation of mutual information for networks with unbounded activation functions and the use of L2 regularization to induce more compression.  The use of information planes to study the training behavior of networks is not new.  This paper addresses the issue of unbounded  hidden state activities.  As the differential mutual information in DNN is ill-defined, the authors proposed to add noise to the hidden activity by using the binning process.   It is not clear in the paper that if the binning is applied just for visualizing the information plane or for computing the activities of hidden units in upper layers.   If it is the latter one, it creates unnecessary distortions to the DNN.  As the authors pointed out, different initializations can lead to different behaviour on the information plane.  It would be difficult to draw conclusions based on the experimental results, even they come from the average of 50 individual networks.  Also, the experiences are performed using a particular task, it is not sure if similar behavior is observed in other tasks.   It is, however, more important to understand what makes the compression.   For the L2 regularization, the compression is expected as the regularization tends to limit the values of the  weights.