Noisy Information Bottlenecks for Generalization

1 INTRODUCTION . Bayesian inference is at the core of many directions of machine learning research in recent years . Applications range from a principled treatment of uncertainty in neural networks ( Gal , 2016 ) over latent variable models for high-dimensional data ( Kingma & Welling , 2013 ; Rezende & Mohamed , 2015 ) to reinterpretations of popular stochastic regularizers ( Kingma et al. , 2015 ; Gal , 2016 ) that inspired more flexible extensions ( Louizos & Welling , 2017 ; Molchanov et al. , 2017 ) . In all but the simplest models , exact inference is intractable , so approximations are necessary . Among those , variational inference ( Wainwright et al. , 2008 ) has been particularly popular , as it is amenable to gradient-based stochastic optimization ( Kingma & Welling , 2013 ; Titsias & Lázaro-Gredilla , 2014 ; Ranganath et al. , 2014 ) and thus highly scalable . Typically , researchers seek to develop more flexible approximate posterior distributions ( Rezende & Mohamed , 2015 ; Kingma et al. , 2016 ; Salimans et al. , 2015 ; Ranganath et al. , 2016 ; Huszár , 2017 ; Chen et al. , 2018 ; Vertes & Sahani , 2018 ; Burda et al. , 2015 ; Cremer et al. , 2017 ) in the hope of more faithfully representing the true posterior . However , recent works have suggested ( Trippe & Turner , 2018 ; Braithwaite & Kleijn , 2018 ; Shu et al. , 2018 ) that restricting the family of variational approximations can , in fact , have a positive regularizing effect , leading to better generalization . In this work , we seek to provide an information theoretic explanation for the observed behaviour of different families variational approximations . By reinterpreting Gaussian mean-field inference as maximum a-posteriori in a noisy model , we can quantify bounds on the mutual information between the data and the parameters variables . We explore the potential of this approach for regularization . Unlike methods relying on information-based objectives ( Tishby et al. , 2000 ; Shamir et al. , 2010 ; Agakov & Barber , 2004 ; Chen et al. , 2016 ; Phuong et al. , 2018 ) , it is compatible with standard variational inference . 2 NOISY INFORMATION BOTTLENECKS . This section characterizes model overfitting within information theory and points out a mitigation strategy that limits the amount of information extracted by the inferred model about the data . We will show this to be implicitly deployed by a common class of inference techniques in section 3 in order to explain observed regularizing effects . 2.1 AN INFORMATION-THEORETIC CHARACTERIZATION OF MODEL OVERFITTING . Intuitively , model overfitting can be understood as memorization or learning too much information about the training data . More formally , if D denote the random variable for all our data and θ all learned parameters and latent variables in a given probabilistic model , we can quantify the amount of information gained about the data D given a particular parameter value θ∗ in I ( D , θ = θ∗ ) : = H ( D ) −H ( D|θ = θ∗ ) ( 1 ) where H ( D|θ = θ∗ ) : = −Ep ( D|θ=θ∗ ) log p ( D|θ = θ∗ ) denotes the entropy of a variable D conditioned on a particular value θ∗ . In the context of variational inference , model overfitting can now be characterized by the symptom that the learned information about the data D Eθ∗∼q ( θ ) I ( D , θ = θ∗ ) ( 2 ) is too large in expectation under the approximately inferred posterior distribution q ( θ ) ≈ p ( θ|D ) . We now motivate this objective through the extreme case of maximum-likelihood learning in the limit of unrestrictedly expressive models for unsupervised tasks . Here , q is a point-mass on the parameter values that maximize the likelihood of the data . Under the assumption of indepenent and identically distributed ( iid ) datapoints , the model will then store all information that can possibly be extracted from the data , as shown in Appendix A . This is all information except the datapoint identity ( the information necessary to distinguish between distinct training samples , e. g. the index ) which can not be learned due to the iid assumption . This will result in perfect overfitting up to the sample identity , thereby impeding generalization . In unsupervised learning , information that can be learned by the model about the data will not be learned by the latent , a phenomenon known as the information preference problem ( Chen et al. , 2016 ; Alemi et al. , 2016 ; Zhao et al. , 2017 ; Phuong et al. , 2018 ) . Again , this can be characterized by the quantity from Equation 2 being too high . 2.2 WHY BOUNDING MUTUAL INFORMATION ? . This observation motivates placing a limit on the expected amount of extracted information given by Equation 2 . We observe that the expectation of I ( D , θ = θ∗ ) under the prior p ( θ ) is just the mutual information I ( D , θ ) as commonly defined in literature : Eθ∗∼p ( θ ) I ( D , θ = θ∗ ) = H ( D ) − Eθ∗∼p ( θ ) H ( D|θ = θ∗ ) = H ( D ) −H ( D|θ ) = I ( D , θ ) ( 3 ) This implies that if we could limit I ( D , θ ) to not be greater than some capacity C , we would obtain the following guarantee : If our dataset d was sampled from the marginal d ∼ p ( D ) =∫ dθp ( θ ) p ( D|θ ) , then a sample from the exact posterior θ∗ ∼ p ( θ|D = d ) will , on average , contain no more information about the data D than our capacity C due to Ed∼p ( D ) Eθ∗∼p ( θ|D=d ) I ( D , θ = θ∗ ) = Eθ∗∼p ( θ ) I ( D , θ = θ∗ ) = I ( D , θ ) ≤ C ( 4 ) This would limit the quantity given in Equation 2 in expectation under the assumptions that the model captures the nature of the generating process in p ( D ) and that our posterior estimate q ( θ ) is close to the true posterior p ( θ|D ) . These are common assumptions necessary to justify any variational Bayesian approach . We derive further motivation for our approach from the fact that limiting mutual information between data and learned parameters provably bounds generalization error ( Xu & Raginsky , 2017 ; Bassily et al. , 2018 ) . While the approach of limiting the amount of extracted information is already used for the purpose of generalization in adaptive analysis ( Feldman & Steinke , 2018 ; Smith , 2017 ; Russo & Zou , 2015 ) , in the following section we give an approach compatible with a variety of variational and deep learning algorithms . 2.3 NOISY INFORMATION BOTTLENECKS . How can we limit I ( D , θ ) ? Due to nonlinearities in typical deep models , it is hard to calculate and therefore limit the mutual information between data and parameters directly . Instead , one way to achieve this is to make the data dependent only on some noisy version θ̃ of the learned variables θ , resulting in a model p ( θ , θ̃ , D ) = p ( θ ) p ( θ̃|θ ) p ( D|θ̃ ) forming a Markov chain as shown in Figure 1a . The prior p ( θ ) and corruption process p ( θ̃|θ ) are chosen to result in a certain finite I ( θ̃ , θ ) . The data processing inequality ( Cover & Thomas , 2012 ) then ensures I ( D , θ ) ≤ I ( θ̃ , θ ) ( 5 ) for any model architecture p ( D|θ̃ ) . C = I ( θ̃ , θ ) now acts as a capacity , bounding the expected amount of information that is extracted about the data . In the next section , we will show that such a bound is implicit in an important class of variational inference techniques . As we will see , the corruption process will be realized through injected noise , and we therefore term this architecture noisy information bottleneck ( NIB ) . Quantifying the corresponding capacity C will help to explain observed regularizing effects in both supervised and unsupervised learning ( as shown in Figure 1b and 1c ) from an information-theoretic perspective . 3 REGULARIZATION THROUGH CONSTRAINED INFERENCE . Most variational approximation schemes such as variational autoencoders ( Kingma et al. , 2015 ; Rezende & Mohamed , 2015 ) and network weight uncertainty ( Blundell et al. , 2015 ) use simple Gaussian mean field inference for tractable training using the reparameterization trick . In this section , we quantify information-theoretic capacity constraints implicit in Gaussian mean field inference that helps to understand observed regularizing effects . We will first illustrate the bound for a fixed-scale distribution and then discuss the more general case where the variance is learned as well . 3.1 FIXED-SCALE GAUSSIAN MEAN FIELD INFERENCE . Let p ( θ , D ) denote a model with parameters θ and data D. The free energy objective now is F ( q ) = Eq ( θ ) log p ( D|θ ) −DKL ( q ( θ ) ||p ( θ ) ) ( 6 ) We consider the case of Gaussian mean field inference with fixed variance σ2 on parameters with component-wise independent priors p ( θ ) = N ( 0 , σ2pI ) . We can write the inferred q ( θ ) in its reparameterized form θ = µ∗ + σ , with µ∗ being the inferred mean and being noise sampled from N ( 0 , I ) , as shown in Figure 2a . We can then write the objective as F ( µ∗ ) = Ep ( ) log p ( D|θ = µ∗ + σ ) − ∑ i µ∗2i 2σ2p + const . ( 7 ) which is optimized with respect to µ∗ . We use i ∈ { 1 , . . . , K } to denote the model parameter index . To show how Gaussian mean field contains an implicit information bottleneck we include the inferred means µ into the generative model , without changing the resulting objective . We therefore define a model p′ ( µ , µ̃ , D ) = p′ ( µ ) p′ ( µ̃|µ ) p′ ( D|µ̃ ) ( Figure 2b ) , where µ ∼ N ( 0 , σ2pI ) represents the parameters of our new model and µ̃ = µ + σ its noise-injected version with ∼ N ( 0 , I ) . We leave the rest of the model p′ ( D|µ̃ ) = p ( D|θ ) unchanged . This model is an instantiation of a noisy information bottleneck as in Figure 1a , where µ and µ̃ take the roles of θ and θ̃ . Inference model q Generative model p Inference model q′ Generative model p′ On this new model , we do variational inference by sampling noise from the prior q′ ( ) = p′ ( ) , while learning a deterministic approximate posterior q′ ( µ ) with point-mass at a particular µ∗ , resulting in a MAP-like objective1 F ′ ( q′ ) = Eq′ ( µ ) q′ ( ) log p′ ( D|µ , ) + ∑ i log p′ ( µ∗i ) ( 8 ) Using above definitions , and Equation 7 , this implies equivalence F ( µ∗ ) = F ′ ( µ∗ ) of both objectives , resulting in exactly equivalent training algorithms . For the new model , we get H ( µ̃ ) = K2 log 2πe ( σ2 + σ2p ) and H ( µ̃|µ = µ∗ ) = K2 log 2πeσ 2 for any µ∗ , inducing a noisy information bottleneck of capacity I ( µ̃ , µ ) = H ( µ̃ ) − Eµ∗∼p ( µ ) H ( µ̃|µ = µ∗ ) = K 2 log ( 1 + σ2p σ2 ) ( 9 ) in the sense of Equation 5 , whereK denotes the number of parameters . This quantity is known as the capacity of channels with Gaussian noise in signal processing ( Cover & Thomas , 2012 ) . Intuitively , a high prior variance σ2p corresponds to a large capacity , while a high noise variance σ 2 reduces it . Simply adjusting the signal-to-noise ratio σ2p σ2 allows to create an information bottleneck of any desired capacity . This suggests the usefulness of applying fixed-scale Gaussian mean field inference to model parameters for regularization . There are relations to existing approaches : When applied to training neural networks , MAP training on this model can be interpreted as applying weight decay combined with a additive noiseN ( 0 , σ2 ) on all network weights . Molchanov et al . ( 2017 ) shows that this results in multiplicative noise on the unit activations . Wang & Manning ( 2013 ) reports that empirical results do not change significantly when dependencies between the different elements of the layer output are ignored , which is then equivalent to scaled Gaussian dropout ( Kingma et al. , 2015 ) .

Summary: This paper studies "Noisy Information Bottlenecks". The overall idea is that, if the mutual information between learned parameters and the data is limited, then this prevents overfitting. It proposes to create a "bottleneck" to limit the mutual information. Specifically, the bottleneck is created by having the data depend on a noisy version of the parameters, rather than the true parameters and invoking the information processing inequality. The paper gives an example of Gaussian mean field inference. Ultimately, the analysis boils down to looking at a signal-to-noise ratio of the algorithm, which looks very much like regularization.