Information Theoretic lower bounds on negative log likelihood

1 INTRODUCTION . A statistician plans to use a latent variable model p ( x ) = ∫ p ( z ) ` ( x|z ) dz , ( 1 ) where p ( z ) is known as the prior over the latent variables , and ` ( x|z ) is the likelihood of the data conditional on the latent variables ; we will often use ( p ( z ) , ` ( x|z ) ) as a shorthand for the model . Frequently , both the prior and the likelihood are parametrized and the statistician ’ s job is to find reasonable parametric families for both - an optimization algorithm then chooses the parameter within those families . The task of designing these parametric families can sometimes be time consuming - this is one of the key modeling tasks when one adopts the representation learning viewpoint in machine learning . In this article we ask the question of how much p ( z ) can be improved if one fixes ` ( x|z ) and viceversa , with the goal of equipping the statistician with tools to make decisions on where to invest her time . One way to answer whether p ( z ) can be improved for fixed ` ( x|z ) is to drop the assumption that p ( z ) must belong to a particular family and ask how a model could improve in an unrestricted setting . Mathematically , given data { x1 , · · · , xN } the first problem we study is the following optimization problem : for a fixed ` ( x|z ) , min p ( z ) − 1 N N∑ i=1 log ∫ p ( z ) ` ( xi|z ) dz ( 2 ) which as we will show , is also indirectly connected to the problem of determining if ` ( x|z ) can be improved for a given fixed p ( z ) . The quantity being optimized in ( 2 ) is called the average negative log likelihood of the data , and is used whenever one assumes that the data { x1 , · · · , xN } have been drawn statistically independently at random . Note that in this paper we are overloading the meaning of p ( z ) : in ( 1 ) it refers to prior in the “ current ” latent variable model in the statistician ’ s hands , and in ( 2 ) and other similar equations , it refers to a prior that can be optimized . We believe that the meaning will be clear depending on the context . Obviously , for any given ` ( x|z ) , p ( z ) , from the definition ( 1 ) we have the trivial upper bound min p ( z ) − 1 N N∑ i=1 log ∫ p ( z ) ` ( xi|z ) dz ≤ − 1 N N∑ i=1 log p ( xi ) ( 3 ) Can we give a good lower bound ? A lower bound could tell us how far we can improve the model by changing the prior . The answer turns out to be in the affirmative . In an important paper , Lindsay ( 1983 ) proved several facts about the problem of optimizing priors in latent variable models . In particular , he showed that min p ( z ) − 1 N N∑ i=1 log ∫ p ( z ) ` ( xi|z ) dz ≥ − 1 N N∑ i=1 log p ( xi ) − sup z log ( 1 N N∑ i=1 ` ( xi|z ) p ( xi ) ) . ( 4 ) This result is very general - it holds for both discrete and continuous latent variable spaces , scalar or vector . It is also sharp - if you plug in the right prior , the upper and lower bounds match . It also has the advantage that the lower bound is written as a function of the trivial upper bound ( 3 ) - if someone proposes a latent variable model p ( x ) which uses a likelihood function ` ( x|z ) , the optimal negative log likelihood value when we optimize the prior is thus known to be within a gap of sup z log ( 1 N N∑ i=1 ` ( xi|z ) p ( xi ) ) ( 5 ) bits . The individual quantities under the sup c ( z ) ∆ = 1 N N∑ i=1 ` ( xi|z ) p ( xi ) ( 6 ) have a specific functional meaning : they can be regarded as multiplicative factors that tell you how to modify the prior to improve the log likelihood ( see the Blahut-Arimoto algorithm ( Blahut , 1972 ) ) : p ( z ) ← p ( z ) c ( z ) . ( 7 ) Lindsay ( 1983 ) derived his results with no apparent reference to earlier published work on ratedistortion theory , which is how information theorists study the problem of lossy compression Shannon ( 1959 ) . There is no reason for why this connection could be reasonably made , as it is not immediately obvious that the problems are connected ; However , the quantity c ( z ) and the lower bound ( 4 ) in fact can be derived from ideas in Berger ’ s book on rate-distortion theory ( Berger , 1971 ) ; in fact Lindsey ’ s classical result that the optimal prior in a latent variable model has finite support with size equal to the size of the training data can be , drawing the appropriate connection , seen as a result of a similar result in rate-distortion theory also contained in Berger ( 1971 ) . With time , the fundamental connection between the log likelihood optimization in latent variable modeling and the computation of rate-distortion functions became more clear . Although not explicitly mentioned in these terms , Rose ( 1998 ) , Neal & Hinton ( 1998 ) restate the optimal log likelihood as a problem of minimizing the variational free energy of a certain statistical system ; this expression is identical to the one that is optimized in rate-distortion theory . The Information Bottleneck method of Tishby et al . ( 1999 ) is a highly successful idea that exists in this boundary , having created a subfield of research that remains relevant nowadays ( Tishby & Zaslavsky , 2015 ) ( Shwartz-Ziv & Tishby , 2017 ) . Slonim & Weiss ( 2002 ) showed a fundamental connection between maximum likelihood and the information bottleneck method : “ every fixed point of the IB-functional defines a fixed point of the ( log ) likelihood and vice versa ” . Banerjee et al . ( 2004 ) defined a rate-distortion problem where the output alphabet Z is finite and is jointly optimized with the test channel . By specializing to the case of where the distortion measure is a Bregman divergence , they showed the mathematical equivalence between this problem and that of maximum likelihood estimation where the likelihood function is an exponential family distribution derived from the Bregman divergence . Watanabe & Ikeda ( 2015 ) study a variation of the maximum likelihood estimation for latent variable models where the maximum likelihood criterion is instead replaced with the entropic risk measure . The autoencoder concept extensively used in the neural network community is arguably directly motivated by the encoder/decoder concepts in lossy/lossless compression . Giraldo & Prı́ncipe ( 2013 ) proposed using a matrix based expression motivated by rate-distortion ideas in order to train autoencoders while avoiding estimating mutual information directly . Recently , Alemi et al . ( 2018 ) exploited the β-VAE loss function of Higgins et al . ( 2017 ) to explicitly introduce a trade-off between rate and distortion in the latent variable modeling problem , where the notions of rate and distortion have similarities to those used in our article . Latent Variable Modeling is undergoing an exciting moment as it is a form of representation learning , the latter having shown to be an important tool in reaching remarkable performance in difficult machine learning problems while simultaneously avoiding feature engineering . This prompted us to look deeper into rate-distortion theory as a tool for developing a theory of representation learning . What excites us is the possibility of using a large repertoire of tools , algorithms and theoretical results in lossy compression in meaningful ways in latent variable modeling ; notably , we believe that beyond what we will state in this article , Shannon ’ s famous lossy source coding theorem , the information theory of multiple senders and receivers , and the rich Shannon-style equalities and inequalities involving entropy and mutual information are all of fundamental relevance to the classical problem ( 1 ) and more complex variations of it . The goal of this article is laying down a firm foundation that we can use to build towards the program above . We start by proving the fundamental equivalence between these two fields by using simple convexity arguments , avoiding the variational calculus arguments that had been used before . We then take an alternate path to proving ( 2 ) also involving simple convexity arguments inspired by earlier results in rate-distortion theory . We then focus on what is a common problem - instead of improving a prior for a fixed likelihood function , improve the likelihood function for a fixed prior but for a fixed prior . Interestingly , ratedistortion theory still is relevant to this problem , although the question that we are able to answer with it is smaller in scope . Through a simple change of variable argument , we will argue that if the negative log likelihood can be improved by modifying the prior , exactly the same negative log likelihood can be attained by modifying the likelihood function instead . Thus if rate-distortion theory predicts that there is scope for improvement for a prior , the same holds for the likelihood function but conversely , while rate-distortion theory can precisely determine when it is that a prior can no longer be improved , the same can not be said for the likelihood function . Finally , we test whether the lower bound derived and the corresponding fundamental quantity c ( z ) are useful in practice when making modeling decisions by applying these ideas to a problem in image modeling for which there have been several recent results involving Variational Autoencoders . 2 TECHNICAL PRELIMINARIES . In our treatment , we will use notation that is commonly used in information theory . If you have two distributions P , Q the KL divergence from P to Q is denoted as DKL ( P‖Q ) which is known to be nonnegative . For a discrete random variable A , we denote its entropy H ( A ) . If you have two random variables A , B , their mutual information is I ( A ; B ) We assume that the data { x1 , · · · , xN } comes from some arbitrary alphabet X which could be continuous or discrete . For example , it could be the set of all sentences of length up to a given number of words , the set of all real valued images of a given dimension , or the set of all real valued time series with a given number of samples . Let X be a random variable distributed uniformly over the training data { x1 , · · · , xN } . The entropy H ( X ) of this random variable is logN , and the fundamental lossless compression theorem of Shannon tells us that any compression technique for independently , identically distributed data samples following the law of X must use at least H ( X ) bits per sample . But what if one is willing to allow losses in the data reconstruction ? This is where rate-distortion theory comes into play . Shannon introduced the idea of a reconstruction alphabet Z ( which need not be equal to X ) , and a distortion function d : X ×Z → R which is used to measure the cost of reproducing an element of X with an element of Z . He then introduced an auxiliary random variable Z , which is obtained by passing X through a channel QZ|X , and defined the rate-distortion function as R ( D ) = min QZ|X : EX , Zd ( X , Z ) ≤D I ( X ; Z ) ( 8 ) Shannon ’ s fundamental lossy source coding theorem in essence shows that R ( D ) plays the same role as the entropy plays for lossless compression - it represents the minimum number of bits per sample needed to compress X within a fidelity D , showing both necessity and sufficiency when the number of samples being compressed approaches infinity . This fundamental result though is not needed for this paper ; its relevance to the modeling problem will be shown in a subsequent publication . What we will use instead is the theory of how you compute rate-distortion functions . In particular , by choosing a uniform distribution over { x1 , · · · , xN } and by not requiring that these are unique , we are in effect setting the source distribution to be the empirical distribution observed in the training data .

Summary: This is an interesting paper that studies the latent variable modeling from an information theoretic perspective. Specifically, the authors argue that the rate-distortion theory for lossy compression provides a natural toolkit for studying latent variable models, and they propose a lower bound (also a gap function) that could be used to assess the goodness of data fitting given a pair of prior distribution over latent factor and a likelihood function. Overall the paper is very well-written, clear to follow, and the authors did a great job in not overclaiming their results.