TherML: The Thermodynamics of Machine Learning

1 INTRODUCTION . Let X , Y be some paired data , for example : a set of images X and their labels Y . We imagine the data comes from some true , unknown data generating process Φ1 , from which we have drawn a training set of N pairs : TN ≡ ( xN , yN ) ≡ { x1 , y1 , x2 , y2 , . . . , xN , yN } ∼ φ ( xN , yN ) . ( 1 ) We further imagine the process is exchangeable2 and the data is conditionally independent given the governing process Φ : p ( xN , yN |φ ) = ∏ i p ( xi|φ ) p ( yi|xi , φ ) . ( 2 ) As machine learners , we believe that by studying the training set , we should be able to infer or predict new draws from the same data generating process . Call a set of M future draws from the data generating process T ′M ≡ { XM , YM } the test set . The predictive information ( Bialek et al. , 2001 ) is the mutual information between the training set and a infinite test set , equivalently the amount of information the training set provides about the generative process itself : Ipred ( TN ) ≡ lim M→∞ I ( TN ; T ′M ) = I ( TN ; Φ ) = I ( XN , Y N ; Φ ) . ( 3 ) The predictive information measures the underlying complexity of the data generating process ( Still , 2014 ) , and is fundamentally limited and must grow sublinearly in the dataset size ( Bialek et al. , 2001 ) . Hence , the predictive information is a vanishing fraction of the total information in the training set 3 : lim N→∞ Ipred ( TN ) H ( TN ) = 0 ( 4 ) A vanishing fraction of the information present in our training data is in any way useful for future tasks . A vanishing fraction of the information contained in the training data is signal , the rest is noise . We claim the goal of learning is to learn a representation of data , both locally and globally that captures the predictive information while being maximally compressed : that separates the signal from the noise . 1 Here we aim to invoke the same philosophy as in the introduction to Watanabe ( 2018 ) . 2 That is , we imagine the data satisfies De Finetti ’ s theorem , for which infinite exchangeable processes usually can be described by products of conditionally independent distributions , but don ’ t want to worry too much about the complicated details since there are subtle special cases ( Accardi , 2018 ) . 3 Here and throughout H ( A ) is used to denote entropies H ( A ) = − ∑ i p ( A ) log p ( A ) . 2 A TALE OF TWO WORLDS . We are primarily interested in learning a stochastic local representation of X , call it Z , defined by some parametric distribution of our own design : p ( zi|xi , θ ) with its own parameters θ . A training procedure is a process that assigns a distribution p ( θ|xN , yN ) to the parameters conditioned on the observed dataset . In this way , the parameters of our local parametric map are themselves a global representation of the dataset . With our augmentations , the world now looks like the graphical model in Figure 3a , denoted World P : Some data generating process Φ generates a dataset ( XN , Y N ) which we perform some learning algorithm on to get some parameters p ( θ|xN , yN ) which we can use to form a parametric local representation p ( zi|xi , θ ) . World P is what we have . It is not necessarily what we want . What we have to contend with is an unknown distribution of our data . What we want is a world that corresponds to the traditional modeling assumptions in which Z acts as a latent factor for X and Y , rending them conditionally independent , leaving no correlations unexplained . Similarly , we would prefer if we could easily marginalize out the dependence on our universal ( Φ ) and model specific ( Θ ) parameters . World Q in Figure 3b is the world we want 4 . We can measure the degree to which the real world aligns with our desires by computing the minimum possible relative information5 between our distribution p and any distribution consistent with the conditional dependencies encoded in graphical model Q6 . It can be shown ( Friedman et al. , 2001 ) that this quantity is given by the difference in multi-informations between the two graphical models , as measured in World P : J ≡ min q∈Q DKL [ p ; q ] = IP − IQ . ( 5 ) The multi-information ( Slonim et al. , 2005 ) of a graphical model is the KL divergence between the joint distribution and the product of all of the marginal distributions , which can be computed as a sum of mutual informations , one for each node in the graph , between itself and its parents : IG ≡ 〈 log p ( gN ) ∏ i p ( gi ) 〉 = ∑ i I ( gi ; Pa ( gi ) ) ( 6 ) In our case : J = I ( Θ ; XN , Y N ) + ∑ i [ I ( Xi ; Φ ) + I ( Yi ; Xi , Φ ) + I ( Zi ; Xi , Θ ) − I ( Xi ; Zi ) − I ( Yi ; Zi ) ] . ( 7 ) 4 We could consider different alternatives , deciding to relax some of the constraints we imposed in World Q , or generalizing World P by letting the representation depend on X and Y jointly , for instance . What follows demonstrates a general sort of calculus that we can invoke for any specified pair of graphical models . In particular Appendices A to C discuss alternatives . 5 Also known as the KL divergence . 6 Note that this is DKL [ p ; q∗ ] where q∗ is the well known reverse-information projection or moment pro- jection : q∗ = argminq∈QDKL [ p ; q ] ( Csiszár & Matúš , 2003 ) . This minimal relative information has two terms outside our control and we can take them to be constant , but which relate to the predictive information : ∑ i [ I ( Xi ; Φ ) + I ( Yi ; Xi , Φ ) ] ≥ ∑ i I ( Yi ; Xi ) + Ipred ( TN ) . ( 8 ) These terms measure the intrinsic complexity of our data . The remaining four terms are : • I ( Xi ; Zi ) - which measures how much information our representation contains about the input ( X ) . This should be maximized to ensure our local representation actually represents the input . • I ( Yi ; Zi ) - which measures how much information our representation contains about our auxiliary data . This should be maximized as well to ensure that our local representation is predictive for the labels . • I ( Zi ; Xi , Θ ) - which measures how much information the parameters and input determine about our representation . This should be minimized to ensure consistency between worlds , and ensure we learn compressed local representations . Notice that this is similar to , but distinct from the first term above . I ( Zi ; Xi , Θ ) = I ( Zi ; Xi ) + I ( Zi ; Θ|Xi ) ( 9 ) by the Chain Rule for mutual information 7 . • I ( Θ ; XN , Y N ) - which measures how much information we store about our training data in the parameters of our encoder . This should also be minimized to ensure we learn compressed global representation , preveting overfitting . These mutual informations are all intractable in general , since we can not compute the necessary marginals in closed form , given that we do not have access to the true data generating distribution . 2.1 FUNCTIONALS . Despite their intractability , we can compute variational bounds on these mutual informations . 2.1.1 ENTROPY . S ≡ 〈 log p ( θ|xN , yN ) q ( θ ) 〉 P ≥ I ( Θ ; XN , Y N ) ( 10 ) The relative entropy in our parameters or just entropy for short measures the relative information between the distribution we assign our parameters in World P after learning from the data ( XN , Y N ) , with respect to some data independent q ( θ ) prior on the parameters . This is an upper bound on the mutual information between the data and our parameters and as such can measure our risk of overfitting our parameters . 2.1.2 RATE Ri ≡ 〈 log p ( zi|xi , θ ) q ( zi ) 〉 P ≥ I ( Zi ; Xi , Θ ) ( 11 ) The rate measures the complexity of our representation . It is the relative information of a sample specific representation zi ∼ p ( z|xi , θ ) with respect to our variational marginal q ( z ) . It measures how many bits we actually encode about each sample , and can measure how our risk of overfitting our representation . We use R ≡ ∑ iRi . 7Given this relationship , we could actually reduce the total number of functions we consider from 4 to 3 , as discussed in Appendix A . 2.1.3 CLASSIFICATION ERROR . Ci ≡ −〈log q ( yi|zi ) 〉P ≥ H ( Yi ) − I ( Yi ; Zi ) = H ( Yi|Zi ) ( 12 ) The classification error measures the conditional entropy of Y left after conditioning on Z . It is a measure of how much information about Y is left unspecified in our representation . This functional measures our supervised learning performance . We use C ≡ ∑ i Ci . 2.1.4 DISTORTION . Di ≡ −〈log q ( xi|zi ) 〉P ≥ H ( Xi ) − I ( Xi ; Zi ) = H ( Xi|Zi ) ( 13 ) The distortion measures the conditional entropy of X left after conditioning on Z . It is a measure of how much information about X is left unspecified in our representation . This functional measures our unsupervised learning performance . We use D ≡ ∑ iDi . 2.2 GEOMETRY . The distributions p ( z|x , θ ) , p ( θ|xN , yN ) , q ( z ) , q ( x|z ) , q ( y|z ) can be chosen arbitrarily . Once chosen , the functionals R , C , D , S take on well described values . The choice of the five distributional families specifies a single point in a four-dimensional space . Importantly , the sum of these functionals is a variational upper bound ( up to an additive constant ) for the minimum possible relative information between worlds ( Appendix D ) : S +R+ C +D ≥ J + ∑ i H ( Xi , Yi|Φ ) ( 14 ) Besides just the upper bound , we can consider the full space of feasible points . Notice that S and R are both themselves upper bounds on mutual informations , and so must be positive semidefinite . If our data is discrete , or if we have discretized it 8 , D and C which are both upper bounds on conditional entropies , must be positive as well . Along with Equation ( 14 ) , given that∑ iH ( Xi , Yi|Φ ) is a positive constant outside our control , the space of possible ( R , C , D , S ) values is at least restricted to be points in the positive orthant with some minimum possible Manhattan distance to the origin : S +R+ C +D ≥ ∑ i H ( Xi , Yi|Φ ) R ≥ 0 S ≥ 0 D ≥ 0 C ≥ 0 ( 15 ) Even in the infinite model family limit , data-processing inequalities on mutual information terms all defined in a set of variables that satisfy some nontrivial conditional dependencies ensure that there are regions in this functional space that are wholly out of reach . The surface of the feasible region maps an optimal frontier , optimal in the degree to which it minimizes mismatch between our two worlds subject to constraints on the relative magnitudes of the individual terms . This convex polytope has edges , faces and corners that are identifiable as the optimal solutions for well known objectives . This story is a generalization of the story presented in Alemi et al . ( 2018 ) , which can be considered a two-dimensional projection of this larger space ( onto R , D ) . Within our larger framework we can derive more specific bounds between subsets of the functionals . For instance : Ri +Di ≥ H ( Xi ) + I ( Zi ; Θ|Xi ) . ( 16 ) This mirrors the bound given in Alemi et al . ( 2018 ) where R+D ≥ H ( X ) , which is still true given that all conditional mutual informations are positive semi-definite ( H ( X ) + I ( Z ; Θ|X ) ≥ H ( X ) ) , but here we obtain a tighter pointwise bound that has a term measuring how much information about our encoding is revealed by the parameters after conditioning on the input itself . This term 8More generally , if we choose some measure m ( x ) , m ( y ) on both X and Y , we can define D and C in terms of that measure e.g . D ≡ − 〈 log q ( x|z ) m ( x ) 〉 P ≥ Hm ( X ) − I ( X ; Z ) = Hm ( X|Z ) I ( Zi ; Θ|Xi ) captures the degree to which our local representation is overly sensitive to the particular parameter settings 910 .

Summary: This paper builds on the (Alemi et al 2018) ICML paper and presents a formal framework for representation learning. The authors use a graphical model for their representation learning task and use basic information theoretic inequalities to upper-bound their measure of performance which is a KL divergence. The authors then define the optimal frontier which corresponds to the lowest possible upper-bound and write it as an optimization problem. Written with Lagrange multipliers, they obtain several known cost functions for different particular choices of these parameters.