TherML: The Thermodynamics of Machine Learning

1 INTRODUCTION . Let X , Y be some paired data , for example : a set of images X and their labels Y . We imagine the data comes from some true , unknown data generating process Φ1 , from which we have drawn a training set of N pairs : TN ≡ ( xN , yN ) ≡ { x1 , y1 , x2 , y2 , . . . , xN , yN } ∼ φ ( xN , yN ) . ( 1 ) We further imagine the process is exchangeable2 and the data is conditionally independent given the governing process Φ : p ( xN , yN |φ ) = ∏ i p ( xi|φ ) p ( yi|xi , φ ) . ( 2 ) As machine learners , we believe that by studying the training set , we should be able to infer or predict new draws from the same data generating process . Call a set of M future draws from the data generating process T ′M ≡ { XM , YM } the test set . The predictive information ( Bialek et al. , 2001 ) is the mutual information between the training set and a infinite test set , equivalently the amount of information the training set provides about the generative process itself : Ipred ( TN ) ≡ lim M→∞ I ( TN ; T ′M ) = I ( TN ; Φ ) = I ( XN , Y N ; Φ ) . ( 3 ) The predictive information measures the underlying complexity of the data generating process ( Still , 2014 ) , and is fundamentally limited and must grow sublinearly in the dataset size ( Bialek et al. , 2001 ) . Hence , the predictive information is a vanishing fraction of the total information in the training set 3 : lim N→∞ Ipred ( TN ) H ( TN ) = 0 ( 4 ) A vanishing fraction of the information present in our training data is in any way useful for future tasks . A vanishing fraction of the information contained in the training data is signal , the rest is noise . We claim the goal of learning is to learn a representation of data , both locally and globally that captures the predictive information while being maximally compressed : that separates the signal from the noise . 1 Here we aim to invoke the same philosophy as in the introduction to Watanabe ( 2018 ) . 2 That is , we imagine the data satisfies De Finetti ’ s theorem , for which infinite exchangeable processes usually can be described by products of conditionally independent distributions , but don ’ t want to worry too much about the complicated details since there are subtle special cases ( Accardi , 2018 ) . 3 Here and throughout H ( A ) is used to denote entropies H ( A ) = − ∑ i p ( A ) log p ( A ) . 2 A TALE OF TWO WORLDS . We are primarily interested in learning a stochastic local representation of X , call it Z , defined by some parametric distribution of our own design : p ( zi|xi , θ ) with its own parameters θ . A training procedure is a process that assigns a distribution p ( θ|xN , yN ) to the parameters conditioned on the observed dataset . In this way , the parameters of our local parametric map are themselves a global representation of the dataset . With our augmentations , the world now looks like the graphical model in Figure 3a , denoted World P : Some data generating process Φ generates a dataset ( XN , Y N ) which we perform some learning algorithm on to get some parameters p ( θ|xN , yN ) which we can use to form a parametric local representation p ( zi|xi , θ ) . World P is what we have . It is not necessarily what we want . What we have to contend with is an unknown distribution of our data . What we want is a world that corresponds to the traditional modeling assumptions in which Z acts as a latent factor for X and Y , rending them conditionally independent , leaving no correlations unexplained . Similarly , we would prefer if we could easily marginalize out the dependence on our universal ( Φ ) and model specific ( Θ ) parameters . World Q in Figure 3b is the world we want 4 . We can measure the degree to which the real world aligns with our desires by computing the minimum possible relative information5 between our distribution p and any distribution consistent with the conditional dependencies encoded in graphical model Q6 . It can be shown ( Friedman et al. , 2001 ) that this quantity is given by the difference in multi-informations between the two graphical models , as measured in World P : J ≡ min q∈Q DKL [ p ; q ] = IP − IQ . ( 5 ) The multi-information ( Slonim et al. , 2005 ) of a graphical model is the KL divergence between the joint distribution and the product of all of the marginal distributions , which can be computed as a sum of mutual informations , one for each node in the graph , between itself and its parents : IG ≡ 〈 log p ( gN ) ∏ i p ( gi ) 〉 = ∑ i I ( gi ; Pa ( gi ) ) ( 6 ) In our case : J = I ( Θ ; XN , Y N ) + ∑ i [ I ( Xi ; Φ ) + I ( Yi ; Xi , Φ ) + I ( Zi ; Xi , Θ ) − I ( Xi ; Zi ) − I ( Yi ; Zi ) ] . ( 7 ) 4 We could consider different alternatives , deciding to relax some of the constraints we imposed in World Q , or generalizing World P by letting the representation depend on X and Y jointly , for instance . What follows demonstrates a general sort of calculus that we can invoke for any specified pair of graphical models . In particular Appendices A to C discuss alternatives . 5 Also known as the KL divergence . 6 Note that this is DKL [ p ; q∗ ] where q∗ is the well known reverse-information projection or moment pro- jection : q∗ = argminq∈QDKL [ p ; q ] ( Csiszár & Matúš , 2003 ) . This minimal relative information has two terms outside our control and we can take them to be constant , but which relate to the predictive information : ∑ i [ I ( Xi ; Φ ) + I ( Yi ; Xi , Φ ) ] ≥ ∑ i I ( Yi ; Xi ) + Ipred ( TN ) . ( 8 ) These terms measure the intrinsic complexity of our data . The remaining four terms are : • I ( Xi ; Zi ) - which measures how much information our representation contains about the input ( X ) . This should be maximized to ensure our local representation actually represents the input . • I ( Yi ; Zi ) - which measures how much information our representation contains about our auxiliary data . This should be maximized as well to ensure that our local representation is predictive for the labels . • I ( Zi ; Xi , Θ ) - which measures how much information the parameters and input determine about our representation . This should be minimized to ensure consistency between worlds , and ensure we learn compressed local representations . Notice that this is similar to , but distinct from the first term above . I ( Zi ; Xi , Θ ) = I ( Zi ; Xi ) + I ( Zi ; Θ|Xi ) ( 9 ) by the Chain Rule for mutual information 7 . • I ( Θ ; XN , Y N ) - which measures how much information we store about our training data in the parameters of our encoder . This should also be minimized to ensure we learn compressed global representation , preveting overfitting . These mutual informations are all intractable in general , since we can not compute the necessary marginals in closed form , given that we do not have access to the true data generating distribution . 2.1 FUNCTIONALS . Despite their intractability , we can compute variational bounds on these mutual informations . 2.1.1 ENTROPY . S ≡ 〈 log p ( θ|xN , yN ) q ( θ ) 〉 P ≥ I ( Θ ; XN , Y N ) ( 10 ) The relative entropy in our parameters or just entropy for short measures the relative information between the distribution we assign our parameters in World P after learning from the data ( XN , Y N ) , with respect to some data independent q ( θ ) prior on the parameters . This is an upper bound on the mutual information between the data and our parameters and as such can measure our risk of overfitting our parameters . 2.1.2 RATE Ri ≡ 〈 log p ( zi|xi , θ ) q ( zi ) 〉 P ≥ I ( Zi ; Xi , Θ ) ( 11 ) The rate measures the complexity of our representation . It is the relative information of a sample specific representation zi ∼ p ( z|xi , θ ) with respect to our variational marginal q ( z ) . It measures how many bits we actually encode about each sample , and can measure how our risk of overfitting our representation . We use R ≡ ∑ iRi . 7Given this relationship , we could actually reduce the total number of functions we consider from 4 to 3 , as discussed in Appendix A . 2.1.3 CLASSIFICATION ERROR . Ci ≡ −〈log q ( yi|zi ) 〉P ≥ H ( Yi ) − I ( Yi ; Zi ) = H ( Yi|Zi ) ( 12 ) The classification error measures the conditional entropy of Y left after conditioning on Z . It is a measure of how much information about Y is left unspecified in our representation . This functional measures our supervised learning performance . We use C ≡ ∑ i Ci . 2.1.4 DISTORTION . Di ≡ −〈log q ( xi|zi ) 〉P ≥ H ( Xi ) − I ( Xi ; Zi ) = H ( Xi|Zi ) ( 13 ) The distortion measures the conditional entropy of X left after conditioning on Z . It is a measure of how much information about X is left unspecified in our representation . This functional measures our unsupervised learning performance . We use D ≡ ∑ iDi . 2.2 GEOMETRY . The distributions p ( z|x , θ ) , p ( θ|xN , yN ) , q ( z ) , q ( x|z ) , q ( y|z ) can be chosen arbitrarily . Once chosen , the functionals R , C , D , S take on well described values . The choice of the five distributional families specifies a single point in a four-dimensional space . Importantly , the sum of these functionals is a variational upper bound ( up to an additive constant ) for the minimum possible relative information between worlds ( Appendix D ) : S +R+ C +D ≥ J + ∑ i H ( Xi , Yi|Φ ) ( 14 ) Besides just the upper bound , we can consider the full space of feasible points . Notice that S and R are both themselves upper bounds on mutual informations , and so must be positive semidefinite . If our data is discrete , or if we have discretized it 8 , D and C which are both upper bounds on conditional entropies , must be positive as well . Along with Equation ( 14 ) , given that∑ iH ( Xi , Yi|Φ ) is a positive constant outside our control , the space of possible ( R , C , D , S ) values is at least restricted to be points in the positive orthant with some minimum possible Manhattan distance to the origin : S +R+ C +D ≥ ∑ i H ( Xi , Yi|Φ ) R ≥ 0 S ≥ 0 D ≥ 0 C ≥ 0 ( 15 ) Even in the infinite model family limit , data-processing inequalities on mutual information terms all defined in a set of variables that satisfy some nontrivial conditional dependencies ensure that there are regions in this functional space that are wholly out of reach . The surface of the feasible region maps an optimal frontier , optimal in the degree to which it minimizes mismatch between our two worlds subject to constraints on the relative magnitudes of the individual terms . This convex polytope has edges , faces and corners that are identifiable as the optimal solutions for well known objectives . This story is a generalization of the story presented in Alemi et al . ( 2018 ) , which can be considered a two-dimensional projection of this larger space ( onto R , D ) . Within our larger framework we can derive more specific bounds between subsets of the functionals . For instance : Ri +Di ≥ H ( Xi ) + I ( Zi ; Θ|Xi ) . ( 16 ) This mirrors the bound given in Alemi et al . ( 2018 ) where R+D ≥ H ( X ) , which is still true given that all conditional mutual informations are positive semi-definite ( H ( X ) + I ( Z ; Θ|X ) ≥ H ( X ) ) , but here we obtain a tighter pointwise bound that has a term measuring how much information about our encoding is revealed by the parameters after conditioning on the input itself . This term 8More generally , if we choose some measure m ( x ) , m ( y ) on both X and Y , we can define D and C in terms of that measure e.g . D ≡ − 〈 log q ( x|z ) m ( x ) 〉 P ≥ Hm ( X ) − I ( X ; Z ) = Hm ( X|Z ) I ( Zi ; Θ|Xi ) captures the degree to which our local representation is overly sensitive to the particular parameter settings 910 .

Summary: This paper attempts to establish a notion of thermodynamics for machine learning. Let me give an attempt at summary. First, an objective function is established based on demanding that the multi-information of two graphical models be small. The first graphical model is supposed to represent the actual dependence of variables and parameters used to learn a latent description of the training data, and the model demands that the latents entirely explain the correlation of the data, with the parameters marginalized out. Then, a variational approximation is made to four subsets of terms in this objective function, defining four "thermodynamic"  functionals. Minimizing the sum of these functionals puts a variational upper bound on the objective. Next, the sum is related to an unconstrained Lagrange multiplier problem making use of the facts (1) that such an objective will likely have many different realizations of the thermodynamic functionals for specific value of the bound and (2) that on the optimal surface the value of one of the functional can be parametrized in terms of the three others. If we pick the entropy functional to be parameterized in terms of the others, we find ourself precisely in the where the solution to the optimization is a Boltzmann distribution; the coefficients of the Lagrange multipliers will then take on thermodynamic interpretations in of temperature, generalized chemical potentials, etc. At this point, the machinery of thermodynamics can be brought to bear, including a first law, Maxwell relations (equality of mixed partial derivatives), etc.