Estimating the Unique Information of Continuous Variables

The integration and transfer of information from multiple sources to multiple targets is a core motive of neural systems . The emerging field of partial information decomposition ( PID ) provides a novel information-theoretic lens into these mechanisms by identifying synergistic , redundant , and unique contributions to the mutual information between one and several variables . While many works have studied aspects of PID for Gaussian and discrete distributions , the case of general continuous distributions is still uncharted territory . In this work we present a method for estimating the unique information in continuous distributions , for the case of one versus two variables . Our method solves the associated optimization problem over the space of distributions with fixed bivariate marginals by combining copula decompositions and techniques developed to optimize variational autoencoders . We obtain excellent agreement with known analytic results for Gaussians , and illustrate the power of our new approach in several brain-inspired neural models . Our method is capable of recovering the effective connectivity of a chaotic network of rate neurons , and uncovers a complex trade-off between redundancy , synergy and unique information in recurrent networks trained to solve a generalized XOR task . 1 Introduction and background . In neural systems , often multiple neurons are driven by one external event or stimulus ; conversely multiple neural inputs can converge onto a single neuron . A natural question in both cases is how multiple variables hold information about the singleton variable . In their seminal work [ 1 ] , Williams and Beer proposed an axiomatic extension of classic information theory to decompose the mutual information between multiple source variables and a single target variable in a meaningful way . For the case of two sources X1 , X2 , their partial information decomposition ( PID ) amounts to expressing the mutual information of X1 , X2 with a target Y as a sum of four non-negative terms , I ( Y : ( X1 , X2 ) ) = U ( Y : X1\X2 ) + U ( Y : X2\X1 ) +R ( Y : ( X1 , X2 ) ) + S ( Y : ( X1 , X2 ) ) , ( 1.1 ) corresponding to unique ( U1 , U2 ) , redundant ( R ) and synergistic ( S ) contributions , respectively . These terms should also obey the consistency equations I ( Y : X1 ) = R ( Y : ( X1 , X2 ) ) + U ( Y : X1\X2 ) , ( 1.2 ) I ( Y : X2 ) = R ( Y : ( X1 , X2 ) ) + U ( Y : X2\X1 ) . ( 1.3 ) The PID has proved useful in understanding information processing by distributed systems in a diverse array of fields including machine learning [ 2 , 3 ] , earth science [ 4 ] and cellular automata [ 5 ] , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and particularly in neuroscience [ 6–10 ] , where notions of synergy and redundancy , traditionally considered mutually exclusive and distinguished by the sign of ∆ = I ( Y : ( X1 , X2 ) ) − I ( Y : X1 ) − I ( Y : X2 ) , = S ( Y : ( X1 , X2 ) ) −R ( Y : ( X1 , X2 ) ) , ( 1.4 ) have long played a central role in the quest to understand how neural circuits integrate information from multiple sources [ 11–14 ] . The novelty of the PID framework here is in separating the measures of synergy and redundancy in ( 1.4 ) . The above abstract formulation of PID provides three equations for four unknowns , and only becomes operational once one of U1 , U2 , R , or S is defined . This has been done in [ 15 ] via a definition of the unique information : Definition 1 ( BROJA [ 15 ] ) . Given three random variables ( Y , X1 , X2 ) with joint probability density p ( y , x1 , x2 ) , the unique information U1 of X1 with respect to Y is U ( Y : X1\X2 ) = min q∈Q Iq ( Y : X1|X2 ) , ( 1.5 ) = min q∈Q ∫ dydx1dx2 q ( y , x1 , x2 ) log ( q ( y , x1|x2 ) q ( y|x2 ) q ( x1|x2 ) ) , ( 1.6 ) where Q = { q ( y , x1 , x2 ) | q ( y , xi ) = p ( y , xi ) , i = 1 , 2 } . ( 1.7 ) In words , we minimize the conditional mutual information I ( Y : X1|X2 ) over the space of density functions that preserve the marginal densities p ( y , x1 ) and p ( y , x2 ) . The above definition implies , along with ( 1.2 ) - ( 1.3 ) , that the unique and redundant information only depend on the marginals p ( y , x1 ) , p ( y , x2 ) , and that the synergy can only be estimated from the full p ( y , x1 , x2 ) . The original definition in [ 15 ] was limited to discrete random variables . Here , we show that the extension to continuous variables is well-defined and can be practically estimated . Motivation from decision theory [ 15 ] . Consider for simplicity discrete variables . A decision maker DM1 can choose an action a from a finite set A , and receives a reward u ( a , y ) based on the selected action and the state y , which occurs with probability p ( y ) . Notably , DM1 has no knowledge of y , but observes instead a random signal x1 sampled from p ( x1|y ) . Choosing the action maximizing the expected reward for each x1 , his maximal expected reward is R1 = ∑ x1 p ( x1 ) max a|x1 ∑ y p ( y|x1 ) u ( a , y ) . ( 1.8 ) DM1 is said to have no unique information about y w.r.t . another decision maker DM2 that observes x2 ∼ p ( x2|y ) – if R2 ≥ R1 for any set A , any distribution p ( y ) , and any reward function u ( a , y ) . A celebrated theorem by Blackwell [ 16 , 17 ] states that such a generic advantage by DM2 occurs iff there exist a stochastic matrix q ( x1|x2 ) which satisfies p ( x1|y ) = ∑ x2 p ( x2|y ) q ( x1|x2 ) . ( 1.9 ) But this occurs precisely when the unique information ( 1.5 ) vanishes , since then there exists a joint distribution q ( y , x1 , x2 ) in Q for which y ⊥ x1|x2 , which implies q ( x1|x2 , y ) = q ( x1|x2 ) , and thus ( 1.9 ) holds . Similar results exist for continuous variables [ 18 , 19 ] . Thus the unique information from Definition 1 quantifies a departure from Blackwell ’ s relation ( 1.9 ) . In this work we present a definition and a method to estimate the BROJA unique information for generic continuous probability densities . Our approach is based on the observation that the constraints ( 1.7 ) can be satisfied with an appropriate copula parametrization , and makes use of techniques developed to optimize variational autoencoders . We only consider one-dimensional Y , X1 , X2 for simplicity , but the method can be naturally extended to higher dimensional cases . In Section 2 we review related works , in Section 3 we present our method and Section 4 contains several illustrative examples . 2 Related works . Partial information decomposition offers a solution to a repeated question that was not addressed by ‘ classical ’ information theory regarding the relations between two sources and a target [ 1 ] . From a mathematical perspective a ’ functional definition ’ has to be made , meaning that such a definition should align with our intuitive notions . Yet , as shown in [ 20 ] , not all intuitively desirable properties of a PID can be realized simultaneously . Thus , different desirable properties are chosen for distinct application scenarios . Thus , various proposals for decomposition measures are not seen as conflicting but as having different operational interpretations . For example , the BROJA approach used here builds on desiderata from decision theory , while other approaches appeal to game theory [ 21 ] or the framework of Kelly gambling [ 22 ] . Yet other approaches use arguments from information geometry [ 23 ] . Other approaches assume agents receiving potentially conflicting or incomplete information about the source variables for the purpose of inference or decryption ( see e.g . [ 24 , 25 ] ) . In [ 26 ] the authors separate the specific operational interpretations of PID measures from the general structure of information decomposition . The actual computation of the BROJA unique information is non-trivial , even for discrete variables . Optimization methods exist for the latter case [ 27–29 ] , and analytic solutions are only known when all the variables are univariate binary [ 30 ] . For continuous probability densities , an earlier definition aligned with the BROJA measure was made by Barret [ 31 ] , but only applies to Gaussian variables . For Barret ’ s measure , an analytic solution is known when p ( y , x1 , x2 ) is a three-dimensional Gaussian density [ 31 ] , but does not generalize to higher dimensional Gaussians [ 32 ] . 3 Bounding and estimating the unique information . We proceed in two steps . We first introduce a parametrization of the optimization space Q in ( 1.7 ) and then introduce and optimize an upper bound on the unique information . 3.1 Parametrizing the optimization space with copulas . To characterize the optimization space Q in ( 1.5 ) - ( 1.7 ) , it is convenient to recall that according to Sklar ’ s theorem [ 33 ] , any n-variate probability density can be expressed as p ( x1 . . . xn ) = p ( x1 ) . . . p ( xn ) c ( u1 . . . un ) , ( 3.1 ) where p ( xi ) is the marginal and ui = F ( xi ) is the CDF of each variable . The dependency structure among the variables is encoded in the function c : [ 0 , 1 ] n → [ 0 , 1 ] . This is a copula density , a probability density on the unit hypercube with uniform marginals [ 34 ] , ∫ [ 0,1 ] n−1 n∏ j=1 , j 6=i duj c ( u1 . . . un ) = 1 ∀i . ( 3.2 ) Note that under univariate reparametrizations z′i = g ( zi ) , the ui ’ s and the copula c remain invariant . For an overview of copulas in machine learning , see [ 35 ] . Proposition 1 . Under the BROJA Definition 1 of unique information , all the terms of the partial information decomposition in ( 1.1 ) - ( 1.3 ) are independent of the univariate marginals p ( x1 ) , p ( x2 ) , p ( y ) , and only depend on the copula c ( uy , u1 , u2 ) . Proof . Expressing q ( y , x1 , x2 ) , q ( x1 , x2 ) , q ( y , x2 ) via copula decompositions ( 3.1 ) , and changing variables as duy = q ( y ) dy , etc. , the objective function in ( 1.6 ) becomes Iq ( Y : X1|X2 ) = ∫ [ 0,1 ] 3 duydu1du2 c ( uy , u1 , u2 ) log ( c ( uy , u1 , u2 ) c ( uy , u2 ) c ( u1 , u2 ) ) . ( 3.3 ) Note that the copula of any marginal distribution is the marginal of the copula : c ( uy , u2 ) = ∫ [ 0,1 ] du1 c ( uy , u1 , u2 ) , c ( u1 , u2 ) = ∫ [ 0,1 ] duy c ( uy , u1 , u2 ) . ( 3.4 ) Thus the optimization objective and the unique information are independent of the univariate marginals . A similar result holds for the mutual information terms in the l.h.s . of ( 1.1 ) - ( 1.3 ) .1 It follows that none of the PID terms in ( 1.1 ) - ( 1.3 ) depend on the univariate marginals , and therefore all the PID terms are invariant under univariate reparametrizations of ( y , x1 , x2 ) . In order to parametrize the optimization space Q in ( 1.7 ) using copulas , consider the factorization p ( y , x1 , x2 ) = p ( x1 ) p ( y|x1 ) p ( x2|y , x1 ) . ( 3.5 ) Using the copula decomposition ( 3.1 ) for n = 2 , the last two factors in ( 3.5 ) can be expressed as p ( y|x1 ) = p ( y , x1 ) p ( x1 ) = p ( y ) p ( x1 ) c ( y , x1 ) p ( x1 ) = c ( uy , u1 ) p ( y ) , ( 3.6 ) and similarly p ( x2|y , x1 ) = p ( x1 , x2|y ) p ( x1|y ) , ( 3.7 ) = c1,2|y ( u1|y , u2|y ) p ( x2|y ) , ( 3.8 ) = c1,2|y ( u1|y , u2|y ) c ( uy , x2 ) p ( x2 ) , ( 3.9 ) where we defined the conditional CDFs , ui|y = F ( ui|uy ) = ∂C ( uy , ui ) ∂uy i = 1 , 2 ( 3.10 ) and C ( uy , ui ) is the CDF of c ( uy , ui ) . Note that the function c1,2|y ( u1|y , u2|y ) in ( 3.8 ) is not the conditional copula c ( u1 , u2|uy ) , but rather the copula of the conditional p ( x1 , x2|y ) . Using expressions ( 3.6 ) and ( 3.9 ) , the full density ( 3.5 ) becomes p ( y , x1 , x2 ) = p ( y ) p ( x1 ) p ( x2 ) c ( uy , u1 , u2 ) , ( 3.11 ) where c ( uy , u1 , u2 ) = c ( uy , u1 ) c ( uy , u2 ) c1,2|y ( u1|y , u2|y ) . ( 3.12 ) This is a simple case of the pair-copula construction of multivariate distributions [ 38–40 ] , which allows to expand any n-variate copula as a product of ( conditional ) bivariate copulas . Proposition 2 . The copula of the conditional , c1,2|y ( · , · ) , parametrizes the space Q in ( 1.7 ) . Proof . Since q ( y , xi ) = p ( y , xi ) ( i = 1 , 2 ) , the copula factors in p ( y , xi ) = p ( y ) p ( xi ) c ( uy , ui ) , i = 1 , 2 ( 3.13 ) are fixed in Q . Therefore , in the copula decomposition ( 3.12 ) for q ( y , x1 , x2 ) ∈ Q , only the last factor can vary in Q . Let us denote by θ the parameters of a generic parametrization for the copula c1,2|y ( u1|y , u2|y ) . Since the latter is conditioned on uy , the parameters can be taken as a function θ ( uy ) . It follows that the copula of q necessarily has the form cθ ( uy , u1 , u2 ) = c ( uy , u1 ) c ( uy , u2 ) c1,2|θ ( uy ) ( u1|y , u2|y ) , ( 3.14 ) and the parameters of the function θ ( uy ) are the optimization variables.2 1The connection between mutual information and copulas was discussed in [ 36 , 37 ] . 2We note that in multivariate pair-copula expansions it is common to assume constant conditioning parame- ters θ [ 41 ] , but we do not make such a simplifying assumption .

Summary: (I am not an expert in information theory) The authors extend partial information decomposition to the case of continuous variables with arbitrary distributions.  My review will focus on the last section, where a recurrent neural network was trained on a generalized XOR task. By decomposing the unique, redundant and synergistic components of each neuron’s contribution to the output, the authors identify different strategies used by these networks.