Estimating the Unique Information of Continuous Variables

The integration and transfer of information from multiple sources to multiple targets is a core motive of neural systems . The emerging field of partial information decomposition ( PID ) provides a novel information-theoretic lens into these mechanisms by identifying synergistic , redundant , and unique contributions to the mutual information between one and several variables . While many works have studied aspects of PID for Gaussian and discrete distributions , the case of general continuous distributions is still uncharted territory . In this work we present a method for estimating the unique information in continuous distributions , for the case of one versus two variables . Our method solves the associated optimization problem over the space of distributions with fixed bivariate marginals by combining copula decompositions and techniques developed to optimize variational autoencoders . We obtain excellent agreement with known analytic results for Gaussians , and illustrate the power of our new approach in several brain-inspired neural models . Our method is capable of recovering the effective connectivity of a chaotic network of rate neurons , and uncovers a complex trade-off between redundancy , synergy and unique information in recurrent networks trained to solve a generalized XOR task . 1 Introduction and background . In neural systems , often multiple neurons are driven by one external event or stimulus ; conversely multiple neural inputs can converge onto a single neuron . A natural question in both cases is how multiple variables hold information about the singleton variable . In their seminal work [ 1 ] , Williams and Beer proposed an axiomatic extension of classic information theory to decompose the mutual information between multiple source variables and a single target variable in a meaningful way . For the case of two sources X1 , X2 , their partial information decomposition ( PID ) amounts to expressing the mutual information of X1 , X2 with a target Y as a sum of four non-negative terms , I ( Y : ( X1 , X2 ) ) = U ( Y : X1\X2 ) + U ( Y : X2\X1 ) +R ( Y : ( X1 , X2 ) ) + S ( Y : ( X1 , X2 ) ) , ( 1.1 ) corresponding to unique ( U1 , U2 ) , redundant ( R ) and synergistic ( S ) contributions , respectively . These terms should also obey the consistency equations I ( Y : X1 ) = R ( Y : ( X1 , X2 ) ) + U ( Y : X1\X2 ) , ( 1.2 ) I ( Y : X2 ) = R ( Y : ( X1 , X2 ) ) + U ( Y : X2\X1 ) . ( 1.3 ) The PID has proved useful in understanding information processing by distributed systems in a diverse array of fields including machine learning [ 2 , 3 ] , earth science [ 4 ] and cellular automata [ 5 ] , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and particularly in neuroscience [ 6–10 ] , where notions of synergy and redundancy , traditionally considered mutually exclusive and distinguished by the sign of ∆ = I ( Y : ( X1 , X2 ) ) − I ( Y : X1 ) − I ( Y : X2 ) , = S ( Y : ( X1 , X2 ) ) −R ( Y : ( X1 , X2 ) ) , ( 1.4 ) have long played a central role in the quest to understand how neural circuits integrate information from multiple sources [ 11–14 ] . The novelty of the PID framework here is in separating the measures of synergy and redundancy in ( 1.4 ) . The above abstract formulation of PID provides three equations for four unknowns , and only becomes operational once one of U1 , U2 , R , or S is defined . This has been done in [ 15 ] via a definition of the unique information : Definition 1 ( BROJA [ 15 ] ) . Given three random variables ( Y , X1 , X2 ) with joint probability density p ( y , x1 , x2 ) , the unique information U1 of X1 with respect to Y is U ( Y : X1\X2 ) = min q∈Q Iq ( Y : X1|X2 ) , ( 1.5 ) = min q∈Q ∫ dydx1dx2 q ( y , x1 , x2 ) log ( q ( y , x1|x2 ) q ( y|x2 ) q ( x1|x2 ) ) , ( 1.6 ) where Q = { q ( y , x1 , x2 ) | q ( y , xi ) = p ( y , xi ) , i = 1 , 2 } . ( 1.7 ) In words , we minimize the conditional mutual information I ( Y : X1|X2 ) over the space of density functions that preserve the marginal densities p ( y , x1 ) and p ( y , x2 ) . The above definition implies , along with ( 1.2 ) - ( 1.3 ) , that the unique and redundant information only depend on the marginals p ( y , x1 ) , p ( y , x2 ) , and that the synergy can only be estimated from the full p ( y , x1 , x2 ) . The original definition in [ 15 ] was limited to discrete random variables . Here , we show that the extension to continuous variables is well-defined and can be practically estimated . Motivation from decision theory [ 15 ] . Consider for simplicity discrete variables . A decision maker DM1 can choose an action a from a finite set A , and receives a reward u ( a , y ) based on the selected action and the state y , which occurs with probability p ( y ) . Notably , DM1 has no knowledge of y , but observes instead a random signal x1 sampled from p ( x1|y ) . Choosing the action maximizing the expected reward for each x1 , his maximal expected reward is R1 = ∑ x1 p ( x1 ) max a|x1 ∑ y p ( y|x1 ) u ( a , y ) . ( 1.8 ) DM1 is said to have no unique information about y w.r.t . another decision maker DM2 that observes x2 ∼ p ( x2|y ) – if R2 ≥ R1 for any set A , any distribution p ( y ) , and any reward function u ( a , y ) . A celebrated theorem by Blackwell [ 16 , 17 ] states that such a generic advantage by DM2 occurs iff there exist a stochastic matrix q ( x1|x2 ) which satisfies p ( x1|y ) = ∑ x2 p ( x2|y ) q ( x1|x2 ) . ( 1.9 ) But this occurs precisely when the unique information ( 1.5 ) vanishes , since then there exists a joint distribution q ( y , x1 , x2 ) in Q for which y ⊥ x1|x2 , which implies q ( x1|x2 , y ) = q ( x1|x2 ) , and thus ( 1.9 ) holds . Similar results exist for continuous variables [ 18 , 19 ] . Thus the unique information from Definition 1 quantifies a departure from Blackwell ’ s relation ( 1.9 ) . In this work we present a definition and a method to estimate the BROJA unique information for generic continuous probability densities . Our approach is based on the observation that the constraints ( 1.7 ) can be satisfied with an appropriate copula parametrization , and makes use of techniques developed to optimize variational autoencoders . We only consider one-dimensional Y , X1 , X2 for simplicity , but the method can be naturally extended to higher dimensional cases . In Section 2 we review related works , in Section 3 we present our method and Section 4 contains several illustrative examples . 2 Related works . Partial information decomposition offers a solution to a repeated question that was not addressed by ‘ classical ’ information theory regarding the relations between two sources and a target [ 1 ] . From a mathematical perspective a ’ functional definition ’ has to be made , meaning that such a definition should align with our intuitive notions . Yet , as shown in [ 20 ] , not all intuitively desirable properties of a PID can be realized simultaneously . Thus , different desirable properties are chosen for distinct application scenarios . Thus , various proposals for decomposition measures are not seen as conflicting but as having different operational interpretations . For example , the BROJA approach used here builds on desiderata from decision theory , while other approaches appeal to game theory [ 21 ] or the framework of Kelly gambling [ 22 ] . Yet other approaches use arguments from information geometry [ 23 ] . Other approaches assume agents receiving potentially conflicting or incomplete information about the source variables for the purpose of inference or decryption ( see e.g . [ 24 , 25 ] ) . In [ 26 ] the authors separate the specific operational interpretations of PID measures from the general structure of information decomposition . The actual computation of the BROJA unique information is non-trivial , even for discrete variables . Optimization methods exist for the latter case [ 27–29 ] , and analytic solutions are only known when all the variables are univariate binary [ 30 ] . For continuous probability densities , an earlier definition aligned with the BROJA measure was made by Barret [ 31 ] , but only applies to Gaussian variables . For Barret ’ s measure , an analytic solution is known when p ( y , x1 , x2 ) is a three-dimensional Gaussian density [ 31 ] , but does not generalize to higher dimensional Gaussians [ 32 ] . 3 Bounding and estimating the unique information . We proceed in two steps . We first introduce a parametrization of the optimization space Q in ( 1.7 ) and then introduce and optimize an upper bound on the unique information . 3.1 Parametrizing the optimization space with copulas . To characterize the optimization space Q in ( 1.5 ) - ( 1.7 ) , it is convenient to recall that according to Sklar ’ s theorem [ 33 ] , any n-variate probability density can be expressed as p ( x1 . . . xn ) = p ( x1 ) . . . p ( xn ) c ( u1 . . . un ) , ( 3.1 ) where p ( xi ) is the marginal and ui = F ( xi ) is the CDF of each variable . The dependency structure among the variables is encoded in the function c : [ 0 , 1 ] n → [ 0 , 1 ] . This is a copula density , a probability density on the unit hypercube with uniform marginals [ 34 ] , ∫ [ 0,1 ] n−1 n∏ j=1 , j 6=i duj c ( u1 . . . un ) = 1 ∀i . ( 3.2 ) Note that under univariate reparametrizations z′i = g ( zi ) , the ui ’ s and the copula c remain invariant . For an overview of copulas in machine learning , see [ 35 ] . Proposition 1 . Under the BROJA Definition 1 of unique information , all the terms of the partial information decomposition in ( 1.1 ) - ( 1.3 ) are independent of the univariate marginals p ( x1 ) , p ( x2 ) , p ( y ) , and only depend on the copula c ( uy , u1 , u2 ) . Proof . Expressing q ( y , x1 , x2 ) , q ( x1 , x2 ) , q ( y , x2 ) via copula decompositions ( 3.1 ) , and changing variables as duy = q ( y ) dy , etc. , the objective function in ( 1.6 ) becomes Iq ( Y : X1|X2 ) = ∫ [ 0,1 ] 3 duydu1du2 c ( uy , u1 , u2 ) log ( c ( uy , u1 , u2 ) c ( uy , u2 ) c ( u1 , u2 ) ) . ( 3.3 ) Note that the copula of any marginal distribution is the marginal of the copula : c ( uy , u2 ) = ∫ [ 0,1 ] du1 c ( uy , u1 , u2 ) , c ( u1 , u2 ) = ∫ [ 0,1 ] duy c ( uy , u1 , u2 ) . ( 3.4 ) Thus the optimization objective and the unique information are independent of the univariate marginals . A similar result holds for the mutual information terms in the l.h.s . of ( 1.1 ) - ( 1.3 ) .1 It follows that none of the PID terms in ( 1.1 ) - ( 1.3 ) depend on the univariate marginals , and therefore all the PID terms are invariant under univariate reparametrizations of ( y , x1 , x2 ) . In order to parametrize the optimization space Q in ( 1.7 ) using copulas , consider the factorization p ( y , x1 , x2 ) = p ( x1 ) p ( y|x1 ) p ( x2|y , x1 ) . ( 3.5 ) Using the copula decomposition ( 3.1 ) for n = 2 , the last two factors in ( 3.5 ) can be expressed as p ( y|x1 ) = p ( y , x1 ) p ( x1 ) = p ( y ) p ( x1 ) c ( y , x1 ) p ( x1 ) = c ( uy , u1 ) p ( y ) , ( 3.6 ) and similarly p ( x2|y , x1 ) = p ( x1 , x2|y ) p ( x1|y ) , ( 3.7 ) = c1,2|y ( u1|y , u2|y ) p ( x2|y ) , ( 3.8 ) = c1,2|y ( u1|y , u2|y ) c ( uy , x2 ) p ( x2 ) , ( 3.9 ) where we defined the conditional CDFs , ui|y = F ( ui|uy ) = ∂C ( uy , ui ) ∂uy i = 1 , 2 ( 3.10 ) and C ( uy , ui ) is the CDF of c ( uy , ui ) . Note that the function c1,2|y ( u1|y , u2|y ) in ( 3.8 ) is not the conditional copula c ( u1 , u2|uy ) , but rather the copula of the conditional p ( x1 , x2|y ) . Using expressions ( 3.6 ) and ( 3.9 ) , the full density ( 3.5 ) becomes p ( y , x1 , x2 ) = p ( y ) p ( x1 ) p ( x2 ) c ( uy , u1 , u2 ) , ( 3.11 ) where c ( uy , u1 , u2 ) = c ( uy , u1 ) c ( uy , u2 ) c1,2|y ( u1|y , u2|y ) . ( 3.12 ) This is a simple case of the pair-copula construction of multivariate distributions [ 38–40 ] , which allows to expand any n-variate copula as a product of ( conditional ) bivariate copulas . Proposition 2 . The copula of the conditional , c1,2|y ( · , · ) , parametrizes the space Q in ( 1.7 ) . Proof . Since q ( y , xi ) = p ( y , xi ) ( i = 1 , 2 ) , the copula factors in p ( y , xi ) = p ( y ) p ( xi ) c ( uy , ui ) , i = 1 , 2 ( 3.13 ) are fixed in Q . Therefore , in the copula decomposition ( 3.12 ) for q ( y , x1 , x2 ) ∈ Q , only the last factor can vary in Q . Let us denote by θ the parameters of a generic parametrization for the copula c1,2|y ( u1|y , u2|y ) . Since the latter is conditioned on uy , the parameters can be taken as a function θ ( uy ) . It follows that the copula of q necessarily has the form cθ ( uy , u1 , u2 ) = c ( uy , u1 ) c ( uy , u2 ) c1,2|θ ( uy ) ( u1|y , u2|y ) , ( 3.14 ) and the parameters of the function θ ( uy ) are the optimization variables.2 1The connection between mutual information and copulas was discussed in [ 36 , 37 ] . 2We note that in multivariate pair-copula expansions it is common to assume constant conditioning parame- ters θ [ 41 ] , but we do not make such a simplifying assumption .

Summary:    This paper presents a novel method for computing the Partial    Information Decomposition (BROJA flavour) in the case of three    continuous random variables. The method uses a variational approach    on a lower bound derived from a copula representation of the joint    PDF of the three variables. The paper presents some application    examples, including two where the proposed method is used to    elucidate the properties of recurrent neural networks.