Title: Gaussian Partial Information Decomposition: Bias Correction and Application to High-dimensional Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: partial information decomposition

estimation

bias

inter-area interaction

neuroscience
Abstract: Recent advances in neuroscientific experimental techniques have enabled us to simultaneously record the activity of thousands of neurons across multiple brain regions. This has led to a growing need for computational tools capable of analyzing how task-relevant information is represented and communicated between several brain regions. Partial information decompositions (PIDs) have emerged as one such tool, quantifying how much unique, redundant and synergistic information two or more brain regions carry about a task-relevant message. However, computing PIDs is computationally challenging in practice, and statistical issues such as the bias and variance of estimates remain largely unexplored. In this paper, we propose a new method for efficiently computing and estimating a PID definition on multivariate Gaussian distributions. We show empirically that our method satisfies an intuitive additivity property, and recovers the ground truth in a battery of canonical examples, even at high dimensionality. We also propose and evaluate, for the first time, a method to correct the bias in PID estimates at finite sample sizes. Finally, we demonstrate that our Gaussian PID effectively characterizes inter-areal interactions in the mouse brain, revealing higher redundancy between visual areas when a stimulus is behaviorally relevant.

Gaussian Partial Information Decomposition: Bias Correction and Application to High-dimensional Data

Abstract

Recent advances in neuroscientific experimental techniques have enabled us to simultaneously record the activity of thousands of neurons across multiple brain regions. This has led to a growing need for computational tools capable of analyzing how task-relevant information is represented and communicated between several brain regions. Partial information decompositions (PIDs) have emerged as one such tool, quantifying how much unique, redundant and synergistic information two or more brain regions carry about a task-relevant message. However, computing PIDs is computationally challenging in practice, and statistical issues such as the bias and variance of estimates remain largely unexplored. In this paper, we propose a new method for efficiently computing and estimating a PID definition on multivariate Gaussian distributions. We show empirically that our method satisfies an intuitive additivity property, and recovers the ground truth in a battery of canonical examples, even at high dimensionality. We also propose and evaluate, for the first time, a method to correct the bias in PID estimates at finite sample sizes. Finally, we demonstrate that our Gaussian PID effectively characterizes inter-areal interactions in the mouse brain, revealing higher redundancy between visual areas when a stimulus is behaviorally relevant.

1 Introduction

Neuroscientific experiments are increasingly collecting large-scale datasets with simultaneous recordings of multiple brain regions with single-unit resolution [1-3]. These experimental advances call for new computational tools that can allow us to probe how multiple brain regions jointly process relevant information in a behaving animal.

Partial Information Decompositions (PIDs) offer a new method for studying how different brain regions carry task-relevant information: they provide measures to quantify the amount of unique , redundant and synergistic information that one region has with respect to another. The information itself could pertain to task-relevant variables such as stimuli, behavioral responses, or information contained in a third region. For example, we may be interested in how much information about a stimulus is communicated or shared (i.e., redundantly present) between two brain regions over time. Or, we might be interested in the extent to which one region's activity uniquely explains that of another, while excluding information corresponding to spontaneous behaviors.

Ideas such as redundancy and synergy have a long history in neuroscience, having been proposed for understanding noise correlations [4] and to understand differences in encoding complexity between different brain regions [5]. PIDs have also been suggested for quantifying how much sensory information is used to execute behaviors [6] and for tracking stimulus-dependent information flows between brain regions [7, 8]. Outside of neuroscience, PID has been used to understand interactions

between different variables in financial markets [9], to quantify the relevance of different features for the purpose of feature selection in machine learning [10], and to define and quantify bias in the field of fair Machine Learning [11].

An important constraint that has limited the broader adoption of PIDs in neuroscience is the computational difficulty of estimating PIDs for high-dimensional data. Many PID definitions that are operationally well-motivated involve solving an optimization problem over a space of probability distributions: the number of optimization variables can thus be exponential in the number of neurons [12]. This has led to the use of poorly motivated PID definitions that are easy to compute (such as the 'MMI-PID' of [13], in works such as [9, 14-16]), or limited analyses to very few dimensions [17]. Furthermore, due to the limited exploration of estimators for PIDs, issues such as the bias and variance of estimates have received no attention so far, to our knowledge.

In this paper, we make the following contributions:

We provide a new and efficient method for computing and estimating a well-known PID definition called the ∼ -PID or the BROJA-PID [18] on Gaussian distributions (Section 3). By restricting our attention to Gaussian distributions, we are able to significantly reduce the number of optimization variables, so that this is just quadratic in the number of neurons, rather than exponential.

We present a set of canonical examples for Gaussian distributions where ground truth is known, and show that our method outperforms others (Section 4).

We also raise (for what we believe is the first time) the issue of bias in PID estimates, propose a method for correcting the bias, and empirically evaluate its performance (Section 5).

Finally, we show that our Gaussian PID estimator closely agrees with ground truth, even on non-Gaussian distributions, and show an example of its use on real neural data (Section 6).

Related work

Our method is based on our earlier work [12], where we also examined PIDs for Gaussian distributions. Our current work differs in a few key aspects: (i) we estimate the PID of a different PID definition, the ∼ -PID rather than the δ -PID, because the δ -PID does not satisfy a basic property called additivity [19] (defined in Sec. 2); (ii) our current method provides an exact upper bound to the PID definition being computed, rather than an approximate upper bound; (iii) we now consider the problem of estimation, not just computation, and explore the issue of the bias of PID estimates; and (iv) our current method is much faster, and we demonstrate agreement with ground truth at much higher dimensionality.

Several other studies have also considered methods for efficiently estimating PIDs: Banerjee et al. [20] and Makkeh et al. [21] address computing discrete PIDs, but their method does not scale to higher dimensions; Pakman et al. [17] estimate PIDs using copulas, but their method would also potentially be computationally prohibitive at high dimensionalities; Liang et al. [22] use convex optimization to directly estimate the ∼ -PID for general high-dimensional distributions, but they do not compare with ground truth at high dimensionality or examine bias in their estimates.

2 Background: An Introduction to PIDs and the ∼ -PID

In this section, we provide an introduction to the concept of partial information decomposition along with an illustrative example. Let M , X and Y be three random variables with joint distribution P MXY . APID decomposes the total mutual information between the message M and two constituent random variables X and Y into a sum of four non-negative components that satisfy [18, 23]:

Here, I ( A ; B ) is the Shannon mutual information [24] between the random variables A and B , and the four terms in the RHS of (1) are respectively the information about M that is (i) uniquely present in X and not in Y ; (ii) uniquely present in Y and not in X ; (iii) redundantly present in both X and Y and can be extracted from either; and (iv) synergistically present in X and in Y , i.e., information which cannot be extracted from either of them individually, but can be extracted from their interaction. For the sake of brevity, we may also refer to these partial information components as UI X , UI Y , RI and SI respectively. Notwithstanding notation, they should all properly be understood to be functions of the joint distribution P MXY .

Now, UI X , UI Y , RI and SI consist of four undefined quantities, subject to the three equations in (1)-(3). In addition, they are typically assumed to be non-negative, RI and SI are each constrained to be symmetric in X and Y , and the functional forms of UI X and UI Y should be identical when exchanging X for Y . Despite the number of constraints, many definitions satisfy all of them, each differing in its motivation and interpretation [18, 23, 25-27] (see [27, 28] for a review), and we need to formally define one of these partial information components to determine the other three.

Example 1. Before we jump into a specific definition, we provide an intuition into what these terms mean using a simple example. Suppose M = [ A,B,C ] , X = [ A,B,C ⊕ Z ] , and Y = [ B,Z ] , where A,B,C,Z ∼ i.i.d. Ber(0.5). 1 Then, X has 1 bit of unique information about M , i.e., A ; Y has no unique information; X and Y both have 1 bit of redundant information, i.e., B , since it can be obtained from either X or Y ; and X and Y have 1 bit of synergistic information, i.e., C , which cannot be obtained from either X or Y individually (since C ⊕ Z ⊥ ⊥ C ), but can only be recovered when both X and Y are known. For more examples on binary variables, we refer the reader to [18].

In this manuscript, we consider a definition that we refer to as the ∼ -PID 2 [18, 25], which is defined below. We chose to build an estimator for this definition for two reasons: (i) it is a Blackwellian PID definition, i.e., it has well-defined operational interpretations based on concepts from statistical decision theory (e.g., see [18, 27, 29] for details); and (ii) it satisfies many desirable properties (e.g., see [18, 30]), and in particular, a property that we call additivity of independent components .

Definition 1 ( ∼ -PID [18]) . The unique information about M present in X and not in Y is given by

where ∆ P := { Q MXY : Q MX = P MX , Q MY = P MY } and I Q ( · ; · | · ) is the conditional mutual information over the joint distribution Q MXY . The remaining ∼ -PID components, ˜ UI ( M : Y \ X ) , ˜ RI ( M : X ; Y ) and ˜ SI ( M : X ; Y ) , follow from equations (1) -(3) .

Property 1 (Additivity of independent components) . Suppose M = [ M 1 , M 2 ] , X = [ X 1 , X 2 ] , and Y = [ Y 1 , Y 2 ] , such that ( M 1 , X 1 , Y 1 ) ⊥ ⊥ ( M 2 , X 2 , Y 2 ) . Then, additivity implies that

and similarly for the other three partial information components, UI Y , RI and SI .

Property 1 stipulates that we should be able to compute the PIDs of two independent systems separately , and then add the components across both systems. In effect, additivity implies that the PID of an isolated system should not depend on the PID of another isolated system, making it an intuitive and highly desirable property (see App. A.1 for concrete examples). Of the many PID definitions examined by Rauh et al. [19], only the ∼ -PID satisfied additivity (as proved in [18]).

3 Computing the ∼ -PID for Gaussian Distributions

The first contribution of this paper is a method to efficiently compute bounds on the ∼ -PID for jointly Gaussian random vectors M , X and Y . To be precise, our method computes an upper bound for ˜ UI X and ˜ UI Y , and lower bounds for ˜ RI and ˜ SI . Similar to our earlier work [12], we present a new PID definition that we call the ∼ G -PID, which characterizes an upper bound on the unique information of the ∼ -PID by restricting the optimization space to jointly Gaussian Q MXY :

Definition 2 ( ∼ G -PID) . Let P MXY be jointly Gaussian. Then, the unique information about M present in X and not in Y is given by

where ∆ P := { Q MXY : Q MXY jointly Gaussian, Q MX = P MX , Q MY = P MY } and I Q is the conditional mutual information over the joint distribution Q MXY .

If the optimal Q MXY in the unrestricted optimization of Definition 1 happens to be Gaussian for some P MXY , then the ∼ G -PID would be identical to the ∼ -PID for that P MXY . We conjecture

that this happens whenever P MXY is Gaussian: for example, in a similar optimization problem for computing the information bottleneck [31], the optimal distribution is Gaussian whenever P is Gaussian [32, 33]. We provide empirical evidence in favor of this conjecture through a sequence of examples with Gaussian P in Sec. 4, where we recover the ground truth even when Q is restricted to be Gaussian. However, we leave a theoretical examination of this conjecture for future work.

In practical terms, restricting the search space to Gaussian Q MXY reduces the number of optimization variables from being exponential in the dimensionality to quadratic (see Appendix A.2), allowing us to compute the ∼ G -PID for much higher dimensionalities of M , X and Y . In what follows, we show how the optimization problem for the ∼ G -PID can be written out in closed-form and then solved using projected gradient descent.

3.1 Notation and Preliminaries

Suppose M , X and Y are jointly Gaussian random vectors of dimensions d M , d X and d Y respectively, with a joint covariance matrix given by Σ MXY . We will make extensive use of the submatrices of Σ MXY , so we explain their notation here:

Σ XY will denote the ( d X + d Y ) × ( d X + d Y ) joint (auto-)covariance matrix of the vector [ X T , Y T ] T .

Σ X,Y (note the comma) will denote the d X × d Y cross-covariance matrix between X and Y .

Σ XY,M will denote the ( d X + d Y ) × d M cross-covariance matrix between the concatenated vector [ X T , Y T ] T and the vector M .

In general, groupings of vectors without commas represent joint covariance, while a comma represents a cross-covariance between the groups on either side of the comma. The same notation will also be used for conditional covariance matrices: for example, Σ XY | M is the conditional joint covariance of ( X,Y ) given M , while Σ X,Y | M is the conditional cross -covariance between X and Y given M .

We will also use an equivalent notation for the joint distribution [12], where P MXY is parameterized as a 'broadcast channel' [24, Ch. 15.6] from M to X and Y :

Here, H X := Σ X,M and H Y := Σ Y,M represent channel gain matrices , while N X and N Y represent additive noise and are not necessarily independent of each other: [ N T X , N T Y ] T ∼ N (0 , Σ XY | M ) .

Remark 1. Without loss of generality, we can assume that M , X and Y are all zero-mean, and that Σ M = I . Further, we explicitly assume that the X and Y channels are individually whitened, i.e., that Σ X | M = I and Σ Y | M = I . This assumption precludes deterministic relationships between M and X or Y , and is required to ensure that information quantities remain finite [12].

3.2 Optimizing the Union Information

Bertschinger et al. [18] showed that the minimizer for the unique information is also the minimizer for the 'union information', I ∪ ( M : X ; Y ) := UI X + UI Y + RI . In other words, we can also solve the following optimization problem, which yields simpler expressions for the objective and gradient:

Now, suppose P MXY is Gaussian with covariance Σ P MXY and the solution Q MXY is also assumed to be Gaussian with covariance Σ Q MXY . Then, the constraint in (8) implies that Σ Q MX = Σ P MX and Σ Q MY = Σ P MY . In other words, Σ M , Σ X , Σ Y , and Σ M,XY are all constant across P and Q . Therefore, the only part of Σ Q MXY that is variable is Σ Q X,Y , or equivalently, Σ Q X,Y | M . 3 In what follows, we will drop the superscripts denoting the distribution, as this will be clear from context. Generally speaking, we will discuss the optimization problem and thus the distribution will be Q .

Proposition 1. The union information for the ∼ G -PID of Definition 2 is given by

where the optimization variable Σ X,Y | M is an off-diagonal block embedded within Σ XY | M ; all other matrices in the objective are constants that are derived from Σ P MXY .

We solve the above optimization problem using projected gradient descent: we analytically derive the gradient and the projection operator for the constraint set as shown below. Then, we use the RProp algorithm [34, 35] for gradient descent, which independently adjusts the learning rates for each optimization parameter (derivations, and details of implementation and complexity are in App. A). Code for our implementation is available on GitHub [36], and details of the compute configuration are given in App. E.

Proposition 2. The objective in Proposition 1 can be simplified to

The gradient of the objective with respect to Σ X,Y | M is given by

A projection operator on to the constraint set Σ XY | M ≽ 0 can be obtained as follows: let Σ XY | M =: V Λ V T be the eigenvalue decomposition of Σ XY | M , with Λ =: diag ( λ i ) . Let λ i := max (0 , λ i ) represent the rectified eigenvalues, and Λ := diag ( λ i ) . Then, define

where Σ X | M , Σ Y | M and Σ X,Y | M are submatrices of Σ XY | M .

4 Canonical Gaussian Examples

In this section, we show how well our ∼ G -PID estimator performs on a series of Gaussian examples of increasing complexity, which have known ground truth. Barrett [13] showed that, for Gaussian distributions, the ∼ -PID reduces to the MMI-PID (defined below), whenever M is scalar. These also happen to be cases when the optimal distribution Q MXY is Gaussian [12], and thus the ∼ G -PID should recover the ground truth. We then leverage additivity (Property 1) to combine two or more simple examples into complex ones, where ground truth continues to be known.

Definition 3 (Minimum Mutual Information (MMI) PID) . Let the redundant information be defined as the minimum of the two mutual informations:

The remaining MMI-PID components, UI MMI ( M : X \ Y ) , UI MMI ( M : Y \ X ) and SI MMI ( M : X ; Y ) , follow from equations (1) -(3) .

We first provide a Gaussian analog of Example 1 in Examples 2-4 (for d M = d X = d Y = 1 ). We will use the channel notation described in Equation (7). Complete derivations for these examples (and some nuances that are omitted here) are presented in App. B.

Example 2 (Pure uniqueness: variable A from Example 1) . Suppose M ∼ N (0 , 1) , H X = 1 and H Y = 0 , with N X , N Y ∼ i.i.d. N (0 , 1) . Here, only X receives information about M , while Y is pure noise. Thus, X has unique information about M ( UI X = I ( M ; X ) > 0 ), with no unique information in Y , and no redundancy or synergy ( UI Y = RI = SI = 0 ).

Example 3 (Pure redundancy: variable B from Example 1) . Ideally, we would set M ∼ N (0 , 1) , X = M and Y = M . However, for continuous random variables, I ( M ; X ) = ∞ when M = X . So instead, we set M ∼ N (0 , 1) , H X = 1 and H Y = 1 , with N X ∼ N (0 , 1) while N Y = N X (i.e., X = Y , so they are both the same noisy version of M ). In this case, X and Y are fully redundant since they both contain exactly the same information about M . Thus, RI = I ( M ; ( X,Y )) > 0 , while UI X = UI Y = SI = 0 .

Example 4 (Pure synergy: variable C from Example 1) . We cannot replicate pure synergy for Gaussian variables, but we can approach it in a limit. Let M ∼ N (0 , 1) , H X = 1 and H Y = 0 , with N X ∼ N (0 , σ 2 ) and N Y = N X (i.e., X = M + Y ). Further, let σ 2 →∞ . In this case, I ( M ; Y ) = 0

|

PIDs for Canonical Gaussian Examples

0.00

1.00

1.78

3.16

5.62

10.00

17.78

31.62

56.23

100.00

Noise in

Y

given

X

,

σY

X

0.0

0.5

1.0

1.5

2.0

2.5

Partial information (bits)

Unique and Redundant

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0.99

Noise correlation,

ρ

Unique and Synergistic

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0.99

Noise correlation,

ρ

Redundant and Synergistic

PID component

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

PID definition

∼

G

-PID

δG

-PID

MMI-PID

Ground truth

0.00

0.33

0.67

1.00

1.33

1.67

2.00

2.33

2.67

3.00

Gain in

X

1

,

α

0.0

0.5

1.0

1.5

2.0

2.5

Partial information (bits)

Increasing Gain in

X

1

0.00

0.33

0.67

1.00

1.33

1.67

2.00

2.33

2.67

3.00

Gain in

X

1

,

α

10

15

10

13

10

11

10

9

10

7

10

5

10

3

10

1

Error (bits)

Error in gain example

0.00

0.10

0.14

0.20

0.28

0.40

0.56

0.79

1.11

1.57

Angle,

θ

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Partial information (bits)

Rotation of

X

w.r.t.

Y

PID component

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

PID definition

∼

G

-PID

δG

-PID

MMI-PID

Ground truth

and I ( M ; X ) → 0 as σ 2 →∞ , so X and Y individually convey little to no information about M . However, we can recover information about M from X and Y together by taking their difference, since X -Y = M . Thus, SI > 0 , while UI Y = RI = 0 and UI X → 0 .

Examples 2, 3 and 4 have been provided solely for intuition. Their PIDs can be inferred directly from Equations (1)-(3). We next describe three one-dimensional examples that each have two non-zero PID components. For lack of space, we only provide a brief description and defer details to App. B. We estimate the ∼ G -PID (as well as the δ G -PID [12] and the ground-truth MMI-PID [13]) for these examples and show that all three are equal (see Fig. 1).

Example 5 (Unique and redundant information) . Let X be a noisy representation of M , and let Y be a noisy representation of X with standard deviation σ Y | X . When Y = X (zero noise), this example reduces to Example 3. As σ Y | X →∞ , RI reduces while UI X approaches I ( M ; X ) .

Example 6 (Unique and synergistic information) . Let M ∼ N (0 , 1) , H X = 1 , H Y = 0 and N X , N Y ∼ N (0 , σ 2 ) such that their correlation is ρ . When σ 2 is finite and ρ = 0 , this example reduces to Example 2, since there can be no synergy between X and Y . As ρ → 1 , X -Y → M ; so the total mutual information I ( M ; ( X,Y )) →∞ , driven by synergy growing unbounded, while the unique component remains constant at I ( M ; X ) .

Example 7 (Redundant and synergistic information) . Let M ∼ N (0 , 1) , H X = H Y = 1 and N X , N Y ∼ N (0 , 1) such that their correlation is ρ . When ρ < 1 , I ( M ; X ) and I ( M ; Y ) are both equal by symmetry, and thus equal to RI (see Def. 3 for the MMI-PID, which is ground truth here). As ρ reduces, the two channels X and Y have noisy representations of M with increasingly independent noise terms. Averaging the two, ( X + Y ) / 2 , will provide more information about M than either one of them individually (i.e., synergy), and thus SI increases as ρ reduces.

Stability of the

-PID over increasing dimensionality

Timing analysis

2

4

8

16

32

64

128

256

512

1024

0.25

0.5

1

2

4

8

16

32

64

128

256

512

1024

I

(

M

; (

X,Y

))

2

4

8

16

32

64

128

256

512

1024

UIX

2

4

8

16

32

64

128

256

512

1024

UIY

2

4

8

16

32

64

128

256

512

1024

RI

2

4

8

16

32

64

128

256

512

1024

SI

∼

G

Dimension of each of

M

,

X

and

Y

:

d

=

dM

=

dX

=

dY

Partial Information (bits)

2

4

8

16

32

64

d

10

2

10

1

10 0

10 1

10 2

Time taken (seconds)

∼

G

-PID

δG

-PID

The next set of examples will use the examples presented above in different combinations. This ensures that, where possible, the ground truth remains known in accordance with Property 1. These examples are also designed to reveal the differences between the ∼ -PID, the MMI-PID and the δ -PID: in particular, they show how the MMI-PID and the δ -PID fail where the ∼ -PID does not. These examples use two-dimensional M , X and Y , i.e., ( d M , d X , d Y ) = (2 , 2 , 2) . A diagrammatic representation of Examples 8 and 9 is given in App. B.2.

Example 8. Let X 1 = αM 1 + N X, 1 , Y 1 = M 1 + N Y, 1 , X 2 = M 2 + N X, 2 and Y 2 = 3 M 2 + N Y, 2 , where M 1 , M 2 , N X,i , N Y,i ∼ i.i.d. N (0 , 1) , i = 1 , 2 . Here, ( M 1 , X 1 , Y 1 ) is independent of ( M 2 , X 2 , Y 2 ) , therefore using Property 1, we can add the PID values from their individual decompositions (which each have known ground truth via the MMI-PID since M 1 and M 2 are scalar). Fig. 2(l) compares the ∼ G -PID, the δ G -PID and the MMI-PID for the joint decomposition of I ( M ; ( X,Y )) , at different values of α , the gain in X 1 . Only the ∼ G -PID matches the ground truth, as it is the only definition here that is additive.

Example 9. Let M and Y be as in Example 8. Suppose X = H X R ( θ ) M , where H X is a diagonal matrix with diagonal entries 3 and 1, and R ( θ ) is a 2 × 2 rotation matrix that rotates M by an angle θ . When θ = 0 , X has higher gain for M 1 while Y has higher gain for M 2 . When θ increases to π/ 2 , X and Y have equal gains for both M 1 and M 2 (barring a difference in sign). Since ( M 1 , X 1 , Y 1 ) is not independent of ( M 2 , X 2 , Y 2 ) for all θ , we know the ground truth only at the end-points. Nonetheless, the example shows a difference between the three definitions, as shown in Fig. 2(r).

Example 10. In this example, we test the stability of the ∼ G -PID as the dimensionality, d := d M = d X = d Y increases. By Property 1, if we take two i.i.d. systems of variables ( M,X,Y ) at dimensionality d and concatenate their respective variables, every PID component of the composite system of dimensionality 2 d should be double that of the original. This process can be repeated, taking two independent 2 d -dimensional systems and concatenating them to create a 4 d -dimensional system. Fig. 3 shows precisely this process starting with the system in Example 8 with d = 2 , and continually doubling its size until d = 1024 . The ∼ G -PID accurately matches ground truth by doubling in value, and remains stable with small relative errors (shown in App. B.5). The ∼ G -PID also runs much faster than the δ G -PID on this example, providing a speed-up of more than 1000 × at d = 64 (see right-most plot in Fig. 3, and extended results in App. B.4).

Remark 2. In the above examples, the ground truth referred to the ∼ -PID of Def. 1, since it was inferred using Def. 3 and additivity (Property 1). Since our method for computing the ∼ G -PID recovers the ground truth ∼ -PID, we can infer that the distribution of the optimal Q MXY for the ∼ -PID is in fact Gaussian in these examples. This provides empirical evidence in support of the conjecture stated in Sec. 3.

5 Estimation and Bias-correction for the ∼ G -PID

Having discussed how to compute the ∼ G -PID and shown that it agrees well with ground truth in several canonical examples, we discuss how the ∼ G -PID may be estimated from data. Given a sample

100

200

300

600

1000

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Partial information value (bits)

Bias correction in the "Bit-of-all" set up

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Partial information value (bits)

Bias correction in the "Fully-redundant" set up

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

Sample size

Sample size

of n realizations of M , X and Y drawn from P MXY , we may estimate the sample joint covariance matrix ˆ Σ MXY . We therefore use the straightforward 'plug-in' estimator for the ∼ G -PID, by using ˆ Σ MXY in place of Σ MXY in the optimization problem in equation (9).

However, it is well-known that estimators of information-theoretic quantities suffer from large biases for moderate sample sizes [37]. Cai et al. [38] characterized the bias in the entropy of a d -dimensional Gaussian random vector, for a fixed sample size n .

Proposition 3 (Bias in Gaussian entropy [38]) . Suppose M ∈ R d M has an auto-covariance matrix Σ M . The entropy of M is H ( M ) = 1 2 log det(2 πe Σ M ) when Σ M is known [24]. For the sample covariance matrix ˆ Σ M , the bias is given by:

For a proof, we refer the reader to [38, Corollary 2]. This result may be naturally extended to compute the bias of each of the mutual information quantities in the LHS of equations (1)-(3):

Corollary 4 (Bias in Gaussian mutual information) . For the joint mutual information I ( M ; ( X,Y )) ,

This follows directly from the fact that I ( M ; ( X,Y )) = H ( M )+ H ( X,Y ) -H ( M,X,Y ) . Similarly, we can compute the bias of ˆ I ( M ; X ) and ˆ I ( M ; Y ) . But this does not uniquely determine the bias in the individual PID components, and as with defining PIDs, we need to decompose the bias in Corollary 4 across the four PID components such that they agree with these constraints. We solve this problem by defining a bias-corrected version of the union information from Proposition 1.

Definition 4 (Bias-corrected Union Information) . We assign the bias in the union information to be the same fraction as the bias in the joint mutual information I ( M ; ( X,Y )) . This gives rise to a bias-corrected estimate of the union information:

We do not analyze theoretically whether the bias-corrected union information is consistent and unbiased, thus, it may still have some residual bias relative to the true union information. However, we find empirically that this bias correction process works reasonably well and appears both consistent and unbiased, in a number of examples. Figure 4 shows biased and bias-corrected PID values for

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

to

X

0.0

0.2

0.4

0.6

0.8

Partial information (bits)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

to

X

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Normalized partial information

PID component

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

PID definition

∼

G

-PID

δG

-PID

MMI-PID

Ground truth

Poisson

∼

-PID

PID values (left) and PID values normalized by

I

(

M

; (

X,Y

))

(right)

in a multivariate Poisson spike-count simulation

1

1

0

50

100

150

200

Time after stimulus onset (ms)

0

2

4

6

8

10

12

14

16

Redundancy,

RI

(VISp : VISl; VISal)

(bits)

****

ns

****

****

****

***

****

****

****

ns

***

ns

****

Redundancy betw 3 visual cortical areas

(top 20 principal components)

Change

Non-change

0

50

100

150

200

Time after stimulus onset (ms)

0.4

0.6

0.8

1.0

1.2

RI

(VISp : VISl; VISal)

/I

(VISp; (VISl

,

VISal))

****

ns

****

****

****

*

****

****

****

ns

ns

ns

****

Redundancy fraction betw 3 visual cortical areas

(top 20 principal components)

Change

Non-change

100 runs of two configurations called 'Bit-of-all' (with a little bit of each PID component) and 'Fully-redundant' (which has predominantly redundancy), with d M = d X = d Y = 10 (details and additional setups in App. C). We find that bias correction brings the PID values closer to their true values even at small sample sizes. In App. C.3, we also include a preliminary analysis of the variance of PID estimates using bootstrap [39, Ch. 8].

6 Application to Simulated and Real Neural Data

So far, we have only considered the ∼ G -PID when applied to Gaussian P MXY . Although Def. 2 strictly applies only to Gaussian P MXY , the estimation process in Sec. 5 relies only on a sample covariance matrix, which is well-defined for a wide variety of non-Gaussian distributions. Many applications where PIDs could be useful have non-Gaussian data. For instance, there is great interest in applying PIDs in neuroscience (e.g., to understand how multiple brain regions jointly encode and communicate information [6, 40]), but spike-count distributions are non-Gaussian.

Simulated neural data. To show that our ∼ G -PID estimates provide reasonable results on nonGaussian spiking neural data, we first simulate spike-count data using Poisson random variables (following [12]; described in App. D.1). We evaluate the ground truth ∼ -PID for this distribution using the discrete PID estimator of Banerjee et al. [20]. The ∼ G -PID is estimated from a sample covariance matrix using 10 6 realizations of M , X and Y . We find that the ∼ G -PID closely matches

the ground truth for a range of parameter values, despite the fact that the ∼ G -PID is effectively computed on a Gaussian approximation of a Poisson distribution (Fig. 5). More examples of the ∼ G -PID applied to non-Gaussian data are provided in App. D.2 (also see App. A.2). We conclude that it is reasonable to use and interpret the ∼ G -PID on non-Gaussian spike count data.

Real neural data. We then applied our bias-corrected ∼ G -PID estimator to the Visual Behavior Neuropixels dataset collected by us at the Allen Institute [41]. We recorded over 80 mice using six neuropixels probes targeting various regions of visual cortex, while the mice were engaged in a visual change-detection task. In the task, images from a set of 8 natural scenes were presented in 250 ms flashes, at intervals of 750 ms; the image would stay the same for a variable number of flashes after which it would change to a new image. The mouse had to lick to receive a water reward when the image changed. Thus, a given image flash could be a behaviorally relevant target if the previous image was different, or not, if the previous image was the same.

We used our bias-corrected PID estimator to understand how information is processed along the visual hierarchy during this task. We estimated the ∼ G -PID to understand how information contained in the spiking activity of primary visual cortex (VISp) was represented in two higher-order visual cortical areas, VISl and VISal. We aligned trials to the onset of a stimulus flash, binned spikes in 50 ms intervals and considered the top 20 principal components (to achieve reasonable estimates at these sample sizes; explained further in App. D.5) from each region in each time bin. We computed the ∼ G -PID on the sample covariance matrix of these principal components (shown in Fig. 6). We found that there was a significantly larger amount of redundant information about VISp activity between VISl and VISal for stimulus flashes corresponding to an image change, compared to flashes that were not changes (Fig. 6(l)). The larger redundancy was also sustained slightly longer for flashes corresponding to changes, than non-change flashes. Both of these effects were maintained even when the redundancy was normalized by the joint mutual information, suggesting that the effect was not purely due to an increase in the total amount of information (Fig. 6(r)). Our results suggest that the visual cortex propagates information throughout the hierarchy more robustly when such information is relevant for behavior.

7 Discussion

In this paper, we proposed a new and efficient method for estimating the ∼ G -PID for Gaussian distributions. We showed that our method recovers the ground truth and suitably corrects for bias through a series of examples. In particular, Fig. 4 showed how large the biases can be at small sample sizes, which makes bias correction particularly important in neuroscientific settings where sample sizes are often small.

We focused on Gaussian distributions, as they have historically been a starting point for many estimators (e.g., correlation is used as a measure of dependence, but zero correlation implies independence only in the Gaussian case). We were able to show ground-truth validation for our ∼ G -PID estimator at high dimensionalities only thanks to the existence of closed-form results on scalar Gaussians.

While our central claims and results applied to Gaussian distributions, our method for computing PIDs did not immediately break down for distributions close to Gaussian in some limit (e.g., Poisson). The effective spiking rate used in our multivariate Poisson distribution was also not particularly large (see App. D.1); we would expect our estimates to improve for higher firing rates, since the Poisson distribution will then be more Gaussian.

Limitations. Our work has several limitations that require further theory and simulations to resolve, the most important of which are: (1) Our estimator is technically a bound on the PID values because we assume Gaussian optimality in Definition 2; (2) Our bias-correction method is heuristic: we do not provide a rigorous theoretical characterization of the bias of PID values.

Broader impacts. Our work is mainly methodological, so the scope for negative impacts depends on how the methods might be used. For example, incorrect interpretations drawn from the use of our PID estimators may affect scientific conclusions. In particular, the PID is inherently a correlational quantity and carries the same caveats: it may not be appropriate to make causal interpretations on the basis of observed PID values. Also, despite our best efforts to explore a variety of systems, we cannot tell how accurate our bias-correction method will be in novel configurations.

Acknowledgments and Disclosure of Funding

We thank Łukasz Ku´ smierz for providing a valuable reference on the bias of Shannon entropy estimates. We also thank Gabe Schamberg and Christof Koch for helpful discussions.

We wish to thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement, and support. Praveen Venkatesh was supported by a Shanahan Family Foundation Fellowship at the Interface of Data and Neuroscience, supported in part by the Allen Institute. Stefan Mihalas was in part supported by NSF 2223725, NIH R01EB029813, and RF1DA055669 grants.

References

Saskia EJ de Vries, Jerome A Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience , 23 (1):138-151, 2020.

Joshua H Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jennifer A Luviano, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature , 592(7852):86-92, 2021.

Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth D Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science , 364(6437): eaav7893, 2019.

Elad Schneidman, William Bialek, and Michael J Berry. Synergy, redundancy, and independence in population codes. Journal of Neuroscience , 23(37):11539-11553, 2003.

Itay Gat and Naftali Tishby. Synergy and redundancy among brain cells of behaving monkeys. Advances in neural information processing systems , 11, 1998.

Giuseppe Pica, Eugenio Piasini, Houman Safaai, Caroline Runyan, Christopher Harvey, Mathew Diamond, Christoph Kayser, Tommaso Fellin, and Stefano Panzeri. Quantifying how much sensory information in a neural code is relevant for behavior. Advances in Neural Information Processing Systems , 30, 2017.

Giuseppe Pica, Mohammadreza Soltanipour, and Stefano Panzeri. Using intersection information to map stimulus information transfer within neural networks. BioSystems , 185:104028, 2019.

Jan Bím, Vito De Feo, Daniel Chicharro, Malte Bieler, Ileana L Hanganu-Opatz, Andrea Brovelli, and Stefano Panzeri. A non-negative measure of feature-related information transfer between neural signals. BioRxiv , page 758128, 2019.

Tomas Scagliarini, Luca Faes, Daniele Marinazzo, Sebastiano Stramaglia, and Rosario N Mantegna. Synergistic information transfer in the global system of financial markets. Entropy , 22(9):1000, 2020.

Patricia Wollstadt, Sebastian Schmitt, and Michael Wibral. A rigorous information-theoretic definition of redundancy and relevancy in feature selection based on (partial) information decomposition. arXiv preprint arXiv:2105.04187 , 2021.

Sanghamitra Dutta, Praveen Venkatesh, Piotr Mardziel, Anupam Datta, and Pulkit Grover. An informationtheoretic quantification of discrimination with exempt features. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3825-3833, 2020.

Praveen Venkatesh and Gabriel Schamberg. Partial information decomposition via deficiency for multivariate Gaussians. In 2022 IEEE International Symposium on Information Theory (ISIT) , pages 2892-2897. IEEE, 2022.

Adam B Barrett. Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Physical Review E , 91(5):052802, 2015.

Nigel Colenbier, Frederik Van de Steen, Lucina Q Uddin, Russell A Poldrack, Vince D Calhoun, and Daniele Marinazzo. Disambiguating the role of blood flow and global signal with partial information decomposition. Neuroimage , 213:116699, 2020.

Tjeerd W Boonstra, Luca Faes, Jennifer N Kerkman, and Daniele Marinazzo. Information decomposition of multichannel EMG to map functional interactions in the distributed motor system. NeuroImage , 202: 116093, 2019.

Jana Krohova, Luca Faes, Barbora Czippelova, Zuzana Turianikova, Nikoleta Mazgutova, Riccardo Pernice, Alessandro Busacca, Daniele Marinazzo, Sebastiano Stramaglia, and Michal Javorka. Multiscale information decomposition dissects control mechanisms of heart rate variability at rest and during physiological stress. Entropy , 21(5):526, 2019.

Ari Pakman, Amin Nejatbakhsh, Dar Gilboa, Abdullah Makkeh, Luca Mazzucato, Michael Wibral, and Elad Schneidman. Estimating the unique information of continuous variables. Advances in neural information processing systems , 34:20295-20307, 2021.

Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy , 16(4):2161-2183, 2014.

Johannes Rauh, Pradeep Kr Banerjee, Eckehard Olbrich, Guido Montúfar, and Jürgen Jost. Continuity and additivity properties of information decompositions. arXiv preprint arXiv:2204.10982 , 2022.

Pradeep Kr Banerjee, Johannes Rauh, and Guido Montúfar. Computing the unique information. In 2018 IEEE International Symposium on Information Theory (ISIT) , pages 141-145. IEEE, 2018.

Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. BROJA-2PID: A robust estimator for bivariate partial information decomposition. Entropy , 20(4):271, 2018.

Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling feature interactions: An information decomposition framework. arXiv preprint arXiv:2302.12247 , 2023.

Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 , 2010.

Thomas M Cover and Joy A Thomas. Elements of Information Theory . John Wiley & Sons, 2012.

Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. In Guided self-organization: inception , pages 159-190. Springer, 2014.

Malte Harder, Christoph Salge, and Daniel Polani. Bivariate measure of redundant information. Physical Review E , 87(1):012130, 2013.

Artemy Kolchinsky. A novel approach to the partial information decomposition. Entropy , 24(3):403, 2022.

Joseph T Lizier, Nils Bertschinger, Jürgen Jost, and Michael Wibral. Information decomposition of target effects from multi-source interactions: perspectives on previous, current and future work. Entropy , 20(4): 307, 2018.

Praveen Venkatesh, Keerthana Gurushankar, and Gabriel Schamberg. Capturing and interpreting unique information. arXiv preprint arXiv:2302.11873 , 2023.

Pradeep Kr Banerjee, Eckehard Olbrich, Jürgen Jost, and Johannes Rauh. Unique informations and deficiencies. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 32-38. IEEE, 2018.

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for Gaussian variables. Advances in Neural Information Processing Systems , 16, 2003.

Amir Globerson and Naftali Tishby. On the optimality of the Gaussian information bottleneck curve. The Hebrew University of Jerusalem, Tech. Rep , page 22, 2004.

Martin Riedmiller and Heinrich Braun. A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In IEEE International Conference on Neural Networks , pages 586-591. IEEE, 1993.

Geoffrey Hinton. Coursera neural networks for machine learning, lecture 6, 2018. URL ❤/a116/a116♣/a115✿✴✴ ✇✇✇✳❝/a115✳/a116♦/a114♦♥/a116♦✳❡❞✉✴⑦/a116✐❥♠❡♥✴❝/a115❝✸✷✶✴/a115❧✐❞❡/a115✴❧❡❝/a116✉/a114❡❴/a115❧✐❞❡/a115❴❧❡❝✻✳♣❞❢ . Also see ❤/a116/a116♣/a115✿✴✴ ♦♣/a116✐♠✐③❛/a116✐♦♥✳❝❜❡✳❝♦/a114♥❡❧❧✳❡❞✉✴✐♥❞❡①✳♣❤♣❄/a116✐/a116❧❡❂❘▼❙/a80/a114♦♣ .

Code for this paper. URL ❤/a116/a116♣/a115✿✴✴❣✐/a116❤✉❜✳❝♦♠✴♣/a114❛✈❡❡♥✈✷✺✸✴❣♣✐❞ .

Liam Paninski. Estimation of entropy and mutual information. Neural computation , 15(6):1191-1253, 2003.

T Tony Cai, Tengyuan Liang, and Harrison H Zhou. Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional Gaussian distributions. Journal of Multivariate Analysis , 137:161-172, 2015.

Larry Wasserman. All of statistics: a concise course in statistical inference , volume 26. Springer, 2004.

Nicholas M Timme and Christopher Lapish. A tutorial for information theory in neuroscience. eneuro , 5 (3), 2018.

Allen Institute. Visual behavior neuropixels dataset overview, 2022. URL ❤/a116/a116♣/a115✿✴✴♣♦/a114/a116❛❧✳❜/a114❛✐♥✲♠❛♣✳ ♦/a114❣✴❡①♣❧♦/a114❡✴❝✐/a114❝✉✐/a116/a115✴✈✐/a115✉❛❧✲❜❡❤❛✈✐♦/a114✲♥❡✉/a114♦♣✐①❡❧/a115 .

Volker Strassen. Gaussian elimination is not optimal. Numerische mathematik , 13(4):354-356, 1969.

Gaussian Partial Information Decomposition: Bias Correction and Application to High-dimensional Data

Appendices

Praveen Venkatesh, Corbett Bennett, Sam Gale, Tamina K. Ramirez, Greggory Heller, Séverine Durand, Shawn Olsen, Stefan Mihalas

A Supplementary Material for Sections 2 and 3

A.1 Implications of the Additivity Property

As mentioned in the main text, additivity states that the PID values of an isolated system should not depend on the PID values of another isolated system. Without additivity, it is not possible to examine independent systems in isolation, since broadening your view to include a different isolated system could change the PID values of the first system.

Hypothetically, suppose we have two separate individuals (labeled 1 and 2) receiving completely independent stimuli, and we examine the PID between the activity in brain regions M , X and Y in each of their brains. Then the total unique information that X 1 and X 2 have about M 1 and M 2 with respect to Y 1 and Y 2 should be equal to the sum of the unique information in each of the individuals taken separately.

As another example, suppose we are trying to examine visual information flow and auditory information flow in a multi-sensory integration task. We may want to understand the degree to which the activity in the sub-regions of the visual and auditory systems depend on each other. A reasonable null model of independence between the two systems would be that the PID values of the joint system will be equal to the sum of the PID values in the two individual systems. Then, measuring the actual degree to which the joint PID value is not equal to the sum will be a meaningful measure of dependence between the systems. This would only be possible with a PID definition that guaranteed additivity of independent sub-systems. However, since the δ -PID does not satisfy the additivity (Property 1), we cannot guarantee that the aforementioned null would be the correct null model, and we would not be able to perform such an analysis.

A.2 Explaining the Exponential Reduction in the Number of Optimization Variables

In Def. 2, we restrict Q MXY to be jointly Gaussian. This reduces the number of optimization variables from being exponential in the dimensionality to quadratic, as we show here.

Previous discrete ∼ -PID estimators [20, 21] have found efficient methods to solve the optimization problem of Def. 1 without reducing the number of optimization variables. If P MXY is discrete in Def. 1, and each dimension of M , X and Y has a support of size K (i.e., M i , X i and Y i can each take one of K discrete values), then the total number of degrees of freedom in ∆ P is O ( K ( d M + d X + d Y ) ) . This corresponds to the total support of a discrete Q MXY , which can be an arbitrarily complex discrete distribution.

In contrast, the ∼ G -PID in Def. 2 reduces the dimensionality of the optimization space by assuming that Q MXY is Gaussian. Q MXY is then completely parameterized by its covariance matrix, Σ Q MXY . In fact, given the additional constraints in Def. 2, the only part of Σ Q MXY that is variable is Σ Q X,Y , as explained in Sec. 3.2. This matrix has only d X × d Y variables. Thus, by restricting the optimization space to jointly Gaussian distributions, we have reduced the number of variables from exponential in the dimensionality to quadratic.

As a concrete example, consider a neuroscientific setting (as in Sec. 6) where M , X and Y represent the spiking activity of neurons in three different brain regions, so that d M , d X and d Y represent the number of neurons, and K represents the maximum number of spikes. Makkeh et al. [21] demonstrated their method for a maximum support size of K = 18 with d M = d X = d Y = 1 , for a total support of approximately 18 3 -1 = 5831 in Q MXY . If we assume a single neuron can have at most K = 10 spikes, then we get an effective dimensionality of log 10 (5831) < 4 . In other words,

these discrete ∼ -PID estimators can handle around 4 neurons across all three regions M , X and Y (which is also what we use in the simulation of Fig. 5 in Sec. 6), whereas our ∼ G -PID estimator can handle several hundreds if not thousands of neurons (as seen in Example 10).

A.3 Proofs of Propositions 1 and 2

Proof of Proposition 1. Firstly, the differential entropy of a Gaussian random variable M with covariance matrix Σ M is given by [43, Thm. 8.4.1]:

Secondly, for a joint Gaussian distribution P MXY parameterized by a covariance matrix Σ MXY , the conditional covariance matrix can be written as [44, Sec. 8.1.3]:

Using these two equations, we can derive the mutual information between M and ( X,Y ) as follows:

where in (a) we used equation (18), in (b) we used equation (20), and in (c) we used the fact that det( I + AB ) = det( I + BA ) .

The remainder of the proof follows from the arguments presented below equation (8). The constraint in Proposition 1 arises because, when optimizing over Σ X,Y | M , we require Σ MXY to be a valid positive semidefinite covariance matrix, i.e., Σ MXY ≽ 0 . This happens if and only if Σ M and its Schur complement in Σ MXY are both positive semidefinite, i.e., Σ M ≽ 0 and Σ M -Σ M,XY Σ -1 XY Σ T M,XY = Σ XY | M ≽ 0 .

Proof of Proposition 2. The proof is divided into three parts consisting of derivations for the objective, the gradient and the projection operator.

Objective. After whitening the P X | M and the P Y | M channels, and assuming that Σ M = I (see Remark 1), without loss of generality we have that

For the sake of brevity, let Σ represent the optimization variable Σ X,Y | M , and let S be its Schur complement in Σ XY | M , I -ΣΣ T . Then, the inverse of Σ XY | M can be written as [44, Sec. 9.1.5]:

Therefore, we get:

Thus, setting B := H X -Σ H Y , the optimization problem in Proposition 1 reduces to

Gradient. Let the objective derived in the previous section be called f (Σ) , where Σ := Σ X,Y | M as before. We can compute the gradient of f with respect to Σ using standard identities from matrix calculus. First, note that the gradient of a scalar function with respect to a matrix is itself a matrix with entries as follows:

Considering each element of this matrix:

where in (a), we have used the identity ∂ log det( X ) = Tr { X -1 ∂ ( X ) } [44, Sec. 2], while in (b), we use the fact that only B and S depend on Σ implicitly, with the other terms being constants.

Expanding the partial derivative alone, we get:

wherein

where J ij is the single-entry matrix , containing a 1 at location ( i, j ) and 0's everywhere else; in (b) and (d), we use the fact that ∂X/∂X ij = J ij [44, Sec. 9.7.6]; and in (c) we use the identity

∂ ( X -1 ) = X -1 ∂ ( X ) X -1 . Therefore, (40) becomes

Putting it all together, and letting A := ( I + H T Y H Y + B T S -1 B ) , (39) becomes

where in (e), we have used the fact that the trace of a matrix product is invariant under cyclic permutations of the matrices within the product.

Finally, using the fact that Tr { W T J ij } = Tr { WJ ij T } = W ij for any matrix W [44, Sec. 9.7.5],

Projection operator. Recall that the optimization variable, Σ := Σ X,Y | M is an off-diagonal block of Σ XY | M , which is the matrix upon which the constraint is defined:

wherein the diagonal blocks are identity due to Remark 1. For the purposes of this section, let us suppose Σ XY | M is a function of Σ , Σ XY | M =: g (Σ) , so that the constraint may be written as g (Σ) ≽ 0 . A suitable projection operator, therefore, will accept a value Σ 0 (that may violate g (Σ 0 ) ≽ 0 ) and find a point Σ proj close to it that satisfies the constraint, i.e., g (Σ proj ) ≽ 0 .

We do not find the 'orthogonal' projection operator, which has the minimum distance ∥ Σ proj -Σ 0 ∥ in some norm. Instead, we propose a simple heuristic to find a Σ proj which satisfies the constraint.

If Σ 0 satisfies the constraint, then we are done, so let us assume that g (Σ 0 ) / ≽ 0 . Then, we can find a matrix Σ XY | M which is close to g (Σ 0 ) and satisfies Σ XY | M ≽ 0 as follows: let the eigenvalue decomposition of g (Σ 0 ) be given by V Λ V T , with Λ =: diag( λ i ) being the diagonal matrix consisting of its eigenvalues λ i . Then, since g (Σ 0 ) is not positive semidefinite, ∃ i s.t. λ i < 0 . We set λ i := 0 for all such i ; effectively, λ i = max { 0 , λ i } ∀ i . We then reconstruct the matrix using these 'rectified' eigenvalues and set it to be Σ XY | M := V Λ V T , where Λ = diag( λ i ) .

Now, we need to find Σ such that g (Σ) = Σ XY | M . However, Σ XY | M may not have identity matrices on its diagonal blocks, i.e., it might not correspond to a whitened channel. We therefore whiten

Σ XY | M as follows:

where Σ X | M and Σ Y | M are the diagonal blocks of Σ XY | M . Crucially, since the matrix multiplying Σ XY | M on either side is itself (the inverse square-root of) a covariance matrix (and hence positive semidefinite), Σ whitened XY | M is also positive semidefinite.

Now, the off-diagonal block of Σ whitened XY | M will satisfy g ( · ) = Σ whitened XY | M ≽ 0 . This off-diagonal block forms the output of our projection operation and can be written as

which comes directly from equation (57).

A.4 Details of ∼ G -PID Optimization and RProp Implementation

The optimization problem for the ∼ G -PID, using projected gradient descent with RProp (mentioned in Section 3), is implemented as follows:

Let Σ := Σ X,Y | M be shorthand for the optimization variable, and let Proj( · ) represent the projection operator defined in Prop. 2. Let Σ ( i ) represent the value of Σ at iteration i of the optimization. Initialize Σ (0) = Proj( H X H + Y ) , where H + Y is the pseudoinverse of H Y .

Evaluate the objective and the gradient as defined in Prop. 2, at the current value of Σ ( i ) . Compute the sign of (each element of) the gradient,

When computing the objective and the gradient, add a small regularization term to the computation of S -1 (as defined in Prop. 2): S -1 = ( (1 + ϵ ) I -ΣΣ T ) -1 , where we take ϵ = 10 -7 .

Update:

where Proj is the projection operator defined in Prop. 2; η ( i ) is a time-varying learning rate vector of the same dimension as Σ , describing the learning rate for each element of Σ ; ⊙ represents an element-wise (or Hadamard) product between vectors; and α := 0 . 999 is a constant, which when raised to the power of i , imposes a slow overall decay of the learning rate to promote convergence.

The matrix inverses embedded in the projection operator are also regularized by modifying Equation (58) as follows:

where γ is slowly increased from 10 -12 , by a factor of 10 in each step, until g (Σ proj X,Y | M ) ≽ 0 .

η (0) is initialized to 10 -3 and η ( i ) is updated as follows:

where β := 0 . 9 is a constant that determines how fast the learning rate increases or decreases; and all operations are carried out element-wise. Note that when some element of the gradient changes in sign, that element of -ψ (Σ ( i +1) ) ⊙ ψ (Σ ( i ) ) will be positive, resulting in a decrease in that element of η ( i ) . On the other hand, if the sign of some element of the gradient remains the same, then the learning rate for that component will increase by a factor of 1 / 0 . 9 .

Stop when the absolute differences between the current objective and the previous objectives from the last 20 consecutive iterations are all less than 10 -6 ('patience'), or when the maximum number of iterations is exceeded (set to 10 4 iterations).

A.5 Computational Complexity of the ∼ G -PID

Suppose for simplicity that M , X and Y all have the same dimensionality d := d M = d X = d Y . Then, Σ MXY and all its submatrices have a side of dimensionality O ( d ) . Thus, the computational complexity of each gradient descent iteration is determined by the complexity of the matrix operations in equations (10)-(13). These operations include matrix multiplication, matrix inverses, log det( · ) and eigenvalue decomposition, which have a worst case complexity of O ( d 3 ) .

Matrix multiplication can potentially be performed at lower complexity for large matrix sizes (e.g., see Strassen [42]), which has downstream implications for each of the other operations as well. However, for our purposes, we take the computational complexity of the objective, the gradient and the projection operator to all be O ( d 3 ) .

B Supplementary Material for Section 4

First, observe that by subtracting equation (2) from equation (1), we have

Similarly, subtracting equations (1) and (3), we get that I ( M ; X | Y ) = UI X + SI . These two equations hold in general, and will be used in what follows.

B.1 Details and Derivations for Examples in Section 4

Example 2 (Pure uniqueness) .

Derivation of PID values in Example 2.

Example 3 (Pure redundancy) .

Derivation of PID values in Example 3.

Example 4 (Pure synergy) .

𝑌 2

𝑋2

𝑋1

𝑌 1

𝑚2

𝑚1

0

𝑌 2

𝑋2

𝑋1

𝑌 1

𝑚2

𝑚1

0

𝜃

Derivation of PID values in Example 4.

It should be noted that certain nuances have been omitted in discussing Examples 2-4 above. For instance, in Example 3, Σ XY | M is rank deficient and hence non-invertible, which would be an issue when computing the objective in Equation (9). Also, in Example 4, I ( M ; ( X,Y )) = ∞ , however this could be corrected by adding some noise to either X or Y so that their difference is a noisy representation of M .

Example 5 (Unique and redundant information) .

Derivation of PID values in Example 5. Essentially, X is a noisy representation of M , while Y is a noisy representation of X . Since M -X -Y forms a Markov chain, I ( M ; Y | X ) = 0 , and hence UI Y = SI = 0 . When σ 2 = 0 , this example reduces to Example 3 with only redundancy being present. For any finite non-zero value of σ 2 , both RI and UI X are present and are non-zero. Since M is scalar, the redundancy for the ∼ -PID is identical to the MMI-PID's redundancy [13]:

since I ( M ; Y ) < I ( M ; X ) , by the data processing inequality. At the limit when σ 2 → ∞ , I ( M ; Y ) → 0 , and therefore RI → 0 , while UI X will become equal to I ( M ; X ) .

Example 6 (Unique and synergistic information) .

Derivation of PID values in Example 6. When ρ = 1 and σ 2 →∞ , this example reduces to Example 4, with only synergy being present. In general, Y ⊥ ⊥ M , therefore, I ( M ; Y ) = UI Y + RI = 0 , meaning UI Y = RI = 0 . For any finite value of σ 2 , X will have some unique information about M given by UI X = I ( M ; X ) > 0 . Correspondingly, SI = I ( M ; X | Y ) -UI X =

0.00

0.33

0.67

0.99

1.33

1.67

2.00

2.33

2.67

3.00

Gain in

X

1

,

α

0

20

40

60

80

Partial information (bits)

∼

- vs.

δ

-PID at

dM

=

dX

=

dY

=64

∼

G

-PID

δG

-PID

I ( M ; X | Y ) -I ( M ; X ) . When ρ = 0 , I ( M ; X | Y ) = I ( M ; X ) and therefore SI = 0 . As ρ → 1 , X -Y → M ; so the total mutual information I ( M ; ( X,Y )) → ∞ , driven by synergy growing unbounded, while the unique component remains finite at I ( M ; X ) .

Example 7 (Redundant and synergistic information) .

Derivation of PID values in Example 7. When ρ = 1 , we once again reduce to Example 3 with only redundancy. When ρ < 1 , we cannot infer the PID values using Equation (1) and non-negativity alone, since none of the individual mutual information values (or conditional mutual information values) go to zero. Instead, we can determine the redundancy using the MMI-PID since M is scalar. Note that I ( M ; X ) and I ( M ; Y ) are both equal by symmetry, and thus equal to RI . This also implies that both UI X and UI Y must be equal to zero. As ρ reduces, the two channels X and Y have noisy representations of M with increasingly independent noise terms. Therefore, their average, ( X + Y ) / 2 will be more informative about M than either one of them individually, meaning that X and Y jointly contain more information than any one individually. This extra information about M is synergistic, given by SI = I ( M ; X | Y ) , and increases as ρ decreases, attaining its maximum possible value at ρ = 0 .

B.2 Diagrams Explaining Examples 8 and 9

Examples 8 and 9 can be understood diagrammatically as shown in Fig. 7(l) and Fig. 7(r), respectively. In both diagrams, we represent the two-dimensional plane describing M , with axes m 1 and m 2 . The colored vectors shown on this plane represent H X and H Y , i.e., the gain with which X and Y represent each value of M . For example, Y 2 captures only M 2 , with a gain corresponding to its length. The gains are directly representative of the signal-to-noise ratio (and hence the amount of information) in each variable, since the noise in each variable is i.i.d., with unit variance. In Example 8, the gain in X 1 is variable, while in Example 9, the angle at which X 1 and X 2 sample M 1 and M 2 is variable.

B.3 Comparison of the ∼ G - and δ G -PIDs in Example 8 at Higher Dimensionality

For completeness, we also present a comparison between the ∼ G - and δ G -PIDs for Example 8 at a higher dimensionality of d = 64 . The results can be found in Fig. 8, and show that both PIDs scale proportionally. We use a gain of α = 0 . 99 in place of α = 1 , since a gain of unity causes certain matrices to become ill-conditioned.

Although the δ G -PID does not obey the additivity property in general (which is why it does not agree with the ∼ G -PID at gains of α > 1 ), it appears to double along with the ∼ G -PID (i.e., additivity across identical independent copies appears to hold, at least in this specific example).

2

4

8

16

32

64

128

256

512

1024

d

, Dimension of

M

,

X

and

Y

10

1

10 0

10 1

10 2

Time taken (seconds)

Timing analysis of the ∼ G -PID for the doubling example

Absolute error in the ∼ G -PID over increasing dimensionality

2

4

8

16

32

64

128

256

512

1024

10 -15

10 -13

10 -11

10 -9

10 -7

10 -5

I

(

M

; (

X,Y

))

2

4

8

16

32

64

128

256

512

1024

UIX

2

4

8

16

32

64

128

256

512

1024

UIY

2

4

8

16

32

64

128

256

512

1024

RI

2

4

8

16

32

64

128

256

512

1024

SI

Abs. error in PID value (bits)

M

Dimension of each of

,

X

and

Y

Relative error in the ∼ G -PID over increasing dimensionality

2

4

8

16

32

64

128

256

512

1024

10 -14

10 -12

10 -10

10 -8

I

(

M

; (

X,Y

))

2

4

8

16

32

64

128

256

512

1024

UIX

2

4

8

16

32

64

128

256

512

1024

UIY

2

4

8

16

32

64

128

256

512

1024

RI

2

4

8

16

32

64

128

256

512

1024

SI

Rel. error in PID value

M

Dimension of each of

,

X

and

Y

B.4 Run Time Analysis of the ∼ G -PID in Example 10

In Fig. 9, we present a complete picture of the time taken to compute the ∼ G -PID, as a function of dimensionality d , in Example 10. Note that we cannot compare with the δ G -PID beyond a dimensionality of d = 64 , because the computation for the δ G -PID failed in our setup for d > 64 . Neither the δ G -PID nor the ∼ G -PID were carefully profiled and optimized, therefore, the runtime analysis in both Fig. 9 and Fig. 3 should be considered preliminary. We leave a more thorough comparison of the run time of both these methods to future work.

B.5 Absolute and Relative Errors in Example 10

Figure 10 shows how the absolute and relative errors in PID values scale with increasing dimensionality in Example 10. The absolute errors increase in proportion to dimensionality, starting under 10 -7

at d = 2 and remaining under 10 -4 at d = 1024 . The relative errors are all roughly constant, and remain under 10 -6 .

C Supplementary Material for Section 5

C.1 Implementation Details for Bias-correction

We use a number of different setups based on sampling from random connectivity matrices for bias correction in Section 5. All of these setups assume that d X = d Y .

The both-unique , fully-redundant and high-synergy setups have the following in common:

Also, the elements of H X are either zero or one, H X ( i, j ) ∼ i.i.d. Ber(0.1). These three setups differ in their definitions of H Y (the channel gain from M to Y ) and Σ W (which controls the extent of correlation between X and Y ).

The both-unique setup draws H Y ( i, j ) ∼ i.i.d. Ber(0.1), with all elements of H Y independent of the elements of H X , and sets Σ W = 0 .

The fully-redundant setup is similar to Example 7, by setting H Y = H X and Σ W = 0 . 9 I (note that Σ W is square, since d X = d Y ). By keeping Σ W close to the identity matrix, we are effectively in the regime with high correlation ρ in Example 7. This allows us to come close to emulating Example 3, without suffering from the issue of non-invertibility of Σ XY | M , mentioned in App. B.

The high-synergy setup is similar to Example 6, by setting H Y = 0 and Σ W = 0 . 8 I . As with the fully-redundant setup, by keeping Σ W close to the identity matrix, we are in the highρ regime. This allows us to come close to emulating Example 4, while not making the synergy or the total mutual information infinite.

The zero-synergy setup is similar to Example 5, and uses the following setup:

Here, H X ( i, j ) ∼ i.i.d. Ber(0.1), while H Y = H ′ Y H X , with H ′ Y ( i, j ) ∼ i.i.d. Ber(0.1), H ′ Y ⊥ ⊥ H X . Defined this way, M -X -Y form a Markov chain, ensuring that I ( M ; X | Y ) = 0 , so that SI = 0 (Refer equation (63)).

The bit-of-all setup is a combination of equal parts of the high-synergy and zero-synergy setups. The variables X and Y are swapped in the zero-synergy setup, so that both X and Y can have some unique information.

Remark 3 (Rectification) . In practice, we observed that the bias correction procedure prescribed in Definition 4 could lead to negative values for certain PID quantities. This occurred because the bias-corrected union information was not guaranteed to satisfy certain bounds, which we enforce below. To prevent the occurrence of negative PID values after bias-correction, we require a form of rectification:

where ˆ I ( · ) | bias-corr represents a bias-corrected mutual information estimate. After the second rectification equation above, the union information is bounded from below by the individual (bias-corrected) mutual information values, and bounded from above by the sum of the individual mutual information values, and by the total mutual information.

C.2 Bias-correction Performance in Additional Setups and at Higher Dimensionality

Plots showing bias correction performance for all setups described in App. C.1 are shown in Fig. 11 for 10-dimensional, and in Fig. 12 for 20-dimensional M , X and Y .

Of the setups we examine, only the case with both X and Y having purely unique information appears to have somewhat poor performance, where our bias-correction method appears to over-correct the bias in unique information, while insufficiently correcting the bias in redundancy and synergy.

C.3 A Preliminary Analysis of the Variance of PID Estimates

In Figures 13 and 14, we present a preliminary analysis of the variance of our PID estimates using bootstrap. The figures represent the true distribution of the PID estimates over multiple sample draws, or over multiple bootstrap sample draws, in the form of box plots. In what follows, we colloquially refer to these box plots as 'confidence intervals'. The true 'confidence intervals' were estimated using 100 runs of bias-corrected PID estimates, i.e., by drawing 100 different samples, each of size n . The bootstrap 'confidence intervals' were estimated using 100 bootstrap samples that were resampled from a single randomly drawn sample of size n .

When correcting for bias in the PID estimates on bootstrap samples, we use the number of unique data points in each bootstrap sample in place of n (refer Corollary 4), rather than the total sample size. This leads to more stable bootstrap-PID estimates.

The quality of the bootstrap 'confidence interval' is affected greatly by the quality of the individual sample used for bootstrap resampling. Nevertheless, we observe a reasonable degree of qualitative agreement between the true 'confidence interval' and the bootstrap 'confidence interval', particularly as the sample size increases. Future work will assess confidence intervals with greater care, using well-defined metrics, and assess how well these confidence intervals are calibrated.

D Supplementary Material for Section 6

D.1 Details Regarding the Multivariate Poisson Spike-count Simulation

We follow our previous paper [12], where this analysis was first presented. In this simulation, M is two-dimensional, consisting of two independent and identically distributed Poisson random variables, M 1 and M 2 . X and Y are each generated through a linear combination of Binomially thinning M 1 and M 2 , along with some Poisson noise:

While our results in Fig. 5 show that our ∼ G -PID estimator comes closest to obtaining the same values as the estimator of Banerjee et al. [20], several remarks are warranted:

The method of Banerjee et al. [20] estimates the ∼ -PID and not the δ -PID. Since there is no discrete estimator for the δ -PID, it is entirely possible that the δ G -PID is equally (or more) accurate with respect to its own true value. That is, the difference between the δ G -PID and the Banerjee et al. [20] estimator is entirely due to a difference in definitions;

We stated in the introduction that the δ G -PID estimates an approximate upper bound, whereas our ∼ G -PID estimates an exact upper bound. The difference due to the approximate nature of the δ G -PID upper bound is also not captured here, nor in Examples 8 and 9. In all these examples, we cannot tell how much of the difference is due to the difference in definitions and the approximate nature of the estimators.

100

200

300

600

1000

Sample size

0

1

2

3

4

5

6

Partial information value (bits)

Bias correction in the "both-unique" set up (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Partial information value (bits)

Bias correction in the "fully-redundant" set up (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0

1

2

3

4

5

Partial information value (bits)

Bias correction in the "high-synergy" set up (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Partial information value (bits)

Bias correction in the "zero-synergy" set up (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Partial information value (bits)

Bias correction in the "bit-of-all" set up (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0

2

4

6

8

10

12

14

16

Partial information value (bits)

Bias correction in the "both-unique" set up (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Partial information value (bits)

Bias correction in the "fully-redundant" set up (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0

5

10

15

20

25

Partial information value (bits)

Bias correction in the "high-synergy" set up (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Partial information value (bits)

Bias correction in the "zero-synergy" set up (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Partial information value (bits)

Bias correction in the "bit-of-all" set up (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

Biased

Bias-corrected

Ground truth

100

200

300

600

1000

Sample size

0

1

2

3

4

5

PID value (bits) with CIs

Bootstrap CIs in the "both-unique" setup (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

PID value (bits) with CIs

Bootstrap CIs in the "fully-redundant" setup (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0

1

2

3

4

5

6

7

PID value (bits) with CIs

Bootstrap CIs in the "high-synergy" setup (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0

1

2

3

4

5

PID value (bits) with CIs

Bootstrap CIs in the "zero-synergy" setup (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0

1

2

3

4

5

6

7

8

PID value (bits) with CIs

Bootstrap CIs in the "bit-of-all" setup (

d

=10

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

PID value (bits) with CIs

Bootstrap CIs in the "both-unique" setup (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0

5

10

15

20

25

PID value (bits) with CIs

Bootstrap CIs in the "fully-redundant" setup (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0

5

10

15

20

25

PID value (bits) with CIs

Bootstrap CIs in the "high-synergy" setup (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

PID value (bits) with CIs

Bootstrap CIs in the "zero-synergy" setup (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

100

200

300

600

1000

Sample size

0

2

4

6

8

10

12

14

16

PID value (bits) with CIs

Bootstrap CIs in the "bit-of-all" setup (

d

=20

)

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

True CI

Bootstrap CI

Ground truth

Finally, we do not know how accurate the 'ground truth' itself is, since this is also assessed using the estimator of Banerjee et al. [20]. Differences between the various estimators could also be due to a poor estimate produced by the discrete PID estimator.

D.2 Additional Simulations on Non-Gaussian Data

In order to evaluate how the ∼ G -PID performs on a greater variety of non-Gaussian distributions, we considered three more setups similar to the one described in App. D.1.

The first additional setup is a multivariate Binomial distribution:

This setup has the same mean as the multivariate Poisson distribution, but has a smaller variance and is therefore more peaked, further from zero. The multivariate Binomial distribution is closer to Gaussian than the multivariate Poisson, and indeed, we see good agreement between the ∼ G -PID and the 'ground truth', as assessed by the estimator of Banerjee et al. [20] (see Fig. 15, top panels).

The next two setups are 'zero-inflated' versions of the multivariate Poisson and Binomial distributions, which are produced by passing each of the aforementioned variables through a 'Z-channel', with zero-out probability 0.3. That is,

where Z i ∼ i.i.d. Ber(0 . 7) , i ∈ { M 1 , M 2 , X, Y } . The primed versions of these variables have an additional probability mass at zero, with a probability of 0.3. This is said to better reflect the statistics of Calcium imaging [45], although we use a different base distribution than the one in the cited paper. The zero-inflated versions of the Poisson and Binomial distributions are bimodal, with one of the peaks at zero, and the Binomial version having more well separated peaks. For both zero-inflated versions, the ∼ G -PID performs poorly at recovering the absolute ∼ -PID values (Fig. 15, middle-left and bottom-left panels). However, the relative PID values (normalized by the total mutual information) are closer to their true values (Fig. 15, middle-right and bottom-right panels), providing hope that a better mutual information estimate (e.g., [46, 47]) could correct the absolute PID values to some extent.

Further validation on non-Gaussian data is necessary to understand the impact of non-Gaussianity on ∼ G -PID estimates. However, a more extensive evaluation of non-Gaussian distributions (especially at higher dimensionalities) is made difficult by the unavailability of ground truth. The 'ground truth' in these examples is obtained using the discrete ∼ -PID estimator of Banerjee et al. [20] (which we assume is more accurate for discrete variables). Our method provides a better result than the δ -PID and the MMI-PID in Fig. 5, at least part of which is probably due to the difference in definitions. Nevertheless, we believe these examples demonstrate that applying our method to non-Gaussian data need not be ruled out, although care should always be taken in interpreting results.

D.3 Implementation Details of the Analysis Pipeline

The Visual Behavior Neuropixels data was analyzed as follows:

We selected mice that had at least 20 units in each brain region of interest. Only mice with both familiar and novel sessions were selected.

From each region we selected units of 'good' quality, with SNR at least 1, and with fewer than 1 inter-spike interval violations.

Trials were aligned to the start of each stimulus flash, and spikes were counted in bins of 50 ms, between 0 and 250 ms after stimulus onset (0-50 ms, 50-100 ms, etc.).

Trials corresponding to a non-change flash were defined as those that occurred between 4 and 10 flashes after the start of a behavioral trial, such that the image remained the same as the original image in this behavioral trial. Flashes corresponding to an omission, flashes after an omission, and flashes during which the animal licked, were all removed. Only flashes that occurred while

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

1

to

X

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Partial information (bits)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

1

to

X

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Normalized partial information

PID component

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

PID definition

∼

G

-PID

Ground truth

(Banerjee et al.

∼

-PID)

PID values (left) and PID normalized by

I

(

M

; (

X,Y

))

(right)

in a multivariate Binomial spike-count simulation

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

1

to

X

0.00

0.05

0.10

0.15

0.20

0.25

Partial information (bits)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

1

to

X

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Normalized partial information

PID component

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

PID definition

∼

G

-PID

Ground truth

(Banerjee et al.

∼

-PID)

PID values (left) and PID normalized by

I

(

M

; (

X,Y

))

(right)

in a zero-inflated Poisson spike-count simulation

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

1

to

X

0.00

0.05

0.10

0.15

0.20

0.25

Partial information (bits)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Weight from

M

1

to

X

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Normalized partial information

PID component

I

(

M

; (

X,Y

))

UIX

UIY

RI

SI

PID definition

∼

G

-PID

Ground truth

(Banerjee et al.

∼

-PID)

PID values (left) and PID normalized by

I

(

M

; (

X,Y

))

(right)

in a zero-inflated Binomial spike-count simulation

0

50

100

150

200

Time after stimulus onset (ms)

0

2

4

6

8

10

12

Redundancy,

RI

(VISp : VISl; VISal)

(bits)

****

ns

****

****

****

***

****

****

****

ns

**

ns

****

Redundancy betw 3 visual cortical areas (top 10 principal components)

Change

Non-change

0

50

100

150

200

Time after stimulus onset (ms)

0.2

0.4

0.6

0.8

1.0

RI

(VISp : VISl; VISal)

/I

(VISp; (VISl

,

VISal))

***

ns

****

ns

ns

*

****

*

****

ns

ns

ns

****

Redundancy fraction betw 3 visual cortical areas (top 10 principal components)

Change

Non-change

0

50

100

150

200

Time after stimulus onset (ms)

0

2

4

6

8

10

12

14

Redundancy,

RI

(VISp : VISl; VISam)

(bits)

****

ns

****

****

****

***

****

****

****

**

*

ns

****

Redundancy betw 3 visual cortical areas

(top 20 principal components)

Change

Non-change

0

50

100

150

200

Time after stimulus onset (ms)

0.2

0.4

0.6

0.8

1.0

RI

(VISp : VISl; VISam)

/I

(VISp; (VISl

,

VISam))

**

ns

****

****

****

ns

****

****

**

ns

ns

ns

****

Redundancy fraction betw 3 visual cortical areas (top 20 principal components)

Change

Non-change

IMXY

UIX

UIY

RI

SI

0

5

10

time = 0

IMXY

UIX

UIY

RI

SI

time = 50

IMXY

UIX

UIY

RI

SI

time = 100

IMXY

UIX

UIY

RI

SI

time = 150

IMXY

UIX

UIY

RI

SI

time = 200

PID values in bits;

M

=VISp

,

X

=VISl

,

Y

=VISal

(top 20 PCs)

PID value (bits)

PID components over time (in ms)

Condition

Change

Non-change

UIX

UIY

RI

SI

0.0

0.2

0.4

0.6

0.8

time = 0

UIX

UIY

RI

SI

time = 50

UIX

UIY

RI

SI

time = 100

UIX

UIY

RI

SI

time = 150

UIX

UIY

RI

SI

time = 200

PID fraction of total mutual information;

M

=VISp

,

X

=VISl

,

Y

=VISal

(top 20 PCs)

PID fraction

Condition

Change

Non-change

PID components over time (in ms)

IMXY

UIX

UIY

RI

SI

0

5

10

time = 0

IMXY

UIX

UIY

RI

SI

time = 50

IMXY

UIX

UIY

RI

SI

time = 100

IMXY

UIX

UIY

RI

SI

time = 150

IMXY

UIX

UIY

RI

SI

time = 200

PID values in bits;

M

=VISp

,

X

=VISl

,

Y

=VISam

(top 20 PCs)

PID value (bits)

PID components over time (in ms)

Condition

Change

Non-change

UIX

UIY

RI

SI

0.0

0.2

0.4

0.6

0.8

time = 0

UIX

UIY

RI

SI

time = 50

UIX

UIY

RI

SI

time = 100

UIX

UIY

RI

SI

time = 150

UIX

UIY

RI

SI

time = 200

PID fraction of total mutual information;

M

=VISp

,

X

=VISl

,

Y

=VISam

(top 20 PCs)

PID fraction

Condition

Change

Non-change

PID components over time (in ms)

the animal was engaged (as measured by an average reward rate of at least 2 rewards/min) were selected.

Trials corresponding to a change flash were defined as those during which an image change occurred, and the animal was engaged (as above).

The top 10 or 20 principal components of neural activity were selected at each time bin, and for each brain region under consideration. Principal component analysis was carried out using the Scikit-learn [48] package in Python.

∼ G -PID estimates were computed on the covariance matrix between principal components across regions, for each time bin.

Data were aggregated across 40 or 41 mice for the figures with VISal, and across 38 mice for the figures with VISam.

Statistical significance was assessed using a two-sided unpaired Mann-Whitney-Wilcoxon test.

D.4 Additional Results

Fig. 16 shows the redundancy about VISp activity, between VISl and VISal. It is identical to Fig. 6, except that it uses only the top-10 PCA components in each of the three visual cortical regions. This figure shows that the result presented in Fig. 6 gracefully degrades with a reduction in the number of PCA components. When using fewer PCA components, the increase in redundancy on change trials is still statistically significant, but the effect size is not as large, and not as sustained, as it is in Fig. 6.

Fig. 17 shows the redundancy about VISp activity between VISl and VISam, the latter of which is a different higher-order cortical region (see, e.g., [49]). This figure shows that the result in Fig. 6 can also be seen for another choice of higher-order cortical region (i.e., VISam rather than VISal).

Figures 18 and 19 show all PID components, not just the redundancy, for the same settings as in Figures 6 and 17 respectively. These show that redundancy is the dominant partial information component, and appears to be the main driver of changes in the overall mutual information. This justifies why we include only a plot of redundancy in Figs. 6, 16 and 17.

D.5 The Necessity for PCA, and Maximizing Dimensionality

In our analysis of real neural data in Sec. 6, we used principal components analysis to reduce dimensionality, despite the fact that our method could scale to much larger dimensionalities. As mentioned briefly in Sec. 6, this was due to the limited number of trials we had. For many mice, there were a larger number of neurons in each region than trials with which to compute the covariance matrix. In other words, directly computing the covariance of the neural activity would have resulted in a covariance matrix that was rank-deficient. We used PCA to reduce dimensionality and ensure that we obtained a reasonable estimate of the covariance matrix, and to minimize the error in our PID estimate, which would naturally be higher with a greater number of PCA dimensions.

For completeness, we analyze here what would happen if we used the maximum possible number of PCA components for each mouse, while ensuring that the covariance matrix is not rank deficient. We also ensure that we had the same number of components across all regions and across both change and non-change conditions (within each mouse), so as to perform a fair comparison. This gives us on average 53 ± 16 PCA components, with a minimum of 22 and a maximum of 84 PCA components. We find that the results of higher and more sustained redundancy on change flashes continues to hold when using the maximum possible number of PCA components for each mouse (see Fig. 20). However, as expected, there is much greater variance across mice, possibly due to variability in the number of PCA components chosen across mice, and due to greater errors in our PID estimates. For the main results presented in Fig. 6, in order to be consistent across mice, we rounded down from the minimum and used the top 20 PCA components for all mice.

D.6 Differences between Change and Non-change Conditions are not an Artifact of Bias-correction

The number of trials corresponding to change flashes is much smaller than the number of trials corresponding to non-change flashes. Accordingly, the sample size used to estimate the covariance matrix is different in each of the two conditions. Bias correction was performed using the appropriate

0

50

100

150

200

Time after stimulus onset (ms)

0

5

10

15

20

Redundancy,

RI

(VISp : VISl; VISal)

(bits)

****

ns

****

****

***

*

****

****

***

ns

*

ns

****

Redundancy betw 3 visual cortical areas

(max principal components)

Change

Non-change

0

50

100

150

200

Time after stimulus onset (ms)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

RI

(VISp : VISl; VISal)

/I

(VISp; (VISl

,

VISal))

ns

ns

**

***

ns

ns

****

****

*

**

ns

*

****

Redundancy fraction betw 3 visual cortical areas

(max principal components)

Change

Non-change

0

50

100

150

200

Time after stimulus onset (ms)

0

2

4

6

8

10

12

14

16

Redundancy,

RI

(VISp : VISl; VISal)

(bits)

****

ns

****

****

***

***

****

****

****

ns

**

ns

****

Redundancy betw 3 visual cortical areas (top 20 principal components)

Change

Non-change

0

50

100

150

200

Time after stimulus onset (ms)

0.2

0.4

0.6

0.8

1.0

1.2

RI

(VISp : VISl; VISal)

/I

(VISp; (VISl

,

VISal))

**

ns

****

****

***

*

****

****

****

ns

ns

ns

****

Redundancy fraction betw 3 visual cortical areas (top 20 principal components)

Change

Non-change

sample size; however, as noted in Section 5, our bias correction process is not perfect, and may leave some residual bias.

In order to show that the results we observed were not an artifact of differences in residual bias caused by different sample sizes, we randomly subsampled the non-change flashes to produce a dataset with equal numbers of trials for change and non-change flashes. Repeating the analysis as before, we found that our conclusions continued to hold even in the setting where both conditions have equal sample sizes, as shown in Figure 21.

D.7 A Note on Inhomogeneous Poisson Processes

Inhomogeneous Poisson processes are those whose mean firing- (or 'emission-') rates change with time. Our method for estimating the PID does not consider the temporal characteristics of the original signal. Rather, the data analyst can choose the random variables M , X and Y to span some time range (or different time ranges) as they please.

In our analysis of real neural data in Sec. 6, we counted the number of spikes in a 50-125 ms window after stimulus onset. Even if the underlying spiking process was an inhomogeneous Poisson process, this spike count would be Poisson distributed, with a mean given by the integral of the emission rate over the fixed window. In general, while analyzing data and computing PID values, one will have to be aware of severe inhomogeneities (e.g., if the distribution changes not just in the same way in each trial, but changes across trials), and account for them separately. For example, in our analysis, we excluded time periods when the mice were not 'engaged' in the task, as defined by not actively consuming rewards at a rate of at least 2 rewards per minute.

E Compute Configuration Used and Code Availability

All analyses were performed on a workstation equipped with an Intel Core i7-10700KF CPU with 8 cores (16 threads), 48 GiB of RAM and data stored on a 1 TB PCIe NVMe solid state drive.

Analysis of the Visual Behavior Neuropixels data (for 84 sessions) with d = 20 took approximately 9 minutes to run. This included loading data for each session and computing 840 PID values on 60 × 60 covariance matrices, implying an average run-time of about 0.64s for each PID estimate (including amortized data-load time).

All code used to compute and estimate the ∼ G -PID and correct for bias, including all examples in this paper and code for neural data analysis, is available on GitHub [36].

Additional References

Thomas M Cover and Joy A Thomas. Elements of Information Theory . John Wiley & Sons, 2012.

K. B. Petersen and M. S. Pedersen. The Matrix Cookbook . Technical University of Denmark, 2012. URL ❤/a116/a116♣✿✴✴✇✇✇✷✳❝♦♠♣✉/a116❡✳❞/a116✉✳❞❦✴♣✉❜❞❜✴♣✉❜/a115✴✸✷✼✹✲❢✉❧❧✳❤/a116♠❧ . Version: November 15, 2012.

Xue-Xin Wei, Ding Zhou, Andres Grosmark, Zaki Ajabi, Fraser Sparks, Pengcheng Zhou, Mark Brandon, Attila Losonczy, and Liam Paninski. A zero-inflated gamma model for post-deconvolved calcium imaging traces. Neurons, Behavior, Data Analysis, and Theory , 3(2):1-21, 2020.

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E , 69(6):066138, 2004.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning , pages 531-540. PMLR, 2018.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Joshua H Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jennifer A Luviano, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature , 592(7852):86-92, 2021.

Peer reviews:
comments: SUMMARY:
The paper proposed a new method for partial information decomposition (PID) on multivariate Gaussian distributions. The issue of bias was discussed, and a correction method was provided. The method was tested on synthetic canonical examples and real data.

STRENGTHS:
1. The introduction clearly lays out the problem.
2. The paper is extremely well written with notations clearly defined.
3. The method extends prior work on PID with new properties. 
4. The method is rigorously tested on simulated data.
5. The method solves an important problem, namely “the extent to which one region’s activity uniquely explains that of another, while excluding information corresponding to spontaneous behaviors” as stimulus could make two regions seem correlated.

WEAKNESSES:
1. The paper does not provide enough real data to show effectiveness in real applications.
2. Testing on higher dimensional settings would be important.

QUESTIONS:
1. Would non-modeled confounding factors affect the results?

Summary after rebuttal
The added simulations for testing higher dimensional data further justify the high score I gave.

LIMITATIONS:
The paper does not have negative societal impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPartial Information Decompositions (PIDs) play an essential role in neuroscience research. One constraint of the broader usability of PIDs is the computational difficulty of computing PIDs for high-dimensional neural recordings. To address this concern, the authors propose a method to compute and estimate a PID efficiently. More specifically, they restrict the optimization space of PIDs to jointly Gaussian, which reduces the number of optimization variables, allowing them to compute PIDs for much higher dimensionalities of neural data. Then, the authors show their method could be written out in closed form and solved by projected gradient descent, and they use nine examples with increasing complexity to show their method has the ability to recover ground truth and stability over increasing dimensionality. The authors also claim it's the first time to correct the bias and variance of estimates in a heuristic way. Finally, the authors evaluate the performance of their method on both synthetic and real neural data.\n\nSTRENGTHS:\n* Computational scalability of ~PID with a basic property called additivity.\n* The authors' exposition of their method and experiments is clear.\n* The first time to raise the issue of bias in PID estimates.\n\nWEAKNESSES:\n* No analysis is done to show the stability of $\\delta$-PID over increasing dimensionality. According to the paper, there are two differences between $\\delta$-PID and $\\sim$-PID. The first one is the \"additivity\" property, which is clearly shown by Examples 8-9 in Section 4. The second one is that $\\sim$-PID uses an exact upper bound, while $\\delta$-PID uses an approximate upper bound.  \nHowever, no analysis is performed to show how the second difference will affect the performance. In other words, people may be curious about the stability of $\\delta$-PID over increasing dimensionality if we use examples with no \"additivity\" property ($\\delta$-PID and $\\sim$-PID both agress with the ground truth).  \nCurrently, it's hard to see the differences between $\\delta$-PID and $\\sim$-PID when applying to high dimensional neural data. I think this point may determine whether the proposed contribution is timely and impactful or a solid technical upgrade without practical consequences in neuroscience.\n\nQUESTIONS:\n* Does $\\delta$-PID also agree with the ground truth even when distributions are not Gaussian?\n* Could you please discuss how the \"additivity\" property would affect the analysis of communication among brain regions?\n\nLIMITATIONS:\nI have highlighted technical limitations and weaknesses above. I have nothing further to add here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new, efficient method for computing Partial Information Decompositions (PIDs) on multivariate Gaussians. They build their approach around the $\\sim$-PID approach, as this allows them to preserve an additivity property (allowing PIDs to be computed on independent systems separately and then added later). They present a number of canonical examples for Gaussians, shaw that their Gaussian PID works even when distributions are non-Gaussian, and show an example of its use on real neural data from the Allen Institute. Finally, they address the issue of bias in PID estimates, propose a bias-correction method, and evaluate it empirically.\n\nSTRENGTHS:\nThe paper is very clearly written and the topics well explained. The literature review seems comprehensive, and although of somewhat specific topical interest, the work seems original and useful.\n\nWEAKNESSES:\nGiven one of the major motivating factors given in the introduction is the need for efficient estimators of PID that can accommodate higher dimensional neural data, I was disappointed that in the end the authors chose to apply their method to a PCA-reduced version of the Allen Institute neuropixel probe data... wasn't the point to be able to address higher dimensional problems? It's thus not completely clear to me that this method, as is, has delivered on the promise of providing a PID approach more applicable to data with thousands of neurons than other PID methods.\n\nQUESTIONS:\nCan the authors method run on thousands of neurons, consistent with the original motivation they provide? \nWhat are the limitations on dimensionality more explicitly for this method? \nWhat is the computational complexity?\nDoes the author's method require using PCA or having high firing rate neurons? \nHow does it deal with low firing rate neurons or highly inhomogenous Poisson variables?\n\nLIMITATIONS:\nThe authors adequately address limitations, although it would be good to hear more related to the questions above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes an upper bound on mutual information (more precisely: on the \"unique information\" / \"union information\" that appear in \"Partial Information Decomposition\") that is easier to compute. This is done by replacing an infimum over all distributions matching given marginals by an infimum over only the ones that are Gaussian.\nToy examples (with Gaussian distributions) are given, and an experiment on real data is performed.\nFurther theory also includes considerations about biases arising from covariance estimators.\n\nSTRENGTHS:\nThe paper is mostly clearly written, with simple examples to help the reader follow.\nThe paper is self-complete, with reminders of definitions.\nConsidering covariance estimator biases is a plus.\n\nWEAKNESSES:\nEDIT: after discussion with the authors, I see that I had misunderstood the scope and contributions of the paper. The weaknesses below do not longer hold (or not as strongly).\n\n-----------\n\nThe main issue is that the definition of the upper bound actually supposes that the distribution MXY that is studied is Gaussian, at least marginally. Indeed the infimum is performed over all possible distributions Q_MXY that are Gaussian and whose marginals satisfy Q_MX = P_MX and Q_MY = P_MY, which implies that P_MX and P_MY have to be Gaussian (otherwise the infimum is performed over an empty set).\nAs a consequence, this upper bound definition cannot be applied to datasets that are not Gaussian (at least marginally). This crucial point is not discussed in the paper.\n\nActually, from Sections 3.2 and 5, it seems that the upper bound is estimated based on covariance matrices only, without using distribution Gaussianity. If this is the case indeed, then the definition could be changed, to rely directly on these covariance matrices and removing the Gaussianity assumption. This however would probably require to significantly rewrite the paper.\n\n\nAnother issue is the lack of sufficient validation, either theoretically or experimentally, in particular regarding distributions that are not Gaussian (i.e., really not Gaussian). This comment could have been tempered if there had been a discussion about general approximative Gaussianity of distributions in the field of study, but there is none. \nAlso, the impact of the introduction of such a Gaussianity constraint in the upper bound estimation should be studied closely: how realistic is this assumption, how far are real optimal Q_MXY from their Gaussian versions, how tight is the upper bound thus obtained, etc.\n\nIf this was a theoretical paper, I would expect theoretical results (guarantees such as a bound on the error made by the upper bound, or proving the conjectures, etc.). If this was an experimental paper, I would expect an extensive validation on many distributions for which the actual mutual informations are estimated (brute force or known solutions) in settings as varied as possible (not just normal distributions + a single real dataset). But the paper, in its current state, does meet these expectations.\n\nQUESTIONS:\nI would be willing to revise my score if the points in the section above were significantly addressed.\n\nEDIT: they were.\n\nLIMITATIONS:\nCf the validation issue in the weakness section above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFirst, I would like to thank the authors for the work they put into contributing to the field. \nIn their paper, the authors provide a new method to estimating a well-known measure, the Partial Information Decomposition (PID). Within the topic of PIDs, their method computes a specific version of PID, the ~-PID. \n\nWhile methods to compute the ~-PID already exist, the authors' method is advantageous in that \n1) it is more efficient to compute (quadratic in number of variables instead of exponential) and \n2) in that it corrects for the bias. \n\nThe author's then go on to show with simulations and examples that \n1) their choice of PID method, the ~-PID, outperforms other PID methods\n2) the bias correction works empirically\n3) their method is stable even for increasing dimensionality\n4) their method can be applied on data which is not Gaussian distributed such as simulated Poisson data and real neural recordings (neuropixel)\n\nSTRENGTHS:\n- The paper addresses a topic which is becoming more and more relevant as the number of neurons that can be recorded from simultaneously is increasing: The question of how information is distributed among neurons and brain areas.\n- The paper is written in a very clear and structured way which successfully guides the reader through every logical step. \n- The mathematical formulation is sound\n- The quality of figures is excellent\n\nWEAKNESSES:\n- The paper is an improvement of an already existing method which makes it important and publishable yet disqualifies it for a very high grade.\n- The claim of generalizability to non-Gaussian data is not backed up strongly enough empirically or mathematically (refer to \"Questions\")\n- Small mistakes/typos in lines: 177, 206, 322\n\nQUESTIONS:\n- In the list of contributions, the authors state in line 50 that they are able to reduce the number of optimization variables from exponential to quadratic. Where can I see this in the equations and derivations? \n\n- The authors show that their method works on non-Gaussian data by applying it to simulated Poisson data and neuropixel spike count data. They explain this by stating that the Gaussian distributions is a good-enough approximation to the Poisson distribution. It is known, however, that neural firing does not follow a Poisson distribution. This is especially pronounced when recorded via 2photon imaging for which the distribution is strongly inflated at zero. Xue-Xin Wei et al. 2020 propose a zero-inflated gamma model to accurately capture calcium imaging traces. Is it feasible for the authors to test their method on such bimodal distributions which might be harder to approximate with a Gaussian distribution? If the rebuttal time for such experiments is too short, can the authors comment on the expected outcome?\n\nLIMITATIONS:\nThe authors discuss and point out most limitations. The claim that their method works on non-Gaussian data is only backed by an experiment with Poisson data, however. In my opinion, there is still room for the method to fail for other relevant distributions (refer to point \"Questions\") which would be worth mentioning as a (possible) limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}