Estimating the Unique Information of Continuous Variables
Authors: Ari Pakman
Amin Nejatbakhsh
Dar Gilboa
Abdullah Makkeh
Luca Mazzucato
Michael Wibral
Elad Schneidman
Venue: NeurIPS 2021 Poster (2021)
Keywords: Recurrent Neural Networks
Partial Information Decomposition

Abstract: The integration and transfer of information from multiple sources to multiple targets is a core motive of neural systems. The emerging field of partial information decomposition (PID) provides a novel information-theoretic lens into these mechanisms by identifying synergistic, redundant, and unique contributions to the mutual information between one and several variables. While many works have studied aspects of PID for Gaussian and discrete distributions, the case of general continuous distributions is still uncharted territory. In this work we present a method for estimating the unique information in continuous distributions, for the case of one versus two variables. Our method solves the associated optimization problem over the space of distributions with fixed bivariate  marginals by combining copula decompositions and techniques developed to optimize variational autoencoders. We obtain excellent agreement with known analytic results for Gaussians, and  illustrate the power of our new approach in several brain-inspired neural models. Our method is capable of recovering the effective connectivity of a chaotic network of rate neurons, and uncovers a complex trade-off between redundancy, synergy and unique information in recurrent networks trained to solve a generalized XOR~task.


Estimating the Unique Information of Continuous Variables
Ari Pakman Columbia University
Amin Nejatbakhsh Columbia University
Dar Gilboa Harvard University
Abdullah Makkeh Georg August University
Luca Mazzucato University of Oregon
Michael Wibral
Elad Schneidman Weizmann Institute
Georg August University
Abstract
The integration and transfer of information from multiple sources to multiple tar- gets is a core motive of neural systems. The emerging field of partial information decomposition (PID) provides a novel information-theoww nisms by identifying synergistic, redundant, and unique contributions to the mutual information between one and several variables. While many works have studied aspects of PID) for Gaussian and discrete distributions, the case of general contin- uous distributions still uncharted territory. this work we present method for estimating the unique information in continuous distributions, for the case of one versus two variables. Our method solves the associated optimization problem over the space of distributions with fixed bivariate marginals by combining copula decompositions and techniques developed to optimize variational autoencoders. We obtain excellent agreement with known analytic results for Gaussians, and illus- trate the power our new approach in several brain- insiinii neural models. Our method capable of recovering the effective connectivity of chaotic network of rate neurons, and uncovers a complex trade-off between redundancy, synergy and unique information in recurrent networks trained solve generalized XOR task.
lens into these mecha-
Introduction and background
In neural systems, often multiple neurons are driven by one external event stimulus; conversely multiple neural inputs can converge onto single neuron. A natural question in both cases is how multiple variables hold information about the singleton variable. their seminal work [1] Williams and Beer proposed an axiomatic extension of classic information theory to decompose the mutual information between multiple source variables and a single target variable in meaningful way. For the case of two sources X1, X2, their partial information decomposition (PID) amounts to expressing the mutual information of X,, X2 with a target Y as sum of four non-negative terms,
1(Y: : X2), U(Y X2) U(Y X2) + R(Y (Xi, X2)) + S(Y Xe),
(1.1)
corresponding unique (U1, U2), redundant (R) and synergistic (S) contributions, respectively. These terms should also obey the consistency equations
T(Y Xx) = R(Y: (X1, + U(Y : :X :X X2), I(Y: X2) R(Y (X,, + U(Y X,)
(1.2)
(1.3)
The PID has proved useful in understanding information processing by distributed systems in diverse array of fields including machine learning 22 3) earth science 4 and cellular automata (5).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and particularly in neuroscience 16-10, where notions of synergy and redundancy, traditionally considered mutually exclusive and distinguished by the sign of
A I(Y: (X1, Xx)) - I(Y X1)
S(Y (X1, - R(Y (X1, X2)),
(1.4)
have long played a central role in the quest to understand how neural circuits integrate information from multiple sources [1H44] The novelty of the PID framework here is in separating the measures of synergy and redundancy in
The above abstract formulation of PID provides three equations for four unknowns, and only becomes operational once one of U1. U2, R. or S is defined. This has been done in [15] via definition of the unique information:
Definition 1 (BROJA ISD. Given three random variables (Y, X1, X2) joint probability den- sity ply, F2, T2), the unique information U1 of X1 with respect to Y U(Y 10Y:X. X2) min 4 xx1(X2),
(1.5)
qEQ
/
au, (122)
=
min gEQ
[dddz d2 (V,, T2) log
((yl22))((
(1.6)
where
Q
= x,), 1,2).
(1.7)
In words, we minimize the conditional mutual information I(Y X1 X2) over the space of density functions that preserve the marginal densities p(y. Ø1) and p(y. The above definition implies, along with 1.2 1.3). that the unique and redundant information only depend on the marginals p(y, T2), and that the synergy only be estimated from the full p(y, 22).
The original definition in [5 was limited to discrete random variables. Here, we show that the extension to continuous variables is well-defined and can be practically estimated.
Motivation from decision theory [15. Consider for simplicity discrete variables. A decision maker DM, can choose an action a from finite set A, and receives a reward u(a, y) based on the selected action and the state y, which occurs with probability p(y). Notably, DM, has no knowledge of but observes instead a random signal æ1 sampled from p(z1ly). Choosing the action maximizing the expected reward for each Ti, his maximal expected reward is
Ri
Σ E((r)) 21
max Σ Er(ulriuu(aa))) a|r1
(1.8)
DM1 is said to have no unique information about y w.r.t. another decision maker DM2 that observes I2 ~ p(æ2ly) - if Rg 2 R1 for any set A. any distribution p(y), and any reward function u(a, y). A celebrated theorem by Blackwell [7] states that such a generic advantage by DM, occurs iff there exist stochastic matrix qx1 (22) which satisfies
p(zily)=
Σ p(2zly)g(<111)))
(1.9)
But this occurs precisely when the unique information 1.5 vanishes, since then there exists a joint
distribution 21, in Q for which y 1 x1 [22 which implies q(01 (22, y) = q( (x2), and thus (I.9) holds. Similar results exist for continuous variables [18 Thus the unique information
from Definition Iqqaantiiees a departure from Blackwell's 's relation (1.9)
In this work we present a definition and method to estimate the BROJA unique information for generic continuous probability densities. Our approach is based on the observation that the constraints (I.7) can be satisfied with appropriate copula parametrization, and makes use of techniques developed to optimize variational autoencoders. We only consider one-dimensional
Y, Y,X1,Xs X₂ for simplicity, but the method can be naturally extended to higher dimensional cases. In
Section we review related works, Section we present our method and Section 4 contains several illustrative examples.

---

2
Related works
Partial information decomposition offers solution to repeated question that not addressed by 'classical' information theory regarding the relations between two sources and a target []] From mathematical perspective 'functional definition' has be made, meaning that such definition should align with our intuitive notions. Yet, as shown in (20), not all intuitively desirable properties of PID can be realized simultaneously. Thus, different desirable properties are chosen for distinet application scenarios. Thus, various proposals for decomposition measures are not seen as conflicting but as having different operational interpretations. For example, the BROJA approach used here builds on desiderata from decision theory, while other approaches appeal to game theory [21] or
the framework of Kelly gambling [22]] Yet other approaches use arguments from information
geometry [23]. Other approaches assume agents receiving potentially conflicting or incomplete
information about the source variables for the purpose inference or decryption (see e.g. 24 25)).
In 26 the authors separate the specific operational interpretations of PID measures from the general
structure of information decomposition.
The actual computation of the BROJA unique information is non-trivial, even for discrete variables. Optimization methods exist for the latter case (27-29), and analytic solutions only known when all the variables are univariate binary 33 For continuous probability densities, an earlier definition aligned with the BROJA measure was made by Barret (31), but only applies to Gaussian variables. For Barret's s measure, an analytic solution is known when ply, 21, T2) is three-dimensional Gaussian density [3]] but does not generalize higher dimensional Gaussians [32
3 Bounding and estimating the unique information
We proceed in two steps. We first introduce a parametrization of the optimization space Q in (.. and then introduce and optimize upper bound on the unique information.
3.1 Parametrizing the optimization space with copulas
To characterize the optimization space Q in 1.5 it is convenient to recall that according Sklar's theorem 33 any n n-variate probability density can be expressed as
p(=1....., ... ... p(x1)
... Un))
(3.1)
where p(x;) is the marginal and u; = F(x;)) is the CDF of each variable. The dependency structure among the variables is encoded in the function c:(0, 1)"d 1]. This is copula density, probability density on the unit hypercuhe with uniform marginals (4).
II du, clu...l.uu ... un) =1 Vi.
(3.2)
[0.1] -0.1 j=1.j
Note that under univariate reparametrizations 2/ (22), the u, 's and the copula e remain invariant. For overview of copulas in machine learning, B5
Proposition 1. Under the BROJA Definition of unique information, all the terms of the partial infor- mation decomposition in [I] [.133 are independent of the univariate marginals p(x1), p(())))))))))))))))))))()))))) ply). and only depend on the copula cl Myy u1, u2).
Proof. Expressing q(y. 1 Ø2), g( 22). gly, .22) via copula decompositions 3.1 and changing variables as du, dly)ddy, etc., the objective function in (1.6) becomes
/ [0.1]a
c(Uy, U1 U2)
LY x11X2)
du duz cluy, un, u2) log c(Uy: u2 )((1), u2)
(3.3)
3

---

Note that the copula of any marginal distribution is the marginal of the copula:
c(uy, u2) = / duj cluy, u1, u2), c(u,, u2) = /
duy c(uy, u1 U2).
(3.4)
[0.1]
[o.1]
Thus the optimization objective and the unique information independent of the univariate marginals. A similar result holds for the mutual information terms in the I.h.s. of (1.1-(133)
It follows that none of the PID terms in (1.I)-1.33 (1.3) depend on the univariate marginals, and therefore
all the PID terms are invariant under univariate reparametrizations of (y, (1,
In order to parametrize the optimization space Q in (1.7) using copulas, consider the factorization p 27 21 ) x1).
ply, 21, x2)
(3.5)
Using the copula decomposition 3.1 for n 2, the last two factors in 3.5 can be expressed as
p(yla1) = x1) (()))((1)((()2)) c(uy, "1)p(y)
(3.6)
p(x1)
p(21)
and similarly
x(21,
(3.7)
p(z11))
(1 u2n))((221)) (12//("1l)= (2 x2) (x)
(3.8)
(3.9)
where we defined the conditional CDFs,
aC(uy: u4)
Uily Fuu (My)
1,2
(3.10)
Oug
and Cluy, u.) the CDF of c(uy, u.). Note that the function C1,21y (Ully: U2/y) 3.8 is not
the conditional copula c(u, u2lu,,) but rather the copula of the conditional p(x1, T2 22))) Using expressions (3.6) and (3.9), the full density (3.5) becomes
(1,,01,222
p(s)p(xr))((s))((u,, u1, uz)
(3.11)
where
cuuy u1, u2) cuy, u1) cluy, (2) y> (2]p)-
(3.12)
This is simple of the pair-copula construction of multivariate distributions [0 which allows to expand any n-variate copula as product of (conditional) bivariate copulas.
Proposition 2. The copula of the conditional, C1,2 (5))), parametrizes the space Q in 1.7
Proof. Since ((), = ply. x4) (i = 1,, the copula factors in p(y,,;) 1000=ppp000 ((,) cuu, u,),
=
(3.13)
are fixed in Q. Therefore, in the copula decomposition (.12 for g(y. 21, E Q, only the last factor can vary in Q. Let us denote by 0 the parameters of generic parametrization for the copula 61 (1,2/1( (Mily U2 Since the latter is conditioned on uy. the parameters can taken as function (u,.). It follows that the copula of necessarily has the form
cy (uys U1, U2) c(uy. U1 c(uy. U2) (1,2)0()))) (Uipp- U2ly).
(3.14)
and the parameters of the function ((u)) are the optimization variables
'The connection between mutual information and copulas was discussed
36 37
We note that in multivariate pair-copula expansions it is common to assume constant conditioning parame- ters 0 41. but do not make such simplifying assumption.

---

3.2 Optimizing an upper bound
Inserting now the expression (3.44) into the objective function (3.3 we get
[[0] (1g( log [c(uy. (1)c1, (uily ]2nny] E (m), ug) log co(co u2).
(3.15)
which is our objective function and satisfies the marginal constraints (17 Note that apart from
the optimization parameters it depends on the bivariate copulas c(Uy. U1 and c(uy, U2) which should be estimated from the observed data. Given D observations {y (8) (1)
) we map each value to [0, 1] via the empirical CDFs of each coordinate (y, #1, (1,21)), Computing the latter has a O(D log D) cost from sorting each coordinate and yields data set {uy 21 u² ) The latter set is used to estimate copula densities c(uy, U1) and d(uy, U2) by fitting several parametric and non non-parametric copula models [42], and choosing the best pair of models using the AIC criterion
222
(1) (8)
From the learned copulas we also get the conditional CDF functions 4iy ((u//u)) that appear in the arguments of the first term in (3.15).
A variational upper bound. Minimizing (3.55) directly w.r.t. 0 is challenging because the second term depends on the copula marginal (u u2) which has no closed form, requires integrating (3.14) w.r.t. "y- We introduce instead an inference distribution ro("g (1, ug), with parameters ø, that approximates the conditional copula cæ( (11, u2), and consider the bound
log ca (u1, u2) = log f du', co cu u1, u1) 2 duly (u/yl/ U2) log co U1, u2) (s(u,/11,, u2)
(3.16)
which follows from Jensen's inequality and is tight when ro(ugllu w1, u2) co(u,, ( U2) This expression gives an upper bound on [0). which can be minimized jointly w.r.t. (0, o).
disadvantage of the bound (3.16) is that its tightness depends strongly on the expressiveness the inference distribution re (1,, W/. u2). This situation can be improved by considering multiple-sample generalization proposed by [4
A
co colug (a) un, u2) T (ub U1 u22)
log co (wi, w2) 2 D u2) = E p(u S (.. log
(3.17)
where the expectation is w.r.t. A independent samples of r ro (u, u2). (ui, coincides with the lower bound in (3.16) for A and satisfies
( u2),
(3.18)
lim
D D uz) = log co (u, u2)
(3.19)
A A-500
Thus, even when (u(u,, U1, u2) o co (uy, (1, uz). the bound can be made arbitrarily tight for large enough A. Inserting 3.177 in (3.15). we get finally
10 s 1 + B2 d).
(3.20)
where
B1 En u2) log [c(uy, un B2 10, Φ) = -E Ees(us ...) Da (ui, u2),
(3.21)
(3.22)
and minimize the r.h.s. of (3200) w.r.t. (0, Φ). Lawvieriance estimates of the gradients to perform the minimization can be obtained with the reparametrization trick [45] 46], as discussed in detail in Appendix A In our examples below use for (1,2)((u.,) bivariate Gaussian copula (reviewed in Appendix B). Such copula has just one parameter 0 E 1-1,++1 +1], and thus the optimization is done over the space of functions 0( (= 1] + [11,+11, which we parametrize with a two-layer neural network. Similarly, we parametrize r& (uy) (u, u2) with two-layer neural network. Details of these networks are in Appendix D
For this fitting/model selection step, used the pyvinecopulib python package 45
5

---

Estimatedd
Exact
Deference
646-
125
646
135
646
60250
153
150
6.8
0.35
633
661
661
667
025
625
0.6
009
6:6
009
60000
Figure 1: Estimated vs. exact values of unique information for Gaussians. For a three- dimensional Gaussian, we show estimates of U(Y xx)) as function of the correlations Py.z. (i 2), compared with the exact results from 31 Only for Gaussian distributions are exact results known for continuous variables.
While the term B2 in our bound is similar to the negative of the ELBO bound in importance weighted autoencoders (IWAEs) 44 there are some differences between the two settings, the most important being that are interested in the precise value of the bound the minimum, rather than the learned
functions ca, rd. Note also that our latent variables (k) are one-dimensional, as opposed to the usual higher dimensional latent distributions of variational autoencoders, and that the empirical expectation over data observations in IWAEs is replaced in Bg by the expectation over co(U1, W2). whose parameters are also optimized.
Estimating the other PID terms In the following adopt the minimal value taken by the upper bound (3.20) as our estimate of U1. The other terms in the partial information decomposition obtained from the consistency relations (L.1..1.3)) after estimating the mutual informations I(Y: : (X,, X2)), I(Y: : I,), X2). There are several methods for the latter. In our examples, the observed data fit additional copulas c(u,, u2) and C12)9(u,) and estimate T(Y Xi)
I El log c(uy ( and similarly for the other terms. Note that all our estimates have sources D
of potential bias. Firstly, the estimation of the parametric copulas subject to model or parameter misspecification, which can be ameliorated by more refined model selection strategies. Secondly, the optimized bound might not saturate, biasing the estimate upwards. This can be improved using higher A values and improving the gradient-based optimizer used.
4 Examples
Comparison with exact results for Gaussians. Consider three-dimensional Gaussian with corre- lations Py.e; between y, Hz for i = 1,2. The exact solution to (I.5) in this case is 33
1 - 1-P
U(Y: X X2) log
1py.a < Py.a.l
(4.1)
Fig. compares the above expression with estimates from our method. Here we know that Cy.1 and C222 are Gaussian copulas, with parameters Py. Py,ag, and we assumed a Gaussian copula for C1,2/y.0 (Milys U2ly) well. For each pair of values P In this and the rest of the experiments, optimized the parameters (8, Φ) using the ADAM algorithm [47] with fixed learning rate 10-2 during 1200 iterations, and using A = 50. The results reported correspond to the mean the bound in the last 100 iterations. The comparison in Fig. shows excellent agreement.
Model systems of three neurons. The nature of information processing of neural systems prominent area of application of the PID framework, since synergy has been proposed as natural measure of information modification 48.. We consider two models:
MI
M2
(X1, ~ NN y= (ttann(w)) w1 X1 W2 X2).
(X,, N NO0.
(4.2)
X2/ wi w

---

1.0
SLY-1K,
0.8
RD,:DAG Kail
0.8
0.5
0.4
0.4
0.2
0.2
0.4
0.2 82 0.8
12
Figure 2: Partial information decomposition for two neural network models. In both mod- els (42)) we fixed wi = 0.5, P12 = 0.3, and show the PID terms as function of the synaptic strength W2. normalized by I(Y: (X1, We show mean (lines) and standard deviations (shaded area around each line) from 3 runs. Left: Model 1: The input of greatest weight conveys all the unique information, and synergy and redundancy both peak wh U22 Right: Model 2: The second input X2 has negligible unique information contribution, but its synaptic strength we modulates the synergistic term, associated to the modification of information the neuron performs [48]
Both models are parameterized by the correlation P12 and weights W1, W2. Model 1 is a particularly simple neural network. The tanh activation does not affect its copula, and even for linear activation function the variables are not jointly Gaussian since Y is deterministic on (X1, X2). Model 2 is
inspired by a normalization operation widely believed to be canonical in neural systems [49] and plays role in common learned image compression methods [50]. The results, presented in Figure 2] are obtained from 3000 samples from each model
Computational aspects of connectivity in recurrent neural circuits. We apply our continuous
variable PID understand computational aspects of the information processing between recurrently coupled neurons (Fig. 3 A large amount of work has been devoted to applying information theoretic
measures for quantifying directed pairwise information transfer between nodes in dynamic networks and neural circuits (51) However, classical information theory only allows for the quantification of information transfer, whereas the framework of PID enables further decomposition of information processing into transfer, storage, and modification, providing further insights into the computation within recurrent system (52]. Transfer entropy (TE) [53] is a popular measure to estimate the directed transfer of information between pairs of neurons 54 55], and is sometimes approximated by linear Granger causality. Intuitively, TE between a process X and a process Y measures how much the past of X, X-, can help to predict the future of Y, Y+, accounting for its past Y- Although TE quantifies how much information is transferred between neurons, it does not shed light on the computation emerging from the interaction of X and Y Simply put, the information transferred from X- could enter Y+, independently of the past state Y-, or it could be fused in a non-trivial way with the information in the state in Y 522 [56). PID decomposes the TE into modified transfer (quantified by S(Y+:X-,,) Y-)) and unique transfer (quantified by U(Y+: U the Appendix for a proof):
(Y-)) terms (see
TE(X + Y)
(Y))
Furthermore, the information kept by the system through time can quantified by the unique storage (given by U(Y+:Y- X-)) and redundant storage (given by R(Y+:X-,Y in PID (8). This perspective a new step towards understanding how the information is processed in recurrent systems beyond merely detecting the direction functional interactions estimated by traditional TE methods (see Appendix G for details). To explore these ideas, we simulated chaotic networks of rate neurons with an a-priori causal structure consisting of two sub-networks X and Y (Fig 3a, see [57] for more details on causal analyses this network model). The sub-network X a Rossler attractor of three neurons obeying the dynamical equations:
X1 = X3
X₂ = X1+ X
(4.3)
X3 = 3 - X3(X1 -7)
7

---

(a)
Signals
MMU
200
600
800
1000
Time
(b)
(c)
Xeos: (x)
X.)
Storage
Yeos: (Xp Yg)
Redundant (R)
Unique (U2)
(d)
Transfer
Transfer Entropy
S+U1
Modified (S)
Unique (U1)
Figure 3: PID uncovers the effective connectivity and allows for the quantification of storage, modification, and transfer of information in chaotic network of rate neurons. Schematics of recurrent network architecture (left) and representative activity (right). b: Schematic of the PID) triplets for each 3 block of the matrices c, d. e: PID decomposition into modified transfer S. unique transfer U1, redundant storage R, and unique storage U2 for the rate network. The future of X neurons only depends on unique information the past of X neurons and their synergistic interactions. The interactions between the X and Y sub-networks only contain synergistic information regarding the future of but no redundant information; the latter only present the interactions confined within each sub-network. d: The transfer entropy (TE), estimated via IDTxl [58). recovers the sum of modified and unique transfer terms S + U1.
where {a, 3. 0.2.7. There are 100 neurons the sub-network Y from which chose the first three, Y1.3. to simulate the effect of unobserved nodes. Neurons within the sub-network obey the dynamical equations
= + 10 10tannnJyyx + Jyy
(4.4)
where Jyx E R 1000x has all its entries equal to 0.1, and Jyy is the recurrent weight matrix of the Y sub-network, sampled as zero-mean, independent Gaussian variables with standard deviation 4. No projections exist from the downstream sub-network Y to the upstream sub-network X. We simulated time series from this network (exhibiting chaotic dynamics, see Fig. (3p) and estimated the PID as unique, redundant, and synergistic contribution of neuron i and neuron, at time / in shaping the future of neuron j at time + 1. For each pair of neurons Z,, Zz E (X1:,, Y1:3) we treated as iid sample and ran PID on these triplets (i,j represent rows and columns in Fig. (3-d). The PID uncovered the functional architecture of the network and further revealed non-trivial interactions between neurons belonging to the different sub-networks, encoded in four matrices: modified transfer S, unique transfer U1. redundant storage R, and unique storage U2 (details in Fig, (d). The sum of the modified and unique transfer terms was found to be consistent with the TE (Fig. 3 TE equal connectivity, consistent with previous results 55 57
(Z), D,, 2 Z "
U1, up estimation bias). The TE itself captured the network effective
Uncovering a plurality of computational strategies in RNNs trained to solve complex tasks. A fundamental goal in neuroscience to understand the computational mechanisms emerging from
*Note that the estimation of the PID from many samples of the triplets (Z), Zz 2 is operationally the same whether such triplets are or, as in our case, temporally correlated. This is similar to estimating expectations w.r.t. the equilibrium distribution a Markov chain by using temporally correlated successive values of the chain. In both cases, the temporal correlations do not introduce bias in the estimator but can increase the variance.

---

(a)
(b)
(c)
readout
1.0
K-4
WW
K=
0.0
K=8
K-10
y=1
0.5
y=0 0
b-O Stimuuus
D=10
Time
Repert
(d)
5
R
(e)
250
Figure 4: PID RNNs trained to solve generalized XOR problem. a: Input data drawn from 2D Gaussian Mixture Model with K mixture components X ~ Σx= N [as, a/) with means lying on the unit circle (grey and black dots represent the two class labels). b: Two layer network with 2D input layer, 5 recurrently connected hidden neurons X and one readout neuron Y; RNN activity unfolds in time (horizontal axis). The imput is presented at time is trained with BPTT to report the decision at f = 10. In this representation, layers correspond to time-steps and weights Wxx are shared between layers. PID between output (t) and pairs of hidden neurons (t X, 1) for 10 yielding S, R. U1. U2 (distribution over 1000 input samples for each task K; 20 networks per task). Harder tasks led an increase PID measures. d: Example receptive fields for network with U > S shows emergence of grand-mother cells in the hidden layer (red and blue colors represent hidden neurons outputs; grandmother cell, second from left). Example receptive fields for a network with S > U, relying on higher synergy between neurons to solve the task.
O, then withdrawn, and the RNN
the collective interactions of recurrent neural circuits leading to cognitive function and behavior. Here, we show that PID opens a new window for assessing how specific computations arise from recurrent neural interactions. Unlike MI TE, the PID quantifies the alternative ways in which neuron determines the information in its output from its inputs, and thus can sensitive marker of different computational strategies. We here trained RNNs models of cortical circuits (59) and used the PID to elucidate how the computations emerging from recurrent neural interactions contribute task performance. We trained RNNs solve generalized version of the classic XOR classification problem with target labels corresponding to odd vs. even mixture components (Fig. 4D) Stimuli were presented for one time step 0) and the network was trained to report the decision at t == 10. By tracking the temporal trajectories of the hidden layer activity we found that the network recurrent dynamics (represented as unfolded in time Fig. 4b) progressively pulls the two input classes in opposite directions along the output weights (see Appendix). We used PID to dissect how plurality of different strategies emerge from recurrent neural interactions in RNNs trained for solving a classification task. The computation emerged from the recurrent interaction between hidden neurons at different time steps. Do all successfully trained networks have similar profile in terms of the PID terms? If so, this hints at single computational strategy across these networks. If not, it is safe to assume that task performance reached via different mechanisms, despite identical network architecture and training algorithm.

---

We found that on average across multiple networks S. R, and U rose with task difficulty (Fig. 4. yet at all difficulties, individual networks differed strongly with respect the ratio S/U, i.c. there were networks with larger average synergy across neuron pairs compared to the average unique information, and vice versa. For simple networks like the ones used here, one can inspect receptive fields to understand the reason for this differential behaviour (Fig. fl-e). Indeed, networks with high average unique information displayed "grandmother-cell" -like neurons, that would alone classify large parts of the sample space, while in networks with higher average synergy such cells were absent (Fig. 4d). The emergence of these "grandmother-cell -like receptive fields is due to the recurrent dynamics. While in feedforward architecture (Wxx = 0) hidden layer receptive fields are captured by hyperplanes in input space, in the RNN the receptive fields are time dependent, where later times are interpreted as deeper layers (Fig. Ap) and thus can capture highly non-lineat features in input space. The advantage of PID versus a manual inspection of receptive fields is twofold: First, the PID framework abstracts and generalizes descriptions of receptive fields as being e.g. "grandmother-cell" -like; thus the concept of unique information stays relevant even in scenarios where the concept of receptive field becomes meaningless, or inaccessible. Second, the quantitative outcomes of a PID rest only on information theory, not specfic assumptions about neural coding or computational strategies, and can be obtained for large numbers of neurons.
Comparison of our PID-based approach with the concept of neuronal selectivity used in neuroscience highlights interesting similarities and differences. Several kinds of selectivity (pure, mixed linear, and mixed non-linear) can be identified by performing regression analysis of neural responses vs. task variables [60]. In this framework, our grand-mother cells correspond to neurons with pure selectivity to the input class labels (a.k.a. schoice-seecctiig that non-lineat mixed selectivity of neurons to the class labels is beneficial when solving the XOR task, by leading to a high- -ighdiinnniioaal representation of the task variables. While selectivity profiles a property of single neuron responses to task variables, our PID measures are property of the combined activity of triplets of neurons and thus reveal emerging functional interactions between units and their computational algorithms (see also 7 and 52 This allowed to characterize functional property of neural systems less studied than task variable selectivity: the computations that require functional mixing of the information from multiple units (measured by the average synergistic information) vs. the computations that rely on the output of individual neurons (measured by the unique information and described as grandmother cells). Concretely, by comparing PID and receptive fields we found that that in networks with high unique information, neurons typically have receptive fields with pure selectivity (grandmother cells, with large unique information the class labels). In networks with high synergy, neurons show complex mixed selectivity to class labels.
neurons). In the XOR task, (60] showed
5 Conclusions
We presented partial information decomposition measure for continuous variables with arbitrary probability densities, thereby extending the popular BROJA PID measure for discrete variables. Extending PID measures to continuous variables drastically broadens the possible applications of the PID framework. This is important as the latter provides key insights into the way complex system represents and modifies information information about target uniquely (such that it can only be obtained from that variable), redundantly. or only synergistically with other variables. Answering these questions is pivotal to understanding distributed computation in complex systems in general, and neural coding in particular. We believe that the methods presented here will allow PIDs to be extended efficiently in neuroscience for multiple continuous sources with potentially complex dependency structures, as would be common in cellular imaging data or activation properties of brain modules or areas in functional imaging. More generally, the approach we presented here would be relevant for other application domains such as machine learning, biomedical science, finance, and the physical sciences.
computation via asking which variables carry
10

---

Acknowledgments
We thank Thibault Vatter and Praveen Venkatesh for conversations. The work of AP is supported by the Simons Foundation, the DARPA NESD program, NSF NeuroNex Award DB11 707398 and The Gatsby Charitable Foundation. DG is supported by Swartz Fellowship. AM and MW supported by Volkswagenstifturg under the program 'Big Data in den Lebenswissenschafter and by the Ministry for Science and Education of Lower Saxony and the Volkswagen Foundation through the Niedersächsisches Vorab'. LM is supported by NINDS Grant NS118461 (BRAIN Initiative). ES supported by the Simons Collaboration the Global Brain (542997) as well as research support from Martin Kushner Schnur and Mr. and Mrs. Lawrence Feis, and is the Joseph and Bessie Feinberg Professorial Chair.
References
[1] Paul L Williams and Randall D Beer, Nonnegative decomposition of multivariate information. arXiv preprini arXiv: 1004.2515, 2010.
[2] Tycho Tax, Pedro AM Mediano, and Murray Shanahan. The partial information decomposition of generative neural network models. Entropy, 19(9):474, 2017.
[3] Patricia Wollstadt, Sebastian Schmitt, and Michael Wibral. A rigorous information-thcordee definition of redundancy and relevancy in feature selection based on (partial) information decomposition. arXiv preprint arXiv:2105.0413 2021.
[4] Allison Goodwell et al. Debates-dock information theory provide a new paradigm for earth science? Water Resources Research, 56())=2009WRR024944 2020.
[5] Benjamin Flecker, Wesley Alford, John M Beggs, Paul L Williams, and Randall D Beer. Partial information decomposition
spatiotemporal filter. Chaos, 21(3):037104, 2011.
[6] Michael Wibral, Joseph T Lizier, and Viola Priesemann. Bits from brains for biologically inspired computing. Frontiers in Robotics and AI, 2:5, 2015.
[7] Nicholas M Timme, Shinya Ito, Maxym Myroshnychenko, Sunny Nigam, Masanori Shimono, Fang-Chin Yeh, Pawel Hottowy, Alan M Litke, and John M Beggs. High-degree neurons feed cortical computations. PLoS computational biology, 12(5)ee104885,, 2016.
[8] Michael Wibral, Viola Priesemann, Jim W Kay, Joseph T Lizier, and William A Phillips. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain and cognition, 1 11:255-34 38, 2017.
[9] Giuseppe Pica, Eugenio Piasini, Houman Safaai, Caroline Runyan, Christopher D Harvey, Mathew E Diamond, Christoph Kayser, Tommaso Fellin, and Stefano Panzeri. Quantifying how much sensory information in a neural code is relevant for behavior. In 2017.
[10] Jim W Kay, WA Phillips, Jaan Aru, Bruce P Graham, and Matthew Larkum. A Bayesian decomposition of BAC firing as mechanism for apical amplification in neocortical pyramidal neurons. bioRxiv, page 604066, 2019.
[11] Itay Gat and Naftali Tishby. Synergy and redundancy among brain cells of behaving monkeys. In Advances in neural information processing systems, pages 111-117, 1999.
[12] Naama Brenner, Steven P Strong, Roland Koberle, William Bialek, and Rob R de Ruyter van Steveninck. Synergy in a neural code. Neural computation, 12(7):153115522. 2000.
[13] Elad Schneidman, William Bialek, and Michael J Berry. Synergy, redundancy, and independence in population codes. Journal of Neuroscience, 23(37), 2003.
[14] Rodrigo Quian Quiroga and Stefano Panzeri. Extracting information from neuronal populations: information theory and decoding approaches. Nature Reviews Neuroscience, 10(3)_173-185 2009.
[15] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 16(4):2161-2183, 2014.
[16] Blackwell. Comparison of experiments. Proc. 2nd Berkeley Symp. Math. Stats, and Probabil- ity, 1951.

---

[17] Moshe Leshno and Yishay Spector. An elementary proof of Blackwell's 's theorem. Mathematical Social Sciences, 25(1):95-98, 1992.
[18]
Erik Torgersen. Comparison of statistical experiments. Cambridge University Press, 1991. Lucien Le Cam. Comparison of experiments: A short review. Lecture Notes-Monograph Series, pages 127-138, 1996.
[19]
[20] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, and Jürgen Jost. Shared information new insights and problems in decomposing information in complex systems. In Thomas Gilbert, Markus Kirkilionis, and Gregoire Nicolis, editors, Proceedings of the European Conference on Complex Systems 2012, pages 251-269, Cham, 2013. Springer International Publishing.
[21] Robin AA Ince. Measuring multivariate redundant information with pointwise common change in surprisal. Entropy. 19(7):318, 2017.
[22] Conor Finn and Joseph T Lizier. Pointwise partial information decomposition using the specificity and ambiguity lattices. Entropy, 20(4):297, 2018.
[23] Malte Harder, Christoph Salge, and Daniel Polani. Bivariate measure of redundant information. Physical Review E, (1))0210 2013.
[24] Johannes Rauh. Secret sharing and shared information. Entropy, 19(11)6001, 2017.
[25]
Abdullah Makkeh, Aaron J Gutknecht, and Michael Wibral. Introducing a differentiable measure of pointwise shared information. Physical Review E, 103(3):032149 2021.
[26] Aaron J Gutknecht, Michael Wibral, and Abdullah Makkeh. Bits and pieces: Understanding information decomposition from part-whole relationships and formal logic. Proceedings of the
Royal Society A, 477(225)): 10, 2021.
[27] Pradeep Kr Banerjee, Johannes Rauh, and Guido Montúfar. Computing the unique information. In ISIT, 2018
[28] Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. Bivariate partial information decompo- sition: The optimization perspective. Entropy, 19(10):530, 2017.
[29] Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. Broja-2pid: robust estimator for bivariate partial information decomposition. Entropy, 20(4):271, 2018.
[30] Johannes Rauh, Maik Schünemann, and Jüergen Jost. Properties of unique information. arXiv preprint arXiv: 1912. /2505, 2019.
[31] Adam Barrett. Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Physical Review E, 91(5), 2015.
[32] Gabriel Schamberg and Praveen Venkatesh. Partial Information Decomposition via Deficiency for Multivariate Gaussians. arXiv preprint arXiv:2 00769, 2021.
[33] M Sklar. Fonetions de repartition n dimensions leurs marges. Publ. inst, statist. univ. Paris, 8:229-20 201, 1959.
[34] Harry Joe. Multivariate Models and Multivariate Dependence Concepts. CRC Press, May 1997.
[35] Gal Elidan. Copulas in machine learning. Copulae mathematical and quantitative finance, pages 39-60. Springer, 2013.
[36] Rafael S Calsaverini and Renato Vicente. An enformation-ttherrel approach to statistical dependence: Copula information. EPL (Europhysics Letters), 88(6))68003, 2009.
[37] Jian Ma and Zengqi Sun. Mutual information is copula entropy. Tsinghua Science & Technology. 16(1):51-54, 20
[38] Tim Bedford and Roger M Cooke. Probability density decomposition for conditionally depen- dent random variables modeled by vines. Annals of Mathematics and Artificial intelligence, 32(1-4):245-268, 2001.
[39] Kjersti Aas, Claudia Czado, Amoodo Frigessi, and Henrik Bakken. Pair-copula constructions of multiple dependence. Insurance: Mathematics and economics, 44(2):182-198, 2009.
[40] Claudia Czado. Pair-copula constructions of multivariate copulas. In Copula theory and its applications, pages 93-109. Springer, 2010.
12

---

[41] Thomas Nagler and Claudia Czado. Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas. Journal of Multivariate Analysis, 151:69-89, 2016.
[42] Roger B Nelsen. An Introduction to Copulas. Springer Science & Business Media, June 2007.
[43] Thomas Nagler and Thibault Vatter. pyvinecopulib, November 2020.
[44] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders.
ICLR 2016, 2016.
[45] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. NIPS 2015.
[46] George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J Maddison. Doubly reparameterized gradient estimators for Monte Carlo objectives. ICLR, 2019.
[47] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR 2015.
[48]
Joseph T Lizier, Benjamin Flecker, and Paul L Williams. Towards a synergy-based approach to measuring information modification. In 2013 IEEE Symposium on Artificial Life (ALIFE), pages 43-51. IEEE, 2013.
[49] Matteo Carandini and David Heeger. Normalization as a canonical neural computation. Nat. Rev. Neurosci,, 13(1):51-62, November 2011.
[50] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compres-
sion. ICLR, 2017.
[51] Andrew T Reid, Drew B Headley, Ravi D Mill, Ruben Sanchez-Romero, Lucina Q Uddin, Daniele Marinazzo, Daniel J Lurie, Pedro A Valdés-Sosa, Stephen José Hanson, Bharat B Biswal, et al. Advancing functional connectivity research from association to causation. Nature 2019.
neuroxcience, 22(1
[52] Michael Wibral, Conor Finn, Patricia Wollstadt, Joseph T Lizier, and Viola Priesemann. Quanti- fying information modification in developing neural networks via partial information decompo- sition. Entropy, 19(9);494, 2017.
[53] Thomas Schreiber. Measuring information transfer. Physical review letters, 85(2):461, 2000.
[54] Raul Vicente, Michael Wibral, Michael Lindner, and Gordon Pipa. Transfer entropy- model-free measure of effective connectivity for the neurosciences. Journal of computational neuroscience, 30(1):45-44))) 67, 2011.
[55] Leonardo Novelli and Joseph T Lizier. Inferring network properties from time series using transfer entropy and mutual information: validation of multivariate versus bivariate approaches. Network Neuroscience, 5(2):373-40X 2021.
[56) Paul L Williams and Randall D Beer. Generalized measures of information transfer. arXiv preprint arXiv: 102. 1507, 2011.
[57] Amin Nejatbakhsh, Francesco Fumarola, Saleh Esteki, Taro Toyoizumi, Roozbeh Kiani, and Luca Mazzucato. Predicting perturbation effects from resting state activity using functional causal flow. bioRxiv. 2020.
[58) Patricia Wollstadt, Joseph T Lizier, Raul Vicente, Conor Finn, Mario Martinez-Zarzuela, Pedro Mediano, Leonardo Novelli, and Michael Wibral. Idtxl: The information dynamics toolkit xl: a python package for the efficient analysis of multivariate information dynamics in networks.
Journal of Open Source Software, 4(84): 2019.
[59] Valerio Mante, David Sussillo, Krishna Shenoy, and William Newsome, Context-dependent computation by recurrent dynamics in prefirontal cortex. nature. 503((474):78 84, 2013.
[60] Mattia Rigotti, Omri Barak, Melissa R Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, (977(751):585--55
2013.
[61] Tom Rainforth, Adam Kosiorek, Tuan Anh Lc, Chris Maddison, Maximilian Igl, Frank Wood, and Yee whye Teh. Tighter variational bounds are not necessarily better. In ICML, 2018
[62] Murray Rosenblatt. Remarks multivariate transformation. The annals of mathematical statistics, 2703):470-47,, 1952.
13


Reviews:
[{"review_id": "wD8Mp9015Ny", "reviewer": "Reviewer_BEa7", "summary": "   This paper presents a novel method for computing the Partial\n   Information Decomposition (BROJA flavour) in the case of three\n   continuous random variables. The method uses a variational approach\n   on a lower bound derived from a copula representation of the joint\n   PDF of the three variables. The paper presents some application\n   examples, including two where the proposed method is used to\n   elucidate the properties of recurrent neural networks.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important issue standing in the way of\npractical application of PID to the analysis of real data. To the best\nof my knowledge, the approach is novel, and I could not identify any\nformal or technical problem. The paper is well written and for the\nmost part does a good job of describing the relationship of this work\nto existing literature. I particularly appreciated the very accessible\nsummary of the decision-theoretic foundation of the BROJA PID. Overall\nthis is in my opinion a good and useful paper that would benefit the\nNeurIPS community.\n\nHaving said the above, I would like to bring two points to the\nattention of the authors.\n\n1. While reading the paper, I got the distinct impression that the\n   title of the paper is somewhat misleading, in the sense that to me\n   the \"meat\" of the paper is clearly the novel PID estimation method,\n   while the recurrent network side is more of a demonstration of\n   potential usefulness. I was confused by this until I happened to\n   see the submission history of the paper (that is, that the RNN side\n   was added partly to address previous criticisms). While I think\n   that the RNN results are interesting, I am of the opinion that the\n   paper would be more likely to reach an interested audience / more\n   likely to have a bigger impact if the title was changed to\n   something more related to the PID itself (as I suspect that the\n   original title must have been in the previous submission). In the\n   same vein, the neural-networks-first presentation in the\n   introduction may be softened.\n2. I couldn't find a link to a software implementation of the proposed\n   technique. I don't think this is a hard requirement, but it would\n   certainly be of great help for the interested reader, even if it is\n   just unpolished set of functions/scripts that allow to at least to\n   reproduce some of the results in the paper.\n\nMinor points:\n- Lines 86--87: \"PID has been previously applied to in vitro neural\n  data to elucidate how single neuron computational properties depend\n  on network topology [ref to Timme et al, PLoS comp bio 2016]\". If I\n  am not mistaken, the PID in Timme et al is based on the old Williams\n  and Beer approach. By contrast, the method proposed by the authors\n  in the current paper is based on the three-variable BROJA PID. An\n  early application of the BROJA PID in neuroscience is Pica et al,\n  NeurIPS 2017, \"Quantifying how much sensory information in a neural\n  code is relevant for behavior.\", so the authors may consider adding\n  this reference too.\n- Line 187: missing period at the end of the paragraph.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "mIArG86chcX", "reviewer": "Reviewer_VXUZ", "summary": "Authors present a partial information decomposition (PID) method that can be applied to continuous variables thereby extending a popular approach for analysis of discrete variables. Authors validate their method  by leveraging known analytic results for Gaussian random variables. Additionally, the authors demonstrate that their approach is able to recover the effective connectivity of a neuronal network with chaotic dynamics. Finally, authors use their method to investigate a complex trade-off between redundant, synergetic, and unique information components in recurrent networks that have been trained to solve the XOR problem.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The algorithm presented in this manuscript is derived from an existing one for discrete variables, but the authors manage to extend the method to include continuous variables, thereby producing a novel and widely applicable technique. There are no obvious flaws that I can see in the theory and experiments presented in this paper. However, I did not review the theory in detail and it is not impossible that I have missed many important details. The paper is written clearly enough and provides enough experimental evidence to back the main claims introduced by the authors. PID for continuous variables can have a number of applications (specially in the field of neuroscience) as indicated by the authors and therefore the contributions presented in this manuscript are quite significant. An interesting addition to this paper would be to add an example application of the method using real data (from the brain), but I do not see this as requirement for this manuscript to be accepted in the conference.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Z_V-tLsDmqA", "reviewer": "Reviewer_7Ede", "summary": "(I am not an expert in information theory)\nThe authors extend partial information decomposition to the case of continuous variables with arbitrary distributions. \nMy review will focus on the last section, where a recurrent neural network was trained on a generalized XOR task. By decomposing the unique, redundant and synergistic components of each neuron’s contribution to the output, the authors identify different strategies used by these networks. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not an expert in information theory, and will therefore focus on the last section – studying a recurrent neural network trained on a generalized XOR task.\n\nThe question of algorithm identification in trained networks is both important and challenging. One aspect of this challenge is the difficulty to define what it means to identify an algorithm. Is it some connectivity motif? An arrangement of fixed points? The geometry of phase space trajectories? The response to novel inputs?\n\nGiven this state of affairs, novel suggestions on how to characterize algorithms can be useful. In the specific example given here, however, the authors did not clearly define what they mean by algorithm. The analysis shows that the properties of individual neurons differ both within networks and across networks. Specifically, some single neurons in the recurrent layer seem to already encode most of the desired solution to the XOR problem.\n\nIn terms of an algorithm, I find this description lacking. How do these neurons obtain such a representation? By definition, the output of the network will have this property. Finding such internal neurons merely moves the question, but does not answer it.\n\nFurthermore, this notion is very similar to that of mixed selectivity (Fusi et al. Curr opinion neuro, 2016). This might imply that linear dimensionality, which is much simpler to compute, could provide similar information on the networks.\n\nI also found figure 7C puzzling. For K=6, both neurons 1 and 5 seem to have very similar receptive fields and can be described as grandmother cells. Neuron 5 has a higher mutual information, and neuron 1 has a higher unique information. It could be useful to use this case to explain the differences and how they originate.\n\nFurthermore, the networks analyzed here are very small (5 neurons) compared to those cited in line 227 (100 neurons). How does this method scale?\n\nOverall, this specific example does not demonstrate the utility of this method in inferring algorithms from trained networks.\n\nOnce again, I would like to stress that this review only relates to one small aspect of the paper.\n\n\n**EDIT: Score raised from 4 to 7 after reading all reviews and author responses.**\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "not relevant", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "QbSDeKvC5VZ", "reviewer": "Reviewer_ToUB", "summary": "This paper extends the popular PID measure of Bertschinger et al. (2014) to continuous random variables, and provides a method to compute and estimate the PID in this case. The authors use a novel method based on copulas to reparameterize the optimization problem used to compute the PID, and estimate it by using an approximation based on a variational upper bound. They also provide several examples to show that their estimates match known PID quantities and that the PID can be useful in various neuroscientifically relevant settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the technical section of the paper remains hard to read for an audience unfamiliar with copulas, despite the efforts of the authors to introduce these concepts. The notation is often times confusing and challenging to parse. However, I believe the paper is of substantial significance, since no methods to compute or estimate PIDs on general continuous distributions currently exist. As the experimental results outlined in the paper illustrate, such a method could be of great interest in several application domains, particularly in neuroscience.\n\nOriginality:\nThe paper is very original, since there are no methods to estimate PIDs for general continuous distributions. Most works discuss computation of the PID, assuming full knowledge of the distributions, rather than estimating the PID, which requires the additional step of inferring distributions from data.\n\n(Technical) Quality:\nBarring the issues with clarity of presentation and notation (which are discussed in detail below), I believe the paper is of high technical quality: the authors have used a number of concepts and have been thorough in their references: they provide a good sampling of the relevant background and literature for each concept. The idea of using copulas for the Bertschinger et al. PID is intuitively appealing.\nI have one technical question: in Figure 1, why is there a large difference in the left bottom corner? The left-bottom should be where the question is easiest: $\\rho_{Y, X_2}$ is close to 1 and $\\rho_{Y, X_1}$ is close to zero, so it should be easy to determine that the unique information (in $X_1$) is zero.\n\nClarity:\nI believe the technical section can be significantly improved in its presentation:\n1. It would be useful to have an short outline at the start of Section 3 briefly describing the steps involved in estimating the unique information. Otherwise, it is hard to see what the various sub-parts are leading towards.\n2. In equation 3.3, is $c(u_y, u_2)$ the distribution that results from summing $c(u_y, u_1, u_2)$ over $u_1$? Please clarify.\n3. Line 107: What does the term \"copula degrees of freedom\" mean? This has not been introduced yet.\n4. Equations 3.5 and 3.8: it is not immediately obvious how the conditional distribution can be written in terms of the copula. An extra line explaining this near Equation 3.1 may be useful.\n5. Line 111: \"Note that the function ...\" - it might be useful to have this sentence appear a little earlier, just after Equation 3.8.\n6. Line 119: \"parameterizing by $\\theta$\"  - it remains unclear after this sentence exactly what $\\theta$ parameterizes. If $\\theta$ is the same as $Q$, why not use that instead?\n7. Line 121: $\\theta(y)$ is now being used as a function. Please clearly explain what $\\theta$ is.\n8. Equation 3.14: $I_q[\\theta]$ is confusing notation, since $q$ does not appear in the RHS. Also, the second term in the RHS implicitly converts an expectation over three variables to one over two. This is only valid if summing $c_{\\theta}(u_y, u_1, u_2)$ over $u_y$ is equal to $c_{\\theta}(u_1, u_2)$.\n9. Line 128: Does the $D \\log D$ cost come from sorting the variables for computing the CDF? Please clarify.\n10. Please mention how long the estimation algorithm took to run, and on what hardware. Running time is an important aspect of the reusability of this work.\n11. Line 214: \"We treated ... as iid samples and ran PID on these triplets\" - why is this a sensible thing to do for the question being explored? Please explain briefly.\n\nMinor comments:\n1. The negative vspace after title should be removed.\n2. Line 81: \"exits\" should be \"exist\"\n3. Figure 3b: Bottom-right quadrant has $X_t$ in place of $Y_t$\n4. Line 251: \"... PID framework abstracts _and_ generalizes ...\"\n\nSignificance:\nI believe this paper is very significant: it provides the first usable PID estimator for general continuous random variables, and is quite likely to be used by a number of works seeking to apply PIDs in different contexts.\n\n**Update after author responses**\n\nBased on the authors' responses, I am happy to fully recommend accepting the paper, and I have raised my score accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations have been adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a new method for partial information decomposition that was found to be novel and interesting. Given the narrowing of the scope (away from the RNN application and more focused on the PID estimation) proposed by the authors in response to the reviewers initial comments, the reviewers agreed that the work was relevant to the NeurIPS community and that the method and applications would be greatly appreciated. I thus am happy to recommend this paper be accepted. 
Decision: Accept (Poster)