Analytically deriving Partial Information Decomposition for affine systems of stable and convolution-closed distributions
Authors: Chaitanya Goswami
Amanda Merkley
Venue: NeurIPS 2024 poster (2024)
Keywords: Partial Information Decomposition
Neuroscience
Multimodal learning
Analytical PID
Primary area: neuroscience_and_cognitive_science
TL;DR: Analytical calculation of Partial Information Decomposition for various well-known distributions, such as Poisson, Cauchy, Gamma, Exponential and many more.

Abstract: Bivariate partial information decomposition (PID) has emerged as a promising tool for analyzing interactions in complex systems, particularly in neuroscience. PID achieves this by decomposing the information that two sources (e.g., different brain regions) have about a target (e.g., a stimulus) into unique, redundant, and synergistic terms. However, the computation of PID remains a challenging problem, often involving optimization over distributions. While several works have been proposed to compute PID terms numerically, there is a surprising dearth of work on computing PID terms analytically. The only known analytical PID result is for jointly Gaussian distributions. In this work, we present two theoretical advances that enable analytical calculation of the PID terms for numerous well-known distributions, including distributions relevant to neuroscience, such as Poisson, Cauchy, and binomial. Our first result generalizes the analytical Gaussian PID result to the much larger class of stable distributions. We also discover a theoretical link between PID and the emerging fields of data thinning and data fission. Our second result utilizes this link to derive analytical PID terms for two more classes of distributions: convolution-closed distributions and a sub-class of the exponential family. Furthermore, we provide an analytical upper bound for approximately calculating PID for convolution-closed distributions, whose tightness we demonstrate in simulation.

Analytically deriving Partial Information Decomposition for affine systems of stable and convolution-closed distributions
Chaitanya Goswami
Amanda Merkley
Electrical & Computer Engineering, Carnegie Mellon University, Pittsburgh, PA-15213.
Electrical & Computer Engineering.
Carnegie Mellon University, Pittsburgh, PA-15213. amerkley@andrew cmu, edu
cgoswami@andre Sandrew cmu edu
Abstract
Bivariate partial information decomposition (PID) has emerged as a promising tool for analyzing interactions in complex systems, particularly in neuroscience. PID achieves this by decomposing the information that two sources (e.g., different brain regions) have about target (e.g., stimulus) into unique, redundant, and synergistic terms. However, the computation of PID remains a challenging problem, often involving optimization over distributions. While several works have been proposed to compute PID terms numerically, there is surprising dearth of work on computing PID terms analytically. The only known analytical PID result is for jointly Gaussian distributions. In this work, we present two theoretical advances that enable analytical calculation the PID terms for numerous well- known distributions, including distributions relevant to neuroscience, such Poisson, Cauchy, and binomial. Our first result generalizes the analytical Gaussian PID result to the much larger class of stable distributions. We also discover theoretical link between PID and the emerging fields of data thinming and data fission. Our second result utilizes this link to derive analytical PID terms for two more classes of distributions: convolution-closed distributions and a sub-class of the exponential family. Furthermore, for approximately computing PID for convolution-closed distributions, whose tightness demonstrate in simulation.
provide analytical upper bound
Introduction
Bivariate partial information decompositior (PID) (PID) is an information theoretic framework developed for answering a central inquiry in many neuroscientific and machine learning studies: how do two sources, X and Y. joinlly process information about a target ? PID answers this question by quantifying the M-specific information contained in different interactions between X and Y. Specifically, it decomposes
total information and have about M into four components: (i) the information about M contained uniquely in X, (ii) the information about M contained uniquely in Y, (iii the redundant information about M contained in both X and Y, and (iv) the synergistic information about M which arises from the interaction between X and Y.
The PID terms offer novel insights for understanding interactions within complex systems, particularly in neuroscience. For instance, PID has been used to understand the firing patterns of grid cells 212 and to study the flow of information in the visual cortex [3]. The following list of works [I
3
demonstrate the application of PID in studying diverse neuroscientific questions.
Throughout this work refer to bivariate PID PID.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Beyond neuroscience, PID has found applications in multimodal learning for interpreting model predictions 9 [0], fair machine learning for defining and quantifying bias
and However, the primary constraint hindering broader adoption of PID is the difficulty of computing and estimating the PID terms. BROIA-PID [16). widely applied PID framework, requires solving a constrained minimization problem over a set of probability distributions (see Sec. 2 This minimiza- tion problem can pose a considerable challenge, particularly when the underlying distributions are continuous. As a result, several recent works have been dedicated to providing numerical algorithms
understanding financial markets
[5]
that solve the aforementioned minimization problem exactly or approximately B 9 with more emphasis on the case of discrete distributions.
88
19, 20)
Despite significant progress in numerically calculating PID (typically through discrete approxima- tions), very few works exist on Sateltticaaly@aalllllllt PID. As of this writing, the only known analytical PID expressions exist for jointly Gaussian M, X, and Y [21 22). A fundamental property of the Gaussian system that one of the unique information (UT) terms its PID is guaranteed to be zero. This property greatly simplifies the computation of PID for Gaussian systems, as the rest of the PID terms can be easily derived by solving desirable linear equations specified in many PID frameworks (see Sec bypassing the need for optimizing over distributions (see Sec.
In this work. we show that numerous systems of random variables M, X, and Y expressing particular "affine dependence structure" also exhibit the same fundamental property that at least one of the UT terms is zero. Consequently, we expand significantly using this fundamental property to analytically compute PID for various systems of M, X, and Y employing well-known distributions such as Poisson, exponential, gamma, beta, negative binomial, multinomial, Cauchy, Lévy-stable and more. The main contributions of this work are:
the existing Gaussian PID result by
We extend the Gaussian PID result to a much larger class of distributions, known as the stable distribution family [23 24 in Sec. 4 These results provide the first known analytical PID for fat-tailed distributions.
2. We highlight theoretical link between PID calculation and the fields of data thinning [25 and data fission [26 in Sec. [ We utilize this link derive analytical PID terms for two more distribution families: convolution-closed distributions (25 and certain exponential family distributions 26
3. For convolution-closed distributions, we further derive an analytical upper bound the objective of the minimization used for computing BROJA-PID. We use this upper bound to approximately compute PID for systems of M, X, and Y having a non-affine dependence structure. We show the goodness of our approximation by a simulation study in Sec. 6
2 Background
We denote the set of all natural numbers, real numbers, and positive real numbers N, R. and RR,, respectively. Vectors are denoted by bold-faced font and an arrow, and matrices are denoted by bold-faced font. Define No- [d] = f1....... d} d E N. We denote I, and O₂ as d-dimensional vectors having all elements as 1 and 0, respectively. For brevity, the probability notations of the form P(A B) and P(A) are always understood to be as P(A = a = b) and P(A = a), respectively. The general term 'distribution' is used to refer to both probability density function (p.d.f.) and probability mass function (p.m.f.) (whichever is appropriate depending upon the context). The symbol A 11. B C denotes that and B conditionally independent given C, and similarly, A IL B implies A and B are independent. The Ly-norm is denoted by II(:)]pp Vp E [0, oo), and [[-] denotes the absolute function.
Notation:
{Ó]. Let La be the identity matrix of size x d and
PID Background: Suppose M, X,Y Y are random variables with joint distribution P(M, X.Y). According to 16 27 three desirable equalities for a bivariate PID are as follows:
(MM [X, = - + (M; + RI(M; SI;;
(1)
PSection 2 precisely detines the notion of "analytical calculating PID" adopted in this work.

---

Here, XX) and Ip(M;Y) Y) denote the mutual information [28] between M and X. and M and Y. respectively, under the distribution P(M, X. Y)). Similarly, (M; [X, YI) denotes the joint mutual information between M and [X; Y] under the distribution P(M, X, Y). TTMM:YY and UT(M; X) are the unique information about M in X and Y. respectively. ST(M; X.Y and RM(M; X, Y) denote the respective synergistic and redundant information shared between X and Y about M. The linear system defined in ( contains four variables in three equations. Hence, only one of UI, RI, or S1 need be defined to evaluate all four PID terms. Proposing a suitable definition M 351 and all our theoretical results applicable for any Blackwellian PID 22 [3)] or PID definition satisfying the assumption (*) of Bertschinger et al. [16] (see Appx. for a formal justification). For example, our results are applicable for the PID definitions proposed in [16 22 27 29 (3). In this work, discuss our results in the context of the BROIA-PID 16 38 which is the definition of PID adopted in previous
of PID is the focus of much research [6 27 29] 30 [1 32
33
works 3 9 20 :
U
= min [Q(M;X|Y) or equivalently UI(M;Y (X) = min (M; I|X), (2) QEAP
QEAr
where p-= X,M, conditional mutual information under the distribution Q(M, Note that the minimizing distribu- tions for both problems shown in (2) are the same [T6), and consequently, both minimization problems are equivalent. We refer the reader to (39 for a review on PID. In this work, we define "analytically calculating PID" as analytically solving 2 by providing an explicit construction of the minimizing distribution (M,, X, Y) that minimizes 2 Note that only the distributions Q" (M, Y) and P(M, X, Y) are needed to compute the BROJA-PID terms.
= ))(M,Y) and Io(M; is the
We briefly describe two distribution families used in Sec. Bec. Sec. (see Appx. N for more details).
Stable distribution family: Stable distributions are family of distributions that naturally arise in the context of generalized central limit theorems. Some well-known members of this family are the Gaussian, Cauchy, Poisson, and Lévy distributions. A defining feature of stable distributions that the sum of two independent copies of a random variable X, denoted as Xj and X2, has the same distribution as translated and scaled version of X [23 24 40) In this work, we consider five sub-classes of stable distributions:
(i) Continuous stable distributions are parameterized through four parameters: stability parameter (0, 2). skewness parameter 3 E 1-,,1], scale parameter 7 (0, oo), and location parameter 4 E I. We denote its p.d.f. as Pcss(a, B,7, p)) Note that all continuous stable distribution (except Gaussian) are fat-tailed.
(ii) Independent component multivariate stable distributions describe the distribution of X consisting of d independent random variables (X,} such that each X3 ~ (as B,, Tjs H2). We denote its
p.d.f. as pcs-ic(aa 3, (i).
(iii) Elliptically-contooie random vectors whose p.d.f. has elliptical contours, e.g., the multivariate Gaussian distribution. We denote its p.d.f. as PCS- f.). Here, Σ is positive definite matrix, fd E Rd, and a E (0,
multivariate stable distributions are the distributions of continuous stable
(iv) Discrete stable distributions are the discrete analogues of the continuous stable distributions. The p.m.f. of discrete stable distributions, denoted as (v, r), are parameterized through two parameters: rate parameter > 0 and exponent 0 < < 1. Discrete stable distributions do not have well-known multivariate generalizations except the Poisson distribution.
(v) Multivariate Poisson distribution: We use the multivariate Poisson distribution proposed in [4]]
42 4 44,, denoted as Poisson(d, d', A),, which represents each random variable in the random vector Ñ sum of independent Poisson random variables. Formally, N Poisson d', Á) if Ñ AÑo, where A = [A1... ....... A Here, A, denotes a d x (9) submatrix having no duplicate columns, where each of its columns contain exactly ones and (d i) zeros 44]. The vector
N [NP .......... No Ni
Ng N
with mutually independent No N ~ Poisson(A,,, (A....))) V (d.....,1,) A4.1 E [d"], where AC ..ib....,JJ), < j2 < .... < fi, and J.....;J E [d], e.g. 43 = ((1, 2), (1,3), (2,3)}. Note that d' < d. and A = [Λ1 da (d' 1)..d] -)..)
3

---

We refer the reader to 23
45 46 47 for more details on stable distributions.
Convolution-closed distributions: Convolution-closet distributions form large class of distribu- tions that are closed under convolution in some parameter 8. Formally, we define the convolution- closed distribution as follows: Let FD denote a family of distributions, where each member dis- tribution f()) E FD is indexed by a parameter E D. Consider X1 ~ f()1), X2 ~ f(62), and X1 IL X2 for some ò1, 82 E D. Then, FD is convolution-closed in the parameter 6 if
X1 X2 ~ f())) * f()2) = f(61 + 62) V 81, 62 E DD such that 61 + 02 E D,
(3)
where denotes the convolution operator. Many well-known distributions can be considered convolution-close in some parameter [25], such as the Poisson distribution, Gaussian distribution,
gamma distribution, etc. Table in [25] lists various examples of convolution-closed distributions.
3 A sufficient condition for computing PID terms analytically
The main focus of this work is to analyze the cases where 2 is analytically solvable. A sufficient condition for solving 2 is to show that Ap contains distribution QMc(M, Y) with the Marko- vian structure M + + Y or M + + X. We briefly discuss the argument justifying this sufficient condition. Consider the case where Qac
Y) E Ap and has the Markovian structure
M + X + Y. First, we note that (M; (X) = minges Ig (M; X| 2 0 due to the non- negativity of conditional mutual information 28 Second, Y IL M|X due to the Markovian structure of QC
(Mr(Ma M, X, Y). which implies that Tomo X) = 0 (conditionally independent random variables have zero conditional mutual information 28 Hence, QMc(M, X, achieves the lower bound of zero for the minimization problem minge A To (M; X), showing that QM (M, X, Y) indeed minimizes (). A similar argument can be made for the case when Q @MccMM X, Y) has the Markovian structure M + Y + X. Proposition I in Appx. A formalizes the above argument.
The above sufficient condition provides an easy way to analytically calculate the PID terms, as it ensures one of the unique information terms is always zero. Consequently, the remaining PID terms can be calculated by substituting zero for the appropriate unique information term in (I) and solving the resultant linear system. Hence, if applicable, the above sufficient condition considerably simplifies the calculation of PID terms by circumventing the need for optimizing over a of distributions. Surprisingly, many well-known distribution families allow intuitive constructions of P(M, X, Y), for which the above sufficient condition is applicable. In the following sections, we provide theorems that specify sufficient conditions under which the existence of these Markov chains in Ap p can be guaranteed for these P(M,X, XY
4 Computing PID for stable distributions
In this section, we extend existing results for analytical PID computation of jointly Gaussian M, X, and Y (2) to the much larger class of stable distributions. Our results utilize two key observations to provide the generalization of the Gaussian results: (i) We first identify that the analytical computa- tion of PID for jointly Gaussian systems is due to their particular "affine dependence" structure; (ii) We show that these particular affine dependence structures are not unique Gaussian systems, but rather extend to many members of the stable distribution family.
For jointly Gaussian M, X, and Y, the conditional distributions P(X M) + xxx)) and P(Y M)=N(cM + ) are also Gaussian distributions, where their means are an affine function of M and their variances are fixed with respect to M. This particular affine dependence of X and Y on M is the key to the analytical calculation of their PID, as it guarantees existence of a Markov chain QMc X, Y)= E Ap. Thus, we can apply the sufficient condition described in Sec. to compute the PID terms. We illustrate through an example: consider P(X xx) and P(YM) M)-NMM o where of < of. and M P(M) for some appropriate P(M). Then, we can explicitly construct (M, X. Y) E Ap with the Markovian structure M + X + Y as follows: Choose Qarc M, E.Y)=P(M)PP
where QMc ) is specified through the addition of independent Gaussian noise, i.e., Y =x X + Here, ~ ~N oz - and 11 (M, X). easy to verify that (Mc (M ) ), which implies that QMc (M) construction, QMC PPM) and hence, (M E Ap.
M)
By

---

The above example is special case of well-known result where, for scalar M, one can always construct a lower signal-to to-nose ratio (SNR) "Gaussian channel". i.e. P(Y M our example, by
adding independent Gaussian noise to higher SNR "Gaussian channel" 22 48 i.e. P(XJM) in
our example, Surprisingly, the technique of adding independent noise to construct Markov chains contained in Ap p can be extended to P(X M) and P(Y|M) as members of stable distributions. Theorems and generalize the above construction for P(X [M) and P( as members of univariate continuous and univariate discrete stable distributions, respectively. Theorems 2 and S consider the case multivariate stable distributions (see Sec. 2 The key technique for proving these theorems is that Markov chain QMc X, Y) Ap can always be constructed by adding appropriate independent noise to a higher SNR P(X MM) to obtain lower SNR P(Y|M) similar to our above example.
4.1 PID univariate affine continuous stable system
Theorem I can be viewed direct generalization of Barrett's Gaussian PID result [21] to stable distributions, showing one of the UI terms is always zero. We begin by formally describing the
univariate affine continuous stable system that generalizes the Gaussian system of Barrett's [2]]. Let the joint distribution of M, X, and Y, denoted as P(M, X, Y), satisfy the following properties: (i) M P(M) with support set M C R. (ii) P(XIM) and P(Y are univariate continuous stable distributions with an affine dependence on M, i.e. P(X[M)= pcs(a, 9x, 1x, aM + b) and P(Y M) -pss (a, By Bysgn(ac), ,y, + d), where a, b, E R.
Theorem Let M, X, and Y be random variables whose joint distribution P,M, Y) described by a univariate affine continuous stable system. Without the loss of generality, assume lalf1x 2 lcl/vr 3x) and 1 + By (2xlel/ lala))
(1 + 3x), then Ap contains Markov chain of the the form M + X -Y and UI( (X) =0.
If - By (7xlel//ylal))
Proof. See Appx. F for the proof. Here, lol/2x and lel/Ty are the SNR analogues.
4.2 PID of multivariate affine continuous stable systems
We analyze two multivariate generalizations of Theorem D in Theorems 2 and 3 Namely, we consider independent component multivariate stable distribution (where all the components of the random vector are independent) and ellpptcclll-contouree elliptically contoured, similar to multivariate Gaussian distributions). We construct two systems employing these two sub-classes of multivariate continuous stable distributions and show that one of the UI terms is always zero for these systems. For both cases, denote the joint distribution of M, X, and Y as P(M, X, Y), where the support set of M is M R. The dimensions of M, X. and Y are 1, dx, and dy, respectively.
multivariate stable distribution (where the p.d.f.
System 1: The random vectors X and Y satisfy the following equations:
x Hx + Axīx + bx and Ý = Hy + Ay Zy + by.
(4)
where Σx ~ Pas pcla δax Ixx, σdx). žy Pcs-ic(a, Oe,, Iev Oay ). Ax and Ay are invertible matrices, Hx, bx E Rdx, and Hy, by E Rdy
Theorem 2. Let System I describe the joint distribution P(M, X. Y) of M, X, and Y. Without
the loss of generality, assume |A llx S "|Ax'Hxlle where k a/a-1 E (1, 2] and
00 V a E (0, 1]. Then, Ap contains Markov chain of the form M + X - Y and
O.
Proof. See Appx.
'H G for the proof. Here, ||A Hylle and ||Ax'' "xlll llx the SNR analogues.
System 2: The conditional distribution of X and Ý conditioned on M are as follows: P(XIM)
PCs- PCs-Eo(a, Σx, Hy M + bx) and P( M) pes- mes-Ec(a, Σy, Hy M + by:,, where Ex and Ey are positive definite matrices, Hx, bx e _x and Hy, by Br
5

---

Theorem 3. Let System 2 describe the joint distribution P(M, X, Y) of X, and Y. Define D 1/2 Vx and by 1/2 as the respective inverses of the matrices Σ and Σ which satisfy: (Σ T = Exx
and (E V " Σy. Without the loss of generality, assume |
HHy 112 2
x_ Hx x xx122 Then,
Ap contains a Markov chain of the form M- -> X Y and UT(M; X =
Proof. See Appx. H for the proof. Here, | 1/2 Hy and || 1/3 Hxx ll2 are the SNR analogues.
4.3 PID of univariate afline discrete stable system
The univariate affine discrete stable system is the discrete counterpart of the univariate affine con- tinuous stable system described in Sec. [4.]] The formal description of the univariate affine discrete stable system is as follows. Let the joint distribution of X, and Y. denoted as P(M. X, Y), satisfy the following properties:
00); (ii) P(X(M) and P(Y (.) are univariate discrete stable distributions with an affine dependence on M, i.e., (
M P(M) with support set M
am + b) and = em + d), where a, b, c, d E (0, 00). Theorem 4. Let M, and Y be random variables whose joint distribution P(M, X, Y) is described by a univariate affine discrete stable system. Without the loss of generality, assume a 2 c. If 2 %dd then Ap contains Markov chain of the form M + X - Y and (I 0.
Proof. See Appx. foor the proof. Here, a and e are the SNR analogues.
4.4 PID of multivariate linear Poisson system
The Poisson distribution is the only the discrete stable distribution with well-known multivariate extension. Hence, we analyze gector-generalizatiing of Theorem
4 only for the Poisson distribution.
We now describe the multivariate linear Poisson system: Let the joint distribution P(M, N,, Y) of
M, X, and Y satisfy the following properties: (i) M P(M) with support set M C (0, o)); (ii)
P(X) M) Poisson(dx, d'x, A xx) and M
d'y, Ay), with:
Ax = Axx dx -
-d) dx
1 Ainnt = x Mi..., M2 j E [dx] and (i1.....2,) 11) dx
Ky DAY N - 11)..dy
Ae.ndy = =my T
M V E [dy] and (d1....,,4y) E A. (5)
Theorem 5. Let the joint distribution PMM,X, Y) of M, X, and Y be described by the multi- variate linear Poisson system defined above. Without the loss of generality, assume d'x 2 dy- If
E(pr )EE 4x Vig....." 2 EGi Wi..., Vj E [dyy], then Ap contains a Markov chain of
the form M , +x + & and UT(M; Y X
.... Proof. See Appx. for the proof. Here, 71 ..... xxx -dx- -1).d-_...xxx"' "..... ... Tav (od--(dd---) dy are the SNR analogues.
Computing PID using data thinning and data fission strategies
Data thinning and data fission are emerging fields in machine learning and statistics dedicated to studying the procedures of splitting a random variable X into N different component random variables. These splitting procedures provide an attractive alternative to the standard splitting of datasets into training, validation, and test splits for performing crossvaaidatiin to select statistical model parameters, as they enable users to perform gross-validation even for the extreme case of dataset containing a single datapoint. We refer the reader to the recent works Neufeld et al. (25 and Leiner et al. [26] for more comprehensive discussion on data thinning and fission, respectively.
The fields of data thinning and data fission share an important theoretical link with analytically calculating PID terms. Specifically, the tools developed for splitting random variables for data thinning and data fission can be readily used to calculate PID terms analytically. To give an intuition, suppose X contains more information about M than Y. Then, we can employ data fission and thinning strategies to decompose into two components, fi [i(X) (X) and f2(X), such that f2(X) follows the same distribution as Y. Then, fx(X) and Y convey the same information about M as they

---

are identically distributed and represent the redundant component of the PID termss Similarly, (X)
contains the information uniquely contained X about M. Thus, (X) represents the unique information term. Theorems 6 and 7 utilize the data thinning and data fission proposed in 25 26
to construct a Markov chain Qa M,X,Y Y) E Ar for several systems of random variables M, X,
and Y. This allows us to use the sufficient condition discussed in Sec. Bo compute their PID.
5.1 PID for convolution-close distribution based on data thinning strategies of Neufeld et al.
Neufeld et al. 25 introduces data thinning for large family of distributions known convolution-
closed distributions (see Sec. 2 An attractive property of convolution-closed distributions is that they provide natural way define dilation/thinning operation. Formally, let X f(6), then we define X. as the e-dilated version of X if X. ~ f(eō) for some E E (0, 1) such that ed D. Furthermore, if we assume (1 e) E D. then -
+ Xi=e), where X1- f((1 - e)6) and X, TT X1-e (see lemma 19 for formal justification). We denote X1
P 2) XX Xi-e = x) as G(cd, (1 -C6,, z). This dilation operation forms the basis of data thinning, as it enables X to be split into its dilated components X, and X1-c, such that X Xx + +Xiee. where X, IL Xi-e. We utilize this dilation operation for analytically calculating
the PID terms of the following linear convolution-closee system:
Let FD be a convolution-close distribution family as described in Sec. | The joint distribution P(M, X, Y) of the random variables M, X, and Y describes a linear convolution-closed system if the distributions P(X MM) and P(Y|M) are defined as follows:
Linear convolution-close system:
P(X|M=m) = f(6)) and P(Y]M=m) f f( such that 5 5% E DDm EMM
(6)
where M is the support of M. Furthermore, we assume 8x = yo% V m E M for some y E RR..
Theorem 6. Let the joint density P(M, X, Y) of random variables M, X, and Y be described by linear convolution-clos
system. Without the loss of generality, assume S 1. If
(a) (1 ()) E DDV m M,
(b) PXX,|X, + X1-,=I, M=m) = G(y (1 - 7)8mm
()0mm) 2) does not depend on m, where PM = f(160)) P(X1-1IM) f((1 7)) and X,, 11 X1-,IM,
then Ap contains a Markov chain of the form M X Y and U U =
The proof of Theorem 6 is provided in Appx. K Theorem 6 enables PID calculation for several
well-known distributions such as gamma, Poisson, beta etc. (sce Appx. for (non-exhaustive) list).
5.2 PID for distributions based on data fission strategies of Leiner et al.
Leiner et al. [26] propose a "conjugate- prior reversal" strategy for splitting a random variable X into two components, f(X) and g(X). for certain exponential family distributions. The distributions pro- posed for performing conjugate- mnnjuatt-prrior reversal provide natural descriptions of P(M, X, Y) (specified in theorem 7 for which PID can be calculated analytically. We briefly describe the distributions used in the conjugate-priol reversal strategy. Let X ~ Perp1 (X), where
Peapl (X x; 02) (0),, 02) exp(0) 0 A(x)),
(7)
for some appropriately defined ((.).)),(-)... 01 and 8g. Furthermore, define a random variable Y through its conditional density p = z)):
p(Y = y/X = r; 0z) = hyy) exp (x (xTT()) - 07 A(x))
(8)
for some T()). and 0z. such that p(Y y| = 01, 02) is well-defined distribution. Then, the decomposition terms f() and g(X) ) are and X, respectively, in the conjugate- reversal strategy. Furthermore, the marginal distribution of Y is expressed as:
Peap2( (Y- 01, O2, 02) = (O,, 02) H (Tu)) (2 0z) (see proof of Theorem I in 26
Theorem Let M, and Y be random variables having the joint distribution P(M, X, X.Y). Furthermore, the conditional distribution of X and Y conditioned on M are as follows: =m)= (X: (x; 0g (m), 02(m)) and P(Y|M= = (Y; 0, (m), (m),02(m), 0z). Then, AP P contains Markov chain of the form M + X Y and UI(M; Y X) = =0.
7

---

The proof of Theorem 7 can be found in Appx. L The proof essentially stems from observing that the joint distribution Q wv (M. X, P X M) Mc (Y lies in Ap. where (Y h(y) exp (x') 0g A(x)) A (non-exhaustive) list of well-known distributions for which the- orem is applicable is provided in Appx. D Leiner et al. [26] also discuss some more strategies for performing data fission that do not follow the conjugate-prior reversal strategy. We provide the corresponding results for computing PID for these remaining data fission strategies and for additional
miscellaneous distributions in Appx.
Upper bound for convolution-closed distributions
b
Analytical MI-MMNNN Nu-uericaall Mi
0.07
0.06
Numerical MI (in bits)
2.0
%6 Difigrence Anolytical
wrr.. Numerical Mi
4%
0.05
1.5
bits)
0.04
3
0.03
1.0
2
0.02
0.5
1%
0.01
0.00
0.0
1
10 15
20
1
10 15
20
10 15 Functions Pair Index
20
Function Pair Index
Function Index
Figure 1: and b, respectively, show the box plot of the difference IQu (M, Lex (M, Y) and the corresponding values of Iqx X,, for the 20 different function pairs across the 75 different P(M) distributions. The light-blue dots show the corresponding data points used for making the box plots. e shows the ratio of the median difference Iq, (M, X, Y) - IQN XM) and the median value of I XM, XY) in percentage, for each function pair.
Several numerical methods, such as 3 20 approximately solve 2 by considering a smaller constraint set minimizing appropriate upper bound for calculating the PID terms. These numerical methods employ general approximations that rely on weak assumptions on the underlying distributions for solving (2). as their goal is estimate PID for large class of distributions. However, for specific applications, it is possible to make stronger assumptions on the underlying distributions (e.g., assuming the Poisson distribution for modeling neural spikes). This section illustrates how our theoretical analysis can benefit these numerical algorithms by providing more refined approximations for solving (2 that harness these stronger assumptions. Specifically, we construct an upper bound for distributions and show that, under certain assumptions, the the objective of (2) for convolution-clos
upper bound can be analytically minimized over Ar in Sec. 6..1 Note that these upper bounds are applicable for more general cases than our theoretical results, as they do not require the sources X and Y to have an affine dependence on M. Consequently, our upper bound is also applicable in cases where both of the UI terms in the PID are non-zero, unlike our results in Sec. 4 and 5 We demonstrate the tightness of our upper bound through a simulation study Sec. 6.2
Notations and assumptions: We consider the minimization problem mingeAp (M; [M; [X, Y)) for deriving the upper bound, as it was shown to be equivalent to (2) in [16). The distribution P(M, X, Y) for which we will construct our upper bound is specified as follows. The random variable M has support over M, the conditional distributions P(X [M) and P are members of some convolution-close distribution family Fo, and there exists some bias: o E D such that
8X
P() and (TY]M=m)=//m (6% 6% 15th (xxa)) (6x
where 6x OY E m E EM, with 5Y. -Ai o D and 5x 8bias =((==(()) ( %
6Y = -1 (6) s%a.)) and 8x m = =m
(3) (6 a.x for em (2)
e E [0, m E M.
The above assumptions ensure that X and Y can always be decomposed into new random variables X', Y" nx, Y', and ny (see (T0). We utilize these decomposed random variables to construct our upper bound. Hence, our upper bound is only applicable for systems satisfying (9. The above assumptions describe a large class of systems. For example, in the Poisson case, our upper bound is applicable for any P(M, X, Y) having P(X|M) = Poissonífi (M)) and P(Y (oisson([2(MM) as long as fi (M) 2 f2(M) over M. Appx. of systems for which (9) holds to illustrate the assumptions of
we provide numerous examples
8

---

6.1
Upper bound construction
First, we consider an arbitrary distribution Q(M,X,Y) E Ap. Therefore, we know and
We use the di- lation properties of convolution-closed distributions (Sec. [5.1 to decompose and Y into their respective dilated versions: (X', "Y", nxx and (Y', ,Y From the results of Appx. 1. we know are mutually conditionally independent given M and y II. [y Hence, we can construct the following Markov chain for any arbitrary Q(M, X, Y) E Ap:
that X = + +nx and Y Y" ny. Furthermore, (X',
M - X Y" Y nx NY 1' (X' +-" +'+ny] = [X rJ".
(10)
We denote the joint distribution of (M, X Y", Y",Y",,, ,Y',"xx," ny) as Q(M, "Y,,X"Y" Y", n,x, ny). We ap- propriately choose the dilation amounts for X and Y such that the respective conditional distributions of (X', Y" Y', nxx nY are as follows:
M-))
8%h - M)as),, Q(Y"|M-m) = f(6m orias);
M - k M)) f( and ((ny|M)) f f(
(11)
The distributions in 11 well-defined due (9. Appx. 1. formally shows that Q, defined as above, exists for each Q E Ap. We use data proc-psocssiing inequality and (1O) to conclude:
(2(M: [X', Y", nx, Y', ny)) 2 (M; [X, YJ) QQ(M,, X,Y) E Ap.
(12)
Hence, 12 provides the desired upper bound for objective (M; [X, Furthermore, the minimization problem minqEa, (M; Y", nx, Y", my]]) is analytically solvable, and the minimizing distribution Q' has the structure:
,",",,,,,,,,"
Y' = Y" and (nx. ny) TL (M, Y').
The corresponding distribution of (M, X), Y), denoted as QA (M, Y), can be found by appropri-
ately manipulating Q", X X' + Y" nx and Y Y + ny. The distribution QA serves as an
approximate solution for the problem minqEA, (M; [X, Yl) and, consequently, 2 Note that if
multiple 6/x and 6)x exist satisfying (.) we optimize over the pairs (d 6% to further refine our approximate solution of (2).
6.2 Simulation study for numerically validating the upper bound
We illustrate the tightness of our upper bound through a simulation study on the Poisson distribution convolution-closed distribution). The reason for choosing the Poisson distribution is two-fold: (i) Poisson distribution is easily approximated as a discrete distribution over finite support, enabling calculation of ground-truth PID terms through numerical solvers such as [10 I7 18). which are not readily available for continuous distributions; (ii) Many practical applications of PID have been in neuroscience, and the Poisson distribution frequently used for modeling neural spikes in neuroscientific studies 49).
We compare the performance of analytical estimate Q (M, X,Y) with the numerical ground- truth estimate QN (M, X) for the Poisson distribution. The simulation setup is as follows: We choose P(M. X,Y) such that P(XJM) Poisson(f1 (M)) and P(Y MM) Poisson fa (M)). We chose 20 different pairs of non-linear (-) and f2(-) (enumerated in Table in Appx. M). such that the assumptions in 9 satisfied. The distribution P(M) is chosen to be discrete distribution. For each function pair, we compare Q X X, Y) and Q~ X, Y) over 75 different distributions of P(M) as shown in Fig. D.. The 75 distributions are further sub-divided into three groups of 25 based on the number of the outcomes of M, which are 2, 4, and 8. For each of the 75 distributions, the values of M are randomly sampled from [1, 4] and the values of (MM) are randomly sampled from simplex of appropriate dimensions. We compare the mutual information IQ (M; [X, Y)) calculated using QA with the numerical ground-truth IQ (M; [X, Y)) calculated using numerical solvers (Fig. [p). For calculating Iq (M; [X, Yl), we approximate the Poisson distribution as finite discrete distribution by terminating its support at the smallest integer where its cumulative distribution is greater than 0.99. We use the code provided in [50] (under Apache 2.0-license) to numerically demonstrates that the analytical upper bound provided by QA A is very tight for the tested function pairs (within < 1% of
solve (2) using this discrete approximation of the Poisson distribution. Fig. [

---

the numerical ground-truth for 16 function pairs) and serves as good approximation for solving ( Furthermore, the tightness of our analytical upper bound suggests that Qn might be an analytical solution of (2) for larger class of systems of M, X, and Y having non-affine dependence on M. Similar results demonstrating the tightness of upper-bound for engative-binomin and binomial distributions are presented in Appx. M.4
7 Discussion and Limitations
In this work, we analytically compute PID for large classes of distributions, greatly expanding upon the analytical result for the Gaussian system. We provide the first known analytical PID for systems employing Poisson, gamma, exponential, Cauchy, beta, Dirichlet, Lévy-stable, binomial, multinomial, negative binomial, and uniform distributions. Furthermore, we generalize the previous Gaussian PID result (21 in an additional way by showing the target M need not be Gaussian. Our stable distribution results provide the first known analytical computation of PID for fat-tailed distributions (all continuous stable distributions have infinite variance except the Gaussian distribution). A practical utility of our analytical results is that they provide a large test bed which the performance of numerical PID estimators can be compared and evaluated. This test bed may benefit future works on numerical PID estimation by enabling more comprehensive testing of PID estimators.
Our results on the Poisson, Cauchy, and binomial are of particular relevance in the neuroscientific context. Poisson and Cauchy distributions are widely used to model neural spikes 4g] and network dynamics in the brain [5I]. Binomial thinning is frequently-used operator in neuroscience [49]. Our generalization of the Gaussian result could be helpful in refining approximations already used in computing PID for neural data [3) e.g., by relaxing the assumption of joint Gaussianity. As continuous stable distributions have been shown to better model Magnetic Resonance Imaging our results may also be helpful in computing PID in this application.
data 52 53
We also connect the fields of data thinning and data fission with PID by using their decomposition strategres for analytically computing the PID of systems based on convolution-closed distributions (see Sec. 5). Conversely, our PID results also suggest decomposition strategies for data thinning/fission purposes, e.g., our stable distribution results suggest that stable random variables can be decomposed by adding independent noise (similar to the Gaussian case discussed in (26).. Convolution-closed distributions are particularly promising for studying PID as they allow intuitive construction of upper bounds that can be analytically minimized (see Sec. (6) These upper bounds can complement the work on numerical estimation of PID by providing more refined approximations. Another promising avenue is to combine our upper bounds with lower bounds (e.g., from (54]) to create branch and bound algorithms (55) for solving ( Overall, our analytical results greatly facilitate the computation of PID, either by directly using the analytical expressions or by providing refined approximations for numerical methods.
Limitations and Future Work: We study PID for univariate M, X, and Y and provide some vector extensions. More vector extensions of our results are a promising direction for subsequent works. Most of our analytical results require P(XJM) and M(Y to depend on some affine functions of M. The existence of analytical solutions for the cases where P(X[M) and P(Y depend on non-affine functions of M remains an open question. The upper bound discussed in Sec. can be further refined with more careful analysis. and more rigorous testing of these upper bounds is required to understand which cases the upper bound is a good approximation for solving 2 We defer the testing and refinement of these upper bounds for subsequent works, our primary goal in this work is to study analytical solutions of PID. Niu & Quinn [56) also propose a duality result between the synergistic and redundant components in the Gaussian broadcast and multiple-acces: channels utilizing the analytical PID expressions of the Gaussian system. It may be possible to derive similar duality results for appropriately defined broadcast and multiple-access channels employing other distributions (e.g., Poisson or Cauchy) with our theoretical results.
Due to their theoretical nature, our results' negative impact primarily depends on how they are used and interpreted. Our theoretical results are applicable only under the specific assumptions outlined in this work, and using these results without ensuring that the theorem's 's assumptions are satisfied can lead to incorrect scientific conclusions. We also caution reader against naively using PID to draw causal inferences, as PID is inherently correlational quantity.
Broader Impact:
10

---

Acknowledgments and Disclosure of Funding
Authors would like to thank Prof. Pulkit Grover, Dr. Praveen Venkatesh, Prof. Alireza Chamanzar, and Neil Ashim Mehta for helpful discussions. This work was supported by National Science Foundation (NSF). Authors declare no competing interests.
References
[1] Praveen Venkatesh and Pulkit Grover. Understanding encoding and redundancy in grid cells using partial information decomposition. Comput. Syst. Courosci.(Coyyree 2020.
[2] Ariel K Feldman, Praveen Venkatesh, Douglas J Weber, and Pulkit Grover. cntommtiion-thoreetie tools to understand distributed source coding in neuroscience. IEEE Journal on Selected Areas in Information Theory, 2024.
[3] Praveen Venkatesh, Corbett Bennett, Sam Gale, Tamina Ramirez, Greggory Heller, Severine Durand, Shawn Olsen, and Stefan Mihalas. Gaussian partial information decomposition: Bias correction and application to nigh-dimensional data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 74602-74635. Curran Associates, Inc., 2023.
[4] Nigel Colenbier, Frederik Van de Steen, Lucina Q Uddin, Russell A Poldrack, Vince D Calhoun, and Daniele Marinazzo. Disambiguating the role blood flow and global signal with partial information decomposition. Neurolmage, 213:1 16699, 2020.
[5] Tjeerd Boonstra, Luca Faes, Jennifer N Kerkman, and Daniele Marinazzo. Information decomposition of multichannel EMG map functional interactions in the distributed motor system. Neurolmage, 202: 16093, 2019.
[6] Jana Krohova, Luca Faes, Barbora Czippelova, Zuzana Turianikova, Nikoleta Mazgutova, Riccardo Pernice, Alessandro Busacca, Daniele Marinazzo, Sebastiano Stramaglia, and Michal Javorka. Multiscale information decomposition dissects control mechanisms of heart rate variability at rest and during physiological stress. Entropy, 21(5):526, 2019.
[7] Giuseppe Pica, Eugenio Piasini, Houman Safaai, Caroline Runyan, Christopher Harvey, Mathew Diamond, Christoph Kayser, Tommaso Fellin, and Stefano Panzeri. Quantifying how much sensory information in a neural code is relevant for behavior. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[8] Praveen Venkatesh, Sanghamitra Dutta, and Pulkit Grover. Information flow in computational systems. IEFF Transactions on Information Theory, 6609):5456-541, 2020. 2020.
[9] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nic, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Russ Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling multimodal interactions: An information decomposition framework. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 27351-27393. Curran Associates, Inc., 2023.
[10] Paul Pu Liang, Yun Cheng, Ruslan Salakhutdinov, and Louis-Philipee Morency. Multimodal fusion interactions: A study of human and automatic quantification. In Proceedings of the 25th International Conference on Multimodal Interaction, pages 425-435. 2023.
[11] Sanghamitra Dutta, Praveen Venkatesh, Piotr Mardziel, Anupam Datta, and Pulkit Grover. An enformatioon-heoretiie quantification of discrimination with exempt features. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3825-3833, 2020.
[12] Sanghamitra Dutta and Faisal Hamman. A review of partial information decomposition in algorithmic fairness and explainability, Entropy, 25(5):795, 2023.

---

[13] Faisal Hamman and Sanghamitra Dutta. Demystifying local & global fairesss trade-off's in federated learning using partial information decomposition. The Twelfth International Conference Learning Representations, 2024.
[14] Faisal Hamman and Sanghamitra Dutta. A unified view of group faimess tradeoffs using partial information decomposition. 2024 IEEE International Symposium on Information Theory (ISIT), pages 214-219, 2024.
[15] Tomas Scagliarini, Luca Faes, Daniele Marinazzo, Sebastiano Stramaglia, and Rosario N Mantegna. Synergistic information transfer in the global system of financial markets. Entropy, 2009);1000, 2020.
[16] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 16(4):2161-2183, 2014.
[17] Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. Bivariate partial information decompo- sition: The optimization perspective. Entropy, 19(10)5530, 2017.
[18] Pradeep Kr Banerjee, Johannes Rauh, and Guido Montúfar. Computing the unique information. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 141-145. IEEE, 2018.
[19] Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. BROIA-2PID: A robust estimator for bivariate partial information decomposition. Entropy, 20(4):271, 2018.
[20] Ari Pakman, Amin Nejatbakhsh, Dar Gilboa, Abdullah Makkeh, Luca Mazzucato, Michael Wibral, and Elad Schneidman. Estimating the unique information of continuous variables. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 20295-20307. Curran Associates, Inc., 2021.
[21] Adam Barrett. Exploration of synergistic and redundant information sharing static and dynamical Gaussian systems. Physical Review E, 91(5):052802, 2015.
[22] Praveen Venkatesh and Gabriel Schamberg. Partial information decomposition via deficiency for multivariate Gaussians. In 2022 IEEE International Symposiun on Information Theory (ISIT), pages
IEEE, 2022.
[23] John P Nolan. Univariate stable distributions. Springer Series in Operations Research and Financial Engineering, 10:978-3, 2020.
[24] Fred W Steutel and Klaas van Harn. Discrete analogues of self-decomposabliity and stability. The Annals of Probability, pages 893-899, 1979.
[25] Anna Neufeld, Ameer Dharamshi, Lucy L. Gao, and Daniela Witten. Data thinning for convolution-close distributions. Journal of Machine Learning Research, 15(57):1-35, 2024.
[26] James Leiner, Boyan Duan, Larry Wasserman, and Aaditya Ramdas. Data fission: Splitting a single data point. Journal of the American Statistical Association, 000):1-12, 2023.
[27] Paul I. Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint AXiv:1004.2515 2010.
[28] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). -ileee-nterscieece, USA, 2006.
[29] Malte Harder, Christoph Salge, and Daniel Polani. Bivariate measure of redundant information. Physical Review E, (11)0113 2013.
[30] Pradeep Kr Banerjee, Eckehard Olbrich, Jürgen Jost, and Johannes Rauh. Unique informations and deficiencies. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 32-38. IEEE, 2018
[31] Jim W Kay and Robin AA Ince. Exact partial information decompositions for Gaussian systems based on dependency constraints. Entropy, 20(4):240, 2018.
12

---

[32] Conor Finn and Joseph T Lizier. Pointwise partial information decomposition using the specificity and ambiguity lattices. Entropy, 20(4):297, 2018.
[33] Xueyan Niu and Christopher J Quinn. A measure of synergy, redundancy, and unique infor- mation using information geometry. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 3127-3133 IEEE, 2019.
[34] Fernando E Rosas, Pedro AM Mediano, Borzoo Rassouli, and Adam B Barrett. An operational information decomposition via synergistic disclosure. Journal of Physics A: Mathematical and Theoretical, 53(48):485001, 2020.
[35] Keerthana Gurushankar, Praveen Venkatesh, and Pulkit Grover. Extracting unique information through Markov relations. In 2022 58th Anmual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1-6. IEEE, 2022.
[36] David Blackwell. Equivalent comparisons of experiments. The Annals of Mathemattiaal Statistics, pages 265-2722 272, 1953.
[37] Praveen Venkatesh, Keerthana Gurushankar, and Gabriel Schamberg. Capturing and interpreting unique information. In 2023 IEEE International Symposium on Information Theory (ISIT), pages 2631-2636 IEEE, 2023.
[38] Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. Guided self- organization: inception, pages 159-190, 2014.
[39] Joseph T Lizier, Nils Bertschinger. Jürgen Jost, and Michael Wibral. Information decomposition of target effects from multi-source interactions: Perspectives on previous, current and future work. Entropy, 20(4):307, 2018
[40] John P Nolan. Stable distributions. 2012.
[41] Jeremie Fish, Jie Sun, and Erik Bollt. Interaction networks from discrete event data by Poisson multivariate mutual information estimation and information flow with applications from gene expression data. Applied Network Science, 7(1):1-22, 2022.
[42] AS Krishnamoorthy. Multivariate binomial and Poisson distributions. Sankhyā: The Indian Journal of Statistics, pages 117-124, 1951.
[43] DM Mahamunulu. A note on regression in the multivariate Poisson distribution. Journal of the American Staristical Association, 62(317):251-255 1967.
[44] Dimitris Karlis and Loukia Meligkotsidou. Multivariate Poisson regression with covariance structure. Statistics and Computing, 15(4):255-20) 2005.
[45] Murad S Taqqu. Stable non-Gaussian random processes: stochastic models with infinite variance. Chapman & Hall, 1994.
[46] S James Press. Multivariate stable distributions. Journal of Multivariate Analysis, 2(4):444 462, 1972.
[47] Norman Lloyd Johnson, Samuel Kotz, and Narayanaswamy Balakrishnan. Discrete multivariate distributions, volume 165. Wiley New York, 1997.
[48] Xiaohu Shang and H Vincent Poor. Noisy-interference sum-rate capacity for vector Gaussian interference channels. IEEE Transactions on Information Theory, 59(1):132-153, 2012.
[49] Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems. MIT press, 2005.
[50] Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. Broja-2pid: robust estimator for bertschinger al.'s bivariate partial information decomposition. Entropy, 20(4):271, 2018.
[51] Viktoras Pyragas and Kestutis Pyragas. Mean-field equations for neural populations with G-Gaussian heterogeneities. Physical Review E, 104(4):044402, 2022.
13

---

[52] Diego Salas-Gonzan González, JM Górriz, Javier Ramírez, Matthias Schloegl. Elmar Wolfgang Lang, and Andrés Ortiz. Parameterization of the distribution of white and grey matter in MRI using the o-stable distribution. Computers in biology and medicine, 53(5):559-55 2013.
[53] Diego Salas-Gonzales Juan M Górriz, Javier Ramírez, and Elmar W Lang. Why using the alpha- stable distribution in Neurolmage? In 2014 International Conference on Signal Processing and Multimedia Applications (SIGMAP), pages 297-301. IEEE, 2014.
[54] Seiya Tokui and Issei Sato. Disentanglement analysis with partial information decomposition. arXiv preprini arXiv: 2tt::2/08.1a0 13733, 2021.
[55] Narendra and Fukunaga. A branch and bound algorithm for feature subset selection. IEEE Transactions on Computers, 100(9):917-922, 1977.
[56) Xueyan Niu and Christopher Quinn. Synergy and redundancy duality between Gaussian multiple access and broadcast channels. In 2020 International Symposium on Information Theory and Applications (ISITA), pages 6-10. IEEE, 2020.
[57] Robert Ash and Catherine A Doléans- Dadee. Probability and measure theory. Academic press, 2000.
[58] Gilbert Strang.
Introduction linear algebra. SIAM, 2022.
[59] Alan V Oppenheim, Alan S Willsky, Syed Hamid Nawab, and Jian-Jiun Ding. Signals and systems, volume 2. Prentice hall Upper Saddle River, NJ, 1997.
[60] Jagadesh Chandramohan and Lung-Kun Kuang Liang. Bernoulli, multinomial and Markov chain thinning of some point processes and some results about the superposition of dependent renewal processes. Journal of Applied Probability, 22(4):828-88) 835, 1985.
[61] Sergio Verdú. Poisson communication theory. International Technion Communication Day in Honor of Israel Bar-David, 66, 1999.
[62] Christian Walck et al. Hand-book on statistical distributions for experimentalists. Stockholms universitet, 1996.
[63] Kalimuthu Krishnamoorthy. Handbook of statistical distributions with applications. Chapman and Hall/CRC, 2006.
[64] Lenka Slámová and Lev B Klebanov. Generalized definitions of discrete stability. arXiv preprint a0Xiv 025888 2015.
14

---

A Formal proof of the sufficient condition for solving ② discussed in Sec.
3
Proposition For the minimization problem defined in 22 if there exists a joint density (M, X,Y) Y) E Ap, having the Markovian structure of the form M + +X - or M - - X, then Que (M, X, X,Y) minimizes 2 Furthermore,
1. If (M, X,Y) has the Markovian structure M - X - Y, then UI(M;Y' = laxe [M;Y|X
0.
2. MMQc (M, X, Y) has the Markovian structure M Y X, then UT(M; X(Y) 1Q (M, x = 0.
Proof. Case I: QM X) Ap having the Markovian structure M X X
By the definition the Markov chain, we know that M and Y are conditionally independent given X. Denote QMe (M, (M,X,Y) X) as the joint distribution of the Markov chain M + X Y. Then,
To (M; Y|X)= = 0,
(13)
where JQMc (Ar; (XX is the conditional mutual information between M and Y given X under the distribution QMc (M, X, Y). Equation (3) follows from a well-known property of conditional mutual information which states that AAA B[C) = 0 iff A TL BIC (88
Since TQ 2 RAQ E Ap, due to the non-negativity of conditional mutual information 28
we can conclude that mingeA, lQ(M, ( 2 O. The joint distribution QMc (M, (M, Y) having the Markovian structures M + X Y achieves the lower bound of for the minimization problem mingeAp (M; X(X) implying QMC M, X, Y) minimizes
Case 2: (M,
Ap having the Markovian structure M -
A similar argument as case 1, with the random variables X and Y switched, would allow us to conclude that the joint distribution QMc(M, specifying the Markov chain M + y + X minimizes 2
B
Other PID definitions for which theorems 17 are valid
Our theoretical results (Theorems 7 hold for a wide range of PID measures. Particularly, our results hold for two broad classes PID definitions: PID definitions satisfying assumption (*) in
Bertschinger et al. [16] and Blackwellian PIDs (see Appendix B C of Venkatesh & Schamberg [2 for a deeper discussion on differences between these two families PID definition). Some examples of PID definitions satisfying assumption (*) in Bertschinger et al. [16] are the Williams & Beer's PID
definition [27), the PID definition proposed in Harder et 229 and I-PID proposed in Venkatesh example of Blackwellinn PID is the 8-PID discussed in Venkatesh & Schamberg [22]. Section III-F of Venkatesh et al. (37] further discusses a family of Blackwellian PID definitions for which our theorems are also applicable. We now provide formal arguments showing that our theoretical results are applicable for PID definitions that either satisfy assumption (*) of Bertschinger et al. [16] or are Blackwellian
et al. (37 Similarly,
PID definitions satisfying assumption (+): We invoke the argument presented in the proof dis- cussed in Section 4.2 of Barrett [21] to show that our results hold for any PID definition satisfying assumption (+) in Bertschinger et al. [16]. Barrett's argument relies on the key observation that the UI calculated using BROJA-PID upper bounds the UI calculated using any other PID definition satisfying assumption
(*) in Bertschinger et al. Entropy' 14. Consequently, if one of the UI atoms of the BROJA- PID is zero then the corresponding UI atom of the other PID definitions must also be zero due to BROJA PID upper-bounding them and the non-negativity
the PID atoms. Since the remaining PID atoms are calculated using ( it must be the case that the PID atoms calculated using any PID definition satisfying assumption (*) must be the same as the PID atoms calculated using BROJA-PID whenever one of the UI terms of BROIA-PID is zero.
The defining feature of a Blackwellian PID that the UI atom is zero iff possible to construct the Markovian structure between the random variables (i.e., M - X + Y or M + Y + X), while preserving their pairwise marginals (see Venkatesh al. (2 for more details), In all our theoretical results, show that the random variables in the analyzed system admit
Blackwellian PID:
15

---

Markovian structure, while preserving the pairwise marginal structure. Hence, the UI atom for the systems analyzed Theorems 7 is zero for any Blackwellian PID. Consequently, as the rest of the PID terms are calculated using ( it must be the case that the PID atoms calculated using any Blackwellian PID must be the same the PID atoms calculated using BROJA-PID in our results.
C Lists of Systems that can solved using Theorem
In this section, we provide examples of systems of random variables (M, X, Y) employing well- known distributions whose PID terms can be obtained through theorem 6 The following list is modified from Table of [25]. Note that for the joint distribution P(M, X, X. Y) of the system of random variable (M, X, Y). we only specify P(M), MM) and (MM as any P(M, Y) possible extensions of the corresponding systems for which the upper bound discussed in Sec. [ojs applicable. Note that the extension of the systems provided not exhaustive, and there exists more systems for which the upper bound applicable. In the following examples, we assume a >
having these marginals can be solved through theorem 6 We also discuss
1. System: The distributions of P(M), P(XIM), and P(Y M) are as follows:
(a) M ~ P(M) having support (0, oc).
(b) P(X[M) Poisson( a.M)
(c) P(Y M) Poisson (bM).
The corresponding G( (1 - 70 x)) specified in theorem 6 is Binomial(z, 7) with
? = 1/a, and a,
Result: The Markov chain M - X Ap and (I 0.
An extension of the system for which the upper bound of Sec. 6 in applicable: The
distributions of P(M), PM) and PMY|M) are follows:
(a) M P(M) having support M.
(b) P(X[M) [ (MM) for (M) 0 A e M.
(c) P(Y [M) Poisson (2)))) Poisson(fa(AM),
for f2(M)
V m M.
(d) fi(m) 2 f2(m) V m M and all the underlying distributions are well-defined.
2. System: The distributions of P(M), P(X and M) are as follows:
(a) M ~ P(M) having support (-20,
(b) P(XIM)=N((ul)) ao²).
(c) P(Y]M) N(bM, 2)
The corresponding G(75%
m. (1 - 7) xx specified in theorem is NM 71 7)
with 7 9/2, and a, b > 0.
Result: The Markov chain M + - Ap and U =
An extension of the system for which the upper bound of Sec. 6 in applicable: The
distributions of P(M), P(X and PMY] are follows:
(a) M P(M) having support M.
(b) P(X[M) P(X
(M)). (MM). Gaussian (MM)
for f1(M) O
(c) P(Y [M) (aassian(ff2MMM)) f2(M)) for f2(M) 0Ym M.
(d) fi(m) 2 f2(m) m M and all the underlying distributions are well-defined.
3. System: The distributions of P(M). P(X and P(Y) M) are as follows:
(a) M ~ P(M) having support (0, 1].
(b) P(X[M) =Negative Binomial M).
(c) P(Y M) Negative Binomial M).
The corresponding G(y6% m2 (1 7)6xX specified in theorem is Beta-Binomial(r, ya, (1-))a) with 7 9/a, and a, b N.
Result: The Markov chain M + - - + Ap. and UI(M; 0. An extension of the system for which the upper bound of Sec. 6 in applicable: The distributions of P(M), P(X]M), and PYY) are follows:
(a) M P(M) having support M.
16

---

(b)
Binomial(fi p). for fi(M) E No M and E [0, 1].
(c)
(ino for f2(M) No mem.. M.
(d) fi(m 2 fa(m) V m MM and all the underlying distributions are well-defined.
4. System: The distributions of P(M), P(X and M) are as follows:
(a) M P(M) having support (0, x)).
(b) P(X M)
(c) P) = Gamma(4, M).
The corresponding G(
m: (1 (1- x) specified theorem 6 is obtained as follows:
G(yō 7) ,) Z ~ Beta(b, (1 (1--b) b)) with 1/a, and a, b (0, 0o). Note
that we are following the Gamma(a, (8) notation.
Result: The Markov chain M + - + E Ap and X = 0.
An extension of the system for which the upper bound of Sec. 6 in applicable: The
distributions of P(M), MM) and P(Y]M) are as follows:
(a) A ~ P(M) having support M.
(b)
(M), 5), for f1(M) > 0 m E M and 3 > 0.
(c) P(Y (M) (M) 8). for fa(M) A E M.
(d) fi(m) 2 f2(m) 4 m M and all the underlying distributions are well-defined.
5. System: The distributions of P(M), P(XJM), and M) are as follows:
(a) M P(M) having support (0, xc).
(b) F
(c) M))M) = Gamma(b, M).
The corresponding G
m: (1 - x) specified theorem is obtained as follows:
G(y8 (1 7) 22 = Z ~ Beta(b, (1 b)), and 0 1. Note that following the Gamma(o, o, 8) notation.
Result: The Markov chain M + X Ap and U 0. An extension of the system for which the upper bound of Sec. 6 in applicable: The
distributions of P(M), P(X]M), and P(Y|M) are as follows:
(a) M ~ P(M) having support M. (
(b)
for fi(M) > O
(c) P(Y [M) Gamma(f2(M) fi (A)), for fa(M) > O M.
(d) f2(m) < 1 V E M and all the underlying distributions are well-defined.
6. System: The distributions of P(M), P(X|M), and P)Y are as follows:
(a) M P(M) having support [0, 1].
(b)
.)..
(c) M) = Binomial(b,, M).
The corresponding G(v6%, (1 7)6X, .) specified in theorem 6 is Hypergeometrri((, ya, (1 - 7)a) with y 1/a, and a, N. Result: The Markov chain M -
Ar and U = 0.
An extension the system for which the upper bound of Sec. 6 in applicable: The
distributions of P(M), P(X and P(Y|M) are follows:
(a) M P(M) having support M.
(b) P(X -
for fi (M( mm M and p E [0, 1]. (c) P(Y M) Bino p). for (2(M) E N, vm M.
(d) (im) Z f2(m) m E .M and all the underlying distributions are well-defined.
7. System: The distributions of P(M), P(X|M), and P(Ý M) are as follows:
(a) M P(M) having support over k-dimensional simplex.
(b) P M) Multinomialy (a, M).
(c) P(Y M) Multinomialg M).
17

---

The corresponding((1000 (1 - 7) m. x) specified in theorem 6 is Multivariate Hypergeo- metric (x, Ty) with y 1/a and a, b N.
Result: The Markov chain X Y Ap and (M; Y X 0.
An extension of the system for which the upper bound Sec. 6 in applicable: The
distributions of P(M), P(X [M), and P)Y are as follows:
(a) M P(M) having support M.
(b) P(X M) (f
for fi(M) No VoME...
(c) P(Y M) = Multinomial(f2(MM p), for f2(M) E Y
(d) [1m) Z (m) A .M, is valid probability vector, and all the underlying
distributions are well-defined.
D Lists of Systems that can solved using Theorem
In this section, provide examples of systems of random variables (M, X,Y) employing well- known distributions whose PID terms can be obtained through thcorem 7 The following list is appropriately modified from Appendix A of (26). Similar to Appx. C we only specify specify P(M), P(X(M), and P(Y of the joint distribution P(M, X, Y). as any P(M, X, Y) having the same marginals can be solved through theorem 7
In the following examples,
assume b.
1. System: The distributions of P(M), P(XIM), and P(Y M) are as follows:
(a) M ~ P(M) having support (0,
(b) P(X[M)= = Exponential(M))
(c) P(Y M) Geometric (M)+ where E (0)
Results: The Markov chain - X + Y E Ap. and UTMM; = 0. The corre-
sponding QMc(Y)X) required for constructing the Markov chain is Poisson(=X).
2. System: The distributions of P(M), P(X|M), and and P(Y are as follows:
(a) M ~ P(M) having support M.
(2M)), where (m)) > m E M. (b) P( Gf (M)),
(c) P(Y [M) Negative Binomial(fi (M), (L(MT/(66())) where T E (0, 00)
Results: The Markov chain M - + Y E Ap. and U = 0. The corre-
sponding QMc(Y)X) required for constructing the Markov chain is is Poisson(rX)
3. System: The distributions of P(M). P(XIM), and P(Y are as follows:
(a) M ~ P(M) having support (0, oc). 1).
(b) (eX]M)=Betal)
(c) P(Y M) is a discrete distribution with over support set (0, 1,..., N), with
Ts + m)N!
P(Y = y/M = m) = m I(N + 1+ m)y!
where I'(.) is the Gamma function.
Results: The Markov chain M X + + Ap. and UI(M; 0. The corre- sponding QM)(Y required for constructing the Markov chain is Binomial(N, X).
4. System: The distributions of ((M), P(X]), and M( are as follows:
(a) M P(M) having support (0, oc).
(b)
(c) P(Y (M) is a discrete distribution with over support set {0, 1...., N). with
Ily + mm)! T(N +1 m)y!!
P(Y y m)) m
where (-)) is the Gamma function.
Results: The Markov chain M + - Y Ap, and (T; (M; The correspond-
ing Qm (Y (X) required for constructing the Markov chain is is 1 -
18

---

5. System: The distributions of P(M), P(X[M), and P(Y M) are as follows:
(a) M P(M) having support (0, oo) for dimension d.
(b) P(X M = -iirricceee(M,,......
(c) P(Y M) a discrete distribution having support {y: E fI,..., vi e
1= 1/i NN, and
EE M1) N a M
P(YM)
r E Mi +N
((M)
where (()) is the Gamma function.
Results: The Markov chain M + X Y Ap, and U Y = 0. The corre-
sponding Qmc required for constructing the Markov chain is Multinomiala(N. X).
E
Miscellaneous PID Results
This section explores the analytical calculation of PID terms for five more systems of random variables (M, X, Y) with joint distribution P(M, X, Y). These five systems' P(M, X, Y) employ well- known distributions, but their respective analytical PID calculation does not result from Theorem I Theorem 7 Of note are the results of P(M, X, Y) employing exponential and uniform distribution, which states that these systems will always have both unique information terms zero.
E.1 PID for the miscellancous system M1.
Miscellaneous system M1: Let (M,X, Y) be a system of random variables with joint distribution P(M, X, Y). The joint distribution P(M, X, Y) describes the miscellaneous system M1 if satisfies the following two properties:
1. M P(M), and has support 00).
2. (0)
M), and P(YIM) (xpoo)nnnnnnnnlyyyyy where 7x, m
Proposition 2. Let M, X, and be random variables whose joint distribution P(M, X, Y) describes the miscellaneous system MI defined above. Then, Ap contains distribution QMC (M, M, X, Y)Y), which admits both Markovian structures, ie., M X +Y and M Y - X. Consequently, both UI(M;Y' Y = U (MM (\))= =.
Proof. We prove proposition by explicitly constructing a joint distribution QMC X,Y) E Ap that admits both Markovian structures, namely, M X + Y and - > X. necessary and sufficient condition for the distribution QMC (M, X, Y) to admit both Markovian structures is that it should satisfy (4..
Mc (M, X, =
(14)
We now briefly outline the structure of the proof. The rest of the proof can be divided into three parts:
1. In the first part, explicitly construct Qarc QM, (M,.Y).
2. In the second part, we show that the QMC M, Y) constructed in the first part lies in Ap.
3. In the third part, we conclude our proof by showing that QMc X, Y) satisfies 14 and, consequently, admits both Markovian structures.
Part 1: Specifying (M,
We specify QMc X, Y) by individually specifying QMc(M), QMc(X and (Mc(Y
We choose QMc(M) and Q (MIMM as follows:
QMc(M) = P(M), and (M M) P(XIM).
(15)
19

---

The distribution (xIX) is specified through the following deterministic transformation:
7x X. 7
Y
(16)
Part 2: Showing (M, Ap
For showing QMc Y) Ar, we first need calculate @Mc
Calculating (Y By lemma D X Exp(A), then aX ~ Exponential (Va). Since
(Mc(X [M) we can use the result of lemma follows:
to calculate Q QMc as
XXx
TV 2 M
QMc(Y = MC
M = Exponential
TY
7X
Exponential (yyM) = P(Y)M).
(17)
Then, from 15 and 17 we can conclude:
Qma (M.X) MMc = P(MP((IM) = P(M.X),
(18)
QMc(M, QMc(MM)Mcc(1IA)) P(M))(YYIM) P(M,Y).
(19)
Hence, QMc((M, Ap.
Part Showing QMc(M, satisfies (4)
is trivial to see that by the construction of QMc((,,XXY given in Part 1, we have:
QMc(M, QucC(M)QxxcXXXMMMM0000CCC
(20)
By the chain rule, we have:
Qmc (MCN5V=)CQQNM)NNNCCCMMMMMMMMCMC
(21)
Hence, for showing that (,, QMc(X M) (xI()
satisfies
14
we only need to show that
Showing QMc(X M) QM)(X Observe that Y 7x/TVX X by construction. Therefore, we can equivalently express X = " "v/Tx3 // Y. Consequently.
QMc(X Y, M) = I[[X = ">/TXX as X = w/xXY,
(22)
where I[+] is the indicator function. Since R.H.S the above equation does not depend on M, we have:
(), = I[x w//xX1 QM)()
(23)
Combining 2T)pss and (33.. we have:
X, Q.C.YY/)-OOcuMMQexeMMMeeeeee
(24)
Combining 20 and 24 we can show that
Y) satisfies 14).
Conclusion:
Combining the results of parts and 3,
conclude that the QMC X, Y)
AP and contains both Markovian structure M - X + Y and M + + X. A unique implication of QMc,M, X, Y) admitting both Markovian structures is that both Lawe (M; IQue (M; YJX) = O. Consequently, both minimization problems presented in (2) are minimized by QMc (M, X) with U UT(M: X minqEA, IQ(M;
and UI(M;
mingEAP IQ (M; X = 0, completing our proof.
Lemma 1. If X Exponential(A) with X E oc), then 7X Exponential( E (0, oo).
Proof. We denote the p.d.f. of the random variable X as px and, by lemma statement, we know:
Ae-zh 2 0
Px [X= = x) = Exponential(A)
(25)
0
otherwise
20

---

Denote the p.d.f. of the function f(X) of the random variable X as P-(x)(). We want to derive P-(x)(-) for f(X) y.X Since, f(X) = 7.X is an invertible function, we can use the change-of-
variable technique for deriving (+)x
(f(x) - px xx (f) 2
y 2 0 otherwise
dy
Exponential
(26)
E.2 PID for the miscellaneous system M2.
Miscellancous System M2: Let XM, Y) be system of random variables with joint distribution
P(M, X, Y). The joint distribution P(M, Y) describes the miscellancous system M2 if it satisfies the following two properties:
1. M P(M) and has support (0, oc).
2. P Binomial( M, px), and P Binomial(M,pyy
This system is adapted from the negative binomial decomposition listed in Appendix A (26).
Proposition 3. Let M, X, and Y be random variables whose joint distribution P(M, X, Y) describes the miscellaneous system M2 defined above.
1. If Px I PY, then Ap contains Markov chain of the form M + X + Y, and U (X) =0
2. If Px 2 py. then Ap contains Markov chain of the form M + Y + X, and UI(M;XXY))
Proof. We only provide an explicit proof of condition 1. The proof of condition 2 follows the same steps as the proof of condition 1 with parameters X and Y switched.
Proof of condition 1: We briefly outline the proof structure. The proof can be divided into two parts:
1. In the first part, we explicitly construct a joint distribution QM, having the Markovian structure M + YY
2. In the second part, conclude proof by showing the (M, constructed in the first part, lies in Ap.
Part 1: Specifying QMc(M,X,Y)
Since QMc(M, is the joint distribution of Markov chain M + - Y, we have (X
Taking advantage of this special structure of QMe (M, Y), we specify QMc(M, X, YY by individually specifying Q (M), QMcc(X)M and (Mc(Y[X) We choose (Mc(M), QMc M) and (Mc(Y)X) follows:
X, X)
(Mc(M) P(M), QMC P(X]M) and QMc(Y Binomial (xoomialXX.)),
where p px pY) py (1 P
(27)
Part 2: Showing QMC (M, X, Y) E Ar
For showing QMc
Ap. we first need calculate QMc
Calculating (M Since Quc (X Negative Binomial [M, px) and QMc(Y|X)
Binomial (N. p), we can use lemma for calculating Q(Y[M):
@M Negative Binomial(M, = P(YIM).
(28)
21

---

Then, from 27 and 28 we can conclude:
QMc X) QMc(MM)Mcc
P(M) X),,
(29)
QMc((, = Qmc(M)QQc((1IM) P(M))(YYIM) = P(M,Y).
(30)
Hence, (M, X, E Ap and consequently, by proposition concludes our proof.
UT(M;Y = 0. This
Lemma 2. Let X Negative Binomial (N, and P(Y Binomial(X, (X,p) with p
Px(1- py] (1 P px PY 2 Px. PX. E 1]. and N No. Then, Y Negative Binomial(N,p)) .p)).
Proof. First, in order for X(Y to legitimate distribution, we need to ensure that p [0, 1], Le., we need to show that:
0 0sps1.
(31)
Since px, (1 (1-pr); (1 - px),pr 2 Oas E (0. 1]. we can easily verify the first inequality:
px(1 pp) (1 px PY
2 O.
(32)
For verifying p 1 will use the assumption that py px:
1 1
px S PY
(33)
PY
px
Subtracting 1 from both sides of the above inequality.
1 PY
1 - PX
15
(34)
PY
px
py
px
Rearranging terms, we obtain:
Px - pr) ( $ <1 as (1 px px)py/
(35)
(1-Px)py
For calculating P(Y), we borrow the result of proposition 13 in 126 which states that
if X Negative Binomial(M,pxx) and P(YIX) Binomial(X,p)) then P(Y)
Negative Binomial
(N, px Px pPx
Hence,
P(Y) Negative Binomial
N,
PX
Negative Binomial
Px N
px + P - PPx
px + p(1 px
Negative Binomial
N.
px
px (I px Jpy
Px (1 pyr)
(1 px)
Negative Binomial
N,
px + px (1-pT] PY
Negative Binomial
N.
PXPY
PxPY + px 11-18 (y-
PXPY
Negative Binomial
N.
Negative Binomial (N, py)
(36)
Px
22

---

E.3 PID for the miscellaneous system M3.
Miscellaneous System M3: Let (M, X., Y) be system of random variables with joint distribution
P(M, X, Y). The joint distribution P(M, Y) describes the miscellaneous system M3 if satisfies the following two properties:
M P(M) and has support (0, oo). 2. P(X]M)=UUiform(( M) and P(Y M Tr M), where 7x: TY (O).
Proposition 4. Let M, X, and be random variables whose joint distribution P(M, X, Y) describes the miscellaneous system M3 defined above. Then, Σp contains distribution Q Mc (M,X, X, Y), which admits both Markovian structures, i.e., M + X + Y and M +Y - Consequently, both UI(MYY (X) U
Proof. We prove proposition 4 by explicitly constructing joint distribution QMc (M,, X,Y) Ap that admits both Markovian structures, namely, . + X + Y and M + + Y X. necessary and sufficient condition for the distribution QMC X, XY) to admit both Markovian structures is that should satisfy (37b:
QMC (M, Y) hww.a//Awwaaocoa0000000aoCcaCocd
(37)
We now briefly outline the structure of the proof. The rest of the proof can be divided into three parts:
1. In the first part, explicitly construct (M,
2. In the second part, show that the (M,c(M,X,Y) constructed in the first part, lies in Ap.
3. In the third part, we conclude our proof by showing that (M, satisfies 37 and,
consequently, admits both Markovian structures.
Part Specifying (M,
We specify (M, (M, X,Y) Y) by individually specifying QMrc(M). QMc((X(MM) and QMc(Y
We choose QM (M) and QMc (X as follows:
(M) = P(M), and ( M) P(X]M).
(38)
The distribution (M is specified through the following deterministic transformation:
Y TY X. 7x
(39)
Part 2: Showing
Ap
For showing QMc X, Ap, we first need to calculate QMc
Calculating QMc M):
Since QMc (X -ni)-onifmmmm/// 7x and Y 7Y/7x we can use
the result of lemma 3 to calculate QMc (M) as follows:
QMc(Y QMe
TY x 7x
M
Uniform (0,
rY 2 7x M) 7x
Uniform (0, M))
P(Y
(40)
Then, from 38 and 440) we can conclude:
@Mc(M, X) QMc(M)) (XIM) P(M)P(X(M) - P(M,)) (M) = (Mc(MM)QNc(YIMM) P(M))(()M)) = P(M,Y).
(41)
(42)
Hence, (M, AA..
Part 3: Showing (M,,,M Y) satisfies 37
23

---

It is trivial to see that by the construction of QMc(MM,X,Y given in Part 1, we have:
(43)
By the chain rule, we have:
QM, Y) QuccCWWA9AacYYTMMMMeecCCCXX...
(44)
Hence, for showing that (M,, (M, X, Y) satisfies (37). we only need to show that QMc(X QMc(X Y).
Showing QMc(XIY,M) = (M Observe that Y =1V/7xX X by construction. Therefore, we can equivalently express 7x xx/1> Y. Consequently,
QMc( Y, M) = I[X 7x/1x Y| as X 2x/TxY,
(45)
where I[+] is the indicator function. Since R.H.S the above equation does not depend on M, we
have:
( = 1[ X 1x//x = Qmc(X
(46)
Combining 44 and (46). have:
= MMN
(47)
Combining 44 and (47). we can show that Quc (M, (M, satisfies (37)
Conclusion:
Combining the results of parts and 3, we can conclude that (M, E
Ap and contains both Markovian structures: M + X Y and M Y X. A unique implication of @Mc(M, X, Y) admitting both Markovian structures that both To (M; X
Towe (M; Y|X) O. Consequently, both minimization problems presented in (2 minimized and (I(M)Y
by (M,
with U
mingeAp IQ(M;
mingeAp IQ(M; (M) = 0, completing our proof.
Lemma 3. Let X Uniform(O, a) for some a E R, then xX Uniform(O, 7a) for E R and a,7 11700
Proof.
We denote the p.d.f. of the random variable X as px (+).
By lemma statement, we know:
1 x E [0
px(X = x) Uniform(O, a)
[
(48)
0
otherwise
Denote the p.d.f. of the function f(X) of the random variable X as (.) We want to derive (-) for f(X) 7X. Since, f(X) = 7X is an invertible function, we can use the change-of- variable technique for deriving (+):
df d
1/iaal 0
y E [0, 7a] Uniform(O, 7a). otherwise
P)x))((XX y) -xx(X = (-)
dy
E.4 PID for the miscellaneous system M4.
Miscellaneous System M4: Let X,, Y) be system of random variables with joint distribution
P(M, X, Y). The joint distribution P(M, X) describes the miscellaneous system M4 if it satisfies the following two properties:
1. M ~ P(M) and has support [0, 1]. 2. M (M), and P(Y M) M 3:p--pM),, where E [0, 1].
This system is adapted from the Bernoulli decomposition listed in Appendix A 26
Proposition 5. Let M, X, and Y be random variables whose joint distribution P(M, X, Y) describes the miscellaneous system M4 defined above. Then, Ap contains a Markov chain of the form M X Y and (IMM; X)= =0
24

---

Proof. We briefly outline the proof structure. The proof can be divided into two parts:
1. In the first part, explicitly construct joint distribution QMc(M, having the Markovian structure M + - -.
2. In the second part, we conclude our proof by showing the QM, Y) constructed in the first part, lies in Ap.
Part 1: Specifying (M,
Since @M, X, Y) is the joint distribution of Markov chain M X Y, we have Qme(M, X, Y)= =
Leveraging the special structure of QMc(M X,,Y, Y), we specify QMc X) by individually specifying QMcc(M), (X)c((M)) and (M)) We choose QMc, QMc [M) as follows:
QMc(M) P(M), and ( M) P(X
(49)
The distribution QM)(Y is specified through the following stochastic transformation:
Z) Z(1 (),-x),
(50)
where Bernoulli(p) and TL MM.
Part 2: Showing QMc X, Y) Ap
For showing (M, E Ap. we first need to calculate QMc
Calculating Q (Y [M): For calculating (M) states that if X" ~ Bernoulli()), Z' Bernoulli(p'). Z' X', and Y (1 - Z') + (1 X') then Y Bernoulli
the result of proposition 10 in 226 which
2p'0).
Noting that QMc(XJM) = Bernoulli(M) (M) and Y = X - 2)+ ((-1)), where 2 Bernoulli (p) and Z 11 X X,M, we can calculate Q (Y
(Y) MM) follows:
QMc(Y|M) = Bernoulli (M + 2pM P(YIM).
(51)
Then, from
449 and (5) we can conclude:
QMc XX) (Mc (X P(M))(X(M) PM,,
(52)
QMc(M,Y) = (Mc(MM)Qcc((IM] = P(M))(Y(M) = P(M,Y).
(53)
Hence, @MccM, XY E ÁP and consequently, by proposition UI(M; 0. This
concludes our proof.
E.5 PID for the miscellaneous system M5.
Miscellaneous System M5: Let (M, X, Y) be system of random variables with joint distribution
P(M, X, Y). The joint distribution P(M,, X, Y) describes the miscellaneous system M5 if satisfies the following two properties:
1. M P(M) and has support over simplex of dimension d.
2. P ((XMM) -
Ma). and P(Y Cateoorl-ca where
di = (1 - M, + p/d.
This system is adapted from the categorical decomposition listed in Appendix 26
Proposition 6. Let M, X, and be random variables whose joint distribution P(M, X, Y) describes the miscellaneous system M5 defined above. Then, Ap contains a Markov chain of the form M + and ((X)=
Proof. We briefly outline the proof structure. The proof can be divided into two parts:
1. In the first part, we explicitly construct joint distribution QMC X, Y) having the Markovian structure M x X
25

---

2. In the second part, we conclude our proof by showing the QMc the first part, lies in Ap.
X, X.Y) constructed in
Part 1: Specitying QMo
Since QMc X, Y) is the joint distribution of Markov chain M X + Y. we have Que (M, X, XYY) (MM structure of QMe (,,X,)) we specify Quc X) by individually specifying Qarc(M).
Taking advantage of this special
OMc(X(M), and Q ()))). We choose QMc(M). QMc(X[M) as follows:
QMc(M) P(M), and Quc (X M) P(X[M).
(54)
The distribution QMc(Y is specified through the following transformation:
= X(1 - -) ZD;
(55)
where Bernoolli(p)) Categorical (....., 2), and D, Z, and (X, M) are jointly indepen- dent.
Part 2: Showing QMC X, (M
For showing (M, E Ap, we first need to calculate QMc(Y
Calculating (Mc(Y (M)) For calculating QMc(Y (M), we use the result of proposition 11 in 26). which states that:
~ Categorical (0....., ..), ~ Bernuuli(y'), D' ~ Categorical (1/,,..,, 1/d), and y (1(1 7') + D'' with X', 7², and D' jointly independent, then y" Categorical (8),-.. ... %2) with of = (1 p0) v/ja.
Ma) and Y X(1 2) + 2D, with X,D, and 7 jointly conditionally independent conditioned on M, have:
Noting that QMC (X MM Categorical (M,
p Q ( Categorical oa), with o, MM(1 - p) d
P(Y M).
(56)
From 54
and 56 we can conclude:
Qmc (M, X) QmC (M) (X P(M)P(XINN) P(M, X),
(57)
Que (M. Y) = [MccMM]Qcc(YIM = P(M)P(YIM)
V).
(58)
Hence, QMc X) E Ap and consequently, by proposition UIMM; Y 0. This
concludes our proof.
E.6 PID for the miscellaneous system M6.
Miscellancous System M6: Let (M, X, Y) be system of random variables with joint distribution
P(M, X, Y). The joint distribution P(M, X, Y) describes the miscellaneous system M6
satisfies the following two properties:
1. M ~ P(M) and has support N.
2. P(X [M)Multinomiala, (X; M,)x), and M (Y: PY).
Proposition 7. Let M be a random variable. Furthermore, assume & and & are random vectors of size dx and dy, respectively, such that
Y) describes miscellaneous system M6.
1. If. min P. 2 minig(dy) xx then there exists Markov chain of the form M - X in Ap, and UI(M; Y X = =0.
2. If. min P. S minie(dx) xx then there exists a Markov chain of the form M -+ Y
X in Ap, and U =
26

---

Proof. We only provide an explicit proof of condition 1. The proof of condition 2 follows the same steps as the proof of condition I with parameters of X and Y switched. We briefly outline the proof structure. The proof can be divided into two major parts:
1. In the first part, explicitly construct joint distribution QMc (M, Y) having the Markovian structure M x Y.
2. In the second part, we conclude our proof by showing the QMc Y) constructed in the first part, lies in Ap.
Part 1: Specifying OMc(M,X
Since QMc X, Y) is the joint distribution of Markov chain M have QMc X, Y) = QxccMM)Qcc(X(MMMQ
X Y.
(Y X). Taking advantage of this special structure of QM (M, X. Y). we specify QMc X, Y) by individually specifying QMc(M), (Mc( and QMC (Y (J)). We choose (M), QMc(X [M as follows:
QMc(M) P(M), and QMc X M) M).
(59)
Specifying Q( X): Let xi = arg aiggdy p and ky = arg min min p.
Then,
-Mulinomaall (Y: Iz. PÝ). where
Pay Ei P.
E oe EieaxIiaz P.
pr
(60)
Eielexiu P
We first show the proposed QMe (Y X) is a valid distribution. By condition 1:
min iE|dy]
Y 2
min p.x
(61)
sEdx
Σ
P 2
Σ
p 22 Vj
(62)
iE x
RE(dy|Ve
O, V [dy]
(63)
Eieldx]Vkg P
where the last inequality is due to px and By forming valid multinomial distributions.
Part 2: Showing QMC X, Ep
For showing QMc X, Y) E Ap, we first need to calculate QMc (M))
Calculating QMO YlM): We use lemma to calculate Q(Ý[M). Note that QMc(X = Multinomially (M, px), hence we have:
@Mc MM) = Multinomialdy (M, )r where,
P. EletaxIVy P
pby EvelxeRay p. Exenxliy P.
Exeaexiay 2e Ex'y, 1 pp
By
Exelxxieg Pd
Exeldx Pi
B (pY p 1- --Eeedy) p. -r
M Multinomiall, (M, pY) P(Y(M).
(64)
Then, from 55 and (64). we can conclude:
QMc(M, X) QMcc(M)MMcl X MM = P(M)P(X(M) = X),
(65)
(Mc Y) QMc(M))Mc (YM P(MM)P( = -MM Y).
(66)
Hence, QMc(M, X, Y) Ap and consequently, by proposition concludes our proof.
UT(M; =.
This
27

---

Lemma 4. Suppose X Multinomialax (n. px) and YIX Multinomialdl where I C (1....,................. dx}. Then Y ~ Multinomialay (n, )²), where
EEEIXX;10
Σ jez Pi) EjEr ps} [ 1s) i= dy
ssiddy
(67)
Proof. Without loss of generality, reshuffle the class indices of Y such that I= -i:1 S w). Nox-1; Di=l Is I n]. By the law of total probability,
where = Let C== E No.
P( y)= Σ P(X x) 8| x)
(68)
SEC
(SEII 23j)
dy
Σ SEC
Pi
(69)
He Vi!
Remove from the sum all terms that do not depend on Note that Ydy = EjEI 23 - dy
y and
Buy (1 2i=1 di 4dv
Jdy
are both functions of y. Let ys Edn yi and 9== -E 9i-
T p.
(ISEz 22)) (1 90) (e)xxar=
n!
p(Ý y)
==
dy ysl
XEC I xi!
(EjEz - Un)
Form a multinomial coefficient outside the sum:
dy
n!
P( y)=
II
Vi
(70)
11 dy Yo! (n vo)!
iml
(n Us)! I
dx pfr (Ijer zi)!
(1 qo) E
(71)
IT xi! (EjEz Us
XEC
Let I = - fi:w < i dx} be the set of indices that are not taken in the number of trials for Y.
Consider the sum over x E C alone (7I). Separating terms that belong and Io,
(Ejera)) 23 I1 Tz!!
Σ REC 1lEE xzy! Ejer - V.)
(n (n-w4 ys)
II JEZ
P3 (1 96) SSe zs-
II JEI
3
Let C" (x' Ndx-1, -dx-1 3y can cquivalently sum over elements in C¹ Perform change of variable with u = ΣjEr Lj and
Idy
dy i=ن Yi
-dy-1 w.). Note x C X E C²,
define B = {zg E X:j E I, ESEI 2 = u + ys). Then the sum becomes
Σ
(n yz)! Jjzzo lu!
II JEZZ
P3 (1 gz)*
Σ EB
W + Us
II P3
(72)
IJi .... Fim
REC
JEI
Note that the inner sum simplifies by the multinomial theorem:
k++
(n vo)!
II
P3 Ty (1 2²
Σ Pj
(73)
1liero 1.ylul JEZ0
&EC'
JEI
V/o
Σ Pj JEI
1i I.y!- u!
(n Ho)!
Pj
25
Σ
Pj
9o
Pj
(74)
XEC*
JEZ
JEI
JEI
28

---

Reapply the multinomial theorem to the sum of x over C":
Σ
Pj
Σ
Pj
Pj
9o Σ
Pj
(75)
1ET
JEZ9
jET
JET.
Vx
R- R-
Σ
Pj
- qs Σ
Pj
(76)
JEI
JET
Inserting 76 into 71 the marginal becomes
2
B-
P( = 3) =
I1 dy-1 y:! . (n Uo)!
n!
9:
1
Σ
Pj
(77)
JEE
JEI
n! 11,2r dy ( CjeTP; Pj -96)
qs
Σ
Pj
(78)
M ys! (n Uss!
JEZ
Multinomiala, q")
(79)
where q" is defined as in 67
F Proof of Theorem
In this section, provide the proof of theorem [ Since theorem provides the analytical PID terms for the univariate affine continuous stable system defined in Sec. we briefly restate certain key properties of the univariate stable distributions and the corresponding univariate affine continuous stable system for convenience.
F.1 Univariate continuous stable distribution
Univariate continuous stable distribution are a large class of distributions that naturally arise in the context of generalized central limit theorems. We refer the reader Appx. N.2 for more details univariate continuous stable distributions. We now list certain key properties of univariate continuous stable distributions that make of in the proof of theorem
1. If X is distributed according to univariate continuous stable distribution, then the sum of two independent copies of X, denoted as X2 and X2, follows the same univariate continuous stable distribution upto a scaling and translation operation, i.c., aX1 + 6X2 cX + d for a, b, and E R.
2. The p.d.f. of univariate continuous stable distributions is characterized by four parameters:
stability parameter denoted as a E (0, 2]. skewness parameter denoted as 3 E [-1, 1]. scale parameters denoted as 7 E (0, 20), and location parameter denoted as μ R. We denote the p.d.f. of a univariate continuous stable distribution as pcs 14
F22 Definition of univariate affine continuous stable system
Let (M,X, Y) be a system of random variables with joint distribution P,M,X,Y, The joint distribution P(M, X, Y) describes the univariate affine continuous stable system if it satisfies the following two properties:
1. M P(M) having some support M C
The conditional distributions of random variables and Y conditioned on M can be expressed through univariate continuous stable distributions with affine dependence on M, ax.aM i.e., P(X -)=Pcs 8x, 7x, aM aM++b b)) and P(Y]M)= sac))(ac), TY, eM d),
where a, b, d E R.
29

---

F.3 Formal proof of Theorem 1
We first discuss and provide the provide the proofs of lemma 5 and lemma 6 that we will use for proving theorem In the following proofs, (g)(x) denotes the sign function:
1
x >
sgn
0
x
(80)
x
<0
Lemma 5. Xf X ~ pas(a, (. then any 1) + and n. E R, we have
Pcs(a, (. Inly, 7/4 k)
a 1
aX 1 pos (o, |177, qp 2/a8mm log(In))) a
Proof. See the proof of proposition 1.4 part (b) in 23
Lemma 6. If X1 pas(a, (,,, B1, H1) and X₂ ~ pas(a, 82, 72, M2) are independent, then X1 X2 PCs(a; B3 7.14) where
B1n + 3212 B
== 72: and μ μl P2-
A + 72
Proof. See the proof proposition 1.4 part (c) in 23
Theorem Let M, X, and be random variables whose joint distribution P(M, X.Y) Y) describes univariate affine continuous stable system. Without the loss of generality, assume lol/xx 2 lol/vy- If - By 2 fxxle//yyaa)) (1 8x) ) and + By 2 (xxlel//xya)) (1 + 3x), then Ap p contains Markov chain of the form M + X Y and (())
Proof. We first note that we can always assume lal/xx 2 Ic//yy without the loss of generality because if 10/TV Jal/2x, then
always switch our nomenclature refer Y as X, and X as Y.
We now briefly outline the proof structure. We divide the proof into two cases ml/7x |cl/nv and 1ol/7x = jol/yy. The proof for both cases follows essentially the same structure consisting of two major parts:
1. In the first part, we explicitly construct a joint distribution Markovian structure M + X
having the
2. In the second part, show that the QMO (Mrc(M,X, X, Y) constructed in the first part, lies in Ap. Therefore, we can then apply the result proposition [ to conclude UI (M;Y Y\ = 0.
Case 1: lol/7x > 10//xr
Part 1: Specifying QMc(M,X,Y)
We explicitly construct the desired Markov chain M + X + Y by constructing a larger Markov chain M - X + X' y' Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M + +
Denote the joint distribution of the Markov chain M + X X' + Y' Y as (Mc X,X', "Y,Y",Y), We can decompose (M, X', "',YY, Y" by utilizing its Markovian structure follows:
QMc(M, X, X',Y = QMc((M MXMMMMMM (C XVT
Consequently, we can specify/construct the distribution Quc (M, X, X', Y Y) by individually
specifying (Mc(M), QMc(XIM), (X'X() (M' and QY
Specifying QM (M) and (MIM: We choose (M and QMc (Mro as follows:
QMc(M) P(M) and Qxc (M) = P(X(M),
(81)
where P(M) and P(X]M) are marginal distributions derived from the original joint distribution P(M, X, Y) (discussed in the theorem statement) over which the bivariate PID is being calculated.
30

---

Specifying (xc(X:)):: The distribution Qx is specified by the following deterministic transformation:
a +
X'=
(82)
1 1x + (23x log (og((/a)))/aaa)
a
We will also derive the distribution Q(X*) before proceeding with our construction, as we will need it later in the proof to show that the constructed (,) E
Deriving M (Xr](X Since Q (XJc X' is scaled and translated version of X, we can use lemma to derive QMc(X' The exact expression of (Mc(X'J. (X m) is provided in (83).
m) pcs( 8x ix,am b) for fixed m and
QMc m) -pcs(a, 8x, (x. 10 where
(/) #+1 sgn 1/o3xx a=1 =gn((/a) -gn sgn(a)8x.
3x
1/1 nx a#1 7x 1///xx
7X'
a=1
|a|
[am + b) log((//a)) /al)//// 19 loo((1//l////
a 1 a=1 a
b
P (am + b) + 79
QMc m) (cs(a, 3xsgn(a). 7x/lal, t/a).
(83)
Specifying (Y")(")"'';; In order to define QM)((" we need to define an auxiliary variable e. The random variable pas(e;a, ,',,,,,,,, v',),, where
=gn(())
(T1/1e1)* By (xx/1al)) Bx (1r/1=1² (1x/1a)"
n'= ((17)
[c|
7x |a|
1
d
(84)
c a
In order for to have legitimate continuous stable distribution, we need to ensure that 3 and μ lie within their appropriate bounds as specified in Appx. F.1 It is trivial to see that μz' E R.
Showing / oo). Under the assumption of the case 1, know that:
|a|
|c|
7x
Manipulating the above inequality, we obtain:
7x
yY
7x
a
Y
7x
a
0
c
[a
c
[a]
Xx
7 > 0.
(85)
[a
Hence, (85) shows that 7/ E (0, oo).
Lastly, showing that 3' E 1]. Note that the condition 3' E [- can be equivalently expressed as |,81| < 1. For showing that 3 [-1, 1]. we will show that the inequality |3'| < is equivalent to
the inequalities: (7xlel/|olxx)" (1 [x) 1 - By and (I + 3x) (7x (xxlcl/jalxx))
< 1 By, where
the last two inequalities hold by the theorem assumptions:
188 + + (a) (T</1<1)" By - (7x/1al)" 3x
(1r/111) By (7x/1al)) 3x
$1
S 1,
(Tr/1c/) (1x/1u))
(Tr/1cl)* (Tx/1al)a
15 (1r/1cl)* By - (1x/11l)) Bx
1.
(86)
(7r/1el)" - (Tx/1al)*
Multiplying (1r/ - (1x/1al)) on both sides of the inequalities in (86).
(xx/1a))* - (Tv/1cl)* (xt/aa)"((tv/1a)0 s (tr/1el)* " By ( Bx & (tr/1el) - (7x/1al)*
(87)
31

---

Adding (xx/lal) Bx on both sides of the inequalities in 87
(1x/1al)* + (1x/1a)) 5x (1r/141)* & (mx/1e) )a By I (nx/1a1) (1x/1aa) + (1x/lal) Bx:
+1++Bx (2x/lal)) - (Tx//a4)* (1x/1e1)* By s (Tx/1a1)* + (xx/lal) (3x 1).
(88)
Dividing (r>/1e)) on both sides of the inequalities in 88
(1 + (x) (xxlec//alm)) - 1y By = 1 + (xxle///alx)) (3x 1).
(89)
Simplifying the inequalities in (89
(1 + 3x) (1xlel//alm) 1 BY,
(90)
By - 1 (xxlel//aly>) (3x - 1).
(91)
Multiplying -1 on both sides in 91
(xxle)/falm)) (1 Bx) S By
(92)
Hence, by above analysis we can conclude that the inequality |38 is equivalent to the inequalities:
(xxlo///alr>) )" (1 Bx) - 1 By and (1+ (xxlel/1alr)) - 1+ By, which ensures S E [[-1, 1].
As 3 1], 7 E oo) and n' E IR, continuous stable distribution. Furthermore, we define IL (M, X,,,X" X'), i.e, Pr(c[M, X, x') = Pr(c) and Pr(M, X, X'(c) = Pr(M, X').
conclude follows legitimate univariate
now use to specify QMccY'IXX as follows:
='= X' +6.
(93)
We also derive the distribution Q(Y'MM) before proceeding with our construction, we will need it later in the proof to show that the constructed QM X, E Ap.
Deriving Q M): We will use the distributions QMc(X'M and Qme(e) to derive the distribution of Q (M'l(')
R = QMo(X' +c=y/(==m)
O k/ M
- -M 3/ -k, M=m)dk.
=
Using the fact that L M and IL X'M,, have:
QMc(Y' = M = =m)= 1
(94)
Since (e) and QMc M m) are univariate continuous stable distributions for a fixed m (see (84) and (83), equation (4 describes convolution of two univariate stable distributions for a fixed m. Convolution of two univariate continuous stable distribution akin to summing two independent random variable having the same univariate continuous stable distributions, and
32

---

consequently (Y'MM) can be derived using lemma 6
Hence,
@Mc | m) pcs(a, (a, 3,7, 7,P)) ))) where j1 = m + b a
+ + = m + b
m
+
a
c
((
a
a
=
xx
t
7x
+
7x
[a]
lal
[c|
a
[c]
8 (1x/1al)" sgn (a) 13x 7/000'' (7x/ al)" +yya
(10/1al)* sgn(a) 8x ((>r/1al)) (xx/ (sgn(a) (or/1a1) (xx/1a1)) (Tx//al) + (>//1=])* (Tx/lal))
(Dx//el)) -(7x/1a1)
= =gn((a) (Tx/Ta))* 8x + (Tr/1el)* By (Tx/Ial)) Bx = =sgn(a)
(7r/1el)* By (Tr/1e*
(TV/1ol)a
= =g
(95)
Specifying QMc Qwe is specified by the following deterministic transformation:
y= cx' (2sg(((g(a))3r log(c)))/<
ex'
a +
(96)
a ===
Finally, we construct the desired X,, from (M Y,,Y) by marginalizing X' and Y'
Part Showing (MrcM,,XYY) Ap Ap
For showing that Q M, XY Ar, we first need to calculate QM
Calculating Q{Y]M): We use lemma 5 to calculate Q(Y[M) as from (95). we know that
QMc [M) = PYssY""a, sgn(a) a,nn() /lel,, M + d/c) is continuous stable distribution for
fixed m, and Y translated and scaled version of y" The exact expression of Q = m) is
provided in (97).
Qucc (YIc = m)) = pas(a, By, Ry. where
By sgn(c)sgn(cn
a+1
a=1 =)) = sgn(ac)8r,
(a)
lcl
a a
d
PV lcl
n
a=1 ,;
c(m + d)
a + 1
JA
a a= = cm + d
c(m + L + By by [g((c)// 2xgn(a)eBy log(lcl//
Que + = m) = pas(a, Bysgn(ac), 7V, cm + = PMY
(97)
From 81 and 197 we can conclude:
P(M)P((IM) @Mc(M, X)) (Mc (CMPXXX))M P(M,X),
(98)
Qmc = (Msc(M)QMc((*) (Y)M) = P(M)) = P(M,Y)
(99)
Hence, QMc Y) Ap and consequently, by proposition UI(M; Y( 0. concluding our proof for case
Case 2: lul/7x = lol/Tv
The proof of case 2 is extremely similar distribution (M, X, Y) without the need of specifying a larger Markov chain.
For case able directly construct the
Part 1: Specifying XM)(M,
Similarly to case 1, we specify (Mr X,Y) Y) by individually specifying QMc(M), QMc(XIM) and (M The distributions (M((M) and QMc(X (M) are as follows:
QMc(M) P(M) and (Mc = P(
(100)
33

---

The distribution (MI(( is defined using the following deterministic transformation:
9/oX - 2t/a + y= X + 3 log (14/al) <b/o d a=
a # #1
(101)
Part 2: Showing (M, (M
For showing that (M,, E Ap, we first need to calculate Quic Calculating Q(Y]M): We use lemma calculate Q(Y as (Mc (M]M-M)Mm is a continuous stable distribution for a fixed m. and translated and scaled version of X. The exact expression of Quc QMc(Y|M = m) is provided in (102).
QMc(Y = m. =pcs(p; a, a, p)
<a/am + bc + d
a aF1
ī ca/am + ct - be + + 20/an log (1%/a) 20/ox log (1%/l)
==1
cm + d,
7 |d/x//al al |c/1r/14 = yr, using the fact that 7x/|a| / by assumption)
3 Bsgn((/a) Bsgn(ac),
(Y |M m) pcs (a, syn -m -
(102)
From 110 and (102). we can conclude:
QMc X) QMc((M)QMc = P(M)P(XIM) = P(M,
(103)
QMc(M, Y) QMcc(M)QMc
PY (M) P(M,Y).
(104)
Hence, (M, X, Y) E A and consequently, by proposition UI((;; = 0, concluding proof for case 2 and theorem
G Proof of Theorem 2
In this section, we provide the proof of theorem 2] Since theorem provides the analytical PID terms for system of the multivariate affine continuous stable system defined in Sec. [ we briefly restate certain key properties the corresponding independent component multivariate stable distributions and the corresponding multivariate affine continuous stable system for convenience.
G.1 Independent component multivariate stable distribution
Independent component multivariate stable distributions are specific multivariate generalization of univariate continuous stable distributions describing a collection of independent random variables, with each random variable distributed according univariate continuous stable distribution. We refer the reader to Appx. N.3 for more details on multivariate continuous stable distributions. We now list certain key properties of independent component multivariate stable distributions that we make use of in the proof of theorem 2
1. If X is distributed according to independent component multivariate stable distribution, then the sum of two independent copies of X, denoted X, and X2, follows an independent component multivariate stable distribution up to scaling and translation operation, i.e., ax1 + 2 cx + d for a, b, c and d E R.
2. We denote the p.d.f. of the independent component multivariate stable continuous distribu- tion as Pcs- rc- (a., (3, 7, jū), where
3 = [3
3a] T with 3, E [[-1,1],7= In
Jaj" with 73 E (0, oo),
i [M pa] _ E IR. and (0,
In general, the f.d.f. of independent component multivariate stable distribution do not have closed-form analytical expression, and are expressed through their characteristic function.
34

---

3. The characteristic function of d-dimensional random vector X having independent compo- nent multivariate distribution PCs-ic(a, 3, 7, (i) is expressed as follows:
1 cxp Σ
/ (55g + ttt F
YEER (105)
where P(()) { tan na
2 (1)((1)
a
*
sgn(()
-1 t<0
0 1=
a
1 1>0
4. The random vector X is essentially a collection of independent random variables {X,} where X₃ ~
[
Xa)" having the distribution pcs- j))
(as B5. 7g. Pz).
G.2 Definition of the system of the multivariate affine continuous stable system
Let the random variable M P(M) with support M S R. X and Y are dx-dimensional and dy-dimmesiooal random vectors, respectively. The joint distribution P(M, X, Y) describes system of the multivariate affine continuous stable system if it satisfies the following two properties
M P(M) having some support set M C
2. The conditional distributions of random variables X and Y conditioned on M are expressed as follows:
X-Hx + Axīx A + +bx and Ý Hy M + Zy + +by,
(106)
where Zx pcs-ic(a, σax Lax, Zy P-c-- Day. Iav, Ody), Ax and Ay are invertible matrices, Hx, bx E Rdx, and Hr, by E Rdy
G.3 Formal proof of Theorem 2
Theorem 2. Let the joint distribution P(M,X X, Y) M, X, and Ý describe the system I of the multivariate affine continuous stable system. Without the loss of generality, assume ||Ay' S
||Axx Hxll where K = a/a-1V a E (1,2] and = do a E (0, 1]. Then, Ap contains a Markov chain of the form M- + X Y andUIMME Y X = O.
Proof. We first note that we can always assume | generality because if Ay Hy |lx H llk. then can always switch nomenclature to refer Y X, and X Y.
Hy llk S || [lx without the loss of
The proof of above theorem relies on the result of lemma 7 and the fact that the linear system described in (106) can always be reduced the special case of the linear system used lemma We briefly outline the proof structure.
1. In the first part, explicitly construct joint distribution QMc X Y) having the Markovian structure M + X + Y.
2. In the second part, we show that the QMr X Therefore, we can then apply the result of proposition to conclude UT(M; Y X) =
Y) constructed the first part, lies in Ap.
Part 1: Specifying (Mc(M, X. Y)
We explicitly construct the desired Markov chain M + X + Y by constructing a larger Markov
chain M + X X + Y + Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M X Y.
35

---

Denote the joint distribution of the Markov chain M 7 7 X + QMc X, X' Y', Y). We can decompose QMc X', Y', Y) by utilizing its Markovian structure as follows:
QMrc(M,X X, X Y Y QMc(MM)Qrr((IIM)))erc |
[X] MC MC Y X (Y))))
Consequently, we can specify/construct the distribution QMC (M, Y Y) by individually specifying QM) Qarc (XIM), (X QMcc(Y'X')) and QMc(YY We specify
Quc(M, X) = P(M, X) and Q(X*(X) using the following deterministic transformation:
A X bxx
(107)
Substituting (066) in the above equation:
X
A Hx AxZx + bix - -6xx = A Zx.
(108)
Specifying QMc (')" We define Y' according to the following affine system:
Y' = Ay Hy M Zy.
(109)
Observe that both X' and Y' can be described as an affine function of M satisfying the special
structure described in lemma 7 (see (115).. Furthermore, (M) = P(M) by construction. Consequently, (Mrc(M) satisfies the properties required by the distribution of M outlined in
lemma as P(M) in both lemma 7 and theorem follow the same properties. Since H S
||A Hx lle by the assumption in the theorem, we can apply the result of lemma
7 to construct
Qmel (YI)),, where Hx and Hy would be replaced by A x Hx and Ay * respectively.
Lastly, specify QMc(Y through the following linear system:
Y Y + +by.
(110)
Finally, we construct the desired QMC (M, X, Y) from @M, Y Y) by marginalizing X' and Y
Part Showing QMC (M, X, Y) E Ap
For showing that QM (M, X, Y) Ap, we first need derive QMc (Y MMM.
Y Deriving Q M)): Substituting 1009 in 110) obtain:
Y Ar Hy M Zr by = Hy Ay + +by.
(111)
Inspecting the above equation, can conclude:
( M.
(112)
From 112 we conclude:
(M, Y)
= =
(113)
We know that by construction:
QMc P)..X).
(114)
Hence, QMc (M, X, Y) E Ap and consequently, by proposition UI(M; Y X) = 0, concluding our proof.
Lemma 7. Let the joint distribution of M, X and V, denoted as P(M, Y), describe the system of the multivariate affine continuous stable system. Furthermore, assume Ax Lax Ay = Lav- bx 0ay and by Odys where L is a d identity matrix and 0g is d-dimensional vector of zeros. Then,
36

---

1. If Hylly S [H where x- = o/a-1 V E (1,2) and x V a E (0, then Ap
contains Markov chain of the form M + X Y and UI 0.
2. If || <
|H where k %/a-1 V E (1. and == 00 V E (0, 11. then Ap
contains Markov chain of the form M + Y X and UI 0.
Proof. We only provide an explicit proof of condition 1. Condition is essentially the same as condition with the parameters of and Y switched. Consequently, the proof of condition follows the same steps as the proof of condition 1, with parameters about X and Y switched.
Proof of condition 1:
We briefly outline the proof structure.
1. In the first part, we explicitly construct joint distribution QM, &, Y) having the Markovian structure + X Y.
2. In the second part, we show that the QMr X, Y) constructed the first part, lies A p. Therefore, we can then apply the result proposition to conclude UT(M; Y (X =
Part 1: Specifying (M, X, Y)
We explicitly construct the desired Markov chain M + X Y by constructing a larger Markov chain M + X + X' Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M X Y.
Denote the joint distribution of the Markov chain M + X
Y as QMc X', Y). We
decompose QMc((M, X, X', Y) by utilizing its Markovian structure follows:
QMc X Y) Me Mc MM) (x))mc((((x))
Consequently, we can specifylconstruet the distribution QMc M, X, X', Y) by individually specify- ing QM) QMc (x.c(MMM) .M. QMc((1 and Qmc (Y)x'.
Under the assumptions of lemma in (106) as follows:
7 we can simplify the linear system describing P(M, X, Y) shown
x X M + Zx and Y Hy M ZZy
(115)
Correspondingly, we can write the characteristic functions of conditional distributions P m) and P(Y M = m) for a fixed m by employing lemma
T c
M M= = cxp
Σ j=1
It a + it
H'
if M |==m]
exp
Σ 1=1
1 a + it
Hym
(116)
where the characteristic function can be derived by realizing that for given m. Hxm and Hym are constants being added to the random vectors Zx PCs-ic(a, 0g I Os.) and Zy
pcs-ic(a, Oz,, Id, ), respectively, and then using the result lemma
Specifying (Mc(M) and QMc(XIM) MM) We specify Qmc (M) and (XICM M) as follows:
QMc(M) = P(M) and Quc X M) P(XIM),
(117)
where P(M) and P(X|M) are marginal distributions derived from the original joint distribution P(M, X, Y) (discussed in the lemma statement) over which the bivariate PID is being calculated.
37

---

Specifying QMc [ X We specify Quec [X X)
through the following deterministic transforma-
tion:
X
H k )"
(118)
where H is as defined in lemma 10 and & E oo].
We will also derive the distribution ( before proceeding with our construction, will need it later in the proof show that the constructed QM, Y) E Ap.
we represent X as function M and Zx in similar manner to in (15), and then use the result of lemma to derive its conditional characteristic function Ele² X' [M], and correspondingly the desired (X/c(X))M)))
Deriving QM (X' M): To derive QMc(X')
We multiply Hy (H ym on both sides of equality in 115 to obtain the following linear equation:
H ) Hy HH (Hx M Zx) Hy Hy (H
" Hx Hy (H+ "
)
Substituting (18 in the above equation, we obtain:
Hy Hy (H Zx X = H
(119)
Using the result of lemma 10 i.e., Hy (H* H Hy, in the above equation we obtain:
X M + (H,)
(120)
From 120). can observe that X' is an affine function of Ex for fixed m. Hence, can employ the result of lemma to derive the distribution of Qasc (X| (XIM) (by deriving its corresponding characteristic function).
Note that the j-th column of Hy (fix)) can be expressed as h2 Hy, where hif is the j-th component of H Consequently:
dx
E
R
M
m
exp
+ + Hym
j=1
exp
MM
+ it
Hym
dx
exp
6ift
Σ
it
Hym
(121)
We substitute the value h² in [12 using the definition of H given in lemma 10 substitution into two cases:
We will divide the
Case 1: k finite, i.e., is E [0, 00). Then, we have h13 16.2 1*/ / lltth. which implies:
dx
h fka
E
x
M = m
=
cxp
H
+ * Hym
H x llitk
j=1
H
dx
hxx ka
+ it T Hym
exp
H
ka
litk
ka
Hif
Hx
exp
ka
+ it
Hym
(122)
aatka
1
38

---

Case 2: k infinite, i.e., k = oo. Then, we have hi = = j * j", and hj. 1/48, which implies:
cat" X'
M = m = exp
H
+ it Hym
H
exp
+ it Hym
(123)
H
Combining 122
and 123
we can re-write E
leit x
M = as:
x x
M = m = =exp (-f(k)
H
+ it Hym). where
c
f(b) {{
(
||Htx Itaa/
[x l
k finite
(124)
1/1
k: infinite
Specifying QMC (Y):)) We specify QMc (Y(x)) through the following stochastic transformation:
Y X +
(125)
where ē follows multivariate stable distribution. Furthermore, we assume that é jointly inde- pendent from (X,X, i.e., 1T follows:
X',
The characteristic function of ē is defined as
dy Σ 10 1(b)
it 22 c
= exp
HEE
(126)
In order for QMc QMa(Ý (XXx)) to be a valid distribution, need to ensure that the distribution of defined in (26)) is a legitimate multivariate stable distribution. We will utilize property for showing that é follows stable multivariate stable distribution. Property states random vector is distributed according to an independent component multivariate stable distribution iff every 1-dimensional projection of this random vector follows a univariate stable continuous distribution. Hence, show that e is distributed according to a legitimate multivariate stable distribution, we will show that every 1-dimensional projection of follows univariate stable continuous distribution.
Let E E Rdy, then the characteristic function of the 1-dimensional projection of é along E, i.e., E can be trivially deduced using (166.
dy f(k)
H3 a
exp
-
j=
(-
|
H )
exp
+
fkk
(127)
Comparing the characteristic function of L`' described in 127 with the standard characteristic
function of a univariate stable characteristic function defined in (33 we can conclude (T ~ ics(a,,((t,,) (f), (()), where:
(t)
1 (kk H (((t) 0, and B(6) = 0.
(128)
In order for ē` to have legitimate univariate stable distribution, we just need to show that (t) > 0 V E Rdy, Note that yt)) = 0 would correspond to the case where all the mass of the distribution is centered at O, which does satisfy definition 2 and is an example of degenerate univariate stable distribution. Let us analyze the function y(t):
(t) = E
[(k) HTp
(129)
39

---

Applying lemma on the term E we obtain:
| H 1ly
a (1,
(130)
|E
Hy 1
a E (0,
Applying (130) in 129
we obtain:
Hy l
a (1,2)
(t))
[ la
[ a [0
(131)
Denote ((t, a) as follows:
| f(+) Hl B a (1,
((T,a)=
(132)
|t (
a (0,
In order to show (t) 2 OYEE Re it suffices show that ((t 2 L R and a E (0, 2).
Case I: a E (1,
For 21, we have:
(tt, a) 6
f(k)
Hy
[E
f(k)
Hy (byl1//--)
1/a-1)
(133)
%/4-1
Since, the above equation holds for any k E [0, oo]. we choose i = 1/(a-1). Substituting the form f(k) from [24 in the above equation, obtain:
%/-1
%a-1
Hy
x
(F.
E
0/--1
1//-1
0/a-1
a
1 -+))
-=
H x
1
%/4-1
n-1
Hy
H y
%
t
(134)
H x
H
1u/a-
a/a-1
Note that
2 0, and by condition we know that 11///11/-111
Hy
11%
[ 11 o/- Hence, using these two previous facts and the above equation, we can conclude:
(( t
a/a
7 ((t,a) 2 O.
(135)
H
20
w//-1
20
Case 2: a E (0, 1].
((ī,
- ((a) -(0)))))))
= a)
|
H 1 (a) ( | s(a))
|
H
(136)
where (a) is derived using the monotonicity of Lp norms in p. see [57
More precisely, we use the inequality ||E||a= 2 ||t|] V (0, 1]. Simplifying the above equation, we have:
([,, a) > |=|_00 (1 /kk) Hr 1
(137)
40

---

Since, the above equation holds for any & o0], we choose k co. Substituting the form fkk from 124 the above equation, we obtain:
I(t, (x) 2 |=l
Hy 2 H x
(138)
Note that { 2 O, and by condition we know that ||Y o0 I |xxl
|Hy I Hx ll
Hence, using these two facts and the above equation, we can conclude:
Hy Y
T(t, 0)) 2
= a(tt,a) 2 O.
(139)
H x
From (135) and (139). we can conclude that I(C, a)) 2 L E R. and a E (0, 2]. which implies r(t) 2 2 gdy Hence, é follows a legitimate multivariate stable distribution.
X' Y) Finally, we construct the desired QM X, Y) from QMc X'X
by marginalizing X'.
Part 2: Showing QMC (M, X, Y) E Ap
For showing that Qe (M, X, Y) Ap, we first need derive Q \lM..
Deriving Q M): We will derive QMO Y [M) by using the fact that é LL X M, hence their conditional characteristic functions would just result in multiplication.
E
e
M
mn = E 1t
M
m
= E
X
+i
M = m
=E 1
CIE² M m
E
x'
M
m
E
M m
(140)
where (a) due 1 X'M Substituting the characteristic functions of X' and from and (166, respectively, in (40):
124
dy
exp
Σ j=1
Ity l + f(k)
H
exp
(-f(() Hyf + +it Hy"")
dy
exp
|4,1 a + s(k))
Afft -(k))
HGT + + Ht'
j=l
dy
exp
|t a + it
Hym
(141)
By comparing the characteristic function shown in 141 with the characteristic function derived using P(Y|M) shown in (16),, we can conclude:
QMc (occ PYY
(142)
From 142 and 117 we can conclude:
(Mc Y Quc(M))Mc P(MPP(YMM) PPMM)
(143)
We know that by construction:
(Mc(M,X) X).
(144)
Hence, QMC (M, X,Y) proof.
Y) E Ap and consequently, by proposition
UT(M; Y X) = 0, concluding
41

---

Lemma 8. Lerä= [a
aal E Rd and = [b
ba] E R then we have:
E (1,2) (0,
b
(145)
where 1l is the standard Lp norm.
Proof. Case I: a E (1,2)
Analysing the LHS of (145)
1
(a)
(b)
a:ba
Labbil
Σ
lai!"
Σ
bb: 1y
[b
(146)
where (a) is due to the sub-additivity of [-)] operator and (b) is due to Hölder's inequality [57] Raising both sides of the inequality to in (146) gives the desired inequality. Since, >> 0 raising both sides of the inequalities to power a does not change the direction of inequality.
S/4
(147)
Case 2: E (0.
Analysing the LHS of (45)
(a)
(B)
a'
a
lasbel
Σ
max iE(l.....d)
[b:|
|a]| [b]lss
(148)
where (a) is again due to the sub-additivity of ((-) operator, and (b) is due to the fact that la,byl < layl max (I.....d) |b, v E {l....,dl d). Raising both sides of the inequality to a in (148) gives us the desired inequality. Since, > 0 raising both sides of the inequalities to power does not change
the direction of incquality.
TT
a
&
(149)
Combining (147) and 149). we obtain the desired inequality.
Lemma 9. Let Z pcs-ic(a, 0a, La. Oa) be d-dimensional random vector, A m.d paxd be x d matrix, and b Rd. Then, the characteristic funrrrcttristiccfniiiinn of AZ + is given by (50)
|a,]",;]
E B 57 (AZ (AZ-6)
6)
exp Σ
a
+ +
v VOERS Rd,
(150)
where ä; is the i-th column of A.
Proof. Analyzing L.H.S of 150 (AZ+6)
A2+
A2
AZ 1
E
EE
(151)
Using the fact that est B
is constant expectation over Z, and linearity of expectation, have:
E e (A2+6)
= = g
exp
(E AZ
= =
exp
(ET
[a]
aul 2
= -
&Tan] 2)]
exp
42

---

Simplifying the term (
t" ad) Z by substituting Z = [2z Zal²
we obtain:
exp
i
(t 2g
Using properties of exponentials, we obtain:
E
II exp
it )
Using the fact {2,} are jointly independent we obtain:
= b
II j=1
it a 2,)]
exp
Using the fact that Zj pas(a, O, 1, 0) and the result of lemma 5
we know that (" 2g ~
Pcs(a, 0, jt"ājl 0). Substituting the corresponding characteristic function of t 2 using (323)
d
( E" (.la)) = t exp Σ
[
exp
a
j=1
exp
iT a
+ b
(152)
Lemma 10. Let Hi
ndd
[1]
Rd and Hg [1(3]
hds
(2)
Rd Define
H as follows:
sgn(h (onhh"))n('p sgn(h (1
A 10,
Furthermore, define H as follows:
j arg max
jE((....d1)
H ph
with
hith h3 =00- E (1....,,d,M. and hg
(153)
Then, we have:
H H2 k [0, oo) and Hg -H
(154)
where [1 is the standard L1+k norm (57). We can alternatively state [54) in a more compact notation:
H H. vk [0, where the case of k oo is to be understood as H HP.
(155)
Proof. Case 1: k E oo), i.e., & is finite,
h (1)
H
H
(sgn(h()))n))))
1+
Σ 1=1
|411 1+k
H, = H2.
litt
H
litt
43

---

Case 2: k = 00,
d H2 H1 = H.
(
@ H (ngen
1 @ H. H
h his
H2
where (a) is due to the fact that all elements of H are except the j" element, and (b) is due to the fact that hef 3/hg0.
H Proof of Theorem
In this section, we provide the proof of theorem 3 Since theorem B provides the analytical PID
terms for the system 2 of the multivariate affine continuous stable system defined in Sec. E we briefly restate certain key properties of the corresponding elliptically-contone multivariate stable distributions and the corresponding multivariate affine continuous stable system for convenience.
H.1 Elliptically-contturre multivariate stable distribution
Illiptically-contturee multivariate stable distributions another example of multivariate general- ization univariate continuous stable distributions. As the name suggests, the defining features of these distributions is that the corresponding p.d.f. has elliptical contours, similar to the multivariate Gaussian distribution. We refer the reader to Appx. NN.3 for more details on multivariate continuous stable distributions. We now list certain key properties of elliptically-contturee multivariate stable distributions that we make use of in the proof of theorem 3
1. We denote the p.d.f. of the cliptically-contourre continuous stable distribution as (a, Σ, (i), where is a positive definite matrix, f E Rd, and a E (0, 2]. In general, the p.d.f. of closed-form analytical expression, and are expressed through their characteristic function.
multivariate stable distribution do not have
2. The characteristic function of d-dimensional random vector X having elliptically- contoured multivariate distribution Pcs Ea(a, E, (i) is expressed as follows:
(- ( (5"TT)) Σt 8/2 + it ;i"
IF
it
x
exp
V VEERd.
(156)
H.2 Definition of the system 2 of the multivariate affine continuous stable system
Let the random variable M P(M) with support M C R. X and Y dx-dimensional and dy-dimensional random vectors, respectively. The joint distribution P(M, X, Y) describes the second system of the multivariate affine continuous stable distribution if it satisfies the following two properties:
M P(M) having some support set M C R.
2. The conditional distributions of X and Y are expressed as follows:
P(X pcs- Ex. H M + bx), and
( = paa, Ey, Hy + by),
(157)
where Σx and Ey positive definite matrices, Hy, bx Rdx, and Hy, By Rdy
H.3 Formal proof of theorem 3
Theorem 3. Let the joint distribution P(M,X. Y) of M, X, and y describe the system 2 of the multivariate affine continuous stable system. Define Σx 1/2 and Σx the respective inverses of the matrices E and Σ which satisfy: (Σ o Ex. and (2 Ex =Ey. Without the loss of generality, assume ||E 3/H ||2 <
| 1/2 Hx l2. Then, Ap contains a Markov chain of the
form M X and UI(M Y = 0.
44

---

Proof. We first note two important observations:
1. We can always assume if y " Hy 2 12x Hxx 112. then we can always switch our nomenclature refer to Y as X, and as Y.
EE Hrll2 < | |
without the loss generality because
1/2
2. Properties of positive definite matrices guarantee the existence of an invertible Σ and
Σ 58
The proof of this theorem is generalization of the proof used for the deriving similar result for the special case of the multivariate Gaussian distribution discussed in [22], which borrows known results from Gaussian interference channels discussed in [48). This proof follows the same structure as carlier proofs.
1. In the first part, we explicitly construct joint distribution QMc X, Y) having the Markovian structure M x Y
2. In the second part, we show that the QMr(M, X, Y) constructed the first part, lies Ap. Therefore, we can then apply the result of proposition to conclude UT(M; Y X) =
Part 1: Specifying QMc M, X, Y)
We explicitly construct the desired Markov chain M + + Y by constructing larger Markov chain M + X x + X" Y - Y, and then marginalizing the larger Markov chain obtain the desired Markov chain M X +Y
Denote the joint distribution of the Markov chain M - x Y as (M, X, X', X", Y Y). We can decompose QMc
X' (M, XM, X' X,
+
Y Y) by utiliz- (A ( (M)
ing its Markovian structure follows: QMc(M, X. Y', Y) Que (X([) QMc(X X"IX")Qee Y" """QMo (YIY") Consequently, we can specify/construct the distribution QMO M, X, X', X Y Y by individually specifying QMc(MM, Qmc (Mc(((M) (XIM),
Que X x)) ( IX',, QMC (Y () and QY'c(Ý(('))
Specifying QMc(M) and QMc M): We choose QMc(M) and (Mc(A as follows:
QMc(M) P(M) and QMc
M) (
P(X
(158)
Specifying QMC (X (X'Ix): (x((X specified through the following deterministic transforma- tion:
X <x> bx).
(159)
We will also derive the distribution Q(**IM)) before proceeding with our construction, will need it later in the proof show that the constructed QMo (M, X, Y) E Ap.
Deriving QMc M): We use lemma to derive QMC X IM m). as Que X|M m) an elliptically contoured multivariate stable distribution for fixed and (159) defines scaling and
45

---

translation operation.
Quc(X M = m) = PCS-EC (a, [xx 1// Ex
(xmx) V2 ((x -xx)),
QMc I = m) = Pcs-
(a, 1
= QMc(X'Ia = m.) = PCs- EC
(,,
(xX 1/2) H Hxm)
+ QMc(X'IA ('MM m) = pcs
(a. xx /2) xx (Ex 7 (Txm)
= QMc X m) = PCS-EC EC
(a,Idx. (Ex 1/3 (H)m)
= QMc (X =m) = PPs--E EC
Jx Kx Rxm)
(160)
where Lax is a dx x dx identity matrix, and Kx = 1xx x
Specifying QMC (xc(X"(X) Qasc(X (Xsc(X"1Xx) (X²) is again specified through a deterministic transformation:
X" Ky '
(161)
where Ky (Ey* and K2 1/1 Kx lb x We also derive the distribution Q(X'(M)
which will be needed for showing that the constructed Qxc X. Y) E Ap.
Deriving QMc(X"IM = m): We again use lemma 11 to calculate QMc(X"IM = m), as QMc(X' [X|M m) is elliptically contoured multivariate stable distribution (see (600) for fixed m. and [199] defines a scaling and translation operation.
Quc X M m) PCS-EC (a. (Ky (K)) Lax (K
(K3) (KX))
PCS-EC (a, (K (KY()) Ky (K) (Kmm)
From lemma 10 we know that Ky (" = Ky, which implies
(
X' M = m) = PCS-EC (a, (Kx(( (T) (.)T))KKkm)))) Kym).
(162)
Specifying (YM We specify Qmc(Y X") through the following stochastic transforma-
tion:
(163)
where é follows multivariate stable distribution. Furthermore, we assume that é is jointly indepen- dent from (X, X
X", M), i.e., 7 1 (X, X" M). The characteristic function of ? is defined as
follows:
cxp (- (E"IAT 2 (F" (("R
(164)
In order for QMC QMc(Y' X") to be a valid distribution, we need to ensure that the distribution of غ defined in (164) is a legitimate multivariate stable distribution. Similarly to the proof of theorem [ we will utilize property for showing that é follows stable multivariate stable distribution. Property [] states a random vector distributed according to multivariate stable distribution iff every dimensional projection of this random vector follows a univariate stable continuous distribution. Hence, to show that is distributed according to legitimate multivariate stable distribution, we will show that every 1-dimensional projection of é follows univariate stable continuous distribution.
Let E Rdy, then the characteristic function of the 1-dimensional projection of é along É, i.e. i'e can be trivially derived from (164).
E
e T
-exp (-("Tla,t) 4/2 + (ERRR((E)))))R 2/2
(165)
46

---

Comparing the characteristic function of t"Tē described in (27) with the standard characteristic
function of a univariate stable characteristic function defined in 323 we can conclude t' ~ pas(a, ((t), (()), u(t)), where:
(C) (i"Id, 4/2 - (R
K (2))) a/2 (u(() = 0, and B(t) (166)
In order for E' to have a legitimate univariate stable distribution, we just need to show that (t) 2 A t E Rdy, Note that y()) = 0 would correspond to the case where all the mass of and is example of degenerate
the distribution is centered O, which does satisfy definition 2 univariate stable distribution.
By the assumption in theorem, use the result of lemma 12 which shows that (Ky( (K2))) = Iay. Using the fact that (Lay - (Kx x
|| |
Kxl2 l2
Hy
| Hx 2l2. Hence, can
("))) is a positive semi-definite matrix, we have:
1r Jav - (Ky
F2 (R
I' E2 Es (ky
Ii'Iar t) 4/2 2 E (
VEE Rdr
-) (a " E (Ky (Ky (K) ()) 4/2
OVEER (167)
Hence, we know that Z follows legitimate multivariate stable distribution. We also derive Qmc(Y'M which is needed for showing QMr Y) E
Deriving QMc Y M): We will derive QMM(YY|[M)) using the fact that ē -T X m, hence their conditional characteristic functions would just result multiplication.
M
m
E "
(X"+2)
M
m
EE"
8"
M m
= E
it " X" T M = m
(a)
M
m
T M
m
(168)
where (a) is due ē 11. I Substituting the characteristic functions of X" and é from (162) and (644). respectively, in (68)
4/2 + +
(ET (K [
cxp
IE E) 9/2 +
(K)) (2)))) 9/2)
ET [ky
exp
4/2 - (F"Idr)
+ Rr"Kkym)
= exp
(169)
By inspecting the characteristic function shown in 169,. we can conclude:
QMc (Y'I -m) pcs pos-sc(a, Iay, K kym).
(170)
Specifying Q Ý': Lastly, we specify QMO Y through the following deterministic trans-
formation:
== - by.
(171)
Finally, construct the desired QMC (,, X, Y) from QMc X X" V Y) by marginaliz- ing X X" and Y
CMM X, Ý) Ap Part Showing QMC (M,
For showing that QMc (M,X, X, Y) Ap, we first need derive QMC (Y (M).
47

---

Deriving Q( We use lemma 11 to calculate Quc M = m), as Que (XIM = m) is an
M):
elliptically contoured multivariate continuous stable distribution for fixed m (see (77)). and defines a scaling and translation operation.
Qucc = m) = PCS-EC
(a. (Zy 1/2
" by)
(a)
Kym +Bx) a; Ev
PCS-EC
(
(a, Ey, (E) / )r Ey 13 Hym + +by)
PCS-EC
(172)
PCS-EC
a, Σy, Hym + by
(173)
where (a) is due to the fact that
Ey, and (b) is due to the fact that Ky = Ey
Comparing (173) with P(Y|M m) shown in Appx. H.2 we can conclude:
Quc Y M) M)
(174)
From 174 and 158 we can conclude:
@Mc(M, Y)
= P(M) = P(M, Y).
(175)
We know that by construction:
QMc(M, X).
(176)
Hence, QMC (M, X, Y) E Ap and consequently, by proposition i UI(M; Y (X) 0,, concluding our proof.
Lemma 11. Let X PCS- PCS-Ec(a, E, p.) be d-dimensional random vector, where E is a d d positive definite matrix, ç Rd, and a E (0,2). Furthermore, let A Rxx and b Rd. Then.
AX b pcs- ( pcs AEAT Aū b).
Proof. Calculating the characteristic function of AX + b:
E exp i (AX +))] (AX+b))) = E exp TE AX
it 6) = E [oxp (i (
exp if 6
Employing linearity of expectation to pull the term exp (it"b) out of the expectation, as it is constant with respect to X:
[ (AX + 6))] exp (it (1E"5) E exp (t AX)]
E
exp
(177)
Substituting
E = F A in 177
E exp ( + b]]] exp (i(Tb) exp (1(())))
Substituting the formula the characteristic function an Elliptically multivariate stable distribution (given in (156)) for calculating
exp (i())) Tx) as X ~ Pcs- E, pi):
E (F (AXX +b)] = exp ( exp (- (-)()) + +i(t')²
Re-substituting i' 2=ETA in the above equation
(it'b) exp (-
[
( (-(TRAEA')) 9/2 +HEAAR)
= =
(FTAEA")) + (A
exp
(178)
48

---

By inspecting the characteristic function described in (78). we can determine that AX + b is distributed according to an elliptically-connouree multivariate stable distribution, i.c. AX PCS-EC (. Aj+
Lemma 12. Let H1 E Rd and H2 E Rdd, Then, ||H 2 H2lle if and only if 3 a matrix A E Rddxdd such that H2 = AH, and AAT 3 Idg, with A = H2 (H))² where B 3 C means
B C positive semi-definite matrix, H as defined in lemma 10 and dd is d2 d2 identity matrix.
Proof. See proof lemma 5 in for special case / 1.
I Proof of Theorem
In this section, we provide the proof of theorem 4
Since theorem 4 provides the analytical PID
terms for the univariate affine discrete stable system defined in Sec. we briefly restate certain key properties of the univariate discrete stable distributions and the corresponding univariate affine discrete stable system for convenience.
I.1 Univariate discrete stable distribution
Univariate discrete stable distributions the discrete analogues of univariate continuous stable dis- tributions. We refer the reader to Appx. We now list certain key properties of univariate continuous stable distributions that make of in the proof of theorem
1.4x..N.4Jor [or more details on univariate discrete stable distributions.
1. If X is distributed according to univariate discrete stable distribution, then X _ y X1 (1 - ~- V o X2, where X1 and X2 are two independent copies of X, (0, 1], n E [0, and denotes the binomial thinning operation.
2. The p.m.f. of discrete stable distributions are characterized by two parameters: exponent v E 1]. and rate parameter т E (0, o0). We denote the p.m.f. of a univariate discrete (D, 7). In general, discrete stable distributions do not have "nice"
stable distribution analytical form consisting of well-known elementary functions. Consequently, the univariate discrete stable distribution typically expressed through their probability generating function.
3. The probability generating function of a discrete random variable having a stable discrete distribution is given by (179):
P2(2) (opp-T(1-=2)))
(179)
1.2 Definition of univariate affine discrete stable system
Let M,X, and Y be a system of random variables with the joint distribution P(M, X,Y). The joint distribution P(M, X, Y) describes the univariate affine discrete stable system if it satisfies the following two properties:
1. M P(M) having some support set M C o2).
2. The conditional distributions of random variables X and Y conditioned on M can be expressed through discrete stable family distributions with an affine dependence on M. Formally, P(X M
am + b) and P(Y]M=m) Ppspps( cm + d), where
a, b, d 00).
I.3 Formal proof of Theorem
Theorem 4. Let M, X, and Y be random variables whose joint distribution P(M, X. X.Y) describes a univariate affine discrete stable system. Without the loss of generality, assume c. 1fa/b 2 s/a, then Ap contains a Markov chain of the form M + X - Y and UI(M;Y 0.
49

---

Proof. We first note that we can always assume a 2 without the loss of generality because if a S then we can always switch our nomenclature refer Y as X, and X Y. We briefty outline the proof structure.
1. In the first part, we explicitly construct joint distribution Markovian structure M + X
having the
Y) constructed in the first part lies in Ar. 2. In the second part, we show that the QMc X, X, Therefore, we can then apply the result of proposition | to conclude UT (T;M; = 0.
Part 1: Specifying QMc(M,
We explicitly construct the desired Markov chain M + X + Y by constructing a larger Markov chain M
X' + Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M + X Y.
Denote the joint distribution of the Markov chain M + X X' Y as QMc X, X', YY. We can decompose QM X, X', by utilizing its Markovian structure as follows:
QMc X. XX, QucetaTPO.cCCAAMMMQCCCCXXXXAAAC
Consequently, we can specify/constrrutt the distribution QMc(M, x, XY,Y by individually specify- ing QMc(M), QMc(X|M), (X')x)) and (Ylc(
Specifying (M) and (M We choose (Mc(M) and QMc(X as follows:
QMc(M)= P(M) and QMc = P(X|M),
(180)
where P(M) and P(X[M) are marginal distributions derived from the original joint distribution P(M,X; Y) (discussed in the theorem statement) over which the bivariate PID is being calculated.
Specifying QMc Qxc(X'IX) is specified through a binomial thinning operation, specifi- cally:
X' = X<(aaa))
(181)
Note that the above binomial thinning operation valid as c S a by the assumption in the theorem statement. We also derive QMc(X') M) it is needed for showing QMc(M,
Ap.
Deriving (Mc We use lemma 13 to derive (X'[ m) as QMc(X = m) describes univariate discrete stable distribution for fixed and X' is binomially-thinned version of X. The exact expression of QMc( = m) provided (82).
OMc(X'I = m) PDs(U, (*/a) ((6))((an (am + t/a)),
(Ds em <//).
(182)
Specifying QYc(Y OMc(Y(X)) is specified through the following stochastic transformation:
Y X
(183)
where Pos (u,d - c/aa. For (MC(" to be a valid distribution, we need to ensure that distributed according to a legitimate discrete stable distribution. order to show e distributed according to legitimate discrete stable distribution, need to ensure that v and d - <b/a lie within their appropriate bounds as specified Sec. It is trivial to see y E (0, 1].
From the assumptions in the theorem statement, we know that
bc
0.
(184)
Hence, e follows legitimate discrete stable distribution. Furthermore, we choose e (M,X X'). Finally, we construct the desired (M,, from
by marginalizing X'.
Part 2: Showing @Mc X,Y) Y) E Ap
For showing that (M, Ap, first need to derive QMC (MC(Y)M),
50

---

Deriving Q(Y]M): Note that e follows discrete stable distribution Pbs(v, d - be/a) and QMel X' M m) is also discrete stable distribution Pos(X'; ; be/a) for fixed m (see 182) Furthermore, IL M Hence, using lemma 14 we have
QMc = m) = PDS(U, em + be/a - be/o + d) = PDs(U, cm + d).
(185)
As P(Y = (DS(U, cM + d) (see Appx. 1.2 we have:
Q = P(YMM).
(186)
From 180 and 186). we can conclude:
QMc x QM) (X P(M) P(M,X),
(187)
Que M, M) = (M = P(M) = P(M,Y).
(188)
Hence, QMC (M,cM, E Ap and consequently, by proposition our proof.
UI(M;; = O, concluding
Lemma 13, Let N ~ PDs(v,T), Then, N1 = y o ~ Pos(u, Vy E [0. 1]. where the binomial thinning operation.
Proof. The probability generating function of N, denoted as PN (2), can be described using (179):
(2) exp (-r(1 =)").
(189)
For calculating the probability generating function of N1. we use the following relationship between the probability generating function of a random variable R and its Ainomially-thimee version R o with p [0, 1]:
P PRop (2) (1 p(1 2)) (see equation 3.3
24
(190)
From 189 and (99).. we can calculate the probability generating function of Nı as follows:
PN, (z) P (2) = Px 711 - =)) = exp (-r (1 7(1 (11=))))²)
exp (-T (7 =))" = exp (-7>" (1-z)").
(191)
Inspecting the probability generating function of N1 in 191). we can conclude that Ni PDs(U, T"T).
Lemma 14. Let Nı PDs(U,T), N2 Pos(U, 12), and N₁ II N2. Then, N1 N₂ N PDS(n; U, T2).
Proof. The probability generating function of N₁ and N2, denoted as PN, (z) and PN,, (2), can be described using (19):
(2) exp (- =)"" and PN,(2), = exp(-12(1 - =)
(192)
Calculating the probability generating function of N N₁ N2.
(a) P (2)lP
(b) exp (1 =) exp (-T2 =2)"),
(=) PN, + No
exp (- + T2) (1 ))"
(193)
where (a) is due to the fact that N₁ IL N2, and (b) is achieved by substituting the explicit forms of FN, (z) and PN, (z) from (922). Inspecting the probability generating function of N, + N in (193)
we can conclude that N1 N₂ N Pns(nu; 72).
Proof of Theorem
In this section, we provide the proof of theorem 5 Since theorem provides the analytical PID terms for the multivariate linear Poisson system defined in Sec. we briefly restate certain key properties of multivariate Poisson distributions and the multivariate linear Poisson system for convenience.
51

---

J.1 Multivariate Poisson Distribution
We use the multivariate Poisson distribution proposed in 42). For more details, see Appx. N.5 We list certain key properties of multivariate Poisson distributions that we make use of in the proof of theorem
1. Under the definition of the multivariate Poisson distribution proposed in [42]. each compo- nent of the random vector is sum of independent Poisson random, i.e.,
Ñ AÑ
(194)
where A is an appropriate matrix of O's and I's. We can decompose A = [A where A, is d x (9) submatrix having duplicate columns, and each of its columns containing exactly i ones and (d - i) zeros N (d-1)d-... (d with Ni.... ~ Poisson(Ai, A (i1,..., E Ag. E [d'].
Ar).
and N [NY No N
Furthermore, the random variables {Ni,...,, Na- (d' 1)...a) mutually independent.
2. We denote the p.m.f. of the multivariate Poisson distribution as Poisson(d, d', A), where d 2 d', and
= [J Ad A12 ........... (d-
(195)
3. For d' 1, we have that Ñ is collection of independent Poisson random variables, and when both d = d' 1, we recover the scalar Poisson distribution: Pr(N = n)
-n Vn E No.
n!
4. Let Ñ Poisson(d, d', A), where Ñ is d-dimensional random vector. Then the corre- sponding p.m.f. of Ñ is described below:
Let n'
[112 ... (id 1)d ... Rd-(dd' (d' -1 1)...] "" and da be the dimension of n', then:
d
P(Ñ=m)=e
9 Σ
II
x
Ass Aiz
E'C
(11,)2)EAg
x Q((
(196)
where C = [í' E Noaa (a (F) < n4 Vi [d]] and
Q (n, a jm2 (s)))) "
with X being the i-th row of the matrix A' [A ....... Ar
J.2 Definition of the multivariate linear Poisson system
Let the random variable M P(M). X and Y are dx di-deennsoo and dimeennnsoo random vectors, respectively. The joint distribution P(M, X, Y) describes the multivariate linear Poisson system if it satisfies the following two properties:
MM P(M) having some support set M S (0,
2. The conditional distributions of random vectors X and Y are multivariate Poisson distributions, i.c., P( [M) -Possonndxx dxx, Ax) and P( M Poissonndy ddy,, dyy, Ay), with:
Äx =
xx xxx -dxx - 1)...x
T
M VVj E [dx] and (i,,.., E Agx, i = = Win.,
Ay by Ady (d'y 1)...y
T
Naniee = x my M V [dy] and (i)) AdY.
N
Here, 7 ..... nā (d'x 1xx..xx .... Tay -dyy 1y...y E Rt.
y
52

---

Furthermore, let Ax, and Ay be the corresponding A A-matrices (defined in (944) associated with P(X[M) and P(Y M), respectively.
J.3 Formal proof of Theorem 5
Theorem 5.
Let the joint distribution P(M, X, Y) of M, X, and Y describe the multivariate linear
Poisson system. Without the loss of generality, assume d'x 2 dy. If I () dx
7 Ki...dy 2
(i EA by Hy... Vj E [dyy], then Ap contains Markov chain of the form M X - Y and UTM Y x) = 0.
Proof. We first note that we can always assume d'x 2 d without the loss of generality because d'x S d'y, then we can always switch our nomenclature to refer to Y as X, and X as Y.
We provide an explicit construction of the Markov chain M + X
Y
+ Y having X
the marginals P(M, X), and P(M, Y). Denote the joint density of M + X
Y Y
QMM,X, X9, Y Y). For ((M) and Q(XIM) we choose them be equal P(M) and
P(X[M) M), i.c. Q(M) = P(M), and Q(XIM) P() Note that due to this construction,
P(M, X) = (M, X) holds trivially.
For [(x9(X), we use the result described in lemma 17 Note that the construction for Q( X) is not explicit but rather implicit. Let dx be the dimension of Åx, then we explic- itly choose: Q( .M) = 6x(X- = AxX²), and Q(Yss)M) xx 1,,11, Note that Q{ Pois-oons d'x- Ax) by construction, and we derive Q(X² X, M) through the Bayes theorem. Here, 6x() (.) is the Kronecker delta function (59). By lemma 17 we know that Q(X9 X, M) = 9 9 X), and hence we have the Markov chain + X
For choosing Q (YYa) x²), we rely on the result of lemma 18 Let us define the random vectors:
x = xg X 1
T
(6 ()44 and Y = Yg ( (5)44
iee. X, and Y, are random vectors containing terms of the form X "y....iy" and y ky""..yy" where (i,, .............. 1n) E A and (ki,..., ki) E Age. respectively. Note that we can write X X 8T.]= and we define Y YT YT Then, we construct Q(Y: X9) as product of dy multinomial distributions, described below:
dy II (Q 1=1
where Q(Y,-V,1X,- -xx) = Multinomial(k; (ky; N₁, P.). and
Nj I'x,, kj yイ
ITx [T
p; pp... ... Pds- 1)...dd
1 Σ (i (5)...))) Pis...y)
V (f.,...,iy)) i;) E
Pii..dy
By construction X consists of mutually conditionally independent Poisson random variables (condi- tioned on M), and the condition of lemma 18 (specified in (219)) is satisfied by condition 1. Therefore, using the result of lemma 18 on the Markov chain M X - Ys (obtained by marginalizing X), conclude that Q(Y MM) Poisson(1, dx,- Ay), where dxy is the dimension of Ay.
We specify Q 99) through the following deterministic transformation
Ay Y to obtain the Markov chain M X X Y +Y. Marginalizing X and X in the above Markov chain,
53

---

we get the following Markov chain: M- + Y + Y. Now, since Y = Ay Y and P(Y*) = x 1, A, P(YIM) Poisson(dyy dy, dyy,A) Ay) (by definition of the multivariate Poisson discussed in Sec. J.1 Since we have ((M) = P(M) and Q(Y[M) P( M). we also have Q(M, Y) P(M, Y). Marginalizing X and Ys from the Markov chain M X X
+ Y results the desired Markov chain - Y having Q(M, X) (MM, X) and Q(M, Y)= PPM, Y), concluding our proof.
Lemma 15. If N ~ Poisson(>) vj E (I....,d) and all N₃ 's are mutually independent, then P(N N2 N N₂ N) (N....... Pa/Ea A3
>a/E Arl).
Proof. See Section 3 of 660
Lemma 16. Let Ñ = N
...N where N₂ ~ Poisson(>) (2 & E fl....,dib di} and all Nj 's are mutually independent from each other. Define P P(N Í
Multinomial NK: where
d1
N =
k =
[N(2)
N(2) de
2j=1
Σイ= de N₂)
j=1
(9)
Σラ 1
p
2
where {1,²) (2) do jj=1 are some positive constants, i.e. 1,9) E (0, oc). such that Di= X3 S ESn A.P
Then, Ñ₂ 2222)
de N20)
where No2)) Poixson(>{2)) ((2),, and all N((2 are mutually indepen-
dent form each other.
Proof. Lemma 16 is straight-forward generalization of lemma 15 Denote the probability of N₂ shown in the lemma statement P(N2). We will prove the above lemma by constructing Markov chain Ñ; Ni + N - N₂ with joint density Q(N,, Ni, N/2 N2). and showing that Q(N₂) = P(N)) and ((Nz(N PN (N.).
Specifying Q(N I We specify ((N through the following deterministic transformation:
Nº ITN Σ NSO
(197)
j=1
Calculating Q(N)): We use the property that the sum of independent Poisson random variables is Poisson random variable with its rate parameter being the sum of the rate parameters of its summands [6]. Hence, Q(N{) Poisson( ( (1)).
Specifying N(): We specify Q( nij) as follows:
Q(N n11 Binomial (Ng: Eie 250)
(198)
Calculating Q(Ng): We use lemma 15 to calculate Q(N)). Note that Q(N))
Poisson Σ/e ビØ=1
((1)
and (( Binomial (N/: n4, E/e //9/ /Σ/,,, x(r) Hence, by lemma 15
we have:
Σ s= 1
de
(N = Poisson
X
Σ di
X (1)
Poisson
X (2)
(199)
54

---

Specifying N1): We specify Q(N2|N2 = n²) as follows:
Q(N2|N2 n24)
Multinomial
N2: n22.
X
(200)
13
Calculating Q(N₂): We again use lemma
to calculate Q(N2). Note that.
d2
Q(N)) = Poisson
and
(4
1)
Q(N2|Ng = n(2) = Multinomial
(201)
191 (3)
Hence, by lemma 15 we have:
2 N
N:22] d2
where .v... Poisson (2(²) VJ
[dz]
(202)
Hence, by 202 we know that Q(N2) = P(Ñ2).
Calculating Q( I;
QN = nl Ñ = -1) = Σ
Q(N = n11 N32 = n³g N₂ = n2 N; = =ū,),
ni m.mn== =0
80
((z (4 1n2).
where (a) is due to the Markovian nature of QN Ny, N₂2 N2) ). We drop the random variable nota- tion for brevity. Let ū = (1(") the exact form of (nn (n))))(())))))) (m)) and Q ( (n2) in the above equation:
(1)
and π. n
(2)
Then, substituting
di
da
EE= ny-nn
8
d
(n)))
I
n
Σ
(ni mnng!!
1
Σラ=1 103)
x,
m
=0
/=1
(2)
(2)
(nb)
1
(2)
122
(2)
d2
(2)
where [-|] is an indicator function. Simplifying the above equation by using the fact that only non-zero term in the above summation is n²
nn) Einnn and n/y
Σラ ンー=1" n (2)
プ 11
(1)
Σ レ==
(2)
E m (
Σ =
12
E n
(2)
2 )== ng
(2)
E 11
dg ا
(2)
Mn/ n 2x (2)
d2
(2)
d2
1 (2)
n
j=1
n
Σイ
I
d2
)
de
j=1
j=1
Collecting all factorial terms together, and rearranging some terms:
E m
(1))
1
d (2)
Σ
-
m
X
di
(1)
d2 (2)
|1115= 1= (m) (3)
do
Σ 1= 10))
-=1 11
"
Σn1 n (2)
Σ j=1 (2)
da
(2)
d
=== 1
Σ
j=1 1
de
/=1
55

---

ΣS Ej n1²)
Substituting
1 /// /E/z 200)
-1 (2 2/00/
EEA= 2
in the above equa-
tion:
E ( (111
1
Σee و= (2))
Σ m
- 1
x
Σ
2 ml
di (1)
E -== "
(2)
! d2 (n)
p d2
di
d2
(2)
(1)
d2
1!!1
j=l
j=1
di
n
Σd a
(1)
Σn m
(1)
- 2 أ n
(2)
!11
d2
(mg (2) )
1
x
1=1 12
j-
j=
d2
(2)
(2)
(1)
d2 (2)
Σ 12 (1)
1 1
122)
Σdj n
Σ
x
(1)
Σラ= njj
(2)
(1
(n, (2)
di
(2)
(203)
(T)
j=l
By inspecting (203).
we can see that Q(N2 Ī = MMllinomial (k; N, p) with,
N = Σ No. k=
di
[MP] [
d2 ΣS N1
j= N₂ T
d2
(204)
j=1
p [M / xg"
19 // 20) 1- Eje, // "0'""
(205)
Hence, by (205) we can conclude that Q(N2/N,) = ((N finishing our proof.
Lemma 17. Let Ñ be a d-dimensional vector and M be positive random variable, i.e. M
P(M), < 0) = 0. Let P(N M) = Poisson(d, d', A), where:
Ā [A ... da d12 ... -1-(er11)..d)""
Xin.. = -K...yyM M.. V (d') and (i1..... ,..).., E AAS.
(206)
Let No = [NT. Ng No. No Ndd1zad...
where P(Ñ
Poisson(d (dx 1, A) and Ñ AÑo, where dA the dimension of A, and A = [A Aa]
as defined in Sec.
J.1 Then P(Ñ M) P(ÑOIN), i.e. M, and N form the following
Markov chain M
N No.
Proof. To prove lemma 17 all we need to show is that P(Ñ M) does not depend on M. Let us to calculate P(Ñ N, M). By Bayes Theorem, we know:
P(Ño M) I(N No, M)
(207)
P(N
Let us now write the expression for P(NIM) using (96)
d'
Ty Mr
M -
(7)
Σ1 II
TT==,%,
1=
(E)...)))EA
(208)
56

---

Canceling M in the above equation for the terms inside the summation, we get:
P[N IT d II m)
ΣI II
Biana
(209)
I 74
J= (N...)))E
Absorbing all terms that do depend upon M into we have
d' B EI II
(210)
[Ti=1 Yha
Then, we can rewrite 209 as:
d
P Be 1s Á (n Be Be- M A
II (76)
Be I' M'
Be P
(211)
where, in (a) further absorb I (7:) into B since it does not depend upon M, and in (b)
substitute Li n4 = H' Similarly, let us write out the expression for P(N9
d' II j=1
P(Ñ = -
I'A
II (Ha.. M m
(212)
Collecting all the M terms, and absorbing all the terms that do not depend upon M into D, we obtain:
P(Ñ - = De- 1 De-1" M D/ E y (ing Now let us analyze the term I Anº:
(213)
i
Afi
21
Ist An (a)
(19) ATT (b)
(n197
(z19)
A
d''.
where (a) uses the fact that 1 A is scalar and hence is equal to its transpose, (b) uses the fact A A= =[A [ A ) and (c) follows from the special structure of A,, i.e., that each column only contains i ones and zeros, and the fact that A is akin to summing up the columns, hence A i = i1. rewrite the above equation as:
Equivalently
T A'A 9
Σ (yE
(jnh.....
(214)
Substituting (214) into 213
P(Ñ = De I MI" Ane
(215)
Now, let us write out the expression for P(ÑIÑe M). Since N = AÑ., P(Ñ(Ñ9) can be represented Kronecker delta function with the condition N = AÑD, , i...
P(N N M) P(N N (i)) 0g (ń Añ
(216)
where (.) is the Kronecker delta function. Substituting 2111 215 and 216) in 207
we get:
De-1" 1 A ARD 5x (n
And
(a) DM And 5x (ń An
P(N M)
Be i i
BMİ"E
DMI'T Añ
5x ñn An
Dōg iñ Amo
(217)
BM AHo
57

---

where we obtain (a) by canceling the term e I'' A (b) by using the fact ñ = A due to the delta function, and (c) by canceling the term M Aãe Since the terms D, and 6x Ano do not depend upon M, we can conclude P(Ñ*IN, M) also does not depend upon M, i.. PIN N M) = P(Ñ9 [N) or M + N No.
XT r
Lemma 18. Let M P(M) have support over (0, co). Define, X
x7
and
Y Y ... d T
where X; and Y, are random vectors of size qi N andrr N, respectively, V E [d,], and j E [d2). Let the conditional distribution of X and Y conditioned on M be specified as follows:
P {
II
P [ M).. and P (YIM) II ==1
P
(Y.IM). where
1= P (x.MM)
Poisson (q,, 1,,,, A, -I Mr
ny Mi T
Vie [dil, and
P
Y (
Poisson
(1,,1,A))) A. = =bAM²
My Vj E [d2].
(218)
If d1 d2 and 219 hold
41 Vi E [dz]
(219)
1=1
j=1
then the distribution Q(M, X, Y) defined in (220 lies in Ap.
-
(220)
where Q( X) is a product of d2 multinomial distributions, i.e Q()(x) II 2(Y,(x,), where
Q(Y, (x)) Mullinoomall (k;; No, Pc) and
N₂
Tijs
K.
ya
Dim
Ej=1 Fij- - En Uii]
j=1
Tym,
1 1
765
Ene
Eje 7
Ene
16 For showing Q(M, X, Y) specified in (220) lies in Ap, we need to show that Q(M,, X) ((M, X) and Q(M, Y)
Proof. Lemma 18 follows from straightforward application of lemma
P(M, Y). (M, X) = P(M, X) is satisfied trivially by construction Q(M, X. Y).
Proving Q(Ý, M) = P(Y, M):
For showing the second condition, we will explicitly show Q(YIM) = P( which, combined with the fact that ((M) = P(M) (by construction), will show Q(Y, M) = P(Y, M) completing proof.
58

---

Calculating X): From the law of total probability, we have:
M = Σ
= -
P
-
M)) II
X =
i=1
d2
d1
Σ
I
P
=
& = x
II
P(X: = M)
X ==1
i=d2+1
d2
di II imdg+1
Σ P(X, - = xx))
Σ 8,
- xxIM)
x=1
da
dg -I
(d)
Σ &
(221)
where (a) is due to the particular structure of P(X and Q X) specified in 218 and (220).
(Y(X)
respectively, (b) is just collecting the terms P(X [M) and Q(Y,[X.) into a single product, (c) is due to spreading the summation of the components of x over their corresponding distribution components, and (d) is due to the fact that sum over all probabilities of all possible x; is 1.
Note that for fixed m, P X m)
Poisson (ni, 1, [xam'
Vin, mim,m" and
Q(Y, Xx)= Multinomial(K; (k,; N;, p.). where
N₂ Tijs
kj [y
yim. ΣØ= Inj - ΣØ Ui) u.)""
j=1
Toms
P
78g
E ==1 769
Note that we can equivalently write p. as follows,
M T M
y Ar
1 Em T6f M²
p.
(222)
M
Σ5 75. M
In the setup of lemma
16 choose dj qi and de rie 1 -n Vj j [ry] and A5")
Mi A V j [4.]. assumption. Hence, we use the result of lemma 15 to conclude ((,,IM) = Poisson(rs. 1, with
From (19). we know the condition EØ= (2) S ΣSn x! is satisfied by
[A Air. 1" with Aij = 7 M
(223)
Comparing Q(Y, specified in 223 with MM) specified in 218). can conclude Q(Y.|M)= = P(Y, [MM) Substituting
( = (Y,((,) in (2T), we obtain:
d2
@ MM = II P(Y MM)
(224)
Comparing (244 with 218
conclude Q M) P P(Y finishing proof.
59

---

K Proof of Theorem
this section, we provide the proof of theorem Since theorem provides the analytical PID terms for the linear connoluuiion-clseee system defined in Sec. 5.1 breflyy restate certain key properties of convolution-closed distributions and the linear convolution-clon system for convenience.
K.1 Convolution-closed distributions
Convolution-closed distributions are a large class of distributions that are defined as follows:
Definition 1. Let FD denote a family of distributions, where each member distribution, f()) E FD. is indexed by parameter 8. Consider X1 f(ōi), X2 f(02), and X1 IL for some 51, 02 D. Then, FD is convolation-closel in the parameter 6,
+2 f(61) * f( = f(1 + 52) 81, 82 E D such that d1 + + E D,
(225)
where: * denotes the convolution operator.
Convolution-closed distributions define natural dilation/thinning operator. Formally, ~ 1(8). Then, define X. the e-dilated version of X if X f(ed) for some (0, 1) such that ed D. Furthermore, if we assume (1 ε) E D. then P(X,(X) can be defined as P(X, Xe + X(1-e)) where X1-c ~ f(11 c)6) and X, TT X1-e. Denote, P x) P(X,XXe ((1-)) x) G(ed, c)6, z).
K.2 Definition of linear convolution-closed system
Let FD be convolution-closed distribution family in parameter 8 D, and M be target/meessage random variable having distribution P(M) over some support set M. Define the conditional distribution of random variables X and Y conditioned M as follows:
P(X|M=m) ( and P(Y[M=m) f6% such that 5 8% DY m E.M. (226)
Furthermore, assume that there exists 7 E (0, x)) such that: 6x = 760 V E M. We denote such a system of random variables (M, Y). having the joint distribution P(M, XY,Y, a linear convolution-clssee system.
K.3 Formal proof of theorem
Theorem 6. Let the joint density P(M,X,Y) of random variables M, X. and Y describe a linear convolution-closed system. Without the loss of generality, assume 7 S If
(a) 7 E DDV m CEmEEM M.
(b) PXX,|X, + XI-,=I, M-m) G(78, (1 7)8% x) does not depend on m, where P(X,IM) ), P(X 5((1 7)) and X., I1. X1-1IM,
then Ap contains a Markov chain of the form M + X +Y and UT(M;Y 0.
Proof. We first note that we can always assume 7 < 1 without the loss generality because if 2 then always switch nomenclature to refer to Y as X, and X as Y.
We briefly outline the proof structure.
1. In the first part, we explicitly construct a joint distribution (M, having the Markovian structure M - X Y.
2. In the second part, we show that the QMc X, Y) constructed in the first part, lies in Ap. Therefore, we can then apply the result of proposition | to conclude U (M;; (X)) = 0.
Part 1: Specifying (Mc(M,
60

---

Denote the joint distribution of the Markov chain M + X + Y as (M, Y). We can decompose Q X, Y) by utilizing its Markovian structure follows:
Qmc X,Y) = Que((MTQxxc((IMMMMNe(((((
Consequently, we can construct the distribution (Mc(M, by individually specifying QMc ( and Q (Y)
Specifying QMc and QMc(X]M): We specify (Mrc(M) and (Mc(X as follows:
QMc(M) P(M) and QMc P(XIM).
(227)
Specifying Q.C (Y]x): By assumption (b) in the theorem, we know that G(y (1 does not depend on m, and consequently, does not depend on 6, Hence, we can simplify notation of G(yō (1 - 7 m) G(y, (1 (1->), 7), We specify [X([x) as follows:
7) .))
Q0 = x) = 1,,,)
(228)
Part 2: Showing QMC QMc (,
For showing that (M, Ap, we first need to derive QMC (Y)
Deriving (MM)Y We use the result of lemma 19 to derive QMc MM). As QMc(XIM
m) = f(6%) is convolution-closed distribution for fixed m. and QMc(Y = v/X = x)
G(v, 1 - 7, x). the result of lemma 19 shows that (Y can be expressed as:
QMcc( = m) = f(r x ) = f(70m))) (a) f(6%) = P( = m)),
(229)
where (a) is due to the assumption 6%xh = 76% (see Appx.
K.2
From
227 and 229 we can conclude:
Quc X)- (Mc((MQMc(XIMM)) P(M) X),,
(230)
QMc(M, (M,Y) = (Mc(MM)Qcc(|MM)) = P(M))(YIM) = P(M,Y).
(231)
Hence, (M, X,M, X,Y) E Ap and consequently, by proposition UI(M;Y(X) 0. concluding
our proof for case 1.
Lemma 19. Let FD be a convolution-closed distribution family convolutioncclosed in the parameter For a 8 DD assume there exists 6 (0,1) such that c5, (1 c)8 E DD. Consider the following random variables:
X ~ [(E)) E FD. Xi_c ~ 5((1-c8)) E FD. and X ((6) EF)
such that X 11 Xi 1f If = x. x. X = x) =x)= P((: = x.IXi + Xi_e xx), then we have
P(X. x.) P(X xZ) c0) or X. $ X', where & implies equality in distribution (57).
Proof. First let us derive the distribution of X' = =x + X Since, X IL Xi we can write P(X') follows:
P(X) f(eō) f((1-c)8) (0) f(eō (1 c)) f(6) P(X),
(232)
where * is the convolution operator, (a) is due to the properties of convolution distribution (see Appx. K.I. and (b) due to the definition of X in the lemma statement. From 232 we
can conclude P P(X x) = P(X' = x), and by assumption we have PXX. P(X
P + X1_,)). Hence, we have:
P(X = X. xx) = P(X = x)P (X. = x.|X = x) P(X' x) = x2|X' = x),
P(X' = (X. x.|Xi + x
=P(X. + = x, =x).
(233)
From 233 we conclude that P(X.) P(X.) = f(ed).
61

---

L Proof of Theorem 7
In this section, we provide the proof of theorem
L.1 Definition of Peapl and Perp2
We restate the definitions of Peapl and Pezp2 defined in Sec. 5.2 which is used in the proof of theorem
Let X Pexpl (X), where
Perpl (X = x; 0, 02) = (01. 82) exp(0f= 07 A(x),
(234)
for some appropriately defined (((--),(()), 01 and 02. Furthermore, define random variable Y through its conditional density p = z):
P(Y y x; 01, 02) h(y) exp (x" A(x).
(235)
for some Λ()). T()). and 03. such that p(Y y|X x; ,,,00, 02) is well-defined distribution. Furthermore, the marginal distribution of is expressed as:
H(0),02)
Perpz (Y = y; 02 O2 0.) = h(w)
(see proof of theorem in 26
(236)
H(O, + T()), 02 + 0z)
L.2 Formal proof of theorem]
Theorem 7. Let M,X, and Y be random variables having the joint distribution P(M. X,Y). Furthermore, the conditional distribution of X and conditioned on M are as follows: (X; 0g (m), 02(m)) and P(Y M=m)= (Y: (m), (m), 0z). Then, Ap contains a Markov chain of the form M
and ( (X) - =0
Proof. We prove theorem by explicitly constructing a Markov chain M + X - Y, having joint distribution (M, X, 7). and showing that Quc E Ap.
Specifying QMc and QMc(XIM): We specify (Mc((M) and (Mc(X as follows:
QMc(M) P(M), and Qurc M) P(XIM).
(237)
Specifying QmC (YJX): specify QMc(V)X) (xp() 0z A(x)). Then, by (236)
we know that
(m) (2)
Q(Y = y|M = m) = Pexpe (y: 01 (m). (2())
(238)
+ (Tl)) + 03(m))
From
237
and 238
we can conclude:
Q QMc(M,Y) ( = P( P(M,Y)
(239)
Qmc QMc(M,X [, =
= P(X = P(M,X).
(240)
Hence, QMC X,Y) Y) Ap and consequently, by proposition our proof.
UT = 0, concluding
M Additional proofs and details for Sec. 6
In this section, we provide the proofs certain statements made in Sec. as well as additional details regarding the simulation study presented in the same section.
M.1 Generating Q for every Q Ap
Our goal in this section is to show that there exists a Q (as described in Sec. for every Q E Ap. where the corresponding P(M, X. Y is as defined in Sec. 6 We first briefly restate the assumptions on P(M, X, Y) described in Sec. 6 The distribution P(M,, X, X,Y) Y) has the following properties: the random variable M has support over M, the conditional distributions P(XIM) and P(Y are
62

---

members of some convolution-closed distribution family FD. and there exists some sess bios E D
such that:
M -MM== f( and P(Y where 8x 8% D m E M, with
8x
8X mm (8 ky for some (
[0, 1 m M.
6 = (2) (6% -oblae) for some
c
(2)
[0, MalImmEM.
m
8xX mm
(3) (5/ o for some m
10, m EM, and
(6 bass) (6 8 (5
6x
(Bias
8)aa))) E.
(241)
As Q(M,X)= and Q(M,Y) - P(M,Y) due to Q E Ap, we have:
Q(X and Q -m= (6 where dy 6% E DDMME M.
(242)
M.1.1 Explicitly constructing Q for every Q E Ap
We show existence of Q for every Q Ap by explicitly constructing Q for each Q E Ap.
We define vectors to simplify our notations:
- yJ - Y" nx Y ny!"
(243)
where X and Y are the random variables defined above. (X', Y" nx), and (Y', ny) are the corresponding dilated versions of X and Y defined in Sec. 6 Note that the joint distribution of (M, X', Y" nx, Y',n',, ny) is Q(M, X', Y" nx, Y',n' ny). We reiterate the two main properties of Q(X', Y". nx, Y', ny),, mentioned in Sec. that want prove:
The marginal conditional distributions (X', Y" nx, Y', Y', my) given M are as follows:
Q(x]M==)=\(63 5% - (5 - ))a))), Q(Y" M==) = 1(5x - % Q(Y' (=)) )) @( (M) (( Q(nv (6)
2. The random variables (X', Y" nx) are jointly conditionally independent given M. Simi- larly, the random variables (Y",, ny) are also conditionally independent given M.
We construct B from Á by specifying the conditional distribution Q A, M). Note that given the values of A and M, we can always use QB A. .M) to construct the random variables B. The goal of this remaining section is to show that the corresponding distribution Q(B, (specified through Q A, M)) satisfies the above two properties. We first explicitly state the structure of Q(B A. M).
Specifying Q(BIA, A, M):
We first impose the following structure on Q(B)A, M) as follows:
Q(BA,, M) = Q(X', Y" m, XX (YY(M)QQY))))) ny YY,),
(244)
where Q(X', Y" nx|XX M) and Q(Y', ny /Y, M) are the conditional distributions through which we specify Q(B]A, M). Since Q(B),, M) product of two well-defined distributions, it is valid distribution. Let us now analyze the joint distribution Q(M, A, B). By chain rule, we have:
Q(M, Q(M, Q(()A M).
(245)
Substituting the fact that (M,A) = (M,X,Y) and using the structure of Q(B|A, M) specified in (244). we have:
Y", "x M)Q(" (Y,
(246)
We now specify the two components of Q(B|A, M).
Specifying Q(YY, nn MY;
63

---

Using the chain rule on Q(Y", ny M) we obtain
Q(Y', my Y, M) - Q)
(247)
Choose:
=y, G(e (2) 8% (1 = )0% Stm.y), y), where = )uas) V m E M, and
Q Bony/l=yyyY==
= (ny = y-
(248)
Here, (x) denotes the Kronecker delta function (59] Marginalizing (X, X', Y", ,xx from (26)).
we obtain:
Q(M,Y,Y' my) =(M))((X1M)Q(( my/ Q(M))((Y) ny
(249)
Consider the distribution Q(Y,Y", My MM). /
Then, we know:
Q
= f(6%),
yy Q -m) 6g ( and = - mm (1 - (1 )o v)-
Q(Y'=u/Y==y
(250)
Comparing the above equation with the structure of Q(Y, Y', IM) specified in proposition g we can conclude that (y", are just dilated versions of Y. Consequently, using the result of proposition 8 we have:
Q(Y'I
f(o - oa Q(y](M) f(0% and Y' 11 ny JM.
(251)
Specifying Q(X', Y", Mx)
We similarly decompose Q('', Y" nx X M) using the chain rule as follows: Q(X', , = MM)xx/XXX))((X(xx,,, M)Q((" |X, X',
(252)
choose:
Q(ny XX -x, M mm) = G((1 - (/) 16xr mm 8x x), where 8x = 13 ( V m E M,
Q(X'I= nxx =
Tnx: M M=m) xx
G((1 e ( b elll Spias). x Tax) where
m 80xs = = ) A m E M, and
Q( IX=I, nx =xxx X'=,', M=m) (1 (1x= =x-x===d - I.n)
(253)
Here 5x() (-) denotes the Kronecker delta function obtain:
(59 Marginalizing (Y, Y Y',y)) from 246 we
Q(M, X,M,X,X', Y",, nxx = Q(M)Q(XXI)Q(XX nx, Y"I,,)) -"MMX/ nx M).
Consider the distribution Q(X, nxx,, MM). Then, we know:
Q(X(M=m) = ((5 and Q(ny|X, M), Q(X'IX(Xxx ax, M), and (X""I,,,,, X', .)) are as defined in (253
(254)
Comparing the above equation with the structure of Q(X,X' nx, Y"l specified in proposition 9 we can conclude that (X', Y" nx) proposition
, are just dilated versions X. Consequently, using the result of
9
we have:
O(X]I==m) = [6x 5%
(5
Q M) 1(5
Q(Y"|M)= = 1(6 -y and (X', ynxx) are jointly conditionally independent conditioned on M.
(255)
This concludes our construction of Q(B[A, M). We obtain our desired Q(M, X', Y" nx, Y", ny-) by marginalizing A out of Q(M, AB From (251) and (255) we know that Q(M, X", Y" nx, Y", ny) satisfies the two main properties specified earlier. Since we did not make any specific assumptions on Q aside from the fact that it lies in Ap, the above construction
nx, always be used to construet nX',,x, Y" Y' ny) from (M, X, Y) using the Q(B|A, M). Note
that due to the particular construction of Q(B [ M). we have:
X' nx + Y and Y Y' + ny.
(256)
64

---

M.1.2 Propositions required for constructing Q
We describe two important propositions that we use to show the existence of Q for every Q E Ap. Proposition Suppose the random variables M and Y have the joint distribution (M,, Y) such that
Q( P(M) and Q(Y = f(6 f(
E
(257)
where M denotes the support of M and of f(6m) E FD for some convolution-closee distribution family FD. Furthermore, there also exists a bias E D such that
6% = (2) = (6 - -8raa) for some e E [0, 1] and (6% - EDVM M.
(258)
Consider two dilated versions of denoted as and ny, defined as follows:
Q( M m=) G(c!22 8% (1 (1-c22)0 v), and ny Y YY'
(259)
Then, we have
1. Y 11 ny M
2. The conditional distributions M) and Q(ny) M) are specified as follows:
Q(YIM m) [( 6% -66% (baac) and
Q(ny|M=m) = ((( - (0 )xx) = 5(8) 4 m E E.M.
(260)
Proof. We essentially modify the argument given in theorem 1 of (25] to prove the above proposition. Consider the conditional distribution of Y. Y', and ny conditioned on M. denoted ((YMM))( (Yr M). By the assumptions in (257) and (259), we know that
Y", my) (M)... Then, by chain rule, have
Q(Y[M=m) = f(
Y = G( (2) 8Y (1 em 16 y), and Q(ny|Y=y,Y'='/.. M) = (n = -y-- -y)),
(261)
where 5x( (-) is the Kronecker delta function
159
Observation 1: Our first key observation is that the distribution Q(Y', Y, (y|MM=) is completely specified through the distributions MM) and Q(Y'Y,, M), as ny is completely determined by Y' and Y and Q( Y', .M) is specified through Kronecker delta function.
Similarly, consider new pair random variables Ykew: ny such that View IL n M and the respective conditional distributions of Y and ny are specified as follows:
Onew((yn M=m) = fc!!2) 5%), and (nz ( M= = f((1 - (2)
E M. (262)
We will now show two key properties for the random variables (Ynews ny near Ynew + uz ) that will form the crux of our argument:
Property 1:
Qnew (Yiew ngee = M-=m)
Let us now calculate the conditional distribution Q(Y + nz y)M))
new = M) Qnew (Ye (Yiew + my
Qnew (Ynew = ki ny new = y) k: k\M)).
(263)
Using conditional dependence of Y and neow we have Qnew (Yren k. nyow y -k(M)
Qnew (Y k|M)) new (npew y k)M):
Qnew (X new + ny new = M) km-oo
Qnewn (Y = k: kkIM) new = - -k|M). (264)
Note that the R.H.S of the above equation defines a convolution of Qnew (Y Nrew IM) and Q( IM). hence:
Qnew (Y + +ny = v| Qnew (Y klM) Qnew (ne = -k|M)
(265)
65

---

where * denotes the convolution operator. Substituting 2662 in the above equation:
Quean (Y + ny M f( f((1 - < )mm)
(266)
Using the properties of convolution-closed distributions:
Quew (Y + - f( + (1 ( )om) f(6m)
(267)
Hence, by comparing above equation with 2611 we have:
Qnew (aeno(Yr
(Ynew + n4 neww y M m) = Q(Y yy = m).
(268)
Property 2: Q(Y =/ YYew
new + ng y m-) = G(em (2) 8 1 - (2)
em 0 v)
By definition of G(e 8% (1 (1-Em (2) )o) =) given in Sec. 5.1 we have:
G(( (1 m (2) 18 2) = Pr(Z (2)
7 (2) + Z (3 2), where
Pr(Z.c) = f(em (2) 8x Pr(Z (1) f((1 ) ) and (0) 1L 2 cm
(269)
In the following steps, we slightly modify the notation of G(em 5% (1 - <m 10% 2) to
(2)
G(z (( 100% (1 cm 60% 2) make notations more explicit as follows:
(2)
G(2 4m 8x (1 - =/22)) 16 2) Pr(Z (2) 2. Lg ( + Z --
(2) 2).
(270)
We use the definition of the conditional distribution to express G( (2
gY (1 (2) 30y 2) as:
m: 2) = Pr(Ze =_(2), Z 2,0) + 2 [2) =)
G(z,ca) (2) ( m. (1 m (2)
1-tm
(271)
Pr(Z (2)
1 9 2)
Pr(Ze = (2)
Pr(Z
7 (2) + 1-6m
(2) = 7 (2) =2 (2)
(272)
Pr(Z 2 (2) + 7 1-6m
(2) =
Using the fact that Pr(Z,g (2) + - = = (2) 2 (2)
= 2,0) = Pr(Z
1-
= 7 2 (2) )
since
cm
cm
cm
cm
cm
2
mm
Pr(Z. 2 Pr(ZZ- (2) = 2
1-em
(273)
Pr(Z (2) + Z, 1-r 2)
Note that by properties of convolution-closs distributions, Pr(Z (2) + Z, - 2)) f(z 6%)
where (2; 6%r) is to be understood as the distribution (6%) evaluated at z. Substituting (669) in the above equation:
f(z ( cm SY ff) - 2,g0 (1- c/ )
(274)
f(2;
Now, note that by (262) and (267). we have:
Qneuc (Y new 2e M= m) [=_00) (2)
m 5xx)
Quew (nnew =2 2, (M=)) m) = f(z 2,00 (1 - (1-c(2)) and
Qnew (Cu (Y +n new = = = M=m) = f(2;
(275)
Hence, combining the above equation with (74)) and renaming 2_(2) as yo and as y. have:
G( (1 -1(2) 1o) v) Qnew (Y
- (MM-mmee m)enn -y 1-1--116 Mm)
Qnew Vhew + ngow= Mm)
(276)
66

---

Noting the fact that " -y - D'(M-n) Qnew (Yewly (Yew, + ny M - m, Ynew =v')
due to new -. ny M and abbreviating notation pertaining to y', we have:
G my (1 (2 1o v) Qnew ( (Y M I [M=m) (Y + n = -ylM=mm
(277)
Qnew (Ynew + ny - y M mm)
Using Bayes rule in the above equation provides us with our desired property:
Gi om (1 (/
1o y) Qnew (Y M m, Ynew ny =y)
(278)
Furthermore, by comparing (259) with the above equation, we have:
new = y, U, - m) = Q(Y = = Qnew (Yn = 3/ VYie + my
M-M)
(279)
Observation 2: Note that the distribution Q(Y nyw,, Y new + ny
nyew M) is also completely deter-
mined by specifying Q(Y + ny (M) and Q(Y Vnew + ny M) since ny completely determined by Yeo + ny and Y nnw
By property and (299). we know that
Qnew (Ynew + +ny new [M) ((Y|M) and Qnew
Sneel(Y (Ynew Veewr Niew + ny ny MM = QYY")r, (280)
Therefore, combining the above equation with observations and 2,we have:
ny Consequently the distribution Q(Y,, ny M) Quea (Ynew we also have Y' IL ny|M Furthermore, by 262
Q(Y', Y, ny M) Q(Y new? Ynew + nl new new IM).
(281)
Since, Yeew IL. ny M,
nnew (Yew [ M=) Q(Y MM f(e (2) 6m V m M, and
new IM =m) = Qnny M) = -((1 - (2) 1 A m EM Quew (n
(282)
By 258 we know that (2) B8 8Y, o0 and (1 - (2) )0% = 8 Hence, [m
Q(YIM)) = (((12))) 516% bY V m E M, and Q(nr|M)- s(1 (2)) )6)))))))))))))m))m))))))))))) = f( mm E M.
(283)
Proposition 9. Suppose the random variables M and X have the joint distribution Q(M, such that
Q( = P(M) and Q(X[M=m) = f ( 4 mme. E.
(284)
where M denotes the support of M and (6x%) FDr some convolution-close distribution family Fo. Furthermore, there also exist 6 E D, and 6% E D E M such that
6x 8x (L) a for some e( [0,11] bias
8. 6m (8:x Sprna) for some em [0, 1]. and
(5th 5Y (5it - &%an)), (8x ( (6% -8 ) E m E M.
(285)
Then consider three dilated versions of X, denoted as X', Y", and nx, defined as follows:
Q(X', Y n X)
M)) nx, M))(( nx, M')', where X=x, X-x, M=m) = =G((1 - (3) ) cm 8X x)),
Q(nx
Xxx M M=m) = =G((1 (1(11-((0) (6 s cm m 8
(8x x
). x - -nz). and
y"=
= X - X nx-
(286)
Then, we have
1. (nx, X', Y") are jointly conditionally independent conditioned M.
67

---

2. The conditional distributions Q(XIMM, MM), and MM) are specified as follows:
Q(X'IM) - -6 8%H -6 - - b)ss)))
Q(Y = =smm ogias), and
Q = f(
(287)
Proof. The proof essentially consists of recursively applying the proof of proposition 8 Consider auxiliary random variable Z X nx. Then, the distribution Q(X, nx Z M) has the following properties:
Q(XIM) (0) (( Q(1x M ( X X ( G((1 _
(3) )o cm 8x 2), and
Q nx nx M m) 8x (Z
Tnx) )
(288)
where 5x(1 is the Kronecker delta function, and the equalities (a) and (b) follow from (284 and (285), respectively. Note that the distribution Q(X, nxx M)M has exactly the same structure as Q(Y, ny .M described in (26T), hence Z and ny are just dilated versions of X. Consequently.
from the result of proposition 8. we know that Z 11. x [M, and the conditional distributions Q (Z)
and (n IM) are described as follows:
Q
S((1 11 and Q M= f(e (3) 8
(289)
From 285
we have em 8x
(3)
5x 8X and (1 cm
)
(3)
0 Consequently,
Q(nx (- f( and Q Q(Z f(
8
(290)
mx We alternatively express Q(X' mx. M) as follows:
X= = G((1 - ( (6 ). (11 (6
& x Inx) (291)
Note that the distribution Ix x=x, M m)) only depends X and nx through their difference X nx. Hence, substituting Z = X nx in the above equation:
X= M=)) Q(x' Z=- Inx M M=m)
=G((1 (6 : (8xx
bi x nx)), (292)
2(X = G((1 - (((1)(()) (5x) 6bins) ), < (1) (5x
8x , 2).
(293)
Hence, consider the following distribution Q(Z,, X', Y" where from 290 and
293
we have:
Q(Z = f(8xx 1 6. b
Q(X' = 2, M m) G((1 (1) )6xx
x ) (6 d 2).
(294)
Furthermore, by 286 we have Y" = X nx X (c) Z X'. where (c) is due to Z = X nxx Hence,
=x) = 8x (Y" = = =z-x). x). Z M, M,
(295)
Comparing 295 and 2244 with 261). we can again conclude that X" and Y' are just dilated we have X' IL Y and
versions of Z. Hence, again using the result of proposition &
Q(X'MM=) f((1 - (1)) ()) and Q(Y"IM=m) = f(el)) (5 d)) (296)
From 285 we have cm (8x ) 8Y 8y and (1 (1) ((xx 8y 8x 8Yr
baas ). Consequently,
M-xm = f(8 5% (8ix Dynn) and M=m) = -(6 -Sh (297)
68

---

Hence. by 2900g and 297 we can conclude that the conditional distributions (EY and @(nx M) are specified as follows:
Q( m) f(8m 8% - -680as opsas))
Q( M) = f(6 0pan)) and
Q(ax (M) f(8tass)
(298)
Showing joint conditional independence of (ny, X', Y") given M:
Consider the distribution Q Z, X', Y Using the chain rule, we have:
Z , MM) Q(nx: (x),, 2),
M",, 2),
Qmxx Y" ( M,,),
(299)
where (b) follows from nx 11 ZIM. For showing (c), observe that Y" Z X', and hence we write (Y, X Q(Y Z, X'). Furthermore, by analyzing ((X''11x Z, M), can conclude:
[xx Z, M) Q("" nxx X nx, M) ((x' XX, M) Q(x' 1 M) (300)
Marginalizing Z out of (299). we obtain:
(()x where (d) follows from X' IL I. By (301). we know that (nx, Y") are jointly conditionally independent given M, concluding our proof.
(301)
M.2 Analytically optimizing the upper bound
Proposition 10. Let Q(M, X', Y", .,,Y' nyy be the distribution derived from a distribution Q E Ap using the construction scheme specified in Appx. M. Then, the minimization problem specified in (302) is analytically solvable with the optimizing Q given by (303)
min (M; [X', Y", .,x,Y Y',n))), (note that Q is function of Q) QEAr
(302)
Q"(M,X',Y' Y" ,,,,,,,,,, Y")
(mx (mx) (m'))"(""IX")) with 6% (6 - - 2 (nx) (
('(M)
Q'(Y"IM=m) -66 - (nv) f(b%aa) and Q" (Y"IX") = I[Y" = =Y',, (303)
where 1[[]] the indicator function. Note that for the minimizing Q"", we have (nx, ny, XM, are jointly independent and Y =
X', X',"")
Proof. First note that by the construction scheme defined in Appx. M.I we know that there is unique Q for every Q E Ap. Hence, we can interpret Q as function of Q. and consequently the minimization problem (302) is well-defined. Using the chain rule of mutual information, we derive the following lower bound for IQ(M; [X', Y" nx. Y", my)
L XX', Y", nx, Y' ny)) (M [X', Y") (M; [nx, Y', ny]][X', "o"lI(X'''") Y'I)
(304)
+lo(M: [X', X"",,,",n" nx, Y myl) Io(M: Y"",,
(305)
where (a) due the non-negativity of conditional mutual information. Furthermore, note that by properties of Q derived in Appx. M.I we know that X' IL y" [M V Q e Ap. Consequently, the term lo (M; [X', "I) does note vary as we choose different Q from Ap as the corresponding Q(M,X' "XXX'"') Y") = Q((M)((X'
M) and the terms:
Q(M) Q(M) = P(M), Q(X ( m= = f(8% - -8Y -8ix @ f -
oy and
(306)
69

---

do not vary for all Q E Ap (see Appx. M.I. Therefore, we have:
(M; Y" nx, Y', ",,,Y,,,,lll my])VQE Ap, I [[M "J -
(307)
where we forgo the subscript Q for Io [X", Y'I to indicate that the term [M; [X", Y
does not vary over Ap. Hence, if we show that the distribution (M, X', Y", nx. Y.,Y ny)
specified in (303) achieves the lower bound in (307). then (M, X', Y", ,1x, Y', ny) ,'y) should minimize the value of lg (M; [X', , Y", nyl) over the set Ap. We now calculate la- (M; Y", nx, Y", ny.)). By the chain rule mutual information:
nx,
1Q. (X', Y", ni,, Y", ny]) = =1 [Mx ny, X', Y"]) + 10. (M; Y I' NY, Y").
Using the fact that Y Y". we know that lg- (0-(AM ny, x,, Y") zero. A sim- ple justification follows from the fact l.. (M; Y'|xx, ny, X, X, H(Y |nx, Ny, X, Y", M), where (.I.)) the conditional entropy function (28] Observe that both conditional entropy terms are zero as Y = Y", and condition entropy of a random variable
(Y,",,,,, Y")
conditioned on itself is zero. Therefore, we can simplify the above equation
Tq- (M; Y",xx mx, Y", nyy) = lQ- (M; [mx, ny, , YI)
(308)
Applying the chain rule of mutual information again, we obtain:
Ih. (M- y",,,, Y", nyl) = 1g- (M; (xx,ny)) + 10- (M; [X', ('''""""Jna"a ml))
(309)
Using the fact that (nx, 1L (", ,"", we know that (M:; [nx, ny]) 0 as mutual information between independent random variables is zero (28. Consequently,
TQ. (M; n", Y', myl] = To. (M; [X', " |nx, my) (b)
Iq. (M: [X', Y"",,
(310)
where (b) again follows from the fact that (mx, n IL [M, Y"), and, consequently, conditioning
(M, X', y" on (nx,ny has no effect. From (310), we see the proposed Q" in (303) does indeed achieve the lower bound specified in (307). and consequently minimizes (302).
The final step remaining in our proof is to ensure that the proposed Q' corresponds to a valid (M, X, Y) lying in Ap, i... there is some ai, X, Y) E Ap that generates Q' using the construction scheme outlined in Appx. M. We know that the relationship between the random variables for (M, X', Y". n,,
nxx,,, Y', ny) having the joint distribution Q and (M, Y) having the distribution Q can be expressed follows: X = X' + Y nx and Y =Y + ny. Hence, these relationships for calculating the conditional marginals Qin(X
(X|M) M) and Qi ( of the cor- responding distribution Qi (M, X, Y) for the proposed distribution QMM,, Y", nxx Y', ,yy):
QUD(XIM = m) = (X' + Y" nx MM
m)
( Q''X'' M = mm)+ Q'(Y'IM = =m)- (m = m)
-(8 6%x - -85 - -oy * * o)ros) f(
4 5(5% 5% - -K - -8% + 88 -8 + 8X
f(xx) = P(XIM- = -m).
(311)
where denotes the convolution operator, and (c) follows from the fact that (X', Y" nx) are jointly conditionally independent given M under Q", and (d) follows from the properties of convolution- closed distributions. Similarly, calculating Qi (Y|M m):
OY m) QX' + +y |M m)
(c) (Q"(""IM = m) * Q* (My|MM = m)
= f(6)) - 5% f
f(6)) SY.s + 8txas)
f(6%) P(Y|M =)),
(312)
where (c) follows from the fact that Y' 11 ny | under Q" It easy to see that (i) =
Q(M) = P(M). Hence, we have (M, Qin(M, = P(M))(X YIM = P(M,Y) which implies (M, E Ap, con- cluding our proof.
P(M))(((M) = P(M,X) and
70

---

M.3 Additional details regarding simulation
The list of different function pairs used in the simulation study are listed in Table | The function pairs were chosen with the only requirement being that they satisfied the assumptions in ( For constructing the analytical estimate Q X, Y), the corresponding o and 8% were chosen as follows:
1. For each distribution and function pair, we first determine the range of possible values for 6 and y
2. For 0 we choose the range as [0, minm EM (mm),, where M is the support set of M for that distribution. Note that we cannot choose 6 > minm f2(m) as the corresponding conditional distribution (")"') = Poisson fz (M) - % (required for the optimal Q" from which QA is constructed, see Appx. M.2 would have a negative rate parameter, which is not allowed. Within this range, we choose 10 different, uniformly spaced values
for 6/m..
3. For each chosen value of oyeas the corresponding range for bias is then chosen as [0, min [minmem fi (m), (f f2(m) + 6 H. Note that choosing the upper limit of op as min minmme [i(m), min - f2(m)) + k ensures that the distribution (X*I) = (o)sson(([((M)) ( - (/2(M) (lias)
)) and a. (required for specifying the optimal Q" from which
(Z( Poisson( ( QA is constructed, see Appx. M.I Appx. | have positive rate parameters. For each value of 6% we again choose 10 different values of o within the above specified range. In total, we chose 100 different pairs of (d V.az)
For each pair of a (. we calculate the analytical QA Y) as follows:
First, we construct the optimal Q* distribution using the structure provided in proposition 10 The exact specification of Q" is described below. We simplify the structure of Q" by using the fact that Y' Y":
Q" (M,,'' Y', ny, Y") = Q' Y', ny)),,
(X" (nx), with (M) P(M),
('('' Poisson (frisson(fi(A) (fi f2(M) -(obra obtan))
Q' (mx) = Poisson (6 (6%
Q'(Y'IM) = Poisson (f2 (M) 6%
(mx) = Poisson ( (
(313)
Then, we use the property that X = X' + Y + nxx, and Y Y' + ny, and use that to appropriately modify Q" obtain QA follows:
We calculate a new distribution QM + +Y, Y,, nx,
Q'M,,X' YY,Y Y nx, ny)
+Y,,M. = =Q' (nx)
M),
(314)
where Z - X' + Y' and alternatively also Z = x nx (as X = X" Y + nx)) First note that (Z( from Appx. M.2 has the form f(x Dxxa) which for the Poisson case is (f 5) Furthermore, we can recognize that (Y = y/ Z = 2, M = m) Q' +Y''' I' + y m) G(y':em) (1) cm + yy which for the Poisson case can be written Binomial Z. pm) where Pm fa(M)) ( (
(1)
Now define S {znx E X 2m Inx +2 = z}. where xm and 2, are the corresponding support sets of the distributions Q* | m) and Q' (Z M m).
71

---

respectively. Then, using the fact that X = Z + nxx we have:
("M==, (M, nx, ny, Y1) = Q' =mmm=mmm--) m))"((x))) (ny)x
Σ xxx 2|M m)Q mm).
(315)
(#nx #2)SS
where know that for the Poisson case:
(*(M) = P(M),Q" = Poisson(O (7)M) Poisson(fi (fi(M) - 8 ):
Z, f2(M) 0
(nxx) Poisson (6 and (Y M) Binomial
(316)
fi [i(M ) 8-X
Using similar procedure, derive Q, from (*, X,,,,,Y Y' .y Y ny + Y'. Define Sy = (Bny ) y Vux + yy, then:
Q,(M, = Q(M, + Y) = Σ (Iay EE)
Q'
= Iny)
After analytically calculating QA for all 100 bias pairs of (o 8% chose QA corre- sponding to the pair (6 8 which results in the smallest value of the corresponding (M: [X,Y]).
Index
f1(M)
f2(M)
Index
fi(M)
f2(M)
3M
3log(1 + M)
11
M
M
1.5M3
0.5e
12
3M²
2M
+ log(MM
0.5M
0.25e M sin ( (MF7s))
13
(iin(At//20)
2
7M
4M sin? (Ma/s)
14
2cosh(M)
Aiii(MM/pp
M
M
15
5M
5VM
6
M3
M
16
e.M
sinh M)
7
5M
(sin(Ma/s)
17
(c)
sinh (M)
M + M
2M
18
2.5 + 5.M
0.5M3 M
9
7M
2 cos (MT/16)
19
1.5cosh(M)
0.5M
10
8(M cOS( (T/s))
2 sin Ma/8))
20
8M +3
7M
Table Table lists the 20 different function pairs used in the simulation study described in Sec. for comparing the analytical estimate of the minimizing distribution of (2) proposed in Sec. with the corresponding ground-truie truth numerical estimate, The index in Table maps the corresponding function pair to the results shown in Fig. To illustrate, the number on the z-axis of all the plots shown in Fig. corresponds to the function pair with index in Table
M.4 Additional results and analysis
We report the tightness of the upper-bound for systems employing binomial and eggative-binomia distributions in Fig. Fia. We find that the results for binomial and negative-binom binomial remain consistent with the Poisson study performed in Sec. 6 and the upper-bound continues remain tight. We also performed further analysis to understand how our upper-bound performs when both of the UI terms are non-zero. We sub-selected the trials in the 20 pairs (analyzed in Section 6 that had both UI 0.01 bits and UI, > 0.01 bits, and report the corresponding median error of these sub-selected trials in Fig. 2 d).
N Distributions
We provide brief descriptions of the distribution families and the corresponding distributions used in this work. We also provide citations to relevant works for the reader interested in learning more about these distributions.
72

---

Difference in Analytical MI
5
2.00
.4%
Binomial
1.75
w.r.t. Numerical
4
Neg. Binomial
1.50
1.25
E 1.00
Jk
0.75
1%
0.50
1
0.25
0.00
10
10
15
Functions Pair Index
Function Pair Index
(a) Upper Bound (UB): Binomial & Neg. Binomial
(b) UIg -oissonn
% Difference in Analytical MI
0.25
w.r.t. Numerical MI
0.20
Uly (in bits)
0.15
0.10
2
0.05
0.00
1
5
10
15
20
1
5
10
15
20
Function Pair Index
Functions Pair Index
(c) Uly-Poisson
(d) UB: Non-zero Poisson UI
Figure 2: (a) shows additional results of simulation study performed in Section 6 of the manuscript for the distributions: Negative Binomial and Binomial. The experimental setup is the same as described Section(6) with the differences being that only function pairs were tested, the number of outcomes of M was restricted to and 4, and the support of M was appropriately modified for Binomial and Negative Binomial distributions (see caption of Table below). The function pairs tested are provided in Table 2 (b) and (c) show the unique information atoms of the systems having non-zero unique information in both X and Y tested the simulation study presented in Section of the manuscript. More specifically, of the 20 pairs analyzed in Section 6 we only selected trials that had both Ul > 0.01 bits and ULy >> 0.01 bits. (d) provides the corresponding median error for the subselected trials (originally shown in Fig M in the manuscript for all trials) for the systems shown in (b) and (c).
Negative Binomial
Binomial
Index
f(MM)
f2(M)
H(M)
f2(M)
1
0.5M²
M 0.25M2
[0.5,r
M
2
M.
[0.04M*
[0.1M2]
3
4M
M sin' Minn (MT/16)+2
4M
M Minn 2 (MT/16))
4
5M
10 isin (MT/16)] +2
5M
10 10]sin(( (sin(M/r/16) +2]
0.05(M M
[0.05(M2 + +
6
7M
0.5.² cos? Mrx +2
7M
[055M2 cos Mr +22
7
M COS(2 (28//))
ssin Ma/18)
[2(M COS( (88(2//8))]
12 sin Ma Ma/40)+21
8
2M
0.001 4/2 +9
2M
0.004 4/345))
9
0.5 2M)
M log(M)
[3.2 2M11
M + log [M]]
10
18sin(M(=/1s)
5VM
19 19sin(MM//an
[3VM]
Table 2: For negative binomial the outcomes of M were uniformly randomly sampled from the
interval [5, 10), and for the binomial experiment the outcomes of M were uniformly chosen from the
set (5, 6, ..., 14). The symbol [-] denotes the rounding to closest integer operation. The corresponding
system models of binomial and negative binomial can be found in Appendix (systems 3 and 6, respectively) of the manuscript.
73

---

N.1 Common distributions used in this work
We employed various well-known distributions in this work. We refer the reader to 62 for review of these distributions. We use the most commonly used notations in literature to represent these distributions. Regardless, we provide explicit definitions of our notations for these distributions to remove ambiguity.
Poisson Distribution Poisson distribution is discrete distribution having support over all non- negative integers. The p.m.f. of the Poisson distribution is described by a single rate parameter À E [0, oo). We define the Poisson distribution with the rate parameter À distribution with all its mass at O. We use (oisson(A) to denote the p.m.f. of a random variable X distributed according the Poisson distribution. The exact definition of the p.m.f. of X is provided below:
degenerate point
P(X- x) Poisson(x; A) ^-
V Nu
Gaussian Distribution Gaussian distribution is a continuous distribution having support over R. The p.d.f. of the Gaussian distribution is described by its mean μ E. and variance o² E [0, oc). We define the Gaussian distribution with the variance its mass μ.. We N (p, o²) to denote the p.d.f. of a random variable X distributed according to the Gaussian distribution. The exact definition of the p.d.f. of X is provided below:
= as a degenerate point distribution with all
1
p(X x) N(;;
2
V WEER
V2TO2
Negative Binomial Distribution Negative binomial distribution discrete distribution having support over all non-negative integers. The p.m.f. of the negative binomial distribution is described by two parameters: the number of successes E N, and the probability use Negative Binomial(r, p) to denote the p.m.f. of a random variable X distributed according to the negative binomial distribution. The exact definition of the p.m.f. of X is provided below:
success p E (0, 1]. We
xr
) (1 p)" p'""" No.
P(X- r) Negative Binomial (x; (rr,p)
x
x (x+--1)
(x (r+ 1)! -
where
Geometric Distribution Geometric distribution a special case of negative binomial distribution with 1. The p.m.f. of the geometric distribution is described by success probability (0, 1]. We use Geometric(p) to denote the p.m.f. of a random variable X distributed according to the geometric distribution. The exact definition of the p.m.f. of X is provided below:
P(X = x)) Geometric(r;p)) p) = p(1 - p)* x E No.
Beta-Binomias Distribution Beta-binomial distribution is discrete distribution having support over (O..... N) for some N E No. The p.m.f. of the beta-binomial distribution described by three parameters: N specifies the support, a E (0, oo) and (0, oo). We use Beta- Beta-Binomial NN 3) to denote the p.m.f. of a random variable X distributed according to the beta-binomial distribution. The exact definition of the p.m.f. of X is provided below:
B(x + a. M.M - x A n E (O.
P(X = x) = Beta--inomiia(2;...
B(a,
(())
where B(a, 3) ((a +))
is the beta function and r(-) is the Gamma function.
Gamma Distribution Gamma distribution is continuous distribution having support over (0, oo). The p.d.f. the gamma distribution is described by its shape parameter a >> and rate parameter O. We use Gamma (., 3) to denote the p.d.f. of random variable X distributed according to the gamma distribution. The exact definition of the p.d.f. of X is provided below:
g
(XX x) Gamma(x; a)
V (0,
T(a)
where I'(-) is the Gamma function.
74

---

Exponential Distribution Exponential distribution is special case of the Gamma distribution with 1. The p.d.f. of the exponential distribution is described by its rate parameter À > We use Exponential X) to denote the p.d.f. of random variable X distributed according to the exponential distribution. The exact definition of the p.d.f. of is provided below:
p(X = x) = Exponential X) = xe A2 v E [0, oc).
(317)
Beta Distribution Beta distribution is continuous distribution having support over (0. 1] (0, 1) depending upon the values of its parameter. The p.d.f. of the beta distribution is described by two shape parameters: a and / > We use Beta(o; (z, B) denote the p.d.f. of random variable X distributed according to the beta distribution. The exact definition of the p.d.f. of X is provided below:
T(a + 3) 11- - E (0, (B)
p(x 2) (2ta(2; a,
where T(-) is the Gamma function. The end-point is included the support of the beta distribution if a 2 and the end-point 1 is included in the support of the beta distribution if , 1.
Dirichlet Distribution Dirichlet distribution is multivariate generalization of the beta distribution. The p.d.f. of Dirichlet distribution is described by d-shape parameters: {ar,...,,ad) where E (0, 0o) V E (d]. We use Dirichlet(an, Ota) to denote the p.d.f. d-dimensional random vector X distributed according to the Dirichlet distribution. The exact definition of the p.d.f. of X is provided below:
I (2 as) p(x = x) Dirichlet(x;; 01, o1....,,ad aa) = I ((ar)
d II 2=1
x T
where [(.) is the Gamma function, and x [ xal with ma E [0, 1] and Σd I; 1.
Binomial Distribution Binomial distribution is discrete distribution having support
{O...., N} for some N E Ng- The p.m.f. of the binomial distribution described by two pa- rameters: the total number of trials, N, and the success probability, p. We use Binomial(N; p) to denote the p.m.f. a random variable X distributed according to the binomial distribution. The exact definition of the p.m.f. of X is provided below:
(N) x:
P(X- x) Binomial(s;; N,p)
p² (1-p) m)=->am vnnnnne (O....,N), N),
(318)
N
N NI(N -2)!
where
Multinomial Distribution Multinomial distribution a multivariate generalization of the binomial distribution. The p.m.f. of the multinomial distribution described by two parameters: the total number of trials, N, and the success probability vector, We use Multinomial (N, p) to denote the p.m.f. of a d-dimensional random vector X distributed according to the multinomial distribution. The exact definition of the p.m.f. of X is provided below:
N!
P(X x) Multinomial,(X; N.P)
where, p = P1 Pa 1T is a d-dimensional probability vector, i.e. S pt 1
and Σ Pi = 1, and x 21 No. V1 < and Dd =n
1" is dddimensiioal categorical vector, i.e.
Hypergeometric Distribution The f.m.f. of the hypergeometric distribution is described by three parameters: the total number of success states in the population S E No, the total number of failure states F e No and the number of draws E {O...., S F}. We use Hypergeometric K, n) to
75

---

denote the p.m.f. of a random variable X distributed according to the hypergeometric distribution. The exact definition of the p.m.f. of is provided below:
P(X = x) = Hypergeometrri(1;; k,)
(2)
V x E {max
(n, (n,
($+P)
A
A!
where
for some A, B E No and B A.
B
BY(A By!
Multivariate Hypergeometric Distribution Multivariate hypergcometric distribution is mul- tivariate generalization of hypergeometric distribution. The p.m.f. of the multivariate hyperge- ometric distribution is described by the following parameters: the total number of object types E Ng, the size of population of each type (ni..... ndn with n, E No, and the number of draws (O...., Ei= m.). Define π = |n n.] Then, we use Multivariate Hypergeometric ni n) denote the p.m.f. d-dimensional random vector X distributed according to the multivariate hypergeometric distribution. The exact definition of the p.m.f. of X is provided below:
P(X x) Multivariate Hypergeometric ni,
1I
(319)
E R4)
where (A)
A! BI(A- By for some A, B No and B S A, and x [2]
xa] with i;
{0, n and >
Uniform Distribution Uniform distribution is continuous distribution having support over (a, b)
where a, TR. We use Uniform(a, to denote the p.d.f. of a random variable X distributed according to the uniform distribution. The exact definition of the p.d.f. of X is provided below:
p(X = = r) Uniform(x;;))
Vr E
a bl
For uniform distribution, abuse notation by not ensuring < b for representing the interval [a, b]. In the context of uniform distribution, the notation [a, b] is to be understood as [a, bj if a S b and [b, a] if 2 b.
Bernoulli Distribution The p.m.f. of the random variable X distributed according to the Bernoulli distribution denoted as Bernoullip)) for p 1]. The exact definition of the p.d.f. of X is provided below:
P(X 0) p, and P(X =) =1 =1pp
(320)
The p.m.f. of random variable X distributed according to the categorical distribution denoted Categorical (pi...., ,aa) for ps E 10 [d] and Σ Pi 1. The exact definition the p.d.f. X is provided below:
Categorical Distribution
P(X = = pi
(321)
N.2 Univariate Stable Distribution Definition
We employ the definition of a univariate stable distribution discussed in 23:
Definition A random variable X said to have a stable continuous distribution if. for two independent copies of X, denoted X, and X2, and positive constants 0 and b > (322) holds for some e > O and d IR.
aX1 6X2 d cX + where d means equality in distribution (77
(322)
76

---

Continuous stable distributions are typically expressed through their characteristic functions, as the analytical form of many continuous stable distributions is not known. The characteristic function of a random variable X having a continuous stable distribution is typically parameterized by four parameters: stability parameter denoted as a E (0, 2), skewness parameter denoted as / E [-1,1] 1]. scale parameter denoted as y E (0, oo), and location parameter denoted as 14 IR. The exact form of the characteristic function is given in (323).
E
[c
(t; a, 9, 71 = exp (itpu - 11t) i.3sg((t)
(323)
-1
t<0
tan E
a + a == sgn(t)
where ((a)=
2 -2 log (
0
t=0
1 t>0
We further denote the p.d.f. of X having the characteristic function defined in 323 as P 3,7,1) Some well-known examples of continuous stable distribution are the Gaus- sian/normal distribution, Cauchy distribution, and Lévy distribution. All univariate continuous stable distributions have infinite variance except the Gaussian distribution, and their tails follow a power law behavior [23]. Univariate stable distributions have been widely used to model many financial and physical phenomena. Chapter 2 of [23] provides a list of applications where stable distributions have been used. For a more in-depth treatment of stable distributions, we refer the reader to (23].
N.3 Multivariate Stable Distribution Definition
multivariate stable distribution is defined similarly to univariate stable distribution described in Appx. N.2 We employ the definition of multivariate distribution given in (23
Definition 3. A non- non-generate d-dimensional random variable X is said to have a multivariate stable continuous distribution if. for two independent copies of X. denoted as X2 and X2 and positive
constants > O and b >> 0, (324) holds for some > O and d E Rdd.
xx, 2
d. where & means equality in distribution (57].
(324)
A main source of complexity while analyzing multivariate continuous stable distributions is the wide range of dependence structures this family of distribution admits (23). We refer the reader [46] 45 for more general treatment of multivariate stable distributions. In this work, we focus on two special classes of multivariate stable distributions: independent component multivariate stable continuous distribution denoted as pcs- (c. 3, 7. ji) and elliptically contoured multivariate stable -BC (a, Σ, 1i). Similarly to the univariate continuous stable distribution, both independent component and elliptically contoured multivariate stable distributions do not have explicit general analytical forms and are expressed through their characteristic functions.
distribution denoted as pCs-pcEE
The characteristic function of d-dimensional random vector X having independent component multivariate distribution pcs-ic B, 7, (i) is expressed as follows:
d hotp 13 (t)))(())))))))(()) iEF i VEE (325)
E 1 exp
where ((a)) and sgn(-) are as defined in 323
B = [8| Bal² with 33 E 1-1, タ =
[T1 7a)" with 7j E (0, o0), p = [M]
pa]" E Rd, and E (0, One can also alterna-
tively think of the random vector X = [X1
Xa)" having the distribution pCs-ic(a, G, 9, ii)
as collection of independent random variables {X,]ş= where X, pcs(a, 8j, (j))
The characteristic function of a d-dimensional random vector x having elliptically multi- variate distribution Pcs- PCs-Ec(a, E, jī) is expressed as follows:
E
e
x = exp
T
Σt
5/2
+ +iET
ViER. Rd,
(326)
where Σ is a positive definite matrix, pi = [0
na] E Rdd and a E (0,2). Note that allowing
Σ to be positive semi-definite will violate the fact that X should be a non-degenerate d-dimensional random variable. We also list an important property of multivariate continuous stable distributions used in this paper.
77

---

Property 1. A d-dimensional random vector X has multivariate stable continuous distribution and only if every 1-dimensional projection has a univariate continuous stable distribution, i.e., XTX pas(a, ((t,,
R
Proof. See the proof of P.1. in 46
N.4 Univariate Discrete Stable Distributions
Discrete stable distributions form the discrete analogues of the continuous stable distributions. We employ the following definition for a univariate discrete distributions taken from [24]:
Definition 4. A random variable X is said to have discrete stable distribution with exponent
(0, 1] for two independent copies of X, denoted as X1 and X2, (32) holds for all y E [0, 1].
7o X1 + (1- X2 - X,
(327)
where d means equality in distribution (57). and implies binomial thinning operation defined as X follows: 7 о X - N2, where N₂ Bernouli(7). and all Njs are jointly independent.
Similar to univariate continuous stable distributions, univariate discrete stable distribution do not have explicit analytical forms, and are typically expressed through their probability generating functions and/or characteristic functions.. The characteristic function a discrete random variable having a stable discrete distribution is given by (328):
E [[e exp (T(E" 1))) ((;;,,T), and (N(== = exp(-r(1- -))))
(328)
where (N r) is the probability generating function of N, 1-1, T> 0, and 0 v 1. We further denote the probability mass function of N as PDs(N = n) (N; A) a standard
Poisson distribution with mean À 24 All discrete stable distribution asymptotically follow a power
law distribution
664
N.5 Multivariate Poisson Distribution
To our knowledge, there are no well-known multivariate generalizations of univariate discrete stable distributions, except for the special case of the Poisson distribution. In this section, we describe popular multivariate extension of the Poisson distribution.
An intuitive way to define multivariate Poisson distribution is represent each random variable in the multivariate Poisson distribution as a sum of independent Poisson random variables [41 42
43 44)
To illustrate, let us construct bivariate Poisson random vector Ñ = [N N²)" where
N1 NI N N₂ = NJ + N
(329)
and NJ, N2. and No2 are mutually independent Poisson random variables with rates Å₁ A2, and A12, respectively. Here, the dependence between Nj and N₂ is expressed through Ni2. with covariance between Ni and N2 being equal to A12 [7]. For dimensions > the Poisson random vector:
Ñ AÑ
(330)
where A is appropriate matrix of 0's and 1's. We decompose A A Aw where A, is a d x (3) submatrix having no duplicate columns, and each of its columns containing exactly i ones and (d - i) zeros [4],, and Ñ [N]... ... No No N22... ... (d-1ja N2 with No ~ Poisson( As....y) A (f1,...,áy) .4) E AS.J E [d"]. Furthermore, the random variables (NI,..., Na- 1)...a} are mutually independent. An intuitive way to think about this multivariate Poisson distribution is to interpret (NT)d J= the "main effect", {Ng (11 as the "two-way covariance effects" and so on until {N? ( (b)....gr)))AA as the d' -way covariance effect in an ANOVA-like fashion 44. For more detailed discussion, see 44 47 42 Note that d' < d. In this work, we denote the d'dimensional random vector N having multivariate Poisson distribution with parameters Ä A> ... Xa d12 ... ... (d' (where d' < d). N Poisson(d (d, d' Ä). Note that for d' = 1, we have that K is collection of independent Poisson random variables, and when both d = d' = 1, we recover the scalar Poisson distribution: Pr(K k) A k No.
specifying up to d'-way covariance effects
78

---

Deriving the p.m.f. of the Multivariate Poisson Distribution
Let Ñ Poisson(d, d'' A), where Ñ a d-dimensional random vector. Then the corresponding
p.m.f. of Ñ is described below:
Let ni' [122 Rid-1)d (d' 1)...) 1)...)), and dæ be the dimension of ñ'' then:
P(N= 1
Σ
II
Ass
x
do
N'EC
(61,82)Egg
Q(n
(331)
IT Aig
where C' = (ni' E No :(a:) H S n4 V E [d]. and
Q II (n; - a
1
$1...
with a being the i-th row of the matrix A' = A2 Aa..
Proof of Ñ ~ (d, d', A) having the p.m.f. described in (331.
We know that Ñ AÑg. where [A1... A.). Let the dimension of No be dNa and Sn {Hi No :k Ak*). Then, using the fact Ñ AÑ the p.m.f. of ÑÑ be expressed using the p.m.f. of No in the following manner:
PN=
Σ P(N ni9).
(332)
Since Ño just collection of mutually independent Poisson random variables, we can write P(N9 = ni as a product of scalar Poisson distributions, i.e.,
P(N9 n9)
II
A
(333)
Substituting (33
into 332
and collecting all e Aupnddyy terms obtain:
P(Ñ-n) n) =E -
1'A Σ II
II
(334)
RP ArESa j=1 1)).
K' where ni [nf ... n and ní contains the rest of the elements n Using the fact that ni E S = π = A we have: ńi = A, + A'n'. Combining this with the fact that A is an identity matrix,
Decompose the matrix A = [A1 A'], where A' = [A .2.... Ag and T [m]]"
we obtain:
i प. A'n's ii ñ A'n
n; (T)"TT n''
(335)
where a' is the i-th row of A' and i E [d]. Substituting 335 into (34).
1 ( d'
P(Ñ=n)= PT
Σ II HEAA
(336)
(n.
WES
where the summation constraint is transformed Syy = n N (a) S n. Viv [d]. The equivalence of the summations over the sets Sk and Sk can be derived by considering that
79

---

ni = - -a and ne 2 0. Collecting all the factorial terms, we get:
Q n²
II
(337)
(n:
a n')! y=2 (f1,..n())E
no "i....
Substituting (337
into 336 we obtain:
P(N n) -
Σ
II
X
(a)
FM)
Q TT).
(338)
W''
R'
LiEA
(1
Taking the term I
Xn
of the summation as it does depend upon n'' we obtain:
P(N -iN=n)=e I"A II
The
Σ
II
II
II
Qū1, (339)
GAA
n'EES
LiEAA
Notice that (a))T ni' contains all the elements of n' which have in their subscript. Hence, expanding
the term II,, X, -aat"" s'' and distributing over the product, we obtain the desired form:
P(Ñ
P'A
Th
Σ
II
Assda di, Aiz
x
D'EC
(i1.12)EAg
Nig IT Aig
II (E
Q(ū,
(340)
80

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The three main results of this work are clearly enumerated in Section (Intro- duction) that demonstrate the main claim of the work which is analytically computing partial information decomposition for numerous systems employing well-known distributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have provided a discussion on the limitations of this work in Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
81

---

Answer: [Yes]
Justification: A formal proof of each theorem is provided in the Appx. F L with clearly stated assumptions. A formal justification of the arguments used in Sec. provided in Appx.[M] Dectionf]diion dissuussss the general proof technique used in proving our main theorems.
We also provide informal discussions on the constructions used for proving theorems in their respective sections, namely Sec. 4 and Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The details of our simulation results are provided in Section 0 followed by more details in Appendix M.3 We have also provided the exact code used for running the simulation study.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
82

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have provided our code with the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guides httpai///uiids//ooeuuuibiissssss onlliicyy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips //nips.ccpput http:liiigguddee//oeeeeeeee
wwwmenssssoniiii ssionpolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have provided all simulation details in Section and Appendix L.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have provided box-plot of our results accompanied by the actual data points used for computing the box-plot in Figure [ stated in Sec. G
The factors of variability are clearly
Guidelines:
The answer NA means that the paper does not include experiments.
83

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: We did not feel the need to provide information on the compute resources required to run our simulation, as our main results are of theoretical nature. Furthermore, all our simulation results can be reproduced on any ordinary personal laptop and do not require any special compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ottsshhhhr//////rrpp....p.///p//
Answer: [Yes]
Justification: The research conducted is mainly theoretical in nature and believe it does not violate the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
84

---

Answer: [Yes]
Justification: While our research is mainly theoretical, we have tried to anticipate how might affect the society negatively and positively.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe our work poses no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided the citation and the corresponding license for the code we used for running our simulation in Section 6.
Guidelines:
The answer NA means that the paper does not existing assets.
85

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing human subjects
86

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
87


Reviews:
[{"review_id": "2nolO8Y8Ix", "reviewer": "Reviewer_Ffcm", "summary": "This paper provides analytical solutions for the Partial Information Decomposition (PID) (specifically, the BROJA PID) for a broad class of continuous distributions. The core proposition of this work lies on the relationship between stable distributions, Markov chains, and the minimisation conditions for Blackwellian PIDs (such as BROJA). In addition to a wealth of novel analytical results, the paper also provides an original and useful link to data fission and data thinning, as well as upper bounds for PID in a broader class of distributions.", "questions": "Why is the restriction of univariate M necessary?\n\nTheorem 1, one of the core results of the paper, has an important \"if\" clause in line 187. What is the intuition behind this condition? When does it hold? Please discuss and provide positive and negative examples.", "limitations": "The authors adequately discuss their limitations at length.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a very strong paper with substantial theoretical results. It fills an important gap in the PID literature, which has so far been (almost, although not fully) confined to either Gaussian or discrete systems. The paper also links PID to other problems in statistics (data fission and data thinning), which grounds the results in a broader context. Although presentation could be improved, the authors do a reasonably good job at presenting a very mathematically involved subject while maintaining some degree of intuition.", "weaknesses": "Despite its title, the paper doesn't _really_ compute any PID analytically. To compute all PID atoms, one needs to calculate the 3 mutual informations in addition to the BROJA unique information. The authors do not show that these quantities can be analytically calculated for the same classes of distributions studied.\n\nSimilarly, the authors only provide sufficient conditions for the BROJA-PID to be zero, but they do not compute it analytically for cases when it is non-zero. This is a very important caveat and, although I still think the paper makes a solid contribution, many of its statements and section titles are somewhat overstated.\n\nI'm aware of the space limitations, but I would still consider it a weakness that the paper doesn't provide much of an illustration of its results. This leaves readers with the impression that many results are within reach, but without any clear intuition of what these results actually are. I would encourage the authors to capitalise more on the benefits they claim, e.g. showing actual analytical PID values from fat-tailed systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s positive review of our work and their constructive criticism which should improve the overall quality of our manuscript.\n\nWe like the reviewer’s suggestion to show analytical PID values/expressions to illustrate our theoretical results. Inspired by this suggestion, we found a pleasing illustration of theorem 1 that utilizes the Cauchy distribution. Particularly, if $M\\sim\\text{Cauchy}(0,\\gamma_M), P(X|M)\\sim\\text{Cauchy}(M, \\gamma_X)$, and $P(Y|M)\\sim\\text{Cauchy}(M, \\gamma_Y)$, then the RI term can be written as $\\min\\\\{ \\log(1+\\gamma_M/\\gamma_X), \\log(1+\\gamma_M/\\gamma_Y)\\\\}$, which is the analog of the the Gaussian PID result of Barrett Phys. Rev. E.’14. Similarly, the non-zero UI term can be written as $\\max\\\\{ \\log(1+\\gamma_M/\\gamma_X), \\log(1+\\gamma_M/\\gamma_Y)\\\\}-\\min\\\\{ \\log(1+\\gamma_M/\\gamma_X), \\log(1+\\gamma_M/\\gamma_Y)\\\\}$. Obtaining the analytical expression of SI turns out to be harder due to I(M;X,Y) not having a “simple” closed-form expression, as P(Y|X) and P(X|Y) are not necessarily Cauchy distributions (in contrast with the Gaussian case where P(X|Y) and P(Y|X) continue to be Gaussian). Regardless, we will try to find informative illustrations that facilitate understanding of our theoretical results, e.g., by providing examples of simple closed-form PID expressions for the systems listed in Appendix B and C, wherever possible.\n\n**Answer to Q1**: It is necessary to restrict M to be univariate as many of the constructions provided in the proofs of theorems 1-7 do not extend to the case where M is multivariate. For example, lemma 10 used in the proof of theorem 2 only works for the case where M is univariate. Similarly, the construction of Q(X’|X) in the proof of theorem 4 only works when M is univariate which we illustrate through a counter-example: Consider the system with vector message $M=[M_1,M_2], P(X|M) =$ Poisson$(aM_1+bM_2)$, and $P(Y|M)=$Poisson$(cM_1+dM_2)$ with a>c and b<d. Then, the binomial thinning operation (used in the proof of theorem 4 to construct the Markov chain) can no longer be used to construct the Markov chain $M\\rightarrow X\\rightarrow Y$ as there exists no $0<p\\leq 1$ such that $Y=X$  o $p$, since no $p$ can simultaneously satisfy $ap=c$ and $bp=d$. A similar situation occurs for convolution-closed distributions where we also rely on a dilation operator akin to binomial thinning.  \n\nIntuitively, univariate M describes a special condition where one of the sources (i.e., X or Y) is always a noisier version of the other. Consequently, constructing Markov chains is comparatively easier for univariate $M$ as one can always construct the noisier source from the less noisy source by adding independent noise, dilating the signal strength, or both. For multivariate $M$, it is not always possible to consider one source as a noisier version of the other source which makes constructing the respective Markov chain (required for showing one of the UI terms is zero) more challenging or even potentially impossible. For example, in the Poisson counter-example system discussed above, we have a>c and b<d, in which case X has access to a less noisy version of $M_1$ but a more noisy version of $M_2$, so hence both X and Y should have access to some unique information about M (X through $M_1$ and Y through $M_2$). Hence, for multivariate M, it is not always possible to show the existence of a Markov chain to obtain the PID terms. \n\nHowever, it is possible to specify majorization conditions (e.g., something akin to a>c and b>d) for multivariate M, where one is still able to obtain PID terms through the Markov chain construction technique used in our work. For example, theorem 3 can be generalized to multivariate M using the full result of lemma 5 of Shang & Poor IEEE Trans. on Info Theory’12. Note that these majorization conditions are more restrictive as they guarantee one of the UI terms to be zero only for specific parameter values of the system compared to the PID results for univariate M, where one of the UI terms is always zero for all system parameter values. We are actively trying to derive these majorization conditions for our theorems, but these conditions are not straightforward to obtain as they require us to construct some matrix-generalization of binomial thinning and new matrix constructions for theorem 2. As a side note, Venkatesh & Schamberg ISIT’21 provides a nice discussion on why M being univariate is essential for jointly Gaussian systems and a corresponding majorization condition for Barrett’s univariate M-Gaussian result for univariate M.\n\n**Answer to Q2**: The “if” clause in theorem 1 accounts for different skewness of the stable family distributions. The parameter $\\beta$ encodes the skewness of the stable distributions, where with $\\beta=0$ indicates no skewness, positive $\\beta$ denotes skewness towards positive reals, and negative $\\beta$ denotes skewness towards negative reals. The intuition behind the condition is that the random variables X and Y, for general stable distributions, also convey information about the random variable M through their skewness in addition to the standard SNR interpretation. Hence, in order for Y to have zero UI about M, X needs to have sufficiently high SNR to overcome the additional information conveyed by Y about M by virtue of its different skewness compared to X. The level of SNR required by X in order to ensure that Y has zero UI is quantified by the inequalities given in the “if” clause of theorem 1. For example, let us assume that SNR of X is 2, SNR of Y is 1, $\\alpha=1$, and $\\beta_X=0$. Then, Y will not have any UI about M if its skewness parameter is  $-½ \\leq \\beta_Y\\leq ½$. Hence, in order for Y to have non-zero UI about M, its skewness must be either bigger than ½ or smaller than -½.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29DImQSEJJ", "reviewer": "Reviewer_2Gx9", "summary": "The paper provides an analytical way of evaluating Partial Information Decomposition (PID) of the BROJA type (BROJA-PID). PID is a framework used in neuroscience and other fields to analyze how two sources of information interact to affect a target. The primary challenge addressed by the paper is the difficulty in computing BROJA-PID, which involves a complex optimization problem over probability distributions. This difficulty is avoidable when the sources and the target obey the Gaussian distribution. By considering the reason why this avoidance is possible, the authors clarify a sufficient condition (certain affine structures are assumed when conditioned on the target) where the analytical evaluation of PID is possible for a larger class of stable distributions. This result is generalized for convolution-closed distributions and a subclass of the exponential family, to show a link with data thinning and data fission. Furthermore, an upper bound of the PID for convolution-closed distributions, which is analytically evaluated and is applicable in generic situations, is derived based on the invented technique. Numerical simulation is conducted to demonstrate the efficiency of the proposed formula.", "questions": "My questions are mainly about the practical implementation/application of the presented result:\n1. How do you recommend handling situations where the underlying distributions are not exactly known and only empirical data is available?\n2. Could you clearly state the situations where your analytical formulas have a clear advantage over numerical methods?\n3. How do the theoretical results apply to high-dimensional data sets, and could you provide an example of such an application?", "limitations": "I think the authors addressed the limitations of the work adequately. For improvement, it is fine to answer and resolve the above equations and weak points.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tInnovative Theoretical Contributions: The paper extends the known analytical PID results from Gaussian distributions to a broader class of stable distributions. This is a significant theoretical advancement as it provides new insights into the information structure of various well-known distributions. An analytical upper bound for PID for convolution-closed distributions is also useful for practical applications.\n2.\tPractical Relevance: The results are particularly relevant to neuroscience and related fields, where PID is used to understand complex interactions in the systems.", "weaknesses": "1.\tAccessibility: The paper is highly theoretical, which might limit its accessibility to a broader audience who may not have a strong background in statistics and information theory.\n2.\tLimited Scope of Applications: While the paper focuses on extending analytical results to stable and convolution-closed distributions, it does not address the applicability of these results to real-world data extensively. The results rely on specific assumptions about the underlying distributions. The practical applicability of these results depends on how well these assumptions hold in real-world scenarios, and more discussions about this point are desired.  Also, some more empirical validation on the real-world applications would enhance the impact of the present work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review of our work. We have provided the replies to the queries raised by the reviewer below:\n\n**Reply to Q1**: Our theoretical results can be applied to empirical data by assuming that the data can be reasonably modeled by one of the distributions discussed in this work (e.g., Poisson, uniform, etc.). The validity of this assumption could be confirmed through, e.g., a goodness of fit test such as Kolmogorov–Smirnov test. After confirming the assumptions on the distribution of the data hold, the corresponding distribution parameters can be estimated, e.g., through maximum likelihood estimation. If the estimated parameters reasonably follow the assumptions of  their corresponding system model, then we can use the corresponding theoretical result to infer that one of the UI terms in the PID is zero. Consequently, the problem of PID estimation can be reduced to estimating I(M;X,Y), I(M;X), and I(M;Y), which can be estimated by using plugin estimators or general-purpose mutual information estimators such as MINE (Belghazi et al. ICML’18) or Kernel Density Estimators (KDE) (Moon et al. Phys. Rev. E.’95). In case a convolution-closed distribution can reasonably model the data, the upper bound presented in Section 6 can also be used to approximately calculate PID atoms if the estimated parameters do not follow the assumptions of the systems discussed in theorem 1-7 but do follow the looser assumptions required by the upper-bound.\n\n**Answer to Q2 and Q3**:  We combined the replies to questions 2 and 3 raised by the reviewer as the replies to both questions are related. First, theorems 2, 3, and 5 discuss the vector-case analysis, where X and Y are assumed to be of arbitrary dimensions, although M is still assumed to be univariate. Theorem 5, which discusses the multivariate Poisson system, is particularly relevant as the Poisson distribution is a common choice for modeling neural spikes. Many neuroscientific experiments record spiking activity from multiple neurons across different brain regions under different behavior conditions, resulting in high-dimensional datasets.  The behavior condition can often be modeled as a univariate discrete random variable encoding different behavior conditions (represented as $M$ in our setup). The neuron spiking activity in different brain regions can be modeled using multiple Poisson random variables. E.g., the vectors X and Y can represent the activity of neurons in brain regions 1 and 2, respectively. For such systems, theorem 5 could be used to calculate the PID of neurons present in different brain regions with respect to the behavior variable $M$. \n\n*Advantage*: The main advantage of using theorem 5 and modeling neural data using the Poisson distribution is data-efficiency. As these datasets tend to be high-dimensional, general-purpose PID estimators must first estimate the high-dimensional distributions P(X|M) and P(Y|M) and then optimize over these estimated high-dimensional distributions to obtain the PID. This procedure can be data inefficient at high dimensions. In contrast, computation of PID atoms using theorem 5 under the Poisson system's assumptions only requires estimating the rate parameters of the different Poisson variables, which is more data-efficient at high dimensions. Note that the corresponding mutual information terms required for calculating PID can be estimated using plug-in estimators as they only depend on P(M) (already known by design of the experiment) and the estimated Poisson rate parameters. Note that to use our method, the data must follow the system model assumptions reasonably well. Our theoretical results could also be useful in calculating PID of fat-tailed distribution systems. Most of the existing PID solvers assume discrete or Gaussian distribution for calculating PID, both of which would be poor assumptions for modeling fat-tailed distributions frequently used to model financial data (Jansen et al. the review of economics and statistics ’91, Haas et al. ‘07). Pakman et al. NeurIPS'21 uses a copula method for estimating PID by optimizing an upper bound, which could potentially be used to estimate the PID of fat-tail systems. However, if the system model assumptions of our theorems hold, then our results can provide an exact PID of the fat-tailed systems instead of Pakman et al.'s approximate solution. \n\nWe also want to point out a *practical benefit* of our theoretical results, which is providing numerous systems on which the performance of different numerical estimators can be benchmarked. Previously, the ground-truth PID was known for very few systems (primarily Gaussian or some hand-designed toy systems), limiting the evaluation of numerical PID estimators as these estimators could not compare their numerical estimates with the ground-truth. With our theoretical results, the numerical PID estimators can extensively test their performance on a large number of systems where the ground-truth PID is known or can be easily calculated. \n\nWhile not directly discussed in this work, another interesting avenue to explore using our theoretical results is to numerically estimate PID for more general system models by extending the method used in Venkatesh et al. NeurIPS’23. Venkatesh et al. NeurIPS’23 builds on existing analytical Gaussian PID results by assuming that Q(Y|M,X) of the minimizing distribution Q(M,X,Y) remains Gaussian for general Gaussian systems. This assumption can be seen as a special case of assuming Q(Y|M,X) of the minimizing distribution Q(M,X,Y) having the form $G(\\epsilon\\delta, (1-\\epsilon)\\delta, x)$ (see Sec. 5 of the manuscript) for convolution-closed distribution. Hence, there is potential for extending the numerical techniques developed in Venkatesh et al. NeurIPS’23 to other convolution-closed distributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QDjZSSdTv0", "reviewer": "Reviewer_vKRJ", "summary": "The paper focuses on analytically expression of the BROJA-PID [12, 33]. By considering the stable distribution family, the authors are able to generalize the previous Gaussian PID result [17] (one of the uniqueness always vanishes for BROJA-PID) to a wider class of distributions. An upper bound for convolution-closed distributions is also given.", "questions": "1. In line 131, it is stated that the main focus of this work is to analyze the cases where (2) is analytically solvable. What is the definition of \"analytically solvable\"?\n2. In section 4, the two systems are constructed in a symmetric manner. How do the results compare to [Niu and Quinn, 2020]?\n3. Do the analytical solutions only hold for BROJA-PID or do they also hold for a wider range of PID measures?\n\n[Niu and Quinn, 2020] Niu, Xueyan, and Christopher J. Quinn. \"Synergy and redundancy duality between gaussian multiple access and broadcast channels.\" 2020 International Symposium on Information Theory and Its Applications (ISITA). IEEE, 2020.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The BROJA-PID [12, 33] has been widely adopted due to the simplicity of its optimization form. This paper extend a famous Gaussian PID result [17] (one of the uniqueness always vanishes for BROJA-PID) to a wider class of distributions. This result simplifies the computation of many PID problems and provides a test bed for numerical PID estimators.", "weaknesses": "The proposed analytical extension is only for the BROJA-PID [12, 33]. There exists other PID measures that diverges from the BROJA-PID even in the bivariate case. The paper does not provide sufficient discussion on this. Further, the title of the paper is misleading, as Partial Information Decomposition is not just BROJA decomposition. The authors should consider revising the title and make this clear in the beginning of the paper. Also, if analytical expression is already available, why \"computing\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive endorsement of our work. We aim to improve our work based on the suggestions made by the reviewer. The answers to the specific queries of the reviewer are provided below:\n\n**Answer to Q1**: The definition we implicitly adopted for “analytically solving” equation (2) is that we are able to provide an explicit construction of the minimizing distribution $Q^*(M,X,Y)$ that minimizes equation (2). In all our proofs, we have provided an explicit construction of the minimizing distribution. We aim to appropriately modify the manuscript so as to clearly and explicitly define this concept. We apologize for any inconvenience/confusion it may have caused the reviewer.\n\n**Answer to Q2**: We compared the two systems proposed in Section 4 (defined on lines 194 and 202) with the results presented in Niu & Quinn ISITA'20. We found that there are subtle but significant differences between the systems we are analyzing and the systems analyzed in Niu & Quinn ISITA'20:\n\n1. The main difference is that Niu & Quinn ISITA'20 analyzes the PID within a Gaussian Broadcast Channel (BC) with the PID within a Gaussian Multiple Access Channel (MAC) and obtains a duality between the PID terms of these *two* different systems, i.e., Synergistic Information (SI) of MAC has the same value as the Redundant Information (RI) of BC and vice-versa. In contrast, we are analyzing the PID of a *single* system consisting of a message (M) passed through two different BCs, i.e., $P(\\mathbf{X}|M)$ and $P(\\mathbf{Y}|M)$, where $\\mathbf{X}$ and $\\mathbf{Y}$ are vectors. The theoretical results for systems 1 and 2 in Section 4 can be thought of as the generalization of Property 1 in Niu & Quinn ISITA’20 for stable distributions, which can be used to compute the UI, RI, and SI w.r.t. $M$ between these two different BCs.\n\n2. One can potentially construct an equivalent MAC model for stable distributions inspired by the BC-type models discussed in Section 4. However, it is not immediately apparent to us how to show the duality discussed in theorems 4.1 and 4.2 in Niu & Quinn’20 for these stable distribution versions of MAC and BC. Particularly, the proof of theorem 4.1 relies on the closed-form expressions of $I(X_1; Y |X_2)$ and $I(X;Y)$, which exist for Gaussian distributions but are not readily available for all stable distributions. Verdu Entropy’23 does provide a similar log(1+SNR)-type expression for the Cauchy distribution which could be potentially used to show similar duality results for the same, but we are not sure if the duality results would continue to hold for all stable distributions. Even among the stable distributions, the Gaussian distribution does tend to be quite special as the only continuous stable distribution with finite moments. \n\n**Answer to Q3**: Yes, our analytical solutions (theorem 1-7) hold for a wider range of PID measures. Particularly, our results hold for two broad classes of PID definitions: PID definitions satisfying assumption ($\\*$) in Bertschinger et al. Entropy’14 and Blackwellian PIDs (see Appendix B-C of Venkatesh & Schamberg ISIT’21 for a deeper discussion on differences between these two families of PID definition). Some examples of PID definitions satisfying assumption ($\\*$) in Bertschinger et al. Entropy’14 are the Williams & Beer’10 PID definition, the PID definition proposed in Harder et al. Phys. Rev. E.’13, and I-PID proposed in Venkatesh et al. ISIT’23. Similarly, an example of a Blackwellian PID is the $\\delta$-PID discussed in Venkatesh & Schamberg ISIT’21. We also point the reviewer to Section III-F of Venkatesh et al. ISIT’23 which discusses a family of Blackwellian PID definitions for which our theorems are also applicable.\n\nWe invoke the argument presented in the proof discussed in Section 4.2 of Barrett Phys. Rev. E.’14 to show that our results hold for any PID definition satisfying assumption ($\\*$) in Bertschinger et al. Entropy’14. Barrett’s argument relies on the key observation that the UI calculated using BROJA-PID upper bounds the UI calculated using any other PID definition satisfying assumption ($\\*$) in Bertschinger et al. Entropy’14. Consequently, if one of the UI atoms of the BROJA-PID is zero then the corresponding UI atom of the other PID definitions must also be zero due to BROJA PID upper-bounding them and the non-negativity of the PID atoms. Since the remaining PID atoms are calculated using equation (1) in our manuscript, it must be the case that the PID atoms calculated using any PID definition satisfying assumption ($\\*$) must be the same as the PID atoms calculated using BROJA-PID whenever one of the UI terms of BROJA-PID is zero. \n\nBy utilizing the definition of a Blackwellian PID, we can demonstrate that our results also hold any Blackwellian PID. For a Blackwellian PID, a UI atom is zero iff it is possible to construct the Markovian structure between the random variables (i.e., $M\\rightarrow X\\rightarrow Y$ or $M\\rightarrow Y\\rightarrow X$) while preserving their pairwise marginals (see Venkatesh et al. ISIT’23 for more details), which is indeed the case for our results. Consequently, theorem 1-7 can be used to show that all Blackwellian PID would have one of the UI atom as zero.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iX8aHkOrga", "reviewer": "Reviewer_zuEu", "summary": "This paper studies analytical computing of Partial Information Decomposition (PID) when the underlying distribution is stable distribution and convolution closed distribution. In particular, the paper considers BROJA-PID framework consisting of minimization problem of conditional mutual information over a set of distributions. Under this framework, computing PID requires to solve the minimization problem.  The minimization problem gives solution zero whenever the underlying random variables are formed Markov chain. \nThe main contributions of this paper are: 1) generalizing the Gaussian PID result to broader distributions by finding sufficient condition under which the mutual information in the minimization problem achieves zero; 2) connecting data fission and thinning results by showing some convolution closed distribution can be useful in construction of a solution to the minimization problem.\nThe authrors also provide list of system examples for which their results are applicable.", "questions": "I am curious how tight the proposed upper bound for other system models except Poisson distribution. Can the authors provide more simulation results under various settings? Particularly, does the upper bound work well for systems that does not include Markov chain property (zero unique information) or the assumptions in the paper?\n\nMoreover, can the authors provide comparisons with other numerical estimators for PID under various settings?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper generalizes the result of PID computing for joint Gaussian distribution to a broader distribution family, some stable distributions and convolution closed distributions. The results in the paper provide a list of system models where the unique information is zero by exploring the properties of Markov chain and by constructing a distribution having Markov chain. \nMoreover, the paper provides a connection to the data fission/thinning and a way to construct upper bound using their results, which might be useful in computing PID for some systems.", "weaknesses": "While this paper extensively explores a sufficient condition (Markov chain) for computing the unique information, I am not sure how much this result contributes on the computing PID for general systems. Some of my concerns are:\n1) Most of results come from the construction of distributions having Markov chain property. In my opinion, the results are just examples of underlying system models that induce Markov chain. \n2) The assumption that the underlying conditional distributions should somehow linearly depend on the target random variable. This assumption seems quite strong in general system models. \n3) Although the results are claimed as computing PID, the results are not applicable for general nonzero unique information. I think zero unique information case seems very restrictive in general system models, so the results are applicable very few cases. \n4) Analysis of the proposed upper bound is not sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments.\n\nWhile we agree that the assumption of linearity can be strong, this assumption has been made in previous works, e.g., Venkatesh et al. NeurIPS’23 uses a Gaussian assumption with linear dependence on M to calculate the PID of experimental neural data. \n\nBased on the reviewer’s suggestion, we have analyzed our proposed upper-bound for two more systems, namely those defined by negative-binomial and binomial distributions (see Fig 1a in the attached document). The setup is the same as the Poisson experiment (Section 6 of the manuscript). The system models used for negative-binomial and binomial are listed in Appx. B of the manuscript. We analyzed 10 different function pairs and considered M to have either  2 and 4 outcomes. We find that the upper-bound continues to be reasonably tight. We used the Poisson experiment in the manuscript to analyze systems that have non-zero UI in both X and Y. To do so, we sub-selected the Poisson systems which have both non-zero UI (shown in Figs 1c and 1d in the attached document), and showed the respective errors for these non-zero UI systems in Fig 1e. We find that the upper-bound continues to hold reasonably well for these systems having both UI terms as non-zero. \n\nIt is not possible to apply the upper-bound when the assumptions in the paper are not satisfied. Particularly, for the Poisson experiment, the distribution shown in equation (316), given by Binomial$(Z, (f_2(M)-\\delta^Y_{bias})/(f_1(M)-\\delta^X_{bias}))$ which is used in constructing the upper-bound, would not be well-defined as it requires $(f_2(M)-\\delta^Y_{bias})/(f_1(M)-\\delta^X_{bias})$ to be less than 1. There is a naive extension to the upper-bound in case the assumptions are not satisfied, which is to construct the upper-bound distribution individually for each outcome of M. For example, assume M takes two values 0 and 1. Let us further assume that $(f_2(M=0)-\\delta^Y_{bias})/(f_1(M=0)-\\delta^X_{bias}) <1$ and $(f_2(M=1)-\\delta^Y_{bias})/(f_1(M=1)-\\delta^X_{bias}) >1$. Then, for M=0, we will construct the upper-bound distribution as detailed in the manuscript, and for M=1, we can swap all the variables pertaining to X with all variables pertaining to Y, which would change the distribution in equation (316) to  Binomial$(Z, (f_1(M=1)-\\delta^X_{bias})/(f_2(M=1)-\\delta^Y_{bias}))$ for M=1 case, ensuring that this binomial distribution continues to be well-defined. Note that we have no theoretical reason to assume that this upper-bound should be tight. We show a similar empirical analysis for the Poisson experiment for 5 function pairs which do not follow the assumptions of the upper-bound discussed in this work (see Fig 1b. in the attached document). We find that this upper-bound is tight for some functions and loose for others.\n\nGiven the time-constraints of author-rebuttal, we are unable to provide comparisons with other numerical estimators. Regardless, all our experiments are performed on discrete distributions for which the optimization problem in (2) is convex, and any numerical estimator used would converge to the global minimum. We choose Makkeh et al. Entropy'18 as the numerical solver as it implements an exponential cone solver to numerically solve equation (2) and empirically tended to be the fastest method for our experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their thoughtful evaluation of our manuscript. We are grateful for the instructive comments and suggestions provided by all the reviewers that would invariably improve our work. Some common remarks by reviewer Ffcm and reviewer vKRJ regarding the presentation of our results seem to stem from us not explicitly defining what we meant by "analytically computing PID". To address these common comments, we will rephrase our manuscript to more carefully state our claims based on their suggestions, e.g., by explicitly acknowledging that all our theoretical results focus on the cases where one of the UI terms is zero, and discussing in more detail the PID definitions for which our results hold. 

The attached document contains the additional analysis on the upper bound suggested by the reviewer zuEu.

Meta-review: This paper provides an analytical means to calculate the bivariate Bertschinger-Rauh-Olbrich-Jost-Ay (BROJA) partial information decomposition (PID) for a wide class of distributions. The key is the notion of stable distributions (or equivalently, the so-called "degradation channels"). With this notion, for three random variables $M,X,Y$ if the marginals of $M,X$ and $M,Y$ are such that there exist serially concatenated channels $M\to X$ and $X\to Y$ to reproduce these marginals, one can assure that the unique information $UI(M;Y\backslash X)$ vanishes, which then allows us to calculate BROJA PID from the values of mutual information $I_P(M;X)$, $I_P(M;Y)$, and $I_P(M;X,Y)$ via solving the linear equation (1).

The ratings/confidences from the four reviewers were 7/4, 7/2, 7/5, and 5/3, with average well above the acceptance threshold. All the reviewers agreed that the contributions of this paper will widen our understanding about when one can evaluate BROJA PID from a set of mutual information values. Despite some weaknesses raised by the reviewers, this paper contains a solid theoretical contribution about PID.
Decision: Accept (poster)