Title: Analytically deriving Partial Information Decomposition for affine systems of stable and convolution-closed distributions
Venue: NeurIPS 2024
Accepted: yes
Keywords: Partial Information Decomposition

Neuroscience

Multimodal learning

Analytical PID
Abstract: Bivariate partial information decomposition (PID) has emerged as a promising tool for analyzing interactions in complex systems, particularly in neuroscience. PID achieves this by decomposing the information that two sources (e.g., different brain regions) have about a target (e.g., a stimulus) into unique, redundant, and synergistic terms. However, the computation of PID remains a challenging problem, often involving optimization over distributions. While several works have been proposed to compute PID terms numerically, there is a surprising dearth of work on computing PID terms analytically. The only known analytical PID result is for jointly Gaussian distributions. In this work, we present two theoretical advances that enable analytical calculation of the PID terms for numerous well-known distributions, including distributions relevant to neuroscience, such as Poisson, Cauchy, and binomial. Our first result generalizes the analytical Gaussian PID result to the much larger class of stable distributions. We also discover a theoretical link between PID and the emerging fields of data thinning and data fission. Our second result utilizes this link to derive analytical PID terms for two more classes of distributions: convolution-closed distributions and a sub-class of the exponential family. Furthermore, we provide an analytical upper bound for approximately calculating PID for convolution-closed distributions, whose tightness we demonstrate in simulation.

Analytically deriving Partial Information Decomposition for affine systems of stable and convolution-closed distributions

Abstract

Bivariate partial information decomposition (PID) has emerged as a promising tool for analyzing interactions in complex systems, particularly in neuroscience. PID achieves this by decomposing the information that two sources (e.g., different brain regions) have about a target (e.g., a stimulus) into unique, redundant, and synergistic terms. However, the computation of PID remains a challenging problem, often involving optimization over distributions. While several works have been proposed to compute PID terms numerically , there is a surprising dearth of work on computing PID terms analytically . The only known analytical PID result is for jointly Gaussian distributions. In this work, we present two theoretical advances that enable analytical calculation of the PID terms for numerous wellknown distributions, including distributions relevant to neuroscience, such as Poisson, Cauchy, and binomial. Our first result generalizes the analytical Gaussian PID result to the much larger class of stable distributions. We also discover a theoretical link between PID and the emerging fields of data thinning and data fission. Our second result utilizes this link to derive analytical PID terms for two more classes of distributions: convolution-closed distributions and a sub-class of the exponential family. Furthermore, we provide an analytical upper bound for approximately computing PID for convolution-closed distributions, whose tightness we demonstrate in simulation.

1 Introduction

Bivariate partial information decomposition 1 (PID) is an information-theoretic framework developed for answering a central inquiry in many neuroscientific and machine learning studies: how do two sources, X and Y , jointly process information about a target M ? PID answers this question by quantifying the M -specific information contained in different interactions between X and Y . Specifically, it decomposes the total information X and Y have about M into four components: (i) the information about M contained uniquely in X , (ii) the information about M contained uniquely in Y , (iii) the redundant information about M contained in both X and Y , and (iv) the synergistic information about M which arises from the interaction between X and Y .

The PID terms offer novel insights for understanding interactions within complex systems, particularly in neuroscience. For instance, PID has been used to understand the firing patterns of grid cells [1, 2] and to study the flow of information in the visual cortex [3]. The following list of works [1, 3, 4, 5, 6, 7, 8] demonstrate the application of PID in studying diverse neuroscientific questions.

Beyond neuroscience, PID has found applications in multimodal learning for interpreting model predictions [9, 10], fair machine learning for defining and quantifying bias [11, 12, 13, 14], and understanding financial markets [15].

However, the primary constraint hindering broader adoption of PID is the difficulty of computing and estimating the PID terms. BROJA-PID [16], a widely applied PID framework, requires solving a constrained minimization problem over a set of probability distributions (see Sec. 2). This minimization problem can pose a considerable challenge, particularly when the underlying distributions are continuous. As a result, several recent works have been dedicated to providing numerical algorithms that solve the aforementioned minimization problem exactly or approximately [3, 9, 17, 18, 19, 20], with more emphasis on the case of discrete distributions.

Despite significant progress in numerically calculating PID (typically through discrete approximations), very few works exist on analytically 2 calculating PID. As of this writing, the only known analytical PID expressions exist for jointly Gaussian M , X , and Y [21, 22]. A fundamental property of the Gaussian system is that one of the unique information (UI) terms in its PID is guaranteed to be zero . This property greatly simplifies the computation of PID for Gaussian systems, as the rest of the PID terms can be easily derived by solving desirable linear equations specified in many PID frameworks (see Sec 2), bypassing the need for optimizing over distributions (see Sec. 3).

In this work, we show that numerous systems of random variables M , X , and Y expressing a particular 'affine dependence structure' also exhibit the same fundamental property that at least one of the UI terms is zero. Consequently, we expand significantly on the existing Gaussian PID result by using this fundamental property to analytically compute PID for various systems of M , X , and Y employing well-known distributions such as Poisson, exponential, gamma, beta, negative binomial, multinomial, Cauchy, Lévy-stable and more. The main contributions of this work are:

We extend the Gaussian PID result to a much larger class of distributions, known as the stable distribution family [23, 24] in Sec. 4. These results provide the first known analytical PID for fat-tailed distributions.

We highlight a theoretical link between PID calculation and the fields of data thinning [25] and data fission [26] in Sec. 5. We utilize this link to derive analytical PID terms for two more distribution families: convolution-closed distributions [25] and certain exponential family distributions [26].

For convolution-closed distributions, we further derive an analytical upper bound on the objective of the minimization used for computing BROJA-PID. We use this upper bound to approximately compute PID for systems of M , X , and Y having a non-affine dependence structure. We show the goodness of our approximation by a simulation study in Sec. 6.

2 Background

Notation : We denote the set of all natural numbers, real numbers, and positive real numbers as N , R , and R + , respectively. Vectors are denoted by bold-faced font and an arrow, and matrices are denoted by bold-faced font. Define N 0 = N ∪ { 0 } . Let I d be the identity matrix of size d × d and [ d ] = { 1 , . . . , d } ∀ d ∈ N . We denote ⃗ 1 d and ⃗ 0 d as d -dimensional vectors having all elements as 1 and 0 , respectively. For brevity, the probability notations of the form P ( A | B ) and P ( A ) are always understood to be as P ( A = a | B = b ) and P ( A = a ) , respectively. The general term 'distribution' is used to refer to both probability density function (p.d.f.) and probability mass function (p.m.f.) (whichever is appropriate depending upon the context). The symbol A ⊥ ⊥ B | C denotes that A and B are conditionally independent given C , and similarly, A ⊥ ⊥ B implies A and B are independent. The L p -norm is denoted by ∥ ( · ) ∥ p ∀ p ∈ [0 , ∞ ) , and | ( · ) | denotes the absolute function.

PID Background : Suppose M,X,Y are random variables with joint distribution P ( M,X,Y ) . According to [16, 27], three desirable equalities for a bivariate PID are as follows:

Here, I P ( M ; X ) and I P ( M ; Y ) denote the mutual information [28] between M and X , and M and Y , respectively, under the distribution P ( M,X,Y ) . Similarly, I P ( M ; [ X,Y ]) denotes the joint mutual information between M and [ X,Y ] under the distribution P ( M,X,Y ) . UI ( M ; X \ Y ) and UI ( M ; Y \ X ) are the unique information about M in X and Y , respectively. SI ( M ; X,Y ) and RI ( M ; X,Y ) denote the respective synergistic and redundant information shared between X and Y about M . The linear system defined in (1) contains four variables in three equations. Hence, only one of UI,RI , or SI need be defined to evaluate all four PID terms. Proposing a suitable definition of PID is the focus of much research [16, 27, 29, 30, 31, 32, 33, 34, 35] and all our theoretical results are applicable for any Blackwellian PID [22, 36] or a PID definition satisfying the assumption ( ∗ ) of Bertschinger et al. [16] (see Appx. B for a formal justification). For example, our results are applicable for the PID definitions proposed in [16, 22, 27, 29, 37]. In this work, we discuss our results in the context of the BROJA-PID [16, 38], which is the definition of PID adopted in previous works [3, 9, 20]:

where ∆ P = { Q ( M,X,Y ): Q ( M,X )= P ( M,X ) , Q ( M,Y ) = P ( M,Y ) } and I Q ( M ; X | Y ) is the conditional mutual information under the distribution Q ( M,X,Y ) . Note that the minimizing distributions for both problems shown in (2) are the same [16], and consequently, both minimization problems are equivalent. We refer the reader to [39] for a review on PID. In this work, we define 'analytically calculating PID' as analytically solving (2) by providing an explicit construction of the minimizing distribution Q ∗ ( M,X,Y ) that minimizes (2). Note that only the distributions Q ∗ ( M,X,Y ) and P ( M,X,Y ) are needed to compute the BROJA-PID terms.

We briefly describe two distribution families used in Sec. 4 and Sec. 5 (see Appx. N for more details).

Stable distribution family : Stable distributions are a family of distributions that naturally arise in the context of generalized central limit theorems. Some well-known members of this family are the Gaussian, Cauchy, Poisson, and Lévy distributions. A defining feature of stable distributions is that the sum of two independent copies of a random variable X , denoted as X 1 and X 2 , has the same distribution as a translated and scaled version of X [23, 24, 40]. In this work, we consider five sub-classes of stable distributions:

(i) Continuous stable distributions are parameterized through four parameters: stability parameter α ∈ (0 , 2] , skewness parameter β ∈ [ -1 , 1] , scale parameter γ ∈ (0 , ∞ ) , and location parameter µ ∈ R . We denote its p.d.f. as p CS ( α, β, γ, µ ) . Note that all continuous stable distribution (except Gaussian) are fat-tailed.

(ii) Independent component multivariate stable distributions describe the distribution of ⃗ X consisting of d independent random variables { X j } d j =1 , such that each X j ∼ p CS ( α, β j , γ j , µ j ) . We denote its p.d.f. as p CS -IC ( α, ⃗ β , ⃗ γ , ⃗ µ ) .

(iii) Elliptically-contoured multivariate stable distributions are the distributions of continuous stable random vectors whose p.d.f. has elliptical contours, e.g., the multivariate Gaussian distribution. We denote its p.d.f. as p CS -EC ( α, Σ , ⃗ µ ) . Here, Σ is a positive definite matrix, ⃗ µ ∈ R d , and α ∈ (0 , 2] .

(iv) Discrete stable distributions are the discrete analogues of the continuous stable distributions. The p.m.f. of discrete stable distributions, denoted as P DS ( ν, τ ) , are parameterized through two parameters: rate parameter τ > 0 and exponent 0 < ν ≤ 1 . Discrete stable distributions do not have well-known multivariate generalizations except the Poisson distribution.

(v) Multivariate Poisson distribution : We use the multivariate Poisson distribution proposed in [41, 42, 43, 44], denoted as Poisson ( d, d ′ , ⃗ Λ ) , which represents each random variable in the random vector ⃗ N as a sum of independent Poisson random variables. Formally, ⃗ N ∼ Poisson ( d, d ′ , ⃗ Λ ) if ⃗ N = A ⃗ N g , where A = [ A 1 . . . A d ′ ] . Here, A i denotes a d × ( d i ) submatrix having no duplicate columns, where each of its columns contain exactly i ones and ( d -i ) zeros [44]. The vector

with mutually independent N g i 1 ...i j ∼ Poisson ( λ i 1 ...i j ) ∀ ( i 1 , . . . , i j ) ∈ A d j , j ∈ [ d ′ ] , where A d i = { ( j 1 , . . . , j i ): j 1 < j 2 < . . . < j i , and j 1 , . . . , j i ∈ [ d ] } , e.g., A 3 2 = { (1 , 2) , (1 , 3) , (2 , 3) } . Note that d ′ ≤ d , and ⃗ Λ = [ λ 1 . . . λ d -( d ′ -1) ...d ] T .

We refer the reader to [23, 24, 40, 42, 44, 45, 46, 47] for more details on stable distributions.

Convolution-closed distributions : Convolution-closed distributions form a large class of distributions that are closed under convolution in some parameter δ . Formally, we define the convolutionclosed distribution as follows: Let F D denote a family of distributions, where each member distribution f ( δ ) ∈ F D is indexed by a parameter δ ∈ D . Consider X 1 ∼ f ( δ 1 ) , X 2 ∼ f ( δ 2 ) , and X 1 ⊥ ⊥ X 2 for some δ 1 , δ 2 ∈ D . Then, F D is convolution-closed in the parameter δ if

where ∗ denotes the convolution operator. Many well-known distributions can be considered convolution-closed in some parameter [25], such as the Poisson distribution, Gaussian distribution, gamma distribution, etc. Table 1 in [25] lists various examples of convolution-closed distributions.

3 A sufficient condition for computing PID terms analytically

The main focus of this work is to analyze the cases where (2) is analytically solvable. A sufficient condition for solving (2) is to show that ∆ P contains a distribution Q MC ( M,X,Y ) with the Markovian structure M → X → Y or M → Y → X . We briefly discuss the argument justifying this sufficient condition. Consider the case where Q MC ( M,X,Y ) ∈ ∆ P and has the Markovian structure M → X → Y . First, we note that UI ( M ; Y \ X ) = min Q ∈ ∆ P I Q ( M ; Y | X ) ≥ 0 due to the nonnegativity of conditional mutual information [28]. Second, Y ⊥ ⊥ M | X due to the Markovian structure of Q MC ( M,X,Y ) , which implies that I Q MC ( M ; Y | X ) = 0 (conditionally independent random variables have zero conditional mutual information [28]). Hence, Q MC ( M,X,Y ) achieves the lower bound of zero for the minimization problem min Q ∈ ∆ P I Q ( M ; Y | X ) , showing that Q MC ( M,X,Y ) indeed minimizes (2). A similar argument can be made for the case when Q MC ( M,X,Y ) has the Markovian structure M → Y → X . Proposition 1 in Appx. A formalizes the above argument.

The above sufficient condition provides an easy way to analytically calculate the PID terms, as it ensures one of the unique information terms is always zero. Consequently, the remaining PID terms can be calculated by substituting zero for the appropriate unique information term in (1) and solving the resultant linear system. Hence, if applicable, the above sufficient condition considerably simplifies the calculation of PID terms by circumventing the need for optimizing over a set of distributions. Surprisingly, many well-known distribution families allow intuitive constructions of P ( M,X,Y ) , for which the above sufficient condition is applicable. In the following sections, we provide theorems that specify sufficient conditions under which the existence of these Markov chains in ∆ P can be guaranteed for these P ( M,X,Y ) .

4 Computing PID for stable distributions

In this section, we extend existing results for analytical PID computation of jointly Gaussian M , X , and Y [21, 22] to the much larger class of stable distributions. Our results utilize two key observations to provide the generalization of the Gaussian results: (i) We first identify that the analytical computation of PID for jointly Gaussian systems is due to their particular 'affine dependence' structure; (ii) We show that these particular affine dependence structures are not unique to Gaussian systems, but rather extend to many members of the stable distribution family.

For jointly Gaussian M , X , and Y , the conditional distributions P ( X | M )= N ( aM + b, σ 2 X ) and P ( Y | M )= N ( cM + d, σ 2 Y ) are also Gaussian distributions, where their means are an affine function of M and their variances are fixed with respect to M . This particular affine dependence of X and Y on M is the key to the analytical calculation of their PID, as it guarantees existence of a Markov chain Q MC ( M,X,Y ) ∈ ∆ P . Thus, we can apply the sufficient condition described in Sec. 3 to compute the PID terms. We illustrate through an example: consider P ( X | M )= N ( M,σ 2 X ) and P ( Y | M )= N ( M,σ 2 Y ) where σ 2 X < σ 2 Y , and M ∼ P ( M ) for some appropriate P ( M ) . Then, we can explicitly construct Q MC ( M,X,Y ) ∈ ∆ P with the Markovian structure M → X → Y as follows: Choose Q MC ( M,X,Y )= P ( M ) P ( X | M ) Q MC ( Y | X ) where Q MC ( Y | X ) is specified through the addition of independent Gaussian noise, i.e., Y = X + ϵ . Here, ϵ ∼ N (0 , σ 2 Y -σ 2 X ) and ϵ ⊥ ⊥ ( M,X ) . It is easy to verify that Q MC ( Y | M )= N ( M,σ 2 Y ) , which implies that Q MC ( M,Y )= P ( M,Y ) . By construction, Q MC ( M,X )= P ( M,X ) , and hence, Q MC ( M,X,Y ) ∈ ∆ P .

The above example is a special case of a well-known result where, for a scalar M , one can always construct a lower signal-to-noise ratio (SNR) 'Gaussian channel', i.e. P ( Y | M ) in our example, by adding independent Gaussian noise to a higher SNR 'Gaussian channel' [22, 48], i.e. P ( X | M ) in our example. Surprisingly, the technique of adding independent noise to construct Markov chains contained in ∆ P can be extended to P ( X | M ) and P ( Y | M ) as members of stable distributions. Theorems 1 and 4 generalize the above construction for P ( X | M ) and P ( Y | M ) as members of univariate continuous and univariate discrete stable distributions, respectively. Theorems 2, 3, and 5 consider the case of multivariate stable distributions (see Sec. 2). The key technique for proving these theorems is that a Markov chain Q MC ( M,X,Y ) ∈ ∆ P can always be constructed by adding appropriate independent noise to a higher SNR P ( X | M ) to obtain a lower SNR P ( Y | M ) , similar to our above example.

4.1 PID of univariate affine continuous stable system

Theorem 1 can be viewed as a direct generalization of Barrett's Gaussian PID result [21] to stable distributions, showing one of the UI terms is always zero. We begin by formally describing the univariate affine continuous stable system that generalizes the Gaussian system of Barrett's [21]. Let the joint distribution of M,X, and Y , denoted as P ( M,X,Y ) , satisfy the following properties: (i) M ∼ P ( M ) with support set M⊆ R ; (ii) P ( X | M ) and P ( Y | M ) are univariate continuous stable distributions with an affine dependence on M , i.e., P ( X | M )= p CS ( α, β X , γ X , aM + b ) and P ( Y | M )= p CS ( α, β Y sgn ( ac ) , γ Y , cM + d ) , where a, b, c, d ∈ R .

Theorem 1. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) is described by a univariate affine continuous stable system. Without the loss of generality, assume | a | / γ X ≥ | c | / γ Y . If 1 -β Y ≥ ( γ X | c | / γ Y | a | ) α (1 -β X ) and 1 + β Y ≥ ( γ X | c | / γ Y | a | ) α (1 + β X ) , then ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

Proof. See Appx. F for the proof. Here, | a | / γ X and | c | / γ Y are the SNR analogues.

4.2 PID of multivariate affine continuous stable systems

We analyze two multivariate generalizations of Theorem 1 in Theorems 2 and 3. Namely, we consider independent component multivariate stable distribution (where all the components of the random vector are independent) and elliptically-contoured multivariate stable distribution (where the p.d.f. is elliptically contoured, similar to multivariate Gaussian distributions). We construct two systems employing these two sub-classes of multivariate continuous stable distributions and show that one of the UI terms is always zero for these systems. For both cases, denote the joint distribution of M , ⃗ X , and ⃗ Y as P ( M, ⃗ X , ⃗ Y ) , where the support set of M is M⊆ R . The dimensions of M , ⃗ X , and ⃗ Y are 1 , d X , and d Y , respectively.

System 1: The random vectors ⃗ X and ⃗ Y satisfy the following equations:

where ⃗ Z X ∼ p CS -IC ( α, ⃗ 0 d X , ⃗ 1 d X , ⃗ 0 d X ) , ⃗ Z Y ∼ p CS -IC ( α, ⃗ 0 d Y , ⃗ 1 d Y , ⃗ 0 d Y ) , A X and A Y are invertible matrices, ⃗ H X , ⃗ b X ∈ R d X , and ⃗ H Y , ⃗ b Y ∈ R d Y .

Theorem 2. Let System 1 describe the joint distribution P ( M, ⃗ X , ⃗ Y ) of M , ⃗ X , and ⃗ Y . Without the loss of generality, assume ∥ A -1 Y ⃗ H Y ∥ κ ≤ ∥ A -1 X ⃗ H X ∥ κ , where κ = α / α -1 ∀ α ∈ (1 , 2] and κ = ∞ ∀ α ∈ (0 , 1] . Then, ∆ P contains a Markov chain of the form M → ⃗ X → ⃗ Y and UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Proof. See Appx. G for the proof. Here, ∥ A -1 Y ⃗ H Y ∥ κ and ∥ A -1 X ⃗ H X ∥ κ are the SNR analogues.

System 2: The conditional distribution of ⃗ X and ⃗ Y conditioned on M are as follows: P ( ⃗ X | M ) = p CS -EC ( α, Σ X , ⃗ H X M + ⃗ b X ) and P ( ⃗ Y | M ) = p CS -EC ( α, Σ Y , ⃗ H Y M + ⃗ b Y ) , where Σ X and Σ Y are positive definite matrices, ⃗ H X , ⃗ b X ∈ R d X , and ⃗ H Y , ⃗ b Y ∈ R d Y .

Theorem 3. Let System 2 describe the joint distribution P ( M, ⃗ X , ⃗ Y ) of M , ⃗ X , and ⃗ Y . Define Σ -1 / 2 X and Σ -1 / 2 Y as the respective inverses of the matrices Σ 1 / 2 X and Σ 1 / 2 Y which satisfy: ( Σ 1 / 2 X ) T Σ 1 / 2 X = Σ X , and ( Σ 1 / 2 Y ) T Σ 1 / 2 Y = Σ Y . Without the loss of generality, assume ∥ Σ -1 / 2 Y ⃗ H Y ∥ 2 ≤ ∥ Σ -1 / 2 X ⃗ H X ∥ 2 . Then, ∆ P contains a Markov chain of the form M → ⃗ X → ⃗ Y and UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Proof. See Appx. H for the proof. Here, ∥ Σ -1 / 2 Y ⃗ H Y ∥ 2 and ∥ Σ -1 / 2 X ⃗ H X ∥ 2 are the SNR analogues.

4.3 PID of univariate affine discrete stable system

The univariate affine discrete stable system is the discrete counterpart of the univariate affine continuous stable system described in Sec. 4.1. The formal description of the univariate affine discrete stable system is as follows. Let the joint distribution of M,X, and Y , denoted as P ( M,X,Y ) , satisfy the following properties: (i) M ∼ P ( M ) with support set M ⊆ (0 , ∞ ) ; (ii) P ( X | M ) and P ( Y | M ) are univariate discrete stable distributions with an affine dependence on M , i.e., P ( X | M = m )= P DS ( ν, am + b ) and P ( Y | M = m )= P DS ( ν, cm + d ) , where a, b, c, d ∈ (0 , ∞ ) .

Theorem 4. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) is described by a univariate affine discrete stable system. Without the loss of generality, assume a ≥ c . If a / b ≥ c / d , then ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

Proof. See Appx. I for the proof. Here, a and c are the SNR analogues.

4.4 PID of multivariate linear Poisson system

The Poisson distribution is the only the discrete stable distribution with a well-known multivariate extension. Hence, we analyze vector-generalizations of Theorem 4 only for the Poisson distribution. We now describe the multivariate linear Poisson system: Let the joint distribution P ( M, ⃗ X , ⃗ Y ) of M, ⃗ X , and ⃗ Y satisfy the following properties: (i) M ∼ P ( M ) with support set M⊆ (0 , ∞ ) ; (ii) P ( ⃗ X | M )= Poisson ( d X , d ′ X , ⃗ Λ X ) and P ( ⃗ Y | M )= Poisson ( d Y , d ′ Y , ⃗ Λ Y ) , with:

Theorem 5. Let the joint distribution P ( M, ⃗ X , ⃗ Y ) of M , ⃗ X , and ⃗ Y be described by the multivariate linear Poisson system defined above. Without the loss of generality, assume d ′ X ≥ d ′ Y . If ∑ ( i 1 ,...,i j ) ∈ A d X j γ X i 1 ...i j ≥ ∑ ( i 1 ,...,i j ) ∈ A d Y j γ Y i 1 ...i j ∀ j ∈ [ d ′ Y ] , then ∆ P contains a Markov chain of the form M → ⃗ X → ⃗ Y and UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Proof. See Appx. J for the proof. Here, γ X 1 , . . . , γ X d X -( d ′ X -1) ...d X , γ Y 1 , . . . , γ Y d Y -( d ′ Y -1) ...d Y ∈ R + are the SNR analogues.

5 Computing PID using data thinning and data fission strategies

Data thinning and data fission are emerging fields in machine learning and statistics dedicated to studying the procedures of splitting a random variable X into N different component random variables. These splitting procedures provide an attractive alternative to the standard splitting of datasets into training, validation, and test splits for performing cross-validation to select statistical model parameters, as they enable users to perform cross-validation even for the extreme case of a dataset containing a single datapoint. We refer the reader to the recent works of Neufeld et al. [25] and Leiner et al. [26] for a more comprehensive discussion on data thinning and fission, respectively.

The fields of data thinning and data fission share an important theoretical link with analytically calculating PID terms. Specifically, the tools developed for splitting random variables for data thinning and data fission can be readily used to calculate PID terms analytically. To give an intuition, suppose X contains more information about M than Y . Then, we can employ data fission and thinning strategies to decompose X into two components, f 1 ( X ) and f 2 ( X ) , such that f 2 ( X ) follows the same distribution as Y . Then, f 2 ( X ) and Y convey the same information about M as they

are identically distributed and represent the redundant component of the PID terms. Similarly, f 1 ( X ) contains the information uniquely contained in X about M . Thus, f 1 ( X ) represents the unique information term. Theorems 6 and 7 utilize the data thinning and data fission proposed in [25, 26] to construct a Markov chain Q MC ( M,X,Y ) ∈ ∆ P for several systems of random variables M , X , and Y . This allows us to use the sufficient condition discussed in Sec. 3 to compute their PID.

5.1 PID for convolution-closed distribution based on data thinning strategies of Neufeld et al.

Neufeld et al. [25] introduces data thinning for a large family of distributions known as convolutionclosed distributions (see Sec. 2). An attractive property of convolution-closed distributions is that they provide a natural way to define a dilation/thinning operation. Formally, let X ∼ f ( δ ) , then we define X ϵ as the ϵ -dilated version of X if X ϵ ∼ f ( ϵδ ) for some ϵ ∈ (0 , 1) such that ϵδ ∈ D . Furthermore, if we assume (1 -ϵ ) δ ∈ D , then P ( X ϵ | X )= P ( X ϵ | X ϵ + X 1 -ϵ ) , where X 1 -ϵ ∼ f ((1 -ϵ ) δ ) and X ϵ ⊥ ⊥ X 1 -ϵ (see lemma 19 for a formal justification). We denote P ( X ϵ | X = x ) = P ( X ϵ | X ϵ + X 1 -ϵ = x ) as G ( ϵδ, (1 -ϵ ) δ, x ) . This dilation operation forms the basis of data thinning, as it enables X to be split into its dilated components X ϵ and X 1 -ϵ , such that X = X ϵ + X 1 -ϵ , where X ϵ ⊥ ⊥ X 1 -ϵ . We utilize this dilation operation for analytically calculating the PID terms of the following linear convolution-closed system:

Linear convolution-closed system : Let F D be a convolution-closed distribution family as described in Sec. 2. The joint distribution P ( M,X,Y ) of the random variables M , X , and Y describes a linear convolution-closed system if the distributions P ( X | M ) and P ( Y | M ) are defined as follows:

where M is the support of M . Furthermore, we assume δ X m = γδ Y m ∀ m ∈ M for some γ ∈ R + .

Theorem 6. Let the joint density P ( M,X,Y ) of random variables M , X , and Y be described by a linear convolution-closed system. Without the loss of generality, assume γ ≤ 1 . If

then ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

The proof of Theorem 6 is provided in Appx. K. Theorem 6 enables PID calculation for several well-known distributions such as gamma, Poisson, beta etc. (see Appx. C for a (non-exhaustive) list).

5.2 PID for distributions based on data fission strategies of Leiner et al.

Leiner et al. [26] propose a 'conjugate-prior reversal' strategy for splitting a random variable X into two components, f ( X ) and g ( X ) , for certain exponential family distributions. The distributions proposed for performing conjugate-prior reversal provide natural descriptions of P ( M,X,Y ) (specified in theorem 7) for which PID can be calculated analytically. We briefly describe the distributions used in the conjugate-prior reversal strategy. Let X ∼ p exp 1 ( X ) , where

for some appropriately defined H ( · , · ) , A ( · ) , θ 1 and θ 2 . Furthermore, define a random variable Y through its conditional density p ( Y | X = x ) :

for some h ( · ) , T ( · ) , and θ 3 , such that p ( Y = y | X = x ; θ 1 , θ 2 ) is a well-defined distribution. Then, the decomposition terms f ( X ) and g ( X ) are Y and X , respectively, in the conjugate-prior reversal strategy. Furthermore, the marginal distribution of Y is expressed as:

Theorem 7. Let M,X, and Y be random variables having the joint distribution P ( M,X,Y ) . Furthermore, the conditional distribution of X and Y conditioned on M are as follows: P ( X | M = m )= p exp 1 ( X ; θ 1 ( m ) , θ 2 ( m )) and P ( Y | M = m )= p exp 2 ( Y ; θ 1 ( m ) , θ 2 ( m ) , θ 3 ) . Then, ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

The proof of Theorem 7 can be found in Appx. L. The proof essentially stems from observing that the joint distribution Q MC ( M,X,Y )= P ( M ) P ( X | M ) Q MC ( Y | X ) lies in ∆ P , where Q MC ( Y | X )= h ( y ) exp ( x T T ( y ) -θ T 3 A ( x ) ) . A (non-exhaustive) list of well-known distributions for which theorem 7 is applicable is provided in Appx. D. Leiner et al. [26] also discuss some more strategies for performing data fission that do not follow the conjugate-prior reversal strategy. We provide the corresponding results for computing PID for these remaining data fission strategies and for additional miscellaneous distributions in Appx. E.

6 Upper bound for convolution-closed distributions

a

c

b

0.07

0.06

Numerical MI (in bits)

2.0

w.r.t.Numerical MI

4%

4

0.05

1.5

(in bits)

0.04

3

0.03

1.0

2

0.02

0.01

0.5

1

1%

0.00

0.0

1

5

10

15

20

1

5

10

15

20

%

0

1

5

10

15

20

FunctionPairIndex

FunctionPairIndex

FunctionsPairIndex

Several numerical methods, such as [3, 9, 20], approximately solve (2) by considering a smaller constraint set or minimizing an appropriate upper bound for calculating the PID terms. These numerical methods employ general approximations that rely on weak assumptions on the underlying distributions for solving (2), as their goal is to estimate PID for a large class of distributions. However, for specific applications, it is possible to make stronger assumptions on the underlying distributions (e.g., assuming the Poisson distribution for modeling neural spikes). This section illustrates how our theoretical analysis can benefit these numerical algorithms by providing more refined approximations for solving (2) that harness these stronger assumptions. Specifically, we construct an upper bound for the objective of (2) for convolution-closed distributions and show that, under certain assumptions, the upper bound can be analytically minimized over ∆ P in Sec. 6.1. Note that these upper bounds are applicable for more general cases than our theoretical results, as they do not require the sources X and Y to have an affine dependence on M . Consequently, our upper bound is also applicable in cases where both of the UI terms in the PID are non-zero, unlike our results in Sec. 4 and 5 We demonstrate the tightness of our upper bound through a simulation study in Sec. 6.2.

Notations and assumptions : We consider the minimization problem min Q ∈ ∆ P I Q ( M ; [ X,Y ]) for deriving the upper bound, as it was shown to be equivalent to (2) in [16]. The distribution P ( M,X,Y ) for which we will construct our upper bound is specified as follows. The random variable M has support over M , the conditional distributions P ( X | M ) and P ( Y | M ) are members of some convolution-closed distribution family F D , and there exists some δ X bias , δ Y bias ∈ D such that

The above assumptions ensure that X and Y can always be decomposed into new random variables X ′ , Y ′′ , n X , Y ′ , and n Y (see (10)). We utilize these decomposed random variables to construct our upper bound. Hence, our upper bound is only applicable for systems satisfying (9). The above assumptions describe a large class of systems. For example, in the Poisson case, our upper bound is applicable for any P ( M,X,Y ) having P ( X | M ) = Poisson ( f 1 ( M )) and P ( Y | M ) = Poisson ( f 2 ( M )) , as long as f 1 ( M ) ≥ f 2 ( M ) over M . In Appx. C, we provide numerous examples of systems for which (9) holds to illustrate the assumptions of (9).

6.1 Upper bound construction

First, we consider an arbitrary distribution Q ( M,X,Y ) ∈ ∆ P . Therefore, we know Q ( X | M = m )= P ( X | M = m )= f ( δ X m ) and Q ( Y | M = m )= P ( Y | M = m )= f ( δ Y m ) . We use the dilation properties of convolution-closed distributions (Sec. 5.1) to decompose X and Y into their respective dilated versions: ( X ′ , Y ′′ , n X ) and ( Y ′ , n Y ) . From the results of Appx. M.1, we know that X = X ′ + Y ′′ + n X and Y = Y ′ + n Y . Furthermore, ( X ′ , Y ′′ , n X ) are mutually conditionally independent given M and Y ′ ⊥ ⊥ n Y | M . Hence, we can construct the following Markov chain for any arbitrary Q ( M,X,Y ) ∈ ∆ P :

We denote the joint distribution of ( M,X ′ , Y ′′ , Y ′ , n X , n Y ) as ¯ Q ( M,X ′ , Y ′′ , Y ′ , n X , n Y ) . We appropriately choose the dilation amounts for X and Y such that the respective conditional distributions of ( X ′ , Y ′′ , Y ′ , n X , n Y ) are as follows:

The distributions in (11) are well-defined due to (9). Appx. M.1 formally shows that a ¯ Q , defined as above, exists for each Q ∈ ∆ P . We use data-processing inequality and (10) to conclude:

Hence, (12) provides us the desired upper bound for our objective I Q ( M ; [ X,Y ]) . Furthermore, the minimization problem min Q ∈ ∆ P I ¯ Q ( M ; [ X ′ , Y ′′ , n X , Y ′ , n Y ]) is analytically solvable, and the minimizing distribution ¯ Q ∗ has the structure:

The corresponding distribution of ( M,X,Y ) , denoted as Q A ( M,X,Y ) , can be found by appropriately manipulating ¯ Q ∗ , as X = X ′ + Y ′′ + n X and Y = Y ′ + n Y . The distribution Q A serves as an approximate solution for the problem min Q ∈ ∆ P I Q ( M ; [ X,Y ]) and, consequently, (2). Note that if multiple δ X bias and δ Y bias exist satisfying (9), we optimize over the pairs ( δ X bias , δ Y bias ) to further refine our approximate solution of (2).

6.2 Simulation study for numerically validating the upper bound

We illustrate the tightness of our upper bound through a simulation study on the Poisson distribution (a convolution-closed distribution). The reason for choosing the Poisson distribution is two-fold: (i) Poisson distribution is easily approximated as a discrete distribution over finite support, enabling calculation of ground-truth PID terms through numerical solvers such as [10, 17, 18], which are not readily available for continuous distributions; (ii) Many practical applications of PID have been in neuroscience, and the Poisson distribution is frequently used for modeling neural spikes in neuroscientific studies [49].

We compare the performance of our analytical estimate Q A ( M,X,Y ) with the numerical groundtruth estimate Q N ( M,X,Y ) for the Poisson distribution. The simulation setup is as follows: We choose a P ( M,X,Y ) such that P ( X | M ) = Poisson ( f 1 ( M )) and P ( Y | M ) = Poisson ( f 2 ( M )) . We chose 20 different pairs of non-linear f 1 ( · ) and f 2 ( · ) (enumerated in Table 1 in Appx. M), such that the assumptions in (9) are satisfied. The distribution P ( M ) is chosen to be a discrete distribution. For each function pair, we compare Q A ( M,X,Y ) and Q N ( M,X,Y ) over 75 different distributions of P ( M ) as shown in Fig. 1a. The 75 distributions are further sub-divided into three groups of 25 based on the number of the outcomes of M , which are 2 , 4 , and 8 . For each of the 75 distributions, the values of M are randomly sampled from [1 , 4] and the values of P ( M ) are randomly sampled from a simplex of appropriate dimensions. We compare the mutual information I Q A ( M ; [ X,Y ]) calculated using Q A with the numerical ground-truth I Q N ( M ; [ X,Y ]) calculated using numerical solvers (Fig. 1b). For calculating I Q N ( M ; [ X,Y ]) , we approximate the Poisson distribution as a finite discrete distribution by terminating its support at the smallest integer where its cumulative distribution is greater than 0 . 99 . We use the code provided in [50] (under Apache 2.0-license) to numerically solve (2) using this discrete approximation of the Poisson distribution. Fig. 1c demonstrates that the analytical upper bound provided by Q A is very tight for the tested function pairs (within < 1% of

the numerical ground-truth for 16 function pairs) and serves as a good approximation for solving (2). Furthermore, the tightness of our analytical upper bound suggests that Q A might be an analytical solution of (2) for a larger class of systems of M , X , and Y having non-affine dependence on M . Similar results demonstrating the tightness of upper-bound for negative-binomial and binomial distributions are presented in Appx. M.4.

7 Discussion and Limitations

In this work, we analytically compute PID for large classes of distributions, greatly expanding upon the analytical result for the Gaussian system. We provide the first known analytical PID for systems employing Poisson, gamma, exponential, Cauchy, beta, Dirichlet, Lévy-stable, binomial, multinomial, negative binomial, and uniform distributions. Furthermore, we generalize the previous Gaussian PID result [21] in an additional way by showing the target M need not be Gaussian. Our stable distribution results provide the first known analytical computation of PID for fat-tailed distributions (all continuous stable distributions have infinite variance except the Gaussian distribution). A practical utility of our analytical results is that they provide a large test bed in which the performance of numerical PID estimators can be compared and evaluated. This test bed may benefit future works on numerical PID estimation by enabling more comprehensive testing of PID estimators.

Our results on the Poisson, Cauchy, and binomial are of particular relevance in the neuroscientific context. Poisson and Cauchy distributions are widely used to model neural spikes [49] and network dynamics in the brain [51]. Binomial thinning is a frequently-used operator in neuroscience [49]. Our generalization of the Gaussian result could be helpful in refining approximations already used in computing PID for neural data [3], e.g., by relaxing the assumption of joint Gaussianity. As continuous stable distributions have been shown to better model Magnetic Resonance Imaging data [52, 53], our results may also be helpful in computing PID in this application.

We also connect the fields of data thinning and data fission with PID by using their decomposition strategies for analytically computing the PID of systems based on convolution-closed distributions (see Sec. 5). Conversely, our PID results also suggest decomposition strategies for data thinning/fission purposes, e.g., our stable distribution results suggest that stable random variables can be decomposed by adding independent noise (similar to the Gaussian case discussed in [26]). Convolution-closed distributions are particularly promising for studying PID as they allow intuitive construction of upper bounds that can be analytically minimized (see Sec. 6). These upper bounds can complement the work on numerical estimation of PID by providing more refined approximations. Another promising avenue is to combine our upper bounds with lower bounds (e.g., from [54]) to create branch and bound algorithms [55] for solving (2). Overall, our analytical results greatly facilitate the computation of PID, either by directly using the analytical expressions or by providing refined approximations for numerical methods.

Limitations and Future Work : We study PID for univariate M , X , and Y and provide some vector extensions. More vector extensions of our results are a promising direction for subsequent works. Most of our analytical results require P ( X | M ) and P ( Y | M ) to depend on some affine functions of M . The existence of analytical solutions for the cases where P ( X | M ) and P ( Y | M ) depend on non-affine functions of M remains an open question. The upper bound discussed in Sec. 6 can be further refined with more careful analysis, and more rigorous testing of these upper bounds is required to understand in which cases the upper bound is a good approximation for solving (2). We defer the testing and refinement of these upper bounds for subsequent works, as our primary goal in this work is to study analytical solutions of PID. Niu & Quinn [56] also propose a duality result between the synergistic and redundant components in the Gaussian broadcast and multiple-access channels utilizing the analytical PID expressions of the Gaussian system. It may be possible to derive similar duality results for appropriately defined broadcast and multiple-access channels employing other distributions (e.g., Poisson or Cauchy) with our theoretical results.

Broader Impact : Due to their theoretical nature, our results' negative impact primarily depends on how they are used and interpreted. Our theoretical results are applicable only under the specific assumptions outlined in this work, and using these results without ensuring that the theorem's assumptions are satisfied can lead to incorrect scientific conclusions. We also caution reader against naively using PID to draw causal inferences, as PID is inherently a correlational quantity.

Acknowledgments and Disclosure of Funding

Authors would like to thank Prof. Pulkit Grover, Dr. Praveen Venkatesh, Prof. Alireza Chamanzar, and Neil Ashim Mehta for helpful discussions. This work was supported by National Science Foundation (NSF). Authors declare no competing interests.

References

Praveen Venkatesh and Pulkit Grover. Understanding encoding and redundancy in grid cells using partial information decomposition. Comput. Syst. Neurosci.(Cosyne) , 2020.

Ariel K Feldman, Praveen Venkatesh, Douglas J Weber, and Pulkit Grover. Information-theoretic tools to understand distributed source coding in neuroscience. IEEE Journal on Selected Areas in Information Theory , 2024.

Praveen Venkatesh, Corbett Bennett, Sam Gale, Tamina Ramirez, Greggory Heller, Severine Durand, Shawn Olsen, and Stefan Mihalas. Gaussian partial information decomposition: Bias correction and application to high-dimensional data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 74602-74635. Curran Associates, Inc., 2023.

Nigel Colenbier, Frederik Van de Steen, Lucina Q Uddin, Russell A Poldrack, Vince D Calhoun, and Daniele Marinazzo. Disambiguating the role of blood flow and global signal with partial information decomposition. NeuroImage , 213:116699, 2020.

Tjeerd W Boonstra, Luca Faes, Jennifer N Kerkman, and Daniele Marinazzo. Information decomposition of multichannel EMG to map functional interactions in the distributed motor system. NeuroImage , 202:116093, 2019.

Jana Krohova, Luca Faes, Barbora Czippelova, Zuzana Turianikova, Nikoleta Mazgutova, Riccardo Pernice, Alessandro Busacca, Daniele Marinazzo, Sebastiano Stramaglia, and Michal Javorka. Multiscale information decomposition dissects control mechanisms of heart rate variability at rest and during physiological stress. Entropy , 21(5):526, 2019.

Giuseppe Pica, Eugenio Piasini, Houman Safaai, Caroline Runyan, Christopher Harvey, Mathew Diamond, Christoph Kayser, Tommaso Fellin, and Stefano Panzeri. Quantifying how much sensory information in a neural code is relevant for behavior. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Praveen Venkatesh, Sanghamitra Dutta, and Pulkit Grover. Information flow in computational systems. IEEE Transactions on Information Theory , 66(9):5456-5491, 2020.

Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Russ R Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling multimodal interactions: An information decomposition framework. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 27351-27393. Curran Associates, Inc., 2023.

Paul Pu Liang, Yun Cheng, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multimodal fusion interactions: A study of human and automatic quantification. In Proceedings of the 25th International Conference on Multimodal Interaction , pages 425-435, 2023.

Sanghamitra Dutta, Praveen Venkatesh, Piotr Mardziel, Anupam Datta, and Pulkit Grover. An information-theoretic quantification of discrimination with exempt features. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3825-3833, 2020.

Sanghamitra Dutta and Faisal Hamman. A review of partial information decomposition in algorithmic fairness and explainability. Entropy , 25(5):795, 2023.

Faisal Hamman and Sanghamitra Dutta. Demystifying local & global fairness trade-offs in federated learning using partial information decomposition. In The Twelfth International Conference on Learning Representations , 2024.

Faisal Hamman and Sanghamitra Dutta. A unified view of group fairness tradeoffs using partial information decomposition. In 2024 IEEE International Symposium on Information Theory (ISIT) , pages 214-219, 2024.

Tomas Scagliarini, Luca Faes, Daniele Marinazzo, Sebastiano Stramaglia, and Rosario N Mantegna. Synergistic information transfer in the global system of financial markets. Entropy , 22(9):1000, 2020.

Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy , 16(4):2161-2183, 2014.

Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. Bivariate partial information decomposition: The optimization perspective. Entropy , 19(10):530, 2017.

Pradeep Kr Banerjee, Johannes Rauh, and Guido Montúfar. Computing the unique information. In 2018 IEEE International Symposium on Information Theory (ISIT) , pages 141-145. IEEE, 2018.

Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. BROJA-2PID: A robust estimator for bivariate partial information decomposition. Entropy , 20(4):271, 2018.

Ari Pakman, Amin Nejatbakhsh, Dar Gilboa, Abdullah Makkeh, Luca Mazzucato, Michael Wibral, and Elad Schneidman. Estimating the unique information of continuous variables. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 20295-20307. Curran Associates, Inc., 2021.

Adam B Barrett. Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Physical Review E , 91(5):052802, 2015.

Praveen Venkatesh and Gabriel Schamberg. Partial information decomposition via deficiency for multivariate Gaussians. In 2022 IEEE International Symposium on Information Theory (ISIT) , pages 2892-2897. IEEE, 2022.

John P Nolan. Univariate stable distributions. Springer Series in Operations Research and Financial Engineering , 10:978-3, 2020.

Fred W Steutel and Klaas van Harn. Discrete analogues of self-decomposability and stability. The Annals of Probability , pages 893-899, 1979.

Anna Neufeld, Ameer Dharamshi, Lucy L. Gao, and Daniela Witten. Data thinning for convolution-closed distributions. Journal of Machine Learning Research , 25(57):1-35, 2024.

James Leiner, Boyan Duan, Larry Wasserman, and Aaditya Ramdas. Data fission: Splitting a single data point. Journal of the American Statistical Association , 0(0):1-12, 2023.

Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 , 2010.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) . Wiley-Interscience, USA, 2006.

Malte Harder, Christoph Salge, and Daniel Polani. Bivariate measure of redundant information. Physical Review E , 87(1):012130, 2013.

Pradeep Kr Banerjee, Eckehard Olbrich, Jürgen Jost, and Johannes Rauh. Unique informations and deficiencies. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 32-38. IEEE, 2018.

Jim W Kay and Robin AA Ince. Exact partial information decompositions for Gaussian systems based on dependency constraints. Entropy , 20(4):240, 2018.

Conor Finn and Joseph T Lizier. Pointwise partial information decomposition using the specificity and ambiguity lattices. Entropy , 20(4):297, 2018.

Xueyan Niu and Christopher J Quinn. A measure of synergy, redundancy, and unique information using information geometry. In 2019 IEEE International Symposium on Information Theory (ISIT) , pages 3127-3131. IEEE, 2019.

Fernando E Rosas, Pedro AM Mediano, Borzoo Rassouli, and Adam B Barrett. An operational information decomposition via synergistic disclosure. Journal of Physics A: Mathematical and Theoretical , 53(48):485001, 2020.

Keerthana Gurushankar, Praveen Venkatesh, and Pulkit Grover. Extracting unique information through Markov relations. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1-6. IEEE, 2022.

David Blackwell. Equivalent comparisons of experiments. The Annals of Mathematical Statistics , pages 265-272, 1953.

Praveen Venkatesh, Keerthana Gurushankar, and Gabriel Schamberg. Capturing and interpreting unique information. In 2023 IEEE International Symposium on Information Theory (ISIT) , pages 2631-2636. IEEE, 2023.

Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. Guided selforganization: inception , pages 159-190, 2014.

Joseph T Lizier, Nils Bertschinger, Jürgen Jost, and Michael Wibral. Information decomposition of target effects from multi-source interactions: Perspectives on previous, current and future work. Entropy , 20(4):307, 2018.

John P Nolan. Stable distributions . 2012.

Jeremie Fish, Jie Sun, and Erik Bollt. Interaction networks from discrete event data by Poisson multivariate mutual information estimation and information flow with applications from gene expression data. Applied Network Science , 7(1):1-22, 2022.

AS Krishnamoorthy. Multivariate binomial and Poisson distributions. Sankhy¯ a: The Indian Journal of Statistics , pages 117-124, 1951.

DMMahamunulu. A note on regression in the multivariate Poisson distribution. Journal of the American Statistical Association , 62(317):251-258, 1967.

Dimitris Karlis and Loukia Meligkotsidou. Multivariate Poisson regression with covariance structure. Statistics and Computing , 15(4):255-265, 2005.

Murad S Taqqu. Stable non-Gaussian random processes: stochastic models with infinite variance . Chapman & Hall, 1994.

S James Press. Multivariate stable distributions. Journal of Multivariate Analysis , 2(4):444-462, 1972.

Norman Lloyd Johnson, Samuel Kotz, and Narayanaswamy Balakrishnan. Discrete multivariate distributions , volume 165. Wiley New York, 1997.

Xiaohu Shang and H Vincent Poor. Noisy-interference sum-rate capacity for vector Gaussian interference channels. IEEE Transactions on Information Theory , 59(1):132-153, 2012.

Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems . MIT press, 2005.

Abdullah Makkeh, Dirk Oliver Theis, and Raul Vicente. Broja-2pid: A robust estimator for bertschinger et al.'s bivariate partial information decomposition. Entropy , 20(4):271, 2018.

Viktoras Pyragas and Kestutis Pyragas. Mean-field equations for neural populations with q-Gaussian heterogeneities. Physical Review E , 105(4):044402, 2022.

Diego Salas-González, JM Górriz, Javier Ramírez, Matthias Schloegl, Elmar Wolfgang Lang, and Andrés Ortiz. Parameterization of the distribution of white and grey matter in MRI using the α -stable distribution. Computers in biology and medicine , 43(5):559-567, 2013.

Diego Salas-Gonzalez, Juan M Górriz, Javier Ramírez, and Elmar W Lang. Why using the alphastable distribution in NeuroImage? In 2014 International Conference on Signal Processing and Multimedia Applications (SIGMAP) , pages 297-301. IEEE, 2014.

Seiya Tokui and Issei Sato. Disentanglement analysis with partial information decomposition. arXiv preprint arXiv:2108.13753 , 2021.

Narendra and Fukunaga. A branch and bound algorithm for feature subset selection. IEEE Transactions on Computers , 100(9):917-922, 1977.

Xueyan Niu and Christopher J Quinn. Synergy and redundancy duality between Gaussian multiple access and broadcast channels. In 2020 International Symposium on Information Theory and Its Applications (ISITA) , pages 6-10. IEEE, 2020.

Robert B Ash and Catherine A Doléans-Dade. Probability and measure theory . Academic press, 2000.

Gilbert Strang. Introduction to linear algebra . SIAM, 2022.

Alan V Oppenheim, Alan S Willsky, Syed Hamid Nawab, and Jian-Jiun Ding. Signals and systems , volume 2. Prentice hall Upper Saddle River, NJ, 1997.

Jagadesh Chandramohan and Lung-Kuang Liang. Bernoulli, multinomial and Markov chain thinning of some point processes and some results about the superposition of dependent renewal processes. Journal of Applied Probability , 22(4):828-835, 1985.

Sergio Verdú. Poisson communication theory. International Technion Communication Day in Honor of Israel Bar-David , 66, 1999.

Christian Walck et al. Hand-book on statistical distributions for experimentalists . Stockholms universitet, 1996.

Kalimuthu Krishnamoorthy. Handbook of statistical distributions with applications . Chapman and Hall/CRC, 2006.

Lenka Slámová and Lev B Klebanov. Generalized definitions of discrete stability. arXiv preprint arXiv:1502.02588 , 2015.

A Formal proof of the sufficient condition for solving (2) discussed in Sec. 3

Proposition 1. For the minimization problem defined in (2) , if there exists a joint density Q MC ( M,X,Y ) ∈ ∆ P , having the Markovian structure of the form M → X → Y or M → Y → X , then Q MC ( M,X,Y ) minimizes (2) . Furthermore,

If Q MC ( M,X,Y ) has the Markovian structure M → X → Y , then UI ( M ; Y \ X ) = I Q MC ( M ; Y | X ) = 0 .

If Q MC ( M,X,Y ) has the Markovian structure M → Y → X , then UI ( M ; X \ Y ) = I Q MC ( M,X | Y ) = 0 .

Proof. Case 1 : Q MC ( M,X,Y ) ∈ ∆ P having the Markovian structure M → X → Y .

By the definition of the Markov chain, we know that M and Y are conditionally independent given X . Denote Q MC ( M,X,Y ) as the joint distribution of the Markov chain M → X → Y . Then,

where I Q MC ( M ; Y | X ) is the conditional mutual information between M and Y given X under the distribution Q MC ( M,X,Y ) . Equation (13) follows from a well-known property of conditional mutual information which states that I ( A ; B | C ) = 0 iff A ⊥ ⊥ B | C [28].

Since I Q ( M ; Y | X ) ≥ 0 ∀ Q ∈ ∆ P , due to the non-negativity of conditional mutual information [28], we can conclude that min Q ∈ ∆ P I Q ( M ; Y | X ) ≥ 0 . The joint distribution Q MC ( M,X,Y ) having the Markovian structures M → X → Y achieves the lower bound of 0 for the minimization problem min Q ∈ ∆ P I Q ( M ; Y | X ) implying Q MC ( M,X,Y ) minimizes (2).

Case 2 : Q MC ( M,X,Y ) ∈ ∆ P having the Markovian structure M → Y → X .

A similar argument as case 1, with the random variables X and Y switched, would allow us to conclude that the joint distribution Q MC ( M,X,Y ) specifying the Markov chain M → Y → X minimizes (2).

B Other PID definitions for which theorems 1-7 are valid

Our theoretical results (Theorems 1-7) hold for a wide range of PID measures. Particularly, our results hold for two broad classes of PID definitions: PID definitions satisfying assumption ( ∗ ) in Bertschinger et al. [16] and Blackwellian PIDs (see Appendix B-C of Venkatesh & Schamberg [22] for a deeper discussion on differences between these two families of PID definition). Some examples of PID definitions satisfying assumption ( ∗ ) in Bertschinger et al. [16] are the Williams & Beer's PID definition [27], the PID definition proposed in Harder et al. [29], and I-PID proposed in Venkatesh et al. [37]. Similarly, an example of a Blackwellian PID is the δ -PID discussed in Venkatesh & Schamberg [22]. Section III-F of Venkatesh et al. [37] further discusses a family of Blackwellian PID definitions for which our theorems are also applicable. We now provide formal arguments showing that our theoretical results are applicable for PID definitions that either satisfy assumption ( ∗ ) of Bertschinger et al. [16] or are Blackwellian

PID definitions satisfying assumption ( ∗ ) : We invoke the argument presented in the proof discussed in Section 4.2 of Barrett [21] to show that our results hold for any PID definition satisfying assumption ( ∗ ) in Bertschinger et al. [16]. Barrett's argument relies on the key observation that the UI calculated using BROJA-PID upper bounds the UI calculated using any other PID definition satisfying assumption ( ∗ ) in Bertschinger et al. Entropy'14. Consequently, if one of the UI atoms of the BROJA-PID is zero then the corresponding UI atom of the other PID definitions must also be zero due to BROJA PID upper-bounding them and the non-negativity of the PID atoms. Since the remaining PID atoms are calculated using (1), it must be the case that the PID atoms calculated using any PID definition satisfying assumption ( ∗ ) must be the same as the PID atoms calculated using BROJA-PID whenever one of the UI terms of BROJA-PID is zero.

Blackwellian PID : The defining feature of a Blackwellian PID is that the UI atom is zero iff it is possible to construct the Markovian structure between the random variables (i.e., M → X → Y or M → Y → X ), while preserving their pairwise marginals (see Venkatesh et al. [22] for more details). In all our theoretical results, we show that the random variables in the analyzed system admit

a Markovian structure, while preserving the pairwise marginal structure. Hence, the UI atom for the systems analyzed in Theorems 1-7 is zero for any Blackwellian PID. Consequently, as the rest of the PID terms are calculated using (1), it must be the case that the PID atoms calculated using any Blackwellian PID must be the same as the PID atoms calculated using BROJA-PID in our results.

C Lists of Systems that can solved using Theorem 6

In this section, we provide examples of systems of random variables ( M,X,Y ) employing wellknown distributions whose PID terms can be obtained through theorem 6. The following list is modified from Table 2 of [25]. Note that for the joint distribution P ( M,X,Y ) of the system of random variable ( M,X,Y ) , we only specify P ( M ) , P ( X | M ) , and P ( Y | M ) , as any P ( M,X,Y ) having these marginals can be solved through theorem 6. We also discuss some possible extensions of the corresponding systems for which the upper bound discussed in Sec. 6 is applicable. Note that the extension of the systems provided are not exhaustive, and there exists more systems for which the upper bound is applicable. In the following examples, we assume a > b .

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support (0 , ∞ ) .

(b) P ( X | M )= Poisson ( aM ) .

(c) P ( Y | M ) = Poisson ( bM ) .

The corresponding G ( γδ X m , (1 -γ ) δ X m , x ) specified in theorem 6 is Binomial ( x, γ ) with γ = b / a , and a, b > 0 .

Result : The Markov chain M → X → Y ∈ ∆ P and UI ( M ; Y \ X ) = 0 .

An extension of the system for which the upper bound of Sec. 6 in applicable : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( X | M )= Poisson ( f 1 ( M )) , for f 1 ( M ) > 0 ∀ m ∈ M .

(c) P ( Y | M ) = Poisson ( f 2 ( M )) , for f 2 ( M ) > 0 ∀ m ∈ M .

(d) f 1 ( m ) ≥ f 2 ( m ) ∀ m ∈ M and all the underlying distributions are well-defined.

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support ( -∞ , ∞ ) .

(b) P ( X | M )= N ( aM,aσ 2 ) .

(c) P ( Y | M ) = N ( bM,bσ 2 ) .

The corresponding G ( γδ X m , (1 -γ ) δ X m , x ) specified in theorem 6 is N ( γx, γ (1 -γ ) σ 2 ) with γ = b / a , and a, b > 0 .

Result : The Markov chain M → X → Y ∈ ∆ P and UI ( M ; Y \ X ) = 0 .

An extension of the system for which the upper bound of Sec. 6 in applicable : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( X | M )= Gaussian ( f 1 ( M ) , f 1 ( M )) , for f 1 ( M ) > 0 ∀ m ∈ M .

(c) P ( Y | M ) = Gaussian ( f 2 ( M ) , f 2 ( M )) , for f 2 ( M ) > 0 ∀ m ∈ M .

(d) f 1 ( m ) ≥ f 2 ( m ) ∀ m ∈ M and all the underlying distributions are well-defined.

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support (0 , 1] .

(b) P ( X | M )= Negative Binomial ( a, M ) .

(c) P ( Y | M ) = Negative Binomial ( b, M ) .

The corresponding G ( γδ X m , (1 -γ ) δ X m , x ) specified in theorem 6 is Beta-Binomial ( x, γa, (1 -γ ) a ) with γ = b / a , and a, b ∈ N .

Result : The Markov chain M → X → Y ∈ ∆ P , and UI ( M ; Y \ X ) = 0 .

An extension of the system for which the upper bound of Sec. 6 in applicable : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( X | M )= Negative Binomial ( f 1 ( M ) , p ) , for f 1 ( M ) ∈ N 0 ∀ m ∈ M and p ∈ [0 , 1] .

(c) P ( Y | M )= Negative Binomial ( f 2 ( M ) , p ) , for f 2 ( M ) ∈ N 0 ∀ m ∈ M .

(d) f 1 ( m ) ≥ f 2 ( m ) ∀ m ∈ M and all the underlying distributions are well-defined.

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support (0 , ∞ ) .

(b) P ( X | M )= Gamma ( a, M ) .

(c) P ( Y | M ) = Gamma ( b, M ) .

The corresponding G ( γδ X m , (1 -γ ) δ X m , x ) specified in theorem 6 is obtained as follows: G ( γδ X m , (1 -γ ) δ X m , x )= xZ, Z ∼ Beta ( b, (1 -b )) with γ = b / a , and a, b ∈ (0 , ∞ ) . Note that we are following the Gamma ( α, β ) notation.

Result : The Markov chain M → X → Y ∈ ∆ P and UI ( M ; Y \ X ) = 0 .

An extension of the system for which the upper bound of Sec. 6 in applicable : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( X | M )= Gamma ( f 1 ( M ) , β ) , for f 1 ( M ) > 0 ∀ m ∈ M and β > 0 .

(c) P ( Y | M ) = Gamma ( f 2 ( M ) , β ) , for f 2 ( M ) > 0 ∀ m ∈ M .

(d) f 1 ( m ) ≥ f 2 ( m ) ∀ m ∈ M and all the underlying distributions are well-defined.

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support (0 , ∞ ) .

(b) P ( X | M )= Exponential ( M ) .

(c) P ( Y | M ) = Gamma ( b, M ) .

The corresponding G ( γδ X m , (1 -γ ) δ X m , x ) specified in theorem 6 is obtained as follows: G ( γδ X m , (1 -γ ) δ X m , x )= xZ, Z ∼ Beta ( b, (1 -b )) , and 0 < b ≤ 1 . Note that we are following the Gamma ( α, β ) notation.

Result : The Markov chain M → X → Y ∈ ∆ P and UI ( M ; Y \ X ) = 0 .

An extension of the system for which the upper bound of Sec. 6 in applicable : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( X | M )= Exponential ( f 1 ( M )) , for f 1 ( M ) > 0 ∀ m ∈ M .

(c) P ( Y | M ) = Gamma ( f 2 ( M ) , f 1 ( M )) , for f 2 ( M ) > 0 ∀ m ∈ M .

(d) f 2 ( m ) ≤ 1 ∀ m ∈ M and all the underlying distributions are well-defined.

6. System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support [0 , 1] .

(b) P ( X | M )= Binomial ( a, M ) .

(c) P ( Y | M ) = Binomial ( b, M ) .

The corresponding G ( γδ X m , (1 -γ ) δ X m , x ) specified in theorem 6 is Hypergeometric ( x, γa, (1 -γ ) a ) with γ = b / a , and a, b ∈ N .

Result : The Markov chain M → X → Y ∈ ∆ P and UI ( M ; Y \ X ) = 0 .

An extension of the system for which the upper bound of Sec. 6 in applicable : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( X | M )= Binomial ( f 1 ( M ) , p ) , for f 1 ( M ) ∈ N 0 ∀ m ∈ M and p ∈ [0 , 1] .

(c) P ( Y | M ) = Binomial ( f 2 ( M ) , p ) , for f 2 ( M ) ∈ N 0 ∀ m ∈ M .

(d) f 1 ( m ) ≥ f 2 ( m ) ∀ m ∈ M and all the underlying distributions are well-defined.

System : The distributions of P ( ⃗ M ) , P ( ⃗ X | ⃗ M ) , and P ( ⃗ Y | ⃗ M ) are as follows:

(a) ⃗ M ∼ P ( ⃗ M ) having support over a k -dimensional simplex.

(b) P ( ⃗ X | ⃗ M )= Multinomial k ( a, ⃗ M ) .

(c) P ( ⃗ Y | ⃗ M ) = Multinomial k ( b, ⃗ M ) .

The corresponding G ( γδ X m , (1 -γ ) δ X m , x ) specified in theorem 6 is Multivariate Hypergeometric ( ⃗ x , γ Y ) with γ = b / a , and a, b ∈ N .

Result : The Markov chain ⃗ M → ⃗ X → ⃗ Y ∈ ∆ and UI ( ⃗ M ; ⃗ Y \ ⃗ X ) = 0

An extension of the system for which the upper bound of Sec. 6 in applicable distributions of P ( M ) , P ( ⃗ X | M ) , and P ( ⃗ Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( ⃗ X | M )= Multinomial k ( f 1 ( M ) , ⃗ p ) , for f 1 ( M ) ∈ N 0 ∀ m ∈ M .

(c) P ( ⃗ Y | M ) = Multinomial ( f 2 ( M ) , ⃗ p ) , for f 2 ( M ) ∈ N 0 ∀ m ∈ M .

(d) f 1 ( m ) ≥ f 2 ( m ) ∀ m ∈ M , ⃗ p is a valid probability vector, and all the underlying distributions are well-defined.

D Lists of Systems that can solved using Theorem 7

In this section, we provide examples of systems of random variables ( M,X,Y ) employing wellknown distributions whose PID terms can be obtained through theorem 7. The following list is appropriately modified from Appendix A of [26]. Similar to Appx. C, we only specify specify P ( M ) , P ( X | M ) , and P ( Y | M ) of the joint distribution P ( M,X,Y ) , as any P ( M,X,Y ) having the same marginals can be solved through theorem 7. In the following examples, we assume a > b .

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support (0 , ∞ ) .

(b) P ( X | M )= Exponential ( M ) .

(c) P ( Y | M ) = Geometric ( M / ( τ + M ) ) , where τ ∈ (0 , ∞ ) .

Results : The Markov chain M → X → Y ∈ ∆ P , and UI ( M ; Y \ X ) = 0 . The corresponding Q MC ( Y | X ) required for constructing the Markov chain is Poisson ( τX ) .

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support M .

(b) P ( X | M )= Gamma ( f 1 ( M ) , f 2 ( M )) , where f 1 ( m ) , f 2 ( m ) > 0 ∀ m ∈ M .

(c) P ( Y | M ) = Negative Binomial ( f 1 ( M ) , f 2 ( M ) / ( f 2 ( M )+ τ ) ) , where τ ∈ (0 , ∞ )

Results : The Markov chain M → X → Y ∈ ∆ P , and UI ( M ; Y \ X ) = 0 . The corresponding Q MC ( Y | X ) required for constructing the Markov chain is is Poisson ( τX ) .

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support (0 , ∞ ) .

(b) P ( X | M )= Beta ( M, 1) .

(c) P ( Y | M ) is a discrete distribution with over support set { 0 , 1 , . . . , N } , with

where Γ( . ) is the Gamma function.

Results : The Markov chain M → X → Y ∈ ∆ P , and UI ( M ; Y \ X ) = 0 . The corresponding Q MC ( Y | X ) required for constructing the Markov chain is Binomial ( N,X ) .

System : The distributions of P ( M ) , P ( X | M ) , and P ( Y | M ) are as follows:

(a) M ∼ P ( M ) having support (0 , ∞ ) .

(b) P ( X | M )= Beta (1 , M ) .

(c) P ( Y | M ) is a discrete distribution with over support set { 0 , 1 , . . . , N } , with

where Γ( . ) is the Gamma function.

Results : The Markov chain M → X → Y ∈ ∆ P , and UI ( M ; Y \ X ) = 0 . The corresponding Q MC ( Y | X ) required for constructing the Markov chain is is Binomial ( N, 1 -X ) .

P . : The

System : The distributions of P ( ⃗ M ) , P ( ⃗ X | ⃗ M ) , and P ( ⃗ Y | ⃗ M ) are as follows:

(a) ⃗ M ∼ P ( ⃗ M ) having support (0 , ∞ ) d for some dimension d .

(b) P ( ⃗ X | ⃗ M )= Dirichlet ( M 1 , . . . , M d ) .

(c) P ( ⃗ Y | ⃗ M ) is a discrete distribution having support { y i ∈ { 1 , . . . , N } ∀ i ∈ [ d ] | ∑ d i =1 y i = N } , and

where Γ( . ) is the Gamma function.

Results : The Markov chain ⃗ M → ⃗ X → ⃗ Y ∈ ∆ P , and UI ( ⃗ M ; ⃗ Y \ ⃗ X ) = 0 . The corresponding Q MC ( ⃗ Y | ⃗ X ) required for constructing the Markov chain is Multinomial d ( N, ⃗ X ) .

E Miscellaneous PID Results

This section explores the analytical calculation of PID terms for five more systems of random variables ( M,X,Y ) with joint distribution P ( M,X,Y ) . These five systems' P ( M,X,Y ) employ wellknown distributions, but their respective analytical PID calculation does not result from Theorem 1Theorem 7. Of note are the results of P ( M,X,Y ) employing exponential and uniform distribution, which states that these systems will always have both unique information terms zero.

E.1 PID for the miscellaneous system M1.

Miscellaneous system M1 : Let ( M,X,Y ) be a system of random variables with joint distribution P ( M,X,Y ) . The joint distribution P ( M,X,Y ) describes the miscellaneous system M1 if it satisfies the following two properties:

M ∼ P ( M ) , and has support (0 , ∞ ) .

P ( X | M )= Exponential ( γ X M ) , and P ( Y | M )= Exponential ( γ Y M ) , where γ X , γ Y ∈ (0 , ∞ ) .

Proposition 2. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) describes the miscellaneous system M1 defined above. Then, ∆ P contains a distribution Q MC ( M,X,Y ) , which admits both Markovian structures, i.e., M → X → Y and M → Y → X . Consequently, both UI ( M ; Y \ X ) = UI ( M ; X \ Y ) = 0 .

Proof. We prove proposition 2 by explicitly constructing a joint distribution Q MC ( M,X,Y ) ∈ ∆ P that admits both Markovian structures, namely, M → X → Y and M → Y → X . A necessary and sufficient condition for the distribution Q MC ( M,X,Y ) to admit both Markovian structures is that it should satisfy (14).

We now briefly outline the structure of the proof. The rest of the proof can be divided into three parts:

In the first part, we explicitly construct Q MC ( M,X,Y ) .

In the second part, we show that the Q MC ( M,X,Y ) constructed in the first part lies in ∆ P .

In the third part, we conclude our proof by showing that Q MC ( M,X,Y ) satisfies (14) and, consequently, admits both Markovian structures.

Part 1: Specifying Q MC ( M,X,Y )

We specify Q MC ( M,X,Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) , and Q MC ( Y | X ) . We choose Q MC ( M ) and Q MC ( X | M ) as follows:

The distribution Q MC ( Y | X ) is specified through the following deterministic transformation:

Part 2: Showing Q MC ( M,X,Y ) ∈ ∆ P

For showing Q MC ( M,X,Y ) ∈ ∆ P , we first need to calculate Q MC ( Y | M ) :

Calculating Q MC ( Y | M ) : By lemma 1, if X ∼ Exp ( λ ) , then aX ∼ Exponential ( λ / a ) . Since Q MC ( X | M )= Exponential ( γ X M ) , we can use the result of lemma 1 to calculate Q MC ( Y | M ) as follows:

Then, from (15) and (17), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P .

Part 3: Showing Q MC ( M,X,Y ) satisfies (14)

It is trivial to see that by the construction of Q MC ( M,X,Y ) given in Part 1, we have:

By the chain rule, we have:

Hence, for showing that Q MC ( M,X,Y ) satisfies (14), we only need to show that Q MC ( X | Y, M ) = Q MC ( X | Y ) .

Showing Q MC ( X | Y, M ) = Q MC ( X | Y ) : Observe that Y = γ X / γ Y X by construction. Therefore, we can equivalently express X = γ Y / γ X Y . Consequently,

where I [ · ] is the indicator function. Since R.H.S of the above equation does not depend on M , we have:

Combining (21) and (23), we have:

Combining (20) and (24), we can show that Q MC ( M,X,Y ) satisfies (14).

Conclusion : Combining the results of parts 2 and 3, we can conclude that the Q MC ( M,X,Y ) ∈ ∆ P and contains both Markovian structure M → X → Y and M → Y → X . A unique implication of Q MC ( M,X,Y ) admitting both Markovian structures is that both I Q MC ( M ; X | Y ) = I Q MC ( M ; Y | X ) = 0 . Consequently, both minimization problems presented in (2) are minimized by Q MC ( M,X,Y ) with UI ( M ; X \ Y ) = min Q ∈ ∆ P I Q ( M ; X | Y ) = 0 and UI ( M ; Y \ X ) = min Q ∈ ∆ P I Q ( M ; Y | X ) = 0 , completing our proof.

Lemma 1. If X ∼ Exponential ( λ ) with λ ∈ (0 , ∞ ) , then γX ∼ Exponential ( λ / γ ) for γ ∈ (0 , ∞ ) .

Proof. We denote the p.d.f. of the random variable X as p X ( · ) and, by lemma statement, we know:

Denote the p.d.f. of the function f ( X ) of the random variable X as p f ( X ) ( · ) . We want to derive p f ( X ) ( · ) for f ( X ) = γX . Since, f ( X ) = γX is an invertible function, we can use the change-ofvariable technique for deriving p f ( X ) ( · ) :

E.2 PID for the miscellaneous system M2.

Miscellaneous System M2 : Let ( M,X,Y ) be a system of random variables with joint distribution P ( M,X,Y ) . The joint distribution P ( M,X,Y ) describes the miscellaneous system M2 if it satisfies the following two properties:

M ∼ P ( M ) and has support (0 , ∞ ) .

P ( X | M )= Negative Binomial ( M,p X ) , and P ( Y | M )= Negative Binomial ( M,p Y ) .

This system is adapted from the negative binomial decomposition listed in Appendix A [26].

Proposition 3. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) describes the miscellaneous system M2 defined above.

If p X ≤ p Y , then ∆ P contains a Markov chain of the form M → X → Y , and UI ( M ; Y \ X ) = 0 .

If p X ≥ p Y , then ∆ P contains a Markov chain of the form M → Y → X , and UI ( M ; X \ Y ) = 0 .

Proof. We only provide an explicit proof of condition 1. The proof of condition 2 follows the same steps as the proof of condition 1 with parameters of X and Y switched.

Proof of condition 1 : We briefly outline the proof structure. The proof can be divided into two parts:

In the first part, we explicitly construct a joint distribution Q MC ( M,X,Y ) having the Markovian structure M → X → Y .

In the second part, we conclude our proof by showing the Q MC ( M,X,Y ) constructed in the first part, lies in ∆ P .

Part 1: Specifying Q MC ( M,X,Y )

Since Q MC ( M,X,Y ) is the joint distribution of a Markov chain M → X → Y , we have Q MC ( M,X,Y ) = Q MC ( M ) Q MC ( X | M ) Q MC ( Y | X ) . Taking advantage of this special structure of Q MC ( M,X,Y ) , we specify Q MC ( M,X,Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) , and Q MC ( Y | X ) . We choose Q MC ( M ) , Q MC ( X | M ) and Q MC ( Y | X ) as follows:

Part 2: Showing Q MC ( M,X,Y ) ∈ ∆ P

For showing Q MC ( M,X,Y ) ∈ ∆ P , we first need to calculate Q MC ( Y | M ) :

Calculating Q MC ( Y | M ) : Since Q MC ( X | M ) = Negative Binomial ( M,p X ) and Q MC ( Y | X ) = Binomial ( N,p ) , we can use lemma 2 for calculating Q ( Y | M ) :

Then, from (27) and (28), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; Y \ X ) = 0 . This concludes our proof.

Lemma 2. Let X ∼ Negative Binomial ( N,p X ) and P ( Y | X ) = Binomial ( X,p ) with p = p X (1 -p Y ) p Y (1 -p X ) , p Y ≥ p X , p Y , p X ∈ [0 , 1] , and N ∈ N 0 . Then, Y ∼ Negative Binomial ( N,p Y ) .

Proof. First, in order for P ( Y | X ) to be a legitimate distribution, we need to ensure that p ∈ [0 , 1] , i.e., we need to show that:

Since p X , (1 -p Y ) , (1 -p X ) , p Y ≥ 0 as p X , p Y ∈ [0 , 1] , we can easily verify the first inequality:

For verifying p ≤ 1 , we will use the assumption that p Y ≥ p X :

Subtracting 1 from both sides of the above inequality.

Rearranging terms, we obtain:

For calculating P ( Y ) , we borrow the result of proposition 13 in [26], which states that if X ∼ Negative Binomial ( M,p X ) and P ( Y | X ) = Binomial ( X,p ) , then P ( Y ) = Negative Binomial ( N, p X p X + p -pp X ) . Hence,

E.3 PID for the miscellaneous system M3.

Miscellaneous System M3 : Let ( M,X,Y ) be a system of random variables with joint distribution P ( M,X,Y ) . The joint distribution P ( M,X,Y ) describes the miscellaneous system M3 if it satisfies the following two properties:

M ∼ P ( M ) and has support (0 , ∞ ) .

P ( X | M )= Uniform (0 , γ X M ) and P ( Y | M )= Uniform (0 , γ Y M ) , where γ X , γ Y ∈ R \{ 0 } .

Proposition 4. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) describes the miscellaneous system M3 defined above. Then, ∆ P contains a distribution Q MC ( M,X,Y ) , which admits both Markovian structures, i.e., M → X → Y and M → Y → X . Consequently, both UI ( M ; Y \ X ) = UI ( M ; X \ Y ) = 0 .

Proof. We prove proposition 4 by explicitly constructing a joint distribution Q MC ( M,X,Y ) ∈ ∆ P that admits both Markovian structures, namely, M → X → Y and M → Y → X . A necessary and sufficient condition for the distribution Q MC ( M,X,Y ) to admit both Markovian structures is that it should satisfy (37):

We now briefly outline the structure of the proof. The rest of the proof can be divided into three parts:

In the first part, we explicitly construct Q MC ( M,X,Y ) .

In the second part, we show that the Q MC ( M,X,Y ) constructed in the first part, lies in ∆ P .

In the third part, we conclude our proof by showing that Q MC ( M,X,Y ) satisfies (37) and, consequently, admits both Markovian structures.

Part 1: Specifying Q MC ( M,X,Y )

We specify Q MC ( M,X,Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) , and Q MC ( Y | X ) . We choose Q MC ( M ) and Q MC ( X | M ) as follows:

The distribution Q MC ( Y | X ) is specified through the following deterministic transformation:

Part 2: Showing Q MC ( M,X,Y ) ∈ ∆ P

For showing Q MC ( M,X,Y ) ∈ ∆ P , we first need to calculate Q MC ( Y | M ) :

Calculating Q MC ( Y | M ) : Since Q MC ( X | M )= Uniform (0 , γ X M ) , and Y = γ Y / γ X X , we can use the result of lemma 3 to calculate Q MC ( Y | M ) as follows:

Then, from (38) and (40), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P .

Part 3: Showing Q MC ( M,X,Y ) satisfies (37)

It is trivial to see that by the construction of Q MC ( M,X,Y ) given in Part 1, we have:

By the chain rule, we have:

Hence, for showing that Q MC ( M,X,Y ) satisfies (37), we only need to show that Q MC ( X | Y, M ) = Q MC ( X | Y ) .

Showing Q MC ( X | Y, M ) = Q MC ( X | Y ) : Observe that Y = γ Y / γ X X by construction. Therefore, we can equivalently express X = γ X / γ Y Y . Consequently,

where I [ · ] is the indicator function. Since R.H.S of the above equation does not depend on M , we have:

Combining (44) and (46), we have:

Combining (43) and (47), we can show that Q MC ( M,X,Y ) satisfies (37).

Conclusion : Combining the results of parts 2 and 3, we can conclude that Q MC ( M,X,Y ) ∈ ∆ P and contains both Markovian structures: M → X → Y and M → Y → X . A unique implication of Q MC ( M,X,Y ) admitting both Markovian structures is that both I Q MC ( M ; X | Y ) = I Q MC ( M ; Y | X ) = 0 . Consequently, both minimization problems presented in (2) are minimized by Q MC ( M,X,Y ) with UI ( M ; X \ Y ) = min Q ∈ ∆ P I Q ( M ; X | Y ) = 0 and UI ( M ; Y \ X ) = min Q ∈ ∆ P I Q ( M ; Y | X ) = 0 , completing our proof.

̸

Lemma 3. Let X ∼ Uniform (0 , a ) for some a ∈ R , then γX ∼ Uniform (0 , γa ) for γ ∈ R and a, γ = 0 .

Proof. We denote the p.d.f. of the random variable X as p X ( · ) . By lemma statement, we know:

Denote the p.d.f. of the function f ( X ) of the random variable X as p f ( X ) ( · ) . We want to derive p f ( X ) ( · ) for f ( X ) = γX . Since, f ( X ) = γX is an invertible function, we can use the change-ofvariable technique for deriving p f ( X ) ( · ) :

E.4 PID for the miscellaneous system M4.

Miscellaneous System M4 : Let ( M,X,Y ) be a system of random variables with joint distribution P ( M,X,Y ) . The joint distribution P ( M,X,Y ) describes the miscellaneous system M4 if it satisfies the following two properties:

M ∼ P ( M ) and has support [0 , 1] .

P ( X | M )= Bernoulli ( M ) , and P ( Y | M )= Bernoulli ( M + p -2 pM ) , where p ∈ [0 , 1] .

This system is adapted from the Bernoulli decomposition listed in Appendix A [26].

Proposition 5. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) describes the miscellaneous system M4 defined above. Then, ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

Proof. We briefly outline the proof structure. The proof can be divided into two parts:

In the first part, we explicitly construct a joint distribution Q MC ( M,X,Y ) having the Markovian structure M → X → Y .

In the second part, we conclude our proof by showing the Q MC ( M,X,Y ) constructed in the first part, lies in ∆ P .

Part 1: Specifying Q MC ( M,X,Y )

Since Q MC ( M,X,Y ) is the joint distribution of a Markov chain M → X → Y , we have Q MC ( M,X,Y ) = Q MC ( M ) Q MC ( X | M ) Q MC ( Y | X ) . Leveraging the special structure of Q MC ( M,X,Y ) , we specify Q MC ( M,X,Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) , and Q MC ( Y | X ) . We choose Q MC ( M ) , Q MC ( X | M ) as follows:

The distribution Q MC ( Y | X ) is specified through the following stochastic transformation:

where Z ∼ Bernoulli ( p ) and Z ⊥ ⊥ ( X,M ) .

Part 2: Showing Q MC ( M,X,Y ) ∈ ∆ P

For showing Q MC ( M,X,Y ) ∈ ∆ P , we first need to calculate Q MC ( Y | M ) :

Calculating Q MC ( Y | M ) : For calculating Q ( Y | M ) , we use the result of proposition 10 in [26], which states that if X ′ ∼ Bernoulli ( θ ) , Z ′ ∼ Bernoulli ( p ′ ) , Z ′ ⊥ ⊥ X ′ , and Y ′ = X ′ (1 -Z ′ ) +(1 -X ′ ) Z ′ , then Y ∼ Bernoulli ( θ + p ′ -2 p ′ θ ) .

Noting that Q MC ( X | M ) = Bernoulli ( M ) and Y = X (1 -Z )+ Z (1 -X ) , where Z ∼ Bernoulli ( p ) and Z ⊥ ⊥ X | M , we can calculate Q MC ( Y | M ) as follows:

Then, from (49) and (51), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; Y \ X ) = 0 . This concludes our proof.

E.5 PID for the miscellaneous system M5.

Miscellaneous System M5 : Let ( ⃗ M , X, Y ) be a system of random variables with joint distribution P ( ⃗ M , X, Y ) . The joint distribution P ( ⃗ M , X, Y ) describes the miscellaneous system M5 if it satisfies the following two properties:

⃗ M ∼ P ( ⃗ M ) and has support over simplex of dimension d .

P ( X | ⃗ M )= Categorical ( M 1 , . . . , M d ) , and P ( Y | ⃗ M )= Categorical ( ϕ 1 , . . . , ϕ d ) , where ϕ i = (1 -p ) M i + p / d .

This system is adapted from the categorical decomposition listed in Appendix A [26].

Proposition 6. Let ⃗ M , X, and Y be random variables whose joint distribution P ( ⃗ M , X, Y ) describes the miscellaneous system M5 defined above. Then, ∆ P contains a Markov chain of the form ⃗ M → X → Y and UI ( ⃗ M ; Y \ X ) = 0 .

Proof. We briefly outline the proof structure. The proof can be divided into two parts:

In the first part, we explicitly construct a joint distribution Q MC ( ⃗ M , X, Y ) having the Markovian structure ⃗ M → X → Y .

In the second part, we conclude our proof by showing the Q MC ( ⃗ M , X, Y ) constructed in the first part, lies in ∆ P .

Part 1: Specifying Q MC ( ⃗ M , X, Y )

Since Q MC ( ⃗ M , X, Y ) is the joint distribution of a Markov chain ⃗ M → X → Y , we have Q MC ( ⃗ M , X, Y ) = Q MC ( ⃗ M ) Q MC ( X | ⃗ M ) Q MC ( Y | X ) . Taking advantage of this special structure of Q MC ( ⃗ M , X, Y ) , we specify Q MC ( ⃗ M , X, Y ) by individually specifying Q MC ( ⃗ M ) , Q MC ( X | ⃗ M ) , and Q MC ( Y | X ) . We choose Q MC ( ⃗ M ) , Q MC ( X | ⃗ M ) as follows:

The distribution Q MC ( Y | X ) is specified through the following transformation:

where Z ∼ Bernoulli ( p ) , D ∼ Categorical ( 1 d , . . . , 1 d ) , and D , Z , and ( X, ⃗ M ) are jointly independent.

Part 2: Showing Q MC ( ⃗ M , X, Y ) ∈ ∆ P

For showing Q MC ( ⃗ M , X, Y ) ∈ ∆ P , we first need to calculate Q MC ( Y | ⃗ M ) :

Calculating Q MC ( Y | ⃗ M ) : For calculating Q MC ( Y | ⃗ M ) , we use the result of proposition 11 in [26], which states that:

If X ′ ∼ Categorical ( θ 1 , . . . , θ d ) , Z ′ ∼ Bernoulli ( p ′ ) , D ′ ∼ Categorical ( 1 / d , . . . , 1 / d ) , and Y ′ = X ′ (1 -Z ′ ) + Z ′ D ′ with X ′ , Z ′ , and D ′ jointly independent, then Y ′ ∼ Categorical ( ϕ ′ 1 , . . . , ϕ ′ d ) with ϕ ′ i = (1 -p ′ ) θ i + p ′ / d .

Noting that Q MC ( X | ⃗ M ) = Categorical ( M 1 , . . . , M d ) and Y = X (1 -Z ) + ZD , with X , D , and Z jointly conditionally independent conditioned on M , we have:

From (54) and (56), we can conclude:

Hence, Q MC ( ⃗ M , X, Y ) ∈ ∆ P and consequently, by proposition 1, UI ( ⃗ M ; Y \ X ) = 0 . This concludes our proof.

E.6 PID for the miscellaneous system M6.

Miscellaneous System M6 : Let ( M, ⃗ X , ⃗ Y ) be a system of random variables with joint distribution P ( M, ⃗ X , ⃗ Y ) . The joint distribution P ( M, ⃗ X , ⃗ Y ) describes the miscellaneous system M6 if it satisfies the following two properties:

M ∼ P ( M ) and has support N .

P ( ⃗ X | M )= Multinomial d X ( ⃗ X ; M,⃗ p X ) , and P ( ⃗ Y | M )= Multinomial d Y ( ⃗ Y ; M,⃗ p Y ) .

Proposition 7. Let M be a random variable. Furthermore, assume ⃗ X and ⃗ Y are random vectors of size d X and d Y , respectively, such that P ( M, ⃗ X , ⃗ Y ) describes miscellaneous system M6 .

If, min i ∈ [ d Y ] p Y i ≥ min i ∈ [ d X ] p X i , then there exists a Markov chain of the form M → ⃗ X → ⃗ Y in ∆ P , and UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

If, min i ∈ [ d Y ] p Y i ≤ min i ∈ [ d X ] p X i , then there exists a Markov chain of the form M → ⃗ Y → ⃗ X in ∆ P , and UI ( M ; ⃗ X \ ⃗ Y ) = 0 .

Proof. We only provide an explicit proof of condition 1. The proof of condition 2 follows the same steps as the proof of condition 1 with parameters of ⃗ X and ⃗ Y switched.

We briefly outline the proof structure. The proof can be divided into two major parts:

In the first part, we explicitly construct a joint distribution Q MC ( M, ⃗ X , ⃗ Y ) having the Markovian structure M → ⃗ X → ⃗ Y .

In the second part, we conclude our proof by showing the Q MC ( M, ⃗ X , ⃗ Y ) constructed in the first part, lies in ∆ P .

Part 1: Specifying Q MC ( M, ⃗ X , ⃗ Y )

Since Q MC ( M, ⃗ X , ⃗ Y ) is the joint distribution of a Markov chain M → ⃗ X → ⃗ Y , we have Q MC ( M, ⃗ X , ⃗ Y ) = Q MC ( M ) Q MC ( ⃗ X | M ) Q MC ( ⃗ Y | ⃗ X ) . Taking advantage of this special structure of Q MC ( M, ⃗ X , ⃗ Y ) , we specify Q MC ( M, ⃗ X , ⃗ Y ) by individually specifying Q MC ( M ) , Q MC ( ⃗ X | M ) , and Q MC ( ⃗ Y | ⃗ X ) . We choose Q MC ( M ) , Q MC ( ⃗ X | M ) as follows:

Specifying Q ( ⃗ Y | ⃗ X ) : Let k ∗ X = arg min i ∈ [ d X ] p X i , and k ∗ Y = arg min i ∈ [ d Y ] p Y i . Then, Q ( ⃗ Y | ⃗ X )= Multinomial d Y ( ⃗ Y ; ∑ i ∈ [ d X ] \ k ∗ X x i , p ∗ Y ) , where

We first show the proposed Q MC ( ⃗ Y | ⃗ X ) is a valid distribution. By condition 1:

where the last inequality is due to ⃗ p X and ⃗ p Y forming valid multinomial distributions.

Part 2: Showing Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P

For showing Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P , we first need to calculate Q MC ( ⃗ Y | M ) :

Calculating Q MC ( ⃗ Y | M ) : We use lemma 4 to calculate Q ( ⃗ Y | M ) . Note that Q MC ( ⃗ X | M ) = Multinomial d X ( M,⃗ p X ) , hence we have:

Q MC ( Y | M ) = Multinomial d Y ( M,⃗ p ′ Y ) , where ,

Then, from (59) and (64), we can conclude:

Hence, Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; ⃗ Y \ ⃗ X ) = 0 . This concludes our proof.

Lemma 4. Suppose ⃗ X ∼ Multinomial d X ( n, ⃗ p X ) and ⃗ Y | ⃗ X ∼ Multinomial d Y ( ∑ j ∈I X j , ⃗ q ) , where I ⊂ { 1 , . . . , d X } . Then ⃗ Y ∼ Multinomial d Y ( n,⃗ q ∗ ) , where

Proof. Without loss of generality, reshuffle the class indices of ⃗ Y such that I = { i :1 ≤ i ≤ w } , where |I| = w . Let C = { ⃗ x ′ ∈ N d X -1 0 : ∑ d X -1 i =1 x i ≤ n } . By the law of total probability,

Remove from the sum all terms that do not depend on ⃗ x . Note that y d Y = ∑ j ∈I x j -∑ d Y -1 i =1 y i and q y d Y d Y = ( 1 -∑ d Y i =1 q i ) y d Y are both functions of ⃗ y . Let y s = ∑ d Y -1 i =1 y i and q s = ∑ d Y -1 i =1 q i .

Form a multinomial coefficient outside the sum:

Let I 0 = { i : w < i ≤ d X } be the set of indices that are not taken in the number of trials for ⃗ Y . Consider the sum over ⃗ x ∈ C alone in (71). Separating terms that belong to I and I 0 ,

Let C ′ = { ⃗ x ′ ∈ N d X -1 0 : ∑ d X -1 i =1 x i -∑ d Y -1 i =1 y i ≤ n -∑ d Y -1 i =1 y i } . Note ⃗ x ∈ C ⇔ ⃗ x ∈ C ′ , so we can equivalently sum over elements in C ′ . Perform a change of variable with u = ∑ j ∈I x j -y s and define B = { x j ∈ ⃗ x : j ∈ I , ∑ j ∈I x j = u + y s } . Then the sum becomes

Note that the inner sum simplifies by the multinomial theorem:

Reapply the multinomial theorem to the sum of ⃗ x over C ′ :

Inserting (76) into (71), the marginal becomes

where ⃗ q ∗ is defined as in (67).

F Proof of Theorem 1

In this section, we provide the proof of theorem 1. Since theorem 1 provides the analytical PID terms for the univariate affine continuous stable system defined in Sec. 4, we briefly restate certain key properties of the univariate stable distributions and the corresponding univariate affine continuous stable system for convenience.

F.1 Univariate continuous stable distribution

Univariate continuous stable distribution are a large class of distributions that naturally arise in the context of generalized central limit theorems. We refer the reader to Appx. N.2 for more details on univariate continuous stable distributions. We now list certain key properties of univariate continuous stable distributions that we make use of in the proof of theorem 1:

If X is distributed according to a univariate continuous stable distribution, then the sum of two independent copies of X , denoted as X 1 and X 2 , follows the same univariate continuous stable distribution upto a scaling and translation operation, i.e., aX 1 + bX 2 d = cX + d for a, b, c > 0 and d ∈ R .

The p.d.f. of univariate continuous stable distributions is characterized by four parameters: stability parameter denoted as α ∈ (0 , 2] , skewness parameter denoted as β ∈ [ -1 , 1] , scale parameters denoted as γ ∈ (0 , ∞ ) , and location parameter denoted as µ ∈ R . We denote the p.d.f. of a univariate continuous stable distribution as p CS ( α, β, γ, µ ) .

F.2 Definition of univariate affine continuous stable system

Let ( M,X,Y ) be a system of random variables with joint distribution P ( M,X,Y ) . The joint distribution P ( M,X,Y ) describes the univariate affine continuous stable system if it satisfies the following two properties:

M ∼ P ( M ) having some support set M⊆ R .

The conditional distributions of random variables X and Y conditioned on M can be expressed through univariate continuous stable distributions with an affine dependence on M , i.e., P ( X | M )= p CS ( α, β X , γ X , aM + b ) and P ( Y | M )= p CS ( α, β Y sgn ( ac ) , γ Y , cM + d ) , where a, b, c, d ∈ R .

F.3 Formal proof of Theorem 1

We first discuss and provide the provide the proofs of lemma 5 and lemma 6 that we will use for proving theorem 1. In the following proofs, sgn ( x ) denotes the sign function:

̸

Lemma 5. If X ∼ p CS ( α, β, γ, µ ) , then for any η = 0 and κ ∈ R , we have

Proof. See the proof of proposition 1.4 part (b) in [23].

Lemma 6. If X 1 ∼ p CS ( α, β 1 , γ 1 , µ 1 ) and X 2 ∼ p CS ( α, β 2 , γ 2 , µ 2 ) are independent, then X 1 + X 2 ∼ p CS ( α, β, γ, µ ) where

Proof. See the proof of proposition 1.4 part (c) in [23].

Theorem 1. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) describes a univariate affine continuous stable system. Without the loss of generality, assume | a | / γ X ≥ | c | / γ Y . If 1 -β Y ≥ ( γ X | c | / γ Y | a | ) α (1 -β X ) , and 1 + β Y ≥ ( γ X | c | / γ Y | a | ) α (1 + β X ) , then ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

Proof. We first note that we can always assume | a | / γ X ≥ | c | / γ Y without the loss of generality because if | c | / γ Y ≥ | a | / γ X , then we can always switch our nomenclature to refer to Y as X , and X as Y .

We now briefly outline the proof structure. We divide the proof into two cases | a | / γ X > | c | / γ Y and | a | / γ X = | c | / γ Y . The proof for both cases follows essentially the same structure consisting of two major parts:

In the first part, we explicitly construct a joint distribution Q MC ( M,X,Y ) having the Markovian structure M → X → Y .

In the second part, we show that the Q MC ( M,X,Y ) constructed in the first part, lies in ∆ P . Therefore, we can then apply the result of proposition 1 to conclude UI ( M ; Y \ X ) = 0 .

Case 1: | a | / γ X > | c | / γ Y

Part 1: Specifying Q MC ( M,X,Y )

We explicitly construct the desired Markov chain M → X → Y by constructing a larger Markov chain M → X → X ′ → Y ′ → Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M → X → Y .

Denote the joint distribution of the Markov chain M → X → X ′ → Y ′ → Y as Q MC ( M,X,X ′ , Y ′ , Y ) . We can decompose Q MC ( M,X,X ′ , Y ′ , Y ) by utilizing its Markovian structure as follows:

Consequently, we can specify/construct the distribution Q MC ( M,X,X ′ , Y ′ , Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) , Q MC ( X ′ | X ) , Q MC ( Y ′ | X ′ ) , and Q MC ( Y | Y ′ ) .

Specifying Q MC ( M ) and Q MC ( X | M ) : We choose Q MC ( M ) and Q MC ( X | M ) as follows:

where P ( M ) and P ( X | M ) are marginal distributions derived from the original joint distribution P ( M,X,Y ) (discussed in the theorem statement) over which the bivariate PID is being calculated.

̸

Specifying Q MC ( X ′ | X ) : The distribution Q MC ( X ′ | X ) is specified by the following deterministic transformation:

̸

We will also derive the distribution Q ( X ′ | M ) before proceeding with our construction, as we will need it later in the proof to show that the constructed Q MC ( M,X,Y ) ∈ ∆ P .

Deriving Q MC ( X ′ | M ) : Since Q MC ( X | M = m ) = p CS ( α, β X , γ X , am + b ) for a fixed m and X ′ is scaled and translated version of X , we can use lemma 5 to derive Q MC ( X ′ | M ) . The exact expression of Q MC ( X ′ | M = m ) is provided in (83).

̸

̸

Specifying Q MC ( Y ′ | X ′ ) : In order to define Q MC ( Y ′ | X ′ ) , we need to define an auxiliary variable ϵ . The random variable ϵ ∼ p CS ( ϵ ; α, β ′ , γ ′ , µ ′ ) , where

In order for ϵ to have a legitimate continuous stable distribution, we need to ensure that β ′ , γ ′ , and µ ′ lie within their appropriate bounds as specified in Appx. F.1. It is trivial to see that µ ′ ∈ R .

Showing γ ′ ∈ (0 , ∞ ) . Under the assumption of the case 1, we know that:

Manipulating the above inequality, we obtain:

Hence, (85) shows that γ ′ ∈ (0 , ∞ ) .

Lastly, showing that β ′ ∈ [ -1 , 1] . Note that the condition β ′ ∈ [ -1 , 1] can be equivalently expressed as | β ′ | ≤ 1 . For showing that β ′ ∈ [ -1 , 1] , we will show that the inequality | β ′ | ≤ 1 is equivalent to the inequalities: ( γ X | c | / | a | γ Y ) α (1 -β X ) ≤ 1 -β Y and (1 + β X ) ( γ X | c | / | a | γ Y ) α ≤ 1 + β Y , where the last two inequalities hold by the theorem assumptions:

Multiplying ( γ Y / | c | ) α -( γ X / | a | ) α on both sides of the inequalities in (86).

̸

Adding ( γ X / | a | ) α β X on both sides of the inequalities in (87).

Dividing ( γ Y / | c | ) α on both sides of the inequalities in (88).

Simplifying the inequalities in (89).

Multiplying -1 on both sides in (91):

Hence, by above analysis we can conclude that the inequality | β ′ | ≤ 1 is equivalent to the inequalities: ( γ X | c | / | a | γ Y ) α (1 -β X ) ≤ 1 -β Y and (1+ β X ) ( γ X | c | / | a | γ Y ) α ≤ 1+ β Y , which ensures β ′ ∈ [ -1 , 1] .

As β ′ ∈ [ -1 , 1] , γ ′ ∈ (0 , ∞ ) and µ ′ ∈ R , we can conclude ϵ follows a legitimate univariate continuous stable distribution. Furthermore, we define ϵ ⊥ ⊥ ( M,X,X ′ ) , i.e., Pr( ϵ | M,X,X ′ ) = Pr( ϵ ) and Pr( M,X,X ′ | ϵ ) = Pr( M,X,X ′ ) .

We now use ϵ to specify Q MC ( Y ′ | X ′ ) as follows:

We also derive the distribution Q ( Y ′ | M ) before proceeding with our construction, as we will need it later in the proof to show that the constructed Q MC ( M,X,Y ) ∈ ∆ P .

Deriving Q MC ( Y ′ | M ) : We will use the distributions Q MC ( X ′ | M ) and Q MC ( ϵ ) to derive the distribution of Q MC ( Y ′ | M ) :

Using the fact that ϵ ⊥ ⊥ M and ϵ ⊥ ⊥ X ′ | M , we have:

Since Q MC ( ϵ ) and Q MC ( X ′ | M = m ) are univariate continuous stable distributions for a fixed m (see (84) and (83)), equation (94) describes a convolution of two univariate stable distributions for a fixed m . Convolution of two univariate continuous stable distribution is akin to summing two independent random variable having the same univariate continuous stable distributions, and

consequently Q MC ( Y ′ | M ) can be derived using lemma 6. Hence,

Specifying Q MC ( Y | Y ′ ) : Q MC ( Y | Y ′ ) is specified by the following deterministic transformation:

̸

Finally, we construct the desired Q MC ( M,X,Y ) from Q MC ( M,X,X ′ , Y ′ , Y ) by marginalizing X ′ and Y ′ .

Part 2: Showing Q MC ( M,X,Y ) ∈ ∆ P

For showing that Q MC ( M,X,Y ) ∈ ∆ P , we first need to calculate Q MC ( Y | M ) .

Calculating Q ( Y | M ) : We use lemma 5 to calculate Q ( Y | M ) as from (95), we know that Q MC ( Y ′ | M ) = p CS ( Y ′ ; α, sgn ( a ) β Y , γ Y / | c | , M + d / c ) is a continuous stable distribution for a fixed m , and Y a translated and scaled version of Y ′ . The exact expression of Q MC ( Y | M = m ) is provided in (97).

̸

From (81) and (97), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; Y \ X ) = 0 , concluding our proof for case 1.

Case 2: | a | / γ X = | c | / γ Y

The proof of case 2 is extremely similar to case 1. For case 2, we are able to directly construct the distribution Q MC ( M,X,Y ) without the need of specifying a larger Markov chain.

Part 1: Specifying Q MC ( M,X,Y )

Similarly to case 1, we specify Q MC ( M,X,Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) and Q MC ( Y | X ) . The distributions Q MC ( M ) and Q MC ( X | M ) are as follows:

̸

̸

The distribution Q MC ( Y | X ) is defined using the following deterministic transformation:

̸

Part 2: Showing Q MC ( M,X,Y ) ∈ ∆ P

For showing that Q MC ( M,X,Y ) ∈ ∆ P , we first need to calculate Q MC ( Y | M ) .

Calculating Q ( Y | M ) : We use lemma 5 to calculate Q ( Y | M ) as Q MC ( X | M = m ) is a continuous stable distribution for a fixed m , and Y a translated and scaled version of X . The exact expression of Q MC ( Y | M = m ) is provided in (102).

̸

From (100) and (102), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; Y \ X ) = 0 , concluding our proof for case 2 and theorem 1.

G Proof of Theorem 2

In this section, we provide the proof of theorem 2. Since theorem 2 provides the analytical PID terms for system 1 of the multivariate affine continuous stable system defined in Sec. 4, we briefly restate certain key properties of the corresponding independent component multivariate stable distributions and the corresponding multivariate affine continuous stable system for convenience.

G.1 Independent component multivariate stable distribution

Independent component multivariate stable distributions are a specific multivariate generalization of univariate continuous stable distributions describing a collection of independent random variables, with each random variable distributed according to a univariate continuous stable distribution. We refer the reader to Appx. N.3 for more details on multivariate continuous stable distributions. We now list certain key properties of independent component multivariate stable distributions that we make use of in the proof of theorem 2:

If ⃗ X is distributed according to an independent component multivariate stable distribution, then the sum of two independent copies of ⃗ X , denoted as ⃗ X 1 and ⃗ X 2 , follows an independent component multivariate stable distribution up to a scaling and translation operation, i.e., a ⃗ X 1 + b ⃗ X 2 d = c ⃗ X + ⃗ d for a, b, c > 0 and ⃗ d ∈ R .

We denote the p.d.f. of the independent component multivariate stable continuous distribution as p CS -IC ( α, ⃗ β , ⃗ γ , ⃗ µ ) , where

In general, the p.d.f. of independent component multivariate stable distribution do not have a closed-form analytical expression, and are expressed through their characteristic function.

The characteristic function of a d -dimensional random vector ⃗ X having independent component multivariate distribution p CS -IC ( α, ⃗ β , ⃗ γ , ⃗ µ ) is expressed as follows:

̸

The random vector ⃗ X = [ X 1 . . . X d ] T having the distribution p CS -IC ( α, ⃗ β , ⃗ γ , ⃗ µ ) is essentially a collection of independent random variables { X j } d j =1 , where X j ∼ p CS ( α, β j , γ j , µ j ) .

G.2 Definition of the system 1 of the multivariate affine continuous stable system

Let the random variable M ∼ P ( M ) with support M ⊆ R . ⃗ X and ⃗ Y are d X -dimensional and d Y -dimensional random vectors, respectively. The joint distribution P ( M, ⃗ X , ⃗ Y ) describes system 1 of the multivariate affine continuous stable system if it satisfies the following two properties

M ∼ P ( M ) having some support set M⊆ R .

The conditional distributions of random variables ⃗ X and ⃗ Y conditioned on M are expressed as follows:

where ⃗ Z X ∼ p CS -IC ( α, ⃗ 0 d X , ⃗ 1 d X , ⃗ 0 d X ) , ⃗ Z Y ∼ p CS -IC ( α, ⃗ 0 d Y , ⃗ 1 d Y , ⃗ 0 d Y ) , A X and A Y are invertible matrices, ⃗ H X , ⃗ b X ∈ R d X , and ⃗ H Y , ⃗ b Y ∈ R d Y .

G.3 Formal proof of Theorem 2

Theorem 2. Let the joint distribution P ( M, ⃗ X , ⃗ Y ) of M , ⃗ X , and ⃗ Y describe the system 1 of the multivariate affine continuous stable system. Without the loss of generality, assume ∥ A -1 Y ⃗ H Y ∥ κ ≤ ∥ A -1 X ⃗ H X ∥ κ , where κ = α / α -1 ∀ α ∈ (1 , 2] and κ = ∞∀ α ∈ (0 , 1] . Then, ∆ P contains a Markov chain of the form M → ⃗ X → ⃗ Y and UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Proof. We first note that we can always assume ∥ A -1 Y ⃗ H Y ∥ κ ≤ ∥ A -1 X ⃗ H X ∥ κ without the loss of generality because if ∥ A -1 Y ⃗ H Y ∥ κ ≥ ∥ A -1 X ⃗ H X ∥ κ , then we can always switch our nomenclature to refer to ⃗ Y as ⃗ X , and ⃗ X as ⃗ Y .

The proof of above theorem relies on the result of lemma 7 and the fact that the linear system described in (106) can always be reduced to the special case of the linear system used in lemma 7. We briefly outline the proof structure.

In the first part, we explicitly construct a joint distribution Q MC ( M, ⃗ X , ⃗ Y ) having the Markovian structure M → ⃗ X → ⃗ Y .

In the second part, we show that the Q MC ( M, ⃗ X , ⃗ Y ) constructed in the first part, lies in ∆ P . Therefore, we can then apply the result of proposition 1 to conclude UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Part 1: Specifying Q MC ( M, ⃗ X , ⃗ Y )

We explicitly construct the desired Markov chain M → ⃗ X → ⃗ Y by constructing a larger Markov chain M → ⃗ X → ⃗ X ′ → ⃗ Y ′ → ⃗ Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M → ⃗ X → ⃗ Y .

Denote the joint distribution of the Markov chain M → ⃗ X → ⃗ X ′ → ⃗ Y ′ → ⃗ Y as Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ′ , ⃗ Y ) . We can decompose Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ′ , ⃗ Y ) by utilizing its Markovian structure as follows:

Consequently, we can specify/construct the distribution Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ′ , ⃗ Y ) by individually specifying Q MC ( M ) , Q MC ( ⃗ X | M ) , Q MC ( ⃗ X ′ | ⃗ X ) , Q MC ( ⃗ Y ′ | ⃗ X ′ ) , and Q MC ( ⃗ Y | ⃗ Y ′ ) . We specify Q MC ( M, ⃗ X ) = P ( M, ⃗ X ) and Q ( ⃗ X ′ | ⃗ X ) using the following deterministic transformation:

Substituting (106) in the above equation:

Specifying Q MC ( ⃗ Y ′ | ⃗ X ′ ) : We define ⃗ Y ′ according to the following affine system:

Observe that both ⃗ X ′ and ⃗ Y ′ can be described as an affine function of M satisfying the special structure described in lemma 7 (see (115)). Furthermore, Q MC ( M ) = P ( M ) by construction. Consequently, Q MC ( M ) satisfies the properties required by the distribution of M outlined in lemma 7, as P ( M ) in both lemma 7 and theorem 2 follow the same properties. Since ∥ A -1 Y ⃗ H Y ∥ κ ≤ ∥ A -1 X ⃗ H X ∥ κ by the assumption in the theorem, we can apply the result of lemma 7 to construct Q MC ( ⃗ Y ′ | ⃗ X ′ ) , where ⃗ H X and ⃗ H Y would be replaced by A -1 X ⃗ H X and A -1 Y ⃗ H Y , respectively.

Lastly, we specify Q MC ( ⃗ Y | ⃗ Y ′ ) through the following linear system:

Finally, we construct the desired Q MC ( M, ⃗ X , ⃗ Y ) from Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ′ , ⃗ Y ) by marginalizing ⃗ X ′ and ⃗ Y ′ .

For showing that Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P , we first need to derive Q MC ( ⃗ Y | M ) .

Deriving Q ( ⃗ Y | M ) : Substituting (109) in (110) we obtain:

Inspecting the above equation, we can conclude:

From (112), we can conclude:

We know that by construction:

Hence, Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; ⃗ Y \ ⃗ X ) = 0 , concluding our proof.

Lemma 7. Let the joint distribution of M , ⃗ X and ⃗ Y , denoted as P ( M, ⃗ X , ⃗ Y ) , describe the system 1 of the multivariate affine continuous stable system. Furthermore, assume A X = I d X , A Y = I d Y , ⃗ b X = ⃗ 0 d X and ⃗ b Y = ⃗ 0 d Y , where I d is a d × d identity matrix and ⃗ 0 d is d -dimensional vector of zeros. Then,

If ∥ ⃗ H Y ∥ κ ≤ ∥ ⃗ H X ∥ κ , where κ = α / α -1 ∀ α ∈ (1 , 2] and κ = ∞∀ α ∈ (0 , 1] , then ∆ P contains a Markov chain of the form M → ⃗ X → ⃗ Y and UI ( M ; ⃗ Y / ⃗ X ) = 0 .

If ∥ ⃗ H X ∥ κ ≤ ∥ ⃗ H Y ∥ κ , where κ = α / α -1 ∀ α ∈ (1 , 2] and κ = ∞∀ α ∈ (0 , 1] , then ∆ P contains a Markov chain of the form M → ⃗ Y → ⃗ X and UI ( M ; ⃗ X / ⃗ Y ) = 0 .

Proof. We only provide an explicit proof of condition 1. Condition 2 is essentially the same as condition 1, with the parameters of X and Y switched. Consequently, the proof of condition 2 follows the same steps as the proof of condition 1, with parameters about X and Y switched.

Proof of condition 1 :

We briefly outline the proof structure.

In the first part, we explicitly construct a joint distribution Q MC ( M, ⃗ X , ⃗ Y ) having the Markovian structure M → ⃗ X → ⃗ Y .

In the second part, we show that the Q MC ( M, ⃗ X , ⃗ Y ) constructed in the first part, lies in ∆ P . Therefore, we can then apply the result of proposition 1 to conclude UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Part 1: Specifying Q MC ( M, ⃗ X , ⃗ Y )

We explicitly construct the desired Markov chain M → ⃗ X → ⃗ Y by constructing a larger Markov chain M → ⃗ X → ⃗ X ′ → ⃗ Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M → ⃗ X → ⃗ Y .

Denote the joint distribution of the Markov chain M → ⃗ X → ⃗ X ′ → ⃗ Y as Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ) . We can decompose Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ) by utilizing its Markovian structure as follows:

Consequently, we can specify/construct the distribution Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ) by individually specifying Q MC ( M ) , Q MC ( ⃗ X | M ) , Q MC ( ⃗ X ′ | ⃗ X ) , and Q MC ( ⃗ Y | ⃗ X ′ ) .

Under the assumptions of lemma 7, we can simplify the linear system describing P ( M, ⃗ X , ⃗ Y ) shown in (106) as follows:

Correspondingly, we can write the characteristic functions of conditional distributions P ( ⃗ X | M = m ) and P ( ⃗ Y | M = m ) for a fixed m by employing lemma 9:

where the characteristic function can be derived by realizing that for a given m , ⃗ H X m and ⃗ H Y m are constants being added to the random vectors ⃗ Z X ∼ p CS -IC ( α, ⃗ 0 d x , ⃗ 1 d x , ⃗ 0 d x ) and ⃗ Z Y ∼ p CS -IC ( α, ⃗ 0 d y , ⃗ 1 d y , ⃗ 0 d y ) , respectively, and then using the result of lemma 9.

Specifying Q MC ( M ) and Q MC ( ⃗ X | M ) : We specify Q MC ( M ) and Q MC ( ⃗ X | M ) as follows:

where P ( M ) and P ( ⃗ X | M ) are marginal distributions derived from the original joint distribution P ( M, ⃗ X , ⃗ Y ) (discussed in the lemma statement) over which the bivariate PID is being calculated.

Specifying Q MC ( ⃗ X ′ | ⃗ X ) : We specify Q MC ( ⃗ X ′ | ⃗ X ) through the following deterministic transformation:

where ⃗ H k X is as defined in lemma 10, and k ∈ [0 , ∞ ] .

We will also derive the distribution Q ( ⃗ X ′ | M ) before proceeding with our construction, as we will need it later in the proof to show that the constructed Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P .

Deriving Q MC ( ⃗ X ′ | M ) : To derive Q MC ( ⃗ X ′ | M ) , we represent ⃗ X ′ as function of M and ⃗ Z X in a similar manner to ⃗ X in (115), and then use the result of lemma 9 to derive its conditional characteristic function E [ e ⃗ t T ⃗ X ′ | M ] , and correspondingly the desired Q MC ( ⃗ X ′ | M ) .

We multiply ⃗ H Y ( ⃗ H k X ) T on both sides of equality in (115) to obtain the following linear equation:

Substituting (118) in the above equation, we obtain:

Using the result of lemma 10, i.e., ⃗ H Y ( ⃗ H k X ) T ⃗ H X = ⃗ H Y , in the above equation we obtain:

From (120), we can observe that ⃗ X ′ is an affine function of ⃗ Z X for a fixed m . Hence, we can employ the result of lemma 9 to derive the distribution of Q MC ( ⃗ X ′ | M ) (by deriving its corresponding characteristic function).

Note that the j -th column of ⃗ H Y ( ⃗ H k X ) T can be expressed as h ′ j ⃗ H Y , where h ′ j is the j -th component of ⃗ H k X . Consequently:

We substitute the value h ′ j in (121) using the definition of ⃗ H k X given in lemma 10. We will divide the substitution into two cases:

Case 1: k is finite, i.e., k ∈ [0 , ∞ ) . Then, we have h ′ j = | h X j | k / ∥ ⃗ H X ∥ 1+ k 1+ k , which implies:

̸

Case 2: k is infinite, i.e., k = ∞ . Then, we have h ′ j = 0 ∀ j = j ∗ , and h ′ j ∗ = 1 / h X 1 , which implies:

Combining (122) and (123), we can re-write E [ e i ⃗ t T ⃗ X ′ ∣ ∣ ∣ M = m ] as:

Specifying Q MC ( ⃗ Y | ⃗ X ′ ) : We specify Q MC ( ⃗ Y | ⃗ X ′ ) through the following stochastic transformation:

where ⃗ ϵ follows a multivariate stable distribution. Furthermore, we assume that ⃗ ϵ is jointly independent from ( ⃗ X , ⃗ X ′ , M ) , i.e., ⃗ ϵ ⊥ ⊥ ( ⃗ X , ⃗ X ′ , M ) . The characteristic function of ⃗ ϵ is defined as follows:

In order for Q MC ( ⃗ Y | ⃗ X ′ ) to be a valid distribution, we need to ensure that the distribution of ⃗ ϵ defined in (126) is a legitimate multivariate stable distribution. We will utilize property 1 for showing that ⃗ ϵ follows a stable multivariate stable distribution. Property 1 states a random vector is distributed according to an independent component multivariate stable distribution iff every 1 -dimensional projection of this random vector follows a univariate stable continuous distribution. Hence, to show that ⃗ ϵ is distributed according to a legitimate multivariate stable distribution, we will show that every 1 -dimensional projection of ⃗ ϵ follows a univariate stable continuous distribution.

Let ⃗ t ∈ R d Y , then the characteristic function of the 1-dimensional projection of ⃗ ϵ along ⃗ t , i.e., ⃗ t T ⃗ ϵ can be trivially deduced using (126).

Comparing the characteristic function of ⃗ t T ⃗ ϵ described in (127) with the standard characteristic function of a univariate stable characteristic function defined in (323), we can conclude ⃗ t T ⃗ ϵ ∼ p CS ( α, β ( ⃗ t ) , γ ( ⃗ t ) , µ ( ⃗ t )) , where:

In order for ⃗ t T ⃗ ϵ to have a legitimate univariate stable distribution, we just need to show that γ ( ⃗ t ) ≥ 0 ∀ ⃗ t ∈ R d Y . Note that γ ( ⃗ t ) = 0 would correspond to the case where all the mass of the distribution is centered at 0 , which does satisfy definition 2, and is an example of a degenerate univariate stable distribution. Let us analyze the function γ ( ⃗ t ) :

Applying lemma 8 on the term ∣ ∣ ∣ ⃗ H T Y ⃗ t ∣ ∣ ∣ , we obtain:

Applying (130) in (129), we obtain:

Denote l ( ⃗ t , α ) as follows:

In order to show γ ( ⃗ t ) ≥ 0 ∀ ⃗ t ∈ R d Y , it suffices to show that l ( ⃗ t , α ) ≥ 0 ∀ ⃗ t ∈ R d Y and α ∈ (0 , 2] . Case 1: α ∈ (1 , 2] .

For α ∈ (1 , 2] , we have:

Since, the above equation holds for any k ∈ [0 , ∞ ] , we choose k = 1 / ( α -1) . Substituting the form f ( k ) from (124) in the above equation, we obtain:

Note that ∥ ∥ ∥ ⃗ t ∥ ∥ ∥ α α ≥ 0 , and by condition 1 we know that ∥ ⃗ H Y ∥ α / α -1 ≤ ∥ ⃗ H X ∥ α / α -1 ⇒∥ ⃗ H Y ∥ α α / α -1 ≤ ∥ ⃗ H X ∥ α α / α -1 . Hence, using these two previous facts and the above equation, we can conclude:

Case 2: α ∈ (0 , 1] .

where ( a ) is derived using the monotonicity of L p norms in p , see [57]. More precisely, we use the inequality ∥ ⃗ t ∥ α ≥ ∥ ⃗ t ∥ 1 ∀ α ∈ (0 , 1] . Simplifying the above equation, we have:

Since, the above equation holds for any k ∈ [0 , ∞ ] , we choose k = ∞ . Substituting the form f ( k ) from (124) in the above equation, we obtain:

Note that ∥ ∥ ∥ ⃗ t ∥ ∥ ∥ α α ≥ 0 , and by condition 1 we know that ∥ ⃗ H Y ∥ ∞ ≤ ∥ ⃗ H X ∥ ∞ ⇒∥ ⃗ H Y ∥ α ∞ ≤ ∥ ⃗ H X ∥ α ∞ . Hence, using these two facts and the above equation, we can conclude:

From (135) and (139), we can conclude that l ( ⃗ t , α ) ≥ 0 ∀ ⃗ t ∈ R d Y and α ∈ (0 , 2] , which implies γ ( ⃗ t ) ≥ 0 ∀ ⃗ t ∈ R d Y . Hence, ⃗ ϵ follows a legitimate multivariate stable distribution.

Finally, we construct the desired Q MC ( M, ⃗ X , ⃗ Y ) from Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ Y ) by marginalizing ⃗ X ′ .

For showing that Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P , we first need to derive Q MC ( ⃗ Y | M ) .

Deriving Q ( ⃗ Y | M ) : We will derive Q MC ( ⃗ Y | M ) by using the fact that ⃗ ϵ ⊥ ⊥ ⃗ X ′ | M , hence their conditional characteristic functions would just result in multiplication.

where ( a ) is due to ⃗ ϵ ⊥ ⊥ ⃗ X ′ | M . Substituting the characteristic functions of ⃗ X ′ and ⃗ ϵ from (124) and (126), respectively, in (140):

By comparing the characteristic function shown in (141) with the characteristic function derived using P ( ⃗ Y | M ) shown in (116), we can conclude:

From (142) and (117), we can conclude:

We know that by construction:

Hence, Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; ⃗ Y \ ⃗ X ) = 0 , concluding our proof.

Lemma 8. Let ⃗ a = [ a 1 . . . a d ] ∈ R d and ⃗ b = [ b 1 . . . b d ] ∈ R d , then we have:

where ∥ ( · ) ∥ p is the standard L p norm.

Proof. Case 1 : α ∈ (1 , 2]

Analysing the LHS of (145)

where ( a ) is due to the sub-additivity of | ( · ) | operator, and ( b ) is due to Hölder's inequality [57]. Raising both sides of the inequality to α in (146) gives us the desired inequality. Since, α > 0 raising both sides of the inequalities to power α does not change the direction of inequality.

Case 2 : α ∈ (0 , 1]

Analysing the LHS of (145)

where ( a ) is again due to the sub-additivity of | ( · ) | operator, and ( b ) is due to the fact that | a i b i | ≤ | a i | max i ∈{ 1 ,...,d } | b i | ∀ i ∈ { 1 , . . . , d } . Raising both sides of the inequality to α in (148) gives us the desired inequality. Since, α > 0 raising both sides of the inequalities to power α does not change the direction of inequality.

Combining (147) and (149), we obtain the desired inequality.

Lemma 9. Let ⃗ Z ∼ p CS -IC ( α, ⃗ 0 d , ⃗ 1 d , ⃗ 0 d ) be a d -dimensional random vector, A = [ a ij ] n,d i,j =1 ∈ R n × d be a n × d matrix, and ⃗ b ∈ R d . Then, the characteristic function of A ⃗ Z + ⃗ b is given by (150)

where ⃗ a i is the i -th column of A .

Proof.

Using the fact that e i ⃗ t T ⃗ b is constant w.r.t expectation over ⃗ Z , and linearity of expectation, we have:

Simplifying the term [ ⃗ t T ⃗ a 1 . . . ⃗ t T ⃗ a d ] ⃗ Z by substituting Z = [ Z 1 . . . Z d ] T , we obtain:

Using properties of exponentials, we obtain:

Using the fact { Z j } d j =1 are jointly independent we obtain:

Using the fact that Z j ∼ p CS ( α, 0 , 1 , 0) and the result of lemma 5, we know that ⃗ t T ⃗ a j Z j ∼ p CS ( α, 0 , | ⃗ t T ⃗ a j | , 0) . Substituting the corresponding characteristic function of ⃗ t T ⃗ a j Z j using (323)

Lemma 10. Let ⃗ H 1 = [ h (1) 1 . . . h (1) d 1 ] T ∈ R d 1 and ⃗ H 2 = [ h (2) 1 . . . h (2) d 2 ] T ∈ R d 2 . Define ⃗ H k 1 as follows:

Furthermore, define ⃗ H ∞ 1 as follows:

Then, we have:

where ∥ ( · ) ∥ 1+ k is the standard L 1+ k norm [57]. We can alternatively state (154) in a more compact notation:

where the case of k = ∞ is to be understood as ⃗ H k 1 = ⃗ H ∞ 1 .

Proof. Case 1 : k ∈ [0 , ∞ ) , i.e., k is finite,

Case 2 : k = ∞ ,

where ( a ) is due to the fact that all elements of ⃗ H ∞ 1 are 0 except the j ∗ element, and ( b ) is due to the fact that h ∞ j ∗ = 1 / h (1) j ∗ .

H Proof of Theorem 3

In this section, we provide the proof of theorem 3. Since theorem 3 provides the analytical PID terms for the system 2 of the multivariate affine continuous stable system defined in Sec. 4, we briefly restate certain key properties of the corresponding elliptically-contoured multivariate stable distributions and the corresponding multivariate affine continuous stable system for convenience.

H.1 Elliptically-contoured multivariate stable distribution

Elliptically-contoured multivariate stable distributions are another example of a multivariate generalization of univariate continuous stable distributions. As the name suggests, the defining features of these distributions is that the corresponding p.d.f. has elliptical contours, similar to the multivariate Gaussian distribution. We refer the reader to Appx. N.3 for more details on multivariate continuous stable distributions. We now list certain key properties of elliptically-contoured multivariate stable distributions that we make use of in the proof of theorem 3:

We denote the p.d.f. of the elliptically-contoured continuous stable distribution as p CS -EC ( α, Σ , ⃗ µ ) , where Σ is a positive definite matrix, ⃗ µ ∈ R d , and α ∈ (0 , 2] . In general, the p.d.f. of Elliptically-contoured multivariate stable distribution do not have a closed-form analytical expression, and are expressed through their characteristic function.

The characteristic function of a d -dimensional random vector ⃗ X having ellipticallycontoured multivariate distribution p CS -EC ( α, Σ , ⃗ µ ) is expressed as follows:

H.2 Definition of the system 2 of the multivariate affine continuous stable system

Let the random variable M ∼ P ( M ) with support M ⊆ R . ⃗ X and ⃗ Y are d X -dimensional and d Y -dimensional random vectors, respectively. The joint distribution P ( M, ⃗ X , ⃗ Y ) describes the second system of the multivariate affine continuous stable distribution if it satisfies the following two properties:

M ∼ P ( M ) having some support set M⊆ R .

The conditional distributions of ⃗ X and ⃗ Y are expressed as follows:

where Σ X and Σ Y are positive definite matrices, ⃗ H X , ⃗ b X ∈ R d X , and ⃗ H Y , ⃗ b Y ∈ R d Y .

H.3 Formal proof of theorem 3

Theorem 3. Let the joint distribution P ( M, ⃗ X , ⃗ Y ) of M , ⃗ X , and ⃗ Y describe the system 2 of the multivariate affine continuous stable system. Define Σ -1 / 2 X and Σ -1 / 2 Y as the respective inverses of the matrices Σ 1 / 2 X and Σ 1 / 2 Y which satisfy: ( Σ 1 / 2 X ) T Σ 1 / 2 X = Σ X , and ( Σ 1 / 2 Y ) T Σ 1 / 2 Y = Σ Y . Without the loss of generality, assume ∥ Σ -1 / 2 Y ⃗ H Y ∥ 2 ≤ ∥ Σ -1 / 2 X ⃗ H X ∥ 2 . Then, ∆ P contains a Markov chain of the form M → ⃗ X → ⃗ Y and UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Proof. We first note two important observations:

We can always assume ∥ Σ -1 / 2 Y ⃗ H Y ∥ 2 ≤ ∥ Σ -1 / 2 X ⃗ H X ∥ 2 without the loss generality because if ∥ Σ -1 / 2 Y ⃗ H Y ∥ 2 ≥ ∥ Σ -1 / 2 X ⃗ H X ∥ 2 , then we can always switch our nomenclature to refer to ⃗ Y as ⃗ X , and ⃗ X as ⃗ Y .

Properties of positive definite matrices guarantee the existence of an invertible Σ 1 / 2 X and Σ 1 / 2 Y [58].

The proof of this theorem is a generalization of the proof used for the deriving a similar result for the special case of the multivariate Gaussian distribution discussed in [22], which borrows known results from Gaussian interference channels discussed in [48]. This proof follows the same structure as our earlier proofs.

In the first part, we explicitly construct a joint distribution Q MC ( M, ⃗ X , ⃗ Y ) having the Markovian structure M → ⃗ X → ⃗ Y .

In the second part, we show that the Q MC ( M, ⃗ X , ⃗ Y ) constructed in the first part, lies in ∆ P . Therefore, we can then apply the result of proposition 1 to conclude UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Part 1: Specifying Q MC ( M, ⃗ X , ⃗ Y )

We explicitly construct the desired Markov chain M → ⃗ X → ⃗ Y by constructing a larger Markov chain M → ⃗ X → ⃗ X ′ → ⃗ X ′′ → ⃗ Y ′ → ⃗ Y , and then marginalizing the larger Markov chain to obtain the desired Markov chain M → ⃗ X → ⃗ Y .

Denote the joint distribution of the Markov chain M → ⃗ X → ⃗ X ′ → ⃗ X ′′ → ⃗ Y ′ → ⃗ Y as Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ X ′′ , ⃗ Y ′ , ⃗ Y ) . We can decompose Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ X ′′ , ⃗ Y ′ , ⃗ Y ) by utilizing its Markovian structure as follows: Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ X ′′ , ⃗ Y ′ , ⃗ Y ) = Q MC ( M ) Q MC ( ⃗ X | M ) Q MC ( ⃗ X ′ | ⃗ X ) Q MC ( ⃗ X ′′ | ⃗ X ′ ) Q MC ( ⃗ Y ′ | ⃗ X ′′ ) Q MC ( ⃗ Y | ⃗ Y ′ ) . Consequently, we can specify/construct the distribution Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ X ′′ , ⃗ Y ′ , ⃗ Y ) by individually specifying Q MC ( M ) , Q MC ( ⃗ X | M ) , Q MC ( ⃗ X ′ | ⃗ X ) , Q MC ( ⃗ X ′′ | ⃗ X ′ ) , Q MC ( ⃗ Y ′ | ⃗ X ′′ ) , and Q MC ( ⃗ Y | ⃗ Y ′ ) .

Specifying Q MC ( M ) and Q MC ( ⃗ X | M ) : We choose Q MC ( M ) and Q MC ( ⃗ X | M ) as follows:

Specifying Q MC ( ⃗ X ′ | ⃗ X ) : Q MC ( ⃗ X ′ | ⃗ X ) is specified through the following deterministic transformation:

We will also derive the distribution Q ( ⃗ X ′ | M ) before proceeding with our construction, as we will need it later in the proof to show that the constructed Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P .

Deriving Q MC ( ⃗ X ′ | M ) : We use lemma 11 to derive Q MC ( ⃗ X ′ | M = m ) , as Q MC ( ⃗ X | M = m ) is an elliptically contoured multivariate stable distribution for a fixed m , and (159) defines a scaling and

translation operation.

where I d X is a d X × d X identity matrix, and ⃗ K X = ( Σ -1 / 2 X ) T ⃗ H X .

Specifying Q MC ( ⃗ X ′′ | ⃗ X ′ ) : Q MC ( ⃗ X ′′ | ⃗ X ′ ) is again specified through a deterministic transformation:

where ⃗ K Y = ( Σ -1 / 2 Y ) T ⃗ H Y and ⃗ K 1 X = 1 / ∥ ⃗ K X ∥ 2 2 ⃗ K X . We also derive the distribution Q ( ⃗ X ′ | M ) which will be needed for showing that the constructed Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P .

Deriving Q MC ( ⃗ X ′′ | M = m ) : We again use lemma 11 to calculate Q MC ( ⃗ X ′′ | M = m ) , as Q MC ( ⃗ X ′ | M = m ) is an elliptically contoured multivariate stable distribution (see (160)) for a fixed m , and (159) defines a scaling and translation operation.

From lemma 10, we know that ⃗ K Y ( ⃗ K 1 X ) T ⃗ K X = ⃗ K Y , which implies

Specifying Q MC ( ⃗ Y ′ | ⃗ X ′′ ) : We specify Q MC ( ⃗ Y ′ | ⃗ X ′′ ) through the following stochastic transformation:

where ⃗ ϵ follows a multivariate stable distribution. Furthermore, we assume that ⃗ ϵ is jointly independent from ( ⃗ X , ⃗ X ′ , ⃗ X ′′ , M ) , i.e., ⃗ ϵ ⊥ ⊥ ( ⃗ X , ⃗ X ′ , ⃗ X ′′ , M ) . The characteristic function of ⃗ ϵ is defined as follows:

In order for Q MC ( ⃗ Y ′ | ⃗ X ′′ ) to be a valid distribution, we need to ensure that the distribution of ⃗ ϵ defined in (164) is a legitimate multivariate stable distribution. Similarly to the proof of theorem 2, we will utilize property 1 for showing that ⃗ ϵ follows a stable multivariate stable distribution. Property 1 states a random vector is distributed according to a multivariate stable distribution iff every 1 -dimensional projection of this random vector follows a univariate stable continuous distribution. Hence, to show that ⃗ ϵ is distributed according to a legitimate multivariate stable distribution, we will show that every 1 -dimensional projection of ⃗ ϵ follows a univariate stable continuous distribution.

Let ⃗ t ∈ R d Y , then the characteristic function of the 1-dimensional projection of ⃗ ϵ along ⃗ t , i.e., ⃗ t T ⃗ ϵ can be trivially derived from (164).

Comparing the characteristic function of ⃗ t T ⃗ ϵ described in (127) with the standard characteristic function of a univariate stable characteristic function defined in (323), we can conclude ⃗ t T ⃗ ϵ ∼ p CS ( α, β ( ⃗ t ) , γ ( ⃗ t ) , µ ( ⃗ t )) , where:

In order for ⃗ t T ⃗ ϵ to have a legitimate univariate stable distribution, we just need to show that γ ( ⃗ t ) ≥ 0 ∀ ⃗ t ∈ R d Y . Note that γ ( ⃗ t ) = 0 would correspond to the case where all the mass of the distribution is centered at 0 , which does satisfy definition 2, and is an example of a degenerate univariate stable distribution.

By the assumption in theorem, ∥ ⃗ K Y ∥ 2 = ∥ Σ -1 / 2 Y ⃗ H Y ∥ 2 ≤ ∥ ⃗ K X ∥ 2 = ∥ Σ -1 / 2 X ⃗ H X ∥ 2 . Hence, we can use the result of lemma 12, which shows that ( ⃗ K Y ( ⃗ K 1 X ) T )( ⃗ K Y ( ⃗ K 1 X ) T ) T ⪯ I d Y . Using the fact that ( I d Y -( ⃗ K Y ( ⃗ K 1 X ) T )( ⃗ K Y ( ⃗ K 1 X ) T ) T ) is a positive semi-definite matrix, we have:

Hence, we know that ⃗ ϵ follows a legitimate multivariate stable distribution. We also derive Q MC ( ⃗ Y ′ | M ) which is needed for showing Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P .

Deriving Q MC ( ⃗ Y ′ | M ) : We will derive Q MC ( ⃗ Y ′ | M ) using the fact that ⃗ ϵ ⊥ ⊥ ⃗ X ′′ | M = m , hence their conditional characteristic functions would just result in multiplication.

where ( a ) is due to ⃗ ϵ ⊥ ⊥ ⃗ X ′′ | M . Substituting the characteristic functions of ⃗ X ′′ and ⃗ ϵ from (162) and (164), respectively, in (168)

By inspecting the characteristic function shown in (169), we can conclude:

Specifying Q MC ( ⃗ Y | ⃗ Y ′ ) : Lastly, we specify Q MC ( ⃗ Y | ⃗ Y ′ ) through the following deterministic transformation:

Finally, we construct the desired Q MC ( M, ⃗ X , ⃗ Y ) from Q MC ( M, ⃗ X , ⃗ X ′ , ⃗ X ′′ , ⃗ Y ′ , ⃗ Y ) by marginalizing ⃗ X ′ , ⃗ X ′′ , and ⃗ Y ′ .

Part 2: Showing Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P

For showing that Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P , we first need to derive Q MC ( ⃗ Y | M ) .

Deriving Q ( ⃗ Y | M ) : We use lemma 11 to calculate Q MC ( ⃗ Y | M = m ) , as Q MC ( ⃗ X | M = m ) is an elliptically contoured multivariate continuous stable distribution for a fixed m (see (170)), and (171) defines a scaling and translation operation.

where ( a ) is due to the fact that ( Σ 1 / 2 Y ) T Σ 1 / 2 Y = Σ Y , and ( b ) is due to the fact that ⃗ K Y = Σ -1 / 2 Y ⃗ H Y . Comparing (173) with P ( ⃗ Y | M = m ) shown in Appx. H.2, we can conclude:

From (174) and (158), we can conclude:

We know that by construction:

Hence, Q MC ( M, ⃗ X , ⃗ Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; ⃗ Y \ ⃗ X ) = 0 , concluding our proof.

Lemma 11. Let ⃗ X ∼ p CS -EC ( α, Σ , ⃗ µ ) be a d -dimensional random vector, where Σ is a d × d positive definite matrix, ⃗ µ ∈ R d , and α ∈ (0 , 2] . Furthermore, let A ∈ R n × d and ⃗ b ∈ R d . Then,

Proof. Calculating the characteristic function of A ⃗ X + ⃗ b :

Employing linearity of expectation to pull the term exp ( i ⃗ t T ⃗ b ) out of the expectation, as it is constant with respect to ⃗ X :

Substituting the formula of the characteristic function of an Elliptically-contoured multivariate stable distribution (given in (156)) for calculating E [ exp ( i ( ⃗ t ′ ) T ⃗ X )] as ⃗ X ∼ p CS -EC ( α, Σ , ⃗ µ ) :

Re-substituting ⃗ t ′ = ⃗ t T A in the above equation

By inspecting the characteristic function described in (178), we can determine that A ⃗ X + ⃗ b is distributed according to an elliptically-contoured multivariate stable distribution, i.e., A ⃗ X + ⃗ b ∼ p CS - EC ( α, AΣA T , A ⃗ µ + ⃗ b ) .

Lemma 12. Let ⃗ H 1 ∈ R d 1 and ⃗ H 2 ∈ R d 2 . Then, ∥ ⃗ H 1 ∥ 2 ≥ ∥ ⃗ H 2 ∥ 2 if and only if ∃ a matrix A ∈ R d 2 × d 1 such that ⃗ H 2 = A ⃗ H 1 , and AA T ⪯ I d 2 , with A = ⃗ H 2 ( ⃗ H 1 1 ) T , where B ⪯ C means B -C is a positive semi-definite matrix, ⃗ H 1 1 is as defined in lemma 10, and I d 2 is d 2 × d 2 identity matrix.

I Proof of Theorem 4

In this section, we provide the proof of theorem 4. Since theorem 4 provides the analytical PID terms for the univariate affine discrete stable system defined in Sec. 4, we briefly restate certain key properties of the univariate discrete stable distributions and the corresponding univariate affine discrete stable system for convenience.

I.1 Univariate discrete stable distribution

Univariate discrete stable distributions are the discrete analogues of univariate continuous stable distributions. We refer the reader to Appx. N.4 for more details on univariate discrete stable distributions. We now list certain key properties of univariate continuous stable distributions that we make use of in the proof of theorem 4:

If X is distributed according to a univariate discrete stable distribution, then X d = γ ◦ X 1 + (1 -γ ν ) 1 / ν ◦ X 2 , where X 1 and X 2 are two independent copies of X , ν ∈ (0 , 1] , γ ∈ [0 , 1] , and ◦ denotes the binomial thinning operation.

The p.m.f. of discrete stable distributions are characterized by two parameters: exponent ν ∈ (0 , 1] , and rate parameter τ ∈ (0 , ∞ ) . We denote the p.m.f. of a univariate discrete stable distribution as P DS ( ν, τ ) . In general, discrete stable distributions do not have a 'nice' analytical form consisting of well-known elementary functions. Consequently, the univariate discrete stable distribution are typically expressed through their probability generating function.

The probability generating function of a discrete random variable N having a stable discrete distribution is given by (179):

I.2 Definition of univariate affine discrete stable system

Let M , X , and Y be a system of random variables with the joint distribution P ( M,X,Y ) . The joint distribution P ( M,X,Y ) describes the univariate affine discrete stable system if it satisfies the following two properties:

M ∼ P ( M ) having some support set M⊆ (0 , ∞ ) .

The conditional distributions of random variables X and Y conditioned on M can be expressed through discrete stable family distributions with an affine dependence on M . Formally, P ( X | M = m )= P DS ( ν, am + b ) and P ( Y | M = m )= P DS ( ν, cm + d ) , where a, b, c, d ∈ (0 , ∞ ) .

I.3 Formal proof of Theorem 4

Theorem 4. Let M,X, and Y be random variables whose joint distribution P ( M,X,Y ) describes a univariate affine discrete stable system. Without the loss of generality, assume a ≥ c . If a / b ≥ c / d , then ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

Proof. We first note that we can always assume a ≥ c without the loss of generality because if a ≤ c , then we can always switch our nomenclature to refer to Y as X , and X as Y .

We briefly outline the proof structure.

In the first part, we explicitly construct a joint distribution Q MC ( M,X,Y ) having the Markovian structure M → X → Y .

In the second part, we show that the Q MC ( M,X,Y ) constructed in the first part lies in ∆ P . Therefore, we can then apply the result of proposition 1 to conclude UI ( M ; Y \ X ) = 0 .

Part 1: Specifying Q MC ( M,X,Y )

We explicitly construct the desired Markov chain M → X → Y by constructing a larger Markov chain M → X → X ′ → Y and then marginalizing the larger Markov chain to obtain the desired Markov chain M → X → Y .

Denote the joint distribution of the Markov chain M → X → X ′ → Y as Q MC ( M,X,X ′ , Y ) . We can decompose Q MC ( M,X,X ′ , Y ) by utilizing its Markovian structure as follows:

Consequently, we can specify/construct the distribution Q MC ( M,X,X ′ , Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) , Q MC ( X ′ | X ) , and Q MC ( Y | X ′ ) .

Specifying Q MC ( M ) and Q MC ( X | M ) : We choose Q MC ( M ) and Q MC ( X | M ) as follows:

where P ( M ) and P ( X | M ) are marginal distributions derived from the original joint distribution P ( M,X,Y ) (discussed in the theorem statement) over which the bivariate PID is being calculated.

Specifying Q MC ( X ′ | X ) : Q MC ( X ′ | X ) is specified through a binomial thinning operation, specifically:

Note that the above binomial thinning operation is valid as c ≤ a by the assumption in the theorem statement. We also derive Q MC ( X ′ | M ) as it is needed for showing Q MC ( M,X,Y ) ∈ ∆ P .

Deriving Q MC ( X ′ | M ) : We use lemma 13 to derive Q MC ( X ′ | M = m ) as Q MC ( X | M = m ) describes a univariate discrete stable distribution for a fixed m , and X ′ is a binomially-thinned version of X . The exact expression of Q MC ( X ′ | M = m ) is provided in (182).

Specifying Q MC ( Y | X ′ ) : Q MC ( Y | X ′ ) is specified through the following stochastic transformation:

where ϵ ∼ P DS ( ν, d -cb / a ) . For Q MC ( Y | X ′ ) to be a valid distribution, we need to ensure that ϵ is distributed according to a legitimate discrete stable distribution. In order to show ϵ is distributed according to a legitimate discrete stable distribution, we need to ensure that ν and d -cb / a lie within their appropriate bounds as specified in Sec. I.1. It is trivial to see ν ∈ (0 , 1] .

From the assumptions in the theorem statement, we know that

Hence, ϵ follows a legitimate discrete stable distribution. Furthermore, we choose ϵ ⊥ ⊥ ( M,X,X ′ ) .

Finally, we construct the desired Q MC ( M,X,Y ) from Q MC ( M,X,X ′ , Y ) by marginalizing X ′ .

For showing that Q MC ( M,X,Y ) ∈ ∆ P , we first need to derive Q MC ( Y | M ) .

Deriving Q ( Y | M ) : Note that ϵ follows discrete stable distribution P DS ( ν, d -bc / a ) and Q MC ( X ′ | M = m ) is also a discrete stable distribution P DS ( X ′ ; ν, cm + bc / a ) for a fixed m (see (182)). Furthermore, ϵ ⊥ ⊥ X ′ | M . Hence, using lemma 14, we have

As P ( Y | M ) = P DS ( ν, cM + d ) (see Appx. I.2), we have:

From (180) and (186), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; Y \ X ) = 0 , concluding our proof.

Lemma 13. Let N ∼ P DS ( ν, τ ) . Then, N 1 = γ ◦ N ∼ P DS ( ν, γ ν τ ) ∀ γ ∈ [0 , 1] , where ◦ is the binomial thinning operation.

Proof. The probability generating function of N , denoted as P N ( z ) , can be described using (179):

For calculating the probability generating function of N 1 , we use the following relationship between the probability generating function of a random variable R and its binomially-thinned version R ◦ p , with p ∈ [0 , 1] :

From (189) and (190), we can calculate the probability generating function of N 1 as follows:

Inspecting the probability generating function of N 1 in (191), we can conclude that N 1 ∼ P DS ( ν, γ ν τ ) .

Lemma 14. Let N 1 ∼ P DS ( ν, τ 1 ) , N 2 ∼ P DS ( ν, τ 2 ) , and N 1 ⊥ ⊥ N 2 . Then, N 1 + N 2 = N ∼ P DS ( n ; ν, τ 1 + τ 2 ) .

Proof. The probability generating function of N 1 and N 2 , denoted as P N 1 ( z ) and P N 2 ( z ) , can be described using (179):

Calculating the probability generating function of N = N 1 + N 2 ,

where ( a ) is due to the fact that N 1 ⊥ ⊥ N 2 , and ( b ) is achieved by substituting the explicit forms of P N 1 ( z ) and P N 2 ( z ) from (192). Inspecting the probability generating function of N 1 + N 2 in (193), we can conclude that N 1 + N 2 = N ∼ P DS ( n 1 ; ν, τ 1 + τ 2 ) .

J Proof of Theorem 5

In this section, we provide the proof of theorem 5. Since theorem 5 provides the analytical PID terms for the multivariate linear Poisson system defined in Sec. 4, we briefly restate certain key properties of multivariate Poisson distributions and the multivariate linear Poisson system for convenience.

J.1 Multivariate Poisson Distribution

We use the multivariate Poisson distribution proposed in [42]. For more details, see Appx. N.5. We list certain key properties of multivariate Poisson distributions that we make use of in the proof of theorem 5:

Under the definition of the multivariate Poisson distribution proposed in [42], each component of the random vector is a sum of independent Poisson random, i.e.,

where A is an appropriate matrix of 0 's and 1 's. We can decompose A = [ A 1 . . . A d ′ ] , where A i is a d × ( d i ) submatrix having no duplicate columns, and each of its columns containing exactly i ones and ( d -i ) zeros [44], and ⃗ N g = [ N g 1 . . . N g d N g 12 . . . N g ( d -1) d . . . N g d -( d ′ -1) ...d ] T , with N g i 1 ...i j ∼ Poisson ( λ i 1 ...i j ) ∀ ( i 1 , . . . , i j ) ∈ A d j , j ∈ [ d ′ ] . Furthermore, the random variables { N 1 , . . . , N d -( d ′ -1) ...d } are mutually independent.

We denote the p.m.f. of the multivariate Poisson distribution as Poisson ( d, d ′ , ⃗ Λ ) , where d ≥ d ′ , and

For d ′ = 1 , we have that ⃗ N is a collection of independent Poisson random variables, and when both d = d ′ = 1 , we recover the scalar Poisson distribution: Pr( N = n ) = e -λ λ n n ! , ∀ n ∈ N 0 .

Let ⃗ N ∼ Poisson ( d, d ′ , ⃗ Λ ) , where ⃗ N is a d -dimensional random vector. Then the corresponding p.m.f. of ⃗ N is described below:

Let ⃗ n ′ = [ n 12 . . . n ( d -1) d . . . n d -( d ′ -1) ...d ] T , and d ⃗ n ′ be the dimension of ⃗ n ′ , then:

where C = { ⃗ n ′ ∈ N d ⃗ n ′ 0 :( ⃗ a ′ i ) T ⃗ n ′ ≤ n i ∀ i ∈ [ d ] } , and

with ⃗ a ′ i being the i -th row of the matrix A ′ = [ A 2 . . . A d ′ ] .

J.2 Definition of the multivariate linear Poisson system

Let the random variable M ∼ P ( M ) . ⃗ X and ⃗ Y are d X -dimensional and d Y -dimensional random vectors, respectively. The joint distribution P ( M, ⃗ X , ⃗ Y ) describes the multivariate linear Poisson system if it satisfies the following two properties:

M ∼ P ( M ) having some support set M⊆ (0 , ∞ ) .

The conditional distributions of random vectors ⃗ X and ⃗ Y are multivariate Poisson distributions, i.e., P ( ⃗ X | M )= Poisson ( d X , d ′ X , ⃗ Λ X ) and P ( ⃗ Y | M )= Poisson ( d Y , d ′ Y , ⃗ Λ Y ) , with:

X X X Y Y Y

Furthermore, let A X , and A Y be the corresponding A -matrices (defined in (194)) associated with P ( ⃗ X | M ) and P ( ⃗ Y | M ) , respectively.

J.3 Formal proof of Theorem 5

Theorem 5. Let the joint distribution P ( M, ⃗ X , ⃗ Y ) of M , ⃗ X , and ⃗ Y describe the multivariate linear Poisson system. Without the loss of generality, assume d ′ X ≥ d ′ Y . If ∑ ( i 1 ,...,i j ) ∈ A d X j γ X i 1 ...i j ≥ ∑ ( i 1 ,...,i j ) ∈ A d Y j γ Y i 1 ...i j ∀ j ∈ [ d ′ Y ] , then ∆ P contains a Markov chain of the form M → ⃗ X → ⃗ Y and UI ( M ; ⃗ Y \ ⃗ X ) = 0 .

Proof. We first note that we can always assume d ′ X ≥ d ′ Y without the loss of generality because if d ′ X ≤ d ′ Y , then we can always switch our nomenclature to refer to ⃗ Y as ⃗ X , and ⃗ X as ⃗ Y .

We provide an explicit construction of the Markov chain M → ⃗ X → ⃗ X g → ⃗ Y g → ⃗ Y having the marginals P ( M, ⃗ X ) , and P ( M, ⃗ Y ) . Denote the joint density of M → ⃗ X → ⃗ X g → ⃗ Y g → ⃗ Y as Q ( M, ⃗ X , ⃗ X g , ⃗ Y g , ⃗ Y ) . For Q ( M ) and Q ( ⃗ X | M ) , we choose them to be equal to P ( M ) and P ( ⃗ X | M ) , i.e. Q ( M ) = P ( M ) , and Q ( ⃗ X | M ) = P ( ⃗ X | M ) . Note that due to this construction, P ( M, ⃗ X ) = Q ( M, ⃗ X ) holds trivially.

For Q ( ⃗ X g | ⃗ X ) , we use the result described in lemma 17. Note that the construction for Q ( ⃗ X g | ⃗ X ) is not explicit but rather implicit. Let d ⃗ Λ X be the dimension of ⃗ Λ X , then we explicitly choose: Q ( ⃗ X | ⃗ X g , M ) = δ K ( ⃗ X = A X ⃗ X g ) , and Q ( ⃗ Y g | M )= Poisson ( d ⃗ Λ X , 1 , ⃗ Λ Y ) . Note that Q ( ⃗ X | M )= Poisson ( d X , d ′ X , ⃗ Λ X ) by construction, and we derive Q ( ⃗ X g | ⃗ X , M ) through the Bayes theorem. Here, δ K ( · ) is the Kronecker delta function [59]. By lemma 17, we know that Q ( ⃗ X g | ⃗ X , M ) = Q ( ⃗ X g | ⃗ X ) , and hence we have the Markov chain M → ⃗ X → ⃗ X g .

For choosing Q ( ⃗ Y g | ⃗ X g ) , we rely on the result of lemma 18. Let us define the random vectors:

i.e., ⃗ X j and ⃗ Y j are random vectors containing all terms of the form X g i 1 ,...,i j , and Y g k 1 ,...,k j , where ( i 1 , . . . , i j ) ∈ A d 1 j and ( k 1 , . . . , k j ) ∈ A d 2 j , respectively. Note that we can write ⃗ X g = [ ⃗ X T 1 . . . ⃗ X T d ′ 1 ] T , and we define ⃗ Y g = [ ⃗ Y T 1 . . . ⃗ Y T d ′ 2 ] T . Then, we construct Q ( ⃗ Y g | ⃗ X g ) as a product of d ′ Y multinomial distributions, described below:

where Q ( ⃗ Y j = ⃗ y j | ⃗ X j = ⃗ x j ) = Multinomial ( k j ; N j , p i ) , and

By construction ⃗ X g consists of mutually conditionally independent Poisson random variables (conditioned on M ), and the condition of lemma 18 (specified in (219)) is satisfied by condition 1. Therefore, using the result of lemma 18 on the Markov chain M → ⃗ X g → ⃗ Y g (obtained by marginalizing ⃗ X ), we conclude that Q ( ⃗ Y g | M ) = Poisson (1 , d ⃗ Λ Y , ⃗ Λ Y ) , where d ⃗ Λ Y is the dimension of ⃗ Λ Y .

We specify Q ( ⃗ Y | ⃗ Y g ) through the following deterministic transformation ⃗ Y = A Y ⃗ Y g to obtain the Markov chain M → ⃗ X → ⃗ X g → ⃗ Y g → ⃗ Y . Marginalizing ⃗ X and ⃗ X g in the above Markov chain,

we get the following Markov chain: M → ⃗ Y g → ⃗ Y . Now, since ⃗ Y = A Y ⃗ Y g and P ( ⃗ Y g | M ) = Poisson ( d ⃗ Λ Y , 1 , ⃗ Λ Y ) , P ( ⃗ Y | M ) = Poisson ( d Y , d ′ Y , Λ Y ) (by definition of the multivariate Poisson discussed in Sec. J.1). Since we have Q ( M ) = P ( M ) and Q ( ⃗ Y | M ) = P ( ⃗ Y | M ) , we also have Q ( M, ⃗ Y ) = P ( M, ⃗ Y ) . Marginalizing ⃗ X g and ⃗ Y g from the Markov chain M → ⃗ X → ⃗ X g → ⃗ Y g → ⃗ Y results in the desired Markov chain M → ⃗ X → ⃗ Y having Q ( M, ⃗ X )= P ( M, ⃗ X ) and Q ( M, ⃗ Y )= P ( M, ⃗ Y ) , concluding our proof.

Lemma 15. If N j ∼ Poisson ( λ j ) ∀ j ∈ { 1 , . . . , d } , and all N j 's are mutually independent, then P ( N 1 , . . . , N j | ∑ d j =1 N j = N ) = Multinomial ( N 1 , . . . , N j ; N, [ λ 1 / ∑ d i =1 λ j . . . λ d / ∑ d i =1 λ j ]) .

Proof.

Lemma 16. Let ⃗ N 1 = [ N (1) 1 . . . N (1) d 1 ] T , where N j ∼ Poisson ( λ (1) j ) ∀ j ∈ { 1 , . . . , d 1 } and all N j 's are mutually independent from each other. Define P ( ⃗ N 2 | ⃗ N 1 ) = Multinomial ( ⃗ k ; N,⃗ p ) , where

where { λ (2) j } d 2 j =1 are some positive constants, i.e., λ (2) j ∈ (0 , ∞ ) , such that ∑ d 2 i =1 λ (2) j ≤ ∑ d 1 j =1 λ (1) j . Then, ⃗ N 2 = [ N (2) 1 . . . N (2) d 2 ] T , where N (2) j ∼ Poisson ( λ (2) j ) , and all N (2) j 's are mutually independent form each other.

Proof. Lemma 16 is a straight-forward generalization of lemma 15. Denote the probability of ⃗ N 2 shown in the lemma statement as P ( ⃗ N 2 ) . We will prove the above lemma by constructing a Markov chain ⃗ N 1 → N ′ 1 → N ′ 2 → ⃗ N 2 with joint density Q ( ⃗ N 1 , N ′ 1 , N ′ 2 , ⃗ N 2 ) , and showing that Q ( ⃗ N 2 ) = P ( ⃗ N 2 ) and Q ( ⃗ N 2 | ⃗ N 1 ) = P ( ⃗ N 2 | ⃗ N 1 ) .

Specifying Q ( N ′ 1 | ⃗ N 1 ) : We specify Q ( N ′ 1 | ⃗ N 1 ) through the following deterministic transformation:

Calculating Q ( N ′ 1 ) : We use the property that the sum of independent Poisson random variables is a Poisson random variable with its rate parameter being the sum of the rate parameters of its summands [61]. Hence, Q ( N ′ 1 ) = Poisson ( ∑ d 1 j =1 λ (1) j ) .

Specifying Q ( N ′ 2 | N ′ 1 ) : We specify Q ( N ′ 2 | N ′ 1 = n ′ 1 ) as follows:

Calculating Q ( N ′ 2 ) : We use lemma 15 to calculate Q ( N ′ 2 ) . Note that Q ( N ′ 1 ) = Poisson ( ∑ d 1 j =1 λ (1) j ) , and Q ( N ′ 2 | N ′ 1 ) = Binomial ( N ′ 2 ; n ′ 1 , ∑ d 2 j =1 λ (2) j / ∑ d 1 j =1 λ (1) j ) . Hence, by lemma 15, we have:

Specifying Q ( N ′ 2 | N ′ 1 ) : We specify Q ( ⃗ N 2 | N ′ 2 = n ′ 2 )

Calculating Q ( ⃗ N 2 ) : We again use lemma 15 to calculate Q ( ⃗ N 2 ) . Note that,

Hence, by lemma 15, we have:

⃗ N 2 = [ N (2) 1 . . . N (2) d 2 ] T , where N (2) j ∼ Poisson ( λ (2) j ) ∀ j ∈ [ d 2 ] . (202) Hence, by (202), we know that Q ( ⃗ N 2 ) = P ( ⃗ N 2 ) . Calculating Q ( ⃗ N 2 | ⃗ N 1 ) :

where ( a ) is due to the Markovian nature of Q ( ⃗ N 1 , N ′ 1 , N ′ 2 , ⃗ N 2 ) . We drop the random variable notation for brevity. Let ⃗ n 1 = [ n (1) 1 . . . n (1) d 1 ] T , and ⃗ n 2 = [ n (2) 1 . . . n (2) d 2 ] T . Then, substituting the exact form of Q ( n ′ 1 | ⃗ n 1 ) , Q ( n ′ 2 | n ′ 1 ) , and Q ( ⃗ n 2 | n ′ 2 ) in the above equation:

where I [ · ] is an indicator function. Simplifying the above equation by using the fact that only non-zero term in the above summation is n ′ 1 = ∑ d 1 j =1 n (1) j and n ′ 2 = ∑ d 1 j =1 n (2) j .

Collecting all factorial terms together, and rearranging some terms:

Substituting ( ∑ d 2 j =1 λ (2) j / ∑ d 1 j =1 λ (1) j ) ∑ d 2 j =1 n (2) j = ∏ d 2 j =1 ( ∑ d 2 j =1 λ (2) j / ∑ d 1 j =1 λ (1) j ) n (2) j in the above equation:

By inspecting (203), we can see that Q ( ⃗ N 2 | ⃗ N 1 ) = Multinomial ( ⃗ k ; N,⃗ p ) with,

Hence, by (205), we can conclude that Q ( ⃗ N 2 | ⃗ N 1 ) = P ( ⃗ N 2 | ⃗ N 1 ) finishing our proof.

Lemma 17. Let ⃗ N be a d -dimensional vector and M be a positive random variable, i.e. M ∼ P ( M ) , P ( M ≤ 0) = 0 . Let P ( ⃗ N | M ) = Poisson ( d, d ′ , ⃗ Λ ) , where:

Let ⃗ N g = [ N g 1 . . . N g d N g 12 . . . N g ( d -1) d . . . N g d -( d ′ -1) ...d ] T , where P ( ⃗ N g | M ) = Poisson ( d ⃗ Λ , 1 , ⃗ Λ ) and ⃗ N = A ⃗ N g , where d Λ is the dimension of ⃗ Λ , and A = [ A 1 . . . A d ′ ] as defined in Sec. J.1. Then P ( ⃗ N g | ⃗ N , M ) = P ( ⃗ N g | ⃗ N ) , i.e. M, ⃗ N g and ⃗ N form the following Markov chain M → ⃗ N → ⃗ N g .

Proof. To prove lemma 17, all we need to show is that P ( ⃗ N g | ⃗ N , M ) does not depend on M . Let us to calculate P ( ⃗ N g | ⃗ N , M ) . By Bayes Theorem, we know:

Let us now write the expression for P ( ⃗ N | M ) using (196):

Canceling M in the above equation for the terms inside the summation, we get:

Absorbing all terms that do not depend upon M into B , we have

Then, we can rewrite (209) as:

where, in (a) we further absorb ∏ d i =1 ( γ i ) n i into B since it does not depend upon M , and in (b) we substitute ∑ d i =1 n i = ⃗ 1 T ⃗ n . Similarly, let us write out the expression for P ( ⃗ N g | M ) :

Collecting all the M terms, and absorbing all the terms that do not depend upon M into D , we obtain:

Now let us analyze the term ⃗ 1 T A⃗ n g :

where (a) uses the fact that ⃗ 1 T A ⃗ n g is a scalar and hence is equal to its transpose, (b) uses the fact A =[ A 1 . . . A d ′ ] , and (c) follows from the special structure of A i , i.e., that each column only contains i ones and d -i zeros, and the fact that A T 1 ⃗ 1 is akin to summing up the columns, hence A T i ⃗ 1 = i ⃗ 1 . Equivalently we can rewrite the above equation as:

Now, let us write out the expression for P ( ⃗ N | ⃗ N g , M ) . Since ⃗ N = A ⃗ N g , P ( ⃗ N | ⃗ N g ) can be represented as a Kronecker delta function with the condition ⃗ N = A ⃗ N g , i.e.,

where δ K ( · ) is the Kronecker delta function. Substituting (211), (215) and (216) in (207), we get:

Substituting (214) into (213):

where we obtain (a) by canceling the term e -⃗ 1 T ⃗ Λ , (b) by using the fact ⃗ n = A ⃗ n g due to the delta function, and (c) by canceling the term M ⃗ 1 T A ⃗ n g . Since the terms D,B and δ K ( ⃗ n = A ⃗ n g ) do not depend upon M , we can conclude P ( ⃗ N g | ⃗ N , M ) also does not depend upon M , i.e. P ( ⃗ N g | ⃗ N , M ) = P ( ⃗ N g | ⃗ N ) , or M → ⃗ N → ⃗ N g .

Lemma 18. Let M ∼ P ( M ) have support over (0 , ∞ ) . Define, ⃗ X = [ ⃗ X T 1 . . . ⃗ X T d 1 ] T and ⃗ Y = [ ⃗ Y 1 . . . ⃗ Y T d 2 ] T , where ⃗ X i and ⃗ Y j are random vectors of size q i ∈ N and r j ∈ N , respectively, ∀ i ∈ [ d 1 ] , and j ∈ [ d 2 ] . Let the conditional distribution of ⃗ X and ⃗ Y conditioned on M be specified as follows:

If d 1 ≥ d 2 and (219) hold

then the distribution ˜ Q ( M, ⃗ X , ⃗ Y ) defined in (220) lies in ∆ P .

where ˜ Q ( ⃗ Y | ⃗ X ) is a product of d 2 multinomial distributions, i.e ˜ Q ( ⃗ Y | ⃗ X ) = ∏ d 2 i =1 ˜ Q ( ⃗ Y i | ⃗ X i ) , where ˜ Q ( ⃗ Y i | ⃗ X i ) = Multinomial ( ⃗ k i ; N i , ⃗ p i ) and

Proof. Lemma 18 follows from a straightforward application of lemma 16. For showing ˜ Q ( M, ⃗ X , ⃗ Y ) specified in (220) lies in ∆ P , we need to show that ˜ Q ( M, ⃗ X ) = P ( M, ⃗ X ) and ˜ Q ( M, ⃗ Y ) = P ( M, ⃗ Y ) . ˜ Q ( M, ⃗ X ) = P ( M, ⃗ X ) is satisfied trivially by construction of ˜ Q ( M, ⃗ X , ⃗ Y ) .

For showing the second condition, we will explicitly show ˜ Q ( ⃗ Y | M ) = P ( ⃗ Y | M ) which, combined with the fact that ˜ Q ( M ) = P ( M ) (by construction), will show ˜ Q ( ⃗ Y , M ) = P ( ⃗ Y , M ) completing our proof.

Calculating ˜ Q ( ⃗ Y | ⃗ X ) : From the law of total probability, we have:

where ( a ) is due to the particular structure of P ( ⃗ X | M ) and ˜ Q ( ⃗ Y | ⃗ X ) specified in (218) and (220), respectively, ( b ) is just collecting the terms of P ( ⃗ X i | M ) and ˜ Q ( ⃗ Y i | ⃗ X i ) into a single product, ( c ) is due to spreading the summation of the components of ˜ x over their corresponding distribution components, and ( d ) is due to the fact that sum over all probabilities of all possible ˜ x i is 1 .

Note that for a fixed m , P ( ⃗ X i | M = m ) ∼ Poisson ( n i , 1 , [ γ i 1 m i . . . γ in i m i ] T ) , and Q ( ⃗ Y i | ⃗ X i ) = Multinomial ( ⃗ k i ; N i , ⃗ p i ) , where

Note that we can equivalently write ⃗ p i as follows,

In the setup of lemma 16, choose d 1 = q i and d 2 = r i , λ (2) j = γ Y ij M i ∀ j ∈ [ r i ] and λ (1) j = γ X ij M i ∀ j ∈ [ q i ] . From (219), we know the condition ∑ d 2 j =1 λ (2) j ≤ ∑ d 2 j =1 λ (2) j is satisfied by assumption. Hence, we use the result of lemma 15 to conclude ˜ Q ( ⃗ Y i | M ) = Poisson ( r i , 1 , ⃗ Λ ′ ) , with

Comparing ˜ Q ( ⃗ Y i | M ) specified in (223) with P ( ⃗ Y i | M ) specified in (218), we can conclude Q ( ⃗ Y i | M ) = P ( ⃗ Y i | M ) . Substituting Q ( ⃗ Y i | M ) = P ( ⃗ Y i | M ) in (221), we obtain:

Comparing (224) with (218), we can conclude ˜ Q ( ⃗ Y | M ) = P ( ⃗ Y | M ) , finishing our proof.

K Proof of Theorem 6

In this section, we provide the proof of theorem 6. Since theorem 6 provides the analytical PID terms for the linear convolution-closed system defined in Sec. 5.1, we briefly restate certain key properties of convolution-closed distributions and the linear convolution-closed system for convenience.

K.1 Convolution-closed distributions

Convolution-closed distributions are a large class of distributions that are defined as follows:

Definition 1. Let F D denote a family of distributions, where each member distribution, f ( δ ) ∈ F D , is indexed by a parameter δ . Consider X 1 ∼ f ( δ 1 ) , X 2 ∼ f ( δ 2 ) , and X 1 ⊥ ⊥ X 2 for some δ 1 , δ 2 ∈ D . Then, F D is convolution-closed in the parameter δ , if

where ∗ denotes the convolution operator.

Convolution-closed distributions define a natural dilation/thinning operator. Formally, let X ∼ f ( δ ) . Then, we define X ϵ as the ϵ -dilated version of X if X ∼ f ( ϵδ ) for some ϵ ∈ (0 , 1) such that ϵδ ∈ D . Furthermore, if we assume (1 -ϵ ) δ ∈ D , then P ( X ϵ | X ) can be defined as P ( X ϵ | X ϵ + X (1 -ϵ ) ) , where X 1 -ϵ ∼ f ((1 -ϵ ) δ ) and X ϵ ⊥ ⊥ X 1 -ϵ . Denote, P ( X ϵ | X = x ) = P ( X ϵ | X ϵ + X (1 -ϵ ) = x ) = G ( ϵδ, (1 -ϵ ) δ, x ) .

K.2 Definition of linear convolution-closed system

Let F D be a convolution-closed distribution family in parameter δ ∈ D , and M be a target/message random variable having distribution P ( M ) over some support set M . Define the conditional distribution of random variables X and Y conditioned on M as follows:

Furthermore, we assume that there exists γ ∈ (0 , ∞ ) such that: δ X m = γδ Y m ∀ m ∈ M . We denote such a system of random variables ( M,X,Y ) , having the joint distribution P ( M,X,Y ) , a linear convolution-closed system .

K.3 Formal proof of theorem 6

Theorem 6. Let the joint density P ( M,X,Y ) of random variables M , X , and Y describe a linear convolution-closed system. Without the loss of generality, assume γ ≤ 1 . If

then ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

Proof. We first note that we can always assume γ ≤ 1 without the loss of generality because if γ ≥ 1 , then we can always switch our nomenclature to refer to Y as X , and X as Y .

We briefly outline the proof structure.

In the first part, we explicitly construct a joint distribution Q MC ( M,X,Y ) having the Markovian structure M → X → Y .

In the second part, we show that the Q MC ( M,X,Y ) constructed in the first part, lies in ∆ P . Therefore, we can then apply the result of proposition 1 to conclude UI ( M ; Y \ X ) = 0 .

Part 1: Specifying Q MC ( M,X,Y )

Denote the joint distribution of the Markov chain M → X → Y as Q MC ( M,X,Y ) . We can decompose Q MC ( M,X,Y ) by utilizing its Markovian structure as follows:

Consequently, we can construct the distribution Q MC ( M,X,Y ) by individually specifying Q MC ( M ) , Q MC ( X | M ) , and Q MC ( Y | X ) .

Specifying Q MC ( M ) and Q MC ( X | M ) : We specify Q MC ( M ) and Q MC ( X | M ) as follows:

Specifying Q MC ( Y | X ) : By assumption (b) in the theorem, we know that G ( γδ X m , (1 -γ ) δ X m , x ) does not depend on m , and consequently, does not depend on δ X m . Hence, we can simplify notation of G ( γδ X m , (1 -γ ) δ X m , m ) as G ( γ, (1 -γ ) , x ) . We specify Q ( Y | X ) as follows:

Part 2: Showing Q MC ( M,X,Y ) ∈ ∆ P

For showing that Q MC ( M,X,Y ) ∈ ∆ P , we first need to derive Q MC ( Y | M ) .

Deriving Q MC ( Y | M ) : We use the result of lemma 19 to derive Q MC ( Y | M ) . As Q MC ( X | M = m ) = f ( δ X m ) is a convolution-closed distribution for a fixed m , and Q MC ( Y = y | X = x ) = G ( γ, 1 -γ, x ) , the result of lemma 19 shows that Q MC ( Y | M ) can be expressed as:

where ( a ) is due to the assumption δ X m = γδ Y m (see Appx. K.2).

From (227) and (229), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; Y \ X ) = 0 , concluding our proof for case 1.

Lemma 19. Let F D be a convolution-closed distribution family convolution-closed in the parameter δ . For a δ ∈ D , assume there exists ϵ ∈ (0 , 1) such that ϵδ, (1 -ϵ ) δ ∈ D . Consider the following random variables:

such that X ′ ϵ ⊥ ⊥ X ′ 1 -ϵ . If P ( X ϵ = x ϵ | X = x ) = P ( X ′ ϵ = x ϵ | X ′ ϵ + X ′ 1 -ϵ = x ) , then we have P ( X ϵ = x ϵ ) = P ( X ′ ϵ = x ϵ ) = f ( x ϵ ; ϵδ ) or X ϵ d = X ′ ϵ , where d = implies equality in distribution [57].

Proof. First let us derive the distribution of X ′ = X ′ 1 -ϵ + X ′ ϵ . Since, X ′ ϵ ⊥ ⊥ X ′ 1 -ϵ , we can write P ( X ′ ) as follows:

where ∗ is the convolution operator, ( a ) is due to the properties of convolution-closed distribution (see Appx. K.1), and ( b ) is due to the definition of X in the lemma statement. From (232), we can conclude P ( X = x ) = P ( X ′ = x ) , and by assumption we have P ( X ϵ | X ) = P ( X ′ ϵ | X ′ ) = P ( X ′ ϵ | X ′ ϵ + X ′ (1 -ϵ ) ) . Hence, we have:

From (233), we can conclude that P ( X ϵ ) = P ( X ′ ϵ ) = f ( ϵδ ) .

L Proof of Theorem 7

In this section, we provide the proof of theorem 7.

L.1 Definition of p exp 1 and p exp 2

We restate the definitions of p exp 1 and p exp 2 defined in Sec. 5.2, which is used in the proof of theorem 7. Let X ∼ p exp 1 ( X ) , where

for some appropriately defined H ( · , · ) , A ( · ) , θ 1 and θ 2 . Furthermore, define a random variable Y through its conditional density p ( Y | X = x ) :

for some h ( · ) , T ( · ) , and θ 3 , such that p ( Y = y | X = x ; θ 1 , θ 2 ) is a well-defined distribution. Furthermore, the marginal distribution of Y is expressed as:

L.2 Formal proof of theorem 7

Theorem 7. Let M,X, and Y be random variables having the joint distribution P ( M,X,Y ) . Furthermore, the conditional distribution of X and Y conditioned on M are as follows: P ( X | M = m )= p exp 1 ( X ; θ 1 ( m ) , θ 2 ( m )) and P ( Y | M = m )= p exp 2 ( Y ; θ 1 ( m ) , θ 2 ( m ) , θ 3 ) . Then, ∆ P contains a Markov chain of the form M → X → Y and UI ( M ; Y \ X ) = 0 .

Proof. We prove theorem 7 by explicitly constructing a Markov chain M → X → Y , having joint distribution Q MC ( M,X,Y ) , and showing that Q MC ( M,X,Y ) ∈ ∆ P .

Specifying Q MC ( M ) and Q MC ( X | M ) : We specify Q MC ( M ) and Q MC ( X | M ) as follows:

Specifying Q MC ( Y | X ) : We specify Q MC ( Y | X ) = h ( y ) exp( x T T ( y ) -θ T 3 A ( x )) . Then, by (236), we know that

From (237) and (238), we can conclude:

Hence, Q MC ( M,X,Y ) ∈ ∆ P and consequently, by proposition 1, UI ( M ; Y \ X ) = 0 , concluding our proof.

M Additional proofs and details for Sec. 6

In this section, we provide the proofs of certain statements made in Sec. 6, as well as additional details regarding the simulation study presented in the same section.

M.1 Generating ¯ Q for every Q ∈ ∆ P

Our goal in this section is to show that there exists a ¯ Q (as described in Sec. 6) for every Q ∈ ∆ P , where the corresponding P ( M,X,Y ) is as defined in Sec. 6. We first briefly restate the assumptions on P ( M,X,Y ) described in Sec. 6. The distribution P ( M,X,Y ) has the following properties: the random variable M has support over M , the conditional distributions P ( X | M ) and P ( Y | M ) are

members of some convolution-closed distribution family F D , and there exists some δ X bias , δ Y bias ∈ D such that:

As Q ( M,X )= P ( M,X ) and Q ( M,Y ) = P ( M,Y ) due to Q ∈ ∆ P , we have:

M.1.1 Explicitly constructing a ¯ Q for every Q ∈ ∆ P

We show existence of a ¯ Q for every Q ∈ ∆ P by explicitly constructing a ¯ Q for each Q ∈ ∆ P . We define two vectors to simplify our notations:

where X and Y are the random variables defined above. ( X ′ , Y ′′ , n X ) , and ( Y ′ , n Y ) are the corresponding dilated versions of X and Y as defined in Sec. 6. Note that the joint distribution of ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) is ¯ Q ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) . We reiterate the two main properties of ¯ Q ( X ′ , Y ′′ , n X , Y ′ , n Y ) , mentioned in Sec. 6, that we want to prove:

The marginal conditional distributions of ( X ′ , Y ′′ , n X , Y ′ , n Y ) given M are as follows:

The random variables ( X ′ , Y ′′ , n X ) are jointly conditionally independent given M . Similarly, the random variables ( Y ′ , n Y ) are also conditionally independent given M .

We construct ⃗ B from ⃗ A by specifying the conditional distribution ¯ Q ( ⃗ B | ⃗ A , M ) . Note that given the values of ⃗ A and M , we can always use ¯ Q ( ⃗ B | ⃗ A , M ) to construct the random variables ⃗ B . The goal of this remaining section is to show that the corresponding distribution ¯ Q ( ⃗ B , M ) (specified through ¯ Q ( ⃗ B | ⃗ A , M ) ) satisfies the above two properties. We first explicitly state the structure of ¯ Q ( ⃗ B | ⃗ A , M ) .

Specifying ¯ Q ( ⃗ B | ⃗ A , M ) :

We first impose the following structure on ¯ Q ( ⃗ B | ⃗ A , M ) as follows:

where ¯ Q ( X ′ , Y ′′ , n X | X,M ) and ¯ Q ( Y ′ , n Y | Y, M ) are the conditional distributions through which we specify ¯ Q ( ⃗ B | ⃗ A , M ) . Since ¯ Q ( ⃗ B | ⃗ A , M ) is a product of two well-defined distributions, it is a valid distribution. Let us now analyze the joint distribution ¯ Q ( M, ⃗ A , ⃗ B ) . By chain rule, we have:

Substituting the fact that ( M, ⃗ A ) = ( M,X,Y ) and using the structure of ¯ Q ( ⃗ B | ⃗ A , M ) specified in (244), we have:

We now specify the two components of ¯ Q ( ⃗ B | ⃗ A , M ) .

Specifying ¯ Q ( Y ′ , n Y | Y, M ) :

Using the chain rule on ¯ Q ( Y ′ , n | Y, M ) , we obtain

Choose:

Here, δ K ( · ) denotes the Kronecker delta function [59]. Marginalizing ( X,X ′ , Y ′′ , n X ) from (246), we obtain:

Consider the distribution ¯ Q ( Y, Y ′ , n Y | M ) . Then, we know:

Comparing the above equation with the structure of ¯ Q ( Y, Y ′ , n Y | M ) specified in proposition 8, we can conclude that ( Y ′ , n Y ) are just dilated versions of Y . Consequently, using the result of proposition 8, we have:

Specifying ¯ Q ( X ′ , Y ′′ , n X | X,M ) :

We similarly decompose ¯ Q ( X ′ , Y ′′ , n X | X,M ) using the chain rule as follows:

We choose:

Here δ K ( · ) denotes the Kronecker delta function [59]. Marginalizing ( Y, Y ′ , n Y ) from (246), we obtain:

Consider the distribution ¯ Q ( X,X ′ , n X , Y ′′ | M ) . Then, we know:

Comparing the above equation with the structure of ¯ Q ( X,X ′ , n X , Y ′′ | M ) specified in proposition 9, we can conclude that ( X ′ , Y ′′ , n X ) are just dilated versions of X . Consequently, using the result of proposition 9, we have:

This concludes our construction of ¯ Q ( ⃗ B | ⃗ A , M ) . We obtain our desired ¯ Q ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) by marginalizing ⃗ A out of ¯ Q ( M, ⃗ A , ⃗ B ) . From (251) and (255) we know that ¯ Q ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) satisfies the two main properties specified earlier. Since we did not make any specific assumptions on Q aside from the fact that it lies in ∆ P , the above construction can always be used to construct ( X ′ , n X , Y ′′ , Y ′ , n Y ) from ( M,X,Y ) using the ¯ Q ( ⃗ B | ⃗ A , M ) . Note that due to the particular construction of ¯ Q ( ⃗ B | ⃗ A , M ) , we have:

M.1.2 Propositions required for constructing ¯ Q

We describe two important propositions that we use to show the existence of a ¯ Q for every Q ∈ ∆ P . Proposition 8. Suppose the random variables M and Y have the joint distribution Q ( M,Y ) such that

where M denotes the support of M and f ( δ Y m ) ∈ F D for some convolution-closed distribution family F D . Furthermore, there also exists a δ Y bias ∈ D such that

Consider two dilated versions of Y , denoted as Y ′ and n Y , defined as follows:

Then, we have

Y ′ ⊥ ⊥ n Y | M

The conditional distributions ¯ Q ( Y ′ | M ) and ¯ Q ( n Y | M ) are specified as follows:

Proof. We essentially modify the argument given in theorem 1 of [25] to prove the above proposition. Consider the conditional distribution of Y , Y ′ , and n Y conditioned on M , denoted as ¯ Q ( Y, Y ′ , n Y | M ) . Then, by chain rule, we have ¯ Q ( Y, Y ′ , n Y | M ) = ¯ Q ( Y | M ) ¯ Q ( Y ′ | M,Y ) ¯ Q ( n Y | Y, Y ′ , M ) . By the assumptions in (257) and (259), we know that

where δ K ( · ) is the Kronecker delta function [59].

Observation 1 : Our first key observation is that the distribution ¯ Q ( Y ′ , Y, n Y | M = m ) is completely specified through the distributions ¯ Q ( Y | M ) and ¯ Q ( Y ′ | Y, M ) , as n Y is completely determined by Y ′ and Y and ¯ Q ( n Y | Y, Y ′ , M ) is specified through a Kronecker delta function.

Similarly, consider a new pair of random variables Y ′ new , n new Y such that Y ′ new ⊥ ⊥ n new Y | M and the respective conditional distributions of Y ′ new and n new Y are specified as follows:

We will now show two key properties for the random variables ( Y ′ new , n new Y , Y ′ new + n new Y ) that will form the crux of our argument:

Let us now calculate the conditional distribution ¯ Q ( Y ′ new + n new Y = y | M ) :

Using conditional dependence of Y ′ new and n new Y , we have ¯ Q new ( Y ′ new = k, n new Y = y -k | M ) = ¯ Q new ( Y ′ new = k | M ) ¯ Q new ( n new Y = y -k | M ) :

Note that the R.H.S of the above equation defines a convolution of ¯ Q new ( Y ′ new | M ) and ¯ Q ( n new Y | M ) , hence:

where ∗ denotes the convolution operator. Substituting (262) in the above equation:

Using the properties of convolution-closed distributions:

Hence, by comparing above equation with (261) we have:

By definition of G ( ϵ (2) m δ Y m , (1 -ϵ (2) m ) δ Y m , z ) given in Sec. 5.1, we have:

In the following steps, we slightly modify the notation of G ( ϵ (2) m δ Y m , (1 -ϵ (2) m ) δ Y m , z ) to G ( z ϵ (2) m ; ϵ (2) m δ Y m , (1 -ϵ (2) m ) δ Y m , z ) to make our notations more explicit as follows:

We use the definition of the conditional distribution to express G ( ϵ (2) m δ Y m , (1 -ϵ (2) m ) δ Y m , z ) as:

Using the fact that Pr( Z ϵ (2) m + Z 1 -ϵ (2) m = z | Z ϵ (2) m = z ϵ (2) m ) = Pr( Z 1 -ϵ (2) m = z -z ϵ (2) m ) , since Z ϵ (2) m ⊥ ⊥ Z 1 -ϵ (2) m :

Note that by properties of convolution-closed distributions, Pr( Z ϵ (2) m + Z 1 -ϵ (2) m = z )) = f ( z ; δ Y m ) , where f ( z ; δ Y m ) is to be understood as the distribution f ( δ Y m ) evaluated at z . Substituting (269) in the above equation:

Now, note that by (262) and (267), we have:

Hence, combining the above equation with (274), and renaming z ϵ (2) m as y ′ and z as y , we have:

Noting the fact that ¯ Q new ( n new Y = y -y ′ | M = m ) = ¯ Q new ( Y ′ new + n new Y = y | M = m,Y ′ new = y ′ ) due to Y ′ new ⊥ ⊥ n new Y | M and abbreviating notation pertaining to y ′ , we have:

Using Bayes rule in the above equation provides us with our desired property:

Furthermore, by comparing (259) with the above equation, we have:

Observation 2 : Note that the distribution ¯ Q ( Y ′ new , n new Y , Y ′ new + n new Y | M ) is also completely determined by specifying ¯ Q ( Y ′ new + n new Y | M ) and ¯ Q ( Y ′ new | Y ′ new + n new Y , M ) since n new Y is completely determined by Y ′ new + n new Y and Y ′ new .

By property 1 and (279), we know that

Therefore, combining the above equation with observations 1 and 2,we have:

Consequently the distribution ¯ Q ( Y ′ , n Y | M ) = ¯ Q new ( Y ′ new , n new Y | M ) . Since, Y ′ new ⊥ ⊥ n new Y | M , we also have Y ′ ⊥ ⊥ n Y | M . Furthermore, by (262),

By (258), we know that ϵ (2) m δ Y m = δ Y m -δ Y bias and (1 -ϵ (2) m ) δ Y m = δ Y bias . Hence,

Proposition 9. Suppose the random variables M and X have the joint distribution Q ( M,X ) , such that

where M denotes the support of M and f ( δ X m ) ∈ F D for some convolution-closed distribution family F D . Furthermore, there also exist δ X bias ∈ D , and δ Y m ∈ D ∀ m ∈ M such that

Then consider three dilated versions of X , denoted as X ′ , Y ′′ , and n X , defined as follows:

Then, we have

( n X , X ′ , Y ′′ ) are jointly conditionally independent conditioned on M .

The conditional distributions ¯ Q ( X ′ | M ) , ¯ Q ( Y ′′ | M ) , and ¯ Q ( n X | M ) are specified as follows:

Proof. The proof essentially consists of recursively applying the proof of proposition 8. Consider a new auxiliary random variable Z = X -n X . Then, the distribution ¯ Q ( X,n X , Z | M ) has the following properties:

where δ K ( · ) is the Kronecker delta function, and the equalities ( a ) and ( b ) follow from (284) and (285), respectively. Note that the distribution ¯ Q ( X,n X , Z | M ) has exactly the same structure as ¯ Q ( Y, Y ′ , n Y | M ) described in (261), hence Z and n Y are just dilated versions of X . Consequently, from the result of proposition 8, we know that Z ⊥ ⊥ n X | M , and the conditional distributions ¯ Q ( Z | M ) and ¯ Q ( n Y | M ) are described as follows:

From (285), we have ϵ (3) m δ X m = δ X m -δ X bias and (1 -ϵ (3) m ) δ X m = δ X bias . Consequently,

We alternatively express ˜ Q ( X ′ | n X , X, M ) as follows:

Note that the distribution ¯ Q ( X ′ | n X = x n X , X = x, M = m ) only depends on X and n X through their difference X -n X . Hence, substituting Z = X -n X in the above equation:

Hence, consider the following distribution ¯ Q ( Z, X ′ , Y ′′ | M ) , where from (290) and (293), we have:

Furthermore, by (286), we have Y ′′ = X -n X -X ′ ( c ) = Z -X ′ , where ( c ) is due to Z = X -n X . Hence,

Comparing (295) and (294) with (261), we can again conclude that X ′′ and Y ′ are just dilated versions of Z . Hence, again using the result of proposition 8, we have X ′ ⊥ ⊥ Y ′′ | M , and

From (285), we have ϵ (1) m ( δ X m -δ X bias ) = δ Y m -δ Y bias and (1 -ϵ (1) m )( δ X m -δ Y bias ) = δ X m -δ Y M -( δ X bias -δ Y bias ) . Consequently,

Hence, by (290) and (297), we can conclude that the conditional distributions ¯ Q ( X ′ | M ) , ¯ Q ( Y ′′ | M ) , and ¯ Q ( n X | M ) are specified as follows:

Showing joint conditional independence of ( n Y , X ′ , Y ′′ ) given M :

Consider the distribution ¯ Q ( n X , Z, X ′ , Y ′′ | M ) . Using the chain rule, we have:

where ( b ) follows from n X ⊥ ⊥ Z | M . For showing ( c ) , observe that Y ′′ = Z -X ′ , and hence we can write ¯ Q ( Y ′′ | M,Z,n Y , X ′ ) = ¯ Q ( Y ′′ | Z, X ′ ) . Furthermore, by analyzing ¯ Q ( X ′ | n X , Z, M ) , we can conclude:

Marginalizing Z out of (299), we obtain:

where ( d ) follows from X ′ ⊥ ⊥ Y ′′ | M . By (301), we know that ( n X , X ′′ , Y ′ ) are jointly conditionally independent given M , concluding our proof.

M.2 Analytically optimizing the upper bound

Proposition 10. Let ¯ Q ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) be the distribution derived from a distribution Q ∈ ∆ P using the construction scheme specified in Appx. M.1. Then, the minimization problem specified in (302) is analytically solvable with the optimizing ¯ Q ∗ given by (303) .

where I [ · ] is the indicator function. Note that for the minimizing ¯ Q ∗ , we have ( n X , n Y , ( M,X ′ , Y ′′ )) are jointly independent and Y ′ = Y ′′ .

Proof. First note that by the construction scheme defined in Appx. M.1, we know that there is a unique ¯ Q for every Q ∈ ∆ P . Hence, we can interpret ¯ Q as a function of Q , and consequently the minimization problem in (302) is well-defined. Using the chain rule of mutual information, we derive the following lower bound for I ¯ Q ( M ; [ X ′ , Y ′′ , n X , Y ′ , n Y ]) :

where ( a ) is due to the non-negativity of conditional mutual information. Furthermore, note that by properties of ¯ Q derived in Appx. M.1, we know that X ′ ⊥ ⊥ Y ′′ | M ∀ Q ∈ ∆ P . Consequently, the term I ¯ Q ( M ; [ X ′ , Y ′′ ]) does note vary as we choose different Q from ∆ P as the corresponding ¯ Q ( M,X ′ , Y ′′ ) = ¯ Q ( M ) ¯ Q ( X ′ | M ) ¯ Q ( Y ′′ | M ) and the terms:

do not vary for all Q ∈ ∆ P (see Appx. M.1). Therefore, we have:

where we forgo the subscript ¯ Q for I ¯ Q ( M ; [ X ′′ , Y ′ ]) to indicate that the term I ¯ Q ( M ; [ X ′′ , Y ′ ]) does not vary over ∆ P . Hence, if we show that the distribution ¯ Q ∗ ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) specified in (303) achieves the lower bound in (307), then ¯ Q ∗ ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) should minimize the value of I ¯ Q ( M ; [ X ′ , Y ′′ , n X , Y ′ , n Y ]) over the set ∆ P . We now calculate I ¯ Q ∗ ( M ; [ X ′ , Y ′′ , n X , Y ′ , n Y ]) . By the chain rule of mutual information:

Using the fact that Y ′ = Y ′′ , we know that I ¯ Q ∗ ( M ; Y ′ | n X , n Y , X, Y ′′ ) is zero. A simple justification follows from the fact I ¯ Q ∗ ( M ; Y ′ | n X , n Y , X, Y ′′ ) = H ( Y ′ | n X , n Y , X, Y ′′ ) -H ( Y ′ | n X , n Y , X, Y ′′ , M ) , where H ( . | . ) is the conditional entropy function [28]. Observe that both conditional entropy terms are zero as Y ′ = Y ′′ , and condition entropy of a random variable conditioned on itself is zero. Therefore, we can simplify the above equation as:

Applying the chain rule of mutual information again, we obtain:

Using the fact that ( n X , n Y ) ⊥ ⊥ ( M,X ′ , Y ′′ ) , we know that I ( M ; [ n X , n Y ]) = 0 as mutual information between independent random variables is zero [28]. Consequently,

where ( b ) again follows from the fact that ( n X , n Y ) ⊥ ⊥ ( M,X ′ , Y ′′ ) , and, consequently, conditioning ( M,X ′ , Y ′′ ) on ( n X , n Y ) has no effect. From (310), we see the proposed ¯ Q ∗ in (303) does indeed achieve the lower bound specified in (307), and consequently minimizes (302).

The final step remaining in our proof is to ensure that the proposed ¯ Q ∗ corresponds to a valid Q ∗ UB ( M,X,Y ) lying in ∆ P , i.e., there is some Q ∗ UB ( M,X,Y ) ∈ ∆ P that generates ¯ Q ∗ using the construction scheme outlined in Appx. M.1. We know that the relationship between the random variables for ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) having the joint distribution ¯ Q and ( M,X,Y ) having the distribution Q can be expressed as follows: X = X ′ + Y ′′ + n X and Y = Y ′ + n Y . Hence, we use these relationships for calculating the conditional marginals Q ∗ UB ( X | M ) and Q ∗ UB ( Y | M ) of the corresponding distribution Q ∗ UB ( M,X,Y ) for the proposed distribution ¯ Q ∗ ( M,X ′ , Y ′′ , n X , Y ′ , n Y ) :

where ∗ denotes the convolution operator, and ( c ) follows from the fact that ( X ′ , Y ′′ , n X ) are jointly conditionally independent given M under ¯ Q ∗ , and ( d ) follows from the properties of convolutionclosed distributions. Similarly, calculating Q ∗ UB ( Y | M = m ) :

where ( e ) follows from the fact that Y ′ ⊥ ⊥ n Y | M under ¯ Q ∗ . It is easy to see that Q ∗ UB ( M ) = ¯ Q ( M ) = P ( M ) . Hence, we have Q ∗ UB ( M,X ) = P ( M ) P ( X | M ) = P ( M,X ) and Q ∗ UB ( M,Y ) = P ( M ) P ( X = Y | M ) = P ( M,Y ) , which implies Q ∗ UB ( M,X,Y ) ∈ ∆ P , concluding our proof.

M.3 Additional details regarding simulation

The list of different function pairs used in the simulation study are listed in Table 1. The function pairs were chosen with the only requirement being that they satisfied the assumptions in (9). For constructing the analytical estimate ˆ Q A ( M,X,Y ) , the corresponding δ X bias and δ Y bias were chosen as follows:

For each distribution and function pair, we first determine the range of possible values for δ X bias and δ Y bias .

For δ Y bias , we choose the range as [0 , min m ∈M f 2 ( m )) , where M is the support set of M for that distribution. Note that we cannot choose δ Y bias > min m ∈M f 2 ( m ) as the corresponding conditional distribution ¯ Q ∗ ( Y ′ | M ) = Poisson ( f 2 ( M ) -δ Y bias ) (required for the optimal ¯ Q ∗ from which Q A is constructed, see Appx. M.2) would have a negative rate parameter, which is not allowed. Within this range, we choose 10 different, uniformly spaced values for δ Y bias .

For each chosen value of δ Y bias , the corresponding range for δ X bias is then chosen as [0 , min { min m ∈M f 1 ( m ) , min m ∈M ( f 1 ( m ) -f 2 ( m ) + δ Y bias ) } ] . Note that choosing the upper limit of δ X bias as min { min m ∈M f 1 ( m ) , min m ∈M ( f 1 ( m ) -f 2 ( m ) + δ Y bias ) } ensures that the distribution ¯ Q ∗ ( X ′ | M ) = Poisson (( f 1 ( M ) -δ X bias ) -( f 2 ( M ) -δ Y bias )) and ¯ Q ∗ ( Z | M ) = Poisson ( f 1 ( M ) -δ X bias ) (required for specifying the optimal ¯ Q ∗ from which Q A is constructed, see Appx. M.1 and Appx. M.2) have positive rate parameters. For each value of δ Y bias , we again choose 10 different values of δ X bias within the above specified range. In total, we chose 100 different pairs of ( δ X bias , δ Y bias ) .

For each pair of ( δ X bias , δ Y bias ) , we calculate the analytical Q A ( M,X,Y ) as follows:

First, we construct the optimal ¯ Q ∗ distribution using the structure provided in proposition 10. The exact specification of ¯ Q ∗ is described below. We simplify the structure of ¯ Q ∗ by using the fact that Y ′ = Y ′′ :

Then, we use the property that X = X ′ + Y ′ + n X , and Y = Y ′ + n Y , and use that to appropriately modify ¯ Q ∗ to obtain Q A as follows:

We calculate a new distribution ¯ Q ∗ ( M,X ′ + Y ′ , Y ′ , n X , n Y ) :

where Z = X ′ + Y ′ and alternatively also Z = X -n X (as X = X ′ + Y ′ + n X ). First note that ¯ Q ∗ ( Z | M ) from Appx. M.2 has the form f ( δ X m -δ X bias ) , which for the Poisson case is Poisson ( f 1 ( M ) -δ X bias ) . Furthermore, we can recognize that ¯ Q ∗ ( Y ′ = y | Z = z, M = m ) = ¯ Q ∗ ( Y ′ = y ′ | X ′ + Y ′ = x ′ + y ′ , M = m ) = G ( y ′ ; ϵ (1) m , 1 -ϵ (1) m , x ′ + y ′ ) , which for the Poisson case can be written as Binomial ( Z, p m ) where p m = f 2 ( M ) -δ Y bias / f 1 ( M ) -δ X bias .

Now define S m x = { x n X ∈ X n m X , z ∈ Z m : x n X + z = x } , where X n m X and Z m are the corresponding support sets of the distributions ¯ Q ∗ ( n X | M = m ) and ¯ Q ∗ ( Z | M = m ) ,

respectively. Then, using the fact that X = Z + n X , we have:

where we know that for the Poisson case:

Using a similar procedure, we derive Q A ( M,X,Y ) from ¯ Q ∗ ( M,X,Y ′ , n Y ) as Y = n Y + Y ′ . Define S m y = { y n y ∈ Y m n X , z ∈ Y ′ m : y n X + y ′ = y } , then:

After analytically calculating Q A for all 100 bias pairs of ( δ X bias , δ Y bias ) , we chose Q A corresponding to the pair ( δ X bias , δ Y bias ) which results in the smallest value of the corresponding I Q A ( M ; [ X,Y ]) .

M.4 Additional results and analysis

We report the tightness of the upper-bound for systems employing binomial and negative-binomial distributions in Fig. 2 (a) . Wefind that the results for binomial and negative-binomial remain consistent with the Poisson study performed in Sec. 6 and the upper-bound continues to remain tight. We also performed further analysis to understand how our upper-bound performs when both of the UI terms are non-zero. We sub-selected the trials in the 20 pairs (analyzed in Section 6) that had both UI x > 0 . 01 bits and UI y > 0 . 01 bits, and report the corresponding median error of these sub-selected trials in Fig. 2 (d) .

N Distributions

We provide brief descriptions of the distribution families and the corresponding distributions used in this work. We also provide citations to relevant works for the reader interested in learning more about these distributions.

(a) Upper Bound (UB): Binomial & Neg. Binomial

(b) UI

x

-Poisson

(c) UI

y

-Poisson

(d) UB: Non-zero Poisson UI

Difference in Analytical MI

5

w.r.t. Numerical MI

4%

Binomial

4

Neg. Binomial

3

乙

工

1%

0

%

1

5

10

FunctionsPairIndex

0.25

0.20

bits)

0.15

0.10

0.05

0.00

1

5

10

15

20

FunctionPairIndex

2.00

1.75

1.50

bits)

1.25

1.00

×

0.75

0.50

0.25

0.00

1

5

10

15

20

Function Pair Index

Difference in Analytical MI

5

w.r.t. Numerical MI

4%

4

2

1%

%

0

1

5

10

15

20

FunctionsPairIndex

N.1 Common distributions used in this work

We employed various well-known distributions in this work. We refer the reader to [62, 63] for a review of these distributions. We use the most commonly used notations in literature to represent these distributions. Regardless, we provide explicit definitions of our notations for these distributions to remove ambiguity.

Poisson Distribution Poisson distribution is a discrete distribution having support over all nonnegative integers. The p.m.f. of the Poisson distribution is described by a single rate parameter λ ∈ [0 , ∞ ) . We define the Poisson distribution with the rate parameter λ = 0 as a degenerate point distribution with all its mass at 0 . We use Poisson ( λ ) to denote the p.m.f. of a random variable X distributed according to the Poisson distribution. The exact definition of the p.m.f. of X is provided below:

Gaussian Distribution Gaussian distribution is a continuous distribution having support over R . The p.d.f. of the Gaussian distribution is described by its mean µ ∈ R and variance σ 2 ∈ [0 , ∞ ) . We define the Gaussian distribution with the variance σ 2 = 0 as a degenerate point distribution with all its mass at µ . We use N ( µ, σ 2 ) to denote the p.d.f. of a random variable X distributed according to the Gaussian distribution. The exact definition of the p.d.f. of X is provided below:

Negative Binomial Distribution Negative binomial distribution is a discrete distribution having support over all non-negative integers. The p.m.f. of the negative binomial distribution is described by two parameters: the number of successes r ∈ N , and the probability of success p ∈ (0 , 1] . We use Negative Binomial ( r, p ) to denote the p.m.f. of a random variable X distributed according to the negative binomial distribution. The exact definition of the p.m.f. of X is provided below:

Geometric Distribution Geometric distribution is a special case of negative binomial distribution with r = 1 . The p.m.f. of the geometric distribution is described by a success probability p ∈ (0 , 1] . We use Geometric ( p ) to denote the p.m.f. of a random variable X distributed according to the geometric distribution. The exact definition of the p.m.f. of X is provided below:

Beta-Binomial Distribution Beta-binomial distribution is a discrete distribution having support over { 0 , . . . , N } for some N ∈ N 0 . The p.m.f. of the beta-binomial distribution is described by three parameters: N specifies the support, α ∈ (0 , ∞ ) and β ∈ (0 , ∞ ) . We use Beta-Binomial ( N,α,β ) to denote the p.m.f. of a random variable X distributed according to the beta-binomial distribution. The exact definition of the p.m.f. of X is provided below:

Gamma Distribution Gamma distribution is a continuous distribution having support over (0 , ∞ ) . The p.d.f. of the gamma distribution is described by its shape parameter α > 0 and rate parameter β > 0 . We use Gamma ( α, β ) to denote the p.d.f. of a random variable X distributed according to the gamma distribution. The exact definition of the p.d.f. of X is provided below:

where Γ( · ) is the Gamma function.

Exponential Distribution Exponential distribution is special case of the Gamma distribution with α = 1 . The p.d.f. of the exponential distribution is described by its rate parameter λ > 0 . We use Exponential ( λ ) to denote the p.d.f. of a random variable X distributed according to the exponential distribution. The exact definition of the p.d.f. of X is provided below:

Beta Distribution Beta distribution is a continuous distribution having support over [0 , 1] or (0 , 1) depending upon the values of its parameter. The p.d.f. of the beta distribution is described by two shape parameters: α > 0 and β > 0 . We use Beta ( α, β ) to denote the p.d.f. of a random variable X distributed according to the beta distribution. The exact definition of the p.d.f. of X is provided below:

where Γ( · ) is the Gamma function. The end-point 0 is included in the support of the beta distribution if α ≥ 1 and the end-point 1 is included in the support of the beta distribution if β ≥ 1 .

Dirichlet Distribution Dirichlet distribution is a multivariate generalization of the beta distribution. The p.d.f. of Dirichlet distribution is described by d -shape parameters: { α 1 , . . . , α d } , where α i ∈ (0 , ∞ ) ∀ i ∈ [ d ] . We use Dirichlet ( α 1 , . . . , α d ) to denote the p.d.f. of a d -dimensional random vector ⃗ X distributed according to the Dirichlet distribution. The exact definition of the p.d.f. of X is provided below:

where Γ( · ) is the Gamma function, and ⃗ x = [ x 1 . . . x d ] T , with x i ∈ [0 , 1] and ∑ d i =1 x i = 1 .

Binomial Distribution Binomial distribution is a discrete distribution having support over { 0 , . . . , N } for some N ∈ N 0 . The p.m.f. of the binomial distribution is described by two parameters: the total number of trials, N , and the success probability, p . We use Binomial ( N,p ) to denote the p.m.f. of a random variable X distributed according to the binomial distribution. The exact definition of the p.m.f. of X is provided below:

Multinomial Distribution Multinomial distribution is a multivariate generalization of the binomial distribution. The p.m.f. of the multinomial distribution is described by two parameters: the total number of trials, N , and the success probability vector, ⃗ p . We use Multinomial d ( N,⃗ p ) to denote the p.m.f. of a d -dimensional random vector ⃗ X distributed according to the multinomial distribution. The exact definition of the p.m.f. of ⃗ X is provided below:

where, ⃗ p = [ p 1 . . . p d ] T is a d -dimensional probability vector, i.e. 0 ≤ p i ≤ 1 , ∀ 1 ≤ i ≤ d and ∑ d i =1 p i = 1 , and ⃗ x = [ x 1 . . . x d ] T is a d -dimensional categorical vector, i.e. x i ∈ N 0 , ∀ 1 ≤ i ≤ d , and ∑ d i =1 x i = n .

Hypergeometric Distribution The p.m.f. of the hypergeometric distribution is described by three parameters: the total number of success states in the population S ∈ N 0 , the total number of failure states F ∈ N 0 and the number of draws n ∈ { 0 , . . . , S + F } . We use Hypergeometric ( S, K, n ) to

denote the p.m.f. of a random variable X distributed according to the hypergeometric distribution. The exact definition of the p.m.f. of X is provided below:

Multivariate Hypergeometric Distribution Multivariate hypergeometric distribution is a multivariate generalization of hypergeometric distribution. The p.m.f. of the multivariate hypergeometric distribution is described by the following parameters: the total number of object types d ∈ N 0 , the size of population of each type { n 1 , . . . , n d } with n i ∈ N 0 , and the number of draws n ∈ { 0 , . . . , ∑ d i =1 n i } . Define ⃗ n = [ n 1 . . . n d ] . Then, we use Multivariate Hypergeometric ( ⃗ n , n ) to denote the p.m.f. of a d -dimensional random vector ⃗ X distributed according to the multivariate hypergeometric distribution. The exact definition of the p.m.f. of ⃗ X is provided below:

where ( A B ) = A ! B !( A -B )! for some A,B ∈ N 0 and B ≤ A , and ⃗ x = [ x 1 . . . x d ] T with x i ∈ { 0 , n i } and ∑ d i =1 x i = n .

Uniform Distribution Uniform distribution is a continuous distribution having support over ( a, b ) where a, b ∈ R . We use Uniform ( a, b ) to denote the p.d.f. of a random variable X distributed according to the uniform distribution. The exact definition of the p.d.f. of X is provided below:

For uniform distribution, we abuse notation by not ensuring a < b for representing the interval [ a, b ] . In the context of uniform distribution, the notation [ a, b ] is to be understood as [ a, b ] if a ≤ b and [ b, a ] if a ≥ b .

Bernoulli Distribution The p.m.f. of the random variable X distributed according to the Bernoulli distribution is denoted as Bernoulli ( p ) for p ∈ [0 , 1] . The exact definition of the p.d.f. of X is provided below:

Categorical Distribution The p.m.f. of a random variable X distributed according to the categorical distribution is denoted as Categorical ( p 1 , . . . , p d ) for p i ∈ [0 , 1] ∀ i ∈ [ d ] and ∑ d i =1 p i = 1 . The exact definition of the p.d.f. of X is provided below:

N.2 Univariate Stable Distribution Definition

We employ the definition of a univariate stable distribution discussed in [23]:

Definition 2. A random variable X is said to have a stable continuous distribution if, for two independent copies of X , denoted as X 1 and X 2 , and positive constants a > 0 and b > 0 , (322) holds for some c > 0 and d ∈ R .

Continuous stable distributions are typically expressed through their characteristic functions, as the analytical form of many continuous stable distributions is not known. The characteristic function of a random variable X having a continuous stable distribution is typically parameterized by four parameters: stability parameter denoted as α ∈ (0 , 2] , skewness parameter denoted as β ∈ [ -1 , 1] , scale parameter denoted as γ ∈ (0 , ∞ ) , and location parameter denoted as µ ∈ R . The exact form of the characteristic function is given in (323).

̸

We further denote the p.d.f. of X having the characteristic function defined in (323) as p CS ( X ; α, β, γ, µ ) . Some well-known examples of continuous stable distribution are the Gaussian/normal distribution, Cauchy distribution, and Lévy distribution. All univariate continuous stable distributions have infinite variance except the Gaussian distribution, and their tails follow a power law behavior [23]. Univariate stable distributions have been widely used to model many financial and physical phenomena. Chapter 2 of [23] provides a list of applications where stable distributions have been used. For a more in-depth treatment of stable distributions, we refer the reader to [23].

N.3 Multivariate Stable Distribution Definition

A multivariate stable distribution is defined similarly to a univariate stable distribution described in Appx. N.2. We employ the definition of multivariate distribution given in [23].

Definition 3. A non-generate d -dimensional random variable ⃗ X is said to have a multivariate stable continuous distribution if, for two independent copies of ⃗ X , denoted as ⃗ X 1 and ⃗ X 2 , and positive constants a > 0 and b > 0 , (324) holds for some c > 0 and ⃗ d ∈ R d .

A main source of complexity while analyzing multivariate continuous stable distributions is the wide range of dependence structures this family of distribution admits [23]. We refer the reader [46, 45] for a more general treatment of multivariate stable distributions. In this work, we focus on two special classes of multivariate stable distributions: independent component multivariate stable continuous distribution denoted as p CS -IC ( α, ⃗ β , ⃗ γ , ⃗ µ ) and elliptically contoured multivariate stable distribution denoted as p CS -EC ( α, Σ , ⃗ µ ) . Similarly to the univariate continuous stable distribution, both independent component and elliptically contoured multivariate stable distributions do not have explicit general analytical forms and are expressed through their characteristic functions.

The characteristic function of a d -dimensional random vector ⃗ X having independent component multivariate distribution p CS -IC ( α, ⃗ β , ⃗ γ , ⃗ µ ) is expressed as follows:

where Φ( α ) and sgn ( · ) are as defined in (323), ⃗ β = [ β 1 . . . β d ] T with β j ∈ [ -1 , 1] , ⃗ γ = [ γ 1 . . . γ d ] T with γ j ∈ (0 , ∞ ) , ⃗ µ = [ µ 1 . . . µ d ] T ∈ R d , and α ∈ (0 , 2] . One can also alternatively think of the random vector ⃗ X = [ X 1 . . . X d ] T having the distribution p CS -IC ( α, ⃗ β , ⃗ γ , ⃗ µ ) as a collection of independent random variables { X j } d j =1 , where X j ∼ p CS ( α, β j , γ j , µ j ) .

The characteristic function of a d -dimensional random vector ⃗ X having elliptically-contoured multivariate distribution p CS -EC ( α, Σ , ⃗ µ ) is expressed as follows:

where Σ is a positive definite matrix, ⃗ µ = [ µ 1 . . . µ d ] T ∈ R d , and α ∈ (0 , 2] . Note that allowing Σ to be positive semi-definite will violate the fact that ⃗ X should be a non-degenerate d -dimensional random variable. We also list an important property of multivariate continuous stable distributions used in this paper.

Property 1. A d -dimensional random vector ⃗ X has a multivariate stable continuous distribution if and only if every 1 -dimensional projection has a univariate continuous stable distribution, i.e., ⃗ t T ⃗ X ∼ p CS ( α, β ( ⃗ t ) , γ ( ⃗ t ) , µ ( ⃗ t )) ∀ ⃗ t ∈ R d .

Proof. See the proof of P.1. in [46].

N.4 Univariate Discrete Stable Distributions

Discrete stable distributions form the discrete analogues of the continuous stable distributions. We employ the following definition for a univariate discrete distributions taken from [24]:

Definition 4. A random variable X is said to have a discrete stable distribution with exponent ν ∈ (0 , 1] if for two independent copies of X , denoted as X 1 and X 2 , (322) holds for all γ ∈ [0 , 1] .

where d = means equality in distribution [57], and ◦ implies a binomial thinning operation defined as follows: γ ◦ X = ∑ X j =1 N j , where N j ∼ Bernoulli ( γ ) , and all N ′ j s are jointly independent .

Similar to univariate continuous stable distributions, univariate discrete stable distribution do not have explicit analytical forms, and are typically expressed through their probability generating functions and/or characteristic functions . The characteristic function of a discrete random variable N having a stable discrete distribution is given by (328):

where P N ( z ; ν, τ ) is the probability generating function of N , i = √ -1 , τ > 0 , and 0 < ν ≤ 1 . We further denote the probability mass function of N as P DS ( N = n ; ν, τ ) . P DS ( N ; 1 , λ ) is a standard Poisson distribution with mean λ [24]. All discrete stable distribution asymptotically follow a power law distribution [64].

N.5 Multivariate Poisson Distribution

To our knowledge, there are no well-known multivariate generalizations of univariate discrete stable distributions, except for the special case of the Poisson distribution. In this section, we describe a popular multivariate extension of the Poisson distribution.

An intuitive way to define multivariate Poisson distribution is to represent each random variable in the multivariate Poisson distribution as a sum of independent Poisson random variables [41, 42, 43, 44]. To illustrate, let us construct a bivariate Poisson random vector ⃗ N = [ N 1 N 2 ] T , where

and N g 1 , N g 2 , and N g 12 are mutually independent Poisson random variables with rates λ 1 , λ 2 , and λ 12 , respectively. Here, the dependence between N 1 and N 2 is expressed through N g 12 , with covariance between N 1 and N 2 being equal to λ 12 [47]. For dimensions d > 2 , the Poisson random vector:

where A is an appropriate matrix of 0 's and 1 's. We can decompose A = [ A 1 . . . A d ′ ] , where A i is a d × ( d i ) submatrix having no duplicate columns, and each of its columns containing exactly i ones and ( d -i ) zeros [44], and ⃗ N g = [ N g 1 . . . N g d N g 12 . . . N g ( d -1) d . . . N g d -( d ′ -1) ...d ] T , with N g i 1 ...i j ∼ Poisson ( λ i 1 ...i j ) ∀ ( i 1 , . . . , i j ) ∈ A d j , j ∈ [ d ′ ] . Furthermore, the random variables { N 1 , . . . , N d -( d ′ -1) ...d } are mutually independent. An intuitive way to think about this multivariate Poisson distribution is to interpret { N g i } d i =1 as the 'main effect', { N g i 1 i 2 } ( i 1 ,i 2 ) ∈ A d 2 as the 'two-way covariance effects' and so on until { N g i 1 ...i d ′ } ( i 1 ,...,i d ′ ) ∈ A d d ′ as the d ′ -way covariance effect in an ANOVA-like fashion [44]. For a more detailed discussion, see [43, 44, 47, 42]. Note that d ′ ≤ d . In this work, we denote the d -dimensional random vector ⃗ N having a multivariate Poisson distribution with parameters ⃗ Λ = [ λ 1 . . . λ d λ 12 . . . λ d -( d ′ -1) ...d ] T specifying up to d ′ -way covariance effects (where d ′ ≤ d ), as ⃗ N ∼ Poisson ( d, d ′ , ⃗ Λ ) . Note that for d ′ = 1 , we have that K is a collection of independent Poisson random variables, and when both d = d ′ = 1 , we recover the scalar Poisson distribution: Pr( K = k ) = e -λ λ k k ! , ∀ k ∈ N 0 .

Deriving the p.m.f. of the Multivariate Poisson Distribution

Let ⃗ N ∼ Poisson ( d, d ′ , ⃗ Λ ) , where ⃗ N is a d -dimensional random vector. Then the corresponding p.m.f. of ⃗ N is described below:

Let ⃗ n ′ = [ n 12 . . . n ( d -1) d . . . n d -( d ′ -1) ...d ] T , and d ⃗ n ′ be the dimension of ⃗ n ′ , then:

where C = { ⃗ n ′ ∈ N d ⃗ n ′ 0 :( ⃗ a ′ i ) T ⃗ n ′ ≤ n i ∀ i ∈ [ d ] } , and

with ⃗ a ′ i being the i -th row of the matrix A ′ = [ A 2 . . . A d ′ ] .

Proof of ⃗ N ∼ Poisson ( d, d ′ , ⃗ Λ ) having the p.m.f. described in (331).

We know that ⃗ N = A ⃗ N g , where A = [ A 1 . . . A d ′ ] . Let the dimension of ⃗ N g be d ⃗ N g and S ⃗ n = { ⃗ n g ∈ N d ⃗ N g 0 : k = A k g } . Then, using the fact ⃗ N = A ⃗ N g , the p.m.f. of ⃗ N can be expressed using the p.m.f. of ⃗ N g in the following manner:

Since ⃗ N g is just a collection of mutually independent Poisson random variables, we can write P ( ⃗ N g = ⃗ n g ) as a product of scalar Poisson distributions, i.e.,

Substituting (333) into (332), and collecting all e λ i 1 ...i j terms we obtain:

Decompose the matrix A = [ A 1 A ′ ] , where A ′ = [ A 2 . . . A d ′ ] and ⃗ n g = [( ⃗ n g 1 ) T ⃗ n ′ T ] T , where ⃗ n g 1 = [ n g 1 . . . n g d ] T and ⃗ n ′ contains the rest of the elements in ⃗ n g . Using the fact that ⃗ n g ∈ S ⃗ n ⇒ ⃗ n = A ⃗ n g we have: ⃗ n = A 1 ⃗ n g 1 + A ′ ⃗ n ′ . Combining this with the fact that A 1 is an identity matrix, we obtain:

where ⃗ a ′ i is the i -th row of A ′ and i ∈ [ d ] . Substituting (335) into (334):

where the summation constraint is transformed to S ⃗ n ′ = { ⃗ n ′ ∈ N d ⃗ n ′ 0 :( ⃗ a ′ i ) T ⃗ n ′ ≤ n i ∀ i ∈ [ d ] } . The equivalence of the summations over the sets S k ′ and S k can be derived by considering that

n g i = n i -( ⃗ a ′ i ) T ⃗ n ′ , and n g i ≥ 0 . Collecting all the factorial terms, we get:

Substituting (337) into (336), we obtain:

Taking the term ∏ i 1 ∈ A 1 λ n i i 1 out of the summation as it does not depend upon ⃗ n ′ , we obtain:

Notice that ( a ′ i ) T ⃗ n ′ contains all the elements of ⃗ n ′ which have i in their subscript. Hence, expanding the term ∏ i 1 ∈ A d 1 λ -( ⃗ a ′ i ) T ⃗ n ′ i 1 and distributing over the product, we obtain the desired form:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The three main results of this work are clearly enumerated in Section 1 (Introduction) that demonstrate the main claim of the work which is analytically computing partial information decomposition for numerous systems employing well-known distributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We have provided a discussion on the limitations of this work in Section 7. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: A formal proof of each theorem is provided in the Appx. F- L, with clearly stated assumptions. A formal justification of the arguments used in Sec. 6 is provided in Appx. M. Section 3 discusses the general proof technique used in proving our main theorems. We also provide informal discussions on the constructions used for proving theorems 1- 7 in their respective sections, namely Sec. 4 and Sec. 5.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details of our simulation results are provided in Section 6, followed by more details in Appendix M.3. We have also provided the exact code used for running the simulation study.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided our code with the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided all simulation details in Section 6 and Appendix L.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification:We have provided a box-plot of our results accompanied by the actual data points used for computing the box-plot in Figure 1. The factors of variability are clearly stated in Sec. 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We did not feel the need to provide information on the compute resources required to run our simulation, as our main results are of theoretical nature. Furthermore, all our simulation results can be reproduced on any ordinary personal laptop and do not require any special compute resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted is mainly theoretical in nature and we believe it does not violate the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: While our research is mainly theoretical, we have tried to anticipate how it might affect the society negatively and positively.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe our work poses no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have provided the citation and the corresponding license for the code we used for running our simulation in Section 6.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies analytical computing of Partial Information Decomposition (PID) when the underlying distribution is stable distribution and convolution closed distribution. In particular, the paper considers BROJA-PID framework consisting of minimization problem of conditional mutual information over a set of distributions. Under this framework, computing PID requires to solve the minimization problem.  The minimization problem gives solution zero whenever the underlying random variables are formed Markov chain. \nThe main contributions of this paper are: 1) generalizing the Gaussian PID result to broader distributions by finding sufficient condition under which the mutual information in the minimization problem achieves zero; 2) connecting data fission and thinning results by showing some convolution closed distribution can be useful in construction of a solution to the minimization problem.\nThe authrors also provide list of system examples for which their results are applicable.\n\nSTRENGTHS:\nThis paper generalizes the result of PID computing for joint Gaussian distribution to a broader distribution family, some stable distributions and convolution closed distributions. The results in the paper provide a list of system models where the unique information is zero by exploring the properties of Markov chain and by constructing a distribution having Markov chain. \nMoreover, the paper provides a connection to the data fission/thinning and a way to construct upper bound using their results, which might be useful in computing PID for some systems.\n\nWEAKNESSES:\nWhile this paper extensively explores a sufficient condition (Markov chain) for computing the unique information, I am not sure how much this result contributes on the computing PID for general systems. Some of my concerns are:\n1) Most of results come from the construction of distributions having Markov chain property. In my opinion, the results are just examples of underlying system models that induce Markov chain. \n2) The assumption that the underlying conditional distributions should somehow linearly depend on the target random variable. This assumption seems quite strong in general system models. \n3) Although the results are claimed as computing PID, the results are not applicable for general nonzero unique information. I think zero unique information case seems very restrictive in general system models, so the results are applicable very few cases. \n4) Analysis of the proposed upper bound is not sufficient.\n\nQUESTIONS:\nI am curious how tight the proposed upper bound for other system models except Poisson distribution. Can the authors provide more simulation results under various settings? Particularly, does the upper bound work well for systems that does not include Markov chain property (zero unique information) or the assumptions in the paper?\n\nMoreover, can the authors provide comparisons with other numerical estimators for PID under various settings?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on analytically expression of the BROJA-PID [12, 33]. By considering the stable distribution family, the authors are able to generalize the previous Gaussian PID result [17] (one of the uniqueness always vanishes for BROJA-PID) to a wider class of distributions. An upper bound for convolution-closed distributions is also given.\n\nSTRENGTHS:\nThe BROJA-PID [12, 33] has been widely adopted due to the simplicity of its optimization form. This paper extend a famous Gaussian PID result [17] (one of the uniqueness always vanishes for BROJA-PID) to a wider class of distributions. This result simplifies the computation of many PID problems and provides a test bed for numerical PID estimators.\n\nWEAKNESSES:\nThe proposed analytical extension is only for the BROJA-PID [12, 33]. There exists other PID measures that diverges from the BROJA-PID even in the bivariate case. The paper does not provide sufficient discussion on this. Further, the title of the paper is misleading, as Partial Information Decomposition is not just BROJA decomposition. The authors should consider revising the title and make this clear in the beginning of the paper. Also, if analytical expression is already available, why \"computing\"?\n\nQUESTIONS:\n1. In line 131, it is stated that the main focus of this work is to analyze the cases where (2) is analytically solvable. What is the definition of \"analytically solvable\"?\n2. In section 4, the two systems are constructed in a symmetric manner. How do the results compare to [Niu and Quinn, 2020]?\n3. Do the analytical solutions only hold for BROJA-PID or do they also hold for a wider range of PID measures?\n\n[Niu and Quinn, 2020] Niu, Xueyan, and Christopher J. Quinn. \"Synergy and redundancy duality between gaussian multiple access and broadcast channels.\" 2020 International Symposium on Information Theory and Its Applications (ISITA). IEEE, 2020.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides an analytical way of evaluating Partial Information Decomposition (PID) of the BROJA type (BROJA-PID). PID is a framework used in neuroscience and other fields to analyze how two sources of information interact to affect a target. The primary challenge addressed by the paper is the difficulty in computing BROJA-PID, which involves a complex optimization problem over probability distributions. This difficulty is avoidable when the sources and the target obey the Gaussian distribution. By considering the reason why this avoidance is possible, the authors clarify a sufficient condition (certain affine structures are assumed when conditioned on the target) where the analytical evaluation of PID is possible for a larger class of stable distributions. This result is generalized for convolution-closed distributions and a subclass of the exponential family, to show a link with data thinning and data fission. Furthermore, an upper bound of the PID for convolution-closed distributions, which is analytically evaluated and is applicable in generic situations, is derived based on the invented technique. Numerical simulation is conducted to demonstrate the efficiency of the proposed formula.\n\nSTRENGTHS:\n1.\tInnovative Theoretical Contributions: The paper extends the known analytical PID results from Gaussian distributions to a broader class of stable distributions. This is a significant theoretical advancement as it provides new insights into the information structure of various well-known distributions. An analytical upper bound for PID for convolution-closed distributions is also useful for practical applications.\n2.\tPractical Relevance: The results are particularly relevant to neuroscience and related fields, where PID is used to understand complex interactions in the systems.\n\nWEAKNESSES:\n1.\tAccessibility: The paper is highly theoretical, which might limit its accessibility to a broader audience who may not have a strong background in statistics and information theory.\n2.\tLimited Scope of Applications: While the paper focuses on extending analytical results to stable and convolution-closed distributions, it does not address the applicability of these results to real-world data extensively. The results rely on specific assumptions about the underlying distributions. The practical applicability of these results depends on how well these assumptions hold in real-world scenarios, and more discussions about this point are desired.  Also, some more empirical validation on the real-world applications would enhance the impact of the present work.\n\nQUESTIONS:\nMy questions are mainly about the practical implementation/application of the presented result:\n1. How do you recommend handling situations where the underlying distributions are not exactly known and only empirical data is available?\n2. Could you clearly state the situations where your analytical formulas have a clear advantage over numerical methods?\n3. How do the theoretical results apply to high-dimensional data sets, and could you provide an example of such an application?\n\nLIMITATIONS:\nI think the authors addressed the limitations of the work adequately. For improvement, it is fine to answer and resolve the above equations and weak points.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides analytical solutions for the Partial Information Decomposition (PID) (specifically, the BROJA PID) for a broad class of continuous distributions. The core proposition of this work lies on the relationship between stable distributions, Markov chains, and the minimisation conditions for Blackwellian PIDs (such as BROJA). In addition to a wealth of novel analytical results, the paper also provides an original and useful link to data fission and data thinning, as well as upper bounds for PID in a broader class of distributions.\n\nSTRENGTHS:\nThis is a very strong paper with substantial theoretical results. It fills an important gap in the PID literature, which has so far been (almost, although not fully) confined to either Gaussian or discrete systems. The paper also links PID to other problems in statistics (data fission and data thinning), which grounds the results in a broader context. Although presentation could be improved, the authors do a reasonably good job at presenting a very mathematically involved subject while maintaining some degree of intuition.\n\nWEAKNESSES:\nDespite its title, the paper doesn't _really_ compute any PID analytically. To compute all PID atoms, one needs to calculate the 3 mutual informations in addition to the BROJA unique information. The authors do not show that these quantities can be analytically calculated for the same classes of distributions studied.\n\nSimilarly, the authors only provide sufficient conditions for the BROJA-PID to be zero, but they do not compute it analytically for cases when it is non-zero. This is a very important caveat and, although I still think the paper makes a solid contribution, many of its statements and section titles are somewhat overstated.\n\nI'm aware of the space limitations, but I would still consider it a weakness that the paper doesn't provide much of an illustration of its results. This leaves readers with the impression that many results are within reach, but without any clear intuition of what these results actually are. I would encourage the authors to capitalise more on the benefits they claim, e.g. showing actual analytical PID values from fat-tailed systems.\n\nQUESTIONS:\nWhy is the restriction of univariate M necessary?\n\nTheorem 1, one of the core results of the paper, has an important \"if\" clause in line 187. What is the intuition behind this condition? When does it hold? Please discuss and provide positive and negative examples.\n\nLIMITATIONS:\nThe authors adequately discuss their limitations at length.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-29"}