Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions
Authors: Wenyuan Zhao
Adithya Balachandran
Chao Tian
Paul Pu Liang
Venue: NeurIPS 2025 poster ()
Keywords: Partial information decomposition
multimodal learning
information theory
Gaussian distributions
Primary area: general_machine_learning

Abstract: The study of multimodality has garnered significant interest in fields where analyzing interactions among multiple information sources can enhance predictive modeling, data fusion, and interpretability. Partial information decomposition (PID) has emerged as a useful information-theoretic framework to quantify the degree to which individual modalities independently, redundantly, or synergistically convey information about a target variable. However, existing PID methods depend on optimizing over a joint distribution constrained by estimated pairwise probability distributions, which are costly and inaccurate for continuous and high-dimensional modalities. Our first key insight is that the problem can be solved efficiently when the pairwise distributions are multivariate Gaussians, and we refer to this problem as Gaussian PID (GPID). We propose a new gradient-based algorithm that substantially enhances computational efficiency for GPID based on an alternative formulation of the underlying optimization problem. To generalize the applicability to non-Gaussian data, we learn information-preserving encoders to transform random variables of arbitrary input distributions into pairwise Gaussian random variables. Along the way, we resolved an open problem regarding the optimality of joint Gaussian solutions for GPID. Empirical validation on diverse synthetic examples demonstrates that our proposed method provides more accurate and efficient PID estimates than existing baselines. We further evaluate on a series of large-scale multimodal benchmarks to show its utility in real-world applications of quantifying PID in multimodal datasets and selecting high-performing models.

Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions
Wenyuan Zhao'
Adithya Balachandran"
Chao Tian'
Paul Pu Liang
Texas A&M University
Massachusetts Institute of Technology
[wyzhao,, chao. ttamu edu, 2'adithyab, ppliang)@mit.
Abstract
The study of multimodality has garnered significant interest in fields where the analysis of interactions among multiple information sources can enhance predictive modeling, data fusion, and interpretability. Partial information decomposition (PID) has emerged as a useful information thhooretic framework to quantify the degree which individual modalities independently, redundantly, or synergistically convey information about target variable. However, existing PID methods depend on optimizing over a joint distribution constrained by estimated pairwise probability distributions, which are costly and inaccurate for continuous and high-dimensional modalities. Our first key insight is that the problem can be solved efficiently when the pairwise distributions are multivariate Gaussians, and we refer to this problem Gaussian PID (GPID). We propose new tradient-based algorithm that substantially improves the computational efficiency of GPID based on an alternative formulation of the underlying optimization problem. To generalize the applicability to non- Gaussian data, we learn information information-preservinng encoders to transform random variables of arbitrary input distributions into pairwise Gaussian random variables. Along the way, we resolved an open problem regarding the optimality of joint Gaussian solutions for GPID. Empirical validation in diverse synthetic examples demonstrates that our proposed method provides more accurate and efficient PID estimates than existing baselines. We further evaluate series of large-scale multimodal benchmarks to show its utility in real-world applications quantifying PID in multimodal datasets and selecting high-performing models.
Introduction
Multimodal machine learning is fast-growing subarea of artificial intelligence research that aims to develop systems capable of integrating and fusing many heterogeneous modalities [2.31 In addition to much empirical progress, there has been a recent drive toward building theoretical foundations to understand when information in individual modalities is important and how this information becomes contextualized in the presence other modalities [32]. This aspect of quantifying interactions provides valuable insights into the significance of different modalities, the necessary amount of data required in each modality, and the methods most suitable for fusing multimodal representations [15) 35 61]. Partial Information Decomposition (PID), an advanced framework rooted in information theory, has been used as a formal framework to analyze how information is distributed
18
26
27
among multiple data sources for target task variable [ [7 55).
One fundamental challenge of PID is high computational complexity, especially when the size and dimensionality of the datasets are large 43 Accurate estimation of information--heorrtie measures, such as mutual information (MI), from empirical data non-trivial and even prohibitive, particularly for high-dimensional or continuous distributions
38
Estimating PID also presents a significant
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

challenge, as analytic approximations of these quantities can only be obtained when the features are pointwise discrete low-dimensional, However, for large-scale datasets, the number of optimization variables can be exponential in the number of neurons [53] The computational burden requires significant approximations or simplifications based on sampling that may compromise the accuracy of
PID estimation [[2]. Therefore, it is necessary to develop PID estimator that applies to continuous and high-dimensional multimodal data.
In this paper, we identify the first key insight that PID for Gaussian distributions, known as Gaussian PID (GPID), scalable for continuous modalities. We first develop new algorithm for GPID, which is exact and efficient in multivariate Gaussian distributions with high dimensionality. The second insight to learn feature encoders to transform arbitrary imput distributions into latent Gaussian space, without violating the information interactions between the original modalities. This transformation can be done by normalizing flows [39], whose invertible bijections preserve information while bringing the joint distributions closer to Gaussians 77. Subsequently, the PID) estimate can be dramatically simplified in latent Gaussian distributions, which enjoy a closed-form analysis of differential entropy and MI. We summarize the following contributions in this paper:
1. We propose new gradient-basee based algorithm, called Thin-PID, which significantly enhances the computational efficiency of PID estimates in latent Gaussian distributions.
2. We develop new framework, called Flow-PID, to generalize Gaussian PID algorithms to arbitrary input modalities using flow-based information-preserving encoders.
3. We demonstrate the improved accuracy and efficiency of our proposed Thin-PID and Flow- PID on diverse synthetic datasets with ground truth information labels. Further evaluation on a series of large-scale multimodal benchmarks shows its utility in real-world applications.
Finally, we release the data and code for Thin-PID and Flow-PID to encourage further studies of multimodal information and modeling at htttt:://giiiti com cww..//arrrreeeaaafoooooooddd
2 Background and related work
Let X1, X2, Y be three random vectors of dimension dx, : dx dy in their respective alphabets Xi, N2, and Y/ Denote A as the set of joint distributions in (A 22. 3). In multimodal learning, we are concerned with inferring on the class label or regression value Y, whether the modalities X and X2 can provide useful information together, beyond their unique information, and how much common information for we can ascribe to both X₁ and X2.
2.1 Partial information decomposition
PID decomposes the total information I,(X,, X2; Y), where Tp denotes the Shannon mutual infor- mation (9] in joint distribution p, between the target Y and two basic features (X1, X2) into four non-negative parts (55):
T,(X,, Y) (Y; U2(Y; X2\x1) R(Y; X1, S(Y: X1, (1)
where the four terms on the right-hand side respectively represent the information regarding Y that uniquely in X1, uniquely in X2, redundantly in either X₁ or X2, synergistically in both X2 and X2. It is also often required that the decomposition satisfy the conditions on individual information: (X2), (Y;
(Y;; (1) R(Y; xx,X,) +
R(Y; x,, X2) + U2(Y; ()))
The decomposition is not unique, since we only have three linear equations to specify four variables. We adopt the definition of PID introduced by Bertschinger al. (5). which appears align effectively with multimodal learning applications and has been shown to facilitate model selection (27].
Definition 2.1 (PID). The redundant, unique, and synergistic information are given by
R = max Xx;;;X2; rEAp
(2)
U1 min
(xxx; U2= = min (X2;; Y|X)), YEA, S = 1X1,, X2; Y) min 1ç(X,, X2; Y);
(3)
EEA,
7EA,
where A, := (g E A g( p(.;, v), Vy E V. E X E (2)) and I, is the mutual information (MI) over the joint distribution (( 22, y). Note that Ap only preserves the marginals p(21,2) y) and p(z2, y), but not necessarily the joint distribution p(z1, 22, y).

---

2.2 Normalizing flows
Estimating MI and differential entropy in PID can be very challenging for high-dimensional data, un- less in multivariate Gaussian distributions. Normalizing flows class of machine learning models that are used to transform a simple, tractable probability distribution (e.g. Gaussian distributions) to more complex distribution, while preserving exact likelihood computation and invertibility [39]
f()) == fx o fn-10 10.10-00000 fi(z),
where f(-) is invertible and differentiable, == fz) and px(x)) pz(=) det 0z 8z _
Unlike generative modeling, we adopt the inverse process of normalizing flows in PID to transform complex input modalities in such a way that MI can be preserved and computed using latent repre- sentations [f(xx fy = I,(X; Y). Subsequently, information interactions are computed efficiently in latent Gaussian distributions. This motivates us to design a novel framework for PID estimators by training normalizing flows that preserve the total information IX, X2: Y) and transform input modalities into latent spaces that are well approximated by multivariate Gaussian representations.
2.3 Related work
There have been some recent efforts on PID estimators. Liang et al. (27 introduced the CVX estimator, which formulates the PID definition as a convex optimization problem that can be solved if X2 and X2 are discrete and small. Liang al. [27] also developed the BATCH algorithm, which approximates PID for large datasets by parameterizing the joint distribution with neural networks and normalization satisfy marginal constraints. Therefore, several of these methods are limited in scope and do not scale well to continuous, high-dimensione data. Venkatesh et al. [52] introduced ~P-, which restricts g(x), #2, E Ap to joint Gaussian distributions, where PID values are easier to estimate given the analytical entropy and MI. However, the optimality of joint Gaussian distributions remains an open question, which implies that the joint Gaussian solution may not be true in general and only provides an optimizing bound for GPID estimation.
PID estimation:
Information theory estimation: Neural estimators of cntematiion-hhoorttie quantities have become essential tools in machine learning due to their ability to scale to high-dimensional. continuous data. Mutual Information Neural Estimator (MINE) leverages the Donsker-Varadhan representation of the KL divergence to construct a variational lower bound on MI [4] Alternative estimators based on f-divergences, including NWJ [36] and InfoNCE (37, provide tighter bounds specific cases. Entropy estimation has also been addressed via neural score matching and normalizing flows [44]. These methods optimize variational bounds via gradient descent, which is parameterized by neural networks [4]]. Such approaches cannot be directly adapted for PID estimation due to the additional optimization problem in Equations to
More recent works include 7 16
cntrmattion-hhoreetee multimodal learning: Estimating multimodal information has been a critical step toward developing better benchmarks and algorithms for multimodal learning. Several bench- marks are categorized by the types of information that modalities contribute, which subsequently inspire research into new multimodal fusion methods (30 Deeper studies of multimodal information have also inspired new ways to guide the collection of pre-traiming data 6 and new multimodal pre-training objectives (34]. Multimodal contrastive learning a popular approach in which repre- sentations of the same concept expressed in different modalities are matched together (i.e., positive
pairs) and those of different concepts are far apart (i.e., negative pairs) 13 25 41). It can be shown that contrastive learning provably captures redundant information across the two views [7] 48 and
recent work has proposed extensions to capture unique and synergistic information [I
A new Gaussian PID theory and algorithm
In this section, we present new PID estimator for Gaussian distributions and establish the notation we use in the rest of the paper. The first contribution of this paper is show that the optimal solution of the restricted PID optimization is jointly Gaussian. Secondly, we propose a new algorithm for GPID, which significantly enhances computational efficiency for high- dimensiiiaii features.
3

---

GPID is exactly the PID problem when the pairwise marginals are known to be Gaussians. We adopt the following notation in the subsequent discussion. Suppose [X .T] are jointly Gaussian vectors. Without loss of generality, we assume that they have zero mean and use the following Definition 3.1 (GPID). Let A be the set of joint distributions, where p(æ1. y) and p(z2, v) are pairwise Gaussian. The synergistic information about in X1 and X2 is given by
S 1,,,
min gEAg
(6)
where 49 := {a A
g( v) p(z,, y), Vy E y, 2i E 2, E [2]]. Subsequently. redundant and unique information can be computed by R max (X1; Y) and U1
minge bg (X1; X2)), U2 mingeag L (X2; { respectively.
notation to denote the covariance matrices. More detailed derivation can be found in Appendix A
Σxy denotes the (dx + dy) x (dx + dy) auto-covariance matrix of the vector [XT, VT). Joff Xy denotes the dx dy cross-covariance matrix (off-diagonal of Exy) between X and Y.
3.1 Optimality of joint Gaussian solution
In [52] instead solving GPID, the authors directly restricted the set A in GPID by placing the additional constraint that g(T1, T2, y) Gaussian, and then optimized within this restricted set, without showing that doing so does not cause any loss of optimality. In other words, PPPIID may only provide lower bound of for GPID in general.
Question: joint Gaussian solution g 22, 22,1) indeed optimal for GPID? Answer: Yes. It is optimal as long as p(æ1, y) and p(æ2, y) are Gaussians.
Optimality: We next show that there always exists jointly Gaussian optimizer g( 22, y) for GPID in Definition 3.1 Recall that we are interested in the following optimization problem:
min GEA6
(X1, X2; Y) min (h, hig (1, 9E
where h-) denotes the differential entropy of the distribution g. Since the marginal g is preserved to be the same as in both p(œ1, y) and p(r2, y). the problem is clearly equivalent to:
max hy X2).
(8)
gEA6
The key to establishing the optimality of Gaussian solutions is the inequality given in Lemma 3.2 whose proof is given in Appendix A.1 It now only remains to argue that the upper bound on the right-hand side of (9) can be achieved with a jointly Gaussian q AG for any g A9. This obvious since q has the same first and second moments as g(F1, 32, y), and we have g E AG Therefore, it is without loss of optimality to restrict the set A to only joint Gaussians.
Lemma 3.2. For any q(01, 122, y) with finite first and second moments, have
hg (Y) < hg (1),,,X2):
(9)
where ((I1, y) is jointly Gaussian distribution with the same first and second moments as q(21, I2, y).
3.2 Thin-PID: new algorithm for GPID
We are now in a position to discuss the problem of how to compute GPID. Recall that Venkatesh and Schamberg (53) used a two-user Gaussian broadcast channel interpret GPID, where Y the transmitter input variable, and X1 X2 are the channel outputs at the two individual receivers:
X = H Y n, and X₂ H2Y + 722-
(10)
Without loss of generality, we assume that m. and n2 are independent additive white noise, i.e., Σ,,, Σx,y = Tax, and Σn, = Σx,r = Tax,- Otherwise, we can perform receiver-side linear transformations through eigenvalue decomposition. For the same reason, we assume

---

Table I: Complexity analysis of different GPID algorithms. The complexities of ED and SVD are cubic in the values shown in the table. Thin-PH -DD- achieves better complexity on any scale of computation.
SVD min(dx, dxa)
Lin-Bqn Solve
Mul
Thin-PIID Tilde-PID
2 min(dx, dx,)
4 * min(dx, dx,) 8 (dx, dxo)
dx, dxa
max dxa)
2 (dx: + dx,)
Y~ ~~N(0, Ey) has zero-mean. Although the constraints p(æ1, y) and p(z2, y) can be specified by Ex,y and Σx,y Y (or equivalent En, and En, [ the joint distribution of n and ne for g(X1, 1, X2, Y) remains to be optimized. However, they be assumed be jointly Gaussian with covariance
Ening as shown in Section 3.1
Using the interpretation of the Gaussian broadcast channel, synergistic information S for GPID can be recast as the cooperative gain with the input signal Y [46]. As a result, min Lg (X1, X2; Y) in S can be determined by optimizing the worst possible correlation between n1 and 122 in the least favorable noise problem [58].
Theorem 3.3 (Thin-PID). The optimization problem of synergistic information S in Definition 3.1
can recast as minimizing the following objective function:
HEy H + Enina
minimize: ( = log
[nnnal
(11)
subject to: En, Eng I1, Ening O,
where H = [H] [H]] is the concatenation of two channel matrices, and Ening is the covariance of tttoo joint noise vectors 1 and n2 with cross-covariance Seff m2z"
In this objective, H and My are constants that can be estimated directly from the marginals and will not affect the optimization. Similarly, En₁ and Eng, which are diagonal block matrices of sizes dx, x dx, and dx, x dxy, can be whitened as identity matrices before optimizing GPID. Therefore, the only variable to be optimized is Injmng which is the off aff-diagonal block matrix of Σn
Proposition 3.4 (Projected gradient descent for Thin-PID). The gradient of the unconstrained Joof given by
objective C( in Equation (11) with respect to 2ning
VL Foff = GT.
rufff 111n
(I-E
Doff
(12)
where B = B2, G G GT G1,2, and we define the block matrix as
G
G1.2
G:
G
G2.2
HEyH Emjngs G1,2 E Edx, xdx2,
(13)
The gradient descent updates on Ynff mnnnn'
D ngng
+ RRrop( minng
[42
The projection operator onto the constraint set can be obtained by laking the singular value decom- position (SVD) on Deff "
SVD (Fof = UAV
(14)
Proj Proj (D UAVT.
(15)
where A = (Xe), No = min (max (0, Ad),11 1) and Ā diag(Ne).
Complexity analysis: In Thin-PID, the computational bottleneck comes from determining SVD off and inverting matrices. SVD on "Rina requires O (min(dx,, :x,,,, )x,)²) complexity. For the inverse matrices, note that G is constant and only needs to be computed once. The other inverse can be computed by solving linear equations with complexity O(d Without loss of generality,
'The joint covariance Σx,y can be fully specified by Σx, = En, since My is constant which can be directly computed from target Y.
5

---

Flows
Latent Gaussian marginals
Interprete as Gaussian PID
Thin-PID objective
Data
Estimater Hj, Hy. Zay
Thin-PID Optimizer
B
(AN1))) to Not. Exg))
X1
min 2
log subject No,
[M, +
f00
E
X
>i,
Data 2
B(XK)
((ay)) to Man Bogry
Gauesian broadcast charnel
Enyy, 20
Figure 1: Flow-PID learns latent Gaussian encoders, parameterized by the Cartesian flow f1 x f2 fy, to transform input modalities (X1, X2. Y) into Gaussian marginal distributions. Then, PID values can be computed efficiently via Thin-PID under the equivalent interpretation of GPID.
assume dx, dx, since we can always exchange input modalities. In contrast, the state-of-the- art Tilde-PiD proposed by Venkatesh et al. [52] requires the eigenvalue decomposition (ED) on Ening with the dominant complexity O ((dx, dx, )9) As shown in Table Thin-PID achieves significant improvement in computational efficiency, especially when the feature dimensions are high and dx, > dx,- More detailed complexity analysis is shown in Appendix A.4
4 Learning latent Gaussian encoder via normalizing flows
When the marginal distributions p(z1, v) and p(12, y) are not Gaussian, computing PID from data samples requires estimating the joint distribution in some manner. We propose a novel approach where we learn feature encoder transforming (X1, X2, Y) into latent space, such that they are well- approximated by Gaussian marginal distributions, then utilizing Thin-PID to perform the computation in an efficient way when modalities are continuous and dineessineii.
= fY( be three transformations that are defined by three neural networks, respectively. Given the dataset D {(a{() " ) 1,2..... .N. ideally the transformations should satisfy the conditions: 1) The transformations are invertible such that the The marginal distributions p(I .) and p( 9) are well-approximated by Gaussian distributions. Our goal is to learn Cartesian product of normalizing flows fi x f2 JY which will preserve the total mutual information as 1e (X1), f)),s (2());;; (Y))) = (X1, X2;
Let = fi X2 f2(X)), y=
MI does not change; 2)
according to Theorem 4.1 We refer to this method as Flow-PID, illustrated in Figure [1]
Theorem 4.1 (Invariance of total MI under bijective mappings). Let X1, N2, Y be absolutely contin-
wous vectors, Si : X + Ræxx, f2 : X₂ + Rdx, and JY y - Rdd be bijective piecewise smooth mappings with tractable Jacobians. Then
fv (v))) = Tx,,,, X2;
Corollary 4.2. Under the same conditions in Theorem 4.1 the PID of f((2)) the same as the PID of X2,
is Corollary 4.3. Let (X, Y) be absolutely continuous with PDF p(ar, y). Let Let v) be PDF defined in the same space as p(æ, y). Then T (X:Y) - (X; < KL. (p(x, ,))l((,,)))
4.1 Gaussian marginal loss
Theorem 4.1 and Corollary 4.2 imply that PID can be solved in latent space through invertible transformations that preserve the total information. However, it could restrict the possible distributions to an unknown family. Instead, we approximate the PDF in latent space via variational Gaussian marginals q y) and q(11, y) with tractable pointwise MI, and train & and fi x f2 x fy to minimize the discrepancy between the real and the approximated total MI.
To regularize the Cartesian flows with Gaussian marginal distributions, we simultaneously min- Ex,y)), where Ex,y and Σx,y are the covariance of the variational Gaussian marginals, and (l) denotes the KL diver- gence. Note that maximizing the likelihood of (X()) x fY(Y) and (x(2) sY(Y) also min- imizes KI. (px,y Y (JT g (xx,,))) and KI (px,y (5z g "')lW(( Ex,y)).
imize (L(p(&1, D) (M Ex,y)) and (L ú) [[[ [I,(((
We refer PPD as Tilde-PID PID in the sequel.

---

Unique and Redundant
Unique and Synergistic
Redundant and Synergistic
PID
Pertial Partial information (aits)
0.3
33
52
PID definition
Thin-PID
60
33
Tide-PID
Ground truth
given
Noise correlation.
Noise correlation. p
Figure 2: PID values for ID Gaussian example with different types of interactions. Thin- Thin-PID and Tilde-PID agree exactly with the ground truth.
Therefore, the Gaussian marginal regularizer is equivalent to maximize the log-likelihood of ((x2)) " such that .;(") = S; (a))), yod -)(())) where (a ((01) are sampled from variational Gaussian distribution N (M4, Ex,r).
Proposition 4.4 (Gaussian marginal loss for Flow-PID). Gaussian marginal objective for p( Ú) is given by
Given data samples {(x{)) 2 y the }.
0f log det
afr(v)) dy
Lar(x)) , EN( g(0) log det
(16)
8z,
where μr and Σx.y are determined by the Gaussian maximum likelihood. The objective function of the latent Gaussian encoder is
CRow X2) (W CC
(17)
5 Synthetic PID validation
In this section, we validate our Thin-PID and Flow-PID on synthetic Gaussian and non- non-Gaussian examples with known ground truth, compared with Tilde-PID designed for GPID (52), CVX, and BATCH designed for features with discrete support 27 Details of experimental settings are provided in Appendix C
5.1 Validating Thin-PID on canonical Gaussian examples
1D broadcast channel: We first validate the accuracy of Thin-PID on canonical Gaussian examples with dx, dx,- = dy 1. Let Y ~ (O 1). We design 3 cases with: 1) unique and redundanr; 2) unique and synergistic; 3) redundant and synergistic information. Detailed settings can be found in Appendix C. The ground truth can be solved exactly by MMI-PID 3 when p(11, 22, y) is ID Gaussian. The degrees of interactions estimated by Thin-PiD and Tilde-PID are shown in Figure 2 We observe that Thin-PID exactly recovers the ground truth in all three canonical Gaussian cases, which corroborates the correctness of our proposed Thin-PID algorithm for GPID.
Let X1,1 aYi + n1,1 3Y2 + 72,2, where Y1, Y2,n1, n22 i.i.d. N/ 1). i= 1, Here, (X1,1, X2,1. 1. Y1) is independent of (X1,2, X2,2, Y2). Using the additive property, we are able to aggregate the PID values derived from their separate decompositions, each of which is associated with known ground truth, as determined by the MMI-PID, considering Yi and Y2 as scalars. (ii Rotation: Let X, H(O),, where H, diagonal matrix with diagonal entries 3 and 1, and R(0) is a 2 x 2 rotation matrix that rotates Y at an angle O. When 0 = 0. X, has higher gain for Vi and X₂ has higher gain for Yg. When 0 increases to T/2, 2, X, and X₂ have equal gains for both Y1 and Y2 (barring difference in sign). Since (X1,1: X2,1, Yi) is not independent of (X1,2, X2.2, Y2) for all 0, we only know the ground truth at the endpoints.
GPID examples at higher dimensionality: (i) Cooperative Gain:
X2,1 = Y1 1, X1,2 Y2 + n1,2, X2.2
Results. The results of examples (i) and (ii) are shown in Figure 3 We observe that Thin-PID
achieves the best accuracy with the error 10 10-12 while the absolute error of Tilde-PID is > 10-8.
7

---

Increasing Gain in X;
Error in qain example
Rotation xf w.r.t. X2
PID component
Partial information (bits)
2.5
10
0.75
2.0
Partial Information
3,50
10
7.00
20
PID definition
10"d
Thin-PID
TIdd-PID
0.75
MMI-PID
10.4
truth
656
0.3
111
in X:,
X1.
Angle,
Figure 3: Left: PID values in Example (i); right: PID values in Example (ii); middle: absolute error between different GPID algorithms and the ground truth. Thin-PID achieves the best accuracy with < 100 error, while the absolute error of Tilde-PID is 10-8
Table 2: Non-Gaussian multi-dimensional examples: Tilde-PID is estimated from sample covariance
on 1e6 realizations of transformed non-G non-Gaussian variables. Flow-PID is estimated on the latent Gaussian representations by normalizing flows. Only Flow-Ph -ID agrees with the truth.
Dim
22,222
110,5,
[30,
10,2]
(100, 60,2)
PID
R
U
U2
5
R
U1
U2
5
R
U,
U2
5
R
U
U2
5
Tide-PID
0.29
0.84
1.19
0.16
0.19
1.48
197
0.13
How-PID
0.62
0.91
0.50
0.11
2.36
0.32
0.19
0.45
2.18
1.13
0.17
4.34
0.36
0
0.25
Truth
0.79
1.66
0.58
0.18
2.96
0.54
0.26
0.58
2.92
2.18
0.25
5.71
1.01
0
0.57
Time analysis: As discussed in Table Thin-PID sig- nificantly improves the computational efficiency when the feature dimension is high. We report the execution time of GPID algorithms when the feature dimension
Thin Tilde PID
200
Execution Time
increases in 2 cases: 1) both dx, and dx, increase; 2) dx, increases from 100 to 1000 with fixed dx,- = 100.
100
Figure 4 shows that Thin- PID costs much less time than Tilde-PID with speed of more than 10x when min(dx, (in(dx,,xx)) 100.
dX1 (dX2 0.6 dX1)
dXL (dX2 100)
5.2 Flow-PID non-Gaussian examples
Figure Time analysis: Thin-PID achieves 10x speed of Tilde-PII.
Multivariate Gaussian with invertible nonlinear transformation: Next we evaluate the Flow-PID when p(z,,)) v) is longer Gaussian. We sample from the joint Gaussian vectors (X1, Y) (0, Ex,x,,, and transform samples to XX,, Y) through absolutely invertible nonlinear func- tion (more details in Appendix C.2). According to Theorem 4.1 the MI of (X1, X², Y) remains the same as that of X2, Y), they are no longer pairwise Gaussian after the transformation. For Tilde-PID, the degree of interactions are computed by directly estimating the covariance of non-Gaussian samples. The "exact" PID is obtained by feeding the Gaussian covariance Σx,x, x,y to GPID directly. From Table 2 Flow-PIID aligns with the exact truth in relative PID values, whereas TiD fails with distorted nature and degree of interactions.
Specialized interactions with discrete targets: Although GPID is designed for continuous modalities and targets, evaluate its generalization to cases with discrete targets. Three latent vectors 21. 22, No0, Σ) are employed to quantify the information unique to both, respectively. [21, 2e] transformed high dimeemsimniill I1 using a fixed transformation T. Similarly, [22, ze] is transformed to via Tz. By assigning different weights to [21. 22, ze], we create ten synthetic datasets with different types of specialized interactions. As indicated in Table 3, Flow-PID not only accurately assigns the prevalent type of interaction, but also provides better quantification of the degrees of specialized interactions compared BATCH. noteworthy observation is that the estimation of PID regarding synergy S proves to be the most challenging and
and common

---

Table 3: Specialized interactions with discrete labels: Flow-PID provides more accurate PID values than BATCH, especially on datasets with high synergistic interactions. The specialized interactions determined by PID estimators are highlighted in bold.
Task
DB
De;
Dir
Dg
PID
11
U1
1/2
01
2/
5
1
U1
U2
5
11
U1
U2
S
BATCH
0.29
0.02
0.02
0.30
0
0
0
0.30
0.11
0.02
0.02
0.15
FlowPID
0.51
0
0.49
0
0
0
0.51
0
0.12
0
0.30
Truth
0.58
0
0
0
0.56
0
0
0
0
0.54
0
0
0
0
0,56
Task
[2),
(2)
PID
R
U
U2
S
U1
02
R
U1
02
BATCH
0.04
0.09
0.06
0.11
0.02
0.02
0.10
0.11
0.02
0.05
Flow-PID
0.06
0.17
0.12
0.20
0
0
0.23
0.10
0
0
0.08
Truth
0
0.25
0.25
0.18
0
0
0.36
0.22
0
0
0.22
Task
[[
Rg.
#2)
f(2g An.
PID
R
U
U2
S
U
U2
R
U,
Ug
BATCH
0.06
0.01
0.01
0.06
0.07
0
0.066
0.19
0.06
Flow-PID>
0.08
0.23
0.10
0
0.30
0.12
Truth
0.13
0
0.27
0.21
0
0.21
0.0
0.34
0
0.17
Table 4: Estimating PID on MultiBench datasets (30
Flow-PID recognizes more modality
interactions than CVX/BATCH, and effectively highlights dominant modalities.
Tamseis Modalies
AVMNIST
UR-FUNNY
Vision, Amiro
Viston,
Audio
Vision, T6xt
Audio, FOST
PID
R
U1
02
05
0/
R
01
0/
U.
0g
CVX/BATCH
0.06
Flow PID
0.12
0.53
0.09
0.19
0.98
0.28
0.04
0
1.19
0.03
0.03
0
1.22
0.12
Dalasess
MOST
MOSET
MUSSARD
Audio
75
755
1/3
01
15
CVXOBATCH
0.17
0.16
0.76
0.22
0.04
0.09
0.13
0.14
0.01
0.001
0.14
0.02
0.01
0.34
6.14
0.01
0.01
0.37
FlowPIDD
0.58
0
0.60
0.44
0.78
3.06
0.66
0.40
0
0.29
0.27
0.45
0
0.54
0.41
0.66
0
0.92
0.30
results in overestimated redundancy R. in BATCH. Conversely, Flow-PID mitigates this issue by achieving lower A and higher S.
Real-world applications of PID
Real-world multimodal benchmarks. We use a collection of real-world multimodal datasets in MultiBench (30), which spans 10 diverse modalities (images, video, audio, text, time-series), 15 prediction tasks, and 5 research areas. These datasets designed to test combination of feature learning and arbitrarily complex interactions under different multimodal fusion models in the real world. For datasets with available modality features (images, text), we use an end-to-end Flow-PID estimator. For other modalities (audio, time-series), we first use pretrained encoders to obtain features before Flow-PID (full dataset and experimental settings are available in Appendix D).
Results. From Table we observe that Flow-PID effectively highlights dominant modalities by assigning higher unique information to sources with stronger predictive contributions. Although the ground truth for real-world datasets is unknown (and may not be determined), this allows us to quantitatively assess which modalities are most informative for the task, offering deeper insight into modality importance beyond standard accuracy metrics. An interesting observation is that the total information (R U1 + U2 + S) recognized by Flow-PID is much larger than BATCH. The total information in a dataset yields an upper bound on multimodal model performance. On many of these datasets, multimodal models achieve over 75% accuracy, while the total information often under 0.5 for BATCH in Figure
Real-world datasets with task-driven and causally relevant interactions. We conducted Flow- PID on 2 real-world datasets with expected interactions, which are causally relevant to the tasks, to demonstrate our method in additional application areas and with unexplored modalities. 1) We quantified the PID of predicting the breast cancer stage from protein expression and nicroRNA expression on the TCGA-BRCA dataset. Flow-PID identified strong uniqueness for the modality of microRNA expression as well as moderate amounts of redundancy and synergy. These results are

---

also in line with modern research, which suggests microRNA changes as a direct result of cancer progression. 2) The results on VQA (Visual Question Answering) show the expected high synergy since the image and question complement each other to predict the answer. The detailed experimental results are shown in Table
Table 5: Flow-PID on real datasets with task-driven interactions.
Dataset
dim-X;
dim-X2
R
U1
U2
Expected interaction
TCGA
217
1881
0.41
0.0
1.07
0.34
VQA2.0
768
1000
0.22
0.26
0.0
0.76
Table 6: Model selection performance on new datasets D compared to the best-performing model.
Synthetic
AV-MNIST
ENRICO
UR-FUNNY
MOSI
MUStARD
Flow-PID
99.76% 99.91%
100% 99.85%
100% 100%
96.72% 98.58%
99.67% 99.35%
98.19% 95.15%
BATCH
Model Selection. After conducting evaluations on diverse multimodal datasets, we interested in whether GPID beneficial in selecting the most ap- propriate model capable of addressing the requisite interactions for a dataset. Given new dataset D, we measure the difference in normalized PID values be- tween D and ''' among a suite of 10 pretrained syn- thetic datasets with different types of specialized in- teractions. For each D', we pretrain 8 different mul- timodal fusions, rank the similarities to the unseen dataset, sD, D') = LIE(R,U, UU.S) 1t Ip/, and recommend the top-3 models on the most sim- ilar dataset D Table @ indicates that the selected models achieve more than 96% of the accuracy of the sess-eerformng model. UR-FUNNY records the comparatively lower accuracy, likely due to the significantly higher amount of unique information in the text modality compared to vision and audio.
Total mutual information (bits)
CVX BATCH FLOW
AAMNET tv. UN-FURY or. uR nn-in (V.
Finer LA, MUSTARD rv. A)
MUSTARD (V. T
MUSTRRD (4
most tv.
MOSE (A.1)
Figure 5: Total mutual information determined by Flow-PID and BATCH/CVX estimators.
7 Conclusion
In this paper, aim to develop dimensional modalities, of which PID estimates could be inaccurate and burdensome. We first identified that PID is casier to solve in latent Gaussian distributions without loss of optimality, and proposed a new GPID algorithm that significantly enhances the computational efficiency compared to the state-of of-the-art algorithms. Secondly, we develop latent Gaussian encoder via normalizing flows to generalize GPID algorithms to non- non-Gaussian cases. Through comprehensive experiments, demonstrated that our proposed method provides more accurate and efficient PID estimates than existing baselines, and showed the utility in diverse multimodal datasets and applications by dataset quantification and model selection.
and efficient PID framework for continuous and high-
Limitations: 1) The latent Gaussian encoders only approximate the marginal distributions through invertible bijective mappings. which introduce bias when the divergence between approximated Gaussian distributions and true underlying marginals is large. 2) Although Thin-PID is exact in GPID cases, the accuracy of Flow-PID largely depends on the performance of latent Gaussian encoders, and the optimization error could increase with more intricate underlying features or fewer data samples. 3) It is challenging rigorously justify the quantification real-world datasets, the generation of multimodal data is unknown.
Future work can leverage Flow-PID to expand datasets with specific objectives, enhance multi-task representation learning within the context of higher-dimensional data and continuous targets, and explore the fine-tuning pretraining of a large model under the guidance of Flow-PID.
10

---

Acknowledgments
The work of Wenyuan Zhao and Chao Tian is partly supported by NSF via grant DMS-2312173. We also acknowledge Nvidia for their GPU support.
References
[1] Allison, P. D. (1977). Testing for interaction in multiple regression. American journal of sociology, 83(1):144-155
[2] Baltrušaitis, T., Ahuja, C., and Morency, L.-P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, 41(2):223-443
[3] Barrett, A. (2015). Exploration synergistic and redundant information sharing in static and dynamical gaussian systems. Physical Review E, 91($):052802.
[4] Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. (2018). Mutual information neural estimation. International conference on machine learning. pages 531-540. PMLR.
[5] Bertschinger, N., Rauh, J., Olbrich, E., Jost, J., and Ay, N. (2014). Quantifying unique information. Eniropy, 16(4):1161-2183.
[6] Birhane, A., Prabhu, V. U., and Kahembwe, E. (2021). Multimodal datasets: misogyny, pornog- raphy, and malignant stereotypes. arXiv preprint 11X.1:21
[7] Butakov, I., Tolmachev, A., Malanchuk, S., Neopryatnaya, A., and Frolov, A. (2024). Mutual information estimation via normalizing flows. Advances in Neural Information Processing Systems, 37:3027-3057.
[8] Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., and Poria, S. (2019). Towards multimodal sarcasm detection (an _obviously_ perfect paper). arXiv preprint arXiv:1906.00 01815.
[9] Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
[10] Ding, Y., Li, S., Narasimhan, B., and Tibshirani, R. (2022). Cooperative learning for multiview analysis. Proceedings of the National Academy of Sciences, 119(38):c22211131
[11] Dufumier, B., Castillo-Navarro, J., Tuia, D., and Thiran, J.-P. (2024). What to align in multi- modal contrastive learning? arXiv preprint arXiv:2409.07402
[12] Ehrlich, D. A., Schick-Poland, K., Makkeh, A., Lanfermann, F., Wollstadt, P., and Wibral, M. (2024). Partial information decomposition for continuous variables based on shared exclusions: Analytical formulation and estimation. Physical Review E, 110(1):014155
[13] Frome, A., Corrado, G. S., Shlens, J., Bengio. S., Dean, J., Ranzato, M., and Mikolov, T. (2013). Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26.
[14] Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. (2016). Mul- timodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint 16Xiv:1606.00 01847.
[15] Gao, J., Li, P., Chen, Z., and Zhang, (2020). A survey on deep learning for multimodal data fusion. Neural computation, 32(5):829-866
[16] Gowri, G., Lun, X.-K., Klein, A. M., and Yin, (2024). Approximating mutual information of high-dimensional variables using learned representations. Advances in Neural Information Processing Systems, 37:132843-1323
[17) Griffith, and Koch, (2014). Quantifying synergistic mutual information. In Guided self-organization: inception, pages 159-190. Springer.

---

[18] Guo, W., Wang, J., and Wang, S. (2019). Deep multimodal representation learning: A survey, leee Access, 7:63373-63394.
[19] Hasan, M. K., Rahman, W., Zadeh, A. B., Zhong, J., Tanveer, M. I., Morency, L.-P., and Hoque, M. E. (2019). Ur-funny: A multimodal language dataset for understanding humor. In Pruceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (EMNLP-IJCNLP) pages 2046-2056.
[20] Hastie, T. and Tibshirani, R. (1987). Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371-388
[21] Hou, M., Tang, J., Zhang, J., Kong. W., and Zhao, Q. (2019). Deep multimodal multilinear fusion with high-order polynomial pooling. Advances in Neural Information Processing Systems, 32.
[22] Hu, Z., Kang. S., Zeng, Q., Huang, K., and Yang, Y. (2024). Infonet: neural estimation of mutual information without test-time optimization. arXiv preprint 1.Xi::2402.10158.
[23] Jaccard, J. and Turrisi, R. (2003). Interaction effects in multiple regression. Number 72. sage.
[24] Jayakumar, S. M., Czarnecki, W. M., Menick, J., Schwarz, J., Rae, J., Osindero, S., Teh, Y. W., Harley, T., and Pascanu, R. (2020). Multiplicative interactions and where to find them. In International conference on learning representations.
[25] Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H.,, Li, Z., and Duerig, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages
PMLR.
[26] Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. (2021). Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705.
[27] Liang, P. P., Cheng, Y., Fan, X., Ling, C. K., Nie, S., Chen, R., Deng, 7., Allen, N., Auerbach, R., Mahmood, F., et al. (2023a). Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems, 36:27351-27393.
[28] Liang, P. P., Deng, Z., Ma, M. Q., Zou, Y., Morency, L.-P., and Salakhutdinov. R. (2023b). Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36:32971-32998.
[29] Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q. Salakhutdinov, R., and Morency, L.-P. (2019). Learning representations from imperfect time series data via tensor rank regularization. arXiv preprint arXiv:1907, 1907.0101..
[30] Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021). Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 2D11
[31] Liang, P. P., Zadeh, A., and Morency, L.-P. (2024). Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10)11-4 42.
[32] Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. (2022). Mind the gap: Under- standing the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 15:17612-77625
[33] Liu. Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P., Zadeh, A. B., and Morency. L.-P. (2018). Efficient low-rank multimodal fusion with modality-speciiff factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2247-2256.
[34] McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Belyi, A., et al. (2024). Mml: methods, analysis and insights from multimodal Imm pre-training. In European Conference on Computer Vision, pages 304-323. Springer.
12

---

[35] Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng. A. Y., et al. (2011). Multimodal deep learning. In ICML, volume 11, pages 689-696.
[36] Nguyen, X., Wainwright, M. J., and Jordan, M. I. (2010). Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-588
[37] Oord, A. v. d., Li, Y., and Vinyals. O. (2018). Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.0348
[38] Pakman, A., Nejatbakhsh. A., Gilboa, D., Makkeh, A., Mazzucato, L., Wibral, M., and Schneid- man, E. (2021). Estimating the unique information of continuous variables. Advances in neural information processing systems, 34:20295-2085
[39] Papamakarios, G., Nalisnick, E.,, Rezende, D. J., Mohamed, S., and Lakshminarayanan, (2021). Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1-64.
[40] Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. (2019). On variational bounds of mutual information. In International conference on machine learning, pages 5171-5180. PMLR.
[41] Radford, A., Kim, J. W. Hallacy, C., Ramesh, A.. Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-8763. PmLR.
[42] Riedmiller, M. and Braun, H. (1993). A direct adaptive method for faster backpropagation learning: The rprop algorithm. In IEEE international conference on neural networks, pages 586-591. IEEE.
[43] Schick-Poland, K., Makkeh, A., Gutknecht, A. J., Wollstadt, P., Sturm, A., and Wibral, M. (2021). A partial information decomposition for discrete and continuous variables. arXiv preprint arXiv:21066 12393.
[44] Song, and Ermon, S. (2019). Understanding the limitations of variational mutual information estimators. arXiv preprint 1rXiv:1910.06222..
[45] Suzuki, M., Nakayama, K., and Matsuo, Y. (2016). Joint multimodal learning with deep generative models. arXiv preprint 1rXiv:1611.0189/
[46] Tian, C. and Shamai, S. (2025). Broadcast channel cooperative gain: An operational interpreta- tion of partial information decomposition. Entropy, 27(3):310.
[47] Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. (2020). What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839.
[48] Tosh, C., Krishnamurthy, A., and Hsu, (2021). Contrastive learning, multi-view redundancy, and linear models. In Algorithuiic Learning Theory, pages 1179-1206. PMLR.
[49] Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. (2019). Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, page 6558.
[50] Tsai, Y.-H. H., Liang, P., Zadeh, A., Moreney, L.PP. and Salakhutdinov, R. (2018). Learning factorized multimodal representations. arXiv preprint arXiv: 180..06166.
[51] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
[52] Venkatesh, P., Bennett, C., Gale, S., Ramirez, T., Heller, G., Durand, S., Olsen, S., and Mihalas, S. (2023). Gaussian partial information decomposition: Bias correction and application to high- dimensional data. Advances in Neural Information Processing Systems, 36.74622-74655.
13

---

[53] Venkatesh, P. and Schamberg, G. (2022). Partial information decomposition via deficiency for multivariate gaussians. In 2022 IEEE International Symposium on Information Theory (ISIT), pages 2892-2897. IEFE.
[54] Vielzeuf, V. Lechervy, A., Pateux, S., and Jurie, F. (2018). Centralnet: multilayer approach for multimodal fusion. In Proceedings of the European conference on computer vision (ECCV) workshops.
[55] Williams, L.. and Beer, R. D. (2010). Nonnegative decomposition of multivariate information. arXiv preprint arXiv: 1004..515.
[56] Wu, M. and Goodman, N. (2018). Multimodal generative models for scalable weakly-supervised learning. Advances in neural information processing systems, 31.
[57] Yao, and Wan, X. (2020). Multimodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguisties.
[58] Yu, W. and Cioffi, J. M. (2004). Sum capacity of gaussian vector broadcast channels. IEEE Transactions on information theory, 5009):1875-1822.
[59] Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. (2017). Tensor fusion network for multimodal sentiment analysis. arXiv preprint 177.02270...7250.
[60] Zadeh, A. B., Liang P. P. Poria, S., Cambria, E., and Morency, L.-P. (2018). Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pages 2236-2246.
[61] Zhang, C., Yang. Z., He, X., and Deng, L. (2020). Multimodal intelligence: Representa- tion learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Processing, 14(3):478-493
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We summarize our contributions of this paper in the abstract and introduction
regarding: theory and algorithm development, experimental validation and applications. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes].
Answer:
Justification:
We discuss the limitations in the last section where conclude the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof? Answer: [Yes].
Justification: We state the assumptions in definitions and theories. The proofs included in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
15

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: We disclose detailed settings of experiments in the appendix due to the page limits. The code for reproducing the results are also provided.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: The code is provided in zip file for supplementary materials. Due to the size limits, we only provide the source to download the data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co ec/
public/guiden http:/uuddes//ddebmiiiieee Soooodeeumisssoooooloooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
16

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpssccccpbccccccceceeceecccccc cetccc/giieee /ttp:///u/eee//eeeuuubbbiii CodeSuomissss wwwihibmmissiiooo ssinniiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: Detailed training and tests are shown in the appendix due to the pape limits. We have a pointer in the main text.
Guidelines:
The answer NA means that the paper does not include experiments. The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer:
[No].
Justification: The error bar for GPID is unnecessary since it is exact. For PID values in real- world datasets, the ground truth is unknown. Thus we do not include such measurements. But diseuss the robustness in the appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative стror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
17

---

Answer: [Yes].
Justification: We put the information on computer resources in the appendix due the page
limits of the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the http///rrrppp.cppppppttttt/ttiia
NeurlPS Code of Ethics
Answer:
[Yes].
Justification: This is mainly a technical paper on designing algorithms and theories. All the datasets used are open to the public.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: This is a technical paper, which does not discuss the societal impact. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system Iearns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA].
Justification: This a technical paper on building theories and algorithms. Such risks are not posed in this paper.
18

---

Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: We cited all the references and the code from existing baselines. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]..
Justification: The code for reproducing experimental results are included as anonymized zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: This is technical paper, and does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
19

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

Appendix
A The Gaussian PID theory and algorithm
Information theory quantifies how much information one variable X offers about another variable Y, which is formulated by Shannon's 's mutual information 19), represented as Y); It reflects the decrease in entropy from HY) to ( given X input. However, extending mutual information (MI) directly to three or more variables presents notable challenges. Specifically, the three-way MI / (X ;; X2; Y) can be both negative and positive, leading to considerable difficulty in its interpretation when quantifying interactions between multiple variables.
was introduced as framework to extend information theory to multiple variables. It decomposes the total information that two variables offer about a task [(X,, X2; Y) into four components: redundancy R shared between X1 and X2. unique information U1 specific to X1 and U2 to X2, and synergy S. These components must collectively fulfill the following consistency equations:
Partial Information Decomposition (PID) (55
R+ U1 Y()
(18)
R+ U2 = I(2;; Y),
(19)
U2 5 =+x5=I(X;:
(20)
((x);
(21)
R- = 12;))
(22)
A definition of R was first proposed by Williams and Beer (55] and subsequently improved by Bertschinger et al. [1. Griffith and Koch [17], which gives the PID definition we adopt in this work.
Definition A.1 (PID (77). The redundant, unique, and synergistic information are given by
R max X2;
(23)
rEAp
U1 min (X1 (22),
U2 = min (|x1), qEAp (X1,
(24)
gEAp
S = 12(X1, X2; y) min &EAp
(25)
where A, :- {g E A
g( 3) p(.;, v), Vy E y.z E Xi, E [2]]. and Ig is the mutual information (MI) over the joint distribution g T2 y). Note that Ap only preserves the marginals p(21,8) y) and p(z2. v), but not necessarily the joint distribution p(z1, 22. 22.
The definition of PID enjoys two properties:
1. Non-negativity: all the four decomposed components (/t, U1, U2, S) are non-negative.
2. Additivity: For two independent subsystems X,.X,,,, X2.1) and 11,, we
have UYY:: X1, \ X2)= = UY2 X1.1 X2.1) + UY2 : X1,2 \ X2,2). This implies that the PID of an isolated system should not depend on another isolated system.
The fundamental challenge of PID is estimating information-theorett measures when the size and dimensionality of the datasets are large (4). Optimizing over the pointwise MI quantities can only be obtained when the features are pointwise discrete or low-dimensional, and the number of optimization variables is exponential in the number of neurons (53]. Our first key insight is that the measurement of MI has closed-form analysis when the pairwise distributions are multivariate Gaussians, and we refer to this problem as Gaussian PID (GPID).
Definition A.2 (GPID). Let A be the set of joint distributions, where p(#1, y) and p(z2,y) pairwise Gaussian. The redundant, unique, and synergistic information are given by
R max (X1;; Y); 4EAG
(26)
U, = min I,(X;; Y|X;)))X2, DEAG
U2 = min YIX)),
(27)
SEAS
S = (X1, X2;
min SEAS
X2;111
(28)
where
4 = bg A q( () = p(..))) Vy E y, x, X, i [2]]
(29)
21

---

Definition A.3 (~e-PII [52)). Let p(11, 22,8) y) be joint Gaussian distribution. The redundant, unique, and synergistic information are given by
R max (x: REA,
(30)
U1 min (x ( (22), ,EA,
U/2 min min 1x(2;; gEdp
(31)
S = 12( X2;
min gEdp
(x;11
(32)
where
p {g is jointly Gaussian {(<;; v) - p( w), Vy E V, x, E X, i E [2]}
(33)
Connections between PID definitions: GPID exactly the PID problem when the pairwise marginals p(z1.y) and p(22, y) are known to be Gaussians. If the optimal q(21, 22, y) in PID Gaussian for some p(11, 22, y), then G identical to PID for that p(11, Z2, y). Venkatesh et al. [52] introduced PPD by directly placing the additional constraint that g 22, y) is Gaus- sian. In Section 3.1, we show that the joint Gaussian solution ~GPPID is also optimal in GPID, but this left as open question in [52].
Broadcast channel interpretation of GPID: Let two Gaussian marginals GPID have covariance Σx, Y and Σx,y, respectively. We can interpret GPID in the following two-user Gaussian broadcast channel:
X1 H2Y + 11
(34)
X2 = H2Y +
(35)
where H1, H2, n2 can be estimated from p(z1, y) and p(z2.v):
[x] Y TIT N(m Ex,r),
(36)
[X]] Y TJ ~ ((22, Ex,r),
(37)
H1 Σa 2xIV,
(38)
H2 ΣX2Y,
(39)
En; Σxi -H H
(40)
Σ Σx22 H2 H
(41)
Full algorithm of Thin-PID: We derived the objective and the projected gradient descent method in Theorem 3.3 and Proposition 3.4, respectively. The complete algorithm of Thin-PID is given as follows.
Algorithm 1 Thin-PID algorithm.
Require:
Channel matrix H [H] JT covariance Σy. Initialize: S = H, H. learning rate 1) a = 0.999, 3 = 0.9
while not converged do
Compute VL Dall (j)
[H H + Emjng) 1 2n up-off from Eq. (12-13) Iall (j+1)
N2
Update Doff
Dpyng using RProp
42
n2
oof (j) an)( VL E (j)
SVD(E (j+1)) U ddiag (d,) from Eq. (14)
Proj off
(2nt (1+1)) = U Udiag (min (max(),, 0), 1)) VT from Eq. (15)
Update 11(-1) n B 3-002
1o (0 sh (( ( == Sgn V L( ((EN(())
end while
return Doll Dng
22

---

A.1 Proof of Lemma 3.2
For any random vectors X1, X2, Y that follow the distribution q 22, 1/)
h (Y|x x X2) by (Y - 1,, (2)] X2) X2)
(42)
S hy (Y - [1
(43)
S hg (A Ey - X2.X2))
(44)
hi (8-
(45)
hi (81111
(46)
where for clarity we also use X1, X2, Y denote the random variables that follow the distribution ((11, 22, y): the second inequality is because Gaussian distributions maximize the differential entropy
with the same second moment. h (N(00, Ey-E(Y x,)x,)) denotes the differential entropy of Gaussian
vector with zero mean and covariance the same as Y IE(Y X1, X2), and the last two equalities are because of the joint Gaussian distribution.
A.2 Proof of Theorem 3.3
The optimization problem we need to solve in Gaussian PID is
S [x1, Y); min gEAs
(x1,,, X2; Y);
(47)
where
4 (q A :q is jointly Gaussian, ( v) p Tz: v), by E V. oi X, i [2]).
(48)
The random variables in the Gaussian PID interpreted by Gaussian broadcast channel can be written
Xi X₂
H H₂
Y
n1 12
(49)
where (0, E.). ni (0, En,). n2 ~ (0, E,,), and H = [H H is the channel matrix which can be directly estimated from Gaussian marginals p(11, y) and p(z2, y).
We firstly consider the objective function 1,, Y): The differential entropy of Gaussian random vector X ~ N(xx, Ex) is given by
MXX)
71 log(2T) + log [Ex| + 2"
(50)
2
where is the dimension of vector X. Therefore, the MI between two random vectors X and Y ~ [Ny, Ey) is given by
(X;Y) = (X) - (XIT)
(51)
1 Σxl
log
(52)
2
Therefore, the objective we need optimize in synergistic information S is
1
EE
I2(X1 X2:
log
(53)
Σx,x,rl
1
HEy HT + +mnnng
log 2
(54)
Eninal
where
Dn
Enjng =
yoll Dngn
(55)
23

---

The second equality is because Xx, = Ening and Σx,x, HEy H + +Emna using properties
of multivariate Gaussian distributions.
Next, we consider the constraints on Gaussian marginals g( y) = p(11. y). (x22 y) p(12, y). Note that p(21, y) and p(22,y) y) are already preserved by En, and Eng. Without loss of generality, we can assume _x Ln, = Lax, and Exy,Y Lin = Ldx, since we can always perform receiver side linear transformations individually whiten the X1 and X2 channels. Therefore, the only optimization variable in 1,,;, Y) is Yoll
Therefore, the optimization problem can be recast
HEy HT + Emina
minimize: L (20t log
(56)
subject to: En, 2nz = I,
Enina ビ
A.3 Proof of Proposition 3.4
Gradient. Let = HEy H + Empng- Assuming that G is positive definite, then we have
V do) G
(57)
Therefore, the gradient of the unconstrained objective function in Equation (11) with respect to Ening given by
Gl Dnnnal
VEn mg
log
G
(58)
Using the block matrix formulas, the gradient with respect Doft is thus given by
VL (Eat (_a(nn) [G-1-
E F-1 ming off
(59)
where the subscript "up-off" denotes the upper off-diagonal block matrix.
We firstly compute the inverse G = HEy H + Vinnas Define the block matrices
G₁1 G1
En Soff
HΣyH Ening =
Eming =
(60)
G
22
En
where
G1,1 = H,Ey H + En,,
(61)
G2, H₁Σ H + Enzy
(62)
G1 = H2 H + yofl -
(63)
Using the block matrix inverse formulas, the upper off-diagonal of G-1 is given by
GT. (G22 C.
(64)
Similarly, the upper of-diagonal of E is given by
offf (I (I- Doft off
nina 2
(65)
if 2n, and Dn, are identity matrices without loss of generality. Therefore, the gradient of the unconstrained objective is given by
V L (2 GT, (G2,2 - GT. + manng (1 (1- 2pyng )
(66)
24

---

Projection operator. The optimization variable Doff the matrix constrained by
Liing is an off dfi-onaa block of En,ng, which is
>a 122 I
Ening
(67)
Note that Ex; and ing can always be whitened by performing receiver-side linear transformations after we estimate H1, H2, My from the data. Therefore, the constraint on the projection we need to consider is
2n
0.
(68)
Pngni
By the Schur complement conditions for positive definiteness,
Engnz =
b ll S 1.
(69)
Therefore, we only need to show that the projection of D onto the spectral norm ball (2alf ||20|| lla s 1 is achieved by shrinking the singular values of 2inz via
Proj( = Udiag (min (max(A,, VT,
(70)
where 2ning To find the projection the spectral norm ball, we want solve
UAV is the SVD of E "nn"z"
min
Σ Doll P
(71)
s.t. E0|2 1.
(72)
Let Doff "nina = UAV with A := diag(Ar). We apply the same decomposition to Σ UAVT, where not necessarily diagonal. However, we can always set the off-diagonal blocks to zeros without increasing the Frobenius norm.
Write
O, where the diagonal matrix of and O is the pure off-diagonal matrix. Since the Frobenius inner product is the usual Euclidean one on entries, the diagonal and off- diagonll subspaces are orthogonal. Hence, we have
Σ
soff
|
II + 1OO1 > |D
(73)
For the feasibility the spectral norm ball, it is obvious because
[|diag max |aa| < ||
(74)
Now it is feasible restrict A be a diagonal matrix. Because the Frobenius norm is unitary-
invariant, we can simplify:
Doll Doin 11 = |
A E -X:)²
(75)
Thus, only need solve:
Σ
(A,
s.t. max À 1
(76)
Note that the SVD of real matrices always gives nonnegative eigenvalues. Therefore, X; and X should be non-negative, and the optimal solution is given by
A min(max 0), 1).
(77)
Then the projection operator is given by
Proj oof
+- Udiag (7x) VT.
(78)
25

---

A.4
Computational complexity
In this section, we discuss the computational complexity of different GPID algorithms. The state- of-the-art GPID algorithm is Tilde-PID [52], which is shown to be faster than other older baselines (MMI-PID B], 8-PID
(53).. We identify the difference between our work and Tilde-PID as follows:
(i) We compute the PID of a different objective function. Although the optimized variable is the same, Emang. the computation of the gradient in each iteration is significantly different. For Thin-PID, Gj11 in Eq. (12) is constant and does not need to be computed in each iteration. The other inverses of the matrix each iteration can be computed by solving a set of linear equations with variables min dx,, , dx,), while Tilde-PID requires solving linear equations with dominant variables of size dx, + dx,-
(ii) We use a different projection operator on Engng. work on the upper off-diagonal Sooff PRang since the diagonal blocks are identity matrices. The Thin-PII) requires SVD Mayn, which has (mmin(dx, dx, )o) complexity. However, Tilde-PID requires the eigenvalue decomposition in En,aa of size (dx, + dx,) x (dx, + dxx)), which has O(dxx, + +dx,)") complexity, and additional SVD in the projector with O(max(dx, dx,) )9) complexity.
n2- Although the constraint is Eming 2 O, we only
Table A.1: Complexity analysis of different GPID algorithms. The complexities of ED and SVD are cubic in the values shown in the table. Thin-PID achieves better complexity on any scale of computation.
ED
SVD
Lin-Eqn Solve
Mul
Thin-PID Tilde-PID
min(xx, (dx, dx.). max (dx,: dx,)
2 * min(dx, ,dx,) 2 (dx: + dx,)
4 min(dx,,,x, 8 (dx, + dx,)
dx, + dxa
B Latent Gaussian encoders and normalizing flows
When the marginal distributions p(11, y) and p(22, y) are not Gaussian, we learn a feature encoder transforming X2, into a latent space such that they are well-approximated by Gaussian marginal distributions, then using ThDD to compute PID values in the GPID problem.
Information informatiin--rrserving encoder: Let X1 (X,), X, = f2(X2), y = (Y) be three transfor- mations defined by three neural networks, respectively. Ideally, the transformations should satisfy the following conditions:
1. Transformations are invertible, so that the MI does not change.
2. p(ã,, f) and p(f2, ú) are well approximated by Gaussian distributions.
Flow-PID: This transformation can be performed by normalizing flows (39, whose invertible
bijections preserve information while bringing the joint distributions closer to Gaussians (7]. We established the theory of invariant total MI and PID under bijective mappings in Theorem 4.1 and Corollary 4.2. Our goal is to learn a Cartesian product of normalizing flows fi x fo x fy that preserves the total mutual information (z(f((X,) fx(x; fy (Y)) Ip(X1, X2; Y). For the constraint set on Ag, we proposed the regularization with the Gaussian marginal loss in Corollary 4.3 and Proposition 4.4.
B.1 Proof of Theorem 4.1
Lemma B.1 (7), Let : Ω - Th' be an absolutely continuous random vector, and let g RRn'' - T2
be an injective piecewise-smoott mapping with Jacobian J, satisfying n' and [et J) + almost everywhere. Let PDFs Pę and Pain exist. Then,
T(E:n) = T(g(E):))
(79)
By Lemma B.
f2())); Y) T(X1, Y);Y). Let (X1, X2) we only need to show (X1), f2(),
be the concatenation of (x() and f2(X2). It is obvious that concatenation is also bijective and invertible. Therefore, we have
(X
(X), Y) (,, x2): Y) 1(X,, where the second equality is because 9(X1, X2) is invertible and bijective.
(80)
26

---

Algorithm 2 Flow-PID algorithm
Require: Multimodal dataset X1 E X2 LT, Y YE)"
Initialize Cartesian flow networks fi x f2 x fy.
while not converged do for sampled batch X, E E 2 E 2 E ym do
Transform latent feature: X, = 1)) X₂ = fx()),, Y = fY().
Compute the marginal loss of (X,, Y) and (X2, Y) using Eq. (16). Compute the sum two marginal losses using Eq. (17).
Perform gradient step on the loss.
end for end while Calculate H1, H2, My from Eq. (10) using (X,, (x, Y) Perform Thin-PID from Eq. (11-15) using H1, H2, Mi return PID values: R, U1 U2.
B.2 Proof of Corollary 4.2
Let X1 = (X1). X2 = f2((2), and y = fr where f1, fr are invertible bijective mappings. Define the set of distributions
Ap { q 0) p(fi'(#1).)) fy (0)) ((02, - p(ff (t)))
Since fI, f2, and fr are bijective mappings. this is well-defined. Moreover, if (X1, X2, Y) are jointly distributed according to p, then (X1, X2, P) are jointly distributed according to
p 2. 1) p(f (.1), f2 (#2). ( (0)).
We now show that there is a bijection between the sets 4, and Ag. Given any E Ap. define i E A by
((ă 22, 1) = x q() (22), (0)).
is straightforward to verify that & satisfies the required marginal constraints in Ap, as the marginals transform correctly under invertible mappings. Conversely, given any I E Ag, we can recover the corresponding E Ap via the inverse transformations. Thus, the mapping between Ap and As is bijection.
We can now prove the main result. By Theorem 4.1, we have
(X, X2; Y) = Tp (x,,, X2:
(81)
The PID solution in the original coordinates is given by minimizing the left-hand side of Equation (8) over Ap, while the PID solution in the transformed coordinates minimizes the right- side over Ap. Because of the bijection between Ap and Ag. the optimization problems equivalent, and the minimum values are the same. Therefore, the synergy values both sets of coordinates are the same. We similarly conclude that all of the PID values equal, desired.
B.3 Proof of Corollary 4.3
Since q(a, y) is the PDF defined on the same space as p(æ,, y), we have
p(z,y) [p(x))(u)
q v)
p(z,y)
g())g())
YY; E log
E Eoog
(82)
(())(())
q(x,v) p(*)p())
"(z,, v)
[(()) +E log
(1v)
I, (X;Y) E log
+ log
(83)
((x, y)
[(xx)
p())
=1 KL
v)) KL (p(x)) @ p(v)ll(()) @ ((v))
(84)
Firstly, since KI. (p(x) 3 p(u) (x))) @ (()) 2 0. we have
- - < KL (p(x.))))
(85)
27

---

Secondly, given the monotonicity of the KL divergence, we have KL (p(z, v)ll((1,))) KL (p(=))a((zz) and KL (p(z, (z v)) 2 KL (p(v))la(v)) Therefore, we have
>
4x(XY)- - 4x;; 2 KL (p(z, ,2 8) - 2 KL (p(a, ,)llo(a,))) v))
(86)
KL (p(2, v) w))
(87)
Combining the two directions, we have
14((X; - (X; S KL v)la(,z) v)).
(88)
B.4 Proof of Proposition 4.4
Let f(x,)) = fx x (Y) be a Cartesian normalizing flow. Using the change of variables formula, we have
logp(a, y) p(ž 9) + log det 0f y) a(x, y)
(89)
For the Cartesian flow f = fx x fy, the Jacobian is block-diagonal. Therefore, we have log del 0f y)
log det (fx(2) + log del 0fr(y) 8y
(90)
0(x,1) 9)
Ox
Given a dataset D = ((x(") (0) yb))),, = (X1) x fr()) and (2( x fY( also minimizes KL (px,y (fi x fr II Ex,y))
22
N}. note that maximizing the likelihood of
and KL (px,y (fz fy (p2, Ex,r)). Therefore, the Gaussian marginal regularizer is equivalent to maximizing the log log-likelihood of ((x(") ) such that x².')) = f: (x())) y
g (g()), where (a!()) g(0) are sampled from variational Gaussian distribution N(H, Ex,y). Therefore,
the maximum-likelihow estimates of p(x1, y) and p(æ22 y) to regularize the supremum in Gaussian marginals
Cxrax( ,10)) Ex(m go0) log det (z1 log det Qfr(v) 8z;
(91)
where CN ($(b)) (o0) is the likelihood of the latent multivariate Gaussian variables
Ni (xX
Σ
1 {
Σ (r((())) N
(92)
N
F=1
(((((()) Mxx )
(m(aa)) 18x,
T
Σx, = N Σ
fy(1)))
HV/
(rr()()) -M
(93)
(1) (00) (o
-
1 log [xx,y]
2 (1.50)
1 (1)
14)
x
( 2
- mi)
(94)
To simultaneously ensure that both p(g 9) and p(12, 9) are approximately Gaussian, we train the flow using the loss function
Lnow Cnow((X,, X2, Y)) = Lxr(((X,, K)J) + Lx(((X2, Y))),
(95)
where Cx((, Y))) is estimated by Monte Carlo sampling
N Σ
1
Cx(f(X,,))) ~
(96)
N
C Experimental details for synthetic datasets
We provide the experimental details of Section data generation, feature processing, model architec- ture, and computing resources.
28

---

C.1 Canonical Gaussian examples
ID broadcast channel: We illustrated results on canonical ID Gaussian examples in Figure and corroborate the correctness of Thin-PID by designing three cases:
Unique and redundant information (left):
Y ~~(00,))
(97)
X1 Y + n1,
B1 (0,
n1 1T Y,
(98)
X2 X1 n2,
n2 ~ ~N(0,0²²,
n2 Il. X1.
(99)
Unique and synergistic information (middle):
Y ~ 10,
(100)
X1 Y + n1,
12, ~ 12, ~N(0 Conr(n1, =
(21, Tl Y,
(101)
= = 122
(102)
Redundant and synergistic information (right):
Y ~ (0, 1) X; = Y 11;
(103)
R2, ~ -N(0,,) 1), Corr(n1, n2) p-
(21, 11 Y,
(104)
X₂ = Y + n22
(105)
Additional experiments on GPID at higher dimensionality: We next design additional examples to validate Thin-PID at higher dimensionality, which are also benchmarks [52].
Case I: Let X1,1 = aYi + nI, X2, = Yi + 1221, X1.2 Y22 + n1,,, X2,2 = 3Y22 + 12,2, where Yi, M1,i, 12, i. (O 1), 1,2. Here. (X1,1, X2,1, ,, Yi) is independent of (X1,2, 22,2 Y2). Using the additive property, we are able to aggregate the PID values derived from their separate decompositions, each of which associated with known ground truth, as determined by the MMI-PID, considering Y1 and X2 as scalars.
Case 2: Let Y and X2 be the same in Case 1. Let H2 R())Y, where H1 is diagonal matrix with diagonal entries 3 and 1, and R(0) () is a 2 x 2 rotation matrix that rotates Y at an angle 8. When 8 0, X1 has higher gain for Y1 and X₂ has higher gain for Y2. When @ increases to T 7/2, X, and X2 have equal gains for both Y, and Y22 (barring difference in sign). Since (X1,1, X2,1, Yi) is not independent of (X1,2, X2,2, Y2) for all 0, we only know the ground truth at the endpoints.
The results Cases 1 and 2 are shown Figure The left and right subfigures show the PID values of different GPID algorithms, and the middle one shows the absolute error between each PID algorithm and the ground truth. We observe that Thin-PID achieves the best accuracy with the error < 10 while the absolute error of PiDde-PID is > 10-8.
Results.
We test the stability of Thin-PID at higher dimensionality of d= dx, = dx, dy. We repeat the process of Case and use additive property to concatenate X2, Therefore, the PID values should be doubled as we double the dimension
Case 3:
Results. The results of GPID with higher dimensionality are shown in Figure C. The PID values of Thin-PID match ground truth by doubling in value when the dimension of (X 1, X2, Y) doubles.
Accuracy, Figure C.2 shows the absolute errors in PID values using the Thin-PID algorithm. is observed that the absolute error of the Thin-PID algorithm remains constrained below around 10 even as the dimensionality extends to 1024. However, the absolute error of Tilde-PID in [52) increases with increasing dimension and will exceed 10-5 when d > 1024. Therefore, Thin-PID is not only more efficient but also more accurate than Tilde-PID.
C.2 Synthetic non-Gaussian examples
Multivariate Gaussian with invertible nonlinear transformation: We start with multivariate Gaussian distribution (0, Ex,x,r) The pointwise dataset {- , x. y " is sampled from the joint Gaussian distribution. The "exact" truth of PID is obtained by calculating Hi, H2, and Σy from E X2y directly, then performing Thin-PID. To show the necessity Flow-PID in non-
(3) Gaussian cases, we transform x 22 and y into non- Gaussian distributions using three nonlinear
29

---

Stability of the Thin-PID over increasing dimensionality
107; (X1, X2
U1
U2
Partial Information (bits)
512
256
128
0.25
Dimension Y,
Figure C.I: GPID results when the dimension d dx, dx, dy increases. Different shadings represent different values of gain in (a) in Case 1. The PID values of Thin-PID doubles every time when d doubles.
Absolute error in the Thin-PID over increasing dimensionality
(X1, X2
Abs. errorin in PID value (bits)
10-1
10-4
Figure C.2: Absolute errors of Thin-PID from Case 3.
invertible transformations कं (x1) of (X1,X2, Y) remains the same as that of (X1, X2, Y), but they are no longer pairwise Gaussian after the transformation.
2/2 and VV. According Theorem 4.1, the MI
Flow-PID learns the flow-based latent Gaussian encoder first, and then performs Thin-PID in the learned Gaussian marginal distributions. For Tilde-PID, the PID values computed by directly estimating the covariance
Non-Gaussian samples (#1 T2. V7 We did not include BATCH as baseline in this case because BATCH requires feature clustering that is not feasible with continuous target y. Therefore, BATCH cannot be generalized to regression or multitask scenarios where the target value is continuous.
The results are shown in Table 2. Flow-PID aligns with the exact truth in the relative PID values, whereas Tilde-PID fails with a distorted nature and degree of interactions.
Table C.I: Full results on non-Gaussian multi-dimennional examples.
Dim
(2 2,
2)
[10, 5,
(30,
10,2)
PID
R
U1
U2
R
U1
U2
$
U1
U2
5
Tilde-PID
0.18
0.29
0.76
0.02
0.84
0
1.19
0.16
1.09
0.88
0.19
Flow-PID
0.62
0.91
0.50
0.11
2.36
0.32
0.19
0.45
2.18
1.13
0
0.17
Truth
0.79
1.46
0.58
0.18
2.96
0.54
0.26
0.58
2.92
2.18
0
0.25
Dim
(100,
60,
(256,
256, 2)
(512, 512, 2)
PID
R
U1
U2
S
R
U1
U2
S
R
U1
U2
S
PID
1.48
0
1,97
0.13
1.94
0
2.31
0.10
1.09
0.88
0.19
Flow-PID
4.34
0.36
0.25
3.79
0.15
0
0.36
0.48
0.98
0.47
0.11
Truth
5.71
1.01
0.57
7.85
0.14
0.05
0.90
0.81
1.45
0.58
0.19
30

---

Specialized interactions with discrete targets: We followed the settings of the synthetic generative model in [27] Let 21, 2c R:50 be fixed set of latent variables from (0, with 0.5.
=1; 22, Re represent latent concepts for the unique information of X1, the unique information of X2 and the common information, respectively. The concatenated variables [21, are transformed into high-dimensional I1 E R R.000 using a fixed weight matrix Ti E R 1100x1000 and also [22, =e] to 22 through T2. The discrete label y is generated by a function of (21, 22, 22, ze). By assigning different
weights, the label can depend on (1) only which reflects pure redundancy, (2) only 2 or 22. which reflects pure uniqueness # or 22, (3) the concatenation of [21, ø2], which reflects pure synergy. More specifically,
X1 Ti [2l,
(106)
X₂ = T2 [22, Zel:
(107)
Ei S [[=1, E2, =cl)s
Y = sigmoid
2 0.5
(108)
where f is fixed nonlinear transformation with dropout rate p 0.1.
As shown in Table 3, generate 10 synthetic datasets, including four specialized datasets {DD: Du, Du,- Ds} with pure redundancy, uniqueness, or synergy. The rest are mixed datasets with y generated from (21, 22, ze) of different weights.
The ground-truth interactions are estimated by the test performance of multimodal models. The test accuracy Pace is converted to the MI between the inputs and the label using the bound:
1X1,, Y) I log Picce + HY)..
(109)
The information in each interaction is computed by dividing the total MI by the interactions involved in the data generation process: if the total MI is 0.6 bits and the label depends on half of the common information between modalities and half from the unique information in æ1, then the ground truth R = 0.3 and U₁= 0.3.
C.3 Compute configuration and code Availability.
All experiments with synthetic datasets are performed on Linux machine, equipped with 48GB RAM and NVIDIA GeForce RTX 4080.
The code used to reproduce results synthetic datasets is included in ZIP file part of the supplementary material.
BATCH and Flow-PID require the training of neural networks (NNs). Before training NNs, we preprocess the feature by standardizing the features and randomly shuffling the mini-batches. BATCH follows the same training recipe as in (27), and the NN architectures used in Flow-PID are given in Table
Table C.2: The NN architectures for Flow-PID.
NN
Architecture
GLOW
x1:
4 (5) splits, GLOW blocks between splits, 16 hidden channels in each block, leaky constant = 0.01 Orthogonal linear layer
x1:
x3 :
ReaNVeR.ffiieCongggggooMMMMM... 64, d)), Permute-swap)
RealNVP x6:
RelllWWrfffiicCoCggggggeeee///dd 64, d)), Permute-swap)
D Experimental details for real-world datasets
We provide the experimental details of Section 6 and introduce the real-world datasets from Multi- Bench 30 For the BATCH baseline, we follow the same experimental settings and training recipes in 27]. We release the data and code in an anonymous ZIP file attached with the supplementary materials.
31

---

Table C.3: Training recipe.
Parameter
Value
Optimizer
Adam
Initial learning rate
le-4
Scheduler
Aonealnnne
Weight decay
le-4
Data augmentation
Normalization
Batch size
128
D.1 Quantifying real-world datasets
MultiBench datasets: We use a collection of real-world multimodal datasets in MultiBench [30], which spans 10 diverse modalities (images, video, audio, text, temesserie), 15 prediction tasks (humor, sentiment, emotions, mortality rate, ICD-9 codes, image-captions, human activities, digits, robot pose, object pose, robot contact, and design interfaces). and 5 research areas (affective computing. healthcare, multimedia, robotics, and HCT). These datasets are designed to test combination of feature learning and arbitrarily complex interactions under different multimodal fusion models the real world.
Table D.I: MultiBench datasets used for quantifying interactions between diverse modalities, tasks, and research areas.
Datasets
Modalities
Size
Prediction task
Research areas
AV-MNIST 54
(image, audio) 60, 000
logits
Multimedia
MOSEI 160
(texi, video, audio) 22, 777
sentiment, emotions Affective Computing
UR-FUNNY 10
(text, video, audio) 514
humor
A Affective Computing
MUSTARD g
(text, video, audio)
690
sarcasm
Affective Computing
Real-world datasets with task-driven and causally relevant interactions: we conducted addi- tional experiments with real-world data to demonstrate method in additional application areas and with unexplored modalities.
TCGA-BRCA is multimodal dataset created to help study the causes and progression of breast cancer. We quantified the PID predicting the breast cancer stage from protein expression and microRNA expression. Flow-PID identified strong uniqueness for the modality of microRNA expression as well as moderate amounts of redundancy and synergy. These results are also in line with modern research, which suggests microRNA changes as a direct result of cancer progression.
VQA (Visual Question Answering) is multimodal dataset consisting of 10,000 images with corresponding yes/no questions and their answers. Under the paradigm of using the image and question to predict the answer, would naturally expect high synergy since the image and question complement each other. Our method, Flow-PID, recovers exactly this we find synergy the dominant interaction between the modalities.
The detailed experimental results are shown in Table D.2 Both of these experiments demonstrate further applications and additional modalities to validate our method.
Table D.2: Additional experimental results Flow-PID on real datasets with task-driven interactions.
Expected
Dataset
dim-X,
dim-X;
R
U1
U2
S
interactions
VQA2.0
768
1000
0.22
0.26
0.0
0.76
TCGA
487
1881
0.41
0.0
1.07
0.34
U2
Information- feature extractor: For datasets with available modality features (images, text), we use the end-to-end PID estimator (Flow-PID, BATCH). For other modalities (audio, time- series), we first use pretrained encoders to extract features. To preserve the MI of the extracted
32

---

features, we add a contrastive loss to the encoder Enc(X) using MINE [4]
N
N
M To (Enc(x(0))) loog V(0) = N
e To (Fno(a (100)))))))))))))))
(110)
N
D.2
Model selection
Given new dataset D, we interested in whether PID estimators beneficial in recommending the most appropriate model without training all models from scratch. We hypothesize that the model with the best performance will likely perform well the dataset most analogous to it, given the similarity in the interactions. Therefore, we select the most similar pre-trained dataset DD from a of base data D (the 10 synthetic data sets presented in Table 3) by measuring the difference in the normalized PID values:
Setup:
D = arg min s(D, D') = arg min D
E
IID - -
(111)
Dr
JE(R,U,
The quality of the model selection is evaluated by the percentage of the performance of the selected model with respect to the performance of the truly best-performing model on D:
Performance(ff f*) ff
Acc(f)/Accl)
(112)
Choices of multimodal models: We implement 10 multimodal fusion models in synthetic datasets and MultiBench datasets.
1. ADDTIIVE: Suitable unimodal models are first applied to each modality before aggregating
the outputs using additive average: y (x) f2(x2))
20
2. AGREE: Add another regularizer as prediction agreement (+A(fI
12,)))²
[TO
3. ALIGN; Add feature alignment (+=Asim(x1,x)) X2) like contrastive learning (11).
4. ELEM: Element-wise interactions for static interactions (i.e., without trainable interaction
parameters): y = f(1 x2)
23
5. TENSOR: Outer-product interactions (i.e., higher-order tensors): y (x
14 59
29]33) 33].
6. MI: Dynamic interactions with learnable weights include multiplicative interactions W:
= f(x, Wx,) 24
7. MULT-Dynamic interactions with learnable weights through cross-modal self-attention,
which is used in multimodal transformers: = f(ootmaax (x| X2 )x1) [51 49 (7).
8. LOWER: Lower-order terms in higher-order interactions to capture unique information B3
59]
9. REC: Reconstruction objectives to encourage maximization of unique information (i.e.. adding an objective Live [ (Zmm) - X1 [g Zmm)
X2 where 91, are auxiliary
decoders mapping 7m to each raw input modality [45] 50, 56
10. EF (early fusion): Concatenating data at the earliest input level, essentially treating it as single modality, and defining a suitable prediction model y f[ (31.
Architectures of multimodal models in different datasets: To make fair comparison between BATCH and Flow-PID, we adopt the same architecture of feature encoder, modality fusion, and training hyperparameters in [27]. For datasets with available modality features, we use data with standard pre proccescsssngg as the input of multimodal models. For other datasets without available modality features FUR-FUNNY MUStARD, MOSED, we first use pretrained encoders to extract features, which are also provided along with those datasets. The NN architectures and training hyperparameters are provided below.
33

---

Table D.3: NN architectures for multi-modal fusion models. The input dimension is decided by extracting d-dimensional features from the data. For datasets with available modality features, feature dim d is identical to the data dim. For other datasets without available modality features, we first use pretrained encoders to obtain features with output dim
Component
Model
Parameter
Value
Identity
Encoder
Feature dim
[d. d]
Linear
Hidden dim
512
Feature dim
d. d
Decoder
Linear
Hidden dim
512
Concat
Elem
Output dim
512
[2
Fusion
Output dim
512
LOWER 33
rank
32
Embed dim
512
MULT 49
Num heads
Identity
Hidden size
512
Classification head
2-Layer MLP
Activation
LeakyReLUU(.2)
Dropout
Loss
Cross Entropy
EF& & ADDITIVE & ELEM & TENSOR MI MULT & LOWER
Batch size
128
Num epochs
100
Optimizer/Leaminn rate
Adam/0.0000
Cross Entropy
Loss
+ Agree/Align Weight
Batch size
128
AGREE & ALIGN
Num epochs
100
Optimizer/I earning rate
Adam/0.0001
Cross Entropy Weight
2.0
Training
Agree/Align Weight
1.0
Cross Entropy
Loss
+ Reconstruction (MSE)
Batch size
128
Num epochs
100
Optimizer
Adam
REC 50
Learning
0.0001
Recon Loss Modality Weight
[1,
Cross Entropy Weight
2.0
MLP [512, 256, 256
Intermediate Modules
MLP [112, 256, 256
34

---

Table D.4: Table of hyperparameters for affective computing datasets.
Component
Model
Parameter
Value
Identity
Encoder
Input size
5, 20, 35, 74, 300, 704 32, 64, 128, 512, 1024 5. 20, 35, 74, 300, 704
GRU
Hidden dim
Input size
Decoder
GRU
Hidden
33, 64, 128, 512, 1024
Concat
Flem
Output dim
[400, 512)
MI 24
Fusion
Tensor Fusion 59
Output dim
512
Embed dim
40
MULT
49
Num heads
8
Identity
Hidden size
512
Classification head
2-Layer MLP
Activation Dropout
LeakyReL.(0.2)
0.1
Loss
L1 Loss
EF ADDITIVE ELEM TENSOR MI MULT & LOWER
Batch size
Num epochs
40
Letrnnzn rate
Adam/0.0001
LI Loss
Loss
Agree/Align Weight 32
Batch size
AGREE & ALIGN
Num epochs
30
Optimizer/lLcampee rate
Adam/0.0001
Agree/Align Weight
0.1
Training
Loss
Loss
+
Reconstruction (MSE)
Batch size
128
Num epochs Optimizer
50
Adam
REC 50
Learning rate
0.001
Recon Loss Modality Weight
[1,
MI.P [600, 300, 300
Intermediate Modules
MLP 600, 300, 300
Table D.5: Table of hyperparameters for AV-MNIST encoders.
Component
Model
Parameter
Value
Filter Sizes
[5, 3,
Num Filters
66, 12, 24, 48]
Image Encoder
LeNet-3
Filter Strides Filter Paddings
1, /[2, 1,
Max Pooling
[2, 2, 2,
Filter Sizes
[4, 4,
Image Decoder
DeLeNet-3
Num Filters
[24, 12, 6]
Filter Strides Filter Paddings
2,, 1,
Filter Sizes
3,3,,,
Num Filters
12, 192)
Audio Encoder
LeNet-S
Filter Strides Filter Paddings
1, 1,1,1,1,1
Max Pooling
2,
Filter Sizes
4,4,4,4,4,5
Audio Decoder
DeLeNet-5
Num Filters
[96, [96,48, 48, 24, 12, 6,
Filter Strides/ Filter Paddings
2,,2, 1,, 1,1,,
35

---

Table D.6: Table of hyperparameters for ENRICO dataset in the HCI domain.
Model
Parameter
Value
Unimodal
Hidden dim
Hidden dim
32
MI-Matrix
Input dims
16, 16
Hidden dim
MI
Input dims
16, 16
Hidden dim
32
Lower
33
Input dims
16, 16
Rank
Loss
Class-weighted Cross Entropy
Batch size
32
Activation
ReLU
Training
Dropout
Optimizer
Adam
Learning Rate
10-5
Num epochs
30
36


Reviews:
[{"review_id": "wdfWi0NxgO", "reviewer": "Reviewer_SRhn", "summary": "This paper presents a novel framework for Partial Information Decomposition (PID) that efficiently quantifies information interactions in multimodal data. The authors introduce **Thin-PID**, a gradient-based algorithm that achieves 10× speedup for Gaussian PID computation, and **Flow-PID**, which uses normalizing flows to transform arbitrary distributions into latent Gaussian spaces while preserving information content. By proving the optimality of joint Gaussian solutions and developing information-preserving encoders, this work overcomes the computational bottleneck of existing PID methods for high-dimensional continuous data. Extensive experiments on synthetic and real-world multimodal benchmarks demonstrate superior accuracy and efficiency compared to state-of-the-art baselines.", "questions": "Given that baseline values ​​of PID in real datasets are unknown, how can we rigorously verify that Flow-PID produces meaningful and accurate decompositions, rather than just computationally convenient decompositions? The model selection task only provides indirect validation - can you propose more direct evaluation metrics or synthetic-to-real transfer experiments to better demonstrate the correctness of your PID estimates on complex real-world data?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The authors provide rigorous mathematical proofs, particularly establishing the optimality of joint Gaussian solutions for GPID, which resolves a previously open theoretical question and ensures their approach doesn't sacrifice accuracy for efficiency.\n2. Thin-PID achieves over 10× speedup compared to state-of-the-art methods by cleverly reducing the optimization problem to a lower-dimensional space, making PID computation feasible for high-dimensional real-world applications.\n3. Flow-PID ingeniously uses normalizing flows as \"information-preserving translators\" to handle non-Gaussian data, extending the benefits of efficient Gaussian computation to arbitrary distributions without information loss.\n4. The paper demonstrates effectiveness across multiple scenarios - from synthetic data with known ground truth to diverse real-world multimodal benchmarks, showing both accuracy improvements and practical utility in model selection tasks.\n\n**Weaknesses**\n\n1. Although the authors demonstrate a 10× speedup on synthetic data, there is a lack of detailed computational time comparisons on real multimodal datasets, especially when the data dimensionality is really high.\n2. The real data experiments are indeed insufficient - they are only tested on a few MultiBench datasets and lack verification in a wider range of real application scenarios.\n3. I think the training complexity of Flow-PID is underestimated, the training of the normalized flow itself can be expensive and unstable, and the authors do not fully discuss this overhead, which may offset the speed advantage of Thin-PID.\n4. The limitations of the Gaussian approximation assumption in the article are not discussed enough. Although the normalized flow is information-preserving in theory, there is a lack of in-depth analysis of how the approximation error in actual training affects the final PID estimation.\n5. I look at the MultiBench datasets and find that many of them may not be truly \"high-dimensional\". For example, the feature dimensions after being processed by the pre-trained encoder may have been reduced, which weakens the persuasiveness of the \"high-dimensional advantage\".", "was_revised": true, "final_justification": "My issue has been largely resolved, so I'm considering raising my rating.", "rebuttal": "We sincerely thank the reviewer for taking the time and effort to provide thoughtful reviews and insightful comments. We appreciate that you consider our work as a \"**novel framework**\" and find our work \"**mathematically rigorous**\" with \"**demonstrated effectiveness across multiple scenarios**\". Below we will address your concerns point by point. \n\n> **[R1]** Although the authors demonstrate a 10× speedup on synthetic data, there is a lack of detailed computational time comparisons on real multimodal datasets.\n\nThank you for this valuable feedback. Below we clarify the efficiency of our method compared with existing methods in detail.\n- **Reduced computational time results from the dimension, not the feature itself**: Thin-PID has less computational complexity of $O(\\min(d_{X_1}, d_{X_2}))$, which is much less than $O((d_{X_1}+d_{X_2}))$ in Tilde-PID. *Figure 3* corroborates this improvement by showing that the empirical execution time of Thin-PID is around $10\\times$ faster than Tilde-PID if they have the same input dimensions. Therefore, **Thin-PID still outperforms Tilde-PID in real datasets if they have the same feature dimension**. For example, when the input dimension is 1000, the execution time of Thin-PID is only 20s, while Tilde-PID takes more than 200s.\n- **Thin-PID is also more stable and accurate** than Tilde-PID. If the dimension of the input modality is larger than 1000, Tilde-PID struggles to converge and can be incomputable, while our proposed Thin-PID can handle dimensions more than 1000. As shown in *Figure C.1* and *C.2* (*Appendix C*), the accuracy of Thin-PID is better than Tilde-PID, especially when the dimension is high.\n\n> **[R2]** The real data experiments are indeed insufficient - they are only tested on a few MultiBench datasets and lack verification in a wider range of real application scenarios.\n\nThank you for raising this concern. We selected the MultiBench benchmark due to its diversity in modality combinations and tasks, which we believe **cover a representative spectrum of multimodal learning scenarios and highlight the strength of our model**. \n\n**The modalities of the chosen datasets are diverse**, including text, vision, audio, physiology, et al. The dimension of the latent Gaussian features in UR-FUNNY, MUStARD, and MOSEI is 512, which is much higher than $53\\pm 16$ PCA components used in Tilde-PID. To further support the applicability of Flow-PID in diverse scenarios, **we provide 2 additional experiments** on TCGA (Cancer genome atlas) and VQA2.0 (visual question answering) with causally relevant interactions. \n\n- **TCGA-BRCA**: we quantified the PID of predicting the breast cancer stage from protein expression and microRNA expression. Flow-PID **identified strong uniqueness for the modality of microRNA expression** as well as moderate amounts of redundancy and synergy. These results are also **in line with modern research** which suggests microRNA changes as a direct result of cancer progression.\n- **VQA**: consists of 10,000 images with corresponding yes/no questions and their answers. Under the paradigm of using the image and question to predict the answer, one would naturally **expect high synergy since the image and question complement each other**. Our method, **Flow-PID**, **recovers exactly this** -- we find synergy is the dominant interaction between the modalities. \n\nThe detailed results are shown in the table below. Both experiments demonstrate further applications and additional modalities to validate our method. \n\n| Dataset | dim-X1 | dim-X2 | R | U1 | U2 | S | Expected interactions |\n|------|-----|-----|-----|-----|-----|-----|-----|\n| TCGA | 487 | 1881 | 0.41 | 0.0 | **1.07** | 0.34 | U2 |\n| VQA2.0 | 768 | 1000 | 0.22 | 0.26 | 0.0 | **0.76** | S |\n\n> **[R3]** I think the training complexity of Flow-PID is underestimated, the training of the normalized flow itself can be expensive and unstable, and the authors do not fully discuss this overhead, which may offset the speed advantage of Thin-PID.\n\nThank you for raising this important point, which is closely related to our motivations. We agree that the training cost of normalizing flows could be significant in general. However, this will NOT offset the speed advantage of Thin-PID. On the contrary, **the very fast speed of Thin-PID (which is less than 20s with dimension 1000) motivates us to train a MI-preserved encoder** (typically, a normalizing flow) to transform the multi-modalities into latent Gaussian distributions so that PID can be solved efficiently in continuous and high-dimensional space.\n\n- **An MI-preserved encoder is necessary** in any case if you want to solve PID in high-dimensional space. The SOTA discrete PID estimator, BATCH, requires to learn unimodal encoders $p(y|x_i)$ first to estimate the marginal distribution of $p(x_i,y)$. On the other hand, GPID can be solved in a continuous space, but it restricts Gaussian marginals of $p(x_i,y)$. Hence, **there's no free lunch** if we want to solve PID in a continuous, high-dimensional, and non-Gaussian space. \n\n- **Flow-PID can reuse pretrained encoders**, while existing PID algorithms (e.g., BATCH) requires additional training to estimate $p(x_i)$. Compared to the post-training of BATCH, the optimization of Thin-PID is much more efficient.\n\n- Modern normalizing flows (e.g., RealNVP, Glow) have been shown to approximate a wide class of continuous densities with sufficient representational power. We view Flow-PID as a step forward toward tractable, expressive, and general-purpose PID estimation. Future work can explore more advanced generative models (e.g., latent diffusion models, transformers) to produce MI-preserved encoders in multimodal learning.\n\n**We put this as one of the limitations in Section 7**. But this is the best way in our knowledge that allows us to estimate PID in a generalized continuous and non-Gaussian space, which is shown to be effective and efficient. \n\n> **[R4]** The limitations of the Gaussian approximation assumption in the article are not discussed enough.\n\nThank you for raising this point. We disagree with question's premise. Although the flow-based approximation error would impact PID estimates, the pairwise Gaussian assumption is NOT a significant limitation. \n-  **Flow-PID is designed to relax pairwise Gaussianity assumptions.** The Gaussian assumption is only used in GPID methods to efficiently estimate PID with continuous Gaussian distributions. Flow-PID was specifically introduced to **move beyond that assumption** by learning flexible transformations via normalizing flows. This enables more accurate modeling of non-Gaussian, complex multimodal dependencies.\n- **Approximation error is inevitable, but empirically tolerable.** Our empirical results show that **PID estimates are robust to moderate approximation error**. We performed internal ablations measuring the divergence between the flow-transformed latent distribution and the target Gaussian, and observed that even **imperfect transformations led to consistent, interpretable decompositions** across synthetic and real-world tasks.\n- The issue of approximation error is challenging across all PID estimators. BATCH approximates unnormalized pointwise $p(x_1, x_2, y)$ by a NN encoder and Sinkhorn's algorithm, which introduces inevitable and heuristic approximation error. However, Thin-PID allows us to theoretically analyze the impact of the approximation error on Gaussian channel $H_1$ and $H_2$.\n\nIn the revised version, we will add a subsection explicitly discussing how approximation error in the learned flow affects downstream Thin-PID estimation. Thank you again for raising this interesting point. \n\n> [R5] I look at the MultiBench datasets and find that many of them may not be truly \"high-dimensional\". \n\nWe thank the reviewer for this thoughtful observation. The collection of multimodal datasets used in this work is selective, which spans diverse modalities with a wide range of feature dimensions. For example, UR-FUNNY, MUStARD, and MOSEI involve with text, video, and audio as their input modalities, each characterized by a pre-trained encoder of dimension 512. We believe 512 is good to show the capability of Flow-PID to deal with high dimensional data since Tilde-PID uses PCA components of dimension $53\\pm 16$. As shown in R2, we provide two additional experiments on VQA2.0 and TCGA, which further support the applicability of Flow-PID with over 1000 feature dimension. \n\n> Question: Given that baseline values of PID in real datasets are unknown, how can we rigorously verify that Flow-PID produces meaningful and accurate decompositions, rather than just computationally convenient decompositions? \n\nWe thank the reviewer for this important question. Evaluating PID estimates on real-world datasets is inherently challenging, since ground-truth decompositions are not observable. To address this, we respond in three parts:\n- We first want to emphasize that **we have a large selection of different synthetic data: Gaussian and non-Gaussian, discrete and continuous, low and high dimensional modalities, and different specialized interactions**, where the ground truth of PID is known. Our proposed model outperforms the SOTA baselines in all these synthetic cases, which partially shows its generalized capability on real datasets.\n- Indirect validation through functional utility: Flow-PID consistently attributes higher unique information to modalities that are empirically more predictive or causally relevant. This provides evidence of interpretive alignment even without ground-truth PID labels. Additional experimental results in [R2] also verify this.\n- Synthetic-to-real transfer: Given that fact that Flow-PID performs well on a large collection of synthetic datasets, good model selection results infer the appropriate PID estimates on complex real-world data."}, {"review_id": "tXvQGpXR1E", "reviewer": "Reviewer_Qrhw", "summary": "The paper proposes a new framework Flow-PID along with a new PID optimization objective (Thin-PID) to estimate the interactions between modalities in multimodal learning. These modules come along with theoretical analysis on the complexity of the proposed algorithm and the optimality of Gaussian joint distribution restriction. The authors provide empirical evaluation on both synthetic data and real-world multimodal data, comparing the proposal with other PID estimation approaches.", "questions": "I’m wondering about the originality and novelty of the paper. Despite soundness and strong foundation, the proposal seems to be a combination of existing solutions for related problems without significant improvements.", "limitations": "See weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n- The idea of transforming to latent distributions using normalizing flow, then optimizing to estimate PID is quite interesting.\n- The authors provide formal definition of the problem and explanation of essential derivation steps.\n- The proposal is built upon strong theoretical works, without loss of optimality guaranteed.\n### Weaknesses:\n- Major\n   - The overall flow of the proposal, while being sound, is presented in a disjointed style. The reviewer assumes that the framework can be trained end-to-end, since the gradient of the Thin-PID can be estimated directly (Proposition 3.4), while Sec.4 and Fig.1 likely indicate the separated training (2 phases). If so, why is end-to-end training not possible ? If not, why is end-to-end training better than separated training ?\n   - The Thin-PID, one of the main contributions, is directly built upon Tilde-PID[1], where the additional points are discussion on validity of Gaussian assumptions[1] and a different way of computing gradients. While both of them have strong foundations and some impacts, the reviewer does not see the significant contributions to the Tilde-PID. The optimization problem and solution approach are quite similar to the original work [1] also.\n   - The second contribution, Flow-PID, combines MI-preserved encoder and ThinPID algorithm to extend the PID estimation to arbitrary distributions of $X_1$, $X_2$, Y. Here, the proposed MI-preserved encoder idea is quite similar to an existing work[2], which applies for a variable pair $(X; Y)$ only. This paper likely extends this framework to multimodal settings by processing $(X_1; X_2; Y)$ as $p(X_1, Y)$ and $p(X_2, Y)$, which is quite straightforward. \n- Minor:\n   - In Sec. 3.2, the author should provide definitions of $H_1, H_2$ for clarity.\n   - There should be reference to Appendix in all derivation steps, the derivations in main text are quite ambiguous.\n   - Can the proposal be used to estimate PID for a dataset with more than 2 modalities ? If so, the author should provide the results, e.g., estimating $U_1, U_2, U_3$ and so on.\n   - For the claim in lines 295-296, can the author provide more results or visualization to show the correlation between performance and estimated PID ? This observation could be an interesting point. \n\n[1] Praveen Venkatesh, Corbett Bennett, Sam Gale, et al. Gaussian Partial Information Decomposition: Bias Correction and Application to High-dimensional Data. Advances in Neural Information Processing Systems 2023.\n\n[2] Butakov I. D., Tolmachev A. D., Malanchuk S. V., Neopryatnaya A. M., Frolov A. A., Mutual Information Estimation via Normalizing Flows, Advances in Neural Information Processing Systems 2024.", "was_revised": true, "final_justification": "Based on the original submission and the authors’ rebuttal, I am inclined to recommend acceptance of this paper.", "rebuttal": "We sincerely thank the reviewer for taking the time and effort to provide thoughtful reviews and insightful comments. We appreciate that you think our proposed Flow-PID and Thin-PID is \"**quite interesting**\" and \"**new**\" built upon \"**strong theoretical works**\". Below we will address your concerns point by point.\n\n> **[R1]** The overall flow of the proposal, while being sound, is presented in a disjointed style. End-to-end training or separated training? Which is better?\n\nWe thank the reviewer for this insightful concern, and this is closely related to our contributions. Below we will address your concern in detail.\n- The flow-based MI-preserved encoder can NOT be trained end-to-end with Thin-PID, and **they should be optimized in two stages**. Thin-PID and flow-based encoder use different strategies. Note that Thin-PID is to **solve an optimization problem with NO need to train a NN model**. Although Thin-PID uses gradient descent to find the optimal solution, it does not involve with training another NN-based model to infer the optimal solution. The advantage is that Thin-PID only takes less than 20s to search the optimal solution even when the dimension of the modality is as large as 1000.\n- **It is meaningless to perform Thin-PID when the flow is not sufficiently trained**. Note that Thin-PID is exact when the pairwise marginals $p(x_i,y)$ are Gaussian. Hence, there's no need to perform Thin-PID when the flow is not sufficiently trained in a good manner. Although Thin-PID itself is very efficient, **we still only need to perform Thin-PID once** after we transform the input into a latent marginal Gaussian space with a sufficiently trained MI-preserved encoder.\n- This separate optimization strategy **allows us to reuse pretrained encoders** to learn latent Gaussian distributions, which will significantly enhance the efficiency of Flow-PID and make it much more faster than existing PID estimators (e.g., CVX, BATCH).\n\nWe sincerely thank the reviewer for pointing out the ambiguity, and we will revise it in the final manuscript.\n\n> **[R2]** The Thin-PID, one of the main contributions, is directly built upon Tilde-PID [1], where the additional points are discussion on validity of Gaussian assumptions [1] and a different way of computing gradients. ... The optimization problem and solution approach are quite similar to the original work [1] also.\n\nWe appreciate the reviewer’s thoughtful comparison to Tilde-PID [1], and we would like to clarify the key technical and conceptual contributions of Thin-PID beyond that prior work.\n- **Our first contribution is to rigorously prove that GPID can be solved by a joint Gaussian distribution without loss of optimality**. Tilde-PID assumes the optimal solution of GPID is a joint Gaussian distribution, which leaves this an open question. On the other words, **Thin-PID is exact**, while Tilde-PID only solves an optimizing bound on true PID values.\n- **RProp and bias correction in [1] are generalized methods** which can be applied in any gradient-based and sampling-based optimization problems. We acknowledge that our Thin-PID is inspired by the idea of GPID proposed in [1], and we borrowed the RProp and bias correction from Tilde-PID. During our experimental tests, we observed that RProp is indeed helpful for convergence. Therefore, we give them the credits here.\n- **The objective function and derivation of gradients in Thin-PID are completely different** from those in Tilde-PID, although both Thin-PID and Tilde-PID solve the GPID problem $\\min I(X_1,X_2;Y)$. We use the same set of notations to provide a clear and unified understanding of the GPID problem. But **the design of the Thin-PID algorithm is new**. Please see Appendix A.2 for a detailed derivation, which is completely different from Tilde-PID. \n- Thanks to our newly designed objective function in Theorem 3.3, Thin-PID enjoys many advantages against Tilde-PID: 1) **Thin-PID has better accuracy**, which is shown in Figure C.1 and C.3 in Appendix C; 2) **Thin-PID has lower computational complexity**, which is shown in Table 1 and Figure 3; 3) **Thin-PID is more stable** when the dimensions of $X_1$ and $X_2$ are large.\n\n[1] Venkatesh, et al. Gaussian Partial Information Decomposition: Bias Correction and Application to High-dimensional Data, 2023.\n\n> **[R3]** The second contribution, Flow-PID, combines MI-preserved encoder and Thin-PID algorithm to extend the PID estimation to arbitrary distributions. Here, the proposed MI-preserved encoder idea is quite similar to an existing work.\n\nWe thank the reviewer for raising this point. We appreciate the opportunity to clarify both the novelty and motivation behind our MI-preserved encoder within the Flow-PID framework. While the idea of preserving MI during representation learning has been explored in prior work, our application and integration in PID estimates is fundamentally different in several aspects:\n- **Targeted for PID-specific inference**: Our encoder is explicitly designed to preserve task-relevant information for PID, not for classification or MI estimation. We preserve MI between two modalities and the target, which is **crucial for estimating unique and redundant information components**. Prior flow-based encoders do not target this objective and are **not suitable out-of-the-box for PID estimation**.\n- Architectural coupling with Thin-PID and Flow modeling: In Flow-PID, the encoder is not an isolated module but a critical component that interfaces with the underlying optimization problem and surrogate gradient estimator in Thin-PID. We want to note that **our goal is NOT to propose a new MI estimator, but to extend PID estimation to arbitrary (non-Gaussian, continuous, multimodal) distributions** - a capability lacking in existing MI-based encoders. \n- New theorems on estimating PID with MI-preserved encoders: we develop a new underlying optimization problem in Flow-PID to minimize the KL divergence between variational Gaussian marginals $q(x_i,y)$ and the true marginals with invertible flow-based transformations. Although approximating MI with normalizing flows [2] has been explored, **tailoring a flow-based encoder to fit for PID estimates** is not trivial, and this is a novel framework.\n\n[2] Butakov, et al, Mutual Information Estimation via Normalizing Flows, 2024.\n\n> **[R4]** In Sec. 3.2, the author should provide definitions of $H_1$, $H_2$ for clarity.\n\nThank you for this suggestion. We will add a clear pointer to Appendix. Since $(X_1, Y)$ is Gaussian in GPID, we can derive the matrix $H_1$ directly from the covariance matrix of $(X_1, Y)$. Similarly, $H_2$ is derived directly from the covariance matrix of $(X_2, Y)$. The detailed interpretation is provided in Appendix A (line 46).\n\n> **[R5]** There should be reference to Appendix in all derivation steps, the derivations in main text are quite ambiguous.\n\nThank you for this suggestion. We include the proofs of theorems and propositions in the appendix. Due to the limited space of the manuscript, we did not include enough clear pointers in the main text. But we will fix this to make sure the audience can have clear access to detailed derivations. \n\n> **[R6]** Can the proposal be used to estimate PID for a dataset with more than 2 modalities?\n\nWhile the PID framework is defined for 2 modalities, it is common to evaluate the PID on each pair of modalities, and Flow-PID can easily be extended for this. We can jointly train an MI-preserved encoder for each modality according to the analogous Gaussian constraints, and then run Thin-PID on each pair of modalities. In fact, since Thin-PID is very efficient, our approach would allow us to compute the PID on several pairs of modalities much more efficiently than BATCH/CVX. We appreciate that the reviewer raises this interesting question, which can future work in this area.\n\n> **[R7]** For the claim in lines 295-296, can the author provide more results or visualization to show the correlation between performance and estimated PID? This observation could be an interesting point.\n\nThank you for this question. We provide more clarity on this claim (although we are unable to provide visualization due to the change in format). When we are able to predict the output with very low probability of error, we expect the mutual information to be very close to the entropy of the output (in fact, for perfect prediction, the mutual information is precisely the entropy of the output). However, with high probability of error, we expect a lower mutual information. As is done in the work by Liang et al., we can also estimate the mutual information via $I(Y; X_1, X_2)$ by $H(Y) + \\log P_{\\text{acc}}$, which is  a bound on the mutual information due to Feder and Merhav [3]. This estimate matches much more closely with the total mutual information determined by Flow-PID rather than BATCH. We see that BATCH tends to recognize much less mutual information, possibly due to it being highly dependent on the accuracy of unimodal and multimodal classifiers. We find this to be an indirect verification that our method is able to identify interactions better than previous works. \n\n[3] Feder, et al. Relations between entropy and error probability, 1994.\n\n> Question: Originality and novelty . Despite soundness and strong foundation, the proposal seems to be a combination of existing solutions for related problems without significant improvements.\n\nThank you for raising this concern. We appreciate the opportunity to clarify our novelty and motivation. **We have detailed responses in [R2] and [R3]**. Here we summarize our contributions and novelty:\n- Solve an open problem in GPID by proving the optimality of joint Gaussian solutions\n- The design of Thin-PID algorithm is novel, with a newly designed objective and less complexity\n- Flow-PID is a novel framework which extends GPID to arbitrary multimodal distributions\n- Compared to other PID algorithms: more accurate, more efficient, more generalized"}, {"review_id": "fbGzxtLuAo", "reviewer": "Reviewer_18V1", "summary": "The paper proposes a novel partial information decomposition (PID) framework to quantify unique, redundant, and synergistic information in multimodal datasets, focusing on continuous and high-dimensional data. The main contributions include 1. Thin-PID, an efficient gradient-based Gaussian PID (GPID) algorithm that achieves linear complexity in some dimensions by reformulating the optimization problem and 2. Flow-PID, which uses normalized flow to transform non-Gaussian data into a latent Gaussian space for efficient PID estimation. The paper verifies the accuracy and efficiency of the method through experiments on synthetic datasets, and demonstrates its application value in tasks such as model selection on multiple real-world multimodal benchmarks.", "questions": "1.\tIn Flow-PID, when the KL divergence between the latent distribution and the Gaussian distribution is large (i.e., when the fit is poor), what kind of systematic deviation will the four components of PID (U1, U2, R, S) show? Is the collaborative information S more likely to be overestimated, or the redundant information R?\n\n2.\tIn experiments on real-world datasets (Figure 4), the total mutual information estimated by Flow-PID is much higher than that of BATCH. In the absence of true labels, how can we be sure that this \"higher\" estimate is \"more accurate\" rather than \"falsely high\" due to overfitting the noise of training samples? Although the model selection experiment (Table 5) provides indirect evidence, is this conclusion robust enough?\n\n3.\tIn Table 3, Flow-PID outperforms BATCH for discrete target tasks with collaborative information. Is this because Flow-PID’s Gaussian channel interpretation is naturally better at capturing this type of \"XOR\"-style interactions? Or is this perhaps an “inductive bias” introduced by the framework itself that favors the discovery of collaborative information?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "First of all, I would like to acknowledge that I am not familiar with the related works. My review and rating is conservative.\n\n**Strength**\n\n1.\tThe problem solved by the paper - designing an efficient PID estimator for high-dimensional continuous data - is of great significance in the fields of multimodal learning, explainable AI, etc. How to quantify and decompose information interaction is a cutting-edge and challenging topic.\n\n2.\tThe paper cleverly combines PID theory, normalized flow, and the classic Gaussian channel model to form a novel Flow-PID framework. This fusion of cross-domain ideas provides a feasible engineering path to solve a difficult theoretical problem.\n\n3.\tExperiments on synthetic datasets and MultiBench show that Flow-PID outperforms baselines such as BATCH and CVX on high-dimensional and high-cooperative interaction data (Tables 2, 3, and 4).\n\n**Weakness**\n\n1.\tThe pairwise Gaussianity assumption (Definition 3.1) is too strict. Real multimodal data (such as text and images) often present complex non-Gaussian distributions, which may not be fully captured even by normalized flow transformation. \n\n2.\tThe entire Flow-PID framework relies on a core assumption that any complex data distribution can be effectively mapped to a latent space with a Gaussian marginal distribution through normalized flow. The paper should further clarify the scope and limitations of this assumption.\n\n3.\tThe technical details (such as normalized flow implementation and optimized reconstruction) are complex and difficult to understand for non-information theory PID experts, limiting the appeal of the paper to a wider audience.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed most of my concerns. I am keeping my original score.", "rebuttal": "We sincerely thank the reviewer for taking the time and effort to provide thoughtful reviews and insightful comments. We appreciate that the reviewer consider our work as a \"**novel PID framework**\" and \"**great significant in the fields of multimodal learning**\" with theoretically and empirically demonstrated improvement. Below we will address your concerns point by point.\n\n> **[R1]** The pairwise Gaussianity assumption is too strict. Real multimodal data may not be fully captured even by normalized flow transformation.\n\nWe appreciate the reviewer's insightful comment. The pairwise Gaussianity assumption is strong but NOT a huge limitation. To clarify, our use of Definition 3.1 is not a requirement for Flow-PID, but rather serves to motivate and analyze the limitations of existing GPID estimators like Tilde-PID. The very motivation for developing Flow-PID is to relax this assumption and enable decomposition under arbitrary non-Gaussian continuous multimodal distributions. Regarding the concern that normalizing flows may still be insufficient to capture highly complex real-world data:\n\n- Flexible approximation beyond Gaussianity: Modern normalizing flows (e.g., RealNVP, Glow) have been shown to **approximate a wide class of continuous densities with sufficient depth and capacity**. In practice, we find that even moderately expressive flows (3-5 layers) already are capable of capturing skewed or heavy-tailed marginals. \n- Empirical validation: We have **a large selection of different synthetic data with known ground truth** designed with high-dimensional input modalities, non-Gaussian transformations, and different specialized interactions. **Flow-PID outperforms existing PID estimators in all these non-Gaussian cases** (*Table 2, 3, C.1*), which shows that normalizing flows can capture the appropriate latent marginal Gaussian distributions we need. \n- Future extensions: We thank the reviewer for raising the point that even flow-based models may have limitations in extremely high-dimensional or discrete-continuous hybrid settings. We view Flow-PID as a step forward toward tractable, expressive, and general-purpose PID estimation. Future work can explore more advanced generative models (e.g., latent diffusion models, transformers) to produce MI-preserved encoders in multimodal learning.\n\n> **[R2]** The entire Flow-PID framework relies on a core assumption that any complex data distribution can be effectively mapped to a latent space with a Gaussian marginal distribution through normalized flow.\n\nWe thank the reviewer for this important observation. We acknowledge that Flow-PID assumes the existence of an invertible mapping from complex data distributions to a Gaussian latent space, and we agree that the scope and limitations of this assumption warrant further clarification, which is listed as one of the limitations in Section 7. However, **this assumption is NOT a significant limitation**, and it does NOT prevent us from estimating PID in latent Gaussian distributions.\n\n- Gaussianity assumption as a practical modeling tool, not a theoretical requirement: the use of normalizing flows in Flow-PID is not to enforce perfect Gaussianity, but rather to enable a tractable surrogate space where gradient-based optimization and Thin-PID become more manageable on continuous data and targets. \n- Expressive power of normalizing flows: Our results show that the approximation error is empirically tolerable since Flow-PID is capable of dealing with real-world features of dimension 512, 1000, 1881, etc.\n- By Optimal Transport (OT) theory, the existence of a flow that transport from a source distribution to the target distribution is guaranteed under fairly mild conditions.\n\n> **[R3]** The technical details (such as normalized flow implementation and optimized reconstruction) are complex and difficult to understand for non-information theory PID experts, limiting the appeal of the paper to a wider audience.\n\nThank you for this valuable feedback. We appreciate the concern regarding accessibility and agree that making the paper more approachable for a broader audience, including those not deeply familiar with information-theoretic PID or normalizing flows, is important. Due to the page limitation, **we put the technical details in the Appendix A and B**. Additionally, **we uploaded the anonymous code and provided the settings and hyperparameters in Appendix C and D for reproducibility**. We will release the data and code for Thin-PID and Flow-PID to encourage further studies of multimodal information and modeling.\n\n> **[R4]** In Flow-PID, when the KL divergence between the latent distribution and the Gaussian distribution is large (i.e., when the fit is poor), what kind of systematic deviation will the four components of PID (U1, U2, R, S) show? Is the collaborative information S more likely to be overestimated, or the redundant information R?\n\nWe thank the reviewer for this thoughtful and technically nuanced question. Empirically when the KL divergence is large, we observe the following tendencies:\n\n- Redundancy (R) tends to be overestimated when the flow fails to decorrelate shared structure between sources. This happens when high-order correlation gets incorrectly interpreted as mutual redundancy in a near-Gaussian latent space.\n- Synergy (S) is often underestimated in such cases, because synergy requires modeling conditional interactions that are especially sensitive to misrepresentation of joint dependencies. \n- Uniqueness (U) may show mild variability, but could be biased if the flow introduces asymmetries between sources (e.g., when one source is mapped more accurately than the other).\n\nIn our experiments on real dataset, the approximation error is inevitable but empirically tolerable. Flow-PID are robust to moderate approximation error and can quantify specialized interactions as we expected.\n\n> [R5] In experiments on real-world datasets (Figure 4), the total mutual information estimated by Flow-PID is much higher than that of BATCH. In the absence of true labels, how can we be sure that this \"higher\" estimate is \"more accurate\" rather than \"falsely high\" due to overfitting the noise of training samples? Although the model selection experiment (Table 5) provides indirect evidence, is this conclusion robust enough?\n\nThank you for raising this question. First of all, we want to emphasize that we have a large collection of synthetic datasets, involving with diverse multimodalities: Gaussian and non-Gaussian, discrete and continuous, low and high dimensional, and different specialized interactions, where the ground truth of PID is known. Our Flow-PID outperforms the SOTA baselines\nin all these synthetic cases, which partially shows its generalized capability on real datasets. As for the total mutual information, we can provide some more clarity on this claim and provide more quantitative evidence (although we are unable to provide visualization due to the change in format). When we are able to predict the output with very low probability of error, we expect the mutual information to be very close to the entropy of the output (in fact, for perfect prediction, the mutual information is precisely the entropy of the output). However, with high probability of error, we expect a lower mutual information. As is done in the work by Liang et al., we can also estimate the mutual information via $I(Y; X_1, X_2)$ by $H(Y) + \\log P_{\\text{acc}}$, which is  a bound on the mutual information due to Feder and Merhav [1]. This estimate matches much more closely with the total mutual information determined by Flow-PID rather than BATCH. We see that BATCH tends to recognize much less mutual information, possibly due to it being highly dependent on the accuracy of unimodal and multimodal classifiers. We find this to be an indirect verification that our method is able to identify interactions better than previous works. Lastly, Flow-PID selects the most appropriate model by measuring the similarity between synthetic and real datasets. Given that fact that Flow-PID performs well on a large collection of synthetic datasets, good model selection results infer the appropriate PID estimates on complex real-world data.\n\n[1] Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transactions on Information theory, 40(1):259–266, 1994.\n\n> [R6] In Table 3, Flow-PID outperforms BATCH for discrete target tasks with collaborative information. Is this because Flow-PID’s Gaussian channel interpretation is naturally better at capturing this type of \"XOR\"-style interactions? Or is this perhaps an “inductive bias” introduced by the framework itself that favors the discovery of collaborative information?\n\nThank you for this excellent question. Theoretically, the Gaussian channel interpretation allows us to solve PID exactly in marginal Gaussian distributions. Empirically, the flow-based encoder has expressive power of learning a wide class of continuous transformations, and Flow-PID are robust to moderate approximation error. On the other hand, BATCH approximates the pointwise discrete $p(x_1,x_2,y)$ using Sinkhorn's algorithm, which may introduce large approximation error. Additionally, it is known that BATCH generally underestimates synergy [2] since it is directly dependent on estimation of $p(y | x_1, x_2)$, which is done via the best performing multimodal classification model. Therefore, it is shown in Table 3 that Flow-PID performs better on tasks involving strong synergistic information, such as XOR-style dependencies. We also want to note that Flow-PID also outperforms BATCH in tasks with specialized redundancy and uniqueness. \n\n[2] Liang, P. P., Cheng, Y., Fan, X., Ling, C. K., Nie, S., Chen, R., Deng, Z., Allen, N., Auerbach, R., Mahmood, F., et al. (2024). Quantifying \\& modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems, 36."}, {"review_id": "w6zPZ8AjPY", "reviewer": "Reviewer_H15Z", "summary": "## Summary\nThis paper introduces a new computational framework for Partial Information Decomposition (PID), focusing on continuous and high-dimensional multimodal data where traditional PID estimation methods struggle. The proposed method is a new efficient gradient-based algorithm for PID under Gaussian assumptions, and a flow-based extension to handle non-Gaussian, high-dimensional data.", "questions": "See Weakness", "limitations": "See Weakness", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n1) This paper proves that when pairwise distributions are Gaussian, the optimal PID solution also lies within the space of joint Gaussians.\n2) This paper demonstrates that PID profiles (from Flow-PID) can help select the best-performing multimodal fusion model for a new task.\n3) The paper is well-formulated and well-grounded. \n4) The results presented are good. \n\n## Weakness:\n1) Flow-PID’s accuracy depends on the quality of the Gaussianization step using normalizing flows.\n2) Hard to rigorously validate PID estimates on real-world datasets where the true PID values are unknown.\n3) Performance may degrade with highly complex feature distributions or small sample sizes.\n4) Due to the fundamental aspect of the proposed method, it would be more impactful if the authors could open-source their code for people to draw connections between the maths and the implementation.", "was_revised": true, "final_justification": "The proposed Partial Information Decomposition seem to have potential in many other machine learning areas due to its mathematical foundation. In the rebuttal stage, the author also addressed my concerns very timely and promised to release code for public research. So I consider giving rating 5 to accept the paper and encourage related research in the following years.", "rebuttal": "We sincerely thank the reviewer for taking the time and effort to provide thoughtful reviews and insightful comments. We appreciate that the reviewer considers our work as a \"**well-formulated and well-grounded paper**\" with \"**good presented results**\". Below we will address your concerns point by point. \n\n> [R1] Flow-PID’s accuracy depends on the quality of the Gaussianization step using normalizing flows.\n\nWe thank the reviewer for raising this concern. The pairwise Gaussianity is NOT a fundamental limitation. Modern normalizing flows (e.g., RealNVP, Glow) have been shown to approximate a wide class of continuous densities with sufficient depth and capacity. A large collection of experiments on synthetic datasets indicates that perfect Gaussianization is not required: we have empirically observed that moderate approximation suffices to recover meaningful and accurate PID decompositions.\n\n> [R2] Hard to rigorously validate PID estimates on real-world datasets where the true PID values are unknown.\n\nWe thank the reviewer for raising this concern. Rigorous validation of PID estimates on real-world datasets is inherently difficult as the generation of multimodal data is unknown. However, we firstly want emphasize that **we have a large collection of synthetic datasets involving with diverse multimodalities**: Gaussian and non-Gaussian, discrete and continuous, low and high dimensional, and different specialized interactions, where the ground truth of PID is known. Our proposed model outperforms the SOTA baselines in all these synthetic cases, which partially shows its generalized capability on real datasets. Secondly, we used Flow-PID to select high-performance model by measuring the similarity between synthetic and real datasets, which helps support the correctness of PID estimates on complex real-world data. Lastly, we also provided 2 additional experiments on VQA2.0 and TCGA with task-driven interactions, which **provides evidence of interpretive alignment that are causally relevant**.\n\n| Dataset | dim-X1 | dim-X2 | R | U1 | U2 | S | Expected interactions |\n|------|-----|-----|-----|-----|-----|-----|-----|\n| TCGA | 487 | 1881 | 0.41 | 0.0 | **1.07** | 0.34 | U2 |\n| VQA2.0 | 768 | 1000 | 0.22 | 0.26 | 0.0 | **0.76** | S |\n\n> [R3] Performance may degrade with highly complex feature distributions or small sample sizes.\n\nWe thank the reviewer for this raising this point. Both distributional complexity and limited data pose general challenges for information-theoretic estimators, including ours. Flow-PID specifically designed to address complex feature distributions by leveraging expressive normalizing flows. Regarding small sample sizes, we acknowledge that overfitting is a risk. To mitigate this, we employ a MI regularization in the encoder and flow. We listed this as one of limitations in Section 7, but we think this is a common challenge in PID society.\n\n> [R4] Due to the fundamental aspect of the proposed method, it would be more impactful if the authors could open-source their code for people to draw connections between the maths and the implementation.\n\nWe thank the reviewer for this valuable feedback. We appreciate the concern regarding accessibility and agree that making the paper more approachable for a broader audience is important. Due to the anonymity, we only attached the anonymous code in the supplemental materials. **We will release the data and code for Thin-PID and Flow-PID to encourage further studies of multimodal information and modeling**."}]

Author rebuttal:

Meta-review: This paper received positive reviews, including 3 borderline accepts and 1 accept. The reviewers reached a consensus that the paper addresses a significant problem in multimodal learning, provides rigorous mathematical proofs, and demonstrates effectiveness across multiple scenarios. After the rebuttal, all reviewers indicated that their concerns were mostly addressed. Therefore, the AC is inclined to recommend acceptance of this work.
Decision: Accept (poster)