SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\stackrel {(c)} {=} \frac {1}{2} \log \det  (I + \Sigma_ {M} ^ {- 1} \Sigma_ {X Y, M} ^ {\top} \Sigma_ {X Y | M} ^ {- 1} \Sigma_ {X Y, M}), \tag {28}
$$
where in (a) we used equation (18), in (b) we used equation (20), and in (c) we used the fact that  $\operatorname{det}(I + AB) = \operatorname{det}(I + BA)$ .
The remainder of the proof follows from the arguments presented below equation (8). The constraint in Proposition 1 arises because, when optimizing over  $\Sigma_{X,Y|M}$ , we require  $\Sigma_{MXY}$  to be a valid positive semidefinite covariance matrix, i.e.,  $\Sigma_{MXY} \succcurlyeq 0$ . This happens if and only if  $\Sigma_M$  and its Schur complement in  $\Sigma_{MXY}$  are both positive semidefinite, i.e.,  $\Sigma_M \succcurlyeq 0$  and  $\Sigma_M - \Sigma_{M,XY} \Sigma_{XY}^{-1} \Sigma_{M,XY}^{\mathsf{T}} = \Sigma_{XY|M} \succcurlyeq 0$ .
Proof of Proposition 2. The proof is divided into three parts consisting of derivations for the objective, the gradient and the projection operator.
Objective. After whitening the  $P_{X|M}$  and the  $P_{Y|M}$  channels, and assuming that  $\Sigma_M = I$  (see Remark 1), without loss of generality we have that
$$
\Sigma_ {X, M} = \mathbb {E} \left[ \left(H _ {X} M + N _ {X}\right) M ^ {\top} \right] = H _ {X} \mathbb {E} [ M M ^ {\top} ] = H _ {X} \tag {29}
$$
$$
\Sigma_ {X Y \mid M} = \left[ \begin{array}{c c} I & \Sigma_ {X, Y \mid M} \\ \Sigma_ {X, Y \mid M} ^ {\intercal} & I \end{array} \right] \tag {30}
$$
$$
\Rightarrow \Sigma_ {M} ^ {- 1} \Sigma_ {X Y, M} ^ {\mathsf {T}} \Sigma_ {X Y | M} ^ {- 1} \Sigma_ {X Y, M} = \left[ \begin{array}{l l} H _ {X} ^ {\mathsf {T}} & H _ {Y} ^ {\mathsf {T}} \end{array} \right] \left[ \begin{array}{c c} I & \Sigma_ {X, Y | M} \\ \Sigma_ {X, Y | M} ^ {\mathsf {T}} & I \end{array} \right] ^ {- 1} \left[ \begin{array}{l} H _ {X} \\ H _ {Y} \end{array} \right] \tag {31}
$$
For the sake of brevity, let  $\Sigma$  represent the optimization variable  $\Sigma_{X,Y|M}$ , and let  $S$  be its Schur complement in  $\Sigma_{XY|M}, I - \Sigma \Sigma^{\top}$ . Then, the inverse of  $\Sigma_{XY|M}$  can be written as [44, Sec. 9.1.5]:
$$
\Sigma_ {X Y \mid M} ^ {- 1} = \left[ \begin{array}{c c} S ^ {- 1} & - S ^ {- 1} \Sigma \\ - \Sigma^ {\top} S ^ {- 1} & I + \Sigma^ {\top} S ^ {- 1} \Sigma \end{array} \right] \tag {32}
$$
Therefore, we get:
$$
\begin{array}{l} \Sigma_ {M} ^ {- 1} \Sigma_ {X Y, M} ^ {\mathsf {T}} \Sigma_ {X Y | M} ^ {- 1} \Sigma_ {X Y, M} = \left[ \begin{array}{l l} H _ {X} ^ {\mathsf {T}} & H _ {Y} ^ {\mathsf {T}} \end{array} \right] \left[ \begin{array}{c c} S ^ {- 1} & - S ^ {- 1} \Sigma \\ - \Sigma^ {\mathsf {T}} S ^ {- 1} & I + \Sigma^ {\mathsf {T}} S ^ {- 1} \Sigma \end{array} \right] \left[ \begin{array}{l} H _ {X} \\ H _ {Y} \end{array} \right] (33) \\ = H _ {Y} ^ {\top} H _ {Y} + \left(H _ {X} - \Sigma H _ {Y}\right) ^ {\top} S ^ {- 1} \left(H _ {X} - \Sigma H _ {Y}\right) (34) \\ \end{array}
$$
Thus, setting  $B \coloneqq H_X - \Sigma H_Y$ , the optimization problem in Proposition 1 reduces to
$$
\min  _ {\Sigma} \frac {1}{2} \log \det  \left(I + H _ {Y} ^ {\top} H _ {Y} + B ^ {\top} S ^ {- 1} B\right) \tag {35}
$$
$$
s. t. \quad \Sigma_ {X Y | M} \succcurlyeq 0
$$
Gradient. Let the objective derived in the previous section be called  $f(\Sigma)$ , where  $\Sigma \coloneqq \Sigma_{X,Y|M}$  as before. We can compute the gradient of  $f$  with respect to  $\Sigma$  using standard identities from matrix calculus. First, note that the gradient of a scalar function with respect to a matrix is itself a matrix with entries as follows:
$$
\left. \nabla f (\Sigma) \right| _ {i j} = \frac {\partial f}{\partial \Sigma_ {i j}} (\Sigma). \tag {36}
$$
Considering each element of this matrix:
$$
\left. \frac {\partial f}{\partial \Sigma_ {i j}} (\Sigma) = \frac {1}{2} \frac {\partial}{\partial \Sigma_ {i j}} \log \det  \left(I + H _ {Y} ^ {\top} H _ {Y} + B ^ {\top} S ^ {- 1} B\right) \right| _ {\Sigma} \tag {37}
$$
$$
\stackrel {(a)} {=} \frac {1}{2} \operatorname {T r} \left\{\left(I + H _ {Y} ^ {\top} H _ {Y} + B ^ {\top} S ^ {- 1} B\right) ^ {- 1} \frac {\partial}{\partial \Sigma_ {i j}} \left(I + H _ {Y} ^ {\top} H _ {Y} + B ^ {\top} S ^ {- 1} B\right) \right\} \Bigg | _ {\Sigma} \tag {38}
$$
$$
\stackrel {(b)} {=} \frac {1}{2} \operatorname {T r} \left\{\left(I + H _ {Y} ^ {\top} H _ {Y} + B ^ {\top} S ^ {- 1} B\right) ^ {- 1} \frac {\partial}{\partial \Sigma_ {i j}} \left(B ^ {\top} S ^ {- 1} B\right) \right\} \Bigg | _ {\Sigma}, \tag {39}
$$
where in (a), we have used the identity  $\partial \log \det(X) = \operatorname{Tr}\{X^{-1} \partial(X)\}$  [44, Sec. 2], while in (b), we use the fact that only  $B$  and  $S$  depend on  $\Sigma$  implicitly, with the other terms being constants.
Expanding the partial derivative alone, we get:
$$
\left. \frac {\partial}{\partial \Sigma_ {i j}} \left(B ^ {\mathsf {T}} S ^ {- 1} B\right) \right| _ {\Sigma} = \left[ \frac {\partial}{\partial \Sigma_ {i j}} \left(B ^ {\mathsf {T}}\right) \cdot S ^ {- 1} B + B ^ {\mathsf {T}} \cdot \frac {\partial}{\partial \Sigma_ {i j}} \left(S ^ {- 1}\right) \cdot B + B ^ {\mathsf {T}} S ^ {- 1} \cdot \frac {\partial}{\partial \Sigma_ {i j}} (B) \right] _ {\Sigma}, \tag {40}
$$
wherein
$$
\left. \frac {\partial}{\partial \Sigma_ {i j}} (B) \right| _ {\Sigma} = \left. \frac {\partial}{\partial \Sigma_ {i j}} \left(H _ {X} - \Sigma H _ {Y}\right) \right| _ {\Sigma} \tag {41}
$$
$$
\stackrel {(b)} {=} - J ^ {i j} H _ {Y}, \tag {42}
$$
$$
\left. \frac {\partial}{\partial \Sigma_ {i j}} \left(S ^ {- 1}\right) \right| _ {\Sigma} \stackrel {(c)} {=} - S ^ {- 1} \frac {\partial S}{\partial \Sigma_ {i j}} S ^ {- 1} \Bigg | _ {\Sigma} \tag {43}
$$
$$
= - S ^ {- 1} \frac {\partial}{\partial \Sigma_ {i j}} \left(I - \Sigma \Sigma^ {\mathsf {T}}\right) S ^ {- 1} \Big | _ {\Sigma} \tag {44}
$$
$$
\stackrel {(d)} {=} - S ^ {- 1} \left(- J ^ {i j} \Sigma^ {\top} - \Sigma J ^ {i j \top}\right) S ^ {- 1}, \tag {45}
$$
where  $J^{ij}$  is the single-entry matrix, containing a 1 at location  $(i,j)$  and 0's everywhere else; in (b) and (d), we use the fact that  $\partial X / \partial X_{ij} = J^{ij}$  [44, Sec. 9.7.6]; and in (c) we use the identity
$\partial (X^{-1}) = X^{-1}\partial (X)X^{-1}$ . Therefore, (40) becomes
$$
\begin{array}{l} \left. \frac {\partial}{\partial \Sigma_ {i j}} \left(B ^ {\mathsf {T}} S ^ {- 1} B\right) \right| _ {\Sigma} (46) \\ = - \left(J ^ {i j} H _ {Y}\right) ^ {\top} S ^ {- 1} B + B ^ {\top} S ^ {- 1} \left(J ^ {i j} \Sigma^ {\top} + \Sigma J ^ {i j \top}\right) S ^ {- 1} B + B ^ {\top} S ^ {- 1} \left(- J ^ {i j} H _ {Y}\right) (47) \\ = - H _ {Y} ^ {\top} J ^ {i j \top} S ^ {- 1} B + B ^ {\top} S ^ {- 1} J ^ {i j} \Sigma^ {\top} S ^ {- 1} B + B ^ {\top} S ^ {- 1} \Sigma J ^ {i j \top} S ^ {- 1} B - B ^ {\top} S ^ {- 1} J ^ {i j} H _ {Y}. (48) \\ \end{array}
$$
Putting it all together, and letting  $A\coloneqq (I + H_Y^\top H_Y + B^\top S^{-1}B)$  , (39) becomes
$$
\frac {\partial f}{\partial \Sigma_ {i j}} (\Sigma) = \frac {1}{2} \operatorname {T r} \left\{A ^ {- 1} \left(- H _ {Y} ^ {\top} J ^ {i j \top} S ^ {- 1} B + B ^ {\top} S ^ {- 1} J ^ {i j} \Sigma^ {\top} S ^ {- 1} B \right. \right.
$$
$$
\left. + \left. B ^ {\top} S ^ {- 1} \Sigma J ^ {i j \top} S ^ {- 1} B - B ^ {\top} S ^ {- 1} J ^ {i j} H _ {Y}\right) \right\} \tag {49}
$$
$$
\begin{array}{l} = \frac {1}{2} \left[ - \operatorname {T r} \left\{A ^ {- 1} H _ {Y} ^ {\top} J ^ {i j \top} S ^ {- 1} B \right\} + \operatorname {T r} \left\{A ^ {- 1} B ^ {\top} S ^ {- 1} J ^ {i j} \Sigma^ {\top} S ^ {- 1} B \right\} \right. \\ \left. + \operatorname {T r} \left\{A ^ {- 1} B ^ {\top} S ^ {- 1} \Sigma J ^ {i j \top} S ^ {- 1} B \right\} - \operatorname {T r} \left\{A ^ {- 1} B ^ {\top} S ^ {- 1} J ^ {i j} H _ {Y} \right\} \right] \tag {50} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(e)} {=} \frac {1}{2} \Big [ - \operatorname {T r} \left\{S ^ {- 1} B A ^ {- 1} H _ {Y} ^ {\top} J ^ {i j \top} \right\} + \operatorname {T r} \left\{\Sigma^ {\top} S ^ {- 1} B A ^ {- 1} B ^ {\top} S ^ {- 1} J ^ {i j} \right\} \\ \left. + \operatorname {T r} \left\{S ^ {- 1} B A ^ {- 1} B ^ {\top} S ^ {- 1} \Sigma J ^ {i j \top} \right\} - \operatorname {T r} \left\{H _ {Y} A ^ {- 1} B ^ {\top} S ^ {- 1} J ^ {i j} \right\} \right], \tag {51} \\ \end{array}
$$
where in (e), we have used the fact that the trace of a matrix product is invariant under cyclic permutations of the matrices within the product.
Finally, using the fact that  $\mathrm{Tr}\{W^{\mathsf{T}}J^{ij}\} = \mathrm{Tr}\{WJ^{ij\mathsf{T}}\} = W_{ij}$  for any matrix  $W$  [44, Sec. 9.7.5],
$$
\begin{array}{l} \frac {\partial f}{\partial \Sigma_ {i j}} (\Sigma) = \frac {1}{2} \left[ - 2 \left(S ^ {- 1} B A ^ {- 1} H _ {Y} ^ {\top}\right) _ {i j} + 2 \left(S ^ {- 1} B A ^ {- 1} B ^ {\top} S ^ {- 1} \Sigma\right) _ {i j} \right] (52) \\ = \left[ S ^ {- 1} B A ^ {- 1} \left(B ^ {\mathsf {T}} S ^ {- 1} \Sigma - H _ {Y} ^ {\mathsf {T}}\right) \right] _ {i j} (53) \\ \Rightarrow \quad \nabla f (\Sigma) = S ^ {- 1} B A ^ {- 1} \left(B ^ {\mathsf {T}} S ^ {- 1} \Sigma - H _ {Y} ^ {\mathsf {T}}\right) (54) \\ = S ^ {- 1} B \left(I + H _ {Y} ^ {\top} H _ {Y} + B ^ {\top} S ^ {- 1} B\right) ^ {- 1} \left(B ^ {\top} S ^ {- 1} \Sigma - H _ {Y} ^ {\top}\right). (55) \\ \end{array}
$$
Projection operator. Recall that the optimization variable,  $\Sigma \coloneqq \Sigma_{X,Y|M}$  is an off-diagonal block of  $\Sigma_{XY|M}$ , which is the matrix upon which the constraint is defined:
$$
\Sigma_ {X Y \mid M} = \left[ \begin{array}{c c} I & \Sigma \\ \Sigma^ {\mathsf {T}} & I \end{array} \right], \tag {56}
$$
wherein the diagonal blocks are identity due to Remark 1. For the purposes of this section, let us suppose  $\Sigma_{XY|M}$  is a function of  $\Sigma$ ,  $\Sigma_{XY|M} \eqqcolon g(\Sigma)$ , so that the constraint may be written as  $g(\Sigma) \succcurlyeq 0$ . A suitable projection operator, therefore, will accept a value  $\Sigma_0$  (that may violate  $g(\Sigma_0) \succcurlyeq 0$ ) and find a point  $\Sigma^{proj}$  close to it that satisfies the constraint, i.e.,  $g(\Sigma^{proj}) \succcurlyeq 0$ .
We do not find the "orthogonal" projection operator, which has the minimum distance  $\|\Sigma^{proj} - \Sigma_0\|$  in some norm. Instead, we propose a simple heuristic to find a  $\Sigma^{proj}$  which satisfies the constraint.
If  $\Sigma_0$  satisfies the constraint, then we are done, so let us assume that  $g(\Sigma_0) \neq 0$ . Then, we can find a matrix  $\overline{\Sigma}_{XY|M}$  which is close to  $g(\Sigma_0)$  and satisfies  $\overline{\Sigma}_{XY|M} \succcurlyeq 0$  as follows: let the eigenvalue decomposition of  $g(\Sigma_0)$  be given by  $V\Lambda V^{\mathsf{T}}$ , with  $\Lambda = \mathrm{diag}(\lambda_i)$  being the diagonal matrix consisting of its eigenvalues  $\lambda_{i}$ . Then, since  $g(\Sigma_0)$  is not positive semidefinite,  $\exists i$  s.t.  $\lambda_{i} < 0$ . We set  $\bar{\lambda}_i \coloneqq 0$  for all such  $i$ ; effectively,  $\bar{\lambda}_i = \max \{0, \lambda_i\} \forall i$ . We then reconstruct the matrix using these "rectified" eigenvalues and set it to be  $\overline{\Sigma}_{XY|M} \coloneqq V\overline{\Lambda} V^{\mathsf{T}}$ , where  $\overline{\Lambda} = \mathrm{diag}(\bar{\lambda}_i)$ .
Now, we need to find  $\Sigma$  such that  $g(\Sigma) = \overline{\Sigma}_{XY|M}$ . However,  $\overline{\Sigma}_{XY|M}$  may not have identity matrices on its diagonal blocks, i.e., it might not correspond to a whitened channel. We therefore whiten
$\overline{\Sigma}_{XY|M}$  as follows:
$$
\bar {\Sigma} _ {X Y | M} ^ {\text {w h i t e n e d}} = \left[ \begin{array}{c c} \bar {\Sigma} _ {X | M} ^ {- 1 / 2} & 0 \\ 0 & \bar {\Sigma} _ {Y | M} ^ {- 1 / 2} \end{array} \right] \bar {\Sigma} _ {X Y | M} \left[ \begin{array}{c c} \bar {\Sigma} _ {X | M} ^ {- 1 / 2} & 0 \\ 0 & \bar {\Sigma} _ {Y | M} ^ {- 1 / 2} \end{array} \right], \tag {57}
$$
where  $\overline{\Sigma}_{X|M}$  and  $\overline{\Sigma}_{Y|M}$  are the diagonal blocks of  $\overline{\Sigma}_{XY|M}$ . Crucially, since the matrix multiplying  $\overline{\Sigma}_{XY|M}$  on either side is itself (the inverse square-root of) a covariance matrix (and hence positive semidefinite),  $\overline{\Sigma}_{XY|M}^{\text{whitened}}$  is also positive semidefinite.
Now, the off-diagonal block of  $\overline{\Sigma}_{XY|M}^{\text{whitened}}$  will satisfy  $g(\cdot) = \overline{\Sigma}_{XY|M}^{\text{whitened}} \succcurlyeq 0$ . This off-diagonal block forms the output of our projection operation and can be written as
$$
\Sigma_ {X, Y \mid M} ^ {\text {p r o j}} = \bar {\Sigma} _ {X \mid M} ^ {- 1 / 2} \bar {\Sigma} _ {X Y \mid M} \bar {\Sigma} _ {Y \mid M} ^ {- 1 / 2}, \tag {58}
$$
which comes directly from equation (57).
A.4 Details of  $\sim_G$ -PID Optimization and RProp Implementation
The optimization problem for the  $\sim_G$ -PID, using projected gradient descent with RProp (mentioned in Section 3), is implemented as follows:
1. Let  $\Sigma \coloneqq \Sigma_{X,Y|M}$  be shorthand for the optimization variable, and let  $\mathrm{Proj}(\cdot)$  represent the projection operator defined in Prop. 2. Let  $\Sigma^{(i)}$  represent the value of  $\Sigma$  at iteration  $i$  of the optimization. Initialize  $\Sigma^{(0)} = \mathrm{Proj}(H_X H_Y^+)$ , where  $H_{Y}^{+}$  is the pseudoinverse of  $H_{Y}$ .
2. Evaluate the objective and the gradient as defined in Prop. 2, at the current value of  $\Sigma^{(i)}$ . Compute the sign of (each element of) the gradient,
$$
\psi \left(\Sigma^ {(i)}\right) := \operatorname {S g n} \left(\nabla f \left(\Sigma^ {(i)}\right)\right). \tag {59}
$$
When computing the objective and the gradient, add a small regularization term to the computation of  $S^{-1}$  (as defined in Prop. 2):  $S^{-1} = \left((1 + \epsilon)I - \Sigma \Sigma^{\mathsf{T}}\right)^{-1}$ , where we take  $\epsilon = 10^{-7}$ .
3. Update:
$$
\Sigma^ {(i + 1)} = \operatorname {P r o j} \left(\Sigma^ {(i)} - \alpha^ {i} \eta^ {(i)} \odot \psi \left(\Sigma^ {(i)}\right)\right), \tag {60}
$$
where  $\mathrm{Proj}$  is the projection operator defined in Prop. 2;  $\eta^{(i)}$  is a time-varying learning rate vector of the same dimension as  $\Sigma$ , describing the learning rate for each element of  $\Sigma$ ;  $\odot$  represents an element-wise (or Hadamard) product between vectors; and  $\alpha := 0.999$  is a constant, which when raised to the power of  $i$ , imposes a slow overall decay of the learning rate to promote convergence.
The matrix inverses embedded in the projection operator are also regularized by modifying Equation (58) as follows:
$$
\Sigma_ {X, Y \mid M} ^ {p r o j} = \left(\gamma I + \bar {\Sigma} _ {X \mid M} ^ {1 / 2}\right) ^ {- 1} \bar {\Sigma} _ {X Y \mid M} \left(\gamma I + \bar {\Sigma} _ {Y \mid M} ^ {1 / 2}\right) ^ {- 1}, \tag {61}
$$
where  $\gamma$  is slowly increased from  $10^{-12}$ , by a factor of 10 in each step, until  $g(\Sigma_{X,Y|M}^{proj})\succcurlyeq 0$ .
4.  $\eta^{(0)}$  is initialized to  $10^{-3}$  and  $\eta^{(i)}$  is updated as follows:
$$
\eta^ {(i + 1)} = \eta^ {(i)} \odot \beta^ {- \psi (\Sigma^ {(i + 1)}) \odot \psi (\Sigma^ {(i)})}, \tag {62}
$$
where  $\beta \coloneqq 0.9$  is a constant that determines how fast the learning rate increases or decreases; and all operations are carried out element-wise. Note that when some element of the gradient changes in sign, that element of  $-\psi(\Sigma^{(i+1)}) \odot \psi(\Sigma^{(i)})$  will be positive, resulting in a decrease in that element of  $\eta^{(i)}$ . On the other hand, if the sign of some element of the gradient remains the same, then the learning rate for that component will increase by a factor of  $1/0.9$ .
5. Stop when the absolute differences between the current objective and the previous objectives from the last 20 consecutive iterations are all less than  $10^{-6}$  ("patience"), or when the maximum number of iterations is exceeded (set to  $10^{4}$  iterations).
A.5 Computational Complexity of the  $\sim_G$ -PID
Suppose for simplicity that  $M, X$  and  $Y$  all have the same dimensionality  $d \coloneqq d_M = d_X = d_Y$ . Then,  $\Sigma_{MXY}$  and all its submatrices have a side of dimensionality  $\mathcal{O}(d)$ . Thus, the computational complexity of each gradient descent iteration is determined by the complexity of the matrix operations in equations (10)-(13). These operations include matrix multiplication, matrix inverses,  $\log \det(\cdot)$  and eigenvalue decomposition, which have a worst case complexity of  $\mathcal{O}(d^3)$ .
Matrix multiplication can potentially be performed at lower complexity for large matrix sizes (e.g., see Strassen [42]), which has downstream implications for each of the other operations as well. However, for our purposes, we take the computational complexity of the objective, the gradient and the projection operator to all be  $\mathcal{O}(d^3)$ .
B Supplementary Material for Section 4
First, observe that by subtracting equation (2) from equation (1), we have
$$
\begin{array}{l} I (M; (X, Y)) - I (M; X) = U I _ {Y} + S I \\ \Rightarrow \quad I (M; Y \mid X) = U I _ {Y} + S I. \tag {63} \\ \end{array}
$$
Similarly, subtracting equations (1) and (3), we get that  $I(M; X \mid Y) = UI_X + SI$ . These two equations hold in general, and will be used in what follows.
B.1 Details and Derivations for Examples in Section 4
Example 2 (Pure uniqueness).
$$
M \sim \mathcal {N} (0, 1) \tag {64}
$$
$$
X = M + N _ {X} \quad N _ {X}, N _ {Y} \sim \text {i . i . d .} \mathcal {N} (0, 1) \tag {65}
$$
$$
Y = N _ {Y} \quad \left(N _ {X}, N _ {Y}\right) \perp M \tag {66}
$$
Derivation of PID values in Example 2.
$$
\begin{array}{l} Y \perp M \Rightarrow I (M; Y) = 0 (67) \\ \Rightarrow U I _ {Y} + R I = 0 (68) \\ U I _ {Y}, R I \geq 0 \Rightarrow U I _ {Y} = R I = 0 (69) \\ \Rightarrow \quad U I _ {X} = I (M; X) (70) \\ \Rightarrow \quad S I = I (M; (X, Y)) - I (M; X) (71) \\ = I (M; X) - I (M; X) = 0. (72) \\ \end{array}
$$
Example 3 (Pure redundancy).
$$
M \sim \mathcal {N} (0, 1) \tag {73}
$$
$$
X = M + N _ {X} \quad N _ {X} \sim \mathcal {N} (0, 1) \tag {74}
$$
$$
Y = M + N _ {X} \quad N _ {X} \perp M \tag {75}
$$
Derivation of PID values in Example 3.
$$
\begin{array}{l} I (M; X \mid Y) = 0 \quad \Rightarrow \quad U I _ {X} + S I = 0 (76) \\ I (M; Y \mid X) = 0 \quad \Rightarrow \quad U I _ {Y} + S I = 0 (77) \\ \Rightarrow \quad R I = I (M; (X, Y)) = I (M; X). (78) \\ \end{array}
$$
Example 4 (Pure synergy).
$$
M \sim \mathcal {N} (0, 1) \tag {79}
$$
$$
X = M + N _ {X} \quad N _ {X} \sim \mathcal {N} (0, \sigma^ {2}) \tag {80}
$$
Figure 7: Diagrams explaining Examples 8 and 9. See App. B.2 for details.
$$
Y = N _ {X} \quad N _ {X} \perp M \tag {81}
$$
Derivation of PID values in Example 4.
$$
\begin{array}{l} Y \perp M \Rightarrow I (M; Y) = 0 (82) \\ \Rightarrow U I _ {Y} + R I = 0 (83) \\ U I _ {Y}, R I \geq 0 \Rightarrow U I _ {Y} = R I = 0 (84) \\ \Rightarrow \quad U I _ {X} = I (M; X) (85) \\ \Rightarrow \quad S I = I (M; (X, Y)) - I (M; X) (86) \\ = \infty - I (M; X) = \infty . (87) \\ \end{array}
$$
It should be noted that certain nuances have been omitted in discussing Examples 2-4 above. For instance, in Example 3,  $\Sigma_{XY|M}$  is rank deficient and hence non-invertible, which would be an issue when computing the objective in Equation (9). Also, in Example 4,  $I(M; (X,Y)) = \infty$ , however this could be corrected by adding some noise to either  $X$  or  $Y$  so that their difference is a noisy representation of  $M$ .
Example 5 (Unique and redundant information).
$$
M \sim \mathcal {N} (0, 1) \tag {88}
$$
$$
X = M + N _ {X} \quad N _ {X} \sim \mathcal {N} (0, 1) \quad N _ {X} \perp M \tag {89}
$$
$$
Y = M + N _ {X} + N _ {Y} ^ {\prime} \quad N _ {Y} ^ {\prime} \sim \mathcal {N} (0, \sigma^ {2}) \quad N _ {Y} ^ {\prime} \perp (N _ {X}, M) \tag {90}
$$
Derivation of PID values in Example 5. Essentially,  $X$  is a noisy representation of  $M$ , while  $Y$  is a noisy representation of  $X$ . Since  $M - X - Y$  forms a Markov chain,  $I(M;Y\mid X) = 0$ , and hence  $UI_{Y} = SI = 0$ . When  $\sigma^2 = 0$ , this example reduces to Example 3 with only redundancy being present. For any finite non-zero value of  $\sigma^2$ , both  $RI$  and  $UI_{X}$  are present and are non-zero. Since  $M$  is scalar, the redundancy for the  $\sim$ -PID is identical to the MMI-PID's redundancy [13]:
$$
R I = \min  \left\{I (M; X), I (M; Y) \right\} = I (M; Y), \tag {91}
$$
since  $I(M;Y) < I(M;X)$ , by the data processing inequality. At the limit when  $\sigma^2 \to \infty$ ,  $I(M;Y) \to 0$ , and therefore  $RI \to 0$ , while  $UI_{X}$  will become equal to  $I(M;X)$ .
Example 6 (Unique and synergistic information).
$$
M \sim \mathcal {N} (0, 1) \tag {92}
$$
$$
X = M + N _ {X} \quad N _ {X}, N _ {Y} \sim \mathcal {N} (0, \sigma^ {2}), \quad \left(N _ {X}, N _ {Y}\right) \perp M \tag {93}
$$
$$
Y = N _ {Y} \quad \operatorname {C o r r} \left(N _ {X}, N _ {Y}\right) = \rho \tag {94}
$$
Derivation of PID values in Example 6. When  $\rho = 1$  and  $\sigma^2 \to \infty$ , this example reduces to Example 4, with only synergy being present. In general,  $Y \perp M$ , therefore,  $I(M; Y) = UI_Y + RI = 0$ , meaning  $UI_Y = RI = 0$ . For any finite value of  $\sigma^2$ ,  $X$  will have some unique information about  $M$  given by  $UI_X = I(M; X) > 0$ . Correspondingly,  $SI = I(M; X|Y) - UI_X =$
Figure 8: A comparison of the  $\sim_{G^{-}}$  and  $\delta_G$ -PIDs at a dimensionality of  $d = 64$ . The plot is virtually identical to that in Fig. 2 (which uses  $d = 2$ ), except that the  $y$ -axis is scaled up by a factor of 32.
$I(M;X|Y) - I(M;X)$ . When  $\rho = 0$ ,  $I(M;X|Y) = I(M;X)$  and therefore  $SI = 0$ . As  $\rho \to 1$ ,  $X - Y \to M$ ; so the total mutual information  $I(M;(X,Y)) \to \infty$ , driven by synergy growing unbounded, while the unique component remains finite at  $I(M;X)$ .
Example 7 (Redundant and synergistic information).
$$
M \sim \mathcal {N} (0, 1) \tag {95}
$$
$$
X = M + N _ {X} \quad N _ {X}, N _ {Y} \sim \mathcal {N} (0, 1), \quad \left(N _ {X}, N _ {Y}\right) \perp M \tag {96}
$$
$$
Y = M + N _ {Y} \quad \operatorname {C o r r} \left(N _ {X}, N _ {Y}\right) = \rho \tag {97}
$$
Derivation of PID values in Example 7. When  $\rho = 1$ , we once again reduce to Example 3 with only redundancy. When  $\rho < 1$ , we cannot infer the PID values using Equation (1) and non-negativity alone, since none of the individual mutual information values (or conditional mutual information values) go to zero. Instead, we can determine the redundancy using the MMI-PID since  $M$  is scalar. Note that  $I(M;X)$  and  $I(M;Y)$  are both equal by symmetry, and thus equal to  $RI$ . This also implies that both  $UI_{X}$  and  $UI_{Y}$  must be equal to zero. As  $\rho$  reduces, the two channels  $X$  and  $Y$  have noisy representations of  $M$  with increasingly independent noise terms. Therefore, their average,  $(X + Y)/2$  will be more informative about  $M$  than either one of them individually, meaning that  $X$  and  $Y$  jointly contain more information than any one individually. This extra information about  $M$  is synergistic, given by  $SI = I(M;X|Y)$ , and increases as  $\rho$  decreases, attaining its maximum possible value at  $\rho = 0$ .
B.2 Diagrams Explaining Examples 8 and 9
Examples 8 and 9 can be understood diagrammatically as shown in Fig. 7(l) and Fig. 7(r), respectively. In both diagrams, we represent the two-dimensional plane describing  $M$ , with axes  $m_{1}$  and  $m_{2}$ . The colored vectors shown on this plane represent  $H_{X}$  and  $H_{Y}$ , i.e., the gain with which  $X$  and  $Y$  represent each value of  $M$ . For example,  $Y_{2}$  captures only  $M_{2}$ , with a gain corresponding to its length. The gains are directly representative of the signal-to-noise ratio (and hence the amount of information) in each variable, since the noise in each variable is i.i.d., with unit variance. In Example 8, the gain in  $X_{1}$  is variable, while in Example 9, the angle at which  $X_{1}$  and  $X_{2}$  sample  $M_{1}$  and  $M_{2}$  is variable.
B.3 Comparison of the  $\sim_G$ - and  $\delta_G$ -PIIDs in Example 8 at Higher Dimensionality