SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For completeness, we also present a comparison between the  $\sim_{G^{-}}$  and  $\delta_G$ -PIDs for Example 8 at a higher dimensionality of  $d = 64$ . The results can be found in Fig. 8, and show that both PIDs scale proportionally. We use a gain of  $\alpha = 0.99$  in place of  $\alpha = 1$ , since a gain of unity causes certain matrices to become ill-conditioned.
Although the  $\delta_G$ -PID does not obey the additivity property in general (which is why it does not agree with the  $\sim_G$ -PID at gains of  $\alpha > 1$ ), it appears to double along with the  $\sim_G$ -PID (i.e., additivity across identical independent copies appears to hold, at least in this specific example).
Figure 9: A plot showing the time taken to compute the  $\sim_{G}$ -PID at different dimensionalities in Example 10. This plot extends what is shown in the right-most plot of Fig. 3 for the  $\sim_{G}$ -PID.
Absolute error in the  $\sim_{G}$ -PID over increasing dimensionality
Relative error in the  $\sim_G$ -PID over increasing dimensionality
Figure 10: Absolute (top) and relative (bottom) errors in computed PID values from Example 10.
B.4 Run Time Analysis of the  $\sim_{G}$ -PID in Example 10
In Fig. 9, we present a complete picture of the time taken to compute the  $\sim_{G}$ -PID, as a function of dimensionality  $d$ , in Example 10. Note that we cannot compare with the  $\delta_{G}$ -PID beyond a dimensionality of  $d = 64$ , because the computation for the  $\delta_{G}$ -PID failed in our setup for  $d > 64$ . Neither the  $\delta_{G}$ -PID nor the  $\sim_{G}$ -PID were carefully profiled and optimized, therefore, the runtime analysis in both Fig. 9 and Fig. 3 should be considered preliminary. We leave a more thorough comparison of the run time of both these methods to future work.
B.5 Absolute and Relative Errors in Example 10
Figure 10 shows how the absolute and relative errors in PID values scale with increasing dimensionality in Example 10. The absolute errors increase in proportion to dimensionality, starting under  $10^{-7}$
at  $d = 2$  and remaining under  $10^{-4}$  at  $d = 1024$ . The relative errors are all roughly constant, and remain under  $10^{-6}$ .
C Supplementary Material for Section 5
C.1 Implementation Details for Bias-correction
We use a number of different setups based on sampling from random connectivity matrices for bias correction in Section 5. All of these setups assume that  $d_{X} = d_{Y}$ .
The both-unique, fully-redundant and high-synergy setups have the following in common:
$$
\Sigma_ {M} = I \tag {98}
$$
$$
\Sigma_ {X \mid M} = I \tag {99}
$$
$$
\Sigma_ {Y \mid M} = I \tag {100}
$$
$$
\Sigma_ {M X Y} = \left[ \begin{array}{c c c} I & H _ {X} ^ {\top} & H _ {Y} ^ {\top} \\ H _ {X} & H _ {X} H _ {X} ^ {\top} + \Sigma_ {X \mid M} & H _ {X} H _ {Y} ^ {\top} + \Sigma_ {W} \\ H _ {X} & H _ {Y} H _ {X} ^ {\top} + \Sigma_ {W} ^ {\top} & H _ {Y} H _ {Y} ^ {\top} + \Sigma_ {Y \mid M} \end{array} \right]. \tag {101}
$$
Also, the elements of  $H_{X}$  are either zero or one,  $H_{X}(i,j)\sim \mathrm{i.i.d.}\mathrm{Ber}(0.1)$ . These three setups differ in their definitions of  $H_{Y}$  (the channel gain from  $M$  to  $Y$ ) and  $\Sigma_W$  (which controls the extent of correlation between  $X$  and  $Y$ ).
The both-unique setup draws  $H_{Y}(i,j) \sim \mathrm{i.i.d.}$  Ber(0.1), with all elements of  $H_{Y}$  independent of the elements of  $H_{X}$ , and sets  $\Sigma_{W} = 0$ .
The fully-redundant setup is similar to Example 7, by setting  $H_{Y} = H_{X}$  and  $\Sigma_W = 0.9I$  (note that  $\Sigma_W$  is square, since  $d_X = d_Y$ ). By keeping  $\Sigma_W$  close to the identity matrix, we are effectively in the regime with high correlation  $\rho$  in Example 7. This allows us to come close to emulating Example 3, without suffering from the issue of non-invertibility of  $\Sigma_{XY|M}$ , mentioned in App. B.
The high-synergy setup is similar to Example 6, by setting  $H_{Y} = 0$  and  $\Sigma_W = 0.8I$ . As with the fully-redundant setup, by keeping  $\Sigma_W$  close to the identity matrix, we are in the high-  $\rho$  regime. This allows us to come close to emulating Example 4, while not making the synergy or the total mutual information infinite.
The zero-synergy setup is similar to Example 5, and uses the following setup:
$$
\Sigma_ {M} = \Sigma_ {X | M} = I \tag {102}
$$
$$
\Sigma_ {X} = H _ {X} H _ {X} ^ {\mathsf {T}} + \Sigma_ {X | M} \tag {103}
$$
$$
\Sigma_ {Y \mid X} = I \tag {104}
$$
$$
\Sigma_ {M X Y} = \left[ \begin{array}{c c c} I & H _ {X} ^ {\top} & H _ {Y} ^ {\top} \\ H _ {X} & \Sigma_ {X} & \Sigma_ {X} H _ {Y} ^ {\prime \top} \\ H _ {X} & H _ {Y} ^ {\prime} \Sigma_ {X} ^ {\top} & H _ {Y} ^ {\prime} \Sigma_ {X} H _ {Y} ^ {\prime \top} + \Sigma_ {Y | X} \end{array} \right]. \tag {105}
$$
Here,  $H_{X}(i,j) \sim \mathrm{i.i.d.} \operatorname{Ber}(0.1)$ , while  $H_{Y} = H_{Y}^{\prime}H_{X}$ , with  $H_{Y}^{\prime}(i,j) \sim \mathrm{i.i.d.} \operatorname{Ber}(0.1)$ ,  $H_{Y}^{\prime} \perp H_{X}$ . Defined this way,  $M—X—Y$  form a Markov chain, ensuring that  $I(M;X|Y) = 0$ , so that  $SI = 0$  (Refer equation (63)).
The bit-of-all setup is a combination of equal parts of the high-synergy and zero-synergy setups. The variables  $X$  and  $Y$  are swapped in the zero-synergy setup, so that both  $X$  and  $Y$  can have some unique information.
Remark 3 (Rectification). In practice, we observed that the bias correction procedure prescribed in Definition 4 could lead to negative values for certain PID quantities. This occurred because the bias-corrected union information was not guaranteed to satisfy certain bounds, which we enforce below. To prevent the occurrence of negative PID values after bias-correction, we require a form of rectification:
$$
\left. \widetilde {I _ {G} ^ {\cup}} \right| _ {\text {r e c t}} ^ {(1)} := \max  \left\{\left. \widetilde {I _ {G} ^ {\cup}} \right| _ {\text {b i a s - c o r r}}, \left. \hat {I} (M; X) \right| _ {\text {b i a s - c o r r}}, \left. \hat {I} (M; Y) \right| _ {\text {b i a s - c o r r}} \right\} \tag {106}
$$
$$
\left. \widetilde {I _ {G} ^ {\cup}} \right| _ {\text {r e c t}} ^ {(2)} := \min  \left\{\left. \widetilde {I _ {G} ^ {\cup}} \right| _ {\text {r e c t}} ^ {(1)}, \left. \hat {I} (M; X) \right| _ {\text {b i a s - c o r r}} + \left. \hat {I} (M; Y) \right| _ {\text {b i a s - c o r r}}, \left. \hat {I} (M; (X, Y)) \right| _ {\text {b i a s - c o r r}} \right\}, \tag {107}
$$
where  $\hat{I}(\cdot)|_{\mathrm{bias - corr}}$  represents a bias-corrected mutual information estimate. After the second rectification equation above, the union information is bounded from below by the individual (bias-corrected) mutual information values, and bounded from above by the sum of the individual mutual information values, and by the total mutual information.
C.2 Bias-correction Performance in Additional Setups and at Higher Dimensionality
Plots showing bias correction performance for all setups described in App. C.1 are shown in Fig. 11 for 10-dimensional, and in Fig. 12 for 20-dimensional  $M$ ,  $X$  and  $Y$ .
Of the setups we examine, only the case with both  $X$  and  $Y$  having purely unique information appears to have somewhat poor performance, where our bias-correction method appears to over-correct the bias in unique information, while insufficiently correcting the bias in redundancy and synergy.
C.3 A Preliminary Analysis of the Variance of PID Estimates
In Figures 13 and 14, we present a preliminary analysis of the variance of our PID estimates using bootstrap. The figures represent the true distribution of the PID estimates over multiple sample draws, or over multiple bootstrap sample draws, in the form of box plots. In what follows, we colloquially refer to these box plots as "confidence intervals". The true "confidence intervals" were estimated using 100 runs of bias-corrected PID estimates, i.e., by drawing 100 different samples, each of size  $n$ . The bootstrap "confidence intervals" were estimated using 100 bootstrap samples that were resampled from a single randomly drawn sample of size  $n$ .
When correcting for bias in the PID estimates on bootstrap samples, we use the number of unique data points in each bootstrap sample in place of  $n$  (refer Corollary 4), rather than the total sample size. This leads to more stable bootstrap-PID estimates.
The quality of the bootstrap "confidence interval" is affected greatly by the quality of the individual sample used for bootstrap resampling. Nevertheless, we observe a reasonable degree of qualitative agreement between the true "confidence interval" and the bootstrap "confidence interval", particularly as the sample size increases. Future work will assess confidence intervals with greater care, using well-defined metrics, and assess how well these confidence intervals are calibrated.
D Supplementary Material for Section 6
D.1 Details Regarding the Multivariate Poisson Spike-count Simulation
We follow our previous paper [12], where this analysis was first presented. In this simulation,  $M$  is two-dimensional, consisting of two independent and identically distributed Poisson random variables,  $M_1$  and  $M_2$ .  $X$  and  $Y$  are each generated through a linear combination of Binomially thinning  $M_1$  and  $M_2$ , along with some Poisson noise:
$$
M _ {1}, M _ {2} \sim \text {P o i s s} (2) \tag {108}
$$
$$
X \sim \operatorname {B i n o m} \left(M _ {1}, \alpha\right) + \operatorname {B i n o m} \left(M _ {2}, 0. 5\right) + \operatorname {P o i s s} (1) \tag {109}
$$
$$
Y \sim \operatorname {B i n o m} \left(M _ {1}, 0. 5\right) + \operatorname {B i n o m} \left(M _ {2}, 0. 5\right) + \operatorname {P o i s s} (1) \tag {110}
$$
While our results in Fig. 5 show that our  $\sim_G$ -PID estimator comes closest to obtaining the same values as the estimator of Banerjee et al. [20], several remarks are warranted:
1. The method of Banerjee et al. [20] estimates the  $\sim$ -PID and not the  $\delta$ -PID. Since there is no discrete estimator for the  $\delta$ -PID, it is entirely possible that the  $\delta_G$ -PID is equally (or more) accurate with respect to its own true value. That is, the difference between the  $\delta_G$ -PID and the Banerjee et al. [20] estimator is entirely due to a difference in definitions;
2. We stated in the introduction that the  $\delta_G$ -PID estimates an approximate upper bound, whereas our  $\sim_G$ -PID estimates an exact upper bound. The difference due to the approximate nature of the  $\delta_G$ -PID upper bound is also not captured here, nor in Examples 8 and 9. In all these examples, we cannot tell how much of the difference is due to the difference in definitions and the approximate nature of the estimators.
Figure 11: Bias correction for various setups described in App. C.1 with  $d \coloneqq d_{M} = d_{X} = d_{Y} = 10$ .
Figure 12: Bias correction for various setups described in App. C.1 with  $d \coloneqq d_{M} = d_{X} = d_{Y} = 20$ .
Figure 13: Bootstrap "confidence intervals" for various setups described in App. C.1 with  $d \coloneqq d_M = d_X = d_Y = 10$ . Note that these are not true confidence intervals, but box-plot representations of the true variance of the estimator (over 100 runs) and the bootstrap estimate of the estimate's variance (from 100 bootstrap resamplings of a single random sample).
Figure 14: Bootstrap "confidence intervals" for various setups described in App. C.1 with  $d \coloneqq d_M = d_X = d_Y = 20$ . Note that these are not true confidence intervals, but box-plot representations of the true variance of the estimator (over 100 runs) and the bootstrap estimate of the estimate's variance (from 100 bootstrap resamplings of a single random sample).
3. Finally, we do not know how accurate the "ground truth" itself is, since this is also assessed using the estimator of Banerjee et al. [20]. Differences between the various estimators could also be due to a poor estimate produced by the discrete PID estimator.
D.2 Additional Simulations on Non-Gaussian Data
In order to evaluate how the  $\sim_{G}$ -PID performs on a greater variety of non-Gaussian distributions, we considered three more setups similar to the one described in App. D.1.
The first additional setup is a multivariate Binomial distribution:
$$
M _ {1}, M _ {2} \sim \operatorname {B i n o m} (4, 0. 5) \tag {111}
$$
$$
X \sim \operatorname {B i n o m} \left(M _ {1}, \alpha\right) + \operatorname {B i n o m} \left(M _ {2}, 0. 5\right) + \operatorname {B i n o m} (2, 0. 5) \tag {112}
$$
$$
Y \sim \operatorname {B i n o m} \left(M _ {1}, 0. 5\right) + \operatorname {B i n o m} \left(M _ {2}, 0. 5\right) + \operatorname {B i n o m} (2, 0. 5) \tag {113}
$$
This setup has the same mean as the multivariate Poisson distribution, but has a smaller variance and is therefore more peaked, further from zero. The multivariate Binomial distribution is closer to Gaussian than the multivariate Poisson, and indeed, we see good agreement between the  $\sim_G$ -PID and the "ground truth", as assessed by the estimator of Banerjee et al. [20] (see Fig. 15, top panels).
The next two setups are "zero-inflated" versions of the multivariate Poisson and Binomial distributions, which are produced by passing each of the aforementioned variables through a "Z-channel", with zero-out probability 0.3. That is,
$$
M _ {1} ^ {\prime} = M _ {1} \times Z _ {M _ {1}} \quad X ^ {\prime} = X \times Z _ {X} \tag {114}
$$
$$
M _ {2} ^ {\prime} = M _ {2} \times Z _ {M _ {2}} \quad Y ^ {\prime} = Y \times Z _ {Y} \tag {115}
$$
where  $Z_{i} \sim$  i.i.d.  $\mathrm{Ber}(0.7)$ ,  $i \in \{M_1, M_2, X, Y\}$ . The primed versions of these variables have an additional probability mass at zero, with a probability of 0.3. This is said to better reflect the statistics of Calcium imaging [45], although we use a different base distribution than the one in the cited paper. The zero-inflated versions of the Poisson and Binomial distributions are bimodal, with one of the peaks at zero, and the Binomial version having more well separated peaks. For both zero-inflated versions, the  $\sim_G$ -PID performs poorly at recovering the absolute  $\sim$ -PID values (Fig. 15, middle-left and bottom-left panels). However, the relative PID values (normalized by the total mutual information) are closer to their true values (Fig. 15, middle-right and bottom-right panels), providing hope that a better mutual information estimate (e.g., [46, 47]) could correct the absolute PID values to some extent.
Further validation on non-Gaussian data is necessary to understand the impact of non-Gaussianity on  $\sim_{G}$ -PID estimates. However, a more extensive evaluation of non-Gaussian distributions (especially at higher dimensionalities) is made difficult by the unavailability of ground truth. The "ground truth" in these examples is obtained using the discrete  $\sim$ -PID estimator of Banerjee et al. [20] (which we assume is more accurate for discrete variables). Our method provides a better result than the  $\delta$ -PID and the MMI-PID in Fig. 5, at least part of which is probably due to the difference in definitions. Nevertheless, we believe these examples demonstrate that applying our method to non-Gaussian data need not be ruled out, although care should always be taken in interpreting results.
D.3 Implementation Details of the Analysis Pipeline
The Visual Behavior Neuropixels data was analyzed as follows:
1. We selected mice that had at least 20 units in each brain region of interest. Only mice with both familiar and novel sessions were selected.
2. From each region we selected units of 'good' quality, with SNR at least 1, and with fewer than 1 inter-spike interval violations.
3. Trials were aligned to the start of each stimulus flash, and spikes were counted in bins of  $50~\mathrm{ms}$  between 0 and  $250~\mathrm{ms}$  after stimulus onset (0-50 ms, 50-100 ms, etc.).
4. Trials corresponding to a non-change flash were defined as those that occurred between 4 and 10 flashes after the start of a behavioral trial, such that the image remained the same as the original image in this behavioral trial. Flashes corresponding to an omission, flashes after an omission, and flashes during which the animal licked, were all removed. Only flashes that occurred while
PID values (left) and PID normalized by  $I(M; (X, Y))$  (right) in a multivariate Binomial spike-count simulation
PID values (left) and PID normalized by  $I(M; (X, Y))$  (right) in a zero-inflated Poisson spike-count simulation
PID values (left) and PID normalized by  $I(M; (X, Y))$  (right) in a zero-inflated Binomial spike-count simulation
Figure 15: PID values and normalized PID values for three different non-Gaussian distributions, as described in App. D.2. The top, middle and bottom panels are respectively for a multivariate Binomial, a zero-inflated Poisson, and a zero-inflated Binomial distribution. The left column represents the PID values in bits, while the right column represents the PID values normalized by the total mutual information. All values are plotted as a function of the gain from  $M_{1}$  to  $X(\alpha)$  on the  $x$ -axis. An explanation of the results can be found in the text.
Figure 16: Redundancy about VISp activity between VIS1 and VISal, as a function of time, for flashes corresponding to an image change (blue) and flashes corresponding to a non-change (orange). Data points are across 41 mice. The plot on the left shows the raw redundancy in bits, while the plot on the right shows the redundancy normalized by the total mutual information.
Figure 17: Redundancy about VISp activity between VIS1 and VISam, as a function of time, for flashes corresponding to an image change (blue) and flashes corresponding to a non-change (orange). Data points are across 38 mice. The plot on the left shows the raw redundancy in bits, while the plot on the right shows the redundancy normalized by the total mutual information.
PID values in bits;  $M = \mathrm{VISp}$ ,  $X = \mathrm{VISl}$ ,  $Y = \mathrm{VISal}$  (top 20 PCs)
PID fraction of total mutual information;  $M = \mathrm{VISp}$ ,  $X = \mathrm{VISl}$ ,  $Y = \mathrm{VISal}$  (top 20 PCs)
Figure 18: All PID components—about VISp activity, between VISl and VISal—in bits (top), and as a fraction of total mutual information (bottom), at various times after stimulus onset, for change and non-change flashes. Here, the label  $I_{MXY}$  in the x-axis of the top plot refers to  $I(M; (X, Y))$ . These plots show that redundancy is the primary driver of mutual information.
Figure 19: All PID components—about VISp activity, between VIS1 and VISam—in bits (top), and as a fraction of total mutual information (bottom), at various times after stimulus onset, for change and non-change flashes. Here, the label  $I_{MXY}$  in the x-axis of the top plot refers to  $I(M; (X, Y))$ . These plots show that redundancy is the primary driver of mutual information.
the animal was engaged (as measured by an average reward rate of at least 2 rewards/min) were selected.
5. Trials corresponding to a change flash were defined as those during which an image change occurred, and the animal was engaged (as above).
6. The top 10 or 20 principal components of neural activity were selected at each time bin, and for each brain region under consideration. Principal component analysis was carried out using the Scikit-learn [48] package in Python.
7.  $\sim_G$ -PID estimates were computed on the covariance matrix between principal components across regions, for each time bin.
8. Data were aggregated across 40 or 41 mice for the figures with VISal, and across 38 mice for the figures with VISam.
9. Statistical significance was assessed using a two-sided unpaired Mann-Whitney-Wilcoxon test.
D.4 Additional Results
Fig. 16 shows the redundancy about VISp activity, between VISl and VISal. It is identical to Fig. 6, except that it uses only the top-10 PCA components in each of the three visual cortical regions. This figure shows that the result presented in Fig. 6 gracefully degrades with a reduction in the number of PCA components. When using fewer PCA components, the increase in redundancy on change trials is still statistically significant, but the effect size is not as large, and not as sustained, as it is in Fig. 6.
Fig. 17 shows the redundancy about VISp activity between VIS1 and VISam, the latter of which is a different higher-order cortical region (see, e.g., [49]). This figure shows that the result in Fig. 6 can also be seen for another choice of higher-order cortical region (i.e., VISam rather than VISal).
Figures 18 and 19 show all PID components, not just the redundancy, for the same settings as in Figures 6 and 17 respectively. These show that redundancy is the dominant partial information component, and appears to be the main driver of changes in the overall mutual information. This justifies why we include only a plot of redundancy in Figs. 6, 16 and 17.
D.5 The Necessity for PCA, and Maximizing Dimensionality
In our analysis of real neural data in Sec. 6, we used principal components analysis to reduce dimensionality, despite the fact that our method could scale to much larger dimensionalities. As mentioned briefly in Sec. 6, this was due to the limited number of trials we had. For many mice, there were a larger number of neurons in each region than trials with which to compute the covariance matrix. In other words, directly computing the covariance of the neural activity would have resulted in a covariance matrix that was rank-deficient. We used PCA to reduce dimensionality and ensure that we obtained a reasonable estimate of the covariance matrix, and to minimize the error in our PID estimate, which would naturally be higher with a greater number of PCA dimensions.
For completeness, we analyze here what would happen if we used the maximum possible number of PCA components for each mouse, while ensuring that the covariance matrix is not rank deficient. We also ensure that we had the same number of components across all regions and across both change and non-change conditions (within each mouse), so as to perform a fair comparison. This gives us on average  $53 \pm 16$  PCA components, with a minimum of 22 and a maximum of 84 PCA components. We find that the results of higher and more sustained redundancy on change flashes continue to hold when using the maximum possible number of PCA components for each mouse (see Fig. 20). However, as expected, there is much greater variance across mice, possibly due to variability in the number of PCA components chosen across mice, and due to greater errors in our PID estimates. For the main results presented in Fig. 6, in order to be consistent across mice, we rounded down from the minimum and used the top 20 PCA components for all mice.
D.6 Differences between Change and Non-change Conditions are not an Artifact of Bias-correction
The number of trials corresponding to change flashes is much smaller than the number of trials corresponding to non-change flashes. Accordingly, the sample size used to estimate the covariance matrix is different in each of the two conditions. Bias correction was performed using the appropriate
Figure 20: Redundancy about VISp activity between VIS1 and VISal, using the maximum possible number of PCA components for each mouse. Data points are across 42 mice, however, not all mice are represented at all points in time and for both conditions, since individual data points may not have converged due to ill-conditioned matrices. The plot on the left shows the raw redundancy in bits, while the plot on the right shows the redundancy normalized by the total mutual information. The increase in redundancy on change trials is still statistically significant, however, the errorbars have a greater spread for reasons described in the text.