SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marco Celotto
University Medical Center Hamburg-Eppendorf  
Istituto Italiano di Tecnologia  
University of Bologna  
marco.celotto@iit.it
Jan Bim
Datamole, Prague
Alejandro Tlaie
Istituto Italiano di Tecnologia
Vito De Feo
Alessandro Toso
University Medical Center Hamburg-Eppendorf
Stefan M. Lemke
Daniel Chicharro
Hamed Nili
Malte Bieler
University College Kristiania
Tobias H. Donner
Ileana L. Hanganu-Opatz
Andrea Brovelli
Aix Marseille Université CNRS
Stefano Panzeri
University Medical Center Hamburg-Eppendorf  
Istituto Italiano di Tecnologia  
s.panzeri@uke.de
Quantifying the amount, content and direction of communication between brain regions is key to understanding brain function. Traditional methods to analyze brain activity based on the Wiener-Granger causality principle quantify the overall information propagated by neural activity between simultaneously recorded brain regions, but do not reveal the information flow about specific features of interest (such as sensory stimuli). Here, we develop a new information theoretic measure termed Feature-specific Information Transfer (FIT), quantifying how much information about a specific feature flows between two regions. FIT merges the Wiener-Granger causality principle with information-content specificity. We first derive FIT and prove analytically its key properties. We then illustrate and test them with simulations of neural activity, demonstrating that FIT identifies, within the total information propagated between regions, the information that is transmitted about specific features. We then analyze three neural datasets obtained with different recording methods, magneto- and electro-encephalography, and spiking activity, to demonstrate the ability of FIT to uncover the content and direction of information flow between brain regions beyond what can be discerned with traditional analytical methods. FIT can improve our understanding of how brain regions communicate by uncovering previously unaddressed feature-specific information flow.
Cognitive functions, such as perception and action, emerge from the processing and routing of information across brain regions [10, 48, 56, 57, 41]. Methods to study within-brain communication [11, 12, 51] are often based on the Wiener-Granger causality principle, which identifies propagation of information between simultaneously recorded brain regions as the ability to predict the current activity of a putative receiving region from the past activity of a putative sending region, discounting the self-prediction from the past activity of the receiving region [21, 61]. While early measures implementing this principle, such as Granger causality [51], capture only linear interactions, successive information theoretic measures (the closely-related Directed Information [35] and Transfer Entropy [50]) are capable of capturing both linear and nonlinear time-lagged interactions between brain regions [6, 7, 58]. Using such measures has advanced our understanding of brain communication [7, 9, 12, 31, 54, 55, 59, 44]; however, they are designed to capture only the overall information propagated across regions, and are insensitive to the content of information flow. Assessing the content of information flow, not only its presence, would be invaluable to understand how complex brain functions, involving distributed processing and flow of different types of information, arise.
Here, we leverage recent progress in Partial Information Decomposition (PID; [62, 32]) to develop a new non-negative measure (Feature-specific Information Transfer; FIT) that quantifies the directed flow of information about a specific feature of interest between neural populations (Fig. 1A). The PID decomposes the total information that a set of source variables encodes about a specific target variable into components representing shared (redundant) encoding between the variables, unique encoding by some of the variables, or synergistic encoding in the combination of different variables. FIT isolates features-specific information flowing from one region to another by identifying the part of the feature information encoded in the current activity of the receiving region that is shared (redundant) with information present in the past activity of the sending region (because a piece of transmitted information is first found in the sender and then in the receiver) and that is new and unique with respect to the information encoded in the past activity of the receiver (because information already encoded would not have come from the sender).
We first mathematically derive a definition of FIT based on PID. We then use it to demonstrate, on simulated data, that it is specifically sensitive to the flow of information about specific features, correctly discarding feature-unrelated transmission. We then demonstrate that FIT is able to track the feature-specific content and direction of information flow using three different types of simultaneous multi-region brain recordings (electroencephalography - EEG, magnetoencephalography - MEG, and spiking activity). We also address how introducing appropriate null hypotheses and defining conditioned versions of FIT can deal with potential confounding factors, such as the time-lagged encoding of information in two regions without actual communication between them.
2 Defining and Computing Feature-specific Information Transfer (FIT)
We consider two time-series of neural activity  $X$  and  $Y$  simultaneously recorded from two brain regions over several experimental trials.  $X$  and  $Y$  might carry information about a feature  $S$  varying from trial to trial, e.g. a feature of a sensory stimulus or a certain action. The activity measured in each region,  $X$  and  $Y$ , may be any type of brain signal, e.g. the spiking activity of single or multiple neurons, or the aggregate activity of neural populations, such as EEG or MEG. We call  $Y_{pres}$  the activity of  $Y$  at the present time point  $t$ , and  $X_{past}$  and  $Y_{past}$  the past activity of  $X$  and  $Y$  respectively (Fig. 1). Established information theoretic measures such as TE [50] use the Wiener-Granger principle to quantify the overall information propagated from a putative sender  $X$  to a putative receiver  $Y$  as the mutual information  $I$  between the receiver's present neural activity  $Y_{pres}$  and the sender's past activity  $X_{past}$ , conditioned on the receiver's past activity  $Y_{past}$  (Fig. 1):
$$
T E (X \rightarrow Y) = I \left(X _ {\text {p a s t}}; Y _ {\text {p r e s}} \mid Y _ {\text {p a s t}}\right) \tag {1}
$$
(see Supplementary Material, SM1.1 for how TE depends on probabilities of past and present activity). TE captures the overall information propagated across regions but lacks the ability to isolate information flow about specific external variables. To overcome this limitation, here we define FIT, which quantifies the flow of information specifically about a feature  $S$  from a putative sending area  $X$  to a putative receiving area  $Y$  (Fig. 1A). We define FIT using the PID [62]. PID decomposes the joint mutual information  $I(S; \underline{X})$ , that a set of  $N$  source variables  $\underline{X} = (X_1, X_2, \dots, X_N)$  carries
about a target variable  $S$ , into non-negative components called information atoms (see SM1.2). For  $N = 2$ , PID breaks down the joint mutual information  $I(S; X_1, X_2)$  into four atoms: the Shared (or redundant) Information  $SI(S; X_1, X_2)$  that both  $X_1$  and  $X_2$  encode about  $S$ ; the two pieces of Unique Information about  $S$ ,  $UI(S; X_1 \setminus X_2)$  and  $UI(S; X_2 \setminus X_1)$ , provided by one source variable but not by the other; and the Complementary (synergistic) information about  $S$ ,  $CI(S; X_1, X_2)$ , encoded in the combination of  $X_1$  and  $X_2$ . Several measures have been proposed to quantify information atoms [62, 5, 18, 30]. Here we use the measure  $I_{min}$  originally defined in [62], which guarantees non-negative values for information atoms for any  $N$  (see SM1.2).
Using  $I_{min}$ , the Shared Information that  $X_{1}$  and  $X_{2}$  carry about  $S$  is defined as follows:
$$
S I (S: X _ {1}, X _ {2}) = \sum_ {s \in S} p (s) \min  _ {X _ {i} \in \left\{X _ {1}, X _ {2} \right\}} I (S = s; X _ {i}) \tag {2}
$$
where  $I(S = s; X_{i})$  is the specific information that source  $X_{i}$  carries about a specific outcome of the target variable  $s \in S$ , and is defined as:
$$
I (S = s; X _ {i}) = \sum_ {x _ {i} \in X _ {i}} p \left(x _ {i} \mid s\right) \left[ \log \frac {p \left(s \mid x _ {i}\right)}{p (s)} \right] \tag {3}
$$
Intuitively, the shared information computed as in eq. 2 quantifies redundancy as the similarity between  $X_{1}$  and  $X_{2}$  in discriminating individual values of the feature  $S$ . In the general case of  $N$  source variables, information atoms are hierarchically ordered in a lattice structure, and  $I_{min}$  can be used to quantify any atom in the decomposition (including the Unique and Complementary information atoms introduced above for the case  $N = 2$ ; see SM1.2).
Figure 1: Sketch of FIT and TE. (A) TE is the established information-theoretic measure to quantify the overall information propagated between two simultaneously recorded brain regions  $X$  (sender) and  $Y$  (receiver). FIT measures the information flowing from  $X$  to  $Y$  about the stimulus feature S. (B) TE and FIT incorporate content-unspecific and content-specific versions of the Wiener-Granger causality principle. TE is the mutual information between the past activity of  $X$  and the present activity of  $Y$  conditioned on the past of  $Y$ . FIT is the feature information in the present of  $Y$  shared with the past information of  $X$  and unique with respect to the past information of  $Y$ .
We wanted FIT to measure the directed flow of information about  $S$  between  $X$  and  $Y$ , rather than the overall propagation of information measured by TE (Fig. 1A). We thus isolated the information about a feature  $S$  in the past of the sender  $X$  that  $Y$  receives at time  $t$ . Because of the Wiener-Granger causality principle, such information should not have been present in the past activity of the receiver  $Y$ . Therefore, we performed the PID in the space of four variables  $S$ ,  $X_{past}$ ,  $Y_{past}$ , and  $Y_{pres}$  to compute information atoms that combine Shared, Unique and Complementary Information carried by three sources about one target [62]. One natural candidate atom to measure FIT is the information about  $S$  that  $X_{past}$  shares with  $Y_{pres}$  and is unique with respect to  $Y_{past}$ :  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  (Fig. 1B; Fig. S1B). This atom is defined as the difference between the shared information that the two source variables  $X_{past}$  and  $Y_{pres}$  carry about  $S$ , and the shared information that the three source variables  $X_{past}$ ,  $Y_{pres}$  and  $Y_{past}$  carry about  $S$ . Redundancy
can only decrease when adding more sources. Hence by removing the information that is also redundant with  $Y_{past}$ ,  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  quantifies a non-negative component of shared information between  $X_{past}$  and  $Y_{pres}$  about  $S$  that is unique with respect to  $Y_{past}$ . Importantly, using unique information to remove the feature information in  $Y_{past}$  is more conservative than conditioning on  $Y_{past}$  as in TE [27].  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  intuitively captures what we are interested in, and satisfies two desirable mathematical properties: it is upper bounded by the feature information encoded in the past of  $X$  ( $I(S; X_{past})$ ) and in the present of  $Y$  ( $I(S; Y_{pres})$ ). This is because the PID defines redundancy between source variables as sub-components of the information about the target carried by each of the sources (see SM1.3.1). However, the information atom  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  has two undesirable properties. The first is that its value can exceed the total amount of information propagated from  $X$  to  $Y$  (TE). This can happen since the unique information in the PID decomposition is a component of the conditional mutual information about the target. However, the target in  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  is the feature  $S$ , which means that this atom is not constrained to be smaller than the TE, which is independent of  $S$  (see eq. 1 and SM1.3.4). This property is undesirable, because the overall information propagation must be an upper bound to the information transmitted about a specific feature. The second is that by construction (see SM1.3.1) this atom depends on  $X_{past}, Y_{pres}, S$  only through the pairwise marginal distributions  $P(X_{past}, S)$  and  $P(Y_{pres}, S)$ , but not through the marginal distribution  $P(X_{past}, Y_{pres})$ , which implies that this atom by itself cannot identify confounding scenarios where both sender and receiver encode feature information at different times with no transmission taking place (see SM1.3.1).
To address these limitations, following [42] we considered the alternative PID taking  $S$ ,  $Y_{past}$ , and  $X_{past}$  as source variables and  $Y_{pres}$  as a target. In this second PID (Fig. S1B), the atom that intuitively relates to FIT is  $SUI(Y_{pres}:X_{past},S\setminus Y_{past})$ , the information about  $Y_{pres}$  that  $X_{past}$  shares with  $S$  that is unique with respect to  $Y_{past}$ . While being intuitively similar to  $SUI(S:X_{past},Y_{pres}\setminus Y_{past})$ ,  $SUI(Y_{pres}:X_{past},S\setminus Y_{past})$  has  $Y_{pres}$  as target variable and hence is upper bounded by TE (but not by  $I(S;X_{past})$ ) and depends on the pairwise marginal distribution  $P(X_{past},Y_{pres})$  (see SM1.3.2). Thus, this second atom has useful properties that complement those of the first atom. Importantly, Shannon information quantities impose constraints that relate PID atoms across decompositions with different targets. We [42] demonstrated that, for PID with  $N = 2$  sources, these constraints reveal the existence of finer information components shared between similar atoms of different decompositions. Here, we extended this approach (see SM1.3.3) to  $N = 3$  sources and demonstrated that the second atom is the only one in the second PID that has a pairwise algebraic relationship with the first atom, indicating that these atoms share a common, finer information component. Therefore we defined FIT by selecting this finer common component by taking the minimum between these two atoms:
$$
F I T = \min  \left[ S U I \left(S: X _ {p a s t}, Y _ {p r e s} \setminus Y _ {p a s t}\right), S U I \left(Y _ {p r e s}: X _ {p a s t}, S \setminus Y _ {p a s t}\right) \right] \tag {4}
$$
With this definition, FIT is upper bounded by  $I(S; X_{past})$ , by  $I(S; Y_{pres})$  and by  $TE(X \to Y)$ . That FIT satisfies such bounds is essential to interpret it as transmitted information. If FIT could be larger than the feature information encoded by sender  $X$  or receiver  $Y$ , or than the total information transmitted  $(TE(X \to Y))$ , then FIT could not be interpreted as feature information transmitted from  $X$  to  $Y$ . Additionally, FIT depends on the joint distribution  $P(S, X_{past}, Y_{pres})$  through all the pairwise margins  $P(S, X_{past}), P(S, Y_{pres})$ , and  $P(X_{past}, Y_{pres})$ , implying that it can rule out, using appropriate permutation tests, false-communication scenarios in which  $X$  and  $Y$  encode the stimulus independently with a temporal lag, without any within-trial transmission (see SM1.3.4).
Note that the definition of FIT holds when defining present and/or past activity as multidimensional variables, potentially spanning several time points. However, use of multidimensional neural responses requires significantly more data for accurate computation of information. For this reason, following [7, 39, 38], in all computations of TE and FIT we computed the present of  $Y$  at a single time point  $t$  and the past of  $X$  and of  $Y$  at individual time points lagged by a delay  $\delta$ :  $X_{past} = X_{t - \delta}$  and  $Y_{past} = Y_{t - \delta}$ . Note also that in all calculations of FIT and TE, we estimated probabilities from empirical occurrences after discretizing both features and neural activities. SM1.6 reports details of the procedure and Table S1 the number of bins used for each analysis. Simulations of accuracy of these estimates as function of the data size are reported in SM2.5 and Fig. S6.
3 Validation of FIT on simulated data
To test the ability of FIT to measure feature-specific information flow between brain regions, we performed simulations in scenarios of feature-related and feature-unrelated information transfer.
We performed (Fig. 2A-B) a simulation (details in SM2.1) in which the encoded and transmitted stimulus feature  $S$  was a stimulus-intensity integer value (1 to 4). The activity of the sender  $X$  was a two-dimensional variable with one stimulus-feature-informative  $X_{stim}$  and one stimulus-uninformative component  $X_{noise}$ . The stimulus-feature-informative dimension had temporally-localized stimulus-dependent activity from 200 to 250ms and had multiplicative Gaussian noise (similar results were found with additive noise, see SM2.1 and Fig. S3). The stimulus-unrelated component was, at any time point, a zero-mean Gaussian noise. The activity of the receiver  $Y$  was the weighted sum of  $X_{stim}$  and  $X_{noise}$  with a delay  $\delta$ , plus Gaussian noise. The delay  $\delta$  was chosen randomly in each simulation repetition in the range 40-60ms. Here and in all simulations, we averaged information values across simulation repetitions we determined their significance via non-parametric permutation tests [33, 7, 19, 29]. For TE, we permuted  $X$  across all trials to test for the presence of significant within-trial transmission between  $X$  and  $Y$  [7, 33]. For FIT, we conducted two different permutation tests: one for the presence of feature information in  $X$  and  $Y$  (shuffling  $S$  across trials), and another for the contribution of within-trial correlations between  $X$  and  $Y$  to the transmission of  $S$  (shuffling  $X$  across trials at fixed stimulus). We set the threshold for FIT significance as the 99th percentile of the element-wise maximum between the two permuted distributions (see SM1.7).
Figure 2: Testing FIT on simulated data. (A) FIT and TE as function of stimulus-feature-related  $(w_{stim})$  and -unrelated  $(w_{noise})$  transmission strength. * indicate significant values ( $p < 0.01$ , permutation test) for the considered parameter set. (B) Dynamics of FIT and TE in a simulation with time-localized stimulus-feature-information transmission. Red area shows the window of stimulus-feature-related information transfer. Results plot mean (lines) and SEM (shaded area) across 50 simulations (2000 trials each). (C) Different neural encoding functions used for the simulations in panels D-E.  $\delta$ : separation of responses to different features;  $\sigma$ : Gaussian noise SD. (D) Sketch of simultaneous stimulus feature information profiles. Different types of information content are color-coded. (E) FIT in the  $X \to Y$  (cyan) and  $Y \to X$  (grey) directions as a function of SNR  $\delta/\sigma$  (plot in log scale). Results plot mean (lines) and SEM (shaded area) across 100 simulations (2000 trials each). Yellow dots in B, E show points with significant FIT ( $p < 0.01$ , permutation test).
We investigated how FIT and TE from  $X$  to  $Y$  depended on the amount of stimulus-feature-related transmission (increased by increasing  $w_{stim}$ ) and of -unrelated transmission (increased by increasing  $w_{noise}$ ). We report values at the first time point in which information in  $Y$  was received from  $X$ , but similar results hold for later time points. Both FIT and TE increased when increasing  $w_{stim}$  (Fig. 2A). However, TE increased with  $w_{noise}$  (Fig. 2A), as expected from a measure that captures the overall information propagation. In contrast, FIT decreased when increasing  $w_{noise}$ , indicating that FIT specifically captures the flow of information about the considered feature.
We then investigated how well TE and FIT temporally localize the stimulus-feature-related information transmitted from  $X$  to  $Y$  (Fig. 2B). We simulated a case in which stimulus-feature-related information was transmitted from  $X$  to  $Y$  only in a specific window ([240, 310]ms) and computed FIT and TE at each time point (see SM2.1 for details). FIT was significant only in the time window in which  $Y$  received the stimulus information from  $X$ . In contrast, TE was significant at any time point, reflecting that noise was transmitted from  $X$  to  $Y$  throughout the whole simulation time.
Importantly, FIT can detect feature-specific information flow even when information is encoded in the sender and receiver with an overlapping timecourse (see SM2.2 for details). To illustrate this, we simulated the activity of two regions  $X$  and  $Y$  encoding an integer stimulus feature  $S$  with the same time course of stimulus-feature information (Fig. 2D), but with feature specific transmission taking place only from  $X$  to  $Y$ . Because FIT could correctly detect that the format of information representation of  $S$  in the present of  $Y$  was equal to that of the past of  $X$  but different to that of the past of  $Y$  (Fig. 2D), it could correctly detect that feature information flows from  $X$  to  $Y$  (Fig. 2E).
We also performed simulations to investigate whether the non-parametric permutation test described above can correctly rule out as non-significant feature-specific transmission the scenario in which  $X$  and  $Y$  independently encode  $S$  without actual communication occurring between them. We simulated a scenario in which feature information was encoded with a temporal lag in  $X$  and  $Y$ , with no transmission from  $X$  to  $Y$ . We found that the resulting values were always non-significant (see SM2.6.1 and Fig. S7C) when tested against a surrogate null-hypothesis distribution (pairing  $X$  and  $Y$  in randomly permuted trials with the same feature) that destroy the within-trial communication between  $X$  and  $Y$  without changing the feature information encoded in  $X$  and  $Y$  (see SM1.7). Importantly, this null hypothesis also ruled out false communication scenarios where the measured FIT and TE were only due to the presence of instantaneous mixing of sources (see SM 2.7).
Finally, we addressed how to remove the confounding effect of transmission of feature information to  $Y$  not from  $X$  but from a third region  $Z$ . In Granger causality or TE analyses, this is addressed conditioning the measures on  $Z$  [1, 37]. In an analogous way, we developed a conditioned version of FIT, called cFIT (see SM1.4), which measures the feature information transmitted from  $X$  to  $Y$  that is unique with respect to the past activity of a third region  $Z$ . We tested its performance in simulations in which both  $X$  and  $Z$  transmitted feature information to  $Y$  and found that cFIT reliably estimated the unique contribution of  $X$  in transmitting feature information to  $Y$ , beyond what was transmitted by  $Z$  (see SM2.6.2 and Fig. S7D).
4 Analysis of real neural data
We assessed how well FIT detects direction and specificity of information transfer in real neural data.
4.1 Flow of stimulus and choice information across the human visual system