SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} T E (X \rightarrow Y) = I \left(X _ {p a s t}; Y _ {p r e s} \mid Y _ {p a s t}\right) \\ = \sum_ {x _ {p a s t}, y _ {p r e s}, y _ {p a s t}} p \left(x _ {p a s t}, y _ {p r e s}, y _ {p a s t}\right) \log \frac {p \left(x _ {p a s t} , y _ {p r e s} \mid y _ {p a s t}\right)}{p \left(x _ {p a s t} \mid y _ {p a s t}\right) p \left(y _ {p r e s} \mid y _ {p a s t}\right)} \tag {S2} \\ \end{array}
$$
where  $p(x_{past}, y_{pres}, y_{past})$  is the joint probability, sampled across experimental trials, of observing the joint occurrence of  $x_{past} \in X_{past}$ ,  $y_{pres} \in Y_{pres}$ , and  $y_{past} \in Y_{past}$ , and the sum spans all possible events. Importantly, TE does not depend on the stimulus feature  $S$  and thus cannot tell how much of the overall information being transmitted from  $X$  to  $Y$  is about  $S$  or about other factors unrelated to  $S$ .
SM1.2 Elements of the PID theory
PID was introduced first in Ref [38] and is a very active field of research [21]. To make our paper self-standing, here we briefly summarize the basic concepts of PID that are most needed for our reasoning and derivations.
In the general case of  $N$  source variables  $\underline{X} = (X_1, \ldots, X_N)$ , PID dissects the joint mutual information that the source variables jointly carry about a target variable  $T$ ,  $I(\underline{X}; T)$ , into non-overlapping pieces of redundant, unique, and synergistic information. Let  $A_1, \ldots, A_M$  be all the non-empty and potentially overlapping subsets of  $\underline{X}$ , that we call sources in the following. PID considers the collections of sources  $\alpha \in P_1(P_1(\underline{X}))$ , where  $P_1(\underline{X})$  denotes the set of all non-empty subsets of  $\underline{X}$ . That is, a collection  $\alpha$  corresponds to a non-empty subset of sources, namely to a non-empty subset of non-empty subsets of source variables. In the following, for brevity, we will call collections the collections of sources. Collections  $\alpha$  are indicated using a bracketed notation (e.g.,  $\alpha = \{X_1X_2\} \{X_1X_3\}$  represents the collection of the two overlapping sources  $\{X_1X_2\}$  and  $\{X_1X_3\}$ ). Importantly, pieces of unique and synergistic information can be defined and computed algorithmically once the redundant information is identified and computed. Thus, in what follows we focus principally on defining and computing redundancies. For each  $\alpha$ , PID defines the amount of information about  $T$  that is redundant between all sources in the collection:  $I_{\cap}(T; \alpha)$ . Conceptually, the redundancy of any collection  $\alpha$  for which a source  $A_i \in \alpha$  is a subset of another source  $A_j \in \alpha$  ( $i \neq j$ ) should be equal to the redundancy of the same collection after removing the superset  $A_j$  [38]. Therefore, the collections of interest to compute  $I_{\cap}(T; \alpha)$  are only those for which no source is a superset of any other, and hence removing any source  $A_i \in \alpha$  could potentially reduce the redundancy. These collections form a domain called  $\mathcal{A}(\underline{X})$ :
$$
\mathcal {A} (\underline {{X}}) = \left\{\alpha \in P _ {1} \left(P _ {1} (\underline {{X}})\right): \forall A _ {i}, A _ {j} \in \alpha , A _ {i} \not \subseteq A _ {j} \right\} \tag {S3}
$$
It is possible to define a partial order over the collections of  $\mathcal{A}(\underline{X})$ . A collection  $\alpha$  precedes another collection  $\beta$  if for each source B in  $\beta$  it exists a source A in  $\alpha$  that is a subset of B, formally:
$$
\forall \alpha , \beta \in \mathcal {A} (\underline {{X}}) (\alpha \preceq \beta \Leftrightarrow \forall B \in \beta \exists A \in \alpha | A \subseteq B) \tag {S4}
$$
Applying the order relationship in eq. S4 to the elements of  $\mathcal{A}(\underline{X})$  produces redundancy lattices, in which a collection that succeeds  $\alpha$  provides at least as much redundant information about  $T$  as  $\alpha$  [38] (see Fig. S1A,B for the lattices for  $N = 2$  and  $N = 3$  source variables). PID allows quantifying the amount of redundant information  $I_{\partial}(T;\alpha)$  that a specific collection  $\alpha$  contributes to the joint mutual information about T, and that is not already redundant in any collections preceding  $\alpha$  (in the following, we will call  $I_{\partial}(T;\alpha)$  the information atom provided by collection  $\alpha$ ).  $I_{\partial}(T;\alpha)$  is implicitly defined by the following relationship [38]:
$$
I _ {\cap} (T; \alpha) = \sum_ {\beta \preceq \alpha} I _ {\partial} (T; \beta) \tag {S5}
$$
Due to the so-called self-redundancy axiom of the PID theory [38], if an individual source  $A_{i}$  appears in collection  $\alpha$ , the redundancy computed on collection  $\alpha$  is equal to the mutual information between all source variables in  $A_{i}$  and the target variable  $T$ :
$$
I _ {\cap} (T; \alpha) = I _ {\cap} (T; \{A _ {i} \}) = I (T; A _ {i}) \tag {S6}
$$
By combining eqs. S5 and S6 we can write Shannon information theoretic quantities as the sum of partial information atoms:
$$
I (T; A _ {i}) = \sum_ {\beta \preceq A _ {i}} I _ {\partial} (T; \beta) \tag {S7}
$$
Eq. S7 will be fundamental to provide upper bounds for FIT in terms of Shannon information quantities. When applied to the trivariate system  $(S, X_1, X_2)$ , taking  $S$  as target and  $(X_1, X_2)$  as source variables, eq. S7 provides the decomposition of the joint mutual information  $I(S; X, Y)$  that we discussed in the main text:
$$
I (S; X _ {1}, X _ {2}) = I _ {\partial} (S; \{X _ {1} \} \{X _ {2} \}) + I _ {\partial} (S; \{X _ {1} \}) + I _ {\partial} (S; \{X _ {2} \}) + I _ {\partial} (S; \{X _ {1} X _ {2} \}) \tag {S8}
$$
where in the main text we called  $I_{\partial}(S; \{X_1\} \{X_2\}) = SI(S : X_1, X_2)$ ,  $I_{\partial}(S; \{X_1\}) = UI(S : X_1 \setminus X_2)$ ,  $I_{\partial}(S; \{X_2\}) = UI(S : X_2 \setminus X_1)$ ,  $I_{\partial}(S; \{X_1X_2\}) = CI(S : X_1, X_2)$  to improve clarity
for readers not familiar with PID.  $SI$  is shorthand for Shared (that is, redundant) Information;  $UI$  is short-hand for Unique information;  $CI$  is shorthand for Complementary (that is, synergistic) information.
Thus far we covered elements of PID theory that hold for a generic redundancy measure  $I_{\cap}$ , but did not discuss how to compute  $I_{\cap}(T; \alpha)$  for a specific collection  $\alpha$ . Several measures of redundant information have been proposed [16; 3; 11; 20], in this work we use the original measure  $I_{min}$  from Williams and Beer [38], as it has the fundamental property of being non-negative for any information atoms for any number  $N$  of source variables (not only for  $N = 2$ ) (for a proof, see Appendix D of Ref [38]). The redundant information  $I_{min}$  for a collection  $\alpha$  is defined as follows:
$$
I _ {\min } (T; \alpha) = \sum_ {t \in T} p (t) \min  _ {A _ {i} \in \alpha} I (T = t; A _ {i}) \tag {S9}
$$
where  $I(T = t; A_i)$  is the specific information that source  $A_i$  carries about a specific outcome of the target variable  $t \in T$ , and is defined as:
$$
I (T = t; A) = \sum_ {a} p (a | t) \left[ \log \frac {p (t | a)}{p (t)} \right] \tag {S10}
$$
Intuitively,  $I_{min}$  quantifies redundancy as the overlap between the sources in the distributions of specific information across individual values of target variable. This corresponds to quantifying the degree to which all sources in collection  $\alpha$  are similarly discriminative about individual values of the target. We decided to use  $I_{min}$  because of its advantages in terms of being defined for an arbitrary number of source variables  $N$  (something that is needed because FIT is defined in terms of  $N = 3$  source variables and cFIT in terms of  $N = 4$  source variables) and being non-negative for all atoms (which is important to guarantee that FIT is interpretable as a measure of information transmission). Importantly, similarly to other redundancy measures [20],  $I_{min}$  satisfies the pairwise marginals property, meaning that  $I_{min}(T; \alpha)$  only depends on the pairwise marginals distributions  $p(T, A_i)$  between the target  $T$  and each source  $A_i \in \alpha$ .
Alternative redundancy measures proposed so far are either not straightforward to generalize beyond  $N = 2$  source variables [3; 16] or can provide negative information atoms [11]. However, these alternative measures have complementary advantages with respect to  $I_{\min}$ , such as satisfying the identity property  $I_{\cap}(X,Y;(X,Y)) = I(X;Y)$  which guarantees that, in a system made of two independent variables, the two variables cannot carry redundant information about the whole system. Despite not satisfying this property, the  $I_{\min}$  measure has been applied to study information processing in simulated neural networks [2], providing insightful and interpretable results.
SM1.3 Derivation of FIT
In this subsection, we derive the definition of FIT. In the main text, we used the notation  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  to denote the atom of information that is shared by variables  $X_{past}$  and  $Y_{pres}$  about target  $S$  but is unique with respect to a third variable  $Y_{past}$ . Using the bracketed notation introduced in Section SM1.2 to denote information atoms,  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  corresponds to  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$ . From eq. S5, this atom is the difference between the information that  $X_{past}$  and  $Y_{pres}$  share about  $S$  minus the information that  $X_{past}$ ,  $Y_{past}$  and  $Y_{pres}$  share about  $S$  (see also eq. S14). In the PID literature, information redundant in set of sources about a target that is not redundant with information from another set, has been termed shared unique information [6; 23]. Therefore, using the bracketed notation to denote the two atoms of shared unique information  $SUI(S: X_{past}, Y_{pres} \setminus Y_{past})$  and  $SUI(Y_{pres}: X_{past}, S \setminus Y_{past})$ , we can write the definition of FIT as:
$$
F I T = \min  \left[ I _ {\partial} \left(S; \left\{X _ {p a s t} \right\} \left\{Y _ {p r e s} \right\}\right), I _ {\partial} \left(Y _ {p r e s}; \left\{X _ {p a s t} \right\} \{S \}\right) \right] \tag {S11}
$$
We first discuss the mathematical properties of  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$ , the first atom appearing in FIT definition. We then discuss the complementary mathematical properties met by  $(I_{\partial}(Y_{pres}; \{X_{past}\} \{S\}))$ , the second atom appearing in FIT definition. To keep our reasoning as general as possible, we discuss properties of the atoms that are valid when the atom is computed using any of the redundancy measures that satisfy the pairwise marginal property (which include the  $I_{min}$  redundancy measure that we implemented here). We then demonstrate that a specific pairwise
PID for 2 source variables
B
PID for 3 source variables and FIT definition
Figure S1: Schematic of the concepts of PID. (A) The information  $I(S; X, Y)$  that two source variables  $X, Y$  carry about a target variable  $S$  can be decomposed into four PID atoms. Left: a set-theoretic diagram of the decomposition. Shared information  $\{X\} \{Y\}$ , darkest shade of blue; unique information  $\{X\}$  and  $\{Y\}$ , lighter shade of blue; synergistic information  $\{X, Y\}$ , lightest blue. Right: the same decomposition plotted as lattice. A link between two regions symbolizes the ordering relationship of eq. S4. (B) FIT is defined on two PID lattices with three sources and one target. Left: The PID lattice with  $S$  as target and  $(X_{past}, Y_{past}, Y_{pres})$  as sources. Right: the PID with  $Y_{pres}$  as target and  $(S, X_{past}, Y_{past})$  as sources. FIT is the minimum between the two atoms highlighted in red. Classical Shannon information theoretic quantities are mapped on the two lattices with different colors (i.e. the sum of all the atoms bounded by a given color is equal to a classical information-theoretic quantity).  $I(S; Y_{pres})$  is mapped using blue,  $I(S; X_{past})$  using purple,  $TE(X \to Y)$  using yellow, and  $I(X_{past}; Y_{pres})$  using green. The  $p$  and  $t$  subscript in the Figure is a shorthand for past and pres respectively.
algebraic relationship exists between these two atoms. This relationship is derived from the Shannon information theoretic quantities that relate atoms in the two decompositions. Importantly, this relationship uncovers the presence of a more refined information component that is shared between the two atoms. Finally, we discuss how taking the minimum between these two atoms ensures that FIT fulfills simultaneously a series of fundamental properties, including being upper bounded at the same time by the feature information encoded in the past activity of the sender  $X$ ,  $I(S;X_{past})$ , and
in the present activity of the receiver  $Y$ ,  $I(S;Y_{pres})$ , and by the total information flowing from  $X$  to  $Y$ , namely  $TE(X\to Y)$ .
SM1.3.1 Properties of the first atom in the FIT definition
Our intuitive definition is that FIT should be the information shared between the past activity of a sender region  $X_{past}$  and the present activity of a receiver region  $Y_{pres}$  about  $S$  that is unique with respect to the past activity of the receiver  $Y_{past}$ . Thus, within PID of the  $(S, X_{past}, Y_{pres}, Y_{past})$  system the most natural candidate is the first atom in eq. S11  $(I_{\partial}(S; \{X_{past}\} \{Y_{pres}\}))$  coming from the decomposition taking  $(X_{past}, Y_{past}, Y_{pres})$  as source variable and  $S$  as target variable. Using eq. S7, we show that the two Shannon information quantities  $I(S; X_{past})$  and  $I(S; Y_{pres})$  (i.e., the feature information encoded in the past values of the sender  $X$  and of the receiver  $Y$ , respectively) set an upper bound on  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$ . Indeed,  $I(S; X_{past})$  and  $I(S; Y_{pres})$  can be written as the sum of information atoms appearing on the lattice having  $S$  as target:
$$
\begin{array}{l} I (S; X _ {p a s t}) = I _ {\partial} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \} \{Y _ {p a s t} \}) + I _ {\partial} (S; \{X _ {p a s t} \} \{Y _ {p a s t} \}) \\ + I _ {\partial} \left(S; \{X _ {p a s t} \} \{Y _ {p r e s} \}\right) + I _ {\partial} \left(S; \{X _ {p a s t} \} \{Y _ {p a s t} Y _ {p r e s} \}\right) \tag {S12} \\ + I _ {\partial} (S; \{X _ {p a s t} \}) \geq I _ {\partial} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \}) \\ \end{array}
$$
$$
\begin{array}{l} I (S; Y _ {p r e s}) = I _ {\partial} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \} \{Y _ {p a s t} \}) + I _ {\partial} (S; \{Y _ {p r e s} \} \{Y _ {p a s t} \}) \\ + I _ {\partial} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \}) + I _ {\partial} (S; \{Y _ {p r e s} \} \{Y _ {p a s t} X _ {p a s t} \}) \tag {S13} \\ + I _ {\partial} (S; \{Y _ {p r e s} \}) \geq I _ {\partial} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \}) \\ \end{array}
$$
which proves that  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$  is upper bounded by both quantities (see Fig. S1B for a graphical depiction of  $I(S; Y_{pres})$ , in blue, and  $I(S; X_{past})$ , in purple, upper bound the first atom). However, eq. S7 does not establish any relationship between  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$  and Shannon information between the source variables of the decomposition, including  $TE(X \to Y)$ . Therefore, the value of the first atom can exceed the total amount of information transmitted from  $X$  to  $Y$ $TE(X \to Y)$ .
Next we prove that, when computed using a redundancy measure that satisfies the pairwise marginals property (see Section SM1.2),  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$  only depends on the probability distribution  $P(S, X_{past}, Y_{pres})$  through the pairwise marginal distributions  $P(S, X_{past})$  and  $P(S, Y_{pres})$ , and does not depend explicitly on  $P(X_{past}, Y_{pres})$ . Indeed, using eq. S5 we can express  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$  as the difference between the redundancy about  $S$  computed on collection  $\{X_{past}\} \{Y_{pres}\}$  minus the redundancy computed on collection  $\{X_{past}\} \{Y_{pres}\} \{Y_{past}\}$ :
$$
I _ {\partial} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \}) = I _ {\cap} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \}) - I _ {\cap} (S; \{X _ {p a s t} \} \{Y _ {p r e s} \} \{Y _ {p a s t} \}) \tag {S14}
$$
If  $I_{\cap}$  satisfies the pairwise marginals property, then the right-hand side of eq. S14 only depends on the full probability distribution  $P(S, X_{past}, Y_{past}, Y_{pres})$  through the pairwise marginal distributions between the target  $S$  and the individual sources  $P(S, X_{past})$ ,  $P(S, Y_{past})$ , and  $P(S, Y_{pres})$ , but not through the pairwise marginals between the sources, including  $P(Y_{pres}, X_{past})$ . This implies that if we partially disrupt the dependency structure of our data to create surrogate data, where the individual dependencies of  $X$  and of  $Y$  on  $S$  are preserved (i.e., the pairwise marginals  $P(S, X_{past})$  and  $P(S, Y_{pres})$  do not change) and the within-trial correlations at a fixed stimulus are disrupted (i.e., the conditional distribution  $P(X_{pres}, Y_{past}|S)$  changes), this atom will retain the same value it had in the original data. Therefore, this atom alone cannot rule out confounding scenarios where  $X$  and  $Y$  encode  $S$  independently with a temporal lag, with no information transfer at fixed stimulus value.
SM1.3.2 Properties of the second atom in the FIT definition
Atoms satisfying mathematical properties that are complementary to the ones of  $I_{\partial}(S; \{X_{past}\} \{Y_{pres}\})$  exist on the decomposition with  $Y_{t}$  as target. On this decomposition one atom that intuitively captures feature-specific information flow is  $I_{\partial}(Y_{pres}; \{X_{past}\} \{S\})$ , i.e. the information that the past activity of the sender  $X$  and the feature  $S$  share about the present activity of the receiver  $Y$  that is unique with respect to the past activity of the receiver  $Y$ . We first prove that this atom is upper bounded by the value of  $TE(X \to Y)$ , and then that this atom depends on  $P(Y_{pres}, X_{past})$ .
To prove that the value of  $TE(X \to Y)$  sets an upper bound to  $I_{\partial}(Y_{pres}; \{X_{past}\} \{S\})$ , we first use the information-theoretic chain rule [9], to write the conditioned mutual information in eq. S2 as
the difference between the joint mutual information that  $X_{past}$  and  $Y_{past}$  carry about  $Y_{pres}$  minus the mutual information between  $Y_{pres}$  and  $Y_{past}$ . Then, we use eq. S7 to write the two information quantities as the sum of non-negative information atoms, including  $I_{\partial}(Y_{pres}; \{X_{past}\} \{S\})$ :
$$
\begin{array}{l} T E (X \rightarrow Y) = I \left(Y _ {p r e s}; X _ {p a s t}, Y _ {p a s t}\right) - I \left(Y _ {p r e s}; Y _ {p a s t}\right) \\ = I _ {\partial} \left(Y _ {p r e s}; \{S \} \{X _ {p a s t} \}\right) + I _ {\partial} \left(Y _ {p r e s}; \{X _ {p a s t} \} \{S Y _ {p a s t} \}\right) \\ \begin{array}{l} + I _ {\partial} \left(Y _ {p r e s}; \{S \} \left\{X _ {p a s t} Y _ {p a s t} \right\}\right) + I _ {\partial} \left(Y _ {p r e s}; \left\{X _ {p a s t} \right\}\right) \\ + I _ {\partial} \left(Y _ {p r e s}; \left\{X _ {p a s t} S \right\} \left\{Y _ {p a s t} S \right\} \left\{X _ {p a s t} Y _ {p a s t} \right\}\right) \end{array} \tag {S15} \\ + I _ {\partial} \left(Y _ {p r e s}; \left\{X _ {p a s t} S \right\} \left\{X _ {p a s t} Y _ {p a s t} \right\}\right) + I _ {\partial} \left(Y _ {p r e s}; \left\{Y _ {p a s t} S \right\} \left\{X _ {p a s t} Y _ {p a s t} \right\}\right) \\ + I _ {\partial} \left(Y _ {p r e s}; \{X _ {p a s t} Y _ {p a s t} \}\right) \geq I _ {\partial} \left(Y _ {p r e s}; \{S \} \{X _ {p a s t} \}\right) \\ \end{array}
$$
which proves that  $I_{\partial}(Y_{pres}; \{X_{past}\} \{S\})$  is upper bounded by  $TE(X \to Y)$  (see Fig. S1B for a graphical depiction of the mapping of  $TE(X \to Y)$ , in yellow, on the lattice to which this second atom belongs).
Similarly to the first atom, this second atom is also upper bounded by  $I(S;Y_{pres})$  (not proven, but see the blue quantity in Fig. S1B for a graphical depiction of this property), however it is not upper bounded by Shannon information quantities between the source variables of the decomposition with  $Y_{pres}$  as target, and in particular by  $I(S;X_{past})$ . This is important because it proves that neither the second atom alone satisfies all the properties that we require for a measure of feature-specific information transfer.
We then prove that the second atom depends on  $P(Y_{pres}, X_{past})$ , a property which makes it suited to rule out confounding scenarios where  $X$  and  $Y$  independently encode the  $S$  but no communication occurs between the two. To do so, we use eq. S5 to write the second atom as the difference between two redundancy terms:
$$
I _ {\partial} \left(Y _ {p r e s}; \left\{X _ {p a s t} \right\} \{S \}\right) = I _ {\cap} \left(Y _ {p r e s}; \left\{X _ {p a s t} \right\} \{S \}\right) - I _ {\cap} \left(Y _ {p r e s}; \left\{X _ {p a s t} \right\} \{S \} \{Y _ {p a s t} \}\right) \tag {S16}
$$
If  $I_{\cap}$  satisfies the pairwise marginals property, then the right-hand side. of eq. S14 depends on the full probability distribution  $P(S,X_{past},Y_{past},Y_{pres})$  through the marginal distributions between the target  $Y_{pres}$  and the individual sources  $P(Y_{pres},X_{past}),P(Y_{pres},S),$  and  $P(Y_{pres},Y_{past})$ . This implies that if we partially disrupt the dependency structure of our data and create surrogate data where the individual dependencies of  $X$  and of  $Y$  on  $S$  are preserved (i.e., the pairwise marginals  $P(S,X_{past})$  and  $P(S,Y_{pres})$  do not change) and the within-trial correlations at a fixed stimulus are disrupted (i.e., the conditional distribution  $P(X_{pres},Y_{past}|S)$  changes), the value of the referenced atom may differ from its original value. This change occurs because this operation generally disrupts  $P(X_{pres},Y_{past})$ . Therefore, this atom can rule out confounding scenarios where  $X$  and  $Y$  encode  $S$  independently with a temporal lag, with no information transfer at fixed stimulus value.
SM1.3.3 The two atoms in the FIT definition are related by Shannon Information theoretic quantities
This Section is structured as follows. First, we present some basic findings from Ref. [27] where, the authors showed that atoms from different decompositions are algebraically constrained by Shannon's information-theoretic quantities and used these constraints to identify, specifically for a trivariate system, a reduced set of finer information components which could describe all atoms across different decompositions. Next, we express the algebraic constraints between two decompositions as a homogeneous linear system of equations. We demonstrate that the reduced set of information components derived for two decompositions in Ref. [27] can be obtained as solutions to this homogeneous system. Finally, we derive the analogous homogeneous system in the case of four variables. One solution to this system relates specifically the two atoms appearing in the FIT definition.
Ref [27] showed that atoms belonging to different decompositions are algebraically constrained by information-theoretic quantities. These constraints derive from fundamental axioms of PID theory, specifically the fact that in the system  $(X_{1},\ldots ,X_{N})$ , we can use eq. S7 to express the mutual information between two variables  $X_{i}$  and  $X_{j}$  (conditioned on up to  $N - 2$  other variables) as the sum of information atoms from both the decomposition with  $X_{i}$  and the decomposition with  $X_{j}$  as target variable. For the trivariate system  $(S,X,Y)$ , Shannon information quantities