SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junhyung Park
Empirical Inference Department
MPI for Intelligent Systems
72076 Tübingen, Germany
junhyung.park@tuebingen.mpg.de
Simon Buchholz
simon.buchholz@tuebingen.mpg.de
Bernhard Scholkopf
bs@tuebingen.mpg.de
Krikamol Muandet
CISPA
Helmholtz Center for Information Security
66123 Saarbrücken, Germany
muandet@cispa.de
Causality is a central concept in a wide range of research areas, yet there is still no universally agreed axiomatisation of causality. We view causality both as an extension of probability theory and as a study of what happens when one intervenes on a system, and argue in favour of taking Kolmogorov's measure-theoretic axiomatisation of probability as the starting point towards an axiomatisation of causality. To that end, we propose the notion of a causal space, consisting of a probability space along with a collection of transition probability kernels, called causal kernels, that encode the causal information of the space. Our proposed framework is not only rigorously grounded in measure theory, but it also sheds light on long-standing limitations of existing frameworks including, for example, cycles, latent variables and stochastic processes.
Causal reasoning has been recognised as a hallmark of human and machine intelligence, and in the recent years, the machine learning community has taken up a rapidly growing interest in the subject [46, 53, 54], in particular in representation learning [55, 41, 62, 57, 11, 40] and natural language processing [35, 19]. Causality has also been extensively studied in a wide range of other research domains, including, but not limited to, philosophy [39, 64, 16], psychology [60], statistics [45, 56] including social, biological and medical sciences [51, 31, 29, 26], mechanics and law [6].
The field of causality was born from the observation that probability theory and statistics (Figure 1a) cannot encode the notion of causality, and so we need additional mathematical tools to support the enhanced view of the world involving causality (Figure 1b). Our goal in this paper is to give an axiomatic framework of the forwards direction of Figure 1b, which currently consists of many competing models (see Related Works). As a starting point, we observe that the forwards direction of Figure 1a, i.e. probability theory, has a set of axioms based on measure theory (Axioms 2.1) that are widely accepted and used<sup>1</sup>, and hence argue that it is natural to take the primitive objects of this framework as the basic building blocks. Despite the fact that all of the existing mathematical frameworks of causality recognise the crucial role that probability plays / should play in any causal
<sup>1</sup>Kolmogorov's axiomatisation is without doubt the standard in probability theory. However, we are aware of other, less popular frameworks, for example, one that is more amenable to Bayesian probability [34], one based on game theory [59] and imprecise probabilities [61].
Figure 1: Data generating processes and data.
theory, it is surprising that few of them try to build directly upon the axioms of probability theory, and those that do fall short in different ways (see Related Works).
On the other hand, we place manipulations at the heart of our approach to causality; in other words, we make changes to some parts of a system, and we are interested in what happens to the rest of this system. This manipulative philosophy towards causality is shared by many philosophers [64], and is the essence behind almost all causal frameworks proposed and adopted in the statistics/machine learning community that we are aware of.
To this end, we propose the notion of causal spaces (Definition 2.2), constructed by adding causal objects, called causal kernels, directly to probability spaces. We show that causal spaces strictly generalise (the interventional aspects of) existing frameworks, i.e. given any configuration of, for example, a structural causal model or potential outcomes framework, we can construct a causal space that can carry the same (interventional) information. Further, we show that causal spaces can seamlessly support situations where existing frameworks struggle, for example those with hidden confounders, cyclic causal relationships or continuous-time stochastic processes.
Related Works We stress that our paper should not be understood as a criticism of the existing frameworks, or that somehow our goal is to replace them. On the contrary, we foresee that they will continue to thrive in whatever domains they have been used in, and will continue to find novel application areas.
Most prominently, there are the structural causal models (SCMs) [43, 46], based most often on directed acyclic graphs (DAGs). Here, the theory of causality is built around variables and structural equations, and probability only enters the picture through a distribution on the exogenous variables [33]. Efforts have been made to axiomatise causality based on this framework [21, 23, 28], but models based on structural equations or graphs inevitably rely on assumptions even for the definitions themselves, such as being confined to a finite number of variables, the issue of solvability in the case of non-recursive (or cyclic) cases, that all common causes (whether latent or observed) are modeled, or that the variables in the model do not causally affect anything outside the model. Hence, these cannot be said to be an "axiomatic definition" in the strictest sense. In a series of examples in Section 4, we highlight cases for which causal spaces have a natural representation but SCMs do not, including common causes, cycles and continuous-time stochastic processes.
The potential outcomes framework is a competing model, most often used in economics, social sciences or medicine research, in which we have a designated treatment variable, whose causal effect we are interested in, and for each value of the treatment variable, we have a separate, potential outcome variable [31, 26]. There are other, perhaps lesser-known approaches to model causality, such as that based on decision theory [17, 52], on category theory [32, 20], on an agent explicitly performing actions that transform the state space [15], or settable systems [63].
Perhaps the works that are the most relevant to this paper are those that have already recognised the need for an axiomatisation of causality based on measure-theoretic probability theory. Ortega [42] uses a particular form of a tree to define a causal space, and in so doing, uses an alternative, Bayesian set-up of probability theory [34]. It has an obvious drawback that it only considers countable sets of "realisations", clearly ruling out many interesting and commonly-occurring cases, and also does not seem to accommodate cycles. Heymann et al. [27] define the information dependency model based on measurable spaces to encode causal information. We find this to be a highly interesting and relevant approach, but the issue of cycles and solvability arises, and again, only countable sets of outcomes are considered, with the authors admitting that the results are likely not to hold with uncountable sets. Moreover, probabilities and interventions require additional work to be taken care of. Lastly, Cabreros and Storey [12] attempt to provide a measure-theoretic grounding to the potential
outcomes framework, but thereby confine attention to the setting of a finite number of variables, and even restrict the random variables to be discrete.
Finally, we mention the distinction between type causality and actual causality. The former is a theory about general causality, involving statements such as "in general, smoking makes lung cancer more likely". Type causality is what we will be concerned with in this paper. Actual causality, on the other hand, is interested in whether a particular event was caused by a particular action, dealing with statements such as "Bob got lung cancer because he smoked for 30 years". It is an extremely interesting area of research that has far-reaching implications for concepts such as responsibility, blame, law, harm [4, 5], model explanation [7] and algorithmic recourse [36]. Many definitions of actual causality have been proposed [25, 22, 24], but the question of how to define actual causality is still not settled [1]. The current definitions of actual causality are all grounded on (variants) of SCMs, and though out of the scope of this paper, it will be an interesting future research direction to consider how actual causality can be incorporated into our proposed framework.
2 Causal Spaces and Interventions
Familiarity with measure-theoretic probability theory is necessary, and we succinctly collect the most essential definitions and results in Appendix A. Most important to note is the definition of a transition probability kernel (also given at the end of Appendix A.1): for measurable spaces  $(E,\mathcal{E})$  and  $(F,\mathcal{F})$ , a mapping  $K:E\times \mathcal{F}\to [0,\infty ]$  is called a transition probability kernel from  $(E,\mathcal{E})$  into  $(F,\mathcal{F})$  if the mapping  $x\mapsto K(x,B)$  is measurable for every set  $B\in \mathcal{F}$  and the mapping  $B\mapsto K(x,B)$  is a probability measure on  $(F,\mathcal{F})$  for every  $x\in E$ . Also worthy of mention is the definition of a measurable rectangle: if  $(E,\mathcal{E})$  and  $(F,\mathcal{F})$  are measurable spaces and  $A\in \mathcal{E}$  and  $B\in \mathcal{F}$ , then the measurable rectangle  $A\times B$  is the set of all pairs  $(x,y)$  with  $x\in A$  and  $y\in B$ . All proofs are deferred to Appendix F. We start by recalling the axioms of probability theory, which we will use as the starting point of our work.
Axioms 2.1 (Kolmogorov [38]). A probability space is a triple  $(\Omega, \mathcal{H}, \mathbb{P})$ , where:
(i)  $\Omega$  is a set of outcomes;
(ii)  $\mathcal{H}$  is a collection of events forming a  $\sigma$ -algebra, i.e. a collection of subsets of  $\Omega$  such that (a)  $\Omega \in \mathcal{H}$ ; (b) if  $A \in \mathcal{H}$ , then  $\Omega \setminus A \in \mathcal{H}$ ; (c) if  $A_1, A_2, \ldots \in \mathcal{H}$ , then  $\cup_n A_n \in \mathcal{H}$ ;
(iii)  $\mathbb{P}$  is a probability measure on  $(\Omega, \mathcal{H})$ , i.e. a function  $\mathbb{P}: \mathcal{H} \to [0,1]$  satisfying (a)  $\mathbb{P}(\emptyset) = 0$ ; (b)  $\mathbb{P}(\cup_n A_n) = \sum_n \mathbb{P}(A_n)$  for any disjoint sequence  $(A_n)$  in  $\mathcal{H}(c) \mathbb{P}(\Omega) = 1$ .
In the development of probability theory, one starts by assuming the existence of a probability space  $(\Omega, \mathcal{H}, \mathbb{P})$ . However, the actual construction of probability spaces that can carry random variables corresponding to desired random experiments is done through (repeated applications of) two main results – those of Ionescu-Tulcea and Kolmogorov [14, p.160, Chapter IV, Section 4]; the former constructs a probability space that can carry a finite or countably infinite chain of trials, and the latter shows the existence of a probability space that can carry a process with an arbitrary index set. In both cases, the measurable space  $(\Omega, \mathcal{H})$  is constructed as a product space:
(i) for a finite set of trials, each taking place in some measurable space  $(E_t,\mathcal{E}_t), t = 1,\dots,n$ , we have  $(\Omega ,\mathcal{H}) = \otimes_{t = 1}^{n}(E_{t},\mathcal{E}_{t})$
(ii) for a countably infinite set of trials, each taking place in some measurable space  $(E_t, \mathcal{E}_t)$ ,  $t \in \mathbb{N}$ , we have  $(\Omega, \mathcal{H}) = \otimes_{t \in \mathbb{N}} (E_t, \mathcal{E}_t)$ ;
(iii) for a process  $\{X_{t}:t\in T\}$  with an arbitrary index set  $T$  , we assume that all the  $X_{t}$  live in the same standard measurable space  $(E,\mathcal{E})$  , and let  $(\Omega ,\mathcal{H}) = (E,\mathcal{E})^{T} = \otimes_{t\in T}(E,\mathcal{E})$
In the construction of a causal space, we will take as our starting point a probability space  $(\Omega, \mathcal{H}, \mathbb{P})$  where the measure  $\mathbb{P}$  is defined on a product measurable space  $(\Omega, \mathcal{H}) = \otimes_{t \in T} (E_t, \mathcal{E}_t)$  with the  $(E_t, \mathcal{E}_t)$  being the same standard measurable space if  $T$  is uncountable. Denote by  $\mathcal{P}(T)$  the power set of  $T$ , and for  $S \in \mathcal{P}(T)$ , we denote by  $\mathcal{H}_S$  the sub- $\sigma$ -algebra of  $\mathcal{H} = \otimes_{t \in T} \mathcal{E}_t$  generated by measurable rectangles  $\times_{t \in T} A_t$ , where  $A_t \in \mathcal{E}_t$  differs from  $E_t$  only for  $t \in S$ . In particular,  $\mathcal{H}_{\emptyset} = \{\emptyset, \mathcal{H}\}$  is the trivial  $\sigma$ -algebra of  $\Omega = \times_{t \in T} E_t$ . Also, we denote by  $\Omega_S$  the subspace  $\times_{s \in S} E_s$  of  $\Omega = \times_{t \in T} E_t$ , and for  $T \supseteq S \supseteq U$ , we let  $\pi_{SU}$  denote the natural projection from  $\Omega_S$  onto  $\Omega_U$ .
Definition 2.2. A causal space is defined as the quadruple  $(\Omega, \mathcal{H}, \mathbb{P}, \mathbb{K})$ , where  $(\Omega, \mathcal{H}, \mathbb{P}) = (\times_{t \in T} E_t, \otimes_{t \in T} \mathcal{E}_t, \mathbb{P})$  is a probability space and  $\mathbb{K} = \{K_S : S \in \mathcal{P}(T)\}$ , called the causal mechanism, is a collection of transition probability kernels  $K_S$  from  $(\Omega, \mathcal{H}_S)$  into  $(\Omega, \mathcal{H})$ , called the causal kernel on  $\mathcal{H}_S$ , that satisfy the following axioms:
(i) for all  $A\in \mathcal{H}$  and  $\omega \in \Omega$
$$
K _ {\emptyset} (\omega , A) = \mathbb {P} (A);
$$
(ii) for all  $\omega \in \Omega$ ,  $A \in \mathcal{H}_S$  and  $B \in \mathcal{H}$ ,
$$
K _ {S} (\omega , A \cap B) = 1 _ {A} (\omega) K _ {S} (\omega , B) = \delta_ {\omega} (A) K _ {S} (\omega , B);
$$
in particular, for  $A\in \mathcal{H}_S$ $K_{S}(\omega ,A) = 1_{A}(\omega)K_{S}(\omega ,\Omega) = 1_{A}(\omega) = \delta_{\omega}(A).$
Here, the probability measure  $\mathbb{P}$  should be viewed as the "observational measure", and the causal mechanism  $\mathbb{K}$ , consisting of causal kernels  $K_{S}$  for  $S\in \mathcal{P}(T)$ , contains the "causal information" of the space, by directly specifying the interventional distributions. We write  $1_A(\omega)$  when viewed as a function in  $\omega$  for a fixed  $A$ , and  $\delta_{\omega}(A)$  when viewed as a measure for a fixed  $\omega \in \Omega$ . Note that  $\mathbb{K}$  cannot be determined "independently" of the probability measure  $\mathbb{P}$ , since, for example,  $K_{\emptyset}$  is clearly dependent on  $\mathbb{P}$  by (i).
Before we discuss the meaning of the two axioms, we immediately give the definition of an intervention. An intervention is carried out on a sub- $\sigma$ -algebra of the form  $\mathcal{H}_U$  for some  $U \in \mathcal{P}(T)$ . In the following, for  $S \in \mathcal{P}(T)$ , we denote  $\omega_S = \pi_{TS}(\omega)$ . Then note that  $\Omega = \Omega_S \times \Omega_{T \setminus S}$  and for any  $\omega \in \Omega$ , we can decompose it into components as  $\omega = (\omega_S, \omega_{T \setminus S})$ . Then  $K_S(\omega, A) = K_S((\omega_S, \omega_{T \setminus S}), A)$  for any  $A \in \mathcal{H}$  only depends on the first  $\omega_S$  component of  $\omega = (\omega_S, \omega_{T \setminus S})$ . As a slight abuse of notation, we will sometimes write  $K_S(\omega_S, A)$  for conciseness.
Definition 2.3. Let  $(\Omega, \mathcal{H}, \mathbb{P}, \mathbb{K}) = (\times_{t \in T} E_t, \otimes_{t \in T} \mathcal{E}_t, \mathbb{P}, \mathbb{K})$  be a causal space, and  $U \in \mathcal{P}(T)$ ,  $\mathbb{Q}$  a probability measure on  $(\Omega, \mathcal{H}_U)$  and  $\mathbb{L} = \{L_V : V \in \mathcal{P}(U)\}$  a causal mechanism on  $(\Omega, \mathcal{H}_U, \mathbb{Q})$ . An intervention on  $\mathcal{H}_U$  via  $(\mathbb{Q}, \mathbb{L})$  is a new causal space  $(\Omega, \mathcal{H}, \mathbb{P}^{\mathrm{do}(U,\mathbb{Q})}, \mathbb{K}^{\mathrm{do}(U,\mathbb{Q},\mathbb{L})})$ , where the intervention measure  $\mathbb{P}^{\mathrm{do}(U,\mathbb{Q})}$  is a probability measure on  $(\Omega, \mathcal{H})$  defined, for  $A \in \mathcal{H}$ , by
$$
\mathbb {P} ^ {\mathrm {d o} (U, \mathbb {Q})} (A) = \int \mathbb {Q} (d \omega) K _ {U} (\omega , A) \tag {1}
$$
and  $\mathbb{K}^{\mathrm{do}(U,\mathbb{Q},\mathbb{L})} = \{K_S^{\mathrm{do}(U,\mathbb{Q},\mathbb{L}}}:S\in \mathcal{P}(T)\}$  is the intervention causal mechanism whose intervention causal kernels are
$$
K _ {S} ^ {\mathrm {d o} (U, \mathbb {Q}, \mathbb {L})} (\omega , A) = \int L _ {S \cap U} \left(\omega_ {S \cap U}, d \omega_ {U} ^ {\prime}\right) K _ {S \cup U} \left(\left(\omega_ {S \setminus U}, \omega_ {U} ^ {\prime}\right), A\right). \tag {2}
$$
The intuition behind these definitions is as follows. Starting from the probability space  $(\Omega, \mathcal{H}, \mathbb{P})$ , we choose a "subspace" on which to intervene, namely a sub- $\sigma$ -algebra  $\mathcal{H}_U$  of  $\mathcal{H}$ . The intervention is the process of placing any desired measure  $\mathbb{Q}$  on this "subspace"  $(\Omega, \mathcal{H}_U)$ , along with an internal causal mechanism  $\mathbb{L}$  on this "subspace". The causal kernel  $K_U$  corresponding to the "subspace"  $\mathcal{H}_U$ , which is encoded in the original causal space, determines what the intervention measure on the whole space  $\mathcal{H}$  will be, via equation (1). For the causal kernels after intervention, the causal effect first takes place within  $\mathcal{H}_U$  via the internal causal mechanism  $\mathbb{L}$ , then propagates to the rest of  $\mathcal{H}$  via equation (2).
The definition of intervening on a  $\sigma$ -algebra of the form  $\mathcal{H}_U$  given in Definition 2.3 sheds light on the two axioms of causal spaces given in Definition 2.2.
Remark 2.4. Trivial Intervention Axiom (i) of causal mechanisms in Definition 2.2 ensures that intervening on the trivial  $\sigma$ -algebra (i.e. not intervening at all) leaves the probability measure intact, i.e. writing  $\mathbb{Q}$  for the trivial probability measure on  $\{\emptyset, \Omega\}$ , we have  $\mathbb{P}^{\mathrm{do}(\emptyset, \mathbb{Q})} = \mathbb{P}$ .
Interventional Determinism Axiom (ii) of Definition 2.2 ensures that for any  $A \in \mathcal{H}_U$ , we have  $\mathbb{P}^{\mathrm{do}(U,\mathbb{Q})}(A) = \mathbb{Q}(A)$ , which means that if we intervene on the causal space by giving  $\mathcal{H}_U$  a particular probability measure  $\mathbb{Q}$ , then  $\mathcal{H}_U$  indeed has that measure with respect to the intervention probability measure.
$^2$ Choosing  $\mathbb{Q}$  to have measure 1 on a single element would correspond to what is known as a "hard intervention" in the SCM literature. Letting  $\mathbb{Q}$  and  $\mathbb{L}$  be arbitrary would allow us to obtain any "soft intervention".
The following example should serve as further clarification of the concepts.
Example 2.5. Let  $E_1 = E_2 = \mathbb{R}$ , and  $\mathcal{E}_1, \mathcal{E}_2$  be Lebesgue  $\sigma$ -algebras on  $E_1$  and  $E_2$ . Each  $e_1 \in E_1$  and  $e_2 \in E_2$  respectively represent the altitude in metres and temperature in Celsius of a random location. For simplicity, we assume a jointly Gaussian measure  $\mathbb{P}$  on  $(\Omega, \mathcal{H}) = (E_1 \times E_2, \mathcal{E}_1 \otimes \mathcal{E}_2)$ , say
with mean vector  $\binom{1000}{10}$  and covariance matrix  $\begin{pmatrix} 300 & -15 \\ -15 & 1 \end{pmatrix}$ . For each
$e_1 \in E_1$  and  $A \in \mathcal{E}_2$ , we let  $K_1(e_1, A)$  be the conditional measure of  $\mathbb{P}$  given
$e_1$ , i.e. Gaussian with mean  $\frac{1200 - e_1}{20}$  and variance  $\frac{1}{4}$ . This represents the fact that, if we intervene by fixing the altitude of a location, then the temperature of that location will be causally affected. However, if we intervene by fixing a temperature of a location, say by manually heating up or cooling down a place, then we expect that this has no causal effect on the altitude of the place. This can be represented by the causal kernel  $K_{2}(e_{2},B) = \mathbb{P}(B)$  for each  $B\in \mathcal{E}_1$ , i.e. Gaussian measure with mean 1000 and variance 300, regardless of the value of  $e_2$ . The corresponding "causal graph" would be Figure 2. If we intervene on  $\mathcal{E}_1$  with measure  $\delta_{1000}$ , i.e. we fix the altitude at 1000m, then the intervention measure  $\mathbb{P}^{\mathrm{do}(1,\delta_{1000})}$  on  $(E_2,\mathcal{E}_2)$  would be Gaussian with mean 10 and variance  $\frac{1}{4}$ . If we intervene on  $\mathcal{E}_2$  with any measure  $\mathbb{Q}$ , the intervention measure  $\mathbb{P}^{\mathrm{do}(2,\mathbb{Q})}$  on  $(E_1,\mathcal{E}_1)$  would still be Gaussian with mean 1000 and variance 300.
Figure 2: Altitude and Temperature.
The following theorem proves that the intervention measure and causal mechanism are indeed valid.
Theorem 2.6. From Definition 2.3,  $\mathbb{P}^{\mathrm{do}(U,\mathbb{Q})}$  is indeed a measure on  $(\Omega ,\mathcal{H})$ , and  $\mathbb{K}^{\mathrm{do}(U,\mathbb{Q},\mathbb{L})}$  is indeed a valid causal mechanism on  $(\Omega ,\mathcal{H},\mathbb{P}^{\mathrm{do}(U,\mathbb{Q})})$ , i.e. they satisfy the axioms of Definition 2.2.
To end this section, we make a couple of further remarks on the definition of causal spaces.
Remark 2.7. (i) We require causal spaces to be built on top of product probability spaces, as opposed to general probability spaces, and causal kernels are defined on sub- $\sigma$ -algebras of  $\mathcal{H}$  of the form  $\mathcal{H}_S$  for  $S \in \mathcal{P}(T)$ , as opposed to general sub- $\sigma$ -algebras of  $\mathcal{H}$ . This is because, for two events that are not in separate components of a product space, one can always intervene on one of those events in such a way that the measure on the other event will have to change, meaning the causal kernel cannot be decoupled from the intervention itself. For example, in a dice-roll with outcomes  $\{1, 2, 3, 4, 5, 6\}$  each with probability  $\frac{1}{6}$ , if we intervene to give measure 1 to roll 6, then the other outcomes are forced to have measure 0. Only if we consider separate components of product measurable spaces can we set meaningful causal relationships that are decoupled from the act of intervention itself.
(ii) We do not distinguish between interventions that are practically possible and those that are not. For example, the "causal effect of sunlight on the moon's temperature" cannot be measured realistically, as it would require covering up the sun, but the information encoded in the causal kernel would still correspond to what would happen when we cover up the sun.
3 Comparison with Existing Frameworks
In this section, we show how causal spaces can encode the interventional aspects of the two most widely-used frameworks of causality, i.e. structural causal models and the potential outcomes.
3.1 Structural Causal Models (SCMs)
Consider an SCM in its most basic form, given in the following definition.
Definition 3.1 ([46, p.83, Definition 6.2]). A structural causal model  $\mathfrak{C} = (\mathbf{S},\tilde{\mathbb{P}})$  consists of a collection  $\mathbf{S}$  of  $d$  (structural) assignments  $X_{j}\coloneqq f_{j}(\mathbf{PA}_{j},N_{j}),j = 1,\ldots ,d$  where  $\mathbf{PA}_j\subseteq$ $\{X_1,\dots,X_d\} \backslash \{X_j\}$  are called the parents of  $X_{j}$  and  $N_{j}$  are the noise variables; and a distribution  $\tilde{\mathbb{P}}$  over the noise variables such that they are jointly independent.
The graph  $\mathcal{G}$  of an SCM is obtained by creating one vertex for each  $X_{j}$  and drawing directed edges from each parent in  $\mathbf{PA}_j$  to  $X_{j}$ . This graph is assumed to be acyclic.
Below, we show that a unique causal space that corresponds to such an SCM can be constructed.