SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mingzhou Liu<sup>1,2</sup>
Xinwei Sun $^{3*}$
Lingjing Hu
Yizhou Wang $^{1,2,5}$
$^{1}$  School of Computer Science, Peking University
$^{2}$  Center on Frontiers of Computing Studies (CFCS), Peking University
$^{3}$  School of Data Science, Fudan University
$^{4}$  Yanjing Medical College, Capital Medical University
Inferring causal structures from time series data is the central interest of many scientific inquiries. A major barrier to such inference is the problem of subsampling, i.e., the frequency of measurement is much lower than that of causal influence. To overcome this problem, numerous methods have been proposed, yet either was limited to the linear case or failed to achieve identifiability. In this paper, we propose a constraint-based algorithm that can identify the entire causal structure from subsampled time series, without any parametric constraint. Our observation is that the challenge of subsampling arises mainly from hidden variables at the unobserved time steps. Meanwhile, every hidden variable has an observed proxy, which is essentially itself at some observable time in the future, benefiting from the temporal structure. Based on these, we can leverage the proxies to remove the bias induced by the hidden variables and hence achieve identifiability. Following this intuition, we propose a proxy-based causal discovery algorithm. Our algorithm is nonparametric and can achieve full causal identification. Theoretical advantages are reflected in synthetic and real-world experiments. Our code is available at https://github.com/lmz123321/proxy_causal_discovery.
Temporal systems are the primary subject of causal modeling in many sciences, such as pathology, neuroscience, and economics. For these systems, a common issue is the difficulty of collecting sufficiently refined timescale data. That is, we can only observe a subsampled version of the true causal interaction. This can cause serious problems for causal identification, since full observation (i.e., causal sufficiency [1]) is believed to be an important condition for causal identification, and the violation of which (e.g., the existence of hidden variables) can induce strong bias [2].
Example 1.1. Take the pathology study of Alzheimer's disease (AD) as an example. In AD, patients suffer from memory loss due to the atrophy of memory-related brain regions such as the Hippocampus [3]. To monitor such atrophy, the standard protocol is to perform a MRI examination on the brain every six months [4]. However, many studies have shown that the disease can progress much more rapidly than this [5, 6]. For this reason, when recovering the causal interactions between brain regions, those at the unobserved time steps constitute hidden variables and induce spurious edges in the causal graph, as shown in Figure 1 (b).
To resolve this problem, many attempts have been made to recover the correct causal relations from subsampled data. However, none of them achieved general identifiability results. For example, in
*Correspondence to sunxinwei@fudan.edu.cn
(a) consider subsampling
Figure 1: Recovered causal pathways in Alzheimer's disease, with (a) our method that explicitly models the subsampling and (b) Dynotears [7] that neglects subsampling. Spurious edges that contradict clinical studies [8, 9] are marked red. Please refer to Section 5.2 and Figure 7 for details.
(b) ignore subsampling
[10, 11, 12], identifiability was achieved only for linear data. As for nonlinear data, only a small part of the causal information (i.e., an equivalence class) could be identified [13, 14, 15, 16]. Indeed, encountering such difficulties is no coincidence, considering the fact that there are many hidden variables in the system. Particularly, the hidden mediators prevent us from distinguishing the direct causation from possible mediation and hence only allow us to recover some ancestral information.
In this paper, we propose a constraint-based algorithm that can identify the entire causal structure from subsampled data, without any parametric assumption. Our departure here is that in time series, every hidden variable has an observed descendant, which is essentially itself at some observable time in the future<sup>1</sup>. Therefore, we can use the observed descendant as its proxy variable [17] and adjust for the bias induced by it. For this purpose, we first represent the ancestral information, which is composed of both direct causation and hidden mediation, with the Maximal Ancestral Graph (MAG) [18]. Then, to distinguish causation from mediation, we use the observed descendants of the hidden mediators as their proxy variables and adjust for their induced mediation bias. We show that our strategy can achieve a complete identification of the causal relations entailed in the time series. It is essentially nonparametric and only relies on the smoothness and completeness of the structural equation to conduct valid proxy-based adjustment [19].
Back to the AD example, as shown in Figure 1, our method can recover causal pathways that align well with existing clinical studies [8, 9], while compared baselines fail to do so.
Our contributions are summarized below:
1. We establish nonparametric causal identification in subsampled time series.
2. We propose an algorithm to practically recover the causal graph.
3. We achieve more accurate causal identification than others on synthetic and real data.
The rest of the paper is organized as follows. First, in Section 2, we introduce the problem setting, basic assumptions, and related literature needed to understand our work. Second, in Section 3, we establish theories on structural identifiability. Then, based on these theories, in Section 4, we introduce the proposed causal discovery algorithm and verify its effectiveness in Section 5. Finally, we conclude the paper and discuss future works in Section 6.
For example, the hidden variable  $X(t_{u})$  at the unobserved time step  $t_{u}$  with  $X(t_{u}) \to X(t_{u} + 1) \to X(t_{u} + 2) \to \dots \to X(t_{o})$  has an observable descendant  $X(t_{o})$  at some observable time  $t_{o}$  in the future.
(a) Summary graph
(b) Full time DAG with  $k = 2$
(c) MAG over observed variables
Figure 2: Graph terminologies: (a) Summary graph. (b) Full time DAG with subsampling factor  $k = 2$ . Variables at observed time steps  $t_1, t_3, t_5$  are marked gray, while those at unobserved time steps  $t_2, t_4$  are dashed. (c) MAG over observed variables. The edges  $A(t_1) \to M(t_3)$  and  $A(t_3) \leftrightarrow M(t_3)$  are (resp.) induced by the inducing paths  $A(t_1) \to A(t_2) \to M(t_3)$  and  $A(t_3) \gets A(t_2) \to M(t_3)$  in the full time DAG. Please refer to the appendix for detailed explanations.
We start by introducing the causal framework and related literature needed to understand our work.
Time series model. Let  $\mathbf{X}(t) \coloneqq [X_1(t), \dots, X_d(t)]$  be a multivariate time series with  $d$  variables defined at discrete time steps  $t = 1, \dots, T$ . We assume the data is generated by a first-order structural vector autoregression (SVAR) process [16]:
$$
X _ {i} (t) = f _ {i} \left(\mathbf {P A} _ {i} (t - 1), N _ {i}\right), \tag {1}
$$
where  $f_{i}$  is the structural function,  $\mathbf{PA}_i$  is the parent of  $X_{i}$ , and  $N_{i}$  is the exogenous noise.
Implicit in (1) is the assumptions that cause precedes effect, and that causation is invariant to time, i.e., the structural function  $f_{i}$  and the causal parents  $\mathbf{PA}_i$  keeps unchanged across time steps. These assumptions carry the fundamental beliefs of temporal precedence and stability on causality [1], therefore are widely adopted by existing works [10, 11, 12, 14, 16].
Subsampling. The subsampling problem means that model (1) can be only observed every  $k$  steps [10, 12, 16]. That is, we can only observe  $\mathbf{X}(1),\mathbf{X}(k + 1),\dots,\mathbf{X}(\lfloor \frac{T}{k}\rfloor k + 1)$ .
Graph terminologies. Causal relations entailed in model (1) can be represented by causal graphs (Figure 2). We first introduce the full time Directed Acyclic Graph (DAG) [20], which provides a complete description of the dynamics in the system.
Definition 2.1 (Full time DAG). Let  $G \coloneqq (\mathbf{V}, \mathbf{E})$  be the associated full time DAG of model (1). The vertex set  $\mathbf{V} \coloneqq \{\mathbf{X}(t)\}_{t=1}^{T}$ , the edge set  $\mathbf{E}$  contains  $X_{i}(t-1) \to X_{j}(t)$  iff  $X_{i} \in \mathbf{PA}_{j}$ .
We assume the full time DAG is Markovian and faithful to the joint distribution  $\mathbb{P}(\mathbf{X}(1),\dots,\mathbf{X}(T))$
Assumption 2.2 (Markovian and faithfulness). For disjoint vertex sets  $\mathbf{A},\mathbf{B},\mathbf{Z}\subseteq \mathbf{V}$ $\mathbf{A}\perp \mathbf{B}|\mathbf{Z}\Leftrightarrow$ $\mathbf{A}\perp_{G}\mathbf{B}|\mathbf{Z}$ , where  $\perp_G$  denotes  $d$ -separation in  $G$ .
In practice, it is often sufficient to know the causal relations between time series as a whole, without knowing precisely the relations between time instants [21]. In this regard, we can summarize the causal relations with the summary graph [20].
Definition 2.3 (Summary graph). Let  $G \coloneqq (\mathbf{V}, \mathbf{E})$  be the associated summary graph of model (1). The vertex set  $\mathbf{V} \coloneqq \mathbf{X}$ , the edge set  $\mathbf{E}$  contains  $X_{i} \to X_{j} (i \neq j)$  iff  $X_{i} \in \mathbf{PA}_{j}$ .
In this paper, our goal is to recover the summary graph, given data observed at  $t = 1, k + 1, \dots, \left\lfloor \frac{T}{k} \right\rfloor k + 1$ .
For this purpose, we need a structure to represent the (marginal) causal relations between observed variables and hence link the observation distribution to the summary graph. Here, we use the Maximal Ancestral Graph (MAG) [22], since it can represent casual relations when unobserved variables exist.
Specifically, for the variable set  $\mathbf{V} := \{\mathbf{X}(t)\}_{t=1}^{T}$ , let  $\mathbf{O} := \{\mathbf{X}(1), \dots, \mathbf{X}(\lfloor \frac{T}{k} \rfloor k + 1)\}$  be the observed subset and  $\mathbf{L} := \mathbf{V} \backslash \mathbf{O}$  be the unobserved subset. Let  $\mathbf{A}\mathbf{n}(A)$  be the ancestor set of  $A$ . Then, given any full time DAG  $G$  over  $\mathbf{V}$ , the corresponding MAG  $M_G$  over  $\mathbf{O}$  is defined as follows:
Definition 2.4 (MAG). In the MAG  $M_G$ , two vertices  $A, B \in \mathbf{O}$  are adjacent iff there is an inducing path.  $^2$  relative to  $\mathbf{L}$  between them in  $G$ . Edge orientation is:
1.  $A\to B$  in  $M_G$  if  $A\in \mathbf{An}(B)$  in  $G$
2.  $A\gets B$  in  $M_G$  if  $B\in \mathbf{An}(A)$  in  $G$
3.  $A\leftrightarrow B$  in  $M_G$  if  $A\notin \mathbf{An}(B)$  and  $B\notin \mathbf{An}(A)$  in  $G$
Remark 2.5. In the MAG, the directed edge  $A \to B$  means  $A$  is the ancestor of  $B$ . The bidirected edge  $A \leftrightarrow B$  means there is an unobserved confounder  $U$  between  $A$  and  $B$ .
Proximal causal discovery. Recent works [23, 19] found that one could use the descendant (i.e., proxy) of an unobserved variable  $M$  to differentiate direct causation  $A \rightarrow B$  from hidden mediation  $A \rightarrow M \rightarrow B$ . Specifically, suppose that:
Assumption 2.6 (Smoothness). For each causal pair  $A \to B$ , the map  $a \mapsto \mathbb{P}(B|A = a)$  is Lipschitz with respect to the Total Variation (TV) distance, that is, there exists a constant  $L$  such that:
$$
\operatorname {T V} \left(\mathbb {P} (B | A = a), \mathbb {P} (B | A = a ^ {\prime})\right) \leq L | a - a ^ {\prime} |.
$$
Assumption 2.7 (Completeness). For each causal pair  $A \to B$ , the conditional distribution  $\mathbb{P}(B|A)$  is complete, that is, for any function  $g$ , we have:
$$
E \{g (b) | a \} = 0 \text {a l m o s t s u r e l y i f a n d o n l y i f} g (b) = 0 \text {a l m o s t s u r e l y}.
$$
Remark 2.8. Roughly, Assumption 2.6 relies on the continuity of the structural function  $f_{i}$  and Assumption 2.7 requires that the characterization function of the exogenous noise  $N_{i}$  is non-zero. Please refer to [19] for a detailed discussion.
Under the above assumptions, [19] showed that:
Theorem 2.9 ([19]). Suppose that  $A$  and  $B$  are mediated by an unobserved variable  $M$ , and that  $M$  has a proxy variable  $M'$  satisfying  $A \perp_G M'|M$ . Then, we can identify whether  $A \to B$  by testing whether  $pr(b|a) \sim pr(m'|a)$  has a linear relation. If the linear relation exists, then  $A \neq B$ , otherwise, we have  $A \to B$ .
Remark 2.10. The linear relation can be tested by checking the least square residual of regressing  $pr(b|a)$  on  $pr(m'|a)$ . Please refer to [19] for details.
To ensure the existence of the proxy variable, we require the following self causation assumption, which states that each variable  $X_{i}(t)$  is influenced by its past  $X_{i}(t - 1)$ .
Assumption 2.11 (Self causation). In model (1),  $X_{i}\in \mathbf{PA}_{i}$  for each  $i = 1,\dots,d$
Remark 2.12. In time series statistics, this is also known as the autocorrelation assumption [24, 25].
3 Structural identifiability
In this section, we establish the identifiability of the summary graph. For this purpose, we connect the observational distribution to the summary graph with the bridge of the MAG.
Specifically, our analysis consists of three progressive results: Proposition 3.1, Proposition 3.3, and Theorem 3.5. First, in Proposition 3.1, we prove that the MAG can be identified from observational data. Then, in Proposition 3.3, we show that the identified MAG almost uniquely reflects the structure of the summary graph, in the sense that only edges connecting vertices and their ancestors cannot be determined due to hidden mediators. Finally, in Theorem 3.5, we identify these undetermined edges with the proxies of the hidden mediators and therefore identify the whole summary graph.
$^2$ A path  $p$  between  $A, B \in \mathbf{O}$  is an inducing path relative to  $\mathbf{L}$  if every non-endpoint vertex on  $p$  is either in  $\mathbf{L}$  or a collider, and every collider on  $p$  is an ancestor of either  $A$  or  $B$ .
(a) Directed path
(b) Confounding structure
Figure 3: Illustration of Definition 3.2. (a) A directed path from  $A$  to  $B$  with length  $l$ . (b) A confounding structure between  $A$  and  $B$  with lengths  $(r, q)$ .
Next, we first introduce Proposition 3.1, which shows that the MAG is identifiable.
Proposition 3.1 (Identifiability of the MAG). Assuming model (1) and Assumption 2.2, then the MAG over the observed variable set  $\mathbf{O}$  is identifiable, i.e., its skeleton and edge orientations can be uniquely derived from the joint distribution  $\mathbb{P}(\mathbf{O})$ .
Since the identifiability of the MAG's skeleton under the faithfulness assumption is a well-known result [18], we focus on explaining the identification of edge orientations. Specifically, we will divide the edges into two classes: the instantaneous edges and the lagged edges, and discuss their orientations respectively.
The instantaneous edge, e.g.,  $A(t_{3}) \leftrightarrow M(t_{3})$ , is an edge that connects two vertices at the same time. Since we assume cause must precede effect, the instantaneous edge does not represent causation but latent confounding, therefore is bidirected according to Definition 2.4. For example, in Figure 2 (c), the instantaneous edge  $A(t_{3}) \leftrightarrow M(t_{3})$  represents the latent confounding  $A(t_{3}) \leftrightarrow A(t_{2}) \rightarrow M(t_{3})$  between  $A(t_{3})$  and  $M(t_{3})$ . On the other hand, the lagged edge, e.g.,  $A(t_{1}) \rightarrow M(t_{3})$ , is an edge that connects two vertices at different time. The lagged edge represents ancestral information and hence is directed  $(\rightarrow)$  from the past to the future.
To connect the identified MAG to the summary graph, we first define the following graph structures, examples of which are shown in Figure 3:
Definition 3.2 (Graph structures). In the summary graph,
1. A directed path  $p_{AB}$  from  $A$  to  $B$  with length  $l$  is a sequence of distinct vertices  $A, V_1, \ldots, V_l, B$  where each vertex points to its successor. Two directed paths  $p_{AB_1}, p_{AB_2}$  are called disjoint if they do not share any non-startpoint vertex.
2. A confounding structure  $c_{AB}$  between  $A$  and  $B$  with lengths  $(r, q)$  consists of a vertex  $U$ , a directed path  $p_{UA}$  from  $U$  to  $A$  with length  $r$ , and a directed path  $p_{UB}$  from  $U$  to  $B$  with length  $q$ , where  $p_{UA}$  and  $p_{UB}$  are disjoint.
In the following, we explain the relationship between the identified MAG and the summary graph. In particular, we will discuss what the directed and bidirected edges in the MAG (resp.) imply about the summary graph.
According to Definition 2.4, the directed edge  $A(t) \to B(t + k)$  in the MAG means  $A$  is the ancestor of  $B$ . Therefore, in the summary graph, there is either  $A \to B$  or a directed path from  $A$  to  $B$ . On the other hand, the bidirected edge  $A(t + k) \leftrightarrow B(t + k)$  in the MAG means there is a latent confounder between them, which can be either  $A(t + 1)$ , i.e.,  $A(t + k) \leftrightarrow A(t + 1) \to \dots \to B(t + k)$ , or a third variable  $U(t + 1)$ , i.e.,  $A(t + k) \leftrightarrow U(t + 1) \to \dots \to B(t + k)$ . Hence, in the summary graph, there is either a directed path from  $A$  to  $B$  or a confounding structure between them.
Summarizing the above observations, we have the following proposition:
Proposition 3.3 (MAG to summary graph). If there are  $A(t) \to B(t + k)$  and  $A(t + k) \leftrightarrow B(t + k)$  in the MAG, then, in the summary graph, there is either
1.  $A\to B$  2.a directed path from  $A$  to  $B$  with length  $l\leq k - 2$  ,or
3. a directed path from  $A$  to  $B$  with length  $l = k - 1$  and a confounding structure between them with lengths  $(r \leq k - 2, q \leq k - 2)$ .
(a) Direct effect  $A\to B$
Summary graph
(b) Indirect effect  $A\to \dots \to B$
Full time DAG
MAG
Figure 4: Distinguishing the direct effect  $(A\to B)$  from the indirect one  $(A\rightarrow \dots \rightarrow B)$  with proxy variables. Note that though (a) and (b) have the same MAG, only the indirect effect  $A(t_{1})\rightarrow M(t_{2})\rightarrow B(t_{3})$  in (b) can be  $d$  -separated by  $M(t_{2})$  . To test the  $d$  -separation, we can use  $M(t_{3})$  as the proxy variable.
Remark 3.4. In Proposition 3.3, the maximum length of the directed path is  $k - 1$ , this is because any path longer than this can not be an inducing path and therefore can not induce  $A(t) \to B(t + k)$  in the MAG. Please refer to the appendix for details.
Proposition 3.3 provides a necessary condition for having  $A \to B$  in the summary graph, i.e., there are both  $A(t) \to B(t + k)$  and  $A(t + k) \leftrightarrow B(t + k)$  in the MAG. It also inspires us that, to make the condition sufficient, we need to distinguish the direct effect  $(A \to B)$  from the indirect one  $(A \to \dots \to B)$ .
For this purpose, we propose to use the proximal causal discovery method. Before preceding any technical detail, we first brief our idea below. The indirect effect, unlike the direct one, relies on mediation and therefore can be  $d$ -separated by the (unobserved) mediator. To test the  $d$ -separation, we can use the observed proxy of the mediator, thanks to the self causation assumption in Assumption 2.11. For example, in Figure 4 (b), the indirect effect  $A(t_{1}) \rightarrow M(t_{2}) \rightarrow B(t_{3})$  from  $A$  to  $B$  can be  $d$ -separated by the mediator  $M(t_{2})$ , who has an observed proxy  $M(t_{3})$ .
For the general case, the separation set is a bit more complicated to ensure all paths except the one representing direct effect are  $d$ -separated. Specifically, for two vertices  $A(t)$  and  $B(t + k)$ , the separation set is the union of two sets  $\mathbf{M}(t + 1) \cup \mathbf{S}(t)$ . The set  $\mathbf{M}(t + 1)$  contains possible mediators, namely  $A(t + 1)$  and any vertex  $M_{i}(t + 1) (M_{i} \neq B)$  such that  $A(t) \to M_{i}(t + k)$  in the MAG and  $M_{i}$  is not  $B$ 's descendant<sup>3</sup>. The set  $\mathbf{S}(t)$  is used to  $d$ -separate possible back-door paths between  $A(t)$  and  $B(t + k)$ , it contains any vertex  $S_{i}(t) (S_{i} \neq A)$  such that  $S_{i}(t) \to B(t + k)$  or  $S_{i}(t) \to M_{j}(t + k)$  for some  $M_{j} \in \mathbf{M}$  in the MAG.
Equipped with the separation set, we then have the following identifiability result:
Theorem 3.5 (Identifiability of the summary graph). Assuming model (1), Assumption 2.2, and Assumptions 2.6, 2.7, 2.11, then the summary graph is identifiable. Specifically,
1. There is  $A \to B$  in the summary graph iff there are  $A(t) \to B(t + k), A(t + k) \leftrightarrow B(t + k)$  in the MAG, and the set  $\mathbf{M}(t + 1) \cup \mathbf{S}(t)$  is not sufficient to d-separate  $A(t), B(t + k)$  in the full time DAG.
2. The condition "the set  $\mathbf{M}(t + 1) \cup \mathbf{S}(t)$  is not sufficient to  $d$ -separate  $A(t)$ ,  $B(t + k)$  in the full time DAG" can be tested by the proxy variable  $\mathbf{M}(t + k)$  of the unobserved set  $\mathbf{M}(t + 1)$ .
<sup>3</sup>This can be justified from the MAG. Specifically, if  $M_{i}$  is the descendant of  $B$ , then there is  $B(t) \to V_{1}(t + k), V_{1}(t) \to V_{2}(t + k), \ldots, V_{l}(t) \to M_{i}(t + k)$  in the MAG.
Theorem 3.5 provides a necessary and sufficient condition for having  $A \to B$  in the summary graph. This condition is testable with the proxy  $\mathbf{M}(t + k)$  of the unobserved set  $\mathbf{M}(t + 1)$  ( $\mathbf{S}(t)$  is observable). The validity of such proxying, i.e.,  $A(t) \perp_G \mathbf{M}(t + k)|\mathbf{M}(t + 1), \mathbf{S}(t)$  is proved in the appendix.
4 Discovery algorithm
Equipped with the identifiability results in Proposition 3.1 and Theorem 3.5, we can practically discover the summary graph with Algorithm 1. Specifically, given the observed data points, Algorithm 1 first recovers the MAG according to Proposition 3.1. Then, for each pair of vertices in the summary graph, it first uses the necessary condition of "having  $A(t) \to B(t + k)$  and  $A(t + k) \leftrightarrow B(t + k)$  in the MAG" to construct underdetermined edges. Finally, among these underdetermined edges, it uses the proxy variable approach to identify direct causation and remove the indirect one, according to Theorem 3.5.
Below, we first introduce the Partially Determined Directed Acyclic Graph (PD-DAG), which will be used to represent the intermediate result of the algorithm.
Definition 4.1 (PD-DAG). A PD-DAG is a DAG with two kinds of directed edges: solid  $(\rightarrow)$  and dashed  $(\dashrightarrow)$ . A solid edge represents a determined causal relation, while a dashed one means the causal relation is still underdetermined. We use a meta-symbol, asterisk  $(\ast \rightarrow)$ , to denote any of the two edges. In a PD-DAG  $G = (\mathbf{V}, \mathbf{E})$ , the vertex  $A$  points to the vertex  $B$  if the edge  $A \ast \rightarrow B \in \mathbf{E}$ . The definition of the directed path and confounding structure are the same as in the summary graph.