SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanyuan Wang
The University of Melbourne  
yuanyuanw2@student.unimelb.edu.au
Xi Geng
The University of Melbourne xi.geng@unimelb.edu.au
The University of Melbourne  
wei.huang@unimelb.edu.au
Biwei Huang
University of California, San Diego  
bih007@ucsd.edu
Mingming Gong *
The University of Melbourne
mingming.gong@unimelb.edu.au
In this paper, we present conditions for identifying the generator of a linear stochastic differential equation (SDE) from the distribution of its solution process with a given fixed initial state. These identifiability conditions are crucial in causal inference using linear SDEs as they enable the identification of the post-intervention distributions from its observational distribution. Specifically, we derive a sufficient and necessary condition for identifying the generator of linear SDEs with additive noise, as well as a sufficient condition for identifying the generator of linear SDEs with multiplicative noise. We show that the conditions derived for both types of SDEs are generic. Moreover, we offer geometric interpretations of the derived identifiability conditions to enhance their understanding. To validate our theoretical results, we perform a series of simulations, which support and substantiate the established findings.
Stochastic differential equations (SDEs) are a powerful mathematical tool for modelling dynamic systems subject to random fluctuations. These equations are widely used in various scientific disciplines, including finance [11, 30, 40], physics [53, 55, 58], biology [2, 8, 61] and engineering [18, 44, 55]. In recent years, SDEs have garnered growing interest in the machine learning research community. Specifically, they have been used for tasks such as modelling time series data [19, 21, 33] and estimating causal effects [5, 36, 47].
To enhance understanding we first introduce the SDEs of our interest, which are multidimensional linear SDEs with additive and multiplicative noise, respectively. Consider an  $m$ -dimensional standard Brownian motion defined on a filtered probability space  $(\Omega, \mathcal{F}, \mathbb{P}, \{\mathcal{F}_t\})$ , denoted by  $W := \{W_t = [W_{1,t}, \dots, W_{m,t}]^\top : 0 \leqslant t < \infty\}$ . Let  $X_t \in \mathbb{R}^d$  be the state at time  $t$  and let  $x_0 \in \mathbb{R}^d$  be a constant vector denoting the initial state of the system, we present the forms of the aforementioned two linear SDEs.
1. Linear SDEs with additive noise.
$$
d X _ {t} = A X _ {t} d t + G d W _ {t}, \quad X _ {0} = x _ {0}, \tag {1}
$$
where  $0 \leqslant t < \infty$ ,  $A \in \mathbb{R}^{d \times d}$  and  $G \in \mathbb{R}^{d \times m}$  are some constant matrices.
2. Linear SDEs with multiplicative noise.
$$
d X _ {t} = A X _ {t} d t + \sum_ {k = 1} ^ {m} G _ {k} X _ {t} d W _ {k, t}, \quad X _ {0} = x _ {0}, \tag {2}
$$
where  $0 \leqslant t < \infty$ ,  $A, G_{k} \in \mathbb{R}^{d \times d}$  for  $k = 1, \ldots, m$  are some constant matrices.
Linear SDEs are wildly used in financial modeling for tasks like asset pricing, risk assessment, and portfolio optimization [3, 10, 24]. Where they are used to model the evolution of financial variables, such as stock prices and interest rates. Furthermore, linear SDEs are also used in genomic research, for instance, they are used for modeling the gene expression in the yeast microorganism Saccharomyces Cerevisiae [17]. The identifiability analysis of linear SDEs is essential for reliable causal inference of dynamic systems governed by these equations. For example, in the case of Saccharomyces Cerevisiae, one aims to identify the system such that making reliable causal inference when interventions are introduced. Such interventions may involve deliberate knockout of specific genes to achieve optimal growth of an organism. In this regard, identifiability analysis plays a pivotal role in ensuring reliable predictions concerning the impact of interventions on the system.
Previous studies on identifiability analysis of linear SDEs have primarily focused on Gaussian diffusions, as described by the SDE (1) [6, 16, 23, 28, 35, 42]. These studies are typically based on observations located on one trajectory of the system and thus require restrictive identifiability conditions, such as the ergodicity of the diffusion or other restrictive requirements on the eigenvalues of matrix  $A$ . However, in practical applications, multiple trajectories of the dynamic system can often be accessed [15, 31, 45, 54]. In particular, these multiple trajectories may start from the same initial state, e.g., in experimental studies where repeated trials or experiments are conducted under the same conditions [9, 12, 26, 29] or when the same experiment is performed on multiple identical units [48]. To this end, this work presents an identifiability analysis for linear SDEs based on the distribution of the observational process with a given fixed initial state. Furthermore, our study is not restricted to Gaussian diffusions (1), but also encompasses linear SDEs with multiplicative noise (2). Importantly, the conditions derived for both types of SDEs are generic, meaning that the set of system parameters that violate the proposed conditions has Lebesgue measure zero.
Traditional identifiability analysis of dynamic systems focuses on deriving conditions under which a unique set of parameters can be obtained from error-free observational data. However, our analysis of dynamic systems that are described by SDEs aims to uncover conditions that would enable a unique generator to be obtained from its observational distribution. Our motivation for identifying generators of SDEs is twofold. Firstly, obtaining a unique set of parameters from the distribution of a stochastic process described by an SDE is generally unfeasible. For example, in the SDE (1), parameter  $G$  cannot be uniquely identified since one can only identify  $GG^{\top}$  based on the distribution of its solution process [17, 28]. Secondly, the identifiability of an SDE's generator suffices for reliable causal inferences for this system. Note that, in the context of SDEs, the main task of causal analysis is to identify the post-intervention distributions from the observational distribution. As proposed in [17], for an SDE satisfying specific criteria, the post-intervention distributions are identifiable from the generator of this SDE. Consequently, the intricate task of unraveling causality can be decomposed into two constituent components through the generator. This paper aims to uncover conditions under which the generator of a linear SDE attains identifiability from the observational distribution. By establishing these identifiability conditions, we can effectively address the causality task for linear SDEs.
In this paper, we present a sufficient and necessary identifiability condition for the generator of linear SDEs with additive noise (1), along with a sufficient identifiability condition for the generator of linear SDEs with multiplicative noise (2).
2 Background knowledge
In this section, we introduce some background knowledge of linear SDEs. In addition, we provide a concise overview of the causal interpretation of SDEs, which is a critical aspect of understanding the nature and dynamics of these equations. This interpretation also forms a strong basis for the motivation of this research.
2.1 Background knowledge of linear SDEs
The solution to the SDE (1) can be explicitly expressed as (cf. [50]):
$$
X _ {t} := X (t; x _ {0}, A, G) = e ^ {A t} x _ {0} + \int_ {0} ^ {t} e ^ {A (t - s)} G d W _ {s}. \tag {3}
$$
Note that in the context of our study, the solution stands for the strong solution, refer to [22] for its detailed definition.
In general, obtaining an explicit expression for the solution to the SDE (2) is not feasible. In fact, an explicit solution can be obtained when the matrices  $A, G_1, \ldots, G_k$  commute, that is when
$$
A G _ {k} = G _ {k} A \quad \text {a n d} \quad G _ {k} G _ {l} = G _ {l} G _ {k} \tag {4}
$$
holds for all  $k, l = 1, \dots, m$  (cf. [25]). However, the conditions described in (4) are too restrictive and impractical. Therefore, this study will focus on the general case of the SDE (2).
We know that both the SDE (1) and the SDE (2) admit unique solutions that manifest as continuous stochastic processes [22]. A  $d$ -dimensional stochastic process is a collection of  $\mathbb{R}^d$ -valued random variables, denoted as  $X = \{X_t; 0 \leqslant t < \infty\}$  defined on some probability space. When comparing two stochastic processes,  $X$  and  $\tilde{X}$ , that are defined on the same probability space  $(\Omega, \mathcal{F}, \mathbb{P})$ , various notions of equality may be considered. In this study, we adopt the notion of equality with respect to their distributions, which is a weaker requirement than strict equivalence, see [22] for relevant notions. We now present the definition of the distribution of a stochastic process.
Definition 2.1. Let  $X$  be a random variable on a probability space  $(\Omega, \mathcal{F}, \mathbb{P})$  with values in a measurable space  $(S, \mathcal{B}(S))$ , i.e., the function  $X: \Omega \to S$  is  $\mathcal{F} / \mathcal{B}(S)$ -measurable. Then, the distribution of the random variable  $X$  is the probability measure  $P^X$  on  $(S, \mathcal{B}(S))$  given by
$$
P ^ {X} (B) = \mathbb {P} (X \in B) = \mathbb {P} \{\omega \in \Omega : X (\omega) \in B \}, \quad B \in \mathcal {B} (S).
$$
When  $X \coloneqq \{X_{t}; 0 \leqslant t < \infty\}$  is a continuous stochastic process on  $(\Omega, \mathcal{F}, \mathbb{P})$ , and  $S = C[0, \infty)$ , such an  $X$  can be regarded as a random variable on  $(\Omega, \mathcal{F}, \mathbb{P})$  with values in  $(C[0, \infty), \mathcal{B}(C[0, \infty)))$ , and  $P^{X}$  is called the distribution of  $X$ . Here  $C[0, \infty)$  stands for the space of all continuous, real-valued functions on  $[0, \infty]$ .
It is noteworthy that the distribution of a continuous process can be uniquely determined by its finite-dimensional distributions. Hence, if two stochastic processes, labelled as  $X$  and  $\tilde{X}$ , share identical finite-dimensional distributions, they are regarded as equivalent in distribution, denoted by  $X \stackrel{\mathrm{d}}{=} \tilde{X}$ . Relevant concepts and theories regarding this property can be found in [22].
The generator of a stochastic process is typically represented by a differential operator that acts on functions. It provides information about how a function evolves over time in the context of the underlying stochastic process. Mathematically, the generator of a stochastic process  $X_{t}$  can be defined as
$$
(\mathcal {L} f) (x) = \lim  _ {s \rightarrow 0} \frac {\mathbb {E} [ f (X _ {t + s}) - f (X _ {t}) | X _ {t} = x ]}{s},
$$
where  $f$  is a suitably regular function.
In the following, we present the generator of the SDEs under consideration. Obviously, both the SDE (1) and the SDE (2) conform to the general form:
$$
d X _ {t} = b \left(X _ {t}\right) d t + \sigma \left(X _ {t}\right) d W _ {t}, \quad X _ {0} = x _ {0}. \tag {5}
$$
where  $b$  and  $\sigma$  are locally Lipschitz continuous in the space variable  $x$ . The generator  $\mathcal{L}$  of the SDE (5) can be explicitly computed by utilizing Itô's formula (cf. [50]).
Proposition 2.1. Let  $X$  be a stochastic process defined by the SDE (5). The generator  $\mathcal{L}$  of  $X$  on  $C_b^2(\mathbb{R}^d)$  is given by
$$
(\mathcal {L} f) (x) := \sum_ {i = 1} ^ {d} b _ {i} (x) \frac {\partial f (x)}{\partial x _ {i}} + \frac {1}{2} \sum_ {i, j = 1} ^ {d} c _ {i j} (x) \frac {\partial^ {2} f (x)}{\partial x _ {i} \partial x _ {j}} \tag {6}
$$
for  $f \in C_b^2(\mathbb{R}^d)$  and  $x \in \mathbb{R}^d$ , where  $c(x) = \sigma(x) \cdot \sigma(x)^\top$  is a  $d \times d$  matrix, and  $C_b^2(\mathbb{R}^d)$  denotes the space of continuous functions on  $\mathbb{R}^d$  that have bounded derivatives up to order two.
2.2 Causal interpretation of SDEs
An important motivation for the identification of the generator of an SDE lies in the desire to infer reliable causality within dynamic models described by SDEs. In this subsection, we aim to provide some necessary background knowledge on the causal interpretation of SDEs. Consider the general SDE framework described as:
$$
d X _ {t} = a \left(X _ {t}\right) d Z _ {t}, \quad X _ {0} = x _ {0}, \tag {7}
$$
where  $Z$  is a  $p$ -dimensional semimartingale and  $a: \mathbb{R}^d \to \mathbb{R}^{d \times p}$  is a continuous mapping. By writing the SDE (7) in integral form
$$
X _ {t} ^ {i} = x _ {0} ^ {i} + \sum_ {j = 1} ^ {p} \int_ {0} ^ {t} a _ {i j} \left(X _ {s}\right) d Z _ {s} ^ {j}, \quad i \leqslant d. \tag {8}
$$
The authors of [17] proposed a mathematical definition of the SDE resulting from an intervention to the SDE (8). In the following,  $X^{(-l)}$  denotes the  $(d - 1)$ -dimensional vector that results from the removal of the  $l$ -th coordinate of  $X \in \mathbb{R}^d$ .
Definition 2.2. [17, Definition 2.4.] Consider some  $l \leqslant d$  and  $\zeta: \mathbb{R}^{d-1} \to \mathbb{R}$ . The SDE arising from (8) under the intervention  $X_{t}^{l} \coloneqq \zeta(X_{t}^{(-l)})$  is the  $(d-1)$ -dimensional equation
$$
\left(Y ^ {(- l)}\right) _ {t} ^ {i} = x _ {0} ^ {i} + \sum_ {j = 1} ^ {p} \int_ {0} ^ {t} b _ {i j} \left(Y _ {s} ^ {(- l)}\right) d Z _ {s} ^ {j}, \quad i \neq l, \tag {9}
$$
where  $b: \mathbb{R}^{d-1} \to \mathbb{R}^{(d-1) \times p}$  is defined by  $b_{ij}(y) = a_{ij}(y_1, \ldots, \zeta(y), \ldots, y_d)$  for  $i \neq l$  and  $j \leqslant p$  and the  $\zeta(y)$  is on the  $l$ -th coordinate.
Definition 2.2 presents a natural approach to defining how interventions should affect dynamic systems governed by SDEs. We adopt the same notations as used in [17]. Assuming (8) and (9) have unique solutions for all interventions, we refer to (8) as the observational SDE, to its solution as the observational process, to the distribution of its solution as observational distribution, to (9) as the post-intervention SDE, to the solution of (9) as the post-intervention process, and to the distribution of the solution of (9) as the post-intervention distribution. The authors in [17] related Definition 2.2 to mainstream causal concepts by establishing a mathematical connection between SDEs and structural equation models (SEMs). Specifically, the authors showed that under regularity assumptions, the solution to the post-intervention SDE is equal to the limit of a sequence of interventions in SEMs based on the Euler scheme of the observational SDE. Despite the fact that the parameters of the SDEs are generally not identifiable from the observational distribution, the post-intervention distributions can be identified, thus enabling causal inference of the system. To this end, Sokol and Hansen [17] derived a condition under which the generator associated with the observational SDE allows for the identification of the post-intervention distributions. We present the corresponding theory as follows.
Lemma 2.1. [17, Theorem 5.3.] Consider the SDEs
$$
d X _ {t} = a \left(X _ {t}\right) d Z _ {t}, \quad X _ {0} = x _ {0}, \tag {10}
$$
$$
d \tilde {X} _ {t} = \tilde {a} (\tilde {X} _ {t}) d \tilde {Z} _ {t}, \quad \tilde {X} _ {0} = \tilde {x} _ {0}, \tag {11}
$$
where  $Z$  is a  $p$ -dimensional Lévy process and  $\tilde{Z}$  is a  $\tilde{p}$ -dimensional Lévy process. Assume that (10) and (11) have the same generator, that  $a: \mathbb{R}^d \to \mathbb{R}^{d \times p}$  and  $\zeta: \mathbb{R}^{d-1} \to \mathbb{R}$  are Lipschitz and that the initial values have the same distribution. Then the post-intervention distributions of doing  $X^l := \zeta(X^{(-l)})$  in (10) and doing  $\tilde{X}^l := \zeta(\tilde{X}^{(-l)})$  in (11) are equal for any choice of  $\zeta$  and  $l$ .
A main task in the causality research community is to uncover the conditions under which the post-intervention distributions are identifiable from the observational distribution. In the context of dynamic systems modelled in SDEs, similar conditions need to be derived. Lemma 2.1 establishes that, for SDEs with a Lévy process as the driving noise, the post-intervention distributions can be identifiable from the generator. Nevertheless, a gap remains between the observational distribution and the SDE generator's identifiability. This work aims to address this gap by providing conditions under which the generator is identifiable from the observational distribution.
In this section, we present some prerequisites first, and then we present the main theoretical results of our study, which include the condition for the identifiability of generator that is associated with the SDE (1) / SDE (2) from the distribution of the corresponding solution process.
3.1 Prerequisites
We first show that both the SDE (1) and the SDE (2) satisfy the conditions stated in Lemma 2.1.
Lemma 3.1. Both the SDE (1) and the SDE (2) can be expressed as the form of (10), with  $Z$  being a  $p$ -dimensional Lévy process, and  $a: \mathbb{R}^d \to \mathbb{R}^{d \times p}$  being Lipschitz.
The proof of Lemma 3.1 can be found in Appendix A.1. This lemma suggests that Lemma 2.1 applies to both the SDE (1) and the SDE (2), given that they meet the specified conditions. Therefore, for either SDE, deriving the conditions that allow for the generator to be identifiable from the observational distribution is sufficient. By applying Lemma 2.1, when the intervention function  $\zeta$  is Lipschitz, the post-intervention distributions can be identified from the observational distribution under these conditions.
We then address the identifiability condition of the generator  $\mathcal{L}$  defined by (6).
Proposition 3.1. Let  $\mathcal{L}$  and  $\tilde{\mathcal{L}}$  be generators of stochastic processes defined by the form of the SDE (5) on  $C_b^2 (\mathbb{R}^d)$ , where  $\mathcal{L}$  is given by (6) and  $\tilde{\mathcal{L}}$  is given by the same expression, with  $\tilde{b} (x)$  and  $\tilde{c} (x)$  substituted for  $b(x)$  and  $c(x)$ . It then holds that the two generators  $\mathcal{L} = \tilde{\mathcal{L}}$  if and only if  $b(x) = \tilde{b} (x)$  and  $c(x) = \tilde{c} (x)$  for all  $x\in \mathbb{R}^d$ .
The proof of Proposition 3.1 can be found in Appendix A.2. This proposition states that for stochastic processes defined by the SDE (5), the generator is identifiable from functions associated with its coefficients:  $b(x)$  and  $c(x) = \sigma(x) \cdot \sigma(x)^{\top}$ .
3.2 Conditions for identifying generators of linear SDEs with additive noise
Expressing the SDE (1) in the form given by (5) yields  $b(x) = Ax$  and  $c(x) = GG^{\top}$ . Therefore, based on Proposition 3.1, we define the identifiability of the generator of the SDE (1) as follows.
Definition 3.1  $((x_0, A, G)$ -identifiability). For  $x_0 \in \mathbb{R}^d$ ,  $A \in \mathbb{R}^{d \times d}$  and  $G \in \mathbb{R}^{d \times m}$ , the generator of the SDE (1) is said to be identifiable from  $x_0$ , if for all  $\tilde{A} \in \mathbb{R}^{d \times d}$  and all  $\tilde{G} \in \mathbb{R}^{d \times m}$ , with  $(A, GG^\top) \neq (\tilde{A}, \tilde{G}\tilde{G}^\top)$ , it holds that  $X(\cdot; x_0, A, G)^2 \neq X(\cdot; x_0, \tilde{A}, \tilde{G})$ .
In the following, we begin by introducing two lemmas that serve as the foundation for deriving our main identifiability theorem.
Lemma 3.2. For  $x_0 \in \mathbb{R}^d$ ,  $A, \tilde{A} \in \mathbb{R}^{d \times d}$  and  $G, \tilde{G} \in \mathbb{R}^{d \times m}$ , let  $X_t := X(t; x_0, A, G)$ ,  $\tilde{X}_t := X(t; x_0, \tilde{A}, \tilde{G})$ , then  $X(\cdot; x_0, A, G) \stackrel{\mathrm{d}}{=} X(\cdot; x_0, \tilde{A}, \tilde{G})$  if and only if the mean  $\mathbb{E}[X_t] = \mathbb{E}[\tilde{X}_t]$  and the covariance  $\mathbb{E}\{(X_{t + h} - \mathbb{E}[X_{t + h}])(X_t - \mathbb{E}[X_t])^\top\} = \mathbb{E}\{(\tilde{X}_{t + h} - \mathbb{E}[\tilde{X}_{t + h}])(\tilde{X}_t - \mathbb{E}[\tilde{X}_t])^\top\}$  for all  $0 \leqslant t < \infty$  and  $0 \leqslant h < \infty$ .
The proof of Lemma 3.2 can be found in Appendix A.3. This lemma states that for stochastic processes modelled by the SDE (1), the equality of the distribution of two processes can be deconstructed as the equality of the mean and covariance of the state variables at all time points. Calculation shows
$$
\begin{array}{l} \mathbb {E} [ X _ {t} ] = e ^ {A t} x _ {0}, \\ V (t, t + h) := \mathbb {E} \left\{\left(X _ {t + h} - \mathbb {E} \left[ X _ {t + h} \right]\right) \left(X _ {t} - \mathbb {E} \left[ X _ {t} \right]\right) ^ {\top} \right\} \tag {12} \\ = e ^ {A h} V (t), \\ \end{array}
$$
where  $V(t) \coloneqq V(t, t)$ . Please refer to the proof A.5 of Theorem 3.4 for the detailed calculations. It can be easily checked that  $\mathbb{E}[X_t]$  follows the linear ordinary differential equation (ODE)
$$
\dot {m} (t) = A m (t), \quad m (0) = x _ {0}, \tag {13}
$$
where  $\dot{m}(t)$  denotes the first derivative of function  $m(t)$  with respect to time  $t$ . Similarly, each column of the covariance  $V(t,t + h)$  also follows the linear ODE (13) but with a different initial state: the corresponding column of  $V(t)$ . This observation allows us to leverage not only the characteristics of the SDE (1), but also the established theories [43, 57] on identifiability analysis for the ODE (13), to derive the identifiability conditions for the generator of the SDE (1).
${}^{2}X\left( {\vdots ;{x}_{0},A,G}\right)  = \left\{  {X\left( {t;{x}_{0},A,G}\right)  : 0 \leq  t < \infty }\right\}$
We adopt the same setting as in [43], discussing the case where  $A$  has distinct eigenvalues. Because random matrix theory suggests that almost every  $A \in \mathbb{R}^{d \times d}$  has  $d$  distinct eigenvalues with respect to the Lebesgue measure on  $\mathbb{R}^{d \times d}$  [43]. And the Jordan decomposition of such a matrix  $A$  follows a straightforward form which is helpful for deriving the geometric interpretation of the proposed identifiability condition. The Jordan decomposition can be expressed as  $A = Q\Lambda Q^{-1}$ , where
$$
\begin{array}{l} \Lambda = \left[ \begin{array}{c c c} J _ {1} & & \\ & \ddots & \\ & & J _ {K} \end{array} \right], \text {w i t h} J _ {k} = \left\{ \begin{array}{l l} \lambda_ {k}, & \text {i f} k = 1, \ldots , K _ {1}  , \\ \left[ \begin{array}{c c} a _ {k} & - b _ {k} \\ b _ {k} & a _ {k} \end{array} \right], & \text {i f} k = K _ {1} + 1, \ldots , K  . \end{array} \right. \\ Q = \left[ Q _ {1} \right| \dots \left| Q _ {K} \right] = \left[ v _ {1} \right| \dots \left| v _ {d} \right], \\ Q _ {k} = \left\{ \begin{array}{l l} v _ {k}, & \text {i f} k = 1, \ldots , K _ {1}  , \\ [ v _ {2 k - K _ {1} - 1} | v _ {2 k - K _ {1}} ], & \text {i f} k = K _ {1} + 1, \ldots , K  , \end{array} \right. \\ \end{array}
$$
where  $\lambda_{k}$  is a real eigenvalue of  $A$  and  $v_{k}$  is the corresponding eigenvector of  $\lambda_{k}$ , for  $k = 1, \dots, K_{1}$ . For  $k = K_{1} + 1, \dots, K$ ,  $[v_{2k - K_{1} - 1}|v_{2k - K_{1}}]$  are the corresponding "eigenvectors" of complex eigenvalues  $a_{k} \pm b_{k}i$ . Inspired by [43, Definition 2.3., Lemma 2.3.], we establish the following Lemma.
Lemma 3.3. Assuming  $A \in \mathbb{R}^{d \times d}$  has  $d$  distinct eigenvalues, with Jordan decomposition  $A = Q \Lambda Q^{-1}$ . Let  $\gamma_j \in \mathbb{R}^d$  and  $\tilde{\gamma}_j \coloneqq Q^{-1}\gamma_j \in \mathbb{R}^d$  for all  $j = 1, \ldots, n$  with  $n \geqslant 2$ . We define
$$
w _ {j, k} := \left\{ \begin{array}{l l} \tilde {\gamma} _ {j, k} \in \mathbb {R} ^ {1}, & f o r   k = 1, \ldots , K _ {1}  , \\ (\tilde {\gamma} _ {j, 2 k - K _ {1} - 1}, \tilde {\gamma} _ {j, 2 k - K _ {1}}) ^ {\top} \in \mathbb {R} ^ {2}, & f o r   k = K _ {1} + 1, \ldots , K  , \end{array} \right.
$$
where  $\tilde{\gamma}_{j,k}$  denotes the  $k$ -th entry of  $\tilde{\gamma}_j$ .  $\mathrm{rank}([ \gamma_1|A\gamma_1|\ldots |A^{d - 1}\gamma_1|\ldots |\gamma_n|A\gamma_n|\ldots |A^{d - 1}\gamma_n]) < d$  if and only if there exists  $k\in \{1,\dots ,K\}$ , such that  $|w_{j,k}| = 0$  for all  $j = 1,\dots ,n$ , where  $|w_{j,k}|$  is the absolute value of  $w_{j,k}$  for  $k = 1,\dots ,K_{1}$ , and the Euclidean norm of  $w_{j,k}$  for  $k = K_{1} + 1,\dots ,K$ .