SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alberto Caron*
acaron@turing.ac.uk
Xitong Liang*
Department of Statistical Sciences
xitong.liang.18@ucl.ac.uk
Samuel Livingstone
samuel.livingstone@ucl.ac.uk
Jim Griffin
j.griffin@ucl.ac.uk
In this paper, we introduce a novel MCMC sampler, PARNI-DAG, for a fully-Bayesian approach to the problem of structure learning under observational data. Under the assumption of causal sufficiency, the algorithm allows for approximate sampling directly from the posterior distribution on Directed Acyclic Graphs (DAGs). PARNI-DAG performs efficient sampling of DAGs via locally informed, adaptive random neighborhood proposal that results in better mixing properties. In addition, to ensure better scalability with the number of nodes, we couple PARNI-DAG with a pre-tuning procedure of the sampler's parameters that exploits a skeleton graph derived through some constraint-based or scoring-based algorithms. Thanks to these novel features, PARNI-DAG quickly converges to high-probability regions and is less likely to get stuck in local modes in the presence of high correlation between nodes in high-dimensional settings. After introducing the technical novelties in PARNI-DAG, we empirically demonstrate its mixing efficiency and accuracy in learning DAG structures on a variety of experiments.
Structure Learning and Causal Discovery are concerned with reconstructing a graphical model underlying a set of random variables from data, in the form of a Directed Acyclic Graph (DAG), provided that causal identifiability assumptions hold [Pearl, 2009, Drton and Maathuis, 2017, Glymour et al., 2019]. Causal relations can typically be inferred if randomized experiments are carried out [Pearl, 2009]. However, in many relevant applied fields (e.g. biology, genomics, ecology, etc.), only observational data are accessible, as performing interventions is usually costly or simply unfeasible. Structure learning from observational data is a challenging problem, with the space of possible DAGs growing super-exponentially in the number of nodes [Robinson, 1977].
Related work. Early contributions on structure learning algorithms include constraint-based methods [Spirtes et al., 2000, Pearl, 2009], such as PC and FCI, that utilize conditional independence
$^{1}$ Code to implement the PARNI-DAG proposal and replicate the experimental sections is available at https://github.com/XitongLiang/The-PARNI-scheme/tree/main/Structure-Learning.
testing to output a Markov Equivalence Class (MEC). Another stream of literature focuses on score-based methods [Geiger and Heckerman, 1994], such as Greedy Equivalent Search (GES) [Chickering, 1996, 2002], which define and consequently maximize a score function associated with different MECs. Alternatively, functional causal models, such as LiNGAM [Shimizu et al., 2006] and Additive Noise Models [Hoyer et al., 2008b], directly output DAGs. Recent advancements falling in either of the classes above have considered methods relying on deep learning techniques [Zheng et al., 2018, Yu et al., 2019], but are specifically suited for large sample regimes.
MCMC-based Bayesian approaches, that account for graph uncertainty by learning a posterior distribution over graphs, include Madigan et al. [1995] and Giudici and Castelo [2003], who introduced the MCMC Model Composition  $(\mathrm{MC}^3)$  scheme and the Add-Delete-Reverse (ADR) scheme respectively. Grzegorczyk and Husmeier [2008] improve ADR's mixing by proposing a new arc reversal move. Another stream of contributions on structure learning MCMC have considered working with score functions in smaller spaces rather than the DAG one. Friedman and Koller [2003] first recasted the problem by operating in the space of "orders" instead of DAGs, so that candidate orders, consistent with multiple DAGs, are proposed. Although this generates considerable benefits in terms of scalability, it introduces sampling bias, as some DAGs are over-represented [Ellis and Wong, 2008]. This bias is addressed in Kuipers and Moffa [2017], who proposed working in the space of "ordered partitions" instead, which introduces scalability issues, making partition MCMC applicable to graphs with a small number of nodes. Further works have focused on improving mixing efficiency and scalability of these score-based MCMC [Niinimaki et al., 2016, Viinikka and Koivisto, 2020, Kuipers et al., 2022]. Recent advances have also considered approximate Variational Inference methods [Lorch et al., 2021, Cundy et al., 2021, Geffner et al., 2022]. More closely related to our work, van den Boom et al. [2022] considered the locally-informed proposal on non-decomposable Gaussian Graphical Models (GGM), but with neighborhoods defined by a subset of the space around the current state of the Markov chain, i.e. neighborhoods of the  $\mathrm{MC}^3$  type, with size growing quadratically in the number of nodes. However, their WWA algorithm does not target the posterior on DAGs directly, unlike PARNI-DAG.
Tangential to this work is also the rich literature on discrete spaces MCMC samplers for Bayesian variable selection. Noteworthy contributions include Zanella [2020], that up-weights a random-walk kernel via local neighborhood information and constructs the locally-informed proposal, and Griffin et al. [2021], that develops an adaptive MCMC algorithm addressing high-dimensional sparse settings. The locally-informed proposal can be viewed as a discrete analog to the gradient MCMC proposals (e.g. the Metropolis-adjusted Langevin algorithm [Roberts and Rosenthal, 1998]), where the gradient of the target distribution does not exist. The adaptive MCMC instead is capable of learning from the shape of the target distribution and exploiting the most important variables. Liang et al. [2022] generalise the locally-informed proposal and the adaptive MCMC to be used within random neighborhood schemes, and introduced the PARNI proposal for Bayesian variable selection.
Contribution. This work presents a novel MCMC sampler for structure learning, PARNI-DAG, that builds on top of PARNI [Liang et al., 2022], but is equipped with new features for efficient sampling in the space of DAGs. PARNI-DAG constructs a random neighborhood of possible DAGs via probabilities proportional to a function of the Posterior Edge Probabilities (PEPs), that is, the probabilities of including a direct edge between two nodes. Then, since full enumeration of all the possible DAGs within the neighborhood is virtually impossible in high-dimensions, it proposes a new candidate DAG via a "point-wise update", i.e. by considering a sequence of intermediate proposed DAGs belonging to a smaller subset of the neighborhood. Finally, to guarantee better scalability with the number of nodes, we propose a procedure to pre-tune PARNI-DAG's parameters and warm-start the chain by utilising a skeleton graph derived through any constraint or score based algorithm (e.g. PC, GES, ...), or the iterative search space expansion proposed in Kuipers et al. [2022].
Motivations. As we will discuss in later sections, PARNI-DAG generates clear advantages over structure MCMC methods such as  $\mathrm{MC}^3$  and ADR in terms of speed of convergence and mixing, thanks to its new improved proposal, while targeting the same posterior distribution over DAGs. As for the comparison to score-based MCMC methods, it is known that order MCMC [Friedman and Koller, 2003] has better scalability than standard structure MCMC methods, but, operating in the smaller space of order, it introduces sampling bias via a non-uniform prior being assigned to the different DAGs compatible with a single order [Ellis and Wong, 2008]. Compared to order MCMC, PARNI-DAG does not incur in any sampling bias as it targets the posterior on DAGs directly, while
at the same time converging faster than ADR. Kuipers and Moffa [2017] propose a variant called partition MCMC, which operates in the space of ordered partitions instead of orders. Partition MCMC is unbiased in terms of sampling, but it is extremely slow and has high computational complexity (can only be used on very few nodes) [Kuipers et al., 2022]. PARNI-DAG, on the other hand, specifically addresses high-dimensional settings with many nodes.
As a solution to scale order and partition MCMC up to high-dimensional scenarios, the hybrid MCMC scheme of Kuipers et al. [2022] proposes to restrict the initial search space by pruning it via a skeleton graph (e.g. PC algorithm derived skeleton)  $\mathcal{H}$ , whose max parent set size is  $m$  per node, so that complexity reduces to  $O(n2^{m})$ . Although this increases MCMC speed dramatically in large graphs, it is likely to introduce bias as some true edges might be deleted in  $\mathcal{H}$ . To tackle this issue they also propose an iterative procedure (Iterative MCMC) to re-populate  $\mathcal{H}$  with additional potential parent nodes, at the expenses of an increased computational time. We couple PARNI-DAG with a similar procedure, but we use a previously derived skeleton  $\mathcal{H}$  (e.g. the Iterative MCMC one) not to restrict the space, and possibly cancel out relevant edges, but to warm-start the chain by pre-tuning some of the MCMC, as we will explain in details in later sections.
We show in the experiments section how PARNI-DAG brings about improvements over the classical structure MCMC ADR [Giudici and Castelo, 2003, Grzegorczyk and Husmeier, 2008] and score-based MCMC methods [Friedman and Koller, 2003, Kuipers and Moffa, 2017, Kuipers et al., 2022] in terms of DAG learning accuracy and MCMC mixing, particularly in settings with a high number of nodes, as it is able to reach high probability regions in very few steps, while addressing the presence of highly correlated neighborhoods of edges where classical samplers might get trapped. Code to implement PARNI-DAG and fully reproduce the experiments is provided<sup>2</sup>.
Consider a graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , made of nodes  $\mathcal{V}$  and edges  $\mathcal{E}$ , suitable to represent probabilistic dependencies (edges) between random variables (nodes) through a probabilistic graphical model [Koller and Friedman, 2009]. Given a collection of  $n = |\mathcal{V}|$  continuous random variables  $(X_{1},\dots,X_{n})$ , a Bayesian Network  $\mathcal{B} = (\mathcal{G},\Phi)$  associated with a DAG  $\mathcal{G}$ , is a probabilistic graphical model utilized to represent a factorization of the joint probability  $p(X_1,\ldots ,X_n)$  into the conditional distributions  $p(X_{1},\ldots ,X_{n}) = \prod_{i = 1}^{n}p_{\phi}\bigl (X_{j}\mid \mathrm{Pa}(X_{j})\bigr)$ , where  $\phi \in \Phi$  are the parameters and  $\mathrm{Pa}(X_j)$  all the parents of node  $X_{j}$ . Suppose that all nodes  $(X_{1},\dots,X_{n})$  are re-scaled to have zero mean and unit variance. The goal in structure learning is to learn the BN factorization of conditional probabilities, given a sample of  $N$  observations on  $\mathcal{D} = \{X_{1,i},\dots,X_{n,i}\}_{i = 1}^{N}$ . In Bayesian structure learning, we specifically want to learn a posterior distribution on possible DAGs, i.e.  $p(\mathcal{G}|\mathcal{D})\propto p(\mathcal{D}|\mathcal{G})p(\mathcal{G})$ . In this work, we define a DAG indicator variable  $\gamma \in \Gamma = \{0,1\}^{n\times n}$ , where each  $\gamma$  implies a unique graph  $\mathcal{G}_{\gamma}$ ;  $\gamma_{ij} = 1$  indicates the presence of an edge from  $X_{i}$  to  $X_{j}$ , while  $\gamma_{ij} = 0$  indicates absence of it. The linear functional model associated with DAG  $\mathcal{G}_{\gamma}$  can be specified as
$$
X = W _ {\gamma} ^ {\top} X + \varepsilon , \quad \text {w h e r e} \mathbb {E} (\varepsilon) = 0, \operatorname {V a r} (\varepsilon) = \operatorname {d i a g} \left(\sigma_ {1} ^ {2}, \dots , \sigma_ {n} ^ {2}\right), \tag {1}
$$
and  $W_{\gamma}$  is a matrix of weights. Within a fully Bayesian approach to the inference problem presented in (1), we consider the following prior specification
$$
\left(W _ {\gamma}\right) _ {i j} \mid \sigma_ {j} ^ {2}, \gamma_ {i j} = 1 \sim \operatorname {N o r m a l} \left(0, g \sigma_ {j} ^ {2}\right),
$$
$$
\left(W _ {\gamma}\right) _ {i j} \mid \sigma_ {j} ^ {2}, \gamma_ {i j} = 0 \sim \delta_ {0}, p (\sigma_ {j}) \propto \sigma_ {j} ^ {- 2}
$$
where  $\delta_0$  is the Dirac measure at 0. Thanks to their conjugate form, the coefficients  $\{W_{ij}\}_{i,j}$  and the heteroskedastic error variances  $\{\sigma_j\}_j$  can be integrated out analytically, leading to a marginal likelihood that depends only on  $\gamma$ , denoted by  $p(\mathcal{D}|\gamma)$ . The last building block we need for posterior  $\pi (\gamma)\equiv p(\gamma |\mathcal{D})$  inference is acyclicity constraints imposed in the prior distribution for the model indicator  $\gamma$ , which is specified by
$$
p (\gamma) \propto \left(\frac {h}{1 - h}\right) ^ {d _ {\gamma}} \times \mathbb {I} \left\{\mathcal {G} _ {\gamma} \text {i s a D A G} \right\} \tag {2}
$$
where the hyperparameter  $h \in (0,1)$  represents the prior edge probability and  $d_{\gamma} = |\mathcal{E}_{\gamma}|$  is the number of edges implied by  $\gamma$ . We are interested in Bayesian inference on  $\gamma$  with posterior distribution  $\pi(\gamma) \propto p(\mathcal{D}|\gamma)p(\gamma)$ .
$^{2}$ Code is provided in the supplementary materials. Github link to be added upon acceptance.
Notice that the likelihood, and thus posterior distribution, could also be replaced by a scoring function typically used in score-based MCMC methods [Friedman and Koller, 2003, Kuipers and Moffa, 2017, Kuipers et al., 2022], such as the BGe score [Geiger and Heckerman, 2002], or the BDe score, which would make PARNI-DAG suitable also for discrete-valued nodes.
3 The novel PARNI-DAG proposal
3.1 Point-wise implementation of Adaptive Random neighborhood Informed proposal
Recently, Liang et al. [2022] introduced the Point-wise Adaptive Random Neighborhood Informed proposal (PARNI) in the context of Bayesian variable selection problems. PARNI is characterized by a Metropolis-Hastings (MH) proposal [Metropolis et al., 1953, Hastings, 1970] which couples the Adaptively scaled Individual Adaption scheme [Griffin et al., 2021] with the locally-informed proposal [Zanella, 2020]. We begin by firstly describing the generic PARNI proposal for DAG learning problems, then we will introduce the modifications needed to make it efficient and that result in the PARNI-DAG proposal.
The PARNI proposal falls into the class of random neighborhood informed proposals, which are characterized by two steps: i) random sampling of a neighborhood  $\mathcal{N}$ , and then ii) proposal of a new DAG within this neighborhood  $\mathcal{N}$  according to a informed proposal [Zanella, 2020]. The random neighborhoods are drawn based on an auxiliary neighborhood indicator variable  $k\in \mathcal{K}$  with conditional distribution  $p(k|\gamma)$ , such that the neighborhood is constructed as a function of  $\mathcal{N}(\gamma ,k)\subseteq \Gamma$ . Let  $\mathcal{K} = \Gamma = \{0,1\}^{n\times n}$ , then in PARNI each  $k\in \mathcal{K}$  value indicates whether the corresponding position of  $\gamma$  is included in the neighborhood. The conditional distribution of  $k$  takes a product form  $p_{\eta}(k|\gamma) = \prod_{i,j}p_{\eta}(k_{ij}|\gamma_{ij})$ , characterized by the set of tuning parameters  $\eta = \{\eta_{ij}\}_{i,j = 1}^n$ , where  $\eta_{ij}\in (\epsilon ,1 - \epsilon)$  for a small  $\epsilon \in (0,1 / 2)$ , and each  $p_{\eta}(k_{ij}|\gamma_{ij})$  is given by
$$
p _ {\eta} (k _ {i j} = 1 | \gamma_ {i j} = 0) = \min \left\{1, \frac {1 - \eta_ {i j}}{\eta_ {i j}} \right\}, p _ {\eta} (k _ {i j} = 1 | \gamma_ {i j} = 1) = \min \left\{1, \frac {\eta_ {i j}}{1 - \eta_ {i j}} \right\}. (3)
$$
The methods for adapting  $\eta$  will be discussed in Section 3.2 specifically. The neighborhood is thus the constructed as
$$
\mathcal {N} (\gamma , k) = \left\{\gamma^ {*} \in \Gamma \mid \gamma_ {i j} ^ {*} = \gamma_ {i j} \forall (i, j) \text {s . t .} k _ {i j} = 0 \right\}. \tag {4}
$$
The neighborhood  $\mathcal{N}(\gamma, k)$  contains  $2^{d_k}$  models where  $d_k$  denotes the number of 1s in  $k$ . The full enumeration over the whole neighborhood is computationally expensive if  $d_k$  is big, and infeasible when  $d_k$  is beyond 30. Liang et al. [2022] considered a point-wise implementation of the algorithm, which dramatically reduces the computational complexity from  $\mathcal{O}(2^{d_k})$  to  $\mathcal{O}(2d_k)$ . The idea of how the point-wise implementation works is the following. Let variable  $K = \{K_r\}_{r=1}^R$  represent the collection of positions where  $k_{ij} = 1$ . Instead of working with the full neighborhood  $\mathcal{N}(\gamma, k)$ , we construct a sequence of smaller neighborhoods  $\{\mathcal{N}(\gamma(r), K_r)\}_{r=1}^R \subset \mathcal{N}(\gamma, k)$ , and a new DAG  $\gamma'$  drawn from these neighborhoods  $\{\mathcal{N}(\gamma(r), K_r)\}_{r=1}^R$  is sequentially proposed according to the sub-proposals  $q_{g, K_r}(\gamma(r-1), \cdot)$  at each time  $r$ . Given the intermediate DAGs  $\gamma = \gamma(0) \to \gamma(1) \to \dots \to \gamma(R) = \gamma'$ , each of their corresponding sub-proposal is defined by
$$
q _ {g, K _ {r}} (\gamma (r - 1), \gamma (r)) = \frac {g \left(\frac {\pi (\gamma (r)) p \left(K _ {r} \mid \gamma (r)\right)}{\pi (\gamma (r - 1) p \left(K _ {r} \mid \gamma (r - 1)\right))}\right) \mathbb {I} \left\{\gamma (r) \in \mathcal {N} (\gamma (r - 1) , K _ {r}) \right\}}{Z _ {r}}. \tag {5}
$$
where  $g:[0,\infty)\to [0,\infty)$  is a continuous function and  $Z_{r}$  is the normalising constant. The construction of  $K$  and of the sub-neighborhoods  $\{\mathcal{N}(\gamma (r),K_r)\}_{r = 1}^R$  will be discussed later in Section 3.3.
The choice of function  $g(\cdot)$  is crucial to the performance of the PARNI proposal. If  $g(x) = 1$ , the sub-proposal  $q_{g,K_r}$  in (5) reverts back to a random walk proposal which proposes a new DAG from  $N(\gamma(r-1),K_r)$  with uniform probability. It would then be hard for it to explore important neighborhoods of edges in high-dimensions, as it is likely to get stuck in neighborhoods with highly correlated nodes. In the locally-informed proposal of Zanella [2020], function  $g(\cdot)$  is chosen to be a non-decreasing and continuous balancing function  $g:[0,\infty)\to [0,\infty)$ , which satisfies  $g(x) = xg(1/x)$ . Zanella [2020] shows that the locally-informed proposal with balancing function  $g(\cdot)$  is asymptotically optimal compared to other choices of  $g(\cdot)$  in terms of Peskun ordering. In this
work, we consider the Hastings' choice (i.e.  $g(x) = \min \{1, x\}$ , satisfying  $g(x) = xg(1/x)$ ) as it has been shown to be the most stable choice for most problems (see the discussion in Supplement B.1.3 of Zanella [2020]).
The informed proposal in this context not only helps speed up convergence, but it also avoids proposing invalid DAGs. This is because the acyclicity checks for candidate models  $\gamma' \in \mathcal{N}(\gamma(r - 1), K_r)$  can be run before computing their posterior model probabilities, and thus one can assign zero mass to their prior (resulting in zero mass posterior) if  $\gamma'$  is not a valid DAG. The resulting proposal kernel is then given by
$$
q _ {g, K} \left(\gamma , \gamma^ {\prime}\right) = \prod_ {r = 1} ^ {R} q _ {g, K _ {r}} \left(\gamma (r - 1), \gamma (r)\right). \tag {6}
$$
In order to construct a  $\pi$ -reversible chain and calculate the MH acceptance probability, we define the new collection of variables  $K' = \{K_r'\}_{r=1}^R$  for the reverse moves, where  $K'$  contains the same element as  $K$  but with reverse order (i.e.  $K_r' = K_{R-r+1}$ ). The MH acceptance probability is then defined as
$$
\alpha \left(\gamma , \gamma^ {\prime}\right) = \min  \left\{1, \frac {\pi \left(\gamma^ {\prime}\right) q _ {g , K ^ {\prime}} \left(\gamma^ {\prime} , \gamma\right)}{\pi (\gamma) q _ {g , K} \left(\gamma , \gamma^ {\prime}\right)} \right\}. \tag {7}
$$
Proposition 3 of Liang et al. [2022] shows that using the above  $K^{\prime}$  can simplify the calculation of the MH acceptance probability in (7).
In the following sections, we will introduce the necessary modifications and developments to efficiently sample DAGs, the combination of which defines the novel PARNI-DAG proposal. The full details and algorithmic pseudo code of PARNI-DAG are also provided in the supplementary material (Appendix D).
3.2 Warm-start of  $\eta$  combined with hybrid scheme
Griffin et al. [2021] and Liang et al. [2022] both suggested using the Posterior Inclusion Probabilities in a Bayesian variable selection context to define the  $\eta_{ij}$  hyperparameters described above. Perfectly analogous, we will make use of the Posterior Edge Probabilities (PEPs) instead, i.e.  $\pi (\gamma_{ij} = 1)$ . The distribution in (3) with the choice of parameters described, encourages to 'flip' edge values based on variables' marginal importance. For example, suppose the posterior for  $\gamma_{ij}$  is  $\pi (\gamma_{ij} = 1) = 0.9$  and the chain is currently at  $\gamma_{ij} = 0$ . The edge should be then assigned higher probability to be included in the DAG (i.e.  $\gamma_{ij}$  should be switched from 0 to 1). Thus, we propose  $k_{ij} = 1$  with probability  $\min \{1,0.9 / 0.1\} = 1$ , so that the new neighborhood features this possible move and  $\gamma_{ij}$  can be effectively switched from 0 to 1 in the next moves. Another advantage of using the conditional distribution featuring the PEPs in (3) is that when all variables are independent this proposal will have acceptance probability 1 and achieve optimal asymptotic variance as showed in Griffin et al. [2021].
The PEPs,  $\pi (\gamma_{ij} = 1)$ , however cannot be directly calculated for most problems. Griffin et al. [2021] considered the Rao-Blackwellised estimates  $\hat{\pi} (\gamma_{ij} = 1|\gamma_{-ij})$ , where  $\gamma_{-ij}$  denote  $\gamma$  excluding the edge indicator  $\gamma_{ij}$ , and  $\eta$  is updated on the fly given the current output of the chain. The Rao-Blackwellisation is however infeasible in the context of DAG sampling because it entails computing  $n^2$  model probabilities at each iteration of the MCMC. This would result in an impractical computational scheme when working with many nodes. Alternatively, one could take the ergodic sum over the output  $\{\gamma^{(l)}\}_{l = 1}^{t}$  as
$$
\hat {\pi} _ {i j} ^ {(t)} = \frac {1}{t} \sum_ {l = 1} ^ {t} \mathbb {I} \left\{\gamma_ {i j} ^ {(l)} = 1 \right\}, \tag {8}
$$
but the ergodic sum would be too slow to converge. This updating scheme would also lead to other issues such as poor exploration and mis-specification of the important edges. It is worth mentioning that the  $\eta$ 's do not have to be the exact PEPs, as we can still draw valid samples from the target posterior distribution  $\pi$  as long as the MH acceptance probability preserves  $\pi$ -reversibility. However, choosing  $\eta$ 's that are close to the true PEPs will cause each component to change and to target the right proportion of their marginal probabilities (higher probability of including important edges).
Our solution then is to approximate the PEPs  $\tilde{\pi}_{ij}$  before actually running the PARNI-DAG's MCMC, and use this approximation to warm-start the chain. While running the algorithm at time  $t$ , we update
the tuning parameters  $\hat{\eta}_{ij}^{(t)}$  adaptively based on
$$
\hat {\eta} _ {i j} ^ {(t)} = \phi_ {t} \tilde {\pi} _ {i j} + (1 - \phi_ {t}) \hat {\pi} _ {i j} ^ {(t)}, \tag {9}
$$
where  $\{\phi_l\}_{l = 1}^t$  is a decreasing sequence of weights which control the trade-off between the provided warm-start PEPs  $\tilde{\pi}_{ij}$  and the ergodic sum of the output  $\hat{\pi}_{ij}^{t}$ . The choice of  $\{\phi_l\}_{l = 1}^t$  is not unique and we give a full specification of our choice in the supplementary material (Appendix B). In what follows, we will briefly describe the methods used to calculate the warm-start approximation  $\tilde{\pi}_{ij}$ .