Reinforcement Learning for Causal Discovery without Acyclicity Constraints
Authors: Bao Duong
Hung Le
Biwei Huang
Thin Nguyen
Venue: Accepted by TMLR ()

Abstract: Recently, reinforcement learning (RL) has proved a promising alternative for conventional local heuristics in score-based approaches to learning directed acyclic causal graphs (DAGs) from observational data. However, the intricate acyclicity constraint still challenges the efficient exploration of the vast space of DAGs in existing methods. In this study, we introduce ALIAS  (reinforced dAg Learning wIthout Acyclicity conStraints), a novel approach to causal discovery powered by the RL machinery. Our method features an efficient policy for generating DAGs in just a single step with an optimal quadratic complexity, fueled by a novel parametrization of DAGs that directly translates a continuous space to the space of all DAGs, bypassing the need for explicitly enforcing acyclicity constraints. This approach enables us to navigate the search space more effectively by utilizing policy gradient methods and established scoring functions. In addition, we provide compelling empirical evidence for the strong performance of ALIAS  in comparison with state-of-the-arts in causal discovery over increasingly difficult experiment conditions on both synthetic and real datasets. Our implementation is provided at https://github.com/baosws/ALIAS.

Reinforcement Learning for Causal Discovery without Acyclicity Constraints
Bao Duong
d.duong edu.
Applied Artificial Intelligence Institute (A²P), Deakin University
Hung Le
thai. we..ke.ddeknnn.ccm www.eaaakkieeee...o......
Applied Artificial Intelligence Institute (A² Deakin University
Biwei Huang
bih007 edu
University of California, San Diego
Thin Nguyen
thin, @deakin edu.
Applied Artificial Intelligence Institute (A²F), Deakin University
Abstract
Recently, reinforcement learning (RL) has proved a promising alternative for conventional local heuristics score-based approaches to learning directed acyclic causal graphs (DAGs) from obsorvational data. However, the intricate acyclicity constraint still challenges the efficient exploration of the vast space of DAGs in existing methods. In this study, we introduce ALIAS (reinforced dAg Learning without Acyclicity conStraints), novel approach to causal discovery powered by the RL machinery. Our method features efficient policy for generating DAGs in just single step with an optimal quadratic complexity, fueled by novel parametrization of DAGs that directly translates a continuous space to the space of all DAGs, bypassing the need for explicitly enforcing acyclicity constraints. This approach cnables us to navigate the scarch space more effectively by utilizing policy gradient methods and established scoring functions. In addition, we provide compelling empirical evidence for the strong performance of ALIAS in comparison with state-of-the-arts in causal discovery over increasingly difficult experiment conditions both synthetic and real datasets. Our implementation is provided at https://githuu: com/ htt//bassss/All ALIAS
1 Introduction
The knowledge of causal relationships is crucial to understanding the nature in many scientific sectors (Sachs et al.] 2005 Hünermund & Bareinboim, 2023 Cao al., 2019). This is especially relevant in intricate situations where randomized experiments impractical, and therefore, over the last decades, it has motivated the development of causal discovery methods that aim to infer causc-offect relationships from purely passive data. Causal discovery is typically formulated as finding the directed acyelic graph (DAG) representing the causal model that most likely generated the observed data. Among the broad literature, score-based methods are one the most well-recognized approaches, which assigns each possible DAG & a "score" S (D,9) quantifying how much it can explain the obscrved data D, and then optimize the score over the space of DAGs:
G arg max S (D,9). GEDAGs
(1)
Solving this oplimization problem is generally NP-hard Chickering 1996), due to the huge combinatorial search space that grows exper-exponentially with the number variables (Robinson) 1977) and the intricate acyclicity constraint that is difficult characterize and maintain efficiently bocause of combinatorial nature. Most methods therefore resort local heuristics, such as GES (Chickering) (202)) which gradually adds edges into graph one-by-one while laboriously maintaining acyclicity. With the introduction of soft DAG

---

Table 1: Positioning ALIAS among the score-based causal discovery literature.
Search
Method (ysar)
Search
Goneration
Generation complexity
Constraint!
Acyelisity
Nonlinear
Differentsble score
type!
space
steps
nosurnnee
data
required
GES (2002)
Chicketiee
2.02
NOTEARS (2020)
X20
Graphs
Soft
Lood
NOCURL (244))
DAGs
None
DAGMA (2022)
Graphs
ScB:
K
BaDAG (2022)
None
COSMO (2X24)
Graphs
None
x
RL BIC (2020)
Zhn ml. 20201
Graphs
Siugle
Quadratic
Sub
x
DCD Neta (2121)
DAGs
Single
Cubie
None
Global
CORL
Wane
Orderings
Multiple (Antongressive)
Cubte
DAG GPN rax22
(ASeon ds di 402
DAGs
Multippe (Attoegressive)
Cuble
Hard
GARL (2025)
Yang 0 ml. A0150
Ordetings
Multipie (Attoegresddee)
Cubic
Hard
RCL-OG (2005)
Yong
Ordetings
Multiple (Autongressive)
Cubic
Hard
ALIAS (Oums)
DAGs
Single
Quadratic
None
"Local metbodd start with in Iottal graph and rpdate it eay frmntoon while global nethods typically ootern with DAU gromaaion paramatees.
"Methodi with Hand conttniness expliiily identity and dicand the actions that leed to eydls whihe Sut contrainns refor to the use DAG regilarierss
"Mochhrlee that seyelliiit only
lisit concitered not aryiliity.
characterizations Zheng et al. 2018 Yu et al 2019 Zhang et al. 2022 Bello et al 2022). the combinatorial optimization problem above is relaxed to a continuous optimization problem, allowing for exploring graphs more effectively, as multiple edges can be added or removed simultaneously in an update. Alternatively, interventional causal discovery methods exploit available interventional data to identify the causal graph
Hauser & Bühlmann 2012 Brouillard et al. 2020 Lippe et al., 2022). However, our focus in this study is the challenging obscrvational causal discovery sctting where no interventional data accessible.
Recently, reinforcement learning (RL) has emerged into score-based causal discovery (hhu et al. 2020: Wang et al. 2021 Yang et al. 2023a b) as the improved search strategy, thanks to its exploration and exploitation abilities. However, existing RL-based methods handle acyclicity either by fusing the soft DAG regularization from Zheng et al.
(2018) into the reward (Zhu et al., 2020), which wastes time for exploring non-DAGs
but still does not prohibit all cycles (Wang et al. 2021 ; or designing autoregressive policies (Wang et al., 2021 Deleu et al.
2022 Yang et al.) 2023a b) Deleu et al. 2024) that hinder parallel DAG generation and necessitates learning the transition policies over a multitude of discrete state-action combinations.
In this study, we address the aforementioned limitations of score-based causal discovery methods with novel RL approach, named ALIAS (reinforced dAg Learning wIthout Acyclicity conStraints). Our approach employs a generative policy that is capable of generating DAGs in single-step fashion without any acyelicity regularization or explicit acyclicity maintenance. This enables us to effectively explore and exploit the full DAG space with arbitrary score functions, rather than the restricted ordering space. Specifically, we make the following contributions in this study:
At the core of ALIAS, taking inspirations from NoCurl Yuu et al. 2021 and subsequent works (Massidda et al., 2024]] Annadaní ct al., 2023), we design Vec2DAG, surjective map from a continuous domain into the space of all DAGs. We prove that given fixed number of nodes, this function can translate unconstrained real-valued vector into a binary matrix that represents valid DAG, and vice versa-there always exists a vector mapped to every possible DAGs.
2. Thanks to Vec2DAG, we are able to deviso policy outputting actions in the continuous domain that are directly associated with high-reward DAGs. The policy is one-step, unconstrained, and costs only a quadratic number of parallel operations w.r.t. the number of nodes, allowing our agent to explore the DAG space very effectively with arbitrary RL method and scoring function. To our knowledge, ALIAS the first scorc-based causal discovery method based on RL that can explore the exact space of DAGs with an efficient one-step generation, rendering it an efficient realivation of Eq. (I).
3. We demonstrate the effectiveness of the proposed ALLAS method in comparison with various state- of-the-arts on systematic set of numerical evaluations on both synthetic and real-world datascts. Empirical evidence shows that our method consistently surpasses all stale-of-the-art baselines under multiple evaluation metrics on varying degrees of nonlinearity, dimensionality, graph density, and model misspecification. For example, our mothod can achieve an SHD 0.2 + 0.2 on very dense graphs with 30 nodes and 8 parents per node on average, and on large graphs with 200 nodes and 400 edges average, ALIAS can still obtain a very low SHD) of 2.0 + 0.9.
2

---

In-degree
15
DAGMA
ALJAS BSC PVO (Chrs)
ALJAS 15 PRO (Our)
CORL
12
SHID (lower better)
10
7
100
5
-0-
2
5000
10000
15000
20000
0
Step
(a)
(b)
ALIAS + BIC PPO
Thue DAG
ALIAS L.S
True DAG
ALIAS BIC A2C
# DAGs explored (x (x10000)
ALIAS LS A2C
# DAGs explored (x1000)
0.0075
Negative Reward
1000
Negative Reward
1000
0.0050
500
0.0025
0.0000
5000
10000
15000
20000
0
5000
10000
150000
20000
Step
Step
(c)
(d)
Figure 1: Using merely observational data, the proposed ALIAS method correctly identifies all edges of (a) very complex causal dataset with extremely dense connections (linear-Gaussian data with Madös-Rényl graph of 30 nodes and expected in-degree of 8). (b) We evaluate DAG learning performance in Structural Hamming Distance (SHD, lower better) on of such datasets with respect to the first 20 000 training steps of four ALIAS variants, by combining scoring functions Bayesian Information Criterion (BIC) Least Squares (T.S) with RL methods PPO (Schulman et al., 2017) & A2C (Mnih et al.] 2016), comparison with the best baselines this setting, namely CORL (Wang et al., 2021) and DAGMA (Bello et al. 2022) (as evaluated in Scction 6.2 The best method in this scenario is ALIAS BIC + PPO variant with zero SHD at the end of the learning process. (c) & (d) For both scores, our method's rewards al always approach those of the ground truth DAGs very sharply, which is made possible largely thanks to our efficient DAG parameterization, as well as the continuous exploitation and exploration of the RL algorithms, especially PPO.
We summarize the advantages of ALIAS compared with the stato-of-tho-arts in causal discovery in Table 1 In addition, in which our method can achieve absolute accuracy, while the best baselines this setting still struggle.
a snapshot of ALIAS's strong performance in a case of highly complex structures,
2 Related Work
Constraiint-bssdd methods like PC, FCT (Spitess & Glymour 1991 Spirtes et al. 2000) and RFCT (Colombo al.) 2012) form prominent class causal discovery approaches. They first exploit conditional independence ionships statistically exhibited in data via a series of hypothesis tests recover the skeleton, which is the undirected version of the DAG, and then orient the remaining edges using probabilistic graphical rules. However, their performance heavily relics on the quality of the conditional independence tests, which can deteriorate rapidly with the number of conditioning variables (Ramdas et al. 2015) rendering them unsuitable for large or dense graphs.
Note that the "constraint" here largely refers statistical constraints, such as conditional independence, while "constraint" in our method refers the acyclicity enforcement.
3

---

Score-based methods is another major class of DAG learners, where each DAG is assigned properly defined score based on its compatibility with observed data, then the DAG learning problem becomes the optimization problem for the DAG yielding the best score. Score-based methods can be further categorized based on the search approach as follows.
Combinatorial greedy search methods such as GES CChikkerigg 2002) and FGES (Ramsey et al, 2017] resort to greedy heuristics to reduce the search space and enforce acyclicity by adding one edge at time after explicitly checking that it would not introduce any cycle, yet. this comes at the cost of the sub-optimality of the result.
Continuous optimization methods improve upon combinatorial optimization methods in scalability by the ingenious smooth acyclicity constraint, introduced and made popular with NOTEARS (Zheng et al. 2018), which turns said combinatorial optimization into a continuous optimization problem. This enables bypassing the adversary between combinatorial enumeration and scalability
allow for exploring the DAG space much more effectively, where multiple edges can be added or removed in an update. Following developments, e.g., Yu et al. (2019): Lee et al. (2020): Zheng et al. (2020): Ng et al.] (2020); Yu ct al. (2021); Zhang et al. (2022) Wei et al. (2020) contribule to extending and improving the soft DAG characterization in scalability and convergence. Notably, unconstrained DAG parameterizations are also proposed by Yu et al. (2021) and Massidda et al. (2024), which simplify the optimization problem from a constrained to an unconstrained problem. However, contimous optimization methods restrict the choices of the score to be differentiable functions, which exclude many well-studied scores such as BIC, BDe, MDL, or independence- scores [Bhhlmann et al. 2014)..
Reinforcement learning methods have emerged in recent years as the promising replacement for the greedy search heuristics discussed so far, thanks to its search ability via exploration and exploitation. As the pioneer in this line of work, Zhu et al. (2020) introduced the first RL agent that is trained to generate high-reward graphs. To handle acyclicity, they incorporate the soft DAG constraint from Zheng et al. (2018) into the reward function to penalize cyclic graphs. Unfortunately, this may not discard all cycles in the solution, but also increase computational cost drastically due the unnocessary reward calculations for non-DAGs. To mitigate this issue, subsequent studies (Wang el. al. 2021 Yang al.) 2023b a turn to finding
the best-scoring causal ordering instead and subsequently apply variable selection onto the result to obtain DAG, which naturally relieves our concerns with cycles. More particularly, CORL (Wang et al. 2021 is the first RL method operating on the ordering space, which defines states as incomplete permutations and actions as the element to be added next. GARL. (Yang et al.) 2023b) is proposed to enhance ordering generation by exploiting prior structural knowledge with the help of graph attention networks. Meanwhile, RCL-OG (Yang et al. 2023a) introduces a notion of order graph that drastically reduces the state space size from (d!) only O (2*). ). It is also worth noting that the emerging generative flow networks (GFlowNett, Bengio et al. 2021 in which the generation of DAGs is also viewed
2023 offer another technique for learning (distributions of) DAGs (Deleu et al., 2022 2024
sequential generation problem, where edges are added one-by-one with explicit exclusions of edges introducing cycles, and the transition probabilities are learned via flow matching objectives. However, the generation of these orderings and DAGs are usually formulated as a Markov decision process, in which elements are itcratively added to the structure in a multiple-step fashion, which prevents efficient concurrent DAG generations and requires learning the transition functions, which is computationally involved given the multitude of discrete state-action combinations.
3 Background
3.1 Functional Causal Model
Let X = (X (kk)
Xa) be the d-dimensional random (column) vector representing the variables of interest,
E dd denotes the k-th observation of X, and D (x)1 fk=1 indicates the observa- tional dataset containing n i.i.d. samples of X. Assuming causal sufficiency, that is, there are no unobserved
(k)
1
4

---

endogenous variables, the causal structure among said variables can be described by DAG G (V,,) where each vertex i E V = -1......0 d) corresponds to random variable X;, and each edge (j i) E CVV. V implies that X, is direct cause of X₁ We also denote the set of all direct causes of a variable as its parents, i.c., pa, = E (j- -
E). The DAG g is also represented algebraically with a binary adjacency matrir A EET 10,1)axe dxd where the (i, entry is iff (i- j) E E. Then, the space of all (adjacency matrices of) DAGs of d nodes is denoted by D2 {0, 13 We follow the Functional Causal Model framework (FCM, Pearl 2009 to assume the data generation process X; == Sa pa,; F.). Vi = lM=11....,4 where the noises Ey are mutually independent. In addition, we also consider causal minimality (Peters et al. 2014 to ensure each function fi is non-constant to any of its arguments. For any joint distribution over E = (EI....,Ed))
the functions {ft]; induce joint distribution over X. The goal of causal discovery is then to recover the acyclic graph G from empirical samples of P (XX).
3.2
DAG Scoring
Among multiple DAG scoring functions well-developed in the literature (Schwarz
1978
Heckerman et al.,
1995: Rissanen 1978). here we focus on the popular Bayesian Information Criterion (BIC) (Schwarz) 1978).
which adopted in many works (Chickering, 2002 Zhu al., 2020 Wang al. 2021
Yang al.) 2023a] for
its flexibility, computational straightforwarddnss, and consisteney.
More particularly, BIC is parametric score that assumess model family for the causal model parameters, which comes with a set of parameters & containing the paramcters of the causal mechanisms and noise distribution. This score is used to approximate the likelihood of data given the model after marginalizing out the model parameters using Laplace's approximation (Schwarz 1978):
e.g.,
(D = In / v. (v db
SBIC
(2)
2
Given a DAG G, the BIC score is defined generally as follows:
ShI (D,9) = 21n (D (. - -1g|Inn.
(3)
where & the maximum-likclihod estimator of p (D v/ 9), |g] is the number of edges in 9, and n the number of samples in D.
The BIC is consistent in the sense that if a causal model is identifiable, asymptotically, the true DAG has the highest score among all other DAGs (Haughton 1988). Mcanwhile, for limited samples, it prevents overfitting by penalizing edges that do not improve the log- likelihood significantly. More formally:
Lemma 1. Let G* be the ground truth DAG of identifiable SCM satisfying causal minimality (Peters et al. 2014) (i.e., there are redundant edges) inducing the dataset DD and let be the sample size of D. Then, in the limit of large SaIC (D,g*) > SBIC () for any G + g*.
The proof can be found in Appendix B.1
As an example, for additive noise models (ANM) x;
:= fa (Xpa,) + E., Vi = 1,..., d with Gaussian noises
E, N (0; the BIC-based score can be specified as
SSR;
SEIC-NV (D,9) = -n In
+ Inn),
where SSR, = Σm= (2( 2 2 is the sum of squared residuals after regressing X; on its parents in G, and we adopt the convention that |g] is the number of edges in G. Additionally assuming equal noise variances gives us with
SBICEE (D.9) = lnddIn Σd SSR,
+ +1 Inn)
nd
5

---

The derivations of BIC scores presented in Appendix A A simpler yet widely adopted alternative is the least squares (T.S) (Zheng et al.
2018 Lachapelle et al. 2020 Yu et al. 2021 Bello et al. 2022 Massidda et al.
2024). With an additional lo regularization, we define the LS score as Sts (D,9) - 2i- SSR; + Ao [9]), where Ào 0 is aarper-parametrr for penalizing dense graphs. In our empirical studies, following common
practices (Zhu et al. 2020: Wang et al. 2021 Yang et al.) 2023b linear regression is used for linear data
and Gaussian process regression is adopted for nonlinear data. That being said, any valid regression technique can be seamlessly integrated into our method.
3.3 DAG Representations
Typically, to search over the space of DAGs, modern causal discovery methods either optimize over directed
graphs with differentiable DAG regularizers (Zheng et al. 2018 Yu et al. 2019 Lee et al. 2020 Zheng et al.
2020 Ng et al.) 2020 Wei et al. 2020 Zhu et al. 2020 Zhang et al. 2022 ,, or search over causal orderings
and then apply variable selection to suppress redundant edges (Cundy et al. 2021 Charpentier et al. 2022
Chen et al. 2019 Wang et al.] 2021 Rolland et. al., 2022 Sanchez et al. 2023 Yang et al. 2023b The
former approach does not guarantee the acyclicity of the returned graph with absolute certainty, while the latter approach faces challenges in efficiently generating permulations. For instance, in CuCCdd et al. (2021) the permutation matrix representing the causal ordering is parametrized by the Sinkhorn operator (Sinkhorn 1964p followed by the Hungarian algorithm (Kuhn
1955
with a considerable cost of O (d9). Other examples Yang et al. 2023a) that cost at least O (d²) just to O (d)) complexity for generating DAG.
include ordering- blasing-bbased based RL methods (Wang et al. 2021
generate single ordering element, thus totaling
Our work takes inspiration from Yu et al. 2021)) where a novel unconstrained characterization of weighted adjacency matrices of DAGs is proposed. Particularly, a "node potential" vector p E IRd is introduced to model an implicit causal ordering, where i precedes j if pj > P;. Hence, the weight matrix
A = W @ ReLU (grad (p)),
(4)
where W E Rdxd and grad (p);; == Py Pt is the gradient flow operator (Lim 2020,, can be shown to , Yu el al., 2021). However, this weight mairix is only applicable for linear models, and the representation is only used as a refinement for the result returned by a constrained optimization problem. An alternative to this characterization is recently introduced by Massidda et al. (2024). where A W @ sigmoid (grad (p) /T), yet this approach only ensures acyclicity at the limit of the annealing temperature + 0+, which is usually not exactly achieved in practice. Additionally, the equivalent. DAG formulation of Annadani et al. 2023) uses the node potential to represent a smooth explicit permutation matrix: 0 (p) := lim,- Sinkhorn pp [1 [1...d// /T), again necessitating a temperature scheduler and the
correspond to a valid DAG (Theorem 2.1,
expensive Sinkhorn operator, which reportedly requires 300 iterations to converge and an O (d') complexity of the Hungarian algorithm, for generating a single DAG.
4 Vec2DAG: Unconstrained Parametrization of DAGs
4.1 The Vec2DAG Operator
Extending from the formulation in Eq. 4 we design a deterministic translation from an unconstrained continuous space to the space of general binary adjacency matrices of all DAGs, not restricted to linear models. To be more specific, in addition to the node potential vector E Rd we introduce strictly upper-triangular "edge potential" matrix E E ddxd which can be described using d-(d-1) parameters. We then combine them with p to create a unified representation vector z E Sa = [dd-(4+1)/2, which the parameter space of all d-node DAGs in our method. Furthermore, we denote by p (z) and E (z) the node and edge potential components constituting z, respectively. Specifically, pzz) represents the node potential vector formed by the first d elements of z, while E(z) is the edge potential matrix, with the elements above the main diagonal derived from the last d'(d-1) clements of (see our code in Figure 5 Then, our unconstrained DAG parametrization 2 Vec2DAG, for d nodes can be defined as follows.
Definition 1. For all d N+ and z E Sy:

---

Vec2DAGa (2) = H (z + E())) @ H (grad (z)))
(5)
if 0, 1
where H (x)
=
known as the Heaviside step function and @ is the Hadamard (element-wise)
0 otherwise.
product operator.
The intuition behind Vec2DAG is that the first term in Eq. 5) defines symmetric binary adjacency matrix, determining whether two nodes are connected. The directions of these connections are then dictated by the second term in Eq. 5), resulting in a binary matrix that represents a directed graph. Additionally, this directed graph is guaranteed to be acyclic due to the use of the gradient flow operator.
The procedure to sample a DAG then denoted as (z)) A Vec2DAGa (z), which can be implemented in few lines of code, illustrated in Figure 5 of the Appendix. The validity of our parametrization is justified by the following theorem.
Theorem 1. For all d Nt, let Vec2DAG, Sa {0, 10,1 be defined as in Eq. (5). Then, Im (Vec2DAGa) BD,, where Im (.) is the Image operator, and Da is the space of all d-node DAGs.
The proof can be found in Appendix B.2 Our formulation directly represents a DAG by real-valued vector, which is in stark contrary to existing unconstrained methods that only aim for a DAG sampler (Cundy
let al.
2021 Wang et al. 2021 Charpentier et al. 2022 Deleu et al.] 2022 Annadani et al.] 2023) Yang 2023b a). More notably, this approach requires no temperature annealing like in Massidda et al, Annadani et al. (2023). and can generate a valid DAG in a single step since sampling can be done instantly in an unconstraiied manner. In addition, this merely costs O (d²2) parallelizable operations
et al.
(2024]
compared with the (d²) cost of sequentially generating permutations using the Sinkhorn operator (Cundy 2021 Charpentier al. 2022 Annadani al.] 2023) and multiple-step RL methods (Wang al. Yang et al. 2023b a). Moreover, our generation technique is one-step, and thus does not require learning any transition function, which vastly reduces the computational burden compared with RL methods based on sequential decisions.
et al.
2021
4.2
Properties of Vec2DAG
In this section, we show that our parameterization Vec2DAG has some important additional properties that set it apart from past formulations.
Lemma 2. (Scaling and Translation Invariance). For all d E N+, let Vec2DAGa Se + 10, axd be
defined as in Eq. 55. Then, for all z Sa, a > 0, and B Sa such that (3), < 1/2min, p (z), - p -z)) and E (9)
< E (2) Vi,j, we have Vec2DAG, (z) Vec2DAG, (a (z + B)).
This insight proven Appendix B.3. Intuitively, this indicates that scaling the potential by any positive constant a results in the same DAG (Vec2DAG(z) Vec2DAG(a z)), and translating the potential by an amount 3 (which can be large, provided it does not change the ordering of p or the element-wise positivity of E) also results in the same DAG (Vec2DAG(z) Vec2DAG(z+8)). In other words, any DAG can be diversely constructed by infinitely many representations, suggesting a dense parameter space where representations of different DAGs are close to each other. This leads us to the next point, which shows an upper bound of the distance between an arbitrary representation with a representation any DAG.
Lemma 3. (Prorimity between DAGs). Let z Sg. Then, for any DAG E Da and e > 0, there exists ZA in the unit ball B (oo; ||z|| c) around such that Vec2DAG, (Z^) = A.
We provide the proof Appendix B.4 This property is not straightforward in existing constrained optimization ion approaches (Zheng et. al. 2018 Lee el al. 2020 Zheng et al.] 2020 Lachapelle el. al.) 2020 Bello et al. 2022), and suggests that the true DAG may be found closer to the initial position if we start from a smaller scale in our framework. We leverage this result in our implementation by restricting Sy to a
hypercube [- d-(d41//22 with relatively small 7 10. This has the effect of regularizing the search space
7

---

but still does not invalidate Theorem i.c., we can still reach every possible DAGs when searching in this restricted space.
5
ALIAS: Reinforced DAG Learning without Acyclicity Constraints
5.1 Motivation for Reinforcement Learning
Using the Vec2DAG representation, the acore-based causal discovery problem may seem to simplify into a maximization problem: z' arg max SBIC (D, Vec2DAG (z)), which could, in principle, be addressod to (2-11)2)
certain extents using off-the-shelf black-box optimization techniques such as Bayesian optimization, which is one of the most popular blackbox optimization methods. However, solving this optimization problem is far from straightforward due to the high dimensionality of the search space, which grows quadratically with the number of nodes (e.g., for 30 nodes, the space is 465-dimension).
Since RL is the only black-box optimization approach that has been studied in the score-based causal discovery literature (at the time writing this manuscript, to the best, of our knowledge), we align with the established line of works [Zhu et al., 2020 Wang et al., 2021 Yang et al. 2023a b) to specifically focus on leveraging RL to solve this optimization problem. The idea that an RI. agent with a stochastic policy can autonomously decide where to explore based on the uncertainty of the learned policy, which continuously updated through incoming reward signals (Zhu al. 2020). That said, we note that, our method, which only involves one-step policy gradient approach rather than a general RL
trajectories shown below, is better perceived method that requires multi-step Markov decision process (MDP).
As shown below, policy gradient provides our method with built-in exploration-exxlootttioon capabilities and linear scalability with respect to both dimensionality and sample size, making it a practical choice for this problem. In addition, our policy gradient point of view also enables the adaptability of various established methods, such as vanilla policy gradient (Sutton al.) 1999). A2C (Mnih et al.) 2016), and PPO (Schulman et al. 2017), to efficiently optimize our objective, effectively establishing clear association between proposed approach and policy gradient in RL.
5.2 Policy Gradient for DAG Search
Policy and Action. Utilizing RL, seek for a policy π that outpuls continuous action z S. (d+1//)/2 which is the parameter space of DAGs of d nodes. In this work, we consider stochastic policies for better exploration, i.e. we parametrize our policy by an isotropic Gaussian distribution with learnable means and variances: To (z) = N (z: /oe diag (aā)). Since our policy generates a DAG representation in just one step, every trajectory starts with the same initial state and terminates after only one transition, so the agent does not need to be aware of the state in our method, We note that, similar to RL-BIC (Zhu et al., 2020), the one-step nature of the environment does not preclude the application of RL in our approach. This is because one-step environment is simply special case of a MDP, which remains compatible with most RL algorithms.
Reward. The reward of an action in our method set as the graph score of the DAG induced by that action with respect to the observed dataset D (Section 3.2 and divided by n x d to maintain numerical stability without modifying the monotonicity of the score:
1
R R = x d S (D, Vec2DAG (z))
(6)
Policy Gradient Algorithm. Since our action space is continuous, we employ policy gradient methods, which are well established for handling continuous actions, rather than the value-based approach as in recent RI-based techniques (Wang et al. 2021 Yang et al.) 2023b a). The training objective is to maximize the
expected return defined as J (0) Exana R (z)..
Under identifiable causal models, causal minimality, and consistent scoring function, the optimal policy obtained by maximizing this objective will return the true DAG:
8

---

Algorithm 1 ALIAS with vanilla policy gradient for causal discovery.
Input: Dataset D {x(')) "" score function S (D,,), batch size B, and learning rate n)
Output: Estimated causal DAG S.
1: while not terminated do
2: Draw a minibatch of B actions from the policy: (2(k) To)==1 k=
3: Collect rewards {r()) == nxad S (D, Vec2DAGa (2(11)) }
Sec. 4.1
kml 4: Update policy as: 0 =- 0 + B Ek Ve (z( ,((t)
b Sec. 5.2
end while
z ~ Te, :== Vec2DAG (z).
Post-process & by pruning if needed and return.
Sec.
5.3
Lemma 4. Assuning causal identifiabillly and carsaal miniimality, that is, there is unique causol model with redundant edges that can produce the observed dotoset, and BIC score is used to define the reward R (z) as in Eq. (c). Let n be the sample size of the observed dataset DD H" E arg maxExa [R (z)], where
no (z) N (z; ieo ding (o)))- Then, as 00; g Vec2DAG (z) is the true DAG, where Tt--
The proof is presented in Appendix B.5 The differential entropy of the policy can also be added to the expected return as a regularization term to encourage exploration (Mnih et al. 2016), however we find in our experiments that the stochasticity offered by the policy suffices for exploration. That said, we also investigate the effect of entropy regularization in our empirical studies. During training, the parameter 0 is updated the direction suggested by the policy gradient algorithm. For example, using vanilla policy gradient, the gradient is given by the policy gradient theorem (Sutton et al. 1999 as: VeJ (0) Ez [V In To (z) R (z)].
Note that since our trajectories function is always equal to the immediate reward, and therefore there is no need for a critic to estimate the value function. Hence, vanilla policy gradient works well out-of-the-box for our framework, yet in practice our method can be implemented with more advanced algorithms for improved training efficiency. Our practical implementation considers the basic policy algorithm Advantage Actor-Crita (A2C, Mnilh et al. 2016) and a more advanced method Proximal Policy Optimization (PPO, (PPO, Schhlmann et al.] (017). In addition, while policy gradient only ensures local convergence under suitable conditions (Sutton et al.) 1999), our empirical evidence remarks that our method can reach the exact ground truth DAG in notably many cases.
one-step and our environment is deterministic, the state-action value
5.3 Post Processing
With limited sample sizes, due to overfitting, redundant edges may still be present in the returned DAG that achieves the highest score. One approach towards suppressing the false discovery rate is to greedily remove edges with non-substantial contributions in the score. For linear models, standard approach is to threshold the absolute values of the estimated weight matrix W at a certain level 8 (Zheng et al. 2018 Ng et al. 2020 Bello et al. 2022), i.e., removing all edges (i + j) with [Wi] < 8. For nonlinear models, the popular CAM pruming method [Bihlmann et al. 2014 can be employed for generalized additive models (GAMs), which performs a GAM regression on the parents set and exclude the parents that do not pass a predefined significance level. An altornative pruning method that does not depend on the causal model is based on conditional independence (CI), i.e., by imposing Faithfulness (Spirtes et. al.) 2000), for each j pa, in the graph found so far, we remove the edge (j- i) if X, IL Xg MpaMM)) which is direct consequence of the Faithfulness assumption and can be realized with available CT tests like KCTT (Zhang et al., 2011). In addition, for the least squares score, we can increase the regularization strength on the number of edges to encourage sparsity during the learning process. Our numerical experiments investigate the effects of all these approaches.
To summarize, Algorithm I highlights the key steps of our ALIAS method for the case with vanilla gradient policy.
9

---

SHD (L)
FDR (L)
TPR (1)
10
1.00
ER-1
NOTEARS
0.95
0.1
DAGMA
NOCURL
0.90
0.0
COSMO
1.0
RL-BIC
0.50
ER-2
50
0.8
RCL-OG
0.25
CORL
ALIAS (Ours)
0.6
5
10
15
20
255
30
5
10
1.5
20
25
30
10
15
20
25
30
Nodes
Nodes
Nodes
Figure 2: Causal Discovery Performance on Sanear-GGausiann Data. ER-1 and ER-2 denote Erdös- Rényi graph models with expected in-degrees of 1 and 2, respectively. The weight range is u ([-5, -2] 5)), which is wider than prior studies, making our setting more challenging due to higher data variance. We compare the proposed ALIAS method with NOTEARS (Zheng et al. 2018), DAGMA (Bello et al. 2022). NOCURL (Yu et al. 2021 COSMO (Massidda et al. 2024), RL-BIC (Zhu et al. 2020), CORL (Wang et al. 2021), and RCL-OG YYng et al. 2023a).. The performance metrics are Structural Hamming Distance (SHD, lower is better), False Detection Rate (FDR, lower is better), and True Positive Rate (TPR, higher is better). Shaded areas depict standard errors over independent runs.
6 Numerical Evaluations
In this section, we validate our method in the causal discovery task across a comprehensive set settings, including different nonlinearities, varying graph types, sizes and densities, varying sample sizes, as well as different degrees of model músspecification on both synthetic and real data. In addition, also analyze the computational efficiency of our method, as well as demonstrate the significance of different components of our method, especially the choice of reinforcement learning as the optimizer, in our extensive ablation studies.
6.1 Experiment Setup
We conduct extensive empirical evaluations both simulated and real datasets, where the ground truth DAGs are available, to compare the efficiency of the proposed ALIAS method with up-to-date state-of-the-arts in causal discovery, including the constrained continuous optimization approaches with soft DAG constraints NOTEARS (Zheng et al. 2018 2020) and DAGMA (Bello et al. 2022), unconstrained continuous optimization approaches NOCURL (Yu et al. 2021) and COSMO (Massidda et al., 2024), as well as three RL-based methods RI-BIC (Zhu et al. 2020). CORL (Wang al.) 2021), and RCL-OG (Yang et al. 2023a brief description of these methods along with their implementation details and hyper-parameter specifications are provided in Appendix and the evaluation metrics are described in Appendix C.2.1 For the main experiments, use the variant with BIC score and PPO algorithm for method, and examine other variants in the ablation studies. We report supplementary results, including additional ablation studies, in Appendix D
6.2 Linear Data with Gaussian and non- aon-Gaussian Noises
For a given number of nodes d, we first generate a DAG following the Erdös-Rényí graph model (Erdös & Rónyi
1960p with an expected in-degree of k E N+, denoted by ER-k. Next, edge weights are randomly sampled from the uniform distribution P (WW,, and the noises are drawn from the standard Gaussian E, ~ (0, 1). To make this sctting more challenging, we use a wider range P (W) U ([-5, U 22 compared with the common range of ([-1,, -0.5 [0.5,2]) in previous studies (Zheng et al., 2018 Zhu et al., 2020 Wang 000 observation ions for each dataset. This causal model is
et al.
2021 Bello et al. 2022 We then sample
10

---

Table 2: Cansal discovery performance on dense graphs (30-node ER-8) and high-dimensional graphs (200-nodc ER-2) with Gincar-Gaussian data. The performance metrics are Structural Hamming Distance (SHD, lower is better), False Delection Rate (FDR, lower is better), and True Positive Rate (TPR, higher is better). The numbers mean standard ervor over 5 independent runs. Bold: best performance, underline: second-best performance. RL-BIC & CORL fail to run high-dimensional tasks.
Dense graphs (30 nodes, 240 edges)
High-dimensional graphs (200 nodes, 400 edges)
Method
SHD (1)
FDR (L)
TPR (t)
SHD (1)
FDR (L)
TPR (*)
NOTEARS
Zheng ot al.
2018
141.2 11.9
0.25 0.03
0.55 0.003
53.8
0.06 + 0.01
0.93 : 0.01
DAGMA
Bello et al
2022
67.6
8.0
0.14
0.02
9.6 2.7
0.02
0.99 + 0.00
NOCURL
Yu el al.
2021
147.6 + 5.7
0.01
0.63
227.6 + 17.5
0.20 0.08
0.59 0.02
COSMO Massidda
al.
20/4
97.4 6.8
0.21
0.00
158.0 19.5
0.25
0.87 0.02
RL-BIC Zhu
2020
180.6
21.7
0.23 0.06
0.14
CORL Wang aL.
2021
82.4 22.3
0.87 0.00
RCL OGL-
Yang ct
2023a
199.7 + 7.1
0.01
0.51 0.00
1076.6
28.8
0.89 + 0.00
0.32 0.01
ALIAS (Ours)
0.2 0.2
0.00 + 0.00
1.00 0.00
2.0 0.9 0.00 0.00
1.00 0.00
ideblifiable due to the equal noise variances Peters el al. 2014 For fairness, we also apply the same pruning procedure with linear regression coefficients thresholded at 0.3 for all methods and use the equal-variance BIC (Section 3.2 for RL-BIC, CORL, RCL-OG, and ALIAS.
Small to moderate graphs. In Figure P we report the causal discovery performance for Gaussin data with small to moderate graph sizes and densities, showing that our method consistently achieves near-perfect performance in all metrics, which can be expected thanks to its ability to explore the DAG space competently. Overall, the closest method with comparable performance to our method in this case is DAGMA, followed by COSMO, which are among the most advanced continuous optimization approaches, while other methods, including RL-based ones, still struggle even in this simplest scenario. addition, the results on Scale-Frec (SF) graphs and the common weight range 21 (1-2, 0.5] U [0.5, 2]) can also be found in
Appendix D
Dense & High-dimensional graphs. We next test the proposed method's 'ss ability to adapt to highly complex scenarios, including the cases with very dense graphs (ER- 8 graphs) and larger number of nodes (200-node graphs).
200
SHD (lower is better)
Nodes
150
20
30
This is to demonstrate the advantages of our proposed method over existing approaches that typically struggle on
100
50
Method
slightly dense graphs of ER-4 at most (Zheng et al. 2018
50
DAGMA
Yu et al. 2021 Bello et al., 2022 Massidda et al. 2024) or small graphs of only tens of nodes (Zhu et al.) 2020 Yang et al. 2023a). In this case,
ALIAS (Ours)
2000
4000
use the common weight range of (1-, -0.5] U [0.5, 2]) to avoid numerical insta- bilities due to more complex graphs. Table 2 depicts that for dense graphs, ALIAS makes almost no mistake while the best baseline in this case, which is DAGMA, still has a significantly large SHD of 67.6 8.0. The performance gap is narrower in the high-dimensional setting with 200 nodes, yet our method remains the leading approach with SHD of only 2, compared with an SHD of nearly 10 for the second- best method DAGMA.
Sample size
Figure 3: Causal Discovery performance (linear-Gaussian data on ER-8 graphs) as function of sample size (100 to 5 500). Shaded areas depict standard errors over 5 independent. runs.
Effect of sample size. We further investigate the behavior of ALIAS under data scarcity and redundancy.
We again consider the difficult configuration of ER-8 graphs, and vary the sample size from very limited (100) to redundant (5 000) in Figure 3. where it is shown that our method with just 100 samples can surpass DAGMA even with 5 000 samples.
11

---

Table Causal discovery performance under noise misspecification on linear data with 30-node ER-2 graphs. The numbers are mean + standard error over runs. Bold: best performance, underline: second-best performance.
SHD (lower better)
Method MatioodNNNiiee
Noise
Exp (1)
Gumbel (0, 1)
Laplace (0, 1)
Uniform (-1, 1)
NOTEARS
Zheng et al.
2018
6.0
1.6
4.0 = 1.9
3.0
1.4
6.4 4.3
DAGMA Bello et.
2022
1.0
1.0
0.2 = 0.2
1.0
1.0
4.8 + 1.5
NOCURL Yu et al.
2021
10.8
0.8
6.6 - 1.7
4.2 1
1.0
29.0 1 3.0
COSMO Massidda
2024
5.4
2.0
5.0 2.3
7.6
2.5
5.6 + 1.8
RL-BIC Zhu et al.
2020
66.8 + 13.2
34.4 9.5
31.4 10.1
31.4 + 11.5
CORL (Wang
202T
12.0 1.4
15.4 2.4
15.6
15.0 + 0.9
RCL-OG Yang et al.
2023a
77.3 13.5
79.8 = 22.8
41.3 + 16.4
57.0 23.1
ALIAS (Ours)
0.4 0.3
0.4 0.4
0.8 +
0.4
0.4 0.3
Table 4: Causal discovery performance on nonlinear data with Gaussian processes on 10-node ER-4 graphs. The performance metrics are Structural Hamming Distance (SHD, lower is better), False Detection Rate (FDR, lower better), and True Positive Rate (TPR, higher is better). The numbers are mean standard error graphs are dense and the noise is additive, we also study the effect of pruning the output graphs with CAM
runs. Bold: best performance, underline: second-best, performance. Since the
pruning (Bühlmann et al. 2014).
No Pruning
CAM Pruning
Method
SHD (L)
FDR (4)
TPR (1)
SHD (L)
FDR (1)
TPR. (t)
NOTEARS
Zheng et 2020
28.4
0.33 0.06
0.33 0.04
28.6 1.0
0.32 0.06
0.32 0.04
DAGMA
Bello et al.
2022
25.8 1.7
0.32 13220000 0.04
0.40
26.0 1.8
0.31 + 0.04
0.39 = 0.06
NOCURL
Yu et 2021
35.2
0.47=0.00
0.05
0.8
0.06
0.15 = 0.04
COSMO
Massidda et al.
2024
26.1
0.04
0.09
2..0
0.28
0.35 - 0.05
RL-BICT Zhu al.
2020
2.00
0.06
0.04
1.
0.05 0.05
0.04 0.04
CORL
Wang et al.
202T
8.4
0.19
0.00
1.7
0.10 0.04
0.82 = 0.04
RCL-OG
Yang et al.
2023a
1.0
0.03
0.00
9.3
0.12 0.04
0.84 0.02
ALIAS (Ours)
0.8 + 0.4
0.01 + 0.01
0.99 + 0.00
4.6 + 0.9
0.01 + 0.01
0.89 - 0.02
Model misspecification. Next, consider the model misspecification scenarios when the Gaussian noise assumption is violated. In Table 3 we benchmark all methods on linear data with four types of non- non-Gaussian noiscs. The results indicate that our method is still the most robust to noise mis-specification, with an SHD of less than one in all four cases, and the lead performer in three out of four configurations. In addition, we further study the performance of our method under different model misspecification scenarios, including mismatched causal model, noisy data, and the presence of hidden confounders, in Appendix D.4
6.3 Nonlinear Data with Gaussian Processes
In this section, to answer the question whether our method can operate beyond the standard linear- linear-Gaussen Gaussian setting, we follow the evaluations in Zhu et al. (2020); Wang et al. (2021); Yang et al. (2023a) sample each causal mechanism Si from a Gaussian process with an RBF kernel of unit bandwidth, and the noises follow normal distributions with different variances sampled uniformly. We also follow Wang et al. (202)); Yang et al. (023a) to apply Gaussian process regression using the RBF kernel with learnable length scale and regularization a = -1 to calculate the BIC with non-equal variances (Section 3.2 for RL-BIC, CORL, RCT-OG, and our ALIAS method. For NOTEARS, DAGMA, and COSMO, we use their nonlinear versions where Multiple-layer Perceptrons (MLP) are used to model nonlinear relationships.
The empirical results reported in Table verify the effectiveness of our method even on nonlinear data. Our method outperforms all other baselines in all metrics, either with without pruning. Most remarkably,
12

---

Table 5: Causal discovery performance on real-world flow cytometry data (Sachs al., 2005) with 11 nodes, 17 edges, and 853 samples. Running time is compared among RL-basod methods. The figures for RL-BIC are as originally reported. Bold: best performance, underline: second-best performance. Since the causal model is potentially non-additive, we also consider CIT-based pruning with KCTT (Zhang
ct al. 2011
CAM Pruning
CIT Pruning
Total edges
Correct (1)
Total edges
Correct (1) edges
Method
SHD (1)
SHD (L)
edges
NOTEARS
Zheng ct al
2020
5
13
7
5
13
DAGMA
Bello
2022
2
15
6
2
15
NOCURL
Yu
2021
15
4
2
COSMO
Massidaa et al.
2014
16
5
2
RI-BIC
Zhu et al.
2020
10
I
11
CORL
Wang al.
2021
3
14
10
3
15
RCI--GG
Yang et
20/3a
13
9
2
ALIAS (Ours)
10
8
10
9
8
9
even without pruning, our method correctly identifies nearly every edge with an expected SHD lower than However, by using CAM pruning, there is slight degrade in performance of most methods, which could be due to CAM's inability to capture complex causal mechanisms drawn from Gaussian processes.
Furthermore, following Lachapelle et al. (2020), we also study the case of causal model misspecification with Post-nonlincar modols (Zhang & Hyvärinen 2009) in Appendix D.4 In addition, nonlincar modols generated using MLPs are also studied in Appendix D.5
6.4 Real Data
Next, to confirm the validity of our method past synthetic data, evaluate it the popular benchmark flow cytometry dataset (Sachs et al.) (205),, which involves protein signaling network based on expression levels of proteins and phospholipids. We employ the observational partition of the dataset with 853 samples, 11 nodes, and 17 edges.
The empirical results provided Table 5 show that our method ALIAS both achieves the best SHD and number of correct edges among all approaches. Specifically, using CAM pruming under the assumption of generalized additive noise models, achieve the lowest SHD of 10 compared with the second-best of 11 by RL-BIC. Meanwhile, when using 9 identified edges are correct. This is a state-of-tho-art level of SHD among existing studies on this datasct.
pruning, we even further reduce the SHD to 9, with 8 out of
6.5 Runtime Analysis
Here, we analyze the efficiency of ALIAS details. First, show the significance of our optimal quadratic complexity for sampling DAGs, our Figure 4a compares the runtime of our policy with the autoregressive sampling approaches in CORL and RCL-OG with cubic complexity. It can be seen that, our DAG sampling policy is much faster than other approaches and does not significantly slow down with increasing graph sizes. Mcanwhile, CORL and RCL-OG are nearly 140 times slower than ALIAS at 50 nodes, and the speedup ratio drastically increases with the growth of the graph. Second, in Figure 4p, we detail the runtime and performance all methods with varying numbers of nodes. For small- to moderate-sized graphs of up to 50 nodes, our method even faster than gradient-based methods NOCURL and COSMO. For larger graphs, RL-BIC and CORL become computationally expensive very rapidly, and while other methods become faster than ALIAS, their performance quickly degrade with significantly larger SHDs than our method.
13

---

Method
NOTEARS
10a
NOCURL
Runtime (Minutes)
COSMO
RI-BIC
RCL-OG
Nodes
ALIAS
CORL
RCL-OG
10"
CORT.
46.7 135.7x)
46.4 133.7xx)
ALIAS (Ours)
100
0.2
146.8
9000))
188.6 (1,155-9Xg gmx)
10-3
(1.261.5x))
471.3
Average
200
0.4
640.8
(1.896.8x)
915.5 (2,262.0x)
10"22
3)
[3, 9)
[9, 50)
25
50
75
100
120
150
175
200
50+
Number of Nodes
(a) Average sampling time for cach DAG in milliscconds.
(b) Causal Discovery Runtime.
Figure 4: Runtime analysis of ALIAS. (a) We demonstrate the efficiency of our O (df) sampling technique compared with the O (d*) approaches in CORL and RCL-OG. (b) We study the runtime of all methods with respect to graph size. The shaded areas depicts standard errors over 5 random linear-aaussaan datasets with the regular weight range 4 (1-2, 0.5] U [0.5, 2]) on ER-2 graphs.
6.6 Ablation Studies
In Figure D we study the elfect of the choice of graph scorer, method, and number of training sleps onto the performance of ALIAS compared with DAGMA and CORL as the representatives for continuous optimization and RI-based approaches. It be scen that all variants of method surpass the basclines, using few as 000 training steps. While all variants perform equivalently well, PPO proves to be better choice than A2C, with both variants PPO + BIC and PPO + LS can reach very close to zero SHD, while those of A2C are not as performant (Figure D)). The influence of other hyper-parameters can be found in Appendix D.3
Furthermore, show that the effectiveness of ALIAS is not only thanks to the Vec2DAG parametrization alone, but also the application of RL, as opposed to the gradient- based optimization approach commonly employed in the literature, replace RL in method with a gradient-based optimizer, which is popular among modern causal discovery methods, and compare the performances. However, since Vec2DAG is discrete, which renders the objective non-differentiabee as is, we make slight modifications to make it amenable for continuous optimization, and the adapted version for linear data is given as:
Vec2DAGcom: (E(z) + E(z) H(grad(((2)),
which still represents the weighted adjacency matrix of DAG. Still, H(-) is not differentiable, so we further use the Straight- Through estimator (Bengio et al. 2013 to estimate its gradients. Specifically, we use (grra((((z)) for the forward pass whereas the gradients of the inner part dgrad(p(z)) used for the d2 backward pass. This is not done similarly for the first term in Vec2DAG because that would require an additional weight matrix to represent linear coefficients, which is redundant compared with the above. Then, since the BIC score used in RL approach is also non-differentiable, we adopt sielihood-basste loss similar to BIC as follows (which is also used in, e.g., GOLEM, Ng et al. 2020
C(z) In
X X Vec2DAGson (z)1 + A A11z))
11
where X the sparsity regularization coefficient. We minimize this loss until convergence using the Adam optimizer (Kingma 2014) (same as NOCURL, COSMO, and the RL algorithm PPO in method).
14

---

Table Role of RL in ALIAS. We replace RL with continuous optimization using Adam (Kingma 2014 and compare with the RL version. The numbers are mean standard ervor over random datascts on 10-node ER-2 graphs.
Method
SHD (4)
FDR (1)
TPR (t)
Vec2DAG + contimuous optimization (Ir 10-2, A1 = 10 10
14.0 0.6
0.52 0.05
0.42 + 0.06
Vec2DAG + continuous optimization (Ir 10-2, A1 = 10-9)
9.6
0.36 + 0.06
0.65 0.08
Vec2DAG + continuous optimization (Ir 10-2,
Å1 10-?)
1..8
0.45
0.48 0.06
Vec2DAG continuous optimization (Ir 10-3
(0-3,0
Åι 10-3)
1..0
0.37 0.05
0.56 + 0.06
Vec2DAG continuous optimization (Ir 10
Y1 10
11.8
0.44 0.06
0.55
Vec2DAG + continuous optimization (Ir 10 3- ÅI
10-?)
8.4
0
0.62
Vec2DAG + contimuous optimization (Ir = 10 4 À1
= 10-3)
1.
0.53
0.49
Vec2DAG + contimuous optimization (Ir = 10-4
Y1 = 10-6)
12.0
0.05
0.59 0.04
Vec2DAG + contimuous optimization (Ir = 10-4,
A1 = =10-?
2.8
0.45
0.05
Vec2DAG + contimuous optimization (Ir = 10
A = 10
12.6
0.49
0.15
Vec2DAG + continuous optimization (Ir 10
5
À1 10-9)
13.0
0.51 0.09
0.51 0.10
Vec2DAG + continuous optimization 10 A 10-
12.2
0.49 0.08
0.51 0.08
Vec2DAG + RL
0.0
0.00 0.00
1.00 0.00
We provide the results in Table with a wide range hyperparameter choices for the above approach. It can be seen that even in this simple case, the continuous optimization approach performs poorly and cannot compete with our RL approach, confirming that ALIAS's effectiveness attributed greatly by RL, not just Vec2DAG alone.
7 Conclusions
In this study, novel causal discovery method based RT proposed. With the introduction of DAG characterization that bridges unconstrained continuous space to the constrained DAG space, devise RL policy that can generate DAGs efficiently without any enforeement of the acyclicity constraint, which helps improve the search for the optimal score drastically. Experiments on a wide array of both synthelic and real datasets confirm the effectiveness our method compared with srate-of-the-ate baselines.
Regarding limitations, the RL approaches in our study, which are online RL methods, may be limited in sample efficiency, exploration data is not effectively ultilized to prioritize visiting promising DAGs, thus potentially requiring more explorations than needed to reach the optimal DAG. Towards this end, more sample-efficient RL approaches, such as Optimistic PPO (Cai et al. 2020), or reward redesign can be considered to enhance exploration in our method, and thus further improve its efficiency.
Future work may involve doepening the understanding on the convergence propertics of our mothod and extending to more intriguing settings like causal discovery with interventional data and hidden variables.
15

---

References
Yashas Annadani, Nick Pawlowski, Joel Jennings, Stefan Baner, Cheng Zhang, and Wenbo Gong. BayesDAG: Gradient-based posterior sampling for causal discovery. In Advances in Neurol Information Processing Systems, 2023.
Kevin Bello, Bryon Aragam, and Pradeep Ravikumar. DAGMA: learning DAGs via M-matrices and a log-determinant acyelicity characterization. Advances in Neural Information Processing Systems, pp. 8226 8239, 2022.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In Advances in Neural Information Processing Systems, pp. 27381 27394, 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprinl arX.v:1308.3432, 2013.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, pp. 1-55, 2023.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. Advances in Neural Information Processing Systems, pp. 21865-21877, 2020.
Peter Bühlmann, Jonas Peters, and Jan Ernest. CAM: Causal additive models, high dighddimessiona order search and penalized regression. The Annals of Statistics, 42:2526-2555 2556, 2014.
Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. Proceedings of the International Conference on Machine Learning, pp. 1283 1294, 2020.
Yinan Cao, Christopher Summerfield, Hame Park, Bruno Lucio Giordano, and Christoph Kayser. Causal inference in the multisensory brain. Newron, 102:1076 1087, 2019.
Bertrand Charpentier, Simon Kibler, and Stephan Günnemann. Differentiable DAG sampling, In Proceedings of the International Conference on Learning Representations. 2022.
Wenyu Chen, Mathias Drton, and Y Samuel Wang. On causal discovery with an equal-variance assumption. Biometrike, 106:973-980) 2019.
David Maxwell Chickering. Learning Bayesian networks is NP-complete. Learning from data: Artificial intelligence and statistics V, pp. 121-130, 1996.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of Machine Learning Research,
2002.
Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high sigh-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pp. 104--21,44 2012.
Chris Cundy, Aditya Grover, and Stefano Ermon. BCD nets: Scalable variational approaches for Bayesian causal discovery. In Advances in Neural Information Processing Systems, pp- 7095-7110, 2021.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Proceedings of the Uncertainty in Artificial Intelligence, pp. 518-528, 2022.
Tristan Deleu, Mizu Nishikawa-Toomey, Jithendaraa Subramanian, Nikolay Malkin, Laurent Charlin, and Yoshua Bengio. Joint Bayesian inference of graphical structure and parameters with single generative flow network. Advances in Newral Informalion Processing Systems, 2024.
Paul Erdös and Alfréd Rényi. On the evolution of random graphs. Publications of the Mathematical Institute of the Hunariin Academy of Sciences, 1960.
16

---

Dominique Marie-Annich Haughton. On the choice of a model to fit data from exponential family, The Annals of Stalislics, pp. 342-33--- 1988.
Alain Hauser and Peter Bühlmann. Characterization and greedy learning of interventional markov equivalence classes of directed acyelic graphs. The Journal of Machine Learning Research, 13:2409 2464, 2012.
David Heckerman, Dan Geiger, and David M Chickering. Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning, pp. 197-243, 1995.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in Newral Information Processing Systems, 2008.
Paul Hünermund and Flias Bareinboim. Causal inference and data fusion econometrics. The Reonomelries Journal, pp. utad008, 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6985 2014.
Harold W Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, pp. 83-97, 1955.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient- neural DAG learning. In Proceedings of the International Conference Learning Representations, 2020.
Hao-Chih Tee, Matteo Danieletto, Riccardo Miotto, Sarah T Cherng, and Joel T Dudley. Scaling structural learning with NO-BEARS to infer causal transcriptome networks. In Proceedings of the Pacific Symposium on Biocomputing, pp. 391-402, 2020.
Lek-Heng Lim. Hodge Laplacians on graphs. Siam Review, 62:683-715, 2020.
Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery without acyclicity constraints. In Proceedings of the International Conference on Learaing Representations, 2022.
Riccardo Massidda, Francesco Landolfi, Martina Cinquini, and Davide Bacciu. Constraint-free structure learning with smooth acyclic orientations. In Proceedings of the International Conference on Learning Representations, 2024.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pp. 1928-1937, 2016.
Ehsan Mokhtarian, Sina Akbari, AmirEmad Ghassami, and Negar Kiyavash. A recursive Markov boundary- based approach to causal structure learning. In The KDD '2 Workshop on Causal Discovery, pp. 26 54, 2021.
Ehsan Mokhtarian, Sina Akbari, Fateme Jamshidi, Jalal Etesami, and Negar Kiyavash. Learning Bayesian networks in the presence of structural side information. In Proceedings of the AAAT Conference on Artificial Intelligence, pp. 7814-7822, 2022.
Ignavier Ng, AmirEmndd Ghassami, and Kun Zhang, On the role of sparsity and DAG constraints for learning linear DAGs. Tn Advances Newral Information Processing Systems, pp. 17943 -19944,, 2020.
Judea Pearl. Causality. Cambridge University Press, 2009.
Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 2014.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselinee Reliable reinforcement learning implementations. Journal of Machine Learning Research, pp. 1-8, 2021.
17

---

Anditya Ramdas, Sashank Jakkam Reddi, Barnabás Póczos, Aarti Singh, and Larry Wasserman. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2015.
Joseph Ramsey, Madelyn Glymour, Ruben Sanchez-Romero, and Clark Glymour. A million variables and more: the Fast Greedy Equivalence Search algorithm for learning digh-dimensional graphical causal models, with an application to functional magnetic resonance images. International Journal of Dala Science and Analytics, 3:121 11121-129, 2017.
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware the simulated dag! causal discovery benchmarks may be easy to game. In Advances in Neural Information Processing Systems, volume 34, pp. 27772 27784, 2021.
Jorma Rissanen. Modeling by shortest data description. Automatica, pp. 465-471, 1978.
Robert W Robinson. Counting unlabeled acyelic digraphs. In Combinatorial Mathematics V, pp. 28-43. Springer, 1977.
Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello. Score matching enables causal discovery of nonlinear additive noise modcls. In Proceedings of the International Conference Machine Learning, pp. 18741-18753, 2022.
Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal prolein-signaling networks derived from multiparameter single-cell data. Science,
2005.
Pedro Sanchez, Xiao Liu, Alison O'Neil, and Sotirios A. Tsaftaris. Diffusion models for causal discovery via topological ordering. In Proceedings of the International Conference Learning Representations, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (cXiv:1007.0344 2017.
Gideon Schwarz. Estimating the dimension of model. The Annals of Statistics, pp. 461-464, 1978.
Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics, pp. 876-879, 1964.
Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social Science Computer Review, 9:62-72, 1991.
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT Press, 2000.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advonces in Neural Information Processing Systems, 1999.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicene Andrea Pierré, Sander Schulhoff, Jun Jet. Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, 2023.
Xiaoqiang Wang. Yali Du, Shengyu Zhu, Liangjun Ke, Zhitang Chen, Jianye Hao, and Jun Wang. Ordering- based causal discovery with reinforcement learning, In Proceedings of the International Joint Conference Artificial Intelligence, pp. 3566-366 3573, 2021.
Dennis Wei, Tian Gao, and Yue Yu. DAGs with No Fears: A closer look at continuous optimization for learning Bayesian networks. In Advances in Neural Information Processing Systems, pp. 3895-3906, 2020.
Dezhi Yang, Guoxian Yu, Jun Wang, Zhengtian Wu, and Maozu Guo. Reinforcement causal structure learning order graph. Tn Proceedings of the AAAT Conference on Artificial Intelligence, pp. 10737-10744 2023a.
18

---

Dezhi Yang, Guoxian Yu, Jun Wang, Zhongmin Yan, and Maozu Guo. Causal discovery by graph attention reinforcement learning. In Proceedings of the SIAM International Conference Dala Mining, pp. 28-36, 2023b.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks.
In Proceedings of the International Conference Machine Learning, pp. 7154-7163, 2019.
Yue Yu, Tian Gao, Naiyu Yin, and Qiang Ji. DAGs with No Curl: An efficient DAG structure learning approach. In Proceedings of the International Conference on Machine Learning, pp. 12156-12166, 2021.
Keli Zhang, Shengyu Zhu, Marcus Kalander, Ignavier Ng, Junjian Ye, Zhitang Chen, and Lujia Pan. gCastle: A python toolbox for causal discovery. arXiv preprinl arXi::2111.15155, 2021.
Kun Zhang and Aapo Hyvärinen. On the identifiability of the post-nonlinear causal model. In Proceedings of the Conference on Uncertainly in Arlificial Intelligence, pp. 647- 2009.
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pp. 804-813, 2011.
Zhen Zhang, Ignavier Ng, Dong Gong, Yuhang Liu, Ehsan Abbasnejad, Mingming Gong, Kun Zhang, and Javen Qinfeng Shi. Truncated matrix power iteration for differentiable DAG learning. In Advances in Neural Information Processing Systems, pp. 18390-18402, 2022.
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGs with NO TEARS: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, pp. 1472-9483, 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. Learning sparse nonparametric DAGs. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2020.
Shengyu Zhu, Tgnavier Ng, and Zhitang Chen. Causal discovery with reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2020.
19

---

def Vec2DAG(2) >> up adarray:
P z[:d] E up. zaroa d))
E[np.triu, indp (d, -1] =[d:] * (d(da)1))/
CE E.T 0) (pt:, None] [None, :))
Figure 5: Unconstrained DAG parameterization. This function takes as input a real-valued vector z E
12/411/ and deterministically transforms into an adjacency matrix of d-node DAG.
Appendix
A Details about BIC scores
A.1 Non-equal variances BIC
Recall that the additive noise model under Gaussian noise is given by X; = fi (Xpa,) + Ei, where Es ~
W (0, This implies X, ~ N (fs (Xpa.) oo) and the log-likelihood of an empirical dataset DD {x(x)) jkm
is given by
C. = 1mppp (,
(7)
1
(K)
fi
T'pa;/
(k)
of
k=1
n Σ o + const,
(9)
2
where the constant does not depend any variable.
The maximum likelihood estimator for fa can be found via least square methods, and that of of can be found
0L by solving do 0, which yields
6
(k)
ft
(k)
(10)
pa,
71
SSR,
Plugging this back to Eqn.
gives
SSR;
C
In
const.
(11)
i=1
Finally, we obtain the BIC score for the non-equal variances case by incorporating this into Eqn.
13
d
SSR
BIC (D,9) = n
In
[ In n + const,
(12)
20

---

A.2 Equal variances BIC
Similarly to the unequal variances case, by assuming 01 od 0, solve for
de = 0 and obtain
0c
0² nd
(k)
f:
2
pa,
(R)
))
(13)
i=1
k=1
SSR,
Substituting this estimate into Equ.
yields with
nd
> 2im1 SSR;
L
(14)
nd
For the last step, the BIC score for the equal variance case is given by substitution as
BIC (D,g) nd In
1= nd
SSR; + [ In const.
(15)
B Proofs
B.1 Proof of Lemma
Lemma. Let G be the ground truth DAG of an identifiable SCM satisfying causal mindmality [Peters et al, 2014) (i.e., there are no redundant edges) inducing the dataset D, and let n be the sample size of D. Then, in the limit of large n, SaIe (D, G*) Saie (D,9) for any + g*
Proof. Let e be the parameter of the causal model generating the dataset D. Because the causal model is identifiable and (*) are the true parameters generating the data D, the likelihood (D ""' is the highest possible. For any incorrect DAG g + G", there only exists parameter & such that p (D 10 "*) (D w. () if g C G because of causal minimality. Otherwise, the difference between the likelihoods are given by
In (D) (6,"") Inp(D) 1 9)
Σ Inp (x) (6""9)) Inp (x(*) 1 (1.)))
(16)
m KL (p (D 10 , 11 (D vb.g)) to(n)
(17)
k(g.9)
where the second equality follows from the asymptotic behavior of the log-likelihoods. Therefore, for any incorrect DAG 626 g*:
SpIC (D, SBIC (D,9) n n
+0) (g + (19) |g") Inn.
(18)
As 0o, if g" g G then the KI. divergence term grows linearly with n, dominating the logarithmic growth of the penalty term. On the other hand, if G* c G then the difference between the likelihoods vanishes and is therefore dominated by the penalty term. Thus, for any g + g", the BIC score satisfies SBIC (D,9*) > SBIC (D,9) as n + 0o.
B.2 Proof of Theorem i
Theorem. For all d N+, let Vec2DAG, : Sa {0, 10.1)xxt be defined as in Eq. Then, Im (Vec2DAG.) Dd.
21

---

To show that Vec2DAG is surjective, we first show that every point in the parameter space maps to directed graph without any cycle, and vice-versa, for any DAG, there always exists a vector that maps to it.
Lemma. For all d Nt, let Vec2DAGa Sa {0, 1} dxd be defined in Eq. 5). Then, Vec2DAG (z) E Da Vz E Sd-
2021 Specifically, let A Vec2DAGa (z) = H E (z) + Ez)) ④ H (grad (p (z))). Then, the necessary condition for an edge from to j to exist is p: < Py. By contradiction, assuming there exists cycle (i + + is + i1) in A, then it follows that ps, <... < Pi < Pii This contradicts with the total-ordering property of real values, thus concluding our argument.
Proof. The proof follows the same argument with Theorem 2.1 of Yu et al.
Lemma. For all d E N+, let Vec2DAGa S + {0, 13 1ddd be defined as in Eq. A Da and > O, there erists (- (d+1)/2 (- C Sa such that Vec2DAG, (z) A.
5
Then, for any DAG
Proof. We prove by construction. Let an, = fi there is a path from j to i} be the set of ancestors of i. It follows that if i + j is in A then every ancestor of is an ancestor of j, so an; C any and construct the unnormalized
an, an; U {i} U (an; an; \ {i)), which means Jan,| < any/- Therefore,
node palentials as yo land] and normalize it to fit into the (-e,6) range
P. min P.
1 2
Pr
X E
(19)
max P: - min pj
We then construct the upper-triangular edge potentials matrix
1/2e ifi<j and Ai + Ass =
Fig- = -1/22 otherwise.
(20)
We now verify that this pair of potentials leads to the exact binary matrix A via Eq. (5) First, Eij + Ej: = 1/2e > 0 if i and are directly connected in A, and Eij + Eji = -1/2e < 0 otherwise, thus H (E+E* is the undirected version of A. Second, for any directed edge j + i in A, we have Jan,| < langl. which leads to grad (p),g = P, - Pt > 0, and thus H (grad (p) encodes the direction for every directed edge in A. Therefore, using the Hadamard product, we mask out the edges in H (grad (p)) that do not exist in A, and at the same time give direction to any available edge in H (E+ + E resulting in exactly A.
Lemmas
B.2 and
B.2 then completes our proof of Theorem 1
B.3 Proof of Lemma
Lemma. (Scaling and Trunslation Trevaiiance). For all d E NN, let Vec2DAG, : Sa - {0, 10xd be defined as Eg. Then, for all z Sd, a > 0, and B E Sa such that (B), < 1/2min, [z); - p (z), and
E (B) < E (2),g) Vi,j, we have Vec2DAG, (z) = Vec2DAG (a (z + B)).
Proof. We first show that Vec2DAG, = (ec2DAG 2) Va > 0. Let A = Vec2DAG, (z). This follows from the fact that
Vec2DAG (z) = p (z), < pz))
(21)
E(z),) E Ez),; >0
(z);;); < ap (z),
(22)
(2) +E (z) >0
+ Vec2DAG, (a 2)11 = =1,
(23)
22

---

equivalently, Vec2DAG, (z) Vec2DAG (a.z).
and E ( E (z);g Vi These conditions state that translating the potentials by an amount , that does not change the ordering of p or the element- wise positivity of E also leads to the same DAG. First, can be seen that if P: < Pj and P/ E (-)/2 - p:) 1/2 (p) p.)), then Pt + p < 1/2 (p: + < Ps + P/j. Therefore, p (3), < 1/2 p (z); p (2), which also applies when p (2); < (z) + p [z); + p (3); < (2),
Next, we prove Vec2DAG, (z) = Vec2DAG, (z 3)) if lp (3),1 < min, P (z); - (pp))-p(2)) pz))]
Ip (8), < 1/2 minj p (z); -
( (2),
Similarly, ( > 0 = E (2),, + E (3); > if (9);;) < E (2).gg) Combining these two points entails that if E (8) < (2),s Vi,j then we oblain
(z); < p()2, E (Z), + E (z),; >0
(x) = 1
(24)
[z); + (p)), <p(z), +p +p (3),
(25)
E (B()); E9)):) (z), + E()))s))) >0
Vec2DAGa (z+ B) 1.
(26)
Finally, combining Eqs.
23 and 26
concludes our proof for Lemma
Vec2DAG, (a (z + ()) = Vec2DAG, (z + 3) Vec2DAG (z)
B.4 Proof of Lemma
Lemma. (Prozimity between DAGs). Let E Sq. Then, for any DAG A E Da and 0, there cxists ZA in the wnit ball B (oc;||z]], + c) around such that Vec2DAG, (aA) A.
Proof. Lemma B.2 established that there exists representation ZA E (-e, (-E,e) (d+1)/2 for any DAG A and > Therefore, the distance between z and ZA is bounded in Too norm by the triangle incquality as 2 ZAles S |z/ + Zal < |2/20 +
B.5 Proof of Lemma 4
Lemma. Assuming causal édentifiability and causal minimality, that is, there is unique cousal model with redundant edges that produce the observed dataset, and BIC score is used to define the reward R (z) as in Eq. OV. Let n be the sample size of the observed dataset D, 6" E arg maxEzan [R (z)], where no (z) N (z: noo diog (o))). Then,
BE0
00, Vec2DAG (z) is the true DAG, where no-
Proof. Let G* be the true DAG. By the identifiability and minimality of the causal model and Lemma > 20, z² E arg max R (z) if and only if Vec2DAG (z*) = This implies that the reward R (z) is only maximized when z corresponds to g".
Now, consider the objective 0" E arg max maxEx-Ta [R (2)). One solution to this maximization problem when fce
Te places all of its probability mass on an arbitrary arg max (z), Le.,, Re- (z) 8 (z z²); where 8 (-) is the Dirac delta function. This is achieved when μé and σ² In this case, any sample z ne-
satisfies z".
23

---

On the other hand, if me assigns probability mass to multiple values of arg max R (z), then it is regular Gaussian distribution and thus will also put non-zero masses on the values with smaller rewards, thus strictly reducing the value of the expected reward. Thus, Te cannot achieve the maximum expected reward unless it is a Dirac delta distribution concentrated at z".
Finally, since Vec2DAG (z*) = =g", we have 9 Vec2DAG (z) = g with probability when z no- Therefore, as n co, Vec2DAG (z) the true DAG.
C Experiment Details
C.1 Datasets
C.1.1 Synthetic Linear-Gaussin Data
To simulate data for a given number of nodes d, we first generale a DAG following the Erdös-Rényí graph model (Erdös & Rényi, 1960) with a random ordering and an expected in-degree of lo E N+, denoted by ER-k. Next, edge weights are randomly sampled from the uniform distribution 4 ([-5, 22 U [2, 5)), giving a weighted matrix W E Raxd where zero entries indicate no connections, then the noises are sampled from the standard Gaussian distribution Fg N (0.
1,0 observations for each dataset following the linear assignment X; Ejepa, W,,X, + E,, While linear- Ganssiann models are non-identifiable in general (Spirtes et al. 20000, the instances with equal noise variances adopted in this experiment 2014). This data generation process is similar to multiple other works such as Zheng et al. (2018): ZZhu et al. 2020);; Wang et al. (2021) and is conducted using the well-established gCastle utility for causality research (Zhang et al. 2021).
Finally, sample
known to be fully identifiable (Peters
et al.
C.1.2 Nonlinear Data with Gaussian Processes
We evaluate the performance of the proposed method ALIAS with competitors on the exact 5 datasets used by Zhu et al. (2020) in their experiment, which produced by Lachapelle et al. (2020) (https //github.com//
MIT license). In addition, we also consider the first datasets of the PNL-GP portion of their datasets for the model misspecification experiment.
C.2
Experiment Details
C.2.1 Evaluation Metrics
The estimated graphs are assessed against ground truth DAGs multiple evaluation metrics, including the commonly employed Structural Hamming Distance (SHD, lower is better), False Detection Rate (FDR, lower better), and True Positive Rate (TPR, higher better). SHD counts the minimal number of edge additions, removals, and reversals in order to turn one graph into another. FDR is the ratio of incorrectly estimated edges over all estimated edges, while TPR is the proportion of correctly identified edges over all true edges. We also utilize gCastle for the caleulations of the aforementioned metrics.
C.2.2 Implementations of Methods
Our set of baseline methods contains a wide range of both well-established and recent approaches, including the constrained continuous optimization approaches with soft DAG constraints NOTEARS ZZeng et al.) 2018 2020) and DAGMA (Bello al.] 2022), unconstrained continuous optimization approaches NOCURL YYu et al. 2021 and COSMO (Massidda et al. 2024). as well as three RL-based methods RL-BIC (Zhu et al. 2020), CORL (Wang et al. 2021 and RCL-OG (Yang et al. 2023a). where the implementations are publicly available. More specifically,
hhttps://github..cm// http://ss..ccc.aaaaaa////sssssss version 1.0.3, Apache-2.0 license.
24

---

2020 is continuous optimization method that optimizes over the continuous space of weighted adjacency W E Rdxd for linear models, with a soft constraint W is DAG + h (W) = e w-W - d = 0, which is solved using augmented Lagrangian methods. For nonlincar data, W is used to mask the input of the MLPs that model the nonlinear cansal mechanisms.
NOTEARS
(Zheng et al. 2018
DAGMA (Blll et al. W is DAG + (W)
2022) is an altornative proposal to NOTEARS, with the acyclicity constraint
log del (sI We W) log * = for all 8 > 0.
2021 is proposed as an unconstrained continuous optimization method that represents the weight matrix linear model by = W @ ReLU (grad (p)), where W E Rdxd and grad (p);3 := Py Pi. Their optimization strategy is to first find solution from an unconstrained method like NOTEARS, then refine it with this new unconstrained reprosentation.
NOCURL Yu et al.
COSMO (Massidda et al. 2024) improves upon NOCURL by parametrizing the adjacency matrix with A = W sigmoid (grad (p) /7), which can be shown to converge to DAG when 7 0+. They optimize directly with this formulation instead of employing the two-stage approach like NOCURL.
RL-BIC (Zhu et al. 2020 is the first RL-basod method, which involves actor-critic agent that learns to output higl-reward graphs. The acyclicity is incorporated into the reward to penalize cyclic graphs.
CORL (Wang et al. 2021 is the first RL method for ordering- based causal discovery, which revolves around an agent that learns to produce causal orderings with high rewards. The policy sequentially generates each element of the ordering, then the causal order is pruned to obtain a DAG.
RCL-OG (Ynng et al. 2023a is an alternative RL approach for ordering based causal discovery, which improves the state space design of CORL. It reduces the state space size from permutations with a factorial size O (d!) to only O (244).
For NOTEARS, RL-BIC, and CORL, we also adopt the implementations from the gCastle package. For RCL-OG, we employ the implementation provided by the authors at https:///ww..st sdu-idea cn/ htd/coes.ppp name=RCL-OG (no license provided). Since RCL-OG is essentially Bayesian method, we use the best-scoring DAG from its 1 1000 posterior samples as the output for fair comparison with other methods.
Default eyper-parameters for each implementation are used unless specifically indicated. The detailed configurations of all methods are provided in Appendix C.2.3
| Our proposed ALIAS method is implemented using the Stable- Basblneas toolset (Rafinn al. 2021 with
the Advantage Actor-Critic (A2C, Mnih et al., 2016) and Proximal Policy Optimization (PPO, Schu maan
et al. 2017) methods, and custom DAG environment built on top of Gymnasium Towers et al. 2023 See Appendix C.2.3 the syper-parameters for our methods.
Experiments are executed on a mix of several machines running Ubuntu 20.04/22.04 with the matching Python environments, including the following configurations:
AMD EPYC 7742 CPU, 1TB of RAM, and Nvidia A100 40GB GPUs.
Intel Xeon Platinum 8452Y CPU, 1TB of RAM, Nvidia H100 80GB GPUs.
Intel Core 19 13900KF CPU, 128GB of RAM, Nvidia 4070Ti Super 16GB GPU.
The first configuration is used for batched exccutions of all methods, while the second is for iestervc-intensive experiments like high-dimensional ones with RL-based baselines, and the last configuration is for prototyping experiments. We find that RL-based basclines heavily rely on GPU with much slower (hours of) runtime on CPU, even on 30-node graphs, while our method is less dependent on GPU and can even handle datasets of up to 100 nodes in 45 minutes purely on CPU (single process).
httpp://gsthhbb. wtw.Dotn//D/eeeeee.o httsttalleeeaaeeeeeeeeeeeeeeeaea baselines MIT license.
https://gittub.. com/ Farama ioniiiattioo//mmmsiinn MIT license.
25

---

Table 7: Default Syper-parameters for our method ALIAS throughout the experiments. Unmentioned hyper-parameters are left unchanged.
Lineur data (Figures Tables
Nonlineur GP data (Table
Real data (Table
Experiment
and
and Table 19
Batch size (No. parallel environments)
64
Training steps
20,000
No. stepe to run for each update
Data standardization (dimension-wissc)
No (variances should remain equal)
No (data std. is near unit)
Yes (data std. is large)
RL method
PPO
Advantage normalization
Yes
Learning rate
Stable Baseline3" default (0.0003 for PPO and 0.0007 for A2C)
Regression method
Linear Regression
Gaussian Process Regression with RBF kernel
Scoring method
BIC equal variances, assuming Gaussian noises
non-oqual variances, assuming Gaussian noises
lo regularization for LS score (not used in comparative experiments)
0.000001
Table Default lyper-parametens for CORL (Wang et al. apper-parameters are left unchanged.
2021
throughout the experiments. Unmentioned
Linee data FFigures
Nonlinear GP data Table and Table 19
Real data (Table 5
Experiment
Tables
Batch size
64
Trainigg steps
10,000
Data standardization (dimension wise)
No (variances should zemain equal)
No (data std. is unit) Yes (data std. is large)
Regression method
Linear Regression
Gaussian Process Regression with RBF kernel
Scoring method
BIC equal variances, assuming Gaussian molses
BIC non-equal variances, assuming Gaussian nolses
C.2.3 Sypper-aaraeters.
We provide the byper--arraeetes for each method and experiment scenario as follows:
ALIAS (Ours): Table
NOTEARS and DAGMA: Table
13
NOCURL: Table
COSMO: Table
12
RL-BIC: Table
CORL: Table
RCL-OG: Table
10
D Additional Results
D.1 Results on ER graphs
In Table 14 we provide detailed numerical results for Figure i.c., effect of dimensionality and sample size Enear-Gaussin data and ER-8 graphs.
26

---

Table 9: Default typer-parameters for RL-BIC hyper-parametent left unchanged.
Zhu et al.
2020
Uhroughout the experiments. Unmentioned
Liurer data (Figures
Nonlinear GP data (Table
Real data (Table 5
Experiment
Tables 2
14
15
16
and
and Table 19
Batch size
64
Training steps
20,000
Data standardization (dimensiinn-wiee)
No (varianoes should remain equal)
No (data std. is near unit) Yes (data strl. is lange)
Regnssion method
Lincar Regression
Gaussian Proces Regression with RBF keruel
Scoring method
BIC equil variances, assuming Gounsiau uoises
BIC nou-equal varianoes, assuming Gaussian noises
Table 10: Default hyper- parameters for RCL-OG (Yang et al.
2023a throughout the experiments. Unmen-
tioned hyper-parte paremearseeeess are left unchanged.
Experiment
Linear data [ Figures
and
3
Nonlinear GP data (Table
1
Real data (Table
Tables
and
and Table 19
Batch size
Training steps
40,000
Data standardization (dimenssion-wise
No (variances should remain equal)
No (data std. near unit) Yes (ditta large)
Regression method
Linear Regression
Gauesian Process Regression with RBF kernel
Scoring method
BIC equal variances, assuming Gaussian noises
BIC non-oqual varianoes, aseuming Gaussian noises
Default typer-paramedt for NOCURL (Yang et al. 2023a) throughout the experiments. The values are the best parameters yielding the lowest SHD over linear- Gaussian datasets with 30-node ER-4 graphs, found via the tuning process provided by COSMO's implementation. Unmentioned hyper-parameters are left unchanged.
Table 11:
Linear data (Figures Tables
Nonlinear GP data (Table
Real data (Table
Experiment
14
15
16
and Table 19
Batch size
Inuer iterations
5000
Data standardization (dimenuionn-wscc)
No (variances should remain equal)
No (data std. unit)
Yes (data std. large)
Learning rate
0.00097477533556688881
Regularization streugth
0.0074786480099966166
Table 12: Default hyper-paramcters for COSMO (Massidda et al. 2024 throughout the experiments. The values are the best parameters yielding the lowest SHD over linear- Gaussian datasets with 30-node ER-4 graphs for linear data, and MLP datasets with 40-node ER-4 graphs for nonlinear data. They are found via the tuning process provided by COSMO's implementation. Unmentioned hyper-parameter are left unchanged.
Experiment
Linear data (F [Figures
BILd
Noulinear GP data (Table
Real data
Tables
14
15
16
and
18
and Table 19
(Table
Batch size
64
Max epochs
5000
2000
Data standardization (dimension-wiss
No (variances should remain equal)
No (data std. is near unit)
Yes (data std. is large)
Learning rate
0.00442477334758977844
0.001 1 180644488676536
regularization strength
0.0007788031586554470066
0. 0.000999704401138736
la regularization strength
0.029171977799955334
9.00114007514255196225
Priority regularization strength
0J00078019726602225771
0.N01455:03387//11006922
Temperature
0.00084074265500009770
0.0007651933117777666
Shift
1.005860556666624477336
0.00932:033533222377
MLP hidden mnits
0
10
27

---

Table 13: Default hyper-parameters for NOTEARS Zheng et al. 2018 2020
and DAGMA Bello et al. 2022
throughout the experiments. Unmentioned hyper- parameters are left unchanged.
Linear data [ Figures
and
Nonlinear GP data (Table
Real data
Experiment
Tables
3
and
and Table 19
(Table
Data standandization (dimension- )
No (varlances should remain equal)
No (data std. is near unit)
Yes (data std. is large)
SEM
Linear
MLP
Loss type
Table 14: Causal discovery performance as function of dimensionalities and sample size on ER-8 graphs. The mumbers are mean + standard error over random datasets.
Nodes
Samples
Method
SHD (1)
FDR (4)
TPR (t)
DAGMA
40.6 = 3.5
0.09 0.01
0.78 0.02
100
ALIAS
24.8 + 4.6
0.04 0.01
0.86 0.02
DAGMA
36.8 3.8
0.08 = 0.01
0.8 + 0.02
500
ALIAS
3.0 1.1
0.01 0.00
0.98 + 0.00
20
DAGMA
35.6 3.6
0.07 0.01
0.8 0.02 0.99 + 0.00
1000
ALIAS
1.0 0.6
0.0 0.00
DAGMA
31.8 3.6
0.07 + 0.01
0.82 + 0.03
2000
ALIAS
0.2
0.0
1.0 + 0.0
DAGMA
31.2142
0.07 0.01
0.83 0.03
5000
ALIAS
0.2 + 0.2
0.0 0.0
1.0 0.0
DAGMA
87.0 + 10.6
0.19 + 0.03
0.78 + 0.02
100
ALIAS
51.8 1 7.9
0.1 + 0.02
0.86 + 0.02
DAGMA
9.5
0.13 + 0.02
0.81 0.02
500
ALIAS
3.6
0.01 0.0
0.99 + 0.0
30
DAGMA
8.0
0.14 + 0.02
0.82 + 0.02
1000
ALIAS
0.2 0.2
0.0 + 0.0
1.0 0.0
DAGMA
65.8 7.9
0.13 + 0.02
0.82 0.02
2000
ALIAS
0.0 1 0.0
0.0 1 0.0
1.0 0.0
DAGMA
68.4 10.2
0.14 0.02
0.81 + 0.03
5000
ALIAS
0.0 0.0
0.0 0.0
1.0 + 0.0
DAGMA
242.0 + 15.6
0.34 + 0.02
0.76 + 0.01
100
ALIAS
94.2 10.1
0.14 + 0.02
0.9 0.01
DAGMA
213.2 + 20.8
0.31 + 0.03
0.78 + 0.01
500
ALIAS
13.6 1 5.5
0.02 + 0.01
0.99 + 0.00
50
DAGMA
215.2
26.6
0.3 + 0.04
0.77 0.02
1000
ALIAS
2.2 1.0
0.0 0.0
1.0 + 0.0
DAGMA
213.2
25.8
0.31 + 0.03
0.78 + 0.02
2000
ALIAS
1.8 0.8
0.0 + 0.0
1.0 0.0
DAGMA
210.2 32.4
0.3 + 0.04
0.78 0.02
5000
ALIAS
0.7 4: 0.5
0.0 1 0.0
1.0 + 0.0
28

---

D.2 Results on SF graphs
Table 15
investigates the effect of dimensionality and sample size on linear-Gaussian data and Scale-Free (SF)
graphs with parents per node.
Table 15: Causal discovery performance as function of dimensionalities and sample size on SF-8 graphs. The numbers are mean + slandard error over random datasets.
Nodes
Samples
Method
SHD (4)
FDR (4)
TPR (t)
DAGMA
14.6 + 3.5
0.07 + 0.03
0.87 + 0.02
100
ALIAS
16.0 + 2.8
0.09 + 0.02
0.89 + 0.03
DAGMA
6.8 + 1.4
0.02 + 0.01
0.93 + 0.01
500
ALIAS
0.6 + 0.3
0.01 + 0.00
1.0 0.00
20
DAGMA
7.0 1.7
0.02 0.01
0.93 + 0.02
1000
ALIAS
0.2 0.2
0.0 0.00
1.0 = 0.00
DAGMA
9.0 + 3.0
0.04 + 0.02
0.91 0.03 1.0 0.00
2000
ALIAS
0.2 0.2
0.0 0.00
DAGMA
6.6 2.5
0.03 = 0.02 0.02
0.94 0.02
5000
ALIAS
0.2 0.2
0.0 0.00
1.0 0.00
DAGMA
1.1
0.15 + 0.04
0.84 + 0.01
100
ALIAS
40.2114.4
0.1 1 0.01
0.92 1 0.01
DAGMA
45.8 7.9
0.16 + 0.03
0.85 + 0.02
500
ALIAS
0.2 0.2
0.0 0.0
1.0 + 0.0
30
DAGMA
43.6 1 10.0
0.14 + 0.04
0.86 + 0.03
1000
ALIAS
0.2 0.2
0.0 + 0.0
1.0 + 0.0
DAGMA
36.6 9.7
0.12 0.04
0.89 + 0.02
2000
ALIAS
0.0 0.0
0.0 0.0
1.0 = 0.0
DAGMA
27.4 4.7
0.09 + 0.03
0.9 + 0.01
5000
ALIAS
0.0 0.0
0.0 0.0
1.0 + 0.0
DAGMA
101.2 +: 18.3
0.18 0.04
0.84 1 0.02 0.92 + 0.01
100
ALIAS
54.2
0.11 + 0.01
DAGMA
68.4 10.6
0.13
0.89 0.01
500
ALIAS
9.1
0.03 - 0.01
0.99 0.00
50
DAGMA
71.8 + 12.6
0.13 + 0.03
0.89 + 0.01
1000
ALIAS
5.2 3.3
0.01 + 0.01
1.0 0.0
DAGMA
6.8
0.12 = 0.02
0.9 0.02
2000
ALIAS
5.4 4.4
0.02 + 0.01
1.0 0.0
DAGMA
75.0 + 25.1
0.14 + 0.05
0.89 + 0.04
5000
ALIAS
8.8 + 5.2
0.02 - 0.01
0.99 + 0.00
D.3
Ablation Studies
D.3.1
Effect of RL method and learning rate
In Table 16 we conduct ablation studies on the effect of the choices of RL method and learning rate on the performance of our ALIAS method with the BIC score.
29

---

Table 16: Performance sensitivity of ALIAS with BIC score subjected to the variations of learning rate. We employ Ganas-Gn datasets with 30 nodes ER-8 graphs and use CORL (Wang et al. 2021 and DAGMA (Bello et al. 2022 reference. For each row, compute the means and standard errors over 5 independent datasets. Bold: better performance than the baselines. Unless otherwise indicated, the
remaining hyper-parameters are used according to Table 7 in the Appendix.
Method RL method
Learning rate
SHD (4)
FDR (1)
TPR (t)
CORL DAGMA
82.4 +
22.3
0.23 + 0.05
0.87 = 0.04
67.6
8.0
0.14 0.02
0.82 0.02
0.00001
79.4 +
6.7
0.22 + 0.02
0.86 + 0.01
0.00005
12.6 +
3.1
0.04 0.01
0.98 + 0.00
0.0001
1.2
0.4
0.00
0.00
1.00 0.00
0.0005
2.2
1.3
0 0.00
1.00
A2C
0.001
22.8
3.9
0.07
0.005
123.8
49.7
0.24 0.16
0.01
223.4
5.6
0.48
0.13 0.04
0.05
232.6
3.0
0.59 0.03
0.11 = 0.01
ALIAS (Ours)
0.00001
77.8
7.5
0.21 + 0.02
0.86 0.01
0.00005
9.0 +
2.9
0.03 + 0.01
0.99 0.00
0.0001
1.2
0.4
0.00
1.00 1.00
0.0005
0.4
0.3
1.00 0.00
PPO
0.001
4.0
2.0
0.01
0.99 0.00
0.005
204.8
38.8
0.65
0.21 0
0.01
225.4
6.8
0
0.11 0.04
0.05
231.4
5.1
+.00
D.3.2 Effect of Entropy regularization
In Table 17 we conduct ablation studies on the effect of the choices of entropy regularization on the performance of our ALIAS method with the BIC score.
30

---

Table 17: Performance sensitivity of ALIAS with BIC score subjected to the variations of Entropy regular- ization weight. We employ lincar-Gaussian datasets with 30 nodes ER-8 graphs and use CORL (Wang
et al. 2021 and DAGMA (Bello et al. 2022) reference. Our scoring function is BIC. For each row, compute the means and standard errors over 5 independent datasets. Bold: better performance than the
baselines. Unless otherwise indicated, the remaining hyper-parameters are used according to Table 7
Method RL. method
Learning rate Entropy Coef.
SHD (L)
FDR (4)
TPR (1)
CORL DAGMA
82.4 + 22.3
0.23 + 0.05
0.87 = 0.04
67.6 8.0
0.14
0.82 = 0.02
0
1.20 + 0.4
0.00 0.00
1.00 0.00
0.001
1.2 - 0.4
0.00
1.00 0.00
0.0001
0.01
1.6 0.5
0.01
1.00
0.1
28.8
0.09
0.90
1
127.0 : 13.1
0
0.78 0.03 0.03
A2C
0
1.3
0.01 + 0.00
0.00
1.00 1.00 0.00
0.001
1.9
0.01 + 0.00
0.99 0.00
0.0005
0.01
0.2
0.0
1.00
0.1
4
0.04
0.93
ALIAS (Ours)
1
8.5
0.40
0.51 = 0.02 0.02
0
1.4
0.00 0.00
1.00 + 0.00
0.001
0.3
0.00
1.00 0.00
0.0001
0.01
0.5
0.01
+.00
0.1
2..
0.09
0.97 0.00
130.2 + 12.1
0.02
0.77 0.03
PPO
0.00 + 0.00
1.00 0.00
0.001
0.00 0.2
1.00
0.0005
0.01
1.3
0.00
1.00
0.1
51.8 11.8
0.15
0.92
9.5
0.38 0.02
0.58 + 0.02 0.02
D.3.3 Effect of sparsity regularization
In Table 18 we conduct ablation studies on the effect of the choices of sparsity regularization strength of our ALIAS method with the LS score.
31

---

Table 18: Performance sensitivity of ALIAS with LS score subjected the variations of Sparsity regularization weight for the LS score. We employ Sinear-Gaussee datasets with 30 nodes ER-8 graphs and use CORL Wang et al.j 2021 and DAGMA [ello et al., 2022 reference. For each row, compute the means and standard errors over 5 independent datasets. Bold: better performance than the baselines. Unless otherwise
indicated, the remaining hyper-parane parameeareeters are used according to Table 7
Method
RL method
Sparsity regularizer A
SHD (4)
FDR (4)
TPR (t)
CORL DAGMA
82.4 + 22.3
0.23 + 0.05
0.87 0.04
67.6 + 8.0
0.14 0.02
0.82 + 0.02
0
8.4 4.2
0.02 + 0.01
0.98 = 0.01
0.000001
2.8 + 1.0
0.01 + 0.00
0.99 + 0.00
A2C
0.0001
50.0 = 27.7
0.07 1.0 0.04
0.83 = 0.10
0.01
189.0 + 14.2
0.34
0.28 1 0.07
1
226.4 4.7
0.50 0.03
0.08 0.02
ALIAS (Ours)
0
2.0 0.5
0.01 + 0.00
1.00 = 0.00
0.000001
1.2 0.6
0.00 +
1.00 + 0.00
PPO
0.0001
42.0 + 26.4
0.04
0.84 - 0.10
0.01
178.8 + 19.4
0.29 + 0.05
0.31 + 0.08
1
221.6 6.3
0.42 0.04
0.09 = 0.02
D.4 Model Misspecification Results
In Table 19 we study causal model misspecification on nonlinear data. We assume data generated via additive noise models with Gaussian process similarly Zhu et al. (2020); Wang et al. (2021) Yang et 2023a), , but test with datasets generated by the Post-nonlinear Gaussian Process model X,
J et al. 2020
Laplace (0, 1)), which is identifiable (Zhang & Hyvärinen 2009) and produced by Lachapelle
Table 19: Causal discovery performance on nonlinear data with PNL-GP model. The data is generated with 10-node ER-4 graphs and post nonlinear Gaussian processes causal mechanisms. The performance metrics are Structural Hamming Distance (SHD), False Detection Rate (FDR), and True Positive Rate (TPR). Lower SHD and FDR values are preferable, while higher values are better for TPR. The numbers mean 1 slandard errors over 5 independent runs. Since the graphs are dense, we also study the effect of pruning the output graphs
No Pruning
CAM Pruning
Method
SHD (1)
FDR (J)
TPR. (1)
SHD (L)
FDR (1)
TPR (t)
NOTEARS
Zheng et al.
2020
29.4 1.1
0.47 0.02
0.34
1.1
0.46 + 0.02
0.34 = 0.03
DAGMA
Bello et al. 2022
27.6
0.066
0.38+0.00
27.4
0.51 + 0.08
0.38 0.06
NOCURL
Yu et al. 2021
3..8111
0.51 0..08
0.00
1.18
0.07
0.15 = 0.03
COSMO
Massidda et al.
2024
2.84
0.00
0.00
2.84
0.41
0.33 0.02
CORL
Wang et al.
2021
0.9
0.002
0.95
4.7
0.08
0.94 = 0.02
RCL--G
Yang al.
20233
8.6
0.19
0.90
7.2
0.13
0.89 0.02
ALIAS (Ours)
6.8113
0.04
0.85
6.8 1.3
0.03 003-0.00
0.84 0.03
In addition, we also study the robustness of our method under noisy data. Specifically, we corrupt the data
by adding Gaussian noises 2o² 0.1) to p% random entries of the design matrix D E Raxd and report the results for the challenging 30-node ER-8 graphs in Table 20 showing that our method still consistently surpasses the baselines, even with increasing noise levels.
32

---

Table 20:
Causal Discovery Performance under Noisy Data. The numbers are mean standard error over random datascts 30-node ER-8 graphs
Method \ p
1%
3%
5%
10%
NOTEARS
181,4 4.0
189.0 2.5
190.8 + 2.8
194.4 3.9
DAGMA
67.0 + 8.5
75.0 + 9.7
79.0 6.5
94.0 6.1
NOCURL
142.2
147.0 4.8
146.0
153.0
COSMO
9.1
112.8
111.0 8.5
136.2 14.3
ALIAS (Ours)
8.4 + 0.8
27.2 + 3.0
41.4 + 5.0
66.0 + 4.0
Moreover, we investigate our method's performance under the dependence of noises, which is equivalent to the existence of hidden confounders. Specifically, we create datasets with hidden variables by generating datasets with & additional variables then remove them. We present the results (for 30-node ER-S graphs) on such datasets in Table 21 where our method also outperforms all baselines, even with more hidden confounders.
Table 21: Causal Discovery Performance under Hidden Confounders. The numbers are mean I standard error over random datasots on 30-node ER-8 graphs.
Method \
COSMO
93.0 16.2
133.6 + 11.2
132.8 + 7.1
136.4 + 8.5
DAGMA
53.0 = 11.9
77.6
+ 9.3
96.0 7.6
124.0 + 5.3
NOCURL
1.7
159.8 + 8.4
1
158.4 + 5.3
NOTEARS
185.2 10.4
172.6 + 5.7
4.0
188.8 4.4
ALIAS (Ours)
29.4 + 6.2
58.2 + 4.9
82.2 + 6.1
106.6 + 2.5
D.5 Results on nonlinear data with MLPs
In this section, study the performance of our ALIAS method nonlinear causal models with neural
networks, which are popular among gradient-based methods (Zheng al. 2018 Bello el al. 2022 Massidda
et a.. 2024). Specifically, similar to the GP data, we consider 10-node ER-4 graphs with multiple-layer perceptron (MLP) causal mechanisms with standard Gaussian noise, as used in NOTEARS and DAGMA.
These MLPs have one hidden layer with 100 units and sigmoid activation, and we generate 1,000 samples per dataset.
We employ the ALIAS variant that GP regressor with the exact configuration in the GP experiments (Section 6.3), but now using the equal variance variant of the BIC score. This is compared against gradient- based baselines that support MLPs as is. Those include NOTEARS, DAGMA, and COSMO, all of which model nonlinearity using MLPs with the same configuration, namely one hidden layer of 10 units with sigmoid activation.
The results are reported in Table 22 showing that despite the potential disadvantage of using GP regression for MLP-based SEMs, our method still achieves the lowest SHD of near zero compared all three baselines, which are specifically designed to model MLP data.
33

---

Table 22: Causal discovery performance on nonlinear data with MLP model. The data is generated with 10-node ER-4 graphs and MLP model with one hidden layer of 100 units and sigmoid activation as causal mechanisms. The performance metrics are Structural Hamming Distance (SHD), False Detection Rate (FDR), and True Positive Rate (TPR). Lower SHD and FDR values are preferable, while higher values are better for TPR. The numbers are mean + standard errors over 5 independent runs. Since the graphs are dense, we also study the effect of pruning the output graphs.
No Pruning
CAM Pruning
Method
SHD (1)
FDR (J)
TPR. (1)
SHD (L)
FDR (1)
TPR (t)
NOTEARS
Zheng et al.
2020
11.0 + 1.2
0.21 0.01
0.93 + 0.04
10.0 1.9
0.11 = 0.03
0.80 = 0.04
DAGMA
(Blll et al.
122
7.6 1.7
0.09 0.03
0.84 +84+0.05
8.8
0.07 + 0.03
0.78 + 0.04
COSMO
Massidda et al.
2024
1.0
0.01
0.87 + 0.03
9.8
0.09 0.02
0.79 + 0.02
ALIAS (Ours)
0.8 0.4
0.01
0.99 - 0.01
3.4
0.00
0.89 = 0.01
D.6 Results on regular weight range 14 (1-2, 00.5] U [0.5. 2))
In addition to our experiments on Gaussian data with the large weight range 4 ( in Figure 2 here we also consider the regular range & (1-2, 0.5) U [0.5. for the linear weights. In Figure G we present the causal discovery results for both weight ranges under varying graph sizes. It can be seen that larger weights pose significant challenges to several methods, including NOTEARS, NOCURL, RL-BIC, RCL-OG, and CORL. Meanwhile, our method consistently identifies true DAG in all cases for both weight ranges, signifying its robustness to the data variance.
5-node ER-2 graphs
10-nodc ER-2 graphs
20-node ER-2 graphs
30-node ER-2 graphs
40
SHD (
10
0.4
FDR (1)
0.4
0.50
0.2
Weight range
[o.5,
0.0
0.0
0.0
0.00
12, 5J
1.0
TPR (1)
0.5
0.5
0.5
NOTBAOS
D4004
caar
at atc
RC3-00
cut
ALAS (oun)
NOTEAS
DA025
a0000
BLaic
RC--20
cau
ALUS (Dwn)
NOTEAES
NOZUL
ROL-20
AUAS [oun)
0.0
NOTBARS
b4644
NOcUEL
atatc
coRl
ALLAS (90m)
Method
Method
Method
Method
Figure 6: Causal Discovery Performance on linear-Gaussial data with small (u (1-2, 0.5] U (0.5, and large (U (1-5, -2] J [2, 5))) weight ranges. The error bars depict standard errors over 5 independent runs.
D.7 Comparison with Constraint- methods
In addition to the score-based basclines considered so far, in this scction, we also investigate the comparative performance of our ALIAS method against constraint-based methods, which are also prominent approach in causal discovery. To be more specific, consider the classical PC method (Spirtes et al. 2000), as well as more recent advances MARVEL (Mokhtarian et al. 2021b and RSL (Mokhtarian et al. 2022 In this case, we emply gCastle's implementation for PC and the official implementations provided at https: http:r/cccccaaaae..e/oo com/ com/ for RSL and MARVEL.
We evaluate these methods on lincar-Gaussian datasets with varying scales and densitics, and the sample size is fixed to 1,000. The basclines were configured according to their recommended settings, including the Fisher's test with significance levels of 0.05 for PC and zor for RSL and MARVEL, where d is the number
34

---

of nodes. Since constraint-based methods may not orient all edges, for fair evaluation, we compare the undirected skeletons of the estimated and true graphs. The evaluation metrics included SHD, precision, recall, and Fl scores. The results presented in Table 23 reveal that our method demonstrates significantly higher accuracy than constraint- based approaches in recovering the skeleton, achieving near-zero skeleton SHD across all scenarios. In contrast, the baseline methods face considerable challenges, particularly when applied large and dense graphs.
Table 23: Causal Discovery Performance in Comparison with Constraint-basee Methods. We compare the proposed method ALIAS with PC (Spirtes et al. 2000), MARVEL (Mokhtarian et al. 2021 and RSL Mokhtarran et al. 2022) on linear-Gaussia datascts with the regular weight range U (1112, - 0.5 U [0.5, 2)). The performance metrics are SHD (lower is better), Precision (higher is better), Recall (higher is better), and Fi score (higher is better) between the skeletons of the estimated and true graphs. The numbers are mean = standard errors over 5 independent runs.
Data
Method
Skeleton SHD (4)
Skeleton Precision (1)
Skeleton Recall (1)
Skeleton Fj (f)
PC (pirtes al.
2008
25.8 2 2.8
0,88 = 0.08
0.61 +: 0.03
0.74 0.13
MARVEL Klokifarian et
2021
15.6 2.1
0,93 0.02
0.79 1. 0.03
0.85 1 0.02
Sparse graphs (30-node ER-2)
RSL
Mokitaraa
13.8
0.86 2:
0.88
ALIAS (Ours)
0.0 0.0
1,0
1.0 0.0
1.0 0.0
PC Spirtes et al. 2000
216.0 3.6
0.63 = 0.02
0.13 0.01
0,21 = 0.02
MARVEL Mochtarian
221.8 3.8
0.03
0.1
Deuse graphs (30-node Elt-8)
RSL Mokitarin
185.4 5.6
0.66 0.02
0.39 1 0.01
0.49 -1 0.01
ALIAS (Ours)
0.4 0.2
1.0 = 0.0
1.0 1:0:00 0.0
1.0 0.0
PC Spirtes et sl.
2000
216.6 10.5
0.87 = 0.01
0.37 = 0,02
0.69 = 0.01
Large graphi (200-node ER-2)
MARVEL Mokiaa
167.2 8.4
0,91
0.67 0.01
0.77
RSL
Mokitaann Gl
167.2 4. 8.8
0.87 1: 0.01
0.7 :: 0.01
0.78 à 0.01
ALIAS (Ours)
0.8 1. 0.6
1.0 0.0
1.0 .1 0.0
1.0 0.0
D.8 On Varsortability
Recently, it was suggested by Reisach et al. 22021 that marginal variances share high dependencies with the topological ordering of the true DAG for simple additive noise models. As such, sorting the variables according to the variances may be sufficient to recover the DAG, voiding the need for more sophisticated mothods. More particularly, the metric Varsortability E [0, 1] is proposed to assess the correlation between the ordering of marginal data variances and the causal ordering the true DAG. Essentially, Varsortability calculates the portion of directed paths in the ground-truth DAG that have increasing marginal variances, and thus, if it is close to 1 then there is a high chance that simply sorting the variances would reveal the true DAG, and vice versa. It was shown that simple additive noise data commonly employed in existing studies, starting with Zheng et al. (2018), exhibit high values of Varsortability.
However, we show here that this is not entirely the case for our study. Particularly, while the majority of existing studies employ sparse graphs for evaluation, where the densest graphs have an expected in-degree of about 4 at most (Zheng et al.] 2018 Yu et al.] 2021 Bello et al.] 2022. Massidda study, we also consider much denser graphs with an expected in-degree of up to 8. This distinction is of high importance because we find that Varsortability decreases rapidly with the graph density and reaches near-zero at ER-8 graphs (Table 24 indicating that our data settings is much more nontrivial, and cannot be resolved simply by sorting the variances. Furthermore, since Varsortability is close to 0 in our data, one may expect that simply sorting the nodes by decreasing variances may work. In Table 25 we show that this is not the case, where the sortnregress algorithm proposed in conjunction to Varsortability (Reisach et al,, 2021)-which which first sorts the nodes by marginal variances and then perform linear feature selection convert. the ordering to a PAG-cannot recover the true DAG regardless of the sorting direction. Nevertheless, under this intricate scenarios, our method still robustly recovers the true DAG with near-zero SHD.
al. 2024). in this
35

---

Table 24: Varsortability as function of graph density. We compute Varsortability (Reisach et al.
2021 for Anear-Gaussian data on 30-node ER graphs (Section. 6.2 with varying expected in-degrees. The
numbers are mean + slandard ervors over 100 simulations.
Expected in-degree
Varsortability
0.97 0.01
2
0.87 + 0.03
4
0.35 = 0.06
0.05 + 0.01
0.01 + 0.00
Table 25: Comparison with sortnregress on dense data. We consider lincar-Gaussian datasets on
dense 30-node ER-8 graphs. The numberr are mean slandard errors over 5 simulations.
Method
SHD (4)
FDR (4)
TPR (t)
sortnregress (decreasing variances)
360.2 + 2.2
0.97+-0.01
0.04+00.01
sortnregress (inereasing variances)
88.2 17.7
0.27+0.05
0.94+0.01
ALIAS (ours)
0.2 + 0.2
0.00 + 0.00
1.00
D.9 Effect of Data Standardization
Given the issues raised by Reisach et al. 2021 as discussed in the previous section, it is interesting to study the performance of causal discovery methods under data standardization, where the variance ordering information completely removed.
Gaassin data. For linear- Gaussian data, the causal graph is identifiable under the equal-noise variance assumption (Peters et al. 2014). Hence, standardizing our linear-Gaussaal data node-wise would render the noise variances unequal and reduce the causal model to a general lincar- Gaussian model, where the causal graph is known to be unrecoverable (Spirtes et al. 2000). As such, it is expected that causal discovery performance on standardized linear data is worse than its non-standardized counterparts. However, in Table. 26 we show that our ALIAS method can still significantly outperform the basclines under this setting with the lowest SHD, which is only half of that of the second-best method, highlighting ALIAS's robustness against data standardization.
Nonlinear data. On the contrary with linear causal models, nonlinear additive noise models generally identifiable regardless of unequal noise variances (Hoyer et al.) 2008), meaning standardizing data should have minimal influence on the causal discovery performance. Indeed, in Table. 26 we show that our ALIAS method still performs equally well when the data is standardized compared to when the data not standardized (Table. 4 with very low SHD of only around 1, which much lower relative to other basclines.
Table 26: Causal Discovery performance on standardized data. We consider Khner-Gaassen data and nonlinear data with Gaussian processes. Observed data standardized dimension-wise to remove variance information. For ALIAS, use the non-equal variances BIC version to reflect the data condition. The numbers are mean 1 standard errors over 5 independent runs. Bold: best performance, underline: second-best, performance.
Lincar data (10-node ER-2 graphs)
Nonlinear data (10-node ER-4 graphs)
Method
SHD (L)
FDR (L)
TPR (1)
SHD ())
FDR (1)
TPR (1)
NOTEARS
Zheng et al.
2020
10.2 + 2.0
0.39 + 0.04
0.84 + 0.02
7.8
0.14 + 0.07
0.91 + 0.01
DAGMA
Rello al.
212
17.4 2.8
0.70 + 0.08
0.39 + 0.07
29.8
0.4+0.16
0.28 + +.15
COSMO
Massidda et
2024
16.4 1.9
0.74 + 0.06
0.29 1 0.06
3..8
0.5
0.27 0.12
ALIAS (ours)
6.0 1.8
0.30 + 0.07
0.79 + 0.05
1.4
0.0
0.97 + 0.03
36


Reviews:
[{"review_id": "n8F9UQZTeM", "reviewer": "Reviewer_y43i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. Use of Vec2DAG and generating DAGs in one step is efficient and interesting.\n2. The paper offers a theoretical proof of the surjectivity of Vec2DAG and scaling/translation invariance.\n3. Extensive experiments on small to large graphs that show that the proposed ALIAS method outperforms other score-based causal discovery methods.\n\n### Weaknesses & questions\n1. No comparisons are made with constraint-based causal discovery methods.\n2. There are no theoretical guarantees on whether the returned DAG from ALIAS is in the MEC.\n3. The ALIAS method is slow, especially on larger graphs, taking more than 1.5 hours to process one graph of 200 nodes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces ALIAS, an RL-based framework for causal discovery (CD) that generates DAGs in a single step without acyclicity constraints by mapping a continuous vector space onto the space of all DAGs, using Vec2DAG. The authors prove surjectivity of their parameterization, integrate it with policy-gradient algorithms, and empirically demonstrate superior performance over previous CD methods on both synthetic and real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be great if the authors could compare their method with constraint-based methods in causal discovery, like the most often-used one, PC (Spirtes et al. (2000)), or the more recent recursive causal discovery methods, such as [RSL or MARVEL (Mokhtarian et al. (2024))](https://rcdpackage.com/).\n2. Could the authors prove that the DAG returned from ALIAS is in the MEC? Or, perhaps that the DAG returned is precisely the underlying DAG given some restrictions on the FCM, such as it being a linear Guassian model.\n\n### Minor comments\n1. The paper title in the PDF on the first page is \"Reinforced DAG Learning without Acyclicity Constraints Journal Submissions.\" I am guessing that the \"Journal Submissions\" part is not a part of the title and is a typo.\n2. The first paragraph of the second page starting with \"from Zheng et al. (2018) into the reward...\" is missing punctuations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ed3rFBVV13", "reviewer": "Reviewer_YrDq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n`Vec2DAG` is well-motivated, as is ALIAS. Moreover, the authors provide substantial empirical evidence that ALIAS works well, with impressive results in a variety of settings. While this method is more expensive than some continuous optimization-based methods, it is considerably faster than existing RL-based methods (and, of course, combinatorial search space methods).\n\n## Weaknesses\n\nMost of my concerns have to do with clarity and some parts of the experimental setup. I anticipate that these are all relatively minor.\n\n**Runtime** On large graphs, the proposed method has a runtime more than 10 times longer than many other methods, and is over 100 times slower than DAGMA, in particular. This is especially a drawback in the linear case, where methods such as NOTEARS or DAGMA simplify considerably and run fast on CPU, whereas the implemented versions of ALIAS still require optimizing an MLP for the policy.\n\n**Details Lacking in Presentation** There are some omitted details that I think should appear in the manuscript. For example, the definition of $E(z)$ and $p(z)$ are not provided; I had to look in the code to realize that $p(z)$ is just the first $d$ components of $z$, and $E(z)$ is an (triangular) unvec of the remaining components. This should be explained more clearly in the text.\n\nLikewise, there are no details provided surrounding the GP regression in the nonlinear case: what kernels are used? Are approximate GPs used to deal with the poor scaling with respect to $\\\\lvert \\\\mathcal{D} \\\\rvert$?\n\n**Originality of `Vec2DAG`** I find the originality of `Vec2DAG` to be overstated, for example in the introduction where NoCurl is not mentioned at all. Introducing the edge potential and computing $E(z) + E(z)^\\\\top$ is the straightforward way to parameterize a symmetric matrix without diagonal elements (which is admissable by Remark 3.6 of Yu et al. (2021)), and the remaining novelty is adding the Heavyside function, which is also a straightforward way to obtain binary adjacency matrices. Likewise, the proofs are essentially the same as in NoCurl. Originality is not a big concern for acceptance here, but I think the inspiration of `Vec2DAG` should be stated earlier and more prominently.\n\n**Lack of Clarity in Some Proofs** I found the proofs sound, but think their clarity can be considerably improved. For example, consider Lemma 4: $p_{i}$ and $E_{ij}$ are given, but it is not explicitly computed that the hypothesized DAG is recovered. The actual computations provide some clarity into why this works, though: taking E as constructed provides the *undirected* version of the causal graph, minus the diagonal. And since \n$\\\\text{grad}(p)\\_{ij} = p\\_{j} - p\\_{i}$, it is clear that $\\\\text{grad}(p)_{ij} > 0$ if and only if $p_j > p_i$, which fits into the ancestry condition mentioned. Thus, in this construction, $E$ constructs the undirected causal graph, and $p$ encodes the directionality. Then, in the Hadamard product, we mask the undirected causal graph by the causal directions. Note that this is quite similar to the interpretation given in the corresponding proof of Yu et al. (2021). Similarly, the conditions on $\\\\beta$ in Lemma 1/5 are not straightforward to interpret, but important for understanding the lemma's actual implications.\n\n## Questions\n\n**Regarding Synthetic Data** In several places, it is claimed that \"to make this setting more challenging\" a wider range of edge weights are used, i.e., generating from $\\\\mathcal{U}([-5, -2] \\\\cup [2, 5])$ instead of $\\\\mathcal{U}([-2, -0.5] \\\\cup [0.5, 2.0])$. I'm concerned that this may inadvertently cause problems with other methods by fundamentally changing the $l_1$ penalization, as well as the optimal learning rate. Could the authors also provide some experiments using the default hyperparameters *and* the standard generating process?\n\n**On Training of Sparsity Regularizers** Likewise, the standard in the literature is to leave a reasonable but untuned sparsity regularization term. Meanwhile, Table 18 shows that the proposed method is extremely sensitive to the $l_0$ penalty term, with the optimal value used in all experiments. What would be the corresponding effect of tuning sparsity penalties in other methods?\n\n**On Runtime Setting** For Section 5.5/Figure 4, what random datasets are used (i.e., linear or nonlinear)? If non-linear, what are the effects of using larger MLPs in DAGMA/NOTEARS, so that all runtimes are similar?\n\n**On GP-Based Nonlinear SEMs** One could argue that using a GP-based generative model in the synthetic data is unfair to DAGMA/NOTEARS/etc., which use MLPs as interpolators. Indeed, in the NOTEARS+ paper (Zheng et al., 2020), NOTEARS seems to be stronger on MLP-based SEMs than GP-based SEMs when compared to other methods (c.f. Figure 1, or Table 2 of Zheng et al., 2020). This is somewhat addressed in Appendix E.4, but it would be better if models were also tested on the \"MLP\" SEMs used in NOTEARS/DAGMA (note that these may, on the other hand, be somewhat favorable to NOTEARS/DAGMA from an identifiability standpoint, c.f. Section VI.C of Waxman et al., 2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces ALIAS, which uses a NoCurl-type parameterization of DAGs in a reinforcement learning approach to causal discovery with observational data. This entails modifying the NoCurl parameterization of DAGs to obtain `Vec2DAG`, which essentially discretizes the NoCurl parameterization to binary adjacency matrices using the Heavyside function. A Bayesian information criterion-based score is then introduced as the reward for use with policy gradient-based methods such as A2C or PPO. Finally, extensive empirical evidence is presented showing the effectiveness of ALIAS using PPO and A2C in linear and nonlinear SEMs, including several ablation studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical Changes\n\n**Regarding Algorithm 1**  Algorithm 1 only presents the vanilla policy gradient. This can be misleading, since VPG is not the method typically used or advocated for (e.g., only A2C and PPO appear in the experiments). Please revise accordingly, for example retitling as \"ALIAS with Vanilla Policy Gradient\", or editing line 4 to say \"Update policy, for example with vanilla policy gradient: [...]\". \n\n**Standard Error vs. Standard Deviation** In the figures, it is claimed that standard error is shown, but in the tables, it is claimed that the standard deviation is shown. But these seem to be quoting the same value; for example, the quoted standard error of DAGMA in Figure 3 with 1000 samples and 30 nodes seems to be ~14, which is closer to the quoted standard deviation of 17.8 than the corresponding standard error of ~8. This would also be more consistent with the existing literature. Please double-check that the correct value is quoted.\n\n**Use `\\citep` Where Appropriate** There are several places that `\\citep` should be used, including: at the top of page 2 for (Zhu et al., 2020) and (Wang et al., 2021); at the beginning of section 2.2 for (Scwarz, 1978; ...); at the beginning of section 3.3 for (Zheng et al., 2018; ...). There might be some others, and I suspect this is because of calls to `\\cite` rather than `\\citet` directly, which should be searchable.\n\n**Title** The title has \"Journal Submissions\" at the end. Please remove this.\n\n**Experiments** Please include additional experimental evidence, as mentioned in Weaknesses.\n\n## Minor Changes\n\n**Regarding Related Work** I found the overview of related works in Sections 1 and 2 somewhat terse; in contrast, I found Appendix A well-written and helpful, especially in understanding the RL-based methods that I'm less familiar with. I think it would help the readability of the manuscript if Appendix A were to appear in the main text, especially since TMLR does not have page limits. This is ultimately subjective, though, and merely a suggestion.\n\nSection 2.2 says that other methods use $l_0$ regularization, but typically actually use $l_1$ regularization.\n\n# References\n\nYu, Y., Gao, T., Yin, N., & Ji, Q. (2021). DAGs with No Curl: An efficient DAG structure learning approach. In International Conference on Machine Learning (pp. 12156-12166). PMLR.\n\nWaxman, D., Butler, K., & Djurić, P. M. (2024). Dagma-DCE: Interpretable, Non-Parametric Differentiable Causal Discovery. IEEE Open Journal of Signal Processing.\n\nZheng, X., Dan, C., Aragam, B., Ravikumar, P., & Xing, E. (2020). Learning sparse nonparametric dags. In International Conference on Artificial Intelligence and Statistics (pp. 3414-3425). PMLR.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vSsbXJtfrw", "reviewer": "Reviewer_w4vc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The contribution is quite solid. It seems this simple, yet novel mapping from vectors to DAGs (Vec2DAG) has strong merits over prior approaches, which is demonstrated both theoretically and experimentally.\n\n2. Overall, the writing is very good and the text is easy to follow. I also think the figures are very pretty and clean.\n\n3. Extensive experiments on both synthetic and real-world data are presented, with plenty of ablations. The authors did an amazing job here.\n   \n**Weaknesses**\n\n1. No source code for reproducing the experiments seems to be available. It would be great if the authors could supply this for maximum reproducibility and to make the lifes of other authors easier in the future.\n\n2. I am not totally convinced that it is justified to associate ALIAS with reinforcement learning. The policy takes only a single step and there are no states, transition dynamics or stochasticity. To my understanding, the latter three points are what fundamentally characterizes reinforcement learning. I see that the authors compare their method to first-order optimization of a differentiable version of Vec2DAG. However, I would still interpret the proposed ALIAS method as some kind of zero-order optimization rather than reinforcement learning. This point should certainly be discussed in greater detail (for instance, having a discussion section for this would be nice).\n\n3. The related work section belongs in the main text. There is no page limit for TMLR, so I do not see a sound reason to put this section in the appendix.\n\nApart from that, I would be glad if the authors could take into account my requested changes for the revised manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a novel parameterization called *Vec2DAG* of a directed acyclic graph (DAG) in terms of a continuous, real-valued vector. The proposed parameterization alleviates the requirement of laboriously maintaining the acyclicity constraint as it is satisfied by construction. The authors prove that Vec2DAG is theoretically sound in that any DAG can be represented with the given parameterization. In addition, the authors propose *ALIAS*, a score-based causal discovery algorithm based on reinforcement learning that optimizes over the continuous vector to estimate the Markov equivalence class of the underlying causal graph, using only observational data. Plenty of experiments and ablation studies on synthetic and real-world data demonstrate the merits of ALIAS in comparison to prior work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I will proceed to list requested changes one-by-one. Points that are critical for meeting the acceptance criteria are highlighted by *(critical)*.\n\n**Title**\n\n- *Reinforced DAG Learning without Acyclicity Constraints Journal Submissions* In my opionion, the title should be modified. Why does it say *Journal Submissions*? Also I am not sure if *reinforced DAG learning* makes sense (I could be wrong). Why not just choose a title like *Reinforcement Learning for Causal Discovery without Acyclicity Constraints*?\n\n**Introduction**\n\n- *The knowledge of causal relationships required to understand the nature in many scientific sectors,* This sentence does not seem gramatically correct. Perhaps the authors mean something like *Obtaining insights into causal relationships has traditionally required deep domain knowledge*. Please clarify.\n\n- *, especially in intricate situations where randomized experiments are impractical, has motivated the development of causal discovery methods that aim to infer cause-effect relationships from purely passive data over the last decade* This should be an individual sentence. The gist is roughly intelligible, but the grammar and sentence structure is quite poor. I kindly ask the authors to rewrite this sentence.\n\n- *and the intricate acyclicity constraint* Is that really true? I believe that removing the acyclicity constraint makes the problem even harder, because even more graphs are considered.\n\n- *brief overview of the literature in Appendix A* Given that there is no page limit, I do not understand why the literature review is in the appendix. I would kindly ask the authors to put it into the main text or clarify why this decision was made.\n\n**Vec2DAG: Unconstrained Parametrization of DAGs**\n\n- Equation 3: It would be great if the authors could go a bit more into the intuition for Vec2DAG. To me, it seems that having both the node potential and the edge potential is redundant. My confusion may have to do with the fact that I do not quite understand where these potentials even come from. Are they learned? It would be great if the authors could clarify this with a few words.\n\n**3.2 Properties of Vec2DAG**\n\n- I do not understand how Lemma 1 is about *scaling and translation invariance*. To me, the statement of the lemma rather reads as some kind of continuity statement: Two sufficiently close vectors w.r.t. to their potentials yield the same DAG. In this light, I also do not understand the remark *representations of different DAGs are close to each other*. Please clarify.\n\n- Lemma 2 does make sense to me with the given explanation, but it seems to show something very different to Lemma 1. It would be nice to have some more explanation for how these two lemmas relate to each other.\n\n**4.1 Policy Gradient for DAG Search**\n\n- (critical) I do not understand how it is justified to bring reinforcement learning into the game here (see weakness 2). Without an explanation, I am not convinced of the soundness of the proposed method. The authors need to explain why the same results could not be achieved via straight-forward optimization.\n\n- (critical) ALIAS seems to optimize only over $\\mathbf{z}$. I did not understand how the edge and node potentials are parameterized by $\\mathbf{z}$ (I.e., what are $\\mathbf{E}(\\mathbf{z})$ and $\\mathbf{p}(\\mathbf{z})$?). This seems to be highly relevant to the method, so it would be important to discuss this.\n\n**4.2 Post Processing**\n\n- *With limited sample sizes, due to overfitting, redundant edges may still be present in the returned DAG whose highest score* This sentence seems gramatically wrong and makes no sense to me. I guess the authors mean something like *With limited sample sizes, due to overfitting, redundant edges may still be present in the returned DAG that achieves the highest score*. Please clarify.\n\n**5.6 Ablation Studies**\n\n- *Furthermore, to show that the effectiveness of ALIAS is not only thanks to the Vec2DAG parametrization alone, but also the application of RL, we replace RL in our method with a continuous optimizer* This is exactly where I am sceptical. I understand that ALIAS is about optimizing a non-differentiable objective function. However, that does not mean that it is about reinforcement learning. I believe that ALIAS is a zero-order optimization algorithm and the comparison should be with respect to other zero-order optimization algorithms. Furthermore, it is not clear to me what a *continuous optimizer* is. I guess the authors mean *first-order optimizer* or *gradient-based optimizer*.\n\n**Appendix**\n\n- (critical) *These methods can only identify the causal graph up to its Markov equivalent class (MEC), which contains potentially many DAGs inducing the same data* I do not think that ALIAS can generally discover the causal graph beyond Markov equivalence class. Score-based methods, from what I know, generally cannot do that. If ALIAS can actually do that, this would be quite amazing. Therefore, if the authors claim that this is true, it must be proved. Otherwise, the claim must be removed.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers after clarifying various aspects are in agreement for its acceptance. I agree with reviewer assessment mostly.

My principle concern that is outstanding is with respect to experiments. It has been pointed out in the literature that if you use random causal models with additive noise, ordering due to marginal variance of the variables is a valid topological order consistent with that of the true causal graph. So it would be great to actually do experiments where this **variance sortability** is not trivially possible. Please see this reference  https://arxiv.org/abs/2102.13647. The reference cited pointed out this issue with prior gradient based methods and there are various papers following this that examine this issue in more detail.

 I only ask authors to consider an experimental setting where this alternative cannot be trivially used for linear methods at least (following the experimental setting recommendation in that paper). I think variance sortability will also be an issue even in the Gaussian Process setting. So that experiment also may need reconsideration

I would like the authors to clarify the above in the revision.
Decision: Accept (poster)