Title: Distributionally Robust Skeleton Learning of Discrete Bayesian Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: structure learning

Bayesian network

robustness
Abstract: We consider the problem of learning the exact skeleton of general discrete Bayesian networks from potentially corrupted data. Building on distributionally robust optimization and a regression approach, we propose to optimize the most adverse risk over a family of distributions within bounded Wasserstein distance or KL divergence to the empirical distribution. The worst-case risk accounts for the effect of outliers. The proposed approach applies for general categorical random variables without assuming faithfulness, an ordinal relationship or a specific form of conditional distribution. We present efficient algorithms and show the proposed methods are closely related to the standard regularized regression approach. Under mild assumptions, we derive non-asymptotic guarantees for successful structure learning with logarithmic sample complexities for bounded-degree graphs. Numerical study on synthetic and real datasets validates the effectiveness of our method.

Distributionally Robust Skeleton Learning of Discrete Bayesian Networks

Abstract

We consider the problem of learning the exact skeleton of general discrete Bayesian networks from potentially corrupted data. Building on distributionally robust optimization and a regression approach, we propose to optimize the most adverse risk over a family of distributions within bounded Wasserstein distance or KL divergence to the empirical distribution. The worst-case risk accounts for the effect of outliers. The proposed approach applies for general categorical random variables without assuming faithfulness, an ordinal relationship or a specific form of conditional distribution. We present efficient algorithms and show the proposed methods are closely related to the standard regularized regression approach. Under mild assumptions, we derive non-asymptotic guarantees for successful structure learning with logarithmic sample complexities for bounded-degree graphs. Numerical study on synthetic and real datasets validates the effectiveness of our method.

1 Introduction

ABayesian network is a prominent class of probabilistic graphical models that encodes the conditional dependencies among variables with a directed acyclic graph (DAG). It provides a mathematical framework for formally understanding the interaction among variables of interest, together with computationally attractive factorization for modeling multivariate distributions. If we impose causal relationships on the edges between variables, the model becomes a causal Bayesian network that encodes the more informative causation. Without such interpretation, a Bayesian network serves as a dependency graph for factorization of a multivariate distribution. We focus on discrete Bayesian networks with purely categorical random variables that are not ordinal, but will discuss related work on both discrete and continuous Bayesian networks for completeness.

The DAG structure of a Bayesian network is typically unknown in practice [Natori et al., 2017, Kitson et al., 2023]. Structure learning is therefore an important task that infers the structure from data. The score-based approach defines a scoring function that measures the goodness-of-fit of each structure and aims to find an optimal DAG that maximizes the score. Unfortunately, the resulting combinatorial optimization problem is known to be NP-hard [Chickering et al., 2004] without distributional assumptions. Representative approaches include those based on heuristic search [Chickering, 2002], dynamic programming [Silander and Myllymäki, 2006], integer linear programming [Jaakkola et al., 2010] or continuous optimization [Zheng et al., 2018], which either yields an approximate solution or an exact solution in worst-case exponential time. The constraintbased approach [Spirtes and Glymour, 1991, Spirtes et al., 1999, Colombo et al., 2014] performs conditional independence tests to determine the existence and directionality of edges. The time complexity is, however, exponential with the maximum in-degree. Furthermore, the independence test results may be unreliable or inconsistent with the true distribution because of finite samples or

Brian D. Ziebart

even corrupted samples. In general, without interventional data or assumptions on the underlying distribution, we can only identify a Markov equivalence class (MEC) the true DAG belongs to from observational data where DAGs in the MEC are Markov equivalent, that is, encoding the same set of conditional independencies.

A super-structure is an undirected graph that contains the skeleton as a subgraph which removes directionality from the true DAG. It has been shown that a given super-structure possibly reduces the search space or the number of independence tests to be performed. For example, exact structure learning of Bayesian networks may be (fixed-parameter) tractable [Downey and Fellows, 1995] if the super-structure satisfies certain graph-theoretic properties such as bounded tree-width [Korhonen and Parviainen, 2013, Loh and Bühlmann, 2014], bounded maximum degree [Ordyniak and Szeider, 2013] and the feedback edge number [Ganian and Korchemna, 2021]. An incomplete super-structure with missing edges also helps improve the learned DAG with a post-processing hill-climbing method [Tsamardinos et al., 2006, Perrier et al., 2008]. Furthermore, a combination of a skeleton and a variable ordering determines a unique DAG structure. Learning the exact skeleton rather than a rough super-structure is desirable in Bayesian network structure learning.

Spirtes and Glymour [1991], Tsamardinos et al. [2006] make use of independence tests to estimate the skeleton. Loh and Bühlmann [2014] learn a super-structure called moralized graph via graphical lasso [Friedman et al., 2008]. Shojaie and Michailidis [2010] learn the skeleton assuming an ordering of variables. Bank and Honorio [2020] leverage linear regression for skeleton recovery in polynomial time. These methods either rely on independence test results, which are unstable, or a regularized empirical risk minimization problem, where regularization is usually heuristically chosen to combat overfitting. In practice, the observational data is commonly contaminated by sensor failure, transmission error or adversarial perturbation [Lorch et al., 2022, Sankararaman et al., 2022, Kitson et al., 2023]. Sometimes only a small amount of data is available for learning. As a result, the existing algorithms are vulnerable to such distributional uncertainty and may produce false or missing edges in the estimated skeleton.

In this paper, we propose a distributionally robust optimization (DRO) method [Rahimian and Mehrotra, 2019] that solves a node-wise multivariate regression problem [Bank and Honorio, 2020] for skeleton learning of general discrete Bayesian networks to overcome the above limitations. We do not assume any specific form of conditional distributions. We take into account the settings with a small sample size and potential perturbations, which makes the true data generating distribution highly uncertain. Our method explicitly models the uncertainty by constructing an ambiguity set of distributions characterized by certain a priori properties of the true distribution. The optimal parameter is learned by minimizing the worst-case expected loss over all the distributions within the ambiguity set so that it performs uniformly well on all the considered distributions. The ambiguity set is usually defined in such a way that it includes all the distributions close to the empirical distribution in terms of some divergence. With an appropriately chosen divergence measure, the set contains the true distribution with high probability. Hence the worst-case risk can be interpreted as an upper confidence bound of the true risk. The fact that a discrete Bayesian network encompasses an exponential number of states may pose a challenge to solve the DRO problem. We develop efficient algorithms for problems with ambiguity sets defined by Wasserstein distances and Kullback-Leibler (KL) divergences. We show that a group regularized regression method is a special case of our approach. We study statistical guarantees of the proposed estimators such as sample complexities. Experimental results on synthetic and real-world datasets contaminated by various perturbations validate the superior performance of the proposed methods.

1.1 Related Work

Bayesian networks have been widely adopted in a number of applications such as gene regulatory networks [Werhli et al., 2006], medical decision making [Kyrimi et al., 2020] and spam filtering [Manjusha and Kumar, 2010].

In addition to the score-based structure learning methods and constraint-based methods discussed in the introduction section, there are a third class of hybrid algorithms leveraging constraint-based methods to restrict the search space of a score-based method [Tsamardinos et al., 2006, Gasse et al., 2014, Nandy et al., 2018]. There is also a flurry of work on score-based methods based on neural networks and continuous optimization [Zheng et al., 2018, Wei et al., 2020, Ng et al., 2020, Yu et al., 2021, Ng et al., 2022, Gao et al., 2022], motivated by differentiable characterization of acyclicity

without rigorous theoretical guarantees. We refer the interested readers to survey papers [Drton and Maathuis, 2017, Heinze-Deml et al., 2018, Constantinou et al., 2021] for a more thorough introduction of DAG structure learning and causal discovery methods.

Recently, there is an emerging line of work proposing polynomial-time algorithms for DAG learning [Park and Raskutti, 2017, Ghoshal and Honorio, 2017, 2018, Chen et al., 2019, Bank and Honorio, 2020, Gao et al., 2020, Rajendran et al., 2021], among which Bank and Honorio [2020] particularly focuses on general discrete Bayesian networks without resorting to independence tests.

Learning a super-structure can be done by independence tests, graphical lasso or regression, as discussed in introduction. Given a super-structure, how to determine the orientation has been studied by Perrier et al. [2008], Ordyniak and Szeider [2013], Korhonen and Parviainen [2013], Loh and Bühlmann [2014], Ng et al. [2021], Ganian and Korchemna [2021].

DROis a powerful framework emerging from operations research [Delage and Ye, 2010, Blanchet and Murthy, 2019, Shafieezadeh-Abadeh et al., 2019, Duchi and Namkoong, 2019] and has seen effective applications in many graph learning problems such as inverse covariance estimation [Nguyen et al., 2022], graphical lasso learning [Cisneros-Velarde et al., 2020], graph Laplacian learning [Wang et al., 2021], Markov random field (MRF) parameter learning [Fathony et al., 2018], MRF structure learning [Li et al., 2022] and causal inference [Bertsimas et al., 2022].

2 Preliminaries

Weintroduce necessary background and a baseline method for skeleton learning of Bayesian networks.

2.1 Notations

We refer to r n s as the index set t 1 , 2 , . . . , n u . For a vector x P R n , we use x i for its i -th element and x S for the subset of elements indexed by S Ď r n s with ¯ i fi r n szt i u . For a matrix A P R n ˆ m , we use A ij , A i ¨ and A ¨ j to denote its p i, j q -th entry, i -th row and j -th column respectively. A ST represents the submatrix of A with rows restricted to S and columns restricted to T Ď r m s . We define a rowpartitioned block matrix as A fi r A 1 A 2 ¨ ¨ ¨ A k s ⊺ P R ř i n i ˆ m where A i P R n i ˆ m . The ℓ p -norm of a vector x is defined as ∥ x ∥ p : ' p ř i | x i | p q 1 { p with |¨| being the absolute value function. The ℓ p,q norm of a matrix A is defined as ∥ A ∥ p,q : ' p ř j ∥ A ¨ j ∥ q p q 1 { q . When p ' q ' 2 , it becomes the Frobenius norm ∥ ¨ ∥ F . The operator norm is written as ~ A ~ p,q : ' sup ∥ v ∥ p ' 1 ∥ Av ∥ q . The block matrix norm is defined as ∥ A ∥ B,p,q : ' p ř k i ' 1 ∥ A i ∥ q p q 1 { q . The inner product of two matrices is designated by x A , B y fi Tr r A ⊺ B s where A ⊺ is the transpose of A . Denote by b the tensor product operation. With a slight abuse of notation, | S | stands for the cardinality of a set S . We denote by 1 ( 0 ) a vector or matrix of all ones (zeros). Given a distribution P on Ξ , we denote by E P the expectation under P . The least c -Lipschitz constant of a function f : Ξ Ñ R with a metric c : Ξ ˆ Ξ Ñ R is written as lip c p f q : ' inf Λ c p f q where Λ c p f q : ' t λ ą 0 : @ ξ 1 , ξ 2 P Ξ | f p ξ 1 q ´ f p ξ 2 q| ď λc p ξ 1 , ξ 2 qu .

2.2 Bayesian Network Skeleton Learning

Let P be a discrete joint probability distribution on n categorical random variables V : ' t X 1 , X 2 , . . . , X n u . Let G : ' p V , E true q be a DAG with edge set E true . We use X i to represent the i -th random variable or node interchangeably. We call p G , P q a Bayesian network if it satisfies the Markov condition, i.e., each variable X r is independent of any subset of its non-descendants conditioned on its parents Pa r . We denote the children of X r by Ch r , its neighbors by Ne r : ' Pa r Y Ch r and the complement by Co r : ' r n s ´ Ne r ´ t r u . The joint probability distribution can thus be factorized in terms of local conditional distributions:

Let G skel : ' p V , E skel q be the undirected graph that removes directionality from G . Given m samples t x p i q u i m ' 1 drawn i.i.d. from P , the goal of skeleton learning is to estimate G skel from the samples.

We do not assume faithfulness [Spirtes et al., 2000] or any specific parametric form for the conditional distributions. The distribution is faithful to a graph if all (conditional) independencies that hold true

in the distribution are entailed by the graph, which is commonly violated in practice [Uhler et al., 2013, Mabrouk et al., 2014]. The unavailability of a true model entails a substitute model. Bank and Honorio [2020] propose such a model based on encoding schemes and surrogate parameters.

Assume that each variable X r takes values from a finite set C r with cardinality | C r | ą 1 . For an indexing set S Ď r n s , define ρ S : ' ř i P S | C i | ´ 1 and ρ ` S : ' ř i P S | C i | . The maximum cardinality minus one is defined as ρ max : ' max i Pr n s | C i | ´ 1 . Let S r : ' Ť i P Ne r t ρ r i ´ 1 s ` 1 , . . . , ρ r i s u be indices for Ne r in ρ r n s and its complement by S c r : ' r ρ r n s s ´ S r ´ t ρ r r ´ 1 s ` 1 , . . . , ρ r r s u . Let E : C r Ñ B ρ r be an encoding mapping with a bounded and countable set B Ă R . We adopt encoding schemes with B ' t´ 1 , 0 , 1 u such as dummy encoding and unweighted effects encoding 2 which satisfy a linear independence condition. With a little abuse of notation, we reuse E for encoding any X r and denote by E p X S q P B ρ S the concatenation of the encoded vectors t E p X i qu i P S . Consider a linear structural equation model for each X r : E p X r q ' W ˚ ⊺ E p X ¯ r q ` e , where W ˚ fi r W ˚ 1 ¨ ¨ ¨ W ˚ r ´ 1 W ˚ r ` 1 ¨ ¨ ¨ W ˚ n s ⊺ P R ρ ¯ r ˆ ρ r with W ˚ i P R ρ i ˆ ρ r is a surrogate parameter matrix and e P R ρ r is a vector of errors not necessarily independent of other quantities. A natural choice of a fixed W ˚ is the solution to the following problem given knowledge of the true Bayesian network:

Therefore W ˚ ' p W ˚ S r ; 0 q with W ˚ S r ¨ ' E P r E p X ¯ r q S r E p X ¯ r q ⊺ S r s ´ 1 E P r E p X ¯ r q S r E p X r q ⊺ s is the optimal solution by the first-order optimality condition assuming that E P r E p X ¯ r q S r E p X ¯ r q ⊺ S r s is invertible. The expression of W ˚ S r ¨ captures the intuitions that neighbor nodes should be highly related to the current node r while the interaction among neighbor nodes should be weak for them to be distinguishable. We further assume that the errors are bounded:

Assumption 1 (Bounded error) . For the error vector, ∥ e ∥ 8 ď σ and ∥ E P r| e |s ∥ 8 ď µ .

Note that the true distribution does not have to follow a linear structural equation model. Equation (1) only serves as a surrogate model to find technical conditions for successful skeleton learning, which will be discussed in a moment.

The surrogate model under the true distribution indicates that ∥ W ˚ i ∥ 2 , 2 ą 0 ùñ X i P Ne r . This suggests a regularized empirical risk minimization (ERM) problem to estimate W ˚ :

where ˜ λ ą 0 is a regularization coefficient, the block ℓ 2 , 1 norm is adopted to induce sparsity and ˜ P m : ' 1 m ř m i ' 1 δ x p i q stands for the empirical distribution with δ x p i q being the Dirac point measure at x p i q . This approach is expected to succeed as long as only neighbor nodes have a non-trivial impact on the current node, namely, ∥ W ˚ i ∥ 2 , 2 ą 0 ðñ X i P Ne r .

Define the risk of some W under a distribution ˜ P as

where ℓ W p¨q is the squared loss function. The Hessian of the empirical risk R ˜ P m p W q is a block diagonal matrix ∇ 2 R ˜ P m p W q fi ˜ H b I ρ r P R ρ r ρ ¯ r ˆ ρ r ρ ¯ r , where ˜ H : ' E ˜ P m r E p X ¯ r q E p X ¯ r q ⊺ s P R ρ ¯ r ˆ ρ ¯ r and I ρ r P R ρ r ˆ ρ r is the identity matrix of dimension ρ r . Similarly under the true distribution, H : ' E P r E p X ¯ r q E p X ¯ r q ⊺ s . As a result, H is independent of the surrogate parameters W ˚ thus conditions on the Hessian translate to conditions on a matrix of cross-moments of encodings, which only depend on the encoding function E and P .

In order for this baseline method to work, we make the following assumptions.

Assumption 2 (Minimum weight) . For each node r , the minimum norm of the true weight matrix W ˚ for neighbor nodes is lower bounded: min i P Ne r ∥ W i ∥ F ě β ą 0 .

Assumption 3 (Positive definiteness of the Hessian) . For each node r , H S r S r ą 0 , or equivalently, Λ min p H S r S r q ě Λ ą 0 where Λ min p¨q denotes the minimum eigenvalue.

Assumption 4 (Mutual incoherence) . For each node r , ∥ H S c r S r H ´ 1 S r S r ∥ B, 1 , 8 ď 1 ´ α for some 0 ă α ď 1 .

Assumption 2 guarantees that the influence of neighbor nodes is significant in terms of a non-zero value bounded away from zero, otherwise they will be indistinguishable from those with zero weight. Assumption 3 ensures that Equation (2) yields a unique solution. Assumption 4 is a widely adopted assumption that controls the impact of non-neighbor nodes on r [Wainwright, 2009, Ravikumar et al., 2010, Daneshmand et al., 2014]. One interpretation is that the rows of H S c r S r should be nearly orthogonal to the rows of H S r S r . Bank and Honorio [2020] show that these assumptions hold for common encoding schemes and finite-sample settings with high probability under mild conditions. They also show that incoherence is more commonly satisfied for the neighbors than the Markov blanket, which justifies the significance of skeleton learning.

Finally, we take the union of all the learned neighbor nodes for each r P r n s by solving Equation (2) to get the estimated skeleton ˜ G : ' p V , ˜ E skel q .

3 Method

As noted in Bank and Honorio [2020], due to model misspecification, even in the infinite sample setting, there is possible discrepancy between the ERM minimizer ˜ W and the true solution W ˚ , resulting in false or missing edges. In the high-dimensional setting ( m ă n ) or the adversarial setting, this issue becomes more serious due to limited knowledge about the data-generating mechanism P .

In this section, we attempt to leverage a DRO framework to incorporate distributional uncertainty into the estimation process. We present efficient algorithms and study the theoretical guarantees of our methods. All technical proofs are deferred to the supplementary materials.

3.1 Basic Formulation

Let X be a measurable space of all states of the Bayesian network p G , P q , i.e., X P X . Let P p X q be the space of all Borel probability measures on X . Denote by X E : ' t E p X q : @ X P X u the space of all the allowed encodings.

Instead of minimizing the empirical risk and relying on regularization, we seek a distributionally robust estimator that optimizes the worst-case risk over an ambiguity set of distributions:

where A Ď P p X q is an ambiguity set typically defined by a nominal probability measure ˜ P equipped with a discrepancy measure div p¨ , ¨q for two distributions A div ε p ˜ P q : ' t Q P P p X q : div p Q , ˜ P q ď ε u , where ε is known as the ambiguity radius or size. This way of uncertainty quantification can be interpreted as an adversary that captures out-of-sample effect by making perturbations on samples within some budget ε . Some common statistical distances satisfy div p Q , P q ' 0 ðñ Q ' P . In this case, if ε is set to zero, Equation (3) reduces to Equation (2) without regularization. We will show that the DRO estimator ˆ W can be found efficiently and encompasses attractive statistical properties with a judicious choice of A .

3.2 Wasserstein DRO

Wasserstein distances or Kantorovich-Rubinstein metric in optimal transport theory can be interpreted as the cost of the optimal transport plan to move the mass from P to Q with unit transport cost c : X ˆ X Ñ R ` . Denote by P p p X q the space of all P P P p X q with finite p -th moments for p ě 1 . Let M p X 2 q be the set of probability measures on the product space X ˆ X . The p -Wasserstein distance between two distributions P , Q P P p p X q is defined as W p p P , Q q : '

We adopt the Wasserstein distance of order p ' 1 as the discrepancy measure, the empirical distribution as the nominal distribution, and cost function c p x , x 1 q ' ∥E p x q ´ E p x 1 q ∥ for some norm

∥ ¨ ∥ . The primal DRO formulation becomes

According to Blanchet and Murthy [2019], the dual problem of Equation (4) can be written as

Strong duality holds according to Theorem 1 in Gao and Kleywegt [2022]. The inner supremum problems can be solved independently for each x p i q . Henceforth, we focus on solving it for some i P r m s :

Equation (6) is a supremum of | X | convex functions of W , thus convex. Since X E is a discrete set consisting of a factorial number of points ( Π i Pr n s ρ i ), unlike the regression problem with continuous random variables in Chen and Paschalidis [2018], we may not simplify Equation (6) into a regularization form by leveraging convex conjugate functions because X E is non-convex and not equal to R ρ r n s . Moreover, since changing the value of x j for some j P ¯ r is equivalent to changing W ⊺ E p x ¯ r q by a vector, unlike Li et al. [2022] where only a set of discrete labels rather than encodings are dealt with, there may not be a greedy algorithm based on sufficient statistics to find the optimal solution to Equation (6). In fact, let the norm be the ℓ 1 norm, we can rewrite Equation (6) by fixing the values of ∥E p x q ´ E p x p i q q ∥ 1 :

If we fix k , Equation (7) is a generalization of the 0-1 quadratic programming problem, which can be transformed into a maximizing quadratic programming (MAXQP) problem. As a result, Equation (6) is an NP-hard problem with proof presented in Proposition 11 in appendix. Charikar and Wirth [2004] develop an algorithm to find an Ω p 1 { log n q solution based on semi-definite programming (SDP) and sampling for the MAXQP problem. Instead of adopting a similar SDP algorithm with quadratic constraints, we propose a random and greedy algorithm to approximate the optimal solution, which is illustrated in Algorithm 1 in appendix, whose per-iteration time complexity is Θ p n 2 mρ max q . It follows a simple idea that for a random node order π , we select a partial optimal solution sequentially from π 1 to π n . We enumerate the possible states of the first node to reduce uncertainty. In practice, we find that this algorithm always finds the exact solution that is NP-hard to find for random data with n ď 12 and ρ max ď 5 in most cases.

Since X E is non-convex and not equal to R ρ r n s , using convex conjugate functions will not yield exact equivalence between Equation (5) and a regularized ERM problem. However, we can draw such a connection by imposing constraints on the dual variables as shown by the following proposition:

Proposition 5 (Regularization Equivalence) . Let : W : ' r W ; ´ I ρ r s ⊺ P R ρ r n s ˆ ρ r with W r ' ´ I ρ r . If γ ě ρ r n s ∥ : W ∥ 2 F , the Wasserstein DRO problem in Equation (5) is equivalent to

which subsumes a linear regression approach regularized by the Frobenius norm as a special case.

This suggests that minimizing a regularized empirical risk may not be enough to achieve distributional robustness. Note that exact equivalence between DRO and regularized ERM in Chen and Paschalidis [2018] requires X E ' R d .

Now we perform non-asymptotic analysis on the proposed DRO estimator ˆ W . First, we would like to show that the solution to the Wasserstein DRO estimator in Equation (4) is unique so that we refer to an estimator unambiguously. Note that Equation (4) is a convex optimization problem but not necessarily strictly convex, and actually never convex in the high-dimensional setting. However, given

a sufficient number of samples, the problem becomes strictly convex and yields a unique solution with high probability. Second, we show that the correct skeleton E skel can be recovered with high probability. This is achieved by showing that, for each node X r , the estimator has zero weights for non-neighbor nodes Co r and has non-zero weights for its neighbors Ne r with high confidence. Before presenting the main results, we note that they are based on several important lemmas.

Lemma 6. Suppose Ξ is separable Banach space and fix P 0 P P p Ξ 1 q for some Ξ 1 Ď Ξ . Suppose c : Ξ Ñ R ě 0 is closed convex, k -positively homogeneous. Suppose f : Ξ Ñ Y is a mapping in the Lebesgue space of functions with finite first-order moment under P 0 and upper semi-continuous with finite Lipschitz constant lip c p f q . Then for all ε ě 0 , the following inequality holds with probability 1 : sup Q P A Wp ε p P 0 q , Q P P p Ξ 1 q ş f p ξ 1 q Q p d ξ 1 q ď ε lip c p f q ` ş f p ξ 1 q P 0 p d ξ 1 q .

Lemma 6 follows directly from Cranko et al. [2021] and allows us to obtain an upper bound between the worst-case risk and empirical risk. It is crucial for the following finite-sample guarantees.

Lemma 7. If Assumption 3 holds, for any Q P A W p ε p ˜ P m q , with high probability, H Q S r S r is positive definite.

Lemma 8. If Assumption 3 and Assumption 4 hold, for any Q P A W p ε p ˜ P m q and α P p 0 , 1 s , with high probability,

The above two lemmas illustrate that Assumption 3 and Assumption 4 hold in the finite-sample setting. Let the estimated skeleton, neighbor nodes and the complement be ˆ G : ' p V , ˆ E skel q , ˆ Ne r and ˆ Co r respectively. We derive the following guarantees for the proposed Wasserstein DRO estimator.

Theorem 9. Given a Bayesian network p G , P q of n categorical random variables and its skeleton G skel : ' p V , E skel q . Assume that the condition ∥ W ˚ ∥ B, 2 , 1 ď ¯ B holds for some ¯ B ą 0 associated with an optimal Lagrange multiplier λ ˚ B ą 0 for W ˚ defined in Equation (1) . Suppose that ˆ W is a DRO risk minimizer of Equation (4) with a Wasserstein distance of order 1 and an ambiguity radius ε ' ε 0 { m where m is the number of samples drawn i.i.d. from P . Under Assumptions 1, 2, 3, 4, if the number of samples satisfies

where C only depends on α , Λ , and if the Lagrange multiplier satisfies

then for any δ P p 0 , 1 s , r P r n s , with probability at least 1 ´ δ , the following properties hold:

(a) The optimal estimator ˆ W is unique.

(b) All the non-neighbor nodes are excluded: Co r Ď ˆ Co r .

(c) All the neighbor nodes are identified: Ne r Ď ˆ Ne r .

(d) The true skeleton is successfully reconstructed: G skel ' ˆ G skel .

Proof sketch. The main idea in the proof follows that in the lasso estimator [Wainwright, 2009]. Based on a primal-dual witness construction method and Lemma 8, it can be shown that if we control λ ˚ B , a solution constrained to have zero weight for all the non-neighbor nodes is indeed optimal. Furthermore, Lemma 7 implies that there is a unique solution given information about the true neighbors. The uniqueness of the aforementioned optimal solution without knowing the true skeleton is then verified via convexity and a conjugate formulation of the block ℓ 2 , 1 norm. Hereby we have shown that the optimal solution to Equation (4) is unique and excluding all the non-neighbor nodes. Next, we derive conditions on λ ˚ B for the estimation bias ∥ ˆ W ´ W ˚ ∥ B, 2 , 8 ă β { 2 to hold, which allows us to recover all the neighbor nodes. In such manner, applying the union bound over all the nodes r P r n s leads to successful exact skeleton discovery with high probability.

The results in Theorem 9 encompass some intuitive interpretations. Compared to Theorem 1 in Bank and Honorio [2020], we make more explicit the relationship among m , λ ˚ B and δ . On one hand, the lower bound of λ ˚ B ensures that a sparse solution excluding non-neighbor nodes is obtained. A large error magnitude expectation µ therefore elicits stronger regularization. On the other hand, the upper bound λ ˚ B is imposed to guarantee that all the neighbor nodes are identified with less restriction on W . There is naturally a trade-off when choosing ¯ B in order to learn the exact skeleton. The sample complexity depends on cardinalities ρ r n s , confidence level δ , the number of nodes n , the ambiguity level ε 0 and assumptions on errors. The dependence on σ indicates that higher uncertainty caused by larger error norms demands more samples whereas the dependence on µ ´ 2 results from the lower bound condition on λ ˚ B with respect to µ . The ambiguity level is set to ε 0 { m based on the observation that obtaining more samples reduces ambiguity of the true distribution. In practice, we find that ε 0 is usually small thus negligible. Note that the sample complexity is polynomial in n . Furthermore, if we assume that the true graph has a bounded degree of d , we find that m ' O p C p ε 0 ` log p n { δ q` log n ` log ρ max q σ 2 ρ 7 max d 3 min p µ 2 , 1 q q is logarithmic with respect to n , consistent with the results in Wainwright [2009].

We introduce constants ¯ B and λ ˚ B in order to find a condition for the statements in Theorem 9 to hold. If there exists a W incurring a finite loss, we can always find a solution ˆ W and let ¯ B fi max ˆ W ∥ ˆ W ∥ B, 2 , 1 be the maximum norm of all solutions. Imposing ∥ W ∥ B, 2 , 1 ď ¯ B is equivalent to the original problem. By Lagrange duality and similar argument for the lasso estimator, there exists a λ ˚ B that finds all the solutions with ∥ ˆ W ∥ B, 2 , 1 ' ¯ B . Therefore we have a mapping between ε and λ ˚ B .

3.3 Kullback-Leibler DRO

In addition to optimal transport, ϕ -divergence is also widely used to construct an ambiguity set for DRO problems. We consider the following definition of a special ϕ -divergence called the KL divergence: D p Q ∥ P q : ' ş X ln Q p d x q P p d x q Q p d x q , where Q P P p X q is absolutely continuous with respect to P P P p X q and Q p d x q P p d x q denotes the Radon-Nikodym derivative. A noteworthy property of ambiguity sets based on the KL divergence is absolute continuity of all the candidate distributions with respect to the empirical distribution. It implies that if the true distribution is an absolutely continuous probability distribution, the ambiguity set will never include it. In fact, any other point outside the support of the nominal distribution remains to have zero probability. Unlike the Wasserstein metric, the KL divergence does not measure some closeness between two points, nor does it have some measure concentration results. However, we argue that adopting the KL divergence may bring advantages over the Wasserstein distance since the Bayesian network distribution we study is a discrete distribution over purely categorical random variables. Moreover, as illustrated below, adopting the KL divergence leads to better computational efficiency.

Let A fi A D ε p ˜ P m q be the ambiguity set, the dual formulation of Equation (3) follows directly from Theorem 4 in Hu and Hong [2013]:

which directly minimizes a convex objective. In contrast to the approximate Wasserstein estimator, this KL DRO estimator finds the exact solution to the primal problem by strong duality.

The worst-case risk over a KL divergence ball can be bounded by variance [Lam, 2019], similar to Lipschitz regularization in Lemma 6. Based on this observation, we derive the following results:

Theorem 10. Suppose that ˆ W is a DRO risk minimizer of Equation (4) with the KL divergence and an ambiguity radius ε ' ε 0 { m . Given the same definitions of p G , P q , G skel , ¯ B , λ ˚ B , m in Theorem 9. Under Assumptions 1, 2, 3, 4, if the number of samples satisfies

where C depends on α , Λ while independent of n , and if the Lagrange multiplier satisfies the same condition as in Theorem 9, then for any δ P p 0 , 1 s , r P r n s , with probability at least 1 ´ δ , the properties (a)-(d) in Theorem 9 hold.

The sample complexities in Theorem 9 and Theorem 10 differ in the constant C due to the difference between the two probability metrics. Note that C is independent of n in both methods. The dependency on 1 {p λ ˚ B q 2 is absorbed in the denominator because we require that λ ˚ B ´ 16 µρ max { α ą 0 . The sample complexities provide a perspective of our confidence on upper bounding the true risk in terms of the ambiguity radius. ε 0 serves as our initial guess on distributional uncertainty and increases the sample complexity only slightly because it is usually dominated by other terms in practice: ε ! log p n { δ q . Even though the samples are drawn from an adversarial distribution with a proportion of noises, the proposed methods may still succeed as long as the true distribution can be made close to an upper confidence bound.

4 Experiments

We conduct experiments 3 on benchmark datasets [Scutari, 2010] and real-world datasets [Malone et al., 2015] perturbed by the following contamination models:

Noisefree model. This is the baseline model without any noises.

Huber's contamination model. In this model, each sample has a fixed probability of ζ to be replaced by a sample drawn from an arbitrary distribution.

Independent failure model. Each entry of a sample is independently corrupted with probability ζ .

We conduct all experiments on a laptop with an Intel Core i7 2.7 GHz processor. We adopt the proposed approaches based on Wasserstein DRO and KL DRO, the group norm regularization method [Bank and Honorio, 2020], the MMPC algorithm [Tsamardinos et al., 2006] and the GRaSP algorithm [Lam et al., 2022] for skeleton learning. Based on the learned skeletons, we infer a DAG with the hill-climbing (HC) algorithm [Tsamardinos et al., 2006]. For the Wasserstein-based method, we leverage Adam [Kingma and Ba, 2014] to optimize the overall objective with β 1 ' 0 . 9 , β 2 ' 0 . 990 , a learning rate of 1 . 0 , a batch size of 500 , a maximum of 200 iterations for optimization and 10 iterations for approximating the worst-case distribution. For the KL-based and standard regularization methods, we use the L-BFGS-B [Byrd et al., 1995] optimization method with default parameters. We set the cardinality of the maximum conditional set to 3 in MMPC. The Bayesian information criterion (BIC) [Neath and Cavanaugh, 2012] score is adopted in the HC algorithm. A random mixture of 20 random Bayesian networks serves as the adversarial distribution for both contamination models. All hyper-parameters are chosen based on the best performance on random Bayesian networks with the same size as the input one. Each experimental result is taken as an average over 10 independent runs. When dealing with real-world datasets, we randomly split the data into two halves for training and testing.

We use the F1-score, or the Dice coefficient (regarding the label of each edge indicating its presence as a binary random variable and considering all possible edges), to evaluate performance on benchmark datasets and BIC for real-world datasets. The results are reported in Table 1 and more results can be found in Table 2 in appendix. We observe that in most cases the proposed DRO methods are comparable to MMPC and MMHC, which are generally the best-performing methods in Bank and Honorio [2020]. We illustrate in Figure 1 the results on earthquake by varying the number of samples, corruption level and ambiguity radius or regularization coefficient. Figure 1 (a) suggests that all the methods perfectly recover the true skeleton given more than 2 , 000 samples. The results in Figure 1 (b-c) indicate that, in highly uncertain settings, Wasserstein DRO as well as KL DRO is superior to other approaches. Meanwhile, Table 1 and Figure 1 (a) suggest that the DRO methods and the regularized ERM approach are comparable to MMPC and GRaSP when clean data is given. The sensitivity analysis (Figure 1 (d)) suggests a trade-off between robustness and target performance (F1-score in our case). All the approaches have similar execution time except that Wasserstein DRO is several times slower due to the combinatorial sub-problem of computing the worst-case distribution.

5 Discussion and Conclusion

In this paper, we put forward a distributionally robust optimization method to recover the skeleton of a general discrete Bayesian network. We discussed two specific probability metrics, developed

Dataset

n

m

Noise

ζ

Wass

KL

Reg

MMPC

GRASP

Wass+HC

KL+HC

Reg+HC

MMPC+HC

GRASP+HC

HC

asia

8

1000

Noisefree

0

0

.

7800

:

0

.

7285

:

0

.

7897

:

0

.

9067

0

.

8167

0

.

5123

0

.

6367

0

.

5743

0

.

6667

0

.

6583

0

.

6550

asia

8

1000

Huber

0

.

2

0

.

7333

:

0

.

7124

:

0

.

7297

:

0

.

5468

0

.

6570

0

.

3943

0

.

3724

0

.

3487

0

.

2907

0

.

3664

0

.

2183

asia

8

1000

Independent

0

.

2

0

.

6933

0

.

6797

0

.

6868

0

.

6359

0

.

3632

:

0

.

2676

0

.

2632

0

.

2581

0

.

2469

0

.

1794

0

.

2443

cancer

5

1000

Noisefree

0

1

.

0000

:

1

.

0000

:

1

.

0000

:

0

.

6133

0

.

6133

0

.

2800

0

.

2800

0

.

2800

0

.

2800

0

.

2800

0

.

2800

cancer

5

1000

Huber

0

.

5

0

.

9156

:

0

.

8933

:

0

.

9092

:

0

.

6133

0

.

5357

0

.

4333

0

.

3833

0

.

4143

0

.

2589

0

.

2714

0

.

2589

cancer

5

1000

Independent

0

.

2

0

.

9048

:

0

.

9029

:

0

.

8992

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

earthquake

5

1000

Noisefree

0

0

.

8447

:

0

.

9333

:

0

.

9778

1

.

0000

0

.

9778

0

.

2000

0

.

2500

0

.

2500

0

.

2500

0

.

2500

0

.

2278

:

earthquake

5

1000

Huber

0

.

2

0

.

7509

:

0

.

7509

:

0

.

7509

:

0

.

5978

0

.

6583

:

0

.

4618

0

.

4618

0

.

4618

0

.

3860

0

.

4547

0

.

3860

earthquake

5

1000

Independent

0

.

2

0

.

6786

:

0

.

6350

:

0

.

6350

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

sachs

11

1000

Noisefree

0

0

.

8357

:

0

.

8402

:

0

.

8374

:

0

.

9697

0

.

7678

:

0

.

4310

:

0

.

4535

:

0

.

4641

:

0

.

5935

0

.

4112

:

0

.

5873

sachs

11

1000

Huber

0

.

2

0

.

7765

0

.

8064

0

.

7893

0

.

7498

0

.

5663

:

0

.

5194

0

.

4815

0

.

4520

0

.

4736

0

.

2380

0

.

5028

sachs

11

1000

Independent

0

.

5

0

.

5268

:

0

.

5208

:

0

.

5172

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

survey

6

1000

Noisefree

0

0

.

6596

0

.

6545

0

.

6506

0

.

6533

0

.

1714

:

0

.

1789

0

.

1789

0

.

1789

0

.

1789

0

.

0571

0

.

1789

survey

6

1000

Huber

0

.

2

0

.

7303

:

0

.

6778

:

0

.

7095

:

0

.

5396

0

.

3810

0

.

1444

0

.

1444

0

.

1444

0

.

1444

0

.

1516

0

.

1444

survey

6

1000

Independent

0

.

2

0

.

6311

:

0

.

6705

:

0

.

6220

:

0

.

2032

0

.

0000

:

0

.

1071

0

.

1071

0

.

1143

0

.

1071

0

.

0000

0

.

1071

alarm

37

1000

Noisefree

0

0

.

4750

:

0

.

7863

:

0

.

8042

:

0

.

8530

0

.

6824

:

0

.

3483

:

0

.

4949

:

0

.

4470

:

0

.

5635

0

.

4976

0

.

4494

:

alarm

37

1000

Huber

0

.

2

0

.

1432

:

0

.

1619

:

0

.

6571

:

0

.

5486

0

.

1945

:

0

.

2192

0

.

1680

:

0

.

3148

0

.

2774

0

.

2092

:

0

.

2582

alarm

37

1000

Independent

0

.

2

0

.

1419

:

0

.

1448

:

0

.

5458

:

0

.

4309

0

.

2830

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

voting

17

216

Noisefree

0

N/A

N/A

N/A

N/A

N/A

´

2451

.

8631

´

2453

.

2737

´

2453

.

4091

´

2475

.

5799

´

2482

.

3835

´

2456

.

1489

voting

17

216

Huber

0

.

2

N/A

N/A

N/A

N/A

N/A

´

4418

.

9731

´

4418

.

9731

´

4487

.

4544

´

4450

.

3941

´

4445

.

0175

´

4418

.

9731

voting

17

216

Independent

0

.

2

N/A

N/A

N/A

N/A

N/A

´

4453

.

8298

´

4453

.

8298

´

4522

.

5521

´

4465

.

1076

´

4473

.

8612

´

4453

.

8298

backache

32

90

Noisefree

0

N/A

N/A

N/A

N/A

N/A

´

1729

.

8364

´

1726

.

8465

´

1710

.

7248

´

1719

.

5002

´

1713

.

7583

´

1729

.

7991

backache

32

90

Huber

0

.

2

N/A

N/A

N/A

N/A

N/A

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

backache

32

90

Independent

0

.

2

N/A

N/A

N/A

N/A

N/A

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

500

1000

1500

2000

Samples

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

F1

Wass

KL

Reg

MMPC

GRaSP

2000

4000

6000

8000

10000

Samples

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10

-

4

10

-

2

10

0

10

2

10

4

Epsilon / Lambda

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

tractable algorithms to compute the estimators. We established the connection between the proposed method and regularization. We derived non-asymptotic bounds polynomial in the number of nodes for successful identification of the true skeleton. The sample complexities become logarithmic for bounded-degree graphs. Empirical results showcased the effectiveness of our methods.

The strength of making no distributional assumptions in our methods inherits from the regularized regression baseline, which is shown to be a special case of DRO. Besides the original benefits in Bank and Honorio [2020], our methods are explicitly robust and able to adjust ε to incorporate our uncertainty about the data generating mechanism.

Since we do not make any specific assumptions on the conditional probability distributions or on the corruption models, our methods may not be superior to the approaches proposed to tackle certain family of noises or parametric distributions. In addition, making assumptions such as an ordinal relationship, continuous values or a Poisson structural equation model (SEM) may lead to more efficient algorithms and tighter bounds. In some cases, adopting an identity encoding mapping is sufficient to learn a continuous Bayesian network [Aragam et al., 2015]. Furthermore, it would be interesting to incorporate prior distributional information into design of the ambiguity set for better performance. Another important topic is whether the underlying structure is identifiable in a robust manner [Sankararaman et al., 2022]. Formulating the complete DAG learning problem as one optimization problem may lead to a non-convex problem. Nonetheless, leveraging an adversarial training approach in continuous optimization for Gaussian Bayesian networks and causal discovery is a promising future direction to pursue.

Acknowledgments and Disclosure of Funding

This material is based upon work supported by the National Science Foundation under Grant No. 1652530.

References

Bryon Aragam, Arash A Amini, and Qing Zhou. Learning directed acyclic graphs with penalized neighbourhood regression. arXiv preprint arXiv:1511.08963 , 2015.

Adarsh Bank and Jean Honorio. Provable efficient skeleton learning of encodable discrete bayes nets in poly-time and sample complexity. In 2020 IEEE International Symposium on Information Theory (ISIT) , pages 2486-2491. IEEE, 2020.

Dimitris Bertsimas, Kosuke Imai, and Michael Lingzhi Li. Distributionally robust causal inference with observational data. arXiv preprint arXiv:2210.08326 , 2022.

Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research , 44(2):565-600, 2019.

Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing , 16(5):1190-1208, 1995.

Moses Charikar and Anthony Wirth. Maximizing quadratic programs: Extending grothendieck's inequality. In 45th Annual IEEE Symposium on Foundations of Computer Science , pages 54-60. IEEE, 2004.

Ruidi Chen and Ioannis C Paschalidis. A robust learning approach for regression models based on distributionally robust optimization. Journal of Machine Learning Research , 19(13), 2018.

Wenyu Chen, Mathias Drton, and Y Samuel Wang. On causal discovery with an equal-variance assumption. Biometrika , 106(4):973-980, 2019.

David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research , 3(Nov):507-554, 2002.

Max Chickering, David Heckerman, and Chris Meek. Large-sample learning of bayesian networks is np-hard. Journal of Machine Learning Research , 5:1287-1330, 2004.

Pedro Cisneros-Velarde, Alexander Petersen, and Sang-Yun Oh. Distributionally robust formulation and model selection for the graphical lasso. In International Conference on Artificial Intelligence and Statistics , pages 756-765. PMLR, 2020.

Diego Colombo, Marloes H Maathuis, et al. Order-independent constraint-based causal structure learning. J. Mach. Learn. Res. , 15(1):3741-3782, 2014.

Anthony C Constantinou, Yang Liu, Kiattikun Chobtham, Zhigao Guo, and Neville K Kitson. Largescale empirical validation of bayesian network structure learning algorithms with noisy data. International Journal of Approximate Reasoning , 131:151-188, 2021.

Zac Cranko, Zhan Shi, Xinhua Zhang, Richard Nock, and Simon Kornblith. Generalised lipschitz regularisation equals distributional robustness. In International Conference on Machine Learning , pages 2178-2188. PMLR, 2021.

Hadi Daneshmand, Manuel Gomez-Rodriguez, Le Song, and Bernhard Schoelkopf. Estimating diffusion network structures: Recovery conditions, sample complexity & soft-thresholding algorithm. In International conference on machine learning , pages 793-801. PMLR, 2014.

Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research , 58(3):595-612, 2010.

Rod G Downey and Michael R Fellows. Fixed-parameter tractability and completeness i: Basic results. SIAM Journal on computing , 24(4):873-921, 1995.

Mathias Drton and Marloes H Maathuis. Structure learning in graphical modeling. Annual Review of Statistics and Its Application , 4:365-393, 2017.

John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. The Journal of Machine Learning Research , 20(1):2450-2504, 2019.

Rizal Fathony, Ashkan Rezaei, Mohammad Ali Bashiri, Xinhua Zhang, and Brian Ziebart. Distributionally robust graphical models. Advances in Neural Information Processing Systems , 31, 2018.

Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics , 9(3):432-441, 2008.

Robert Ganian and Viktoriia Korchemna. The complexity of bayesian network learning: Revisiting the superstructure. Advances in Neural Information Processing Systems , 34:430-442, 2021.

Ming Gao, Yi Ding, and Bryon Aragam. A polynomial-time algorithm for learning nonparametric causal graphs. Advances in Neural Information Processing Systems , 33:11599-11611, 2020.

Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research , 2022.

Tian Gao, Debarun Bhattacharjya, Elliot Nelson, Miao Liu, and Yue Yu. Idyno: Learning nonparametric dags from interventional dynamic data. In International Conference on Machine Learning , pages 6988-7001. PMLR, 2022.

Maxime Gasse, Alex Aussem, and Haytham Elghazel. A hybrid algorithm for bayesian network structure learning with application to multi-label learning. Expert Systems with Applications , 41 (15):6755-6772, 2014.

Asish Ghoshal and Jean Honorio. Learning identifiable gaussian bayesian networks in polynomial time and sample complexity. Advances in Neural Information Processing Systems , 30, 2017.

Asish Ghoshal and Jean Honorio. Learning linear structural equation models in polynomial time and sample complexity. In International Conference on Artificial Intelligence and Statistics , pages 1466-1475. PMLR, 2018.

Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity. Monographs on statistics and applied probability , 143:143, 2015.

Christina Heinze-Deml, Marloes H Maathuis, and Nicolai Meinshausen. Causal structure learning. Annual Review of Statistics and Its Application , 5:371-391, 2018.

Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online , 1(2):9, 2013.

Tommi Jaakkola, David Sontag, Amir Globerson, and Marina Meila. Learning bayesian network structure using lp relaxations. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 358-365. JMLR Workshop and Conference Proceedings, 2010.

Leonid Vasilevich Kantorovich and SG Rubinshtein. On a space of totally additive functions. Vestnik of the St. Petersburg University: Mathematics , 13(7):52-59, 1958.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Neville Kenneth Kitson, Anthony C Constantinou, Zhigao Guo, Yang Liu, and Kiattikun Chobtham. A survey of bayesian network structure learning. Artificial Intelligence Review , pages 1-94, 2023.

Janne Korhonen and Pekka Parviainen. Exact learning of bounded tree-width bayesian networks. In Artificial Intelligence and Statistics , pages 370-378. PMLR, 2013.

Evangelia Kyrimi, Mariana Raniere Neves, Scott McLachlan, Martin Neil, William Marsh, and Norman Fenton. Medical idioms for clinical bayesian network development. Journal of Biomedical Informatics , 108:103495, 2020.

Henry Lam. Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization. Operations Research , 67(4):1090-1105, 2019.

Wai-Yin Lam, Bryan Andrews, and Joseph Ramsey. Greedy relaxations of the sparsest permutation algorithm. In Uncertainty in Artificial Intelligence , pages 1052-1062. PMLR, 2022.

Yeshu Li, Zhan Shi, Xinhua Zhang, and Brian Ziebart. Distributionally robust structure learning for discrete pairwise markov networks. In International Conference on Artificial Intelligence and Statistics , pages 8997-9016. PMLR, 2022.

Po-Ling Loh and Peter Bühlmann. High-dimensional learning of linear causal networks via inverse covariance estimation. The Journal of Machine Learning Research , 15(1):3065-3105, 2014.

Lars Lorch, Scott Sussex, Jonas Rothfuss, Andreas Krause, and Bernhard Schölkopf. Amortized inference for causal structure learning. Advances in Neural Information Processing Systems , 35: 13104-13118, 2022.

Ahmed Mabrouk, Christophe Gonzales, Karine Jabet-Chevalier, and Eric Chojnacki. An efficient bayesian network structure learning algorithm in the presence of deterministic relations. In ECAI 2014 , pages 567-572. IOS Press, 2014.

Brandon M Malone, Matti Järvisalo, and Petri Myllymäki. Impact of learning strategies on the quality of bayesian networks: An empirical evaluation. In UAI , pages 562-571. Citeseer, 2015.

K Manjusha and Rakesh Kumar. Spam mail classification using combined approach of bayesian and neural network. In 2010 International Conference on Computational Intelligence and Communication Networks , pages 145-149. IEEE, 2010.

Preetam Nandy, Alain Hauser, and Marloes H Maathuis. High-dimensional consistency in score-based and hybrid structure learning. The Annals of Statistics , 46(6A):3151-3183, 2018.

Kazuki Natori, Masaki Uto, and Maomi Ueno. Consistent learning bayesian networks with thousands of variables. In Advanced Methodologies for Bayesian Networks , pages 57-68. PMLR, 2017.

Andrew A Neath and Joseph E Cavanaugh. The bayesian information criterion: background, derivation, and applications. Wiley Interdisciplinary Reviews: Computational Statistics , 4(2): 199-203, 2012.

Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing Systems , 33:17943-17954, 2020.

Ignavier Ng, Yujia Zheng, Jiji Zhang, and Kun Zhang. Reliable causal discovery with improved exact search and weaker assumptions. Advances in Neural Information Processing Systems , 34: 20308-20320, 2021.

Ignavier Ng, Shengyu Zhu, Zhuangyan Fang, Haoyang Li, Zhitang Chen, and Jun Wang. Masked gradient-based causal structure learning. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM) , pages 424-432. SIAM, 2022.

Viet Anh Nguyen, Daniel Kuhn, and Peyman Mohajerin Esfahani. Distributionally robust inverse covariance estimation: The wasserstein shrinkage estimator. Operations Research , 70(1):490-515, 2022.

Sebastian Ordyniak and Stefan Szeider. Parameterized complexity results for exact bayesian network structure learning. Journal of Artificial Intelligence Research , 46:263-302, 2013.

Gunwoong Park and Garvesh Raskutti. Learning quadratic variance function (qvf) dag models via overdispersion scoring (ods). J. Mach. Learn. Res. , 18:224-1, 2017.

Eric Perrier, Seiya Imoto, and Satoru Miyano. Finding optimal bayesian network given a superstructure. Journal of Machine Learning Research , 9(10), 2008.

Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659 , 2019.

Goutham Rajendran, Bohdan Kivva, Ming Gao, and Bryon Aragam. Structure learning in polynomial time: Greedy algorithms, bregman information, and exponential families. Advances in Neural Information Processing Systems , 34:18660-18672, 2021.

Pradeep Ravikumar, Martin J Wainwright, and John D Lafferty. High-dimensional ising model selection using ℓ 1 -regularized logistic regression. The Annals of Statistics , 38(3):1287-1319, 2010.

Karthik A Sankararaman, Anand Louis, and Navin Goyal. Robust identifiability in linear structural equation models of causal inference. In Uncertainty in Artificial Intelligence , pages 1728-1737. PMLR, 2022.

Marco Scutari. Learning bayesian networks with the bnlearn r package. Journal of Statistical Software , 35:1-22, 2010.

Soroosh Shafieezadeh-Abadeh, Daniel Kuhn, and Peyman Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research , 20(103):1-68, 2019.

Ali Shojaie and George Michailidis. Penalized likelihood methods for estimation of sparse highdimensional directed acyclic graphs. Biometrika , 97(3):519-538, 2010.

Tomi Silander and Petri Myllymäki. A simple approach for finding the globally optimal bayesian network structure. In Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence , pages 445-452, 2006.

Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review , 9(1):62-72, 1991.

Peter Spirtes, Christopher Meek, and Thomas Richardson. An algorithm for causal inference in the presence of latent variables and selection bias. Computation, causation, and discovery , 21: 211-252, 1999.

Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search . MIT press, 2000.

Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine learning , 65(1):31-78, 2006.

Caroline Uhler, Garvesh Raskutti, Peter Bühlmann, and Bin Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics , pages 436-463, 2013.

Martin J Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso). IEEE transactions on information theory , 55(5): 2183-2202, 2009.

Xiaolu Wang, Yuen-Man Pun, and Anthony Man-Cho So. Distributionally robust graph learning from smooth signals under moment uncertainty. arXiv e-prints , pages arXiv-2105, 2021.

Dennis Wei, Tian Gao, and Yue Yu. Dags with no fears: A closer look at continuous optimization for learning bayesian networks. Advances in Neural Information Processing Systems , 33:3895-3906, 2020.

Adriano V Werhli, Marco Grzegorczyk, and Dirk Husmeier. Comparative evaluation of reverse engineering gene regulatory networks with relevance networks, graphical gaussian models and bayesian networks. Bioinformatics , 22(20):2523-2531, 2006.

Yue Yu, Tian Gao, Naiyu Yin, and Qiang Ji. Dags with no curl: An efficient dag structure learning approach. In International Conference on Machine Learning , pages 12156-12166. PMLR, 2021.

Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems , 31, 2018.

A Algorithms

The pseudo-code of the greedy algorithm for solving Equation (6) in Wasserstein DRO is illustrated in Algorithm 1.

Algorithm 1 Greedy Algorithm for the Wasserstein Worst-case Risk

B Optimization Details

Define

The Lagrangian dual problem of the Wasserstein DRO problem is

One of its sub-gradients can be computed as

For the DRO problem based on the KL divergence:

a sub-gradient of which can be computed as

C Technical Proofs

Proposition 11 (NP-hardness of Wasserstein DRO Supremum) . The problem in Equation (6) is NP-hard.

Proof. Recall the MAXQP problem:

In Equation (6), let γ ' 0 , E p x r q ' 0 , x ¯ r correspond to n binary variables taking values in {-1, 1} and E p x ¯ r q ' x ¯ r . Let W P R n ˆ n 2 . For all i, j P r n s , let k ' p i ´ 1 q n ` j . The k -the column of W satisfies W ik ' 1 , W jk ' a ij { 2 and 0 for the other elements. We have obtained a polynomial-time reduction from an NP-hard problem to Equation (6).

Proposition 5 (Regularization Equivalence) . Let : W : ' r W ; ´ I ρ r s ⊺ P R ρ r n s ˆ ρ r with W r ' ´ I ρ r . If γ ě ρ r n s ∥ : W ∥ 2 F , the Wasserstein distributionally robust regression problem in Equation (5) is equivalent to

which subsumes a linear regression approach regularized by the Frobenius norm as a special case.

Proof. Recapitulating on Equation (6):

Observe that

Therefore, for any x ‰ x p i q ,

which implies that the supremum can always be achieved at x ' x p i q . Minimizing over γ leads to

Lemma 6. Suppose Ξ is separable Banach space and fix P 0 P P p Ξ 1 q for some Ξ 1 Ď Ξ . Suppose c : Ξ Ñ R ě 0 is closed convex, k -positively homogeneous. Suppose f : Ξ Ñ Y is a mapping in the Lebesgue space of functions with finite first-order moment under P 0 and upper semi-continuous with finite Lipschitz constant lip c p f q . Then for all ε ě 0 , the following inequality holds with probability 1 :

Proof. The result follows directly from Theorem 1 in Cranko et al. [2021]:

Since Ξ 1 Ď Ξ , observe

Lemma 7. If Assumption 3 holds, for any Q P A W p ε p ˜ P m q , with probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q , we have

Proof. The minimum eigenvalue of the true covariance matrix H S r S r satisfies:

where ∥ u ∥ 2 ' 1 is an eigenvector of H Q S r S r with minimum eigenvalue.

Therefore, Λ min p H Q S r S r q can be lower bounded as follows:

due to the fact that

We can obtain an upper bound on | u ⊺ p ˜ H S r S r ´ H Q S r S r q u | based on Lemma 6:

because for function g p E p x qq : ' u ⊺ H S r S r u , it can be shown that for any ∥E p x q ´ E p x 1 q ∥ 1 ' k and some | S | ' k ,

Recall that we assume that the encoding schemes take values in B ' t´ 1 , 0 , 1 u . Therefore lip c p g q ' 4 | S r | 1 2 .

We derive an upper bound of ∥ p H S r S r ´ ˜ H S r S r q ∥ F as follows. Consider a random variable and its expectation

By Hoeffding's inequality, we observe

for t ą 0 . Setting t ' t | S r | for all i, j P S r and applying the union bound,

To conclude, with probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q , we have

Lemma 8. If Assumption 3 and Assumption 4 hold, for any Q P A W p ε p ˜ P m q and α P p 0 , 1 s , with probability at least 1 ´ O p exp p´ Cm ρ 2 max | S r | 3 ` log | S c r | ` log | S r |qq and ε ď C ρ max | S r | 3 { 2 , where C only depends on α , Λ min p H S S q .

Proof. We would like to obtain an upper bound for H Q S c S r p H Q S r S r q ´ 1 B, 1 , 8 . We may write

for t ą 0 . Taking t ' t ρ i | S r | and applying the union bound over i P Co r , we observe that

Similarly, taking t ' t | S r | ,

In order to bound ∥ H Q S c r S r ´ H S c r S r ∥ B, 1 , 8 , for Q ‰ ˜ P , consider ∥ H Q S c r S r ´ ˜ H S c r S r ∥ B, 1 , 8 ď ∥ H Q S c r S r ∥ B, 1 , 8 ` ∥ ˜ H S c r S r ∥ B, 1 , 8 ď E Q ∥E p X ¯ r q S c r E p X ¯ r q ⊺ S r ∥ B, 1 , 8 ` E ˜ P m ∥E p X ¯ r q S c r E p X ¯ r q ⊺ S r ∥ B, 1 , 8 ' sup ˜ P 1 m , Q 1 P A Wp ε p ˜ P 1 m q | E Q 1 ξ 1 ∥E p X ¯ r q S c r E p X ¯ r q ⊺ S r ∥ B, 1 , 8 ´ E ˜ P 1 m ξ 2 ∥E p X ¯ r q S c r E p X ¯ r q ⊺ S r ∥ B, 1 , 8 | ,

where Q 1 and ˜ P 1 m are probability measures on X ˆ Ξ with Ξ ' t´ 1 , ` 1 u and identical marginals as Q and ˜ P m respectively. We assume that Q ‰ ˜ P because otherwise ∥ H Q S c r S r ´ ˜ H S c r S r ∥ B, 1 , 8 ' 0 holds trivially. In this way, the equality is always achieved by some Q 1 , ˜ P 1 m , i.e., setting Q 1 p X , ξ ' 1 q ' 1 and ˜ P 1 m p X , ξ ' ´ 1 q ' 1 .

Define the transport cost function in the ambiguity set A W p ε p ˜ P 1 m q to be c 1 pp X 1 , ξ 1 q , p X 2 , ξ 2 qq : ' ∥E p X 1 q ´ E p X 2 q ∥ 1 with zero cost for ξ . Let g p X , ξ q : ' ξ 1 ∥E p X ¯ r q S c r E p X ¯ r q ⊺ S r ∥ B, 1 , 8 . Consider the Lipschitz constants of g :

Therefore, by the Kantorovich-Rubinstein theorem [Kantorovich and Rubinshtein, 1958],

Similarly,

Based on the above two inequalities, we find that

with probability at least 1 ´ 2 | S c r || S r | exp p´ mt 2 2 ρ 2 max | S r | 2 q , and

with probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q .

Based on Equation (8), we also have

with probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q .

Next we look at the upper bound on the difference between the inverses of H Q S r S r and H S r S r . Observe that

According to Lemma 7, with probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q , we have

Let t ' 1 2 Λ min p H S r S r q and ε ď Λ min p H S r S r q 16 | S r | 1 2 . We get that, with probability at least 1 ´ 2 | S r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q ,

Set t ' t Λ min p H S r S r q 4 ? | S r | and ε ď t Λ min p H S r S r q 8 | S r | ? | S r | in Equation (12), we get that, with probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 p Λ min p H S r S r qq 2 32 | S r | 3 q ,

Now we are ready to obtain upper bounds for the four terms recapitulated here:

We derive the bounds separately.

For the first term, based on Assumption 4, consider

Taking t ' α 24 p 1 ´ α q Λ min p H S r S r q | S r | and ε ď α 48 p 1 ´ α q| S r | Λ min p H S r S r q | S r | in Equation (11) and adopting Equation (13), we conclude that, with probability at least 1 ´ 2 | S r | 2 exp p´ mα 2 Λ min p H S r S r q 1152 p 1 ´ α q 2 | S r | 3 q ´ 2 | S r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q and ε ď min p α 48 p 1 ´ α q| S r | b Λ min p H S r S r q | S r | , Λ min p H S r S r q 16 | S r | 1 2 q , ∥ H S c r S r rp H Q S r S r q ´ 1 ´p H S r S r q ´ 1 s ∥ B, 1 , 8 ď α 6 .

For the second term, rewrite it as

For the third term, we obtain the upper bound

Taking t ' a α 6 in Equation (14). Taking t ' 1 2 a α 6 and 2 ερ max | S r | ď 1 2 a α 6 in Equation (10). We establish the upper bound that, with probability at least 1 ´ 2 | S c r || S r | exp p´ mα 48 ρ 2 max | S r | 2 q ´ 2 | S r | 2 exp p´ mα p Λ min p H S r S r qq 2 192 | S r | 3 q ´ 2 | S r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q and ε ď min p 1 4 ρ max | S r | a α 6 , Λ min p H S r S r q 8 | S r | b α 6 | S r | , Λ min p H S r S r q 16 | S r | 1 2 q ,

For the fourth term, in accordance with Assumption 4,

In conclusion, we have shown that, with probability at least 1 ´ 2 | S r | 2 exp p´ mα 2 Λ min p H S r S r q 1152 p 1 ´ α q 2 | S r | 3 q ´ 2 | S r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q´ 2 | S c r || S r | exp p´ mα 2 Λ min p H S r S r q 288 ρ 2 max | S r | 3 q´ 2 | S c r || S r | exp p´ mα 48 ρ 2 max | S r | 2 q´ 2 | r | 2 exp p´ mα p Λ min p H S r S r qq 2 192 | S r | 3 q ´ 2 | r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q and

the mutual incoherence condition holds for any worst-case distributions:

Simplifying the above expressions, with probability at least 1 ´ O p exp p´ Cm ρ 2 max | S r | 3 ` log | S c r | ` log | S r |qq and ε ď C ρ max | S r | 3 { 2 ,

where C only depends on α , Λ min p H S r S r q .

Lemma 12. If Assumption 1 holds, then for any Q P A W p ε p ˜ P m q and α P p 0 , 1 s , with probability at least 1 ´| S r | ρ r exp p´ mµ 2 2 σ 2 q , ε ď µ σ and λ ˚ B ą 32 µ ? ρ r p 1 ´ α { 2 q α , we have

With probability at least 1 ´| Co r | ρ r exp p´ mµ 2 2 σ 2 q , ε ď µ σ and λ ˚ B ą 32 µ ? ρ max ρ r α , we have

Proof. We start with ~ E Q E p X ¯ r q S r e ⊺ ~ 2 , 8 . After some algeraic manipulation, we find that

Since | e j | is a bounded random variable according to Assumption 1, we apply Hoeffding's inequality to get

Base on a similar argument as Equation (9), we can derive

Taking the union bound over all i P S r and j P ρ r , we find that

Setting t ' µ and ε ď µ σ while requiring λ ˚ B ą 32 µ ? ρ r p 1 ´ α { 2 q α . With probability at least 1 ´ | S r | ρ r exp p´ mµ 2 2 σ 2 q , we have

Then we consider ∥ E Q E p X ¯ r q S c r e ⊺ ∥ B, 2 , 8 :

Similarly, applying Hoeffding's inequality and the Kantorovich-Rubinstein theorem gives us

Let t ' µ , ε ď µ σ and λ ˚ B ą 32 µ ? ρ max ρ r α hold, we have, with probability at least 1 ´ | Co r | ρ r exp p´ mµ 2 2 σ 2 q ,

Theorem 9. Given a Bayesian network p G , P q of n categorical random variables and its skeleton G skel : ' p V , E skel q . Assume that the condition ∥ W ˚ ∥ B, 2 , 1 ď ¯ B holds for some ¯ B ą 0 associated with an optimal Lagrange multiplier λ ˚ B ą 0 for W ˚ defined in Equation (1) . Suppose that ˆ W is a DRO risk minimizer of Equation (4) with a Wasserstein distance of order 1 and an ambiguity radius which leads to

ε ' ε 0 { m where m is the number of samples drawn i.i.d. from P . Under Assumptions 1, 2, 3, 4, if the number of samples satisfies

where C only depends on α , Λ , and if the Lagrange multiplier satisfies

then for any δ P p 0 , 1 s , r P r n s , with probability at least 1 ´ δ , the following properties hold:

(a) The optimal estimator ˆ W is unique.

(b) All the non-neighbor nodes are excluded: Co r Ď ˆ Co r .

(c) All the neighbor nodes are identified: Ne r Ď ˆ Ne r .

(d) The true skeleton is successfully reconstructed: G skel ' ˆ G skel .

Proof. We prove the statements in this theorem in several steps. In order to prove (a) and (b), we will show that the DRO problem is strictly convex if true non-neighbors are known so that there is an optimal solution. Next we would like to demonstrate that this solution with a non-neighbor constraint is indeed unique for all the solutions without constraints. The proof for uniqueness comes with a conclusion that we do not accidentally include any edge between the current node and its non-neighbors. Next, to prove (c), we present a generalization bound for the DRO estimator in terms of its true risk, which leads to a ℓ 8 bound of the difference between the estimator ˆ W and the true weight matrix W ˚ . Combined with the assumption on the minimum weight, it implies that we include all the neighbor nodes successfully. Finally, by taking a union bound for all the nodes, we could conclude that the correct skeleton is recovered with high probability, which proves (d).

(i) Given the true non-neighbors, there is a unique solution.

We start with the Wasserstein DRO problem, which we recapitulate here for convenience:

The objective is convex because it is a supremum of convex functions.

For now, we assume that the non-neighbor nodes Co r are given. We can then explicitly restrict W i ' 0 for all i P Co r . The Hessian of W S r ¨ is a block diagonal matrix reads

where

is the covariance matrix of encodings of X ¯ r under some distribution Q P A W p ε p ˜ P m q .

Since W S c r ¨ is fixed to be zero and ∇ 2 R Q p W S r ¨ q is a block diagonal matrix, we focus on showing that H Q S r S r ą 0 .

We apply Lemma 7 to get the bound

with probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q . Λ min p H S r S r q ´ 4 ε | S r | 1 2 ´ t ą 0 will guarantee that the DRO problem in Equation (4) has a unique solution when the W i ' 0 is satisfied for non-neighbor nodes.

(ii) Given the true non-neighbors, the solution is optimal.

We would like to show that the solution to Equation (4) with true non-neighbor constraints is optimal. In this way, we do not recover any non-neighbor nodes in the skeleton. We adopt the primal-dual witness (PDW) [Wainwright, 2009] method to show optimality for the constrained unique solution.

Recall that we assume ∥ W ∥ B, 2 , 1 ď ¯ B . To begin with, we write the dual problem as

where λ B is the Lagrange multiplier for the norm constraint on W .

ˆ W is optimal if and only if there exists p Q ˚ , Z ˚ , λ ˚ B q that satisfies the KKT condition:

Note that we assume that the constraint ∥ W ∥ B, 2 , 1 ď ¯ B is active such that λ ˚ B ą 0 . This assumption is only for convenience of theoretical analysis and not restrictive. If it is not active, we have ∥ ˆ W ∥ B, 2 , 1 ' ˇ B ă ¯ B for some ˇ B and λ ˚ B ' 0 , which leads to an unconstrained problem similar to the ordinary least square problem, which is known to suffer from overfitting. Instead, we are usually interested in solutions that have finite norms so we can always find ¯ B ' ˇ B ´ ϵ ă ˇ B for some small positive constant ϵ ą 0 to make the constraint active and thus λ ˚ B ą 0 .

Substituting E p X r q ' W ˚ ⊺ E p X ¯ r q ` e into the first-order optimality condition yields

Solving for Z ˚ S c r ¨ , we find that

which can be bounded such that

Note that

Recall that α in Assumption 4. Based on Lemma 8 and Lemma 12, we may write

0 ă ď 1 ˚ ˚

with high probability and certain conditions on λ ˚ B and ε .

Henceforth, ∥ Z ˚ S c r ¨ ∥ B, 2 , 8 ă 1 satisfies strict dual feasibility and we must have ∥ ˆ W S c r ¨ ∥ B, 2 , 1 ' 0 according to complementary slackness: x Z ˚ , ˆ W y ' ∥ ˆ W ∥ B, 2 , 1 . In other words, we have

with high probability. This guarantees that we do not recover any node that is not a neighbor of r with high probability.

(iii) Without information about the true skeleton, we have a unique and optimal solution. We follow the proof of Lemma 11.2 in Hastie et al. [2015].

We have shown that ˆ W satisfying ˆ W i ' 0 @ i P Co r is an optimal solution with optimal dual variables ∥ Z ˚ S c r ¨ ∥ B, 2 , 8 ă 1 .

To avoid clutter of notations, we define

Let p ˇ W , ˇ λ q be any other optimal solution to inf W sup λ L DRO p W q ` λ p ∥ W ∥ B, 2 , 1 ´ ¯ B q . By definition,

The first-order optimality condition for ˆ W says

which implies

By definition, ∥ ˇ W ∥ B, 2 , 1 ´ ¯ B ' 0 and λ ˚ B ą 0 . Since L DRO p¨q is convex, the RHS of the above equation should be non-positive, or equivalently,

Therefore, the equality holds for the above inequalities, which leads to

Recall that ∥ Z ˚ S c r ¨ ∥ B, 2 , 8 ă 1 . In order for ∥ ˇ W ∥ B, 2 , 1 ' x Z ˚ , ˇ W y to hold, we must have

On the other hand,

In that wise, all the optimal solutions ˇ W have

This implies that we have a unique solution that excludes all the non-neighbor nodes without information about the true skeleton. Until now, we have proven properties (a) and (b).

(iv) The set of correct neighbors is recovered.

Consider again the first-order optimality condition in Equation (17),

According to Equation (13), with probability at least 1 ´ 2 | S r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q and ε ď Λ min p H S r S r q 16 | S r | 1 2 ,

According to Equation (15), with probability at least 1 ´ | S r | ρ r exp p´ mµ 2 2 σ 2 q , ε ď µ σ and λ ˚ B ą 32 µ ? ρ r p 1 ´ α { 2 q α , we have

On that account, with probability at least 1 ´ 2 | S r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q ´ | S r | ρ r exp p´ mµ 2 2 σ 2 q and ε ď min p Λ min p H S r S r q 16 | S r | 1 2 , µ σ q while requiring λ ˚ B ą 32 µ ? ρ r p 1 ´ α { 2 q α ,

By Assumption 2, if the condition λ ˚ B ă β 2 p α 8 p 1 ´ α { 2 q ` 1 q ρ max ? | S r | b Λ min p H S r S r q 4 is satisfied, the following inequality holds:

In this way, we are able to recover all the neighbor nodes with a threshold β { 2 . This proves (c).

(v) The true skeleton is recovered with high probability.

The above arguments tell us that with high probability and certain conditions for ε and λ ˚ B satisfied, for each node r , we do not recover any non-neighbor and we do recover all the neighbor nodes. The correct Ne r and Co r are thus identified. Now we are ready to prove (d).

Putting everything together and taking the the union bound for all nodes r P r n s , with probability at least 1 ´ O p n exp p´ Cmµ 2 σ 2 ρ 4 max ρ 3 r n s ` 2 log ρ r n s qq , ε ď Cµ σρ max ρ 3 { 2 r n s and 32 µρ max α ă λ ˚ B ă

β 2 p α 8 p 1 ´ α { 2 q ` 1 q ρ max ? ρ r n s b Λ 4 , where C only depends on α , Λ , we have

Setting ε ' ε 0 m and making the dependence on the sample size more explicit. We draw the conclusion that, if the number of samples satisfies

where C only depends on α , Λ , and if λ ˚ B satisfies

then with probability at least 1 ´ δ for δ P p 0 , 1 s :

Moreover, if we assume that the target graph has a bounded degree of d , the sample complexity becomes logarithmic in n :

Theorem 10. Suppose that ˆ W is a DRO risk minimizer of Equation (4) with the KL divergence and an ambiguity radius ε ' ε 0 { m . Given the same definitions of p G , P q , G skel , ¯ B , λ ˚ B , m in Theorem 9. Under Assumptions 1, 2, 3, 4, if the number of samples satisfies

where C depends on α , Λ while independent of n , and if the Lagrange multiplier satisfies the same condition as in Theorem 9, then for any δ P p 0 , 1 s , r P r n s , with probability at least 1 ´ δ , the properties (a)-(d) in Theorem 9 hold.

Proof. Define

According to Theorem 7 in Lam [2019], the worst-case risk with a KL divergence ambiguity set can be bounded as follows:

where ¯ ℓ W ' 1 m ř i Pr m s ℓ W p x p i q q and C ą 0 is constant independent of n .

Consider

Define ε max : ' max p ? ε, ε q . Therefore, we find that

Similar to the Wasserstein robust risk, we observe that the following results hold for any Q P A D ε p ˜ P m q .

With probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q , we have

With probability at least 1 ´ 2 | S c r || S r | exp p´ mt 2 2 ρ 2 max | S r | 2 q ,

With probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 2 | S r | 2 q ,

With probability at least 1 ´ 2 | S r | 2 exp p´ mt 2 p Λ min p H S r S r qq 2 32 | S r | 3 q ´ 2 | S r | 2 exp p´ m p Λ min p H S r S r qq 2 8 | S r | 2 q and ε max ď C min p t Λ min p H S r S r q 8 | S r | ? | S r | , Λ min p H S r S r q 16 | S r | 1 2 q ,

With probability at least 1 ´ O p exp p´ Cm ρ 2 max | S r | 3 ` log | S c r | ` log | S r |qq and ε max ď C ρ max | S r | 3 { 2 ,

where C only depends on α , Λ min p H S r S r q .

Thanks to the boundedness of the error term e , we have similar conclusions to Lemma 12 if ε max ď µ σ holds.

In such wise, the properties in Theorem 9 hold with the same condition on λ ˚ B and the condition on ε max that ε max ď Cµ σρ max ρ 3 { 2 r n s . Since we set ε ' ε 0 m and define ε max : ' max p ? ε, ε q , the condition on ε max implies that

Dataset

n

m

Noise

ζ

Wass

KL

Reg

MMPC

GRASP

Wass+HC

KL+HC

Reg+HC

MMPC+HC

GRASP+HC

HC

asia

8

1000

Noisefree

0

0

.

7800

:

0

.

7285

:

0

.

7897

:

0

.

9067

0

.

8167

0

.

5123

0

.

6367

0

.

5743

0

.

6667

0

.

6583

0

.

6550

asia

8

1000

Huber

0

.

2

0

.

7333

:

0

.

7124

:

0

.

7297

:

0

.

5468

0

.

6570

0

.

3943

0

.

3724

0

.

3487

0

.

2907

0

.

3664

0

.

2183

asia

8

1000

Independent

0

.

2

0

.

6933

0

.

6797

0

.

6868

0

.

6359

0

.

3632

:

0

.

2676

0

.

2632

0

.

2581

0

.

2469

0

.

1794

0

.

2443

cancer

5

1000

Noisefree

0

1

.

0000

:

1

.

0000

:

1

.

0000

:

0

.

6133

0

.

6133

0

.

2800

0

.

2800

0

.

2800

0

.

2800

0

.

2800

0

.

2800

cancer

5

1000

Huber

0

.

5

0

.

9156

:

0

.

8933

:

0

.

9092

:

0

.

6133

0

.

5357

0

.

4333

0

.

3833

0

.

4143

0

.

2589

0

.

2714

0

.

2589

cancer

5

1000

Independent

0

.

2

0

.

9048

:

0

.

9029

:

0

.

8992

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

earthquake

5

1000

Noisefree

0

0

.

8447

:

0

.

9333

:

0

.

9778

1

.

0000

0

.

9778

0

.

2000

0

.

2500

0

.

2500

0

.

2500

0

.

2500

0

.

2278

:

earthquake

5

1000

Huber

0

.

2

0

.

7509

:

0

.

7509

:

0

.

7509

:

0

.

5978

0

.

6583

:

0

.

4618

0

.

4618

0

.

4618

0

.

3860

0

.

4547

0

.

3860

earthquake

5

1000

Independent

0

.

2

0

.

6786

:

0

.

6350

:

0

.

6350

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

sachs

11

1000

Noisefree

0

0

.

8357

:

0

.

8402

:

0

.

8374

:

0

.

9697

0

.

7678

:

0

.

4310

:

0

.

4535

:

0

.

4641

:

0

.

5935

0

.

4112

:

0

.

5873

sachs

11

1000

Huber

0

.

2

0

.

7765

0

.

8064

0

.

7893

0

.

7498

0

.

5663

:

0

.

5194

0

.

4815

0

.

4520

0

.

4736

0

.

2380

0

.

5028

sachs

11

1000

Independent

0

.

5

0

.

5268

:

0

.

5208

:

0

.

5172

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

survey

6

1000

Noisefree

0

0

.

6596

0

.

6545

0

.

6506

0

.

6533

0

.

1714

:

0

.

1789

0

.

1789

0

.

1789

0

.

1789

0

.

0571

0

.

1789

survey

6

1000

Huber

0

.

2

0

.

7303

:

0

.

6778

:

0

.

7095

:

0

.

5396

0

.

3810

0

.

1444

0

.

1444

0

.

1444

0

.

1444

0

.

1516

0

.

1444

survey

6

1000

Independent

0

.

2

0

.

6311

:

0

.

6705

:

0

.

6220

:

0

.

2032

0

.

0000

:

0

.

1071

0

.

1071

0

.

1143

0

.

1071

0

.

0000

0

.

1071

alarm

37

1000

Noisefree

0

0

.

4750

:

0

.

7863

:

0

.

8042

:

0

.

8530

0

.

6824

:

0

.

3483

:

0

.

4949

:

0

.

4470

:

0

.

5635

0

.

4976

0

.

4494

:

alarm

37

1000

Huber

0

.

2

0

.

1432

:

0

.

1619

:

0

.

6571

:

0

.

5486

0

.

1945

:

0

.

2192

0

.

1680

:

0

.

3148

0

.

2774

0

.

2092

:

0

.

2582

alarm

37

1000

Independent

0

.

2

0

.

1419

:

0

.

1448

:

0

.

5458

:

0

.

4309

0

.

2830

:

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

0

.

0000

barley

48

1000

Noisefree

0

0

.

1521

:

0

.

2632

:

0

.

4913

:

0

.

5847

0

.

5636

0

.

1995

0

.

1970

:

0

.

2503

0

.

2510

0

.

2245

0

.

2526

barley

48

1000

Huber

0

.

2

0

.

1452

:

0

.

1592

:

0

.

4027

:

0

.

4522

0

.

4000

:

0

.

1396

0

.

1151

0

.

1658

0

.

1463

0

.

1530

0

.

1685

barley

48

1000

Independent

0

.

2

0

.

1463

:

0

.

1501

:

0

.

2767

:

0

.

4273

0

.

4923

:

0

.

0598

0

.

0769

0

.

0838

0

.

0727

0

.

0840

0

.

0838

voting

17

216

Noisefree

0

N/A

N/A

N/A

N/A

N/A

´

2451

.

8631

´

2453

.

2737

´

2453

.

4091

´

2475

.

5799

´

2482

.

3835

´

2456

.

1489

voting

17

216

Huber

0

.

2

N/A

N/A

N/A

N/A

N/A

´

4418

.

9731

´

4418

.

9731

´

4487

.

4544

´

4450

.

3941

´

4445

.

0175

´

4418

.

9731

voting

17

216

Independent

0

.

2

N/A

N/A

N/A

N/A

N/A

´

4453

.

8298

´

4453

.

8298

´

4522

.

5521

´

4465

.

1076

´

4473

.

8612

´

4453

.

8298

backache

32

90

Noisefree

0

N/A

N/A

N/A

N/A

N/A

´

1729

.

8364

´

1726

.

8465

´

1710

.

7248

´

1719

.

5002

´

1713

.

7583

´

1729

.

7991

backache

32

90

Huber

0

.

2

N/A

N/A

N/A

N/A

N/A

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

´

3186

.

5001

backache

32

90

Independent

0

.

2

N/A

N/A

N/A

N/A

N/A

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

´

2800

.

9386

connect-4_6000

43

6000

Noisefree

0

N/A

N/A

N/A

N/A

N/A

´

38956

.

4300

´

38956

.

4300

´

38954

.

9501

´

39004

.

8512

´

39933

.

6041

:

´

38956

.

4300

connect-4_6000

43

6000

Huber

0

.

2

N/A

N/A

N/A

N/A

N/A

´

99616

.

2848

´

99616

.

2848

´

102878

.

2766

´

99673

.

5320

´

100212

.

9773

´

99616

.

2848

connect-4_6000

43

6000

Independent

0

.

2

N/A

N/A

N/A

N/A

N/A

´

107403

.

2543

´

107403

.

2543

´

107403

.

2543

´

107403

.

2543

´

107403

.

2543

´

107403

.

2543

The final sample complexity becomes

D More Empirical Results

Table 2 lists the complete experimental results.

Peer reviews:
comments: SUMMARY:
The authors give novel algorithms to learn the structure of a Bayes net from noisy data.

STRENGTHS:
Originality:
Work seems original.

Quality:
Quality is high, writing-wise and content-wise; see questions.

Clarity:
Writing is clear; see questions.

Significance:
Topic is significant.

WEAKNESSES:
No significant issues detected; see questions.

QUESTIONS:
Line 21:
What do you mean by "ordinal?"

Line 69 to Line 70:
Unclear.

Line 127:
Can you please elaborate on faithfulness?

Line 135:
Please define "unweighted effects."

Can you please explain Equation (2)?

Can you please justify Assumption 2?

Can you please explain Equation (5)?

Proof of Theorem 9:
I do not understand the union bound argument.

Line 304:
Why is it always possible to find such a solution?

Line 313 to Line 314:
Unclear.

Figure 1:
Can you please add some short *further* explanation in the caption?

Experiments look OK!

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose methods for learning Bayesian network skeletons based on discrete data when the data is corrupted by arbitrary noise.\n\nThe authors show guarantees on discovering the correct skeleton in terms of the required sample size. Some empirical experiments are also presented.\n\nSTRENGTHS:\n+ rigorous mathematical derivations\n\nWEAKNESSES:\n- very dense mathematical derivations\n- limited and inconclusive empirical results\n\nQUESTIONS:\nTo be honest, the level of mathematical detail in the paper is too much for me. It's not that I wouldn't be able to refresh my measure theory and calculus but I just can't afford to spend a week on a review. This is of course primarily my problem, but I have reviewed and published at NeurIPS before, and based on my experience, this paper is mathematically too dense for at least 90% of the NeurIPS community. But perhaps this is fine in case the 10% find it relevant and useful -- I just couldn't possibly tell.\n\nIn Sec. 4, explain what exactly you mean my the F1 score: the F1 score is used for binary classification, so I'm guessing you're perhaps looking at presence/absence of each of the edges in the DAG. Please clarify further which methods are proposed in this paper (Wass + KL?) and which are benchmarks (Reg, MMPC, GRASP?) so that it's easier to quickly grasp. \n\nAm I right in that the punchline is Figs 1.b and 1.c? If so, please make it easier for the reader to see this or otherwise they'll miss it. \n\nI take it the results indicate that the proposed methods are otherwise about as good as prior work except that they're better when the failure probability is close to 0.5 (Figs. 1.b and 1.c)? If so, doesn't this require an extremely high rate of failures, to the extent that the data is mostly random noise?\n\ndetailed comments:\n- p. 1 \"DAG structure of a Bayesian network is usually unknown\": citation needed (or maybe just say \"often\")\n- p. 2 \"contains as subgraphs the skeleton\": I'm confused by the plural/singular tenses here: if there's only one skeleton, how is it contained as subgraphs\n- p. 2: \"(fixed-parameter) tractable\": what does this mean?\n- p. 2: \"observational data is commonly contaminated [...]\": again, citation needed\n- p. 2: \"may produce false edges\": may it also miss edges?\n- p. 2: \"do not assume any specific form of conditional distributions\" vs \"set of distributions characterized by certain a priori properties\": aren't these statements contradictory? please elaborate\n- p. 2: \"small amount of samples (high-dimensional) and potential perturbations\": the word 'high-dimensional' is grammatically speaking placed in an odd spot; also, does \"small amount \" refer only to samples or also to perturbations -- please clarify the sentence\n\nLIMITATIONS:\nok", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors revisit the problem of structure learning of Bayesian networks. Here, we are interested to learn the underlying DAG structure that encodes the conditional independence properties of a Bayes net from random samples. A number of algorithms have been proposed to solve this problem over the years which can be broadly classified into two groups: score-based and constraint-based. Despite a lot of research activity over the last several decades the general problem is not yet known to be solvable in polynomial time and several approaches are shown to be NP-hard. \n\nThe authors of this paper start with an approach first proposed by Bank and Honorio in 2020: to reduce this discrete structure learning problem to a continuous regression problem based on encoding schemes and surrogate parameters. The idea is that once cast as a regression problem, techniques from structure learning in continuous models such as regression can be employed. The main contribution of this paper is a distributionally robust optimization (DRO) method that builds upon the aforesaid framework to handle noisy data. In this proposed method, the best structure is searched among a ball of certain radius around the empirical distribution that minimizes the regression cost. Specifically, balls according to the Wasserstein distance and KL divergence are considered. The authors give two theorems for each of the above two distance functions under certain assumptions. They further verify the effectiveness of their methods by experimenting with benchmark real and synthetic datasets.\n\nSTRENGTHS:\nThe problem considered in extremely important from a theoretical and practical standpoint. The approach followed by the authors to reduce it to a regression problem as in the previous work could be promising for certain datasets.\n\nWEAKNESSES:\nA major weakness of the paper is that a number of assumptions are made in order to derive the results, which looks ad-hoc and too restrictive in my opinion. See page 4 of the paper for these assumptions. See also line number 271. \n\nI could not help but notice several vague statements throughout the paper, a few of which are listed below. \n\nLine number 224-233 the NP-hardness argument is very vague. Looks like you are using a brute-force approach, if so, the algorithms wont scale beyond 15 nodes. \n\nLine number 240-241: the citation of Chen and Paschalidis [2018] here is for the first time and sudden. What question are they solving? How is it related to your problem? \n\nUnfortunately, I am not convinced from the experimental results that the proposed algorithms are overall performing better than the existing methods. Many of the entries in table 1 show that other methods are performing superior to the proposed algorithms. \n\nFinally, I found the paper very hard to follow, maybe it reads more complicated than it actually is. The author may try to see whether writing it in a simpler manner with small examples makes the paper more readable or not.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for learning the structure of discrete Bayesian networks from potentially corrupted data. It utilizes distributionally robust optimization and regression, optimizing the worst-case risk over a set of distributions. The approach works with general categorical variables without assuming specific conditions and provides efficient algorithms. Non-asymptotic guarantees for successful structure learning are derived, and numerical experiments validate its effectiveness.\n\nSTRENGTHS:\nOriginality:\nThe paper addresses the challenge of missing data in distributionally robust skeleton learning. This aspect adds a unique and valuable contribution to the field.\n\nQuality:\nThe quality of writing in the paper is commendable. The ideas and concepts are presented effectively, and the paper demonstrates a strong grasp of the subject matter.\n\nClarity:\nThe paper's clarity is notable, as it is organized in a logical manner where each section builds upon the previous ones. The flow of information is well-structured, making it easy for readers to follow the paper's content.\n\nSignificance:\nThe task of building discrete Bayesian networks from potentially corrupted data holds significant importance, as it has wide-ranging applications in various fields. By addressing the challenge of distributionally robust skeleton learning, the paper contributes to the advancement of techniques that can handle data uncertainties and enable more accurate modeling and inference in real-world scenarios. The significance of this research lies in its potential to enhance decision-making processes, predictive modeling, and knowledge discovery across diverse domains.\n\nWEAKNESSES:\n- The author's assertion that they had limited space to relate their work to the existing literature is understandable. However, it is crucial to provide a clear positioning of their own work within the broader research landscape. Emphasizing the novelty and distinguishing features of their approach will strengthen the paper's contribution and demonstrate its unique value.\n\n- The paper lacks clarity on whether cross-validation or hyperparameter tuning was performed during the experiments. It would be beneficial to address this issue explicitly to ensure the robustness of the results. Additionally, to further improve the reliability of the findings, it is recommended to conduct a rigorous statistical test, such as the corrected Student's t-test proposed by Nadeau and Bengio [1], using 15 hold-out folds. Incorporating such statistical analyses will enhance the credibility and validity of the experimental results.\n\n[1] https://link.springer.com/article/10.1023/A:1024068626366\n\nQUESTIONS:\nl 105: How is T defined in A_{ST}?\n\nLIMITATIONS:\n- The proposed method accounts for the effect of outliers by optimizing the worst-case risk over a family of distributions within bounded Wasserstein distance or KL divergence. However, the extent to which the method can effectively handle outliers and their impact on the learned network structure should be further explored and evaluated.\n- The proposed approach is designed for general categorical random variables without assuming faithfulness, an ordinal relationship, or a specific form of conditional distribution. While this makes the approach flexible, it may also limit its applicability to certain domains or types of data. Investigating the generalizability of the approach to a broader range of variable types or distributions could be a potential limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper considers the problem of learning the exact skeleton of general discrete Bayesian networks from potentially corrupted data. The estimator optimizes the worst case risk over a family of distributions within bounded Wasserstein distance and KL divergence to the empirical distribution. Under mild conditions, the paper presents non-asymptotic guarantees for successful structure learning with logarithmic sample complexities for bounded degree graphs. Numerical studies on synthetic and real datasets validates the performance of the estimator.

QUESTIONS:
1) Is it possible to extend the DRO framework to learn the structure of continuous Bayesian networks from corrupted samples. A high level discussion regarding this would shed more light on the limitations of the present contributions.

2) Since no assumptions on the underlying true distribution is made, it is only possible to recover the structure upto a Markov equivalence class. However the proposed approach overcomes this limitation and recovers the true skeleton with high probability. A discussion as to why DRO framework achieves this will be highly appreciated.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-31