SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{A} \subseteq \mathcal{P}(\mathcal{X})$  is an ambiguity set typically defined by a nominal probability measure  $\tilde{\mathbb{P}}$  equipped with a discrepancy measure  $\mathrm{div}(\cdot, \cdot)$  for two distributions  $\mathcal{A}_{\varepsilon}^{\mathrm{div}}(\tilde{\mathbb{P}}) \coloneqq \{\mathbb{Q} \in \mathcal{P}(\mathcal{X}) : \mathrm{div}(\mathbb{Q}, \tilde{\mathbb{P}}) \leqslant \varepsilon\}$ , where  $\varepsilon$  is known as the ambiguity radius or size. This way of uncertainty quantification can be interpreted as an adversary that captures out-of-sample effect by making perturbations on samples within some budget  $\varepsilon$ . Some common statistical distances satisfy  $\mathrm{div}(\mathbb{Q}, \mathbb{P}) = 0 \iff \mathbb{Q} = \mathbb{P}$ . In this case, if  $\varepsilon$  is set to zero, Equation (3) reduces to Equation (2) without regularization. We will show that the DRO estimator  $\hat{W}$  can be found efficiently and encompasses attractive statistical properties with a judicious choice of  $\mathcal{A}$ .
3.2 Wasserstein DRO
Wasserstein distances or Kantorovich-Rubinstein metric in optimal transport theory can be interpreted as the cost of the optimal transport plan to move the mass from  $\mathbb{P}$  to  $\mathbb{Q}$  with unit transport cost  $c: \mathcal{X} \times \mathcal{X} \to \mathbb{R}_+$ . Denote by  $\mathcal{P}_p(\mathcal{X})$  the space of all  $\mathbb{P} \in \mathcal{P}(\mathcal{X})$  with finite  $p$ -th moments for  $p \geqslant 1$ . Let  $\mathcal{M}(\mathcal{X}^2)$  be the set of probability measures on the product space  $\mathcal{X} \times \mathcal{X}$ . The  $p$ -Wasserstein distance between two distributions  $\mathbb{P}, \mathbb{Q} \in \mathcal{P}_p(\mathcal{X})$  is defined as  $W_p(\mathbb{P}, \mathbb{Q}) := \inf_{\Pi \in \mathcal{M}(\mathcal{X}^2)} \left\{\left[\int_{\mathcal{X}^2} c^p(\boldsymbol{x}, \boldsymbol{x}') \Pi(\mathrm{d}\boldsymbol{x}, \mathrm{d}\boldsymbol{x}')\right]^{\frac{1}{p}}: \Pi(\mathrm{d}\boldsymbol{x}, \mathcal{X}) = \mathbb{P}(\mathrm{d}\boldsymbol{x}), \Pi(\mathcal{X}, \mathrm{d}\boldsymbol{x}') = \mathbb{Q}(\mathrm{d}\boldsymbol{x}')\right\}$ .
We adopt the Wasserstein distance of order  $p = 1$  as the discrepancy measure, the empirical distribution as the nominal distribution, and cost function  $c(\pmb{x},\pmb{x}^{\prime}) = \| \mathcal{E}(\pmb{x}) - \mathcal{E}(\pmb{x}^{\prime})\|$  for some norm
$\| \cdot \|$  . The primal DRO formulation becomes
$$
\hat {\boldsymbol {W}} \in \arg \inf  _ {\boldsymbol {W}} \sup  _ {\mathbb {Q} \in \mathcal {A} _ {\varepsilon} ^ {W _ {p}} (\tilde {\mathbb {P}} _ {m})} \frac {1}{2} \mathbb {E} _ {\mathbb {Q}} \| \mathcal {E} \left(\boldsymbol {X} _ {r}\right) - \boldsymbol {W} ^ {\intercal} \mathcal {E} \left(\boldsymbol {X} _ {\bar {r}}\right) \| _ {2} ^ {2}. \tag {4}
$$
According to Blanchet and Murthy [2019], the dual problem of Equation (4) can be written as
$$
\inf  _ {\boldsymbol {W}, \gamma \geqslant 0} \gamma \varepsilon + \frac {1}{m} \sum_ {i = 1} ^ {m} \sup  _ {\boldsymbol {x} \in \mathcal {X}} \frac {1}{2} \| \mathcal {E} \left(x _ {r}\right) - \boldsymbol {W} ^ {\intercal} \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}}\right) \| _ {2} ^ {2} - \gamma \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} \left(\boldsymbol {x} ^ {(i)}\right) \|. \tag {5}
$$
Strong duality holds according to Theorem 1 in Gao and Kleywegt [2022]. The inner supremum problems can be solved independently for each  $\pmb{x}^{(i)}$ . Henceforth, we focus on solving it for some  $i \in [m]$ :
$$
\sup  _ {\boldsymbol {x} \in \mathcal {X}} \frac {1}{2} \| \mathcal {E} \left(x _ {r}\right) - \boldsymbol {W} ^ {\top} \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}}\right) \| _ {2} ^ {2} - \gamma \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} \left(\boldsymbol {x} ^ {(i)}\right) \|. \tag {6}
$$
Equation (6) is a supremum of  $|\mathcal{X}|$  convex functions of  $\pmb{W}$ , thus convex. Since  $\mathcal{X}^{\mathcal{E}}$  is a discrete set consisting of a factorial number of points  $(\Pi_{i\in [n]}\rho_i)$ , unlike the regression problem with continuous random variables in Chen and Paschalidis [2018], we may not simplify Equation (6) into a regularization form by leveraging convex conjugate functions because  $\mathcal{X}^{\mathcal{E}}$  is non-convex and not equal to  $\mathbb{R}^{\rho_{[n]}}$ . Moreover, since changing the value of  $x_{j}$  for some  $j\in \bar{r}$  is equivalent to changing  $\pmb{W}^{\top}\pmb {\mathcal{E}}(\pmb{x}_{\bar{r}})$  by a vector, unlike Li et al. [2022] where only a set of discrete labels rather than encodings are dealt with, there may not be a greedy algorithm based on sufficient statistics to find the optimal solution to Equation (6). In fact, let the norm be the  $\ell_1$  norm, we can rewrite Equation (6) by fixing the values of  $\| \mathcal{E}(\pmb {x}) - \mathcal{E}(\pmb{x}^{(i)})\| _1$ :
$$
\sup  _ {\boldsymbol {x} \in \mathcal {X}, 0 \leqslant k \leqslant \rho_ {[ n ]} ^ {+}, \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} \left(\boldsymbol {x} ^ {(i)}\right) \| _ {1} = k} \frac {1}{2} \| \mathcal {E} \left(x _ {r}\right) - \boldsymbol {W} ^ {\intercal} \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}}\right) \| _ {2} ^ {2} - \gamma k. \tag {7}
$$
If we fix  $k$ , Equation (7) is a generalization of the 0-1 quadratic programming problem, which can be transformed into a maximizing quadratic programming (MAXQP) problem. As a result, Equation (6) is an NP-hard problem with proof presented in Proposition 11 in appendix. Charikar and Wirth [2004] develop an algorithm to find an  $\Omega(1/\log n)$  solution based on semi-definite programming (SDP) and sampling for the MAXQP problem. Instead of adopting a similar SDP algorithm with quadratic constraints, we propose a random and greedy algorithm to approximate the optimal solution, which is illustrated in Algorithm 1 in appendix, whose per-iteration time complexity is  $\Theta(n^2 m \rho_{\max})$ . It follows a simple idea that for a random node order  $\pi$ , we select a partial optimal solution sequentially from  $\pi_1$  to  $\pi_n$ . We enumerate the possible states of the first node to reduce uncertainty. In practice, we find that this algorithm always finds the exact solution that is NP-hard to find for random data with  $n \leqslant 12$  and  $\rho_{\max} \leqslant 5$  in most cases.
Since  $\mathcal{X}^{\mathcal{E}}$  is non-convex and not equal to  $\mathbb{R}^{\rho[n]}$ , using convex conjugate functions will not yield exact equivalence between Equation (5) and a regularized ERM problem. However, we can draw such a connection by imposing constraints on the dual variables as shown by the following proposition:
Proposition 5 (Regularization Equivalence). Let  $\ddot{W} := [W; -I_{\rho_r}]^\top \in \mathbb{R}^{\rho_{[n]}\times \rho_r}$  with  $W_{r} = -I_{\rho_{r}}$ . If  $\gamma \geqslant \rho_{[n]} \| \ddot{W} \|_F^2$ , the Wasserstein DRO problem in Equation (5) is equivalent to
$$
\inf  _ {\boldsymbol {W}} \mathbb {E} _ {\tilde {\mathbb {P}} _ {m}} \frac {1}{2} \| \mathcal {E} (X _ {r}) - \boldsymbol {W} ^ {\intercal} \mathcal {E} (\boldsymbol {X} _ {\bar {r}}) \| _ {2} ^ {2} + \varepsilon \rho_ {[ n ]} \| \ddot {\boldsymbol {W}} \| _ {F} ^ {2},
$$
which subsumes a linear regression approach regularized by the Frobenius norm as a special case.
This suggests that minimizing a regularized empirical risk may not be enough to achieve distributional robustness. Note that exact equivalence between DRO and regularized ERM in Chen and Paschalidis [2018] requires  $\mathcal{X}^{\mathcal{E}} = \mathbb{R}^{d}$ .
Now we perform non-asymptotic analysis on the proposed DRO estimator  $\hat{W}$ . First, we would like to show that the solution to the Wasserstein DRO estimator in Equation (4) is unique so that we refer to an estimator unambiguously. Note that Equation (4) is a convex optimization problem but not necessarily strictly convex, and actually never convex in the high-dimensional setting. However, given
a sufficient number of samples, the problem becomes strictly convex and yields a unique solution with high probability. Second, we show that the correct skeleton  $\mathcal{E}_{\mathrm{skel}}$  can be recovered with high probability. This is achieved by showing that, for each node  $X_{r}$ , the estimator has zero weights for non-neighbor nodes  $\mathbf{C o}_r$  and has non-zero weights for its neighbors  $\mathbf{N e}_r$  with high confidence. Before presenting the main results, we note that they are based on several important lemmas.
Lemma 6. Suppose  $\Xi$  is separable Banach space and fix  $\mathbb{P}_0\in \mathcal{P}(\Xi ')$  for some  $\Xi^{\prime}\subseteq \Xi$ . Suppose  $c:\Xi \to \mathbb{R}_{\geqslant 0}$  is closed convex,  $k$ -positively homogeneous. Suppose  $f:\Xi \rightarrow \mathcal{V}$  is a mapping in the Lebesgue space of functions with finite first-order moment under  $\mathbb{P}_0$  and upper semi-continuous with finite Lipschitz constant  $\mathrm{lip}_c(f)$ . Then for all  $\varepsilon \geqslant 0$ , the following inequality holds with probability 1:  $\sup_{\mathbb{Q}\in \mathcal{A}_{\varepsilon}^{W_p}(\mathbb{P}_0),\mathbb{Q}\in \mathcal{P}(\Xi ')}\int f(\xi ') \mathbb{Q}(\mathrm{d}\xi ')\leqslant \varepsilon \mathrm{clip}_c(f) + \int f(\xi ') \mathbb{P}_0(\mathrm{d}\xi ')$ .
Lemma 6 follows directly from Cranko et al. [2021] and allows us to obtain an upper bound between the worst-case risk and empirical risk. It is crucial for the following finite-sample guarantees.
Lemma 7. If Assumption 3 holds, for any  $\mathbb{Q} \in \mathcal{A}_{\varepsilon}^{W_p}(\tilde{\mathbb{P}}_m)$ , with high probability,  $\pmb{H}_{\mathcal{S}_r,\mathcal{S}_r}^{\mathbb{Q}}$  is positive definite.
Lemma 8. If Assumption 3 and Assumption 4 hold, for any  $\mathbb{Q} \in \mathcal{A}_{\varepsilon}^{W_p}(\tilde{\mathbb{P}}_m)$  and  $\alpha \in (0,1]$ , with high probability,
$$
\| \boldsymbol {H} _ {\mathcal {S} _ {r} ^ {c} \mathcal {S} _ {r}} ^ {\mathbb {Q}} (\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}) ^ {- 1} \| _ {B, 1, \infty} \leqslant 1 - \frac {\alpha}{2}.
$$
The above two lemmas illustrate that Assumption 3 and Assumption 4 hold in the finite-sample setting. Let the estimated skeleton, neighbor nodes and the complement be  $\hat{\mathcal{G}} \coloneqq (\mathcal{V}, \hat{\mathcal{E}}_{\mathrm{skel}})$ ,  $\hat{\mathbf{N}}\mathbf{e}_r$  and  $\hat{\mathbf{C}}\mathbf{o}_r$  respectively. We derive the following guarantees for the proposed Wasserstein DRO estimator.
Theorem 9. Given a Bayesian network  $(\mathcal{G},\mathbb{P})$  of  $n$  categorical random variables and its skeleton  $\mathcal{G}_{skel} := (\mathcal{V},\mathcal{E}_{skel})$ . Assume that the condition  $\| \pmb{W}^{*}\|_{B,2,1} \leqslant \bar{B}$  holds for some  $\bar{B} > 0$  associated with an optimal Lagrange multiplier  $\lambda_B^* > 0$  for  $\pmb{W}^*$  defined in Equation (1). Suppose that  $\hat{\pmb{W}}$  is a DRO risk minimizer of Equation (4) with a Wasserstein distance of order 1 and an ambiguity radius  $\varepsilon = \varepsilon_0 / m$  where  $m$  is the number of samples drawn i.i.d. from  $\mathbb{P}$ . Under Assumptions 1, 2, 3, 4, if the number of samples satisfies
$$
m = \mathcal {O} (\frac {C (\varepsilon_ {0} + \log (n / \delta) + \log \rho_ {[ n ]}) \sigma^ {2} \rho_ {m a x} ^ {4} \rho_ {[ n ]} ^ {3}}{\min (\mu^ {2} , 1)},
$$
where  $C$  only depends on  $\alpha$ ,  $\Lambda$ , and if the Lagrange multiplier satisfies
$$
\frac {3 2 \mu \rho_ {m a x}}{\alpha} <   \lambda_ {B} ^ {*} <   \frac {\beta}{(\alpha / (4 - 2 \alpha) + 2) \rho_ {m a x} \sqrt {\rho_ {[ n ]}}} \sqrt {\frac {\Lambda}{4}},
$$
then for any  $\delta \in (0,1]$ ,  $r\in [n]$ , with probability at least  $1 - \delta$ , the following properties hold:
(a) The optimal estimator  $\tilde{W}$  is unique.
(b) All the non-neighbor nodes are excluded:  $\mathbf{C}\mathbf{o}_r\subseteq \hat{\mathbf{C}}\mathbf{o}_r$
(c) All the neighbor nodes are identified:  $\mathbf{N}\mathbf{e}_r\subseteq \hat{\mathbf{N}}\mathbf{e}_r$
(d) The true skeleton is successfully reconstructed:  $\mathcal{G}_{skel} = \hat{\mathcal{G}}_{skel}$ .
Proof sketch. The main idea in the proof follows that in the lasso estimator [Wainwright, 2009]. Based on a primal-dual witness construction method and Lemma 8, it can be shown that if we control  $\lambda_B^*$ , a solution constrained to have zero weight for all the non-neighbor nodes is indeed optimal. Furthermore, Lemma 7 implies that there is a unique solution given information about the true neighbors. The uniqueness of the aforementioned optimal solution without knowing the true skeleton is then verified via convexity and a conjugate formulation of the block  $\ell_{2,1}$  norm. Hereby we have shown that the optimal solution to Equation (4) is unique and excluding all the non-neighbor nodes. Next, we derive conditions on  $\lambda_B^*$  for the estimation bias  $\| \hat{\boldsymbol{W}} - \boldsymbol{W}^* \|_{B,2,\infty} < \beta / 2$  to hold, which allows us to recover all the neighbor nodes. In such manner, applying the union bound over all the nodes  $r \in [n]$  leads to successful exact skeleton discovery with high probability.
The results in Theorem 9 encompass some intuitive interpretations. Compared to Theorem 1 in Bank and Honorio [2020], we make more explicit the relationship among  $m$ ,  $\lambda_B^*$  and  $\delta$ . On one hand, the lower bound of  $\lambda_B^*$  ensures that a sparse solution excluding non-neighbor nodes is obtained. A large error magnitude expectation  $\mu$  therefore elicits stronger regularization. On the other hand, the upper bound  $\lambda_B^*$  is imposed to guarantee that all the neighbor nodes are identified with less restriction on  $W$ . There is naturally a trade-off when choosing  $\bar{B}$  in order to learn the exact skeleton. The sample complexity depends on cardinalities  $\rho_{[n]}$ , confidence level  $\delta$ , the number of nodes  $n$ , the ambiguity level  $\varepsilon_0$  and assumptions on errors. The dependence on  $\sigma$  indicates that higher uncertainty caused by larger error norms demands more samples whereas the dependence on  $\mu^{-2}$  results from the lower bound condition on  $\lambda_B^*$  with respect to  $\mu$ . The ambiguity level is set to  $\varepsilon_0 / m$  based on the observation that obtaining more samples reduces ambiguity of the true distribution. In practice, we find that  $\varepsilon_0$  is usually small thus negligible. Note that the sample complexity is polynomial in  $n$ . Furthermore, if we assume that the true graph has a bounded degree of  $d$ , we find that  $m = O\left(\frac{C(\varepsilon_0 + \log(n / \delta) + \log n + \log\rho_{\max})\sigma^2\rho_{\max}^7d^3}{\min(\mu^2,1)}\right)$  is logarithmic with respect to  $n$ , consistent with the results in Wainwright [2009].
We introduce constants  $\bar{B}$  and  $\lambda_B^*$  in order to find a condition for the statements in Theorem 9 to hold. If there exists a  $W$  incurring a finite loss, we can always find a solution  $\hat{\boldsymbol{W}}$  and let  $\bar{B} \triangleq \max_{\hat{\boldsymbol{W}}} \| \hat{\boldsymbol{W}} \|_{B,2,1}$  be the maximum norm of all solutions. Imposing  $\| \boldsymbol{W} \|_{B,2,1} \leqslant \bar{B}$  is equivalent to the original problem. By Lagrange duality and similar arguments for the lasso estimator, there exists a  $\lambda_B^*$  that finds all the solutions with  $\| \hat{\boldsymbol{W}} \|_{B,2,1} = \bar{B}$ . Therefore we have a mapping between  $\varepsilon$  and  $\lambda_B^*$ .
3.3 Kullback-Leibler DRO
In addition to optimal transport,  $\phi$ -divergence is also widely used to construct an ambiguity set for DRO problems. We consider the following definition of a special  $\phi$ -divergence called the KL divergence:  $D(\mathbb{Q} \parallel \mathbb{P}) \coloneqq \int_{\mathcal{X}} \ln \frac{\mathbb{Q}(\mathrm{d}\boldsymbol{x})}{\mathbb{P}(\mathrm{d}\boldsymbol{x})} \mathbb{Q}(\mathrm{d}\boldsymbol{x})$ , where  $\mathbb{Q} \in \mathcal{P}(\mathcal{X})$  is absolutely continuous with respect to  $\mathbb{P} \in \mathcal{P}(\mathcal{X})$  and  $\frac{\mathbb{Q}(\mathrm{d}\boldsymbol{x})}{\mathbb{P}(\mathrm{d}\boldsymbol{x})}$  denotes the Radon-Nikodym derivative. A noteworthy property of ambiguity sets based on the KL divergence is absolute continuity of all the candidate distributions with respect to the empirical distribution. It implies that if the true distribution is an absolutely continuous probability distribution, the ambiguity set will never include it. In fact, any other point outside the support of the nominal distribution remains to have zero probability. Unlike the Wasserstein metric, the KL divergence does not measure some closeness between two points, nor does it have some measure concentration results. However, we argue that adopting the KL divergence may bring advantages over the Wasserstein distance since the Bayesian network distribution we study is a discrete distribution over purely categorical random variables. Moreover, as illustrated below, adopting the KL divergence leads to better computational efficiency.
Let  $\mathcal{A} \triangleq \mathcal{A}_{\varepsilon}^{D}(\tilde{\mathbb{P}}_{m})$  be the ambiguity set, the dual formulation of Equation (3) follows directly from Theorem 4 in Hu and Hong [2013]:
$$
\inf _ {\boldsymbol {W}, \gamma > 0} \gamma \ln \left[ \frac {1}{m} \sum_ {i \in [ m ]} e ^ {\frac {1}{2} \| \boldsymbol {\mathcal {E}} (\boldsymbol {x} _ {r} ^ {(i)}) - \boldsymbol {W} ^ {\intercal} \boldsymbol {\mathcal {E}} (\boldsymbol {x} _ {\bar {r}} ^ {(i)}) \| _ {2} ^ {2} / \gamma} \right] + \gamma \varepsilon ,
$$
which directly minimizes a convex objective. In contrast to the approximate Wasserstein estimator, this KL DRO estimator finds the exact solution to the primal problem by strong duality.
The worst-case risk over a KL divergence ball can be bounded by variance [Lam, 2019], similar to Lipschitz regularization in Lemma 6. Based on this observation, we derive the following results:
Theorem 10. Suppose that  $\bar{W}$  is a DRO risk minimizer of Equation (4) with the KL divergence and an ambiguity radius  $\varepsilon = \varepsilon_0 / m$ . Given the same definitions of  $(\mathcal{G},\mathbb{P})$ ,  $\mathcal{G}_{skel}$ ,  $\bar{B}$ ,  $\lambda_B^*$ ,  $m$  in Theorem 9. Under Assumptions 1, 2, 3, 4, if the number of samples satisfies
$$
m = \mathcal {O} (\frac {C (\varepsilon_ {0} + \log (n / \delta) + \log \rho_ {[ n ]}) \sigma^ {2} \rho_ {m a x} ^ {4} \rho_ {[ n ]} ^ {3}}{\min (\mu^ {2} , 1)}).
$$
where  $C$  depends on  $\alpha$ ,  $\Lambda$  while independent of  $n$ , and if the Lagrange multiplier satisfies the same condition as in Theorem 9, then for any  $\delta \in (0,1]$ ,  $r \in [n]$ , with probability at least  $1 - \delta$ , the properties (a)-(d) in Theorem 9 hold.
The sample complexities in Theorem 9 and Theorem 10 differ in the constant  $C$  due to the difference between the two probability metrics. Note that  $C$  is independent of  $n$  in both methods. The dependency on  $1 / (\lambda_B^*)^2$  is absorbed in the denominator because we require that  $\lambda_B^* - 16\mu \rho_{\mathrm{max}} / \alpha > 0$ . The sample complexities provide a perspective of our confidence on upper bounding the true risk in terms of the ambiguity radius.  $\varepsilon_0$  serves as our initial guess on distributional uncertainty and increases the sample complexity only slightly because it is usually dominated by other terms in practice:  $\varepsilon \ll \log (n / \delta)$ . Even though the samples are drawn from an adversarial distribution with a proportion of noises, the proposed methods may still succeed as long as the true distribution can be made close to an upper confidence bound.
We conduct experiments on benchmark datasets [Scutari, 2010] and real-world datasets [Malone et al., 2015] perturbed by the following contamination models:
- Noisefree model. This is the baseline model without any noises.
- Huber's contamination model. In this model, each sample has a fixed probability of  $\zeta$  to be replaced by a sample drawn from an arbitrary distribution.
- Independent failure model. Each entry of a sample is independently corrupted with probability  $\zeta$ .
We conduct all experiments on a laptop with an Intel Core i7 2.7 GHz processor. We adopt the proposed approaches based on Wasserstein DRO and KL DRO, the group norm regularization method [Bank and Honorio, 2020], the MMPC algorithm [Tsamardinos et al., 2006] and the GRaSP algorithm [Lam et al., 2022] for skeleton learning. Based on the learned skeletons, we infer a DAG with the hill-climbing (HC) algorithm [Tsamardinos et al., 2006]. For the Wasserstein-based method, we leverage Adam [Kingma and Ba, 2014] to optimize the overall objective with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.990$ , a learning rate of 1.0, a batch size of 500, a maximum of 200 iterations for optimization and 10 iterations for approximating the worst-case distribution. For the KL-based and standard regularization methods, we use the L-BFGS-B [Byrd et al., 1995] optimization method with default parameters. We set the cardinality of the maximum conditional set to 3 in MMPC. The Bayesian information criterion (BIC) [Neath and Cavanaugh, 2012] score is adopted in the HC algorithm. A random mixture of 20 random Bayesian networks serves as the adversarial distribution for both contamination models. All hyper-parameters are chosen based on the best performance on random Bayesian networks with the same size as the input one. Each experimental result is taken as an average over 10 independent runs. When dealing with real-world datasets, we randomly split the data into two halves for training and testing.
We use the F1-score, or the Dice coefficient (regarding the label of each edge indicating its presence as a binary random variable and considering all possible edges), to evaluate performance on benchmark datasets and BIC for real-world datasets. The results are reported in Table 1 and more results can be found in Table 2 in appendix. We observe that in most cases the proposed DRO methods are comparable to MMPC and MMHC, which are generally the best-performing methods in Bank and Honorio [2020]. We illustrate in Figure 1 the results on earthquake by varying the number of samples, corruption level and ambiguity radius or regularization coefficient. Figure 1 (a) suggests that all the methods perfectly recover the true skeleton given more than 2,000 samples. The results in Figure 1 (b-c) indicate that, in highly uncertain settings, Wasserstein DRO as well as KL DRO is superior to other approaches. Meanwhile, Table 1 and Figure 1 (a) suggest that the DRO methods and the regularized ERM approach are comparable to MMPC and GRaSP when clean data is given. The sensitivity analysis (Figure 1 (d)) suggests a trade-off between robustness and target performance (F1-score in our case). All the approaches have similar execution time except that Wasserstein DRO is several times slower due to the combinatorial sub-problem of computing the worst-case distribution.
In this paper, we put forward a distributionally robust optimization method to recover the skeleton of a general discrete Bayesian network. We discussed two specific probability metrics, developed
3Our code is publicly available at https://github.com/DanielLeee/drslbn.