SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yeshu Li*
Alibaba Group
liyeshu.lys@alibaba-inc.com
Brian D. Ziebart
bziebart@uic.edu
We consider the problem of learning the exact skeleton of general discrete Bayesian networks from potentially corrupted data. Building on distributionally robust optimization and a regression approach, we propose to optimize the most adverse risk over a family of distributions within bounded Wasserstein distance or KL divergence to the empirical distribution. The worst-case risk accounts for the effect of outliers. The proposed approach applies for general categorical random variables without assuming faithfulness, an ordinal relationship or a specific form of conditional distribution. We present efficient algorithms and show the proposed methods are closely related to the standard regularized regression approach. Under mild assumptions, we derive non-asymptotic guarantees for successful structure learning with logarithmic sample complexities for bounded-degree graphs. Numerical study on synthetic and real datasets validates the effectiveness of our method.
A Bayesian network is a prominent class of probabilistic graphical models that encodes the conditional dependencies among variables with a directed acyclic graph (DAG). It provides a mathematical framework for formally understanding the interaction among variables of interest, together with computationally attractive factorization for modeling multivariate distributions. If we impose causal relationships on the edges between variables, the model becomes a causal Bayesian network that encodes the more informative causation. Without such interpretation, a Bayesian network serves as a dependency graph for factorization of a multivariate distribution. We focus on discrete Bayesian networks with purely categorical random variables that are not ordinal, but will discuss related work on both discrete and continuous Bayesian networks for completeness.
The DAG structure of a Bayesian network is typically unknown in practice [Natori et al., 2017, Kitson et al., 2023]. Structure learning is therefore an important task that infers the structure from data. The score-based approach defines a scoring function that measures the goodness-of-fit of each structure and aims to find an optimal DAG that maximizes the score. Unfortunately, the resulting combinatorial optimization problem is known to be NP-hard [Chickering et al., 2004] without distributional assumptions. Representative approaches include those based on heuristic search [Chickering, 2002], dynamic programming [Silander and Myllymäki, 2006], integer linear programming [Jaakkola et al., 2010] or continuous optimization [Zheng et al., 2018], which either yields an approximate solution or an exact solution in worst-case exponential time. The constraint-based approach [Spirtes and Glymour, 1991, Spirtes et al., 1999, Colombo et al., 2014] performs conditional independence tests to determine the existence and directionality of edges. The time complexity is, however, exponential with the maximum in-degree. Furthermore, the independence test results may be unreliable or inconsistent with the true distribution because of finite samples or
*Work done when Yeshu was a PhD student at UIC.
even corrupted samples. In general, without interventional data or assumptions on the underlying distribution, we can only identify a Markov equivalence class (MEC) the true DAG belongs to from observational data where DAGs in the MEC are Markov equivalent, that is, encoding the same set of conditional independencies.
A super-structure is an undirected graph that contains the skeleton as a subgraph which removes directionality from the true DAG. It has been shown that a given super-structure possibly reduces the search space or the number of independence tests to be performed. For example, exact structure learning of Bayesian networks may be (fixed-parameter) tractable [Downey and Fellows, 1995] if the super-structure satisfies certain graph-theoretic properties such as bounded tree-width [Korhonen and Parviainen, 2013, Loh and Buhlmann, 2014], bounded maximum degree [Ordyniak and Szeider, 2013] and the feedback edge number [Ganian and Korchemna, 2021]. An incomplete super-structure with missing edges also helps improve the learned DAG with a post-processing hill-climbing method [Tsamardinos et al., 2006, Perrier et al., 2008]. Furthermore, a combination of a skeleton and a variable ordering determines a unique DAG structure. Learning the exact skeleton rather than a rough super-structure is desirable in Bayesian network structure learning.
Spirtes and Glymour [1991], Tsamardinos et al. [2006] make use of independence tests to estimate the skeleton. Loh and Buhlmann [2014] learn a super-structure called moralized graph via graphical lasso [Friedman et al., 2008]. Shojaie and Michailidis [2010] learn the skeleton assuming an ordering of variables. Bank and Honorio [2020] leverage linear regression for skeleton recovery in polynomial time. These methods either rely on independence test results, which are unstable, or a regularized empirical risk minimization problem, where regularization is usually heuristically chosen to combat overfitting. In practice, the observational data is commonly contaminated by sensor failure, transmission error or adversarial perturbation [Lorch et al., 2022, Sankararaman et al., 2022, Kitson et al., 2023]. Sometimes only a small amount of data is available for learning. As a result, the existing algorithms are vulnerable to such distributional uncertainty and may produce false or missing edges in the estimated skeleton.
In this paper, we propose a distributionally robust optimization (DRO) method [Rahimian and Mehrotra, 2019] that solves a node-wise multivariate regression problem [Bank and Honorio, 2020] for skeleton learning of general discrete Bayesian networks to overcome the above limitations. We do not assume any specific form of conditional distributions. We take into account the settings with a small sample size and potential perturbations, which makes the true data generating distribution highly uncertain. Our method explicitly models the uncertainty by constructing an ambiguity set of distributions characterized by certain a priori properties of the true distribution. The optimal parameter is learned by minimizing the worst-case expected loss over all the distributions within the ambiguity set so that it performs uniformly well on all the considered distributions. The ambiguity set is usually defined in such a way that it includes all the distributions close to the empirical distribution in terms of some divergence. With an appropriately chosen divergence measure, the set contains the true distribution with high probability. Hence the worst-case risk can be interpreted as an upper confidence bound of the true risk. The fact that a discrete Bayesian network encompasses an exponential number of states may pose a challenge to solve the DRO problem. We develop efficient algorithms for problems with ambiguity sets defined by Wasserstein distances and Kullback-Leibler (KL) divergences. We show that a group regularized regression method is a special case of our approach. We study statistical guarantees of the proposed estimators such as sample complexities. Experimental results on synthetic and real-world datasets contaminated by various perturbations validate the superior performance of the proposed methods.
Bayesian networks have been widely adopted in a number of applications such as gene regulatory networks [Werhli et al., 2006], medical decision making [Kyrimi et al., 2020] and spam filtering [Manjusha and Kumar, 2010].
In addition to the score-based structure learning methods and constraint-based methods discussed in the introduction section, there are a third class of hybrid algorithms leveraging constraint-based methods to restrict the search space of a score-based method [Tsamardinos et al., 2006, Gasse et al., 2014, Nandy et al., 2018]. There is also a flurry of work on score-based methods based on neural networks and continuous optimization [Zheng et al., 2018, Wei et al., 2020, Ng et al., 2020, Yu et al., 2021, Ng et al., 2022, Gao et al., 2022], motivated by differentiable characterization of acyclicity
without rigorous theoretical guarantees. We refer the interested readers to survey papers [Drton and Maathuis, 2017, Heinze-Deml et al., 2018, Constantinou et al., 2021] for a more thorough introduction of DAG structure learning and causal discovery methods.
Recently, there is an emerging line of work proposing polynomial-time algorithms for DAG learning [Park and Raskutti, 2017, Ghoshal and Honorio, 2017, 2018, Chen et al., 2019, Bank and Honorio, 2020, Gao et al., 2020, Rajendran et al., 2021], among which Bank and Honorio [2020] particularly focuses on general discrete Bayesian networks without resorting to independence tests.
Learning a super-structure can be done by independence tests, graphical lasso or regression, as discussed in introduction. Given a super-structure, how to determine the orientation has been studied by Perrier et al. [2008], Ordyniak and Szeider [2013], Korhonen and Parviainen [2013], Loh and Buhlmann [2014], Ng et al. [2021], Ganian and Korchemna [2021].
DRO is a powerful framework emerging from operations research [Delage and Ye, 2010, Blanchet and Murthy, 2019, Shafieezadeh-Abadeh et al., 2019, Duchi and Namkoong, 2019] and has seen effective applications in many graph learning problems such as inverse covariance estimation [Nguyen et al., 2022], graphical lasso learning [Cisneros-Velarde et al., 2020], graph Laplacian learning [Wang et al., 2021], Markov random field (MRF) parameter learning [Fathony et al., 2018], MRF structure learning [Li et al., 2022] and causal inference [Bertsimas et al., 2022].
We introduce necessary background and a baseline method for skeleton learning of Bayesian networks.
We refer to  $[n]$  as the index set  $\{1,2,\ldots ,n\}$ . For a vector  $\pmb {x}\in \mathbb{R}^n$ , we use  $x_{i}$  for its  $i$ -th element and  $\pmb{x}_{S}$  for the subset of elements indexed by  $\mathcal{S}\subseteq [n]$  with  $\bar{i}\triangleq [n]\backslash \{i\}$ . For a matrix  $\pmb {A}\in \mathbb{R}^{n\times m}$ , we use  $A_{ij},A_{i}$  and  $A_{j}$  to denote its  $(i,j)$ -th entry,  $i$ -th row and  $j$ -th column respectively.  $\pmb{A}_{\mathcal{ST}}$  represents the submatrix of  $\pmb{A}$  with rows restricted to  $\mathcal{S}$  and columns restricted to  $\mathcal{T}\subseteq [m]$ . We define a row-partitioned block matrix as  $\pmb {A}\triangleq [\pmb {A}_1\pmb {A}_2\dots \pmb {A}_k]^{\intercal}\in \mathbb{R}^{\sum_i n_i\times m}$  where  $\pmb {A}_i\in \mathbb{R}^{n_i\times m}$ . The  $\ell_p$ -norm of a vector  $\pmb{x}$  is defined as  $\| x\| _p\coloneqq (\sum_i|x_i|^p)^{1 / p}$  with  $|\cdot |$  being the absolute value function. The  $\ell_{p,q}$  norm of a matrix  $\pmb{A}$  is defined as  $\| A\|_{p,q}\coloneqq (\sum_j\| A_{\cdot j}\| _p^q)^{1 / q}$ . When  $p = q = 2$ , it becomes the Frobenius norm  $\| \cdot \| _F$ . The operator norm is written as  $\| A\|_{p,q}\coloneqq \sup_{\| \boldsymbol {v}\| _p = 1}\| A\boldsymbol {v}\| _q$ . The block matrix norm is defined as  $\| A\|_{B,p,q}\coloneqq (\sum_{i = 1}^{k}\| A_{i}\|_{p}^{q})^{1 / q}$ . The inner product of two matrices is designated by  $\langle A,B\rangle \triangleq \mathrm{Tr}[A^{\intercal}B]$  where  $A^{\intercal}$  is the transpose of  $A$ . Denote by  $\otimes$  the tensor product operation. With a slight abuse of notation,  $|\mathcal{S}|$  stands for the cardinality of a set  $S$ . We denote by 0 (0) a vector or matrix of all ones (zeros). Given a distribution  $\mathbb{P}$  on  $\Xi$ , we denote by  $\mathbb{E}_{\mathbb{P}}$  the expectation under  $\mathbb{P}$ . The least  $c$ -Lipschitz constant of a function  $f:\Xi \to \mathbb{R}$  with a metric  $c:\Xi \times \Xi \to \mathbb{R}$  is written as  $\mathrm{lip}_c(f)\coloneqq \inf \Lambda_c(f)$  where  $\Lambda_c(f)\coloneqq \{\lambda >0:\forall \xi_1,\xi_2\in \Xi |f(\xi_1) - f(\xi_2)|\leqslant \lambda c(\xi_1,\xi_2)\}$ .
2.2 Bayesian Network Skeleton Learning
Let  $\mathbb{P}$  be a discrete joint probability distribution on  $n$  categorical random variables  $\mathcal{V} := \{X_1, X_2, \ldots, X_n\}$ . Let  $\mathcal{G} := (\mathcal{V}, \mathcal{E}_{\mathrm{true}})$  be a DAG with edge set  $\mathcal{E}_{\mathrm{true}}$ . We use  $X_i$  to represent the  $i$ -th random variable or node interchangeably. We call  $(\mathcal{G}, \mathbb{P})$  a Bayesian network if it satisfies the Markov condition, i.e., each variable  $X_r$  is independent of any subset of its non-descendants conditioned on its parents  $\mathbf{P}\mathbf{a}_r$ . We denote the children of  $X_r$  by  $\mathbf{Ch}_r$ , its neighbors by  $\mathbf{Ne}_r := \mathbf{P}\mathbf{a}_r \cup \mathbf{Ch}_r$  and the complement by  $\mathbf{Co}_r := [n] - \mathbf{Ne}_r - \{r\}$ . The joint probability distribution can thus be factorized in terms of local conditional distributions:
$$
\mathbb {P} (\boldsymbol {X}) = \mathbb {P} (X _ {1}, X _ {2}, \dots , X _ {n}) \triangleq \prod_ {i = 1} ^ {n} \mathbb {P} (X _ {i} | \mathbf {P a} _ {i}).
$$
Let  $\mathcal{G}_{\mathrm{skel}} \coloneqq (\mathcal{V}, \mathcal{E}_{\mathrm{skel}})$  be the undirected graph that removes directionality from  $\mathcal{G}$ . Given  $m$  samples  $\{\pmb{x}^{(i)}\}_{i=1}^{m}$  drawn i.i.d. from  $\mathbb{P}$ , the goal of skeleton learning is to estimate  $\mathcal{G}_{\mathrm{skel}}$  from the samples.
We do not assume faithfulness [Spirtes et al., 2000] or any specific parametric form for the conditional distributions. The distribution is faithful to a graph if all (conditional) independencies that hold true
in the distribution are entailed by the graph, which is commonly violated in practice [Uhler et al., 2013, Mabrouk et al., 2014]. The unavailability of a true model entails a substitute model. Bank and Honorio [2020] propose such a model based on encoding schemes and surrogate parameters.
Assume that each variable  $X_{r}$  takes values from a finite set  $\mathcal{C}_r$  with cardinality  $|\mathcal{C}_r| > 1$ . For an indexing set  $\mathcal{S} \subseteq [n]$ , define  $\rho_{\mathcal{S}} \coloneqq \sum_{i \in \mathcal{S}} |\mathcal{C}_i| - 1$  and  $\rho_{\mathcal{S}}^{+} \coloneqq \sum_{i \in \mathcal{S}} |\mathcal{C}_i|$ . The maximum cardinality minus one is defined as  $\rho_{\max} \coloneqq \max_{i \in [n]} |\mathcal{C}_i| - 1$ . Let  $\mathcal{S}_r \coloneqq \bigcup_{i \in \mathbf{N}_{\mathcal{R}_r}} \{\rho_{[i-1]} + 1, \ldots, \rho_{[i]}\}$  be indices for  $\mathbf{N}_{\mathcal{R}}$  in  $\rho_{[n]}$  and its complement by  $\mathcal{S}_r^c \coloneqq [\rho_{[n]}] - \mathcal{S}_r - \{\rho_{[r-1]} + 1, \ldots, \rho_{[r]}\}$ . Let  $\mathcal{E}: \mathcal{C}_r \to \mathcal{B}^{\rho_r}$  be an encoding mapping with a bounded and countable set  $\mathcal{B} \subset \mathbb{R}$ . We adopt encoding schemes with  $\mathcal{B} = \{-1, 0, 1\}$  such as dummy encoding and unweighted effects encoding which satisfy a linear independence condition. With a little abuse of notation, we reuse  $\mathcal{E}$  for encoding any  $X_{r}$  and denote by  $\mathcal{E}(X_{\mathcal{S}}) \in \mathcal{B}^{\rho_{\mathcal{S}}}$  the concatenation of the encoded vectors  $\{\mathcal{E}(X_i)\}_{i \in \mathcal{S}}$ . Consider a linear structural equation model for each  $X_{r}$ :  $\mathcal{E}(X_{r}) = W^{*^{\intercal}}\mathcal{E}(X_{\bar{r}}) + e$ , where  $W^{*} \triangleq [W_1^* \cdots W_{r-1}^* W_{r+1}^* \cdots W_n^*]^{\intercal} \in \mathbb{R}^{\rho_{\bar{r}} \times \rho_r}$  with  $W_i^* \in \mathbb{R}^{\rho_i \times \rho_r}$  is a surrogate parameter matrix and  $e \in \mathbb{R}^{\rho_r}$  is a vector of errors not necessarily independent of other quantities. A natural choice of a fixed  $W^*$  is the solution to the following problem given knowledge of the true Bayesian network:
$$
\boldsymbol {W} ^ {*} \in \arg \inf  _ {\boldsymbol {W}} \frac {1}{2} \mathbb {E} _ {\mathbb {P}} \| \mathcal {E} \left(\boldsymbol {X} _ {r}\right) - \boldsymbol {W} ^ {\intercal} \mathcal {E} \left(\boldsymbol {X} _ {\bar {r}}\right) \| _ {2} ^ {2} \quad \text {s . t .} \quad \boldsymbol {W} _ {i} = \boldsymbol {0} \quad \forall i \in \mathbf {C o} _ {r}. \tag {1}
$$
Therefore  $\boldsymbol{W}^{*} = (\boldsymbol{W}_{\mathcal{S}_r}^*;0)$  with  $\boldsymbol{W}_{\mathcal{S}_r}^* = \mathbb{E}_{\mathbb{P}}[\mathcal{E}(\boldsymbol{X}_{\bar{r}})_{\mathcal{S}_r}\mathcal{E}(\boldsymbol{X}_{\bar{r}})_{\mathcal{S}_r}^{\intercal}]^{-1}\mathbb{E}_{\mathbb{P}}[\mathcal{E}(\boldsymbol{X}_{\bar{r}})_{\mathcal{S}_r}\mathcal{E}(\boldsymbol{X}_r)^{\intercal}]$  is the optimal solution by the first-order optimality condition assuming that  $\mathbb{E}_{\mathbb{P}}[\mathcal{E}(\boldsymbol{X}_{\bar{r}})_{\mathcal{S}_r}\mathcal{E}(\boldsymbol{X}_{\bar{r}})_{\mathcal{S}_r}^{\intercal}]$  is invertible. The expression of  $\boldsymbol{W}_{\mathcal{S}_r}^*$  captures the intuitions that neighbor nodes should be highly related to the current node  $r$  while the interaction among neighbor nodes should be weak for them to be distinguishable. We further assume that the errors are bounded:
Assumption 1 (Bounded error). For the error vector,  $\| e \|_{\infty} \leqslant \sigma$  and  $\| \mathbb{E}_{\mathbb{P}}[|e|] \|_{\infty} \leqslant \mu$ .
Note that the true distribution does not have to follow a linear structural equation model. Equation (1) only serves as a surrogate model to find technical conditions for successful skeleton learning, which will be discussed in a moment.
The surrogate model under the true distribution indicates that  $\| \mathbf{W}_i^*\|_{2,2} > 0\Rightarrow X_i\in \mathbf{N}\mathbf{e}_r$ . This suggests a regularized empirical risk minimization (ERM) problem to estimate  $\mathbf{W}^{*}$ :
$$
\tilde {\boldsymbol {W}} \in \arg \inf  _ {\boldsymbol {W}} \tilde {L} (\boldsymbol {W}) := \frac {1}{2} \mathbb {E} _ {\tilde {\mathbb {P}} _ {m}} \| \mathcal {E} (X _ {r}) - \boldsymbol {W} ^ {\intercal} \mathcal {E} (\boldsymbol {X} _ {\bar {r}}) \| _ {2} ^ {2} + \tilde {\lambda} \| \boldsymbol {W} \| _ {B, 2, 1}, \tag {2}
$$
where  $\tilde{\lambda} > 0$  is a regularization coefficient, the block  $\ell_{2,1}$  norm is adopted to induce sparsity and  $\tilde{\mathbb{P}}_m \coloneqq \frac{1}{m} \sum_{i=1}^{m} \delta_{\boldsymbol{x}^{(i)}}$  stands for the empirical distribution with  $\delta_{\boldsymbol{x}^{(i)}}$  being the Dirac point measure at  $\boldsymbol{x}^{(i)}$ . This approach is expected to succeed as long as only neighbor nodes have a non-trivial impact on the current node, namely,  $\| \boldsymbol{W}_i^* \|_{2,2} > 0 \iff X_i \in \mathbf{Ne}_r$ .
Define the risk of some  $W$  under a distribution  $\hat{\mathbb{P}}$  as
$$
R ^ {\tilde {\mathbb {P}}} (\boldsymbol {W}) := \mathbb {E} _ {\mathbb {P}} \ell_ {\boldsymbol {W}} (\boldsymbol {X}) := \mathbb {E} _ {\mathbb {P}} \frac {1}{2} \| \boldsymbol {\mathcal {E}} (X _ {r}) - \boldsymbol {W} ^ {\intercal} \boldsymbol {\mathcal {E}} (\boldsymbol {X} _ {\bar {r}}) \| _ {2} ^ {2},
$$
where  $\ell_{\pmb{W}}(\cdot)$  is the squared loss function. The Hessian of the empirical risk  $R^{\mathbb{P}_m}(\pmb {W})$  is a block diagonal matrix  $\nabla^{2}R^{\tilde{\mathbb{P}}_{m}}(\pmb {W})\triangleq \tilde{\pmb{H}}\otimes \pmb{I}_{\rho_{r}}\in \mathbb{R}^{\rho_{r}\rho_{\bar{r}}\times \rho_{r}\rho_{\bar{r}}}$ , where  $\tilde{H} := \mathbb{E}_{\tilde{\mathbb{P}}_m}[\mathcal{E}(\pmb {X}_{\bar{r}})\mathcal{E}(\pmb {X}_{\bar{r}})^{\intercal}]\in$ $\mathbb{R}^{\rho_{\bar{r}}\times \rho_{\bar{r}}}$  and  $\pmb{I}_{\rho_r}\in \mathbb{R}^{\rho_r\times \rho_r}$  is the identity matrix of dimension  $\rho_{r}$ . Similarly under the true distribution,  $\pmb {H}\coloneqq \mathbb{E}_{\mathbb{P}}[\mathcal{E}(\pmb {X}_{\bar{r}})\mathcal{E}(\pmb {X}_{\bar{r}})^{\intercal}]$ . As a result,  $\pmb{H}$  is independent of the surrogate parameters  $\pmb{W}^{*}$  thus conditions on the Hessian translate to conditions on a matrix of cross-moments of encodings, which only depend on the encoding function  $\mathcal{E}$  and  $\mathbb{P}$ .
In order for this baseline method to work, we make the following assumptions.
Assumption 2 (Minimum weight). For each node  $r$ , the minimum norm of the true weight matrix  $W^{*}$  for neighbor nodes is lower bounded:  $\min_{i\in \mathbf{Ne}_r}\| W_i\| _F\geqslant \beta >0$
Assumption 3 (Positive definiteness of the Hessian). For each node  $r$ ,  $H_{\mathcal{S}_r\mathcal{S}_r} > 0$ , or equivalently,  $\Lambda_{\min}(H_{\mathcal{S}_r\mathcal{S}_r}) \geqslant \Lambda > 0$  where  $\Lambda_{\min}(\cdot)$  denotes the minimum eigenvalue.
2If there are four variables, dummy encoding may adopt  $\{(1,0,0),(0,1,0),(0,0,1),(0,0,0)\}$  whereas unweighted effects encoding may adopt  $\{(1,0,0),(0,1,0),(0,0,1),(-1,-1,-1)\}$  as encoding vectors.
Assumption 4 (Mutual incoherence). For each node  $r$ ,  $\| H_{\mathcal{S}_r^c\mathcal{S}_r}H_{\mathcal{S}_r\mathcal{S}_r}^{-1}\|_{B,1,\infty}\leqslant 1 - \alpha$  for some  $0 < \alpha \leqslant 1$ .
Assumption 2 guarantees that the influence of neighbor nodes is significant in terms of a non-zero value bounded away from zero, otherwise they will be indistinguishable from those with zero weight. Assumption 3 ensures that Equation (2) yields a unique solution. Assumption 4 is a widely adopted assumption that controls the impact of non-neighbor nodes on  $r$  [Wainwright, 2009, Ravikumar et al., 2010, Daneshmand et al., 2014]. One interpretation is that the rows of  $H_{S_r S_r}$  should be nearly orthogonal to the rows of  $H_{S_r S_r}$ . Bank and Honorio [2020] show that these assumptions hold for common encoding schemes and finite-sample settings with high probability under mild conditions. They also show that incoherence is more commonly satisfied for the neighbors than the Markov blanket, which justifies the significance of skeleton learning.
Finally, we take the union of all the learned neighbor nodes for each  $r \in [n]$  by solving Equation (2) to get the estimated skeleton  $\tilde{\mathcal{G}} := (\mathcal{V}, \tilde{\mathcal{E}}_{\mathrm{skel}})$ .
As noted in Bank and Honorio [2020], due to model misspecification, even in the infinite sample setting, there is possible discrepancy between the ERM minimizer  $\tilde{W}$  and the true solution  $W^{*}$ , resulting in false or missing edges. In the high-dimensional setting ( $m < n$ ) or the adversarial setting, this issue becomes more serious due to limited knowledge about the data-generating mechanism  $\mathbb{P}$ .
In this section, we attempt to leverage a DRO framework to incorporate distributional uncertainty into the estimation process. We present efficient algorithms and study the theoretical guarantees of our methods. All technical proofs are deferred to the supplementary materials.
3.1 Basic Formulation
Let  $\mathcal{X}$  be a measurable space of all states of the Bayesian network  $(\mathcal{G},\mathbb{P})$ , i.e.,  $X\in \mathcal{X}$ . Let  $\mathcal{P}(\mathcal{X})$  be the space of all Borel probability measures on  $\mathcal{X}$ . Denote by  $\mathcal{X}^{\mathcal{E}}\coloneqq \{\mathcal{E}(\boldsymbol {X}):\forall \boldsymbol {X}\in \mathcal{X}\}$  the space of all the allowed encodings.
Instead of minimizing the empirical risk and relying on regularization, we seek a distributionally robust estimator that optimizes the worst-case risk over an ambiguity set of distributions:
$$
\hat {\boldsymbol {W}} \in \arg \inf  _ {\boldsymbol {W}} \sup  _ {\mathbb {Q} \in \mathcal {A}} \frac {1}{2} \mathbb {E} _ {\mathbb {Q}} \| \mathcal {E} \left(\boldsymbol {X} _ {r}\right) - \boldsymbol {W} ^ {\intercal} \mathcal {E} \left(\boldsymbol {X} _ {\bar {r}}\right) \| _ {2} ^ {2}, \tag {3}
$$