SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparisons of F1 scores for benchmark datasets and BIC for real-world datasets (backache, voting). BIC is not applicable to skeletons. The best and runner-up results are marked in bold. Significant differences are marked by  $\dagger$  (paired t-test,  $p < 0.05$ ).
Dataset	n	m	Noise	ζ	Wass	KL	Reg	MMPC	GRASP	Wass+HC	KL+HC	Reg+HC	MMPC+HC	GRASP+HC	HC
asia	8	1000	Noisefree	0	0.7800†	0.7285†	0.7897†	0.9067	0.8167	0.5123	0.6367	0.5743	0.6667	0.6583	0.6550
asia	8	1000	Huber	0.2	0.7333†	0.7124†	0.7297†	0.5468	0.6570	0.3943	0.3724	0.3487	0.2907	0.3664	0.2183
asia	8	1000	Independent	0.2	0.6933	0.6797	0.6868	0.6359	0.3632†	0.2676	0.2632	0.2581	0.2469	0.1794	0.2443
cancer	5	1000	Noisefree	0	1.0000†	1.0000†	1.0000†	0.6133	0.6133	0.2800	0.2800	0.2800	0.2800	0.2800	0.2800
cancer	5	1000	Huber	0.5	0.9156†	0.8933†	0.9092†	0.6133	0.5357	0.4333	0.3833	0.4143	0.2589	0.2714	0.2589
cancer	5	1000	Independent	0.2	0.9048†	0.9029†	0.8992†	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
earthquake	5	1000	Noisefree	0	0.8447†	0.9333†	0.9778	1.0000	0.9778	0.2000	0.2500	0.2500	0.2500	0.2500	0.2278†
earthquake	5	1000	Huber	0.2	0.7509†	0.7509†	0.7509†	0.5978	0.6583†	0.4618	0.4618	0.4618	0.3860	0.4547	0.3860
earthquake	5	1000	Independent	0.2	0.6786†	0.6350†	0.6350†	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
sachs	11	1000	Noisefree	0	0.8357†	0.8402†	0.8374†	0.9697	0.7675†	0.4310†	0.4535†	0.4641†	0.5935	0.4112†	0.5873
sachs	11	1000	Huber	0.2	0.7765	0.8064	0.7893	0.7498	0.5663†	0.5194	0.4815	0.4520	0.4736	0.2380	0.5028
sachs	11	1000	Independent	0.5	0.5268†	0.5208†	0.5172†	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
survey	6	1000	Noisefree	0	0.6596	0.6545	0.6506	0.6533	0.1714†	0.1789	0.1789	0.1789	0.1789	0.0571	0.1789
survey	6	1000	Huber	0.2	0.7303†	0.6778†	0.7095†	0.5396	0.3810	0.1444	0.1444	0.1444	0.1444	0.1516	0.1444
survey	6	1000	Independent	0.2	0.6311†	0.6705†	0.6220†	0.2032	0.0000†	0.1071	0.1071	0.1143	0.1071	0.0000	0.1071
alarm	37	1000	Noisefree	0	0.4750†	0.7863†	0.8042†	0.8530	0.6824†	0.3483†	0.4949†	0.4470†	0.5635	0.4976	0.4494†
alarm	37	1000	Huber	0.2	0.1432†	0.1619†	0.6571†	0.5486	0.1945†	0.2192	0.1680†	0.3148	0.2774	0.2092†	0.2582
alarm	37	1000	Independent	0.2	0.1419†	0.1448†	0.5458†	0.4309	0.2830†	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
voting	17	216	Noisefree	0	N/A	N/A	N/A	N/A	N/A	-2451.8631	-2453.2737	-2453.4091	-2475.5799	-2482.3835	-2456.1489
voting	17	216	Huber	0.2	N/A	N/A	N/A	N/A	N/A	-4418.9731	-4418.9731	-4487.4544	-4450.3941	-4445.0175	-4418.9731
voting	17	216	Independent	0.2	N/A	N/A	N/A	N/A	N/A	-4453.8298	-4453.8298	-4522.5521	-4465.1076	-4473.6612	-4453.8298
backache	32	90	Noisefree	0	N/A	N/A	N/A	N/A	N/A	-1729.8364	-1726.8465	-1719.7248	-1719.5002	-1713.7583	-1729.7991
backache	32	90	Huber	0.2	N/A	N/A	N/A	N/A	N/A	-3186.5001	-3186.5001	-3186.5001	-3186.5001	-3186.5001	-3186.5001
backache	32	90	Independent	0.2	N/A	N/A	N/A	N/A	N/A	-2800.9386	-2800.9386	-2800.9386	-2800.9386	-2800.9386	-2800.9386
Figure 1: Skeleton estimation results in F1-score on the earthquake dataset for Wasserstein DRO, KL DRO, regularized linear regression, MMPC and GRaSP. From left to right: (a) noisefree, varying number of samples; (b) Huber with  $\zeta = 0.5$  noisy data, varying number of samples; (c) independent failure, varying  $\zeta$ , the probability of noise; (d) independent failure with  $\zeta = 0.2$ , varying  $\varepsilon$  the ambiguity radius, or  $\tilde{\lambda}$  the regularization coefficient.
tractable algorithms to compute the estimators. We established the connection between the proposed method and regularization. We derived non-asymptotic bounds polynomial in the number of nodes for successful identification of the true skeleton. The sample complexities become logarithmic for bounded-degree graphs. Empirical results showcased the effectiveness of our methods.
The strength of making no distributional assumptions in our methods inherits from the regularized regression baseline, which is shown to be a special case of DRO. Besides the original benefits in Bank and Honorio [2020], our methods are explicitly robust and able to adjust  $\varepsilon$  to incorporate our uncertainty about the data generating mechanism.
Since we do not make any specific assumptions on the conditional probability distributions or on the corruption models, our methods may not be superior to the approaches proposed to tackle certain family of noises or parametric distributions. In addition, making assumptions such as an ordinal relationship, continuous values or a Poisson structural equation model (SEM) may lead to more efficient algorithms and tighter bounds. In some cases, adopting an identity encoding mapping is sufficient to learn a continuous Bayesian network [Aragam et al., 2015]. Furthermore, it would be interesting to incorporate prior distributional information into design of the ambiguity set for better performance. Another important topic is whether the underlying structure is identifiable in a robust manner [Sankararaman et al., 2022]. Formulating the complete DAG learning problem as one optimization problem may lead to a non-convex problem. Nonetheless, leveraging an adversarial training approach in continuous optimization for Gaussian Bayesian networks and causal discovery is a promising future direction to pursue.
This material is based upon work supported by the National Science Foundation under Grant No. 1652530.
Pradeep Ravikumar, Martin J Wainwright, and John D Lafferty. High-dimensional ising model selection using  $\ell_1$ -regularized logistic regression. The Annals of Statistics, 38(3):1287-1319, 2010.
Karthik A Sankararaman, Anand Louis, and Navin Goyal. Robust identifiability in linear structural equation models of causal inference. In Uncertainty in Artificial Intelligence, pages 1728-1737. PMLR, 2022.
Marco Scutari. Learning bayesian networks with the bnlearn r package. Journal of Statistical Software, 35:1-22, 2010.
Soroosh Shafieezadeh-Abadeh, Daniel Kuhn, and Peyman Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research, 20(103):1-68, 2019.
Ali Shojaie and George Michailidis. Penalized likelihood methods for estimation of sparse high-dimensional directed acyclic graphs. Biometrika, 97(3):519-538, 2010.
Tomi Silander and Petri Myllymäki. A simple approach for finding the globally optimal bayesian network structure. In Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, pages 445–452, 2006.
Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review, 9(1):62-72, 1991.
Peter Spirtes, Christopher Meek, and Thomas Richardson. An algorithm for causal inference in the presence of latent variables and selection bias. Computation, causation, and discovery, 21: 211-252, 1999.
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine learning, 65(1):31-78, 2006.
Caroline Uhler, Garvesh Raskutti, Peter Buhlmann, and Bin Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics, pages 436-463, 2013.
Martin J Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using  $\ell_1$ -constrained quadratic programming (lasso). IEEE transactions on information theory, 55(5): 2183-2202, 2009.
Xiaolu Wang, Yuen-Man Pun, and Anthony Man-Cho So. Distributionally robust graph learning from smooth signals under moment uncertainty. arXiv e-prints, pages arXiv-2105, 2021.
Dennis Wei, Tian Gao, and Yue Yu. Dags with no fears: A closer look at continuous optimization for learning bayesian networks. Advances in Neural Information Processing Systems, 33:3895-3906, 2020.
Adriano V Werhli, Marco Grzegorczyk, and Dirk Husmeier. Comparative evaluation of reverse engineering gene regulatory networks with relevance networks, graphical gaussian models and bayesian networks. Bioinformatics, 22(20):2523-2531, 2006.
Yue Yu, Tian Gao, Naiyu Yin, and Qiang Ji. Dags with no curl: An efficient dag structure learning approach. In International Conference on Machine Learning, pages 12156-12166. PMLR, 2021.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
A Algorithms
The pseudo-code of the greedy algorithm for solving Equation (6) in Wasserstein DRO is illustrated in Algorithm 1.
Algorithm 1 Greedy Algorithm for the Wasserstein Worst-case Risk
Input:  $W,\gamma ,\pmb{x}^{(i)}$    
Output: a solution  $\hat{\pmb{x}}$  to Equation (6)   
Initialize  $\hat{\pmb{x}} = \pmb{x}^{(i)}$    
for all  $(j,x_j^t)\in [n]\times \mathcal{C}_j$  do Get a random permutation  $\pi$  over  $[n]$  with  $\pi_1 = j$  for  $k\coloneqq 2$  to  $n$  do  $x_{\pi_j}^t\gets \arg \sup_{x_{\pi_k}^t}\ell_W(x_{\pi_{[k]}}^t) - \gamma \| \mathcal{E}(x_{\pi_{[k]}}^t) - \mathcal{E}(x_{\pi_{[k]}}^{(i)})\|$  end for if  $x^{t}$  yields a greater objective than  $\hat{x}$  then  $\hat{\pmb{x}}\gets \pmb{x}^{t}$  end if   
end for
B Optimization Details
$$
\ell_ {\boldsymbol {W}} (\boldsymbol {X}) := \frac {1}{2} \| \mathcal {E} \left(\boldsymbol {X} _ {r}\right) - \boldsymbol {W} ^ {\intercal} \mathcal {E} \left(\boldsymbol {X} _ {\bar {r}}\right) \| _ {2} ^ {2}.
$$
The Lagrangian dual problem of the Wasserstein DRO problem is
$$
\inf  _ {\boldsymbol {W}, \gamma \geqslant 0} f (\boldsymbol {W}, \gamma) := \gamma \varepsilon + \frac {1}{m} \sum_ {i = 1} ^ {m} \sup  _ {\boldsymbol {x} \in \mathcal {X}} \ell_ {\boldsymbol {W}} (\boldsymbol {x}) - \gamma \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} (\boldsymbol {x} ^ {(i)}) \| _ {1}.
$$
One of its sub-gradients can be computed as
$$
\begin{array}{l} \frac {1}{m} \sum_ {i = 1} ^ {m} \mathcal {E} \left(\hat {\boldsymbol {x}} _ {\bar {r}} ^ {(i)}\right) \mathcal {E} \left(\hat {\boldsymbol {x}} _ {\bar {r}} ^ {(i)}\right) ^ {\intercal} \boldsymbol {W} - \mathcal {E} \left(\hat {\boldsymbol {x}} _ {\bar {r}} ^ {(i)}\right) \mathcal {E} \left(\hat {\boldsymbol {x}} _ {r} ^ {(i)}\right) ^ {\intercal} \in \frac {\partial}{\partial \boldsymbol {W}} f \\ \varepsilon - \frac {1}{m} \sum_ {i = 1} ^ {m} \| \mathcal {E} (\hat {\pmb {x}} ^ {(i)}) - \mathcal {E} (\pmb {x} ^ {(i)}) \| _ {1} \in \frac {\partial}{\partial \gamma} f. \\ \end{array}
$$
For the DRO problem based on the KL divergence:
$$
\inf  _ {\boldsymbol {W}, \gamma > 0} f (\boldsymbol {W}, \gamma) := \gamma \ln \left[ \frac {1}{m} \sum_ {i \in [ m ]} e ^ {\ell \boldsymbol {w} (\boldsymbol {x} ^ {(i)}) / \gamma} \right] + \gamma \varepsilon ,
$$
a sub-gradient of which can be computed as
$$
\begin{array}{l} \frac {\sum_ {i \in [ m ]} e ^ {\ell \boldsymbol {W} (\boldsymbol {x} ^ {(i)}) / \gamma} \left(\mathcal {E} \left(\boldsymbol {x} _ {\bar {r}} ^ {(i)}\right) \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}} ^ {(i)}\right) ^ {\intercal} \boldsymbol {W} - \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}} ^ {(i)}\right) \mathcal {E} \left(\boldsymbol {x} _ {r} ^ {(i)}\right) ^ {\intercal}\right)}{\sum_ {i \in [ m ]} e ^ {\ell \boldsymbol {W} (\boldsymbol {x} ^ {(i)}) / \gamma}} \in \frac {\partial}{\partial \boldsymbol {W}} f \\ \ln (\frac {1}{m} \sum_ {i \in [ m ]} e ^ {\ell \boldsymbol {W} (\boldsymbol {x} ^ {(i)}) / \gamma}) - \frac {\sum_ {i \in [ m ]} e ^ {\ell \boldsymbol {W} (\boldsymbol {x} ^ {(i)}) / \gamma} \cdot \ell \boldsymbol {W} (\boldsymbol {x} ^ {(i)})}{\gamma \sum_ {i \in [ m ]} e ^ {\ell \boldsymbol {W} (\boldsymbol {x} ^ {(i)}) / \gamma}} + \epsilon \in \frac {\partial}{\partial \gamma} f. \\ \end{array}
$$
C Technical Proofs
Proposition 11 (NP-hardness of Wasserstein DRO Supremum). The problem in Equation (6) is NP-hard.
Proof. Recall the MAXQP problem:
$$
\sum_ {i, j = 1} ^ {n} a _ {i j} x _ {i} x _ {j}, \quad \text {s . t .} x _ {i} \in \{- 1, 1 \} \forall i.
$$
In Equation (6), let  $\gamma = 0$ ,  $\mathcal{E}(x_{r}) = 0$ ,  $\pmb{x}_{\bar{r}}$  correspond to  $n$  binary variables taking values in  $\{-1,1\}$  and  $\mathcal{E}(\pmb{x}_{\bar{r}}) = \pmb{x}_{\bar{r}}$ . Let  $\pmb{W} \in \mathbb{R}^{n \times n^2}$ . For all  $i,j \in [n]$ , let  $k = (i - 1)n + j$ . The  $k$ -the column of  $\pmb{W}$  satisfies  $W_{ik} = 1$ ,  $W_{jk} = a_{ij}/2$  and  $0$  for the other elements. We have obtained a polynomial-time reduction from an NP-hard problem to Equation (6).
Proposition 5 (Regularization Equivalence). Let  $\ddot{\pmb{W}} := [\pmb{W}; -\pmb{I}_{\rho_r}]^\top \in \mathbb{R}^{\rho_{[n]}\times \rho_r}$  with  $\pmb{W}_r = -\pmb{I}_{\rho_r}$ . If  $\gamma \geqslant \rho_{[n]}\| \ddot{\pmb{W}} \|_F^2$ , the Wasserstein distributionally robust regression problem in Equation (5) is equivalent to
Proof. Recapitulating on Equation (6):
$$
\sup _ {\boldsymbol {x} \in \mathcal {X}} \frac {1}{2} \| \mathcal {E} (\boldsymbol {x} _ {r}) - \boldsymbol {W} ^ {\intercal} \mathcal {E} (\boldsymbol {x} _ {\bar {r}}) \| _ {2} ^ {2} - \gamma \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} (\boldsymbol {x} ^ {(i)}) \| _ {1}.
$$
$$
\begin{array}{l} \left\| \mathcal {E} \left(x _ {r}\right) - \boldsymbol {W} ^ {\intercal} \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}}\right) \right\| _ {2} ^ {2} \triangleq \left\| \ddot {\boldsymbol {W}} ^ {\intercal} \mathcal {E} \left(\boldsymbol {x} _ {[ n ]}\right) \right\| _ {2} ^ {2} \\ \leqslant \| \ddot {\boldsymbol {W}} ^ {\intercal} \| _ {\infty , 2} ^ {2} \\ \leqslant \| \ddot {W} \| _ {1, 2} ^ {2} \\ \leqslant \rho_ {[ n ]} \| \ddot {\boldsymbol {W}} \| _ {F} ^ {2} \\ \leqslant \gamma . \\ \end{array}
$$
Therefore, for any  $\pmb{x} \neq \pmb{x}^{(i)}$ ,
$$
\begin{array}{l} \frac {1}{2} \| \mathcal {E} (x _ {r}) - \boldsymbol {W} ^ {\top} \mathcal {E} (\boldsymbol {x} _ {\bar {r}}) \| _ {2} ^ {2} - \gamma \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} (\boldsymbol {x} ^ {(i)}) \| _ {1} - \left(\frac {1}{2} \| \mathcal {E} (x _ {r} ^ {(i)}) - \boldsymbol {W} ^ {\top} \mathcal {E} (\boldsymbol {x} _ {\bar {r}} ^ {(i)}) \| _ {2} ^ {2} - \gamma \| \mathcal {E} (\boldsymbol {x} ^ {(i)}) - \mathcal {E} (\boldsymbol {x} ^ {(i)}) \| _ {1}\right) \\ \leqslant \frac {1}{2} \left(\| \mathcal {E} \left(x _ {r}\right) - \boldsymbol {W} ^ {\top} \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}}\right) \| _ {2} ^ {2} - \| \mathcal {E} \left(x _ {r} ^ {(i)}\right) - \boldsymbol {W} ^ {\top} \mathcal {E} \left(\boldsymbol {x} _ {\bar {r}} ^ {(i)}\right) \| _ {2} ^ {2}\right) - \gamma \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} \left(\boldsymbol {x} ^ {(i)}\right) \| _ {1} \\ \leqslant \frac {1}{2} (2 \gamma) - \gamma \| \mathcal {E} (\boldsymbol {x}) - \mathcal {E} (\boldsymbol {x} ^ {(i)}) \| _ {1} \\ \leqslant \gamma - \gamma \\ = 0, \\ \end{array}
$$
which implies that the supremum can always be achieved at  $\pmb{x} = \pmb{x}^{(i)}$ . Minimizing over  $\gamma$  leads to
$$
\inf  _ {\boldsymbol {W}} \mathbb {E} _ {\tilde {\mathbb {P}} _ {m}} \frac {1}{2} \| \boldsymbol {\mathcal {E}} (X _ {r}) - \boldsymbol {W} ^ {\intercal} \boldsymbol {\mathcal {E}} (\boldsymbol {X} _ {\bar {r}}) \| _ {2} ^ {2} + \varepsilon \rho_ {[ n ]} \| \ddot {\boldsymbol {W}} \| _ {F} ^ {2}.
$$
Lemma 6. Suppose  $\Xi$  is separable Banach space and fix  $\mathbb{P}_0\in \mathcal{P}(\Xi ')$  for some  $\Xi^{\prime}\subseteq \Xi$ . Suppose  $c:\Xi \to \mathbb{R}_{\geqslant 0}$  is closed convex,  $k$ -positively homogeneous. Suppose  $f:\Xi \rightarrow \mathcal{V}$  is a mapping in the Lebesgue space of functions with finite first-order moment under  $\mathbb{P}_0$  and upper semi-continuous with finite Lipschitz constant  $\operatorname {lip}_c(f)$ . Then for all  $\varepsilon \geqslant 0$ , the following inequality holds with probability 1:
$$
\sup_{\mathbb{Q}\in \mathcal{A}_{\varepsilon}^{W_{p}}(\mathbb{P}_{0}),\mathbb{Q}\in \mathcal{P}(\Xi^{\prime})}\int f(\xi^{\prime})\mathbb{Q}(\mathrm{d}\xi^{\prime})\leqslant \varepsilon l i p_{c}(f) + \int f(\xi^{\prime})\mathbb{P}_{0}(\mathrm{d}\xi^{\prime}).
$$
Proof. The result follows directly from Theorem 1 in Cranko et al. [2021]:
$$
\sup_{\mathbb{Q}\in \mathcal{A}^{W_{p}}_{\varepsilon}(\mathbb{P}_{0}),\mathbb{Q}\in \mathcal{P}(\Xi)}\int f(\xi)\mathbb{Q}(\mathrm{d}\xi)\leqslant \varepsilon \operatorname{lip}_{c}(f) + \int f(\xi^{\prime})\mathbb{P}_{0}(\mathrm{d}\xi^{\prime}).
$$
Since  $\Xi' \subseteq \Xi$ , observe
$$
\sup_{\mathbb{Q}\in \mathcal{A}_{\varepsilon}^{W_{p}}(\mathbb{P}_{0}),\mathbb{Q}\in \mathcal{P}(\Xi^{\prime})}\int f(\xi^{\prime})\mathbb{Q}(\mathrm{d}\xi^{\prime})\leqslant \sup_{\mathbb{Q}\in \mathcal{A}_{\varepsilon}^{W_{p}}(\mathbb{P}_{0}),\mathbb{Q}\in \mathcal{P}(\Xi)}\int f(\xi)\mathbb{Q}(\mathrm{d}\xi).
$$
Lemma 7. If Assumption 3 holds, for any  $\mathbb{Q} \in \mathcal{A}_{\varepsilon}^{W_p}(\tilde{\mathbb{P}}_m)$ , with probability at least  $1 - 2|\mathcal{S}_r|^2 \exp \left(-\frac{mt^2}{2|\mathcal{S}_r|^2}\right)$ , we have
$$
\Lambda_ {m i n} \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}\right) \geqslant \Lambda_ {m i n} \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) - 4 \varepsilon | \mathcal {S} _ {r} | ^ {\frac {1}{2}} - t.
$$
Proof. The minimum eigenvalue of the true covariance matrix  $H_{\mathcal{S}_r\mathcal{S}_r}$  satisfies:
$$
\begin{array}{l} \Lambda_ {\min } \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) \triangleq \min  _ {\| \boldsymbol {v} \| _ {2} = 1} \boldsymbol {v} ^ {\intercal} \boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} \boldsymbol {v} \\ = \min  _ {\| \boldsymbol {v} \| _ {2} = 1} \boldsymbol {v} ^ {\intercal} \boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}} \boldsymbol {v} + \boldsymbol {v} ^ {\intercal} \left(\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}\right) \boldsymbol {v} + \boldsymbol {v} ^ {\intercal} \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) \boldsymbol {v} \\ \leqslant \Lambda_ {\min} (\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}) + \boldsymbol {u} ^ {\intercal} (\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}) \boldsymbol {u} + \boldsymbol {u} ^ {\intercal} (\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}) \boldsymbol {u}, \\ \end{array}
$$
where  $\| \pmb{u}\| _2 = 1$  is an eigenvector of  $H_{\mathcal{S}_r\mathcal{S}_r}^{\mathbb{Q}}$  with minimum eigenvalue.
Therefore,  $\Lambda_{\min}(\pmb{H}_{\mathcal{S}_r\mathcal{S}_r}^{\mathbb{Q}})$  can be lower bounded as follows:
$$
\begin{array}{l} \Lambda_ {\min } \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}\right) \geqslant \Lambda_ {\min } \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) - \boldsymbol {u} ^ {\intercal} \left(\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}\right) \boldsymbol {u} - \boldsymbol {u} ^ {\intercal} \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) \boldsymbol {u} \\ \geqslant \Lambda_ {\min} (\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}) - | \boldsymbol {u} ^ {\intercal} (\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}) \boldsymbol {u} | - \| (\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}) \| _ {F}, \\ \end{array}
$$
due to the fact that
$$
\boldsymbol {u} ^ {\intercal} \boldsymbol {H} \boldsymbol {u} \leqslant \Lambda_ {\max } (\boldsymbol {H}) \leqslant \sqrt {\sum_ {i} (\Lambda_ {i} (\boldsymbol {H})) ^ {2}} \leqslant \| \boldsymbol {H} \| _ {2, 2}.
$$
We can obtain an upper bound on  $|\pmb{u}^{\intercal}(\tilde{\pmb{H}}_{\mathcal{S}_r\mathcal{S}_r} - \pmb{H}_{\mathcal{S}_r\mathcal{S}_r}^{\mathbb{Q}})\pmb{u}|$  based on Lemma 6:
$$
\left| \boldsymbol {u} ^ {\intercal} \left(\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} - \boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}\right) \boldsymbol {u} \right| \leqslant 4 | \mathcal {S} _ {r} | ^ {\frac {1}{2}} \varepsilon ,
$$
because for function  $g(\mathcal{E}(\pmb{x})) \coloneqq \pmb{u}^{\top}\pmb{H}_{\mathcal{S}_r\mathcal{S}_r}\pmb{u}$ , it can be shown that for any  $\| \mathcal{E}(\pmb{x}) - \mathcal{E}(\pmb{x}')\| _1 = k$  and some  $|\mathcal{S}| = k$ ,
$$
| g (\mathcal {E} (\boldsymbol {x})) - g (\mathcal {E} (\boldsymbol {x} ^ {\prime})) | \leqslant \sum_ {k \in \mathcal {S}} \sum_ {i \in \mathcal {S} _ {r}} | H _ {i k} - H _ {i k} ^ {\prime} | u _ {i} u _ {k} + | H _ {k i} - H _ {k i} ^ {\prime} | u _ {k} u _ {i} \leqslant 4 k | \mathcal {S} _ {r} | ^ {\frac {1}{2}}.
$$
Recall that we assume that the encoding schemes take values in  $\mathcal{B} = \{-1,0,1\}$ . Therefore  $\mathrm{lip}_c(g) = 4|\mathcal{S}_r|^{\frac{1}{2}}$ .
We derive an upper bound of  $\| (\pmb{H}_{\mathcal{S}_r\mathcal{S}_r} - \tilde{\pmb{H}}_{\mathcal{S}_r\mathcal{S}_r})\| _F$  as follows. Consider a random variable and its expectation
$$
\begin{array}{l} Z _ {i j} := (\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}) _ {i j} = \frac {1}{m} \sum_ {l = 1} ^ {m} \mathcal {E} (\boldsymbol {x} _ {\bar {r}} ^ {(l)}) _ {i} \mathcal {E} (\boldsymbol {x} _ {\bar {r}} ^ {(l)}) _ {j} \in [ - 1 / m, 1 / m ] \\ \mathbb {E} _ {\mathbb {P}} Z _ {i j} = \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) _ {i j}. \\ \end{array}
$$
By Hoeffding's inequality, we observe
$$
\operatorname {P r o b} \left(\left| \left(\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) _ {i j} - \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) _ {i j} \right| \geqslant t\right) \leqslant 2 \exp \left(- \frac {m t ^ {2}}{2}\right),
$$
for  $t > 0$ . Setting  $t = \frac{t}{|\mathcal{S}_r|}$  for all  $i, j \in \mathcal{S}_r$  and applying the union bound,
$$
\operatorname {P r o b} \left(\| \left(\tilde {\boldsymbol {H}} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) - \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) \| _ {F} \geqslant t\right) \leqslant 2 \left| \mathcal {S} _ {r} \right| ^ {2} \exp \left(- \frac {m t ^ {2}}{2 \left| \mathcal {S} _ {r} \right| ^ {2}}\right). \tag {8}
$$
To conclude, with probability at least  $1 - 2|\mathcal{S}_r|^2\exp \left(-\frac{mt^2}{2|\mathcal{S}_r|^2}\right)$ , we have
$$
\Lambda_ {\min } \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}\right) \geqslant \Lambda_ {\min } \left(\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}}\right) - 4 \varepsilon | \mathcal {S} _ {r} | ^ {\frac {1}{2}} - t.
$$
Lemma 8. If Assumption 3 and Assumption 4 hold, for any  $\mathbb{Q} \in \mathcal{A}_{\varepsilon}^{W_p}(\tilde{\mathbb{P}}_m)$  and  $\alpha \in (0,1]$ , with probability at least  $1 - \mathcal{O}\left(\exp \left(-\frac{Cm}{\rho_{max}^2|\mathcal{S}_r|^3} + \log |\mathcal{S}_r^c| + \log |\mathcal{S}_r|\right)\right)$  and  $\varepsilon \leqslant \frac{C}{\rho_{max}|\mathcal{S}_r|^{3/2}}$ ,
$$
\| \boldsymbol {H} _ {\mathcal {S} _ {r} ^ {c} \mathcal {S} _ {r}} ^ {\mathbb {Q}} (\boldsymbol {H} _ {\mathcal {S} _ {r} \mathcal {S} _ {r}} ^ {\mathbb {Q}}) ^ {- 1} \| _ {B, 1, \infty} \leqslant 1 - \frac {\alpha}{2},
$$
where  $C$  only depends on  $\alpha$ ,  $\Lambda_{min}(\pmb{H}_{S_r S_r})$ .
Proof. We would like to obtain an upper bound for  $\| H_{\mathcal{S}_r^c\mathcal{S}_r}^{\mathbb{Q}}(H_{\mathcal{S}_r\mathcal{S}_r}^{\mathbb{Q}})^{-1}\|_{B,1,\infty}$ . We may write