SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}, \quad s _ {i, s, a} \stackrel {i. i. d} {\sim} P ^ {0} (\cdot \mid s, a), \quad i = 1, 2, \dots , N. \tag {9}
$$
The total sample size therefore is  $NSA$ .
Goal. Given the collected samples, the task is to learn the robust optimal policy for the RMDP with some uncertainty set  $\mathcal{U}_{\rho}^{\sigma}(P^{0})$  around the nominal kernel accurately using as few samples as possible. Specifically, given some accuracy level  $\varepsilon > 0$ , the goal is to seek an  $\varepsilon$ -optimal robust policy  $\widehat{\pi}$  obeying  $V^{\star,\sigma}(s) - V^{\widehat{\pi},\sigma}(s) \leq \varepsilon$  for all  $s \in S$ .
3 Model-based algorithm: distributionally robust value iteration
We consider a model-based strategy, which first constructs an empirical nominal transition kernel based on the collected samples, and then applies distributionally robust value iteration (DRVI) to recover the optimal robust policy.
Empirical nominal kernel. The empirical nominal transition kernel  $\widehat{P}^0\in \mathbb{R}^{SA\times S}$  can be constructed using the empirical frequency of visits, i.e.
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}: \quad \widehat {P} ^ {0} \left(s ^ {\prime} \mid s, a\right) := \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} \left\{s _ {i, s, a} = s ^ {\prime} \right\}, \tag {10}
$$
which leads to an empirical RMDP  $\widehat{\mathcal{M}}_{\mathrm{rob}} = \{\mathcal{S},\mathcal{A},\mathcal{U}_{\rho}^{\sigma}(\hat{P}^{0}),\gamma ,r\}$ . Analogously, we can define the corresponding robust value function (resp. robust Q-function) of policy  $\pi$  in  $\widehat{\mathcal{M}}_{\mathrm{rob}}$  as  $\hat{V}^{\pi ,\sigma}$  (resp.  $\hat{Q}^{\pi ,\sigma}$ ) (cf. (5)). In addition, we denote the corresponding optimal robust policy as  $\widehat{\pi}^{\star}$  and the optimal robust value function (resp. optimal robust  $Q$  -function) as  $\widehat{V}^{\star ,\sigma}$  (resp.  $\widehat{Q}^{\star ,\sigma}$ ) (cf. (5)), which satisfies the robust Bellman optimality equation:
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}: \quad \widehat {Q} ^ {\star , \sigma} (s, a) = r (s, a) + \gamma \inf  _ {\mathcal {P} \in \mathcal {U} _ {\rho} ^ {\sigma} \left(\widehat {P} _ {s, a} ^ {0}\right)} \mathcal {P} \widehat {V} ^ {\star , \sigma}. \tag {11}
$$
Algorithm 1: Distributionally robust value iteration (DRV) for infinite-horizon RMDPs.
1 input: empirical nominal transition kernel  $\widehat{P}^0$ ; reward function  $r$ ; uncertainty level  $\sigma$ ; number of iterations  $T$ .
2 initialization:  $\widehat{Q}_0(s,a) = 0,\widehat{V}_0(s) = 0$  for all  $(s,a)\in S\times \mathcal{A}$
3 for  $t = 1,2,\dots ,T$  do
4 for  $s\in S,a\in \mathcal{A}$  do 5 Set  $\widehat{Q}_t(s,a)$  according to (13); 6 for  $s\in S$  do 7 Set  $\widehat{V}_t(s) = \max_a\widehat{Q}_t(s,a)$
$\mathbf{s}$  output:  $\widehat{Q}_T, \widehat{V}_T$  and  $\widehat{\pi}$  obeying  $\widehat{\pi}(s) \coloneqq \arg \max_a \widehat{Q}_T(s, a)$ .
Equipped with  $\widehat{P}^0$ , define the empirical robust Bellman operator  $\widehat{\mathcal{T}}^\sigma$  as
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}: \widehat {T} ^ {\sigma} (Q) (s, a) := r (s, a) + \gamma \inf  _ {\mathcal {P} \in \mathcal {U} _ {\rho} ^ {\sigma} (\widehat {P} _ {s, a} ^ {0})} \mathcal {P} V, \text {w i t h} V (s) := \max  _ {a} Q (s, a). \tag {12}
$$
DRVl: distributionally robust value iteration. To solve for the fixed point of  $\widehat{\mathcal{T}}^{\sigma}$ , we introduce distributionally robust value iteration (DRVl), which is summarized in Algorithm 1. Starting from some initialization  $\widehat{Q}_0 = 0$ , the update rule at the  $t$ -th ( $t \geq 1$ ) step can be formulated as:
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}: \quad \widehat {Q} _ {t} (s, a) = \widehat {\mathcal {T}} ^ {\sigma} \left(\widehat {Q} _ {t - 1}\right) (s, a) = r (s, a) + \gamma \inf  _ {\mathcal {P} \in \mathcal {U} _ {\rho} ^ {\sigma} \left(\widehat {P} _ {s, a} ^ {0}\right)} \mathcal {P} \widehat {V} _ {t - 1}, \tag {13}
$$
where  $\widehat{V}_{t-1}(s) = \max_{a} \widehat{Q}_{t-1}(s, a)$  for all  $s \in S$ . However, directly solving (13) is computationally prohibitive since it involves optimization over an  $S$ -dimensional probability simplex at each iteration, especially when the dimension of the state space  $S$  is prohibitive. Fortunately, in view of strong duality (Iyengar, 2005), (13) can be equivalently solved using its dual problem, which concerns optimizing of a scalar dual variable and thus can be solved efficiently. The specific form of the dual problem depends on the choice of the divergence  $\rho$ , which we discuss in a more detailed version.
4 Theoretical guarantees: sample complexity analyses
We now present our main results, which concern the sample complexities of learning RMDPs when the uncertainty set is specified using the TV distance or the  $\chi^2$  divergence. Surprisingly, the choice of the uncertainty set can lead to dramatic consequence in the sample size requirement.
4.1 The case of TV distance: RMDP is easier than standard MDP
We start with the case when the uncertainty set is measured via the TV distance, where Theorem 1 presents the sample complexity upper bound above which DRVI is able to find an  $\varepsilon$ -optimal robust policy in a small number of iterations; the key challenge of the analysis is to carefully control the robust value function  $V^{\pi ,\sigma}$  as a function of uncertainty level  $\sigma$ .
Theorem 1 (Upper bound using TV distance). Fix the uncertainty set  $\mathcal{U}_{\rho}^{\sigma}(\cdot) = \mathcal{U}_{\mathrm{TV}}^{\sigma}(\cdot)$  using the TV distance in (7). Consider any discount factor  $\gamma \in \left[\frac{1}{4},1\right)$ , uncertainty level  $\sigma \in (0,1)$ , and  $\delta \in (0,1)$ . With probability at least  $1 - \delta$ , the output  $\widehat{\pi}$  from Algorithm 1 with at most  $T = C_1\log \left(N(1 - \gamma)\right)$  iterations yields  $V^{\star ,\sigma}(s) - V^{\widehat{\pi},\sigma}(s)\leq \varepsilon$  for any  $\varepsilon \in \left(0,\sqrt{1 / \max\{1 - \gamma,\sigma\}}\right]$ , as long as the total number of samples obeys NSA  $\geq \frac{C_2SA}{(1 - \gamma)^2\max\{1 - \gamma,\sigma\}\varepsilon^2}\log \left(\frac{SAN}{(1 - \gamma)\delta}\right)$ . Here,  $C_1,C_2 > 0$  are some large enough universal constants.
Before discussing the implications of Theorem 1, we present a matching minimax lower bound that confirms the optimality of the upper bound, which in turn pins down the sample complexity requirement for learning RMDPs with TV distance.
Theorem 2 (Lower bound using TV distance). Consider any tuple  $(S, A, \gamma, \sigma, \varepsilon)$  obeying  $\sigma \in (0, 1 - c_0]$  with  $0 < c_0 \leq \frac{1}{8}$  being any small enough positive constant,  $\gamma \in \left[\frac{1}{2}, 1\right)$ , and  $\varepsilon \in \left(0, \frac{c_0}{256(1 - \gamma)}\right]$ . We
can construct two infinite-horizon RMDPs  $\mathcal{M}_0, \mathcal{M}_1$  defined by the uncertainty set  $\mathcal{U}_{\rho}^{\sigma}(\cdot) = \mathcal{U}_{\mathrm{TV}}^{\sigma}(\cdot)$ , an initial state distribution  $\varphi$ , and a dataset with  $N$  independent samples for each state-action pair over the nominal transition kernel (for  $\mathcal{M}_0$  and  $\mathcal{M}_1$  respectively), such that
$$
\inf  _ {\widehat {\pi}} \max  \left\{\mathbb {P} _ {0} \big (V ^ {\star , \sigma} (\varphi) - V ^ {\widehat {\pi}, \sigma} (\varphi) > \varepsilon \big), \mathbb {P} _ {1} \big (V ^ {\star , \sigma} (\varphi) - V ^ {\widehat {\pi}, \sigma} (\varphi) > \varepsilon \big) \right\} \geq \frac {1}{8},
$$
provided that  $NSA \leq \frac{c_0 SA \log 2}{8192(1 - \gamma)^2 \max\{1 - \gamma, \sigma\} \varepsilon^2}$ . The infimum is taken over all estimators  $\widehat{\pi}$ , and  $\mathbb{P}_0$  (resp.  $\mathbb{P}_1$ ) denotes the probability when the RMDP is  $\mathcal{M}_0$  (resp.  $\mathcal{M}_1$ ).
Below, we interpret the above theorems and highlight several key implications about the sample complexity requirements for learning RMDPs with TV distance.
Near minimax-optimal sample complexity. Theorem 1 shows that the total number of samples required for DRVI to yield  $\varepsilon$ -accuracy is
$$
\widetilde {O} \left(\frac {S A}{(1 - \gamma) ^ {2} \max  \{1 - \gamma , \sigma \} \varepsilon^ {2}}\right). \tag {14}
$$
Taking together with the minimax lower bound asserted in Theorem 2, this confirms the near minimax-optimality of the sample complexity up to some logarithmic factor almost over the full range of the uncertainty level  $\sigma$ , which scales linearly with respect to the size of the state-action space.
RMDPs is easier than standard MDPs with TV distance. Recall that the sample complexity requirement for standard MDP (Agarwal et al., 2020; Li et al., 2020) to yield  $\varepsilon$  accuracy is  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^3\varepsilon^2}\right)$ . Comparing with the sample complexity requirement in (14) for RMDPs with TV distance, this confirms that the latter is at least as easy as—if not easier—than standard MDPs. In particular, when  $\sigma \lesssim 1 - \gamma$  is small, the sample complexity of RMDPs is the same as the standard MDPs, which is expected since the RMDP reduces to the standard MDP when  $\sigma = 0$ . On the other hand, when  $1 - \gamma \lesssim \sigma < 1$ , the sample complexity of RMDPs becomes  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^2\sigma\varepsilon^2}\right)$ , which is smaller than that of standard MDPs by a factor of  $\sigma / (1 - \gamma)$ .
Comparison with state-of-the-art bounds. For the upper bound, our results (cf. Theorem 1) significantly improves over the prior art  $\widetilde{O}\left(\frac{S^2A}{(1 - \gamma)^4\varepsilon^2}\right)$  of Panaganti and Kalathil (2022) by at least a factor of  $\frac{S}{1 - \gamma}$  and even  $\frac{S}{(1 - \gamma)^2}$  when the uncertainty level  $1 - \gamma \lesssim \sigma < 1$  is large. Turning to the lower bound side, Yang et al. (2022) developed a lower bound for RMDPs under the TV distance, which scales as  $\widetilde{O}\left(\frac{SA(1 - \gamma)}{\varepsilon^2} \min \left\{\frac{1}{(1 - \gamma)^4}, \frac{1}{\sigma^4}\right\}\right)$ . Clearly, this is worse than ours by a factor of  $\frac{\sigma^3}{(1 - \gamma)^3} \in \left(1, \frac{1}{(1 - \gamma)^3}\right)$  in the regime where  $1 - \gamma \lesssim \sigma < 1$ .
4.2 The case of  $\chi^2$  distance: RMDP can be harder than standard MDP
We now move onto the case when the uncertainty set is measured via the  $\chi^2$  distance, where Theorem 3 presents the sample complexity upper bound above which DRVI is able to find an  $\varepsilon$ -optimal robust policy in a small number of iterations.
Theorem 3 (Upper bound using  $\chi^2$  distance). Fix the uncertainty set  $\mathcal{U}_{\rho}^{\sigma}(\cdot) = \mathcal{U}_{\chi^2}^{\sigma}(\cdot)$  using the  $\chi^2$  distance in (8). Consider any uncertainty level  $\sigma \in (0,\infty)$ , and  $\delta \in (0,1)$ . With probability at least  $1 - \delta$ , the output policy  $\widehat{\pi}$  from Algorithm 1 with at most  $T = c_{1}\log (N(1 - \gamma))$  iterations yields  $V^{\star ,\sigma}(s) - V^{\widehat{\pi},\sigma}(s)\leq \varepsilon$  for any  $\varepsilon \in \left(0,\frac{1}{1 - \gamma}\right]$ , as long as the total number of samples obeying  $NSA\geq \frac{c_2SA(1 + \sigma)}{(1 - \gamma)^4\varepsilon^2}\log \left(\frac{SAN}{\delta}\right)$ . Here,  $c_{1},c_{2} > 0$  are some large enough universal constants.
In addition, in order to gauge the tightness of Theorem 3, and understand the minimal sample complexity requirement for learning RMDPs with  $\chi^2$  divergence, we further develop a minimax lower bound as follows.
Theorem 4 (Lower bound using  $\chi^2$  divergence). Consider any  $(S, A, \gamma, \sigma, \varepsilon)$  obeying  $\gamma \in [\frac{3}{4}, 1)$ ,  $\sigma \in (0, \infty)$ , and
$$
\varepsilon \leq c _ {3} \left\{ \begin{array}{l l} \frac {1}{1 - \gamma} & i f \sigma \in \left(0, \frac {1 - \gamma}{4}\right), \\ \max  \left\{\frac {1}{\sigma (1 - \gamma)}, 1 \right\} & i f \sigma \in \left[ \frac {1 - \gamma}{4}, \infty\right), \end{array} \right. \tag {15}
$$
for some small universal constant  $c_3 > 0$ . Then we can construct two infinite-horizon RMDPs  $\mathcal{M}_0, \mathcal{M}_1$  defined by the uncertainty set  $\mathcal{U}_{\rho}^{\sigma}(\cdot) = \mathcal{U}_{\chi^2}^{\sigma}(\cdot)$ , an initial state distribution  $\rho$ , and a dataset with  $N$  independent samples for each  $(s,a)$  pair over the nominal transition kernel (for  $\mathcal{M}_0$  and  $\mathcal{M}_1$  respectively), such that
$$
\inf  _ {\hat {\pi}} \max  \left\{\mathbb {P} _ {0} \left(V ^ {\star , \sigma} (\rho) - V ^ {\hat {\pi}, \sigma} (\rho) > \varepsilon\right), \mathbb {P} _ {1} \left(V ^ {\star , \sigma} (\rho) - V ^ {\hat {\pi}, \sigma} (\rho) > \varepsilon\right) \right\} \geq \frac {1}{8}, \tag {16}
$$
provided that the total number of samples
$$
N S A \leq c _ {4} \left\{ \begin{array}{l l} \frac {S A}{(1 - \gamma) ^ {3} \varepsilon^ {2}} & i f \sigma \in \left(0, \frac {1 - \gamma}{4}\right), \\ \frac {\sigma S A}{\min  \left\{1 , (1 - \gamma) ^ {4} (1 + \sigma) ^ {4} \right\} \varepsilon^ {2}} & i f \sigma \in \left[ \frac {1 - \gamma}{4}, \infty\right) \end{array} \right. \tag {17}
$$
for some universal constant  $c_4 > 0$ .
We are now positioned to highlight some key implications of the above theorems about the sample complexity requirements for learning RMDPs with  $\chi^2$  divergence.
Nearly tight sample complexity. To achieve  $\varepsilon$ -accuracy for RMDPs with  $\chi^2$  distance, Theorem 3 shows that a total number of samples on the order of  $\widetilde{O}\left(\frac{SA(1 + \sigma)}{(1 - \gamma)^4\varepsilon^2}\right)$  is sufficient for DRVI. Taking it together with the minimax lower bound in Theorem 4 confirms that the sample complexity is near-optimal up to a polynomial factor of the effective horizon  $1 / (1 - \gamma)$  over the entire range of the uncertainty level  $\sigma$ . In particular, when  $\sigma \asymp 1$ , our sample complexity  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^4\varepsilon^2}\right)$  is tight and matches with the lower bound; when  $\sigma \gtrsim \frac{1}{(1 - \gamma)^3}$ , our sample complexity correctly predicts the linear dependency with  $\sigma$ , suggesting that more samples are needed when one plans for larger  $\chi^2$ -based uncertainty sets.
RMDPs can be much harder than standard MDPs with  $\chi^2$  divergence. The minimax lower bound developed in Theorem 4 exhibits a surprising non-monotonic behavior of the sample size requirement over the entire range of the uncertainty level  $\sigma \in (0,\infty)$  when the uncertainty set is measured via the  $\chi^2$  divergence. When  $\sigma \lesssim 1 - \gamma$ , the lower bound reduces to  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^3\varepsilon^2}\right)$ , which matches with that of standard MDPs, as  $\sigma = 0$  corresponds to standard MDP. However, two additional regimes are worth calling out:
$$
1 - \gamma \lesssim \sigma \lesssim \frac {1}{(1 - \gamma) ^ {1 / 3}}: \widetilde {O} \left(\frac {S A}{(1 - \gamma) ^ {4} \varepsilon^ {2}} \min \left\{\sigma , \frac {1}{\sigma^ {3}} \right\}\right), \mathrm {a n d} \sigma \gtrsim \frac {1}{(1 - \gamma) ^ {3}}: \widetilde {O} \left(\frac {S A \sigma}{\varepsilon^ {2}}\right),
$$
both of which are greater than that of standard MDPs, indicating learning RMDPs with  $\chi^2$  divergence can be much harder.
Comparison with state-of-the-art bounds. Our upper bound significantly improves over the prior art  $\widetilde{O}\left(\frac{S^2A(1 + \sigma)}{(1 - \gamma)^4\varepsilon^2}\right)$  of Panaganti and Kalathil (2022) by a factor of  $S$ , and provides the first finite-sample complexity that scales linearly with respect to  $S$  for discounted infinite-horizon RMDPs, which typically exhibit more complicated statistical dependencies than the finite-horizon counterpart. On the other hand, Yang et al. (2022) established a lower bound on the order of  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^2\sigma\varepsilon^2}\right)$  when  $\sigma \gtrsim 1 - \gamma$ , which is always smaller than the requirement of standard MDPs, and diminishes when  $\sigma$  grows. Consequently, Yang et al. (2022) does not lead to the rigorous justification that RMDPs can be harder than standard MDPs, nor the correct linear scaling of the sample size when  $\sigma$  grows towards infinity.
Finite-sample guarantees for standard RL. There has been a considerable amount of research into non-asymptotic sample analysis of standard RL for a variety of settings; partial examples include, but are not limited to, the works via probably approximately correct (PAC) bounds for the generative model setting (Kearns and Singh, 1999; Beck and Srikant, 2012; Li et al., 2022a; Chen et al., 2020; Azar et al., 2013; Sidford et al., 2018; Agarwal et al., 2020; Li et al., 2023, 2020; Wainwright, 2019) and the offline setting (Rashidinejad et al., 2021; Xie et al., 2021; Yin et al., 2021; Shi et al., 2022; Li
et al., 2022b; Jin et al., 2021; Yan et al., 2022), as well as the online setting via both regret-based and PAC-base analyses (Jin et al., 2018; Bai et al., 2019; Li et al., 2021; Zhang et al., 2020b; Dong et al., 2019; Jin et al., 2020; Li et al., 2022a; Jafarnia-Jahromi et al., 2020; Yang et al., 2021; Woo et al., 2023).
Robustness in RL. To address the challenges of deployed environment uncertainty, an emerging line of works begin to address robustness of RL algorithms with respect to the uncertainty or perturbation over different components of MDPs — state, action, reward, and the transition kernel; see Moos et al. (2022) for a recent review. Besides the framework of distributionally robust MDPs (RMDPs) (Iyengar, 2005) adopted by this work, to promote robustness in RL, there exist various other works including but not limited to Zhang et al. (2020a, 2021); Han et al. (2022); Qiaoben et al. (2021); Sun et al. (2021); Xiong et al. (2022) investigating the robustness w.r.t. state uncertainty. Besides, Tessler et al. (2019); Tan et al. (2020) considered the robustness w.r.t. the uncertainty of the action, and Ding et al. (2023) tackles robustness against spurious correlations.
Distributionally robust RL. Rooted in the literature of distributionally robust optimization, which has primarily been investigated in the context of supervised learning (Rahimian and Mehrotra, 2019; Gao, 2020; Bertsimas et al., 2018; Duchi and Namkoong, 2018; Blanchet and Murthy, 2019), distributionally robust dynamic programming and RMDPs have attracted considerable attention recently (Wolff et al., 2012; Kaufman and Schaefer, 2013; Ho et al., 2018; Smirnova et al., 2019; Ho et al., 2021; Goyal and Grand-Clement, 2022; Derman and Mannor, 2020; Tamar et al., 2014). In the context of RMDPs, both empirical and theoretical studies have been widely conducted, although most prior theoretical analyses focus on planning with an exact knowledge of the uncertainty set (Iyengar, 2005; Xu and Mannor, 2012; Tamar et al., 2014), or are asymptotic in nature (Roy et al., 2017).
Resorting to the tools of high-dimensional statistics, various recent works begin to shift attention to understand the finite-sample performance of provable robust RL algorithms, under diverse data generating mechanisms and forms of the uncertainty set over the transition kernel. Besides the infinite-horizon setting, finite-sample complexity bounds for RMDPs with the TV distance and the  $\chi^2$  divergence are also developed for the finite-horizon setting in Xu et al. (2023); Dong et al. (2022). In addition, many other forms of uncertainty sets have been considered associated with different divergence function including but not limited to Wasserstein distance, R-contamination, KL divergence, Wang and Zou (2021); Yang et al. (2022); Panaganti and Kalathil (2022); Zhou et al. (2021); Shi and Chi (2022); Xu et al. (2023); Wang et al. (2023a); Blanchet et al. (2023); Liu et al. (2022); Wang et al. (2023c); Liang et al. (2023); Xu et al. (2023); Badrinath and Kalathil (2021); Ramesh et al. (2023); Panaganti et al. (2022); Ma et al. (2022). Moreover, various other related problems or issues have been explored such as the difference of various uncertainty types (Wang et al., 2023b), the iteration complexity of the policy-based methods (Li et al., 2022c; Kumar et al., 2023), the cases when the uncertainty level is instance-dependent small enough (Clavier et al., 2023), and regularization-based robust RL (Yang et al., 2023; Zhang et al., 2023).
This work studies sample complexity bounds for learning RMDPs when the uncertainty set is measured via the TV distance and the  $\chi^2$  divergence under the generative model. Our sample complexity bounds not only strengthen the prior art in both directions of upper and lower bounds, but also reveal new insights on how the additional consideration of distributional robustness fundamentally changes the sample complexity of RL in a surprising manner. It turns out that RMDPs are not necessarily harder nor easier than standard MDPs, but the answer is far more nuanced and highly dependent on both the size and shape of the uncertainty set under consideration. These findings could help to guide the practice of RMDPs, by raising awareness that the choice of the uncertainty set not only represents a preference in robustness, but also influences the statistical complexity of the problem.
The work of L. Shi and Y. Chi is supported in part by the grants ONR N00014-19-1-2404, NSF CCF-2106778, DMS-2134080, and CNS-2148212. L. Shi is also gratefully supported by the Leo Finzi Memorial Fellowship, Wei Shen and Xuehong Zhang Presidential Fellowship, and Liang
Ji-Dian Graduate Fellowship at Carnegie Mellon University. The work of Y. Wei is supported in part by the NSF grants DMS-2147546/2015447, CAREER award DMS-2143215, CCF-2106778, and the Google Research Scholar Award. The work of Y. Chen is supported in part by the Alfred P. Sloan Research Fellowship, the Google Research Scholar Award, the AFOSR grant FA9550-22-1-0198, the ONR grant N00014-22-1-2354, and the NSF grants CCF-2221009 and CCF-1907661. The authors also acknowledge Zuxin Liu and He Wang for valuable discussions.