SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhuodong Yu $^{1}$  Ling Dai $^{1}$  Shaohang Xu $^{1}$  Siyang Gao $^{2}$  Chin Pang Ho $^{1}$ \
 $^{1}$ School of Data Science, City University of Hong Kong, Hong Kong\
 $^{2}$ Department of Systems Engineering, City University of Hong Kong, Hong Kong\
{zhuodonyu2-c, lingdai5-c, shaohanxu2-c}@my.cityu.edu.hk\
{siyangao, clint ho}@cityu.edu.hk
Markov decision processes (MDPs) often suffer from the sensitivity issue under model ambiguity. In recent years, robust MDPs have emerged as an effective framework to overcome this challenge. Distributionally robust MDPs extend the robust MDP framework by incorporating distributional information of the uncertain model parameters to alleviate the conservative nature of robust MDPs. This paper proposes a computationally efficient solution framework for solving distributionally robust MDPs with Wasserstein ambiguity sets. By exploiting the specific problem structure, the proposed framework decomposes the optimization problems associated with distributionally robust Bellman updates into smaller subproblems, which can be solved efficiently. The overall complexity of the proposed algorithm is quasi-linear in both the numbers of states and actions when the distance metric of the Wasserstein distance is chosen to be  $L_{1}$ ,  $L_{2}$ , or  $L_{\infty}$  norm, and so the computational cost of distributional robustness is substantially reduced. Our numerical experiments demonstrate that the proposed algorithms outperform other state-of-the-art solution methods.
Markov Decision Processes (MDPs) provide a flexible and powerful modeling framework for sequential decision-making problems under uncertainty (Puterman, 2014; Sutton and Barto, 2018). However, the standard MDP model assumes that the exact knowledge of the transition kernel is available, which is not the case for most real-world applications. While these model parameters can be estimated from data, it is well-known that the optimal policy of MDP is sensitive to estimation errors because of the dynamic nature of the problem. In particular, small errors in estimating transition kernels could lead to catastrophic failures in practice (Iyengar, 2005; Nilim and El Ghaoui, 2005; Le Tallec, 2007).
To overcome the aforementioned challenge in model ambiguity, robust MDPs assume that the transition kernels belong to ambiguity sets (Iyengar, 2005; Nilim and El Ghaoui, 2005; Xu and Mannor, 2006). Through optimizing the worst-case performance, robust MDPs compute robust policies to avoid disappointing performance due to the sensitivity issue caused by inaccurate transition kernels. However, robust MDPs are often too conservative because they always adopt the worst possible transition kernels without any consideration of their likelihood. To mitigate this undesirable side effect of robust MDPs, distributionally robust MDPs are proposed to incorporate distributional information of model ambiguity and maximize the expected reward under the most adversarial probability distribution (Xu and Mannor, 2010; Yu and Xu, 2015).
In a distributionally robust MDP, we assume that all the plausible distributions of the transition kernel belong to a prescribed ambiguity set, which could be formalized in different types, such as moment ambiguity sets (Delage and Ye, 2010) and Wasserstein ambiguity sets (Mohajerin Esfahani and
Kuhn, 2018). In this paper, we focus on the Wasserstein ambiguity sets, which have been a popular choice for distributionally robust data-driven optimization in recent years because of their outstanding empirical performance as well as nice theoretical properties, such as consistency in optimality and finite-sample bounds (Mohajerin Esfahani and Kuhn, 2018; Gao and Kleywegt, 2022). By using Wasserstein distance to formulate the ambiguity set, Yang (2017) shows that there exists an optimal policy that is stationary and Markovian for the corresponding Wasserstein distributionally robust MDP.
While (distributionally) robust MDPs can be solved by extending standard solution methods in classical MDPs to their (distributionally) robust counterparts, these solution methods become much more computationally demanding. For example, each Bellman update for (distributionally) robust MDPs can be formulated as a convex optimization problem. Without making use of any specific problem structure, one would need to use generic convex optimization solvers to compute these Bellman updates, which have to be evaluated numerous times for computing the (distributionally) robust value function. This computational challenge restricted the application of (distributionally) robust MDPs to small or medium size of problems. In recent years, many efficient algorithms are proposed for solving robust MDPs to address this issue (Iyengar, 2005; Nilim and El Ghaoui, 2005; Ho et al., 2018; Behzadian et al., 2021; Grand-Clément and Kroer, 2021b; Ho et al., 2022).
On the other hand, however, distributionally robust MDPs have received limited attention on their computational efficiency. To the best of our knowledge, (Grand-Clément and Kroer, 2021a) is the only pioneer work that proposes a first-order method to efficiently solve Wasserstein distributionally robust MDPs. However, being restricted by the nature of first-order methods, the proposed algorithm in (Grand-Clément and Kroer, 2021a) struggles when high accuracy is needed or when the discount factor is close to one, as it would take unsatisfactorily many numbers of iterations to compute the distributionally robust value function.
In this paper, we take an alternative approach to develop fast algorithms for solving Wasserstein distributionally robust MDPs. In particular, we reformulate the optimization problem of the distributionally robust Bellman update to a structural form, and by exploiting the specific problem structure, we propose decomposition schemes in which the problem of interest can be decomposed into smaller subproblems, which can be solved by our customized algorithms in quasi-linear time. As we will show later, the time complexities of the proposed algorithms are linear in the numbers of actions and kernels and quasi-linear in the number of states, which are significantly better than the state-of-the-art solution methods (Xu and Mannor, 2010; Grand-Clément and Kroer, 2021a).
The rest of this paper is organized as follows. Section 2 introduces the related work, and Section 3 describes the setting of Wasserstein distributionally robust MDPs. In Section 4, we illustrate our decomposition scheme for computing the distributionally robust Bellman updates when the Wasserstein distance in the ambiguity set is chosen to be  $L_{q}$  norm. In Section 5, customized fast algorithms and their time complexities are particularly provided for the three most well-known cases:  $L_{1}, L_{2}$ , and  $L_{\infty}$ . The numerical experiments in Section 6 verify that our methods outperform other state-of-the-art algorithms.
Notations We use boldface lowercase and uppercase letters to denote vectors and matrices, respectively. We denote by  $[N]$  the set of natural numbers from 1 to  $N$ ; that is,  $[N] = \{1,2,\dots,N\}$ . The vector  $\pmb{e}$  is denoted as the vector of all ones while the dimension depends on the context. The vector  $\pmb{e}_j$  represents the vector of all zeros except the  $j$ -th component is one. Probability simplex is denoted as  $\Delta_N = \{\pmb{x} \in \mathbb{R}^N : \pmb{e}^\top \pmb{x} = 1, \pmb{x} \geq \mathbf{0}\}$ . The smallest component of a vector  $\pmb{x}$  is denoted by  $\min\{\pmb{x}\}$ . Given a set  $\mathcal{X}$ , we call the set of all the Borel probability measures as  $P(\mathcal{X})$ . The general  $L_q$  norm for a vector  $\pmb{x} \in \mathbb{R}^N$  is denoted by  $\| \pmb{x} \|_q = \left( \sum_{i=1}^{N} |x_i|^q \right)^{1/q}$  for  $1 \leq q < \infty$ , and the  $L_\infty$  norm is  $\| \pmb{x} \|_\infty = \max_{i \in [N]} |x_i|$ . The Dirac distribution at  $\pmb{x} \in \mathbb{R}^N$  is denoted by  $\delta_{\pmb{x}}$ . For optimization problems with multiple constraints, we indicate the decision variables on the last line of the constraints.
Research on Markov decision processes under model ambiguity can be traced back to the seventies (Satia and Lave Jr, 1973). In recent years, much progress has been made in the development of
robust MDPs. The most fundamental solution scheme to solve robust MDPs is robust value iteration, which is the robust counterpart of the standard value iteration and is first developed in (Givan et al., 2000; Iyengar, 2005; Nilim and El Ghaoui, 2005) for robust MDPs with  $(s,a)$ -rectangular ambiguity sets. Wiesemann et al. (2013) introduce the  $s$ -rectangular ambiguity sets; the authors reformulate the robust Bellman updates as convex optimization problems and solve them by using off-the-shelf solvers that have polynomial-time complexity. Researchers have been investigating the theoretical properties of robust MDPs, such as the error bound of applying robust MDPs with state aggregation (Petrik and Subramanian, 2014), the relationship between robustness and regularization (Derman et al., 2021), and the geometry of value function in robust MDPs (Wang et al., 2022). Other than the  $s$ - and  $(s,a)$ -ambiguity sets, different types of ambiguity sets have been also proposed to mitigate the side effect of conservatism, such as  $k$ -rectangularity (Mannor et al., 2016) and  $r$ -rectangularity (Goyal and Grand-Clément, 2023). Different related models are also proposed to incorporate robustness in different settings, such as robust baseline regret (Ghavamzadeh et al., 2016), imitation learning (Brown et al., 2020), and soft-robust model (Lobo et al., 2020). In terms of algorithmic development, Ho et al. (2018) propose fast algorithms for solving  $s$ -rectangular robust MDPs with weighted  $L_{1}$  norm ambiguity sets and propose to compute robust Bellman updates using the combination of bisection method and homotopy method, and Behzadian et al. (2021) further extend the solution scheme for the unweighted  $L_{\infty}$  norm cases (Delgado et al., 2016). A first order method is introduced in (Grand-Clément and Kroer, 2021b) to approximate robust Bellman updates for robust MDPs with ellipsoidal and Kullback-Leibler (KL)  $s$ -rectangular ambiguity sets. Recently, Ho et al. (2022) propose the  $\phi$ -divergence ambiguity set, which generalizes multiple popular choices of ambiguity sets, and they propose an unified solution scheme to compute robust Bellman updates efficiently. On the other hand, robust policy iteration alternates between robust policy evaluation and policy improvement (Iyengar, 2005) and it has better empirical performance compared to robust value iteration. Kaufman and Schaefer (2013) propose a modified policy iteration for  $(s,a)$ -rectangular robust MDPs. Ho et al. (2021) introduce partial policy iteration for  $s$ - and  $(s,a)$ -rectangular robust MDPs with  $L_{1}$  norm ambiguity sets. Kumar et al. (2022) study the connection between robust MDPs and regularized MDPs, and propose robust policy iteration for  $s$ -rectangular robust MDPs. Other than robust policy iteration, Li et al. (2022) propose a policy gradient method for solving rectangular robust MDPs. In order to solve large-scale problems, Tamar et al. (2014) apply value function approximation and propose robust approximate dynamic programming for robust MDPs.
While this paper is focused on the model-based setting, recently there is an active line of research focusing on robust reinforcement learning (RL) algorithms (Roy et al., 2017; Badrinath and Kalathil, 2021; Wang and Zou, 2021, 2022; Panaganti and Kalathil, 2022). Liu et al. (2022) propose a robust Q-learning algorithm for  $(s,a)$ -rectangular robust MDPs. A function approximation approach is proposed in (Panaganti et al., 2022) for  $(s,a)$ -rectangular robust MDPs with offline datasets. The combination of robust RL and deep RL is also studied in (Pinto et al., 2017; Mankowitz et al., 2019; Zhang et al., 2020) although they do not provide theoretical guarantees of the learned policies. We would like to emphasize that both model-based and model-free methods have their own merits: while model-free methods are powerful for applications where data could be generated without much risk and cost, model-based methods are suitable for applications with very limited data or settings where the (time and financial) cost of obtaining new data is high.
For distributionally robust MDPs with the nested-set structured parameter ambiguity sets, Xu and Mannor (2010) propose a Bellman type backward induction to compute the optimal policy, and Yu and Xu (2015) further extend it to a more general state-wise ambiguity set following the unified framework proposed in (Wiesemann et al., 2014). The Wasserstein ambiguity sets are first introduced for distributionally robust MDPs in (Yang, 2017), and a convex formulation is proposed based on Kantorovich duality. Chen et al. (2019) propose a general formulation to combine both moment-based and statistical-distance-based ambiguity sets together for distributionally robust MDPs. Derman and Mannor (2020) study the connection between Wasserstein distributionally robust MDPs and regularization. Abdullah et al. (2019) propose a reinforcement learning framework with distributionally robustness. As mentioned before, however, (Grand-Clément and Kroer, 2021a) is the only work that focuses on the algorithmic development of solving Wasserstein distributionally robust MDPs, which is also the focus of this paper.
A distributionally robust MDP is a tuple  $(\mathcal{S},\mathcal{A},\pmb {p}_0,\pmb {r},\lambda ,\mathbb{M})$  , where  $\mathcal{S}$  and  $\mathcal{A}$  are the sets of states and actions, respectively. We assume that the state space  $S = \{1,\dots ,S\}$  and the action space  $\mathcal{A} = \{1,\ldots ,A\}$  are finite. The initial state  $s_0$  follows a given initial distribution  $\pmb {p}_0\in \Delta_S$  . When the decision maker takes an action  $a\in \mathcal{A}$  in state  $s\in S$  , the MDP will transit to the next state randomly according to the probability distribution  $\pmb{p}_{sa}\in \Delta_{S}$  , which is assumed to be unknown; the transition from state  $s\in S$  under action  $a\in \mathcal{A}$  to the next state  $s^{\prime}\in S$  will induce the reward  $r_{sas'}$  We use the vector  $\pmb {r}_{sa}\in \mathbb{R}^{S}$  to represent the connection of all rewards given state  $s$  and action  $a$  We denote by  $\lambda \in (0,1)$  the discount factor. In distributionally robust MDPs, the transition kernel  $\prod_{s\in S,a\in \mathcal{A}}\pmb{p}_{sa}\in (\Delta_S)^{S\times A}$  is assumed to be uncertain, and it is governed by an unknown probability distribution  $\mu$  which is assumed to reside in a prescribed ambiguity set  $\mathbb{M}$  that is calibrated from historical data. Note that we also use the notation  $\pmb {p}_s = \prod_{a\in \mathcal{A}}\pmb{p}_{sa}\in (\Delta_S)^A$  for the transition kernel given the state  $s\in S$
The goal of a distributionally robust MDP is to maximize the worst-case expected return; that is,
$$
\max  _ {\boldsymbol {\pi} \in (\Delta_ {A}) ^ {S}} \min  _ {\mu \in \mathbb {M}} \mathbb {E} _ {\boldsymbol {\pi}} \left[ \mathbb {E} _ {\boldsymbol {p} \sim \mu} \left[ \sum_ {t = 0} ^ {\infty} \lambda^ {t} r _ {s _ {t} a _ {t} s _ {t + 1}}: s _ {0} \sim \boldsymbol {p} _ {0} \right] \right]. \tag {1}
$$
Here, the decision variable  $\pi \in (\Delta_A)^S$  is called a (randomized) policy where for any given  $s \in S$ , its sub-vector  $\pi_s \in \Delta_A$  represents the probability distribution of taking action  $a \in \mathcal{A}$  in state  $s$ .
It is worth mentioning that the above formulation of distributionally robust MDPs is a generalization of both robust MDPs and standard MDPs. By specifying the ambiguity set  $\mathbb{M}$  to be  $P(\mathcal{U})$  where  $\mathcal{U}$  is the ambiguity set of a robust MDP, the problem (1) is equivalent to the robust MDP with ambiguity set  $\mathcal{U}$ . If the set  $\mathcal{U}$  is a singleton, then problem (1) with  $\mathbb{M} = P(\mathcal{U})$  is equivalent to the standard MDP with transition kernel from the singleton  $\mathcal{U}$ .
Similar to the case of robust MDPs, we consider the common assumption of rectangularity for the sake of tractability, as otherwise problem (1) is NP-hard (Wiesemann et al., 2013). In particular, we consider the following  $s$ -rectangular  $q$ -Wasserstein ambiguity set (Grand-Clément and Kroer, 2021a)
$$
\mathbb {M} ^ {q} = \left\{\mu \in P \left((\Delta_ {S}) ^ {S \times A}\right): \mu = \bigotimes_ {s \in \mathcal {S}} \mu_ {s}, \mu_ {s} \in \mathbb {M} _ {s} ^ {q} \subseteq P \left((\Delta_ {S}) ^ {A}\right), \forall s \in \mathcal {S} \right\},
$$
where  $\mathbb{M}_s^q$  is the marginal ambiguity set for the distribution  $\mu_s$  that governs the transition kernel at the state  $s\in S$ . The superscript  $q$  indicates that  $\mathbb{M}_s^q$  is defined by  $q$ -Wasserstein distance; that is,
$$
\mathbb {M} _ {s} ^ {q} = \left\{\mu_ {s} \in P \left(\left(\Delta_ {S}\right) ^ {A}\right): W _ {q} \left(\mu_ {s}, \nu_ {s}\right) \leq \theta \right\} \text {f o r} 1 \leq q \leq \infty ,
$$
where  $\nu_{s}$  is the reference distribution at the center of the Wasserstein ball and
$$
W _ {q} \left(\mu_ {s}, \nu_ {s}\right) = \min  _ {\kappa \in P \left((\Delta_ {S}) ^ {A} \times (\Delta_ {S}) ^ {A}\right)} \left\{\left(\underset {(x, y) \sim \kappa} {\mathbb {E}} \left(d _ {q} (x, y)\right) ^ {q}\right) ^ {1 / q}: \kappa \in \Gamma \left(\mu_ {s}, \nu_ {s}\right) \right\},
$$
for  $1 \leq q < \infty$ , where  $\Gamma(\mu_s, \nu_s)$  is the set of all the couplings of  $\mu_s$  and  $\nu_s$  (Ambrosio and Nicola Gigli, 2005), and  $d_q$  is induced by the  $q$ -norm  $d_q(x, y) = \|x - y\|_q$  throughout this work. We define the  $\infty$ -Wasserstein distance  $W_{\infty}(\mu_s, \nu_s)$  by taking  $q \to \infty$ .
We adopt the usual setting in distributionally robust data-driven optimization where  $\nu_{s} = \frac{1}{N}\sum_{i = 1}^{N}\delta_{\hat{\pmb{p}}_{s}^{i}}$  is set to be the empirical distribution and estimated by the samples  $\{\hat{p}_s^i\}_{i = 1}^N$  where  $\hat{\pmb{p}}_s^i\in (\Delta_S)^A$  for every  $i\in [N]$ . Then, the optimal value function of (1),  $\pmb{v}^{\star}$ , satisfies the following distributionally robust Bellman equation (Yu and Xu, 2015; Yang, 2017; Grand-Clément and Kroer, 2021a)
$$
v _ {s} ^ {\star} = \max  _ {\boldsymbol {\pi} _ {s} \in \Delta_ {A}} \min  _ {\boldsymbol {p} _ {s} \in \mathbb {B} _ {s} ^ {q}} \left(\sum_ {a \in \mathcal {A}} \pi_ {s a} \boldsymbol {p} _ {s a} ^ {\top} \left(\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v} ^ {\star}\right)\right) \quad \forall s \in \mathcal {S}, \tag {2}
$$
where  $\mathbb{B}_s^q$  is the set of expected kernels (Yang, 2017; Bertsimas et al., 2018; Xie, 2020) of the following forms
$$
\mathbb {B} _ {s} ^ {q} = \left\{\frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {p} _ {s} ^ {i}: \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {p} _ {s} ^ {i} - \hat {\boldsymbol {p}} _ {s} ^ {i} \right\| _ {q} ^ {q} \leq \theta^ {q}, \boldsymbol {p} _ {s} ^ {i} \in (\Delta_ {S}) ^ {A}, \forall i \in [ N ] \right\} \text {f o r} 1 \leq q <   \infty ,
$$
$$
\mathbb {B} _ {s} ^ {\infty} = \left\{\frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {p} _ {s} ^ {i}: \| \boldsymbol {p} _ {s} ^ {i} - \hat {\boldsymbol {p}} _ {s} ^ {i} \| _ {\infty} \leq \theta , \boldsymbol {p} _ {s} ^ {i} \in \left(\Delta_ {S}\right) ^ {A}, \forall i \in [ N ] \right\}. \tag {3}
$$
4 The Decomposition Algorithm for Distributionally Robust Bellman Update
In this section, we focus on computing the distributionally robust Bellman update, which is the most fundamental operator for solving the distributionally robust Bellman equation (2); we define the distributionally robust Bellman operator  $\mathfrak{T}(\cdot)$  where for any  $\pmb{v}\in \mathbb{R}^{S}$ ,
$$
\left[ \mathfrak {T} (\boldsymbol {v}) \right] _ {s} = \max  _ {\boldsymbol {\pi} _ {s} \in \Delta_ {A}} \min  _ {\boldsymbol {p} _ {s} \in \mathbb {B} _ {s} ^ {q}} \sum_ {a \in \mathcal {A}} \pi_ {s a} \cdot \boldsymbol {p} _ {s a} ^ {\top} \left(\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}\right) \quad \forall s \in \mathcal {S}. \tag {4}
$$
Given the above definition of  $\mathfrak{T}(\cdot)$ , it is well-known that (2) can be solved by iteratively applying  $\mathfrak{T}$ , which is a variant of standard value iteration for solving distributionally robust MDPs (Yu and Xu, 2015; Grand-Clément and Kroer, 2021a). For any initial guess  $\pmb{v}^0 \in \mathbb{R}^S$ , we have  $\lim_{t \to \infty} \pmb{v}^t \rightarrow \pmb{v}^\star$  where  $\pmb{v}^t = \mathfrak{T}(\pmb{v}^{t-1})$  for  $t = 1, 2, \ldots$  and  $\pmb{v}^\star$  satisfies (2). Therefore, the efficiency of evaluating  $[\mathfrak{T}(\pmb{v})]_s$  is crucial to the computation of solving distributionally robust MDPs.
However, computing  $[\mathfrak{T}(\boldsymbol{v})]_s$  using generic convex optimization solvers is much more computationally demanding compared to the case of classical MDPs, which only have time complexity  $\mathcal{O}(SA)$  for computing their Bellman updates. In this section, we exploit the specific problem structure in (4) and reformulate the optimization problem to a form that could be decomposed into smaller problems. We first focus on the case of  $q \in [1,\infty)$  in Section 4.1 and then consider the case of  $q = \infty$  in Section 4.2, where  $q$  indicates the type of Wasserstein distance used in the ambiguity set. As we will show later, combined with the customized fast algorithms in Section 5 for solving the subproblems, one can compute  $[\mathfrak{T}(\boldsymbol{v})]_s$  in time complexity that is quasi-linear in  $S$  and linear in  $A$  and  $N$ . All the proofs of propositions and theorems in this section are provided in Appendix A.1.
4.1 Nested bisection method for  $q$ -Wasserstein ambiguity set with  $q \in [1, \infty)$
We consider the optimization problem in (4) for the case where  $q \in [1, \infty)$ . By applying the minimax theorem, we obtain the following result.
Proposition 4.1. Consider the Bellman updates (4) with  $q \in [1, \infty)$ . Then,
$$
[ \mathfrak {T} (\boldsymbol {v}) ] _ {s} = \left[ \begin{array}{c c} \operatorname {m i n i m i z e} & \gamma \\ \text {s u b j e c t t o} & \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}\right) ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \leq \gamma , \forall a \in \mathcal {A} \\ & \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {a \in \mathcal {A}} \left\| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {q} ^ {q} \leq \theta^ {q} \\ & \gamma \in \mathbb {R}, \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ], \forall a \in \mathcal {A}. \end{array} \right] \quad \forall s \in \mathcal {S}. \tag {5}
$$
The above proposition reformulates the maximin optimization problem in (4) into a convex minimization problem, which can be solved by commercial convex optimization solvers. Moreover, problem (5) can be solved via bisection on  $\gamma$ ; that is, we seek for the lowest possible  $\gamma$  such that  $\frac{1}{N}\sum_{i=1}^{N}(\boldsymbol{r}_{sa} + \lambda \boldsymbol{v})^\top \boldsymbol{p}_{sa}^i \leq \gamma$  for each  $a \in \mathcal{A}$  while  $\boldsymbol{p}_{sa}^i$  satisfies the other constraints in (5), for every  $a \in \mathcal{A}$ . To this end, we introduce the following subproblem
$$
\mathfrak {P} \left(\left\{\hat {\boldsymbol {p}} _ {s a} ^ {i} \right\} _ {i = 1} ^ {N}; \boldsymbol {b} _ {s a}, \gamma\right) = \left[ \begin{array}{c c} & \text {m i n i m i z e} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {q} ^ {q} \\ & \text {s u b j e c t t o} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \leq \gamma \\ & \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ] \end{array} \right]. \tag {6}
$$