SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By setting  $\pmb{b}_{sa} = \pmb{r}_{sa} + \lambda \pmb{v}$ , the above problem  $\mathfrak{P}\left(\{\hat{\pmb{p}}_{sa}^i\}_{i=1}^N; \pmb{b}_{sa}, \gamma\right)$  allows us to seek for decision variable  $\{\pmb{p}_{sa}^i\}_{i=1}^N$  that satisfies the first and the last line of constraints in (5) while minimizing the "budget" used in the second line of constraint in (5). In particular, for any fixed  $\gamma' \in \mathbb{R}$ , we distinguish among the following two cases:
(a) If  $\sum_{a\in A}\mathfrak{P}\left(\{\hat{p}_{sa}^i\}_{i = 1}^N;b_{sa},\gamma '\right)\leq \theta^q$  , then  $\gamma^\prime$  is feasible for (5) and it is an upper bound of the optimal objective value in (5).
(b) If  $\sum_{a\in \mathcal{A}}\mathfrak{P}\left(\{\hat{p}_{sa}^i\}_{i = 1}^N;b_{sa},\gamma '\right) > \theta^q$  , then  $\gamma^\prime$  must be infeasible for problem (5) and so it is a lower bound of the optimal objective value in (5).
To apply the bisection method, we derive the initial upper and lower bounds of the optimal  $\gamma^{\star}$  in (5).
Proposition 4.2. Consider the distributionally robust Bellman update (5). The optimal objective value  $\gamma^{\star}$  is bounded by
$$
\max  _ {a \in \mathcal {A}} \left\{\min  \left\{\boldsymbol {b} _ {s a} \right\} \right\} \leq \gamma^ {\star} \leq \max  _ {a \in \mathcal {A}} \left\{\frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {b} _ {s a} ^ {\top} \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\}. \tag {7}
$$
As opposed to (5), the size of problem (6) is independent of  $A$ . By using the aforementioned bisection method, the overall time complexity is now only linear in  $A$  since we only need to solve (6)  $\mathcal{O}(A\log \epsilon_1)$  times, where  $\epsilon_{1}$  is the tolerance for the bisection method.
While one can solve problem (6) using off-the-shelf solvers and enjoy the reduced complexity by using the above bisection method, this subproblem itself turns out to be another structural optimization problem. As we will show in the following proposition, by applying duality on (6), the reformulation could be further decomposed via another bisection method on the dual variable.
Proposition 4.3. Consider the problem  $\mathfrak{P}\left(\{\hat{p}_{sa}^i\}_{i=1}^N; \boldsymbol{b}_{sa}, \gamma\right)$  in (6). If  $\gamma > \min \{\boldsymbol{b}_{sa}\}$ , then
$$
\mathfrak {P} \left(\left\{\hat {\boldsymbol {p}} _ {s a} ^ {i} \right\} _ {i = 1} ^ {N}; \boldsymbol {b} _ {s a}, \gamma\right) = \max  _ {0 \leq \alpha \leq \bar {\alpha}} - \alpha \gamma + \frac {1}{N} \sum_ {i = 1} ^ {N} \mathfrak {D} _ {q} \left(\hat {\boldsymbol {p}} _ {s a} ^ {i}, \boldsymbol {b} _ {s a}, \alpha\right), \tag {8}
$$
where  $\bar{\alpha} = \max_{i\in [N]}\left\| \pmb {e}_j - \hat{\pmb{p}}_sa^i\right\| _q^q /(\gamma -\min \left\{\pmb {b}_{sa}\right\})$  and
$$
\mathfrak {D} _ {q} \left(\hat {\boldsymbol {p}} _ {s a} ^ {i}, \boldsymbol {b} _ {s a}, \alpha\right) = \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {q} ^ {q} + \alpha \cdot \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i}, \tag {9}
$$
where  $j\in \operatorname *{arg  min}_{s^{\prime}\in S}b_{sa s^{\prime}}$
Notice that we focus on the case where  $\gamma >\min \left\{\pmb{b}_{sa}\right\}$  since  $\min \left\{\pmb{b}_{sa}\right\}$  is not larger than the lower bound of  $\gamma$  in (7), and the case  $\gamma = \min \left\{\pmb{b}_{sa}\right\}$  will not occur because  $\gamma$  is always taken to be the average of the upper and lower bounds in our bisection method. We refer interested readers to Appendix for more details.
Proposition 4.3 indicates that for any fixed feasible  $\alpha$  in (8), the subproblems (9) can be solved separately. Therefore, by applying the bisection method on  $\alpha$  in problem (8), one can naturally decompose the problem (8) into  $N$  smaller problems that have  $S$  variables and  $S + 1$  constraints. As a consequence, the overall time complexity is linear in the number of kernels  $N$ .
By combining both decomposition strategies above, we obtain the proposed nested bisection method, whose pseudocode could be found in Algorithm 1. More details of the algorithm is provided in the Appendix A.1.
Theorem 4.4. Suppose  $\gamma'$  is the value returned by Algorithm 1, and  $\gamma^{\star}$  be the optimal value of (5). With the inputs provided in Algorithm 1 and user-specified tolerances  $\epsilon_1, \epsilon_2 > 0$ , we have
$$
| \gamma^ {\prime} - \gamma^ {\star} | \leq \frac {\epsilon_ {1}}{2} + \frac {A \epsilon_ {2} \left(\max  _ {a \in \mathcal {A}} \max  \{\boldsymbol {b} _ {s a} \} - \underline {{\gamma}}\right) \left(\max  _ {a \in \mathcal {A}} \max  \{\boldsymbol {b} _ {s a} \} + \bar {\gamma}\right)}{\theta^ {q}}.
$$
Theorem 4.5. Algorithm 1 computes (5) in time  $\mathcal{O}\left(h_q(S)NA\log \epsilon_1^{-1}\log \epsilon_2^{-1} + AS\right)$ , where  $h_q(S)$  is the time complexity for solving (9).
Input: Tolerance  $\epsilon_{1}$  for outer bisection method and  $\epsilon_{2}$  for inner bisection method;
Initialization: Set lower bound  $\gamma$  and upper bound  $\bar{\gamma}$  that are specified in (7);
Algorithm 1: Nested bisection method to compute distributionally robust Bellman update (5)
while  $\bar{\gamma} -\gamma >\epsilon_{1}$  do
end
Compute  $\hat{\gamma} = (\bar{\gamma} +\underline{\gamma}) / 2$  and  $r_0 = 0$  ..   
for  $a = 1,\ldots ,A$  do Set lower bound  $\underline{\alpha}_a = 0$  and compute the upper bound  $\bar{\alpha}_a$  in Proposition 4.3; while  $\bar{\alpha}_a - \underline{\alpha}_a > \epsilon_2$  do Compute  $\hat{\alpha}_{a} = (\bar{\alpha}_{a} + \underline{\alpha}_{a}) / 2$  . Solve  $\mathfrak{D}_q(\hat{p}_{sa}^i,\boldsymbol {b}_{sa},\hat{\alpha}_a)$  , for every  $i\in [N]$  . Compute the slope of (8):  $m = -\hat{\gamma} +\frac{1}{N}\sum_{i = 1}^{N}\boldsymbol{b}_{sa}^{\top}\boldsymbol{p}_{sa}^{i,\star}$  . if  $m\leq 0$  then set  $\bar{\alpha}_a = \hat{\alpha}_a$  else set  $\underline{\alpha}_a = \hat{\alpha}_a$  end Set  $\alpha_{a}^{\star} = (\bar{\alpha}_{a} + \underline{\alpha}_{a}) / 2$  . Solve  $\mathfrak{D}_q(\hat{p}_{sa}^i,\boldsymbol {b}_{sa},\alpha_a^\star)$  , for every  $i\in [N]$  . Set  $r_a = r_{a - 1} - \alpha_a^\star \hat{\gamma} +\frac{1}{N}\sum_{i = 1}^{N}\mathfrak{D}_q(\hat{p}_{sa}^i,\boldsymbol {b}_{sa},\alpha_a^\star)$  end if  $r_A > \theta^q$  then set  $\underline{\gamma} = \hat{\gamma}$  else set  $\bar{\gamma} = \hat{\gamma}$
Result: Optimal objective value of (5):  $\gamma = (\bar{\gamma} +\underline{\gamma}) / 2$
As shown in the above theorem, the proposed nested bisection method has a time complexity that is linear in both  $A$  and  $N$ , but the overall complexity depends on the subproblem (9) where its complexity depends on the choice of  $q$  and the number of states  $S$ . In Section 5, we will derive  $h_1(S)$  and  $h_2(S)$ , which are the two most common cases. The complexities associated with solving equations (5) and (8) using general convex optimization methods are also provided in the Appendix B.
4.2 Decomposition scheme for  $\infty$ -Wasserstein ambiguity set
As opposed to the case where  $q \in [1, \infty)$ , for  $\infty$ -Wasserstein ambiguity set, the corresponding Bellman update can be naturally decomposed without using any bisection method.
Proposition 4.6. Consider the Bellman update (4) with  $q = \infty$ . Then,
$$
[ \mathfrak {T} (\boldsymbol {v}) ] _ {s} = \frac {1}{N} \max  _ {a \in \mathcal {A}} \sum_ {i = 1} ^ {N} \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\{\boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i}: \| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \| _ {\infty} \leq \theta \right\} \quad \forall s \in \mathcal {S}. \tag {10}
$$
The above reformulation requires solving  $NA$  inner minimization problems that have  $S$  variables and  $S + 2$  constraints. Hence, the time complexity of Bellman update (10) is linear in  $A$  and  $N$ . Similar to Section 4.1, we will discuss the complexity of solving the inner minimization in the next section.
5 Efficient Algorithms for Subproblems
Section 4 offers decomposition schemes for distributionally robust MDPs with  $q$ -Wasserstein ambiguity set. As shown in Theorem 4.5 and Proposition 4.6, the overall complexity of computing distributionally robust Bellman update is linear in both  $A$  and  $N$  but it depends on complexities of solving the subproblem (9) and the inner minimization problem in (10). In this section, we consider the common cases where  $q \in \{1,2,\infty\}$  and discuss the time complexity of solving the subproblem in each case.
As we will show below, the overall time complexities of our decomposition algorithms are only  $\mathcal{O}\left(SA\log S + NAS\log \epsilon_1^{-1}\log \epsilon_2^{-1}\right)$  for  $q = 1$ ,  $\mathcal{O}(NAS\log S\log \epsilon_1^{-1}\log \epsilon_2^{-1})$  for  $q = 2$ , and  $\mathcal{O}(AS(\log S + N))$  when  $q = \infty$ . These complexities are much lower than  $\mathcal{O}(N^{3.5}A^{3.5}S^{3.5}\log (\epsilon^{-1}))$  and  $\mathcal{O}(NA^{2.5}S^{2.5}\log (S)\epsilon^{-1.5})$  from the state-of-the-art solution methods in (Xu and Mannor, 2010) and (Grand-Clément and Kroer, 2021a), respectively. The proofs of
the following propositions, theorems, and corollaries and the details of the proposed algorithms are relegated to Appendix A.2.
5.1 1-Wasserstein ambiguity set
When  $q = 1$ , the corresponding subproblem (9) has an equivalent form as follows.
Proposition 5.1. Suppose  $q = 1$ . The minimization problem (9) is equivalent to
$$
\min  _ {v \geq 0} \left\{v + \alpha \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i}: \| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \| _ {1} \leq v \right\}. \tag {11}
$$
While problem (11) does not appear to be trivial at first glance, it has nice mathematical properties which allow us to solve this problem efficiently.
Theorem 5.2. The objective function in (11) is piecewise-linear in  $v$ , and it can be solved in time  $\mathcal{O}(S\log S)$ .
Therefore, the proposed Algorithm 1 can compute the Bellman update with the time complexity linear in  $A$  and  $N$  and quali-linear in  $S$ . We obtain the following result by combining Theorem 4.5 and Theorem 5.2.
Corollary 5.2.1. The distributionally robust Bellman update with 1-Wasserstein ambiguity set can be computed in time  $\mathcal{O}\left(SA\log S + NAS\log \epsilon_1^{-1}\log \epsilon_2^{-1}\right)$ , where  $\epsilon_{1}$  and  $\epsilon_{2}$  are the user-specified tolerances in Algorithm 1.
5.2 2-Wasserstein ambiguity set
When  $q = 2$ , the following result shows that (9) could be transformed to an Euclidean projection problem onto the probability simplex.
Theorem 5.3. Suppose  $q = 2$ . The minimization problem (9) is equivalent to
$$
- \frac {\alpha^ {2} \| \boldsymbol {b} _ {s a} \| _ {2} ^ {2}}{4} + \alpha \boldsymbol {b} _ {s a} ^ {\top} \hat {\boldsymbol {p}} _ {s a} ^ {i} + \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\| \boldsymbol {p} _ {s a} ^ {i} - \frac {2 \hat {\boldsymbol {p}} _ {s a} ^ {i} - \alpha \boldsymbol {b} _ {s a}}{2} \right\| _ {2} ^ {2}, \tag {12}
$$
which can be solved in time  $\mathcal{O}(S\log S)$
Euclidean projection problem onto the probability simplex can be solved via the existing algorithm (Wang and Carreira-Perpinan, 2013). By combining the results in Theorem 4.5 and Theorem 5.3, we obtain the following result.
Corollary 5.3.1. The distributionally robust Bellman update with 2-Wasserstein ambiguity set can be computed in time  $\mathcal{O}\left(NAS\log S\log \epsilon_1^{-1}\log \epsilon_2^{-1}\right)$ , where  $\epsilon_{1}$  and  $\epsilon_{2}$  are the user-specified tolerances in Algorithm 1.
5.3  $\infty$ -Wasserstein ambiguity set
When  $q = \infty$ , the corresponding subproblem
$$
\min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\{\boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i}: \| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \| _ {\infty} \leq \theta \right\} \tag {13}
$$
can be reformulated to a linear program with box constraints and a linear equality constraint which can be solved efficiently.
Theorem 5.4. The minimization problem (13) is solved in time  $\mathcal{O}(S\log S)$ .
Therefore, we obtain the overall complexity as follows.
Corollary 5.4.1. The distributionally robust Bellman update with  $\infty$ -Wasserstein ambiguity set can be computed in time  $\mathcal{O}$  ( $AS(\log S + N)$ ).
Figure 1: Comparisons of all algorithms for  $q = 1$ , where (left)  $N = 50$ , (middle)  $A = 70$ ,  $N = 50$ , and (right)  $S = A = 70$ .
Figure 2: Comparisons of all algorithms for  $q = 2$ , where (left)  $N = 50$ , (middle)  $A = 70$ ,  $N = 50$ , and (right)  $S = A = 70$ .
Figure 3: Comparisons of all algorithms for  $q = \infty$ , where (left)  $N = 50$ , (middle)  $A = 70$ ,  $N = 50$ , and (right)  $S = A = 70$ .
We experiment on the performance of the proposed algorithms (Fast), Gurobi (Gurobi Optimization, LLC, 2023), and the first-order method (FOM) proposed in (Grand-Clement and Kroer, 2021a) with different sizes of the randomly generated distributionally robust MDPs. Due to page limit, we only report the results of the experiments on a single Bellman update, and we provide additional experimental results in the Appendix C, which also contains the detailed settings of all the experiments.
Figure 1 reports the average computation times of Bellman updates with 1-Wasserstein ambiguity sets. For the first-order method (Grand-Clément and Kroer, 2021a), we report the computation times of the third Bellman iteration, and these computation times increase at every Bellman iteration of the first-order method. One can see that all three algorithms perform similarly when problem size is small. However, as  $S$ ,  $A$ , or  $N$  increases, the runtimes of both Gurobi and first-order method increase rapidly, while the proposed algorithm remains scalable.
The results with 2-Wasserstein ambiguity sets and  $\infty$ -Wasserstein ambiguity sets are shown in Figure 2 and Figure 3, respectively. These results are similar to the case where  $q = 1$ , which are consistent to our theoretical results on complexities. As expected, the proposed algorithms are several orders of magnitude faster than the existing state-of-the-art solution methods.
This paper studies distributionally robust MDPs with Wasserstein ambiguity sets. In particular, we propose fast algorithms to compute distributionally robust Bellman updates. We show that the proposed algorithms outperform other existing methods in both theory and experiments. Future work could address the extensions to approximate dynamic programming for distributionally robust MDPs as well as model-free settings.
We thank the anonymous reviewers for their supportive comments. This work was supported, in part, by the CityU Start-Up Grant (Project No. 9610481), the National Natural Science Foundation of China (Project No. 72032005), and Chow Sang Sang Group Research Fund sponsored by Chow Sang Sang Holdings International Limited (Project No. 9229076).
Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196, 2019.
Luigi Ambrosio and Giuseppe Savaré Nicola Gigli. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2005.
TW Archibald, KIM McKinnon, and LC Thomas. On the generation of markov decision processes. Journal of the Operational Research Society, 46(3):354-361, 1995.
Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning, pages 511-520. PMLR, 2021.
Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for  $l_{\infty}$ -constrained s-rectangular robust mdps. Advances in Neural Information Processing Systems, 34:25982-25992, 2021.
Dimitri P. Bertsekas. *Nonlinear programming*. Athena Scientific Belmont. Massachusetts, USA, 1999.
Dimitris Bertsimas, Shimrit Shtern, and Bradley Sturt. A data-driven approach for multi-stage linear optimization. Available at Optimization Online, 2018.
Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 45(11):2471-2482, 2009.
Daniel Brown, Scott Niekum, and Marek Petrik. Bayesian robust optimization for imitation learning. Advances in Neural Information Processing Systems, 33:2479-2491, 2020.
Zhi Chen, Pengqian Yu, and William B Haskell. Distributionally robust optimization for sequential decision-making. Optimization, 68(12):2397-2426, 2019.
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595-612, 2010.
Karina V Delgado, Leliane N De Barros, Daniel B Dias, and Scott Sanner. Real-time dynamic programming for markov decision processes with imprecise probabilities. Artificial Intelligence, 230:192-223, 2016.
Esther Derman and Shie Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint arXiv:2003.02894, 2020.
Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized mdps and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34: 22274-22287, 2021.
Insoon Yang. A convex optimization approach to distributionally robust markov decision processes with Wasserstein distance. IEEE control systems letters, 1(1):164-169, 2017.
Pengqian Yu and Huan Xu. Distributionally robust counterpart in markov decision processes. IEEE Transactions on Automatic Control, 61(9):2538-2543, 2015.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024-21037, 2020.
A Appendix: Proofs and Algorithms
A.1 Proofs of results in Section 4
Proof of Proposition 4.1. Plug  $\mathbb{B}_s^q$  in (3) into (4), and apply the minimax theorem, the original problem  $[\mathfrak{T}(\pmb {v})]_s$ ,  $\forall s\in S$  is given by:
$$
\begin{array}{l} \max  _ {\boldsymbol {\pi} _ {s} \in \Delta_ {A}} \quad \min  _ {\boldsymbol {p} _ {s} ^ {1}, \dots , \boldsymbol {p} _ {s} ^ {N}} \sum_ {a \in \mathcal {A}} \pi_ {s a} \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {p} _ {s a} ^ {i}\right) ^ {\top} (\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}) \\ \text {s . t .} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {p} _ {s} ^ {i} - \hat {\boldsymbol {p}} _ {s} ^ {i} \right\| _ {q} ^ {q} \leq \theta^ {q}, \\ \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ], \forall a \in \mathcal {A} \\ = \quad \min  \quad \max  _ {\boldsymbol {\pi} _ {s} \in \Delta_ {A}} \frac {1}{N} \sum_ {a \in \mathcal {A}} \pi_ {s a} \sum_ {i = 1} ^ {N} \left(\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}\right) ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ \text {s . t .} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {p} _ {s} ^ {i} - \hat {\boldsymbol {p}} _ {s} ^ {i} \right\| _ {q} ^ {q} \leq \theta^ {q}, \\ \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ], \forall a \in \mathcal {A} \\ = \quad \min  \quad \max  _ {a \in \mathcal {A}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}\right) ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ \text {s . t .} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {a \in \mathcal {A}} \left\| \boldsymbol {p} _ {s} ^ {i} - \hat {\boldsymbol {p}} _ {s} ^ {i} \right\| _ {q} ^ {q} \leq \theta^ {q}, \\ \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ], \forall a \in \mathcal {A}. \\ \end{array}
$$
Hence (5) is a direct consequence of above formulation by introducing the following epigraph variable
$$
\gamma \text {w h i c h s a t i s f i e s} \gamma \geq \max  _ {a \in \mathcal {A}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}\right) ^ {\top} \boldsymbol {p} _ {s a} ^ {i}.
$$
Proof of Proposition 4.2. To prove the upper bound of  $\gamma^{\star}$ , we consider  $\bar{\gamma} = \max_{a\in \mathcal{A}}\frac{1}{N}\sum_{i = 1}^{N}\pmb{b}_{sa}^{\top}\hat{\pmb{p}}_{sa}^{i}$ , which equals to  $\bar{\gamma}\geq \frac{1}{N}\sum_{i = 1}^{N}\pmb{b}_{sa}^{\top}\hat{\pmb{p}}_{sa}^{i},\forall a\in \mathcal{A}$ . This implies  $\{\hat{p}_{sa}^i\}_{i = 1}^N$  satisfies every constraint in problem (6) with the lowest possible objective value 0, for every  $a\in \mathcal{A}$ . Therefore,  $\sum_{a\in \mathcal{A}}\mathfrak{P}\left(\{\hat{p}_{sa}^i\}_{i = 1}^N;r_{sa} + \lambda v,\bar{\gamma}\right) = 0 < \theta^q$ . At this time,  $\bar{\gamma}$  is feasible for (5). Hence, we provide an upper bound  $\bar{\gamma} = \max_{a\in \mathcal{A}}\frac{1}{N}\sum_{i = 1}^{N}\pmb{b}_{sa}^{\top}\hat{\pmb{p}}_{sa}^{i}$ .
To prove the lower bound of  $\gamma^{\star}$ , we assume the contrary  $\max_{a\in \mathcal{A}}\{\min \{\pmb{b}_{sa}\} \} > \gamma^{\star}$ . So there exists  $\hat{a}\in \mathcal{A}$  with  $\gamma^{\star} < \min \{\pmb{b}_{\hat{s}\hat{a}}\}$ . Then there is no  $\pmb{p}_{s\hat{a}}^{i}$  satisfies the first constraint in (5) for  $a = \hat{a}$ . This contradiction verifies the lower bound  $\underline{\gamma} = \max_{a\in \mathcal{A}}\{\min \{\pmb{b}_{sa}\}\}$ .
Proof of Proposition 4.3. By definition, for any fixed  $s \in S$  and  $a \in \mathcal{A}$ , we have
$$
\begin{array}{l} \mathfrak {P} \left(\left\{\hat {\boldsymbol {p}} _ {s a} ^ {i} \right\} _ {i = 1} ^ {N}; \boldsymbol {b} _ {s a}, \gamma\right) = \min  \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {q} ^ {q} \\ \text {s . t .} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \leq \gamma , \\ \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ]. \\ \end{array}
$$
Then, we introduce the dual variable  $\alpha \geq 0$  for the constraint  $\frac{1}{N}\sum_{i = 1}^{N}\pmb{b}_{sa}^{\top}\pmb{p}_{sa}^{i}\leq \gamma$  and obtain
$$
\begin{array}{l} \mathfrak {P} \left(\left\{\hat {\boldsymbol {p}} _ {s a} ^ {i} \right\} _ {i = 1} ^ {N}; \boldsymbol {b} _ {s a}, \gamma\right) = \max  _ {\alpha \geq 0} \left\{\min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ]} \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {q} ^ {q} + \alpha \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} - \gamma\right) \right\} \\ = \max  _ {\alpha \geq 0} - \alpha \gamma + \frac {1}{N} \sum_ {i = 1} ^ {N} \mathfrak {D} _ {q} (\hat {\boldsymbol {p}} _ {s a} ^ {i}, \boldsymbol {b} _ {s a}, \alpha), \\ \end{array}
$$
where  $\mathfrak{D}_q(\hat{\pmb{p}}_{sa}^i,\pmb{b}_{sa},\alpha)$  is defined in the Proposition 4.3.
To show  $\bar{\alpha}$  defined in the proposition is indeed an upper bound of the optimal  $\alpha^{\star}$ , we denote
$$
f (\alpha) = - \alpha \gamma + \frac {1}{N} \sum_ {i = 1} ^ {N} \mathfrak {D} _ {q} \left(\hat {\boldsymbol {p}} _ {s a} ^ {i}, \boldsymbol {b} _ {s a}, \alpha\right).
$$
Notice that  $\forall \alpha \geq \max_{i\in [N]}\frac{\left\|e_j - \hat{p}_{sa}^i\right\|_q^q}{\gamma - \min\{\boldsymbol{b}_{sa}\}}$  where  $j\in \arg \min_{s'\in S} b_{sas'}$ :
$$
\begin{array}{l} f (\alpha) \leq - \alpha \gamma + \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {e} _ {j} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {q} ^ {q} + \alpha \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {e} _ {j} \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {e} _ {j} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {q} ^ {q} + (\min  \left\{\boldsymbol {b} _ {s a} \right\} - \gamma) \alpha \\ \leq 0 = f (0), \\ \end{array}
$$