SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {k = 1} ^ {K + 1} \hat {p} _ {s a k} ^ {i} > \frac {v}{2} \geq \sum_ {k \in \mathcal {N}} \hat {p} _ {s a k} ^ {i} - \bar {p} _ {k} \geq \sum_ {k = 1} ^ {K + 1} \hat {p} _ {s a k} ^ {i}.
$$
Here the first inequality is due to the selection of  $v$ , the second inequality has been deduced in the first case and the third inequality is from  $\bar{p}_k = 0$ ,  $\forall k \in [K + 1]$ . So we are able to find  $\hat{k} \leq K$  with  $\bar{p}_{\hat{k}} > 0$ . By moving probability  $\varepsilon = \min \left\{\frac{\bar{p}_{\hat{k}}}{2}, \frac{\hat{p}_{sa(K + 1)}^i}{2}\right\}$  from  $\bar{p}_{\hat{k}}$  to  $\bar{p}_{K + 1}$ , we can achieve smaller objective value while keeping the feasibility, hence we get the contradiction, which implies that  $\bar{p}_{K + 1} > 0$  and  $\bar{p}_S < 1$ . By applying the similar procedures in the first case (moving some probability from  $\bar{p}_{K + 1}$  or to  $\bar{p}_S$ ), we can show that  $\bar{p}_k = \hat{p}_{sa_k}^i = r_k$  for  $K + 2 \leq k \leq S - 1$ .
Next we prove that  $\bar{p}_k = r_k = 0$  for  $k\in [K]$ . Suppose the contrary is true; that is,  $\bar{p}_{\hat{k}} > 0$  for some  $\hat{k}\leq K$ . Then we are able to apply the same procedures as before which illustrate that  $\bar{p}_k = \hat{p}_{sak}^i$ ,  $\forall \hat{k} < k < S$ . Provided this, one can verified that the optimal strategy is putting all the rest probability  $1 - \sum_{k = \hat{k} +1}^{S - 1}\hat{p}_{sak}^i$  to  $\bar{p}_S$  since  $b_{sa1} > b_{sa2} > \dots >b_{saS}$ , and  $v\geq 2\sum_{k = 1}^{K}\hat{p}_{sak}^{i}\geq 2\sum_{k = 1}^{\hat{k}}\hat{p}_{sak}^{i}$ , which implies  $\bar{p}_{sa\hat{k}} = 0$  and we get a contradiction. Hence  $\bar{p}_k = r_k = 0$ ,  $\forall k\in [K]$ .
Finally, we verify the rest two components. We introduce the variables  $d_{K + 1} = \bar{p}_{K + 1} - \hat{p}_{sa(K + 1)}^i$  and  $d_S = \bar{p}_S - \hat{p}_{saS}^i$ , then we get the equivalent reformulation of the inner minimization in  $F(v)$ :
$$
\begin{array}{l} \min  _ {d _ {K + 1}, d _ {S}} \quad b _ {s a (K + 1)} d _ {K + 1} + b _ {s a S} d _ {S} \\ \begin{array}{l l} \text {s . t .} & d _ {K + 1} + d _ {S} = \sum_ {k = 1} ^ {K} \hat {p} _ {s a k} ^ {i}, | d _ {K + 1} | + | d _ {S} | \leq v - \sum_ {k = 1} ^ {K} \hat {p} _ {s a k} ^ {i}. \end{array} \\ \end{array}
$$
The optimal  $d_{K + 1}$  and  $d_S$  are given by  $-\frac{v - 2\sum_{k = 1}^{K}\hat{p}_{sak}^i}{2}$  and  $\frac{v}{2}$  respectively. Hence we proved  $\pmb{p}^{\star}$  is an optimal solution.
To prove the third case of the claim, we notice that the optimal solution  $\pmb{e}_S$  to  $\min_{\pmb{p}_{sa}^i \in \Delta_S} \pmb{b}_{sa}^\top \pmb{p}_{sa}^i$  is also feasible for the inner minimization problem in  $F(v)$ , hence it becomes the optimal solution we desire, which implies  $F(v) = v + \alpha b_{saS}$  at this time. This finishes the proof of our claim.
Our claim directly illustrates that  $F(v)$  is a piecewise-linear function in  $v$  with breakpoints  $\{0\} \cup \{2\sum_{k=1}^{K}\hat{p}_{sak}^i : \forall K \in [S-1]\}$ . Furthermore, based on the provided formulation of  $F(v)$ , we can compute the difference of value for  $F(\cdot)$  between any two adjacent breakpoints, given by
$$
F (v _ {K}) - F (v _ {K - 1}) = 2 \hat {p} _ {s a K} ^ {i} + \alpha (b _ {s a S} - b _ {s a K}) \hat {p} _ {s a K} ^ {i} \quad \forall K \in [ S - 1 ],
$$
where  $v_{K} = 2\sum_{k = 1}^{K}\hat{p}_{sak}^{i}, \forall K\in [S - 1]$  and  $v_{0} = 0$ .
Hence we provide Algorithm 2 to compute (11), whose time complexity is  $\mathcal{O}(S\log S)$  generally and can be reduced to  $\mathcal{O}(S)$  if the sorted  $\pmb{b}_{sa}$  is provided.
Proof of Corollary 5.2.1. We can see that Algorithm 2 is in time complexity  $\mathcal{O}(S)$  if  $\cup_{a\in \mathcal{A}}b_{sa}$  are sorted, which can be done at the Initialization step in Algorithm 1 with time complexity  $\mathcal{O}(AS\log S)$ . So by Theorem 4.5, we get the overall complexity is  $\mathcal{O}\left(NAS\log \epsilon_1^{-1}\log \epsilon_2^{-1} + AS\log S\right)$ .
Algorithm 3: Fast algorithm to solve the inner minimization problem in (10)
Input:  $\mathbf{v}$ ,  $\mathbf{r}_{sa}$ , and  $\hat{\mathbf{p}}_{sa}^{i} \in \Delta_{S}$ .
Initialization:  $p_{sa}^{i,\star} = 0$
Sort  $\pmb{b}_{sa}$  as  $b_{san_1} \leq \dots \leq b_{san_s}$ .
Find the smallest  $k$  such that  $\sum_{j=1}^{k}\left(\hat{p}_{san_j}^i + \theta\right) \geq 1$ .
Set  $p_{san_j}^{i,\star} = \hat{p}_{san_j}^i + \theta$  for  $j \leq k - 1$  and  $p_{san_k}^{i,\star} = 1 - \sum_{j=1}^{k-1} \left( \hat{p}_{san_j}^i + \theta \right)$ .
$$
r = \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i, \star}.
$$
Result: Optimal objective value  $r$  and optimal solution  $p_{sa}^{\star}$  of the inner minimization problem in (10).
A.2.2 Proof of results in Section 5.2
Proof of Theorem 5.3. From problem (9), we can get the minimization problem for  $q = 2$
$$
\begin{array}{l} \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\| \boldsymbol {p} _ {s a} ^ {i} - \boldsymbol {\tilde {p}} _ {s a} ^ {i} \right\| _ {2} ^ {2} + \alpha \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ = \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\| \boldsymbol {p} _ {s a} ^ {i} \right\| _ {2} ^ {2} - 2 \boldsymbol {p} _ {s a} ^ {i} ^ {\top} \hat {\boldsymbol {p}} _ {s a} ^ {i} + \left\| \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {2} ^ {2} + \alpha \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ = \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\| \boldsymbol {p} _ {s a} ^ {i} \right\| _ {2} ^ {2} - \left(2 \hat {\boldsymbol {p}} _ {s a} ^ {i} - \alpha \boldsymbol {b} _ {s a}\right) ^ {\top} \boldsymbol {p} _ {s a} ^ {i} + \left\| \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {2} ^ {2} \\ = - \frac {\alpha^ {2} \| \boldsymbol {b} _ {s a} \| _ {2} ^ {2}}{4} + \alpha \boldsymbol {b} _ {s a} ^ {\top} \hat {\boldsymbol {p}} _ {s a} ^ {i} + \min  _ {\boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}} \left\| \boldsymbol {p} _ {s a} ^ {i} - \frac {2 \hat {\boldsymbol {p}} _ {s a} ^ {i} - \alpha \boldsymbol {b} _ {s a}}{2} \right\| _ {2} ^ {2}. \\ \end{array}
$$
So it suffices to solve  $\min_{\pmb{p}_{sa}^i \in \Delta_S} \left\| \pmb{p}_{sa}^i - \frac{2\hat{\pmb{p}}_{sa}^i - \alpha \pmb{b}_{sa}}{2} \right\|_2^2$ , which can be done by Euclidean projection algorithm (Wang and Carreira-Perpinan, 2013) with time complexity  $\mathcal{O}(S \log S)$ .
Proof of Corollary 5.3.1. The result is the direct consequence of Theorem 4.5 with  $h_2(S)$  is  $\mathcal{O}(S \log S)$ , provided by Theorem 5.3.
A.2.3 Proof of results in Section 5.3
Proof of Theorem 5.4. We claim that the Algorithm 3 solves problem (13) with time complexity  $\mathcal{O}(S\log S)$ . By expanding the  $\infty$ -norm, we formulate (13) as the following box constraints problem.
$$
\begin{array}{l l} \min  & \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ \text {s . t .} & \max  \left\{\boldsymbol {0}, \hat {\boldsymbol {p}} _ {s a} ^ {i} - \theta \boldsymbol {e} \right\} \leq \boldsymbol {p} _ {s a} ^ {i} \leq \min  \left\{\boldsymbol {e}, \hat {\boldsymbol {p}} _ {s a} ^ {i} + \theta \boldsymbol {e} \right\}, \\ & \boldsymbol {e} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} = 1, \\ & \boldsymbol {p} _ {s a} ^ {i} \in \mathbb {R} ^ {S}. \end{array}
$$
To get the optimal solution, we put the probability on the index where  $\pmb{b}_{sa}$  is small as much as possible. Specifically, we assume  $b_{sa1} < b_{sa2} < \dots < b_{saS}$  w.l.o.g., and assume  $k$  is the smallest index such that  $\sum_{j=1}^{k} (\hat{p}_{sa j}^i + \theta) \geq 1$ . We claim that
$$
p _ {s a j} ^ {i, \star} = \left\{ \begin{array}{l l} & \hat {p} _ {s a j} ^ {i} + \theta \quad \text {i f} 1 \leq j \leq k - 1 \\ 1 - & \sum_ {\ell = 1} ^ {k - 1} \left(\hat {p} _ {s a \ell} ^ {i} + \theta\right) \quad \text {i f} j = k \\ & 0 \quad \text {o t h e r w i s e}. \end{array} \right.
$$
is the optimal solution to the above formulation. To see this, suppose  $\bar{\pmb{p}}$  is optimal and there is some  $\hat{j} \in [k - 1]$  with  $\bar{p}_{\hat{j}} \neq \hat{p}_{sa j}^{i} + \theta$ . By above box constraint, we get  $\bar{p}_{\hat{j}} < \hat{p}_{sa\hat{j}}^{i} + \theta$ , so there exists  $\bar{j} \geq k$  such that  $\bar{p}_{\bar{j}} > p_{sa\bar{j}}^{i,\star}$ . By moving the probability from  $\bar{p}_{\bar{j}}$  to  $\bar{p}_{\hat{j}}$ , we can achieve strictly smaller objective value, which is a contradiction with that  $\bar{\pmb{p}}$  is optimal. This gives us an optimal solution with the first  $k - 1$  components coincides  $p_{sa}^{i,\star}$ . Then we can get  $p_{sa}^{i,\star}$  is indeed optimal by putting the
extra  $1 - \sum_{\ell = 1}^{k - 1}\left(\hat{p}_{sa\ell}^i +\theta\right)$  probability on  $p_{sak}^{i,\star}$ , since  $b_{sak}\leq \dots \leq b_{saS}$ .
The major time complexity of Algorithm 3 is sorting the vector  $\pmb{b}_{sa} \in \mathbb{R}^S$ , which is  $\mathcal{O}(S\log S)$ .
Proof of Corollary 5.4.1. As for each  $a \in \mathcal{A}$ , we need to sort  $\pmb{b}_{sa}$ , which costs  $\mathcal{O}(AS\log S)$ , then we need to solve  $NA$  subproblems, which costs  $\mathcal{O}(NAS)$ , so the whole problem is computed in time  $\mathcal{O}(AS\log S + NAS)$ .
B Appendix: Computational Complexity for General Convex Optimization
To compare our algorithm with general convex optimization algorithm, we use general convex optimization to compute the problem (5) and problem (8). The time complexities will be discussed in different situations:
- Suppose  $q = 1$ : For general convex optimization problem, problem (5) is equivalent with the following problem:
$$
[ \mathfrak {T} (\pmb {v}) ] _ {s} = \left[ \begin{array}{l l} & {\mathrm {m i n i m i z e}} \quad \gamma \\ & {\mathrm {s u b j e c t t o}} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} (\pmb {r} _ {s a} + \lambda \pmb {v}) ^ {\top} \pmb {p} _ {s a} ^ {i} \leq \gamma , \forall a \in \mathcal {A} \\ & {\quad \quad \quad \quad \quad \quad \quad \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {a \in \mathcal {A}} \sum_ {s ^ {\prime} \in \mathcal {S}} \left| p _ {s a s ^ {\prime}} ^ {i} - \hat {p} _ {s a s ^ {\prime}} ^ {i} \right| \leq \theta} \\ & {\quad \quad \quad \quad \quad \quad \gamma \in \mathbb {R}, \pmb {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ], \forall a \in \mathcal {A}.} \end{array} \right] \quad \forall s \in \mathcal {S}.
$$
By introducing the variables  $t_{\text{sas}'}^i = \left| p_{\text{sas}'}^i - \hat{p}_{\text{sas}'}^i \right|$ ,  $\forall i \in [N]$ ,  $\forall a \in \mathcal{A}$ ,  $\forall s' \in S$ , the above problem is equivalent with
$$
\begin{array}{l} \begin{array}{l l} \text {m i n i m i z e} & \gamma \\ \text {s u b j e c t t o} & \frac {1}{N} \sum_ {i = 1} ^ {N} (\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}) ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \leq \gamma ,   \forall a \in \mathcal {A} \end{array} \\ \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {a \in \mathcal {A}} \sum_ {s ^ {\prime} \in \mathcal {S}} t _ {s a s ^ {\prime}} ^ {i} \leq \theta \\ t _ {s a s ^ {\prime}} ^ {i} \geq p _ {s a s ^ {\prime}} ^ {i} - \hat {p} _ {s a s ^ {\prime}} ^ {i}, t _ {s a s ^ {\prime}} ^ {i} \geq \hat {p} _ {s a s ^ {\prime}} ^ {i} - p _ {s a s ^ {\prime}} ^ {i}, \forall i \in [ N ], \forall a \in \mathcal {A}, \forall s ^ {\prime} \in \mathcal {S} \\ \gamma \in \mathbb {R}, \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, t _ {s a s ^ {\prime}} ^ {i} \in \mathbb {R}, \forall i \in [ N ], \forall a \in \mathcal {A}, \forall s ^ {\prime} \in S. \\ \end{array}
$$
There are  $1 + NSA + NSA = \mathcal{O}(NSA)$  decision variables, and the number of bits in the input is  $\mathcal{O}(1) + \mathcal{O}(NAS) + \mathcal{O}(NAS) + \mathcal{O}(NAS) + \mathcal{O}(NAS) + \mathcal{O}(NAS) = \mathcal{O}(NAS)$ . So by (Karmarkar, 1984), the complexity of solving this LP is  $\mathcal{O}(N^{4.5}S^{4.5}A^{4.5})$ .
We utilize our outer bisection, solving (6) directly using convex optimization. Typically, for each fixed  $a \in \mathcal{A}$  and  $\gamma$ , (6) is equivalent with the LP that
$$
\begin{array}{l} \mathrm {m i n i m i z e} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {s ^ {\prime} \in \mathcal {S}} t _ {s a s ^ {\prime}} ^ {i} \\ \text {s u b j e c t} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \leq \gamma \\ t _ {s a s ^ {\prime}} ^ {i} \geq p _ {s a s ^ {\prime}} ^ {i} - \hat {p} _ {s a s ^ {\prime}} ^ {i}, t _ {s a s ^ {\prime}} ^ {i} \geq \hat {p} _ {s a s ^ {\prime}} ^ {i} - p _ {s a s ^ {\prime}} ^ {i}, \forall i \in [ N ], \forall s ^ {\prime} \in \mathcal {S} \\ \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, t _ {s a s ^ {\prime}} ^ {i} \in \mathbb {R}, \forall i \in [ N ], \forall s ^ {\prime} \in S. \\ \end{array}
$$
There are  $2NS$  decision variables, and the number of bits in the input is  $\mathcal{O}(NS + NS + NS + NS) = \mathcal{O}(NS)$ . So the complexity of solving this LP is  $\mathcal{O}(N^{4.5}S^{4.5})$ . Together with the outer bisection, the total complexity for each Bellman update with  $\epsilon_{1}$  tolerance is  $\mathcal{O}(N^{4.5}S^{4.5}A\log \epsilon_{1}^{-1})$ .
We utilize our nested bisection scheme, solving (9) using the general convex optimization algorithm. Typically, for each fixed  $a \in \mathcal{A}$ ,  $i \in [N]$  and  $\alpha$ , (9) is equivalent with
$$
\begin{array}{l} \text {m i n i m i z e} \quad \sum_ {s ^ {\prime} \in \mathcal {S}} t _ {s a s ^ {\prime}} ^ {i} + \alpha \cdot \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ \begin{array}{l l} \text {s u b j e c t t o} & t _ {s a s ^ {\prime}} ^ {i} \geq p _ {s a s ^ {\prime}} ^ {i} - \hat {p} _ {s a s ^ {\prime}} ^ {i},   t _ {s a s ^ {\prime}} ^ {i} \geq \hat {p} _ {s a s ^ {\prime}} ^ {i} - p _ {s a s ^ {\prime}} ^ {i},   \forall s ^ {\prime} \in \mathcal {S} \\ & \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S},   t _ {s a s ^ {\prime}} ^ {i} \in \mathbb {R},   \forall s ^ {\prime} \in \mathcal {S}. \end{array} \\ \end{array}
$$
There are  $2S$  decision variables, and the number of bits in the input is  $\mathcal{O}(S + S + S) = \mathcal{O}(S)$ . So the complexity of solving this LP is  $\mathcal{O}(S^{4.5})$ . Together with the nested bisection, the total complexity for each Bellman update where the tolerances of bisections are  $\epsilon_{1}$  and  $\epsilon_{2}$ , is  $\mathcal{O}(NS^{4.5}A\log \epsilon_1^{-1}\log \epsilon_2^{-1})$ .
- Suppose  $q = 2$ : For general convex optimization problem, problem (5) is equivalent with the following SOCP:
$$
[ \mathfrak {T} (\boldsymbol {v}) ] _ {s} = \left[ \begin{array}{c c} \text {m i n i m i z e} & \gamma \\ \text {s u b j e c t t o} & \frac {1}{N} \sum_ {i = 1} ^ {N} (\boldsymbol {r} _ {s a} + \lambda \boldsymbol {v}) ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \leq \gamma , \forall a \in \mathcal {A} \\ & \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {a \in \mathcal {A}} \| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \| _ {2} ^ {2} \leq \theta^ {2} \\ & \gamma \in \mathbb {R}, \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ], \forall a \in \mathcal {A}. \end{array} \right] \quad \forall s \in \mathcal {S}.
$$
There are  $1 + NSA = \mathcal{O}(NSA)$  decision variables, and the number of constraints are  $A + 1 + NA(S + 2) = \mathcal{O}(NAS)$ . So the complexity of solving the SOCP with  $\epsilon$ -accuracy is  $\mathcal{O}(\sqrt{NAS}\log \epsilon^{-1}\cdot (NSA)^2 (NAS + 1 + 2(A + NA(S + 2)))) = \mathcal{O}(N^{3.5}S^{3.5}A^{3.5}\log \epsilon^{-1})$ .
We utilize our outer bisection, solving (6) directly using convex optimization. Typically, for each fixed  $a \in \mathcal{A}$  and  $\gamma$ , (6) is equivalent with the SOCP that
$$
\text {m i n i m i z e} \quad \delta
$$
$$
\begin{array}{l} \text {s u b j e c t} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \leq \gamma \\ \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {2} ^ {2} \leq \delta \\ \delta \in \mathbb {R}, \boldsymbol {p} _ {s a} ^ {i} \in \Delta_ {S}, \forall i \in [ N ]. \\ \end{array}
$$
There are  $1 + NS = \mathcal{O}(NS)$  decision variables, and the number of constraints are  $\mathcal{O}(NS)$ . So the complexity of solving the above SOCP with  $\epsilon$ -accuracy is  $\mathcal{O}(\sqrt{NS}\log \epsilon^{-1}\cdot N^2 S^2 (NS + 1 + 2(1 + N(S + 2)))) = \mathcal{O}(N^{3.5}S^{3.5}\log \epsilon^{-1})$ , hence the total complexity of the Bellman update is  $\mathcal{O}(N^{3.5}S^{3.5}A\log \epsilon^{-1}\log \epsilon_1^{-1})$ .
We utilize our nested bisection scheme, solving problem (9) using the general convex optimization algorithm. For each fixed  $a \in \mathcal{A}$ ,  $i \in [N]$  and  $\alpha$ , problem (9) is equivalent with
$$
\begin{array}{l} \text {m i n i m i z e} \quad \delta + \alpha \cdot \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ \text {s u b j e c t} \quad \left\| \boldsymbol {p} _ {s a} ^ {i} - \hat {\boldsymbol {p}} _ {s a} ^ {i} \right\| _ {2} ^ {2} \leq \delta \\ \boldsymbol {p} _ {s a} ^ {t} \in \Delta_ {S}, \delta \in \mathbb {R}. \\ \end{array}
$$
There are  $1 + S = \mathcal{O}(S)$  decision variables, and the number of constraints are  $\mathcal{O}(S)$ . So the complexity of solving the above SOCP with  $\epsilon$ -accuracy is  $\mathcal{O}(\sqrt{S}\log \epsilon^{-1}\cdot S^2 (S + 1 + 2(1 + S))) = \mathcal{O}(S^{3.5}\log \epsilon^{-1})$ , hence the total complexity of the Bellman update is  $\mathcal{O}(NS^{3.5}A\log \epsilon^{-1}\log \epsilon_1^{-1}\log \epsilon_2^{-1})$ .
- Suppose  $q = \infty$ : We can also consider solving the inner problem in (10) using general convex optimization, which is equivalent with
$$
\begin{array}{l} \text {m i n i m i z e} \quad \boldsymbol {b} _ {s a} ^ {\top} \boldsymbol {p} _ {s a} ^ {i} \\ \mathrm {s u b j e c t t o} p _ {s a s ^ {\prime}} ^ {i} - \hat {p} _ {s a s ^ {\prime}} ^ {i} \leq \theta , \hat {p} _ {s a s ^ {\prime}} ^ {i} - p _ {s a s ^ {\prime}} ^ {i} \leq \theta , \forall s ^ {\prime} \in \mathcal {S} \\ \boldsymbol {p} _ {s a} ^ {\iota} \in \Delta_ {S}, \\ \end{array}
$$
Table 1: Comparisons of runtime (average and standard deviation)(second) of Bellman updates for all algorithms in  $L_{1}$  norm.
Algorithm\N=S=AlGORITHM	10	20	30	40
Fast	0.0253 (0.02)	0.1243 (0.02)	0.2794 (0.02)	0.5112 (0.02)
Gurobi	0.039 (0.02)	0.7084 (0.03)	4.3617 (0.05)	22.6013 (0.92)
FOM(3 its)	0.644 (1.0193)	4.0821 (1.05)	12.6608 (1.00)	29.22 (1.18)
Table 2: Comparisons of runtime (average and standard deviation)(second) of Bellman updates for all algorithms in  $L_{2}$  norm.
Algorithm\N=S=AlGORITHM	50	60	70	80
Fast	7.3717 (0.11)	10.8293 (0.12)	16.8888 (0.12)	24.6080 (0.07)
Gurobi	5.1669 (0.09)	11.6760 (0.14)	23.6852 (0.34)	44.3305 (0.18)
FOM(3 its)	9.1936 (1.83)	14.5653 (2.01)	21.3651 (1.97)	31.1326 (1.79)
where  $i\in [N]$  and  $a\in \mathcal{A}$  are fixed.
There are  $\tilde{S}$  decision variables, and the number of bits in the input is  $\mathcal{O}(S + S + S) = \mathcal{O}(S)$ . So the complexity of solving this LP is  $\mathcal{O}(S^{4.5})$ . Then the total complexity for each Bellman update is  $\mathcal{O}(NS^{4.5}A)$ .
C Appendix: Details for Numerical Experiments
We compare our fast algorithm with the state-of-the-art solver Gurobi with version v10.0.1rc0 (Gurobi Optimization, LLC, 2023) and the first-order method of (Grand-Clément and Kroer, 2021a). All experiments are implemented in Python 3.8, and they are run on a 2.3 GHz 4-Core Intel Core i7 CPU with 32 GB 3733 MHz DDR4 main memory. We will release our code to ensure reproducibility. https://github.com/Chill-zd/Fast-Bellman-Updates-DRMDP
Our algorithms are tested on some random instances generated by the Generalized Average Reward Non-stationary Environment Test-bench (Garnet MDPs) (Archibald et al., 1995; Bhatnagar et al., 2009). The Garnet MDPs are a collection of assessment problems designed to assess the performance of reinforcement learning algorithms in non-stationary environments. It is convenient to construct and implement these problems. We utilize the parameter  $n_b$  to regulate the proportion of the next states accessible for each state-action pair  $(s, a)$ . Following the same setting as (Grand-Clément and Kroer, 2021a), we set  $n_b$  to be 0.2 and random uniform rewards to be in [0, 10]. The discount factor  $\lambda$  is fixed at 0.8, and parameter  $\epsilon = 0.1$ . For each MDP instance, we generate the sampled kernels  $p^1, \ldots, p^N$ , considering  $N$  small random (Garnet) perturbations around the nominal kernel  $p^0$ . We set parameter  $\theta$  in Proposition 4.1 to be  $\sqrt{n_bA}$ .
To test the speed of Bellman update, we run the random instances 50 times for all the algorithms, and show the average time of them in tables 1, 2 and 3. We can see that our algorithm performs better than Gurobi and the first-order method. When the states number increases, the running time of our algorithm keeps a small standard deviation.
We also compare the speed of value iteration for all algorithms using the same convergence criteria:  $\| \pmb{v} - \pmb{v}^{\star}\|_{\infty}\leq 2\lambda \epsilon (1 - \lambda)^{-1}$ , which follows (Grand-Clément and Kroer, 2021a). The results are shown in tables 4, 5 and 6. The runtimes that exceed 4000s for  $L_{1}$  and  $L_{\infty}$  case or exceed 10000s for
Table 3: Comparisons of runtime (average and standard deviation) (milliseconds) of Bellman updates for all algorithms in  $L_{\infty}$  norm.
Algorithm\N=S=AlGORITHM	10	20	30	40
Fast	0.06 (0.1)	0.21 (0.1)	0.47 (0.2)	0.80 (0.4)
Gurobi	0.92 (0.1)	7.33 (0.8)	26.50 (3.8)	65.22 (7.9)
FOM	144.92 (981.1)	166.19 (966.5)	237.24 (978.2)	366.74 (976.6)
$L_{2}$  case will be shown as “-”. We can see from tables 4, 5 and 6 that our algorithm always performs better than Gurobi and the first-order method. We point out that our algorithm generally becomes better as the state number increases.
Table 4: Runtime (second) of value iteration for all algorithms in  $L_{1}$  norm.
Algorithm N = S = A	10	20	30	40
Fast	3.6164	34.0930	116.4698	281.9007
Gurobi	5.7131	202.0723	1967.9552	-
FOM	202.4039	3578.1377	-	-
Table 5: Runtime (second) of value iteration for all algorithms in  $L_{2}$  norm.
Algorithm N = S = A	40	50	60	70
Fast	2198.5413	4847.4994	7223.7663	12387.5601
Gurobi	2543.9588	4861.4114	7949.4562	22096.7578
FOM	-	-	-	-
Table 6: Runtime (second) of value iteration for all algorithms in  $L_{\infty}$  norm.
Algorithm N = S = A	10	20	30	40
Fast	0.0045	0.0348	0.1371	0.3181
Gurobi	0.0907	0.8746	3.9243	11.5589
FOM	16.4344	97.44293	1131.5260	3933.8585