SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Observe that  $d_P^\pi$  can be approximated as  $\sum_{n=0}^n (\gamma P^\pi)^n$  for large  $n$  enough, and  $Q_{P,R}^\pi$  can be approximated by dynamic programming [25]. Notably, this step and the second step are negligible as compared to the first step.
Robust Value Iteration by LP
sa-rectangular robust MDPs. We first consider sa-rectangular  $L_{1}$  constrained uncertainty set  $\mathcal{U}_p^{\mathrm{sa}} = \mathcal{P} \times \mathcal{R}$ . Robust Bellman operator is given by
$$
(\mathcal{T}_{\mathcal{U}_{p}^{\mathrm{sa}}}^{\pi}v)(s) = \max_{a}\underbrace{\min_{p\in\mathcal{P}_{sa},r\in\mathcal{R}_{sa}}\left[  r + \gamma\sum_{s^{\prime}}p(s^{\prime})v(s^{\prime})\right]}_{\text{LP with $S$ variable}}.
$$
Note that the above can be solved by  $A$  LPs as uncertainty set  $\mathcal{U}_p^{\mathrm{sa}} = \mathcal{P} \times \mathcal{R}$  induces linear constraint and the objective is also linear with  $S$  variables. Hence, evaluation of  $\mathcal{T}_{\mathcal{U}_p^{\mathrm{sa}}}^\pi v$  requires solving  $SA$  LPs with  $S$  variable each.
$s$ -rectangular robust MDPs. We now consider  $s$ -rectangular  $L_{1}$  constrained uncertainty set  $\mathcal{U}_p^{\mathrm{s}} = \mathcal{P} \times \mathcal{R}$ . Robust Bellman operator is given by
$$
(\mathcal{T}_{\mathcal{U}_{p}^{s}}^{\pi}v)(s) = \underbrace{\min_{p\in\mathcal{P}_{s},r\in\mathcal{R}_{s}}\sum_{a}\pi_{s}(a)\left[  r(a) + \gamma\sum_{s^{\prime}}p(s^{\prime}|a)v(s^{\prime})\right]}_{\text{LP with SA variable}}.
$$
Note that the above can be solved by one LP as uncertainty set  $\mathcal{U}_p^s = \mathcal{P} \times \mathcal{R}$  induces linear constraint and the objective is also linear with SA variables. Hence, evaluation of  $\mathcal{T}_{\mathcal{U}_p^s}^\pi v$  requires solving  $S$  LPs with SA variable each.