SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the continuous setting, we conduct empirical evaluations to assess the robustness of four heuristic methods: deep-bottom-left (DBL) method (Karabulut and Inceoglu, 2005), best-match-first (BMF) method (Li and Zhang, 2015), least-surface-area heuristic (LSAH) (Hu et al., 2017), and online bin packing heuristics (OnlineBPH) (Ha et al., 2017). In addition, we evaluated the packing configuration tree (PCT) algorithm (Zhao et al., 2022a) for the DRL-based methods, as other DRL-based methods are limited to spatially discretized grid world. We also evaluate the robustness of each algorithm under varying attack capabilities by selecting different values for  $N_B$ , including 5, 10, 15, and 20. Since except PCT, other methods can observe only the first item in the item sequence at each time step, the observable number of items of each packing strategy is fixed at 1 to ensure a fair comparison between these packing strategies.
Table 1 displays the performance of different packing strategies against permutations generated by their corresponding permutation-based attackers. We can see that the LSAH algorithm exhibits superior space utilization performance across various experimental settings with different values of  $N_B$ , even though it does not perform as well as the OnlineBPH and DBL algorithms in the nominal dynamics. In contrast, the BMF method is the most susceptible to attacks across all settings with varying numbers of observable items used by permutation-based attackers. With the increase of the attack capability, the PCT algorithm consistently outperforms the heuristic methods in terms of space utilization and the number of packed items, which can be attributed to its generalizability across a large number of problem instances. Similar to the results in the discrete setting, increasing the value of  $N_B$  results in more performance degradation for all methods, which suggests an opportunity to explore more challenging problem instances, and to design proper robust algorithms accordingly.
Table 1: The performance of existing methods under the perturbation in the continuous setting.
Methods	NB=5	NB=10	NB=15	NB=20	w/o attack
Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.
DBL	37.8	8.7	17.3	30.7	8.3	16.1	27.1	6.5	16.1	23.0	7.0	15.9	48.3	10.5	18.6
BMF	27.5	8.6	13.3	21.6	7.3	12.5	17.4	6.5	12.0	14.1	4.7	11.0	42.1	8.2	16.4
LSAH	38.7	10.9	16.7	36.9	9.1	17.2	33.2	10.0	16.3	27.1	8.4	17.6	47.5	8.6	18.4
OnlineBPH	33.8	11.6	15.3	30.2	7.4	16.5	26.4	7.2	16.2	23.3	6.8	15.6	49.1	9.7	19.0
PCT	47.7	8.4	20.4	39.0	8.7	18.5	36.4	9.5	18.8	30.4	8.8	17.9	65.6	8.7	25.0
Figure 1 depicts item distributions under perturbations from different attackers using PCT packing policies in the discrete setting. Since the sizes  $s^d$  of items follow a uniform discrete distribution, the frequency of the sampled sizes are approximately equal, as illustrated in the leftmost bar chart in Figure 1. It is apparent that as the number of observable items of the attacker increases, the attacker
Figure 1: Item distributions in discrete settings with varying values of  $N_B$  for different attackers.
tends to select smaller items. Furthermore, as the attack capability increases, the underlying item distribution undergoes more significant changes. Therefore, in our setting, restricting the number of observable items is a practical approach to create permuted new problem instances that are closer to challenging real-world scenarios, as opposed to allowing an unlimited number of observable items.
In Figure 2, the item distributions from time step 1 to time step 20 are illustrated over different problem instances under perturbations from different attackers, using PCT packing policies in the discrete setting with  $N_B = 15,20$ . These item distributions not only partially reflect the problem instance distribution, but also reveal the preference of the attacker at each time step. At the beginning, the attacker favors items with smaller sizes, but as the time steps increase, it tends to sample items uniformly. This variation tendency illustrates the behavior of the attacker. However, it does not necessarily imply that rearranging the item sequence in ascending order based on item volumes would be an optimal or even effective attack. As selecting smaller items intuitively reduces the likelihood of forming supportive planes for future incoming items, attackers tend to choose smaller items to minimize the space utilization of the packing policy. Therefore, essentially, the attacker aims to select items that cannot be used to construct a supportive plane, rather than blindly choosing smaller items. This also explains why the probability of larger items in the beginning phase does not decrease to zero in Figure 2. As shown in Figure 3, we use a toy example to demonstrate that rearranging the item sequence in ascending order based on item sizes would not be an optimal or even effective attack. In this example, the container sizes  $S^d$ ,  $d \in \{x,y,z\}$  are set to 2, and the item sequence consists of four items, denoted as  $\mathbf{B} = \{b^{1},b^{2},b^{3},b^{4}\}$ . The sizes of the items are as follows:  $b^{1} = (1\times 2\times 2)$ ,  $b^{2} = b^{3} = (1\times 1\times 1)$ , and  $b^{4} = (1\times 1\times 2)$ . We generate a new item sequence  $\mathbf{B}' = \{b^{2},b^{3},b^{4},b^{1}\}$  by rearranging  $\mathbf{B}$  in ascending order based on item sizes. However, this rearrangement is not an effective attack since the optimal solution can still be obtained using a packing strategy. By comparison, moving both  $b^{1}$  and  $b^{4}$  from  $\mathbf{B}$  to the middle between  $b^{2}$  and  $b^{3}$  results in  $\mathbf{B}''$ , which is an effective attack for achieving minimal space utilization. In such a case, the packing agent cannot pack  $b^{1}$  into the container at the third time step.
A.1.2 Performance of AR2L Algorithm
In the continuous setting, we benchmark four other RL algorithms that are adaptable to online 3D-BPP, including the packing configuration tree (PCT) (Zhao et al., 2022a) method, the CVaR-Proximal-Policy-Optimization (CPPO) (Ying et al., 2022) algorithm, the robust adversarial reinforcement learning (RARL) (Pinto et al., 2017) algorithm, and the robust  $f$ -divergence MDP (RfMDP) (Ho et al., 2022; Panaganti et al., 2022) algorithm. Since both the exact AR2L (ExactAR2L) and approximate AR2L (ApproxAR2L) algorithms can adjust the value of  $\alpha$  to develop policies that vary in their level of robustness, we use different values of  $\alpha = 0.3, 0.5, 0.7, 1.0$  to explore the robustness trends with varying values of  $\alpha$  in this setting. Likewise, we construct mixture datasets by randomly selecting  $\beta \%$  nominal box sequences and rearranging them using the learned corresponding permutation-based attacker for each packing policy.
$Uti$ . Table 2 displays the performance of different robust methods that are compatible with online 3D-BPP on datasets with varying number of challenging problem instances in the continuous setting. ExactAR2L  $(\alpha)$  represents the ExactAR2L algorithm taking different values for  $\alpha$ , and similarly for ApproxAR2L  $(\alpha)$ . In the continuous setting, increasing the value of  $\alpha$  can lead to ExactAR2L producing more robust policies against their corresponding permutation-based attackers, as the larger value of  $\alpha$  provides more worst-case problem instances during the training of the packing policy. However, they cannot consistently improve their performance on the nominal dynamics. As
(b)  $N_B = 20$
Figure 2: Item distribution at each time step in the discrete setting. From the top left chart to the bottom right chart, it represents the item distribution from time step 1 to time step 20 in sequence.
Figure 3: The toy example of packing processes with different item sequences.
the number of observable items of the attacker increases, there is a corresponding increase in the distribution deviation between the perturbed data and the nominal data  $(\beta = 0)$ . For the scenario when  $N_B = 5, 10$ , the distribution deviation is comparatively small. Thus, it is acceptable to incorporate more challenging instances by increasing the value of  $\alpha$  to 0.7. This adjustment will further enhance the generalization of the model and lead to improved performance in terms of space utilization compared to PCT. By comparison, in the cases of  $N_B = 15$  and  $N_B = 20$ , ExactAR2L tends to favor  $\alpha = 0.3$  to produce satisfactory performance for  $\beta = 0$ . Thus, by incorporating the perturbed data with small deviation from the nominal data during training, the generalization of the packing policy can be enhanced, leading to the performance improvement. But the larger deviation can degenerate the performance of the packing policy on the nominal datasets, as demonstrated in RARL. Likewise to the tendency in discrete setting, the ExactAR2L algorithm prefers the larger value of  $\alpha$  as the value of  $\beta$  increases. Furthermore, due to the value estimation error, ApproxAR2L cannot perform as well as its exact counterpart in most of tasks under the continuous setting. However, to our surprise, we observed that in the case of  $N_B = 5$ , ApproxAR2L surpasses its corresponding exact counterpart in terms of performance. This could be attributed to the relatively small estimation error resulting from the small deviation in dynamics in this setting, where removing the mixture-dynamics model results in a more stable learning process compared to the exact algorithm. In the remaining tasks, the estimation error in ApproxAR2L can result in more unstable learning, which makes the trend in performance across different values of  $\alpha$  less clear compared to the ExactAR2L algorithm.
$Std$ . As shown in Table 2, in almost all tasks, ExactAR2L with a larger value of  $\alpha$  ( $\alpha = 1.0$ ) exhibits superiority over PCT in terms of  $Std$ . Thus, ExactAR2L can indeed improve the robustness in the continuous setting. However, it should be noted that ExactAR2L does not consistently yield smaller values of  $Std$ . Compared to RARL, this is because the ExactAR2L policy tends to be less conservative than the RARL policy in the majority of tasks. Likewise, ApproxAR2L algorithm prefers larger values of  $\alpha$  to consistently achieve smaller  $Std$ . in different tasks.
Num. In comparison to PCT, the ExactAR2L(1.0) algorithm demonstrates the ability to pack more items in 19 tasks, with a minor decrease observed in 1 task. However, when compared to RARL, the ExactAR2L(1.0) policy only outperforms in 9 tasks. This discrepancy can primarily be attributed to the performance drop observed in the worst-case dynamics of the ExactAR2L algorithm. The ApproxAR2L policy tends to prefer  $\alpha = 0.7$  over the RfMDP policy when it comes to packing more items across all the tasks.
Hyperparameter Configurations. Based on the observations presented in Table 2, it can be concluded that  $\alpha = 1.0$  is the optimal choice for achieving a desired balance between policy performance in both nominal and worst-case scenarios. The ExactAR2L(1.0) algorithm demonstrates competitive robustness and average performance when compared to both PCT and RARL. Compared to the baseline method RfMDP, ApproxAR2L chooses  $\alpha = 0.7$  for the best performance.
In addition, although the convergence rate is not our primary concern in evaluating the RL algorithm, we still intend to observe the trend in the learning curves across varying attack capabilities. Our ExactAR2L algorithm shows greater improvement in terms of convergence rate in both the discrete and continuous settings as the number of observable items of the attacker increases, as demonstrated clearly in Figure 4 5.
A.1.3 Visualization Results
Figure 8 shows a qualitative comparison of the visualized bin packing results obtained by the ExactAR2L, RARL, and PCT policies under the nominal and worst-case dynamics with varying numbers of observable items  $N_B = 15$ , 20 in the discrete setting. As shown in Figure 8(a) 8(b), the ExactAR2L algorithm allows more smaller margin spaces in the container to minimize sizes of those unoccupied spaces. By comparison, the RARL algorithm attempts to create larger spaces, which may pose a risk of future items not fitting into those spaces. The visualization results of the ExactAR2L and PCT algorithms under their respective worst-case dynamics are depicted in Figure 8(c) and 8(d).
(a)  $N_B = 5$
(b)  $N_B = 10$
(c)  $N_B = 15$
(d)  $N_B = 20$
Figure 4: The learning curves of robust RL algorithms for nominal problem instances with different values of  $N_B$  in the discrete setting.
(a)  $N_B = 5$
Figure 5: The learning curves of robust RL algorithms for nominal problem instances with different values of  $N_B$  in the continuous setting.
One observation is that attackers designed for packing policies tend to prioritize smaller items at the beginning, as a majority of smaller items are likely to occur at the bottom of the container. In such a scenario, the ExactAR2L policy prioritizes the creation of supportive planes for future items, even at the cost of producing smaller margin spaces.
A.1.4 Baselines Comparisons
Since we implement PCT with different training configurations, we conducted a comparison between our implementation and the official implementation in this section. We conducted the comparison of the algorithms in terms of convergence rate across various values of  $N_B$ , specifically  $N_B = 1, 5, 10, 15, 20$ . To ensure a fair and consistent comparison, we employed the same number of problem instances during training. Each iteration utilized a fixed rollout length of 30, and we maintained a constant number of 64 parallel processes throughout the experiments. As shown in Figure 6 and Figure 7, our results clearly demonstrate that our implementation exhibits a faster convergence rate across different settings, considering various numbers of observable items.
A.1.5 Practicability of AR2L Algorithm
To validate the practicality of exact AR2L algorithm in real-world scenarios, we directly evaluated our model without retraining on the Mixed-item Dataset (MI Dataset) (Yuan et al., 2023; Elhedhli et al., 2019) which follows the generation scheme proposed by Elhedhli et al. (2019) for the realistic 3D-BPP instance generator. MI dataset has 10 thousand items, with 4668 species, and occurrences vary from 1 to 15. The pallet dimensions is set to the size often used in practice:  $S^x = 120$ ,  $S^y = 100$ , and  $S^z = 100$ . The results are presented in the Table 3. It is evident that in both settings of  $N_B = 15$  and  $N_B = 20$ , our exact AR2L algorithm demonstrates superior performance compared to PCT and RARL across various metrics in the real-world dataset.
Table 3: The performance of PCT, RARL and exact AR2L evaluated on the Mixed-Item Dataset
Methods	NB=15	NB=20
Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.
PCT	48.3	8.5	16.8	48.7	10.1	16.9
RARL	48.8	8.8	16.9	48.8	8.3	16.9
AR2L	50.2	8.5	17.4	52.9	6.5	18.3
(a)  $N_B = 1$
(b)  $N_B = 5$
(c)  $N_B = 10$
(d)  $N_B = 15$
(e)  $N_B = 20$
Figure 6: The training curves of our implementation and the official implementation for PCT in the discrete setting.
(a)  $N_B = 1$
(d)  $N_B = 15$
Figure 7: The training curves of our implementation and the official implementation for PCT in the continuous setting.
(a) ExactAR2L and RARL algorithms in the nominal dynamics with  $N_B = 15$ .
(b) ExactAR2L and RARL algorithms in the nominal dynamics with  $N_B = 20$ .
(c) ExactAR2L and PCT algorithms in the worst-case dynamics with  $N_B = 15$ .
(d) ExactAR2L and PCT algorithms in the worst-case dynamics with  $N_B = 20$ .
Figure 8: Visualization results of various methods. The space utilization values for each sub-figure are displayed alongside the respective visualization.
A.2 Proofs
Lemma 1. [Achiam et al. (2017), Appendix.10.1.1] Given a policy  $\pi (\cdot |\cdot)$  , a transition model  $P(\cdot |s,a)$  , and an initial state distribution  $\mu (s)$  , the discounted state visitation distribution  $d_{\pi ,P}(s)$  under policy  $\pi$  can be described as:
$$
d _ {\pi , P} (s) = (1 - \gamma) \mu (s) + \gamma \sum_ {s ^ {\prime}, a} d _ {\pi , P} \left(s ^ {\prime}\right) \pi \left(a \mid s ^ {\prime}\right) P \left(s \mid s ^ {\prime}, a\right) \tag {1}
$$
Theorem 1. The model discrepancy between two models can be described as  $d(P^1 ||P^2) \triangleq \max_{s,a} D_{TV}(P^1(\cdot | s, a)) || P^2(\cdot | s, a))$ . The lower bound for the newly defined objective is derived as follows:
$$
\eta (\pi , P ^ {o}) + \alpha \eta (\pi , P ^ {w}) \geq (1 + \alpha) \eta (\pi , P ^ {m}) - \frac {2 \gamma | r | _ {\max }}{(1 - \gamma) ^ {2}} \left(d \left(P ^ {m} \| P ^ {o}\right) + \alpha d \left(P ^ {m} \| P ^ {w}\right)\right) \tag {2}
$$
Proof. Given two dynamics models  $P^1(\cdot|s, a)$  and  $P^2(\cdot|s, a)$ , the absolute value of the difference of returns of a policy  $\pi$  can be represented as
$$
\left| \eta (\pi , P ^ {1}) - \eta (\pi , P ^ {2}) \right| = \left| \mathrm {E} _ {s \sim \mu (s)} [ V ^ {P ^ {1}} (s) - V ^ {P ^ {2}} (s) ] \right| \leq \mathrm {E} _ {s \sim \mu (s)} | V ^ {P ^ {1}} (s) - V ^ {P ^ {2}} (s) |, \quad (3)
$$
Here, we set  $\Delta V(s) = V^{P^1}(s) - V^{P^2}(s)$ , and it can be transformed as:
$$
\begin{array}{l} \Delta V (s) = \mathrm {E} _ {a \sim \pi} [ r (s, a) + \gamma \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} (V ^ {P ^ {1}} (s ^ {\prime})) ] - \mathrm {E} _ {a \sim \pi} [ r (s, a) + \gamma \mathrm {E} _ {s ^ {\prime} \sim P ^ {2}} (V ^ {P ^ {2}} (s ^ {\prime})) ] \\ = \gamma \underbrace {\mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \left(V ^ {P ^ {1}} \left(s ^ {\prime}\right)\right) - \gamma \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {2}} \left(V ^ {P ^ {2}} \left(s ^ {\prime}\right)\right)} _ {①} \tag {4} \\ \end{array}
$$
The term ① can be firstly written as:
$$
\begin{array}{l} \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \big (V ^ {P ^ {1}} (s ^ {\prime}) \big) - \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {2}} \big (V ^ {P ^ {2}} (s ^ {\prime}) \big) \\ = \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \big (V ^ {P ^ {1}} \big (s ^ {\prime} \big) \big) - \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \big (V ^ {P ^ {2}} \big (s ^ {\prime} \big) \big) + \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \big (V ^ {P ^ {2}} \big (s ^ {\prime} \big) \big) - \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {2}} \big (V ^ {P ^ {2}} \big (s ^ {\prime} \big)) \\ = \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \Delta V (s ^ {\prime}) + \mathrm {E} _ {a \sim \pi} \sum_ {s ^ {\prime}} \left(P ^ {1} \left(s ^ {\prime} \mid s, a\right) - P ^ {2} \left(s ^ {\prime} \mid s, a\right)\right) V ^ {P ^ {2}} \left(s ^ {\prime}\right) \tag {5} \\ \end{array}
$$
We then derive an upper bound of the second term in Equation 5 as:
$$
\begin{array}{l} \mathrm {E} _ {a \sim \pi} \sum_ {s ^ {\prime}} (P ^ {1} (s ^ {\prime} | s, a) - P ^ {2} (s ^ {\prime} | s, a)) V ^ {P ^ {2}} (s ^ {\prime}) \\ \leq \max _ {s ^ {\prime}} V ^ {P ^ {2}} (s ^ {\prime}) \mathrm {E} _ {a \sim \pi} \sum_ {s ^ {\prime}} (P ^ {1} (s ^ {\prime} | s, a) - P ^ {2} (s ^ {\prime} | s, a)) \\ \leq \frac {2 | r | _ {\operatorname* {m a x}}}{1 - \gamma} \frac {1}{2} \mathrm {E} _ {a \sim \pi} \sum_ {s ^ {\prime}} \left(P ^ {1} \left(s ^ {\prime} \mid s, a\right) - P ^ {2} \left(s ^ {\prime} \mid s, a\right)\right) \tag {6} \\ = \frac {2 | r | _ {\mathrm {m a x}}}{1 - \gamma} \mathrm {E} _ {a \sim \pi} D _ {T V} (P ^ {1} (\cdot | s, a) | | P ^ {2} (\cdot | s, a)). \\ \end{array}
$$
Next, Equation 5 and 6 can be plugged into Equation 4, written as:
$$
\Delta (s) \leq \gamma \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \Delta V \left(s ^ {\prime}\right) + \frac {2 \gamma | r | _ {\max }}{1 - \gamma} \mathrm {E} _ {a \sim \pi} D _ {T V} \left(P ^ {1} (\cdot | s, a) | | P ^ {2} (\cdot | s, a)\right) \tag {7}
$$
Thus, the expectation of  $\Delta (s)$  can be further upper bounded by:
$$
\begin{array}{l} \operatorname {E} _ {s \sim d _ {\pi , P 1}} \Delta (s) \\ \leq \mathrm {E} _ {s \sim d _ {\pi , P ^ {1}}} \left[ \gamma \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \Delta V (s ^ {\prime}) + \frac {2 \gamma | r | _ {\max}}{1 - \gamma} \mathrm {E} _ {a \sim \pi} D _ {T V} (P ^ {1} (\cdot | s, a) | | P ^ {2} (\cdot | s, a)) \right] \\ = \gamma \mathrm {E} _ {s \sim d _ {\pi , P ^ {1}}} \left[ \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \Delta V (s ^ {\prime}) \right] + \frac {2 \gamma | r | _ {\max}}{1 - \gamma} \mathrm {E} _ {s \sim d _ {\pi , P ^ {1}}} \left[ \mathrm {E} _ {a \sim \pi} D _ {T V} \left(P ^ {1} (\cdot | s, a) | | P ^ {2} (\cdot | s, a)\right) \right] \tag {8} \\ \leq \gamma \mathrm {E} _ {s \sim d _ {\pi , P ^ {1}}} [ \mathrm {E} _ {a \sim \pi} \mathrm {E} _ {s ^ {\prime} \sim P ^ {1}} \Delta V (s ^ {\prime}) ] + \frac {2 \gamma | r | _ {\max}}{1 - \gamma} \max _ {s, a} D _ {T V} (P ^ {1} (\cdot | s, a) | | P ^ {2} (\cdot | s, a)) \\ \end{array}
$$
According to Lemma 1, Inequality 8 can be simplified as:
$$
\mathrm {E} _ {s \sim d _ {\pi , P ^ {1}}} \Delta (s) \leq \frac {2 \gamma | r | _ {\max }}{1 - \gamma} \max  _ {s, a} D _ {T V} \left(P ^ {1} (\cdot | s, a) \| P ^ {2} (\cdot | s, a)\right) + \sum_ {s ^ {\prime}} \left[ d _ {\pi , P ^ {1}} \left(s ^ {\prime}\right) - (1 - \gamma) \mu \left(s ^ {\prime}\right) \right] \Delta V \left(s ^ {\prime}\right) \tag {9}
$$
Consequently,  $\mathrm{E}_{s\sim d_{\pi ,P^1}}\Delta (s)$  can be eliminated, and we can obtain:
$$
\left| \eta \left(\pi , P ^ {1}\right) - \eta \left(\pi , P ^ {2}\right) \right| \leq \mathrm {E} _ {s \sim \mu (s)} \left| V ^ {P ^ {1}} (s) - V ^ {P ^ {2}} (s) \right| \leq \frac {2 \gamma | r | _ {\max }}{(1 - \gamma) ^ {2}} \max  _ {s, a} D _ {T V} \left(P ^ {1} (\cdot | s, a) \right\rvert   \| P ^ {2} (\cdot | s, a)) \tag {10}
$$
For the newly defined objective function, we can introduce any unknown dynamics  $P^m$  to obtain:
$$
\eta (\pi , P ^ {o}) - \eta (\pi , P ^ {m}) \geq - \max  _ {s, a} D _ {T V} (P ^ {o} (\cdot | s, a) | | P ^ {m} (\cdot | s, a))
$$
$$
\eta (\pi , P ^ {w}) - \eta (\pi , P ^ {m}) \geq - \max  _ {s, a} D _ {T V} \left(P ^ {w} (\cdot | s, a) \mid \mid P ^ {m} (\cdot | s, a)\right) \tag {11}
$$
Thus, the lower bound of the objective function can be derived as:
$$
\eta (\pi , P ^ {o}) + \alpha \eta (\pi , P ^ {w}) \geq (1 + \alpha) \eta (\pi , P ^ {m}) - \frac {2 \gamma | r | _ {\max }}{(1 - \gamma) ^ {2}} \left(d \left(P ^ {m} \mid \mid P ^ {o}\right) + \alpha d \left(P ^ {m} \mid \mid P ^ {w}\right)\right), \tag {12}
$$
where  $d(P^{1}||P^{2})\triangleq \max_{s,a}D_{TV}(P^{1}(\cdot |s,a)||P^{2}(\cdot |s,a))$
Theorem 2. For any given policy  $\pi$  and its corresponding worst-case dynamics  $P^w$ , the adjust Bellman operator  $\mathcal{T}_a$  is a contraction whose fixed point is  $V_a^\pi$ . The operator equation  $\mathcal{T}_a V_a^\pi = V_a^\pi$  has a unique solution.
Proof. For any given policy  $\pi$  and its corresponding worst-case dynamics  $P^w$ , the adjustable robust Bellman operator  $\mathcal{T}_a$  is defined as:
$$
\mathcal {T} _ {a} V _ {a} ^ {\pi} (s) = \mathbb {E} _ {a \sim \pi} [ r (s, a) + \gamma \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} \left[ V _ {a} ^ {\pi} \left(s ^ {\prime}\right) \right], \tag {13}
$$
where the uncertainty set is defined as  $\mathcal{P}^m = \otimes_{(s,a)\in \mathcal{S}\times \mathcal{A}}\mathcal{P}_{s,a}^m$ ;  $\mathcal{P}_{s,a}^m = \{P_{s,a}\in \Delta (\mathcal{S}): D_{TV}(P_{s,a}||P_{s,a}^o) + \alpha D_{TV}(P_{s,a}||P_{s,a}^w)\leq \rho '\}$ .
For any two adjustable robust value functions  $V_{1}$  and  $V_{2}$ , we have: