SlowGuess/NeurIPS_2023_OCR huggingface dataset:
dynamics for the policy evaluation, and improves the policy under the resulting mixture dynamics. Therefore, we define the adjustable robust value function as  $V_{a}^{\pi} = \sup_{P^{m}\in \mathcal{P}^{m}}V_{a}^{\pi ,P^{m}}$  for the policy evaluation. And the adjustable robust Bellman operator  $\mathcal{T}_a$  can be defined as
$$
\mathcal {T} _ {a} V _ {a} ^ {\pi} (s) = \mathbb {E} _ {a \sim \pi} [ r (s, a) + \gamma \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} \left[ V _ {a} ^ {\pi} \left(s ^ {\prime}\right) \right] ]. \tag {5}
$$
The adjustable robust Bellman operator assumes both the nominal and worst-case dynamics corresponding to  $\pi$  are available for constructing the uncertainty set  $\mathcal{P}^m$ . Unlike the traditional robust Bellman operator (Iyengar, 2005) designed for the minimal value, our proposed operator focus on maximizing the value of a given policy by updating the mixture dynamics. Meanwhile, to avoid the optimistic evaluation, the uncertainty set for the mixture dynamics used in Equation 13 is constrained with regard to both nominal and worst-case environment. Following policy evaluation, the policy is improved using samples drawn from the mixture dynamics. In addition, the following theorem guarantees that  $\mathcal{T}_a$  can facilitate the convergence of the value function to a fixed point.
Theorem 2. For any given policy  $\pi$  and its corresponding worst-case dynamics  $P^w$ , the adjustable robust Bellman operator  $\mathcal{T}_a$  is a contraction whose fixed point is  $V_a^\pi$ . The operator equation  $\mathcal{T}_a V_a^\pi = V_a^\pi$  has a unique solution.
Exact AR2L Algorithm. The AR2L algorithm evaluates and improves policies based on the mixture dynamics that exist in the neighborhoods of both the nominal and worst-case scenarios. However, this algorithm remains impractical since the adjustable robust Bellman operator defined in Equation 13 involves computing the expectations w.r.t. all models in the uncertainty set  $\mathcal{P}^m$ . This potentially results in high computational complexity. Inspired by the classic RARL algorithm (Pinto et al., 2017) which trains a robust policy utilizing perturbations injected by a learned optimal adversary, we propose an iterative training approach that involves training a mixture-dynamics model  $\pi_{\mathrm{mix}}$  alongside the packing policy  $\pi_{\mathrm{pack}}$  and the permutation-based attacker  $\pi_{\mathrm{perm}}$  to obtain the exact AR2L algorithm. The mixture-dynamics model is corresponding to the mixture dynamics, and it permutes the observable item sequences  $\mathbf{B}_t$  to  $\mathbf{B}_t''$  based on observed packed items  $\mathbf{C}_t$ . As illustrated in Figure 2(a), such a model receives the same reward as the packing policy ( $r_t^{\mathrm{mix}} = r_t^{\mathrm{pack}}$ ). Based on Equation 3, the mixture-dynamics model strives to maximize the return under a given policy, while also penalizing deviations between dynamics. As  $P^m$  and  $P^w$  have the same form with the same underlying item distribution  $p_b$ , we can only measure the discrepancy between  $P^m$  and  $P^w$  by evaluating the deviation between  $\pi_{\mathrm{mix}}$  and  $\pi_{\mathrm{perm}}$ . In practice, the Kullback-Leibler (KL) divergence is adopted to constrain the deviation between distributions. Thus, the training loss is
$$
\mathcal {L} _ {\text {m i x}} = - \eta \left(\pi_ {\text {p a c k}}, \pi_ {\text {m i x}}\right) + \left(D _ {K L} \left(\pi_ {\text {m i x}} \mid \mid \mathbb {1} _ {\{x = b _ {t + 1, 1} \}}\right) + \alpha D _ {K L} \left(\pi_ {\text {m i x}} \mid \mid \pi_ {\text {p e r m}}\right)\right), \tag {6}
$$
where  $D_{KL}$  is the KL divergence, and  $\mathbb{1}_{\{x = b_{t + 1,1}\}}$  denotes the indicator function. Thus, the mixture-dynamics model is optimized by minimizing both the DRL loss (the first term) and the distance loss (the second term). Based on this, the exact AR2L algorithm iterates through three stages to progressively improve the packing policy. First, the permutation-based attacker is optimized for a given packing policy, as is done in RARL. Next, the mixture-dynamics model  $\pi_{\mathrm{mix}}$  is learned using the loss function defined in Equation 6. Finally, the packing policy is evaluated and improved using problem instances that are permuted by  $\pi_{\mathrm{mix}}$ .
Approximate AR2L Algorithm. Training the mixture-dynamics model in exact AR2L introduces additional computation to the entire framework. We thus propose the approximate AR2L algorithm, which uses samples from both the nominal and worst-case dynamics to estimate adjustable robust values for the policy iteration as shown in Figure 2(b). Inspired by RfMDP (Ho et al., 2022; Panaganti et al., 2022), we use the dual reformulation of the adjustable robust Bellman operator, as given below.
Proposition 1. The policy evaluation of AR2L can be formulated as a constrained optimization problem. The objective is to maximize the value function over the uncertainty set  $\mathcal{P}^m$ , subject to the constraint defined in Equation 4. By representing the TV distance using the  $f$ -divergence (Shapiro, 2017), the adjustable robust Bellman operator  $\mathcal{T}_a$  given in Equation 13 can be equivalently written as
$$
\begin{array}{l} \mathcal {T} _ {a} V _ {a} ^ {\pi} (s) = \mathbb {E} _ {a \sim \pi} [ r (s, a) + \frac {\gamma}{1 + \alpha} \inf  _ {\lambda , \mu_ {1}, \mu_ {2}, \mu} \left(\mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {o}} \left[ V _ {a} ^ {\pi} \left(s ^ {\prime}\right) - \mu_ {1} \left(s ^ {\prime}\right) \right] _ {+} + \right. \tag {7} \\ \alpha \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {w}} \left[ V _ {a} ^ {\pi} (s ^ {\prime}) - \mu_ {2} (s ^ {\prime}) \right] _ {+} + \mu + \lambda \rho (1 + \alpha)) ], \\ \end{array}
$$
where  $[x]_{+}$  denotes  $\max \{x,0\}$ ,  $\mu_1(s)$  and  $\mu_2(s)$  are Lagrangian multipliers for each  $s\in S$ ;  $\mu = \mu_1(s) + \alpha \mu_2(s)$  holds for each  $s\in S$ , and  $\lambda = \max_s\{V_a^\pi (s) - \mu_1(s),V_a^\pi (s) - \mu_2(s),0\}$ .
The approximate AR2L algorithm eliminates the requirement of using samples from the mixture dynamics for policy evaluation, yet it introduces estimation errors into the process. In practice, this results in mild performance degradation and sometimes unstable learning. Nevertheless, it remains valuable as the approximate AR2L still demonstrates superior performance compared to RfMDP.
In order to implement the AR2L algorithm, we opt to use the PPO algorithm (Schulman et al., 2017) to train packing policy  $\pi_{\mathrm{pack}}$ , permutation-based attacker  $\pi_{\mathrm{perm}}$  and mixture-dynamics model  $\pi_{\mathrm{mix}}$ . The packing policy is implemented using the PCT method (Zhao et al., 2022a) to accommodate the continuous solution space. All of the relevant implementation details, pseudocode for the algorithms, and both derivations and proofs are included in the Appendix.
Training and Evaluation Settings In the online 3D-BPP setting, the container sizes  $S^d$ ,  $d \in \{x,y,z\}$  are equal for each dimension ( $S^x = S^y = S^z$ ), and the item sizes  $s^d$ ,  $d \in \{x,y,z\}$  are limited to no greater than  $S^d /2$  to create more complex scenarios. The stability of each item is checked based on constraints used in (Zhao et al., 2022a,b). Moreover, we adopt two different popular settings used in (Zhao et al., 2022a,b) for training DRL-based policies. In the discrete setting, where item sizes are uniformly generated from a discrete set, i.e.,  $s^d \in \{1,2,3,4,5\}$ , resulting in a total of 125 types of items, and the container size  $S^d$  is fixed at 10. In the continuous setting, the container size  $S^d$  is set to 1, and item sizes  $s^d$ ,  $d \in \{x,y\}$  follow a continuous uniform distribution  $U(0.1,0.5)$ ; and  $s^z$  is uniformly sampled from a finite set  $\{0.1,0.2,0.3,0.4,0.5\}$ . Then, discrete and continuous datasets are created following instructions from the two aforementioned training settings to evaluate both heuristic and DRL-based methods. Each dataset consists of 3,000 problem instances, where each problem instance contains 150 items. We use three metrics to evaluate the performance of various packing strategies: Uti. represents the average space utilization rate of the bin's volume; Std. is the standard deviation of space utilization which evaluates the algorithm's reliability across all instances; Num. evaluates the average number of packed items.
5.1 Algorithm Robustness under Permutation-based Attacks
In our experiment, we evaluate the robustness of six heuristic methods, including deep-bottom-left (DBL) method (Karabulut and Inceoglu, 2005), best-match-first (BMF) method (Li and Zhang, 2015), least-surface-area heuristics (LSAH) (Hu et al., 2017), online bin packing heuristics (OnlineBPH) (Ha et al., 2017), heightmap-minimization (HMM) method (Wang and Hauser, 2019) and maximize-accessible-convex-space (MACS) method (Hu et al., 2020). For the DRL-based methods, we evaluate the constrained deep reinforcement learning (CDRL) (Zhao et al., 2021) algorithm, and the packing configuration tree (PCT) method (Zhao et al., 2022a). We train specific permutation-based attackers for each of them. Since the capacity of the permutation-based attacker is highly related to the number of observable items of the attacker, we choose  $N_B = 5$ , 10, 15, 20 to obtain different attackers. Given that the number of observable items for the packing policy is typically limited to the first item in most methods, except for PCT, we can establish a consistent approach by setting the number of observable items for the packing policy to 1 in this case.
Table 1: The performance of existing methods under the perturbation in the discrete setting.
Methods	NB=5	NB=10	NB=15	NB=20	w/o attack
Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.	Uti.(%)	Std.	Num.
DBL	40.4	14.3	18.4	28.5	16.1	13.9	26.0	14.9	14.9	21.3	12.6	8.5	63.6	11.9	25.8
BMF	46.9	11.5	21.0	40.9	12.2	22.2	38.6	12.9	21.1	33.7	11.9	22.1	62.0	9.2	24.8
LSAH	46.0	10.1	20.3	42.2	11.3	20.9	38.6	11.0	19.4	35.6	11.8	21.3	60.9	10.9	24.6
OnlineBPH	47.1	21.0	19.8	44.0	18.9	22.8	29.8	18.5	14.6	22.4	12.7	14.1	64.1	8.9	25.8
HMM	49.1	11.1	22.5	46.5	13.8	22.5	43.4	13.0	21.4	40.4	10.0	24.1	56.1	10.4	22.6
MACS	43.0	9.7	21.9	40.8	9.0	24.8	39.0	9.8	23.7	38.3	9.0	27.0	53.0	10.8	21.5
CDRL	61.5	7.8	27.9	56.1	6.9	28.7	54.6	7.6	31.3	51.0	7.1	29.9	74.1	7.3	29.1
PCT	63.6	9.9	27.3	58.7	11.3	25.8	50.9	13.1	25.6	40.5	15.3	21.8	76.6	6.0	30.0
Table 1 presents the performance of various methods under perturbations from corresponding permutation-based attackers in the discrete setting. We can observe that though the HMM algorithm does not perform the best in the nominal scenario compared to other heuristic methods, it exhibits superior performance under attackers with different attack capacities. By comparison, the DBL method is the most vulnerable heuristic method. As attack capabilities increase, the CDRL algorithm demonstrates greater robustness compared to the PCT method, consistently achieving a higher average number of packed items with smaller variance of volume ratio. It is worth noting that the larger the value of  $N_B$  is, the more the performance can be degraded for all methods, indicating that harder problem instances can be explored by increasing the value of  $N_B$ . In addition, the number of packed items does not necessarily decrease as attack capacity increases. This is because permutation-based attackers aim to minimize space utilization, which means they may select smaller items to make the packing problem more challenging for the packing policy.
5.2 Performance of AR2L Algorithm
We then benchmark four other RL algorithms compatible with online 3D-BPP tasks, which include packing configuration tree (PCT) (Zhao et al., 2022a) method, the CVaR-Proximal-Policy-Optimization (CPPO) (Ying et al., 2022) algorithm, the robust adversarial reinforcement learning (RARL) algorithm (Pinto et al., 2017), and the robust  $f$ -divergence MDP (RfMDP) (Ho et al., 2022; Panaganti et al., 2022). The PCT algorithm serves as the baseline method and is used as the packing policy for all other methods in our implementations. The CPPO algorithm trains the packing policy with worst-case returns as a constraint, which can potentially improve both average and worst-case performance. The RARL algorithm is the baseline model for the exact AR2L algorithm due to its reliance on worst-case samples from the attacker. Likewise, the RfMDP framework serves as the baseline model for the approximate AR2L algorithm because it involves approximating values of the unknown dynamics. As both the exact AR2L (ExactAR2L) algorithm and the approximate AR2L algorithm (ApproxAR2L) can develop policies with different levels of robustness by adjusting the value of  $\alpha$ , we choose  $\alpha = 0.3, 0.5, 0.7, 1.0$  to explore the relationship between  $\alpha$  and policy robustness. Furthermore, we use different attackers with varying attack capabilities ( $N_B = 5, 10, 15, 20$ ) to investigate the robustness of packing policies. It is important to note that in this scenario, the packing policy is permitted to observe the same number of items as its corresponding attacker. During testing, we construct mixture datasets by randomly selecting  $\beta\%$  nominal box sequences and reordering them using the learned permutation-based attacker for each packing policy. The empirical results in the continuous setting are included in the Appendix. Since we implement PCT with different training configurations, we conducted a comparison between our implementation and the official implementation in the Appendix.
$Uti$ . Table 2 presents the results of evaluating multiple robust methods on various datasets that differ in the number of problem instances perturbed by different attackers. We denote the different values of  $\alpha$  used by the ExactAR2L algorithm as ExactAR2L(α), and similarly for ApproxAR2L(α). The results unequivocally indicate that the ExactAR2L algorithm, when employed with a higher value of  $\alpha$ , exhibits superior performance on nominal cases and competitive robustness in terms of space utilization when compared to other robust methods. When the value of  $\alpha$  is increased, it allows the packing policy to encounter and learn from more worst-case samples during training. By experiencing a wider range of adversarial situations, the packing policy can better adapt to mitigate the impact of attackers, enhancing its overall robustness. However, ExactAR2L cannot consistently improve its average performance across all the settings with different values of  $N_B$ . For the scenario
where  $N_B = 5$ , the distribution deviation between the perturbed data and the nominal data ( $\beta = 0$ ) is comparatively minimal. This indicates that increasing the value of  $\alpha$  and incorporating more challenging instances into the training settings of the nominal dynamics is an acceptable approach. By doing so, the generalization of the model can be enhanced, allowing it to handle a wider range of scenarios. However, the distribution deviation between the perturbed data and the nominal data ( $\beta = 0$ ) increases as the number of observable items increases. In such cases, it becomes less desirable to significantly increase the value of  $\alpha$ . As a result, in the cases of  $N_B = 10$  and  $N_B = 15$  under  $\beta = 0$ , ExactAR2L tends to favor  $\alpha = 0.5$ . Furthermore, in the scenario of  $N_B = 20$ , ExactAR2L
(a)  $N_B = 15$
(b)  $N_B = 20$
Figure 3: 3(a) 3(b) depict the learning curves of robust RL algorithms for nominal problem instances. 3(c) 3(d) show the influence of  $\rho$  on the performance of the ApproxAR2L algorithm.
(c)  $N_B = 5,\alpha = 1.0$
(d)  $N_B = 10, \alpha = 0.5$
cannot outperform PCT due to the larger distribution deviation caused by the increased number of observable items of the attacker. In a word, by including perturbed data with a small deviation from the nominal data during training, the enhanced generalization leads to the performance improvement in nominal cases  $(\beta = 0)$ . Conversely, the large distribution deviation between the perturbed data and the nominal data can degenerate the performance in nominal cases, as indicated by the RARL algorithm. We also observe that as the value of  $\beta$  increases, the ExactAR2L algorithm tends to favor a larger value of  $\alpha$ . Furthermore, the ApproxAR2L algorithm exhibits a similar performance tendency in both nominal and worst-case scenarios as the ExactAR2L algorithm. Due to the introduced value estimation error, the ApproxAR2L algorithm cannot perform as well as the ExactAR2L algorithm. Despite this, the ApproxAR2L algorithm still performs better than its baseline model, the RfMDP algorithm. Additionally, as demonstrated in Figures 3(a) 3(b), the ExactAR2L algorithm can learn faster than other robust RL algorithms.
Std. As shown in Table 2, ExactAR2L demonstrates its superiority over PCT with smaller Std. in 17 tasks, while producing a slightly larger Std. in 3 tasks. Thus, ExactAR2L can indeed improve the robustness. Furthermore, we observe when  $N_B = 5, 10$ , ExactAR2L tends to choose  $\alpha = 0.7$  for smaller Std.. On the other hand, for  $N_B = 15, 20$ , ExactAR2L favors  $\alpha = 1.0$ . Since ExactAR2L is trained on both nominal and worst-case dynamics, while RARL is trained only on the worst-case dynamics, the ExactAR2L policy is less conservative than the RARL policy. While the conservativeness may result in smaller Std. in most tasks, it produces worse results in terms of Uti. under the nominal dynamics. It is worth noting that the value of Std. from ExactAR2L is the closest to that of RARL. This observation shows ExactAR2L can trade off between conservative and risky behavior, as the Std. from ExactAR2L is between that of RARL and PCT. Similarly, ApproxAR2L is less conservative than RfMDP, which causes ApproxAR2L cannot achieve a smaller Std. in all tasks.
Num. The ExactAR2L(1.0) algorithm can pack more items in 17 tasks compared to PCT, and shows a slight drop in 3 tasks, where the average drop is 0.2. We found that to pack more items, ExactAR2L consistently favors  $\alpha = 1.0$  across various tasks. Compared to RARL, ExactAR2L(1.0) can pack at least the same number of items in 16 tasks. Thus ExactAR2L(1.0) can produce competitive results compared to RARL and PCT in terms of Num.. Compared to the baseline method RfMDP, ApproxAR2L(0.5) can pack more items in 16 tasks, and shows a slight drop in only 4 tasks, where the average drop is 0.25.
Hyperparameter Configurations. Based on observations from Table 2,  $\alpha = 1.0$  is the best choice for ExactAR2L across different test settings. When  $\beta = 50, 75$ , ExactAR2L(1.0) performs the best compared to baselines and ExactAR2L with other values of  $\alpha$  (with a slight drop compared to ExactAR2L(0.7) when  $\beta = 50, N_B = 15$ ). If  $\beta = 100$ , ExactAR2L(1.0) can still produce competitive results compared to RARL and significantly outperforms PCT. When  $\beta = 25$ , although  $\alpha = 1.0$  is not the optimal choice, ExactAR2L(1.0) can still outperform other baselines. When  $\beta = 0$ , ExactAR2L(1.0) significantly outperforms RARL, and the slight drop compared to PCT is acceptable, as our goal is to improve the robustness while maintaining average performance at an acceptable level.  $\rho$  is only used in ApproxAR2L algorithm. As shown in Figures 3(c) 3(d), we choose different values of  $\rho$  in different settings. We found that  $\rho = 0.1$  is a trustworthy choice for ApproxAR2L. Based on the observations from Table 2 and Figures 3(c) 3(d), we conclude that  $\rho = 0.1$  and  $\alpha = 0.5$  are the best choice for ApproxAR2L, as it outperforms its corresponding baseline in almost all the tasks.
6 Conclusions and Limitations
In this work, we propose a novel reinforcement learning approach, Adjustable Robust Reinforcement Learning (AR2L), for solving the online 3D Bin Packing Problem (BPP). AR2L achieves a balance between policy performance in nominal and worst-case scenarios by optimizing a weighted sum of returns. We use a surrogate task of identifying a policy with the largest lower bound of the return to optimize the objective function. We can seamlessly bridge AR2L into the RARL and RfMDP algorithms to obtain exact and approximate AR2L algorithms. Extensive experiments show that the exact AR2L algorithm improves the robustness of the packing policy while maintaining acceptable performance, but may introduce additional computational complexity due to the mixture-dynamics model. The approximate AR2L algorithm estimates values without samples from the mixture dynamics, yet performance is not up to our exact AR2L agent due to the existence of estimation error. Though our AR2L framework is designed for online 3D-BPP, it can be also adapted to other decision-making tasks. We will investigate more efficient and generalizable AR2L framework.
We appreciate the anonymous reviewers, (S)ACs, and PCs of NeurIPS2023 for their insightful comments to further improve our paper and their service to the community. We would like to thank the Turing AI Computing Cloud (TACC) (Xu et al., 2021) and HKUST iSING Lab for providing us computation resources on their platform.
Zhao, H., She, Q., Zhu, C., Yang, Y., and Xu, K. (2021). Online 3d bin packing with constrained deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 741-749.
Zhao, H., Yu, Y., and Xu, K. (2022a). Learning efficient online 3d bin packing on packing configuration trees. In International Conference on Learning Representations.
Zhao, H., Zhu, C., Xu, X., Huang, H., and Xu, K. (2022b). Learning practically feasible policies for online 3d bin packing. Science China Information Sciences, 65(1):112105.
A.1 Experiment
In addition to results in the discrete setting presented earlier in the main texts, this section focuses on introducing additional results in the continuous setting. For the continuous setting, the container size  $S^d$ ,  $d \in \{x,y,z\}$  is fixed at 1, and the sizes of items  $s^d$ ,  $d \in \{x,y\}$  follow a continuous uniform distribution  $U(0.1,0.5)$ . To prevent impractical scenarios where packed items cannot form a supportive plane for incoming items, the  $s^z$  dimension of each item is uniformly sampled from a finite set of values  $\{0.1,0.2,0.3,0.4,0.5\}$  (Zhao et al., 2022a). Likewise, Each dataset consists of 3,000 problem instances, where each problem instance contains 150 items. We use three metrics to evaluate the performance of various packing strategies:  $Uti$  represents the average space utilization rate of the bin's volume;  $Std$  is the standard deviation of space utilization across all instances;  $Num$  evaluates the average number of packed items. In the field of online 3D-BPP (Zhao et al., 2021, 2022a; Song et al., 2023), the standard deviation of space utilization across a large number of problem instances ( $>1,000$ ) is commonly used to assess the reliability and stability of a solver, rather than the error bar generated from multiple experiment runs (Kim et al., 2022; Choo et al., 2022; Qiu et al., 2022). In addition, all the models are developed using PyTorch (Paszke et al., 2017) and trained on a Nvidia RTX 3090 GPU and an Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz.
A.1.1 Algorithm Robustness under Permutation-based Attacker