SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxin Pan<sup>1</sup> Yize Chen<sup>2*</sup> Fangzhen Lin<sup>3*</sup>
<sup>1</sup>EMIA, The Hong Kong University of Science and Technology  
<sup>2</sup>AI Thrust, The Hong Kong University of Science and Technology (Guangzhou)  
<sup>3</sup>CSE, The Hong Kong University of Science and Technology  
yuxin.pan@connect.ust.hk yizechen@ust.hk flin@cs.ust.hk
Designing effective policies for the online 3D bin packing problem (3D-BPP) has been a long-standing challenge, primarily due to the unpredictable nature of incoming box sequences and stringent physical constraints. While current deep reinforcement learning (DRL) methods for online 3D-BPP have shown promising results in optimizing average performance over an underlying box sequence distribution, they often fail in real-world settings where some worst-case scenarios can materialize. Standard robust DRL algorithms tend to overly prioritize optimizing the worst-case performance at the expense of performance under normal problem instance distribution. To address these issues, we first introduce a permutation-based attacker to investigate the practical robustness of both DRL-based and heuristic methods proposed for solving online 3D-BPP. Then, we propose an adjustable robust reinforcement learning (AR2L) framework that allows efficient adjustment of robustness weights to achieve the desired balance of the policy's performance in average and worst-case environments. Specifically, we formulate the objective function as a weighted sum of expected and worst-case returns, and derive the lower performance bound by relating to the return under a mixture dynamics. To realize this lower bound, we adopt an iterative procedure that searches for the associated mixture dynamics and improves the corresponding policy. We integrate this procedure into two popular robust adversarial algorithms to develop the exact and approximate AR2L algorithms. Experiments demonstrate that AR2L is versatile in the sense that it improves policy robustness while maintaining an acceptable level of performance for the nominal case.
The offline 3D bin packing problem (3D-BPP) is a classic NP-hard combinatorial optimization problem (COP) (de Castro Silva et al., 2003), which aims to optimally assign cuboid-shaped items with varying sizes to the minimum number of containers while satisfying physical constraints (Martello et al., 2000). In such a setting, a range of physical constraints can be imposed to meet diverse packing requirements and preferences (Gzara et al., 2020). The primary constraint requires items to be packed stably, without any overlaps, and kept within the bin. The online counterpart of 3D-BPP is more prevalent and practical in logistics and warehousing (Wang and Hauser, 2020), as it does not require complete information about all the unpacked items in advance. In this setting, only a limited number of upcoming items on a conveyor can be observed, and items must be packed after the preceding items are allocated (Seiden, 2002). Thus, besides physical constraints already present
*Co-corresponding Authors
in the offline counterpart, the item permutation on the conveyor introduces additional constraints, while the solution approach must take packing order into consideration.
Solution techniques for online 3D-BPP can be broadly categorized into heuristic and learning-based methods. As heuristics often heavily rely on manually designed task-specific score functions (Chazelle, 1983; Ha et al., 2017), they have limitations in expressing complex packing preferences and adapting to diverse scenarios. By comparison, learning-based methods usually involve utilizing deep reinforcement learning (DRL) techniques to develop a more adaptable packing policy capable of accommodating diverse packing requirements (Zhao et al., 2021, 2022a; Song et al., 2023). While emerging DRL-based methods for online 3D-BPP are effective at optimizing average performance, they do not account for worst-case scenarios. This is because these methods often fail on "hard" box sequences arising from the inherent uncertainty in incoming box sequences. In addition, few studies have investigated the robustness of these methods, as incoming item sequences can vary a lot in practice. These limit the practical usage of learning-based approaches.
To study the algorithm robustness under worst-case scenarios, one plausible approach is to design an attacker that can produce perturbations commonly encountered in real-world settings. Various proposed attackers for perturbing RL models often add continuous-value noise to either received observations or performed actions (Tessler et al., 2019; Sun et al., 2021). Yet such methods are mostly not well-suited for online 3D-BPP, as the state is defined by the bin configuration and the currently observed items, and adding continuous-value noise may not correspond to real-world perturbations. As previously discussed, item permutations can significantly impact the performance of a given packing strategy, and in real-world scenarios, the packing strategy may face challenging box sequences. By contrast, perturbing the bin configuration may violate practical constraints or packing preferences. Therefore, we argue that using diverse item permutations to evaluate the model's robustness is a more suitable approach. Some of general robust reinforcement learning frameworks against perturbations can be applied to online 3D-BPP (Pinto et al., 2017; Ying et al., 2022; Ho et al., 2022; Panaganti et al., 2022), yet these methods may develop overly conservative policies, as they usually prioritize improving worst-case performance at the expense of average performance. This motivates us to design a robust algorithm that can effectively handle worst-case problem instances while attaining satisfactory average performance.
In this paper, we develop a novel permutation-based attacker to practically evaluate robustness of both DRL-based and heuristic bin packing policies. We propose an Adjustable Robust Reinforcement Learning (AR2L) framework, which preserves high performance in both nominal and worst-case environments. Our learnable attacker selects an item and place it to the most preceding position in the observed incoming item sequence, as only the first item in this sequence has the priority to be packed first. Essentially, this attacker attempts to identify a problem instance distribution that is challenging for a given packing policy by simply reordering item sequences. The developed AR2L incorporates such novel attacker along with an adjustable robustness weight parameter. Specifically, to consider both the average and worst-case performance, we formulate the packing objective as a weighted sum of expected and worst-case returns defined over space utilization rate. We derive a lower bound for the objective function by relating it to the return under a mixture dynamics, which guarantees AR2L's performance. To realize this lower bound, we turn to identifying a policy with the highest lower bound as the surrogate task. In this task, we use an iterative procedure that searches for the associated mixture dynamics for the policy evaluation, and improves the policy under the resulting mixture dynamics. To further put AR2L into practice, we find the connecting link between AR2L with the Robust Adversarial Reinforcement Learning (ARL) algorithm (Pinto et al., 2017) and the Robust  $f$ -Divergence Markov Decision Process (RfMDP) algorithm (Ho et al., 2022; Panaganti et al., 2022), resulting in the exact AR2L and the approximate AR2L algorithms respectively. Empirical evidence shows our method is able to achieve competitive worst-case performance in terms of space utilization under the worst-case attack, while still maintaining good average performance compared to all previously proposed robust counterparts.
Numerous heuristic Ha et al. (2017); Karabulut and Inceoglu (2005); Wang and Hauser (2019); Li and Zhang (2015); Hu et al. (2017, 2020) and learning-based methods (Hu et al., 2017; Verma et al., 2020; Zhao et al., 2021, 2022b; Yang et al., 2021; Zhao et al., 2022a; Song et al., 2023) have been proposed to solve online 3D-BPP. However, these methods typically only consider average performance and
may not be robust against perturbations. For more detailed reviews of related algorithms on 3D-BPP, please refer to the Appendix.
Adversarial Attacks in Deep RL. More recently, several studies have demonstrated that deep RL algorithms are vulnerable to adversarial perturbations from attackers, and robust policies can be trained accordingly (Huang et al., 2017; Zhang et al., 2020, 2021; Sun et al., 2021; Ding et al., 2023). Yet standard DRL attack schemes cannot generate realistic attacks for 3D-BPP due to the setting of added perturbations. In the field of COP, Lu et al. (2023) used a RL-based attacker to modify the underlying graph. Yet this method is limited to offline COPs that can be formulated as graph. Kong et al. (2019) employed a generative adversarial network (GAN) (Goodfellow et al., 2020) to generate some worst-case problem instances from Gaussian noise under restricted problem setting.
Robust RL. In order to find policies that are robust against various types of perturbations, a great number of studies have investigated different strategies, such as regularized-based methods (Zhang et al., 2020; Shen et al., 2020; Oikarinen et al., 2021; Kumar et al., 2021; Kuang et al., 2022), attack-driven methods (Kos and Song, 2017; Pattanaik et al., 2017; Behzadan and Munir, 2017), novel bellman operators (Liang et al., 2022; Wang and Zou, 2021, 2022), and conditional value at risk (CVaR) based methods (Chow et al., 2017; Tang et al., 2019; Hiraoka et al., 2019; Ying et al., 2022). However, these methods are generally intended to deal with  $l_{p}$ -norm perturbations on vectorized observations, which could limit their applicability to online 3D-BPP. Built upon robust MDP framework (Iyengar, 2005), Pinto et al. (2017) modeled the competition between the agent and the adversary as a zero-sum two-player game, but such game formulation has an excessive prioritization of the worst-case performance. Jiang et al. (2021) introduced Monotonic Robust Policy Optimization (MRPO) to enhance domain generalization by connecting worst-case and average performance. Yet such approach imposes Lipschitz continuity assumptions, which are unaligned with 3D-BPP. The recently developed robust  $f$ -divergence MDP can approximate the worst-case value of a policy by utilizing nominal samples instead of relying on samples from an adversary (Ho et al., 2022; Panaganti et al., 2022). However, this method still cannot account for expected cases, as its focus is solely on learning values in worst-case scenarios.
MDP Formulation of Online 3D-BPP. To learn a highly effective policy via RL, the online 3D-BPP is formulated as an MDP. Inspired by PCT (Zhao et al., 2022a), in this formulation, the state  $s_t^{\mathrm{pack}} \in S^{\mathrm{pack}}$  observed by the packing agent consists of the already packed  $N_C$  items  $\mathbf{C}_t$  in the bin, the observed incoming  $N_B$  items  $\mathbf{B}_t$ , and a set of potential positions  $\mathbf{L}_t$ . The packed items in  $\mathbf{C}_t$  have spatial properties like sizes and coordinates, while each item  $b_{t,i} \in \mathbf{B}_t, i \in \{1,..,N_B\}$  only provides size information. The potential positions are typically generated for the most preceding item in  $\mathbf{B}_t$  using heuristic methods (Martello et al., 2000; Crainic et al., 2008; Ha et al., 2017). Then, the action  $a_t^{\mathrm{pack}} \in \mathcal{A}^{\mathrm{pack}}$  is to choose one position  $l_{t,j} \in \mathbf{L}_t, j \in \{1,..,N_L\}$  for the first item in  $\mathbf{B}_t$ . In the termination state  $s_T$  ( $T$  is the episode length), the agent cannot pack any more items. As a result, the agent receives a delayed reward  $r_T$  that represents the space utilization at  $s_T$ , instead of immediate rewards ( $r_t = 0, t < T$ ). The discount factor  $\gamma$  here is set to 1. The aim is to maximize the space utilization by learning a stochastic packing policy  $\pi_{\mathrm{pack}}(l_{t,j}|\mathbf{C}_t,\mathbf{B}_t,\mathbf{L}_t)$ .
Robust MDP. The goal of robust MDP is to find the optimal robust policy that maximizes the value against the worst-case dynamics from an uncertainty set  $\mathcal{P}^w$ . The uncertainty set is defined in the neighborhood of the nominal dynamics  $P^o = (P_{s,a}^o,(s,a)\in S\times \mathcal{A})$  and satisfies rectangularity condition (Iyengar, 2005), defined as  $\mathcal{P}^w = \otimes_{(s,a)\in S\times \mathcal{A}}\mathcal{P}_{s,a}^w$ ,  $\mathcal{P}_{s,a}^w = \{P_{s,a}\in \Delta (\mathcal{S}):D_{TV}(P_{s,a}||P_{s,a}^o)\leq \rho \}$ , where  $D_{TV}(\cdot)$  denotes the total variation (TV) distance,  $\otimes$  is the Cartesian product,  $\Delta (\mathcal{S})$  is a set of probability measures defined on  $\mathcal{S}$ , and  $\rho \in [0,1]$  is the radius of the uncertainty set. Consequently, the robust value function under a policy  $\pi$  is  $V_r^\pi = \inf_{P^w\in \mathcal{P}^w}V_r^{\pi ,P^w}$ . And the robust Bellman operator  $\mathcal{T}_r$  is defined as  $\mathcal{T}_rV_r^\pi (s) = \mathbb{E}_{a\sim \pi}[r(s,a) + \gamma \inf_{P_{s,a}^w\in \mathcal{P}_{s,a}^w}\mathbb{E}_{s'\sim P_{s,a}^w}[V_r^\pi (s')]]$ , where  $s'$  denotes the next state. To empirically solve the robust MDP, Pinto et al. (2017) proposed RARL that learns the robust policy under the environment perturbed by a learned optimal adversary.
Robust  $f$ -divergence MDP. To avoid the need for a costly trained adversary, RfMDP (Ho et al., 2022; Panaganti et al., 2022) formulates the problem as a tractable constrained optimization task to approximate the robust value using samples from  $P^o$ . The objective is to find a transition distribution
that minimizes the robust value function, subject to the constraint in  $\mathcal{P}^w$  described by the  $f$ -divergence. As a result, a new robust Bellman operator is introduced by solving the dual form of this constrained optimization problem through the Lagrangian multiplier method.
4 Adjustable Robust Reinforcement Learning
In this section, we begin by introducing a novel yet practical adversary capable of generating worst-case problem instances for the online 3D-BPP. Next, we present the adjustable robust reinforcement learning (AR2L) framework to address the robustness-performance tradeoff. Finally, we integrate AR2L into both the RARL algorithm and the RfMDP algorithm to derive the exact and approximate AR2L algorithms in a tractable manner.
4.1 Permutation-based Attacker
In online 3D-BPP, a problem instance is comprised of a sequence of items, and these items are randomly permuted and placed on a conveyor. Such randomness can result in certain instances where trained policy may fail. Therefore, this observation motivates us to design a simple yet realistic adversary called permutation-based attacker. By reordering the item sequence for a given packing policy, our approach can explore worst-case instances without compromising realism (See Figure 1 for attacker overview). In contrast to the approach
Figure 1: Overview of our attack framework.
of directly permuting the entire item sequence as described in (Kong et al., 2019), our method involves permuting the observable item sequence, thereby progressively transforming the entire item sequence. Through the behavior analysis of our permutation-based attacker in the Appendix, we observe the attacker appears to prefer smaller items when constructing harder instances as the number of observable items increases. This aligns with the findings mentioned in (Zhao et al., 2022a), where larger items simplify the scenario while smaller items introduce additional challenges. As a result, the act of permuting the entire sequence might enable the attacker to trickily select certain types of items, and thus carries potential risks associated with shifts in the underlying item distribution. However, we aim to ensure that the attacker genuinely learns how to permute the item sequence to identify and create challenging scenarios. To mitigate these risks, we limit the attacker's capacity by restricting the number  $N_B$  of observable items to proactively address concerns related to changes in the item distribution.
To target any type of solver intended for solving the online 3D-BPP for robustness evaluation, we train a RL-based policy which acts as the permutation-based attacker. The permuted state  $s_t^{\mathrm{perm}} \in S^{\mathrm{perm}}$  of the attacker is comprised of the current bin configuration  $\mathbf{C}_t$  and the item sequence  $\mathbf{B}_t$ . The action  $a_t^{\mathrm{perm}} \in \mathcal{A}^{\mathrm{perm}}$  involves moving one item from  $\mathbf{B}_t$  to a position ahead of the other observed items, resulting in the reordered item sequence  $\mathbf{B}_t' = \{b_{t,i}'\}$ . Then, the packing policy will pack the most preceding one into the bin based on the perturbed observation. The reward for the attacker is defined as  $r_t^{\mathrm{perm}} = -r_t^{\mathrm{pack}}$ , since the objective is to minimize the space utilization by training an optimal stochastic policy  $\pi_{\mathrm{perm}}(b_{t,i}|\mathbf{C}_t,\mathbf{B}_t)$ . The online packing process allows only the first item in the observable item sequence to be packed at each step. To address the issue of exponential growth in the action space as the number of observable items increases, the permutation-based attacker strategically selects one item and positions it ahead of the other items. We can consider a simplified value function represented as  $V^{\pi_{\mathrm{pack}}}(\mathbf{C}_t,\mathbf{B}_t') = r_t^{\mathrm{pack}} + V^{\pi_{\mathrm{pack}}}(\mathbf{C}_{t + 1},\mathbf{B}_{t + 1}')$ . This function indicates that the value of  $\pi_{\mathrm{pack}}$  depends on  $r_t^{\mathrm{pack}}$ ,  $\mathbf{C}_{t + 1}$ , and  $\mathbf{B}_{t + 1}'$ . Furthermore,  $r_t^{\mathrm{pack}}$  and  $\mathbf{C}_{t + 1}$  are influenced by the first item in  $\mathbf{B}_t'$ , while the remaining items in  $\mathbf{B}_t'$  combine with a new item to form a new sequence, which undergoes further permutations as  $\mathbf{B}_{t + 1}'$ . As a result, the permutation of the remaining items at time step  $t$  is disregarded in the attack process. To model such
an attacker, we use the Transformer (Vaswani et al., 2017) that is capable of capturing long-range dependencies in spatial data and sequential data involved in the attacker's observation. Please refer to the Appendix for more details on the implementations.
4.2 Adjustable Robust Reinforcement Learning
For online 3D-BPP, item observed at each timestep is independently generated from a stationary distribution  $p_b(b_{t,i})$  that does not change over time. Once an item is placed into the container, the bin configuration  $\mathbf{C}_t$  becomes deterministic, and a new item  $b_{t+1,N_B}$  is appended to the observable item sequence to construct  $\mathbf{B}_{t+1}$ . Thus, the nominal environment is defined as  $P^o(\mathbf{C}_{t+1},\mathbf{B}_{t+1}|\mathbf{C}_t,\mathbf{B}_t,a_t^{\mathrm{pack}}) = p_b(b_{t+1,N_B})$ . We omit  $\mathbf{L}_t$  for brevity. Since we use the permutation-based attacker to reorder the observable item sequence, the worst-case environment transition is  $P^w(\mathbf{C}_{t+1},\mathbf{B}_{t+1}'|\mathbf{C}_t,\mathbf{B}_t',a_t^{\mathrm{pack}}) = p_b(b_{t+1,N_B})\pi_{\mathrm{perm}}(b_{t+1,1}'|\mathbf{C}_{t+1},\mathbf{B}_{t+1})$ , with  $P^w$  as the worst-case dynamics.
Existing DRL-based methods are developed to learn a packing policy that maximizes space utilization (cumulative reward) under the nominal distribution  $P^o$ . However, in real-world scenarios where the order of items can be adversarially permuted, it is not sufficient to only consider the expected return under  $P^o$ . In contrast, the existing general robust methods that can be deployed to online 3D-BPP overly prioritize robustness at the cost of average performance by solely focusing on the return under the worst-case dynamics  $P^w$ . To address these issues, we should be aware of the returns under both the average and worst-case scenarios. Given that nominal cases are more common than worst-case scenarios in the online 3D-BPP setting, the objective function is defined as
$$
\pi^ {*} = \arg \max  _ {\pi \in \Pi} \eta (\pi , P ^ {o}) + \alpha \eta (\pi , P ^ {w}) \tag {1}
$$
where  $\alpha \in (0,1]$  is the weight of robustness,  $\eta (\cdot)$  is the return, and  $P^w$  is built on the optimal permutation-based attacker. Here symbols without superscripts represent those used by the packing agent (e.g.,  $\pi_{\mathrm{pack}} = \pi$ ).
However, how can we learn such a policy with the presence of both  $P^o$  and  $P^{w?}$ ? To address this matter, we derive a lower bound for the objective function defined in Equation 1 by relating it to the return under an unknown mixture dynamics defined as  $P^m$ , as shown in the following theorem.
Theorem 1. The model discrepancy between two models can be described as  $d(P^1 ||P^2) \triangleq \max_{s,a} D_{TV}(P^1(\cdot|s,a)||P^2(\cdot|s,a))$ . The lower bound for objective 1 is derived as follows:
$$
\eta (\pi , P ^ {o}) + \alpha \eta (\pi , P ^ {w}) \geq (1 + \alpha) \eta (\pi , P ^ {m}) - \frac {2 \gamma | r | _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}} \left(d \left(P ^ {m} \right| \left| P ^ {o}\right) + \alpha d \left(P ^ {m} \right| \left| P ^ {w}\right)\right) \tag {2}
$$
The RHS of Inequality 2 provides the lower bound for the objective function defined in Equation 1, where the first term represents the expected return of policy  $\pi$  under the mixture dynamics  $P^m$ , while the second term denotes the weighted sum of deviations of  $P^m$  from both  $P^o$  and  $P^w$ . Motivated by Theorem 1, we turn to maximizing the RHS of Inequality 2 to concurrently improve both the average and worst-case performance. Therefore, the primal problem in Equation 1 is equivalently reformulated to a surrogate problem where we expect to identify an optimal policy with the maximal lower bound, represented as
$$
\pi^ {*} = \underset {\pi \in \Pi} {\arg \max } \underset {P ^ {m} \in \mathcal {P}} {\max } \eta (\pi , P ^ {m}) - (d (P ^ {m} | | P ^ {o}) + \alpha d (P ^ {m} | | P ^ {w})); \tag {3}
$$
The objective of this optimization problem is to maximize the return by updating both the policy and the mixture dynamics controlled by robustness weight  $\alpha$ , given the nominal and worst-case dynamics. Furthermore, the second term in Equation 3 can be viewed as a constraint that penalizes the deviations between environments, resulting in a constrained optimization problem written as  $\pi^{*} = \arg \max_{\pi \in \Pi}\{\max_{P^{m}\in \mathcal{P}}\eta (\pi ,P^{m}):d(P^{m}||P^{o}) + \alpha d(P^{m}||P^{w})\leq \rho^{\prime}\}$ , where  $\rho^{\prime}\in [0,1 + \alpha ]$ . However, this constraint renders the constrained form of Equation 3 impractical to solve. Instead, a heuristic approximation is used for this constraint, which considers the TV distance at each state-action pair, leading to an uncertainty set for the mixture dynamics:
$$
\mathcal {P} ^ {m} = \otimes_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \mathcal {P} _ {s, a} ^ {m}; \mathcal {P} _ {s, a} ^ {m} = \left\{P _ {s, a} \in \Delta (\mathcal {S}): D _ {T V} \left(P _ {s, a} \| P _ {s, a} ^ {o}\right) + \alpha D _ {T V} \left(P _ {s, a} \| P _ {s, a} ^ {w}\right) \leq \rho^ {\prime} \right\}. \tag {4}
$$
The uncertainty set  $\mathcal{P}^m$  satisfies the rectangularity condition Iyengar (2005). To solve the constrained optimization problem, we carry out an iterative procedure which searches for the associated mixture
(a) Exact AR2L Algorithm
(b) Approximate AR2L Algorithm
Figure 2: Implementations of the AR2L algorithm. Left: The exact AR2L algorithm requires to learn a mixture-dynamics model to generate problem instances for the training of the packing policy. Right: The approximate AR2L algorithm relies on the samples from both the nominal dynamics and the permutation-based attacker to estimate adjustable robust values of the packing policy.