Title: Adjustable Robust Reinforcement Learning for Online 3D Bin Packing
Venue: NeurIPS 2023
Accepted: yes
Keywords: online 3D bin packing problem

combinatorial optimization problem

reinforcement learning
Abstract: Designing effective policies for the online 3D bin packing problem (3D-BPP) has been a long-standing challenge, primarily due to the unpredictable nature of incoming box sequences and stringent physical constraints.  While current deep reinforcement learning (DRL) methods for online 3D-BPP have shown promising results in optimizing average performance over an underlying box sequence distribution, they often fail in real-world settings where some worst-case scenarios can materialize. Standard robust DRL algorithms tend to overly prioritize optimizing the worst-case performance at the expense of performance under normal problem instance distribution. To address these issues, we first introduce a permutation-based attacker to investigate the practical robustness of both DRL-based and heuristic methods proposed for solving online 3D-BPP. Then, we propose an adjustable robust reinforcement learning (AR2L) framework that allows efficient adjustment of robustness weights to achieve the desired balance of the policy's performance in average and worst-case environments. Specifically, we formulate the objective function as a weighted sum of expected and worst-case returns, and derive the lower performance bound  by relating to the return under a mixture dynamics. To realize this lower bound, we adopt an iterative procedure that searches for the associated mixture dynamics and improves the corresponding policy. We integrate this procedure into two popular robust adversarial algorithms to develop the exact and approximate AR2L algorithms. Experiments demonstrate that AR2L is versatile in the sense that it improves policy robustness while maintaining an acceptable level of performance for the nominal case.

Adjustable Robust Reinforcement Learning for Online 3D Bin Packing

Abstract

Designing effective policies for the online 3D bin packing problem (3D-BPP) has been a long-standing challenge, primarily due to the unpredictable nature of incoming box sequences and stringent physical constraints. While current deep reinforcement learning (DRL) methods for online 3D-BPP have shown promising results in optimizing average performance over an underlying box sequence distribution, they often fail in real-world settings where some worst-case scenarios can materialize. Standard robust DRL algorithms tend to overly prioritize optimizing the worst-case performance at the expense of performance under normal problem instance distribution. To address these issues, we first introduce a permutation-based attacker to investigate the practical robustness of both DRL-based and heuristic methods proposed for solving online 3D-BPP. Then, we propose an adjustable robust reinforcement learning (AR2L) framework that allows efficient adjustment of robustness weights to achieve the desired balance of the policy's performance in average and worst-case environments. Specifically, we formulate the objective function as a weighted sum of expected and worst-case returns, and derive the lower performance bound by relating to the return under a mixture dynamics. To realize this lower bound, we adopt an iterative procedure that searches for the associated mixture dynamics and improves the corresponding policy. We integrate this procedure into two popular robust adversarial algorithms to develop the exact and approximate AR2L algorithms. Experiments demonstrate that AR2L is versatile in the sense that it improves policy robustness while maintaining an acceptable level of performance for the nominal case.

1 Introduction

The offline 3D bin packing problem (3D-BPP) is a classic NP-hard combinatorial optimization problem (COP) (de Castro Silva et al., 2003), which aims to optimally assign cuboid-shaped items with varying sizes to the minimum number of containers while satisfying physical constraints (Martello et al., 2000). In such a setting, a range of physical constraints can be imposed to meet diverse packing requirements and preferences (Gzara et al., 2020). The primary constraint requires items to be packed stably, without any overlaps, and kept within the bin. The online counterpart of 3D-BPP is more prevalent and practical in logistics and warehousing (Wang and Hauser, 2020), as it does not require complete information about all the unpacked items in advance. In this setting, only a limited number of upcoming items on a conveyor can be observed, and items must be packed after the preceding items are allocated (Seiden, 2002). Thus, besides physical constraints already present

in the offline counterpart, the item permutation on the conveyor introduces additional constraints, while the solution approach must take packing order into consideration.

Solution techniques for online 3D-BPP can be broadly categorized into heuristic and learningbased methods. As heuristics often heavily rely on manually designed task-specific score functions (Chazelle, 1983; Ha et al., 2017), they have limitations in expressing complex packing preferences and adapting to diverse scenarios. By comparison, learning-based methods usually involve utilizing deep reinforcement learning (DRL) techniques to develop a more adaptable packing policy capable of accommodating diverse packing requirements (Zhao et al., 2021, 2022a; Song et al., 2023). While emerging DRL-based methods for online 3D-BPP are effective at optimizing average performance, they do not account for worst-case scenarios. This is because these methods often fail on 'hard' box sequences arising from the inherent uncertainty in incoming box sequences. In addition, few studies have investigated the robustness of these methods, as incoming item sequences can vary a lot in practice. These limit the practical usage of learning-based approaches.

To study the algorithm robustness under worst-case scenarios, one plausible approach is to design an attacker that can produce perturbations commonly encountered in real-world settings. Various proposed attackers for perturbing RL models often add continuous-value noise to either received observations or performed actions (Tessler et al., 2019; Sun et al., 2021). Yet such methods are mostly not well-suited for online 3D-BPP, as the state is defined by the bin configuration and the currently observed items, and adding continuous-value noise may not correspond to real-world perturbations. As previously discussed, item permutations can significantly impact the performance of a given packing strategy, and in real-world scenarios, the packing strategy may face challenging box sequences. By contrast, perturbing the bin configuration may violate practical constraints or packing preferences. Therefore, we argue that using diverse item permutations to evaluate the model's robustness is a more suitable approach. Some of general robust reinforcement learning frameworks against perturbations can be applied to online 3D-BPP (Pinto et al., 2017; Ying et al., 2022; Ho et al., 2022; Panaganti et al., 2022), yet these methods may develop overly conservative policies, as they usually prioritize improving worst-case performance at the expense of average performance. This motivates us to design a robust algorithm that can effectively handle worst-case problem instances while attaining satisfactory average performance.

In this paper, we develop a novel permutation-based attacker to practically evaluate robustness of both DRL-based and heuristic bin packing policies. We propose an Adjustable Robust Reinforcement Learning (AR2L) framework, which preserves high performance in both nominal and worst-case environments. Our learnable attacker selects an item and place it to the most preceding position in the observed incoming item sequence, as only the first item in this sequence has the priority to be packed first. Essentially, this attacker attempts to identify a problem instance distribution that is challenging for a given packing policy by simply reordering item sequences. The developed AR2L incorporates such novel attacker along with an adjustable robustness weight parameter. Specifically, to consider both the average and worst-case performance, we formulate the packing objective as a weighted sum of expected and worst-case returns defined over space utilization rate. We derive a lower bound for the objective function by relating it to the return under a mixture dynamics, which guarantees AR2L's performance. To realize this lower bound, we turn to identifying a policy with the highest lower bound as the surrogate task . In this task, we use an iterative procedure that searches for the associated mixture dynamics for the policy evaluation, and improves the policy under the resulting mixture dynamics. To further put AR2L into practice, we find the connecting link between AR2L with the Robust Adversarial Reinforcement Learning (RARL) algorithm (Pinto et al., 2017) and the Robust f -Divergence Markov Decision Process (RfMDP) algorithm (Ho et al., 2022; Panaganti et al., 2022), resulting in the exact AR2L and the approximate AR2L algorithms respectively. Empirical evidence shows our method is able to achieve competitive worst-case performance in terms of space utilization under the worst-case attack, while still maintaining good average performance compared to all previously proposed robust counterparts.

2 Related Work

Numerous heuristic Ha et al. (2017); Karabulut and ˙ Inceo˘ glu (2005); Wang and Hauser (2019); Li and Zhang (2015); Hu et al. (2017, 2020) and learning-based methods (Hu et al., 2017; Verma et al., 2020; Zhao et al., 2021, 2022b; Yang et al., 2021; Zhao et al., 2022a; Song et al., 2023) have been proposed to solve online 3D-BPP. However, these methods typically only consider average performance and

may not be robust against perturbations. For more detailed reviews of related algorithms on 3D-BPP, please refer to the Appendix.

Adversarial Attacks in Deep RL. More recently, several studies have demonstrated that deep RL algorithms are vulnerable to adversarial perturbations from attackers, and robust policies can be trained accordingly (Huang et al., 2017; Zhang et al., 2020, 2021; Sun et al., 2021; Ding et al., 2023). Yet standard DRL attack schemes cannot generate realistic attacks for 3D-BPP due to the setting of added perturbations. In the field of COP, Lu et al. (2023) used a RL-based attacker to modify the underlying graph. Yet this method is limited to offline COPs that can be formulated as graph. Kong et al. (2019) employed a generative adversarial network (GAN) (Goodfellow et al., 2020) to generate some worst-case problem instances from Gaussian noise under restricted problem setting.

Robust RL. In order to find policies that are robust against various types of perturbations, a great number of studies have investigated different strategies, such as regularized-based methods (Zhang et al., 2020; Shen et al., 2020; Oikarinen et al., 2021; Kumar et al., 2021; Kuang et al., 2022), attack-driven methods (Kos and Song, 2017; Pattanaik et al., 2017; Behzadan and Munir, 2017), novel bellman operators (Liang et al., 2022; Wang and Zou, 2021, 2022), and conditional value at risk (CVaR) based methods (Chow et al., 2017; Tang et al., 2019; Hiraoka et al., 2019; Ying et al., 2022). However, these methods are generally intended to deal with l p -norm perturbations on vectorized observations, which could limit their applicability to online 3D-BPP. Built upon robust MDP framework (Iyengar, 2005), Pinto et al. (2017) modeled the competition between the agent and the adversary as a zero-sum two-player game, but such game formulation has an excessive prioritization of the worst-case performance. Jiang et al. (2021) introduced Monotonic Robust Policy Optimization (MRPO) to enhance domain generalization by connecting worst-case and average performance. Yet such approach imposes Lipschitz continuity assumptions, which are unaligned with 3D-BPP. The recently developed robust f -divergence MDP can approximate the worst-case value of a policy by utilizing nominal samples instead of relying on samples from an adversary (Ho et al., 2022; Panaganti et al., 2022). However, this method still cannot account for expected cases, as its focus is solely on learning values in worst-case scenarios.

3 Preliminaries

MDP Formulation of Online 3D-BPP. To learn a highly effective policy via RL, the online 3DBPP is formulated as an MDP. Inspired by PCT (Zhao et al., 2022a), in this formulation, the state s pack t ∈ S pack observed by the packing agent consists of the already packed N C items C t in the bin, the observed incoming N B items B t , and a set of potential positions L t . The packed items in C t have spatial properties like sizes and coordinates, while each item b t,i ∈ B t , i ∈ { 1 , .., N B } only provides size information. The potential positions are typically generated for the most preceding item in B t using heuristic methods (Martello et al., 2000; Crainic et al., 2008; Ha et al., 2017). Then, the action a pack t ∈ A pack is to choose one position l t,j ∈ L t , j ∈ { 1 , .., N L } for the first item in B t . In the termination state s T ( T is the episode length), the agent cannot pack any more items. As a result, the agent receives a delayed reward r T that represents the space utilization at s T , instead of immediate rewards ( r t = 0 , t < T ). The discount factor γ here is set to 1. The aim is to maximize the space utilization by learning a stochastic packing policy π pack ( l t,j | C t , B t , L t ) .

Robust MDP. The goal of robust MDP is to find the optimal robust policy that maximizes the value against the worst-case dynamics from an uncertainty set P w . The uncertainty set is defined in the neighborhood of the nominal dynamics P o = ( P o s,a , ( s, a ) ∈ S × A ) and satisfies rectangularity condition (Iyengar, 2005), defined as P w = ⊗ ( s,a ) ∈S×A P w s,a , P w s,a = { P s,a ∈ ∆( S ) : D TV ( P s,a || P o s,a ) ≤ ρ } , where D TV ( · ) denotes the total variation (TV) distance, ⊗ is the Cartesian product, ∆( S ) is a set of probability measures defined on S , and ρ ∈ [0 , 1] is the radius of the uncertainty set. Consequently, the robust value function under a policy π is V π r = inf P w ∈P w V π,P w r . And the robust Bellman operator T r is defined as T r V π r ( s ) = E a ∼ π [ r ( s, a ) + γ inf P w s,a ∈P w s,a E s ′ ∼ P w s,a [ V π r ( s ′ )]] , where s ′ denotes the next state. To empirically solve the robust MDP, Pinto et al. (2017) proposed RARL that learns the robust policy under the environment perturbed by a learned optimal adversary.

Robust f -divergence MDP. To avoid the need for a costly trained adversary, RfMDP (Ho et al., 2022; Panaganti et al., 2022) formulates the problem as a tractable constrained optimization task to approximate the robust value using samples from P o . The objective is to find a transition distribution

that minimizes the robust value function, subject to the constraint in P w described by the f -divergence. As a result, a new robust Bellman operator is introduced by solving the dual form of this constrained optimization problem through the Lagrangian multiplier method.

4 Adjustable Robust Reinforcement Learning

In this section, we begin by introducing a novel yet practical adversary capable of generating worstcase problem instances for the online 3D-BPP. Next, we present the adjustable robust reinforcement learning (AR2L) framework to address the robustness-performance tradeoff. Finally, we integrate AR2L into both the RARL algorithm and the RfMDP algorithm to derive the exact and approximate AR2L algorithms in a tractable manner.

4.1 Permutation-based Attacker

In online 3D-BPP, a problem instance is comprised of a sequence of items, and these items are randomly permuted and placed on a conveyor. Such randomness can result in certain instances where trained policy may fail. Therefore, this observation motivates us to design a simple yet realistic adversary called permutation-based attacker. By reordering the item sequence for a given packing policy, our approach can explore worstcase instances without compromising realism (See Figure 1 for attacker overview). In contrast to the approach

Packing Policy

Bin

Configuration

𝐂

!

Observable Items

𝐁

!

"

Potential Positions

𝐋

!

Bin Configuration

𝐂

!

Observable Items

𝐁

!

𝑏

!,#

Observable Items

𝐁

!

"

Permutation-based

Attacker

Probability

Bin Configuration

𝐂

!#$

𝐁

!

is reordered to

𝐁

!

"

𝑏

!,#

𝑏

!,$

𝑏

!,%

𝑏

!,#

𝑏

!,%

𝑏

!,$

𝑏

!,$

𝑏

!,%

of directly permuting the entire item sequence as described in (Kong et al., 2019), our method involves permuting the observable item sequence, thereby progressively transforming the entire item sequence. Through the behavior analysis of our permutation-based attacker in the Appendix, we observe the attacker appears to prefer smaller items when constructing harder instances as the number of observable items increases. This aligns with the findings mentioned in (Zhao et al., 2022a), where larger items simplify the scenario while smaller items introduce additional challenges. As a result, the act of permuting the entire sequence might enable the attacker to trickily select certain types of items, and thus carries potential risks associated with shifts in the underlying item distribution. However, we aim to ensure that the attacker genuinely learns how to permute the item sequence to identify and create challenging scenarios. To mitigate these risks, we limit the attacker's capacity by restricting the number N B of observable items to proactively address concerns related to changes in the item distribution.

To target any type of solver intended for solving the online 3D-BPP for robustness evaluation, we train a RL-based policy which acts as the permutation-based attacker. The permuted state s perm t ∈ S perm of the attacker is comprised of the current bin configuration C t and the item sequence B t . The action a perm t ∈ A perm involves moving one item from B t to a position ahead of the other observed items, resulting in the reordered item sequence B ′ t = { b ′ t,i } . Then, the packing policy will pack the most preceding one into the bin based on the perturbed observation. The reward for the attacker is defined as r perm t = -r pack t , since the objective is to minimize the space utilization by training an optimal stochastic policy π perm ( b t,i | C t , B t ) . The online packing process allows only the first item in the observable item sequence to be packed at each step. To address the issue of exponential growth in the action space as the number of observable items increases, the permutation-based attacker strategically selects one item and positions it ahead of the other items. We can consider a simplified value function represented as V π pack ( C t , B ′ t ) = r pack t + V π pack ( C t +1 , B ′ t +1 ) . This function indicates that the value of π pack depends on r pack t , C t +1 , and B ′ t +1 . Furthermore, r pack t and C t +1 are influenced by the first item in B ′ t , while the remaining items in B ′ t combine with a new item to form a new sequence, which undergoes further permutations as B ′ t +1 . As a result, the permutation of the remaining items at time step t is disregarded in the attack process. To model such

an attacker, we use the Transformer (Vaswani et al., 2017) that is capable of capturing long-range dependencies in spatial data and sequential data involved in the attacker's observation. Please refer to the Appendix for more details on the implementations.

4.2 Adjustable Robust Reinforcement Learning

For online 3D-BPP, item observed at each timestep is independently generated from a stationary distribution p b ( b t,i ) that does not change over time. Once an item is placed into the container, the bin configuration C t becomes deterministic, and a new item b t +1 ,N B is appended to the observable item sequence to construct B t +1 . Thus, the nominal environment is defined as P o ( C t +1 , B t +1 | C t , B t , a pack t ) = p b ( b t +1 ,N B ) . We omit L t for brevity. Since we use the permutation-based attacker to reorder the observable item sequence, the worst-case environment transition is P w ( C t +1 , B ′ t +1 | C t , B ′ t , a pack t ) = p b ( b t +1 ,N B ) π perm ( b ′ t +1 , 1 | C t +1 , B t +1 ) , with P w as the worst-case dynamics.

Existing DRL-based methods are developed to learn a packing policy that maximizes space utilization (cumulative reward) under the nominal distribution P o . However, in real-world scenarios where the order of items can be adversarially permuted, it is not sufficient to only consider the expected return under P o . In contrast, the existing general robust methods that can be deployed to online 3D-BPP overly prioritize robustness at the cost of average performance by solely focusing on the return under the worst-case dynamics P w . To address these issues, we should be aware of the returns under both the average and worst-case scenarios. Given that nominal cases are more common than worst-case scenarios in the online 3D-BPP setting, the objective function is defined as

where α ∈ (0 , 1] is the weight of robustness, η ( · ) is the return, and P w is built on the optimal permutation-based attacker. Here symbols without superscripts represent those used by the packing agent (e.g., π pack = π ).

However, how can we learn such a policy with the presence of both P o and P w ? To address this matter, we derive a lower bound for the objective function defined in Equation 1 by relating it to the return under an unknown mixture dynamics defined as P m , as shown in the following theorem.

Theorem 1. The model discrepancy between two models can be described as d ( P 1 || P 2 ) ≜ max s,a D TV ( P 1 ( ·| s, a ) || P 2 ( ·| s, a )) . The lower bound for objective 1 is derived as follows:

The RHS of Inequality 2 provides the lower bound for the objective function defined in Equation 1, where the first term represents the expected return of policy π under the mixture dynamics P m , while the second term denotes the weighted sum of deviations of P m from both P o and P w . Motivated by Theorem 1, we turn to maximizing the RHS of Inequality 2 to concurrently improve both the average and worst-case performance. Therefore, the primal problem in Equation 1 is equivalently reformulated to a surrogate problem where we expect to identify an optimal policy with the maximal lower bound, represented as

The objective of this optimization problem is to maximize the return by updating both the policy and the mixture dynamics controlled by robustness weight α , given the nominal and worst-case dynamics. Furthermore, the second term in Equation 3 can be viewed as a constraint that penalizes the deviations between environments, resulting in a constrained optimization problem written as π ∗ = arg max π ∈ Π { max P m ∈P η ( π, P m ) : d ( P m || P o )+ αd ( P m || P w ) ≤ ρ ′ } , where ρ ′ ∈ [0 , 1+ α ] . However, this constraint renders the constrained form of Equation 3 impractical to solve. Instead, a heuristic approximation is used for this constraint, which considers the TV distance at each state-action pair, leading to an uncertainty set for the mixture dynamics:

The uncertainty set P m satisfies the rectangularity condition Iyengar (2005). To solve the constrained optimization problem, we carry out an iterative procedure which searches for the associated mixture

Permutation-based

Attacker

DRL Loss

Distance Loss

Item Distribution

𝑝

!

Nominal Dynamics

Mixture-Dynamics

Model

Packing Policy

𝐁

!

, 𝐂

!

𝐁

!

""

𝑎

!

#$%&

𝐂

!

𝐁

!

, 𝐂

!

𝐁

!

"

𝐁

!

""

𝑟

!

'()

𝐁

!

(a) Exact AR2L Algorithm

Gradient Flow

Data Flow

Permutation-based

Attacker

DRL Loss

Item Distribution

𝑝

!

Nominal Dynamics

Packing Policy

𝐁

!

, 𝐂

!

𝑎

!

"#$%

𝑉

&

(𝐁

!

, 𝐂

!

)

Adjustable Robust

Value Function

𝐁

!

'

𝐁

!

, 𝐂

!

,

𝑟

!

"#$%

𝐁

!

, 𝐂

!

(b) Approximate AR2L Algorithm

dynamics for the policy evaluation, and improves the policy under the resulting mixture dynamics. Therefore, we define the adjustable robust value function as V π a = sup P m ∈P m V π,P m a for the policy evaluation. And the adjustable robust Bellman operator T a can be defined as

The adjustable robust Bellman operator assumes both the nominal and worst-case dynamics corresponding to π are available for constructing the uncertainty set P m . Unlike the traditional robust Bellman operator (Iyengar, 2005) designed for the minimal value, our proposed operator focus on maximizing the value of a given policy by updating the mixture dynamics. Meanwhile, to avoid the optimistic evaluation, the uncertainty set for the mixture dynamics used in Equation 13 is constrained with regard to both nominal and worst-case environment. Following policy evaluation, the policy is improved using samples drawn from the mixture dynamics. In addition, the following theorem guarantees that T a can facilitate the convergence of the value function to a fixed point.

Theorem 2. For any given policy π and its corresponding worst-case dynamics P w , the adjustable robust Bellman operator T a is a contraction whose fixed point is V π a . The operator equation T a V π a = V π a has a unique solution.

Exact AR2L Algorithm. The AR2L algorithm evaluates and improves policies based on the mixture dynamics that exists in the neighborhoods of both the nominal and worst-case scenarios. However, this algorithm remains impractical since the adjustable robust Bellman operator defined in Equation 13 involves computing the expectations w.r.t. all models in the uncertainty set P m . This potentially results in high computational complexity. Inspired by the classic RARL algorithm (Pinto et al., 2017) which trains a robust policy utilizing perturbations injected by a learned optimal adversary, we propose an iterative training approach that involves training a mixture-dynamics model π mix alongside the packing policy π pack and the permutation-based attacker π perm to obtain the exact AR2L algorithm. The mixture-dynamics model is corresponding to the mixture dynamics, and it permutes the observable item sequences B t to B ′′ t based on observed packed items C t . As illustrated in Figure 2(a), such a model receives the same reward as the packing policy ( r mix t = r pack t ). Based on Equation 3, the mixture-dynamics model strives to maximize the return under a given policy, while also penalizing deviations between dynamics. As P m and P w have the same form with the same underlying item distribution p b , we can only measure the discrepancy between P m and P w by evaluating the deviation between π mix and π perm . In practice, the Kullback-Leibler (KL) divergence is adopted to constrain the deviation between distributions. Thus, the training loss is

L mix = -η ( π pack , π mix ) + ( D KL ( π mix || ✶ { x = b t +1 , 1 } ) + αD KL ( π mix || π perm )) , (6) where D KL is the KL divergence, and ✶ { x = b t +1 , 1 } denotes the indicator function. Thus, the mixturedynamics model is optimized by minimizing both the DRL loss (the first term) and the distance loss (the second term). Based on this, the exact AR2L algorithm iterates through three stages to progressively improve the packing policy. First, the permutation-based attacker is optimized for a given packing policy, as is done in RARL. Next, the mixture-dynamics model π mix is learned using the loss function defined in Equation 6. Finally, the packing policy is evaluated and improved using problem instances that are permuted by π mix .

Approximate AR2L Algorithm. Training the mixture-dynamics model in exact AR2L introduces additional computation to the entire framework. We thus propose the approximate AR2L algorithm, which uses samples from both the nominal and worst-case dynamics to estimate adjustable robust values for the policy iteration as shown in Figure 2(b). Inspired by RfMDP (Ho et al., 2022; Panaganti et al., 2022), we use the dual reformulation of the adjustable robust Bellman operator, as given below.

Proposition 1. The policy evaluation of AR2L can be formulated as a constrained optimization problem. The objective is to maximize the value function over the uncertainty set P m , subject to the constraint defined in Equation 4. By representing the TV distance using the f -divergence (Shapiro, 2017), the adjustable robust Bellman operator T a given in Equation 13 can be equivalently written as

where [ x ] + denotes max { x, 0 } , µ 1 ( s ) and µ 2 ( s ) are Lagrangian multipliers for each s ∈ S ; µ = µ 1 ( s ) + αµ 2 ( s ) holds for each s ∈ S , and λ = max s { V π a ( s ) -µ 1 ( s ) , V π a ( s ) -µ 2 ( s ) , 0 } .

The approximate AR2L algorithm eliminates the requirement of using samples from the mixture dynamics for policy evaluation, yet it introduces estimation errors into the process. In practice, this results in mild performance degradation and sometimes unstable learning. Nevertheless, it remains valuable as the approximate AR2L still demonstrates superior performance compared to RfMDP.

In order to implement the AR2L algorithm, we opt to use the PPO algorithm (Schulman et al., 2017) to train packing policy π pack , permutation-based attacker π perm and mixture-dynamics model π mix . The packing policy is implemented using the PCT method (Zhao et al., 2022a) to accommodate the continuous solution space. All of the relevant implementation details, pseudocode for the algorithms, and both derivations and proofs are included in the Appendix.

5 Experiments

Training and Evaluation Settings In the online 3D-BPP setting, the container sizes S d , d ∈ { x, y, z } are equal for each dimension ( S x = S y = S z ), and the item sizes s d , d ∈ { x, y, z } are limited to no greater than S d / 2 to create more complex scenarios. The stability of each item is checked based on constraints used in (Zhao et al., 2022a,b). Moreover, we adopt two different popular settings used in (Zhao et al., 2022a,b) for training DRL-based policies. In the discrete setting, where item sizes are uniformly generated from a discrete set, i.e., s d ∈ { 1 , 2 , 3 , 4 , 5 } , resulting in a total of 125 types of items, and the container size S d is fixed at 10. In the continuous setting, the container size S d is set to 1, and item sizes s d , d ∈ { x, y } follow a continuous uniform distribution U (0 . 1 , 0 . 5) ; and s z is uniformly sampled from a finite set { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } . Then, discrete and continuous datasets are created following instructions from the two aforementioned training settings to evaluate both heuristic and DRL-based methods. Each dataset consists of 3,000 problem instances, where each problem instance contains 150 items. We use three metrics to evaluate the performance of various packing strategies: Uti. represents the average space utilization rate of the bin's volume; Std. is the standard deviation of space utilization which evaluates the algorithm's reliability across all instances; Num. evaluates the average number of packed items.

5.1 Algorithm Robustness under Permutation-based Attacks

In our experiment, we evaluate the robustness of six heuristic methods, including deep-bottom-left (DBL) method (Karabulut and ˙ Inceo˘ glu, 2005), best-match-first (BMF) method (Li and Zhang, 2015), least-surface-area heuristics (LSAH) (Hu et al., 2017), online bin packing heuristics (OnlineBPH) (Ha et al., 2017), heightmap-minimization (HMM) method (Wang and Hauser, 2019) and maximizeaccessible-convex-space (MACS) method (Hu et al., 2020). For the DRL-based methods, we evaluate the constrained deep reinforcement learning (CDRL) (Zhao et al., 2021) algorithm, and the packing configuration tree (PCT) method (Zhao et al., 2022a). We train specific permutation-based attackers for each of them. Since the capacity of the permutation-based attacker is highly related to the number of observable items of the attacker, we choose N B = 5 , 10 , 15 , 20 to obtain different attackers. Given that the number of observable items for the packing policy is typically limited to the first item in most methods, except for PCT, we can establish a consistent approach by setting the number of observable items for the packing policy to 1 in this case.

Table 1 presents the performance of various methods under perturbations from corresponding permutation-based attackers in the discrete setting. We can observe that though the HMM algorithm does not perform the best in the nominal scenario compared to other heuristic methods, it exhibits superior performance under attackers with different attack capacities. By comparison, the DBL method is the most vulnerable heuristic method. As attack capabilities increase, the CDRL algorithm demonstrates greater robustness compared to the PCT method, consistently achieving a higher average number of packed items with smaller variance of volume ratio. It is worth noting that the larger the value of N B is, the more the performance can be degraded for all methods, indicating that harder problem instances can be explored by increasing the value of N B . In addition, the number of packed items does not necessarily decrease as attack capacity increases. This is because permutation-based attackers aim to minimize space utilization, which means they may select smaller items to make the packing problem more challenging for the packing policy.

5.2 Performance of AR2L Algorithm

We then benchmark four other RL algorithms compatible with online 3D-BPP tasks, which include packing configuration tree (PCT) (Zhao et al., 2022a) method, the CVaR-Proximal-PolicyOptimization (CPPO) (Ying et al., 2022) algorithm, the robust adversarial reinforcement learning (RARL) algorithm (Pinto et al., 2017), and the robust f -divergence MDP (RfMDP) (Ho et al., 2022; Panaganti et al., 2022). The PCT algorithm serves as the baseline method and is used as the packing policy for all other methods in our implementations. The CPPO algorithm trains the packing policy with worst-case returns as a constraint, which can potentially improve both average and worst-case performance. The RARL algorithm is the baseline model for the exact AR2L algorithm due to its reliance on worst-case samples from the attacker. Likewise, the RfMDP framework serves as the baseline model for the approximate AR2L algorithm because it involves approximating values of the unknown dynamics. As both the exact AR2L (ExactAR2L) algorithm and the approximate AR2L algorithm (ApproxAR2L) can develop policies with different levels of robustness by adjusting the value of α , we choose α = 0 . 3 , 0 . 5 , 0 . 7 , 1 . 0 to explore the relationship between α and policy robustness. Furthermore, we use different attackers with varying attack capabilities ( N B = 5 , 10 , 15 , 20 ) to investigate the robustness of packing policies. It is important to note that in this scenario, the packing policy is permitted to observe the same number of items as its corresponding attacker. During testing, we construct mixture datasets by randomly selecting β % nominal box sequences and reordering them using the learned permutation-based attacker for each packing policy. The empirical results in the continuous setting are included in the Appendix. Since we implement PCT with different training configurations, we conducted a comparison between our implementation and the official implementation in the Appendix.

Uti. Table 2 presents the results of evaluating multiple robust methods on various datasets that differ in the number of problem instances perturbed by different attackers. We denote the different values of α used by the ExactAR2L algorithm as ExactAR2L( α ), and similarly for ApproxAR2L( α ). The results unequivocally indicate that the ExactAR2L algorithm, when employed with a higher value of α , exhibits superior performance on nominal cases and competitive robustness in terms of space utilization when compared to other robust methods. When the value of α is increased, it allows the packing policy to encounter and learn from more worst-case samples during training. By experiencing a wider range of adversarial situations, the packing policy can better adapt to mitigate the impact of attackers, enhancing its overall robustness. However, ExactAR2L cannot consistently improve its average performance across all the settings with different values of N B . For the scenario

where N B = 5 , the distribution deviation between the perturbed data and the nominal data ( β = 0 ) is comparatively minimal. This indicates that increasing the value of α and incorporating more challenging instances into the training settings of the nominal dynamics is an acceptable approach. By doing so, the generalization of the model can be enhanced, allowing it to handle a wider range of scenarios. However, the distribution deviation between the perturbed data and the nominal data ( β = 0 ) increases as the number of observable items increases. In such cases, it becomes less desirable to significantly increase the value of α . As a result, in the cases of N B = 10 and N B = 15 under β = 0 , ExactAR2L tends to favor α = 0 . 5 . Furthermore, in the scenario of N B = 20 , ExactAR2L

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(a)

N

B

= 15

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(b)

N

B

= 20

0

25

50

75

100

55

60

65

70

75

80

space utilization(%)

=0.1

=0.2

=0.3

=0.4

(c)

N

B

= 5

, α

= 1

.

0

0

25

50

75

100

50

55

60

65

70

75

80

space utilization(%)

=0.1

=0.2

=0.3

=0.4

(d)

N

B

= 10

, α

= 0

.

5

cannot outperform PCT due to the larger distribution deviation caused by the increased number of observable items of the attacker. In a word, by including perturbed data with a small deviation from the nominal data during training, the enhanced generalization leads to the performance improvement in nominal cases ( β = 0 ). Conversely, the large distribution deviation between the perturbed data and the nominal data can degenerate the performance in nominal cases, as indicated by the RARL algorithm. We also observe that as the value of β increases, the ExactAR2L algorithm tends to favor a larger value of α . Furthermore, the ApproxAR2L algorithm exhibits a similar performance tendency in both nominal and worst-case scenarios as the ExactAR2L algorithm. Due to the introduced value estimation error, the ApproxAR2L algorithm cannot perform as well as the ExactAR2L algorithm. Despite this, the ApproxAR2L algorithm still performs better than its baseline model, the RfMDP algorithm. Additionally, as demonstrated in Figures 3(a) 3(b), the ExactAR2L algorithm can learn faster than other robust RL algorithms.

Std. As shown in Table 2, ExactAR2L demonstrates its superiority over PCT with smaller Std. in 17 tasks, while producing a slightly larger Std. in 3 tasks. Thus, ExactAR2L can indeed improve the robustness. Furthermore, we observe when N B = 5 , 10 , ExactAR2L tends to choose α = 0 . 7 for smaller Std. . On the other hand, for N B = 15 , 20 , ExactAR2L favors α = 1 . 0 . Since ExactAR2L is trained on both nominal and worst-case dynamics, while RARL is trained only on the worst-case dynamics, the ExactAR2L policy is less conservative than the RARL policy. While the conservativeness may result in smaller Std. in most tasks, it produces worse results in terms of Uti. under the nominal dynamics. It is worth noting that the value of Std. from ExactAR2L is the closest to that of RARL. This observation shows ExactAR2L can trade off between conservative and risky behavior, as the Std. from ExactAR2L is between that of RARL and PCT. Similarly, ApproxAR2L is less conservative than RfMDP, which causes ApproxAR2L cannot achieve a smaller Std. in all tasks.

Num. The ExactAR2L(1.0) algorithm can pack more items in 17 tasks compared to PCT, and shows a slight drop in 3 tasks, where the average drop is 0.2. We found that to pack more items, ExactAR2L consistently favors α = 1 . 0 across various tasks. Compared to RARL, ExactAR2L(1.0) can pack at least the same number of items in 16 tasks. Thus ExactAR2L(1.0) can produce competitive results compared to RARL and PCT in terms of Num. . Compared to the baseline method RfMDP, ApproxAR2L(0.5) can pack more items in 16 tasks, and shows a slight drop in only 4 tasks, where the average drop is 0.25.

Hyperparameter Configurations. Based on observations from Table 2, α = 1 . 0 is the best choice for ExactAR2L across different test settings. When β = 50 , 75 , ExactAR2L(1.0) performs the best compared to baselines and ExactAR2L with other values of α (with a slight drop compared to ExactAR2L(0.7) when β = 50 , N B = 15 ). If β = 100 , ExactAR2L(1.0) can still produce competitive results compared to RARL and significantly outperforms PCT. When β = 25 , although α = 1 . 0 is not the optimal choice, ExactAR2L(1.0) can still outperform other baselines. When β = 0 , ExactAR2L(1.0) significantly outperforms RARL, and the slight drop compared to PCT is acceptable, as our goal is to improve the robustness while maintaining average performance at an acceptable level. ρ is only used in ApproxAR2L algorithm. As shown in Figures 3(c) 3(d), we choose different values of ρ in different settings. We found that ρ = 0 . 1 is a trustworthy choice for ApproxAR2L. Based on the observations from Table 2 and Figures 3(c) 3(d), we conclude that ρ = 0 . 1 and α = 0 . 5 are the best choice for ApproxAR2L, as it outperforms its corresponding baseline in almost all the tasks.

6 Conclusions and Limitations

In this work, we propose a novel reinforcement learning approach, Adjustable Robust Reinforcement Learning (AR2L), for solving the online 3D Bin Packing Problem (BPP). AR2L achieves a balance between policy performance in nominal and worst-case scenarios by optimizing a weighted sum of returns. We use a surrogate task of identifying a policy with the largest lower bound of the return to optimize the objective function. We can seamlessly bridge AR2L into the RARL and RfMDP algorithms to obtain exact and approximate AR2L algorithms. Extensive experiments show that the exact AR2L algorithm improves the robustness of the packing policy while maintaining acceptable performance, but may introduce additional computational complexity due to the mixture-dynamics model. The approximate AR2L algorithm estimates values without samples from the mixture dynamics, yet performance is not up to our exact AR2L agent due to the existence of estimation error. Though our AR2L framework is designed for online 3D-BPP, it can be also adapted to other decision-making tasks. We will investigate more efficient and generalizable AR2L framework.

Acknowledgments and Disclosure of Funding

We appreciate the anonymous reviewers, (S)ACs, and PCs of NeurIPS2023 for their insightful comments to further improve our paper and their service to the community. We would like to thank the Turing AI Computing Cloud (TACC) (Xu et al., 2021) and HKUST iSING Lab for providing us computation resources on their platform.

References

Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 22-31. PMLR.

Behzadan, V. and Munir, A. (2017). Whatever does not kill deep reinforcement learning, makes it stronger. arXiv preprint arXiv:1712.09344 .

Chazelle, B. (1983). The bottomn-left bin-packing heuristic: An efficient implementation. IEEE Transactions on Computers , 32(08):697-707.

Choo, J., Kwon, Y.-D., Kim, J., Jae, J., Hottung, A., Tierney, K., and Gwon, Y. (2022). Simulationguided beam search for neural combinatorial optimization. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems .

Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. (2017). Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research , 18(1):6070-6120.

Crainic, T. G., Perboli, G., and Tadei, R. (2008). Extreme point-based heuristics for three-dimensional bin packing. Informs Journal on computing , 20(3):368-384.

de Castro Silva, J., Soma, N., and Maculan, N. (2003). A greedy search for the three-dimensional bin packing problem: the packing static stability case. International Transactions in Operational Research , 10(2):141-153.

Ding, W., Shi, L., Chi, Y., and Zhao, D. (2023). Seeing is not believing: Robust reinforcement learning against spurious correlation. arXiv preprint arXiv:2307.07907 .

Elhedhli, S., Gzara, F., and Yildiz, B. (2019). Three-dimensional bin packing and mixed-case palletization. INFORMS Journal on Optimization , 1(4):323-352.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2020). Generative adversarial networks. Communications of the ACM , 63(11):139-144.

Gzara, F., Elhedhli, S., and Yildiz, B. C. (2020). The pallet loading problem: Three-dimensional bin packing with practical constraints. European Journal of Operational Research , 287(3):1062-1074.

Ha, C. T., Nguyen, T. T., Bui, L. T., and Wang, R. (2017). An online packing heuristic for the three-dimensional container loading problem in dynamic environments and the physical internet. In Applications of Evolutionary Computation: 20th European Conference, EvoApplications 2017, Amsterdam, The Netherlands, April 19-21, 2017, Proceedings, Part II 20 , pages 140-155. Springer.

Hiraoka, T., Imagawa, T., Mori, T., Onishi, T., and Tsuruoka, Y. (2019). Learning robust options by conditional value at risk optimization. Advances in Neural Information Processing Systems , 32.

Ho, C. P., Petrik, M., and Wiesemann, W. (2022). Robust phi-divergence mdps. arXiv preprint arXiv:2205.14202 .

Hu, H., Zhang, X., Yan, X., Wang, L., and Xu, Y. (2017). Solving a new 3d bin packing problem with deep reinforcement learning method. arXiv preprint arXiv:1708.05930 .

Hu, R., Xu, J., Chen, B., Gong, M., Zhang, H., and Huang, H. (2020). Tap-net: transport-and-pack using reinforcement learning. ACM Transactions on Graphics (TOG) , 39(6):1-15.

Huang, S., Papernot, N., Goodfellow, I., Duan, Y., and Abbeel, P. (2017). Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284 .

Iyengar, G. N. (2005). Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280.

Jiang, Y., Li, C., Dai, W., Zou, J., and Xiong, H. (2021). Monotonic robust policy optimization with model discrepancy. In International Conference on Machine Learning , pages 4951-4960. PMLR.

Karabulut, K. and ˙ Inceo˘ glu, M. M. (2005). A hybrid genetic algorithm for packing in 3d with deepest bottom left with fill method. In Advances in Information Systems: Third International Conference, ADVIS 2004, Izmir, Turkey, October 20-22, 2004. Proceedings 3 , pages 441-450. Springer.

Kim, M., Park, J., and Park, J. (2022). Sym-NCO: Leveraging symmetricity for neural combinatorial optimization. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems .

Kong, W., Liaw, C., Mehta, A., and Sivakumar, D. (2019). A new dog learns old tricks: Rl finds classic optimization algorithms. In International conference on learning representations .

Kos, J. and Song, D. (2017). Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452 .

Kuang, Y., Lu, M., Wang, J., Zhou, Q., Li, B., and Li, H. (2022). Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7247-7254.

Kumar, A., Levine, A., and Feizi, S. (2021). Policy smoothing for provably robust reinforcement learning. arXiv preprint arXiv:2106.11420 .

Li, X. and Zhang, K. (2015). A hybrid differential evolution algorithm for multiple container loading problem with heterogeneous containers. Computers & Industrial Engineering , 90:305-313.

Liang, Y., Sun, Y., Zheng, R., and Huang, F. (2022). Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. arXiv preprint arXiv:2210.05927 .

Lu, H., Li, Z., Wang, R., Ren, Q., Li, X., Yuan, M., Zeng, J., Yang, X., and Yan, J. (2023). Roco: A general framework for evaluating robustness of combinatorial optimization solvers on graphs. In The Eleventh International Conference on Learning Representations .

Martello, S., Pisinger, D., and Vigo, D. (2000). The three-dimensional bin packing problem. Operations research , 48(2):256-267.

Oikarinen, T., Zhang, W., Megretski, A., Daniel, L., and Weng, T.-W. (2021). Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems , 34:26156-26167.

Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. arXiv preprint arXiv:2208.05129 .

Parreño, F., Alvarez-Valdés, R., Tamarit, J. M., and Oliveira, J. F. (2008). A maximal-space algorithm for the container loading problem. INFORMS Journal on Computing , 20(3):412-422.

Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017). Automatic differentiation in pytorch.

Pattanaik, A., Tang, Z., Liu, S., Bommannan, G., and Chowdhary, G. (2017). Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632 .

Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. (2017). Robust adversarial reinforcement learning. In International Conference on Machine Learning , pages 2817-2826. PMLR.

Qiu, R., Sun, Z., and Yang, Y. (2022). DIMES: A differentiable meta solver for combinatorial optimization problems. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems .

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

Seiden, S. S. (2002). On the online bin packing problem. Journal of the ACM (JACM) , 49(5):640-671.

Shapiro, A. (2017). Distributionally robust stochastic programming. SIAM Journal on Optimization , 27(4):2258-2275.

Shen, Q., Li, Y., Jiang, H., Wang, Z., and Zhao, T. (2020). Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning , pages 8707-8718. PMLR.

Song, S., Yang, S., Song, R., Chu, S., Zhang, W., et al. (2023). Towards online 3d bin packing: Learning synergies between packing and unpacking via drl. In Conference on Robot Learning , pages 1136-1145. PMLR.

Sun, Y., Zheng, R., Liang, Y., and Huang, F. (2021). Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. arXiv preprint arXiv:2106.05087 .

Tang, Y. C., Zhang, J., and Salakhutdinov, R. (2019). Worst cases policy gradients. arXiv preprint arXiv:1911.03618 .

Tessler, C., Efroni, Y ., and Mannor, S. (2019). Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30.

Veliˇ ckovi´ c, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2017). Graph attention networks. arXiv preprint arXiv:1710.10903 .

Verma, R., Singhal, A., Khadilkar, H., Basumatary, A., Nayak, S., Singh, H. V., Kumar, S., and Sinha, R. (2020). A generalized reinforcement learning algorithm for online 3d bin-packing. arXiv preprint arXiv:2007.00463 .

Wang, F. and Hauser, K. (2019). Stable bin packing of non-convex 3d objects with a robot manipulator. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8698-8704. IEEE.

Wang, F. and Hauser, K. (2020). Robot packing with known items and nondeterministic arrival order. IEEE Transactions on Automation Science and Engineering , 18(4):1901-1915.

Wang, Y. and Zou, S. (2021). Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems , 34:7193-7206.

Wang, Y. and Zou, S. (2022). Policy gradient method for robust reinforcement learning. In International Conference on Machine Learning , pages 23484-23526. PMLR.

Xu, K., Wan, X., Wang, H., Ren, Z., Liao, X., Sun, D., Zeng, C., and Chen, K. (2021). Tacc: A fullstack cloud computing infrastructure for machine learning tasks. arXiv preprint arXiv:2110.01556 .

Yang, Z., Yang, S., Song, S., Zhang, W., Song, R., Cheng, J., and Li, Y. (2021). Packerbot: Variablesized product packing with heuristic deep reinforcement learning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5002-5008. IEEE.

Ying, C., Zhou, X., Su, H., Yan, D., Chen, N., and Zhu, J. (2022). Towards safe reinforcement learning via constraining conditional value-at-risk. arXiv preprint arXiv:2206.04436 .

Yuan, J., Zhang, J., and Yan, J. (2023). Towards solving industrial sequential decision-making tasks under near-predictable dynamics via reinforcement learning: an implicit corrective value estimation approach.

Zhang, H., Chen, H., Boning, D., and Hsieh, C.-J. (2021). Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452 .

Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. (2020). Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems , 33:21024-21037.

Zhao, H., She, Q., Zhu, C., Yang, Y., and Xu, K. (2021). Online 3d bin packing with constrained deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 741-749.

Zhao, H., Yu, Y., and Xu, K. (2022a). Learning efficient online 3d bin packing on packing configuration trees. In International Conference on Learning Representations .

Zhao, H., Zhu, C., Xu, X., Huang, H., and Xu, K. (2022b). Learning practically feasible policies for online 3d bin packing. Science China Information Sciences , 65(1):112105.

A Appendix

A.1 Experiment

In addition to results in the discrete setting presented earlier in the main texts, this section focuses on introducing additional results in the continuous setting. For the continuous setting, the container size S d , d ∈ { x, y, z } is fixed at 1, and the sizes of items s d , d ∈ { x, y } follow a continuous uniform distribution U (0 . 1 , 0 . 5) . To prevent impractical scenarios where packed items cannot form a supportive plane for incoming items, the s z dimension of each item is uniformly sampled from a finite set of values { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } (Zhao et al., 2022a). Likewise, Each dataset consists of 3,000 problem instances, where each problem instance contains 150 items. We use three metrics to evaluate the performance of various packing strategies: Uti. represents the average space utilization rate of the bin's volume; Std. is the standard deviation of space utilization across all instances; Num. evaluates the average number of packed items. In the field of online 3D-BPP (Zhao et al., 2021, 2022a; Song et al., 2023), the standard deviation of space utilization across a large number of problem instances (>1,000) is commonly used to assess the reliability and stability of a solver, rather than the error bar generated from multiple experiment runs (Kim et al., 2022; Choo et al., 2022; Qiu et al., 2022) . In addition, all the models are developed using PyTorch (Paszke et al., 2017) and trained on a Nvidia RTX 3090 GPU and an Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz.

A.1.1 Algorithm Robustness under Permutation-based Attacker

In the continuous setting, we conduct empirical evaluations to assess the robustness of four heuristic methods: deep-bottom-left (DBL) method (Karabulut and ˙ Inceo˘ glu, 2005), best-match-first (BMF) method (Li and Zhang, 2015), least-surface-area heuristic (LSAH) (Hu et al., 2017), and online bin packing heuristics (OnlineBPH) (Ha et al., 2017). In addition, we evaluated the packing configuration tree (PCT) algorithm (Zhao et al., 2022a) for the DRL-based methods, as other DRL-based methods are limited to spatially discretized grid world. We also evaluate the robustness of each algorithm under varying attack capabilities by selecting different values for N B , including 5, 10, 15, and 20. Since except PCT, other methods can observe only the first item in the item sequence at each time step, the observable number of items of each packing strategy is fixed at 1 to ensure a fair comparison between these packing strategies.

Table 1 displays the performance of different packing strategies against permutations generated by their corresponding permutation-based attackers. We can see that the LSAH algorithm exhibits superior space utilization performance across various experimental settings with different values of N B , even though it does not perform as well as the OnlineBPH and DBL algorithms in the nominal dynamics. In contrast, the BMF method is the most susceptible to attacks across all settings with varying numbers of observable items used by permutation-based attackers. With the increase of the attack capability, the PCT algorithm consistently outperforms the heuristic methods in terms of space utilization and the number of packed items, which can be attributed to its generalizability across a large number of problem instances. Similar to the results in the discrete setting, increasing the value of N B results in more performance degradation for all methods, which suggests an opportunity to explore more challenging problem instances, and to design proper robust algorithms accordingly.

Figure 1 depicts item distributions under perturbations from different attackers using PCT packing policies in the discrete setting. Since the sizes s d of items follow a uniform discrete distribution, the frequency of the sampled sizes are approximately equal, as illustrated in the leftmost bar chart in Figure 1. It is apparent that as the number of observable items of the attacker increases, the attacker

1

2

3

4

5

0

0.05

0.1

0.15

0.2

0.25

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

w/o attack

Size

Frequency

tends to select smaller items. Furthermore, as the attack capability increases, the underlying item distribution undergoes more significant changes. Therefore, in our setting, restricting the number of observable items is a practical approach to create permuted new problem instances that are closer to challenging real-world scenarios, as opposed to allowing an unlimited number of observable items.

In Figure 2, the item distributions from time step 1 to time step 20 are illustrated over different problem instances under perturbations from different attackers, using PCT packing policies in the discrete setting with N B = 15 , 20 . These item distributions not only partially reflect the problem instance distribution, but also reveal the preference of the attacker at each time step. At the beginning, the attacker favors items with smaller sizes, but as the time steps increase, it tends to sample items uniformly. This variation tendency illustrates the behavior of the attacker. However, it does not necessarily imply that rearranging the item sequence in ascending order based on item volumes would be an optimal or even effective attack. As selecting smaller items intuitively reduces the likelihood of forming supportive planes for future incoming items, attackers tend to choose smaller items to minimize the space utilization of the packing policy. Therefore, essentially, the attacker aims to select items that cannot be used to construct a supportive plane, rather than blindly choosing smaller items. This also explains why the probability of larger items in the beginning phase does not decrease to zero in Figure 2. As shown in Figure 3, we use a toy example to demonstrate that rearranging the item sequence in ascending order based on item sizes would not be an optimal or even effective attack. In this example, the container sizes S d , d ∈ { x, y, z } are set to 2, and the item sequence consists of four items, denoted as B = { b 1 , b 2 , b 3 , b 4 } . The sizes of the items are as follows: b 1 = (1 × 2 × 2) , b 2 = b 3 = (1 × 1 × 1) , and b 4 = (1 × 1 × 2) . We generate a new item sequence B ′ = { b 2 , b 3 , b 4 , b 1 } by rearranging B in ascending order based on item sizes. However, this rearrangement is not an effective attack since the optimal solution can still be obtained using a packing strategy. By comparison, moving both b 1 and b 4 from B to the middle between b 2 and b 3 results in B ′′ , which is an effective attack for achieving minimal space utilization. In such a case, the packing agent cannot pack b 1 into the container at the third time step.

A.1.2 Performance of AR2L Algorithm

In the continuous setting, we benchmark four other RL algorithms that are adaptable to online 3D-BPP, including the packing configuration tree (PCT) (Zhao et al., 2022a) method, the CVaR-ProximalPolicy-Optimization (CPPO) (Ying et al., 2022) algorithm, the robust adversarial reinforcement learning (RARL) (Pinto et al., 2017) algorithm, and the robust f -divergence MDP (RfMDP) (Ho et al., 2022; Panaganti et al., 2022) algorithm. Since both the exact AR2L (ExactAR2L) and approximate AR2L (ApproxAR2L) algorithms can adjust the value of α to develop policies that vary in their level of robustness, we use different values of α = 0 . 3 , 0 . 5 , 0 . 7 , 1 . 0 to explore the robustness trends with varying values of α in this setting. Likewise, we construct mixture datasets by randomly selecting β % nominal box sequences and rearranging them using the learned corresponding permutation-based attacker for each packing policy.

Uti. Table 2 displays the performance of different robust methods that are compatible with online 3D-BPP on datasets with varying number of challenging problem instances in the continuous setting. ExactAR2L( α ) represents the ExactAR2L algorithm taking different values for α , and similarly for ApproxAR2L( α ). In the continuous setting, increasing the value of α can lead to ExactAR2L producing more robust policies against their corresponding permutation-based attackers, as the larger value of α provides more worst-case problem instances during the training of the packing policy. However, they cannot consistently improve their performance on the nominal dynamics. As

Frequency

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

Size

Frequency

(a)

N

B

= 15

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

1

2

3

4

5

0

0.2

0.4

Size

(b)

N

B

= 20

Packing Process

b1

b2

63

64

Step1

Step 2

Step3

Step 4

Item Sequence B'

Packing Process

b2

b3

b4

b1

Step 1

Step3

Step 4

Packing Process

Kcannot be packed

b2

b4

b1

b3

Step1

Step 3

the number of observable items of the attacker increases, there is a corresponding increase in the distribution deviation between the perturbed data and the nominal data ( β = 0 ). For the scenario when N B = 5 , 10 , the distribution deviation is comparatively small. Thus, it is acceptable to incorporate more challenging instances by increasing the value of α to 0.7. This adjustment will further enhance the generalization of the model and lead to improved performance in terms of space utilization compared to PCT. By comparison, in the cases of N B = 15 and N B = 20 , ExactAR2L tends to favor α = 0 . 3 to produce satisfactory performance for β = 0 . Thus, by incorporating the perturbed data with small deviation from the nominal data during training, the generalization of the packing policy can be enhanced, leading to the performance improvement. But the larger deviation can degenerate the performance of the packing policy on the nominal datasets, as demonstrated in RARL. Likewise to the tendency in discrete setting, the ExactAR2L algorithm prefers the larger value of α as the value of β increases. Furthermore, due to the value estimation error, ApproxAR2L cannot perform as well as its exact counterpart in most of tasks under the continuous setting. However, to our surprise, we observed that in the case of N B = 5 , ApproxAR2L surpasses its corresponding exact counterpart in terms of performance. This could be attributed to the relatively small estimation error resulting from the small deviation in dynamics in this setting, where removing the mixture-dynamics model results in a more stable learning process compared to the exact algorithm. In the remaining tasks, the estimation error in ApproxAR2L can result in more unstable learning, which makes the trend in performance across different values of α less clear compared to the ExactAR2L algorithm.

Std. As shown in Table 2, in almost all tasks, ExactAR2L with a larger value of α ( α = 1 . 0 ) exhibits superiority over PCT in terms of Std. . Thus, ExactAR2L can indeed improve the robustness in the continuous setting. However, it should be noted that ExactAR2L does not consistently yield smaller values of Std. compared to RARL. This is because the ExactAR2L policy tends to be less conservative than the RARL policy in the majority of tasks. Likewise, ApproxAR2L algorithm prefers larger value of α to consistently achieve smaller Std. in different tasks.

Num. In comparison to PCT, the ExactAR2L(1.0) algorithm demonstrates the ability to pack more items in 19 tasks, with a minor decrease observed in 1 task. However, when compared to RARL, the ExactAR2L(1.0) policy only outperforms in 9 tasks. This discrepancy can primarily be attributed to the performance drop observed in the worst-case dynamics of the ExactAR2L algorithm. The ApproxAR2L policy tends to prefer α = 0 . 7 over the RfMDP policy when it comes to packing more items across all the tasks.

Hyperparameter Configurations. Based on the observations presented in Table 2, it can be concluded that α = 1 . 0 is the optimal choice for achieving a desired balance between policy performance in both nominal and worst-case scenarios. The ExactAR2L(1.0) algorithm demonstrates competitive robustness and average performance when compared to both PCT and RARL. Compared to the baseline method RfMDP, ApproxAR2L chooses α = 0 . 7 for the best performance.

In addition, although the convergence rate is not our primary concern in evaluating the RL algorithm, we still intend to observe the trend in the learning curves across varying attack capabilities. Our ExactAR2L algorithm shows greater improvement in terms of convergence rate in both the discrete and continuous settings as the number of observable items of the attacker increases, as demonstrated clearly in Figure 4 5.

A.1.3 Visualization Results

Figure 8 shows a qualitative comparison of the visualized bin packing results obtained by the ExactAR2L, RARL, and PCT policies under the nominal and worst-case dynamics with varying numbers of observable items N B = 15 , 20 in the discrete setting. As shown in Figure 8(a) 8(b), the ExactAR2L algorithm allows more smaller margin spaces in the container to minimize sizes of those unoccupied spaces. By comparison, the RARL algorithm attempts to create larger spaces, which may pose a risk of future items not fitting into those spaces. The visualization results of the ExactAR2L and PCT algorithms under their respective worst-case dynamics are depicted in Figure 8(c) and 8(d).

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(a)

N

B

= 5

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(b)

N

B

= 10

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(c)

N

B

= 15

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(d)

N

B

= 20

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

RARL

ExactAR2L(

=0.7)

ApproxAR2L(

=0.7)

CPPO

RfMDP

(a)

N

B

= 5

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=0.7)

CPPO

RfMDP

(b)

N

B

= 10

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(c)

N

B

= 15

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

RARL

ExactAR2L(

=1.0)

ApproxAR2L(

=1.0)

CPPO

RfMDP

(d)

N

B

= 20

One observation is that attackers designed for packing policies tend to prioritize smaller items at the beginning, as a majority of smaller items are likely to occur at the bottom of the container. In such a scenario, the ExactAR2L policy prioritizes the creation of supportive planes for future items, even at the cost of producing smaller margin spaces.

A.1.4 Baselines Comparisons

Since we implement PCT with different training configurations, we conducted a comparison between our implementation and the official implementation in this section. We conducted the comparison of the algorithms in terms of convergence rate across various values of N B , specifically N B = 1 , 5 , 10 , 15 , 20 . To ensure a fair and consistent comparison, we employed the same number of problem instances during training. Each iteration utilized a fixed rollout length of 30, and we maintained a constant number of 64 parallel processes throughout the experiments. As shown in Figure 6 and Figure 7, our results clearly demonstrate that our implementation exhibits a faster convergence rate across different settings, considering various numbers of observable items.

A.1.5 Practicability of AR2L Algorithm

To validate the practicality of exact AR2L algorithm in real-world scenarios, we directly evaluated our model without retraining on the Mixed-item Dataset (MI Dataset) (Yuan et al., 2023; Elhedhli et al., 2019) which follows the generation scheme proposed by Elhedhli et al. (2019) for the realistic 3D-BPP instance generator. MI dataset has 10 thousand items, with 4668 species, and occurrences vary from 1 to 15. The pallet dimensions is set to the size often used in practice: S x = 120 , S y = 100 , and S z = 100 . The results are presented in the Table 3. It is evident that in both settings of N B = 15 and N B = 20 , our exact AR2L algorithm demonstrates superior performance compared to PCT and RARL across various metrics in the real-world dataset.

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

Our Implementation

Official Implementation

(a)

N

B

= 1

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

Our Implementation

Official Implementation

(b)

N

B

= 5

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

Our Implementation

Official Implementation

(c)

N

B

= 10

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

Our Implementation

Official Implementation

(d)

N

B

= 15

1

20

40

60

80

100 120 140

iterations(×200)

60

65

70

75

80

space utilization(%)

convergence

Our Implementation

Official Implementation

(e)

N

B

= 20

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

Our Implementation

Official Implementation

(a)

N

B

= 1

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

Our Implementation

Official Implementation

(b)

N

B

= 5

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

Our Implementation

Official Implementation

(c)

N

B

= 10

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

Our Implementation

Official Implementation

(d)

N

B

= 15

1

20

40

60

80

100 120 140

iterations(×200)

50

55

60

65

70

space utilization(%)

convergence

Our Implementation

Official Implementation

(e)

N

B

= 20

76.6%

80.3%

78.7%

72.4%

69.8%

80.6%

79.7%

84.0%

81.1%

79.6%

73.8%

84.6%

ExactAR2L

RARL

(a) ExactAR2L and RARL algorithms in the nominal dynamics with

N

B

= 15

.

78.4%

73.4%

64.9%

78.0%

74.6%

68.8%

RARL

ExactAR2L

84.8%

78.4%

71.5%

82.9%

81.3%

75.4%

(b) ExactAR2L and RARL algorithms in the nominal dynamics with

N

B

= 20

.

67.5%

55.3%

61.6%

49.8%

57.4%

56.4%

PCT

72.7%

59.9%

63.6%

68.9%

59.5%

60.5%

ExactAR2L

(c) ExactAR2L and PCT algorithms in the worst-case dynamics with

N

B

= 15

.

51.3%

52.7%

60.4%

58.9%

52.0%

64.1%

PCT

68.4%

56.0%

65.8%

68.5%

63.9%

75.0%

ExactAR2L

(d) ExactAR2L and PCT algorithms in the worst-case dynamics with

N

B

= 20

.

A.2 Proofs

Lemma 1. [Achiam et al. (2017), Appendix.10.1.1] Given a policy π ( ·| s ) , a transition model P ( ·| s, a ) , and an initial state distribution µ ( s ) , the discounted state visitation distribution d π,P ( s ) under policy π can be described as:

Theorem 1. The model discrepancy between two models can be described as d ( P 1 || P 2 ) ≜ max s,a D TV ( P 1 ( ·| s, a ) || P 2 ( ·| s, a )) . The lower bound for the newly defined objective is derived as follows:

Proof. Given two dynamics models P 1 ( ·| s, a ) and P 2 ( ·| s, a ) , the absolute value of the difference of returns of a policy π can be represented as

Here, we set ∆ V ( s ) = V P 1 ( s ) -V P 2 ( s ) , and it can be transformed as:

The term 1 ⃝ can be firstly written as:

We then derive an upper bound of the second term in Equation 5 as:

Next, Equation 5 and 6 can be plugged into Equation 4, written as:

Thus, the expectation of ∆( s ) can be further upper bounded by:

E

s

∼

d

π,P

1

∆(

s

)

According to Lemma 1, Inequality 8 can be simplified as:

Consequently, E s ∼ d π,P 1 ∆( s ) can be eliminated, and we can obtain:

For the newly defined objective function, we can introduce any unknown dynamics P m to obtain:

Thus, the lower bound of the objective function can be derived as:

where d ( P 1 || P 2 ) ≜ max s,a D TV ( P 1 ( ·| s, a ) || P 2 ( ·| s, a )) .

Theorem 2. For any given policy π and its corresponding worst-case dynamics P w , the adjust Bellman operator T a is a contraction whose fixed point is V π a . The operator equation T a V π a = V π a has a unique solution.

Proof. For any given policy π and its corresponding worst-case dynamics P w , the adjustable robust Bellman operator T a is defined as:

where the uncertainty set is defined as P m = ⊗ ( s,a ) ∈S×A P m s,a ; P m s,a = { P s,a ∈ ∆( S ) : D TV ( P s,a || P o s,a ) + αD TV ( P s,a || P w s,a ) ≤ ρ ′ } .

For any two adjustable robust value functions V 1 and V 2 , we have:

(14)

Let us assume the existence of two different fixed points V ∗ 1 and V ∗ 2 for the adjustable Bellman operator. As both of these are fixed points, the following equality holds:

According to the proof in 14, the following inequality holds:

Thus, for both Equation 15 and Equation 16 to strictly hold true simultaneously, it must be the case that V ∗ 1 is equal to V ∗ 2 .

Definition 1. Let f : (0 , ∞ ) → R be a convex function with f (1) = 1 . Let P and Q be two probability distributions on a measurable space. Then, the f -divergence is defined as

where d P d Q is a Radon-Nikodym derivative.

Proposition 1. The policy evaluation of AR2L can be formulated as a constrained optimization problem. The objective is to maximize the value function over the uncertainty set P m , subject to the constraint defined in P m . By representing the TV distance using the f -divergence (Shapiro, 2017), the adjustable robust Bellman operator T a can be equivalently written as

where [ x ] + denotes max { x, 0 } , µ 1 ( s ) and µ 2 ( s ) are Lagrangian multipliers for each s ∈ S ; µ = µ 1 ( s ) + αµ 2 ( s ) holds for each s ∈ S , and λ = max s { V π a ( s ) -µ 1 ( s ) , V π a ( s ) -µ 2 ( s ) , 0 } .

Proof. The f -divergence can be utilized to express the constraint defined in the uncertainty set P m as follows:

where ζ : S → R + is the measurable function. ζ o ( s ) = d P ( s ) d P o ( s ) measures the difference between two dynamics at each sample s ∈ S , similarly for ζ w .

In the policy evaluation step of the adjustable robust Bellman operator, the goal is to obtain a mixture dynamics P m s,a that maximizes the adjustable value function while satisfying the constraint defined in Equation 19. We hereby formulate this procedure as a constrained optimization problem, written as:

where the value function V : S → R is also a measurable function. We use P o and P w to denote P o s,a and P w s,a for brevity, ρ = ρ ′ 1+ α is the normalized radius constant, and ϕ ( x ( s )) = 1 2 | x ( s ) -1 | is

a convex and lower semicontinuous function. We thus can use the Lagrangian multiplier method to solve the optimization problme in 20. The Lagrangian function is written as:

The Lagrangian dual of problem 20 is

Since the measurable function space L ( S , Σ( S ) , P ) is decomposable, the supremum in 22 can be taken inside the integral (Shapiro, 2017), that is

We can observe that the two terms inside the integral are identical to the definition of the conjugate function ϕ ∗ ( y ) = sup x ∈ R + { xy -ϕ ( x ) } . And ϕ ( x ) , as a convex, semicontinuous function, is defined as 1 2 | x ( s ) -1 | , resulting in ϕ ∗ ( y ) = max { y, -1 2 } . The dual problem can be transformed as:

We use [ x ] + to represent max { x, 0 } for brevity. It is then transformed as

Next, we set µ ′ 1 ( s ) = µ 1 + µ 3 ( s ) -1 2 λ and µ ′ 2 ( s ) = µ 2 -1 α µ 3 ( s ) -1 2 λ . And we observe that ∀ s ∈ S , µ ′ 1 ( s ) + αµ ′ 2 ( s ) = µ 1 -λ 2 +( µ 2 -λ 2 ) α . We thus set µ = µ ′ 1 ( s ) + αµ ′ 2 ( s ) .

In addition, based on the analysis of the conjugate function ϕ ∗ ( y ) in (Panaganti et al., 2022), we can derive the following two inequalities:

We thus can obtain λ = max s { V π a ( s ) -µ 1 ( s ) , V π a ( s ) -µ 2 ( s ) , 0 } . Building on the proofs above, we can define the adjustable robust Bellman operator as shown in Equation 18.

A.3 Pseudocode of Algorithms

The pseudocodes for both the exact AR2L algorithm and the approximate AR2L algorithm are presented in Algorithm 1 and Algorithm 2, respectively. Note that in practice, the Kullback-Leibler (KL) divergence is adopted to constrain the deviation between distributions in the exact AR2L algorithm.

Algorithm 1 Exact AR2L Algorithm

Initialize : packing policy π pack , permutation-based attacker π perm , mixture-dynamics model π mix , and their corresponding value functions V π pack , V π perm , V π mix ;

A.4 Implementation Details

To implement a packing policy that can work in both the discretized grid world and the continuous solution space, we use the packing configuration tree (PCT)(Zhao et al., 2022a) to represent the state, and employ a transformer(Vaswani et al., 2017) network as the packing policy. PCT is utilized to represent the bin configuration and available empty spaces for the most recently packed item, while the versatile transformer network is capable of handling the variable number of nodes in PCT. We adopt the same network configuration as reported in PCT for the packing policy, with the addition of positional encoding to encode the position information of each item in the observed item sequence.

In addition, those available spaces are generated using a heuristic method that takes into account the packing constraints and preferences for the item to be packed. Specifically, in the discrete setting, we utilize the empty maximal spaces (EMSs) (Ha et al., 2017) heuristics to generate potential positions, and subsequently apply the same

𝑂

𝑦

𝑥

constraints used in PCT to check the stability of these positions, thereby identifying the feasible positions L . In the continuous setting, we propose a new heuristic methods called Intersection Points (IPs) to generate potential positions. Given the 2D packed items in the xoy plane, the intersection points are generated from the intersection of the sides of cuboid-shaped items and the bin, as shown in Figure 9. To extend this method from 2D to 3D cases, we allow candidate positions to be obtained from the intersection points of the sides in the top view.

Algorithm 2 Approximate AR2L Algorithm

Initialize : packing policy π pack , permutation-based attacker π perm , and their corresponding value π pack π perm

In our implementations of both the permutation-based attacker and the mixture-dynamics model, we utilize the transformer (Vaswani et al., 2017) network to model these policies. This choice is motivated by the variable number of observed boxes in different real packing systems, as well as the dynamic changes in the number of packed items during each episode. The transformer is capable of capturing long-range dependencies in spatial data and sequential data. Specifically, we first use two independent element-wise fully-connected (FC) layers to project heterogenous elements of the state into homogenous features. Then, the scaled dot-product attention is relied to calculate the weighted sum of features based on the relevance between given input features. Next, we use the skip-connection operation and the element-wise feed-forward FC layer to obtain the final features x = { x C , x B } ∈ R d x × N , where x C ∈ R d x × N C and x B ∈ R d x × N B denote features of C t and B t respectively, d x is the feature size, and N = N C + N B denotes the variable feature number. Finally, the probability distribution over B t is described as

where ¯ x = 1 N ∑ N i =1 x i and c temp is the temperature constant.

As the number of packed items C and the number of available feasible positions L both vary at different time steps and have irregular shapes, we combine this data into one batch by filling C and L to fixed lengths of 80 and 120 in the discrete setting, and 100 and 120 in the continuous setting, respectively, using dummy elements. Each valid element in C is a vector that specifies the size and position of an item that has already been packed. Each vector in B represents the sizes of an item that has yet to be packed. Each vector in L contains the position of a feasible empty space that can accommodate a new item. To permute the observed item sequence, we feed both C and B to a transformer network that serves as the attacker. In this transformer network, the sizes of the two element-wise fully connected (FC) layers are fixed at 64. Additionally, there is one attention

layer with one head attention, and the sizes of the query, key, and value vectors are set to 64. The size of the element-wise feed-forward FC layer is also set to 64. In our experiment, we utilize the PPO (Schulman et al., 2017) algorithm to effectively train the packing policy, permutation-based attacker, and mixture-dynamics model. PPO employs multiple parallel processes (64 in this case) to interact with their respective environments and collect data. The rollout length for each iteration is set to 30, and the learning rate is set to 0.0003. To ensure a fair comparison, we keep the same hyperparameter settings mentioned above for each of the methods that we have reproduced.

A.5 Related Work

Heuristic Methods. In heuristic methods proposed for solving 3D-BPP, the score function is often designed to rank item placements based on expert knowledge, which can represent the packing requirements and preferences to some extent. Ha et al. (2017) relied on the deep-bottom-left (DBL) (Karabulut and ˙ Inceo˘ glu, 2005) rule to sort empty maximal spaces (EMSs) (Parreño et al., 2008) for the current item. The best-match-first strategy (Li and Zhang, 2015) gives priority to feasible placements that have the smallest margin for the observed item. Hu et al. (2017) introduced the least surface area heuristic, which selects the maximal empty space and orientation that result in the smallest surface area. The Heightmap-Minimization method (Wang and Hauser, 2019) favors items placement that result in the smallest occupied volume as observed from the loading direction. The maximize-accessible-convex-space method, as described by Hu et al. (2020), aims to optimize the empty space available for future, potentially large items. Despite their simplicity and effectiveness, these methods find difficulties in handling complex packing preferences and adapting to diverse scenarios for solving online 3D-BPP. This is because their packing strategies rely heavily on expert experience and may not be flexible enough to accommodate different situations.

DRL-based Methods. To further develop highly effective policies, DRL-based methods are broadly proposed by formulating the online 3D-BPP as a sequential decision-making problem. In this formulation, the state usually includes the bin configuration, which contains information about both the container and the packed items, as well as the observed item on the conveyor. To learn a DRL-based policy, the container is discretized to represent the bin configuration, and a convolutional neural network (CNN) is served as a policy (Hu et al., 2017; Verma et al., 2020; Zhao et al., 2021, 2022b; Yang et al., 2021; Song et al., 2023). However, these learning-based methods only work in a grid world with limited spatial discretization accuracy, which reduces their practical applicability. To deploy DRL-based methods on solving online 3D-BPP with continuous solution space, Zhao et al. (2022a) utilized the Packing Configuration Tree (PCT) to represent the bin configuration and potential available spaces. They then employed a graph attention network (Veliˇ ckovi´ c et al., 2017) to encode the PCT and observed boxes, to encode the PCT and observed boxes, and to select a location from available spaces generated for the most preceding item. However, these methods often focus on optimizing average performance and do not account for worst-case scenarios.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a novel adjustable and robust Reinforcement Learning framework for Online 3D Bin Packing task. The proposed method can achieve balance between performance and the worst-case environment.\n\nSTRENGTHS:\nThe writing is clear, and the experimental results demonstrate that the proposed method works well compared with other RL methods for the targeted 3D Bin Packing problem.\n\nWEAKNESSES:\n- In the section of “Training and Evaluation Setting”, the author mentioned that they are using two different settings which are discrete and continuous. However, I did not find the table showing the experimental results using continuous setting.\n\n- The author mentioned that they use three metrics(U_ti, Std and Num) to evaluate the performance. But only the best U_ti are highlighted in Table 2. I do think the analysis for Table 2 needs more details especially for the Std and Num.\n\n- The computation complexity is also crucial for the RL algorithms. The comparison of computation complexity should be analyzed.\n\nQUESTIONS:\n- Table 2 is the main result, where larger N_B and larger beta represent the harder problem, and we can see that throughout the table, the baseline RARL often achieves the best results, and the proposed Exact AR2L and Approx AR2L is inferior. What is your interpretation?\n\n- How to ensure the Approx AR2L can approximate the AR2L in the \"real-world\" setting? Since you do not perform experiments in the real world? I mean, can your added randomness correspond to a real-world robot arm use case (in some industry), and if so, what kind of distribution can characterize such randomness? I believe this must be addressed by data-driven experiments; however I do not see this in your paper. \n\n- Attacker: I understand the \"attacker\" is used to generate proper samples to train your DRL algorithm for policy improvement. However, do you have a better name for such adversarial or adversary other than an attacker, as they are used in a normal setting. If there is a true attacker of your DRL training, they will probably attack in other places the let the framework down. I understand this might be due to the term already used in the literature, I just want to not confusing the reader about technical terms.\n\nLIMITATIONS:\nThere is no such limitation in this aspect for this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor solving the online 3D Bin Packing Problem (BPP), the authors employ an iterative procedure to search for relevant hybrid dynamics and refine their corresponding strategies. By optimizing the weighted sum of returns, AR2L algorithm which improves the robustness of the packing policy achieves a balance between nominal insurance performance and worst case insurance performance.\n\nSTRENGTHS:\nIn experiment, the authors evaluate the robustness of six heuristic method. Moreover, this paper gives the exact AR2L algorithm and Approximate AR2L algorithm which has positive results. This two part of solving online 3D Bin Packing Problem has been proved to have relatively good results theoretically and experimentally.\n\nWEAKNESSES:\nExperimentation is not sufficient. Continuously valued perturbations for addition may not correspond to distributions in the physical world. Is there any difference between the response of the baseline to the disturbance of these continuous values and the method in the paper to the disturbance of these continuous values. Whether there is a simulation or a robotic arm, a demo for dealing with extreme situations.\n\nQUESTIONS:\n1.\tFor Figure 2, it should be ensured that the positions of the same modules remain unchanged, and the comparison changes should be highlighted, so as to achieve of the exact and approximate AR2L algorithm\n2.\tThe relationship between offline and online BPP introduced by introduce should be clarified, if not needed, offline should be deleted. Because the whole article does not focus on online BPP\n3.\tAt the same time, please supplement the experimental part in weaknesses\n\nLIMITATIONS:\ndue to the mixture-dynamics attacker, it will introduce additional calculations to increase complexity. At the same time for the latency of the whole process will be increased as well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the online three-dimensional bin packing problem (3D-BPP) and extends the PCT algorithm by proposing an adjustable robust reinforcement learning (AR2L) framework that balances the performance of policies in average and worst-case scenarios. The paper designs a permutation-based adversary and introduces an objective function that combines expected return and worst-case return with weights. A lower bound is derived to guide policy learning. The paper presents two algorithms to implement the AR2L framework, one exact that requires training a hybrid dynamics adversary, and another approximate that samples from the original dynamics and permutation-based adversary. Experiments are conducted in both discrete and continuous settings, demonstrating the effectiveness and superiority of the AR2L framework.\n\nSTRENGTHS:\n1. The problem addressed in the paper is highly significant. The authors extend the state-of-the-art PCT algorithm by optimizing its worst-case performance through an adversarial approach. I also have a strong impression of the PCT work, as it intuitively and effectively solves the important online bin packing problem. It is great to see the functionality of PCT being further enhanced through this work.\n2. I appreciate the organization of the paper. The authors demonstrate a strong understanding of packing and reinforcement learning, supporting each argument with data or theory. The paper is clear, understandable, and reasonable, making it enjoyable to read.\n3. The proposed approach is clever. The authors introduce a novel robust reinforcement learning framework, where a permutation-based adversary is designed to generate worst-case problem instances. This is an innovative and practical method. Additionally, the paper derives a lower bound for an objective function and utilizes it to guide policy learning, providing a theoretical contribution.\n4. The paper presents two algorithms to implement the AR2L framework, one exact and one approximate, catering to different scenarios and requirements.\n5. The paper conducts extensive experiments, comparing the AR2L framework with multiple packing and benchmark methods, demonstrating its advantages in terms of average performance and robustness.\n\nWEAKNESSES:\nThe paper does not provide guidance or suggestions on how to choose the hyperparameters α and ρ. I noticed that the optimal values of α vary when β changes in the best-performing AR2L algorithm. Perhaps this could be considered as future work.\n\nQUESTIONS:\n1. If I understand correctly, in Table 2 of the main text and Table 2 in the appendix, each trained policy is still tested using data generated by the attacker (as the performance of the baseline algorithms varies with different beta values, indicating a change in the tested data). So, if we remove the attacker and everyone uses the same data for testing, what would be the performance of AR2L? I am very interested in the practical implications of packing policies and would like to know if the packing policy learned through adversarial training still has advantages when tested on normal data (e.g., object sizes following a uniform distribution). This inquiry is purely out of curiosity and will not affect my scoring.\n2. In the appendix, the authors mention that they failed to reproduce some PCT results. Based on my own attempt, I believe PCT is quite reliable. I suggest that the authors contact the PCT authors as they may be able to assist in successfully reproducing the results. It is possible that there are some issues related to the usage or implementation, as packing involves various complex settings that require consideration of multiple aspects, such as discrete domain, continuous domain, stability checks, and data types. The functionality of the code can be quite complex. I believe the authors of PCT will be willing to help you successfully reproduce the results, and I have sent them a message to notify them and draw their attention to possible coming inquiry emails.\n3. A quick question: I would like to reproduce the results mentioned in the paper. In the code provided, what does \"bal_policy\" represent? Does the code refer to the code for ExactAR2L or ApproxAR2L, or is there an option to switch between the two algorithms?\n\nLIMITATIONS:\nThe paper only considers one type of permutation-based attacker and does not explore other possible attack methods, such as adding or removing items, modifying the size or shape of items, etc.\n\n----------------------After Rebuttal----------------------\n\nThe reviewer appreciates the author's response. However, the reviewer is still considering whether it is necessary to propose a new candidate generation method. During the rebuttal period, the reviewer downloaded the official implementation of PCT. The final test results were even higher than those reported in the PCT paper. Because the reviewer does not see a direct response in the rebuttal, holding onto this doubt, the reviewer intends to slightly lower the rating.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of 3D bin packing problem (3D-BPP). Specifically, it develops a permutation-based attacker and subsequently proposes an adjustable robust reinforcement learning (AR2L) framework. This allows an algorithm to consider both the average and worst-case performance with the attacker, where the packing objective is a weighted sum of expected and worst-case returns over the space utilization rate. The proposed framework is integrated with prior work to develop the exact and approximate variant of the proposed AR2L.\n\n--------------------------\nI acknowledge the author's effort in the rebuttal and have made changes to the review accordingly.\n\nSTRENGTHS:\n+ This work proposes to train a novel permutation-based attacker (RL-based policy) that can re-order the sequence of the observed items to reduce the space utilization of bin packaging tasks. The permutation-based attacker is used to quantitatively evaluate a given algorithm's performance under the worst-case scenario. The empirical results show new insight into the robustness of existing works.\n \n+ This work developed an adjustable robust reinforcement learning (AR2L) approach, which learns a packing policy based on both average scenarios and worst-case scenarios. This can be achieved via a surrogate problem where an optimal policy can be identified with the maximal lower bound. \n\n+ Two forms of implementation of the AR2L algorithm are shown in this work. The exact AR2L algorithm performs better worst-case performance under the worst-case attack with additional computation to the entire framework. On the other hand, the approximate AR2L algorithm introduces estimation errors with the cost of performance drop. For both variants, it outperforms the corresponding baseline methods.\n\nWEAKNESSES:\n- This work shows comprehensive evaluation with different combination of \\alpa, \\beta, and N_B parameters. While this provides valuable insights, it is unclear what would be the most optimal configuration in real-world deployment since the conditions could vary across task.\n\nQUESTIONS:\n- In eqn 2, should the last term be  \\alpha d( P^m || P^w )?\n\nLIMITATIONS:\nNo immediate limitations or impact raised from the novelties from this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is in the category of work that uses reinforcement learning for combinatorial optimization problems. More specifically, it focuses on the online version of the 3d-bin-packing problem (3D-BPP), and proposes a robust RL solution to address the uncertainty that arises from the permutation of an adversarial nature. At the core of their solution, Adjustable Robust Reinforcement Learning (AR2L), is 1) a new attacker that is based on permutation of the observable sequence of the items, 2) a new objective function that adjusts weights between nominal performance and worst-case performance, and 3) theories and algorithms that solve the new optimization.\n\nSTRENGTHS:\n+ RL for combinatorial optimization is an interesting and trending topic. It can potentially draw interest from the RL/OR community.\n\n+ Robustness in 3D-BPP is a practical and important issue. \n\n+ Permutation-based attacker makes sense. \n\n+ The adjustable objective makes sense. \n\n+ The evaluation of robustness for existing methods is applausible.\n\n+ Connections of AR2L to RARA and RfMDP are interesting.\n\nWEAKNESSES:\n- The attacker capacity seems questionable. Because of the following reasons, the problem setting seems rather impractical. These make the formulation, theorems, and evaluation somewhat trivial. \n1) If you formulate the nature as an adversary, then the nature should have control over the permutation of the entire sequence of items that are not presented to the packing policy, not just the observable ones. \n2) Also, it seems that the evaluation assumes the packing policy only observes one item. This is a bit weired because in this case the attacker does not have any capacity. \n3) Moreover, the attacker's policy is to change only one item to the most preceding item, but the attacker should be able to change the entire permutation. This is really how the attacker makes the training robust, and evaluations should be conducted against such an attacker. \n4) Finally, the setting that the sequence of the observable items can be changed is also weired. This is because the packing agent/policy has already seen the set of observable items, you cannot suddenly change them without the notice from the packing agent. \n\n- The paper, along with its formulation, seems to be too much restricted to robustness to permutation-based attack in 3D-BPP. Do the theories/results generalize to other settings? There is a lack of discussion from that aspect. This makes the technical contribution rather limited. The paper would be stronger if it can shed light on how the algorithms are generally applicable to more combinatorial optimization problems with uncertainty.\n\n- Theorem 1: $\\gamma$ is not explained -- is it the discount factor? Isn't is assumed to be 1 in the Preliminaries section? If so, the denominator is 0. At least $1-\\gamma$ is a very small value, making the second term on the RHS of Equation (2) dominating the entire RHS. This significantly restricts the meaningfulness of Theorem 1 (a major theoretical contribution). It seems that the objective in Equation (3) makes sense by itself, but without Theorem 1 this is only a heuristic. Also, why the coefficients are thrown out in Equation (2) and turned into Equation (3) was not explained.\n\n- The empirical results are not convincing \n1) it looks like that the average performance of AR2L is outperformed by the baselines (esp. RARL) in many cases (esp. in the most challenging setting where $\\beta=100$. It is surprising that PCT is not the best even for the nominal setting ($\\beta=0$). These have not been discussed and explained. \n2) Also, although the paper is about robustness, it primarily focuses only on average performance (Uti), discussions of Std are limited even thought the results are shown. \n3) I was expecting to see some guidelines/lessons-learned from what values I should take for the hyperparameters $\\alpha$ and $\\rho$, but did not see. If I were to use the algorithms, how do I choose them? It seems that if I choose the wrong ones, I might end up with a solution that is worse than the baselines. The mysterious performances regarding hyperparas are very un-convincing.\n\nMinor issues:\n* Refereces to equations should be Equation (1), instead of 1. See e.g., Line 205. \n* A typo at the end of Equation (2). $P^0$ -> $P^w$.\n*\n\nQUESTIONS:\nSee questions in the weakness part.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}