Learning Efficient Online 3D Bin Packing on Packing Configuration Trees

1 INTRODUCTION . As one of the most classic combinatorial optimization problems , the 3D bin packing problem usually refers to packing a set of cuboid-shaped items i ∈ I , with sizes sxi , s y i , s z i along x , y , z axes , respectively , into the minimum number of bins with sizes Sx , Sy , Sz , in an axis-aligned fashion . Traditional 3D-BPP assumes that all the items to be packed are known a priori ( Martello et al. , 2000 ) , which is also called offline BPP . The problem is known to be strongly NP-hard ( De Castro Silva et al. , 2003 ) . However , in many real-world application scenarios , e.g. , logistics or warehousing ( Wang & Hauser , 2019a ) , the upcoming items can not be fully observed ; only the current item to be packed is observable . Packing items without the knowledge of all upcoming items is referred to as online BPP ( Seiden , 2002 ) . Due to its obvious practical usefulness , online 3D-BPP has received increasing attention recently . Given the limited knowledge , the problem can not be solved by usual search-based methods . Different from offline 3D-BPP where the items can be placed in an arbitrary order , online BPP must place items following their coming order , which imposes additional constraints . Online 3D-BPP is usually solved with either heuristic methods ( Ha et al. , 2017 ) or learning-based ones ( Zhao et al. , 2021 ) , with complementary pros and cons . Heuristic methods are generally not limited by the size of action space , but they find difficulties in handling complex practical constraints such as packing stability or specific packing preferences . Learning-based approaches usually perform better than heuristic methods , especially under various complicated constraints . However , the learning is hard to converge with a large action space , which has greatly limited the applicability of learning-based methods due to , e.g. , the limited resolution of spatial discretization ( Zhao et al. , 2021 ) . We propose to enhance learning-based online 3D-BPP towards practical applicability through learning with a novel hierarchical representation –– packing configuration tree ( PCT ) . PCT is a dynamically growing tree where the internal nodes describe the space configurations of packed items and ∗Work conducted while the author was visiting the National Key Lab for Novel Software Technology . †Kai Xu is the corresponding author . leaf nodes the packable placements of the current item . PCT is a full-fledged description of the state and action space of bin packing which can support packing policy learning based on deep reinforcement learning ( DRL ) . We extract state features from PCT using graph attention networks ( Velickovic et al. , 2018 ) which encodes the spatial relations of all space configuration nodes . The state feature is input into the actor and critic networks of the DRL model . The actor network , designed based on pointer mechanism , weighs the leaf nodes and outputs the action ( the final placement ) . During training , PCT grows under the guidance of heuristics such as Corner Point ( Martello et al. , 2000 ) , Extreme Point ( Crainic et al. , 2008 ) , and Empty Maximal Space ( Ha et al. , 2017 ) . Although PCT is expanded with heuristic rules , confining the solution space to what the heuristics could explore , our DRL model learns a discriminant fitness function ( the actor network ) for the candidate placements , resulting in an effective and robust packing policy exceeding the heuristic methods . Furthermore , the size of the packing action space is proportional to the number of leaf nodes , making the DRL model easy to train and well-performing even with continuous solution space where the packing coordinates are continuous values . Through extensive evaluation , we demonstrate that our method outperforms all existing online 3D-BPP methods and is versatile in terms of incorporating various practical constraints such as isle friendliness and load balancing ( Gzara et al. , 2020 ) . Our work is , to our knowledge , the first that deploys the learning-based method on solving online 3DBPP with continuous solution space successfully . 2 RELATED WORK . Offline 3D-BPP The early interest of 3D-BPP mainly focused on its offline setting . Offline 3DBPP assumes that all items are known as a priori and can be placed in an arbitrary order . Martello et al . ( 2000 ) first solved this problem with an exact branch-and-bound approach . Limited by exponential worst-case complexity of exact approaches , lots of heuristic and meta-heuristic algorithms are proposed to get an approximate solution quickly , such as guided local search ( Faroe et al. , 2003 ) , tabu search ( Crainic et al. , 2009 ) , and hybrid genetic algorithm ( Kang et al. , 2012 ) . Hu et al . ( 2017 ) decompose the offline 3D-BPP into packing order decisions and online placement decisions . The packing order is optimized with an end-to-end DRL agent and the online placement policy is a handdesigned heuristic . This two-step fashion is widely accepted and followed by Duan et al . ( 2019 ) , Hu et al . ( 2020 ) , and Zhang et al . ( 2021 ) . Heuristics for Online 3D-BPP Although offline 3D-BPP has been well studied , their searchbased approaches can not be directly transferred to the online setting . Instead , lots of heuristic methods have been proposed to solve this problem . For reasons of simplicity and good performance , the deep-bottom-left ( DBL ) heuristic ( Karabulut & Inceoglu , 2004 ) has long been a favorite . Ha et al . ( 2017 ) sort the empty spaces with this DBL order and place the current item into the first fit one . Wang & Hauser ( 2019b ) propose a Heightmap-Minimization method to minimize the volume increase of the packed items as observed from the loading direction . Hu et al . ( 2020 ) optimize the empty spaces available for the packing future with a Maximize-Accessible-Convex-Space method . DRL for Online 3D-BPP The heuristic methods are intuitive to implement and can be easily applied to various scenarios . However , the price of good flexibility is that these methods perform mediocrely , especially for online 3D-BPP with specific constraints . Designing new heuristics for specific classes of 3D-BPP is heavy work since this problem has an NP-hard solution space , many situations need to be premeditated manually by trial and error . Substantial domain knowledge is also necessary to ensure safety and reliability . To automatically generate a policy that works well on specified online 3D-BPP , Verma et al . ( 2020 ) ; Zhao et al . ( 2021 ) employ the DRL method on solving this problem , however , their methods only work in small discrete coordinate spaces . Despite their limitations , these works are soon followed by Hong et al . ( 2020 ) ; Yang et al . ( 2021 ) ; Zhao et al . ( 2022 ) for logistics robot implementation . Zhang et al . ( 2021 ) adopt a similar online placement policy for offline packing needs referring to Hu et al . ( 2017 ) . All these learning-based methods only work in a grid world with limited discretization accuracy , which reduces their practical applicability . Practical Constraints The majority of literature for 3D-BPP ( Martello et al. , 2000 ) only considers the basic non-overlapping constraint 1 and containment constraint 2 : pdi + s d i ≤ pdj + Sd ( 1− edij ) i 6= j , i , j ∈ I , d ∈ { x , y , z } ( 1 ) 0 ≤ pdi ≤ Sd − sdi i ∈ I , d ∈ { x , y , z } ( 2 ) Where pi means the front-left-bottom coordinate of item i and d the coordinate axis , eij takes value 1 otherwise 0 if item i precedes item j along d. The algorithms for 3D-BPP are of limited practical applicability if no even basic real-world constraints , e.g. , stability ( Ramos et al. , 2016 ) , are considered . Zhao et al . ( 2022 ) propose a fast stability estimation method for DRL training and test their learned policies with real logistics boxes . The flaw of their work is the heightmap ( the upper frontier of packed items ) state representation like Zhang et al . ( 2021 ) is still used , while the underlying constraints between packed items are missed . The unavailability of underlying spatial information makes their problem a partially observable Markov Decision Process ( Spaan , 2012 ) which is not conducive to DRL training and limits the performance on 3D-BPP instances with more complex practical constraints , like isle friendliness and load balancing ( Gzara et al. , 2020 ) . 3 METHOD . In this section , we first introduce our PCT concept in Section 3.1 for describing the online packing process . The parameterization of the tree structure and the leaf node selection policy are introduced in Section 3.2 and Section 3.3 respectively . In Section 3.4 , we formulate online 3D-BPP as Markov Decision Process based on PCT , followed by the description of the training method . 3.1 PACKING CONFIGURATION TREE . When a rectangular item nt is added to a given packing with position ( pxn , p y n , p z n ) at time step t , it introduces a series of new candidate positions where future items can be accommodated , as illustrated in Figure 1 . Combined with the axis-aligned orientation o ∈ O for nt based on existing positions , we get candidate placements ( i.e . position and orientation ) . The packing process can be seen as a placement node being replaced by a packed item node , and new candidate placement nodes are generated as children . As the packing time step t goes on , these nodes are iteratively updated and a dynamic packing configuration tree is formed , denoted as T . The internal node set Bt ∈ Tt represents the space configurations of packed items , and the leaf node set Lt ∈ Tt the packable candidate placements . During the packing , leaf nodes that are no longer feasible , e.g. , covered by packed items , will be removed from Lt . When there is no packable leaf node that makes nt satisfy the constraints of placement , the packing episode ends . Without loss of generality , we stipulate a vertical top-down packing within a single bin ( Wang & Hauser , 2019b ) . Traditional 3D-BPP literature only cares about the remaining placements for accommodating the current item nt , their packing policies can be written as π ( Lt|Lt , nt ) . If we want to promote this problem for practical demands , 3D-BPP needs to satisfy more complex practical constraints which also act on Bt . Taking packing stability for instance , a newly added item nt has possibly force and torque effect on the whole item set Bt ( Ramos et al. , 2016 ) . The addition of nt should make Bt a more stable spatial distribution so that more items can be added in the future . Therefore , our packing policy over Lt is defined as π ( Lt|Tt , nt ) , which means probabilities of selecting leaf nodes from Lt given Tt and nt . For online packing , we hope to find the best leaf node selection policy to expand the PCT with more relaxed constraints so that more future items can be appended . Leaf Node Expansion Schemes The performance of online 3D-BPP policies has a strong relationship with the choice of leaf node expansion schemes –– which incrementally calculate new candidate placements introduced by the just placed item nt . A good expansion scheme should reduce the number of solutions to be explored while not missing too many feasible packings . Meanwhile , polynomially computability is also expected . Designing such a scheme from scratch is non-trivial . Fortunately , several placement rules independent from particular packing problems have been proposed , such as Corner Point ( Martello et al. , 2000 ) , Extreme Point ( Crainic et al. , 2008 ) , and Empty Maximal Space ( Ha et al. , 2017 ) . We extend these schemes which have proven to be accurate and efficient to our PCT expansion . The performance of learned policies will be reported in Section 4.1 .

Summary: This work proposes a tree-based learning method for online 3D packing problem. Packing configuration tree nodes is constructed using heuristic-based tree expansion, which acts as the action space of deep reinforcement learning. The tree search schema is interesting, but this work still has lots of space to improve in terms of its methodology and experiments.