Learning Efficient Online 3D Bin Packing on Packing Configuration Trees

1 INTRODUCTION . As one of the most classic combinatorial optimization problems , the 3D bin packing problem usually refers to packing a set of cuboid-shaped items i ∈ I , with sizes sxi , s y i , s z i along x , y , z axes , respectively , into the minimum number of bins with sizes Sx , Sy , Sz , in an axis-aligned fashion . Traditional 3D-BPP assumes that all the items to be packed are known a priori ( Martello et al. , 2000 ) , which is also called offline BPP . The problem is known to be strongly NP-hard ( De Castro Silva et al. , 2003 ) . However , in many real-world application scenarios , e.g. , logistics or warehousing ( Wang & Hauser , 2019a ) , the upcoming items can not be fully observed ; only the current item to be packed is observable . Packing items without the knowledge of all upcoming items is referred to as online BPP ( Seiden , 2002 ) . Due to its obvious practical usefulness , online 3D-BPP has received increasing attention recently . Given the limited knowledge , the problem can not be solved by usual search-based methods . Different from offline 3D-BPP where the items can be placed in an arbitrary order , online BPP must place items following their coming order , which imposes additional constraints . Online 3D-BPP is usually solved with either heuristic methods ( Ha et al. , 2017 ) or learning-based ones ( Zhao et al. , 2021 ) , with complementary pros and cons . Heuristic methods are generally not limited by the size of action space , but they find difficulties in handling complex practical constraints such as packing stability or specific packing preferences . Learning-based approaches usually perform better than heuristic methods , especially under various complicated constraints . However , the learning is hard to converge with a large action space , which has greatly limited the applicability of learning-based methods due to , e.g. , the limited resolution of spatial discretization ( Zhao et al. , 2021 ) . We propose to enhance learning-based online 3D-BPP towards practical applicability through learning with a novel hierarchical representation –– packing configuration tree ( PCT ) . PCT is a dynamically growing tree where the internal nodes describe the space configurations of packed items and ∗Work conducted while the author was visiting the National Key Lab for Novel Software Technology . †Kai Xu is the corresponding author . leaf nodes the packable placements of the current item . PCT is a full-fledged description of the state and action space of bin packing which can support packing policy learning based on deep reinforcement learning ( DRL ) . We extract state features from PCT using graph attention networks ( Velickovic et al. , 2018 ) which encodes the spatial relations of all space configuration nodes . The state feature is input into the actor and critic networks of the DRL model . The actor network , designed based on pointer mechanism , weighs the leaf nodes and outputs the action ( the final placement ) . During training , PCT grows under the guidance of heuristics such as Corner Point ( Martello et al. , 2000 ) , Extreme Point ( Crainic et al. , 2008 ) , and Empty Maximal Space ( Ha et al. , 2017 ) . Although PCT is expanded with heuristic rules , confining the solution space to what the heuristics could explore , our DRL model learns a discriminant fitness function ( the actor network ) for the candidate placements , resulting in an effective and robust packing policy exceeding the heuristic methods . Furthermore , the size of the packing action space is proportional to the number of leaf nodes , making the DRL model easy to train and well-performing even with continuous solution space where the packing coordinates are continuous values . Through extensive evaluation , we demonstrate that our method outperforms all existing online 3D-BPP methods and is versatile in terms of incorporating various practical constraints such as isle friendliness and load balancing ( Gzara et al. , 2020 ) . Our work is , to our knowledge , the first that deploys the learning-based method on solving online 3DBPP with continuous solution space successfully . 2 RELATED WORK . Offline 3D-BPP The early interest of 3D-BPP mainly focused on its offline setting . Offline 3DBPP assumes that all items are known as a priori and can be placed in an arbitrary order . Martello et al . ( 2000 ) first solved this problem with an exact branch-and-bound approach . Limited by exponential worst-case complexity of exact approaches , lots of heuristic and meta-heuristic algorithms are proposed to get an approximate solution quickly , such as guided local search ( Faroe et al. , 2003 ) , tabu search ( Crainic et al. , 2009 ) , and hybrid genetic algorithm ( Kang et al. , 2012 ) . Hu et al . ( 2017 ) decompose the offline 3D-BPP into packing order decisions and online placement decisions . The packing order is optimized with an end-to-end DRL agent and the online placement policy is a handdesigned heuristic . This two-step fashion is widely accepted and followed by Duan et al . ( 2019 ) , Hu et al . ( 2020 ) , and Zhang et al . ( 2021 ) . Heuristics for Online 3D-BPP Although offline 3D-BPP has been well studied , their searchbased approaches can not be directly transferred to the online setting . Instead , lots of heuristic methods have been proposed to solve this problem . For reasons of simplicity and good performance , the deep-bottom-left ( DBL ) heuristic ( Karabulut & Inceoglu , 2004 ) has long been a favorite . Ha et al . ( 2017 ) sort the empty spaces with this DBL order and place the current item into the first fit one . Wang & Hauser ( 2019b ) propose a Heightmap-Minimization method to minimize the volume increase of the packed items as observed from the loading direction . Hu et al . ( 2020 ) optimize the empty spaces available for the packing future with a Maximize-Accessible-Convex-Space method . DRL for Online 3D-BPP The heuristic methods are intuitive to implement and can be easily applied to various scenarios . However , the price of good flexibility is that these methods perform mediocrely , especially for online 3D-BPP with specific constraints . Designing new heuristics for specific classes of 3D-BPP is heavy work since this problem has an NP-hard solution space , many situations need to be premeditated manually by trial and error . Substantial domain knowledge is also necessary to ensure safety and reliability . To automatically generate a policy that works well on specified online 3D-BPP , Verma et al . ( 2020 ) ; Zhao et al . ( 2021 ) employ the DRL method on solving this problem , however , their methods only work in small discrete coordinate spaces . Despite their limitations , these works are soon followed by Hong et al . ( 2020 ) ; Yang et al . ( 2021 ) ; Zhao et al . ( 2022 ) for logistics robot implementation . Zhang et al . ( 2021 ) adopt a similar online placement policy for offline packing needs referring to Hu et al . ( 2017 ) . All these learning-based methods only work in a grid world with limited discretization accuracy , which reduces their practical applicability . Practical Constraints The majority of literature for 3D-BPP ( Martello et al. , 2000 ) only considers the basic non-overlapping constraint 1 and containment constraint 2 : pdi + s d i ≤ pdj + Sd ( 1− edij ) i 6= j , i , j ∈ I , d ∈ { x , y , z } ( 1 ) 0 ≤ pdi ≤ Sd − sdi i ∈ I , d ∈ { x , y , z } ( 2 ) Where pi means the front-left-bottom coordinate of item i and d the coordinate axis , eij takes value 1 otherwise 0 if item i precedes item j along d. The algorithms for 3D-BPP are of limited practical applicability if no even basic real-world constraints , e.g. , stability ( Ramos et al. , 2016 ) , are considered . Zhao et al . ( 2022 ) propose a fast stability estimation method for DRL training and test their learned policies with real logistics boxes . The flaw of their work is the heightmap ( the upper frontier of packed items ) state representation like Zhang et al . ( 2021 ) is still used , while the underlying constraints between packed items are missed . The unavailability of underlying spatial information makes their problem a partially observable Markov Decision Process ( Spaan , 2012 ) which is not conducive to DRL training and limits the performance on 3D-BPP instances with more complex practical constraints , like isle friendliness and load balancing ( Gzara et al. , 2020 ) . 3 METHOD . In this section , we first introduce our PCT concept in Section 3.1 for describing the online packing process . The parameterization of the tree structure and the leaf node selection policy are introduced in Section 3.2 and Section 3.3 respectively . In Section 3.4 , we formulate online 3D-BPP as Markov Decision Process based on PCT , followed by the description of the training method . 3.1 PACKING CONFIGURATION TREE . When a rectangular item nt is added to a given packing with position ( pxn , p y n , p z n ) at time step t , it introduces a series of new candidate positions where future items can be accommodated , as illustrated in Figure 1 . Combined with the axis-aligned orientation o ∈ O for nt based on existing positions , we get candidate placements ( i.e . position and orientation ) . The packing process can be seen as a placement node being replaced by a packed item node , and new candidate placement nodes are generated as children . As the packing time step t goes on , these nodes are iteratively updated and a dynamic packing configuration tree is formed , denoted as T . The internal node set Bt ∈ Tt represents the space configurations of packed items , and the leaf node set Lt ∈ Tt the packable candidate placements . During the packing , leaf nodes that are no longer feasible , e.g. , covered by packed items , will be removed from Lt . When there is no packable leaf node that makes nt satisfy the constraints of placement , the packing episode ends . Without loss of generality , we stipulate a vertical top-down packing within a single bin ( Wang & Hauser , 2019b ) . Traditional 3D-BPP literature only cares about the remaining placements for accommodating the current item nt , their packing policies can be written as π ( Lt|Lt , nt ) . If we want to promote this problem for practical demands , 3D-BPP needs to satisfy more complex practical constraints which also act on Bt . Taking packing stability for instance , a newly added item nt has possibly force and torque effect on the whole item set Bt ( Ramos et al. , 2016 ) . The addition of nt should make Bt a more stable spatial distribution so that more items can be added in the future . Therefore , our packing policy over Lt is defined as π ( Lt|Tt , nt ) , which means probabilities of selecting leaf nodes from Lt given Tt and nt . For online packing , we hope to find the best leaf node selection policy to expand the PCT with more relaxed constraints so that more future items can be appended . Leaf Node Expansion Schemes The performance of online 3D-BPP policies has a strong relationship with the choice of leaf node expansion schemes –– which incrementally calculate new candidate placements introduced by the just placed item nt . A good expansion scheme should reduce the number of solutions to be explored while not missing too many feasible packings . Meanwhile , polynomially computability is also expected . Designing such a scheme from scratch is non-trivial . Fortunately , several placement rules independent from particular packing problems have been proposed , such as Corner Point ( Martello et al. , 2000 ) , Extreme Point ( Crainic et al. , 2008 ) , and Empty Maximal Space ( Ha et al. , 2017 ) . We extend these schemes which have proven to be accurate and efficient to our PCT expansion . The performance of learned policies will be reported in Section 4.1 .

Summary: This paper addresses the problem of online 3D bin packing where the order of objects is out of the model's control and it must make placement decisions one object at a time. Training is framed as a deep RL problem closely following recent work [Zhao et al]. The main contribution is a rethinking of the state and action space which yields much better performance. Prior work trained models to predict placement in a voxelized grid that led to a large action space that could not scale well. Instead, this work represents existing objects and potential placement locations as nodes in a tree which are processed with a graph network. Given a new object, placement nodes can be heuristically instantiated to cover valid locations, and the model uses attention to output a distribution over these nodes and determine the object placement.