Solving Packing Problems by Conditional Query Learning

1 INTRODUCTION . How to pack boxes with the smallest bin size ? With the development of globalization and ECommerce , this question becomes more and more important . Boxes are packed to various bins such as shipping container and boxcar . Many of these boxes are made by paper or plastic ; packing boxes in a more efficient way can greatly reduce material cost or shipping energy . The Bin Packing Problem ( BPP ) is one of the classic integer combinatorial optimization problems and it has been extensively studied for decades ( Wu et al. , 2010 ) . It is shown that BPP is a strongly NP-hard problem ( Martello et al. , 2000 ) , which requires exponential time to generate the optimal solution . Some heuristic algorithms ( Baltacioglu et al. , 2006 ) try to obtain a nearly optimal solution within polynomial time , but these methods require explicit rules for every specific problem setting . When the setting changes even very slightly , the original method can not work properly . On contrast , the explicit or hand-crafted rules can be interpreted as policies by neural networks to make decisions , which are insensitive to problem settings . Neural networks have achieved great success in many domains , such as computer vision ( Liu et al. , 2017 ) , natural language processing ( Vaswani et al. , 2017 ) , speech recognition ( Chan et al. , 2016 ) , etc . Inspired by these booming techniques , many studies ( Bello et al. , 2016 ) adopt the neural networks and the recent advances in artificial intelligence to solve the classic combinatorial optimization problems , such as the Travelling Salesman Problem ( TSP ) , the Vehicle Routing Problem ( VRP ) , etc . Some latest works propose the pointer network ( Vinyals et al. , 2015b ) and utilize the attention mechanism with reinforcement learning ( Nazari et al. , 2018 ; Kool et al. , 2019 ) to solve the TSP and the routing problems respectively . There are also some learning-based attempts for the packing problem , which utilize reinforcement learning in the neural network model since the optimal solution is unknown beforehand . For example , MTSL ( Duan et al. , 2019 ) separates selecting and rotating step by selected learning , and applies the transitional greedy method to perform final positioning step . Laterre et al . ( 2018 ) enlighten by AlphaZero ( Silver et al. , 2017 ) adopt Monte Carlo Tree Search ( MCTS ) with self-competition reinforcement learning to solve the packing problem , but it is restrained to pack boxes that have been preliminarily divided from a bin . Cai et al . ( 2019 ) simply use reinforcement learning to get some packing results , which serve as the initialization to accelerate the original heuristic algorithms . However , these approaches either miss the connection between sub-actions or combine handicraft rules with a learning algorithm . Without the sub-action connection , the learning process becomes partial observable Markov Decision Process ( MDP ) ( Sutton & Barto , 2018 ) , which is hard to generalize a good policy for the lack of information . Some methods generate all possible sub-action sequences at the same time , it is still non-trivial for a neural network model to produce many mutual related outputs in a single forward prorogation even though the setting of these methods are strict MDP . These methods combining handicraft rules are not only difficult to achieve an optimal solution , but also sensitive to problem settings . To the best of our knowledge , there is no end-to-end learning algorithm that solves standard packing problems . In this paper , we propose a Conditional Query Learning ( CQL ) model that directly addresses the gap between sub-actions . With the inner conditional query mechanism , the learning process becomes fully observable MDP , which makes the problem easier to apply the reinforcement learning . Compared with the model that outputs several sub-actions simultaneously , CQL model has smaller action space per forward step . Benefit from the small action space feature , we can apply a simpler model to fit the policy , which work more efficiently . In addition , it would not sacrifice performance with the small action space . As a result , we do not require handicraft rules to feed the gaps between sub-actions since they are provided by conditional queries . Specifically , the packing problem requires three mutual conditioned sub-actions : box selecting , rotating and positioning . To fill the gap between sub-actions , we adopt the CQL model as follows . First of all , the packing problem is formulated as an MDP to apply reinforcement learning . Then the previous sub-actions are embedded as a query to the next sub-action decoder . After all three sub-actions are generated , the environment performs one packing step and updates the observation . Finally , we adopt the actor-critic algorithm to update the model parameters . We conduct extensive experiments to evaluate the models and the results show that the CQL model greatly outperforms the vanilla model which produces sub-actions in one step forward propagation without query . In addition , the CQL model achieves lower bin gap ratio in both 2D and 3D packing compared with extant learning approaches and heuristic algorithms . Specifically , our model improves 7.2 % space utilization ratio in 3D packing ( 30 boxes ) compared with genetic algorithm , and reduces more than 10 % bin gap ratio in almost every case compared with the state-of-the-art learning approaches . Furthermore , numerical results show that CQL greatly outperforms other methods when the scale of the problem increases . The learning carve and the variance of results also illustrate that CQL makes the training process more stable . The contributions of this paper are summarized as follows : • We propose the first end-to-end learning algorithm that solves the standard packing problem for both 2D and 3D settings ; • We propose the conditional query learning ( CQL ) model to address the packing problem that has mutual conditioned multi-dimension actions ; • We combine the conditional query learning with an attention mechanism to construct a new learning framework ; • We conduct extensive experiments and the results show that our model outperforms both heuristic algorithms and the state-of-the-art learning-based models . We also release our model implementation and 2D & 3D packing environment for the community to test their algorithms . The rest of the paper is organized as follows . We introduce related works in the next section . We introduce the preliminaries in Section 3 . Design of CQL model is presented in Section 4 , We implement CQL model and illustrate the comparison results in Section 5 . Finally , we conclude this paper in section 6 . 2 RELATED WORKS . Getting an optimal solution of combinatorial optimization problems are computationally heavy , optimal labeled data for supervised learning is expensive . Hence , when using Neural Networks ( NNs ) to solve these problems , one solution is to apply heuristic algorithm results as labeled data , but the performance of this approach can not be better than the performance of the heuristic algorithm . The other solution is to apply reinforcement learning that makes the algorithm learn from its own experience , which is possible to produce better results than existing algorithms . Here we focus on the reinforcement learning approaches and introduce some related works of reinforcement learning and neural combinatorial optimization . 2.1 SEQUENCE PROBLEM IN NEURAL COMBINATORIAL OPTIMIZATION . Enlighten by the recent success of Neural Networks ( NNs ) , especially the fast progress in sequence to sequence model , which is initially used in Neural Machine Translation ( NMT ) ( Bahdanau et al. , 2014 ; Vinyals et al. , 2015a ; Luong et al. , 2015 ; Vaswani et al. , 2017 ; Shen et al. , 2018 ) . Because many combinatorial optimization problems have similar input and output structure as NMT , many studies adopt NNs to solve sequential combinatorial optimization problems . Vinyals et al . ( 2015b ) propose Pointer Networks , which uses attention as a pointer to select a member of the input sequence as the output . Bello et al . ( 2016 ) and ( Nazari et al. , 2018 ) view TSP and VRP as MDP respectively , and they both apply policy gradient algorithm to train their models . Kool et al . ( 2019 ) further improve the result of routing problem using attention model ( Vaswani et al. , 2017 ) . 2.2 REINFORCEMENT LEARNING WITH PARAMETERIZED ACTION SPACE . Different from routing problems that only require one object selected from input sequence per step , the packing problem requires three sub-actions to pack a box into the bin . This kind of action space is called parameterized action space ( Masson et al. , 2016 ) in reinforcement learning , which requires the agent to select multiple types of action every action step . Hausknecht & Stone ( 2015 ) expand DDPG to parameterized action space and test it on RoboCup soccer game . But this approach suffers from tanh saturation problem in continuous action space , so they apply inverse gradients trick to address it . Masson et al . ( 2016 ) introduce Q-PAMDP algorithm , which alternates learning action selection and parameter selection polices to make the training process more stable , but the parameter policy have to output all the parameters for all discrete actions . The output size of the parameter policy can be explosion when the problem has large high dimensional parameters with large discrete actions . The authors of Wei et al . ( 2018 ) propose a hierarchical approach , they condition the parameter policy on the output of discrete action policy , and they apply Variational Auto-Encoders ( VAE ) trick to make the model differentiable . However , all these researches only divide action to two parts , namely , a discrete action and a set of actions as parameters that may include discrete or continuous actions . But the problems like packing contains several mutual conditioned actions that can not directly view as conventional parameterized action space problem , otherwise the number of outputs of the second model will be the product of the candidate output of each sub-actions , which makes the model has too many output options and is hard to generalize the problem and learn to produce a good solution . 2.3 PACKING PROBLEM . As mentioned before , the packing problems are such problem that have mutually conditioned subactions , and there are some works trying to solve this problem by NCO . Enlighten by AlphaZero Silver et al . ( 2017 ) , Laterre et al . ( 2018 ) apply MCTS self play to search the better solution and learn how to pack , but their method only applies to the dataset which boxes are divided by random cuts from a regular bin . Duan et al . ( 2019 ) propose a selected learning approach solving 3D flexible bin packing problem which balances sequence and orientation complexity . They adopt pointer networks with reinforcement learning to get the box selection and rotation results , and apply greedy algorithm to select the location of boxes , which is not end-to-end learning method and can not get better than greedy algorithm . More importantly , this hybrid method do not view the entire packing problem as one complete optimization process , so the learning process only tries to optimize part of problem . At the same time , different algorithms of sub-actions may have conflict goal in optimization process . Different from previous methods , we simply embed the previous actions as an attention query for the model to reason the subsequent actions and after the full action step is finished , we perform one-step optimization along every sub-action model . In this way , we reduce the total output size to the sum of all sub-action candidate output , which makes the model smaller and easier to learn .

Summary: This paper aims at solving geometric bin packing (2D or 3D) problems using a deep reinforcement learning framework. Namely, the framework is based on the actor-critic paradigm, and uses a conditional query learning model for performing composite actions (selections, rotations) in geometric bin packing. Experiments are performed on several instances of 2D-BPP and 3D-BPP,