Solving Packing Problems by Conditional Query Learning

1 INTRODUCTION . How to pack boxes with the smallest bin size ? With the development of globalization and ECommerce , this question becomes more and more important . Boxes are packed to various bins such as shipping container and boxcar . Many of these boxes are made by paper or plastic ; packing boxes in a more efficient way can greatly reduce material cost or shipping energy . The Bin Packing Problem ( BPP ) is one of the classic integer combinatorial optimization problems and it has been extensively studied for decades ( Wu et al. , 2010 ) . It is shown that BPP is a strongly NP-hard problem ( Martello et al. , 2000 ) , which requires exponential time to generate the optimal solution . Some heuristic algorithms ( Baltacioglu et al. , 2006 ) try to obtain a nearly optimal solution within polynomial time , but these methods require explicit rules for every specific problem setting . When the setting changes even very slightly , the original method can not work properly . On contrast , the explicit or hand-crafted rules can be interpreted as policies by neural networks to make decisions , which are insensitive to problem settings . Neural networks have achieved great success in many domains , such as computer vision ( Liu et al. , 2017 ) , natural language processing ( Vaswani et al. , 2017 ) , speech recognition ( Chan et al. , 2016 ) , etc . Inspired by these booming techniques , many studies ( Bello et al. , 2016 ) adopt the neural networks and the recent advances in artificial intelligence to solve the classic combinatorial optimization problems , such as the Travelling Salesman Problem ( TSP ) , the Vehicle Routing Problem ( VRP ) , etc . Some latest works propose the pointer network ( Vinyals et al. , 2015b ) and utilize the attention mechanism with reinforcement learning ( Nazari et al. , 2018 ; Kool et al. , 2019 ) to solve the TSP and the routing problems respectively . There are also some learning-based attempts for the packing problem , which utilize reinforcement learning in the neural network model since the optimal solution is unknown beforehand . For example , MTSL ( Duan et al. , 2019 ) separates selecting and rotating step by selected learning , and applies the transitional greedy method to perform final positioning step . Laterre et al . ( 2018 ) enlighten by AlphaZero ( Silver et al. , 2017 ) adopt Monte Carlo Tree Search ( MCTS ) with self-competition reinforcement learning to solve the packing problem , but it is restrained to pack boxes that have been preliminarily divided from a bin . Cai et al . ( 2019 ) simply use reinforcement learning to get some packing results , which serve as the initialization to accelerate the original heuristic algorithms . However , these approaches either miss the connection between sub-actions or combine handicraft rules with a learning algorithm . Without the sub-action connection , the learning process becomes partial observable Markov Decision Process ( MDP ) ( Sutton & Barto , 2018 ) , which is hard to generalize a good policy for the lack of information . Some methods generate all possible sub-action sequences at the same time , it is still non-trivial for a neural network model to produce many mutual related outputs in a single forward prorogation even though the setting of these methods are strict MDP . These methods combining handicraft rules are not only difficult to achieve an optimal solution , but also sensitive to problem settings . To the best of our knowledge , there is no end-to-end learning algorithm that solves standard packing problems . In this paper , we propose a Conditional Query Learning ( CQL ) model that directly addresses the gap between sub-actions . With the inner conditional query mechanism , the learning process becomes fully observable MDP , which makes the problem easier to apply the reinforcement learning . Compared with the model that outputs several sub-actions simultaneously , CQL model has smaller action space per forward step . Benefit from the small action space feature , we can apply a simpler model to fit the policy , which work more efficiently . In addition , it would not sacrifice performance with the small action space . As a result , we do not require handicraft rules to feed the gaps between sub-actions since they are provided by conditional queries . Specifically , the packing problem requires three mutual conditioned sub-actions : box selecting , rotating and positioning . To fill the gap between sub-actions , we adopt the CQL model as follows . First of all , the packing problem is formulated as an MDP to apply reinforcement learning . Then the previous sub-actions are embedded as a query to the next sub-action decoder . After all three sub-actions are generated , the environment performs one packing step and updates the observation . Finally , we adopt the actor-critic algorithm to update the model parameters . We conduct extensive experiments to evaluate the models and the results show that the CQL model greatly outperforms the vanilla model which produces sub-actions in one step forward propagation without query . In addition , the CQL model achieves lower bin gap ratio in both 2D and 3D packing compared with extant learning approaches and heuristic algorithms . Specifically , our model improves 7.2 % space utilization ratio in 3D packing ( 30 boxes ) compared with genetic algorithm , and reduces more than 10 % bin gap ratio in almost every case compared with the state-of-the-art learning approaches . Furthermore , numerical results show that CQL greatly outperforms other methods when the scale of the problem increases . The learning carve and the variance of results also illustrate that CQL makes the training process more stable . The contributions of this paper are summarized as follows : • We propose the first end-to-end learning algorithm that solves the standard packing problem for both 2D and 3D settings ; • We propose the conditional query learning ( CQL ) model to address the packing problem that has mutual conditioned multi-dimension actions ; • We combine the conditional query learning with an attention mechanism to construct a new learning framework ; • We conduct extensive experiments and the results show that our model outperforms both heuristic algorithms and the state-of-the-art learning-based models . We also release our model implementation and 2D & 3D packing environment for the community to test their algorithms . The rest of the paper is organized as follows . We introduce related works in the next section . We introduce the preliminaries in Section 3 . Design of CQL model is presented in Section 4 , We implement CQL model and illustrate the comparison results in Section 5 . Finally , we conclude this paper in section 6 . 2 RELATED WORKS . Getting an optimal solution of combinatorial optimization problems are computationally heavy , optimal labeled data for supervised learning is expensive . Hence , when using Neural Networks ( NNs ) to solve these problems , one solution is to apply heuristic algorithm results as labeled data , but the performance of this approach can not be better than the performance of the heuristic algorithm . The other solution is to apply reinforcement learning that makes the algorithm learn from its own experience , which is possible to produce better results than existing algorithms . Here we focus on the reinforcement learning approaches and introduce some related works of reinforcement learning and neural combinatorial optimization . 2.1 SEQUENCE PROBLEM IN NEURAL COMBINATORIAL OPTIMIZATION . Enlighten by the recent success of Neural Networks ( NNs ) , especially the fast progress in sequence to sequence model , which is initially used in Neural Machine Translation ( NMT ) ( Bahdanau et al. , 2014 ; Vinyals et al. , 2015a ; Luong et al. , 2015 ; Vaswani et al. , 2017 ; Shen et al. , 2018 ) . Because many combinatorial optimization problems have similar input and output structure as NMT , many studies adopt NNs to solve sequential combinatorial optimization problems . Vinyals et al . ( 2015b ) propose Pointer Networks , which uses attention as a pointer to select a member of the input sequence as the output . Bello et al . ( 2016 ) and ( Nazari et al. , 2018 ) view TSP and VRP as MDP respectively , and they both apply policy gradient algorithm to train their models . Kool et al . ( 2019 ) further improve the result of routing problem using attention model ( Vaswani et al. , 2017 ) . 2.2 REINFORCEMENT LEARNING WITH PARAMETERIZED ACTION SPACE . Different from routing problems that only require one object selected from input sequence per step , the packing problem requires three sub-actions to pack a box into the bin . This kind of action space is called parameterized action space ( Masson et al. , 2016 ) in reinforcement learning , which requires the agent to select multiple types of action every action step . Hausknecht & Stone ( 2015 ) expand DDPG to parameterized action space and test it on RoboCup soccer game . But this approach suffers from tanh saturation problem in continuous action space , so they apply inverse gradients trick to address it . Masson et al . ( 2016 ) introduce Q-PAMDP algorithm , which alternates learning action selection and parameter selection polices to make the training process more stable , but the parameter policy have to output all the parameters for all discrete actions . The output size of the parameter policy can be explosion when the problem has large high dimensional parameters with large discrete actions . The authors of Wei et al . ( 2018 ) propose a hierarchical approach , they condition the parameter policy on the output of discrete action policy , and they apply Variational Auto-Encoders ( VAE ) trick to make the model differentiable . However , all these researches only divide action to two parts , namely , a discrete action and a set of actions as parameters that may include discrete or continuous actions . But the problems like packing contains several mutual conditioned actions that can not directly view as conventional parameterized action space problem , otherwise the number of outputs of the second model will be the product of the candidate output of each sub-actions , which makes the model has too many output options and is hard to generalize the problem and learn to produce a good solution . 2.3 PACKING PROBLEM . As mentioned before , the packing problems are such problem that have mutually conditioned subactions , and there are some works trying to solve this problem by NCO . Enlighten by AlphaZero Silver et al . ( 2017 ) , Laterre et al . ( 2018 ) apply MCTS self play to search the better solution and learn how to pack , but their method only applies to the dataset which boxes are divided by random cuts from a regular bin . Duan et al . ( 2019 ) propose a selected learning approach solving 3D flexible bin packing problem which balances sequence and orientation complexity . They adopt pointer networks with reinforcement learning to get the box selection and rotation results , and apply greedy algorithm to select the location of boxes , which is not end-to-end learning method and can not get better than greedy algorithm . More importantly , this hybrid method do not view the entire packing problem as one complete optimization process , so the learning process only tries to optimize part of problem . At the same time , different algorithms of sub-actions may have conflict goal in optimization process . Different from previous methods , we simply embed the previous actions as an attention query for the model to reason the subsequent actions and after the full action step is finished , we perform one-step optimization along every sub-action model . In this way , we reduce the total output size to the sum of all sub-action candidate output , which makes the model smaller and easier to learn .

Summary: This paper proposes an end-to-end deep reinforcement learning-based algorithm for the 2D and 3D bin packing problems. Its main contribution is conditional query learning (CQL) which allows effective decision over mutually conditioned action spaces through policy expressed as a sequence of conditional distributions. Efficient neural architectures for modeling of such a policy is proposed. Experiments validate the effectiveness of the algorithm through comparisons with genetic algorithm and vanilla RL baselines.