Improving Learning to Branch via Reinforcement Learning

1 INTRODUCTION . Mixed Integer Programming ( MIP ) has been applied widely in many real-world problems , such as scheduling ( Barnhart et al. , 2003 ) and transportation ( Melo & Wolsey , 2012 ) . Branch and Bound ( B & B ) is a general and widely used paradigm for solving MIP problems ( Wolsey & Nemhauser , 1999 ) . B & B recursively partitions the solution space into a search tree and compute relaxation bounds along the way to prune subtrees that provably can not contain an optimal solution . This iterative process requires sequential decision makings : node selection : selecting the next solution space to evaluate , variable selection : selecting the variable by which to partition the solution space ( Achterberg & Berthold , 2009 ) . In this work , we focus on learning a variable selection strategy , which is the core of the B & B algorithm ( Achterberg & Wunderling , 2013 ) . Very often , instances from the same MIP problem family are solved repeatedly in industry , which gives rise to the opportunity for learning to improve the variable selection policy ( Bengio et al. , 2020 ) . Based on the human-designed heuristics , Di Liberto et al . ( 2016 ) learn a classifier that dynamically selects an existing rule to perform variable selection ; Balcan et al . ( 2018 ) consider a weighted score of multiple heuristics and analyse the sample complexity of finding such a good weight . The first step towards learning a variable selection policy was taken by Khalil et al . ( 2016 ) , who learn an instance customized policy in an online fashion , as well as Alvarez et al . ( 2017 ) and Hansknecht et al . ( 2018 ) who learn a branching rule offline on a collection of similar instances . Those methods need extensively feature engineering and require strong domain knowledge in MIP . To avoid that , Gasse et al . ( 2019 ) propose a graph convolutional neural network approach to obtain competitive performance , only requiring raw features provided by the solver . In each case , the branching policy is learned by imitating the decision of strong branching as it consistently leads to the smallest B & B trees empirically ( Achterberg et al. , 2005 ) . In this work , we argue that strong branching is not a good expert to imitate . The excellent performance ( the smallest B & B tree ) of strong branching relies mostly on the information obtained in solving branch linear programming ( LP ) rather than the decision it makes . This factor prevents learning a good policy by imitating only the decision made by strong branching . To obtain more effective and non-myopic policies , i.e . minimizing the total solving nodes rather than maximizing the immediate duality gap gap , we use reinforcement learning ( RL ) and model the variable selection process as a Markov Decision Process ( MDP ) . Though the MDP formulation for MIP has been mentioned in the previous works ( Gasse et al. , 2019 ; Etheve et al. , 2020 ) , the advantage of RL has not been demonstrated clearly in literature . The challenges of using RL are multi-fold . First , the state space is a complex search tree , which can involve hundreds or thousands of nodes ( with a linear program on each node ) and evolve over time . In the meanwhile , the objective of MIP is to solve problems faster . Hence a trade-off between decision quality and computation time is required when representing the state and designing a policy based on this state representation . Second , learning a branching policy by RL requires rolling out on a distribution of instances . Moreover , for each instance , the solving trajectory could contain thousands of steps and actions can have long-lasting effects . These result in a large variance in gradient estimation . Third , each step of variable selection can have hundreds of candidates . The large action set makes the exploration in MIP very hard . In this work , we address these challenges by designing a policy network inspired by primal-dual iteration and employing a novelty search evolutionary strategy ( NS-ES ) to improve the policy . For efficiency-effectiveness trade-off , the primal-dual policy ignores the redundant information and makes high-quality decisions on the fly . For reducing variance , the ES algorithm is an attractive choice as its gradient estimation is independent of the trajectory length ( Salimans et al. , 2017 ) . For exploration , we introduce a new representation of the B & B solving process employed by novelty search ( Conti et al. , 2018 ) to encourage visiting new states . We evaluate our RL trained agent over a range of problems ( namely , set covering , maximum independent set , capacitated facility location ) . The experiments show that our approach significantly outperforms stateof-the-art human-designed heuristics ( Achterberg & Berthold , 2009 ) as well as imitation based learning methods ( Khalil et al. , 2016 ; Gasse et al. , 2019 ) . In the ablation study , we compare our primal-dual policy net with GCN ( Gasse et al. , 2019 ) , our novelty based ES with vanilla ES ( Salimans et al. , 2017 ) . The results confirm that both our policy network and the novelty search evolutionary strategy are indispensable for the success of the RL agent . In summary , our main contributions are the followings : • We point out the overestimation of the decision quality of strong branching and suggest that methods other than imitating strong branching are needed to find better variable selection policy . • We model the variable selection process as MDP and design a novel policy net based on primal-dual iteration over reduced LP relaxation . • We introduce a novel set representation and optimal transport distance for the branching process associated with a policy , based on which we train our RL agent using novelty search evolution strategy and obtain substantial improvements in empirical evaluation . 2 BACKGROUND . Mixed Integer Programming . MIP is an optimization problem , which is typically formulated as minx∈Rn { cTx : Ax ≤ b , ` ≤ x ≤ u , xj ∈ Z , ∀j ∈ J } ( 1 ) where c ∈ Rn is the objective vector , A ∈ Rm×n is the constraint coefficient matrix , b ∈ Rm is the constraint vector , ` , u ∈ Rn are the variable bounds . The set J ⊆ { 1 , · · · , n } is an index set for integer variables . We denote the feasible region of x as X . Linear Programming Relaxation . LP relaxation is an important building block for solving MIP problems , where the integer constraints are removed : minx∈Rn { cTx : Ax ≤ b , ` ≤ x ≤ u } . ( 2 ) Algorithm 1 : Branch and Bound Input : A MIP P in form Equation 1 Output : An optimal solution set x∗ and optimal value c∗ 1 Initialize the problem set S : = { PLP } . where PLP is in form Equation 2 . Set x∗ = φ , c∗ =∞ ; 2 If S = φ , exit by returning x∗ and c∗ ; 3 Select and pop a LP relaxation Q ∈ S ; 4 Solve Q with optimal solution x̂ and optimal value ĉ ; 5 If ĉ ≥ c∗ , go to 2 ; 6 If x̂ ∈ X , set x∗ = x̂ , c∗ = ĉ , go to 2 ; 7 Select variable j , split Q into two subproblems Q+j and Q − j , add them to S and go to 3 ; Branch and Bound . LP based B & B is the most successful method in solving MIP . A typical LP based B & B algorithm for solving MIP looks as Algorithm 1 ( Achterberg et al. , 2005 ) . It consists of two major decisions : node selection , in line 3 , and variable selection , in line 7 . In this paper , we will focus on the variable selection . Given a LP relaxation and its optimal solution x̂ , the variable selection means selecting an index j . Then , branching splits the current problem into two subproblems , each representing the original LP relaxation with a new constraint xj ≤ bx̂jc for Q−j and xj ≥ dx̂je for Q + j respectively . This procedure can be visualized by a binary tree , which is commonly called search tree . We give a simple visualization in Section A.1 . Evolution Strategy . Evolution Strategies ( ES ) is a class of black box optimization algorithm ( Rechenberg , 1978 ) . In this work , we refer to the definition in Natural Evolution Strategies ( NES ) ( Wierstra et al. , 2008 ) . NES represents the population as a distribution of parameter vectors θ characterized by parameters φ : pφ ( θ ) . NES optimizes φ to maximize the expectation of a fitness f ( θ ) over the population Eθ∼pφ [ f ( θ ) ] . In recent work , Salimans et al . ( 2017 ) outlines a version of NES applied to standard RL benchmark problems , where θ parameterizes the policy πθ , φt = ( θt , σ ) parameterizes a Gaussian distribution pφ ( θ ) = N ( θt , σ2I ) and f ( θ ) is the cumulative reward R ( θ ) over a full agent interaction . At every iteration , Salimans et al . ( 2017 ) apply n additive Gaussian noises to the current parameter and update the population as θt+1 = θt + α 1 nσ n∑ i=1 f ( θt + σ i ) i ( 3 ) To encourage exploration , Conti et al . ( 2018 ) propose Novelty Search Evolution Strategy ( NS-ES ) . In NSES , the fitness function f ( θ ) = λN ( θ ) + ( 1−λ ) R ( θ ) is selected as a combination of domain specific novelty score N and cumulative reward R , where λ is the balancing weight . 3 WHY IMITATING STRONG BRANCHING IS NOT GOOD . Strong branching is a human-designed heuristic , which solves all possible branch LPs Q+j , Q − j ahead of branching . As strong branching usually produces the smallest B & B search trees ( Achterberg , 2009 ) , many learning-based variable selection policy are trained by mimicking strong branching ( Gasse et al. , 2019 ; Khalil et al. , 2016 ; Alvarez et al. , 2017 ; Hansknecht et al. , 2018 ) . However , we claim that strong branching is not a good expert : the reason strong branching can produce a small search tree is the reduction obtained in solving branch LP , rather than its decision quality . Specifically , ( i ) Strong branching can check lines 5 , 6 in Algorithm 1 before branching . If the pruning condition is satisfied , strong branching does not need to add the subproblem into the problem set S. ( ii ) Strong branching can strengthen other LP relaxations in the problem set S via domain propagation ( Rodosek et al. , 1999 ) and conflict analysis ( Achterberg , 2007 ) . For example , if strong branching finds x1 ≥ 1 and x2 ≥ 1 can be pruned during solving branch LP , then any other LP relaxations containing x1 ≥ 1 can be strengthened by adding x2 ≤ 0 . These two reductions are the direct consequence of solving branch LP , and they can not be learned by a variable selection policy . ( iii ) Strong branching activates primal heuristics ( Berthold , 2006 ) after solving LPs . To examine the decision quality of strong branching , we employ vanilla full strong branching ( Gamrath et al. , 2020 ) , which takes the same decision as full strong branching , while the side-effect of solving branch LP is switched off . Experiments in Section 5.2 show that vanilla full strong branching has poor decision quality . Hence , imitating strong branching is not a wise choice for learning variable selection policy .

Summary: The paper proposes a model for *variable selection* in *Mixed Integer Programming (MIP)* solvers. While this problem is clearly a sequential decision making task, modeling it as an MDP is challenging. As a result, existing works use other approaches such as ranking or imitation learning. This paper overcomes these challenges by introducing a new problem representation.