Partition-Based Formulations for Mixed-Integer Optimization of Trained ReLU Neural Networks

1 Introduction . Many applications use mixed-integer linear programming ( MILP ) to optimize over trained feedforward ReLU neural networks ( NNs ) [ 5 , 14 , 18 , 23 , 32 , 36 ] . A MILP encoding of a ReLU-NN enables network properties to be rigorously analyzed , e.g. , maximizing a neural acquisition function [ 33 ] or verifying robustness of an output ( often classification ) within a restricted input domain [ 6 ] . MILP encodings of ReLU-NNS have also been used to determine robust perturbation bounds [ 9 ] , compress NNs [ 28 ] , count linear regions [ 27 ] , and find adversarial examples [ 17 ] . The so-called bigM formulation is the main approach for encoding NNs as MILPs in the above references . Optimizing the resulting MILPs remains challenging for large networks , even with state-of-the-art software . Effectively solving a MILP hinges on the strength of its continuous relaxation [ 10 ] ; weak relaxations can render MILPs computationally intractable . For NNs , Anderson et al . [ 3 ] showed that the big-M formulation is not tight and presented formulations for the convex hull ( i.e. , the tightest possible formulation ) of individual nodes . However , these formulations require either an exponential ( w.r.t . inputs ) number of constraints or many additional/auxiliary variables . So , despite its weaker continuous relaxation , the big-M formulation can be computationally advantageous owing to its smaller size . Given these challenges , we present a novel class of MILP formulations for ReLU-NNs . The formulations are hierarchical : their relaxations start at a big-M equivalent and converge to the convex hull . Intermediate formulations can closely approximate the convex hull with many fewer variables/constraints . The formulations are constructed by viewing each ReLU node as a two-part 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . disjunction . Kronqvist et al . [ 21 ] proposed hierarchical relaxations for general disjunctive programs . This work develops a similar hierarchy to construct strong and efficient MILP formulations specific to ReLU-NNs . In particular , we partition the inputs of each node into groups and formulate the convex hull over the resulting groups . With fewer groups than inputs , this approach results in MILPs that are smaller than convex-hull formulations , yet have stronger relaxations than big-M. Three optimization tasks evaluate the new formulations : optimal adversarial examples , robust verification , and ` 1-minimally distorted adversaries . Extensive computation , including with convexhull-based cuts , shows that our formulations outperform the standard big-M approach with 25 % more problems solved within a 1h time limit ( average 2.2X speedups for solved problems ) . Related work . Techniques for obtaining strong relaxations of ReLU-NNs include linear programming [ 16 , 34 , 35 ] , semidefinite programming [ 12 , 25 ] , Lagrangian decomposition [ 7 ] , combined relaxations [ 30 ] , and relaxations over multiple nodes [ 29 ] . Recently , Tjandraatmadja et al . [ 31 ] derived the tightest possible convex relaxation for a single ReLU node by considering its multivariate input space . These relaxation techniques do not exactly represent ReLU-NNs , but rather derive valid bounds for the network in general . These techniques might fail to verify some properties , due to their non-exactness , but they can be much faster than MILP-based techniques . Strong MILP encoding of ReLU-NNs was also studied by Anderson et al . [ 3 ] , and a related dual algorithm was later presented by De Palma et al . [ 13 ] . Our approach is fundamentally different , as it constructs computationally cheaper formulations that approximate the convex hull , and we start by deriving a stronger relaxation instead of strengthening the relaxation via cutting planes . Furthermore , our formulation enables input node dependencies to easily be incorporated . Contributions of this paper . We present a new class of strong , yet compact , MILP formulations for feed-forward ReLU-NNs in Section 3 . Section 3.1 observes how , in conjunction with optimizationbased bound tightening , partitioning input variables can efficiently incorporate dependencies into MILP formulations . Section 3.2 builds on the observations of Kronqvist et al . [ 21 ] to prove the hierarchical nature of the proposed ReLU-specific formulations , with relaxations spanning between big-M and convex-hull formulations . Sections 3.3–3.4 show that formulation tightness depends on the specific choice of variable partitions , and we present efficient partitioning strategies based on both theoretical and computational motivations . The advantages of the new formulations are demonstrated via extensive computational experiments in Section 4 . 2 Background . 2.1 Feed-forward Neural Networks . A feed-forward neural network ( NN ) is a directed acyclic graph with nodes structured into k layers . Layer k receives the outputs of nodes in the preceding layer k − 1 as its inputs ( layer 0 represents inputs to the NN ) . Each node in a layer computes a weighted sum of its inputs ( known as the preactivation ) , and applies an activation function . This work considers the ReLU activation function : y = max ( 0 , wTx + b ) ( 1 ) where x ∈ Rη and y ∈ [ 0 , ∞ ) are , respectively , the inputs and output of a node ( wTx + b is termed the preactivation ) . Parameters w ∈ Rη and b ∈ R are the node weights and bias , respectively . 2.2 ReLU Optimization Formulations . In contrast to the training of NNs ( where parameters w and b are optimized ) , optimization over a NN seeks extreme cases for a trained model . Therefore , model parameters ( w , b ) are fixed , and the inputs/outputs of nodes in the network ( x , y ) are optimization variables instead . Big-M Formulation . The ReLU function ( 1 ) is commonly modeled [ 9 , 23 ] : y ≥ ( wTx + b ) ( 2 ) y ≤ ( wTx + b ) − ( 1− σ ) LB0 ( 3 ) y ≤ σUB0 ( 4 ) where y ∈ [ 0 , ∞ ) is the node output and σ ∈ { 0 , 1 } is a binary variable corresponding to the on-off state of the neuron . The formulation requires the bounds ( big-M coefficients ) LB0 , UB0 ∈ R , which should be as tight as possible , such that ( wTx + b ) ∈ [ LB0 , UB0 ] . Disjunctive Programming [ 4 ] . We observe that ( 1 ) can be modeled as a disjunctive program : [ y = 0 wTx + b ≤ 0 ] ∨ [ y = wTx + b wTx + b ≥ 0 ] ( 5 ) The extended formulation for disjunctive programs introduces auxiliary variables for each disjunction . Instead of directly applying the standard extended formulation , we first define z : = wTx and assume z is bounded . The auxiliary variables za ∈ R and zb ∈ R can then be introduced to model ( 5 ) : wTx = za + zb ( 6 ) za + σb ≤ 0 ( 7 ) zb + ( 1− σ ) b ≥ 0 ( 8 ) y = zb + ( 1− σ ) b ( 9 ) σLBa ≤ za ≤ σUBa ( 10 ) ( 1− σ ) LBb ≤ zb ≤ ( 1− σ ) UBb ( 11 ) where again y ∈ [ 0 , ∞ ) and σ ∈ { 0 , 1 } . Bounds LBa , UBa , LBb , UBb ∈ R are required for this formulation , such that za ∈ σ [ LBa , UBa ] and zb ∈ ( 1−σ ) [ LBb , UBb ] . Note that the inequalities in ( 5 ) may cause the domains of za and zb to differ . The summation in ( 6 ) can be used to eliminate either za or zb ; therefore , in practice , only one auxiliary variable is introduced by formulation ( 6 ) – ( 11 ) . Importance of relaxation strength . MILP is often solved with branch-and-bound , a strategy that bounds the objective function between the best feasible point and its tightest optimal relaxation . The integral search space is explored by “ branching ” until the gap between bounds falls below a given tolerance . A tighter , or stronger , relaxation can reduce this search tree considerably . 3 Disaggregated Disjunctions : Between Big-M and the Convex Hull . Our proposed formulations split the sum z = wTx into partitions : we will show these formulations have tighter continuous relaxations than ( 6 ) – ( 11 ) . In particular , we partition the set { 1 , ... , η } into subsets S1 ∪ S2 ∪ ... ∪ SN = { 1 , ... , η } ; Sn ∩ Sn′ = ∅ ∀n 6= n′ . An auxiliary variable is then introduced for each partition , i.e. , zn = ∑ i∈Sn wixi . Replacing z = w Tx with ∑N n=1 zn , the disjunction ( 5 ) becomes : [ y = 0∑N n=1 zn + b ≤ 0 ] ∨ [ y = ∑N n=1 zn + b y ≥ 0 ] ( 12 ) Assuming now that each zn is bounded , the extended formulation can be expressed using auxiliary variables zan and z b n for each zn . Eliminating z a n via the summation zn = z a n + z b n results in our proposed formulation ( complete derivation in appendix A.1 ) : ∑ n ( ∑ i∈Sn wixi − zbn ) + σb ≤ 0 ( 13 ) ∑ n zbn + ( 1− σ ) b ≥ 0 ( 14 ) y = ∑ n zbn + ( 1− σ ) b ( 15 ) σLBan ≤ ∑ i∈Sn wixi − zbn ≤ σUB a n , ∀n = 1 , ... , N ( 16 ) ( 1− σ ) LBbn ≤ zbn ≤ ( 1− σ ) UB b n , ∀n = 1 , ... , N ( 17 ) where y ∈ [ 0 , ∞ ) and σ ∈ { 0 , 1 } . We observe that ( 13 ) – ( 17 ) exactly represents the ReLU node ( given that the partitions zn are bounded in the original disjunction ) : the sum zn = ∑ i∈Sn wixi is substituted in the extended convex hull formulation [ 4 ] of disjunction ( 12 ) . Note that , compared to the general case presented by Kronqvist et al . [ 21 ] , both sides of disjunction ( 12 ) can be modeled using the same partitions , resulting in fewer auxiliary variables . We further note that domains [ LBan , UB a n ] and [ LBbn , UB b n ] may not be equivalent , owing to inequalities in ( 12 ) . 3.1 Obtaining and Tightening Bounds . The big-M formulation ( 2 ) – ( 4 ) requires valid bounds ( wTx + b ) ∈ [ LB0 , UB0 ] . Given bounds for each input variable , xi ∈ [ x¯ i , x̄i ] , interval arithmetic gives valid bounds : LB0 = ∑ i ( x ¯ i max ( 0 , wi ) + x̄imin ( 0 , wi ) ) + b ( 18 ) UB0 = ∑ i ( x̄imax ( 0 , wi ) + x¯ i min ( 0 , wi ) ) + b ( 19 ) But ( 18 ) – ( 19 ) do not provide the tightest valid bounds in general , as dependencies between the input nodes are ignored . Propagating the resulting over-approximated bounds from layer to layer leads to increasingly large over-approximations , i.e. , propagating weak bounds through layers results in a significantly weakened model . These bounds remain in the proposed formulation ( 13 ) – ( 17 ) in the form of bounds on both the output y and the original variables x ( i.e. , outputs of the previous layer ) . Optimization-Based Bound Tightening ( OBBT ) or progressive bounds tightening [ 32 ] , tightens variable bounds and constraints [ 15 ] . For example , solving the optimization problem with the objective set to minimize/maximize ( wTx + b ) gives bounds : min ( wTx + b ) ≤ wTx + b ≤ max ( wTx + b ) . To simplify these problems , OBBT can be performed using the relaxed model ( i.e. , σ ∈ [ 0 , 1 ] rather than σ ∈ { 0 , 1 } ) , resulting in a linear program ( LP ) . In contrast to ( 18 ) – ( 19 ) , bounds from OBBT incorporate variable dependencies . We apply OBBT by solving one LP per bound . The partitioned formulation ( 13 ) – ( 17 ) requires bounds such that zan ∈ σ [ LB a n , UB a n ] , z b n ∈ ( 1 − σ ) [ LBbn , UB b n ] . In other words , [ LB a n , UB a n ] is a valid domain for z a n when the node is inactive ( σ = 1 ) and vice versa . These bounds can also be obtained via OBBT : min ( ∑ i∈Sn wixi ) ≤ z a n ≤ max ( ∑ i∈Sn wixi ) ; w Tx + b ≤ 0 . The constraint on the right-hand side of disjunction ( 12 ) can be similarly enforced in OBBT problems for zbn . In our framework , OBBT additionally captures dependencies within each partition Sn . Specifically , we observe that the partitioned OBBT problems effectively form the convex hull over a given polyhedron of the input variables , in contrast to the convex hull formulation , which only considers the box domain defined by the min/max of each input node [ 3 ] . Since ∑ i∈Sn wixi = z a n + z b n and z a nz b n = 0 , the bounds [ min ( ∑ i∈Sn wixi ) , max ( ∑ i∈Sn wixi ) ] are valid for both z a n and z b n. These bounds can be from ( 18 ) – ( 19 ) , or by solving two OBBT problems for each partition ( 2N LPs total ) . This simplification uses equivalent bounds for zan and z b n , but tighter bounds can potentially be found by performing OBBT for z a n and z b n separately ( 4N LPs total ) . Figure 1 compares the proposed formulation with bounds from interval arithmetic ( top row ) vs OBBT ( bottom row ) . The true model outputs ( blue ) , and minimum ( orange ) and maximum ( green ) outputs of the relaxed model are shown over the input domain . The NNs comprise two inputs , three hidden layers with 20 nodes each , and one output . As expected , OBBT greatly improves relaxation tightness .

Summary: The paper introduces a class of novel mixed-integer programming formulations for optimizing the output of a trained ReLU network. Its approach is to model a ReLU as a disjunction where the preactivation function is partitioned in its indices. This results a parameterized formulation that can range from the traditional big-M formulation to an ideal formulation at its extremes. Since the former is weak but light and the latter is strong but large, this allows us to find a trade-off where the formulation is both reasonably strong and not too large so that its performance is better than either extreme. Moreover, the paper presents an optimization-based bounds tightening approach that can be applied to these formulation. The authors show that this formulation has good computational performance compared to its extremes for three classes of problems in the area of neural network verification.