Partition-Based Formulations for Mixed-Integer Optimization of Trained ReLU Neural Networks

1 Introduction . Many applications use mixed-integer linear programming ( MILP ) to optimize over trained feedforward ReLU neural networks ( NNs ) [ 5 , 14 , 18 , 23 , 32 , 36 ] . A MILP encoding of a ReLU-NN enables network properties to be rigorously analyzed , e.g. , maximizing a neural acquisition function [ 33 ] or verifying robustness of an output ( often classification ) within a restricted input domain [ 6 ] . MILP encodings of ReLU-NNS have also been used to determine robust perturbation bounds [ 9 ] , compress NNs [ 28 ] , count linear regions [ 27 ] , and find adversarial examples [ 17 ] . The so-called bigM formulation is the main approach for encoding NNs as MILPs in the above references . Optimizing the resulting MILPs remains challenging for large networks , even with state-of-the-art software . Effectively solving a MILP hinges on the strength of its continuous relaxation [ 10 ] ; weak relaxations can render MILPs computationally intractable . For NNs , Anderson et al . [ 3 ] showed that the big-M formulation is not tight and presented formulations for the convex hull ( i.e. , the tightest possible formulation ) of individual nodes . However , these formulations require either an exponential ( w.r.t . inputs ) number of constraints or many additional/auxiliary variables . So , despite its weaker continuous relaxation , the big-M formulation can be computationally advantageous owing to its smaller size . Given these challenges , we present a novel class of MILP formulations for ReLU-NNs . The formulations are hierarchical : their relaxations start at a big-M equivalent and converge to the convex hull . Intermediate formulations can closely approximate the convex hull with many fewer variables/constraints . The formulations are constructed by viewing each ReLU node as a two-part 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . disjunction . Kronqvist et al . [ 21 ] proposed hierarchical relaxations for general disjunctive programs . This work develops a similar hierarchy to construct strong and efficient MILP formulations specific to ReLU-NNs . In particular , we partition the inputs of each node into groups and formulate the convex hull over the resulting groups . With fewer groups than inputs , this approach results in MILPs that are smaller than convex-hull formulations , yet have stronger relaxations than big-M. Three optimization tasks evaluate the new formulations : optimal adversarial examples , robust verification , and ` 1-minimally distorted adversaries . Extensive computation , including with convexhull-based cuts , shows that our formulations outperform the standard big-M approach with 25 % more problems solved within a 1h time limit ( average 2.2X speedups for solved problems ) . Related work . Techniques for obtaining strong relaxations of ReLU-NNs include linear programming [ 16 , 34 , 35 ] , semidefinite programming [ 12 , 25 ] , Lagrangian decomposition [ 7 ] , combined relaxations [ 30 ] , and relaxations over multiple nodes [ 29 ] . Recently , Tjandraatmadja et al . [ 31 ] derived the tightest possible convex relaxation for a single ReLU node by considering its multivariate input space . These relaxation techniques do not exactly represent ReLU-NNs , but rather derive valid bounds for the network in general . These techniques might fail to verify some properties , due to their non-exactness , but they can be much faster than MILP-based techniques . Strong MILP encoding of ReLU-NNs was also studied by Anderson et al . [ 3 ] , and a related dual algorithm was later presented by De Palma et al . [ 13 ] . Our approach is fundamentally different , as it constructs computationally cheaper formulations that approximate the convex hull , and we start by deriving a stronger relaxation instead of strengthening the relaxation via cutting planes . Furthermore , our formulation enables input node dependencies to easily be incorporated . Contributions of this paper . We present a new class of strong , yet compact , MILP formulations for feed-forward ReLU-NNs in Section 3 . Section 3.1 observes how , in conjunction with optimizationbased bound tightening , partitioning input variables can efficiently incorporate dependencies into MILP formulations . Section 3.2 builds on the observations of Kronqvist et al . [ 21 ] to prove the hierarchical nature of the proposed ReLU-specific formulations , with relaxations spanning between big-M and convex-hull formulations . Sections 3.3–3.4 show that formulation tightness depends on the specific choice of variable partitions , and we present efficient partitioning strategies based on both theoretical and computational motivations . The advantages of the new formulations are demonstrated via extensive computational experiments in Section 4 . 2 Background . 2.1 Feed-forward Neural Networks . A feed-forward neural network ( NN ) is a directed acyclic graph with nodes structured into k layers . Layer k receives the outputs of nodes in the preceding layer k − 1 as its inputs ( layer 0 represents inputs to the NN ) . Each node in a layer computes a weighted sum of its inputs ( known as the preactivation ) , and applies an activation function . This work considers the ReLU activation function : y = max ( 0 , wTx + b ) ( 1 ) where x ∈ Rη and y ∈ [ 0 , ∞ ) are , respectively , the inputs and output of a node ( wTx + b is termed the preactivation ) . Parameters w ∈ Rη and b ∈ R are the node weights and bias , respectively . 2.2 ReLU Optimization Formulations . In contrast to the training of NNs ( where parameters w and b are optimized ) , optimization over a NN seeks extreme cases for a trained model . Therefore , model parameters ( w , b ) are fixed , and the inputs/outputs of nodes in the network ( x , y ) are optimization variables instead . Big-M Formulation . The ReLU function ( 1 ) is commonly modeled [ 9 , 23 ] : y ≥ ( wTx + b ) ( 2 ) y ≤ ( wTx + b ) − ( 1− σ ) LB0 ( 3 ) y ≤ σUB0 ( 4 ) where y ∈ [ 0 , ∞ ) is the node output and σ ∈ { 0 , 1 } is a binary variable corresponding to the on-off state of the neuron . The formulation requires the bounds ( big-M coefficients ) LB0 , UB0 ∈ R , which should be as tight as possible , such that ( wTx + b ) ∈ [ LB0 , UB0 ] . Disjunctive Programming [ 4 ] . We observe that ( 1 ) can be modeled as a disjunctive program : [ y = 0 wTx + b ≤ 0 ] ∨ [ y = wTx + b wTx + b ≥ 0 ] ( 5 ) The extended formulation for disjunctive programs introduces auxiliary variables for each disjunction . Instead of directly applying the standard extended formulation , we first define z : = wTx and assume z is bounded . The auxiliary variables za ∈ R and zb ∈ R can then be introduced to model ( 5 ) : wTx = za + zb ( 6 ) za + σb ≤ 0 ( 7 ) zb + ( 1− σ ) b ≥ 0 ( 8 ) y = zb + ( 1− σ ) b ( 9 ) σLBa ≤ za ≤ σUBa ( 10 ) ( 1− σ ) LBb ≤ zb ≤ ( 1− σ ) UBb ( 11 ) where again y ∈ [ 0 , ∞ ) and σ ∈ { 0 , 1 } . Bounds LBa , UBa , LBb , UBb ∈ R are required for this formulation , such that za ∈ σ [ LBa , UBa ] and zb ∈ ( 1−σ ) [ LBb , UBb ] . Note that the inequalities in ( 5 ) may cause the domains of za and zb to differ . The summation in ( 6 ) can be used to eliminate either za or zb ; therefore , in practice , only one auxiliary variable is introduced by formulation ( 6 ) – ( 11 ) . Importance of relaxation strength . MILP is often solved with branch-and-bound , a strategy that bounds the objective function between the best feasible point and its tightest optimal relaxation . The integral search space is explored by “ branching ” until the gap between bounds falls below a given tolerance . A tighter , or stronger , relaxation can reduce this search tree considerably . 3 Disaggregated Disjunctions : Between Big-M and the Convex Hull . Our proposed formulations split the sum z = wTx into partitions : we will show these formulations have tighter continuous relaxations than ( 6 ) – ( 11 ) . In particular , we partition the set { 1 , ... , η } into subsets S1 ∪ S2 ∪ ... ∪ SN = { 1 , ... , η } ; Sn ∩ Sn′ = ∅ ∀n 6= n′ . An auxiliary variable is then introduced for each partition , i.e. , zn = ∑ i∈Sn wixi . Replacing z = w Tx with ∑N n=1 zn , the disjunction ( 5 ) becomes : [ y = 0∑N n=1 zn + b ≤ 0 ] ∨ [ y = ∑N n=1 zn + b y ≥ 0 ] ( 12 ) Assuming now that each zn is bounded , the extended formulation can be expressed using auxiliary variables zan and z b n for each zn . Eliminating z a n via the summation zn = z a n + z b n results in our proposed formulation ( complete derivation in appendix A.1 ) : ∑ n ( ∑ i∈Sn wixi − zbn ) + σb ≤ 0 ( 13 ) ∑ n zbn + ( 1− σ ) b ≥ 0 ( 14 ) y = ∑ n zbn + ( 1− σ ) b ( 15 ) σLBan ≤ ∑ i∈Sn wixi − zbn ≤ σUB a n , ∀n = 1 , ... , N ( 16 ) ( 1− σ ) LBbn ≤ zbn ≤ ( 1− σ ) UB b n , ∀n = 1 , ... , N ( 17 ) where y ∈ [ 0 , ∞ ) and σ ∈ { 0 , 1 } . We observe that ( 13 ) – ( 17 ) exactly represents the ReLU node ( given that the partitions zn are bounded in the original disjunction ) : the sum zn = ∑ i∈Sn wixi is substituted in the extended convex hull formulation [ 4 ] of disjunction ( 12 ) . Note that , compared to the general case presented by Kronqvist et al . [ 21 ] , both sides of disjunction ( 12 ) can be modeled using the same partitions , resulting in fewer auxiliary variables . We further note that domains [ LBan , UB a n ] and [ LBbn , UB b n ] may not be equivalent , owing to inequalities in ( 12 ) . 3.1 Obtaining and Tightening Bounds . The big-M formulation ( 2 ) – ( 4 ) requires valid bounds ( wTx + b ) ∈ [ LB0 , UB0 ] . Given bounds for each input variable , xi ∈ [ x¯ i , x̄i ] , interval arithmetic gives valid bounds : LB0 = ∑ i ( x ¯ i max ( 0 , wi ) + x̄imin ( 0 , wi ) ) + b ( 18 ) UB0 = ∑ i ( x̄imax ( 0 , wi ) + x¯ i min ( 0 , wi ) ) + b ( 19 ) But ( 18 ) – ( 19 ) do not provide the tightest valid bounds in general , as dependencies between the input nodes are ignored . Propagating the resulting over-approximated bounds from layer to layer leads to increasingly large over-approximations , i.e. , propagating weak bounds through layers results in a significantly weakened model . These bounds remain in the proposed formulation ( 13 ) – ( 17 ) in the form of bounds on both the output y and the original variables x ( i.e. , outputs of the previous layer ) . Optimization-Based Bound Tightening ( OBBT ) or progressive bounds tightening [ 32 ] , tightens variable bounds and constraints [ 15 ] . For example , solving the optimization problem with the objective set to minimize/maximize ( wTx + b ) gives bounds : min ( wTx + b ) ≤ wTx + b ≤ max ( wTx + b ) . To simplify these problems , OBBT can be performed using the relaxed model ( i.e. , σ ∈ [ 0 , 1 ] rather than σ ∈ { 0 , 1 } ) , resulting in a linear program ( LP ) . In contrast to ( 18 ) – ( 19 ) , bounds from OBBT incorporate variable dependencies . We apply OBBT by solving one LP per bound . The partitioned formulation ( 13 ) – ( 17 ) requires bounds such that zan ∈ σ [ LB a n , UB a n ] , z b n ∈ ( 1 − σ ) [ LBbn , UB b n ] . In other words , [ LB a n , UB a n ] is a valid domain for z a n when the node is inactive ( σ = 1 ) and vice versa . These bounds can also be obtained via OBBT : min ( ∑ i∈Sn wixi ) ≤ z a n ≤ max ( ∑ i∈Sn wixi ) ; w Tx + b ≤ 0 . The constraint on the right-hand side of disjunction ( 12 ) can be similarly enforced in OBBT problems for zbn . In our framework , OBBT additionally captures dependencies within each partition Sn . Specifically , we observe that the partitioned OBBT problems effectively form the convex hull over a given polyhedron of the input variables , in contrast to the convex hull formulation , which only considers the box domain defined by the min/max of each input node [ 3 ] . Since ∑ i∈Sn wixi = z a n + z b n and z a nz b n = 0 , the bounds [ min ( ∑ i∈Sn wixi ) , max ( ∑ i∈Sn wixi ) ] are valid for both z a n and z b n. These bounds can be from ( 18 ) – ( 19 ) , or by solving two OBBT problems for each partition ( 2N LPs total ) . This simplification uses equivalent bounds for zan and z b n , but tighter bounds can potentially be found by performing OBBT for z a n and z b n separately ( 4N LPs total ) . Figure 1 compares the proposed formulation with bounds from interval arithmetic ( top row ) vs OBBT ( bottom row ) . The true model outputs ( blue ) , and minimum ( orange ) and maximum ( green ) outputs of the relaxed model are shown over the input domain . The NNs comprise two inputs , three hidden layers with 20 nodes each , and one output . As expected , OBBT greatly improves relaxation tightness .

Summary: This paper presents a family of mixed-integer optimization formulations for trained neural networks (or, rather, individual neurons in the network), with primary application to verification and adversarial analysis/training. The family traces out a spectrum between a small, weak formulation (the big-M) and a larger, stronger formulation (the "convex hull"). The family is parameterized by a partition of the multivariate input to a single ReLU neuron. A handful of strategies for doing this partition are presented, along with computational experiments.