Title: Scalable Neural Network Verification with Branch-and-bound Inferred Cutting Planes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neuron Network Verification; AI Safety; Robustness; Formal Methods
Abstract: Recently, cutting-plane methods such as GCP-CROWN have been explored to enhance neural network verifiers and made significant advancements. However, GCP-CROWN currently relies on ${\it generic}$ cutting planes ("cuts") generated from external mixed integer programming (MIP) solvers. Due to the poor scalability of MIP solvers, large neural networks cannot benefit from these cutting planes. In this paper, we exploit the structure of the neural network verification problem to generate efficient and scalable cutting planes ${\it specific}$ to this problem setting. We propose a novel approach, Branch-and-bound Inferred Cuts with COnstraint Strengthening (BICCOS), that leverages the logical relationships of neurons within verified subproblems in the branch-and-bound search tree, and we introduce cuts that preclude these relationships in other subproblems. We develop a mechanism that assigns influence scores to neurons in each path to allow the strengthening of these cuts. Furthermore, we design a multi-tree search technique to identify more cuts, effectively narrowing the search space and accelerating the BaB algorithm. Our results demonstrate that BICCOS can generate hundreds of useful cuts during the branch-and-bound process and consistently increase the number of verifiable instances compared to other state-of-the-art neural network verifiers on a wide range of benchmarks, including large networks that previous cutting plane methods could not scale to.

Scalable Neural Network Verification with Branch-and-bound Inferred Cutting Planes

Abstract

Recently, cutting-plane methods such as GCP-CROWN have been explored to enhance neural network verifiers and made significant advances. However, GCPCROWNcurrently relies on generic cutting planes ('cuts') generated from external mixed integer programming (MIP) solvers. Due to the poor scalability of MIP solvers, large neural networks cannot benefit from these cutting planes. In this paper, we exploit the structure of the neural network verification problem to generate efficient and scalable cutting planes specific for this problem setting. We propose a novel approach, Branch-and-bound Inferred Cuts with COnstraint Strengthening (BICCOS), which leverages the logical relationships of neurons within verified subproblems in the branch-and-bound search tree, and we introduce cuts that preclude these relationships in other subproblems. We develop a mechanism that assigns influence scores to neurons in each path to allow the strengthening of these cuts. Furthermore, we design a multi-tree search technique to identify more cuts, effectively narrowing the search space and accelerating the BaB algorithm. Our results demonstrate that BICCOS can generate hundreds of useful cuts during the branch-and-bound process and consistently increase the number of verifiable instances compared to other state-of-the-art neural network verifiers on a wide range of benchmarks, including large networks that previous cutting plane methods could not scale to. BICCOS is part of the α,β -CROWN verifier, the VNN-COMP 2024 winner . The code is available at https://github.com/Lemutisme/BICCOS .

1 Introduction

Formal verification of neural networks (NNs) has emerged as a critical challenge in the field of artificial intelligence. In canonical settings, the verification procedure aims to prove certified bounds on the outputs of a neural network given a specification, such as robustness against adversarial perturbations or adherence to safety constraints. As these networks become increasingly complex and are deployed in sensitive domains, rigorously ensuring their safety and reliability is paramount. Recent research progress on neural network verification has enabled safety or performance guarantees in several mission-critical applications [55, 53, 51, 13, 60, 57].

The branch and bound (BaB) procedure has shown great promise as a verification approach [12, 59, 21]. In particular, the commonly used ReLU activation function exhibits a piecewise linear behavior, and this property makes ReLU networks especially suitable for branch and bound techniques. During the branching process, one ReLU neuron is selected and two subproblems are created. Each subproblem refers to one of the two ReLU states (inactive or active) and the bounds of the neurons' activation value are tightened correspondingly. BaB systematically branches ReLU neurons, creates a search tree of subproblems, and prunes away subproblems whose bounds are tight enough to guarantee the verification specifications. Although our work focuses on studying the canonical ReLU settings, branch-and-bound has also demonstrated its power in non-ReLU cases [48]. Despite its success in many tasks, the efficiency and effectiveness of the branch-and-bound procedure heavily depend on the number of subproblems that can be pruned.

38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Recent work has explored the use of cutting plane methods in neural network verification. Cutting planes ('cuts') are additional (linear) constraints in the optimization problem for NN verification to tighten the bound without affecting soundness. By introducing carefully chosen cuts, they can significantly tighten the bounds in each subproblem during BaB, leading to more pruned subproblems and faster verification. However, it is quite challenging to generate effective cutting planes for large-scale NN verification problems. For example, GCP-CROWN [61] relied on strong cutting planes generated by a mixed integer programming (MIP) solver, which includes traditional generic cutting planes such as the Gomory cuts [24], but its effectiveness is limited to only small neural networks. The key to unlocking the full potential of BaB lies in the development of scalable cutting planes that are specific to the NN verification problem and also achieve high effectiveness.

In this paper, we propose a novel methodology, Branch-and-bound Inferred Cuts with COnstraint Strengthening (BICCOS), that can produce effective and scalable cutting planes specifically for the NN verification problem. Our approach leverages the information gathered during the branch-andbound process to generate cuts on the fly. First, we show that by leveraging any verified subproblems (e.g., a subproblem is verified when neurons A and B are both split to active cases), we can deduce cuts that preclude certain combinations of ReLU states (e.g., neurons A and B cannot be both active). This cut may tighten the bounds for subproblems in other parts of the search tree, even when neurons A and B have not yet been split. Second, to make these cuts effective during the regular BaB process, we show that they can be strengthened by reducing the number of branched neurons in a verified subproblem (e.g., we may conclude that setting neuron B to active is sufficient to verify the problem, drop the dependency on A, and remove one variable from the cut). BICCOS can find and strengthen cuts on the fly during the BaB process, adapting to the specific characteristics of each verification instance. Third, we propose a pre-solving strategy called 'multi-tree search' which proactively looks for effective cutting planes in many shallow search trees before the main BaB phase starts. Our main contributions can be summarized as follows:

· We first identify the opportunity of extracting effective cutting planes during the branch-and-bound process in NN verification. These cuts are specific to the NN verification setting and scalable to large NNs that previous state-of-the-art cutting plane methods such as GCP-CROWN with MIP-cuts cannot handle. Our cuts can be plugged into existing BaB-based verifiers to enhance their performance.

· We discuss several novel methods to strengthen the cuts and also find more effective cuts. Strengthening the cuts is essential for finding effective cuts on the fly during the regular BaB process that can tighten the bounds in the remaining domains. Performing a multi-tree search allows us to identify additional cuts before the regular BaB process begins.

· We conduct empirical comparisons against many verifiers on a wide range of benchmarks and consistently outperform state-of-the-art baselines in VNN-COMP [4, 40, 10]. Notably, we can solve the very large models like those for cifar100 and tinyimagenet benchmarks and outperform all state-of-the-art tools, to which GCP-CROWN with MIP-based cuts cannot scale.

2 Background

Notations Bold symbols such as x ( i ) denote vectors; regular symbols such as x ( i ) j indicate scalar components of these vectors. [ N ] represents the set { 1 , . . . , N } , and W ( i ) : ,j is the j -th column of the matrix W ( i ) . A calligraphic font X denotes a set and R n represents the n dimensional real number space.

The NN Verification Problem An L -layer ReLU-based DNN can be formulated as f : x ↦→ x ( L ) , s.t. { x ( i ) = W ( i ) ˆ x ( i -1) + b ( i ) , ˆ x ( i ) = σ ( x ( i ) ) , x (0) = x ; i ∈ [ L ] } , where σ represents the ReLU activation function, with input x =: ˆ x (0) ∈ R d (0) and the neuron network parameters weight matrix W ( i ) ∈ R d ( i ) × d ( i -1) and bias vector b ( i ) ∈ R d ( i ) for each layer i . This model sequentially processes the input x through each layer by computing linear transformations followed by ReLU activations. The scalar values ˆ x ( i ) j and x ( i ) j represent the post-activation and pre-activation values of the j -th neuron in the i -th layer, respectively. We write ˆ f ( i ) j ( x ) and f ( i ) j ( x ) for ˆ x ( i ) j and x ( i ) j , respectively, when they depend on a specific x .

̸

In practical applications, the input x is confined within a perturbation set X , often defined as an ℓ p norm ball. The verification task involves ensuring a specified output property for any x ∈ X . For example, it may be required to verify that the logit corresponding to the true label f i ( x ) is consistently higher than the logit for any other label f j ( x ) , thus ensuring f i ( x ) -f j ( x ) > 0 , ∀ j = i . By

integrating the verification specification as an additional layer with only one output neuron, we define the canonical verification problem as an optimization problem:

where the optimal value f ⋆ ≥ 0 confirms the verifiable property. We typically use the ℓ ∞ norm to define X := { x : ∥ x -x 0 ∥ ∞ ≤ ϵ } , with x 0 as a baseline input. However, extensions to other norms and conditions are also possible [44, 58]. When we compute a lower bound f ⋆ ≤ f ⋆ , we label the problem UNSAT if the property was verified and f ⋆ ≥ 0 , i.e. the problem of finding a concrete input that violates the property is unsatisfiable , or in other words infeasible . If f ⋆ < 0 , it is unclear whether the property might hold or not. We refer to this as unknown .

MIP Formulation and LP Relaxation In the optimization problem (1), a non-negative f ⋆ indicates that the network can be verified. However, due to the non-linearity of ReLU neurons, this problem is non-convex, and ReLU neurons are typically relaxed with linear constraints to obtain a lower bound for f ⋆ . One possible solution is to encode the verification problem using Mixed Integer Programming (MIP), which encodes the entire network architecture, using x ∈ R d ( i ) to denote the pre-activation neurons , ˆ x ∈ R d ( i ) to denote post-activation neurons for each layer, and binary ReLU indicator z ( i ) j ∈ { 0 , 1 } to denote (in)active neuron for each unstable neuron. A lower bound can be computed by letting z ( i ) j ∈ [0 , 1] and therefore relaxing the problem to an LP formulation. There is also an equivalent Planet relaxation [20]. We provide the detailed definition in Appendix A. In practice, this approach is computationally too expensive to be scalable.

Branch-and-bound Instead of solving the expensive LP for each neuron, most existing NN verifiers use cheaper methods such as abstract interpretation [49, 23] or bound propagation [16, 62, 54, 59] due to their efficiency and scalability. However, because of those relaxations, the lower bound for f ⋆ might eventually become too weak to prove f ⋆ ≥ 0 . To overcome this issue, additional constraints need to be added to the optimization, without sacrificing soundness.

The branch-and-bound (BaB) framework, illustrated in Figure 1, is a powerful approach for neural network verification that many state-of-the-art verifiers are based on [12, 54, 16, 29, 21]. BaB systematically tightens the lower bound of f ⋆ by splitting unstable ReLU neurons into two cases: x j ≥ 0 and x j ≤ 0 (branching step), which defines two subproblems with additional constraints. In each subproblem, neuron x j does not need to be relaxed, leading to a tighter lower bound (bounding step). Note that { x ( i ) j ≤ u ( i ) j = 0 } and { x ( i ) j ≥ l ( i ) j = 0 } in the Planet relaxation are equivalent to { z ( i ) j = 0 } and { z ( i ) j = 1 } in the LP relaxation, respectively, see Lemma A.1 in Appendix A. Subproblems with a positive lower bound are successfully verified, and no further splitting is required. The process repeats on subproblems with negative lower bounds until all unstable neurons are split, or all subproblems are verified. If there are still domains with negative bounds after splitting all unstable neurons, a counter-example can be constructed.

quire further branching. General Cutting Planes (GCP) in NN verification A (linear) cutting plane ('cut') is a linear inequality that can be added to a MIP problem, which does not eliminate any feasible integer solutions but will tighten the LP relaxation of this MIP. For more details on cutting plane methods, we refer readers to the literature on integer programming [7, 14]. In the NN verification setting, it may involve variables x ( i ) (pre-activation), ˆ x ( i ) (post-activation) from any layer i , and z ( i ) (binary ReLU indicators) from any unstable neuron. Given N cutting planes in matrix form as [61]:

where H ( i ) , G ( i ) , and Q ( i ) are the matrix coefficients corresponding to the cutting planes. Based on this formulation, one can introduce arbitrary valid cuts to tighten the relaxation. GCP-CROWN [61]

allows us to lower bound (1) with arbitrary linear constraints in (2) using GPU-accelerated bound propagation, without relying on an LP solver.

In [61], the authors propose to find new cutting planes by employing an MIP solver. By encoding (1) as a MIP problem, the MIP solver will identify potentially useful cutting planes. Usually, these cuts would be used by the solver itself to solve the MIP problem. Instead, [61] applied these cuts using GPU-accelerated bound propagation. This approach shows great improvements on many verification problems due to the powerful cuts, but it depends on the ability of the MIP solver to identify relevant cuts. As the networks that are analyzed increase in size, the respective MIP problem increases in complexity, and the MIP solver may not return any cuts before the timeout is reached. In the next chapter, we will describe a novel approach to generate effective and scalable cutting planes.

3 Branch-and-bound Inferred Cuts with Constraint Strengthening (BICCOS)

3.1 Branch-and-bound Inferred Cuts

The first key observation in our algorithm is that the UNSAT subproblems in the BaB search tree include valuable information. If the lower bound of a subproblem leads this subproblem to be UNSAT (e.g., subproblems with green ticks in Fig. 1), it signifies that restricting the neurons along this path to the respective positive/negative regimes allows us to verify the property. A typical BaB algorithm would stop generating further subproblems at this juncture, and continue with splitting only those nodes that have not yet been verified. Crucially, no information from the verified branch is transferred to the unverified domains. However, sometimes, this information can help a lot.

Example. As shown in Fig. 1, assume that after splitting x 1 ≤ 0 and x 3 ≤ 0 , the lower bound of this subproblem is found to be greater than 0 , indicating infeasibility. From this infeasibility, we can infer that the neurons x 1 and x 3 cannot simultaneously be in the inactive regime. To represent this relationship, we use the relaxed ReLU indicator variables z 1 , z 3 ∈ [0 , 1] in the LP formulation and form the inequality z 1 + z 3 ≥ 1 . This inequality ensures that both z 1 and z 3 cannot be 0 simultaneously. If we were to start BaB with a fresh search tree, this constraint has the potential to improve the bounds for all subproblems by tightening the relaxations and excluding infeasible regions.

We propose to encode the information gained from a verified subproblem as a new cutting plane. These cutting planes will be valid globally across the entire verification problem and all generated subproblems. We present the general case of this cutting plane below:

Proposition 3.1. For a verified, or UNSAT, subproblem in a BaB search tree, let Z + and Z -be the set of neurons restricted to the positive and negative regimes respectively. These restrictions were introduced by the BaB process. Then, the BaB inferred cut can be formulated as:

Proof deferred to Appendix B.1. The BaB inferred cut (3) will exclude the specific combination of positive and negative neurons that were proven to be infeasible in the respective branches. An example is shown in Fig. 2a.

Note that while similar cuts were explored in [9], they were not derived from UNSAT problems within the BaB process. In our framework, these cuts can theoretically be incorporated as cutting planes in the form of (2) and work using GCP-CROWN. However, a limitation exists: all elements in our cut are ReLU indicator z , although GCP-CROWN applies general cutting planes during the standard BaB process. It was not originally designed to handle branching decisions on ReLU indicator variables z that may have (partially) been fixed already in previous BaB steps. When cuts are added, they remain part of the GCP-CROWN formulation. However, during the BaB process, some z variables may be fixed to 0 or 1 due to branching, effectively turning them into constants This situation poses a challenge because the original GCP-CROWN formulation presented in [61] does not accommodate constraints involving these fixed z variables, potentially leading to incorrect results. To address this issue, we need to extend GCP-CROWN to handle BaB on the ReLU indicators z , ensuring that the constraints and cuts remain valid even when some z variables are fixed during branching.

Extension of GCP-CROWN To address this limitation, we propose an extended form of bound propagation for BaB inferred cuts that accommodates splitting on z variables. In the original GCPCROWN formulation, I ( i ) represents the set of initially unstable neurons in layer i , for which z cuts

were added due to their instability. However, during the branch-and-bound process, some of these neurons may be split, fixing their corresponding z variables to 0 or 1 , and thus their z variables no longer exist in the original formulation. While updating all existing cuts by fixing these z variables is possible, it is costly since the cuts for each subproblem must be fixed individually, as the neuron splits in each subproblem is different. Our contribution is to handle these split neurons by adding them to the splitting set Z in the new formulation below, without removing or modifying the existing cuts (all subdomains can still share the same set of cuts). This approach allows us to adjust the original bound propagation in [61, Theorem 3.1] to account for the fixed z variables and their influence on the Lagrange dual problem, without altering the existing cuts. Suppose the splitting set for each layer i is Z +( i ) ∪ Z -( i ) := Z ( i ) ⊆ I ( i ) , and the full split set is Z = ⋃ i ∈ [ L -1] Z ( i ) . The modifications to the original GCP-CROWN theorem are highlighted in brown in the following theorem.

Theorem 3.2. [BaB Inferred Cuts Bound Propagation]. Given any BaB split set Z , optimizable parameters 0 ≤ α ( i ) j ≤ 1 and β , µ , τ ≥ 0 , π ( i ) j ⋆ is a function of Q ( i ) : ,j :

Here, ˆ ν ( i ) j , π ( i ) j ⋆ and τ ( i ) j are defined for each initially unstable neuron that has not been split: j ∈ I ( i ) \ Z ( i ) , and h ( i ) j ( β ) is defined for all unstable neurons j ∈ I ( i ) .

Proof in Appendix B.2. The brown-highlighted modifications ensure that the bound propagation correctly accounts for the influence of the fixed ReLU indicator z variables-resulting from branching decisions in the BaB process-on each layer's coefficients and biases. Notably, if we remove all cutting planes, this propagation method reduces to β -CROWN [54]. In this context, the new dual variables µ and τ correspond to the dual variable β used for x splits in β -CROWN. To handle splits, we only need to specify the sets Z +( i ) and Z -( i ) for each layer. By adjusting the dual variables and functions to reflect the fixed states of certain neurons, the extended bound propagation maintains tightness and correctness in the computed bounds.

3.2 Improving BaB Inferred Cuts via Constraint Strengthening and Multi-Tree Search

Theorem 3.2 allows us to use the cheap bound propagation method to search the BaB tree quickly to discover UNSAT paths to generate BaB inferred cuts by Proposition 3.1. However, our second key observation is that naively inferred cuts will not be beneficial in a regular BaB search process, illustrated in Fig. 2a, for example, all the subproblems after the split x 1 ≥ 0 (all nodes on the right after the first split) imply z 1 = 1 , and thus z 1 + z 3 ≥ 1 always holds. Thus, we have to strengthen these cuts to make them more effective. We propose the Branch-and-bound Inferred Cut with COnstraint Strengthening (BICCOS) to solve this.

(a) Based on the UNSAT path { x 1 ≤ 0 , x 3 ≤ 0 } , we can infer the cut 3 as - z 1 - z 3 ≤ - 1 ⇒ z 1 + z 3 ≥ 1 . However, none of the other subproblems would vio- late this constraint anyway, so the bounds do not improve.

(b) Starting from the UNSAT path { x 1 ≤ 0 , x 5 ≤ 0 , x 7 ≥ 0 } , { x 1 ≤ 0 } is not actually needed, as { x 5 ≤ 0 , x 7 ≥ 0 } is sufficient to prove that this subproblem is UNSAT. We can infer a new cut z 7 - z 5 ≥ 1 ,

(c) The root node is split multiple ways. In the sub-tree where the root node is split on x 1 , we extract the cut z 1 + z 3 ≥ 1 . This cut is then applied to all other sub-trees as well. In the sub-tree with the root node split on x 8 , inducing another cut z 8 ≥ 1 that will be shared across sub-trees.

2g +  ≤ 1

Z7

A verified path

Zi

=

Split z1

Z3

Z1

Z8

Split z'3

A verified path)

Remove

8 ≥ 1

+3≥1

the split

1=0

on z1 then

already satisfied!

'Z3

Z3

Re-verify

Z5=0

Search multi-tree simultaneously

Bounds do not improve!

Each verified path infers a cut.

Z7

0≥ sz - lz

Z5=0

Z7=

一

0≥ sz

If the reduced path

=1

is still verifiable, a

new cut is inferred

In a known UNSAT subproblem with multiple branched neurons, it is possible that a subset of the branching neurons is sufficient to prove UNSAT, shown in Fig. 2b. By focusing on this subset, we can strengthen the BaB inferred cuts by reducing the number of z variables involved. From an optimization perspective, each cut corresponds to a hyperplane that separates feasible solutions from infeasible ones. Simplifying the cut by involving fewer z variables reduces the dimensionality of the hyperplane, making it more effective at excluding infeasible regions without unnecessarily restricting the feasible space. Thus, we can draw a corollary:

Corollary 3.3. Formally, consider the two cuts from Eq. (3):

If either Z + 1 ⊂ Z + 2 and Z -1 ⊆ Z -2 , or Z + 1 ⊆ Z + 2 and Z -1 ⊂ Z -2 , then any assignment { z i } satisfying the cut associated with the smaller split set Z also satisfies the other cut. This implies that the cut with the smaller set is strictly stronger than the one with the larger set.

Proof deferred to Appendix B.3. This demonstrates that cuts with fewer variables can be more powerful because they operate in lower-dimensional spaces, allowing the hyperplane to more effectively position itself to exclude infeasible regions.

Now, the challenge is determining which unnecessary branches to remove to produce a cut with as few z variables as possible. An important observation from Theorem 3.2 is that the dual variables µ and τ , associated with the fixed ReLU indicators z ∈ Z +( i ) and z ∈ Z -( i ) respectively, reflect the impact of these fixed activations on the bound. Specifically, a positive dual variable µ > 0 for some z ∈ Z +( i ) indicates that the fixed active state of these neurons contributes significantly to tightening the bound. Similarly, a positive τ > 0 for some z ∈ Z -( i ) signifies that the fixed inactive neurons are influential in optimizing the bound. However, neurons with zero dual variables ( µ = 0 or τ = 0 ) may not contribute to the current bound optimization. This observation suggests that these neurons might be candidates for removal to simplify the cut. But it's important to note that a zero dual variable does not guarantee that the corresponding neuron has no impact under all circumstances-it only indicates no contribution in the current optimization context. Simply removing all neurons with zero dual

variables might overlook their potential influence in other parts of the search space or under different parameter settings. Therefore, the challenge lies in deciding which neurons with zero or negligible dual variables can be safely removed without significantly weakening the cut. This decision requires a careful heuristic that considers not only the current values of the dual variables but also the overall structure of the problem and the potential future impact of these neurons. By intelligently selecting which z variables to exclude, we aim to produce a stronger cut that is both effective in pruning the search space and efficient in computing. To do this, we use a heuristic to determine whether each neuron should be tentatively dropped from the list of constraints. Algorithm 1 shows the constraint strengthening with the neuron elimination heuristic.

Neuron Elimination Heuristic for Constraint Strengthening First, we compute a heuristic influence score for each neuron to assess its impact on the verification objective. This score is based on the improvement in the lower bound of f ⋆ before and after introducing the neuron's split in the BaB tree. By recording the computed lower bounds at each node, we can measure how beneficial each constraint is to the verification process. Second, we rank the neurons according to their influence scores and tentatively drop those that contribute least to improving the lower bound. We will retain neurons whose corresponding Lagrange multipliers, µ or τ , are greater than zero. For neurons with µ or τ equal to zero, we remove a certain percentile of neurons with the lowest scores, as indicated by the 'drop_percentage' parameter in Algorithm 1. When a neuron is dropped, its split is canceled. Then, we perform a re-verification step using only the reduced subset of constraints. This involves recomputing the lower bound on f ⋆ based solely on the selected constraints. If the verification still succeeds-that is, the lower bound remains non-negative-the reduced set induces a new cutting plane. This new cutting plane is applied to all subproblems within the BaB process after further strengthening. Finally, we can iteratively repeat the process of reducing the constraint set, aiming to generate even stronger cuts. The process terminates when the property can no longer be verified with the current subset of constraints. This iterative refinement is designed to focus on the most influential neurons, potentially enhancing the efficiency of the verification. Once the new cuts determined, we merge pairs of cuts if possible (e.g. merging z 1 + z 2 ≥ 0 and z 1 -z 2 ≥ 0 to z 1 ≥ 0 ).

Multi-Tree Search Traditionally, the BaB process generates a single search tree. We propose augmenting this approach by performing multiple BaB processes in parallel as a presolving step, with each process exploring a different set of branching decisions. At each branching point, we initialize multiple trees and apply various branching decisions simultaneously. While this initially increases the number of subproblems, the cutting planes generated in one tree are universally valid and can be applied to all other trees. These newly introduced cuts can help prove UNSAT for nodes in other trees, thereby inducing additional cutting planes and amplifying the pruning effect across the entire search space, illustrated in Fig. 2c.

Since computational resources must be allocated across multiple trees, we prioritize nodes for further expansion that have the highest lower bound on the optimization objective. This strategy ensures that more promising trees receive more computational resources. After a predefined short timeout, we

consolidate our efforts by pruning all but one of the trees and proceed with the standard BaB process augmented with BICCOS on the selected tree. We choose the tree that has been expanded the most frequently, as this indicates that its bounds are closest to verifying the property.

BaB Tree Searching Strategy In the standard BaB process (e.g., in β -CROWN), branches operate independently without sharing information, so the order in which they are explored does not affect the overall runtime. For memory access efficiency, there is a slight preference for implementing BaB as a depth-first search (DFS), where constraints are added until unsatisfiability (UNSAT) can be proven [61, 54]. This approach focuses the search on deeper branches before returning to shallower nodes.

However, in the context of BICCOS, our objective is to generate strong cutting planes that can prune numerous branches across different subproblems. To maximize the generality of these cutting planes, they need to be derived from UNSAT nodes with as few constraints as possible. While constraint strengthening techniques can simplify the constraints, this process is more straightforward when the original UNSAT node already has a minimal set of constraints. Even if only a few constraints are eliminated, the resulting cutting plane can significantly impact many other subproblems. To facilitate this, we propose performing the BaB algorithm using a breadth-first search (BFS) strategy. By exploring nodes that are closest to the root and have the fewest neuron constraints, we can generate more general and impactful cutting planes earlier in the search process.

Algorithm 2 Branch-and-bound Inferred Cuts with Constraint Strengthening (BICCOS).

3.3 BICCOS Summary

Algorithm 2 summarizes our proposed BICCOS algorithm, with the modifications to the standard BaB algorithm highlighted in brown. First (line 2), instead of exploring a single tree, we explore multiple trees in parallel as a presolving step. This process may involve constraint strengthening and utilizes cut inference analogous to the procedures in lines 3-15 of the algorithm. After several iterations, we prune all but one of the trees. From this point forward, only the selected tree is expanded further, following the regular BaB approach.

Until all subdomains of this tree have been verified, BICCOS selects batches of unverified subdomains add additional branching decisions, and attempt to prove the verification property. Unlike regular BaB, it then applies constraint strengthening to all identified UNSAT nodes and infers the corresponding cutting planes. These cutting planes are added to all currently unverified subdomains, potentially improving their lower bounds enough to complete the verification process. If BICCOS fails to identify helpful cutting planes, it effectively behaves like the regular BaB algorithm. We have implemented BICCOS in the α , β -CROWN toolbox. Notably, the cuts found by BICCOS are compatible with those from MIP solvers in GCP-CROWN, and all cuts can be combined in cases where MIP cuts are beneficial.

4 Experiments

We evaluate our verifier, BICCOS, on several popular verification benchmarks from VNN-COMP [4, 40, 10] and on the SDP-FO benchmarks used in multiple studies [15, 54, 41]. In the following discussion, α,β -CROWNrefers to the verification tool that implements various verification techniques, while β -CROWN and GCP-CROWN denote specific algorithms implemented within α,β -CROWN. To ensure the comparability of our method's effects and to minimize the influence of hardware and equipment advances, we rerun β -CROWN and GCP-CROWN with MIP cuts for each experiment. Additionally, we use the same BaB algorithm as in β -CROWN and employ filtered smart branching

*

Results from VNN-COMP 2021 report [4]. † Results from VNN-COMP 2022 report [43] ‡ Results from VNN-COMP 2024 website[1]

* MN-BaB with 600s timeout threshold for all models. '-' indicates that we could not run a model due to unsupported model structure or other errors. We run β -CROWN, GCP-CROWN with MIP cuts and BICCOS with a shorter 200s timeout for all models. The increased timeout for MN-BaB may increase the percentage of verified instances. However, we can still achieve better verified accuracy than all other baselines. Other results are reported from [61].

(FSB) [16] as the branching heuristic in all experiments. Note that in experiments GCP-CROWN refers to GCP-CROWN solver with MIP cuts. We also conduct ablation studies to identify which components of BICCOS contribute the most, including analyses of verification accuracy & time, number of cuts generated, and number of domains visited. Experimental settings are described in Appendix C.1.

Results on VNN-COMP benchmarks We first evaluate BICCOS on many challenging benchmarks with large models, including two VNN-COMP 2024 benchmarks: cifar100-2024 and tinyimagenet-2024 ; two VNN-COMP 2022 benchmarks: cifar100-tinyimagenet-2022 and oval22 . Shown in Table 1, our proposed method, BICCOS, outperforms most other verifiers on the tested benchmarks, achieving the highest number of verified instances in four benchmark sets. BICCOS consistently outperforms the baseline α, β -CROWN verifier (the β -CROWN and GCPCROWN (MIP cuts) lines), verifying more instances across almost all benchmark sets. In particular, GCP-CROWN with MIP cuts cannot scale to the larger network architectures in the cifar100 and tinyimagenet benchmarks with network sizes between 14.4 and 31.6 million parameters, due to its reliance on an MIP solver. BICCOS, on the other hand, can infer cutting planes without the need for an MIP solver and noticeably outperforms the baseline on cifar100 and tinyimagenet . Note that the increase in average runtime (e.g., on the cifar100-tinyimagenet-2022 benchmark) is expected. The instances that could not be verified at all previously but can be verified using BICCOS tend to require runtimes that are below the timeout but above the baseline's average runtime.

Results on SDP-FO benchmarks We further evaluated BICCOS, on the challenging SDP-FO benchmarks introduced in previous studies [15, 54]. These benchmarks consist of seven predominantly adversarial trained MNIST and CIFAR models, each containing numerous instances that are difficult for many existing verifiers. Our results, detailed in Table 2, demonstrate that BICCOS significantly improves verified accuracy across all the tested models when compared to current stateof-the-art verifiers. On both MNIST and CIFAR dataset, BICCOS not only surpasses the performance of methods like β -CROWN and GCP-CROWN on the CNN-A-Adv model but also approaches the empirical robust accuracy upper bound, leaving only a marginal gap. A slight increase in average time in some cases is attributed to the higher number of solved instances.

Ablation Studies on BICCOS Components. To evaluate the contributions of individual components of BICCOS, we performed ablation studies summarized in Table 3. The BICCOS base version with BaB inferred cuts and constraint strengthening already shows competitive performance in many models. Cuts from MIP solvers are compatible with BICCOS and can be added for smaller models that can be handled by MIP solver. Integrating Multi-Tree Search (MTS) significantly boosts performance. On the CIFAR CNN-B-Adv model, verified accuracy rises to 54.5%, outperforming

* We run our BICCOS in different ablation studies with a shorter 200s timeout for all models and compare it to β -CROWN and GCP-CROWN, it achieves better verified accuracy than all other baselines.

GCP-CROWN with MIP cuts's 49%. We also design an adaptive BICCOS configuration (BICCOS auto), which automatically turns on MTS and/or MIP-based cuts according to neural network and verification problem size and quantity, achieves the highest verified accuracies across most of models and is used as the default option of the verifier when BICCOS is enabled. A detailed table with the numbers of cuts and domains visited is provided in Appendix C.2.

5 Related work

Our work is based on the branch and bound framework for neural network verification [12, 17, 59, 28, 54, 16, 37, 21], which is one of the most popular approaches that lead to state-of-the-art results [4, 10, 40]. Most BaB-based approaches do not consider the correlations among subproblems - for example, in β -CROWN [54], the order of visiting the nodes in the BaB search tree does not change the verification outcome as the number of leaf nodes will be the same regardless of how the leaves are split. Our work utilizes information on the search tree and can gather more effective cuts when shallower nodes are visited first.

Exploring the dependency or correlations among neurons has also been identified as a potential avenue to enhance verification bounds. While several studies have investigated this aspect [2, 16, 42, 50], their focus has primarily been on improving the bounding step without explicitly utilizing the relationships among ReLUs during the branching process. Venus [9] considers the implications among neurons with constraints similar to our cutting planes. However, their constraints were not discovered using the verified subproblems during BaB or multi-tree search, and cannot be strengthened. On the other hand, cutting plane methods encode dependency among neurons as general constraints [61, 35], and our work developed a new cutting plane that can be efficiently constructed and strengthened during BaB, the first time in literature.

In addition, some NN verifiers are based on the satisfiability modulo theories (SMT) formulations [32, 46, 19, 36], which may internally use an SAT-solving procedure [8] such as DPLL [22] or CDCL [38]. These procedures may discover conflicts in boolean variable assignments, corresponding to eliminating certain subproblems in BaB. However, they differ from BICCOS in two significant aspects: first, although DPLL or CDCL may discover constraints to prevent some branches with neurons involved in these constraints, they cannot efficiently use these constraints as cutting planes that may tighten the bounds for subproblems never involving these neurons; second, DPLL or CDCL works on the abstract problem where each ReLU is represented as a boolean variable, and cannot take full advantage of the underlying bound propagation solver to strengthen constraints as we did in Alg. 1. Based on our observation in Sec. 3, the constraints discovered during BaB are often unhelpful without strengthening unless in a different search tree, so their effectiveness is limited. However, the learned conflicts can be naturally translated into cuts 3, making this a future work.

More related works on SMT, MIP solvers, nogood learning and cutting plane method in VNN [18, 45, 26, 19, 11, 2, 47, 52, 30, 61, 35] are discussed in Appendix D.

6 Conclusion

We exploit the structure of the NN verification problem to generate efficient and scalable cutting planes, leveraging neuron relationships within verified subproblems in a branch-and-bound search tree. Our experimental results demonstrate that the proposed BICCOS algorithm achieves very good scalability while outperforming many other tools in the VNN-COMP, and can solve benchmarks that existing methods utilizing cutting planes could not scale to. Limitations are discussed in Appendix E.

Acknowledgment Huan Zhang is supported in part by the AI2050 program at Schmidt Sciences (AI2050 Early Career Fellowship) and NSF (IIS-2331967). Grani A. Hanasusanto is supported in part by NSF (CCF-2343869 and ECCS-2404413). Computations were performed with computing resources granted by RWTH Aachen University under project rwth1665. We thank the anonymous reviewers for helping us improve this work.

References

VNN-COMP 2024 website. https://sites.google.com/view/vnn2024 . Accessed: 202410-30.

Ross Anderson, Joey Huchette, Will Ma, Christian Tjandraatmadja, and Juan Pablo Vielma. Strong mixed-integer programming formulations for trained neural networks. Mathematical Programming , 183(1-2):3-39, 2020.

Stanley Bak. nnenum: Verification of relu neural networks with optimized abstraction refinement. In NASA Formal Methods Symposium , pages 19-36. Springer, 2021.

Stanley Bak, Changliu Liu, and Taylor Johnson. The second international verification of neural networks competition (vnn-comp 2021): Summary and results. arXiv preprint arXiv:2109.00498 , 2021.

Stanley Bak, Hoang-Dung Tran, Kerianne Hobbs, and Taylor T. Johnson. Improved geometric path enumeration for verifying relu neural networks. In Proceedings of the 32nd International Conference on Computer Aided Verification . Springer, 2020.

Egon Balas and Robert Jeroslow. Canonical cuts on the unit hypercube. SIAM Journal on Applied Mathematics , 23(1):61-69, 1972.

Dimitris Bertsimas and Robert Weismantel. Optimization over integers. Athena Scientific, 2005.

Armin Biere, Marijn Heule, and Hans van Maaren. Handbook of satisfiability , volume 185. IOS press, 2009.

Elena Botoeva, Panagiotis Kouvaros, Jan Kronqvist, Alessio Lomuscio, and Ruth Misener. Efficient verification of relu-based neural networks via dependency analysis. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3291-3299, 2020.

Christopher Brix, Stanley Bak, Changliu Liu, and Taylor T Johnson. The fourth international verification of neural networks competition (vnn-comp 2023): Summary and results. arXiv preprint arXiv:2312.16760 , 2023.

Rudy Bunel, Jingyue Lu, Ilker Turkaslan, Philip HS Torr, Pushmeet Kohli, and M Pawan Kumar. Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Research , 21(42):1-39, 2020.

Rudy R Bunel, Ilker Turkaslan, Philip Torr, Pushmeet Kohli, and Pawan K Mudigonda. A unified view of piecewise linear neural network verification. Advances in Neural Information Processing Systems , 31, 2018.

Shaoru Chen, Lekan Molu, and Mahyar Fazlyab. Verification-aided learning of neural network barrier functions with termination guarantees. arXiv preprint arXiv:2403.07308 , 2024.

Michele Conforti, Gérard Cornuéjols, and Giacomo Zambelli. Integer programming . Graduate texts in mathematics, . 271. Springer, Cham, 2014.

Sumanth Dathathri, Krishnamurthy Dvijotham, Alexey Kurakin, Aditi Raghunathan, Jonathan Uesato, Rudy R Bunel, Shreya Shankar, Jacob Steinhardt, Ian Goodfellow, Percy S Liang, et al. Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Alessandro De Palma, Harkirat Singh Behl, Rudy Bunel, Philip H. S. Torr, and M. Pawan Kumar. Scaling the convex barrier with active sets. International Conference on Learning Representations (ICLR) , 2021.

Alessandro De Palma, Rudy Bunel, Alban Desmaison, Krishnamurthy Dvijotham, Pushmeet Kohli, Philip HS Torr, and M Pawan Kumar. Improved branch and bound for neural network verification via lagrangian decomposition. arXiv preprint arXiv:2104.06718 , 2021.

Rina Dechter. Enhancement schemes for constraint processing: Backjumping, learning, and cutset decomposition. Artificial Intelligence , 41(3):273-312, 1990.

Hai Duong, Linhan Li, ThanhVu Nguyen, and Matthew Dwyer. A dpll (t) framework for verifying deep neural networks. arXiv preprint arXiv:2307.10266 , 2023.

Ruediger Ehlers. Formal verification of piece-wise linear feed-forward neural networks. In Automated Technology for Verification and Analysis: 15th International Symposium, ATVA 2017, Pune, India, October 3-6, 2017, Proceedings 15 , pages 269-286. Springer, 2017.

Claudio Ferrari, Mark Niklas Muller, Nikola Jovanovic, and Martin Vechev. Complete verification via multi-neuron relaxation guided branch-and-bound. arXiv preprint arXiv:2205.00263 , 2022.

Harald Ganzinger, George Hagen, Robert Nieuwenhuis, Albert Oliveras, and Cesare Tinelli. Dpll (t): Fast decision procedures. In Computer Aided Verification: 16th International Conference, CAV 2004, Boston, MA, USA, July 13-17, 2004. Proceedings 16 , pages 175-188. Springer, 2004.

Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP) . IEEE, 2018.

Paul C Gilmore and Ralph E Gomory. A linear programming approach to the cutting stock problem-part ii. Operations research , 11(6):863-888, 1963.

Julien Girard-Satabin, Michele Alberti, Francois Bobot, Zakaria Chihani, and Augustin Lemesle. Caisar: A platform for characterizing artificial intelligence safety and robustness. arXiv preprint arXiv:2206.03044 , 2022.

Carla P Gomes, Bart Selman, Henry Kautz, et al. Boosting combinatorial search through randomization. AAAI/IAAI , 98(1998):431-437, 1998.

LLC Gurobi Optimization. Gurobi optimizer - gurobi, 2022.

Patrick Henriksen and Alessio Lomuscio. Efficient neural network verification via adaptive refinement and adversarial search. In ECAI 2020 , pages 2513-2520. IOS Press, 2020.

Patrick Henriksen and Alessio Lomuscio. Deepsplit: An efficient splitting method for neural network verification via indirect effect analysis. In Proceedings of the 30th international joint conference on artificial intelligence (IJCAI21) , pages 2549-2555, 2021.

Joey Huchette, Gonzalo Muñoz, Thiago Serra, and Calvin Tsay. When deep learning meets polyhedral theory: A survey. arXiv preprint arXiv:2305.00241 , 2023.

IBM. Ibm ilog cplex optimizer, 2022.

Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zelji´ c, et al. The marabou framework for verification and analysis of deep neural networks. In International Conference on Computer Aided Verification (CAV) , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR) , 2015.

Panagiotis Kouvaros and Alessio Lomuscio. Towards scalable complete verification of relu neural networks via dependency-based branching. In IJCAI , pages 2643-2650, 2021.

Jianglin Lan, Yang Zheng, and Alessio Lomuscio. Tight neural network verification via semidefinite relaxations and linear reformulations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7272-7280, 2022.

Zongxin Liu, Pengfei Yang, Lijun Zhang, and Xiaowei Huang. DeepCDCL: An CDCL-based neural network verification framework. arXiv preprint arXiv:2403.07956 , 2024.

Jingyue Lu and M Pawan Kumar. Neural network branching for neural network verification. arXiv preprint arXiv:1912.01329 , 2019.

Joao Marques-Silva, Inês Lynce, and Sharad Malik. Conflict-driven clause learning SAT solvers. In Handbook of satisfiability , pages 133-182. ios Press, 2021.

Christoph Müller, Francois Serre, Gagandeep Singh, Markus Püschel, and Martin Vechev. Scaling polyhedral neural network verification on gpus. Proceedings of Machine Learning and Systems , 3:733-746, 2021.

Mark Niklas Müller, Christopher Brix, Stanley Bak, Changliu Liu, and Taylor T Johnson. The third international verification of neural networks competition (vnn-comp 2022): Summary and results. arXiv preprint arXiv:2212.10376 , 2022.

Mark Niklas Müller, Gleb Makarchuk, Gagandeep Singh, Markus Püschel, and Martin Vechev. Precise multi-neuron abstractions for neural network certification. arXiv preprint arXiv:2103.03638 , 2021.

Mark Niklas Müller, Gleb Makarchuk, Gagandeep Singh, Markus Püschel, and Martin Vechev. Prima: general and precise neural network certification via scalable convex hull approximations. Proceedings of the ACM on Programming Languages , 6(POPL):1-33, 2022.

Mark Niklas Müller, Christopher Brix, Stanley Bak, Changliu Liu, and Taylor T. Johnson. The third international verification of neural networks competition (vnn-comp 2022): Summary and results, 2023.

Chongli Qin, Brendan O'Donoghue, Rudy Bunel, Robert Stanforth, Sven Gowal, Jonathan Uesato, Grzegorz Swirszcz, Pushmeet Kohli, et al. Verification of non-linear specifications for neural networks. arXiv preprint arXiv:1902.09592 , 2019.

Tuomas Sandholm and Robert Shields. Nogood learning for mixed integer programming. In Workshop on Hybrid Methods and Branching Rules in Combinatorial Optimization, Montréal , volume 20, pages 21-22, 2006.

Karsten Scheibler, Leonore Winterer, Ralf Wimmer, and Bernd Becker. Towards verification of artificial neural networks. In MBMV , pages 30-40, 2015.

Thiago Serra and Srikumar Ramalingam. Empirical bounds on linear regions of deep rectifier networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5628-5635, 2020.

Zhouxing Shi, Qirui Jin, J Zico Kolter, Suman Jana, Cho-Jui Hsieh, and Huan Zhang. Formal verification for neural networks with general nonlinearities via branch-and-bound. 2nd Workshop on Formal Verification of Machine Learning , 2023.

Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages (POPL) , 2019.

Harkirat Singh, M Pawan Kumar, Philip Torr, and Krishnamurthy Dj Dvijotham. Overcoming the convex barrier for simplex inputs. In Advances in Neural Information Processing Systems , 2021.

Chuangchuang Sun, Dong-Ki Kim, and Jonathan P How. Romax: Certifiably robust deep multiagent reinforcement learning via convex relaxation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 5503-5510. IEEE, 2022.

Calvin Tsay, Jan Kronqvist, Alexander Thebelt, and Ruth Misener. Partition-based formulations for mixed-integer optimization of trained relu neural networks. Advances in neural information processing systems , 34:3068-3080, 2021.

Andreas Venzke and Spyros Chatzivasileiadis. Verification of neural network behaviour: Formal guarantees for power system applications. IEEE Transactions on Smart Grid , 12(1):383-397, 2020.

Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification. Advances in Neural Information Processing Systems , 34:29909-29921, 2021.

Eric Wong, Tim Schneider, Joerg Schmitt, Frank R Schmidt, and J Zico Kolter. Neural network virtual sensors for fuel injection quantities with provable performance specifications. In 2020 IEEE Intelligent Vehicles Symposium (IV) , pages 1753-1758. IEEE, 2020.

Haoze Wu, Omri Isac, Aleksandar Zelji´ c, Teruhiro Tagomori, Matthew Daggitt, Wen Kokke, Idan Refaeli, Guy Amir, Kyle Julian, Shahaf Bassan, et al. Marabou 2.0: a versatile formal analyzer of neural networks. In International Conference on Computer Aided Verification , pages 249-264. Springer, 2024.

Junlin Wu, Andrew Clark, Yiannis Kantaros, and Yevgeniy Vorobeychik. Neural lyapunov control for discrete-time systems. Advances in Neural Information Processing Systems , 36:29392955, 2023.

Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems , 33:1129-1141, 2020.

Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. International Conference on Learning Representations (ICLR) , 2021.

Lujie Yang, Hongkai Dai, Zhouxing Shi, Cho-Jui Hsieh, Russ Tedrake, and Huan Zhang. Lyapunov-stable neural control for state and output feedback: A novel formulation for efficient synthesis and verification. arXiv preprint arXiv:2404.07956 , 2024.

Huan Zhang, Shiqi Wang, Kaidi Xu, Linyi Li, Bo Li, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. General cutting planes for bound-propagation-based neural network verification. Advances in Neural Information Processing Systems , 35:1656-1670, 2022.

Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. Advances in neural information processing systems , 31, 2018.

Appendix

In Sec. A, we introduce the Mixed Integer Programming (MIP) formulation for Neural Network Verification and explain how to get the lower bound by solving the LP relaxation and Planet relaxation. In Sec. B, we provide the complete proof for Proposition 3.1 and Theorem 3.2 and Corollary 3.3. Then, in Sec. C, we present the configuration, ablation study, and additional experimental results. Finally in Sec. D and Sec. E, we discuss more related works and limitations of our method.

A MIP Formulation and LP & Planet Relaxation

The MIP Formulation The mixed integer programming (MIP) formulation is the root of many NN verification algorithms. Given the ReLU activation function's piecewise linearity, the model requires binary encoding variables, or ReLU indicators z only for unstable neurons. We formulate the optimization problem aiming to minimize the function f ( x ) , subject to a set of constraints that encapsulate the DNN's architecture and the perturbation limits around a given input x , as follows:

z

(

j

∈ {

0

,

1

}

;

j

∈ I

, i

∈

[

L

-

1]

(12)

Here, the set J ( i ) comprises all neurons in the layer i , which are further categorized into three distinct classes: 'active' ( I +( i ) ), 'inactive' ( I -( i ) ), and 'unstable' ( I ( i ) ); we further let l ( i ) j = min x ∈X f ( i ) j ( x ) , u ( i ) j = max x ∈X f ( i ) j ( x ) , ∀ j ∈ J ( i ) , i ∈ [ L -1] . The MIP approach is initialized with pre-activation bounds l ( i ) ≤ x ( i ) ≤ u ( i ) for each neuron within the feasible set X , across all layers i . These bounds can be calculated recursively through the MIP from the first layer to the final layer. However, MIP problems are generally NP-hard as they involve integer variables.

The LP and Planet relaxation By relaxing the binary variables 12 to z ( i ) j ∈ [0 , 1] , j ∈ I ( i ) , i ∈ [ L -1] , we can get the LP relaxation formulation. By replacing the constraints in 10, 11, 12 with

we can eliminate the z variables and get the well-known Planet relaxation formulation. Both of these two relaxations are solvable in polynomial time to yield lower bounds.

Lemma A.1. In the LP relaxation, splitting an unstable ReLU neuron ( i, j ) based on z ( i ) j = 0 is equivalent to setting x ( i ) j ≤ u ( i ) j = 0 in the Planet relaxation with the relaxed variable z ∈ [0 , 1] projected out. Correspondingly, splitting based on z ( i ) j = 1 is equivalent to setting x ( i ) j ≥ l ( i ) j = 0 .

Proof: An unstable neuron ( i, j ) is one where the pre-activation bounds satisfy l ( i ) j < 0 < u ( i ) j . It could be represented as the following system in LP relaxation: 8, 10, 11, 12, and the system in Planet

i

)

(

i

)

relaxation: 8, 15. We will now analyze the two cases of fixing z ( i ) j to 0 and 1 in the LP relaxation and show their equivalence to modifying the bounds u ( i ) j and l ( i ) j in the Planet relaxation.

1. z ( i ) j = 0 in LP relaxation ⇔ u ( i ) j = 0 in Planet Relaxation.

( ⇒ ) Assume z ( i ) j = 0 in LP relaxation. From 10, ˆ x ( i ) j ≤ u ( i ) j × 0 = 0 = ⇒ ˆ x ( i ) j ≤ 0 . From 8: ˆ x ( i ) j ≥ 0 . Combining the above, we get ˆ x ( i ) j = 0 . From 11, ˆ x ( i ) j ≤ x ( i ) j -l ( i ) j (1 -0) = x ( i ) j -l ( i ) j . Since ˆ x ( i ) j = 0 and 9, this simplifies to 0 ≤ x ( i ) j -l ( i ) j = ⇒ 0 ≥ x ( i ) j ≥ l ( i ) j This is consistent with the pre-activation bounds of the neuron. Therefore, fixing z ( i ) j = 0 in the LP relaxation forces the activation ˆ x ( i ) j to be zero, effectively modeling the neuron as inactive.

( ⇐ ) Assume u ( i ) j = 0 in the Planet relaxation. Set u ( i ) j = 0 . The constraint 15 becomes ˆ x ( i ) j ≤ 0 0 -l ( i ) j ( x ( i ) j -l ( i ) j ) = 0 . From 8 ˆ x ( i ) j ≥ 0 . Combining the above, we get ˆ x ( i ) j = 0 .

Thus, setting u ( i ) j = 0 in the Planet relaxation also forces ˆ x ( i ) j to zero, mirroring the effect of fixing z ( i ) j = 0 in the LP relaxation.

B Proof of Main Results

B.1 Proof of Proposition 3.1

Proof: Due to the relationship between the x and z in Lemma A.1, i.e. for any unstable neuron i , x i ≥ l i = 0 ⇐⇒ z i = 1 and x i ≤ u i = 0 ⇐⇒ z i = 0 ⇐⇒ (1 -z i ) = 1 , we have the following equation to represent an UNSAT path:

By taking negation of Eq 16 to exclude this path, we have:

Notice that z ∈ [0 , 1] |Z| , implies that the sum of z should be bounded by its cardinality, ∑ i ∈Z + z i + ∑ i ∈Z -(1 -z i ) ≤ |Z + | + |Z -| , and the binary indicator property, thus we only consider inequality 18:

Thus, the BaB inferred cut 3 can be used to exclude this UNSAT status.

B.2 Proof of Theorem 3.2

To derive the bound propagation, first, we assign Lagrange dual variables to each non-trivial constraint in LP relaxation with any BaB split set Z . Next, we analyze how splitting on the ReLU indicator z influences the generation of these dual variables. This analysis helps us determine the impact of the splits on the dual formulation. Then, we formulate the Lagrangian dual as a min -max problem. By applying strong duality and eliminating the inner minimization problem, we can simplify the dual formulation. Finally, this results in a dual formulation that we use to perform bound propagation effectively.

□

Theorem 3.2. Given any BaB split set Z , optimizable parameters 0 ≤ α ( i ) j ≤ 1 and β , µ , τ ≥ 0 , π ( i ) j ⋆ is a function of Q ( i ) : ,j :

where variables ν ( i ) are obtained by propagating ν ( L ) = -1 throughout all i ∈ [ L -1] :

Here ˆ ν ( i ) j , π ( i ) j ⋆ and τ ( i ) j are defined for each unstable and without split neuron j ∈ I ( i ) \ Z ( i ) , and h ( i ) j ( β ) is defined for all unstable neurons j ∈ I ( i ) .

Proof: First we write down the primal formulation, and assign the Lagrange multipliers:

Note that for some constraints, their dual variables are not created because they are trivial to handle in the steps. And from Lemma A.1, some unstable neuron constraints 8, 9, 10, 11 will be eliminated during BaB with split on z , i.e. when z ( i ) j = 1 , the unstable neuron constraints 8 will be reduced to

ˆ x ( i ) j = x ( i ) j ≥ 0 ⇒ µ ( i ) j ∈ R + , other dual variables for constraints 9, 10, 11 will not be generated, and when z ( i ) j = 0 , only τ ( i ) j ∈ R + will be generated. Rearrange the equation and swap the min and max (strong duality) gives us:

Here W ( i +1) : ,j denotes the j -th column of W ( i +1) . Note that for the term involving j ∈ I +( i ) ∪Z +( i ) we have replaced ˆ x ( i ) with x ( i ) to obtain the above equation. For the term x ( i ) j , j ∈ I -( i ) ∪ Z -( i ) it is always 0 so it does not appear. Then solving the inner minimization gives us the dual formulation:

where h ( i ) j in the objective 21 is:

The ReLU( · ) term comes from minimizing over z ( i ) j with the constraint 0 ≤ z ( i ) j ≤ 1 . And when splitting on z ∈ Z ( i ) , as we discussed above, based on Lemma A.1, there will be no π ( i ) j , γ ( i ) j assigned for the constraints. The ∥ · ∥ 1 comes from the dual norm form minimizing over the infinite norm of the input x with the constraint x 0 -ϵ ≤ x ≤ x 0 + ϵ .

Thus, the rest of the proof follows from [61, Lemma A.1, Theorem 3.1], which completes the proof of Theorem 3.2.

B.3 Proof of Corollary 3.3

Proof. Let us consider any assignment of the variables { z i } where z i ∈ { 0 , 1 } that satisfies cut 3.1. We aim to show that this assignment also satisfies cut 3.2.

Since Z + 1 ⊂ Z + 2 , there exists at least one neuron k such that k ∈ Z + 2 but k / ∈ Z + 1 . Let Z (∆) + = Z + 2 \ Z + 1 denote the additional neurons in the larger cut 3.2. Similarly, define Z (∆) -= Z -2 \ Z -1 . In cut 3.2, the left-hand side (LHS) can be expressed as:

Similarly, the right-hand side (RHS) of cut 3.2 is:

Subtracting Eq. 3.1 from cut 3.2, we get:

Therefore, the difference between the LHS and RHS of the two cuts is:

Since each z i ∈ { 0 , 1 } , the maximum value of ∑ i ∈Z (∆) + z i is |Z (∆) + | , and the minimum value of ∑ i ∈Z (∆) -z i is 0 . Thus, the maximum possible value of the LHS difference is |Z (∆) + | , which equals the RHS difference.

Therefore, regardless of the values of z i for i ∈ Z (∆) + ∪ Z (∆) -, the inequality:

is always satisfied. This means that satisfying cut 3.1 ensures that cut 3.2 is also satisfied.

Conversely, consider an assignment where cut 3.2 is satisfied but cut 3.1 is violated. This can happen if the additional variables in Z (∆) + and Z (∆) -compensate for the violation in the original variables. For instance, setting z i = 1 for all i ∈ Z (∆) + and z i = 0 for all i ∈ Z (∆) -can decrease the LHS of cut 3.2, making it easier to satisfy even if cut 3.1 is not satisfied.

Therefore, the feasible region defined by cut 3.1 is a subset of the feasible region defined by cut 3.2, confirming that cut 3.1 is stronger.

C Experiments

C.1 Experiment Settings

Our experiments are conducted on a server with an Intel Xeon 8468 Sapphire CPU, one NVIDIA H100 GPU (96 GB GPU memory), and 480 GB CPU memory. Our implementation is based on the open-source α,β -CROWN verifier 1 with cutting plane related code added. All experiments use 24 CPU cores and 1 GPU. The MIP cuts are acquired by the cplex [31] solver (version 22.1.0.0).

We use the Adam optimizer [33] to solve both α , β , µ , τ . For the SDP-FO benchmarks, we optimize those parameters for 20 iterations with a learning rate of 0.1 for α and 0.02 for β , µ , τ . We decay the learning rates with a factor of 0.98 per iteration. The timeout is 200s per instance. For the VNN-COMP benchmarks, we use the same configuration as α , β -CROWN used in the respective competition and the same timeouts. During constraint strengthening, we set drop_percentage = 50% . We only perform one round of strengthening, with no recursive strengthening attempts. We perform constraint strengthening for the first 40 BaB iterations. For multi-tree search, we perform 5 branching iterations, where we each time pick the current best 50 domains and split them to generate 400 new sub-domains.

C.2 Ablation Studies: Number of Cuts and Branch Visited Analysis

The ablation studies provide insights into the impact of different components of the BICCOS algorithm on the verification process, focusing on the average number of branches (domains) explored and the number of cuts generated, as shown in Table 4. A lower number of branches typically indicates a more efficient search process, leading to computational savings and faster verification times.

Across the benchmarks, we observe that the BICCOS configurations, particularly BICCOS (auto), often explore fewer branches compared to the baseline β -CROWN and GCP-CROWN methods. For instance, on the CIFAR CNN-A-Adv model, BICCOS (auto) reduces the average number of branches from 63,809.77 (for β -CROWN) and 28,558.70 (for GCP-CROWN) down to 9,902.96. This reduction is attributed to the cutting planes introduced by the BICCOS algorithm, which tighten the relaxations and more effectively prune suboptimal regions of the search space.

The multi-tree search (MTS) strategy, which explores multiple branch-and-bound trees in parallel, demonstrates its effectiveness in improving verification efficiency. While MTS may increase the total number of branches explored-since domains from multiple trees are counted-the exploration within each tree is optimized, leading to faster convergence and reduced overall computation time. We also design an adaptive BICCOS configuration (BICCOS (auto)), which automatically enables MTS and/or Mixed Integer Programming (MIP)-based cuts based on the neural network and verification problem size. BICCOS (auto) achieves the highest verified accuracies across models and is used as the default option of the verifier when BICCOS is enabled.

* We run our BICCOS in different ablation studies with a shorter 200s timeout for all models and compare it to β -CROWN and GCP-CROWN, it achieves better verified accuracy than all other baselines.

However, it is worth noting that on certain benchmarks, such as the CIFAR CNN-A-Mix-4 model, the number of branches explored by different BICCOS configurations does not always decrease compared to the baselines. In some cases, the base BICCOS version explores more branches than β -CROWN. This could be attributed to the models' inherent complexity and the nature of the verification problem, where the benefits of the cutting planes and MTS may be less pronounced.

Overall, the analysis of the average number of branches in the ablation studies highlights the effectiveness of the BICCOS algorithm and its components in improving the efficiency of the verification process. The combination of cutting planes, multi-tree search, and other optimizations enables BICCOS to achieve high verified accuracy while often exploring fewer branches compared to baseline methods, ultimately leading to computational savings and faster verification times.

C.3 Comparison with MIP based Verifier

To evaluate the performance of the BICCOS cuts in MIP-based solvers, Fig 3 shows comparison with Venus2 [9, 34]. We emphasize on making a fair comparison on the strengths of cuts. Since Venus2 uses an MILP solver to process its cuts, in these experiments we do not use the efficient GCP-CROWN solver. Instead, we also use an MILP solver to handle the BICCOS cuts we found. This ensures that the speedup we achieve is not coming from the GPU-accelerated GCP-CROWN solver. Since our cut generation relies on the process with BaB, we first run BICCOS to get the cuts, and then import the cuts into the MILP solver.

We note that Venus uses branching over ReLU activations to create multiple strengthened MILP problems. On the other hand, we only create one single MILP and do not perform additional branching. Therefore, our MILP formulation is weaker. The fact that we can outperform Venus2 anyway underlines the strength of the generated cuts by BICCOS.

C.4 Overhead Analysis

We note that the multi-tree search (MTS) and cut strengthening procedure incur additional overheads compared to the baseline verifiers shown as Fig 4. The MTS has about 2 - 10 seconds overhead in the benchmarks we evaluated, which may increase the verification time of very easy instances that can be sequentially immediately verified in branch and bound. The BICCOS cut strengthening is called during every branch-and-bound iteration, but its accumulative cost is about 3 - 20 seconds for hard instances in each benchmark, which run a few hundred branch-and-bound iterations. We note that typically, the performance of the verifier is gauged by the number of verified instances within a fixed timeout threshold, thus when the threshold is too low, the added cutting planes may not have time to show their performance gains due to the lack of sufficient time for the branch-and-bound procedure.

We note that such a shortcoming is also shared with other verifiers which require additional steps for bound tightening; for example, the GCP-CROWN verifiers have the extra overhead of a few seconds of calling and retrieving results from the MIP solver, even when the MIP solver is running in parallel with the verifier. The final verified accuracy can justify whether the overhead is worth paying.

D More Related Works

Furthermore, nogood learning and conflict analysis techniques from constraint programming and SAT solving have been adapted to neural network verification to improve solver efficiency [18, 45, 26, 19,

NumberofInstancesVerifiedvs.Time

NumberofInstancesVerifiedvs.Time

200

200

175

MILPwithBICCOS

175

MILPwithBICCOS

150

Time Limit (200)

150

Time Limit(200)

125

125

Time

Time

100

100

75

75

50

50

25

25

0

20

40

60

80

0

20

40

60

80

100

NumberofInstancesVerifiedincifar_cnn_a_adv

NumberofInstancesVerifiedincifar_cnn_a_adv4

NumberofInstancesVerifiedvs.Time

NumberofInstancesVerifiedvs.Time

200

200

175

MILPwithBICCOS

175

150

TimeLimit(200)

150

Time

125

Time

125

MILPwithBICCOS

100

100

Time Limit (200)

75

75

50

50

25

25

口

0

10

20

OE

40

50

60

20

40

09

80

100

NumberofInstancesVerifiedincifar_cnn_a_mix

NumberofInstancesVerifiedincifarcnnamix4

11]. These methods record conflicts encountered during the search-known as nogoods-to prevent revisiting the same conflicting states, effectively pruning the search space. While nogood learning can reduce redundant exploration, it typically operates on discrete abstractions of neural networks, representing ReLU activations as boolean variables. This abstraction limits their ability to leverage the properties of neural network activations. In contrast, our approach not only identifies conflicts but also translates them into cutting planes within the continuous relaxation of the verification problem. This allows us to tighten the bounds for unexplored subproblems, enhancing the overall efficiency of the verification process.

Mixed Integer Programming (MIP) solvers, such as cplex [31] and gurobi [27], have also been applied to neural network verification by formulating the problem as a mixed-integer linear program [6]. These solvers inherently utilize branch-and-bound algorithms and incorporate sophisticated cutting plane generation techniques to tighten the feasible region. However, in practice, applying general-purpose MIP solvers to large-scale neural networks presents significant challenges. For instance, in our experiments with benchmarks like CIFAR100 and Tiny-ImageNet-where network sizes range from 5.4 million to over 30 million parameterscplex was unable to solve the initial LP relaxation within the 200-second timeout threshold. As a result, the branch-and-bound process and cut generation did not commence, underscoring the limitations of using off-the-shelf MIP solvers for large-scale neural network verification tasks.

In terms of cutting plane methods, prior work has explored integrating cuts derived from convex hull constraints within MIP formulations of neural networks [2, 47, 52, 30, 61, 35]. For example, [2] propose cutting planes that tighten the relaxation of ReLU constraints in MILP formulations, improving solution quality. While these methods can enhance optimization, they often incur significant computational overhead in generating and integrating cuts and rely heavily on the capabilities of the underlying MIP solver. In contrast, our approach develops new cutting planes that can be efficiently constructed and strengthened during the branch-and-bound process, leveraging the problem's structure to achieve better scalability and performance.

NumberofInstancesVerifiedvs.Time

200

BetaCROWN

175

Biccos_base

150

Biccos_MTS

GCPCROWN

125

Time

Biccos_auto

100

Time Limit (200)

75

50

25

75

08

85

90

95

NumberofInstancesVerifiedincifar_cnn_a_adv

NumberofInstancesVerifiedvs.Time

200

BetaCROWN

175

Biccos_base

150

Biccos_MTS

GCPCROWN

125

Time

Biccos_auto

100

Time Limit(200)

75

50

25

50

09

70

08

90

NumberofInstancesVerifiedincifar_cnn_a_mix

NumberofInstancesVerifiedvs.Time

200

Beta CROWN

175

Biccos_base

150

Biccos_MTS

GCPCROWN

125

Time

Biccos_auto

100

Time Limit(200)

75

50

25

50

60

70

80

90

100

110

NumberofInstancesVerifiedincifar_cnn_b_adv

NumberofInstancesVerifiedvs.Time

200

BetaCROWN

175

Biccos_base

150

Biccos_MTS

GCPCROWN

125

Time

Biccos_auto

100

Time Limit(200)

75

50

25

84

86

88

90

92

6

96

NumberofInstancesVerified incifar_cnn_a_adv4

NumberofInstancesVerifiedvs.Time

200

BetaCROWN

175

Biccos_base

150

Biccos_MTS

GCPCROWN

125

Time

Biccos_auto

100

Time Limit(200)

75

50

25

08

85

100

105

110

NumberofInstancesVerifiedincifar_cnn_a_mix4

NumberofInstancesVerifiedvs.Time

200

BetaCROWN

175

BICCOS_Base

150

Biccos_MTS

GCPCROWN

125

Time

Biccos_Auto

100

Time Limit(200)

75

50

25

0

20

40

60

80

100

120

NumberofInstancesVerifiedincifar_cnn_b_adv4

Our work bridges the gap between conflict analysis methods like nogood learning and cutting plane techniques by integrating conflict-derived constraints into the continuous relaxation of the verification problem. This integration allows for both effective pruning of the search space and tightening of the problem relaxation, leading to improved verification efficiency. By tailoring our approach to the specifics of neural network verification, we address the limitations observed in general-purpose MIP solvers when applied to large-scale networks, offering a scalable and efficient solution.

E Limitations

The approach relies on the piecewise linear nature of ReLU networks (same as most other papers in BaB), and its applicability to other types of neural architectures requires further extension such as those in [48]. Additionally, the effectiveness of the generated cutting planes may vary depending on the specific structure and complexity of the neural network being verified, and there may be cases where the algorithm's performance gains are less pronounced.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims match our theoretical and empirical results. We present results across several benchmarks to demonstrate that they can generalize to other settings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations have been explicitly discussed in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theoretical results have been proven.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Algorithms have been clearly described with technical details discussed. Parameters for experiments have been included in the appendix. The code will be opensourced.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: BICCOS has been added to the α , β -CROWN toolkit.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details have been discussed in the Experiments section and also in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Verification results are deterministic on the benchmarks, and no error bars need to be provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Time has been reported in tables and we only need one machine and one GPU to run all experiments. The specific hardware details are reported in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and acknowledged the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Societal impacts have been discussed explicitly in Section 7.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No data or models are released.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We've cited the sources of the benchmarks we used, such as those from the VNN-COMP.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are introduced in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFollowing the discussion with the authors, I am increasing my score to weak accept.\n----\n\nThis paper presents a new branch-and-bound-based verification algorithm for neural networks with ReLU activation. The main idea is to produce additional constraints to the verification problem by identifying small combinations of neuronal states which would not lead to an adversarial input, so that these combinations are not explored again in another part of the branch-and-bound tree. Because smaller sets are always better, the authors develop an algorithm for reducing the number of neurons involved in such combinations. They also leverage multiple branch-and-bound trees explored in breadth-first search as a means of producing constraints involving fewer neurons.\n\nSTRENGTHS:\nThe ideas presented by the authors are intuitively reasonable and unsurprisingly effective as a consequence. In fact, they have been studied in other contexts with other names. Most notably, the idea of producing a constraint based on a path of the branch-and-bound tree is commonly known as nogood learning (see [1] for an early use in constraint programming, such as with the concept of a conflict set in page 9; and [2] for the first application to mixed-integer programming). The canonical application has been to identify infeasible nodes, but it is not a big stretch to apply the same idea for when the objective function is likely suboptimal (positive objective function when what we need is a negative value). Moreover, the idea of exploring multiple branch-and-bound trees in parallel is not far from solver restarts [3], which have been widely used in constraint programming and subsequently adopted in mixed-integer programming as well.\n\nAll in all, this is a paper that speaks for itself because the right ideas were used and better numbers were obtained. Issues come next.\n\n[1] https://ftp.cs.ucla.edu/pub/stat_ser/r77-II-reprint.pdf\n\n[2] https://www.cs.cmu.edu/~sandholm/nogoodsForMip.techReport06.pdf\n\n[3] http://www.cs.cornell.edu/selman/papers/pdf/98.aaai.boost.pdf\n\nWEAKNESSES:\nOne problematic omission in this paper is that nowhere it is said that MIP solvers also use branch-and-bound. Otherwise, it sounds as if MIP solvers are some sort of black box magic that tries to solve problems in some inefficient way. There is also some broad generalizations such as \"MIP solver may not return any cuts before timeout\" (149): Which solver? When? Under which conditions? All that a generic MIP solver needs to generate a Gomory cut is the solution of the LP relaxation and the corresponding tableau. \n\nThe fact that branches are defined in terms of neuron outputs is also problematic because there is an overlap between the two resulting subproblems when x=0. The effectiveness of branch-and-bound comes from partitioning the feasible set, and moreover from slicing off the parts of the LP relaxation that are not feasible according to the MIP formulation (such as when z=0 is one subproblem, z=1 is another subproblem, and 0<z<1 gets thrown away). Because your implementation is extending someone else's work, I believe that this might be a mismatch between the code and what the paper describes. Since the z variables are nevertheless a way to parameterize the LP problems to be solved along the way, there would be no problem in defining the constraints directly in terms of z instead of x. In fact, this is exactly what you do when you finally formalize the cut that you are using.\n\nWhen it gets to Proposition 3.1, the statement is unnecessarily complex: why define a diagonal matrix only to multiply it by a vector of ones immediately after? In terms of notation, it doesn't make sense to have output variables indexing which neurons should be removed. Instead, you could use only the corresponding indices. Hence, Z = {1, 3} instead of Z = {x_1, x_3}. Moreover, this is a commonly known cutting plane in MIP, perhaps first introduced in 1972 by [4] and at this point widely known and applied in the MIP community. It would have been better to use the conventional notation $\\sum_{i \\in Z^+} z_i - \\sum_{i \\in Z^-} z_i \\leq |Z^+| - 1$ directly in the paper rather than using it in the appendix and keeping a less readable version in the paper.\n\n[4] https://epubs.siam.org/doi/10.1137/0123007\n\nAlgorithm 1 relies on a threshold, but values or experiments to find the right one are not mentioned anywhere.\n\nThe paper also talks very briefly about prior work, to the point that it sounds as if specialized cutting planes for MIP problems involving neural networks were never proposed before. However, your reference 1 (Anderson et al.) does exactly that. The same is true about [5-6], and a broader perspective about this area can be found in Section 4 of the survey [7].\n\n[5] https://arxiv.org/abs/1810.03370\n\n[6] https://arxiv.org/abs/2102.04373\n\n[7] https://arxiv.org/abs/2305.00241\n\nOther major comments about the writing follow below.\n\nA) The \"The NN Verification Problem\" paragraph in Section 2 is not very precise because it confuses values with variables and do not properly qualify functions when values are dependent on inputs. What follows is a possible correction to that. In line 84, replace \"scalar values\" with \"scalar variables\". In line 85, add \"the values of\" after \"for\". In line 86, replace \", when they depend on a specific x\" with \"for a given input x\". In line 87, add to \"that limit [the post-activation values of] individual neurons\".\n\nB) The text talks about an example in which you no longer want to see both neurons 1 and 3 inactive at the same time, but the example in Figure 1 shows neuron 3 active.\n\nOther minor comments about writing follow below.\n\n10-11: \"cutting planes constraints\": this is not an adequate expression (all constraints define a plane); use either cutting planes, valid inequalities, constraints, or cuts. \n\n21-22: First sentence is a strong statement with no reference provided and never discussed again.\n\n32: \"statuses\"   ->   states (again in 53)\n\n67: \"discussed\"   ->   discuss (not past tense)\n\n98: \"> 0\"   ->   < 0?\n\n98: \"pro[p]erty\"\n\n125: \"status\"   ->   state\n\n139: remove \"BIC\"\n\n149: \"and [the] MIP solver\"\n\n162: \"positive regime\": active regime?\n\n163: \"in [the] LP formulation\"\n\nAuthors repeated twice in reference 11.\n\nReferences 13 and 14 are the same.\n\nUse {CDCL} instead of CDCL in reference 28 and {SAT} instead of SAT in reference 30 to keep the casing in the output, since otherwise what you get is cdcl and sat.\n\nQUESTIONS:\n1) In light of the comments about nogood learning and restarts, can the authors please reframe their contribution in the paper in terms of the existing related work?\n\n2) Can you please qualify your discussion about MIP solvers by being more precise in what they can and cannot do?\n\n3) Are you branching on the output of the neurons (x) , or on the state of the neurons (z)?\n\n4) Can you simplify Proposition 3.1?\n\n5) Can you please describe what thresholds were used for Algorithm 1, and how they were obtained?\n\n6) Can you please rectify the discussion about cutting planes for neural networks by acknowledging prior work on this?\n\n7) Can you please comment on incorporating the major and minor corrections to writing?\n\n8) In lines 145-146, what do you mean by \"[49] solved these cuts using GPU-accelerated bound propagation\"? What is the concept of solving a cut? Do you mean generating the cut? Do you mean solving the LP relaxation after the cuts are included?\n\nLIMITATIONS:\nThe authors were very upfront about limitations to their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents BICCOS: a method to derive cutting planes for use within a state-of-the-art neural network verification framework based on branch and bound (BaB). Given verified (UNSAT) subproblems, BICCOS tries to find a subset of the employed branching choices that led to the verification result, and applies a cutting plane that prunes this subtree from the rest of the BaB procedure. Engineering improvements (going over multiple BaB tree and branching choices in parallel as a pre-processing step) are also presented. The experimental results suggest moderate improvements upon the state-of-the-art over the considered benchmarks.\n\nSTRENGTHS:\nThe idea behind BICCOS is fairly simple, yet relatively novel in the context of neural network verification. Given the additional overhead linked to the \"strengthening\" procedure (recomputing bounds after removing branching decisions), which is required for the overall algorithm, one may think that the overall approach may not pay off. The experiments show that it does, although somewhat marginally, I would believe.\n\nWEAKNESSES:\n**Presentation.**\nThe paper feels quite rushed, and the quality of the presentation definitely needs to improve to meet the NeurIPS bar. \nThe figures are fairly small (especially Figure 2) and fairly hard to read on paper. I would suggest that the authors remove the shadows too, which make things harder. The Tables are also fairly hard to read on paper. The text still has some typos (e.g., \"BICwhere\" in line 139). The example from Figure 1 does not correspond to the text in lines 160-166 (x_3 >= 0 vs x <= 0) or to Figure 2a. In lines 308-310 the text suggests that the comprehensive BICCOS configuration performs the best in all cases: this is not what appears from Table 3.\n\n**Feasibility.**\nThis is linked to the presentation but it's important enough to stand as a separate point. Page 4 repeatedly speaks of infeasibility in a context where I think it's technically incorrect. I think that the fact that a subproblem lower bound is positive does not imply infeasibility: it could very well be that both its lower and upper bounds are positive (with UB > LB), simply suggesting that the minimum for that subproblem (but of course not necessarily for the original problem) is positive. This means that the counter-example search is infeasible, but not the variable assignment (the series of branching decisions). In order to prove infeasibility of the subproblem, one would need to either show that the local UB is smaller than the subproblem LB, or show that the subproblem LB would go to infinity in the limit for iterations (that is, the underlying dual problem is unbounded). All the arguments being made still apply even for feasible yet verified subproblems: the goal is simply to exclude BaB subregions which we know already will lead to positive lower bounds (hence pruning the tree). But this terminology should be adapted to avoid any confusions.\n\n**Results.**\nWhile the fact that the proposed approach works is interesting (see *strengths* above), I do not think the presented experimental results are particularly impressive. Most of the improvements over GCP-CROWN (or Beta-CROWN, when GCP-CROWN can't be applied because of scalability issues) are fairly small. Furthermore, more granularity in the results would be needed (see questions). While this should not be a problem for acceptance on its own, I think it is when combined with the presentation issues above.\n\n--------\n\n**Post-discussion.** I am increasing my score to 6 following the discussion with the authors. I encourage them to acknowledge the shortcomings of the proposed approach in the next version, and to improve the presentation as discussed.\n\nQUESTIONS:\n1) It is repeatedly claimed that the procedure is specific to NN verification, but I think such an approach would apply more generally to any BaB procedure (finding subsets of branching decisions that led to a positive verification result and using that as cutting plane), or at least on any BaB procedure for MILPs. Could you please elaborate on this?\n2) As commonly done in previous work (for instance, Beta-CROWN), plots showing the number of verified properties within a given runtime are needed to fully assess the trade-offs associated to the proposed approach. For instance, how much does it slow verification down on easier properties?\n3) It seems to me that Table 2 reports the best configuration across those in Table 3, for each BICCOS row. Could the authors clarify this?\n\nLIMITATIONS:\nLimitations are appropriately addressed in the conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends GCP-CROWN, an existing toolkit for the verification of neural networks which is based on GPU-accelerated bound propagation combined with a branch-and-bound (BaB) approach. The strength of the existing algorithm is its ability to incorporate cutting planes into the bound propagation process. GCP-CROWN uses cutting planes generated by a Mixed Integer Linear Programming (MILP) solver which is run in parallel to the bound propagation, however, MILP solvers generally do not scale to large problems and only generate generic, problem-independent cutting planes.\nThe authors propose a new approach to generate cutting planes called BICCOS which works by exploiting information from verified branches in the BaB tree. Once a branch is verified, the idea is to remove a number of constraints from the branch in an attempt to obtain a subset of constraints that is sufficient for obtaining a \"verified\" result. If successful, a cut which is valid for all other branches in the BaB tree can be generated based on these constraints. The generation of cuts is run during the normal verification process, but the authors propose adding a presolve step that initially generates multiple shallow BaB trees in an attempt to create a pool of cuts before starting the standard BaB phase from GCP-CROWN.\nThe experimental evaluation shows that BICCOS scales well, is able to generate cuts for problems that the MILP solver employed in GCP-CROWN can't scale to, and outperforms many competing tools.\n\nSTRENGTHS:\n- Neural Network Verification is a relevant research topic\n- A cut generation method that scales to larger networks as well as cuts that are problem-specific and less generic than those generated by a MILP solver are useful contributions.\n- The method outperforms most other toolkits in the experimental evaluation\n\nWEAKNESSES:\n- The work is somewhat incremental compared to GCP-CROWN\n- When comparing the performance of GCP-CROWN and BICCOS (base) in Table 3, this seems to indicate that the newly introduced cuts are weaker than the MILP cuts. Including presolve (BICCOS(with multi-tree)) improves performance compared to GCP-CROWN in only some instances. It's good to have a cut generation method which scales to larger networks, but the method would be a lot stronger if the BICCOS cuts alone outperformed the generic MILP cuts.\n- A comparison with existing cuts, such as the ones from [1], is missing. In the related work section the authors only state that Venus (which implements the cuts from [1]) delivers weaker empirical results than their approach. However, the contribution of this paper are the new cuts and comparing the BICCOS cuts in a GPU-enabled bound propagation framework to the cuts by Botoeva et al. implemented in a MILP-based verifier (which can't make use of GPU acceleration) is not fair since it is well-known in the literature that bound propagation frameworks outperform MILP verifiers. To assess whether the BICCOS cuts are more effective than the cuts in [1] they should be implemented in the same general framework. Without this comparison it is hard to judge the contribution of this work.\n- Appendix A2: This part of the appendix is either very unclear or has a lot of typos. There are a lot of expressions like $\\sum_{i \\in \\mathcal{Z}^+} z_i + \\sum_{j \\in \\mathcal{Z}^-} (1 - z_i)$. Do the authors mean to write $\\sum_{i \\in \\mathcal{Z}^+} \\left ( z_i + \\sum_{j \\in \\mathcal{Z}^-} (1 - z_i) \\right )$? If so, the extra set of brackets should be added. If the authors actually do mean to write $\\sum_{i \\in \\mathcal{Z}^+} \\left ( z_i \\right ) + \\sum_{j \\in \\mathcal{Z}^-} (1 - z_i)$ then the $z_i$ in the second sum makes no sense, should this be $z_j$ then? This unclarity/mistake appears in the two equations between line 516 and 517 (which aren't labeled so I can't refer to them), in line 518, in the first equation between line 518 and line 519 and the second equation between line 518 and 519 (these also aren't labeled so I can't refer to them directly).\n\n### Minor points\n- Line 10-11: cutting planes constraints --> cutting **plane** constraints\n- Line 60: proposed --> **propose**\n- Line 67: discussed --> **discuss**\n- Line 98: The paper states \"If $f^* > 0$, it is unclear whether the property might hold\" --> Shouldn't this be \"If $f^* < 0$\" (i.e. the inequality being flipped?)\n- Line 98: proerty --> pro**p**erty\n- Line 109: Remove \"the\" (sentence should be \"most existing NN verifiers use cheaper methods such as (...)\")\n- Line 139: Remove **BIC** at the beginning of the line\n- Line 143: they --> **the authors**\n- Figure 2a): these subproblems already includes the constraint --> these subproblems already **include** the constraint\n- Line 194: along --> **alone**\n- Line 198-199: using as fewer variables as possible --> using as **few** variables as possible\n- Line 202: we performs a re-verification step, where it recomputes the lower bound --> we **perform** a re-verification step **which** recomputes the lower bound\n- Line 208: we propose --> **W**e propose\n- Table 3: For MNIST, CNN-A-Adv the \"Ver%\" for BICCOS (with MIP cuts) is $0.71$. Is this a typo, what is the correct number?\n- Appendix A.2: The authors derive two equations by \"taking a negation of this equation\". I find this part a bit unclear, do they mean that if the equation holds then this is equivalent to one of the two new equations holding? Or do both of the two new equations need to hold?\n- Line 547-549 in Appendix A4: The authors write \"on the CIFAR CNN-B-Adv model, BICCOS with multi-tree search explores $2.54 \\times 10^3$ branches, significantly lower than the $1.57 \\times 10^3$ branches explored by the base BICCOS version.\" However, as far as my understanding goes, $2.54 \\times 10^3$ is not a **smaller** but a **larger** number than $1.57 \\times 10^3$ so the sentence here makes no sense. I tried to double-check this but the same numbers are reported in the table below (I assume that \"BICCOS with multi-tree search\" in the text is the same as \"BICCOS (with Presolve)\" in the table). Could the authors clarify what their point is here?\n\n### References\n[1] Botoeva, E., Kouvaros, P., Kronqvist, J., Lomuscio, A. & Misener, R. (2020) Efficient Verification of ReLU-Based Neural Networks via Dependency Analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence. 3 April 2020 pp. 3291–3299. doi:10.1609/aaai.v34i04.5729.\n\nQUESTIONS:\n- Table 3: Why does BICCOS with multi-tree perform worse than BICCOS (base) for CNN-A-Adv-4, do the authors have an explanation for this?\n- Why is BICCOS run with a shorter timeout compared to other algorithms? Also why is MN-BaB run with a 600s timeout and then $\\beta$-CROWN, GCP-CROWN and BICCOS are run with a 200s timeout in Table 2, but then in Table 3 the footnote seems to suggest that BICCOS is run with a 200s timeout while $\\beta$-CROWN and GCP-CROWN use a longer timeout? This seems inconsistent. The experiments would be more informative if all algorithms were run with the same time budget as is usual practice e.g. in VNNComp.\n- In Table 1/2 does BICCOS use cuts from a MILP solver (if the solver scales to the problem) or only the newly introduced cuts?\n- Table 3: For CNN-A-Mix-4 the BICCOS-MIP approach has a verified accuracy of 56.5% but BICCOS-all has 56%. What is the authors' intuition here regarding why adding the multi-tree approach worsens performance, do they think this is an issue/can be avoided?\n- Line 302-310: Could the authors clarify what each variant of the algorithm includes here? The text makes it sound a bit like the authors start from BICCOS (base) and then gradually add other components, but does BICCOS (with multi-tree) also include MIP cuts? If so, the performance drop from 52 to 51.5% on e.g. CIFAR CNN-B-Adv would be surprising, any explanations?\n\nLIMITATIONS:\nLimitations are sufficiently addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new approach to produce cutting planes in the context of branch-and-bound-based solvers for neural network verification. Whenever an infeasible subproblem is encountered in branch-and-bound, this method generates a cut from the conflicting assignment that led to the infeasible subproblem (initially redundant w.r.t. the remainder of the tree), and attempts to strengthen this cut by heuristically dropping some of the assignments and rechecking for infeasibility via the lower bound. This is further enhanced by using several parallel shallow trees to produce stronger cuts. This method is implemented on top of the $\\alpha,\\beta$-CROWN framework and provides meaningful computational improvements compared to various baselines on a set of benchmarks.\n\nSTRENGTHS:\nThis paper provides a solid contribution to the area of neural network verification methods. It builds on top of cut-based branch-and-bound verification solvers by presenting a method to quickly infer cuts, which appear to be novel and computationally useful. In particular, they nicely leverage the fact that there are fast methods to produce bounds in NN verification, allowing us to quickly recheck cut validity. This makes for a clean and simple method, which has the advantage of not being too complicated to integrate with an existing cut-based BaB verification solver.\n\nBoth the set of benchmarks and baselines are reasonably extensive, and we can observe improvements in verifiability that are sufficient for a meaningful computational contribution, especially in the CIFAR instances in both VNN-COMP and SDP-FO, without much additional cost in computational time.  The paper is overall clearly written and the figures and algorithms are helpful.\n\nWEAKNESSES:\nIn some of the benchmarks, the computational results may be somewhat incremental, but overall they are positive. There is potential room for improvement in parts of the methods (see Questions section below); in particular, it is not clear if the authors have explored variations of their constraint strengthening approach. In general, I do not see major weaknesses in this paper, though minor concerns are expanded on below.\n\nQUESTIONS:\nGeneral comments:\n\n1. I see that your variable-to-drop selection heuristic is based on their improvements to the lower bound in the tree. While this seems reasonable as a fast heuristic since you already have all the data, these improvements are not independent from each other since they are constrained over previous assignments, and thus there is some bias depending on the depth (i.e. if the assignments were done in a different order in the tree, you'd select different variables, but the constraint is the same). This makes me wonder if there is a better heuristic. Have you considered other approaches?\n\n2. It seems that you try to continue strengthening the cuts based on a fixed drop percentage. I am curious if you have tried something more like a binary search approach over the verification bound? You can also add some sort of tolerance on the bound to stop searching when you are close enough to zero.\n\n3. In Algorithm 1, lines 12-13, you add the cut, and then try to strengthen it again. It sounds like you could just add the best cut here, instead of adding all cuts throughout strengthening, since the best cut dominates the other ones.\n\n4. I appreciate the explanation of the differences between this method and [7] and DPLL/CDCL in the Related Work section, as both of these were in my mind as I was reading the paper. However, I'd like to comment that the reasoning for CDCL makes it sounds like it is impossible in practice to use learned clauses from CDCL as cuts; rather, I believe this is more of a challenging engineering task than not being practically viable. Much like branch-and-bound has been customized for NN verification in a more effective way than generic MIP solvers, I do not see a reason why one would not be able to customize ideas from CDCL for your cut generation procedure. The learned conflicts can be naturally translated to the type of cuts you have in Sec. 3.1 of this paper (except already stronger) and then further strengthened in the same way. Given that CDCL is a tried-and-true method to produce good conflicts, I suspect that this might lead to better cuts and may be interesting future work that would already fit very well with what you have so far.\n\n5. I would have liked to see some data to better understand the cuts generated. In particular, what is the fraction of infeasible nodes from which you were able to produce a (non-redundant) cut, and what was the total number of cuts? What was the average number of assignments that you were able to drop? This sort of data would reveal more information on the overhead of these cuts and how easy or hard it is to find them. I wonder if some cut selection procedure would make sense here, but I do not know if you have many or few cuts.\n\nComments on text:\n\n6. In the MIP section, it would be useful to mention that MIP is also based on branch-and-bound, and modify the last sentence to include why a custom branch-and-bound is more effective than MIP branch-and-bound in practice (e.g. because MIP is based on solving LPs which can be expensive, etc.).\n\n7. In Sec 3.1., I suggest preparing the reader to the fact that the cuts from Sec. 3.1 are redundant w.r.t. the tree until strengthened, instead of waiting until Sec. 3.2 to mention that. This is a question one would naturally have while reading Sec. 3.1, and it would make the reading easier if they already know the answer to that question.\n\n8. Figure 2 is a bit too small to read especially when printed out. If possible, please make it larger.\n\n9. Can you include in the text that your bound computation when strengthening a cut includes the global set of cuts? I see it in Algorithm 1, but I didn't see it in the text.\n\n10. Could you include exactly how the trees differ from each other in the multi-tree approach? The text mentions that they explore a different set of branching decisions, but not how.\n\n11. Could you expand on which methods/baselines use GPUs, and how they are used? This is important for a proper comparison.\n\n12. Typos: Remove \"BIC\" in line 139, capitalize \"we\" in line 208, \"instances\" and double period in caption of Table 4.\n\nLIMITATIONS:\nThe Limitations section is reasonable, covering cases where this approach does not work well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}