Sound and Complete Verification of Polynomial Networks
Authors: Elias Abad Rocamora
Mehmet Fatih Sahin
Fanghui Liu
Grigorios Chrysos
Volkan Cevher
Venue: NeurIPS 2022 Accept (2022)
Keywords: branch and bound
adversarial robustness
adversarial examples
certified robustness
polynomial network verification

Abstract: Polynomial Networks (PNs) have demonstrated promising performance on face and image recognition recently. However, robustness of PNs is unclear and thus obtaining certificates becomes imperative for enabling their adoption in real-world applications. Existing verification algorithms on ReLU neural networks (NNs) based on classical branch and bound (BaB) techniques cannot be trivially applied to PN verification. In this work, we devise a new bounding method, equipped with BaB for global convergence guarantees, called Verification of Polynomial Networks or VPN for short. One key insight is that we obtain much tighter bounds than the interval bound propagation (IBP) and DeepT-Fast [Bonaert et al., 2021] baselines. This enables sound and complete PN verification with empirical validation on MNIST, CIFAR10 and STL10 datasets. We believe our method has its own interest to NN verification. The source code is publicly available at https://github.com/megaelius/PNVerification.

Sound and Complete Verification of Polynomial Networks
Elias Abad Rocamora" LIONS, EPFL Lausanne, Switzerland abad. el wlwas0l0gmaal..ccmmmmmmmmmmmm com
Mehmet Fatih Sahin LIONS, EPFL Lausanne, Switzerland mehmet. swwin@ephh.cccmmm ch
Fanghui Liu LIONS, EPFL
Lausanne, Switzerland fanghui. wwwluepfl..... ch
Grigorios G Chrysos LIONS, EPFL
Volkan Cevher LIONS, EPFL
Lausanne, Switzerland
Lausanne, Switzerland
grigorios chrysos@epfl.ch
volkan. wewiaan.eeeee@@ppl.coom ch
Abstract
Polynomial Networks (PNs) have demonstrated promising performance on face and image recognition recently. However, robustness of PNs is unclear and thus obtaining certificates becomes imperative for enabling their adoption real-world applications. Existing verification algorithms ReLU neural networks (NNs) based on classical branch and bound (BaB) techniques cannot be trivially applied PN verification. In this work, devise new bounding method, equipped with BaB for global convergence guarantees, called Verification of Polynomial Networks VPN for short. One key insight is that obtain much tighter bounds than the interval bound propagation (IBP) and DeepT-Fast [Bonaert et al. 2021] baselines. This enables sound and complete PN verification with empirical validation on MNIST, CIFARIO and STL10 datasets. We believe our method has its own interest NN verification. The source code is publicly available https: htw.eh/....mmmmmeeeese/////eeeee
Introduction
Polynomial Networks (PNs) have demonstrated promising performance across image recognition and generation [Chyysos et al. 2021 Chrysos and Panagakis 2020) being state-of-the-ar on large-scale face recognition Unlike the conventional Neural Networks (NNs). where non- mineeriir is introduced with the use of activation functions [LeCun et al.) 2015). PNs are able to learn non-linear mappings without the need of activation functions by exploiting multiplicative interactions (Hadamard products). Recent works have uncovered interesting properties of PNs in terms of model expressivity (Fan et al.) 202T] and spectral bias [Choraria et al. 2022). However, one critical issue before considering PNs for real-world applications their robustness.
Neural networks are prone small (often imperceptible to the human eye), but malicious perturbations in the input data points (Szegedy et al. 2014 Goodfellow et al. 2015). Those perturbations can have detrimental effect on image recognition systems, e.g., as illustrated in face recognition [Goswami
et al.) 2019 Zhong and Deng] 2019 Dong et al. 2019 Li et al. 2020]. Guarding against such attacks has so far proven futile (Shaahii et 2019 Dou al. 2018 Instead, flurry of research
'Work developed during an exchange coming from Universitat Politècnica Catalunya (UPC), Spain.
Currently at Universidad Carlos de Madrid (UC3M).
htcpps:/ppaprrsstthcoode wttpeeeecoooe/tteeeeeeeeeeeeeeee http://ccc..eeeiiiaattatooooooo.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

has been published on certifying robustness of NNs against this performance degradation [Katz
al. 2017 Ehlers 2017 Tjeng et al. 2019 Bunel et al. 2020b Wang al., 2021 Ferrari et al.
2022
However, most of the verification algorithms for NNs are developed for the ReLU activation
function by exploiting its piecewise linearity property and might not trivially extend to other nonlinear
activation functions [Wang et al.) 2021]. Indeed, [Zhu et al. [2022) illustrate that guarding PNs against adversarial attacks is challenging. Therefore, we pose the following question:
Can obtain certifiable performance for PNs against adversarial attacks?
In this work, we answer affirmatively and provide method for the verification of PNs. Concretely, we take advantage of the wiice-differentiable nature of PNs to build lower bounding method based on a-convexification [Adjiman and Floudas 1996), which is integrated into a Branch and Bound (BaB) algorithm [Land and Doig 1960] to guarantee completeness of our verification method. In order to use e-convexification, a lower bound a of the minimum eigenvalue of the Hessian matrix over the possible perturbation set is needed. We use interval bound propagation together with the theoretical properties of the lower bounding Hessian matrix [Adjiman et al., 1998]. in order to develop an algorithm to efficiently compute a.
Our contributions can be summarized as follows: (i) We propose the first algorithm for the verifi- cation of PNs. (ti) We thoroughly analyze the performance of our method by comparing with black-box solver, with an interval bound propagation (IBP) BaB algorithm and with the zonotope- based abstraction method DeepT-Fast [Bonaert et 2021]. (zii) We empirically show that using a-convexification for lower bounding provides tighter bounds than IBP and Deep'T-Fast for PN verifi- cation. To encourage the community improve the verification of PNs, we make our code publicly available in htttps://githn
com/ hempmmeaellln liteeellluuue wwWeeriiiii... cirriccatioon, The proposed approach can practically verify PNs, while it could also theoretically applied for sound and complete verification of any twice-differentiable network.
Notation: We use the shorthand [n] = 21, n) for positive integer n. We use bold capital (lowercase) letters, c.g., X (æ) for representing matrices (vectors). The 7 column of matrix X is given by æ.j. The element in the ith row and , column given by zij. similarly, the 7t element of vector æ is given by 24. The element-wise (Hadamard) product, symbolized with x, of two matrices (or vectors) in R xd2 (or IRd) gives another matrix (or vector) in R xds (or RR). The fo norm of a vector æ E Rd is given by: || = maxie(d [ Lastly, the operators L and L/ give the lower and upper bounds of a scalar, vector or matrix function by IBP, see Section 3.1
We provide the necessary background by introducing the PN architecture and formalizing the Robustness Verification problem in Section Section 3 provides sound and complete method called VPN to tackle PN verification problem. Section is devoted to experimental validation. Additional experiments, details and proof's are deferred the appendix.
Roadmap:
2 Background
We give an overview of the PN architecture in Section 2.1 and the robustness verification problem in Section
2.1 Polynomial Networks (PNs)
Polynomial Networks (PNs) are inspired by the fact that any smooth function can be approximated via a polynomial expansion [Stone 1948). However, the number of parameters increases exponentially with the polynomial degree, which makes it intractable to use high degree polynomials for high- dimensional data problems such as image classification where the input can be in the order of 10 (Deng al. 2009]. Chhyysos al. [202T] introduce a joint factorization of polynomial coefficients in a low-rank manner, reducing the number of parameters to linear with the polynomial degree and allowing the expression neural network (NN). We briefly recap one fundamental factorization below.
Let N be the polynomial degree, E d be the input vector, d, k and be the input, hidden and output sizes, respectively. The recursive equation of PNs can be expressed as:
x (n) (W [1] (a-1) 1) + x (n-1) Vn E [N].
(I)

---

W 2
W S
3
(3)
W11 [1
C
f(z)
Figure 1: Third degree PN architecture. Blue boxes depict learnable parameters, yellow depict mathematical operations, the green and red boxes are the input and the output respectively. Note that no activation functions are involved, only element-wise (Hadamard) products and additions +. This figure represents the recursive formula of Eq.
where x: W Wl| f() + and denotes the Hadamard product. Wm E Bdxk and Ce Bo are weight matrices, 3 E R is a bias vector. A graphical representation of third degree PN architecture corresponding to Eq. {I) can be found in Fig. [ Further details on the factorization (as well as other factorizations) are deferred to the Appendix (Appendix C.2
2.2 Robustness Verification
Robustness verification [Bastani et al. 2016 Liu al. 2021 consists of verifying that property regarding the input and output of a NN is satisfied, e.g. checking whether or not small perturbation in the input will produce change in the network output that makes it classify the input into another class. Let f [0, 1j IR be a function, e.g., NN or a PN, that classifies the input z into a class c, such that arg max f(=). Our target is to verify that for any input satisfying set of constraints Cin. the output of the network will satisfy set of output constraints Cout- Mathematically.
Cin
f(z) E Cout-
In this work we focus on adversarial robustness (Szegedy et al., 2014 Carlini and Wagner 2017] in classification. Given observation Ep, let arg max f(z)) be the correct class, goal is to check whether every input neighbourhood of zo. classified as f. In this work, we focus on adversarial attacks restricted to neighbourhoods defined in terms of &go norm, which is popular norm-bounded attack in the verification community [Liu et al. 2021). Then, the constraint sets become:
Cin (= || =0 e, E [) (d]
= (z max (0, z0: - -e} S x S min zos + +cJ, Vi e
(3)
Cout (y 91 カy Vj t).
In other words, we need an algorithm that given function f. an input zo and an adversarial budget checks whether Eq. (2) is satisfied. In the case of ReLU NNs, this has been proven to be an constrained optimization
NP-complete problem [Katz al. 2017). This can be reformulated problem. For every adversarial class 7 F f = arg max f(zu), we can solve:
g(z) f(z), - f(z), s.t. 2 E Cin min
the solution x" with v² = f(x)) < f(z); f(z),, Vz E Cin satisfics robustness is verified for the adversarial class 7.
> 0 then
There are two main properties that a verification algorithm admits: soundness and completeness, An algorithm is sound (complete) if every time it verifies (falsifies) a property, it is guaranteed to be the correct answer. In practice, when an algorithm guaranteed to provide the exact global minima of Eq. (4) i.e., v², is said to be sound and complete (usually referred in the literature as simply complete Ferrari al. 2022), ), whereas if lower bound of it is provided sound but not complete. In our work, we will not consider just complete verification, which simply aims at looking for adversarial examples, e.g., Madry al. (018). For a deeper discussion on soundness and completeness, refer to Liu et al. 2021
< v", the algorithm

---

3 Method
Our method, called VPN, be categorized in the the Branch and Bound (BaB) framework [Land and Doig, 1960], a well known approach to global optimization [Horst and Tuy 1996] and NN verification [uneel et al. 2020b]. This kind of algorithms ensures finding a global minima of the problem in Eq. 4 by recursively splitting the original feasible set into smaller sets (branching) where upper and lower bounds of the global minima computed (bounding). This mechanism can be used to discard subsets where the global minima cannot be achieved (its lower bound is greater than the upper bound of another subset).
Our method is based on a variant of BaB algorithm. i.c., a- G-BaB [Adjiman et al., 1998]. which is characterized for using a-convexification [Adjiman and Floudas [996] for computing a lower bound of the global minima of each subset. To be specific, a-convexificattin aims to obtain a convex lower bounding function of any wwice-differentiable function IR R. In Adjiman and Floudas 1996 they propose two methods:
Uniform diagonal shift (single (x)
Ia ,, g(2) E(= 1,)(z; ui); i=1
(5)
is its a-convexified version, note that 24 is the th element of vector z. Let (z)z) V2 V2=((2 be the Hessian matrix of 9. Ia is convex in [1, for a 2 max - 2 min { Amia (=) : E [1,,)}, where Amia is the minimum eigenvalue. Moreover, it holds that (a(2;; at u) S g(z)), E u..
Non-uniform diagonal shift (multiple a's)
Ja a = g + Σ ass2 - 1,)(z - uz),
(6)
i=1
is its a-convexified version. In Adjiman al. setting
1998]. they show that for any vector d > 0,
1 [(hg(z)a)) - Σ max ((h, 2
dj d,
05 max
0,
makes Ja convex in E [7, u.. The choice of the vector d is arbitrary. but affects the final result. For example, taking d = u - yields better results than d 1 in AAdjiman et al. [1998] We need to remark that, though Eq. (5) is special case of Eq. (6). the lower bound of a in Eq. (5) via minimum eigenvalue of the lower bounding Hessian matrix cannot be regarded special case of Eq.
To make PN verification feasible via a-convexification, we need to study IBP for PNs and design an efficient estimate on the a (a in the case of Non-uniform diagonal shift) parameter, which are our main technical contributions in the algorithmic aspect. our case, every feasible set, starting with the input set C,n (Eq. (3). D, is split by taking the widest variable interval and dividing it in two by the middle point. This rather simple, but theoretically powerful strategy, see Lemma g in Appendix F Then, the upper bound of each subproblem is given by applying standard Projected Gradient Descent (PGD) [Kelley 1999) over the original objective function. This is a common approach to find adversarial examples Madry et al.] 2018), but as the objective is non-convex, it is not sufficient for sound and complete verification. The lower bound is given by applying PGD over the a-convexified objective ga (ga), it is convex, PGD converges to the global minima and a lower bound of the original objective. The a (a) parameter computed only once per verification problem. Further details on the algorithm and the proof of convergence of Eq. (4) exist in Appendix D A schematic of our method is available in Fig. 2
In Sections 3.1 to 3.3 we detail our method under the uniform diagonal shift case to compute a lower bound on the minimum eigenvalue of the Hessian matrix into three main components: interval propagation, lower bounding Hessians, and fast estimation on such lower bounding via power method. To conclude the description of our method, in Section 3.4 we describe the a estimation for the non-uniform diagonal shift case.

---

Branch
min g(z)
S branch C
PGD
C1,C2t
Input set Cin
C
0
Subsets emply
Subsets
Check bounds
Verified
True class
Adversarial class
24
PGD
minlged (o(=)
Falsified
Compute (Sections 3.2 und
Figure 2: Overview of our branch and bound verification algorithm. Given a trained PN f, an input set Cin, the true class t and an adversarial class we check if an adversarial example exists (Falsified) or not (Verified). Note that the branching of a subset C' provides two smaller subsets C₁ and Cz. Also note that when L >> 0, no subset is added to subsets.
3.1 Interval Bound Propagation through PN
Interval bound propagation (IBP) is key ingredient of our verification algorithm. Suppose we have an input set defined by an -nomm ball like in Eq. (3). This can be represented as vector of intervals [t, u] (l1,, ui] [2,, w2] ... [la ux] ) E I dx2 where [4, uz] are the lower and upper bound for the 7th coordinate. Let L and U be the lower and upper bound IBP operators. Given this input set, we would like to obtain bounds on the output of the network (f(z)a the gradient (V=f(z))) and the Hessian (V2=f(=) for any z E [L, u]. The operators L(g(z)) and ((g())) of any function g Rd -> IR satisfy:
C(g(z)) S g(x), U(g(z)) 2 g() ul.
We will define these upper and lower bound operators in terms of the operations present in a PN. Using interval propagation [Moore al. 2009), denote the positive part wtt max w:} and the negative part w min 00 w. h2(2) real-valued function of 2, E [d]. we define:
[(x)) = 4
Identity
(U(z)) = 14
2 Σ wx)((()) = Σ ip L(ha(z)) + w U(hx(z))
linear mapping
u
wrhi(=)) = Σ we [(h((z)) wT u(h((z))
(9)
((h)(z)) ((h1(22);
((h) (z))2((2((=)))
=
.
u(h)(z)) ((h2(z)),
multiplication
(()( ((h(()))2())) min 5,
(z)(22)))(2) max S
where - is the set cardinality. Note that the set S is equivalent to:
S = {ab] Va (C()) (=))) Ab (((he())),
With these basic operations, one can define bounds on any intermediate output, gradient or Hessian of PN. For instance, the lower bound the recursive formula from Eq. (I) can be expressed as:
[(((")) C((wT))22) (n-1)
C(( + 1)=("-1)), Vi [k], (2....,NN N). x; (n-11))
(10)
5

---

which only consists on linear mapping and multiplication of intervals. We extend the upper and lower bound (C(-) and (())) operators to vectors and matrices an entry-wise style:
((g(2)))
((g(2)n)
L(g(z)im)
((g(z)2)
[(g(z))
E ERT =(G(z))=
Rmxx
(()())ma)
(lg())mm)
(()
(11)
Note that Eq. 11) is not limited to squared matrices and can hold for arbitrary matrix dimensions. One can directly use IBP obtain bounds on the verification objective from Eq. ( with a single forward pass of the bounds through the network and obtaining L(g(z)) = [(f(z); U(f(2))) In fact IBP is a common practice in NN verification to obtain fast bounds [Wang et al. 2018a].
3.2 Lower bound of the minimum eigenvalue of the Hessian
Here describe method to compute a lower bound on the minimum eigenvalue of the Hessian matrix in the feasible set. Before deriving the lower bound, we need the first and second order partial derivatives of PNs.
Let g(z) = f()) - a. the objective function for = arg max fzz) and any a * f. order to compute the parameter for performing a-convexification, we need to know the structure of our objective function. In this section we compute the first and second order partial derivatives of the PN. The gradient and Hessian matrices of the objective function (see Eq. (4)) are given by:
A:
<=9(z)= - E(oo (cu - 22 (N), Hg(z) Σ (ou (cu- E(eu
(12)
1=1
1=1
We now define the gradients
Vzz;"" (n) and Hessians V 2 of Eq.
in recursive way:
(n) 2t
Wija x;
(n-1) (wip 2 + V Ve Ven (n-1)
(13)
V (n) V
2
1) T
[n]s
(V V
D mnja) + (n)a 1)V2
(n-1)
(14)
with - (1) wi) and V2 22 (1) Oaxd being all-zero matrix. In the next, we are ready to compute a lower bound on the minimum eigenvalue of the Hessian matrix in the feasible set.
Firstly, for any z E [t, us] and any polynomial degree N, we can express the set of possible Hessians
H. = (H (z) = E [1, u]} as an interval matrix. An interval matrix is a tensor [M] E Rdxax2
where every position [m]i = [L(m))) (, is an interval. Therefore, if H (2)z) is bounded for z E u). then we can represent H {H (2)) H Hz)z) E [M] = [ 2) L(m;) I Hz(z); S (m)m)) Ji,j j E [d].
Let L(M) and ((M) ) be the element-wise lower and upper bounds of Hessian matrix, the lower bounding Hessian defined as follows:
LH L(M) + U(M) + diag
[((M)-
((MMI-U(M) -((M)
(15)
2
2
where an all-one vector and (v) is a diagonal matrix with the vector v the diagonal. Described in Adjiman et al. 1998), this matrix satisfies that Amin (LH) S Amin (H (z)), WH, (z)
H. E [1, u].
Then, we can obtain the spectral radius pl Lx) with power method [ises and Pollaczek Pollaczek-Geiringer
1929
As the spectral radius satisfies pl p(LH)= Z (LH)), Vi E [d], the following inequality holds:
-p(LH) S Amia ( (LH) 5 Amia H (=)),, AH, (Hg
E H, z ze [.
(16)
allowing
p(Ln) 2 max(0, min Amin H (z)) [Z. ul)).
3.3 Efficient power method for spectral radius computation of the lower bounding Hessian
By using interval propagation, one can easily compute sound lower and upper bounds on each position of the Hessian matrix, compute the lower bounding Hessian and perform a power method with it to

---

obtain the spectral radius p. However, this method would not scale well to high dimensional scenarios. For instance, the STL10 dataset [Coates et al. 2011 (with input dimension in each color image, our Hessian matrix would require an order of O(d?) = O(10%) real numbers to be stored. This makes intractable to perform a power method over such an humongous matrix, or even to compute the lower bounding Hessian. Alternativlyy, we take advantage of the possibility of expressing the μ matrix as a sum of rank-1 matrices, to enable performing a power method over it.
96-96-3 27, 648)
Standard power method for spectral radius computation
Given any squared and real valued matrix M Rxdd and initial vector Uy R that is not an eigenvector M, the sequence:
M M(Mu.-i
U/
(17)
MM
converges to the eigenvector with the largest eigenvalue in absolute value, i.e. the eigenvector where the spectral radius is attained, being the spectral radius p(M) (MM [Mises and Dollaczek-Geiringe 1929].
Power method over lower bounding Hessian of PNs
We can employ IBP (Section 3.1 in order to obtain expression of the lower bounding Hessian
(L)) and evaluate Eq. (17) as:
U(H (z) v + ((H) + -((H))))) vv.
LH
(18)
2
Applying IBP on Eq. 12 obtain:
((Hg(z)) (cu gi) L(V 22 (o
(N)
(cu - cns) -(y)
U1 U( 2 2. (N))
(19)
b
k
U(H) (ou - cni) C(V 23
(oi- c,.) u V2 (
We can recursively evaluate -(V2rr1"") v and u(v axa" expressed as a sum of rank-1 matrices below. Proposition 1. Let 8 ((5) be a real-valued weight, the matrix-vector products L(ō V2 (1) you and U(5 Vi_m;"" (m)
(1) yo efficiently as these matrices can be
(m)
)u can be evaluated as:
[(6 2
(1)
(m) ju = L(6. V
(a-1)
Ju +T Injes v U(6 V2 (n-1) w TT nn:a
+ w Wnia L(6 VeN
(n-1) [2a(n-1) )u + w/j u(6 V V2ui (1) y0
(20)
+ 2 #
(n-1)) 1) )u,
u(6 (2 = L66 V
(m-1) v + u(8 V2 ( pn]is
ww +T
[n]i
+ Wn]s L(8 V
(n-1) jo Winji u(6 V (n )0 (21)
+ 2 (m-11))0 y0
where 8' E (C( (00(())))(((u) [C(W] + 1), [w x 1)] and vectors L(8 Veu(" and u(8
V=n("=1) can be obtained through IBP Eq.
13
Lastly, by applying recursively Proposition I from n N to n 1, starting with d 1. we can substitute the results on Eq. (19) and then on Eq. (8) to efficiently evaluate step of the power method (Eq. (17)) without needing store the lower bounding Hessian matrix or needing to perform expensive matrx-vectes products.
Overall, our lower bounding method consists in computing valid value of a that satisfies that the a-convexified objective g. convex, following Eqs. 4 and (5). particular, we use a p(L,)) p(LH) is computed via a power method, where the main operation L is evaluated without the need to compute or store the L matrix. Provided this valid a, we perform PGD over ga and this provides a lower bound of the global minima of Eq. (4)
2
7

---

3.4 Non-uniform diagonal shift
In order to obtain an estimate of the a parameter as defined by Adjiman al. [1998] in Eq. 7 we make use of the rank- 1 matrices IBP rules defined in Appendix F.I We also define the operator ((-) max
I [() [(()) and certain useful properties about it in Appendix F22 Thanks to Lemmas 4 to we can obtain an expression for M (H)) This will be used to compute the vector ar in the Non-uniform diagonal shift scenario for a-convexification.
Theorem 1. Let f be a N Negree CCP PN defined as in Eq. ( Let g(z)= f(2); -((2 for any t + 7, [o]. E [o], Let (z) be the Hessian matrix of g. the operation M (H) results in:
M (z) S Σ on c ( (V2)e!")
(22)
where for = 2, N, we can express:
M (V222?")) {(1))
S M V ( /wimll I + // M (V=a)
(23)
+M (wo + +1) M ( (
Lastly, for n =1 M V2 V22 (1) Oaxd is d x d matrix full of zeros.
As one can observe, in Theorem the matrix M (z) is expressed as a sum of rank-1 matrices. This allows to efficiently compute Σ ن (xhy(z)ss) di which is necessary for the right term in di
Eq. 7.. For the left term in Eq. 77) we can efficiently compute the lower bound of the diagonal of the Hessian matrix by using the rules present in Section 3.1 and Appendix F.
Experiments
In this Section we show the efficiency of our method by comparing against a simple Black- box solver. Tightness of bounds also analyzed in comparison with IBP and DeepTT-Fast (Bonaert et al.) 2021], a zonotope based verification method able to handle multiplications tighter than IBP. Finally, study of the performance of our method in different scenarios performed. Unless otherwise specified, every network is trained for 100 epochs with Stochastic Gradiend Descent (SGD), with learning rate of 0.001, which is divided by 10 at epochs [40, 60, 80], momentum 0.9, weight decay 5- 5010 and batch size 128. We thoroughly evaluate our method over the popular image classification datasets MNIST [LeCun et al. [998]. CIFARIO (Krizheskyy et al. [014] and STL [Coates et al., 2011]. Every experiment is done over the first T000 images of the test dataset, this is common practice in verification (Singh et al. 2019]. For images that are correctly classified by the network, we sequentially verify robustness against the remaining classes in decreasing order of network output. Each verification problem is given a maximum execution time of 60 seconds, we include experiments with different time limits in Appendix D Note that the execution time can be longer as execution is cut in an asynchronous way, i.e., after we finish the iteration of the BaB algorithm where the time limit is reached. All of our experiments were conducted on single GPU node equipped with 32 GB NVIDIA V100 PCIe.
4.1 Comparison with a Black-box solver
In this experiment, we compare the performance of our BaB verification algorithm with the Black- box solver Gurobi [Gurobi Optimization, L.C. 2022). Gurobi can globally solve Quadratically Constrained Quadratic Programs whether they are convex or not. As this solver cannot extend to higher degree polynomial functions, we train 2nd degree PNs with hidden size k 16 to compare the verification time of our method with Gurobi. In order to do so, we express the verification objective as Qz q + this together with the input constraints z E [7, u4] is fed to Gurobi and optimized until convergence.
a quadratic form g(z) f(=), f(z)a
The black-box solver approach neither scales to higher-dimensions inputs nor to higher polynomial degrees. With this approach we need (d2) memory to store the quadratic form, which makes unfeasible for datasets with higher resolution images than CIFAR10. On the contrary, as seen in TT our approach does not need so much memory and can scale to datasets with larger input sizes like STL10.
8

---

Table 1: Verification results for 2nd degree PNs. Columns #F, #T and #t refer to the number of images where robustness is falsified, verified and timed-out respectively. When comparing with black-box solver, our method is much faster and can scale to higher dimensional inputs. This is due our efficient exploitation of the low-rank factorization of PNs.
VPN (Our method)
Gurobi
Dataset MNIST (1 x 28 x 28)
Model
Correct
time
F
T
t.o.
time
F
T
t.O.
x 16
961
0.00725
1.76
37
921
0
16.6
37
924
0.013
1.78
71
890
0
15.13
71
890
0.05
1.43
682
267
12
6.25
691
270
0
0.06
1.5
790
155
16
4.47
799
162
CIFARIO
2 x 16
460
1/610
1.03
90
370
0
328.0
90
370
(3 x 32 32)
1/255
1.0
183
277
0
250.07
183
277
4/255
0.92
427
28
5
87.93
429
31
STL10
2 x 16
362
1/610
5.06
142
220
0
(3 x 96 96)
1/255
3.61
246
113
3
out of memory
4/255
1.39
360
1
1
4.2
Comparison with IBP and DeepT-Faast
In this experiment compare the tightness of the lower bounds provided by IBP, DeepT-Fast and a-convexification and their effectiveness when employed for verification. This is done by executing upper bounding step with PGD and one lower bounding step for each lower bounding method over the initial feasible set provided by c (see Eq. (3)). We compare the average of the distance from each lower bound to the PGD upper bound over the first 1000 images of the MNIST dataset for PNs with hidden size k = 25 and degrees ranging from 2 to 7. We also evaluate verified accuracy of 2nd 2onv2))) and 4th (PN_ 2__Cnv4)) PNs with IBP, DeepT-Fast and a-convexification in the Uniform diagonal shift setup. We employ maximum time of 120 seconds. For details on the architecture of these networks, we refer to Appendix
0.0000
01001
0.01
0.1
10
a-cony, single
10
a-cony, muki
10
IBP
Deet
3
3
6
2
3
6
7
2
3
d Dogree
Degree
Degree
Degree
Figure Average difference in log-scale between PGD upper bound (l) and lower bound (C) provided by BP (red), Deep'T-Fast [Bonaert et al. 2021 (orange), a-convexification with Uniform diagonal shift (blue) and a-convexification with Non-uniform diagonal shift (green) of the first 1000 images of the MNIST dataset. a-convexification bounds are significantly tighter than IBP and DeepT-Fast for small e values and PN degrees from to
When using IBP, we get much looser lower bound than with a-convexification, see Fig. 3 Only for high-degree, high-e combinations IBP lower bounds are closer the PGD upper bound. practice, this is not problem for verification, as for epsilons in the order of 0.1, it is really easy to find adversarial examples with PGD and there will be no accuracy left to verify. DecpT-Fast significantly outperforms IBP bounds across all degrees and values. But, observed in Fig. 3 except for big values, its performance is still far from the one provided by both a-convexification methods. When comparing both a-convexification methods (blue and green lines in Fig. 3 observe that for small degree PNs (N < 5), in the Uniform diagonal shift case we are able to obtain tighter bounds.
The looseness of the IBP lower bound is confirmed when comparing the verified accuracy with IBP and the rest of lower bounding methods, see Table With a-convexification, we are able to verify the accuracy of 2nd and thh order PNs almost exactly (almost no gap between the verified accuracy and its upper bound) every studied dataset, while with IBP, we are not able verify robustness for single image in any network-e pair, confirming the fact that IBP cannot be used

---

Table 2: Verification results with our method employing IBP, DeepT-Fastt and a-convexification for lower bounding the objective. Acc.% % is the clean accuracy of the network, Ver % is the
verified accuracy and U.B. its upper bound. When using a-convexification bounds get
verified accuracies really close to the upper bound, while when using IBP verified accuracy is 0 for every network-e pair, which makes unsuitable for PN verification.
DecpT-Fast
VPN (ours)
IBP
2021 Ver.e
(a-convexilicttoo))
Dataset
Model
Acc:%
Time(s)
Ver.
(imets)
Time(s)
Ver.
U.B.
0.015
0.3
2.3
91.3
50
96.3
96.4
MNIST
PN_Comv4
98.6
0.026
0.4
0.0
3.7
59.7
69
92.9
94.8
0.7
13.8
0.0
2N_C2
63.5
1/255
0.3
0.0
2.0
23.3
136.2
44.4
44.6
CIFARIO
2/255
0.5
0.6
1.4
89.2
25.4
27.5
PN_Cons4
62.6
2/255
1/255
0.4
0.0
0.5
2.2
19.5
0.5
224.1
274.6
45.5
16.5
30.5
46.7
STLI0"
PN_Comv4
38.1
1/255
3.4
0.0
26.0
14.7
2481.0
21.7
21.9
Results obtained in the first 360 images of the dataset due the longer running times because of the larger input size of STLI0.
for PN verification. The improvements in the bounds when utilizing Deep'T-Fast instead of IBP is clearly seen in verification results in Table 2 With FeepT-Fast, we able effectively verify PNs faster than with a-convexification, but achieving much lower verified accuracy (Ver%) than with a-convexification. As reference, for CIFAR1O CNnv2 at 2/255, with a-convexification obtain 25.4% verified accuracy, while with D-epT-Fast, we can just obtain 1.4% verified accuracy. It worth highlighting that DeepT-Fast can also scale verify networks trained on STL10.
5
Conclusion
We propose a novel o-BaB global optimization algorithm to verify polynomial networks (PNs). We exhibit that our method outperforms existing methods, such as black-box solvers, IBP and DeepT-Fast [Bonaert et al. 2021). Our method enables verification in datasets such as STL10, which includes RGB images of 96 96 resolution. This is larger than the images typically used previous verification methods. Note that existing methods like IBP and Deep'--Fast are also able to scale to STL10 buth with lower verified accuracy. Our method can further encourage the community to extend verification to broader class of functions well as conduct experiments in datasets of higher resolution. We believe our method can be extended to cover other wiice-diffcrentialle networks in the future.
Limitations
As discussed in Appendix E.2, our verification method does not scale to high-degree PNs. Even though we can verify high-accuracy PNs (see Table 2. we are still far from verifying the top performing deep PNs studied in Chrysos et al. [021) Another problem that we share with ReLU NN verifiers is the scalability to networks with larger input size (Wang al. 202T]. In this work are able to verify networks trained in STL10 [Coates et al. 2011 but these networks are shallow, yet
their verification still takes long time, see Table 2
Acknowledgements
We are deeply thankful to the reviewers for providing constructive feedback. Research was sponsored by the Army Research Office and was accomplished under Grant Number 1911NFF119.-0440. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement number 725594 time-data). This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021_ This project has received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement n 725594 time- data). This work was supported by Zeiss. This work was supported by SNF project - Deep Optimisation of the Swiss National Science Foundation (SNSF) under grant number 200021 205011.
10

---

References
Claire S. Adjiman and Christodoulos A. Floudas. Rigorous convex underestimators for general twice-differentiable problems. Journal of Global Optimization, 9(1):33-40, Jul 1996.
Claire S. Adjiman, Stefan Dallwig, Christodoulos A. Floudas, and Arool Neumaier. A global optimization method, abb, for general cibeeeiifee constrained NLPs - theoretical advances. Computers & Chemical Engineering, 22: 11:-1137-11,, 1998.
Greg Anderson, Shankara Pailoor, Isil Dillig, and Swarat Chaudhuri. Optimization and abstraction: A synergistic approach for analyzing neural network robustness. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDDI 2019, page 731-744, New York, NY, USA, 2019. Association for Computing Machinery.
Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi. Measuring neural net robustness with constraints. In Advances in Neural Information Processing Systems (NeurIPS), NIPS' 16, page 2621 Red Hook, NY, USA, 2016. Curran Associates Inc.
Gregory Bonaert, Dimitar I. Dimitrov, Maximilian Baader, and Martin Vechev. Fast and precise certification of transformers. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2021, page 466-481, New York, NY, USA, 2021. Association for Computing Machinery.
Rudy Bunel, Alessandro De Palma, Alban Desmaison, Krishnamurthy Dvijotham, Pushmeet Kohli, Philip Torr, and M. Pawan Kumar. Lagrangian decomposition for neural network verification. In Jonas Peters and David Sontag, editors, Conference Uncerlainty in Artificial Intelligence (UAT), volume 124 of Proceedings of Machine Learning Research, pages 370-379. PMLR, 03-06 Aug 2020a.
Rudy Bunel, Jingyue Lu, llker Turkaslan, Philip H.S. Torr, Pushmeet Kohli, and M. Pawan Kumar. Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Research, 21(42):1-39 2020b.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57, 2017.
Moulik Choraria, Leello Tadesse Dadi, Grigorios Chrysos, Julien Mairal, and Volkan Cevher. The spectral bias of polynomial neural networks. In International Conference on Learning Representa- tions (ICLR), 2022.
Grigorios G Chrysos and Yannis Panagakis. Naps: Non-adversarial polynomial synthesis. Pattern Recognition Letters, 140:318-324, 2020.
Grigorios G. Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos P Zafeiriou. Deep polynomial neural networks. IEEE Transactions on Puttern Analysis and Machine Intelligence (T-PAMI), page 1-1,2021
Grigorios G Chrysos, Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks. In European Conference on Computer Vision (ECCV), 2022.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudik, editors, International Conference on Artificial Intelligence and Statisties, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 248 255, 2009.
Yinpeng Dong, Hang Su, Baoyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu. Efficient decision- based black-box adversarial attacks on face recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 7714-7722, 2019.
11

---

Zehao Dou, Stanley J Osher, and Bao Wang. Mathematical analysis of adversarial attacks. arXiv preprint arXiv:1811.06492, 2018.
Rüdiger Ehlers. Formal verification of piece-wise linear feed-forward neural networks. CoRR, abs/1705.01320, 2017.
Fenglei Fan, Jinjun Xiong, and Ge Wang. Universal approximation with quadratic deep networks. Neural Networks, 124:383-392. 2020.
Fenglei Fan, Mengzhou Li, Fei Wang, Rongje Lai, and Ge Wang. Expressivity and trainability of quadratic networks. CoRR, abs/2110.0600 2021.
Claudio Ferrari, Mark Niklas Mueller, Nikola Jovanović, and Martin Vechev. Complete verification via multi-neuron relaxation guided samnhhanddbouune In International Conference on Learning Representations (ICLR), 2022.
Timon Gehr, Matthew Mirman, Dana Drachsler-Cohes Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium Security and Privacy (SP), pages 3-18, 2018.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, International Conference on Learning Representations (ICLR), 2015.
Gaurav Goswami, Akshay Agarwal, Nalini K. Ratha, Richa Singh, and Mayank Vatsa. Detecting and mitigating ad adversarial perturbations for robust face recognition. International Journal of Computer Vision (IJCV), 127:719-742. 2019.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2022. URL https://wmi gurobi com
Reiner Horst and Hoang Tuy. Global Optimization: Deterministic Approaches. Springer, Berlin, Heidelberg, 1996.
Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks, 2017.
C. T. Kelley. Iterative Methods for Optimization. Society for Industrial and Applied Mathematics, 1999.
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Review, 51(3): 455-500, 2009.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/ifar. html, 55, 2014.
A. H. Land and A. G. Doig. An automatic method of solving discrete programming problems. Econometrica, 28(3):497-520 1960.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) (1(11))227822222 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444 2015.
Qizhang Li, Yiwen Guo, and Hao Chen. Practical no-box adversarial attacks against dnns. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Changliu Liu, Tomer Amonn Christopher Lazarus, Christopher Strong, Clark Barrett, and Mykel J. Kochenderfer. Algorithms for verifying deep neural networks. Foundations and Trends@ in Optimization, 4(3-4):244 404, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations (ICLR), 2018.
12

---

R. V. Mises and H. Pollaczek- Pollaezek-Geiringer Geiringer. Praktische verfahren der gleichungsauflisung ZAMM Journal of Applied Mathematics and Mechanics Zeitschrift für Angewandte Mathematik und Mechanik, 9(2): 152-164, 1929.
Ramon E. Moore, R. Baker Kearfott, and Michael J. Cloud. Introduction to Interval Analysis. Society for Industrial and Applied Mathematics, 2009.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and aceuracy. In International Conference Machine Learning (ICML), pages 7909-7919, 2020.
Vicenç Rúbies Royo, Roberto Calandra, Dušan M. Stipanović, and Claire J. Tomlin. Fast neural network verification via shadow prices. ArXiv, abs/1902.07247,, 2019.
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In International Conference on Learning Representations (ICLR), 2019.
Gagandeep Singh, Rupanshu Ganvir, Markus Püschel, and Martin Vechev. Beyond the single neuron convex barrier for neural network certification. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019.
M. H. Stone. The generalized weierstrass approximation theorem. Mathematics Magazine, 21(4):
167-184, 1948.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference Learning Representations (ICLR), 2019.
Shiqi Wang, Kexin Pei, Justin Whitchouse, Junfeng Yang, and Suman Jana. Formal security analysis of neural networks using symbolic intervals. In USENIX Conference on Security Symposium, SEC'18, 18, page 1599-16144 USA, 2018a. USENIX Association.
Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-CROWN: Efficient bound propagation with per-neuron split constraints for neural network robustness verification. In A. Beygelzimer. Y. Dauphin, Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems (NeurIPS), 2021.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 7794-7803, 2018b.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), pages 7472-7482, 2019.
Yaoyao Zhong and Weihong Deng. Adversarial learning with margin-based triplet embedding regularization. In International Conference on Computer Vision (ICCV), pages 6549-6558, 2019.
Zhenyu Zhu, Fabian Latorre, Grigorios Chrysos, and Volkan Cevher. Controlling the complex- ity and lipschitz constant improves polynomial nets. In International Conference on Learning Representations (ICLR), 2022.
13


Reviews:
[{"review_id": "uNmN2eH8Bw1", "reviewer": "Reviewer_bVTt", "summary": "This paper is a direct application of complete formal verification  against robustness applied to polynomial networks. They propose a new efficient computation of the Hessian bound in order to guarantees property wit the BaB method (a recent MILP verification method). They applied their method to 3 datasets, MNIST, CIFAR10 and STL10 for 1 model and different noise level +  comparison with IBP for multiple PN models on MNIST.", "questions": "Explicit the last point of weakness (Poor explications of why PN is an interesting family to explore (compared to BNN for example) in term of theory or application (NLP - graphNN - image generation...)) can improve my opinion on the paper.\nBut I think the verifiable accuracy is too low compared to state of the art.\nHowever, if there is a clear proof of PN model application for industry then  ", "limitations": "cf. last point on question.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nNovelty of the approach: complete verification of DNN specific family \nImportant topic\nSound proof of the Hessian bounds\nClarity of the paper\nVery fast verification\n\nWeaknesses\nPoor experiments comparison: Not comparison with SAT based method, no comparison with other  BaB methods. I understand it is not PN based verification but still important (https://arxiv.org/abs/2005.03597 and https://arxiv.org/abs/2103.06624). For example evry verification paper gives the resultas for 0.3 MNST and 2/255 for cfar10 \nPoor verifiable accuracy results compared to results above.\nPoor explications of why PN is an interesting family to explore (compared to BNN for example) in term of theory or application (NLP - graphNN - image generation...)\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SaQzHV47K9_", "reviewer": "Reviewer_sZV9", "summary": "This work introduces a method to prove robustness of polynomial networks against $l^\\infty$ perturbations. The method relies upon finding multivariate polynomials of degree 2 called $f_\\alpha$ lower bounding the differences in the classification score $g(z) = f(z)_t - f(z)_a$. The lower binding function $f_\\alpha$ relies on the parameter $\\alpha$ which is estimated via a combination of branch and bound and interval arithmetic. Crucially, the approach relies on the twice differentiable structure of polynomial networks and the combination with branch and bound over the input make the method complete. The approach is evaluated on MNIST, CIFAR10 and STL10 and compared against standard of the shelf solvers (Gurobi) as well as interval arithmetic. The code was provided together with the submission. ", "questions": "- Have the authors considered the possibility to split the sum in Eq. 5, thus maybe enabling different $\\alpha$ to increase precision of the approach? Would this be possible? If yes, how would the runtime complexity change? \n- In Eq. 13: how is A defined?\n- In Eq. 15: usually for the power method, the sequence is defined by $v_k = \\tfrac{M v_{k-1}}{\\| M v_{k-1} \\|}$. Is there a specific reason to have $M$ twice as in Eq. 15? \n- The power method numerical, thus has an approximation error. Is this approximation error handled soundly for the later analysis? Similarly, in Line 127-129, for PGD also there is a numerical approximation issue. \n- Do I assume correctly that the method is not floating point sound? (I think this is not central but I am curious anyhow)\n- Can more complex networks used for CIFAR10 and STL10 that offer higher precision?\n- Please speculate: Can and if how can you use your certification method to train the polynomial network? \n- Could the input splitting technique be improved by i.e. weighting the interval distance in each dimension by the diagonal entry of the hessian? (i.e. weighting by the second derivatives of the dimension in question?)\n- in Line 110: are the sets really disjoint? Wouldn’t the interval [1,3] be spliced in [1,2] and [2,3] having non-empty intersection?", "limitations": "The limitations are appropriately addressed. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Verifying new architectures is a challenging, interesting and important field of research. Polynomial networks show strong performance along a range of tasks, justifying the authors interest. The paper is mostly well written and the notation is clearly explained. Further, the Limitations are stated clearly in the section for this purpose. The might also be applicable to ReLU networks, as the authors state. Input refinement based approach to make sound verification complete where also considered earlier (see https://arxiv.org/pdf/1804.10829.pdf, https://arxiv.org/abs/1809.08098), the authors may want to have a look. \n\nThe approach does not seem to scale to large models or challenging datasets, which is also clearly stated by the authors. However, it would have been interesting, so see some larger (more accurate) models if feasible for CIFAR10 and STL10 in order to increase the number of correct classified inputs. Further, the hardware (GPU) is not specified. A comparison to other complete & non complete methods (evaluated on other networks) would have given an indication how polynomial networks with this certification compares to say standard ReLU convolutional networks performing the same task. Some details of the approach remained unclear, an explicit example illustrating the method would have been helpful. \n\n## After the rebuttal:\nThe authors clarified all my questions and provided in the general response an informative comparison, solidifying their evaluation. I think that this paper presents new and useful ideas for the field. I recommend acceptance. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IkPaYACYA1I", "reviewer": "Reviewer_rPvn", "summary": "The paper considers developing a sound and complete verification method for robustness certification of polynomial networks (PNs) against norm-based perturbations. The verifier developed here combines input splitting with $\\alpha$-convexification. The main algorithmic contribution is in computing this convexification which involves interval bound propagation (IBP) to bound the lower bounding hessian $L_H$ of the PNs and employing a standard power method to compute the $\\alpha$ on the $L_H$. Comparisons against Gurobi and IBP show that the proposed method can enable more precise and scalable verification. ", "questions": "* On line 119, $\\lambda_{min}$ is undefined (I assume its minimum eigenvalue). In eq. (5) how is $z_i$ related to $z$ (I assume these are components of z)? What is $\\mathcal{U}(A)$ in (13)? \n\n* In Section 3.3, how is the initial vector $v_0$ chosen? Does the initialization affect the speed of the method? Do you always converge or using a value before convergence can still lead to sound verification? \n\n* The input splitting strategy used by your algorithm appears suboptimal. There are better strategies available in the literature that can improve the results of your method (e.g., Efficient Verification of ReLU-Based Neural Networks via Dependency Analysis, AAAI 2020). Any particular reason why you chose the suboptimal strategy?\n\n* How difficult it is to compute the lower bound of the eigenvalue of the Hessian with relational analysis such as DeepZ/Fast-Lin or DeepPoly/CROWN?\n\n* For ReLU-based networks, do you need to use Clark jacobians for your method to work as done in https://dl.acm.org/doi/abs/10.1145/3498718, or non-differentiability at x=0 does not affect your method? If it does not, then why?\n\n* \"We take advantage of the low-rank decomposition characterizing PNs to efficiently perform a power method over the lower bounding Hessian.\" so your method works for only PNs with this property? If yes, then an extension to regular NN with Sigmoid/Tanh activation may not be possible?", "limitations": "The authors adequately addressed the limitations of their method in Section 6. The proposed work has no potential negative societal impact. \n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The problem considered here is novel and well-motivated. There is not much work on the certification of polynomial networks which have also been shown to be vulnerable to adversarial attacks. \n\n+ While $\\alpha$-convexification and IBP are known techniques in the literature, their combination has not been applied for NN verification before to the best of my knowledge. Further, the proposed framework is general and can be extended to other neural network architectures with twice differentiable activation functions.\n\n+ While there are some issues with the writing that I point out later, I enjoyed reading the paper overall. It is well-organized and relatively easy to follow. \n\n\nWeakness:\n\n- Since the primary operation in PNs is multiplication, I am not sure why the authors cannot compare with other methods in the literature (e.g., https://dl.acm.org/doi/abs/10.1145/3453483.3454056, https://openreview.net/forum?id=BJxwPJHFwS) that can handle multiplication more precisely than IBP. In my opinion, IBP is a very weak verifier for networks that are not trained to be easy to verify. Adding comparisons against these baselines will make the evaluation more compelling. Note I am not so concerned about the size of the networks/datasets. In many real-world applications, the small NNs are trained on datasets with much smaller dimensionality than STL10 (e.g., see security classifiers here: https://arxiv.org/abs/2105.11363, or the ones in Bio https://ieeexplore.ieee.org/abstract/document/9680252), therefore even though your verifier does not scale to large Vision/NLP datasets yet, there is still some practical value to your work. In fact, I also encourage the authors to consider these domains for their method in the future to enable more real-world impact of their work instead of focusing only on vision/NLP models which are often limited to academic/industrial competition and still not widely employed in the safety-critical settings due to their brittleness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept