Title: Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Adversarial Robustness

Certified Radius

Lipschitz Constants
Abstract: To improve the robustness of deep classifiers against adversarial perturbations, many approaches have been proposed, such as designing new architectures with better robustness properties (e.g., Lipschitz-capped networks), or modifying the training process itself (e.g., min-max optimization, constrained learning, or regularization). These approaches, however, might not be effective at increasing the margin in the input (feature) space. In this paper, we propose a differentiable regularizer that is a lower bound on the distance of the data points to the classification boundary. The proposed regularizer requires knowledge of the model's Lipschitz constant along certain directions. To this end, we develop a scalable method for calculating guaranteed differentiable upper bounds on the Lipschitz constant of neural networks accurately and efficiently.  The relative accuracy of the bounds prevents excessive regularization and allows for more direct manipulation of the decision boundary. Furthermore, our Lipschitz bounding algorithm exploits the monotonicity and Lipschitz continuity of the activation layers, and the resulting bounds can be used to design new layers with controllable bounds on their Lipschitz constant. Experiments on the MNIST, CIFAR-10, and Tiny-ImageNet data sets verify that our proposed algorithm obtains competitively improved results compared to the state-of-the-art.

Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularization

Abstract

To improve the robustness of deep classifiers against adversarial perturbations, many approaches have been proposed, such as designing new architectures with better robustness properties (e.g., Lipschitz-capped networks), or modifying the training process itself (e.g., min-max optimization, constrained learning, or regularization). These approaches, however, might not be effective at increasing the margin in the input (feature) space. In this paper, we propose a differentiable regularizer that is a lower bound on the distance of the data points to the classification boundary. The proposed regularizer requires knowledge of the model's Lipschitz constant along certain directions. To this end, we develop a scalable method for calculating guaranteed differentiable upper bounds on the Lipschitz constant of neural networks accurately and efficiently. The relative accuracy of the bounds prevents excessive regularization and allows for more direct manipulation of the decision boundary. Furthermore, our Lipschitz bounding algorithm exploits the monotonicity and Lipschitz continuity of the activation layers, and the resulting bounds can be used to design new layers with controllable bounds on their Lipschitz constant. Experiments on the MNIST, CIFAR-10, and Tiny-ImageNet data sets verify that our proposed algorithm obtains competitively improved results compared to the state-of-the-art.

1 Introduction

Motivated by the vulnerability of deep neural networks to adversarial attacks [1], i.e., imperceptible perturbations that can drastically change a model's prediction, researchers and practitioners have proposed various approaches to enhance the robustness of deep neural networks, including adversarial training [2-4], regularization [5, 6], constrained learning [7], randomized smoothing [810], relaxation-based defenses [11, 12], and model ensembles [13, 14]. These approaches modify the model architecture, the optimization procedure (loss function and algorithm), the inference mechanism, or the dataset itself [15] to enhance accuracy on both natural and adversarial examples.

However, most of these methods typically do not directly operate on input margins and rather target the output margin or its surrogates. For example, it is an established property that deep models trained with the cross-entropy loss, which is a surrogate for maximizing the output margin (see Appendix A.2), are prone to adversarial attacks [1]. From this perspective, it is critical to design regularized loss functions that can explicitly and effectively target the margin in the input space [16-18].

Our Contribution (1) Using first principles, we design a novel regularized loss function for training adversarially robust deep classifiers. We design a differentiable regularizer, using the Lipschitz constants of the logit differences, that is a lower bound on the input margin. We empirically show that this regularizer promotes larger margins in the input space. (2) We develop a scalable method for calculating guaranteed analytic and differentiable upper bounds on the Lipschitz constant of deep networks accurately and efficiently. Our method, called LipLT, hinges on the idea of Loop Transformation on the nonlinearities, which allows us to exploit the monotonicity and Lipschitz continuity of activations functions effectively. We prove that the resulting upper bound is better than the product of the Lipschitz constants of all layers (the so-called naive bound), and in practice, it is significantly better. Furthermore, our Lipschitz bounding algorithm can be used to design new layers with controllable bound on their Lipschitz constant. (3) We integrate our Lipschitz estimation algorithm within the proposed training loss to develop a robust training algorithm that eliminates the need for any inner-loop optimization subroutine. We utilize the recurring structure of the calculations that enable parallelized implementation on GPUs. When integrated into training, the relative accuracy of the bounds prevents excessive regularization and allows for more direct manipulation of the decision boundary.

Experiments on the MNIST, CIFAR-10, and Tiny-ImageNet data sets verify that our proposed algorithm obtains competitively improved results compared to the state-of-the-art. Code available on https://github.com/o4lc/CRM-LipLT.

1.1 Related Work

In the interest of space, we only review the most relevant literature and defer the comprehensive version to the supplementary materials.

Enhancing Robustness via Optimization The most well-known approach in adversarial defenses is perhaps adversarial training (AT) [4] and its certified variants [11, 19-23], which involves minimizing a worst-case loss (or its approximation) using uniformly-bounded perturbations to the training data. Since these approaches might hurt accuracy on non-adversarial examples, several regularization-based methods have been proposed to trade off adversarial robustness against natural accuracy [5, 6, 24, 25]. Intuitively, these approaches aim to control the variations of the model close to the decision boundary or around the training data points. In general, the main challenge is to formulate computationally efficient differentiable regularizers that can directly increase the margin of the classifier. Of notable recent work in this direction is [18], in which the authors design a regularizer that directly increases the margin in the input space and prioritizes more vulnerable points by leveraging the dynamics of the decision boundary during training. Our loss function also takes advantage of this approach, but rather than using an iterative algorithm to compute the margin, we use Lipschitz continuity arguments to find closed-form differentiable lower bounds on the margin.

Robustness and Lipschitz Regularity To obtain efficient and scalable certificates of robustness one can bound or control the global Lipschitz constant of the model. To train robust networks, the global Lipschitz bound is typically computed as the product of the spectral norm of linear layers (the naive bound), which provides computationally efficient but overly conservative certificates [5, 19, 26]. If these bounds are localized (e.g., in a neighborhood of each training data point), conservatism can be mitigated at the expense of losing computational efficiency [20, 27]. However, it is not clear if this approach provides any advantages over other local bounding schemes [28, 29] that can be more accurate with comparable complexity. Hence, for training purposes, it is highly desirable to have global but less conservative differentiable Lipschitz bounds. Of notable work in this direction is the semidefinite programming (SDP) approach of [30] (as well as its local version [31]) to provide accurate numerical bounds, which has also been leveraged in training low-Lipschitz networks [32]. However, these SDP-based approaches are restricted to small-scale models. In comparison, LipLT is less accurate than LipSDP but can scale to significantly larger models and larger input dimensions. Compared to the naive method, LipLT is significantly more accurate but has a comparable practical complexity thanks to a specialized GPU implementation that exploits the recursive nature of the algorithm.

Lipschitz Layers Another approach to improving robustness is to design new layers with controllable Lipschitz constant. Various methods have been proposed to design the so-called 1-Lipschitz networks [7, 33-37]. Inspired by the LipSDP framework of [30], the recent work [38] designs 1-Lipschitz layers that generalize many of the aforementioned 1-Lipschitz designs. However, their proposed approach is currently not applicable to multi-layer networks. In [39], the authors propose a reparameterization that directly satisfies the SDP constraint of LipSDP, but the proposed method cannot handle skip connections. [21] introduces ℓ ∞ distance neurons that are 1-Lipschitz and provide inherent robustness against ℓ ∞ perturbations.

1.2 Preliminaries and Notation

The log-sum-exp function is defined as LSE ( x ) = log( ∑ n i =1 exp( x i )) . For any t > 0 , the perspective [40] of the log-sum-exp function satisfies the inequalities max i ( x i ) < t -1 LSE ( tx ) ≤ max i ( x i ) + t -1 log( n ) . We denote the cross-entropy loss by CE ( q, p ) = -∑ K k =1 p k log( q k ) , where p, q ∈ R K are on the probability simplex. For an integer y ∈ { 1 , · · · , K } , we define u y ∈ R K as the y -th standard unit vector. For a vector a ∈ R K , we have ∥ a ∥ p = ( ∑ K i =1 | a i | p ) 1 /p . Additionally, we have ∥ a ⊤ ∥ p = ∥ a ∥ q , where 1 /p +1 /q = 1 . When p = 1 , it is understood that q = ∞ . For a matrix A ∈ R m × n , ∥ A ∥ denotes a matrix norm of A . In this paper we focus on ∥ A ∥ 2 = σ max ( A ) , i.e., the spectral norm of the matrix A . We, however, note that except for the implementation notes regarding the power iteration, the rest of the Lipschitz estimation algorithm is valid in any other matrix norm as well. We denote the indicator function of the event E as 1 E , where 1 E outputs 1 if the event E is realized and 0 otherwise.

2 Certified Radius Maximization (CRM)

Consider a K -class deep classifier C ( x ; θ ) = arg max 1 ≤ i ≤ K f i ( x ; θ ) , where f ( x ; θ ) = softmax( z ( x ; θ )) is the vector of class probabilities, and z ( x ; θ ) is a deep network. If a data point x is classified correctly as y ∈ { 1 , · · · K } , then the logit margin γ ( z ( x ; θ ) , y ) , the difference between the largest and second-largest logit, is γ ( z ( x ; θ ) , y ) = z y ( x ; θ ) -max j = y z j ( x ; θ ) .

̸

Several existing loss functions for training classifiers are differentiable surrogates for maximizing the logit margin, including the cross entropy, and its combination with label smoothing (see Appendix A.2 for more details). However, maximizing the logit margin, or its surrogates, does not necessarily increase the margin in the input space. This can happen, for example, when the model has rapid variations close to the decision boundary. Thus, we need a regularizer targeted for maximizing the input margin.

Regularization Based on Certified Radius For a correctly-classified data x with label y , the distance of x to the decision boundary can be calculated by solving the optimization problem [18],

where the constraint enforces ˆ x to be on the decision boundary. 2 To ensure a large margin in the input space, we must, in principle, maximize the radius R ( x, y ; θ ) of all correctly classified data points. To achieve this goal, we can add a regularizer that penalizes small input margins,

where λ > 0 is the regularization constant and g : R → R is a decreasing function to promote larger certified radii. See appendix A.2 for more discussion on the properties of g .

To optimize (2), we must compute and differentiate through R ( x, y ; θ ) . For ℓ ∞ norm and ReLU activations, we can reformulate (1) as a mixed-integer linear program by using a binary representation

of the activation functions. However, it is not affordable to solve this optimization problem during training. Hence, we must resort to computing lower bounds on R ( x, y ; θ ) instead.

By replacing the logit margin with its soft lower bound in the constraint of (1), we obtain a lower bound on R ( x, y ; θ ) ,

̸

To solve this non-convex problem, the authors of [18] adapt the successive linearization algorithm of [41], which in turn does not provide a provable lower bound. To avoid this iterative algorithm and to provide sound lower bounds, we propose to use Lipschitz continuity arguments.

Lipschitz-Based Surrogates for Certified Radius Suppose we add a norm-bounded perturbation δ to a data point x that is classified correctly as y . For the label of x + δ to not change, we must enforce

̸

Suppose the logit difference x ↦→ ∆ z yi ( x ) is Lipschitz continuous with constant L yi (in p norm), implying | ∆ z yi ( x + δ ; θ ) -∆ z yi ( x ) | ≤ L yi ∥ δ ∥ p , ∀ x, δ . Then we can write

̸

̸

The right-hand side remaining positive for all i = y is a sufficient condition for the correct classification of x + δ as y . This sufficient condition yields a lower bound on R ( x, y ; θ ) as follows,

̸

This lower bound is the pointwise minimum of logit differences normalized by their Lipschitz constants. This lower bound is not differentiable everywhere, but similar to the logit margin, we propose a smooth underapproximation of the min operator using scaled LSE:

̸

In addition to differentiability, another advantage of using this soft lower bound is that as opposed to R ( x, y ; θ ) , which involves only two logits, the soft lower bound R soft t ( x, y ; θ ) includes all the logits, and hence, makes more effective use of information.

Proposition 1 We have the following relationship between R soft t ( x, y ; θ ) defined in (5) and R ( x, y ; θ ) defined in (4).

In summary, we use the following loss function to train our classifier,

The first and the second terms are differentiable surrogates to the negative logit margin, and the input margin, respectively. For example, as we show in Appendix A.2 the negative cross-entropy loss is a differentiable lower bound on the logit margin, i.e., we can choose γ ( z ( x ; θ ) , y ) = -CE ( z ( x ; θ ) , u y ) .

It now remains to estimate the L yi 's (the Lipschitz constants of x ↦→ ∆ z yi ( x ) ) that are needed to compute the second term in the loss function. While any guaranteed upper bound on L yi would suffice to preserve the lower bound property in (4), a more accurate upper bound prevents excessive regularization and allows for more direct manipulation of the decision boundary. To achieve this goal, we propose a new method which we will discuss next.

3 Scalable Estimation of Lipschitz Constants via Loop Transformation (LipLT)

̸

In this section, we propose a general-purpose algorithm for computing a differentiable upper bound on the Lipschitz constant of deep neural networks, which is also of independent interest. For simplicity in the exposition, we first consider single hidden layer neural networks of the form h ( x ) = W 1 ϕ ( W 0 x ) , where W 1 , W 0 have compatible dimensions, and the bias terms are ignored without loss of generality. The activation layers ϕ are of the form ϕ ( z ) = ( φ ( z 1 ) , · · · , φ ( z n 1 )) z ∈ R n 1 , where φ : R → R is the activation function, which we assume to be monotone and Lipschitz continuous, implying α ≤ ( φ ( x ) -φ ( x ′ )) / ( x -x ′ ) ≤ β ∀ x = x ′ for some 0 ≤ α ≤ β < ∞ [30, 42]. In [30], the authors propose an SDP for computing an upper bound on the global Lipschitz constant of multi-layer neural networks when ℓ 2 norm is considered in both input and output domains. This result for the single hidden layer case is formally stated in the following theorem.

Theorem 1 ([30]) Consider a single-layer neural network described by h ( x ) = W 1 ϕ ( W 0 x ) . Suppose ϕ ( x ): R n 1 → R n 1 = [ φ ( x 1 ) · · · φ ( x n 1 )] , where φ is slope-restricted over R with parameters 0 ≤ α ≤ β < ∞ . Suppose there exist ρ > 0 and diagonal T ∈ S n 1 + such that the matrix inequality

The key advantage of this SDP formulation is that we can exploit several properties of the structure, namely monotonicity ( α ≥ 0 ) and Lipschitz continuity ( β < ∞ ) of the activation functions, as well as using the same activation function in the activation layer. However, solving this SDP and enforcing them during training can be challenging even for small-scale neural networks. The recent work [43] has exploited the chordal structure of the resulting SDP imposed by the sequential structure of the network to solve the SDP for larger instances. However, these approaches are still unable to scale to larger problems and are not suitable for training purposes.

To guide the search for analytic solutions to the linear matrix inequality (LMI) of Theorem 1, the authors in [38] consider the following residual structure,

Then it can be shown that the LMI condition (7) generalizes to condition (9) (see Appendix A.3 for details).

By choosing ρ = 1 , H = I and G = -( α + β ) W ⊤ T , then the LMI (9) simplifies to ( α + β ) 2 TWW ⊤ T ⪯ 2 T (all blocks in the LMI except for the lower diagonal block become zero). When we restrict T to be positive definite, then the latter condition is equivalent to ( α + β ) 2 WW ⊤ ⪯ 2 T -1 , which can be satisfied analytically using various choices of T [38]. In summary, the function h ( x ) = x -( α + β ) W ⊤ Tϕ ( Wx ) is guaranteed to be 1 -Lipschitz as long as ( α + β ) 2 WW ⊤ ⪯ 2 T -1 .

A potential issue with the above parameterization (and 1-Lipschitz networks in general) is that, since the true Lipschitz constant of the layer can be less than one, the multi-layer concatenation can become overly contractive. One way to resolve this limitation is to modify the parameterization as follows.

Proposition 2 Suppose WW ⊤ ⪯ 2 ρ ( α + β ) 2 T -1 for some ρ > 0 and some diagonal positive definite T . Then the following function is √ ρ -Lipschitz.

Now if we make a cascade connection of L ≥ 2 layers of the form (10) (each having its own ρ ), a naive bound on the Lipschitz constant of the corresponding deep network would be ∏ L i =1 ρ 1 / 2 i . However, this upper bound can still be very crude. We now propose an alternative approach to compute an analytic upper bound on the Lipschitz constant of (8). For the multi-layer case, we then show that our method can capture the coupling between different layers to improve the naive bound. For the sake of space, we defer all the proofs to the supplementary materials.

<latexi sh

1_b

64="yL0W/

S

pXK

Q

fdT

nFJOAw

>

B

H

c

VD

gN

E

z

G

Mr

Y

8

3

Z

UC

m

R

k

I

o

q

v

2

j

7

P

9

u

+

5

<latexi sh

1_b

64="H

Z

M2G

5n

J

7A

jL

F

U

>

B

c

VD

SgN

EOy

r

f

Y9

oP

K

8

z

p

wQ

+

WTv k3

I

Xd

u

/

0

q

C

m

<latexi sh

1_b

64="9KIBLfoy

ESDPk

YU

c

2g

>A

+3

Z

N

F

q

W

rw

C

5

V

0

m

7

M

v

X

OG

z

j

n

/

d

u

8

J

Q

p

H

T

R

<latexi sh

1_b

64="T

gwWDd3V2Z

+c

y

oF8zH

YI

>A

CKX

LS

M

U

v

u

k

5

m

O

q

9

N

f

BR

G

n

0

7

Jp

r

Pj

E

Q

/

<latexi sh

1_b

64="w/FGN

n

3

T

OuKgqHmJ8

U

>A

B

c

VDLS

E

z

Mr

Y

dQ

y

7

Xv0

2

P

j

R

9

I

f

W

p

+

o

Z

5

<latexi sh

1_b

64="AYI

9B5GM

S

y

QNrdjF

g+w

>

3

c

VDL

2p fUZ

u

ov

q

T

7KC

m

O

R

J

/

PX

H

W

z

nE

k

0

8

<latexi sh

1_b

64="yL0W/

S

pXK

Q

fdT

nFJOAw

>

B

H

c

VD

gN

E

z

G

Mr

Y

8

3

Z

UC

m

R

k

I

o

q

v

2

j

7

P

9

u

+

5

<latexi sh

1_b

64="rGk

0IY

P

n9

zgoD

/C

K

>A

B+3

c

Z

LS

N

F

q

W

8E

uX

w

2

m

7d

JM5

v

f

O

j

H

p

y

Q

T

R

U

V

<latexi sh

1_b

64="VXUJwB0

2RvQ

D

uz

Fj

M

E

>A

H

c

LSgN

O

G

r

Y

8

3

d

C5

m

n

Z

I

7

Pp

k

o

q

T

W/

f

+

y

K

9

<latexi sh

1_b

64="zK

H

+mS

v

p

ro9PF/8

>A

B7n

c

VDL

gM

3

W

u

JN

U2

k0

Z

GX

R

Y

f

w

T

q

Q

I

C

j

5

y

O

<latexi sh

1_b

64="H

Z

M2G

5n

J

7A

jL

F

U

>

B

c

VD

SgN

EOy

r

f

Y9

oP

K

8

z

p

wQ

+

WTv k3

I

Xd

u

/

0

q

C

m

<latexi sh

1_b

64="

LX

oIvJ

>A

B+H

ZD

S

F

P

m8E

C

/

u

w

2

0

O

G

K

R3

Y

n

5

zr

9

U

Q

y

f

<latexi sh

1_b

64="HqoG

ImVp fZ2y

cD

uC5

UK

M

>A

RX

7SwNBF

n

r

Td

3

j

v

Y

/

Ok

0

gW

9

+

J

Q

E

P

8

L

z

<latexi sh

1_b

64="N

7G

zk y2q5or

XC

dU

>A

n

c

VDLS

FJ3

v

M

gEQ

8

u

B

f

p

O

m

Y

j

K

+

0

w

H

9

T

W

/

P

R

Z

I

<latexi sh

1_b

64="nT

P

Fw8Yfc2B

JO

U

V

>A

C

X

DLS

N

3

R

KEk

d

y

Z

m

up

GQ

Hjr7

o

/

+

0

M

9

5

g

I

<latexi sh

1_b

64="w/FGN

n

3

T

OuKgqHmJ8

U

>A

B

c

VDLS

E

z

Mr

Y

dQ

y

7

Xv0

2

P

j

R

9

I

f

W

p

+

o

Z

5

<latexi sh

1_b

64="AYI

9B5GM

S

y

QNrdjF

g+w

>

3

c

VDL

2p fUZ

u

ov

q

T

7KC

m

O

R

J

/

PX

H

W

z

nE

k

0

8

<latexi sh

1_b

64="yL0W/

S

pXK

Q

fdT

nFJOAw

>

B

H

c

VD

gN

E

z

G

Mr

Y

8

3

Z

UC

m

R

k

I

o

q

v

2

j

7

P

9

u

+

5

<latexi sh

1_b

64="F

NoDB

ru cmR

Z+

8

f

k

>A

H

LS

I

P

q3W

M

Qw

Up

G2

0

y

X

j

n

O

/

Ydz5v

V

C7

E

K

9

g

J

T

<latexi sh

1_b

64="VXUJwB0

2RvQ

D

uz

Fj

M

E

>A

H

c

LSgN

O

G

r

Y

8

3

d

C5

m

n

Z

I

7

Pp

k

o

q

T

W/

f

+

y

K

9

𝑥

+

-

+

+

Ψ

ψ

(

x

)

x

x

(

x

1

, ϕ

(

x

1

))

(

x

2

, ϕ

(

x

2

))

(

x

2

, ψ

(

x

2

))

(

x

1

, ψ

(

x

1

))

x

G

Φ

W

H

H

G

W

Φ

α

+

β

2

I

α

+

β

2

GW

α

ϕ

(

x

2

)

-

ϕ

(

x

1

)

x

2

-

x

1

β

-

β

-

α

2

ψ

(

x

2

)

-

ψ

(

x

1

)

x

2

-

x

1

β

-

α

2

1.00

0.75

0.50

0.25

0.00

9

0.25

0.50

0.75

1.00

2.0

-1.51.0

0.5

0.5

1.0

1.5

2.0

0.2

0.1

0.0

0.1

0.2

-2.0

-1.5-1.0

0.5

0.5

1.0

1.5

2.0

The power of loop transformation Starting from (8), using the fact that ϕ is β -Lipschitz, a global Lipschitz constant of h can be computed as

This upper bound is pessimistic as it does not exploit the monotonicity ( α ≥ 0 ) of activation functions. In other words, this bound would not change as long as α ≥ -β . To inform the bound with monotonicity, we perform a loop transformation [44, 45] to bring the non-linearities to the symmetric sector [ -( β -α ) / 2 , ( β -α ) / 2] , resulting in the representation

where ψ ( x ) := ϕ ( x ) -α + β 2 x is the loop transformed nonlinearity, which is no longer monotone, and is β -α 2 -Lipschitz- see Figure 1. We can now compute a Lipschitz constant as

As we show below, this loop transformation does improve the naive bound. In Appendix A.3 we also prove the optimality of the transformation.

Proposition 3 Consider a single-layer residual structure given by h ( x ) = Hx + Gϕ ( Wx ) . Suppose ϕ : R n 1 → R n 1 , ϕ ( x ) = [ φ ( x 1 ) · · · φ ( x n 1 )] , where φ is slope restricted in [ α, β ] . Then h is Lipschitz continuous with constant ∥ H + α + β 2 GW ∥ + β -α 2 ∥ G ∥∥ W ∥ ≤ L naive .

Bound refinement. For the case of the ℓ 2 norm, we can further improve the derived bound in (13). Specifically, we can bound the Lipschitz constant of the second term of h in (12) analytically using LipSDP, which we summarize in the following theorem. 3 .

Theorem 2 Suppose there exists a diagonal T that satisfies G ⊤ G ⪯ T . Then a Lipschitz constant of (8) in ℓ 2 norm is given by

By optimizing over the choice of diagonal T satisfying G ⊤ G ⪯ T , we can find the best Lipschitz bound achievable by this method. In this paper, however, we are more interested in analytical choices of T . Using the same techniques outlined in [38], we have the following choices:

Spectral Normalization (SN): We can satisfy G ⊤ G ⪯ T simply by choosing T = T SN := ∥ G ∥ 2 2 I . In this case, L LT-SDP ( T SN ) = ∥ H + α + β 2 GW ∥ 2 + β -α 2 ∥ W ∥ 2 ∥ G ∥ 2 = L LT.

AOL [36]: Choose T ii = ∑ n j =1 | G ⊤ G | ij . As a result, T -G ⊤ G is real-symmetric and diagonally dominant with positive diagonal elements, and thus, is positive semidefinite.

SLL [38]: Choose T ii = ∑ n j =1 | G ⊤ G | ij q j q i , where q i > 0 . [38] uses the Gershgorin circle theorem to prove that this choice of T satisfies G ⊤ G ⪯ T .

3.1 Multi-layer Neural Networks

We now extend the proposed technique to the multi-layer case. Consider the following structure,

with W L = I for consistency. Then y L = x L is the output of the neural network. To compute a Lipschitz constant, we first apply loop transformation to each activation layer, resulting in the following representation.

Lemma 1 Consider the sequences in (14). Define ˆ H k := H k + α + β 2 G k W k for k = 0 , · · · , L -1 . 4 Then

We are now ready to state our result for the multi-layer case.

Theorem 3 Let m k , k ≥ 1 be defined recursively as

with m 0 = ∥ W 0 ∥ . Then m k is a Lipschitz constant of x 0 ↦→ y k , k = 0 , · · · , L -1 . In particular, m L is a Lipschitz constant of the neural network x 0 ↦→ y L = x L .

Similar to the single-layer case, we can refine the bounds for the ℓ 2 norm. For the sake of space, we defer this to the Appendix, and we use the bounds in (16) in our experiments.

Complexity For an L -layer network, the recursion in (16) requires O ( L 2 ) number of matrix norm calculations. For the naive bound, this number is exactly L . Despite this increase in complexity, we utilize the recurring structure of the calculations that enable parallelized implementation on GPUs. This utilization results in a reduction of the time complexity to O ( L ) , the same as the naive Lipschitz estimation algorithm. We provide a more detailed analysis of the computational and time complexity and the GPU implementation in Appendix A.4.

4 Experimental Results

In this section, we evaluate our proposed method for training deep classifiers on the MNIST [46], CIFAR-10 [47] and Tiny-Imagement [48] datasets. We compare our results with the closely related state-of-the-art methods. We further analyze the efficiency of our improved Lipschitz bounding algorithm and leverage it to compute the certified robust radii for the test dataset.

4.1 ℓ 2 -Robustness

Experimental setup We train using our novel loss function to certify robustness (6), using crossentropy as the differentiable surrogate, against ℓ 2 perturbations of size 1 . 58 on MNIST and 36 / 255 on CIFAR-10 and Tiny-Imagenet 5 . We train convolutional neural networks of the form mCnF with ReLU activation functions, as in [20], where m and n denote the number of convolutional and fully connected layers, respectively. The details of the architectures, training process, and most hyperparameters are deferred to the supplementary materials.

Baselines For baselines, we consider recent state-of-the-art methods that have provided results on convolutional neural networks of similar size. We consider: (1) GloRo [5] which uses naive Lipschitz estimation with a smoothness regularizer inspired by TRADES; (2) Local-lip-B/G (+ MaxMin) [20] which uses local Lipschitz constant calculation along with cut-off ReLU and MaxMin activation functions; (3) LipConvnet-5/10/15 [34] that uses 1-Lipschitz convolutional networks with GNP Householder activations; (4) SLL-small (SDP-based Lipschitz Layers) [38] which is a much larger 27 layer 1-Lipschitz network; (5) AOL-Small/Medium [36] which presents much larger 1-Lipschitz networks trained to have almost orthogonal layers.

Evaluation We evaluate the accuracy of the trained networks under 3 criteria; standard accuracy, adversarial accuracy, and certified accuracy. For adversarial accuracy we perform PGD attacks using [4] (hyperparameters provided in supplementary material). For certified accuracy, we calculate the certified radii using (4) and verify if they are larger than the priori-assigned perturbation budget. For the methods of recent literature, we report their best numbers as per the manuscript.

Results The results of the experiments are presented in Table 1. On MNIST, we outperform the state-of-the-art by a margin of 7.5% on verified accuracy whilst maintaining the same standard accuracy. On CIFAR-10 we surpass all of the state-of-the-art in verified accuracy. Furthermore, on methods using networks of similar sizes, i.e., 6C2F or LipConvnet-5, we surpass by a margin of around 4%. The networks of the recent literature AOL [36] and SLL [38] are achieving slightly worse certified accuracy even with much larger networks. On Tiny-Imagenet, our current best results are on par with the state-of-the-art.

4.1.1 Lipschitz Estimation

A key part of our method is the new improved Lipschitz estimation algorithm (LipLT) and the effective use of pairwise Lipschitz constants. Unlike previous works that estimate the pairwise Lipschitz between class i and j by the upper bound √ 2 L z [20], where L z is the Lipschitz constant of the whole network, or as L i + L j [5], where L i is the Lipschitz constant of the i -th class, we approximate this value directly as L ij by considering the map z i ( x ; θ ) -z j ( x ; θ ) . Table 2 shows the average statistic for Lipschitz constants estimated using different methods. Our improved Lipschitz calculation algorithm provides near an order-of-magnitude improvement over the naive method. Furthermore, Table 2 portrays the superiority of using pairwise Lipschitz constants instead of other proxies. Figure 2a illustrates the difference between using LipLT versus the naive method to calculate the certified radius. In this experiment, the naive method barely certifies one percent of the data points at the perturbation level of 1.58. However, using LipLT, we can certify a significant portion of

∗ Due to a size mismatch that occurs in power iteration, we had to modify the architecture slightly by changing the padding of some of the convolutional layers. The number of neurons are the same as that of the original architecture in [20]. More details in the supplementary material

(a)

(b)

(c)

800

700

LipLT

Naive

Probability

0.8

0.6

1.0

LipLT

Naive

400

300

Frennenrv

Probability

0.6

0.4

LipLT

Naive

400

500

300

Frequency

600

0.2

0.4

100

200

0.2

100

200

200

500

400

300

Probability

0.8

1.0

0.6

0.4

0.0

0

1

Z

3

4

LipLT

Naive

600

400

0

Frenenrv

Probability

0.0

0.8

0.6

0.4

0.0

0.5

1.0

1.5

2.0

LipLT

Naive

600

0

400

100

0.2

200

0.2

200

0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

0.0

0.00

0.01

0.02

0.03

0.04

0

0.0

0.000

0.001

0.002

0.003

0.004

0

E

E

E

the data points.

We further conduct an analysis of the certified radii of the data points for regularized versus unregularized networks for MNIST and CIFAR-10. Figures 2b and 2c illustrate the distribution of the certified radii. These radii were computed using the direct pairwise Lipschitz bounding. When comparing the results of the CRM-trained models (top figures) with the standard models (bottom figures), it becomes evident that our loss function enhances the certified radius.

5 Limitations

Although our proposed formulation has an elegant mathematical structure, there exist some limitations: (1) We are still using global Lipschitz constants to bound the margin, which can be conservative. We will investigate how we can localize our calculations without a significant increase in computation. (2) Computation of pairwise Lipschitz bounds for very deep architectures and or a large number of classes can become computationally intensive for training purposes (see Appendix A.4 for further discussions). (3) Several hyper-parameters require manual tuning. It is highly desirable to explore adaptive approaches to automatically select these hyper-parameters.

6 Conclusion

Adversarial defense methods are numerous and their approaches are different, but they all attempt to explicitly or implicitly increase the margin of the classifier, which is a measure of adversarial robustness (or vulnerability). From this perspective, it is highly desirable to develop adversarial defenses that can manipulate the decision boundary and increase the margin effectively and efficiently. We attempt to maximize the input margin by penalizing the Lipschitz constant of the neural network along vulnerable directions. Additionally, we develop a new method for calculating guaranteed analytic and differentiable upper bounds on the Lipschitz constant of the deep network. LipLT is provably better than the naive Lipschitz constant. We have also provided a parallelized implementation of LipLT using the recurring structure of the calculations, which is fast and scalable. Our proposed method achieves competitive results in terms of verified accuracy on the MNIST and CIFAR-10 datasets.

References

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 'Intriguing properties of neural networks'. In: arXiv preprint arXiv:1312.6199 (2013).

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 'Explaining and harnessing adversarial examples'. In: arXiv preprint arXiv:1412.6572 (2014).

Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 'Adversarial machine learning at scale'. In: arXiv preprint arXiv:1611.01236 (2016).

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 'Towards deep learning models resistant to adversarial attacks'. In: arXiv preprint arXiv:1706.06083 (2017).

Klas Leino, Zifan Wang, and Matt Fredrikson. 'Globally-robust neural networks'. In: International Conference on Machine Learning . PMLR. 2021, pp. 6212-6222.

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 'Theoretically principled trade-off between robustness and accuracy'. In: International conference on machine learning . PMLR. 2019, pp. 7472-7482.

Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. 'Parseval networks: Improving robustness to adversarial examples'. In: International Conference on Machine Learning . PMLR. 2017, pp. 854-863.

Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 'Certified adversarial robustness via randomized smoothing'. In: international conference on machine learning . PMLR. 2019, pp. 13101320.

Aounon Kumar, Alexander Levine, Soheil Feizi, and Tom Goldstein. 'Certifying confidence via randomized smoothing'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 5165-5177.

Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. 'Provably robust deep learning via adversarially trained smoothed classifiers'. In: Advances in Neural Information Processing Systems 32 (2019).

Eric Wong and Zico Kolter. 'Provable defenses against adversarial examples via the convex outer adversarial polytope'. In: International Conference on Machine Learning . PMLR. 2018, pp. 5286-5295.

Sumanth Dathathri, Krishnamurthy Dvijotham, Alexey Kurakin, Aditi Raghunathan, Jonathan Uesato, Rudy R Bunel, Shreya Shankar, Jacob Steinhardt, Ian Goodfellow, Percy S Liang, et al. 'Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 53185331.

Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. 'Defense against adversarial attacks using high-level representation guided denoiser'. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2018, pp. 1778-1787.

Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 'Ensemble adversarial training: Attacks and defenses'. In: arXiv preprint arXiv:1705.07204 (2017).

Nikolaos Tsilivis, Jingtong Su, and Julia Kempe. 'Can we achieve robustness from data alone?' In: arXiv preprint arXiv:2207.11727 (2022).

Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. 'Mma training: Direct input space margin maximization through adversarial training'. In: arXiv preprint arXiv:1812.02637 (2018).

Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. 'Geometry-aware instance-reweighted adversarial training'. In: arXiv preprint arXiv:2010.01736 (2020).

Yuancheng Xu, Yanchao Sun, Micah Goldblum, Tom Goldstein, and Furong Huang. 'Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustness'. In: arXiv preprint arXiv:2302.03015 (2023).

Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. 'Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks'. In: Advances in neural information processing systems 31 (2018).

Yujia Huang, Huan Zhang, Yuanyuan Shi, J Zico Kolter, and Anima Anandkumar. 'Training certifiably robust neural networks with efficient local lipschitz bounds'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 22745-22757.

Bohang Zhang, Tianle Cai, Zhou Lu, Di He, and Liwei Wang. 'Towards certifying l-infinity robustness using neural networks with l-inf-dist neurons'. In: International Conference on Machine Learning . PMLR. 2021, pp. 12368-12379.

Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. 'On the effectiveness of interval bound propagation for training verifiably robust models'. In: arXiv preprint arXiv:1810.12715 (2018).

Sungyoon Lee, Jaewook Lee, and Saerom Park. 'Lipschitz-certifiable training with a tight outer bound'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 16891-16902.

Judy Hoffman, Daniel A Roberts, and Sho Yaida. 'Robust learning with jacobian regularization'. In: arXiv preprint arXiv:1908.02729 (2019).

Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. 'Regularisation of neural networks by enforcing lipschitz continuity'. In: Machine Learning 110 (2021), pp. 393-416.

Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. 'Agreement-on-the-line: Predicting the performance of neural networks under distribution shift'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 19274-19289.

Zhouxing Shi, Yihan Wang, Huan Zhang, J Zico Kolter, and Cho-Jui Hsieh. 'Efficiently computing local Lipschitz constants of neural networks via bound propagation'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 2350-2364.

Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 'An abstract domain for certifying neural networks'. In: Proceedings of the ACM on Programming Languages 3.POPL (2019), pp. 1-30.

Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. 'Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 29909-29921.

Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. 'Efficient and accurate estimation of lipschitz constants for deep neural networks'. In: Advances in Neural Information Processing Systems 32 (2019).

Navid Hashemi, Justin Ruths, and Mahyar Fazlyab. 'Certifying incremental quadratic constraints for neural networks via convex optimization'. In: Learning for Dynamics and Control . PMLR. 2021, pp. 842-853.

Patricia Pauli, Anne Koch, Julian Berberich, Paul Kohler, and Frank Allgöwer. 'Training robust neural networks using Lipschitz bounds'. In: IEEE Control Systems Letters 6 (2021), pp. 121-126.

Sahil Singla and Soheil Feizi. 'Skew orthogonal convolutions'. In: International Conference on Machine Learning . PMLR. 2021, pp. 9756-9766.

Sahil Singla, Surbhi Singla, and Soheil Feizi. 'Improved deterministic l2 robustness on CIFAR10 and CIFAR-100'. In: arXiv preprint arXiv:2108.04062 (2021).

Cem Anil, James Lucas, and Roger Grosse. 'Sorting out Lipschitz function approximation'. In: International Conference on Machine Learning . PMLR. 2019, pp. 291-301.

Bernd Prach and Christoph H Lampert. 'Almost-orthogonal layers for efficient general-purpose Lipschitz networks'. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXI . Springer. 2022, pp. 350-365.

Asher Trockman and J Zico Kolter. 'Orthogonalizing convolutional layers with the cayley transform'. In: arXiv preprint arXiv:2104.07167 (2021).

Alexandre Araujo, Aaron Havens, Blaise Delattre, Alexandre Allauzen, and Bin Hu. 'A unified algebraic perspective on lipschitz neural networks'. In: arXiv preprint arXiv:2303.03169 (2023).

Ruigang Wang and Ian Manchester. 'Direct parameterization of lipschitz-bounded deep networks'. In: International Conference on Machine Learning . PMLR. 2023, pp. 3609336110.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Francesco Croce and Matthias Hein. 'Minimally distorted adversarial examples with a fast adaptive boundary attack'. In: International Conference on Machine Learning . PMLR. 2020, pp. 2196-2205.

Mahyar Fazlyab, Manfred Morari, and George J Pappas. 'Safety verification and robustness analysis of neural networks via quadratic constraints and semidefinite programming'. In: IEEE Transactions on Automatic Control 67.1 (2020), pp. 1-15.

Anton Xue, Lars Lindemann, Alexander Robey, Hamed Hassani, George J Pappas, and Rajeev Alur. 'Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks'. In: 2022 IEEE 61st Conference on Decision and Control (CDC) . IEEE. 2022, pp. 3389-3396.

Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear matrix inequalities in system and control theory . Vol. 15. Siam, 1994.

Charles A Desoer and Mathukumalli Vidyasagar. Feedback systems: input-output properties . SIAM, 2009.

Yann LeCun. 'The MNIST database of handwritten digits'. In: http://yann. lecun. com/exdb/mnist/ (1998).

Alex Krizhevsky, Geoffrey Hinton, et al. 'Learning multiple layers of features from tiny images'. In: (2009).

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 'Imagenet: A largescale hierarchical image database'. In: 2009 IEEE conference on computer vision and pattern recognition . Ieee. 2009, pp. 248-255.

Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit S Dhillon, and Luca Daniel. 'Towards Fast Computation of Certified Robustness for ReLU Networks'. In: arXiv preprint arXiv:1804.09699 (2018).

Trevor Avant and Kristi A Morgansen. 'Analytical bounds on the local Lipschitz constants of affine-ReLU functions'. In: arXiv preprint arXiv:2008.06141 (2020).

Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. 'Evaluating the robustness of neural networks: An extreme value theory approach'. In: arXiv preprint arXiv:1801.10578 (2018).

Aladin Virmaux and Kevin Scaman. 'Lipschitz regularity of deep neural networks: analysis and efficient estimation'. In: Advances in Neural Information Processing Systems . 2018, pp. 3835-3844.

Fabian Latorre, Paul Rolland, and Volkan Cevher. 'Lipschitz constant estimation of Neural Networks via sparse polynomial optimization'. In: International Conference on Learning Representations . 2020.

Tong Chen, Jean B Lasserre, Victor Magron, and Edouard Pauwels. 'Semialgebraic optimization for lipschitz constants of relu networks'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 19189-19200.

Matt Jordan and Alexandros G Dimakis. 'Exactly computing the local lipschitz constant of relu networks'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 7344-7353.

Patrick L Combettes and Jean-Christophe Pesquet. 'Lipschitz certificates for layered network structures driven by averaged activation operators'. In: SIAM Journal on Mathematics of Data Science 2.2 (2020), pp. 529-557.

Gal Chechik, Geremy Heitz, Gal Elidan, Pieter Abbeel, and Daphne Koller. 'Max-margin Classification of Data with Absent Features.' In: Journal of Machine Learning Research 9.1 (2008).

Johan AK Suykens and Joos Vandewalle. 'Least squares support vector machine classifiers'. In: Neural processing letters 9 (1999), pp. 293-300.

Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. 'Large margin deep networks for classification'. In: Advances in neural information processing systems 31 (2018).

Ziang Yan, Yiwen Guo, and Changshui Zhang. 'Deep defense: Training dnns with improved adversarial robustness'. In: Advances in Neural Information Processing Systems 31 (2018).

Yiwen Guo and Changshui Zhang. 'Recent advances in large margin learning'. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).

Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, Masashi Sugiyama, et al. 'Probabilistic margins for instance reweighting in adversarial training'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 23258-23269.

Matthew Mirman, Timon Gehr, and Martin Vechev. 'Differentiable abstract interpretation for provably robust neural networks'. In: International Conference on Machine Learning . 2018, pp. 3575-3583.

Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. 'Efficient neural network robustness certification with general activation functions'. In: Advances in neural information processing systems . 2018, pp. 4939-4948.

Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. 'Semidefinite relaxations for certifying robustness to adversarial examples'. In: Advances in Neural Information Processing Systems . 2018, pp. 10900-10910.

Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 'Certified defenses against adversarial examples'. In: arXiv preprint arXiv:1801.09344 (2018).

Yuhao Mao, Mark Niklas Müller, Marc Fischer, and Martin Vechev. 'TAPS: Connecting Certified and Adversarial Training'. In: arXiv preprint arXiv:2305.04574 (2023).

Diederik P Kingma and Jimmy Ba. 'Adam: A method for stochastic optimization'. In: arXiv preprint arXiv:1412.6980 (2014).

Peer reviews:
comments: SUMMARY:
This paper proposes a new bound for margin maximization, as well as tighter estimation of Lipschitz constant, to improve the adversarial robustness of neural networks. Experiments on three benchmark datasets are presented to show the proposed method can improve the certified radius.

STRENGTHS:
- This paper is written well and clearly
- New bound for input margin as regularization term
- New estimation of Lipschitz constant for multi-layer networks 
- Experiments are conducted to show improved robust radius

WEAKNESSES:
- Regarding the improved bound for input margin, I think the contribution is limited since the similar strategy has be used frequently in recently works. 
- The authors argued that the estimation of Lip. constant is scalable for multi-layer networks, but experimental results are not verified.

QUESTIONS:
See above.

LIMITATIONS:
The novelty of margin regularization is limited and the experimental results are not sufficient.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore how a model’s robustness to local pertubations in the input space can be enhanced through regularization at a low cost on the clean accuracy. \nThe goal is to have large margins on the desicion boundaries in input space.\nSince adding a regularization directly on the size of these margins is very costly, the authors introduce an alternative and efficient proxy to regularize this.\nThey use the Lipschitz constants of the logit difference function between pairwise classes to identify the most important directions to regularize.\nSince computing the constants is exactly is costly, but naive estiamtes are very conservative as they do not take into account the correlation between the trained layers, the authors introduce a new approximation that is efficient to compute.\nThey show that the method is comparing well with SOTA adversarial robustness methods.\n\nSTRENGTHS:\n1. The authors explore a new regularizer that is efficient to compute and can trade off the margin maximation in the input space via Lipschitz regularization which is still efficient to be used on classical toy datasets. They show that this regularizer is on par with existing approaches and may even outperform them.\n2. They provide a new strategy to efficiently approximate Lipschitz constants which is of independent interest.\n3. The paper overall is well written and structured; and can be followed comfortably.\n\nWEAKNESSES:\n1. In the experimental evaluation, I would find it valuable to study your model also independently to see how the accuracy of the improved Lipschitz estimate affects the final accuracy outcome. The claim that the less conservative estimates lead to a better regularizer can be checked independently of other state-of-the-art methods. I think interesting experiments to support that claim could be: Comparing how the estimated (and naive) Lipschitz constant evolves throughout training for training with and without regularizer for the same architecture; replacing the improved estimate of the Lipschitz constant with the naive one; introducing a global hyperparameter that replaces L_{ij} and tunes it at the computational budget available to the Lipschitz constant estimation. These are just a few suggestions that would improve the understanding of the actual efficiency of the introduced loss regularizer.\n2. The method relies on estimates for Lipschitz constants for every pair of classes, this scaling becomes unreasonable for datasets with many different classes.\n\nQUESTIONS:\n1. In L.3 you mention Lipschitz-capped networks and argue on to say that these approaches are not good at increasing the input margin. Why is that exactly, does your own approach not follow a similar idea?\n2. L.90, can you elaborate in a half-sentence on the problems of applying this to multilayer networks?\n3. Can you comment at some point, on why you exclude wrongly classified examples from the regularizer and if this sequential inclusion is expected to have some effect?\n4. I like the notation with the underline for the differentiable functions. However, since \\underline{R}_t^soft is not computing the same version as the R_t^soft and is more closely related to \\underline{R} I find the repetition of the names not optimal. Especially in L.249-252 the notation is confusing. First, in L. 251 there is a repetition (replace … with … with) and also it is not clear to me if h(x) in L.249 takes x=R_t^soft(x’,y,theta) so that the condition is a double application of R_t(R_t^soft(…)). I tried to understand it in different ways but I settled on the following to interpret the results: replace R(..) by x only in the case distinction in L.250 and the second ‘with' in L.51 to ‘and’. Could you please clarify if that is correct?\n5. It would be nice if you could spend 2 more sentences explaining what you mean by loop transformation. To me the term was new and difficult to google, I checked the appendix to understand what was going on.\n6. In (17) you use the function psi(.) which is not introduced in the main draft, only in the appendix - maybe you can explain that jointly with the loop transformation.\n7. I did not find a definition of the Standard baseline, so I am just guessing that it is normal training without any regularization…\n8. Can you give an intuition why you chose the specific values of l_2 perturbations in L.231/232? Is there a good reason why this was not tested for other values?\n9. Can you give a clearer comparison how costly the computation of the approximate Lipschitz constants is in comparison to the backpropagation in terms of GPU time in your expeiments? O(L) is mentioned but it would be nice to understand what this means in practice. \n\n\nMinor comments:\n1. In L.81 introduce the abbreviation SDP\n2. L.88 constantS\n3. L.95 To keep with convention, iterate over small i instead of small k in the CE\n4. L.158 eqref missing\n5. In (10) you just say ‘’compatible dimensions’ but then use dimensions in Thm.3.1 without introduction  — it would be nice to make them clear from the start if you are going to use them anyways.\n6. L.185 Abbreviation of LMI not introduced\n7. L.189 blockS\n8. The numbering of the equations in the Supplementary material is very confusing, as theorems and text have different numberings but are not distinguished (so the eq numbers are somehow in seemingly random order)\n\nLIMITATIONS:\nThe authors are clear about the limitations of their method in the paper. I would like to add that it is not immediately clear to me how more complex architectures would be amenable to similar estimates of the Lipschitz constants. Adversarial robust training which only relies on adapting the training data can achieve this for any architecture, so if this is not given, I would include it as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper developed a robust training algorithm whose objective is to increase the margin while regularizing the Lipschitz constant of the model along vulnerable directions. The authors also developed an improved Lipschitz estimation for a multi-layer networks with residual connections. Experiments on MNIST and CIFAR-10 shows superior performance of the proposed method compared to the state-of-the-art.\n\nSTRENGTHS:\n- A novel regularized loss function that directly promote larger margins in the input space is proposed.\n- A scalable Lipschitz bounding algorithm is proposed, which is tighter than the naive bound while being efficient. \n- Experiments on MNIST and CIFAR-10 showed the effectiveness of the proposed methods.\n\nWEAKNESSES:\n- This paper proposed many interesting methods, but I sometimes find it hard to see the motivation of each proposed method and the connections between them. It may be good to spend some more space to set up the preliminaries: \n    - The name is section 2 is a unified perspective on margin maximization, I'd hope to see what margin maximization algorithms it unified.  For instance, the variant of (6) may result in different algorithms, replacing L_{yi} with \\sqrt(2)L will results to Lipschitz margin training [Tsuzuku et al. 2018]. \n    - It would be good to decouple the effectiveness of each component: the use of pairwise Lipschitz, the use of log-sum-exp to make max/min to be differentiable, the use of the new Lipschitz bounding algorithm. It would be interesting to see the contribution from each of these to the certified accuracy.\n\n- It would be good if the authors can discuss the sensitivity of the proposed method wrt the choice of hyper-parameters. An ablation study on CIFAR-10 with various hyper-parameters and the corresponding certified robustness performance would be helpful. The proposed method is not performing as well on Tiny-Imagenet, I understand that it may be because of the inferior choice of hyper-parameters, but it may be good to debug more into that, especially giving table 2, the improved pair-wise Lipchitz of Tiny Imagenet seems to be much better than other Lipschitz, it is surprising that it does not lead to better certified robustness.\n\n- It would be nice to provide the computational cost of the proposed method in terms of Flops or running time.\n\nQUESTIONS:\n- In section 3, the proposed scalable Lipschitz estimation method seems to be work on neural networks with residual connections (Eq 16). But in the experiments, the model architectures seems to not have residual connections. How to apply the proposed bounds then?  \n\n- What is loop transformation? I am not familiar with the term.\n\n- Line 221, Recurring structure can reduce the complexity by dynamic programing, but the operations are still sequential rather than parallel since we need m_j, j<=k to compute m_{k+1}. How does the parallelized implementation on GPU work?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a loss to induce robustness w.r.t. adversarial attacks. This loss is composed of a term for enlarging the logit difference (such as classical  Cross-Entropy Loss), and a surrogate term to  maximize the robust radius. The latter imply to compute an upper bound of the Lipschitz constant (called L_ij) of the difference of logits z_y-z_i (y is the true class and any other class i).\nInspired by work in [39], the authors propose to take into consideration the motonicity of activation functions to compute a tigher lower bound of a residual-like layer than the classical one. They also propose an extension for computing an upper bound of the Lipschitz constant of a multilayer residual NN.\nNote that Batchnorm layers and dropout are not supported. \nThey also claim to compute these L_ij with the same complexity as computing the Lipschitz constant of each layer (supplementary materials)\nExperiments are done on MNIST,CIFAR10 and TinyImagenet. Comparison is provided with several methods, reporting clean accuracy, adversarial accuracy (PGD attack) and certified accuracy, and report higher certified accuracy. They also compare the Lipschitz estimation to classical bounds (sqrt(2)Lz, Li+Lj and theirs Lij) and claims an improvement of up to 200.\n\nSTRENGTHS:\nThe algorithm described in section 3.1 seems new and results presented in table 1 indicate that it enhance the lipschitz estimation..\n\nWEAKNESSES:\n-Clean performances are low and equivalent to 1LipchitzNN but author indicate in the related work that it should have a higher expressivity\n\n-No evaluation of computational complexity is provided. Scalability os the method is not clear\n\nQUESTIONS:\nThe paper is rather easy to read. Even if without the supplementary materials, \nSeveral errors or inexactitude can be found in  Notation and equation and proof\nComputation of a tight upper bound of NN seems new and interesting. However enlarging the margin under lipschitz constraint or regularization is not new. And besides main loss and regularization may have contradictory objectives (see detail comments)\n\nDetailed remarks:\n\nL31: correlation between robustness and lipschitz constant of NN is an established property , cite[1]\nL33: “constraining the model (e.g. 1-Lipschitz models) can lead to excessive regularization in addition to potentially limiting expressivity.”: it has been shown that 1-Lipschitz NN have the the same expressivity has non constrained networks (see “pay attention to your loss”, Bethune et al. Neurips’22). Thus this sentence is false.\nL43: “hinges on the idea of loop transformation on the nonlinearities”: the notion of “loop transformation” is not clear. Please clarify or add a reference.\nL54: choice of tiny-imagenet is not justified. Authors claim for scalability (L84) but restrain the learning to tiny-Imagenet.This must be explained\nL92: notation for matrix norm  is not described in this section, and ||.|| can be confusing: frobenius nor, spectral norm,…\nL100: “A unified perspective on margin maximization” : this section title is not justified. Please explain which unification is provided in this section?\nL111: “we note that maximizing the logit margin…..in the input space”: this is a well known property ([1])\nL125: “many choices for h, we use the same function proposed in [17] (see supplementary materials) “ but no information is given in the supplementary matrials\nL132: eq 4 this equation is not use in the following, confusing in notation with eq (7), and not evaluated. Probably useless for paper understanding\nL134: .”Solving (4) still requires an iterative inner loop within the outer loop of training”: transformation from inf formulation to outer loop is unclear. Please precise if this loop is an adversarial search or any other method\nL146: “This lower bound is not differentiable”: min function is differentiable. A least subderivative. Justification l149 is a better one. \nEq 9: it is known that CE loss aims to increase the Lipschitz constant of the NN during training (see paper Bethune et al cited earlier), and the regularization in (6) aims to reduce this Lipschitz constant. Please explain how these two antagonist objectives will be balanced. \nL169: inequality soes not represent the lispchitz constraint. A divide operator is missing\nTheorem 3.1: this theorem isbased on supposition of existence of \\rho and T, but in the following justifications of this existence is not clear\nL185: “analytic solutions to the LMI of Theorem 3.1,”: acronym LMI is not defined\nL192: same remark as on line L33\nL204: ||H|| isnot clear can be ether frobenius norm or spectral norm?\nL247: “,we calculate the certified radii using(6)” (6) is only applicable for ReLU like activation. Please justify how this certified robustness is computed for NN such as MaxMin (2)\nL249: “CRM hyperparameters”: first use of this acronym. Please detail\nL251: contradiction of justification given in eq (8). Please justify why the choice of h is not the same for each dataset.\nL255: “For Tiny-Imagenet, as the number of classes is far larger, bounding the pairwise Lipschitz would be computationally heavy.” Such a sentence and footnote 4  need justification. Please detail the computational complexity induced by your method\nTable 1: In related works, author indicate that 1-Lipschitz NN are harder to train and have a lower expressivity. However, clean performance of the proposed solution has performances equivalent to   1Lip solutions and far from classical performances. Please justify\nNo software provided to verify the correctness of the paper \n\n\nSupplementary materials:\nL532: “However, the expressive power of 1-Lipschitz networks needs to be addressed and some works study their limitations [47]. ” this sentence is false. See remark on line 33\nL580: \\gamma is non differentiable. : Max operation is differentiable?\nL598: h function was supposed to be described in the supplementary materials \nL605 (equation) replace (1-t) by (1-t^hat)\nL615 (equation) i\\neqy is replaced in this eq by i\\in[1,K-1] pleae justify?\nL638: please justify te existenceof T and \\rho?\nL660 m_k is not the lipschitz constant, but an upper bound of the Lipschitz constant\nL660 (eq) m_{k+1} should be lower than \nL673: eq 33 : lack ||\nL719: “   Calculating the ℓ 2 norm of a general rectangular matrix can be performed by \nusing the power iteration”: confusing notations: power iteration is for spectral norm and is often in general notes ||.||_2 and not l_2. Confusing with frobenius norm\n\nreferences: clean references for papers that are published and not preprint\n\nLIMITATIONS:\nLimitations are addressed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}