Title: ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Networks

Lipschitz Constant

Robustness
Abstract: The Lipschitz constant plays a crucial role in certifying the robustness of neural networks to input perturbations. Since calculating the exact Lipschitz constant is NP-hard, efforts have been made to obtain tight upper bounds on the Lipschitz constant. Typically, this involves solving a large matrix verification problem, the computational cost of which grows significantly for both deeper and wider networks. In this paper, we provide a compositional approach to estimate Lipschitz constants for deep feed-forward neural networks. We first obtain an exact decomposition of the large matrix verification problem into smaller sub-problems. Then, leveraging the underlying cascade structure of the network, we develop two algorithms. The first algorithm explores the geometric features of the problem and enables us to provide Lipschitz estimates that are comparable to existing methods by solving small semidefinite programs (SDPs) that are only as large as the size of each layer. The second algorithm relaxes these sub-problems and provides a closed-form solution to each sub-problem for extremely fast estimation, altogether eliminating the need to solve SDPs. The two algorithms represent different levels of trade-offs between efficiency and accuracy. Finally, we demonstrate that our approach provides a steep reduction in computation time (as much as several thousand times faster, depending on the algorithm for deeper networks) while yielding Lipschitz bounds that are very close to or even better than those achieved by state-of-the-art approaches in a broad range of experiments. In summary, our approach considerably advances the scalability and efficiency of certifying neural network robustness, making it particularly attractive for online learning tasks.

ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks

Abstract

The Lipschitz constant plays a crucial role in certifying the robustness of neural networks to input perturbations. Since calculating the exact Lipschitz constant is NP-hard, efforts have been made to obtain tight upper bounds on the Lipschitz constant. Typically, this involves solving a large matrix verification problem, the computational cost of which grows significantly for both deeper and wider networks. In this paper, we provide a compositional approach to estimate Lipschitz constants for deep feed-forward neural networks. We first obtain an exact decomposition of the large matrix verification problem into smaller sub-problems. Then, leveraging the underlying cascade structure of the network, we develop two algorithms. The first algorithm explores the geometric features of the problem and enables us to provide Lipschitz estimates that are comparable to existing methods by solving small semidefinite programs (SDPs) that are only as large as the size of each layer. The second algorithm relaxes these sub-problems and provides a closed-form solution to each sub-problem for extremely fast estimation, altogether eliminating the need to solve SDPs. The two algorithms represent different levels of trade-offs between efficiency and accuracy. Finally, we demonstrate that our approach provides a steep reduction in computation time (as much as several thousand times faster, depending on the algorithm for deeper networks) while yielding Lipschitz bounds that are very close to or even better than those achieved by stateof-the-art approaches in a broad range of experiments * . In summary, our approach considerably advances the scalability and efficiency of certifying neural network robustness, making it particularly attractive for online learning tasks.

1 Introduction

The Lipschitz constant, which quantifies how a neural networks output varies in response to changes in its inputs, is a crucial measure in providing robustness certificates [1, 2] on downstream tasks such as ensuring resilience against adversarial attacks [3, 4], stability of learning-based models or systems with neural network controllers [5-9], enhancing generalizability [10], improving gradientbased optimization methods and controlling the rate of learning [11][12]. The problem of calculating the exact Lipschitz constant is NP-hard [13]. Therefore, efforts have been made to estimate tight upper bounds for the Lipschitz constant of feed-forward neural networks (FNNs) [14-18] and other architectures such as convolutional neural networks (CNNs) [19-21]. Typical approaches include formulating a polynomial optimization problem [22] or bounding the Lipschitz constant via quadratic constraints and semidefinite programming (SDP) [14], which in turn requires solving a large-scale matrix verification problem whose computational complexity grows significantly with

both the depth and width of the network. These approaches have also motivated the development of methods to design neural networks with certifiable robustness guarantees [19, 23-25].

Contribution. In this paper, we provide a scalable compositional approach to estimate Lipschitz constants for deep feed-forward neural networks. We demonstrate steep reductions in computation time (as much as several thousand times faster than the state-of-the-art depending on the experiment), while obtaining Lipschitz estimates that are very close to or even better than those achieved by state-of-the-art approaches. Specifically, we develop two algorithms, representing different levels in the trade-off between accuracy and efficiency, allowing for application-specific choices. The first algorithm, ECLipsE , involves estimating the Lipschitz constant through a compositional layer-bylayer solution of small SDPs that are only as large as the weight matrix in each layer. The second algorithm, ECLipsE-Fast , provides a closed-form solution to estimate the Lipschitz constant, completely eliminating the need to solve any matrix inequality SDPs. Both algorithms provably guarantee the existence of solutions at each step to generate tight Lipschitz estimates. In summary, our work significantly advances scalability and efficiency in certifying neural network robustness, making it applicable to a variety of online learning tasks.

Theoretical Approach. We begin with the large matrix verification SDP for Lipschitz constant estimation under the well-known framework LipSDP [14]. To avoid handling a large matrix inequality, we employ a sequential Cholesky decomposition technique to obtain an exact decomposition of the large matrix verification problem into a series of smaller, more manageable sub-problems that are only as large as the size of the weight matrix in each layer. Then, observing the cascade structure of the neural network, we develop (i) algorithm ECLipsE , which characterizes the geometric features of the optimization problem and enables us to provide an accurate Lipschitz estimate and (ii) algorithm ECLipsE-Fast , which further relaxes the sub-problems, and yields a closed-form solution for each sub-problem that altogether eliminates the need to solve any SDPs, resulting in extremely fast implementations.

Related Work. The simplest way to estimate the Lipschitz constans is to provide a naive upper bound using the product of induced weight norms, which is rather conservative [26]. Another approach is to utilize automatic differentiation to approximate a bound, which is not a strict upper bound, although it is often so in practice [13]. Additionally, compositions of nonexpansive averaged operators and affine operators [16], Clarke Jacobian based approaches and other methods focusing on local Lipschitz constants [17][27] have also been studied. Recently, optimization-based approaches such as sparse polynomial optimization [22] and SDP methods such as the canonical LipSDP framework [14] have been successful in providing tighter Lipschitz bounds. SDP-based methods specifically exploit the slope-restrictedness of the activation functions to cast the problem of estimating a Lipschitz constant as a linear matrix verification problem. However, the computational cost of such methods explodes as the number of layers increases. A common strategy to address this is to ignore some coupling constraints among the neurons to reduce the number of decision variables, yielding a more scalable algorithm at the expense of estimation accuracy [14]. Another strategy is to exploit the sparsity of the SDP using graph-theoretic approaches to decompose it into smaller linear matrix inequalities (LMI) [15][28]. Along similar lines, [21] and [29] employ a dissipativitybased method and dynamic convolutional partition respectively to derive layer-wise LMIs that are applicable to both FNNs and CNNs. Very recent developments also focus on enhancing the scalability of SDP-based implementations through eigenvalue optimization and memory improvement [20], which are compatible with autodiff frameworks such as PyTorch and TensorFlow.

2 Problem Formulation and Background

Notation. We define Z N = { 1 , . . . , N } , where N is a natural number excluding zero. A symmetric positive-definite matrix P ∈ R n × n is represented as P > 0 (and as P ≥ 0 , if it is positive semidefinite). We denote the largest singular value or the spectral norm of matrix A by σ max ( A ) . The set of positive semi-definite diagonal matrices is written as D + .

2.1 Problem Formulation

We consider a feedforward neural network (FNN) of l layers with input z ∈ R d 0 and output y ∈ R d l defined as y = f ( z ) . The function f is recursively formulated with layers L i , i ∈ Z l , defined as

where v ( i ) = W i z ( i -1) + b i with W i and b i representing the weight and bias for layer L i respectively, and ϕ : R d i → R d i is a nonlinear activation function that acts element-wise on its argument. The last layer L l is termed the output layer . We denote the number of neurons in layer L i by d i , i ∈ Z l . Definition 1. A function f : R d 0 → R d l is Lipschitz continuous on Z ⊆ R d 0 if there exists a constant L > 0 such that ∥ f ( z 1 ) -f ( z 2 ) ∥ 2 ≤ L ∥ z 1 -z 2 ∥ 2 , ∀ z 1 , z 2 ∈ Z . The smallest positive L satisfying this inequality is termed the Lipschitz constant of the function f .

̸

Without loss of generality, we assume W i = 0 , i ∈ Z l , as any weights being 0 will lead to the trivial case where the output corresponding to any input will remain the same after that layer. Our goal is to provide a scalable approach to give an efficient and accurate upper bound for the Lipschitz constant L > 0 . Note that the proofs of all the theoretical results in this paper are included in Appendix A.

2.2 Preliminaries

We begin with a slope-restrictedness property satisfied by most activation functions, which is typically leveraged to to derive SDPs for Lipschitz certificates [14].

Assumption 1 (Slope-restrictedness) . For the neural network defined in (1) , the activation function ϕ is slope-restricted in [ α, β ] , α < β in the sense that ∀ v 1 , v 2 ∈ R n , we have α ( v 1 -v 2 ) ≤ ϕ ( v 1 ) -ϕ ( v 2 ) ≤ β ( v 1 -v 2 ) element-wise. Consequently, we have that for ∀ Λ ∈ D + ,

Now, we can obtain an upper bound for the Lipschitz constant as follows; this result is equivalent to the well-known LipSDP framework [14].

Theorem 1 (LipSDP) . For the FNN (1) satisfying Assumption 1, if there exists F > 0 and positive diagonal matrices Λ i ∈ D + , i ∈ Z l -1 such that with p = αβ and m = α + β 2 , then ∥ ∥ ∥ z ( l ) 2 -z ( l ) 1 ∥ ∥ ∥ 2 ≤ √ 1 /F ∥ ∥ ∥ z (0) 2 -z (0) 1 ∥ ∥ ∥ 2 , which provides a sufficient condition for the Lipschitz constant L to be upper bounded by √ 1 /F .

Remark 1 . LipSDP provides three variants that tradeoff accuracy and efficiency, namely, LipSDPNetwork, LipSDP-Neuron, and LipSDP-Layer, whose scalability increases sequentially at the expense of decreased accuracy. However, [30] provides a counterexample showing that the Lipschitz estimate from LipSDP-Network is not a strict upper bound; thus, only LipSDP-Neuron, and LipSDPLayer are valid. Theorem 1 here directly corresponds to LipSDP-Neuron. If all Λ i , i ∈ Z l -1 in (3) are set to multiples of identity matrices, that is, λ i I , i ∈ Z l -1 , then it corresponds to LipSDP-Layer.

Assumption 1 holds for all commonly used activation functions; for example, it holds with α = 0 , β = 1 , that is, p = 0 , m = 1 / 2 for the ReLU, sigmoid, tanh, exponential linear functions. Therefore, we focus on this case in this work.

3 Methodology

We now develop two fast compositional algorithms based on LipSDP-Layer and Lipschitz-Neuron respectively. Both algorithms are not only scalable and significantly faster, but also provide comparable estimates for the Lipschitz constant.

3.1 Exact Decomposition

We circumvent direct solution of the large matrix inequality in (3), which becomes computationally prohibitive as the FNN (1) grows deeper. Instead, we develop a sequential block Cholesky decomposition method, akin to the technique introduced in [31], also expanded in [32, 33]. We first restate Lemma 2 of [31] below.

Theorem 2 (Restatement of Lemma 2 of [31]) . A symmetric block tri-diagonal matrix defined as is positive definite if and only if X i > 0 , ∀ i ∈ { 0 } ∪ Z l -1 , where

Theorem 3. Let P l be defined as in (3) with p = 0 , m = 1 / 2 . Then, the Lipschitz certificate P l > 0 holds if and only if the following sequence of matrix inequalities is satisfied:

where

Theorem 3 provides an exact decomposition of (3), and allows us to establish necessary and sufficient conditions through small matrix inequalities that scale with the size of the weight matrices of each layer, rather than that of the entire network. To accurately estimate the Lipschitz constant, we need to decide on Λ i , i ∈ Z 1 -1 that generate a tight upper bound at the last stage. In other words, we want M l -1 -FW T l W l > 0 to yield the smallest estimate for √ 1 /F . In the following subsection, we provide compositional algorithms to decide the appropriate Λ i , i ∈ Z 1 -1 sequentially, so that we only need to solve one small problem corresponding to each layer.

3.2 Compositional Algorithms

We first propose two practical algorithms here. The theory supporting the algorithms and the geometric intuition are deliberately deferred, and will be thoroughly discussed in a the next subsection.

The first algorithm, ECLipsE , explores the geometric features that enables us to provide an accurate Lipschitz estimate by solving small semidefinite programs (SDPs), which are of the size of the weight matrices on each layer. The second algorithm, ECLipsE-Fast relaxes the sub-problems at each stage and yields a closed-form solution for each sub-problem that makes it extremely fast. These algorithms represent different trade-offs between efficiency and accuracy; one may choose ECLipsE if pursuing accuracy, and ECLipsE-Fast for applications where time is of the essence.

We observe in (7) that M i is obtained in a recursive manner and depends on Λ i and M i -1 , i ∈ Z l -1 . Therefore, we decide Λ i and then calculate M i for i ∈ Z l -1 sequentially. Thus, these two algorithms can be implemented layer-by-layer in a compositional manner.

Concretely, for ECLipsE , we obtain Λ i , i ∈ Z l -1 at each stage i using the information from the next layer, i.e. W i +1 , by solving the following small SDP :

For ECLipsE-Fast , Λ i is reduced to λ i I , i ∈ Z l -1 and λ i is calculated in closed-form as

Note that this completely eliminates the need to solve matrix inequality SDPs altogether. At last, after all Λ i s, i ∈ Z l -1 are decided, we obtain the smallest 1 /F , which yields the smallest Lipschitz estimate L = √ 1 /F , as follows

Remark 2 . We choose to directly calculate the smallest 1 /F rather than first derive the largest F . This is because obtaining the largest F first involves taking the inverse of W T l W l , which can cause numerical issues due to potential singularity of W T l W l . In contrast, directly calculating the smallest 1 /F involves taking the inverses of M l -1 , which is already guaranteed to be strictly positive definite at layer l -1 when deciding Λ l -1 .

We summarize the algorithms as one in Algorithm 1. Algorithms ECLipsE and ECLipsE-Fast are respectively preferable based on whether the priority is on accuracy or speed.

Algorithm 1 ECLipsE and ECLipsE-Fast

3.3 Theory

Now we dive into the cascade structure of feed-forward neural networks and demonstrate the theory behind the two algorithms. We analyze the compositional algorithms in Section 3.2 in a backward manner, starting with the output layer. After all Λ i , i ∈ Z l -1 are decided, M i > 0 , i ∈ Z l -2 hold. From Theorem 3, it remains to guarantee that M l -1 -FW T l W l > 0 , and consequently, (10), for which we state the following result.

Proposition 1. For given Λ i , i ∈ Z l -1 that satisfies M i > 0 , i ∈ Z l -2 , the tightest upper bound for Lipschitz constant is L = √ σ max ( W T l W l ( M l -1 ) -1 ) .

Lemma 1. If M i > 0 , then W T i +1 W i +1 ( M i ) -1 and W i +1 ( M i ) -1 ( W i +1 ) T share the same non-zero eigenvalues.

Now, at stage l -1 , when deciding Λ l -1 , Λ i , i ∈ Z l -2 are fixed and thus M l -2 is fixed. According to Proposition 1, we would like to choose Λ l -1 such that σ max ( W T l W l ( M l -1 ) -1 ) , where M l -1 is a function of Λ l -1 , is as small as possible. We have the following result.

Note that at stage i , it is guaranteed that M i > 0 . Taking i = l -1 , Lemma 1 infers that it is equivalent to minimize σ max ( W l ( M l -1 ) -1 W T l ) when deciding on Λ l -1 . Note that M l -1 > 0 , and consequently, the existence of M -1 l -1 is already guaranteed when we reach the last stage. For the sake of conciseness, we define F i ≜ W i ( M i -1 ) -1 W T i i ∈ Z l -1 . From (7), M i = Λ i -1 4 Λ i F i Λ i . We further write out the recursive expression for F i as

Lemma 2. For any constant γ ∈ (0 , 1) , any Λ i ∈ D + that satisfies M i = Λ i -1 4 Λ i F i Λ i > 0 is also a feasible solution for ˜ M i ≜ Λ i -1 4 Λ i ( γ F i )Λ i > 0 . In other words, the feasible region { Λ i : M i > 0 , Λ i ∈ D + } ⊆ { Λ i : ˜ M i > 0 , Λ i ∈ D + } .

Lemma 2 gives us the observation that a contraction F i → γ F i , γ ∈ (0 , 1) yields a larger feasible space for Λ i ∈ D + to ensure M i > 0 . Meanwhile, (11) shows that for any given Λ i , a

smaller F i leads to a smaller F i +1 for the next stage. We can characterize how 'small' F i is by its spectral norm σ max ( F i ) . Then, minimizing σ max ( F i ) aligns with our goal of minimizing σ max ( W T l W l ( M l -1 ) -1 ) = σ max ( W l ( M l -1 ) -1 W T l ) = σ max ( F l ) at the last stage. In other words, a smaller F 1 at the start will generally translate to a tighter Lipschitz estimate at output layer if we always choose to minimize the spectral norm σ max ( F i ) at each stage.

Now we focus on how to specifically optimize Λ i , i ∈ Z l -1 . At stage i , the goal is to seek for the Λ i that minimizes σ max ( F i +1 ) , where F i +1 = W i +1 (Λ i -1 4 Λ i F i Λ i ) -1 W T i +1 as in (11). Note that M i -1 and F i are already fixed and can be regarded as constants at the i -th stage.

Proposition 2. If there exists a singular matrix N ≥ 0 such that M i = c i W T i +1 W i +1 + N , with constant c i > 0 , then σ max ( F i +1 ) = 1 /c i , ∀ i ∈ Z l -1 .

In other words, we need to find the largest c i > 0 to minimize σ max ( F i +1 ) = 1 /c i . Recall that M i = Λ i -1 4 Λ i F i Λ i is a function of Λ i . We state the following proposition that is used to derive the small sub-problems at each stage.

Proposition 3. Consider the following optimization problem for ∀ i ∈ Z l -1 .

Then, the optimal value c i is the largest constant such that M i can be written as M i = c i W T i +1 W i +1 + N , where N is some singular matrix such that N ≥ 0 . Moreover, the feasible region for the optimization problem is always nonempty.

Geometric Analysis: We illustrate the process of achieving the largest c i > 0 in Fig. 1. We geometrically represent a positive semidefinite matrix by the ellipsoid generated by the transformation of a unit ball in the Euclidean space by the matrix. For simplicity of exposition, we refer to this ellipsoid as the 'shape' of the matrix. We plot the shapes of M i and W T i +1 W i +1 in green and blue, respectively, in 2D. The positive definiteness of the constraint in (12) is equivalent to the ellipsoid of W T i +1 W i +1 being contained in the ellipsoid corresponding to M i /c i . Specifically, when c i > 1 , Fig. 1a demonstrates the maximum contraction of M i , corresponding to the largest c i , such that ellipsoid of W T i +1 W i +1 is still contained in ellipsoid of c i M i . Similarly, for the case where c i < 1 , Fig. 1b demonstrates the minimum extent (the smallest 1 /c i ) to which M i needs to expand, such that the ellipsoid of W T i +1 W i +1 is contained. Algebraically, in both cases, c i is the ratio of the lengths of the green and pink arrows. By Proposition 2, the resulting ellipsoid (depicted in pink) is M i /c i = W T i +1 W i +1 + N/c i for both cases, and is tangent to the ellipsoid of W T i +1 W i +1 . Moreover, the vector pointing from the origin to the tangency point aligns with the direction of eigenvectors (the grey vector v in the plots) corresponding to the zero eigenvalues of the singular matrix N ≥ 0 .

M;

WTWi+1

N

Ci

—+

1

WtWi+1

(a) Intuition with c i > 1

WWi+1 +

N

M;

Ci

WTWi+1

(b) Intuition with c i < 1

Combining Proposition 2 and 3, we can derive an optimization problem to sequentially find the appropriate Λ i , i ∈ Z l -1 . The first constraint in (12) is quadratic in Λ i , which makes it unattractive for practical purposes. Therefore, we apply the Schur Complement to transform it into the linear matrix inequality (LMI) constraint in (8). Thus, the optimization problem in Proposition 3 becomes equivalent to the SDP in (8), yielding algorithm ECLipsE . Notice that there are several ways to write the Schur complement of the constraint in (12). We choose this specific structure to avoid singularity of the diagonal entries and ensure positive definiteness.

ECLipsE-Fast achieves remarkable speed by further reducing Λ i , i ∈ Z l -1 to a multiple of identity matrix λ i I , where λ i > 0 , and by relaxing the sub-problems. While our goal remains to minimize σ max ( F i +1 ) = σ max ( W i +1 (Λ i -1 4 Λ i F i Λ i ) -1 W T i +1 ) , we intentionally disregard information from W i +1 , and instead focus solely on minimizing the spectral norm of (Λ i -1 4 Λ i F i Λ i ) -1 .

Roughly speaking, a smaller σ max ( (Λ i -1 4 Λ i F i Λ i ) -1 ) yields a smaller σ max ( F i +1 ) . This relaxation allows us to derive a closed-form solution for Λ i , i ∈ Z l -1 as follows.

Proposition 4. Choosing λ i = 2 σ max ( F i ) > 0 minimizes σ max ( (Λ i -1 4 Λ i F i Λ i ) -1 ) where Λ i = λ i I under the constraint that M i = Λ i -1 4 Λ i F i Λ i > 0 . Moreover, this closed-form solution for λ i always satisfies M i > 0 , i ∈ Z l -1 .

By the definition of F i , Proposition 4 matches with (9), yielding algorithm ECLipsE-Fast . Although this relaxation may result in a loss of tightness, the closed-form solution offers the advantage of significantly increased computational speed.

Geometric Analysis: We now demonstrate the geometric analysis behind the development of ECLipsE-Fast and compare it with ECLipsE in the case where c i > 1 (Fig. 2). We also include the case c i < 1 in Appendix B. The key idea behind ECLipsE-Fast is that instead of keeping the shape of M i fixed, and contracting the ellipsoid itself, as in ECLipsE , we first find the largest inscribed ball (dark green) for the ellipsoid of M i . Then, we contract this ball to the maximum extent such that it still contains W T i +1 W i +1 . The resulting ball (dark blue) is precisely the smallest circumscribing ball for the ellipsoid of W T i +1 W i +1 . Note that this approach serves as an approximation for the process of contraction depicted in Fig. 7b (corresponding to ECLipsE ), thus yielding a smaller c i . We use this approximation to achieve a closed-form solution, which significantly increases the computational speed.

WTWi+1

Ci

Mt

(a) Geometric Intuition of

ECLipsE-Fast

Mi

WTtWi+1

N

V

Ci

!

WitIWi+1

Remark 3 . In Lemma 2, the analysis initially fixes the shape of F i . However, when optimizing Λ i , the shape of the feasible region depends on F i , which can vary with different Λ i -1 , i ∈ Z l . Thus, this approximation, which allows for a scalable distributed algorithm to solve the centralized problem (3) introduces an unavoidable but minor tradeoff in achieving global optimality.

4 Experiments

We implement our algorithms † on randomly generated neural networks and ones trained on the MNIST dataset. The details of the experimental setup, and training of the neural networks (both randomly generated and trained on the MNIST dataset) are described in Appendix D.

Baselines. ‡ For ECLipsE , Λ i , i ∈ Z l -1 can have different diagonal entries, which benchmarks to LipSDP-Neuron. For ECLipsE-Fast , Λ i = λ i I , i ∈ Z l -1 , which benchmarks to LipSDP-Layer. Additionally, we compare our Lipschitz estimates to the naive upper bound L naive = ∏ l i =1 ∥ W i ∥ 2 [26], CPLip [16] and LipDiff [20]. The codes for these baselines are available at [34, 35, 20]. Note that LipDiff is accelerated using a node with 2 NVIDIA A100 GPUs (80G) and 512 GB of memory.

4.1 Randomly Generated Neural Networks

We first consider randomly generated networks, where the number of layers are chosen from { 2 , 5 , 10 , 20 , 30 , 50 , 75 , 100 } , and number of neurons are chosen from { 20 , 40 , 60 , 80 , 100 } , amounting to a total of 40 experiments for each algorithm (including the baselines). We quantify the computation time and tightness of the Lipschitz bounds (raw data in Appendix E). The Lipschitz bounds presented in the following figures are normalized to the trivial upper bound for ease of comparison.

Case 1: Varying network depth (number of layers). We select a network with 80 neurons per layer, and demonstrate the scalability of our algorithm as network depth increases. Note that all baseline approaches fail to provide a Lipschitz estimate within a computational cutoff time of 15 min for networks larger than this size (see results in Appendix E). As the number of layers increases, the computation time for CPLip algorithm explodes (the algorithm does not return a Lipschitz estimate within the cutoff time beyond 20 layers); however, CPLip provides the most accurate estimates in smaller networks. LipDiff provides inadmissible Lipschitz estimates even for moderate networks, returning as much as 10-100 times the trivial bound (see Table 2a, Appendix E for the estimates). Also, while LipDiff has similar computational time for smaller networks, computational time grows for deeper networks as recorded in Appendix E Table 2b. Consequently, we do not include these results in the plots. LipSDP-Neuron and LipSDP-Layer are also scalable to some extent; however, they fail for a networks of 30 and 50 layers respectively. In contrast, the computation time for ECLipsE and ECLipsE-Fast stays low and grows only linearly with respect to the number of layers (Fig. 3b). Notably, ECLipsE-Fast is significantly faster (thousands of times) than LipSDP-Layer, owing to the closed-form solution at each stage, while ECLipsE is also considerably faster than LipSDP-Neuron. The Lipschitz estimates given by algorithms ECLipsE and ECLipsE-Fast are very close to the ones from LipSDP-Neuron and LipSDP-Layer respectively (Fig. 3a), and outperform the trivial bound. As the number of layers increases, the normalized Lipschitz estimates are smaller, indicating that our algorithms are well-suited to very deep networks.

Tightness of Lipschitz Estimates for FNN with 80 Neurons

0

20

40

60

80

100

Number of Layers

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ECLipsE

ECLipsE-Fast

CPLip

(a) Lipschitz estimates normalized to trivial bound

0

20

40

60

80

100

Number of Layers

0

50

100

150

200

250

300

350

400

Time

ECLipsE

ECLipsE-Fast

CPLip

(b) Computation time (seconds)

Time Used for FNN with 80 Neurons

Case 2: Varying neural network width (number of neurons per layer). Wenowexamine the performance of our algorithms for wider (more hidden neurons per layer), rather than deeper networks (with more layers), and demonstrate the results for networks with 20 and 50 layers respectively (Fig. 4). While the complete raw data is presented in Appendix E, we discuss the results for 20 and 50 layer networks here, since they represent the network sizes where different baselines fail to return Lipschitz estimates beyond the computation cutoff time of 15 min. Note that while LipDiff also manages to generate estimates for all network sizes in our 50 layers case, it once again provides inadmissible Lipschitz constants, returning as much as 10 4 -10 6 times the trivial bound. Therefore, we do not include these results in Fig. 4 (see Tables 3a and 3b in Appendix E for the estimates and computation time.) We can observe from Figs. 4b and 4d that the computation time needed for CPLip, LipSDP-Layer, and LipSDP-Neuron significantly increases with the number of neurons, while the computation time of our method still grows linearly. Meanwhile, the Lipschitz estimates from algorithms ECLipsE and ECLipsE-Fast are close to the ones from LipSDP-Neuron and LipSDP-Layer respectively (Figs. 4a and 4c). Thus, we can conclude that our method significantly improves scalability for wider neural networks.

Case 3: Comparison with LipSDP implementations. In order to address the scalability issue as the size of the network grows, LipSDP utilizes a splitting approach, where the network is split into smaller sub-networks and the Lipschitz constants for each sub-network are composed at the end to obtain the final estimate. We benchmark our approach with respect to the performance of LipSDPLayer and LipSDP-Neuron considering different sub-network sizes. Note that our algorithms do not require any splitting , since they remain scalable to large networks. As the FNNs are larger than the ones in previous cases, we change the cutoff time to 30 minutes. We conduct two sets of experiments

Tightness of Lipschitz Estimates for FNN with 20 Layers

20

30

40

50

60

70

80

90

100

Number of Neurons

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ECLipsE

ECLipsE-Fast

CPLip

(a) Lipschitz estimates normalized to trivial bound

Tightness of Lipschitz Estimates for FNN with 50 Layers

20

30

40

50

60

70

80

90

100

Number of Neurons

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ECLipsE

ECLipsE-Fast

CPLip

(c) Lipschitz estimates normalized to trivial bound

20

30

40

50

60

70

80

90

100

Number of Neurons

0

100

200

300

400

500

600

700

800

Time

ECLipsE

ECLipsE-Fast

CPLip

Time Used for Lipschitz Estimates for FNN with 20 Layers

(b) Computation time (seconds)

Time Used for Lipschitz Estimates for FNN with 50 Layers

20

30

40

50

60

70

80

90

100

Number of Neurons

0

100

200

300

400

500

600

700

Time

ECLipsE

ECLipsE-Fast

CPLip

(d) Computation time (seconds)

to study how our algorithms perform on considerably deep neural networks and how network width affects these results.

In the first set of experiments, we consider FNNs with 100 layers, with the number of neurons chosen from the set {80,100,120,140,160}. The splitting sizes for LipSDP-Neuron and LipSDP-Layer are 3, 5 and 10. We represent different FNN sizes by shapes and different algorithms by the color in

0.95

80neurons

0.9

100 neurons

120 neurons

140 neurons

0.85

160neurons

NormalizedEstimates

ECLipsE

0.8

ECLipsE-Fast

0.75

0.7

0.65

0.6

0.55

0

200

400

600

800

1000

1200

1400

1600

1800

Time Used

Fig. 5. By plotting the normalized Lipschitz estimates and computation times on the two axes, we illustrate how efficient and accurate an algorithm is by how close the corresponding data point is to the origin. We observe that all the data points for ECLipsE-Fast are at the leftmost extreme of the plot, indicating that it is the most efficient algorithm. Further, ECLipsE-Fast also outperforms the red cluster (LipSDP-layer with the network split into 3) in both tightness and speed. Comparing data points of the same shape, ECLipsE-Fast outperforms LipSDPLayer for all sub-network splits both in terms of the Lipschitz estimate and the computation time. Finally, the data points corresponding to ECLipsE are clustered at the bottom left, demonstrating that it is relatively more accurate and efficient than all LipSDP methods, no matter how the network is split.

In the second set of experiments, we explore even wider networks. Specifically, we choose a fairly deep neural network with 50 layers and vary the width from 150 to 1000. The splitting size for LipSDP-Neuron and LipSDP-Layer is 5. The resulting Lipschitz estimates (normalized with respect to trivial upper bounds) and the computation time are provided in Tables 4a and 4b of Appendix

E due to space limitations. From these results, we observe that ECLipsE-Fast is extremely fast even for very wide networks, with a running time of only 15.63 seconds for a network width of 1000, while the computation time for LipSDP-Layer grows significantly. Also, while ECLipsE fails when the width reaches 300, it is comparable to LipSDP-Neuron split into 5 sub-networks in terms of time performance.

Remark 4 . We notice that when the neural networks are significantly wide, ECLipsE takes more than 30 minutes while ECLipsE-Fast remains efficient. This observation can be explained by examining the computational complexity of these algorithms. Note that we directly state the computational complexity of each algorithm here for brevity; the detailed derivations are included in Appendix C. Suppose a neural network has n hidden layers with m neurons. Then, the computational cost for LipSDP and ECLipsE are O ( n 4 m 4 ) and O ( nm 4 ) respectively. We can observe that the complexity is significantly decreased in terms of the depth, but is the same in terms of the width, immediately indicating the advantage for deep networks. Nevertheless, as m grows, the difference between O ( n 4 m 4 ) and O ( nm 4 ) is still drastically enhanced, especially with large n . More importantly, for ECLipsE-Fast , the computational cost drops to O ( nm 3 ) . This is the fastest one can expect if the weights on each layer are treated as a whole.

4.2 Neural Networks Trained on MNIST.

We now demonstrate our algorithms on four networks trained on the MNIST dataset (see Appendix D for details) to achieve an accuracy of at least 97%. The resulting networks are not very deep (3 layers), with 100, 200, 300, and 400 neurons. We set a computational cutoff time of 30 min to obtain Lipschitz estimates. As described in the note on Baselines earlier in this section, ECLipsE is benchmarked against LipSDP-Neuron and ECLipsE-Fast is benchmarked against the faster LipSDP-Layer due to their mathematical structure. From Fig. 6b, we can see that ECLipsE-Fast is significantly faster than LipSDP-Layer, while ECLipsE is also considerably faster than LipSDPNeuron. Note that all algorithms provide very similar Lipschitz estimates (Fig. 6a). Therefore, for networks that are not very deep, such as those in this example, ECLipsE-Fast is the optimal choice, since it significantly outperforms all algorithms in terms of speed, while the approximation error due to the closed-form solution is not too significant compared to the baselines.

Tightness of Lipschitz Estimates for FNN Trained on MNIST

100

150

200

250

300

350

400

Number of Neurons

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ECLipsE

ECLipsE-Fast

100

150

200

250

300

350

400

Number of Neurons

0

200

400

600

800

1000

1200

1400

1600

Time

ECLipsE

ECLipsE-Fast

(b) Computation time (seconds)

Time Used for Lipschitz Estimates for FNN Trained on MNIST

5 Conclusion

Wepropose a scalable approach to estimate Lipschitz constants for deep neural networks by developing a new matrix decomposition that yields two fast algorithms. Our experiments demonstrate that our algorithms significantly outperform the state-of-the-art in terms of computation speed, while providing comparable Lipschitz estimates. We envision that further computational speedup can be achieved through sparse matrix multiplication and eigenvalue estimation techniques, and leveraging autodiff frameworks, along the lines of [20]. While we can unroll the convolutional layers in CNN structure to a large fully connected neural network layer to apply ECLipsE and ECLipsE-Fast to estimate Lipschitz constant, better compositional methods that are tailored to feature the convolutional layers are expected for future work. Similarly, other architectures, such as residual networks, present additional challenges due to their unique structures and will be considered in future research.

Acknowledgment . This work was partially supported by the Air Force Office of Scientific Research grant, FA9550-23-1-0492.

References

M. Fazlyab, M. Morari, and G. J. Pappas, 'Safety verification and robustness analysis of neural networks via quadratic constraints and semidefinite programming,' IEEE Transactions on Automatic Control , vol. 67, no. 1, pp. 1-15, 2020.

T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, 'Spectral normalization for generative adversarial networks,' in International Conference on Learning Representations , 2018.

C. Finlay, A. M. Oberman, and B. Abbasi, 'Improved robustness to adversarial examples using lipschitz regularization of the loss,' 2018.

Y. Tsuzuku, I. Sato, and M. Sugiyama, 'Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks,' Advances in neural information processing systems , vol. 31, 2018.

L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, 'Safe learning in robotics: From learning-based control to safe reinforcement learning,' Annual Review of Control, Robotics, and Autonomous Systems , vol. 5, pp. 411-444, 2022.

H. Yin, P. Seiler, and M. Arcak, 'Stability analysis using quadratic constraints for systems with neural network controllers,' IEEE Transactions on Automatic Control , vol. 67, no. 4, pp. 19801987, 2021.

A. Aswani, H. Gonzalez, S. S. Sastry, and C. Tomlin, 'Provably safe and robust learning-based model predictive control,' Automatica , vol. 49, no. 5, pp. 1216-1226, 2013.

W. G. Y. Tan and Z. Wu, 'Robust machine learning modeling for predictive control using lipschitz-constrained neural networks,' Computers & Chemical Engineering , vol. 180, p. 108466, 2024.

Y. Xu and S. Sivaranjani, 'Learning dissipative neural dynamical systems,' IEEE Control Systems Letters , vol. 7, pp. 3531-3536, 2023.

P. L. Bartlett, D. J. Foster, and M. J. Telgarsky, 'Spectrally-normalized margin bounds for neural networks,' Advances in neural information processing systems , vol. 30, 2017.

H. Zhang, Z. Zheng, and J. Lavaei, 'Gradient-based algorithms for convex discrete optimization via simulation,' Operations research , vol. 71, no. 5, pp. 1815-1834, 2023.

C. Herrera, F. Krach, and J. Teichmann, 'Estimating full lipschitz constants of deep neural networks,' stat , vol. 1050, p. 8, 2020.

A. Virmaux and K. Scaman, 'Lipschitz regularity of deep neural networks: analysis and efficient estimation,' Advances in Neural Information Processing Systems , vol. 31, 2018.

M. Fazlyab, A. Robey, H. Hassani, M. Morari, and G. Pappas, 'Efficient and accurate estimation of lipschitz constants for deep neural networks,' Advances in Neural Information Processing Systems , vol. 32, 2019.

A. Xue, L. Lindemann, A. Robey, H. Hassani, G. J. Pappas, and R. Alur, 'Chordal sparsity for lipschitz constant estimation of deep neural networks,' in 2022 IEEE 61st Conference on Decision and Control (CDC) , pp. 3389-3396, IEEE, 2022.

P. L. Combettes and J.-C. Pesquet, 'Lipschitz certificates for layered network structures driven by averaged activation operators,' SIAM Journal on Mathematics of Data Science , vol. 2, no. 2, pp. 529-557, 2020.

M. Jordan and A. G. Dimakis, 'Exactly computing the local lipschitz constant of relu networks,' Advances in Neural Information Processing Systems , vol. 33, pp. 7344-7353, 2020.

Z. Wang, G. Prakriya, and S. Jha, 'A quantitative geometric approach to neural-network smoothness,' Advances in Neural Information Processing Systems , vol. 35, pp. 34201-34215, 2022.

M. Fazlyab, T. Entesari, A. Roy, and R. Chellappa, 'Certified robustness via dynamic margin maximization and improved lipschitz regularization,' Advances in Neural Information Processing Systems , vol. 36, 2024.

Z. Wang, B. Hu, A. J. Havens, A. Araujo, Y. Zheng, Y. Chen, and S. Jha, 'On the scalability and memory efficiency of semidefinite programs for lipschitz constant estimation of neural networks,' in The Twelfth International Conference on Learning Representations , 2024.

P. Pauli, D. Gramlich, and F. Allgöwer, 'Lipschitz constant estimation for 1d convolutional neural networks,' in Learning for Dynamics and Control Conference , pp. 1321-1332, PMLR, 2023.

F. Latorre, P. Rolland, and V. Cevher, 'Lipschitz constant estimation of neural networks via sparse polynomial optimization,' arXiv preprint arXiv:2004.08688 , 2020.

R. Wang and I. Manchester, 'Direct parameterization of lipschitz-bounded deep networks,' in International Conference on Machine Learning , pp. 36093-36110, PMLR, 2023.

A. Araujo, A. Havens, B. Delattre, A. Allauzen, and B. Hu, 'A unified algebraic perspective on lipschitz neural networks,' arXiv preprint arXiv:2303.03169 , 2023.

A. Havens, A. Araujo, S. Garg, F. Khorrami, and B. Hu, 'Exploiting connections between lipschitz structures for certifiably robust deep equilibrium models,' Advances in Neural Information Processing Systems , vol. 36, 2024.

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, 'Intriguing properties of neural networks,' arXiv preprint arXiv:1312.6199 , 2013.

Z. Shi, Y. Wang, H. Zhang, J. Z. Kolter, and C.-J. Hsieh, 'Efficiently computing local lipschitz constants of neural networks via bound propagation,' Advances in Neural Information Processing Systems , vol. 35, pp. 2350-2364, 2022.

M. Newton and A. Papachristodoulou, 'Exploiting sparsity for neural network verification,' in Learning for Dynamics and Control , pp. 715-727, PMLR, 2021.

Y. Sulehman and T. Mu, 'Scalable lipschitz estimation for cnns,' arXiv preprint arXiv:2403.18613 , 2024.

P. Pauli, A. Koch, J. Berberich, P. Kohler, and F. Allgöwer, 'Training robust neural networks using lipschitz bounds,' IEEE Control Systems Letters , vol. 6, pp. 121-126, 2021.

E. Agarwal, S. Sivaranjani, V. Gupta, and P. Antsaklis, 'Sequential synthesis of distributed controllers for cascade interconnected systems,' in 2019 American Control Conference (ACC) , pp. 5816-5821, IEEE, 2019.

E. Agarwal, S. Sivaranjani, V. Gupta, and P. J. Antsaklis, 'Distributed synthesis of local controllers for networked systems with arbitrary interconnection topologies,' IEEE Transactions on Automatic Control , vol. 66, no. 2, pp. 683-698, 2020.

E. Agarwal, S. Sivaranjani, V. Gupta, and P. Antsaklis, 'Compositional verification of passivity for cascade interconnected nonlinear systems,' in 2020 28th Mediterranean Conference on Control and Automation (MED) , pp. 319-324, IEEE, 2020.

A. Robey, 'Lipsdp: Lipschitz estimation for neural networks.' https://github.com/ arobey1/LipSDP , 2024. Accessed: 2024-05-22.

A. Xue, 'Chordal-lipsdp: A chordally sparse formulation of the lipsdp technique for bounding lipschitz constants of a feedforward neural network.' https://github.com/ AntonXue/chordal-lipsdp , 2024. Accessed: 2024-05-22.

Appendices

A Technical Proofs

Proof of Relation in Assumption 1. We show how the slope-restrictedness of the activation function implies (2). The inequality α ( v 1 -v 2 ) ≤ ϕ ( v 1 ) -ϕ ( v 2 ) ≤ β ( v 1 -v 2 ) that holds elementwise yields

where subscript i indexes the i th element of the vector. Summing all the inequalities for i ∈ R n and letting Λ = diag ( λ 1 , λ 2 , ..., λ n ) , we have

In other words, we have the following quadratic inequality

which can be directly rewritten as (2) with p = αβ and m = α + β 2 .

Proof of Theorem 1 . For any two inputs z (0) 1 and z (0) 2 , let z ( i ) 1 , z ( i ) 2 , v ( i ) 1 , v ( i ) 2 , i ∈ Z l be computed as in (1). Define ∆ z ( i ) = z ( i ) 1 -z ( i ) 2 , i ∈ { 0 } ∪ Z l and ∆ v ( j ) = v ( j ) 1 -v ( j ) 2 , j ∈ Z l . By both left and right multiplying the left hand side in (3) by the vector [∆ z (0) , ∆ z (1) , ..., ∆ z ( l -1) ] T , we have

Now, we show that every summand in the above inequality (13) is negative semidefinite. In fact, with Assumption 1, using notation ∆ v ( i ) = v ( i ) 1 -v ( i ) 2 and ∆ ϕ ( v ( i ) ) = ϕ ( v ( i ) 1 ) -ϕ ( v ( i ) 2 ) , i ∈ Z l , and taking Λ = Λ i , we can write

Note that ∆ v ( i ) = ( W i z ( i -1) 1 + b i ) -( W i z ( i -1) 2 + b i ) = W i ∆ z ( i -1) and ∆ ϕ ( v i ) = ∆ z i . We can express these relationships in matrix form as

Substituting (15) into (14), we have

Combining (13) and (16), we have

Similarly, as the last layer is a linear layer, ∆ z ( l ) = ∆ v ( l ) = W l ∆ z ( l -1) . Then (17) is exactly

yielding a upper bound √ 1 /F for the Lipschitz constant.

Proof of Theorem 2 . Applying Lemma 2 in [31], we define

and

Then, with P i and R i defined above, we directly have

In other words, the sufficient and necessary condition X i > 0 , ∀ i ∈ { 0 } ∪ Z l -1 is equivalent to

which is the same as (21).

Proof of Proposition 1 . As M i > 0 , i ∈ Z l -2 has been guaranteed, it remains to ensure that M l -1 -FW T l W l > 0 by Theorem 3. This is equivalent to M l -1 /F > W T l W l . Therefore, the smallest possible 1 /F is σ max ( W T l W l ( M l -1 ) -1 ) . Then by Theorem 1, the upper bound for the Lipschitz constant is √ 1 /F = √ σ max ( W T l W l ( M l -1 ) -1 ) .

Proof of Lemma 1 . M i > 0 indicates ( M i ) -1 > 0 . Then for ∀ v 0 = 0 ,

̸

̸

meaning that all eigenvalues of W i +1 ( M i ) -1 W T i +1 are non-negative. As W i +1 = 0 , we know that the largest eigenvalue is positive and should be the same as σ max ( W i +1 ( M i ) -1 W T i +1 ) > 0 . Now consider any non-zero eigenvalue λ a of matrix W i +1 ( M i ) -1 W T i +1 and let v a = 0 be its corresponding eigenvector. Then,

̸

Left multiplying both sides with W T i +1 , we have

̸

As λ a = 0 , we know that W T i +1 v a = 0 from (23). (Otherwise, λ a v a = 0 with v a = 0 will lead to λ a = 0 ). With W T i +1 v a = 0 , (24) implies that λ a is also an eigenvalue of W T i +1 W i +1 ( M i ) -1 corresponding to eigenvector W T i +1 v a = 0 .

̸

̸

̸

Conversely, for any non-zero eigenvalue λ b of W T i +1 W i +1 ( M i ) -1 corresponding to eigenvector v b = 0 , we have

̸

̸

Let v c = 1 λ b W i +1 ( M i ) -1 v b . We have v b = W T i +1 v c . Then we substitute v b on the both sides in (25) and obtain

Left multiplying both side with W i +1 ( M i ) -1 , we further have

Let v d = W i +1 ( M i ) -1 W T i +1 v c , we finally have

̸

Similarly, we can conclude that λ b = 0 is the eigenvalue of W i +1 ( M i ) -1 W T i +1 and v d is the corresponding eigenvector.

Proof of Lemma 2 . To prove that { Λ i : M i > 0 , Λ i ∈ D + } ⊆ { Λ i : ˜ M i > 0 , Λ i ∈ D + } , it suffices to show that ˜ M i -M i ≥ 0 for any γ ∈ (0 , 1) . In fact,

At stage i , M i -1 > 0 is guaranteed. Then, similar to (22), we know that F i = W i ( M i -1 ) -1 W T i ≥ 0 Since Λ i ∈ D + and 0 < γ < 1 , implying 1 -γ 4 > 0 , we have 1 -γ 4 Λ i F i Λ i ≥ 0 .

Proof of Proposition 2 . Recall that by definition, F i +1 = W i +1 ( M i ) -1 W T i +1 . Applying Lemma 1, we have σ max ( F i +1 ) = σ max ( W T i +1 W i +1 ( M i ) -1 ) . Meanwhile, with M i = c i W T i +1 W i +1 + N ,

(27)

WIth N ≥ 0 and M i > 0 after Λ i is decided, we show that N c i ( M i ) -1 only has non-negative eigenvalues. As M i > 0 is guaranteed to symmetric according to (7), there exists a symmetric square root for ( M i ) -1 and we denote it to be ( M i ) -1 2 . Then N ( M i ) -1 is similar to ( M i ) -1 2 N ( M i ) -1 2 , thus sharing the same eigenvalues. Furthermore, for ∀ x = 0 ,

̸

It indicates ( M i ) -1 2 N ( M i ) -1 2 only has non-negative eigenvalues. The first equation holds by the symmetry of ( M i ) -1 2 and the second inequality is because of the definition of positive semidefiniteness of N . Therefore, with c i > 0 , the eigenvalues of N c i ( M i ) -1 = 1 c i N ( M i ) -1 are all non-negative. and all eigenvalues of W T i +1 W i +1 ( M i ) -1 should be less than or equal to 1 /c i . On the other hand, as N is singular, N ( M i ) -1 is also singular, thus having eigenvalue 0. Therefore,

̸

Proof of Proposition 3 . We first show that if M i -1 > 0 , the feasible region is always non-empty for ∀ i ∈ Z l -1 . We use σ to denote σ max ( W i ( M i -1 ) -1 W T i ) = σ max ( F i ) . We take Λ i = 2 σ I and c i = 0 . 9 σ · σ max ( W T i +1 W i +1 ) . As W i = 0 , i ∈ Z l , we have σ > 0 and σ max ( F i ) > 0 , ensuring Λ i ∈ D + and c i > 0 . Further,

Therefore, the feasible region at least includes the Λ i and c i we specified, and is thus not empty. To make the feasibility complete, we prove that M i > 0 , ∀ i ∈ { 0 } ⋃ Z l -1 by induction. When i = 0 , M i = M 0 = I > 0 . When it comes to stage i , we have by induction that M i -1 > 0 is true. As Λ i are obtained satisfying (12) with c i > 0 and recall the recursive relation for M i , i ∈ Z l -1 to be M i = Λ i -1 4 Λ i W i ( M i -1 ) -1 W T i Λ i , we have M i > c i W T i +1 W i +1 ≥ 0 .

We now prove by contradiction that the optimal value c i is the largest constant such that M i can be written as M i = c i W T i +1 W i +1 + N , where N is some singular matrix that N ≥ 0 . M i = c i W T i +1 W i +1 + N . Suppose there exists a ˆ c i > c i such that it satisfies all the constraints. Then, from the first constraint in (12), we have

Let v = 0 be the eigenvector of N corresponding to eigenvalue 0 . Observe that

̸

The last step holds because c i -ˆ c i < 0 by definition of ˆ c i . It contradicts (29).

Proof of Proposition 4 . When Λ i = λ i I ,

As M i = Λ i -1 4 Λ i F i Λ i > 0 , we have

Minimizing the above spectrum is equivalent to maximizing the denominator λ i -λ 2 i σ max ( F i ) / 4 in (31), which is quadratic in λ i . To find the optimal λ i , we set the derivative of the denominator with respect to λ i to be 0, and obtain the closed-form solution λ i = 2 σ max ( F i ) .

Moreover, with Λ i = 2 σ max ( F i ) I and F i > 0 guaranteed at stage i , we have

B Geometric Analysis for ECLipsE-Fast

The geometric analysis for algorithm ECLipsE-Fast analogous to Fig. 2, comparing the case where c i > 1 and in other cases are shown in Fig. 7.

(a) For

c

i

>

1

(b) For other cases

Mi

Ci

WW+1

WtWiti

Ci

Mi

C Computational Complexity Derivation

We derive the time complexity for both ECLipsE and ECLipsE-Fast in detail here. Suppose a neural network has n hidden layers with m neurons. Then, the large matrix in Theorem 1 has dimension nm + O (1) and the decision variable is of size nm + O (1) . Note that the computational complexity for solving an LMI with the size of the matrix constraint being size A and the number of decision variables being B is O ( A 3 + A 2 B 2 ) . Therefore, the computational cost for LipSDP is O (( nm + O (1)) 3 + ( nm + O (1)) 2 ( nm + O (1)) 2 ) = O ( n 4 m 4 ) . Contrarily, ECLipsE solves n sub-problems as in Eq. (8), each involving a matrix of size O ( m ) and m decision variables. The corresponding total computational cost is n × ( O ( m 3 + m 2 m 2 )) = O ( nm 4 ) . This directly indicates the advantage of ECLipsE for deep networks. Also, as m grows, the difference between O ( n 4 m 4 ) and O ( nm 4 ) is still significantly enhanced, especially with large n . Regarding ECLipsE-Fast , we note that we do not need to solve any SDPs and the computational cost drops to n × O ( m 3 ) = O ( nm 3 ) . This is the fastest one can expect if the weights on each layer are treated as a whole.

D Experimental Setup and Data Generation

Experimental Setup. All experiments are implemented on a Windows laptop with a 12-core CPU with 16GB of RAM.

Randomly Generated Neural Networks. We set the input dimension to be 4 and the output dimension to be 1. The activation functionsare chosen to be ReLU, and the number of neurons in each hidden layer is set to be the same. We randomly generate weights for each layer to follow the normal

distribution. Also, in order to avoid the case where the Lipschitz constant is too large or too small and may cause numerical issues, we scale the weights on each layer such that the is norm randomly chosen in [0 . 4 , 1 . 8] , following a uniform distribution.

MNIST. For training on the dataset MNIST, the input dimension is 784 and output dimension is 10, which is compatible with the dataset. he activation functionsare chosen to be ReLU, and the number of neurons in each hidden layer is set to be the same. We train neural networks using the SGD optimizer with a learning rate of 0.01 and momentum of 0.9 until they achieve at least 97% accuracy on test data.

E Additional Experimental Results

The Lipschitz constant estimates and computation times for randomly generated neural networks with the number of layers chosen from { 2 , 5 , 10 , 20 , 30 , 50 , 75 , 100 } , and number of neurons are chosen from { 20 , 40 , 60 , 80 , 100 } , are provided below.

F Broader Impacts

This work is primarily theoretical and pertains to obtaining upper bounds on the Lipschitz constant, which can serve as a measure of the robustness of deep neural networks, and does not have any direct societal impact.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in our abstract and introduction accurately reflect the paper's scope and contributions. All the theoretical and experimental results are aligned with the claims made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly discuss all the theoretical assumptions behind our work and the classes of networks for which this work is applicable. We also discuss and demonstrate the computational efficiency of the proposed algorithms for different network sizes.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the complete list of assumptions behind the theoretical results in Sections 2 and 3 of the main paper. We also provide the key ideas and geometric intuition behind the proofs in Section 3. The detailed proofs of all results are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the experimental details are described in the Appendix, and all the code and data are submitted along with the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the code and data are submitted along with the paper, and will be opensourced upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, the paper details the experimental setting and benchmarks in Section 4 and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our algorithms to compute Lipschitz constant bounds are deterministic, and always yield the same result for a given neural network up within numerical accuracy bounds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources and computation time required for each experiment are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics, and ensured that our paper conforms to these regulations.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper is primarily theoretical, and does not have any immediate societal impact. We discuss this in the Appendix section F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not pose any such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the datasets used to train our neural networks, and the code for the papers utilized as benchmarks to evaluate our algorithms are cited in the main text and in our code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: All the code and data are submitted along with the paper, and will be released publicly with detailed documentation upon publication.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper is mainly theoretical and does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper is mainly theoretical and does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents two novel algorithms for computing the Lipschitz constant of feedforward neural networks (NN). The starting point is a previously-known semi-definite programming (SDP) problem which enables to compute the Lipschitz constant. The paper proposes a decomposition of this SDP in sequential subproblems over layers, then relax the subproblems to enable iterative computations across layers, instead of solving a joint problem on all layers. This approach scales much better with both width and depth, as demonstrated by experiments on neural networks at initialization and after training on MNIST.\n\nSTRENGTHS:\n- The paper is overall well-written (see caveat in the weakness section). \n- The question of computing the Lipschitz constant of neural networks is important for a number of downstream tasks. The proposed method provides estimates that are experimentally on-par with approaches based on SDP methods, while being order of magnitude faster.\n\nWEAKNESSES:\n[EDIT (Aug.7): the rebuttal answered my questions adequately. In particular, the method does provide provably-correct upper-bounds.]\n\n- Theoretical results in Section 3.3 are a bit hard to follow, because the section gives the story behind the proposed relaxation, as well as geometric interpretation, but does not provide a main result summarizing the theoretical guarantees of the proposed approach. This is a key point, because provably correct upper-bounds on the Lipschitz constant are of course much preferable. Although it is suggested that the proposed algorithms are indeed provably correct, it is not clearly stated in the paper. So the paper would highly benefit from a clear statement on this fact, as well as a summary of the theoretical results into a theorem (see also Questions).\n- The comparison with methods in the literature is limited to SDP methods, which is OK given that the main contribution is to provide a clever relaxation of these methods, but still a broader comparison would have been interesting. \n- The approach only applies to feedforward NN.\n\nQUESTIONS:\n- I believe that Proposition 4 shows that ECLipsE-Fast provably gives an upper-bound on the true (unknown) Lipschitz constant of the neural network. Is this correct?\n- Does ECLipsE also always give an upper-bound on the true Lipschitz constant? If so, are there assumptions for this to hold? If not, at which steps are approximations made? I guess this should more or less follow from Lemmas 1 and 2, and Propositions 2 and 3, but it is not clearly stated in the paper.\n\nMinor remarks:\n- Authors could consider typesetting their algorithms as “Eclipse” and “Eclipse-Fast” to improve readability.\n- line 199: “such” missing.\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of computing the Lipschitz constant of a neural network. Since computing the exact Lipschitz constant is NP-hard, efforts have been made to obtain tight upper bounds on the Lipschitz constant. This paper builds on the work of LipSDP [1], which involves solving a large matrix verification problem. Since that the large matrix verification problem grows significantly for both deeper and wider networks, this paper proposes a compositional approach to estimate the Lipschitz constants of deep feed-forward neural networks more efficiently. First, the authors obtain an exact decomposition of the large matrix verification problem into smaller sub-problems and, then, exploiting the underlying cascade structure of the network the authors develop two algorithms to compute a bound on the Lipschitz constant:\n- The first algorithm explores the geometric features of the problem and provides a tight estimate of the Lipschitz constant by solving small semidefinite programs (SDPs) that are only as large as the size of each layer.\n- The second algorithm relaxes these subproblems and provides a closed-form solution to each subproblem for extremely fast estimation, altogether eliminating the need to solve SDPs altogether.\n\nFinally, the authors provide extensive experiments to show the different levels of tradeoffs between efficiency and accuracy of the two algorithms. They show that their approach provides a steep reduction in computation time while yielding Lipschitz bounds that are very close to, or even better than, those achieved by state-of-the-art approaches.\n\nSTRENGTHS:\n- The paper is clear and well written. The problem of providing a scalable algorithm for computing the Lipschitz of neural networks is interesting and important. \n- The exact decomposition of the large matrix verification problem into smaller subproblems is very interesting.\n- The two algorithms for computing the sequence of inequalities provide interesting trade-offs. The first algorithm (ECLipsE) looks, if I understand correctly, like a direct improvement of LipSDP, since ECLipsE provides the same value as LipSDP in a more efficient way. \n- The second algorithm also looks interesting as it provides a way to compute the Lipschitz constant without SDPs.\n\nWEAKNESSES:\n- it looks like the approach is restricted to a very limited set of neural networks (feedforward neural networks), can the approach be used for convolutional neural networks?\n- In the experiments, the authors use randomly generated neural networks for their first set of experiments, in my experience it is usually easier to compute SDP on random weight matrices than on trained weight matrices due to conditioning, could the authors provide results of these experiments with trained networks? \n- Can the authors clarify if ECLipsE has to compute all subproblems at the same time or if a sequential approach is possible? \n- I assume that the ECLipsE algorithm uses Matlab for SPD optimization, have the authors tried using a deep learning framework (e.g. PyTorch) for ECLipsE-Fast? Could ECLipsE-Fast be used during training, e.g. for regularization?\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two algorithms, ECLipsE and ECLipsE-Fast, to estimate the Lipschitz constant of a feed-forward neural network. The estimation of the Lipschitzness plays a crucial role in certifying the robustness of neural networks and is known to be an NP-hard problem. The proposed algorithms are based on the LipSDP of Fazlyab et al. (2019), which describes the semidefinite program (SDP) that an upper bound of such a Lipschitz constant should generally satisfy. The authors decompose the original large SDP into smaller layer-wise SDPs to improve the scalability of the original approach. The validity of the resulting methods, ECLipsE and ECLipsE-Fast, is supported by theoretical analyses. Experiments show a steep reduction in computation time while maintaining competitive accuracy compared to the LipSDP.\n\nSTRENGTHS:\nOriginality: The proposed algorithm is novel and clearly distinguishes itself from prior works.\n\nQuality: The motivation of the work is clearly stated and explained. Prior works are also well-discussed. The authors provide thorough and sound mathematical justification and geometrical intuitions for the two algorithms.\n\nClarity: The paper is well-written including the methodology, and the motivation is clear.\n\nSignificance: Compared to LipSDP, the proposed algorithm provides significant improvement in terms of efficiency and addresses concerns stated in the motivations of the paper in the beginning.\n\nOverall, I feel this is a good paper with a promising approach equipped with a thorough and interesting mathematical justification.\n\nWEAKNESSES:\nOverall, in my view, the main weakness of this paper is that there is no explicit comparison with other works trying to improve the scalability of LipSDP (such as [20]). As a result, while the paper indeed improves the original LipSDP in a new way, it is unclear how significant this work is taking into account existing literature. In addition, the limitations should be more carefully discussed. See below for further detailed comments and advice.\n\n### **Quality**\n \n(W-Q1) There are some typos: l.60 (constans), p.3 eqation (3) (index of the bottom right element is i+1 but should be l), l.172 (in the matrix WMW there are 2 unnecessary “L”), l.264 (computatinoal), between l.420 and l.421 (i\\in \\mathbb{R}^n), p.13 equation (18) (if i=0 should be if i=1?), l.477 (functionsare), l.480 (the is norm), \n\n(W-Q2) I feel that some use of words is misleading.  \n\n1. l.60 “We develop a sequential Cholesky decomposition technique to obtain […]“: If this is about Theorem 2, it directly uses the result of Agarwal et al., so maybe “use, employ” would be better than “develop”.\n\n2. l.10 “The first algorithm [...] enables us to provide a *tight* Lipschitz constant”, l.56 “ algorithm [...] enables us to provide an *accurate* Lipschitz”... What do you mean by “tight” and “accurate”? Since the proposed algorithms are using some simplifications, I think that those adjectives should be relative, i.e., only used in comparison with something else. Notably, the experiments (e.g., Figure 3) show that CPLip is far more precise than the proposed algorithms so the authors should clarify the meaning of “tight” and “accurate” (or delete them if there is no justification) when describing their own algorithms.\n\n3. Between l.462 and l.463, you write “$N/ci(M_i)^{-1}\\ge0$”. Semi-positive definiteness was only defined for symmetric matrices but this one is not necessarily symmetric. There should be an easy fix, but it is ambiguous what you mean by this sign. \n\n(W-Q3) In Proposition 3, the authors simplify Theorem 2 by setting $M_i$ to $c_iW_{i+1}^\\top  W_{i+1}+N$ without any proper discussion about this choice (Q2).\n\n(W-Q4) The proof of positive definiteness of $M_i$ is missing in Proposition 3 (Q3). \n\n(W-Q5) (minor) Some references should be adjusted: [2] was accepted at ICLR2018, some capital letters are missing (L of lipschitz)...\n\n### **Clarity**\n\n(W-C1) Some parts may require clarifications (See also Questions):\n\n1. (minor) l.111 “[30] provides a counterexample to the most accurate approach”: The concrete property of the most accurate approach disproved by [30] could be explained in a few words.\n2. (minor) A mathematical comparison of the computational complexity between your approach and LipSDP will largely help the reader to quickly understand the difference in scalability.\n\n(W-C2) While limitations of proposed algorithms are all stated, they are dispersed throughout the paper. A short subsection summarizing them would be useful. (See also Limitations)\n\n(W-C3) It was a bit difficult to understand the idea of the proposed algorithms from Subsection 3.2. Perhaps showing Algorithm 1 in the main text would be better.\n\n### **Significance**\n \n(W-S1) (major) One of the main contributions of this work is the scalability of the proposed algorithms. However, the authors do not compare their method with other accelerations of LipSDP. Therefore, it is difficult to situate their work within the broader context of efforts to improve the scalability of LipSDP. At least, comparing the algorithms with that of Wang et al. (2024) [20] is important to clarify these points. \n\n(W-S2) The algorithms were run on medium-scale neural networks, and it is difficult to imagine the scalability of ECLipsE and ECLipsE-Fast. Experiments with even larger architectures (for example, those for training on CIFAR or ImageNet) would be more convincing.  \n\n(W-S3) I feel experiments of Subsection 4.2 are a little bit redundant (Q4). \n\n(W-S4) Limitations of the algorithms should be discussed in more detail and more explicitly.\n\nQUESTIONS:\nQ1: Is Theorem 1 *necessary* and sufficient? If not, this should be added to the limitations of the work.\n\nQ2: Why did the authors set $M_i$ to $c_iW_{i+1}^\\top  W_{i+1}+N$ in Proposition 3?\n\nQ3: How is the positive definiteness of $M_i$ guaranteed in Equation (6) and Proposition 3?\n\nQ4: What was the motivation to run experiments of Subsection 4.2?\n\nLIMITATIONS:\nThere are several limitations of the algorithms that are worth mentioning:\n\n1. The algorithms (Lemma 1) need that the last weight matrix is full row rank. So, we cannot blindly apply them to any feed-forward neural network.\n\n2. There is a simplification when transforming Theorem 2 into Proposition 3 by limiting the expression of  $M_i $ to $c_iW_{i+1}^\\top  W_{i+1}+N$. This may lead to looser bounds than the original LipSDP.\n\n3. The algorithms cannot be applied to CNNs and residual networks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two novel Lipschitz constant estimation algorithms ECLipsE and ECLipsE-Fast. They are supported by a new decomposition theory developed for the Lip-SDP framework, derived by applying an existing theory (Lemma 2 of [31]). Experiments demonstrate the estimation accuracy and acceleration using toy radom networks and networks trained on MNIST data, by comparing with classical Lipschitz constant estimation method.\n\nSTRENGTHS:\nThe targeted research problem is important and useful.\n\nThe decomposition theory is new and the two estimation algorithms are novel. I highly appreciate the beauty of the application of Lemma 2 of [31] in the proposed theory development.\n\nThe achieved result improvement is satisfactory for deep networks.\n\nWEAKNESSES:\n(1) The proposed algorithm is efficient at addressing network depth, but does not look at the width.  The theory behind self-explains the success of its capability of handling depth.  However, layers with very high numbers of neurons still pose challenges for the sub-problems, e.g., solving for Eq. (6) and Eq. (7). There is a lack of mentioning of this. Also the experiments only studied some modest widths up to 100 neurons. It would be good to see more empirical results with higher neuron numbers in each layer,  to understand the limit of the proposed algorithms on network width.\n\n(2) In experiments, there seems a missing comparison with the “parallel implementation by splitting” version of Lip-SDP as reported in their paper [14], which was proposed to address the depth issue.\n\n(3) The  method description can be improved, e.g., being more organised. The paper can present information that is more important and helpful to practitioners and general readers’ understanding in main paper, while leave some analysis and supporting lemmas to appendix. Personally, I find it helpful to see in the main paper a description of the existing Lemma 2 in [31] and the pseudo code of the proposed algorithm.\n\nQUESTIONS:\nThe authors are invited to address my comments (1) and (2) in weakness section. If possible, it would be good to see some added results to help understand more the width capacity of the proposed algorithms.\n\nLIMITATIONS:\nDiscussion on discussion is pretty limited. For instance, it  can be improved around the network width issue.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors are able to decompose a particular case of LipSDP-neuron exactly into a series of sub-problems leading to the proposed algorithm ECLipSE. In the case of the relaxed LipSDP-layer, it can be shown that each sub-problem can be solved analytically and eliminate the need for solving an SDP. The proposed algorithm ECLipSE-fast can provide Lipschitz estimates of deep-NN quite fast at the expense of increased conservativeness.\n\nSTRENGTHS:\n-The paper is well written and the theoretical arguments are well motivated and connected to the the numerical experiments. The paper mainly builds upon LipSDP, but I believe the theoretical insights to be novel.\n \n-The insight that LipSDP may be simplified into sub-problems is a significant contribution and advances the practical value of LipSDP. In particular, the proposal of ECLipsE-Fast which is an analytical solution to a relaxed sub-problem shows a significant improvement over the previous naive product bound and is very scalable.\n\nWEAKNESSES:\n-It seems that the approach doesn’t yet apply to residual networks or CNN commonly found in state-of-the-art vision models. For this reason, it seems difficult to show improved certified robustness on common image classification benchmarks which would benefit the most from increased scalability (CIFAR10-100, Tiny-imagenet, etc).\n\n-Certified robustness is mentioned as an application at several points in the paper. While the scalable and tighter Lipschitz estimates on random networks and MNIST certainly suggest some improvements, no practical measures of certified robustness are presented (e.g. robust accuracy).  For certified robust accuracy on MNIST, it is common to use 1-Lipschitz parameterization (SLL, AOL, other direct parameterizations) which eliminates the need for Lipschitz estimation. Would applying ECLipSE to composed 1-Lipschitz networks provide significantly tighter estimates in this case?\n\nQUESTIONS:\n-How does the full row rank assumption limit the applications of ECLipSE’s? Is it possible to relax this assumption? It seems like networks considered in this paper are only of constant width or decreasing in output dimension in the case of MNIST which will usually satisfy the full row-rank assumption.\n\n-Can you comment on the slight gap between the Lipschitz constant estimates shown in Appendix E between ECLipsE and LipSDP-Neuron? Is it that LipSDP is not as accurate in larger settings or is ECLipsE somehow slightly conservative?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}