RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Measurements
Our main baseline tool is LiPopt (Latorre et al . , 2020 ) , which is an \ell_{\infty} -FGL estimation tool. 2 2 2 Another method was proposed by Chen et al . ( 2020 ) , however, the code is not available and we are not able to compare it with GeoLIP. Notice that LiPopt is based on the Python Gurobi solver (Gurobi Optimization, LLC, 2022 ) , while we use the MATLAB CVX and MOSEK solver.
LiPopt relies on a linear programming (LP) hierarchy for the polynomial optimization problem. We use LiPopt-k to denote the k -th degree of the LP hierarchy. BruF stands for an brute-force exhaustive enumeration of all possible activation patterns. This is the ground truth for FGL estimations. However, this is an exponential-time search, so we can only run it on networks with a few hidden units. Sample means that we randomly sample 200,000 points in the input space and compute the gradient norm at those points. Notice that this is a lower bound of the true Lipschitz constant, and thus a lower bound of the FGL. MP stands for the weight-matrix-norm-product method. This is a naive upper bound of FGL. We use NGeoLIP to denote the natural SDP relaxations devised in Section 3 , and DGeoLIP to denote the dual SDP Equation 12 for \ell_{\infty} -FGL estimation. Notice that NGeoLIP only applies to two-layer networks.

We use “—” in the result tables to denote that the experimental setting is not in the scope of the tool’s application, and “N/A” to denote the computation takes too much time ( >20 hours).
Network setting
We run the experiments on fully-connected feed-forward neural networks, trained with the MNIST dataset for 10 epochs using the ADAM optimizer (Kingma and Ba, 2015 ) . All the trained networks have accuracy greater than 92\% on the test data. For two-layer networks, we use 8 , 16 , 64 , 128 , 256 hidden nodes. For multiple-layer networks, we consider 3 , 7 , 8 -layer networks, and each hidden layer has 64 ReLU units.
Because MNIST has 10 classes, we report the estimated FGL with respect to label 8 as in Latorre et al . ( 2020 ) , and the average running time per class: we record the total computation time for all 10 classes from each tool, and report the average time per class.
Discussion
We present selected results in Tables 1 and 2 , and related major discussions here. The full results, more experimental setup and additional discussions can be found in Appendix C .
RQ1
In the experiments of LiPopt, we only used LiPopt-2. In theory, if one can go higher in the LP hierarchy in LiPopt, the result becomes more precise.
However, in the case of fully-connected neural networks, using degree-3 in LiPopt is already impractical.
For example, on the simplest network that we used, i.e., the single-hidden-layer neural network with 8 hidden units, using LiPopt-3, one \ell_{\infty} -FGL computation needs at least 200 hours projected by LiPopt. As a result, for all the LiPopt-related experiments, we were only able to run LiPopt-2.
As Latorre et al . ( 2020 ) pointed out, the degree has to be at least the depth of the network to compute a valid bound, so we have to use at least LiPopt-k for k -layer networks. LiPopt is unable to handle neural networks with more than two layers because this requires LiPopt with degrees beyond 2.
Even if we only consider LiPopt-2 on two-layer networks, the running time is still much higher compared to GeoLIP. This demonstrates the great advantage of GeoLIP in terms of scalability compared with LiPopt.
If we compare LiPopt-2 with GeoLIP on two-layer networks from Table 1 , it is clear that GeoLIP produces more precise results. For networks with depth greater than 2 , we can only compare GeoLIP with the matrix-norm-product method. As we can see from all experiments, GeoLIP’s estimation on the FGL is always much lower than MP.

We have also shown that the two-layer network \ell_{\infty} -FGL estimation from GeoLIP has a theoretical guarantee with the approximation factor K_{G}<1.783 ( Theorem 3.3 ). If we compare the two-layer network results from GeoLIP and Sampling in Table 1 , which is a lower bound of true Lipschitz constant, the ratio is within 1.783 . This validates our theoretical claim.
RQ2
In Section 4 , we have demonstrated the duality between NGeoLIP and LipSDP for the \ell_{2} -FGL estimation on two-layer networks, even though the approaches appear drastically different. The experiments show that on two-layer networks, LipSDP and NGeoLIP for \ell_{2} -FGL estimations ( Table 7 in the appendix), and DGeoLIP and NGeoLIP for \ell_{\infty} -FGL estimations produce the same values. These results empirically validate the duality arguments, and also all the related SDP programs.
SDP relaxation
Applying SDP on intractable combinatorial optimization problem was pioneered by the seminal Goemans-Williamson algorithm for the Max-Cut problem (Goemans and Williamson, 1995 ) . For two-layer networks, we have reduced the FGL estimation to the mixed-norm problem, and provide approximation algorithms with ratios compatible with the known optimal constants in the corresponding mixed-norm problems. Improving them can be a very hard task. We also provide a compositional SDP interpretation of LipSDP-Neuron. Although Pauli et al . ( 2022 ) demonstrated the flaw in LipSDP-Network, our compositional SDP interpretation shows that LipSDP-Neuron is correct. In fact, from the compositional SDP interpretation, the program is only constrained by the underlying perturbation geometry and the layer-wise restriction from each hidden unit, so the constraints and objective exactly encode the FGL-estimation problem without additional assumptions. Because often the SDP relaxation for intractable problems gives the optimal known algorithms, we conjecture that GeoLIP and LipSDP are also hard to improve on FGL estimations.

Latorre et al . ( 2020 ) used polynomial optimization to address the \ell_{\infty} -FGL estimation. We argue that approaching FGL-estimations from the perspective of polynomial optimization loses the accurate characterization of this problem. For example, for two-layer networks, we have provided constant approximation algorithms to estimate FGLs in both \ell_{\infty} and \ell_{2} cases. However, for a general polynomial optimization problem on a cube, we cannot achieve constant approximation. For example, the maximum independent set of a graph can be encoded as a polynomial optimization problem over a cube (Motzkin and Straus, 1965 ) , but the maximum independent set problem cannot be approximated within a constant factor in polynomial time unless \mathsf{P}=\mathsf{NP} (Trevisan, 2004 ) .
Related work
Chen et al . ( 2020 ) employed polynomial optimization to compute the true Lipschitz constant of ReLU-networks for \ell_{\infty} -perturbations, and used Lasserre’s hierarchy of SDPs (Lasserre, 2001 ) to solve the polynomial optimization problem. However, their approach is highly tailored to ReLU networks, while ours, like LipSDP, can handle common activations, such as sigmoid and ELU.

Latorre et al . ( 2020 ) also proposed to use LiPopt to estimate the local Lipschitz constant. However, estimating this quantity is not the problem studied in our work, and there are tools specifically designed for local perturbations and the Lipschitz constant (Laurel et al . , 2022 ; Zhang et al . , 2019 ) .

Lipschitz regularization of neural networks is an important task, and recent works (Aziznejad et al . , 2020 ; Bungert et al . , 2021 ; Gouk et al . , 2021 ; Krishnan et al . , 2020 ; Terjék, 2020 ) have investigated this problem. However, here we study a related but different problem, i.e., Lipschitzness measurement of neural networks. Our work can motivate new Lipschitz regularization
techniques.
Conclusion
In this work, we have provided a quantitative geometric framework for FGL estimations, and also algorithms for the \ell_{\infty} -FGL estimation. One important lesson is that when transferring techniques from one perturbations to another ones, we should also transfer the underlying geometry. One future work is to train smooth neural networks using the SDPs proposed in this paper.
Acknowledgments and Disclosure of Funding
The authors thank Vijay Bhattiprolu for introducing recent progress on the mixed-norm problems. The work is partially supported by Air Force Grant FA9550-18-1-0166, the National Science Foundation (NSF) Grants CCF-FMitF-1836978, IIS-2008559, SaTC-Frontiers-1804648, CCF-2046710
and CCF-1652140, and ARO grant number W911NF-17-1-0405. Zi Wang and Somesh Jha are
partially supported by the DARPA-GARD problem under agreement number 885000.
Gradient as operator
If a function g:\mathbb{R}^{m}\rightarrow\mathbb{R} is a differentiable function at a\in\mathbb{R}^{m} , then the total derivative of g at a is

Dg(a)=[\frac{\partial g}{\partial x_{1}}(a),\ldots,\frac{\partial g}{\partial x_{m}}(a)],

and the gradient of g at a is \nabla g(a) is the transpose of Dg(a) . The linear approximation of g at a is \langle Dg(a),dx\rangle . Equivalently, we can view the change of a function with respect to an infinitesimal perturbation as the inner product of \nabla g(a) and dx . In this sense, the gradient acts as an operator on the perturbation.
Differentiable activation
Because we want to upper bound the true Lipschitz constant, we only need to show that the quantity considered in the paper indeed upper bounds the true Lipschitz constant considered in the paper. If the activation function is differentiable, then the neural network f is also differentiable, so Equation 3 is trivially true, as proved and applied in Latorre et al . ( 2020 , Theorem 1 and Equation 4) .
ReLU activation
For ReLU networks, it is true if we have [a,b]=[0,1] . One can consider the (Clarke) generalized Jacobian as in Jordan and Dimakis ( 2020 ) . At each input point, the Clarke Jacobian is contained in \{(W^{1})^{T}\cdot\text{diag}(v^{2})\cdot\cdots\cdot\text{diag}(v^{d})(W^{d})^{T}\mid v^{i}\in[a,b]^{n_{i}}\} . Alternatively, we can also use the perturbation propagation argument in Section 4 to see this upper bound. Note that Raghunathan et al . ( 2018 ) used this interval representation for ReLU’s derivative.
Maximum over hypercube
Now we want to show that the optimization problems over hypercubes considered in this work attain the maximum at the vertices. Without of loss of generality, let us assume the hypercube is [-1,1]^{n} . Otherwise, we can transform the hypercube to [-1,1]^{n} .
Let A\in\mathbb{R}^{m\times n} , x\in\mathbb{R}^{n} , y\in\mathbb{R}^{m} , and z\in\mathbb{R}^{n} .

We will use the following facts

||x||_{1}=\max_{z\in\{-1,1\}^{n}}\langle x,z\rangle ;

\ell_{\infty} is the dual of \ell_{1} ;

Let U\subseteq\mathbb{R}^{n} . When \max_{x,z\in U}\langle Ax,Az\rangle is well-defined, we have \max_{x\in U}\langle Ax,Ax\rangle=\max_{x,z\in U}\langle Ax,Az\rangle .

The first fact is from ||x||_{1}=|x_{1}|+\ldots+|x_{n}|=\max_{z\in\{-1,1\}^{n}}\langle x,z\rangle . The second fact is from Hölder’s inequality for finite-dimensional vector space. For the third one, \langle Ax,Az\rangle is maximized only when Ax=Az . Now we can show that the maximization problems considered in this paper attain the maximum at the hypercube vertices.

\begin{split}\max_{||x||_{\infty}=1}||Ax||_{1}&=\max_{||x||_{\infty}=1,y\in\{-1,1\}^{m}}\langle Ax,y\rangle\quad\quad(From fact (1))\ &=\max_{||x||_{\infty}=1,y\in\{-1,1\}^{m}}\langle x,A^{T}y\rangle\ &=\max_{y\in\{-1,1\}^{m}}||A^{T}y||_{1}\quad\quad\quad\quad\;\;\;(From fact (2))\ &=\max_{x\in\{-1,1\}^{n},y\in\{-1,1\}^{m}}\langle x,A^{T}y\rangle\ &=\max_{x\in\{-1,1\}^{n},y\in\{-1,1\}^{m}}\langle Ax,y\rangle.\end{split} (14)

\begin{split}\max_{||x||_{\infty}=1}||Ax||_{2}^{2}&=\max_{||x||_{\infty}=1}\langle Ax,Ax\rangle\ &=\max_{||x||_{\infty}=1,||z||_{\infty}=1}\langle Ax,Az\rangle\;\;\;\;\;\;\;\;(From fact (3))\ &=\max_{||x||_{\infty}=1,||z||_{\infty}=1}\langle A^{T}Ax,z\rangle.\end{split}

Using the similar idea in Equation 14 , we have

\max_{||x||_{\infty}=1}||Ax||_{2}^{2}=\max_{x\in\{-1,1\}^{n},z\in\{-1,1\}^{n}}\langle Ax,Az\rangle=\max_{x\in\{-1,1\}^{n}}\langle Ax,Ax\rangle.

More generally, in the bilinear forms considered above, if x=x_{1}\otimes\cdots\otimes x_{d} is generated by the tensor product of variables over cubes, we can fix one variable and write x as a matrix product, and then move the fixed variable to the hypercube vertices. We can repeat this process to move all variables to the vertices.
A.2 Additional definitions
For any neural network f , let OPT(f) be the optimal value of Equation 3 . We say an algorithm \mathcal{A} is an approximation algorithm for Equation 3 with approximation ratio \alpha>1 , if OPT(f)\leq\mathcal{A}(f)\leq\alpha OPT(f) .
A.3 SDP for the \infty\rightarrow 1 mixed-norm problem
Recall that for v\in\mathbb{R}^{m} , ||v||_{1}=\max_{||u||_{\infty}=1}\langle u,v\rangle . We can reformulate the mixed-norm problem as follows:

\max_{x\in\{-1,1\}^{n}}||Ax||_{1}=\max_{(x,y)\in\{-1,1\}^{n+m}}\langle Ax,y\rangle.

If we let z=\begin{pmatrix}x^{T}&y^{T}\end{pmatrix} , we can have

\max_{(x,y)\in\{-1,1\}^{n+m}}\langle Ax,y\rangle=\max_{z\in\{-1,1\}^{n+m}}z\cdot B\cdot z^{T},

where B is a (m+n)\times(m+n) matrix. The last m rows and first n columns of B is A , and the rest are 0 : B=\begin{pmatrix}0&0\\
A&0\end{pmatrix} .

The natural SDP relaxation of the \infty\rightarrow 1 mixed-norm problem is:

\maxtr(BX) s.t.\;X\succeq 0,X_{ii}=1, i\in[n+m].

In other words, we treat X as the SDP matrix relaxed from the rank-1 matrix z^{T}\cdot z .
A.4 SDP relaxation and Grothendieck inequalities
In this work, we used the Grothendieck inequality as in Theorem 3.1 :

\max_{u_{i},v_{j}\in B(H)}\sum_{i,j}A_{ij}\langle u_{i},v_{j}\rangle_{H}\leq K_{G}||A||_{\infty\rightarrow 1}, (15)

for any A\in\mathbb{R}^{n\times m} ; and the little Grothendieck inequality:

\max_{u_{i},v_{j}\in B(H)}\sum_{i,j}(A^{T}A)_{ij}\langle u_{i},v_{j}\rangle_{H}\leq\frac{\pi}{2}||A||^{2}_{\infty\rightarrow 2}

Notice that A^{T}A is a PSD matrix.

As discussed in Section A.3 ,

||A||_{\infty\rightarrow 1}=\max_{z\in\{-1,1\}^{n+m}}z\cdot B\cdot z^{T}.

The natural SDP relaxation is

\maxtr(BX) s.t.\;X\succeq 0,X_{ii}=1, i\in[n+m].

Because X\succeq 0 , X=MM^{T} for some M\in\mathbb{R}^{(m+n)\times d} , where d\geq 1 . Let M_{i} be the i -th row vector of M . X_{ij}=\langle M_{i},M_{j}\rangle , and X_{ii}=1 means \langle M_{i},M_{i}\rangle=1 . As a result, \text{tr}(BX)=\sum_{i,j}A_{ij}X_{ij}=\sum_{i,j}A_{ij}\langle M_{i},M_{j}\rangle_{H} , where H is the Hilbert space of \mathbb{R}^{d} equipped with the canonical inner product. Thus, Equation 15 implies that K_{G} is the approximation ratio in the SDP relaxation for the mixed-norm problem.

In contrast, in the mixed-norm problem, the variable to B_{ij} is z_{i}z_{j} , the product of two scalars.
If d=1 in the SDP relaxation, M is a column vector, and X is a rank-1 matrix. In this case, the SDP coincides with the combinatorial problem, because the inner product degenerates to the multiplication of two scalars. Hence, the SDP relaxation can be viewed as a continuous relaxation of a discrete problem, and Equation 15 quantifies this geometric transformation. Another interpretation for the SDP relaxation is that SDP drops the rank-1 constraint in the quadratic formulation of the mixed-norm problem.
A.5 Rescaling from 0-1 cube to norm-1 cube
Now let us show how we transform the 0-1 cube in Equation 4 to a norm-1 cube, and formulate an equivalent optimization problem. As a result, we can apply the SDP program in Equation 5 to compute an upper bound of the \ell_{\infty} -FGL.

Let x_{i}=(t_{i}+1)/2 , where t_{i}\in\{-1,1\} . We have

\begin{split}&\max_{x\in\{0,1\}^{n}}||Ax||_{1}\ =&\max_{x\in\{0,1\}^{n},y\in\{-1,1\}^{m}}y^{T}Ax\ =&\max_{t\in\{-1,1\}^{n},y\in\{-1,1\}^{m}}\frac{1}{2}y^{T}A(t+e_{n}).\end{split} (17)

Let OPT_{1} be the optimal value of

\max_{(t,y)\in\{-1,1\}^{n+m}}y^{T}A(t+e_{n}).

Introduce another variable \tau\in\{-1,1\} , and let OPT_{2} be the optimal value of

\max_{(t,y,\tau)\in\{-1,1\}^{n+m+1}}y^{T}A(t+\tau e_{n})

OPT_{1}=OPT_{2} .

Clearly OPT_{2}\geq OPT_{1} .

Now if (\hat{t},\hat{y},\tau=-1) is an optimal solution to Equation 18 , then (-\hat{t},-\hat{y},\tau=1) is also an optimal solution, so OPT_{2}\leq OPT_{1} .
∎

Now let z=(t,\tau) , and we can verify that y^{T}A(t+\tau e_{n})=y^{T}Bz , where B=\begin{pmatrix}A&Ae_{n}\end{pmatrix} .

As a result, the semidefinite program to the \ell_{\infty} -FGL constant is

\begin{split}\max\;&\frac{1}{2}tr(BX)\ s.t.\;X\succeq 0,X_{ii}=1&,i\in[n+m+1],\end{split}

where B is a (n+1+m)\times(n+1+m) matrix, and B=\begin{pmatrix}0&0&0\\
A&Ae_{n}&0\end{pmatrix} .
A.6 Proof of Theorem 3.2
We will use the cube rescaling techniques introduced in Section A.5 . Alon and Naor ( 2004 ) showed that matrix cut-norm is \mathsf{MAXSNP} -hard. We will show that if one can solve the FGL estimation problem, then one can find the cut norm of a matrix.

Given a matrix A , the cut norm of a matrix A\in\mathbb{R}^{m\times n} is defined as

CN(A)=\max_{x\in\{0,1\}^{n},y\in\{0,1\}^{m}}\langle Ax,y\rangle.

We need to transform y from 0-1 cube to norm-1 cube, so similarly let y_{i}=(t_{i}+1)/2 , where t_{i}\in\{-1,1\} . The we will have

CN(A)=\max_{x\in\{0,1\}^{n},y\in\{0,1\}^{m}}\langle Ax,y\rangle=\frac{1}{2}\max_{x\in\{0,1\}^{n},t\in\{-1,1\}^{m}}\langle Ax,(t+e_{m})\rangle.

Let B=\begin{pmatrix}A\\
e_{m}^{T}A\end{pmatrix} . From above we know that

\max_{x\in\{0,1\}^{n},t\in\{-1,1\}^{m}}\langle Ax,(t+e_{m})\rangle=\max_{x\in\{0,1\}^{n},(t,\tau)\in\{-1,1\}^{m+1}}\langle Bx,(t,\tau)\rangle.

One can then construct a two layer neural network, where the first weight matrix is B^{T} , and the second weight matrix is (1,\ldots,1)\in\mathbb{R}^{n} . Because the network we consider has only one output, the second weight matrix is only a vector. The FGL of this network is exactly twice of the cut norm of A .
∎
A.7 Proof of Theorem 3.3
Let B=\begin{pmatrix}0&0&0\\
A&Ae_{n}&0\end{pmatrix} . Combing Equations 17 , 18 and 5 , the approximation algorithm for Equation 4 where q=1 is induced by the following SDP program:

\begin{split}\max\;&\frac{1}{2}tr(BX)\ s.t.\;X\succeq 0,X_{ii}=1&,i\in[n+m+1].\end{split}

∎
A.8 Natural SDP relaxation of \ell_{2} -FGL estimation
Now let q=2 in Equation 4 , we will have:

\max_{y\in\{0,1\}^{n}}||Ay||_{2}.

In other words, we only need to solve the following program:

\max_{z\in\{0,1\}^{n}}z^{T}(A^{T}A)z

Let M=A^{T}A , then M is a PSD matrix. We have demonstrated the scaling techniques in Section A.5 . Let x\in\{-1,1\}^{n+1} , one can verify that

\max_{z\in\{0,1\}^{n}}z^{T}Mz=\frac{1}{4}\max_{x\in\{-1,1\}^{n+1}}x^{T}\hat{M}x, (20)

where \hat{M}=\begin{pmatrix}M&Me_{n}\\
e_{n}^{T}M&e_{n}^{T}Me_{n}\end{pmatrix} .

It is easy to verify that if M is PSD, \hat{M} is also PSD. Because M=A^{T}A , \hat{M}=(A,Ae_{n})^{T}\cdot(A,Ae_{n}) . Now we can consider the following natural SDP relaxation to \max_{x\in\{-1,1\}^{n+1}}x^{T}\hat{M}x :

\begin{split}\maxtr(\hat{M}X)\ s.t.\;X\succeq 0,X_{ii}=1,i\in&[n+1].\end{split} (21)

This SDP relaxation admits a \frac{\pi}{2} -approximation factor from Equation 16 Rietz ( 1974 ); Nesterov ( 1998 ) .
A.9 Proof of Theorem 3.4
Let \hat{M}=\begin{pmatrix}M&Me_{n}\\
e_{n}^{T}M&e_{n}^{T}Me_{n}\end{pmatrix} , where M=A^{T}A . Combining Equations 19 , 20 and 21 , the approximation algorithm for Equation 4 where q=2 is induced by the following SDP program:

\begin{split}\max\frac{1}{2}\sqrt{tr(\hat{M}X)}\ s.t.\;X\succeq 0,X_{ii}=1,i\in&[n+1].\end{split} (22)

∎
A.10 Comparison with Raghunathan et al . ( 2018 )
Raghunathan et al . ( 2018 ) formulated the following SDP to upper bound the \ell_{\infty} -FGL on two-layer neural networks:

\begin{split}\max\frac{1}{4}tr(CX)\ s.t.\;X\succeq 0,X_{ii}=1,i\in&[n+m+1],\end{split} (23)

where C is a (m+n+1)\times(m+n+1) matrix, and C=\begin{pmatrix}0&0&A^{T}\\
0&0&e_{n}^{T}A^{T}\\
A&Ae_{n}&0\end{pmatrix} .

If we compare Equations 6 and 23 , C=B+B^{T} . Because X is symmetric, \text{tr}(CX)=2\text{tr}{(BX)} . Therefore, Equations 6 and 23 produce the same result.
A.11 Equivalence between the new optimization program and Equation 19
Notice that because u\in\mathbb{R}^{1\times n} , u\Delta\sigma(x) is a scalar. We can view each z_{i} in Equation 19 as \frac{\Delta\sigma(x)_{i}}{\Delta y_{i}} , the derivative of \sigma(x)_{i} without the limit. Therefore, \Delta\sigma(x)_{i}=z_{i}\Delta y_{i} . Recall that from Section 4 , \Delta y_{i}=w_{i}\Delta x , so \Delta\sigma(x)_{i}=w_{i}z_{i}\Delta x , then u\Delta\sigma(x)=\Delta x\sum_{i}^{n}u_{i}z_{i}w_{i}=\Delta x(Az) , where A=W^{T}\text{diag}(u) as defined in Equation 4 .

As a result, from Cauchy–Schwarz inequality, the above objective is

\begin{split}\max_{\Delta x,\Delta\sigma(x)}\frac{(u\Delta\sigma(x))^{2}}{(\Delta x)^{2}}&=\max_{z}(Az)^{2},\ s.t.\;z\in&[a,b]^{n}.\end{split}

This demonstrates the equivalence between the new optimization program and Equation 19 when [a,b]=[0,1] for \sigma=\text{ReLU} .
Tools
We obtain the LiPopt implementation from https://github.com/latorrefabian/lipopt , under the MIT License.
Server specification
All the experiments are run on a workstation with forty-eight Intel ® Xeon ® Silver 4214 CPUs running at 2.20GHz, and 258 GB of memory, and eight Nvidia GeForce RTX 2080 Ti GPUs. Each GPU has 4352 CUDA cores and 11 GB of GDDR6 memory.
Dataset and split
We used the standard MNIST dataset from the PyTorch package (Paszke et al . , 2019 ) . We used the “train” parameter in the MNIST function to split training and testing data.
Single hidden layer
We consider the \ell_{\infty} -FGL estimation on two layer neural networks with different numbers of hidden units. The results are summarized in Tables 3 and 4 .
Multiple hidden layers
We consider the \ell_{\infty} -FGL estimation on 3 , 7 , 8 -layer neural networks. Each hidden layer in the network has 64 ReLU units. The results are summarized in Tables 5 and 6 .
\ell_{2} -FGL estimation
We measure the \ell_{2} -FGL on two-layer networks mainly to compare whether Equation 7 and LipSDP produce the same result. Additionally, we also want to empirically examine the approximation guarantee from Theorem 3.4 . Still, we consider networks with 8 , 16 , 64 , 128 , 256 hidden nodes. The results are summarized in Tables 7 and 8 .
Duality
The results in Table 7 show that the results of LipSDP and GeoLIP on two-layer-network \ell_{2} -FGL estimation are exactly the same, which empirically demonstrates the duality between LipSDP and GeoLIP, as discussed in Section 4 . Though Pauli et al . ( 2022 ) showed that the most precise version of LipSDP is invalid for estimating an upper bound of \ell_{2} -FGL, our dual-program argument shows that the less precise version of LipSDP is correct.
Precision
We showed that GeoLIP’s approximation factor for the \ell_{2} -FGL estimation on two layer networks is \sqrt{\frac{\pi}{2}}\approx 1.253 in Theorem 3.4 . The \ell_{2} -FGL from GeoLIP is very close to the sampled lower bound of true Lipschitz constant in Table 7 . On the other hand, because the result from GeoLIP is an upper bound of FGL, and this result is not much greater than the sampled lower bound of true Lipschitz constant, this empirically demonstrates that the true Lipschitz constant is not very different from the FGL on two-layer networks.
Running time
If we compare the running time in Tables 4 and 8 , the dual program takes more time to solve than the natural relaxation. This is particularly true when the number of hidden neurons increases. From the reported numbers of variables and equality constraints by CVX, the dual program and natural relaxation have similar numbers. It is also observed that the CPU usage is higher when the natural relaxation is being solved.
We want to point out that the running time and optimization algorithm are solver-dependent, and efficiently solving SDP is beyond the scope of this work. It is an interesting future direction to exploit the block structure of the dual programs, and develop algorithms that are compatible with those programs, because training smooth networks is a critical task, and it is promising to incorporate the SDP programs.