RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Contributions
To summarize, we have made the following contributions:

We provide a unified theoretical framework to address FGL estimations, which immediately yields the computational hardness and SDP-based approximation algorithms on two-layer networks ( Section 3 ).

We demonstrate the relations between our algorithms and other SDP-based tools, which in return inspires us to design the algorithms for multi-layer networks. This provides more insightful and compositional interpretations of existing works, and makes them easier-to-generalize ( Sections 4 and 5 ).

We implement the algorithms and name the tool GeoLIP. We empirically validate our theoretical claims, and compare GeoLIP with existing methods to estimate FGL for \ell_{\infty} -perturbations. The result shows that GeoLIP provides a tighter bound (20%-60% improvements) than existing tools on small networks, and much better results than the naive matrix-norm-product method for deep networks, which existing tools cannot handle ( Section 6 ).
Notation
Let [n]=\{1,\ldots,n\} . For two functions f and g , f\circ g(x)=f(g(x)) . A 0-1 cube is \{0,1\}^{n} , and a norm-1 cube is \{-1,1\}^{n} for some integer n>0 . \mathbb{R}_{+}=[0,\infty) .
For any vector v\in\mathbb{R}^{n} , \text{diag}(v) is an n\times n diagonal matrix, with diagonal values v . Let e_{n}=(1,\ldots,1)\in\mathbb{R}^{n} be an n -dimensional vector of all 1 ’s; and I_{n}=\text{diag}(e_{n}) , the identity matrix. Let ||v||_{p} denote the \ell_{p} norm of v . We use q to denote the Hölder conjugate of p as a convention, i.e., \frac{1}{p}+\frac{1}{q}=1 . If v is an operator in the \ell_{p} -space, the operator norm of v is then ||v||_{q} . Throughout the paper, we consider the \ell_{p} -norm of the input’s perturbation, and therefore, the \ell_{q} -norm of the gradient, which acts as an operator on the perturbation.
A square matrix X\succeq 0 means that X is positive semidefinite (PSD). Let \text{tr}(X) be the trace of a square matrix X . If a,b\in\mathbb{R}^{n} , let \langle a,b\rangle be the inner product of a and b .
Lipschitz function
Given two metric spaces (X,d_{X}) and (Y,d_{Y}) , a function f:X\rightarrow Y is Lipschitz continuous if there exists K>0 such that for all x_{1},x_{2}\in X ,

d_{Y}(f(x_{2}),f(x_{1}))\leq Kd_{X}(x_{2},x_{1})

The smallest such K satisfying Equation 1 , denoted by K_{f} , is called the Lipschitz constant of f .
For neural networks, X is in general \mathbb{R}^{m} equipped with the \ell_{p} -norm. We will only consider the case when Y=\mathbb{R} . In actual applications such as a classification task, a neural network has multiple outputs. The prediction is the class with the maximum score. One can then use the margin between each pair of class predictions and its Lipschitz constant to certify the robustness of a given prediction (Raghunathan et al . , 2018 ; Leino et al . , 2021 ) .
From Rademacher’s theorem, if f is Lipschitz continuous, then f is is almost everywhere differentiable, and K_{f}=\sup_{x}||\nabla f(x)||_{q} .
Neural network as function
A neural network f:\mathbb{R}^{m}\rightarrow\mathbb{R} is characterized by:

f_{1}(x)=W^{1}x+b_{1};\;f_{i}(x)=W^{i}\sigma(f_{i-1}(x))+b_{i},i=2,\ldots,d.

where W^{i}\in\mathbb{R}^{n_{i+1}\times n_{i}} is the weight matrix between the layers, n_{1}=m , d is the depth of the neural network, \sigma denotes an activation function, b_{i}\in\mathbb{R}^{n_{i+1}} is the bias term, and f=f_{d}\circ\cdots\circ f_{1} . Because we only consider the \mathbb{R} as the codomain of f , W^{d}=u\in\mathbb{R}^{1\times n_{d}} is a vector. From chain rule, the gradient of this function is

abla f(x)=(W^{1})^{T}[diag(\sigma^{\prime}(f_{1}(x)))(W^{2})^{T}\cdotsdiag(\sigma^{\prime}(f_{d-1}(x)))(W^{d})^{T}]

Common activation functions, including ReLU (Nair and Hinton, 2010 ) , sigmoid functions, and ELU (Clevert et al . , 2016 ) are almost everywhere differentiable. As a result, we are interested in the supremum operator norm of Equation 2 .

However, checking all possible inputs x is infeasible, and common activation functions have bounded derivative values, say [a,b] . We are then interested in the following value instead:

\max_{v^{i}\in[a,b]^{n_{i}}}||(W^{1})^{T}\cdotdiag(v^{2})\cdot\cdots\cdotdiag(v^{d})(W^{d})^{T}||_{q}, (3)

where n_{i} is the dimension of each \text{diag}(v^{i}) . We call this value the formal global Lipschitz constant (FGL) because we treat all activation functions independent but in reality not all activation patterns are feasible. Therefore, this is an upper bound of the true global Lipschitz constant of the neural network. However, it is the value studied in most global Lipschitzness literature (Scaman and Virmaux, 2018 ; Fazlyab et al . , 2019 ; Latorre et al . , 2020 ) , and also turns out useful in certifying the robustness of neural networks (Raghunathan et al . , 2018 ; Leino et al . , 2021 ; Pauli et al . , 2022 ) . We use \ell_{p} -FGL to denote the FGL for \ell_{p} -perturbations.

In this paper, we focus on the \ell_{p} -perturbation on the input, where p=\infty or p=2 . Because q is the Hölder conjugate of p , we are interested in the value of Equation 3 , when q=1 (for p=\infty ), and q=2 (for p=2 ).
Notice that in the ReLU -network case, [a,b] is [0,1] . We will use ReLU -networks as the illustration for the rest of the paper because of the popularity of ReLU in practice and the easy presentation of the 0-1-cube. However, the algorithms presented in this work can be adapted with minor adjustments to other common activation functions.

FGL considers all possible activation patterns on the hidden layers, while some of the activation patterns might be unachievable in reality. Therefore, FGL is an upper bound of the true Lipschitz constant. Notice that the activation pattern induced from an input is also decided by the bias term. Therefore, to find the true Lipschitz constant, one has to incorporate the information from the bias term.
Two-layer neural networks
In this section, we consider the two-layer neural network case. We reduce the FGL estimation to the matrix mixed-norm problem. This immediately yields the computational complexity and approximation algorithms for FGL estimations. In Section A.1 , we show that we can consider \{0,1\} instead of [0,1] in Equation 3 for two-layer networks.
Problem description
For a two-layer network where W^{1}=W\in\mathbb{R}^{n\times m} and W^{d}=u\in\mathbb{R}^{1\times n} ,
its FGL (as in Equation 3 ) is \max_{y\in\{0,1\}^{n}}||W^{T}\text{diag}(y)u^{T}||_{q} , where we use y to denote v^{1} in this case.
If we expand the matrix multiplication, it is easy to check that this equals to \max_{y\in\{0,1\}^{n}}||W^{T}\text{diag}(u)y||_{q} . Let A=W^{T}\text{diag}(u) , then the \ell_{p} -FGL is

\max_{y\in\{0,1\}^{n}}||Ay||_{q}
\ell_{\infty} -FGL estimation
We consider a natural SDP relaxation to Equation 4 when q=1 , and analyze the result using the celebrated Grothendieck Inequality , which is a fundamental tool in functional analysis.
Mixed-norm problem
The \infty\rightarrow 1 mixed-norm of a matrix is defined as

||A||_{\infty\rightarrow 1}=\max_{||x||_{\infty}=1}||Ax||_{1}.

The mixed-norm problem appears similar to Equation 4 when q=1 , except for that instead of a norm- 1 -cube, the cube in Equation 4 is a 0-1-cube. Alon and Naor ( 2004 ) showed that it is \mathsf{NP} -hard, specifically \mathsf{MAXSNP} -hard, to compute the \infty\rightarrow 1 mixed-norm of a matrix A , via a reduction to the graph Max-Cut problem.
Moreover, Alon and Naor ( 2004 ) constructed a natural SDP relaxation for the mixed-norm problem:

\begin{split}\maxtr(BX)\ s.t.\;X\succeq 0,X_{ii}=1,&i\in[n+m],\end{split} (5)

where A is a submatrix of B . We provide the detailed derivation of this relaxation in Section A.3 .
In fact, this relaxation admits a constant approximation factor. Grothendieck ( 1956 ) developed the local theory of Banach spaces, and showed that there exists an absolute value K_{G} such that

For any m,n\geq 1 , A\in\mathbb{R}^{n\times m} , and any Hilbert space H , the following holds:

\max_{u_{i},v_{j}\in B(H)}\sum_{i,j}A_{ij}\langle u_{i},v_{j}\rangle_{H}\leq K_{G}||A||_{\infty\rightarrow 1},

where B(H) denotes the unit ball of the Hilbert space.

The precise value of K_{G} is still an outstanding open problem, and it is known that K_{G}<1.783 (Krivine, 1979 ; Braverman et al . , 2011 ) .
The approximation factor of the SDP relaxation in Equation 5 is K_{G} .
Similar to the mixed-norm problem, we show that the \ell_{\infty} -FGL estimation is \mathsf{MAXSNP} -hard and provide an SDP relaxation, which also admits the K_{G} -approximation ratio. We provide a detailed explanation on why K_{G} is the approximation ratio and how we can view the SDP relaxation as a geometric transformation in Section A.4 .

\ell_{\infty} -FGL estimation is \mathsf{MAXSNP} -hard.
From 0-1 cube to norm-1 cube
If we can transform the 0-1 cube in Equation 4 to a norm-1 cube, and formulate an equivalent optimization problem, then one can apply the SDP program in Equation 5 to compute an upper bound of the FGL. Indeed, we provide a cube rescaling technique, and it allows us to construct the SDP for the \ell_{\infty} -FGL estimation. We provide the full detail of this technique in Section A.5 , and the result SDP for the \ell_{\infty} -FGL estimation is

\begin{split}\max\;\frac{1}{2}&tr(BX)\ s.t.\;X\succeq 0,X_{ii}=1&,i\in[n+m+1],\end{split} (6)

where B is a (n+1+m)\times(n+1+m) matrix, and B=\begin{pmatrix}0&0&0\\
A&Ae_{n}&0\end{pmatrix} . As a result, we have:

There exists a polynomial-time approximation algorithm to estimate the \ell_{\infty} -FGL of two-layer neural networks, moreover, the approximation ratio is K_{G} .
\ell_{2} -FGL estimation
Scaman and Virmaux ( 2018 ) showed that the \ell_{2} -FGL estimation is \mathsf{NP} -hard. If q=2 in Equation 4 , the objective becomes similar to the \infty\rightarrow 2 mixed-norm problem. This is a quadratic optimization problem with a PSD weight matrix over a cube, and can be viewed as a generalization of the graph Max-Cut problem. In the quadratic optimization formulation of Max-Cut, the weight matrix is the Laplacian of the graph, a special PSD matrix (Goemans and Williamson, 1995 ) . Nesterov ( 1998 ) generalized Goemans-Williamson’s technique and analyzed the case when the weight matrix is PSD, showing that the natural SDP relaxation in this case has a \frac{\pi}{2} -approximation ratio. This provides a \sqrt{\frac{\pi}{2}} -approximation algorithm for the \ell_{2} -FGL estimation. The approximation ratio comes from a similar inequality to the one in Theorem 3.1 , known as the Little Grothendieck Inequality .
The SDP for \ell_{2} -FGL estimation is:

\begin{split}\max\frac{1}{2}&\sqrt{tr(\begin{pmatrix}A^{T}A&A^{T}Ae_{n}\ e_{n}^{T}A^{T}A&e_{n}^{T}A^{T}Ae_{n}\end{pmatrix}X)}\ s.t.\;&X\succeq 0,X_{ii}=1,i\in[n+1].\end{split} (7)

The full derivation is provided in Section A.8 , and we have the following theorem:

There exists a polynomial-time approximation algorithm to estimate the \ell_{2} -FGL of two-layer neural networks with an approximation factor \sqrt{\frac{\pi}{2}} .

As we have discussed, for two-layer networks, the \ell_{p} -FGL estimation is essentially the \infty\rightarrow q mixed-norm problem. Indeed the mixed-norm problem is an outstanding topic in theoretical computer science. As discussed in Bhattiprolu et al . ( 2018 ) , the \infty\rightarrow q mixed norm problem has constant approximation algorithms if q\leq 2 , and is hard to approximate within almost polynomial factors when q>2 . Because when q>2 , its Hölder conjugate p<2 . This implies that for two-layer networks, the FGL estimation can be much harder for \ell_{p} -perturbations when p<2 .

Briët et al . ( 2017 ) showed that it is \mathsf{NP} -hard to approximate the \infty\rightarrow 2 mixed-norm problem better than \sqrt{\frac{\pi}{2}} . Raghavendra and Steurer ( 2009 ) proved that assuming the unique games conjecture (Khot, 2002 ) , it is \mathsf{NP} -hard to approximate the \infty\rightarrow 1 mixed-norm problem better than K_{G} . These optimal approximation ratios match our SDP relaxations for FGL estimations accordingly.
Relations to existing SDP works
Before introducing our approach for multi-layer networks, we first examine some existing SDP works on FGL estimations, and discuss their relationships with our natural SDP relaxations in Section 3 .
\ell_{\infty} -FGL estimation
Raghunathan et al . ( 2018 ) formulated an SDP that only works for two-layer networks. Theirs is essentially the same as ours in Equation 6 (See the detailed comparison in Section A.10 ).
However, we provide a rigorous derivation and simpler formulation, and also a sound theoretical analysis of the bound, which illustrate more insights to this problem. Raghunathan et al . ( 2018 ) treated the SDP relaxation as a heuristic to a hard quadratic
programming problem. We prove that this relaxation is not only a heuristic, but in fact induces an approximation algorithm with a tight bound.
\ell_{2} -FGL estimation
Fazlyab et al . ( 2019 ) proposed LipSDP, another SDP-based algorithm for the \ell_{2} -FGL estimation problem. Fazlyab et al . ( 2019 ) provided several variants of LipSDP to balance the precision and scalability. Pauli et al . ( 2022 ) demonstrated that the most precise version of LipSDP, LipSDP-Network , fails to produce an upper bound for \ell_{2} -FGL. In this paper, all the references of LipSDP are to LipSDP-Neuron , the less precise version.
Surprisingly, even though the approach in LipSDP appears quite different from Equation 7 , we show that LipSDP is dual of Equation 7 to estimate the \ell_{2} -FGL on two-layer networks.
LipSDP for two-layer networks is:

\min_{\zeta,\lambda}\Big{\{}\sqrt{\zeta}:\begin{pmatrix}-2abW^{T}WT-\zeta I_{m}&(a+b)W^{T}T\ (a+b)TW&-2T+u^{T}u\end{pmatrix}\preceq 0,\lambda_{i}\geq 0\Big{\}},

where T=\text{diag}(\lambda) for \lambda\in\mathbb{R}_{+}^{n} ; a and b are the lower and upper bounds of the activation’s derivative.

We will construct a new quadratic program, which we show is equivalent to Equation 4 when q=2 , and LipSDP is its dual SDP relaxation.

Let the input of the i -th activation node on \text{diag}(y) be y_{i} , and w_{i} be the row vector of W . Hence, y_{i}=w_{i}x . Let \Delta x\in\mathbb{R}^{m} be a perturbation on x , so \Delta y_{i}=w_{i}\Delta x . Let \Delta\sigma(y)\in\mathbb{R}^{n} denote the induced perturbation on \text{diag}(y) .
The constraint from the activation function is \frac{\Delta\sigma(y)_{i}}{\Delta y_{i}}\in[a,b] , in other words, \frac{\Delta\sigma(y)_{i}}{w_{i}\Delta x}\in[a,b] .
One can write the range constraint as

(\Delta\sigma(y)_{i}-a\cdot w_{i}\Delta x)(\Delta\sigma(y)_{i}-b\cdot w_{i}\Delta x)\leq 0.

This can be written in the quadratic form:

\begin{pmatrix}w_{i}\Delta x\ \Delta\sigma(y)_{i}\end{pmatrix}^{T}\begin{pmatrix}-2ab&a+b\ a+b&-2\end{pmatrix}\begin{pmatrix}w_{i}\Delta x\ \Delta\sigma(y)_{i}\end{pmatrix}\geq 0,\;\;\;\;\forall i\in[n]

Since for the two layer network, f(x)=u\sigma(y) , then \Delta f(x)=u\Delta\sigma(y) . The objective for \ell_{2} -FGL estimation is \max_{\Delta x,\Delta\sigma(y)}\sqrt{\frac{(u\Delta\sigma(y))^{2}}{(\Delta x)^{2}}} . The equivalence between this program and Equation 4 when q=2 is presented in Section A.11 .

Another interpretation for the quadratic program is that we want to quantify how the output changes given a data-independent input change, i.e., \Delta x . In other words, we want to analyze the effect of \Delta x propagating from the input to the output, with symbolic values rather than actual inputs. The idea is similar to symbolic execution from program analysis (Baldoni et al . , 2018 ) .
Duality to LipSDP
Now we will show that LipSDP is the dual SDP to the program formulated above. The dual SDP derivation is of similar form in Ben-Tal and Nemirovski ( 2001 , Ch.4.3.1) . Let us introduce a variable \zeta such that \zeta-\frac{(u\Delta\sigma(y))^{2}}{(\Delta x)^{2}}\geq 0 .
In other words,

\zeta(\Delta x)^{2}-(u\Delta\sigma(y))^{2}\geq 0

For each constraint in Equation 8 , let us introduce a dual variable \lambda_{i}\geq 0 . Multiply each constraint with \lambda_{i} , then \begin{pmatrix}\Delta x\\
\Delta\sigma(y)_{i}\end{pmatrix}^{T}\begin{pmatrix}-2ab\lambda_{i}w_{i}^{T}w_{i}&(a+b)\lambda_{i}w_{i}^{T}\\
(a+b)\lambda_{i}w_{i}&-2\lambda_{i}\end{pmatrix}\begin{pmatrix}\Delta x\\
\Delta\sigma(y)_{i}\end{pmatrix}\geq 0,\;\;\;\;\forall i\in[n].

Sum all of them, then we have \begin{pmatrix}\Delta x\\
\Delta\sigma(y)\end{pmatrix}^{T}\begin{pmatrix}-2abW^{T}WT&(a+b)W^{T}T\\
(a+b)TW&-2T\end{pmatrix}\begin{pmatrix}\Delta x\\
\Delta\sigma(y)\end{pmatrix}\geq 0 ,
where T=\text{diag}(\lambda) is the n\times n diagonal matrix of dual variables \lambda_{1},\ldots,\lambda_{n} .

Equation 9 can be rewritten as: \begin{pmatrix}\Delta x\\
\Delta\sigma(y)\end{pmatrix}^{T}\begin{pmatrix}\zeta I_{m}&0\\
0&-u^{T}u\end{pmatrix}\begin{pmatrix}\Delta x\\
\Delta\sigma(y)\end{pmatrix}\geq 0. As a result, the dual program for the new optimization program is

\min_{\zeta,\lambda}\Big{\{}\sqrt{\zeta}:\begin{pmatrix}-2abW^{T}WT-\zeta I_{m}&(a+b)W^{T}T\ (a+b)TW&-2T+u^{T}u\end{pmatrix}\preceq 0,\lambda_{i}\geq 0\Big{\}}.

In Remark 3.5 , we mention that \sqrt{\frac{\pi}{2}} is the optimal approximation ratio for the \infty\rightarrow 2 mixed-norm problem, which matches the approximation ratio in Theorem 3.4 . Hence, improving the natural SDP relaxation in Equation 7 can be very hard. The duality provides another evidence of LipSDP-Neuron’s correctness, and hints that LipSDP-Network, the improved variant, may be wrong.
\ell_{\infty} -FGL estimation for multi-layer networks
For a multi-layer neural network, the formal gradient becomes a high-degree polynomial, and its \ell_{q} -norm estimation becomes a high-degree polynomial optimization problem over a cube, which is in general a hard problem (Lasserre, 2015 ) . We provide a discussion of the polynomial optimization approach of FGL estimation in Appendix B . Here we provide an SDP dual program of the \ell_{\infty} -FGL estimation inspired by the dual SDP approach in Section 4 . The difference is that now we consider \ell_{\infty} -perturbations to the input space instead of \ell_{2} . Hence, the objective becomes

\max_{\Delta x,\Delta\sigma(y)}\frac{|u\Delta\sigma(y)|}{||\Delta x||_{\infty}}.

If we add an extra constraint ||\Delta x||_{\infty}=1 , the above objective becomes

\max_{\Delta x,\Delta\sigma(y)}\frac{1}{2}(u\Delta\sigma(y)+u\Delta\sigma(y))

The constraints are

\begin{pmatrix}w_{i}\Delta x\ \Delta\sigma(y)_{i}\end{pmatrix}^{T}\begin{pmatrix}2ab&-(a+b)\ -(a+b)&2\end{pmatrix}\begin{pmatrix}w_{i}\Delta x\ \Delta\sigma(y)_{i}\end{pmatrix}\leq 0,\;\Delta(x)_{j}^{2}\leq 1,\;\forall i\in[n],j\in[m].

We can write \Delta(x)_{j}^{2}\leq 1 as \begin{pmatrix}1\\
\Delta x_{j}\end{pmatrix}^{T}\begin{pmatrix}1&0\\
0&-1\end{pmatrix}\begin{pmatrix}1\\
\Delta x_{j}\end{pmatrix}\geq 0 .

Now let us introduce n+m non-negative dual variables (\tau,\lambda) , where \tau\in\mathbb{R}^{m}_{+} and \lambda\in\mathbb{R}^{n}_{+} . If we multiply each dual variable with the constraint and add all the constraints together, we will have

\begin{pmatrix}1\ \Delta x\ \Delta\sigma(y)\end{pmatrix}^{T}\begin{pmatrix}\sum_{j=1}^{m}\tau_{j}&0&0\ 0&-2abW^{T}WT_{2}-T_{1}&(a+b)W^{T}T_{2}\ 0&(a+b)T_{2}W&-2T_{2}\end{pmatrix}\begin{pmatrix}1\ \Delta x\ \Delta\sigma(y)\end{pmatrix}\geq 0,

where T_{1}=\text{diag}(\tau) and T_{2}=\text{diag}(\lambda) .
As a result, we can incorporate the objective Equation 10 and obtain the dual SDP for the \ell_{\infty} -FGL estimation:

\min_{\zeta,\lambda,\tau}\Big{\{}\frac{\zeta}{2}:\begin{pmatrix}\sum_{j=1}^{m}\tau_{j}-\zeta&0&u\ 0&-2abW^{T}WT_{2}-T_{1}&(a+b)W^{T}T_{2}\ u^{T}&(a+b)T_{2}W&-2T_{2}\end{pmatrix}\preceq 0,\lambda_{i},\tau_{j}\geq 0\Big{\}}

The SDP programs in Section 3 are strictly feasible because the identity matrix is a positive definite solution. Hence, Slater’s condition is satisfied and strong duality holds.
Multi-layer extension
We can simply extend the dual program to multiple-layer networks. We first vectorize all the units in the input layer and hidden layers, and then constrain them using layer-wise inequalities to formulate an optimization problem.
Let us consider a general d -layer multi-layer network, where W^{i}\in\mathbb{R}^{n_{i+1}\times n_{i}} for i\in[d-1] , and W^{d}=u\in\mathbb{R}^{1\times n_{d}} .
Let \Delta x denote the perturbation on the input layer, \Delta z^{i} be the perturbation on the i -th hidden layer, and w^{i}_{j} be the j -th row vector of W^{i} . The only difference between two layer networks and multi-layer networks is that we have the additional constraints:

\begin{pmatrix}\Delta z^{i}\ \Delta z^{i+1}_{j}\end{pmatrix}^{T}\begin{pmatrix}-2ab(w^{i+1}_{j})^{T}w^{i+1}_{j}&(a+b)(w^{i+1}_{j})^{T}\ (a+b)w^{i+1}_{j}&-2\end{pmatrix}\begin{pmatrix}\Delta z^{i}\ \Delta z^{i+1}_{j}\end{pmatrix}\geq 0.

Let \Lambda_{i}\in\mathbb{R}^{n_{i}}_{+} and T_{i}=\text{diag}(\Lambda_{i}) for i\in[d] . Following the similar SDP dual approach, we can add all the constraints together and formulate the following SDP program:

\min_{\zeta,\Lambda_{i}}\Big{\{}\frac{\zeta}{2}:(L+N)\preceq 0,i\in[d]\Big{\}}, (12)

where

L=\begin{pmatrix}0&0&0&\dots&0&0\ 0&-2ab(W^{1})^{T}W^{1}T_{2}&(a+b)(W^{1})^{T}T_{2}&\dots&0&0\ 0&(a+b)T_{2}W^{1}&-2T_{2}-2ab(W^{2})^{T}W^{2}T_{3}&\dots&0&0\ \vdots&\vdots&\vdots&\ddots&\vdots&\vdots\ 0&0&0&\dots&(a+b)T_{d}W^{d-1}&-2T_{d}\ \end{pmatrix}, (13)

N=\begin{pmatrix}\sum_{k=1}^{n_{1}}\Lambda_{1k}-\zeta&0&\dots&u\ 0&-T_{1}&\dots&0\ \vdots&\vdots&\ddots&\vdots\ u^{T}&0&\cdots&0\end{pmatrix}.

If we expand the matrix inequality derived from the compact neural-network representation in Fazlyab et al . ( 2019 , Theorem 2) , we will have exactly the same matrix for network constraints as L ( Equation 13 ) in the dual program formulation. In other words, we provide a compositional optimization interpretation to the compact neural-network representation in LipSDP. With this interpretation, one can extend the SDP to beyond feed-forward structures, such as skip connections (He et al . , 2016 ) . Notice that if we apply the similar reasoning to the multi-layer network \ell_{2} -FGL estimation, we will obtain LipSDP-Neuron.
Evaluation and discussion
The primary goal of our work is to provide a theoretical framework, and also algorithms for \ell_{\infty} -FGL estimations on practically used networks. The \ell_{2} -FGL can be computed using LipSDP. We have implemented the algorithms using MATLAB (MATLAB, 2021 ) , the CVX toolbox (CVX Research, 2020 ) and MOSEK solver (ApS, 2019 ) , and name the tool GeoLIP. To validate our theory and the applicability of our algorithms, we want to empirically answer the following research questions:

As we shall see, GeoLIP is indeed better than existing methods in terms of precision and scalability; and the dual SDP programs produce the same values as their natural-SDP-relaxation counterparts.
Experimental design
To answer RQ1 , we will run GeoLIP and existing tools that measure the \ell_{\infty} -FGL on various feed-forward neural networks trained with the MNIST dataset (LeCun and Cortes, 2010 ) . We will record the computed \ell_{\infty} -FGL to compare the precision, and the computation time to compare the scalability.

To answer RQ2 , we will run the natural SDP relaxations for \ell_{p} -FGL estimations proposed in Section 3 , LipSDP for \ell_{2} -FGL estimation, and the dual program Equation 11 for \ell_{\infty} -FGL on two-layer neural networks, and compare their computed FGLs.