Introduction
The past decade has witnessed the unprecedented success of deep learning in many machine learning tasks (Krizhevsky et al . , 2017 ; Mikolov et al . , 2013 ) . Despite the growing popularity of deep learning, researchers have also found that neural networks are very vulnerable to adversarial attacks (Szegedy et al . , 2014 ; Goodfellow et al . , 2015 ; Papernot et al . , 2016 ) . As a result, it is important to train neural networks that are robust against those attacks Madry et al . ( 2018 ) . In recent years, the deep learning community starts to focus on certifiably robust neural networks Albarghouthi ( 2021 ); Hein and Andriushchenko ( 2017 ); Katz et al . ( 2017 ); Cohen et al . ( 2019 ); Raghunathan et al . ( 2018 ); Wang et al . ( 2022 ); Leino et al . ( 2021 ) .
One way to achieve certified robustness is to estimate the smoothness of neural networks, where the smoothness is measured by the Lipschitz constant of the neural network. Recent works have found that to achieve both high accuracy and low Lipschitzness, the network has to significantly increase the model capacity (Bubeck and Sellke, 2021 ) . This implies that there is an intrinsic tension between the accuracy and smoothness of neural networks.

Commonly considered adversarial attacks are the \ell_{\infty} and \ell_{2} -perturbations in the input space. Leino et al . ( 2021 ); Cohen et al . ( 2019 ) have successfully trained networks with low \ell_{2} -Lipschitz constant, and Huang et al . ( 2021 ) trained networks with low local Lipschitzness for \ell_{\infty} -perturbations.
There are few well-established techniques to train neural networks with low global Lipschitzness for \ell_{\infty} -perturbations. The techniques for the \ell_{2} -perturbation do not easily transfer to the \ell_{\infty} -case Leino et al . ( 2021 ) . One critical step is to measure the Lipschitz constant more precisely and efficiently. Jordan and Dimakis ( 2020 ) showed that for ReLU networks, it is \mathsf{NP} -hard to approximate the Lipschitz constant for \ell_{\infty} -perturbations within a constant factor , and proposed an exponential-time algorithm to compute the exact Lipschitz constant. However, researchers are interested in more scalable approaches to certify and train networks. In this work, we consider the Formal Global Lipschitz constant (FGL) (See Equation 3 ), which is roughly the maximum of the gradient operator norm, assuming all activation patterns on hidden layers are independent and possible. FGL is an upper bound of the exact Lipschitz constant and has been used in Raghunathan et al . ( 2018 ); Fazlyab et al . ( 2019 ); Latorre et al . ( 2020 ) .

We address the Lipschitz constant estimation from the quantitative geometric perspective. Quantitative geometry aims to understand geometric structures via quantitative parameters, and has connections with many mathematical disciplines, such as functional analysis and combinatorics (Naor, 2013 ) . In computer science, quantitative geometry plays a central role in understanding the approximability of discrete optimization problems. We approach those hard discrete optimization problems by considering the efficiently solvable continuous counterparts, and analyze the precision loss due to relaxation, which is often the SDP relaxation (Goemans and Williamson, 1995 ; Nesterov, 1998 ; Alon and Naor, 2004 ) . The natural SDP relaxations for the intractable problems usually induce the optimal known polynomial time algorithms (Bhattiprolu et al . , 2022 ) . By adopting the quantitative geometric approach, we can immediately understand the computational hardness and approximability of FGL estimations. Our algorithms on two-layer networks are the natural SDP relaxations from the quantitative geometric perspective.

Latorre et al . ( 2020 ) employed polynomial optimization methods on the FGL estimation for \ell_{\infty} -perturbations.
Polynomial optimization is a very general framework, and many problems can be cast in this framework (Motzkin and Straus, 1965 ; Goemans and Williamson, 1995 ) . Therefore, we argue that this is not a precise characterization of the FGL-estimation problem. On the other hand, there are also several SDP-based techniques for FGL estimations. Raghunathan et al . ( 2018 ) proposed an SDP algorithm to estimate the FGL for \ell_{\infty} -perturbations of two layer networks, and Fazlyab et al . ( 2019 ) devised an SDP algorithm to estimate the \ell_{2} -FGL. We will demonstrate the intricate relationships between our algorithms and these existing SDPs on two-layer networks.

Several empirical studies have found that techniques on one \ell_{p} -perturbations often do not transfer to another ones, even though the authors claim that in theory these techniques should transfer (Fazlyab et al . , 2019 ; Leino et al . , 2021 ) . This in-theory claim usually comes from a qualitative perspective. In finite-dimensional space, one can always bound one \ell_{p} -norm from another one, so techniques for one \ell_{p} -perturbations can also provide another bound for a different \ell_{p} -perturbations. However, this bound is loose and in practice not useful (Latorre et al . , 2020 ) . Instead, we believe that when transferring techniques from one norm to another one, we should consider the quantitative geometric principle: we should separate the geometry-dependent component from the geometry-independent one in those techniques, and modify the geometry-dependent component accordingly for a different normed space. As we will demonstrate, our whole work is guided by this principle. We hope that our unified quantitative geometric framework can bring insights to the empirical hard-to-transfer observations, and new tools to address these issues.