SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuriy Biktairov Jyotirmoy Deshmukh
{biktairo, jdeshmuk}@usc.edu
Finding tight linear bounds for activation functions in neural networks is an essential part of several state of the art neural network robustness certification tools. An activation function is an arbitrary, nonlinear, scalar function  $f: \mathbb{R}^d \to \mathbb{R}$ . In the existing work on robustness certification, such bounds have been computed using human ingenuity for a handful of the most popular activation functions. While a number of heuristics have been proposed for bounding arbitrary functions, no analysis of the tightness optimality for general scalar functions has been offered yet, to the best of our knowledge. We fill this gap by formulating a concise optimality criterion for tightness of the approximation which allows us to build optimal bounds for any function convex in the region of interest  $R$ . For a more general class of functions Lipschitz-continuous in  $R$  we propose a sampling-based approach (SOL) which, given an instance of the bounding problem, efficiently computes the tightest linear bounds within a given  $\varepsilon > 0$  threshold. We leverage an adaptive sampling technique to iteratively build a set of sample points suitable for representing the target activation function. While the theoretical worst case time complexity of our approach is  $O(\varepsilon^{-2d})$ , it typically only takes  $O(\log^{\beta} \frac{1}{\varepsilon})$  time for some  $\beta \geq 1$  and is thus sufficiently fast in practice. We provide empirical evidence of SOL's practicality by incorporating it into a robustness certifier and observing that it produces similar or higher certification rates while taking as low as quarter of the time compared to the other methods.
Neural networks (NNs) have become the pervasive machine learning model for various tasks including classification, function approximation, and generative modeling. In a typical classification task using supervised learning, the NN is given as input a high-dimensional data object (such as an image or a large piece of text) and its corresponding class label (from a finite set of labels). The NN classifier is deemed robust if for reasonable perturbations to a given input, the predicted label for the input does not change. It has been previously shown that NNs are susceptible to adversarial perturbations in inputs. These approaches take advantage of the non-robustness of a trained NN; for example, in image classification, by injecting a small amount of noise imperceptible to a human, the predicted label for an input image can be changed. In safety-critical applications where object detection and classification may be used as input for decision-making (e.g., in an autonomous mobile robot), such misclassifications can have safety implications. Similarly, when used in a function approximation task (e.g., for regression), it is important that small changes to the NN input do not lead to large changes in the predicted output. Checking the robustness of neural networks has thus emerged as an important problem in recent years [28, 3, 9, 1, 13, 23].
A number of different approaches have been developed for certifying the robustness of the neural networks with popular architectures. The majority of these approaches fall into one of three overar-
ching categories: the algorithms using approximate calculations in order to speed the certification time up [25, 5, 33, 23, 13], the algorithms using exact calculations via some sort of constraint solver [10, 11, 29] and the algorithms taking advantage of both techniques [26, 24, 30, 32]. The most practical approaches which can handle the largest neural networks are usually built on top of the approximate techniques and rely heavily on being able to find tight linear bounds of non-linear scalar functions.
A lot of research has been dedicated to increasing the quality of linear bounds of the most popular activation functions such as ReLU[19], sigmoid, hyperbolic tangent and others[32, 25, 34, 36, 23, 13]. But not much has been done in the direction of efficiently bounding arbitrary functions. This led to the current situation where a number of well-performing novel activation functions like GeLU[8, 21] and Swish[22] and the neural networks containing them are effectively uncertifiable due to their linear bounding not being supported by the robustness certifiers.
To this end we introduce a novel approach for efficient optimal linear bounding of general scalar functions. The considered optimality is with respect to the conventional measure of tightness – the volume of the discrepancy region between the function and the linear bound. At the core of our approach is a simple idea of leveraging the Lipschitz continuity of the activation function in order to build a near-optimal bound using appropriately constructed sample of points from the region of interest. The major contributions of the presented paper are the following
- we analyze the problem of optimal linear bounding of scalar functions and derive a simple optimality criterion which allows us to optimally bound arbitrary functions convex or concave in the region of interest;
- we propose a novel optimal linear bounding approach called SOL which unlike any other known method efficiently produces linear bounds which are arbitrarily close to optimum for the general class of functions Lipschitz-continuous in the region of interest;
- we show that using SOL for the robustness certification is viable by benchmarking its performance on a synthetic dataset of bounding problems and by comparing it to the known alternatives on real neural network certification tasks.
To the best of our knowledge, there are only two approaches presented in the literature capable of producing linear bounds for arbitrary scalar functions. The first one is implemented inside the AutoLiRPA[35] framework for the robustness certification. The second is the LinSyn[20] approach.
AutoLiRPA supports bounding several general binary operations: addition, subtraction, multiplication and division. It is also capable of bounding compositions of these operations, which thus allows it to bound an arbitrary function decomposable into a sequence of some elementary operations. While this class of functions is indeed quite general and contains all the novel activation functions, the decompositional nature of the approach inevitably leads to producing bounds which are significantly suboptimal.
The authors of LinSyn, on the other hand, use an idea reminiscent of what we propose in this paper: they take a uniform sample of points from the region of interest, find the tightest bound for this finite set of points and then repeatedly adjust the obtained bound until its soundness is successfully certified by an SMT solver[4]. Unlike SOL, however, their approach does not provide any optimality guarantees for the generated bounds and heavily relies on the use of the SMT solver, which might limit the method's running time performance.
We provide an empirical robustness certification performance comparison between these two approaches and SOL in section 5.3 of this paper.
3 Optimal linear bounding
In this section we give a formal definition of the optimal linear bounding problem and derive the optimality criterion. We then use this criterion to give a recipe for the optimal linear bounding of functions which are convex or concave in the region of interest.
Figure 1: Linear bounding example.
Figure 2: Optimal bounding criterion.
Without the loss of generality from now on we are only going to discuss the optimal upper bounding. The lower bounding problem can easily be reduced to the former by negating the function and then negating the values of the bound.
Definition (Optimal linear bounding problem). Given a function  $f: R \to \mathbb{R}$  defined in a convex bounded region  $R \subset \mathbb{R}^d$ , we aim to find such linear function  $g(\boldsymbol{x}) = \boldsymbol{a}^\top \boldsymbol{x} + b$ , that
1.  $g(\pmb{x})$  upper-bounds the original function:  $g(\pmb{x}) \geq f(\pmb{x})$  for every  $\pmb{x} \in R$ ;
2. the volume  $Y[g(\pmb{x})] = \int_{\pmb{x} \in R} |f(\pmb{x}) - g(\pmb{x})| d\pmb{x}$  of the discrepancy region between the function graphs is as small as possible:  $g_{opt}(\pmb{x}) = \arg \min_{g(\pmb{x}) = \pmb{a}^\top \pmb{x} + b} Y[g(\pmb{x})]$ .
The value of the discrepancy volume defined above is the conventionally accepted measure of tightness of the linear bound and is known to correlate well with the performance of robustness certification [36, 25, 13]. The notion is illustrated in the figure 1. We are going to call any linear function satisfying the two conditions an "optimal bound". Upon a detailed examination it can be noted that the discrepancy volume may be expressed in a more convenient form.
Proposition 1. For any function  $g(\pmb{x})$  upper-bounding the target function  $f(\pmb{x})$  the equality  $Y[g(\pmb{x})] = g(\pmb{x}_c) \cdot V + C$  holds, where  $V = \int_{\pmb{x} \in R} d\pmb{x}$  is the volume of  $R$ ,  $\pmb{x}_c = \frac{1}{V} \int_{\pmb{x} \in R} \pmb{x} d\pmb{x}$  is the region's center of mass and  $C$  is a constant independent of the function  $g(\pmb{x})$ .
Proof. Indeed, for any upper bound  $g(\pmb{x})$  the integrand in the definition of  $Y[g(\pmb{x})]$  can be simplified
$$
\int_ {\boldsymbol {x} \in R} | f (\boldsymbol {x}) - g (\boldsymbol {x}) | d \boldsymbol {x} = \int_ {\boldsymbol {x} \in R} (g (\boldsymbol {x}) - f (\boldsymbol {x})) d \boldsymbol {x} = \int_ {\boldsymbol {x} \in R} g (\boldsymbol {x}) d \boldsymbol {x} + C,
$$
where  $C = -\int_{\pmb{x} \in R} f(\pmb{x}) d\pmb{x}$  does not depend on  $g(\pmb{x})$ . The straightforward integration
$$
\int_ {\boldsymbol {x} \in R} g (\boldsymbol {x}) d \boldsymbol {x} = \int_ {\boldsymbol {x} \in R} \left(\boldsymbol {a} ^ {\top} \boldsymbol {x} + b\right) d \boldsymbol {x} = \boldsymbol {a} ^ {\top} \boldsymbol {x} _ {c} V + b V = g \left(\boldsymbol {x} _ {c}\right) \cdot V
$$
then yields the desired expression.
Since  $R$  is convex,  $\mathbf{x}_c$  always lies in  $R$ . Then,  $g(\mathbf{x})$  being an upper bound for the target function gives us a lower bound on the discrepancy volume  $Y[g(\mathbf{x})] = V \cdot g(\mathbf{x}_c) + C \geq V \cdot f(\mathbf{x}_c) + C$ . For a concave target function  $f(\mathbf{x})$  this inequality concludes the problem of optimal linear bounding. Indeed, for a concave target function there always exists such an upper bound  $g^*(\mathbf{x}_c)$  that  $g^*(\mathbf{x}_c) = f(\mathbf{x}_c)$ . This upper bound is then optimal due to the inequality.
Now, let  $\mathrm{conv}_R f(\pmb{x})$  be the upper convex hull of the function  $f(\pmb{x})$  in the region  $R$  - a function of  $\pmb{x}$  representing the upper boundary of the convex hull of  $f$ 's graph as is illustrated in figure 2 by the dashed line. The following then holds for arbitrary, possibly non-concave, functions  $f(\pmb{x})$ .
Proposition 2. A linear function  $g(\pmb{x})$  upper-bounds the target function  $f(\pmb{x})$  in the region  $R$  iff it upper-bounds the convex hull  $\operatorname{conv}_R f(\pmb{x})$  of the target function in the region  $R$ .
Proof. If  $g(\pmb{x})$  does not upper-bound the target function  $f(\pmb{x})$ , then it does not upper-bound its convex hull either since  $\mathrm{conv}_R f(\pmb{x}) \geq f(\pmb{x})$  for every  $\pmb{x} \in R$ . Conversely, if  $g(\pmb{x})$  does not upper-bound the convex hull there is at least one point  $\pmb{x}_0$  such that  $f(\pmb{x}_0) > g(\pmb{x}_0)$ . Otherwise the whole convex hull would lie below the linear function.
This proposition allows us to reduce the problem of bounding an arbitrary function  $f(\pmb{x})$  to bounding a concave function  $\mathrm{conv}_R f(\pmb{x})$ . This reduction combined with the reasoning suggested for the concave target functions above gives us the following concise optimality criterion for the linear upper-bounding problem.
Theorem 1. A linear upper bound  $g(\pmb{x})$  of a function  $f(\pmb{x})$  is optimal iff  $g(\pmb{x}_c) = \mathrm{conv}_R f(\pmb{x}_c)$ .
Figure 2 illustrates the criterion showing a portion of  $\operatorname{conv}_R f(\boldsymbol{x}_c)$  distinct from  $f(\boldsymbol{x})$  with a dashed line. Theorem 1 immediately provides us with two efficient optimal bounding algorithms for concave and convex functions.
For concave and differentiable  $f(x)$  the convex hull  $\operatorname{conv}_R f(x)$  coincides with the function itself. Finding an optimal upper bound is then simply a matter of finding a hyperplane tangent to  $f(x)$  at the center of mass  $x_c$ . This only takes  $O(d)$  operations if we have the access to the function's gradient.
For convex  $f(\pmb{x})$  and  $R$  being a polytope one can show that only the values  $f(\pmb{x}_i)$  of the function in the vertices  $\pmb{x}_i$  of  $R$  matter. The problem of upper-bounding then essentially becomes discrete - a linear program with  $d$  variables and  $|\{\pmb{x}_i\}|$  constraints. In the typical case of a cubic  $R$  this means having  $O(2^d)$  constraints which can still be viable for reasonably high-dimensional functions. The process of solving this kind of linear programs efficiently is discussed in details and analyzed experimentally in sections 4.1 and 5.1 of this paper.
The two algorithms combined form a complete optimal upper- and lower-bounding suite to be used with arbitrary convex or concave differentiable activation functions. Optimal bounding of both sides of the function can be done exactly in this case with no need for approximations. Among the suitable popular activation functions are: softplus[37], ELU[2] and SELU[12] with appropriate choices of parameters. The approach can easily be generalized to piecewise-differentiable convex or concave functions as well. This extends the applicability to such functions as ReLU[19], leaky ReLU[16], SELU with a wider set of suitable parameters and the max pooling function (although not an activation function, also frequently used in neural networks).
4 SOL
In order to solve the problem of optimal linear bounding for the functions which are neither convex nor concave in  $R$  we introduce the SOL approach: Sampling-based Optimal Linear bounding algorithm. In contrast to the approaches discussed above where an exactly optimal solution could be found in finite time, here we only aim at synthesizing an upper bound within a certain required tolerance target  $\varepsilon$  of the optimum in terms of the discrepancy volume. In return, we get an algorithm which efficiently upper-bounds a very general class of activation functions – functions Lipschitz-continuous in  $R$ . Some of the popular non-convex and non-concave Lipschitz-continuous activation functions are: Sigmoid, Tanh, GeLU [8], Swish [22], Log Log[6], Mish [18], ELU and SeLU with  $\alpha > 1$ .
The key idea of SOL is to perform multiple iterations alternating between constructing an appropriate sample of points  $S \subset R$  and solving the discrete version of the optimal bounding problem for the constructed sample until properly adjusted solution of the discrete problem meets the required accuracy target. During these iterations the generated sample progressively adapts to the target function increasing its density near the points where the optimal bound " touches" the function which allows for higher accuracy. Figure 3 illustrates the adaptive nature of these iterations by showing sample  $S$  in the first 3 iterations of a SOL run.
4.1 Discrete problem
We define the discrete linear upper-bounding problem mentioned above in the following way.
Definition (Discrete optimal linear bounding problem). Given a set of points  $S = \{(\pmb{x}_i, y_i)\}$ , where  $\pmb{x}_i \in \mathbb{R}^d$  and  $y_i \in \mathbb{R}$ , and a center point  $\pmb{x}_c \in \mathbb{R}^d$ , we aim to find such linear function  $g(\pmb{x}) = \pmb{a}^\top \pmb{x} + b$ , that
1.  $g(\pmb {x})$  upper-bounds the sample points:  $g(\pmb {x}_i)\geq y_i$  for every  $(\pmb {x}_i,y_i)\in S$
2. the value of the bound at  $\pmb{x}_c$  is as small as possible:  $g_{opt}(\pmb{x}) = \arg \min_{g(\pmb{x}) = \pmb{a}^\top \pmb{x} + b} g(\pmb{x}_c)$ .
Solving an instance of this problem with appropriately set  $x_{c}$  can be seen as relaxing an instance of the continuous optimal bounding problem to only contain a finite subset of the constraints while maintaining the original objective.
Clearly, the discrete problem is an instance of the linear programming (LP) optimization problem. It can be solved quite efficiently with one of many known approaches. We benchmark several popular general-purpose LP solvers as well as our implementation of two specific algorithms in the section 5.1 of this paper. It is worth noting that when the number of variables is bounded one can solve an LP in time linear in the number of constraints[17]. For our particular application it means that an instance of the discrete optimal linear bounding can be solved in  $O(|S|)$  for a fixed  $d$ , which makes it as easy asymptotically as simply drawing the sample.
4.2 Soundness conditions
Relaxing the constraints of an optimization problem generally leads to a solution which is not feasible with respect to the original constraints. In case of SOL we hope to reason about the soundness of the bound in a continuous region based on the values of the target function in a finite set of points. One way to enable such reasoning is to limit the analysis to functions conforming to a certain smoothness criterion.
If the target function is "smooth enough" then given a subregion  $U \subseteq R$ , a point  $\pmb{x}^*$  within it and slope  $\pmb{a}$  of the solution to the discrete problem  $g_{S}(\pmb{x})$  there exists a certain threshold  $\Delta (U, \pmb{a}, f) > 0$  such that a big enough gap  $g_{S}(\pmb{x}^{*}) - f(\pmb{x}^{*}) > \Delta$  implies  $g_{S}(\pmb{x}) > f(\pmb{x})$  for every  $\pmb{x} \in U$ . Having such a threshold allows us to obtain a sound upper bound in  $U$  by appropriately shifting  $g_{S}(\pmb{x})$  up if needed. We analyze two different classes of target functions and show how the value of the threshold  $\Delta$  depends on the parameters of the region and the smoothness of the target function.
4.2.1  $f$  is Lipschitz continuous in  $R$
Let  $L_{1} > 0$  be an upper bound on the Lipschitz constant of the function  $f$ , meaning that
$$
\left| f \left(\boldsymbol {x} _ {1}\right) - f \left(\boldsymbol {x} _ {2}\right) \right| \leq L _ {1} \left| \boldsymbol {x} _ {1} - \boldsymbol {x} _ {2} \right|
$$
for  $\pmb{x}_1, \pmb{x}_2 \in R$ . Let  $g(\pmb{x}) = \pmb{a}^\top \pmb{x} + b$  be the solution of the discrete problem associated with a particular sample of points  $S \subset R$  containing  $\pmb{x}^*$ . It can be shown that  $|\pmb{a}| \leq L_1 \cdot G$ , where  $G$  is a geometric factor determined by the shape of the region  $R$  and the structure of the initial sample  $S$ . Importantly, in one-dimensional case (e.g., all traditional activation functions) the factor  $G$  is exactly 1 for any region and any reasonable sample  $S$ . For an arbitrary point  $\pmb{x} \in U$  the following inequalities then hold
$$
g (\boldsymbol {x}) - f (\boldsymbol {x}) \geq g \left(\boldsymbol {x} ^ {*}\right) - f \left(\boldsymbol {x} ^ {*}\right) - | \boldsymbol {a} | | \boldsymbol {x} - \boldsymbol {x} ^ {*} | - L _ {1} | \boldsymbol {x} - \boldsymbol {x} ^ {*} | \geq g \left(\boldsymbol {x} ^ {*}\right) - f \left(\boldsymbol {x} ^ {*}\right) - (1 + G) L _ {1} \delta ,
$$
where  $\delta = \sup_{\pmb{x} \in U} |\pmb{x}^* - \pmb{x}|$  is the "width" of the region. Hence, as long as
$$
g \left(\boldsymbol {x} ^ {*}\right) - f \left(\boldsymbol {x} ^ {*}\right) \geq \Delta = (1 + G) L _ {1} \delta
$$
the bound is sound for the whole subregion  $U$ .
The connection between  $|\pmb{a}|$  and  $L_{1}$  originates from the following inequality which is based on  $g(\pmb{x})$  being a discrete upper bound for the  $L_{1}$ -continuous function  $f(\pmb{x})$
$$
| \boldsymbol {a} | \leq L _ {1} \left(\inf  _ {\{\boldsymbol {x} _ {i} \}, \boldsymbol {v}} \sup  _ {\boldsymbol {x} \in S, i} \boldsymbol {v} \cdot \frac {\boldsymbol {x} - \boldsymbol {x} _ {i}}{| \boldsymbol {x} - \boldsymbol {x} _ {i} |}\right) ^ {- 1} = G (R, S) L _ {1},
$$
where the infimum is taken over all finite subsets  $\{\pmb{x}_i\} \subset R$  such that the center  $\pmb{x}_c$  lies in the subset's convex hull and all unitary vectors  $\pmb{v}$ . The factor  $G(R, S)$  depends on the geometry of the target region  $R$  and on the structure and the density of the sample  $S$ . Note that this factor decreases monotonically when we add new points to the sample, reaching a certain limit  $G(R)$  when  $S = R$ .
Algorithm 1 Adaptive SOL
$\delta \gets \delta_0$ $\{U_i\} \leftarrow$  uniform partition into cubes of size  $\delta$ $x_{c}\gets$  center of mass of  $R$ $K\gets \emptyset$    
do   
 $\{U_i\} \gets$  SPLITCUBES(\{U_i\},K)   
 $\pmb {a},\pmb {b}\gets$  SOLVEDISCRETE((  $\{\pmb {(x_i,f(x_i))}\} ,\pmb {x}_c)$ $\{\Delta_i\} \gets$  GETGAPTHRESHODLS(\{U_i\},  $\{\pmb {x}_i\}$  ，  $\pmb {a},\pmb{b})$ $K\gets \{i|\Delta_i - (g_S(\pmb {x}_i) - f(\pmb {x}_i)) > \frac{\varepsilon}{V}\}$    
while  $K\neq \emptyset$ $b = b + \frac{\varepsilon}{V}$    
return  $\mathbf{a},b$
4.2.2  $\nabla f$  is Lipschitz continuous in  $R$
Let  $L_{2} > 0$  be an upper bound on the Lipschitz constant of the gradient  $\nabla f$  of the target function. Given a bounded  $R$  this immediately implies the Lipschitzness of the target function  $f$  itself. Let  $L_{1}$  be an upper bound on the Lipschitz constant of  $f$ .
The functions  $f(\pmb{x})$  of this type behave "approximately linearly" near the sample point  $\pmb{x}^*$ . If we denote the linearized version of  $f(\pmb{x})$  as  $f_{l}(\pmb{x}) = f(\pmb{x}^{*}) + \nabla f(\pmb{x}^{*}) \cdot (\pmb{x} - \pmb{x}^{*})$ , the inequality  $|f(\pmb{x}) - f_{l}(\pmb{x})| \leq \frac{1}{2} L_{2} |\pmb{x} - \pmb{x}^{*}|^{2}$  holds for an arbitrary point  $\pmb{x}$ .
As a consequence, the following inequality holds for the discrete bound  $g(\pmb{x})$
$$
g (\boldsymbol {x}) - f (\boldsymbol {x}) \geq g (\boldsymbol {x} ^ {*}) - f (\boldsymbol {x} ^ {*}) - | \nabla f (\boldsymbol {x} ^ {*}) - \nabla g (\boldsymbol {x} ^ {*}) | \delta + \frac {1}{2} L _ {2} \delta^ {2}.
$$
Combined with the previous condition which is still applicable in this setting the inequality gives us the following expression for the gap threshold
$$
\Delta = \min  ((1 + G) L _ {1} \delta , | \nabla f (\boldsymbol {x} ^ {*}) - \nabla g (\boldsymbol {x} ^ {*}) | \delta + \frac {1}{2} L _ {2} \delta^ {2})
$$
Intuitively, the introduction of this additional condition matters the most near the "touching points" between the function and the optimal linear bound. Around these points the gradients in the expression are going to be close to each other, so the second term in the minimization will allow for a significantly smaller gaps than those of the first term.
4.3 The algorithm
Now we have all the parts necessary to define the SOL algorithm as well as its simplified version.
At all times during the runtime of the algorithm we are going to maintain the partition of the original region into a collection of cubes  $U_{i} = \left\{\pmb{x} \in \mathbb{R}^{d} \mid \bigwedge_{j=1}^{d} x_{i}^{j} - l_{i} \leq x^{j} \leq x_{i}^{j} + l_{i}\right\}$  with side lengths of  $2l_{i}$  such that  $R \subseteq \bigcup_{i} U_{i}$ . The sample  $S = \{\pmb{x}_{i}\}$  we are going to use for the formulation of the discrete problem at a particular iteration is going to consist of the center points of these cubes.
Given the solution  $g_{S}(\pmb{x})$  of the discrete problem we are going to calculate the gap thresholds  $\Delta_{i}$  associated with all  $(U_{i},\pmb{x}_{i})$  pairs according to one of the soundness conditions. The shift of value  $\eta = \max_{i}\left[\Delta_{i} - (g_{S}(\pmb{x}_{i}) - f(\pmb{x}_{i}))\right]$  can then be applied to the discrete bound to obtain a bound sound in the entire target region  $R$ :  $g_{S}^{*}(\pmb{x}) = g_{S}(\pmb{x}) + \eta$ .