SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Importantly, since  $g_{S}(\pmb{x})$  is a solution of the relaxed problem its discrepancy is a lower bound for the optimal discrepancy. On the other hand,  $g_{S}^{*}(\pmb{x})$  is a feasible bound for the continuous optimization problem, hence its discrepancy is an upper bound for the optimal discrepancy. Combining these two observations with the proposition 1 we can conclude that  $g_{S}^{*}(\pmb{x})$  is within  $\eta \cdot V$  of the optimum discrepancy volume-wise. The value of the shift can be directly controlled by varying the density of the cubes in the partition since  $\eta \leq \max_i \Delta_i \leq (1 + G)L_1\max_i\delta_i$ , where  $\delta_i = \sup_{\pmb{x} \in U_i}|\pmb{x}_i - \pmb{x}| = l_i\sqrt{d}$  is the size of cube  $U_i$ .
This observation inspires the uniform version of SOL. It only performs a single iteration with a uniform partition of  $R$  into equally-sized cubes of size  $\delta \leq \frac{\varepsilon}{(1 + G)L_1V}$ , where  $\varepsilon$  is the desired accuracy target. While being attractively simple this version turns out to be too slow in practice. Indeed, in a typical case we would only need such a high density of sample near the touching points of the optimal upper bound. Having sampled the whole region with a uniform density we end up with an intractably big discrete problem.
The solution to this issue is to start with a sparse regular lattice of cubes as a partition and gradually change it according to the values of  $\Delta_{i}$  obtained after solving a discrete problem. Specifically, we would want to increase the density of cubes in such subregions  $U_{i}$  that
$$
\Delta_ {i} - \left(g _ {S} \left(\boldsymbol {x} _ {i}\right) - f \left(\boldsymbol {x} _ {i}\right)\right) > \frac {\varepsilon}{V}, \tag {1}
$$
as these regions require too big of a shift in order to guarantee the soundness. In order to locally increase the density we split the target cube  $U_{i}$  into  $2^{d}$  evenly spread cubes of half the size of the original. An example of the split procedure can be seen in figure 3. This modification concludes the adaptive version of the SOL algorithm – algorithm 1.
4.4 On the complexity of SOL
Note that the cubes of size  $\delta \leq \delta_c = \frac{\varepsilon}{(1 + G)L_1V}$  cannot violate the inequality (1) and, consequently, are never split. The maximum total number of points in  $S$  achievable until the accuracy target is met is then defined by the critical cell size  $\delta_c$  as  $N_{tot} = O\left(\frac{V}{\delta_c^d}\right) = O(\varepsilon^{-d})$ , where we only track the complexity's dependence on  $\varepsilon$ . Given that each iteration increases the sample size by at least 1 and solving the discrete problem as well as updating  $S$  can be done in  $O(|S|)$ , we can bound the worst case complexity of SOL by  $O(N_{tot}^2) = O(\varepsilon^{-2d})$ .
While this runtime bound may seem excessively conservative, there exist linear bounding problem instances on which SOL is going to take similar  $O(\varepsilon^{-\gamma})$  amount of time for certain values of  $\gamma$ . One particular example is bounding a linear function, where SOL would have to split all cubes down to the size of  $\delta_c$  or  $\delta_c' = \sqrt{\frac{2\varepsilon}{L_2V}}$  depending on which soundness criterion is used. This example would give us the lower complexity bounds of  $\Omega(\varepsilon^{-d})$  and  $\Omega(\varepsilon^{-\frac{d}{2}})$  respectively just for the final iteration of SOL alone.
The situation is quite different, however, for many practical instances of the linear bounding problem. In a typical scenario the optimal bound  $g_{opt}(\pmb{x})$  only " touches" the target function  $f(\pmb{x})$  in  $d$  separate points and  $f(\pmb{x})$  has nondegenerate Hessian in each of them. The high sample density of  $\delta_c^{-d}$  or  $(\delta_c')^{-d}$  is then only necessary in the vicinity of these touching points. Taking the continuous limit one can show that verifying an  $\varepsilon$ -optimal linear bound in this case only requires  $O(\log \frac{1}{\varepsilon})$  points in the sample  $S$  when using the  $L_2$  soundness condition. This logarithmic estimate of the number of points required for the verification is the primary reason we introduce the  $L_2$  condition. Provided that the adaptive sampling procedure "finds" the optimal sample "fast enough" by splitting the cells in appropriate regions, one can hope to achieve the total running time of  $O(\log^{\beta} \frac{1}{\varepsilon})$  for some  $\beta \geq 1$  in arbitrary dimensionality  $d$ . We show that this is indeed the case in practice in section 5.2 where we measure the running time of SOL in a synthetic dataset of bounding problems involving popular activation functions.
In this section we study the performance of SOL empirically by measuring its runtime and the quality of the linear bounds on a synthetic dataset containing randomly generated instances of the bounding problem. We also show the practical applicability of SOL by incorporating it into AutoLiRPA and measuring its robustness certification performance on several image classification neural network architectures.
5.1 Benchmarking discrete problem solvers
Each run of SOL may involve dozens of invocations of the discrete problem solver. There are numerous different LP solvers that can be used for handling the discrete problem. Their performance
Figure 3: Adaptive sampling.
Figure 4: Runtime of LP solvers.
Algorithm 2 1D bisect algorithm
$u\gets \max \{y_i\}$ $l\gets \min \{y_i\}$ $a\gets 0$    
while  $u - l > \varepsilon$  do  
 $m\leftarrow \frac{l + u}{2}$ $l_{max}\gets \max \left\{\frac{m - y_i}{x_c - x_i} |x_i <   x_c\right\}$ $r_{max}\gets \max \left\{\frac{y_i - m}{x_i - x_c} |x_i > x_c\right\}$   
if  $r_{max}\leq l_{max}$  then  
 $u\gets m$ $a\gets \frac{l_{max} + r_{max}}{2}$   
else  
 $l\gets m$   
end if  
end while  
return  $a,u$
is known to vary drastically between the different algorithms used at their core and the distributions of problems targeted. We benchmark 4 approaches on a synthetic dataset of the discrete problem instances to find the one most suitable for our applications.
The dataset was generated by sampling 2000 discrete problem instances for each of the following activation functions: GeLU, Log Log, Swish. We use these three activation functions as they have become quite popular in recent years[8, 21, 22, 27] but don't have optimal linear bounds handcrafted for them specifically. Each problem instance has the region's  $R = [l,r]$  boundaries sampled uniformly from the  $[-2,2]$  interval and contains 500 points sampled uniformly from the region. This number of points was chosen for the experiments as corresponding to the typical final sample size of SOL in application to the robustness certification. Although the uniform distribution of points might not give an accurate representation of the samples encountered by SOL, the difference between the running times of the solvers is pronounced enough to ignore this inaccuracy.
The approaches we investigate include two well-known optimization libraries capable of solving LP: Gurobi[7] and SciPy[31]. The other two approaches are our implementation of Megiddo's linear-time algorithm[17] and a specific bisection procedure which takes advantage of the linear ordering of the sample points in the one-dimensional case. The idea behind the bisection procedure is that a particular  $g(\boldsymbol{x}_c) = h$  guess on the objective value can be validated in linear time by checking whether
(a) Running time comparison.
(b) Optimality gap.
(c) Running time in log-scale.
Figure 5: SOL benchmarking results.
$(x_{c},h)$  lies inside the convex hull of the sample or not. A pseudocode implementing the procedure is shown in the algorithm 2. Note that this procedure is only applicable for  $d = 1$
The benchmarking results are shown in the figure 4. The plot shows the dependence of solver's average runtime across all the problems in the dataset on the targeted optimality tolerance  $\varepsilon$ . Interestingly, both general LP solvers show slower performance than the alternatives, which highlights the specific nature of linear programs encountered by SOL. Both of their running times seem almost independent of the optimality tolerance required. One possible reason would be the presence of significant overhead of setting up sophisticated optimization routines.
The results of the Megiddo's algorithm show expectedly saturating runtime dependence on  $\varepsilon$  since it's supposed to find the exact solution in  $O(|S|)$  time. The only dependence on  $\varepsilon$  is due to the presence of an early stopping upon reaching the target accuracy. The bisection approach shows the best results with a significant margin along the entire bandwidth of  $\varepsilon$  values. Although its dependence on  $\varepsilon$  is asymptotically worse than that of the Megiddo's algorithm  $-O(\log \frac{1}{\varepsilon})$  vs  $O(1)$  - the constant factor of the latter appears to be too high for it to outperform the bisection procedure at practical values of optimality tolerance. Taking into consideration this result, we use the bisect algorithm as the discrete problem solver in all of the remaining evaluations.
5.2 Benchmarking SOL
In order to study the average runtime and the bounding performance of SOL on the linear bounding problem instances encountered in practice we generate another synthetic dataset. This time we sample 1000 instances of the problem for each of the same three functions. The region bounds for the problems are sampled uniformly from the  $[-3, 3]$  interval.
We compare the SOL's performance on this benchmark to the performance of LinSyn[20] as their approach also aims to build bounds close to the optimal ones. It does not, however, give any guaranties on the optimality gap and does not have parameters controlling the accuracy target. Its metrics are thus independent of the optimality target  $\varepsilon$ .
Figures 5a and 5b show the average runtime and the average optimality gap respectively as functions of the optimality target  $\varepsilon$ . The optimality gap is estimated by taking the most accurate linear bound produced by SOL and shifting in down appropriately to obtain a tight underestimate of the optimal value of objective. Notably, there appears to be a wide range of accuracy targets  $\varepsilon \in [10^{-7}, 10^{-3}]$  where SOL gives better average bounds while taking less time than LinSyn. For example, at  $\varepsilon = 10^{-5}$  the SOL bounds are 10 times tighter than these of LinSyn while the runtime is 3 time shorter.
While figure 5a already hints on the form of the asymptotic dependency of the running time  $t$  on the accuracy target  $\varepsilon$ , we make the analysis clearer by investigating the relation between  $\log t$  and  $\log \log \varepsilon$  instead. This dependence shown in figure 5c turns out to be quite close to linear with the slope of  $a \approx 2.0$  estimated by the linear regression. The linear relation with this slope corresponds to the complexity of  $O(\log^2 \frac{1}{\varepsilon})$  and agrees with our hypothesis on the typical running time of SOL.
Table 1: Certification performance comparison. The upper number is the fraction of properties certified, the lower is the estimate of the running time (with std  $\leq 3\%$ ). Neural network architectures are denoted by the activation function used and their respective depths.
Approach	Dataset
MNIST [15]	CIFAR [14]
GeLU 4l	Log Log 4l	Swish 4l	GeLU 5l	Log Log 5l	Swish 5l
AutoLiRPA [35]	0.01	0.0	0.34	0.0	0.59	0.03
230s	23s	9s	700s	50s	35s
LinSyn [20]	0.72	0.23	0.76	0.31	0.69	0.35
430s	450s	410s	580s	360s	550s
SOL (ε = 10-3)	0.73	0.23	0.76	0.31	0.69	0.37
100s	96s	100s	150s	87s	150s
SOL (ε = 10-5)	0.73	0.24	0.76	0.31	0.69	0.37
140s	140s	130s	190s	95s	170s
SOL (ε = 10-7)	0.73	0.24	0.76	0.31	0.69	0.37
310s	300s	290s	400s	160s	340s
5.3 Robustness certification performance
Finally, we study the performance of SOL as a part of a robustness certification pipeline by using SOL as an activation function bounding subroutine in the AutoLiRPA[35] framework. We follow the protocol introduced in LinSyn and compare the robustness certification performance of our approach against that of LinSyn and that of AutoLiRPA with their default decompositional linear bounding approach. In these experiments we attempt the robustness certification of 6 image classification neural networks with convolutional architectures. Three of them were trained on MNIST[15] while the other three - on CIFAR[14].
Each certification task consists of proving that any  $l_{\infty}$ -bounded perturbation of a particular correctly-classified input image retains the same label prediction as the image itself. For each neural network we randomly sample 100 images from the test part of the dataset, filter out the misclassified ones and attempt to certify the remaining images. We use perturbation magnitude of 8/255 for networks trained on MNIST and 1/255 for CIFAR networks as is common in the literature. We have also conducted experiments with larger perturbations of 2/255 and 4/255 for the CIFAR models, but found the performance too weak for a proper analysis. Qualitatively the results for these larger perturbations were similar to the presented results.
The results are shown in table 1. Both LinSyn and SOL show much higher certification rates compared to AutoLiRPA since they aim at producing tight linear bounds while the default bounding procedure of AutoLiRPA systematically ends up with loose bounds.
The SOL variants show similar or slightly higher certification rates than LinSyn while taking as low as a quarter of the time. Lowering the accuracy target down from  $10^{-5}$  noticeably increases the runtime of SOL without any improvement to the success rate of the certification. Notably, although not depicted in the table, the average sizes of the final bounds on the logit outputs of the networks only improve by a fraction of  $1\%$  upon changing  $\varepsilon$  from  $10^{-3}$  to  $10^{-7}$  which is negligible compared to the improvement of x2-x5 from switching from AutoLiRPA to either LinSyn or SOL.
In this paper we have introduced SOL - a sampling-based approach for finding linear bounds arbitrarily close to optimum in terms of tightness for Lipschitz-continuous functions. We have shown experimentally that despite having rather pessimistic theoretical running time guarantees, it is quite fast in practice. As a part of a neural network robustness certification framework it increases the certification rates and takes significantly smaller time compared to the alternative approaches.
The authors would like to thank the anonymous reviewers for their feedback. This work was supported by the National Science Foundation through the following grants: CAREER award (SHF-2048094), CNS-1932620, CNS-2039087, FMitF-1837131, CCF-SHF-1932620, funding by Toyota R&D and Siemens Corporate Research through the USC Center for Autonomy and AI, an Amazon Faculty Research Award, and the Airbus Institute for Engineering Research.
The authors would also like to acknowledge early discussions with Prof. Chao Wang and Dr. Brandon Paulsen that influenced this work.
[1] M. Alzantot, Y. Sharma, A. Elghary, B.-J. Ho, M. Srivastava, and K.-W. Chang. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998, 2018.
[2] D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
[3] L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. Exploring the landscape of spatial robustness. In International conference on machine learning, pages 1802-1811. PMLR, 2019.
[4] S. Gao, S. Kong, and E. M. Clarke. dreal: An smt solver for nonlinear theories over the reals. In Automated Deduction-CADE-24: 24th International Conference on Automated Deduction, Lake Placid, NY, USA, June 9-14, 2013. Proceedings 24, pages 208-214. Springer, 2013.
[5] T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2018.
[6] G. S. d. S. Gomes and T. B. Ludermir. Complementary log-log and probit: activation functions implemented in artificial neural networks. In 2008 Eighth International Conference on Hybrid Intelligent Systems, pages 939-942. IEEE, 2008.
[7] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://www.gurobi.com.
[8] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[9] C. Kanbak, S.-M. Moosavi-Dezfooli, and P. Frossard. Geometric robustness of deep networks: analysis and improvement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4441–4449, 2018.
[10] G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, pages 97-117. Springer, 2017.
[11] G. Katz, D. A. Huang, D. Ibeling, K. Julian, C. Lazarus, R. Lim, P. Shah, S. Thakoor, H. Wu, A. Zeljić, et al. The marabou framework for verification and analysis of deep neural networks. In Computer Aided Verification: 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part I 31, pages 443-452. Springer, 2019.
[12] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
[13] C.-Y. Ko, Z. Lyu, L. Weng, L. Daniel, N. Wong, and D. Lin. Popcorn: Quantifying robustness of recurrent neural networks. In International Conference on Machine Learning, pages 3468-3477. PMLR, 2019.
[14] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[33] L. Weng, H. Zhang, H. Chen, Z. Song, C.-J. Hsieh, L. Daniel, D. Boning, and I. Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning, pages 5276-5285. PMLR, 2018.
[34] Y. Wu and M. Zhang. Tightening robustness verification of convolutional neural networks with fine-grained linear approximation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11674-11681, 2021.
[35] K. Xu, Z. Shi, H. Zhang, Y. Wang, K.-W. Chang, M. Huang, B. Kailkhura, X. Lin, and C.-J. Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems, 33:1129-1141, 2020.
[36] H. Zhang, T.-W. Weng, P.-Y. Chen, C.-J. Hsieh, and L. Daniel. Efficient neural network robustness certification with general activation functions. Advances in neural information processing systems, 31, 2018.
[37] H. Zheng, Z. Yang, W. Liu, J. Liang, and Y. Li. Improving deep neural networks using softplus units. In 2015 International joint conference on neural networks (IJCNN), pages 1-4. IEEE, 2015.