SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By optimizing over the choice of diagonal  $T$  satisfying  $G^{\top}G\preceq T$ , we can find the best Lipschitz bound achievable by this method. In this paper, however, we are more interested in analytical choices of  $T$ . Using the same techniques outlined in [38], we have the following choices:
1. Spectral Normalization (SN): We can satisfy  $G^{\top}G \preceq T$  simply by choosing  $T = T_{SN} \coloneqq \| G\|_2^2 I$ . In this case,  $L_{\mathrm{LT - SDP}}(T_{SN}) = \| H + \frac{\alpha + \beta}{2} GW\|_2 + \frac{\beta - \alpha}{2}\| W\|_2\| G\|_2 = L_{\mathrm{LT}}$ .
2. AOL [36]: Choose  $T_{ii} = \sum_{j=1}^{n} |G^{\top} G|_{ij}$ . As a result,  $T - G^{\top} G$  is real-symmetric and diagonally dominant with positive diagonal elements, and thus, is positive semidefinite.
3. SLL [38]: Choose  $T_{ii} = \sum_{j=1}^{n} |G^{\top} G|_{ij} \frac{q_j}{q_i}$ , where  $q_i > 0$ . [38] uses the Gershgorin circle theorem to prove that this choice of  $T$  satisfies  $G^{\top} G \preceq T$ .
3.1 Multi-layer Neural Networks
We now extend the proposed technique to the multi-layer case. Consider the following structure,
$$
y _ {k} = W _ {k} x _ {k} \quad k = 0, \dots , L
$$
$$
x _ {k + 1} = H _ {k} x _ {k} + G _ {k} \phi \left(y _ {k}\right) k = 0, \dots , L - 1 \tag {14}
$$
with  $W_{L} = I$  for consistency. Then  $y_{L} = x_{L}$  is the output of the neural network. To compute a Lipschitz constant, we first apply loop transformation to each activation layer, resulting in the following representation.
Lemma 1 Consider the sequences in (14). Define  $\hat{H}_k\coloneqq H_k + \frac{\alpha + \beta}{2} G_kW_k$  for  $k = 0,\dots ,L - 1$ . Then
$$
y _ {k + 1} = W _ {k + 1} \hat {H} _ {k} \dots \hat {H} _ {0} x _ {0} + \sum_ {j = 0} ^ {k} W _ {k + 1} \hat {H} _ {k} \dots \hat {H} _ {j + 1} G _ {j} \psi (y _ {j}) \quad k = 0, \dots , L - 1. \tag {15}
$$
We are now ready to state our result for the multi-layer case.
Theorem 3 Let  $m_k, k \geq 1$  be defined recursively as
$$
m _ {k + 1} = \left\| W _ {k + 1} \hat {H} _ {k} \dots \hat {H} _ {0} \right\| + \frac {\beta - \alpha}{2} \sum_ {j = 0} ^ {k} \left\| W _ {k + 1} \hat {H} _ {k} \dots \hat {H} _ {j + 1} G _ {j} \right\| m _ {j} \tag {16}
$$
with  $m_0 = \| W_0\|$ . Then  $m_{k}$  is a Lipschitz constant of  $x_0 \mapsto y_k, k = 0, \dots, L - 1$ . In particular,  $m_L$  is a Lipschitz constant of the neural network  $x_0 \mapsto y_L = x_L$ .
Similar to the single-layer case, we can refine the bounds for the  $\ell_2$  norm. For the sake of space, we defer this to the Appendix, and we use the bounds in (16) in our experiments.
Complexity For an  $L$ -layer network, the recursion in (16) requires  $\mathcal{O}(L^2)$  number of matrix norm calculations. For the naive bound, this number is exactly  $L$ . Despite this increase in complexity, we utilize the recurring structure of the calculations that enable parallelized implementation on GPUs. This utilization results in a reduction of the time complexity to  $\mathcal{O}(L)$ , the same as the naive Lipschitz estimation algorithm. We provide a more detailed analysis of the computational and time complexity and the GPU implementation in Appendix A.4.
3The Theorem of [30] assumes that  $\alpha \geq 0$  (monotonicity), but it can be shown that the theorem holds as long as  $-\infty < \alpha$ , which is the case for the loop-transformed nonlinearity.
4Note that we let  $\hat{H}_k\cdot \cdot \cdot \hat{H}_{j + 1}\big|_{j = k} = I$
In this section, we evaluate our proposed method for training deep classifiers on the MNIST [46], CIFAR-10 [47] and Tiny-Imagement [48] datasets. We compare our results with the closely related state-of-the-art methods. We further analyze the efficiency of our improved Lipschitz bounding algorithm and leverage it to compute the certified robust radii for the test dataset.
4.1  $\ell_2$ -Robustness
Experimental setup We train using our novel loss function to certify robustness (6), using cross-entropy as the differentiable surrogate, against  $\ell_2$  perturbations of size 1.58 on MNIST and 36/255 on CIFAR-10 and Tiny-Imagenet  $^5$ . We train convolutional neural networks of the form mCnF with ReLU activation functions, as in [20], where  $m$  and  $n$  denote the number of convolutional and fully connected layers, respectively. The details of the architectures, training process, and most hyperparameters are deferred to the supplementary materials.
Baselines For baselines, we consider recent state-of-the-art methods that have provided results on convolutional neural networks of similar size. We consider: (1) GloRo [5] which uses naive Lipschitz estimation with a smoothness regularizer inspired by TRADES; (2) Local-lip-B/G (+ MaxMin) [20] which uses local Lipschitz constant calculation along with cut-off ReLU and MaxMin activation functions; (3) LipConvnet-5/10/15 [34] that uses 1-Lipschitz convolutional networks with GNP Householder activations; (4) SLL-small (SDP-based Lipschitz Layers) [38] which is a much larger 27 layer 1-Lipschitz network; (5) AOL-Small/Medium [36] which presents much larger 1-Lipschitz networks trained to have almost orthogonal layers.
Evaluation We evaluate the accuracy of the trained networks under 3 criteria; standard accuracy, adversarial accuracy, and certified accuracy. For adversarial accuracy we perform PGD attacks using [4] (hyperparameters provided in supplementary material). For certified accuracy, we calculate the certified radii using (4) and verify if they are larger than the priori-assigned perturbation budget. For the methods of recent literature, we report their best numbers as per the manuscript.
Results The results of the experiments are presented in Table 1. On MNIST, we outperform the state-of-the-art by a margin of  $7.5\%$  on verified accuracy whilst maintaining the same standard accuracy. On CIFAR-10 we surpass all of the state-of-the-art in verified accuracy. Furthermore, on methods using networks of similar sizes, i.e., 6C2F or LipConvnet-5, we surpass by a margin of around  $4\%$ . The networks of the recent literature AOL [36] and SLL [38] are achieving slightly worse certified accuracy even with much larger networks. On Tiny-Imagenet, our current best results are on par with the state-of-the-art.
4.1.1 Lipschitz Estimation
A key part of our method is the new improved Lipschitz estimation algorithm (LipLT) and the effective use of pairwise Lipschitz constants. Unlike previous works that estimate the pairwise Lipschitz between class  $i$  and  $j$  by the upper bound  $\sqrt{2} L_z$  [20], where  $L_z$  is the Lipschitz constant of the whole network, or as  $L_i + L_j$  [5], where  $L_i$  is the Lipschitz constant of the  $i$ -th class, we approximate this value directly as  $L_{ij}$  by considering the map  $z_i(x;\theta) - z_j(x;\theta)$ . Table 2 shows the average statistic for Lipschitz constants estimated using different methods. Our improved Lipschitz calculation algorithm provides near an order-of-magnitude improvement over the naive method. Furthermore, Table 2 portrays the superiority of using pairwise Lipschitz constants instead of other proxies. Figure 2a illustrates the difference between using LipLT versus the naive method to calculate the certified radius. In this experiment, the naive method barely certifies one percent of the data points at the perturbation level of 1.58. However, using LipLT, we can certify a significant portion of
We selected these values for the perturbation budget for consistency with prior work.
Table 1: Comparison with recent certified training algorithms. Best certified training accuracies are highlighted in bold.
* Due to a size mismatch that occurs in power iteration, we had to modify the architecture slightly by changing the padding of some of the convolutional layers. The number of neurons are the same as that of the original architecture in [20]. More details in the supplementary material
Method	Model	Clean (%)	PGD (%)	Certified(%)
MNIST (ε = 1.58)
Standard	4C3F	99.0	45.4	0.0
GloRo	4C3F	92.9	68.9	50.1
Local-Lip	4C3F	96.3	78.2	55.8
CRM (ours)	4C3F	96.27	88.04	63.37
CIFAR-10 (ε = 36/255)
Standard	6C2F	87.5	32.5	0.0
GloRo	6C2F	77.0	69.2	58.4
Local-Lip-G	6C2F	76.4	69.2	51.3
Local-Lip-B	6C2F	70.7	64.8	54.3
Local-Lip-B + MaxMin	6C2F	77.4	70.4	60.7
LipConvnet	5-CR	75.31	-	60.37
LipConvnet	10-CR	76.23	-	62.57
LipConvnet	15-CR	76.39	-	62.96
SLL	Small	71.2	-	62.6
AOL	Small	69.8	-	62.0
AOL	Medium	71.1	-	63.8
CRM (ours)	6C2F	74.82	72.31	64.16
Tiny-Imagenet (ε = 36/255)
Standard	8C2F	35.9	19.4	0.0
Local-Lip-G	8C2F	37.4	34.2	13.2
Local-Lip-B	8C2F	30.8	28.4	20.7
GloRo	8C2F	35.5	32.3	22.4
Local-Lip-B + MaxMin	8C2F	36.9	33.3	23.4
SLL	Small	26.6	-	19.5
CRM (ours)	8C2F*	23.97	23.04	17.98
(b)
Figure 2: (a) Distribution of certified radii calculated using direct pairwise Lipschitz constants for the MNIST test dataset for the network trained using CRM. (b-c) Comparison of the distribution of the certified radii for a model trained using CRM (top) versus a standard trained model (bottom) for MNIST (b) and CIFAR-10 (c). For any given  $\epsilon$ , the probability curves denote the empirical probability of a data point from that data set having a certified radius of at least  $\epsilon$ .
the data points.
We further conduct an analysis of the certified radii of the data points for regularized versus unregularized networks for MNIST and CIFAR-10. Figures 2b and 2c illustrate the distribution of the certified radii. These radii were computed using the direct pairwise Lipschitz bounding. When comparing the results of the CRM-trained models (top figures) with the standard models (bottom figures), it becomes evident that our loss function enhances the certified radius.
Table 2: Comparison of average pairwise Lipschitz constant calculated using the naive and improved method on networks trained using CRM. We use the same architectures as Table 1. To calculate the averages, we form all unique pairs and calculate the Lipschitz constant according to the relevant formulation and then take the average.
MNIST	CIFAR10	Tiny-Imagenet
Lij	Li + Lj	√2L	Lij	Li + Lj	√2L	Lij	Li + Lj	√2L
Naive	266.08	1285.18	2759.63	93.52	131.98	139.75	313.62	485.62	2057.00
Improved	64.10	212.68	419.83	11.26	18.30	23.43	9.46	14.77	60.37
Although our proposed formulation has an elegant mathematical structure, there exist some limitations: (1) We are still using global Lipschitz constants to bound the margin, which can be conservative. We will investigate how we can localize our calculations without a significant increase in computation. (2) Computation of pairwise Lipschitz bounds for very deep architectures and or a large number of classes can become computationally intensive for training purposes (see Appendix A.4 for further discussions). (3) Several hyper-parameters require manual tuning. It is highly desirable to explore adaptive approaches to automatically select these hyper-parameters.
Adversarial defense methods are numerous and their approaches are different, but they all attempt to explicitly or implicitly increase the margin of the classifier, which is a measure of adversarial robustness (or vulnerability). From this perspective, it is highly desirable to develop adversarial defenses that can manipulate the decision boundary and increase the margin effectively and efficiently. We attempt to maximize the input margin by penalizing the Lipschitz constant of the neural network along vulnerable directions. Additionally, we develop a new method for calculating guaranteed analytic and differentiable upper bounds on the Lipschitz constant of the deep network. LipLT is provably better than the naive Lipschitz constant. We have also provided a parallelized implementation of LipLT using the recurring structure of the calculations, which is fast and scalable. Our proposed method achieves competitive results in terms of verified accuracy on the MNIST and CIFAR-10 datasets.
[62] Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, Masashi Sugiyama, et al. "Probabilistic margins for instance reweighting in adversarial training". In: Advances in Neural Information Processing Systems 34 (2021), pp. 23258-23269.
[63] Matthew Mirman, Timon Gehr, and Martin Vechev. "Differentiable abstract interpretation for provably robust neural networks". In: International Conference on Machine Learning. 2018, pp. 3575-3583.
[64] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. "Efficient neural network robustness certification with general activation functions". In: Advances in neural information processing systems. 2018, pp. 4939-4948.
[65] Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. "Semidefinite relaxations for certifying robustness to adversarial examples". In: Advances in Neural Information Processing Systems. 2018, pp. 10900-10910.
[66] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. "Certified defenses against adversarial examples". In: arXiv preprint arXiv:1801.09344 (2018).
[67] Yuhao Mao, Mark Niklas Müller, Marc Fischer, and Martin Vechev. "TAPS: Connecting Certified and Adversarial Training". In: arXiv preprint arXiv:2305.04574 (2023).
[68] Diederik P Kingma and Jimmy Ba. "Adam: A method for stochastic optimization". In: arXiv preprint arXiv:1412.6980 (2014).