SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\langle \cdot ,\cdot \rangle$  is the Frobenius inner product. As discussed in e.g. [31, 55], the above equations combined with the well-known representation of  $T_{W}\mathcal{M}_{r}$  yield a system of three gradient flow equations for the individual factors
$$
\left\{ \begin{array}{l l} \dot {U} = - G _ {1} (U), & G _ {1} (U) = P _ {U} ^ {\perp} \nabla_ {U} \mathcal {L} (U S V ^ {\top}) (S S ^ {\top}) ^ {- 1} \\ \dot {V} = - G _ {2} (V), & G _ {2} (V) = P _ {V} ^ {\perp} \nabla_ {V} \mathcal {L} (U S V ^ {\top}) (S ^ {\top} S) ^ {- \top} \\ \dot {S} = - G _ {3} (S), & G _ {3} (S) = \nabla_ {S} \mathcal {L} (U S V ^ {\top}) \end{array} \right. \tag {5}
$$
where  $P_U^\perp = (I - UU^\top)$  and  $P_V^\perp = (I - VV^\top)$  are the projection operators onto the space orthogonal to the span of  $U$  and  $V$ , respectively.
Based on the system of gradient flows above, we propose a training scheme that at each iteration and for each layer parametrized by the tuple  $\{U,S,V\}$  proceeds as follows:
1. update  $U$  and  $V$  by numerically integrating the gradient flows  $\dot{U} = -G_{1}(U)$  and  $\dot{V} = -G_{2}(V)$
2. project the resulting  $U, V$  onto the Stiefel manifold of matrices with  $r$  orthonormal columns
3. update the  $r\times r$  weight  $S$  by integrating  $\dot{S} = -G_3(S)$
4. for a fixed robustness tolerance  $\tau$ , project the computed  $S$  onto  $\Sigma_s(\varepsilon)$ , choosing  $s$  and  $\varepsilon$  so that
-  $s$  is the best constant approximation to  $S^{\top}S$ , i.e.  $s = \operatorname{argmin}_{\alpha}\| S^{\top}S - \alpha^{2}I\|_{F}$
-  $\varepsilon$  is such that the  $\mathrm{cond}_2$  of the projection of  $S$  does not exceed  $1 + \tau$
Note that the coefficients  $s, \varepsilon$  at point 4 can be obtained explicitly by setting  $s = \sqrt{\sum_{j=1}^{r} s_j(S)^2 / r}$ , the second moment of the singular values  $s_j(S)$  of  $S$ , and  $\varepsilon = \tau s / (2 + \tau)$ . Note also that, in the differential equations for  $U, S, V$  in (5), the four steps above can be implemented in parallel for each of the three variables. The detailed pseudocode of the training scheme is presented in Algorithm 1. We conclude with several remarks about its implementation.
Remarks, implementation details, and limitations
Each step of Algorithm 1 requires three optimization steps at lines 2, 4, 6. These steps can be implemented using standard first-order optimizers such as SGD with momentum or ADAM. Standard techniques can be used to project onto the Stiefel manifold at lines 3 and 5 of Algorithm 1, see e.g. [4, 70]. Here, we use the QR decomposition. As for the projection onto  $\Sigma_s(\varepsilon)$  at line 9, we
compute the SVD of the small factor  $S$  and set to  $s + \varepsilon$  or  $s - \varepsilon$  the singular values that fall outside the interval  $[s - \varepsilon, s + \varepsilon]$ . Note that, when  $\tau = 0$ , i.e. when we require perfect conditioning for the layer weight  $W = USV^{\top}$ , then the SVD of  $S$  can be replaced by a QR step or any other Stiefel manifold projection. Indeed, we can equivalently set  $s = \sqrt{\mathrm{trace}(S^{\top}S) / r}$ , and then project onto  $\Sigma_s(0)$  by rescaling by a factor  $s$  the projection of  $S$  onto the Stiefel manifold. In this case, the system (5) further simplifies, as we can replace  $(SS^{\top})^{-1}$  and  $(S^{\top}S)^{-\top}$  with the scalar  $1 / s^2$ .
Overall, the compressed low-rank network has  $r(n + m + r)$  parameters per each layer, where  $n$  and  $m$  are the number of input and output neurons. Thus, choosing  $r$  so that  $1 - r(n + m + r) / (nm) = \alpha$  can yield a desired compression rate  $0 < \alpha < 1$  on the number of network parameters, i.e. the number of parameters one eliminates with respect to the full baseline. For example, in our experiments we will choose  $r$  so that  $\alpha = 0.5$  or  $\alpha = 0.8$ .
Computational complexity. Each pass of Alg.1 is done against a batch  $x_{\mathrm{batch}}$ . In order to obtain minimal computational costs for each step in the algorithm, we evaluate  $USV^{\top}x_{\mathrm{batch}}$  sequentially: first  $v = V^{\top}x_{\mathrm{batch}}$ , then  $u = Sv$ , and finally  $Uu$ . Assuming the size of the batch is negligible with respect to  $n$  and  $m$ , the cost of these steps is  $O(rm), O(r), O(rn)$ , respectively. Adding the bias term and evaluating the activation function requires  $O(n)$  operations. Hence, overall we have a cost per layer of  $O(r(n + m + 1))$ . Taping the forward evaluation to compute the gradient with respect to  $U, S, V$  does not affect the asymptotic costs. The QR decompositions used for  $U$  and  $V$  require  $O(r^{2}n)$  and  $O(r^{2}m)$  operations respectively,  $O(r^{2}(n + m))$  overall. Finally, computing the SVD in the projection step for  $S$  requires a worst-case cost of  $O(r^{3})$ . Hence the overall cost per layer is  $O(r(1 + r)(n + m) + r^{3})$  as opposed to the dense network training, which requires  $O(nm)$  operations per layer. If  $r \ll n, m$  then the low-rank method is cheaper than the full baseline. For example, if  $n = m$ , this happens provided  $r < \sqrt{n}$ .
Limitations. As the rank parameter  $r$  has to be chosen a-priori for each layer of the network, a limitation of the proposed approach is the potential need for fine-tuning such parameter, even though the proposed analysis in Table 1 shows competitive performance for both  $50\%$  and  $80\%$  compression rates. Also, if the layer size  $n \times m$  is not large enough, the compression ratio  $\mathrm{cr} = 1 - r(n + m + r) / (nm)$  might be limited. Thus the method works well only for wide-enough networks  $(n, m \gg r$ , so that  $\mathrm{cr} > 0$ ). Finally, a standard way to obtain better adversarial performance would be to combine the proposed conditioning-based robustness with adversarial training strategies [16, 63, 73]. However, the cost of producing adversarial examples during training is not negligible, especially when based on multi-step attacks, and thus the way to incorporate adversarial training without affecting the benefits obtained with low-rank compression is not straightforward.
4.3 Approximation guarantees
Optimization methods over the manifold of low-rank matrices are well-known to be affected by the stiff intrinsic geometry of the constraint manifold which has very high curvature around points where  $W \in \mathcal{M}_r$  is almost singular [4, 31, 70]. This implies that even very small changes in  $W$  may yield very different tangent spaces, and thus different training trajectories, as shown by the result below:
Lemma 1 (Curvature bound, Lemma 4.2 [31]). For  $W \in \mathcal{M}_r$  let  $s_{\min}(W) > 0$  be its smallest singular value. For any  $W' \in \mathcal{M}_r$  arbitrarily close to  $W$  and any matrix  $B$ , it holds
$$
\left\| P _ {W} B - P _ {W ^ {\prime}} B \right\| _ {F} \leq C s _ {\min } (W) ^ {- 1} \left\| W - W ^ {\prime} \right\| _ {F},
$$
where  $C > 0$  depends only on  $B$ .
In our gradient flow terminology, this phenomenon is shown by the presence of the matrix inversion in (5). While this is often an issue that may dramatically affect the performance of low-rank optimizers (see also Section 5), the proposed regularization step that enforces bounded singular values allows us to move along paths that avoid stiffness points. Using this observation, here we provide a bound on the quality of the low-rank neural network computed via Algorithm 1, provided there exists an optimal trajectory leading to an approximately low-rank network. We emphasize that this assumption is well-aligned with recent work showing the existence of high-performing low-rank nets in e.g. deep linear models [7, 17, 25, 49].
Assume the training is performed via gradient descent with learning rate  $\lambda > 0$ , and let  $W(t)$  be the full gradient flow (4). Further, assume that for  $t \in [0, \lambda]$  and a given  $\varepsilon > 0$ , for each layer there exists
$E(t)$  and  $\widetilde{W}(t) \in \mathcal{M}_r \cap \Sigma_s(\varepsilon)$  such that
$$
W (t) = \widetilde {W} (t) + E (t),
$$
where  $s$  is the second moment of the singular values of  $W(t)$  and  $E(t)$  is a perturbation that has bounded variation in time, namely  $\| \dot{E}(t) \| \leq \eta$ . In other words, we assume there exists a training trajectory that leads to an approximately low-rank weight matrix  $W(t)$  with almost constant singular values. Because the value  $s$  is bounded by construction, the parameter-dependent matrix  $\widetilde{W}(t)$  possesses singular values exhibiting moderate lower bound. Thus,  $W(t)$  is far from the stiffness region of (5) and we obtain the following bound, based on [31] (proof moved to Appendix C in the supplementary material)
Theorem 1. Let  $U_{k}S_{k}V_{k}^{\top}$  be a solution to (5) computed with  $k$  steps of Algorithm 1. Assume that
- The low-rank initialization  $U_0S_0V_0^\top$  coincides with the low-rank approximation  $\widetilde{W}(0)$ .
- The norm of the full gradient is bounded, i.e.,  $\| \nabla_W\mathcal{L}(W(t))\| \leq \mu$
The learning rate is bounded as  $\lambda \leq \frac{s - \varepsilon}{4\sqrt{2\mu n}}$
Then, assuming no numerical errors, the following error bound holds
$$
\left\| U _ {k} S _ {k} V _ {k} ^ {\top} - W (\lambda k) \right\| \leq 3 \lambda \eta .
$$
Note that if  $f$  is smooth enough (e.g. Lipschitz) then by the previous theorem we directly obtain an equivalent bound for the functional distance  $\| f(U_k S_k V_k^\top) - f(W(\lambda k))\|$ .
We illustrate the performance of Algorithm 1 on a variety of test cases. All the experiments can be reproduced with the code in PyTorch available at https://github.com/COMPiLELab/CondLR. In order to assess the combined compression and robustness performance of the proposed method, we compare it against both full and low-rank baselines.
For all models, we compute natural accuracy and robust accuracy. Let  $\{(x_i,y_i)\}_{i = 1,\dots ,n}$  be the set of test images and the corresponding labels and let  $f$  be the neural network model, with output  $f(x)$  on the input  $x$ . We quantify the test set robust accuracy as:
$$
\mathrm {r o b u s t \_ a c c} (\delta) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} _ {\{y _ {i} \}} (f (x _ {i} + \delta_ {i}))
$$
where  $\delta = (\delta_{i})_{i = 1,\dots,n}$  are the adversarial perturbation associated to each sample. Notice that, in the unperturbed case with  $\| \delta_i\| = 0$ , the definition of robust accuracy exactly coincides with the definition of test accuracy. In our experiments, adversarial perturbations are produced by both the fast gradient sign method (FGSM) [19] and the projected gradient descent attack (PGD) [47], with  $\| \delta_i\|_{\infty} = \epsilon$ , and  $\epsilon$  controls perturbation strength. As images in our set-up have input entries in  $[0,1]$ , the perturbed input is then clamped to that interval. Note that, for the same reason, the value of  $\epsilon$  controls in our case the relative size of the perturbation.
Datasets. We consider MNIST, CIFAR10, and CIFAR100 [33] datasets for evaluation purposes. The first contains 60,000 training images, the second one contains 50,000 training images, while the third one contains 50,000 training images. All the datasets have 10,000 test images, first two have 10 classes and last one has 100 classes. No data-augmentation is performed.
Models. We use LeNet5 [35] for MNIST dataset, VGG16 [58] and WideResnet (WRN16-4) [78] for CIFAR10 and CIFAR100. The general architecture for all the used networks is preserved across the models, while the weight-storing structures and optimization frameworks differ.
Methods. Our baseline network is the one done with the standard implementation. Cayley SGD [39] and Projected SGD [3] are Riemannian optimization-based methods that train the network weights over the Stiefel manifold of matrices with orthonormal columns. Thus, both methods ensure  $\mathrm{cond}_2(W) = 1$  for all layers. The former uses an iterative estimation of the Cayley transform, while the latter uses QR-based projection to retract the Riemannian gradient onto the Stiefel manifold. Both methods have no compression and use full-weight matrices. DLRT, SVD prune, and Vanilla are low-rank methods that ensure compression of the model parameters during training. DLRT [55] is based on a low-rank gradient flow model similar to the proposed Algorithm 1. SVD prune [75] is based on a regularized loss with a low-rank-promoting penalty term. This approach was designed to
compress the ranks of the network after training, but we imposed a fixed compression rate from the beginning for a fair comparison. "Vanilla" denotes the obvious low-rank approach, that parametrizes the layers as  $W = UV^{\top}$  and performs alternate descent steps over  $U$  and  $V$ , as done in e.g. [29, 71]. All models are implemented with fixed training compression ratios  $\alpha = 0.5$  and  $\alpha = 0.8$ , i.e. we only use  $50\%$  and  $20\%$  of the parameters the full model would use during training, respectively. Finally, we implement CondLR as in Algorithm 1 for three choices of the conditioning tolerance  $\tau \in \{0, 0.1, 0.5\}$ . We also implement a modified version of Algorithm 1 in which  $S$  is directly projected onto the Stiefel manifold  $\Sigma_1(0)$ , i.e. the parameter  $s$  is fixed to one.
Training. Each method and model was trained for 120 epochs of stochastic gradient descent with a minibatch size of 128. We used a learning rate of 0.1 for LeNet5 and 0.05 for VGG16 with momentum 0.3 and 0.45, respectively, and a learning rate scheduler with factor = 0.3 at 70 and 100 epochs.
Results. For comparison, we measure robust accuracy for all chosen combinations of datasets, models, and methods using FGSM and PGD attacks with relative perturbation budget  $\epsilon \in \{0.01, 0.02, 0.030, 0.04, 0.05, 0.06\}$  for MNIST,  $\epsilon \in \{0.001, 0.002, 0.003, 0.004, 0.005, 0.006\}$  for CIFAR10 VGG16,  $\epsilon \in \{0.0003, 0.0006, 0.001, 0.0013, 0.0016\}$  for CIFAR10 and CIFAR100 with WRN16-4.
The results are summarized in Table 1 for LeNet5 MNIST and VGG16 CIFAR10 with FGSM and a subset of the perturbation budget, and in Table 2 for WRN16-4 CIFAR10 with both FGSM and PGD. The additional results are presented in Tables 3–6 and are moved to Appendix A. In the tables, we highlight (in gray) the best-performing method for each range of compression rates  $\alpha \in \{0\%, 50\%, 80\% \}$ , and the best method overall (in bold). The experiments show that the proposed method meaningfully conserves the accuracy as compared to the baseline for all datasets; moreover, the robustness is improved from the baseline for  $50\%$  compression rate and, additionally, in the case of MNIST for  $80\%$  compression rate. Compared to orthogonal non-compressed methods, CondLR with  $50\%$  compression rate also outperforms Cayley SGD and Projected SGD. At the same time CondLR is compressed by design, so we reduce memory costs alongside gaining robustness. As anticipated in Section 4.1, low-rank methods without regularization deteriorate the condition number and thus the robustness of the model. This is consistent with the results of Table 1, where we observe that compression without regularization does not work well in terms of robust accuracy, in particular DLRT, Vanilla and SVD prune exhibit a drop in the performance in comparison with baseline and, consequently, CondLR. Specifically, per each fixed compression ratio  $\alpha$ , our method exhibits the highest robustness; moreover, the robustness of CondLR with  $\alpha = 0.8$  is higher than the robustness of any other method with an even lower compression ratio,  $\alpha = 0.5$ .
As a further experimental evaluation, we analyze the robustness of low-rank training models with respect to small singular values. As shown by Theorem 1, CondLR is not affected by the steep curvature of the low-rank manifold  $\mathcal{M}_r$ , i.e. accuracy and convergence rate of CondLR do not deteriorate when the conditioning of the weight matrices explodes. In contrast, small singular values may significantly affect alternative low-rank training models. In Figure 2 we show the behavior of loss, accuracy, and condition number as functions of the iteration steps, when the network is initialized with ill-conditioned layer matrices. Precisely, similar to what is done in [29], we randomly sample Gaussian initial weights, compute their SVD to define the initial  $U$  and  $V$  factors, and then force the singular values of the initial  $S$  factor to decay exponentially. We observe that CondLR and DLRT (which is also based on a low-rank gradient flow formulation) are the most robust, while the performance of the other methods deteriorates dramatically. This confirms that, as also observed in [29, 55], most low-rank approaches require specific fine-tuned choices of the initialization, while the proposed robust low-rank training model ensures solid performance independent of the initialization.
In recent years, extensive effort has been put into (a) studying different forms of implicit bias in deep learning, including bias towards low-rank parameter weights [7, 9, 18], and (b) designing modified training techniques that take advantage of the implicit low-rank structure to reduce training costs and compress the network [29, 55, 71, 75]. Based on the notion of condition number of a network, as a form of local Lipschitz constant, in this work we observe that training directly on the manifold of matrices with a fixed rank may lead to instabilities due to bad conditioning, going against the intuition that low-rank networks may be more robust to adversarial attacks. Thus, we propose an
algorithm able to mitigate this phenomenon in a computationally affordable way, allowing us to train directly on the manifold of matrices of a fixed rank while controlling the condition number.
Table 1: Method comparison results
Rel. perturbation ε	LeNet5 MNIST	VGG16 Cifar10	c.r. (%)
0.0	0.02	0.04	0.06	0.0	0.002	0.004	0.006
Baseline	0.9872	0.9793	0.9655	0.9407	0.9104	0.752	0.5822	0.4592	0
Cayley SGD	0.9874	0.9804	0.9688	0.9486	0.8962	0.7446	0.5816	0.4529	0
Projected SGD	0.9878	0.9807	0.968	0.9476	0.897	0.7455	0.5832	0.4574	0
CondLR	τ=0	0.9883	0.9825	0.9732	0.958	0.9099	0.7456	0.5711	0.4292
τ=0.1	0.9877	0.9828	0.9763	0.9677	0.9093	0.7411	0.5985	0.4878
τ=0.5	0.9867	0.9802	0.9724	0.9598	0.8997	0.7225	0.6019	0.5017
stief	0.986	0.9809	0.9721	0.9586	0.9138	0.7322	0.546	0.4054
DLRT	0.967	0.9573	0.939	0.9078	0.8425	0.599	0.441	0.3691	50
Vanilla	0.9875	0.9773	0.9603	0.94	0.8997	0.6771	0.4886	0.3849	50
SVD prune	0.9883	0.9793	0.9639	0.9414	0.8992	0.673	0.4777	0.3698	50
CondLR	τ=0.0	0.9882	0.9801	0.9676	0.9452	0.9066	0.7263	0.541	0.4
τ=0.1	0.9877	0.9795	0.966	0.9444	0.9048	0.7123	0.5262	0.4013
τ=0.5	0.9858	0.9768	0.9613	0.9342	0.8933	0.6823	0.4854	0.3666
stief	0.9815	0.9729	0.9581	0.9399	0.9067	0.7184	0.5289	0.3861
DLRT	0.9649	0.9517	0.9281	0.8865	0.8092	0.5839	0.4178	0.3086	80
Vanilla	0.9862	0.972	0.9464	0.9194	0.881	0.6424	0.4266	0.299	80
SVD prune	0.9864	0.9737	0.9512	0.9281	0.8799	0.6357	0.4206	0.2927	80
Table 2: WRN16-4 Cifar10
Rel. perturbation ε	0.0	0.0003	0.0006	0.001	0.0013	0.0016	cr (%)
Baseline, FGSM	0.9129	0.885	0.8531	0.8056	0.7667	0.7263	0
CondLR	τ = 0.0	0.9254	0.8958	0.865	0.8092	0.7669	0.7236	50
τ = 0.1	0.9271	0.8974	0.8612	0.8117	0.7697	0.7223	50
τ = 0.0	0.9106	0.8807	0.844	0.7874	0.7413	0.697	80
τ = 0.1	0.9146	0.8833	0.8497	0.8004	0.7568	0.7142	80
Baseline, PGD10	0.9129	0.8898	0.8619	0.8188	0.7828	0.7435	0
CondLR	τ = 0.0	0.9254	0.8997	0.8714	0.8241	0.7791	0.7375	50
τ = 0.1	0.9271	0.9018	0.8684	0.8233	0.782	0.7374	50
τ = 0.0	0.9106	0.8838	0.8512	0.8014	0.7565	0.7064	80
τ = 0.1	0.9146	0.8861	0.8582	0.813	0.7705	0.724	80
Figure 2: Evolution of loss, accuracy, and  $\prod_{i}\mathrm{cond}(W_{i})$  for Lenet5 on MNIST dataset, for ill-conditioned initial layers whose singular values are forced to decay exponentially with powers of two.
[1] P. Ablin and G. Peyré. Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference on Artificial Intelligence and Statistics, pages 5636-5657. PMLR, 2022.
[2] P.-A. Absil, R. Mahony, and R. Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2008.
[3] P.-A. Absil and J. Malick. Projection-like retractions on matrix manifolds. SIAM Journal on Optimization, 22(1):135-158, 2012.
[4] P.-A. Absil and I. V. Oseledets. Low-rank retractions: a survey and new results. Computational Optimization and Applications, 62(1):5-29, 2015.
[5] C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning, pages 291-301. PMLR, 2019.
[6] M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120-1128. PMLR, 2016.
[7] S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[8] A. Ashok, N. Rhinehart, F. Beainy, and K. M. Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning. In International Conference on Learning Representations, 2018.
[9] B. Bah, H. Rauhut, U. Terstiege, and M. Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers, 2020.
[10] N. Bansal, X. Chen, and Z. Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
[11] P. L. Bartlett, D. J. Foster, and M. J. Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
[12] G. Ceruti and C. Lubich. An unconventional robust integrator for dynamical low-rank approximation. BIT Numerical Mathematics, 62(1):23-44, 2022.
[13] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, pages 854-863. PMLR, 2017.
[14] F. H. Clarke. Optimization and nonsmooth analysis. SIAM, 1990.
[15] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pages 1310-1320. PMLR, 2019.
[16] J. Ding, T. Bu, Z. Yu, T. Huang, and J. Liu. Snn-rat: Robustness-enhanced spiking neural network through regularized adversarial training. Advances in Neural Information Processing Systems, 35:24780-24793, 2022.
[17] R. Feng, K. Zheng, Y. Huang, D. Zhao, M. Jordan, and Z.-J. Zha. Rank diminishing in deep neural networks. arXiv:2206.06072, 2022.
[18] T. Galanti, Z. S. Siegel, A. Gupte, and T. Poggio. Sgd and weight decay provably induce a low-rank bias in neural networks, 2023.
[19] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2015.
[20] S. Gui, H. Wang, H. Yang, C. Yu, Z. Wang, and J. Liu. Model Compression with Adversarial Robustness: A Unified Optimization Framework. 2019.
[78] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
[79] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482. PMLR, 2019.