Robust Learning with Jacobian Regularization
Authors: Judy Hoffman
Daniel A. Roberts
Sho Yaida
Venue:  (2020)
Keywords: Supervised Representation Learning
Few-Shot Learning
Regularization
Adversarial Defense
Deep Learning

Abstract: Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of models against corruption of input data. In order to maximize stability, we analyze and develop a computationally efficient implementation of Jacobian regularization that increases classification margins of neural networks. The stabilizing effect of the Jacobian regularizer leads to significant improvements in robustness, as measured against both random and adversarial input perturbations, without severely degrading generalization properties on clean data.

Under review as a conference paper at ICLR 2020
ROBUST LEARNING WITH JACOBIAN REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of models against corruption of input data. In order to maximize stability, we analyze and develop a computationally efficient implementation of Jacobian regularization that increases classification margins of neural networks. The stabilizing effect of the Jacobian regularizer leads to significant improvements in robustness, measured against both random and adversarial input perturbations, without severely degrading generalization properties on clean data.
INTRODUCTION
Stability analysis lies at the heart of many scientific and engineering disciplines. In an unstable system, infinitesimal perturbations amplify and have substantial impacts on the performance of the system. It especially critical perform thorough stability analysis on complex engineered systems deployed in practice, or else what may seem like innocuous perturbations can lead to catastrophic consequences such as the Tacoma Narrows Bridge collapse (Amman et al.) [194]) and the Space Shuttle Challenger disaster (Feynman and Leighton 2001). As rule of thumb, well-engineered systems should be robust against any input shifts expected or unexpected.
Most models in machine learning are complex nonlinear systems and thus no exception to this rule. For instance, reliable model must withstand shifts from training data to unseen test data, bridging the so-called generalization gap. This problem is severe especially when training data are strongly biased with respect to test data, as in domain-adaptatin true underlying distribution is available, as in few-shot learning. Any instability in the system can further be exploited by adversaries to render trained models utterly useless (Szegedy et al. 2013
tasks, or when only sparse sampling of
Goodfellow al. 2014 Moosavi-Dezzooli et al. 2016 Paapemoo et al., (2066a] Kurakin al. 2016
1017// Gilmer et al.) 2018). is thus of utmost importance to ensure that models be stable against perturbations in the input space.
Madry et al. 2017 7 Carlini and Wagner) [2017]
Various regularization schemes have been proposed to improve the stability of models. For linear classifiers and support vector machines (Cortes and Vapnik, 1995), this goal is attained via 1,2 regularization which maximizes classification margins and reduces overfitting to the training data. This regularization technique has been widely used for neural networks as well and shown to promote generalization (Hinton, 1987 Krogh and Hertz 1992 Zhang et al.) 2018). However, it remains unclear whether or not I. regularization increases classification margins and stability of a network, especially for deep architectures with intertwining nonlinearity.
In this paper, we suggest ensuring robustness of nonlinear models via a Jacobian regularization scheme. We illustrate the intuition behind our regularization approach by visualizing the classification margins of a simple MNIST digit classifier Figure (see Appendix A for more). Decision cells of neural network, trained without regularization, are very rugged and can be unpredictably unstable (Figure Ia). On average, L regularization smooths out these rugged boundaries but does not necessarily increase the size of decision cells, i.e., does not increase classification margins (Figure Tb). In contrast, Jacobian regularization pushes decision boundaries farther away from each training data point, enlarging decision cells and reducing instability (Figure Ic).
The goal of the paper is to promote Jacobian regularization as a generic scheme for increasing robustness while also being agnostic the architecture, domain, or task to which it is applied. In

---

Under review as a conference paper at ICLR 2020
6
3
Without regularization
(b) With L regularization
With Jacobian regularization
Cross sections of decision cells in the input space. To make these sections for LeNet' models trained on the MNIST dataset, a test sample (black dot) and two-dimensional hyperplane C. 12884 passing through it are randomly chosen. Different colors indicate the different classes predicted by these models, transparency and contours are set by maximum of the softmax values, and the circle around the test sample signifies distance to the closest decision boundary in the plane. (a) Decision cells are rugged without regularization. (b) Training with L regularization leads to smoother decision cells, but does not necessarily ensure large cells. (c) Jacobian regularization pushes boundaries outwards and embiggens decision cells.
Figure 1:
support this, after presenting the Jacobian regularizer, we evaluate its effect both isolation as well as in combination with multiple existing approaches that intended promote robustness and generalization. Our intention to showcase the ease of use and complimentary nature of our proposed regularization. Domain experts in each field should be able quickly incorporate regularizer into their learning pipeline as simple way of improving the performance of their state-of-the-art system.
The rest of the paper is structured as follows. Section we motivate the usage Jacobian regularization and develop a computationally efficient algorithm for its implementation. Next, the effectiveness of this regularizer empirically studied in Section 3 regularlizers constrain the learning problem, we first verify that the introduction of our regularizer does not adversely affect learning the case when input data remain unperturbed. Robustness against both random and adversarial perturbations is then evaluated and shown to receive significant improvements from the Jacobian regularizer. We contrast our work with the literature Section and conclude in Section
2
METHOD
Here we introduce scheme for minimizing the norm of an input-output Jacobian matrix as technique for regularizing learning with stochastic gradient descent (SGD). We begin by formally defining the input-output Jacobian and then explain efficient algorithm for computing the Jacobian regularizer using standard machine learning frameworks.
2.1 STABILITY ANALYSTS AND INPUT-OUTPUT JACOBIAN
Let consider the set of classification functions, f. which take vectorized sensory signal, R. as input and outputs a score vector, z f(æ) E R where each element, Ze, is associated with likelihood that the input is from category, c this work, focus on learning this classification function as a neural network with model parameters 0, though our findings should generalize to any parameterized function. Our goal is to learn the model parameters that minimize the classification objective on the available training data while also being stable against perturbations in the input space so as to increase classification margins.
1 'Throughout the paper, the vector denotes the logit before applying softmax layer. The probabilistic *fo/T
output of softmax Pe relates to z. via Pe
Eg 2077 with temperature typically set unity.
2

---

Under review as a conference paper at ICLR 2020
The input-output Jacobian matrix naturally emerges in the stability analysis of the model predictions against input perturbations. Let us consider a small perturbation vector, e E R', of the same dimension as the input. For perturbed input ã e, the corresponding output values shift to
ofc O())) ic Dr.
+
Jen( 0
(1)
where in the second equality the function was Taylor-expanded with respect to the input perturbation and in the third equality the input-output Jacobian matrix,
Ofe
Jes(
(a)) =
ar,
was introduced. As the function ʃ is typically almost everywhere analytic, for sufficiently small perturbations the higher-order terms be neglected and the stability of the prediction is governed by the input-output Jacobian.
2.2 ROBUSTNESS THROUGH INPUT-OUTPUT JACOBIAN MINIMIZATION
is straightforward to see that the larger the components of the Jacobian are, the more unstable the model prediction with respect input perturbations. A natural way reduce this instability then is to decrease the magnitude for each component of the Jacobian matrix, which be realized by minimizing the square of the Frobenius norm of the input-output Jacobian
From Equation (T).
Σ Mex (2)) (w)) R.C
(3)
For linear models, this reduces exactly to L regularization that increases classification margins of these models. For nonlinear models, however, Jacobian regularization does not equate L regularization, and we expect these schemes to affect models differently. In particular, predictions made by models trained with the Jacobian regularization do not vary much as inputs get perturbed and hence decision cells enlarge average. This increase in stability granted by the Jacobian regularization visualized in Figure [ which depicts a cross section of the decision cells for the MNIST digit classification problem using nonlinear neural network (LeCun et al. 1998).
The Jacobian regularizer in Equation 33) can combined with any loss objective used for training parameterized models. Concretely, consider supervised learning problem modeled by neural network and optimized with SGD. At each iteration, a mini-batch consists of a set labeled examples, (x" nd " J.eg, and supervised loss function, Luppers is optimized possibly together with some other regularizer R(0)- such L regularizer Awn 0² over the function parameter space, by minimizing the following bare loss function
Liare (fx", 3 JaEB; 0)
Σ aEB
Couper [f(x)): R(0). ((()))
(4)
To integrate our Jacobian regularizer into training, one instead optimizes the following joint loss
AJR
C f (0) = Lhare (1x", 8²) JOEB; 0)
||(x>)))
(5)
B
aEB
where λJR is hyperparameter that determines the relative importance of the Jacobian regularizer. By minimizing this joint loss with sufficient training data and a properly chosen AIR. we expect models learn both correctly and robustly.
Minimizing the Frobenius norm will also reduce the L' norm, since these norms satisfy the incqualities [=]]|] < E.e Jes (x) & VIC [J())( We prefer to minimize the Frobenius nomm over the L' -noom because the ahility express the former as trace leads efficient algorithm [see Equations 6 through (8)
3

---

then be efficiently Under review as a conference paper at ICLR 2020
2.3 EFFICIENT APPROXIMATE ALGORITHM
In the previous section we have argued for minimizing the Frobenius norm of the input-output Jacobian improve robustness during learning. The main question that follows is how efficiently compute and implement this regularizer in such a way that its optimization can seamlessly be incorporated into any existing learning paradigm. Recently, (Sokol et al. (2017) also explored the idea of regularizing the Jacobian matrix during learning, but only provided an inefficient al algorithm requiring an increase in computational cost that scales linearly with the number of output classes, C, compared to the bare optimization problem (see explanation below). In practice, such an overhead will be prohibitively expensive for many large-scale learning problems, e.g. ImageNet classification has C' 1000 target classes (Deng et al. 2009). (Our scheme, in contrast, can be used for ImageNet: see Appendix H
Here, offer different solution that makes of random projections to efficiently approximate the Frobenius norm of the Jacobian [3] This only introduces a constant time overhead and can be made very small in practice. When considering such an approximate algorithm, one naively must trade off efficiency against accuracy for computing the Jacobian, which ultimately trades computation time for robustness. Prior work by Varga et al. (2017) briefly considers an approach based on random projection, but without providing any analysis on the quality of the Jacobian approximation. Here, we describe our algorithm, analyze theoretical convergence guarantees, and verify empirically that there is only negligible difference in model solution quality between training with the exact computation of the Jacobian as compared to training with the approximate algorithm, even when using a single random projection (see 2
Given that optimization is commonly gradient based, it essential to efficiently compute gradients of the joint loss in Equation (5) and in particular of the squared Frobenius norm of the Jacobian. First, we note that automatic differentiation systems implement a function that computes the derivative of a vector such with respect to any variables on which it depends, if the vector is first contracted with another fixed vector. To take advantage of this functionality, we rewrite the squared Frobienus norm
- - 1 (e x)
- (0)
(6)
dx
where constant orthonormal basis, {e}. of the C-dimensional output space was inserted in the second equality and the last equality follows from definition (2) and moving the constant vector inside the derivative. For each basis vector e, the quantity in the last parenthesis computed by differentiating the product, e 2, with respect to input parameters, æ. Recycling that computational graph, the derivative of the squared Frobenius norm with respect to the model parameters, 0, can be computed through backpropagation with any use of automatic differentiation. Sokolić al. (2017) essentially considers this exact computation, which requires backpropagating gradients through the model C' times to iterate over the C' orthonormal basis vectors {e}. Ultimately, this incurs computational overhead that scales linearly with the output dimension C.
Instead, we further rewrite Equation 66 in terms of the expectation of an unbiased estimator
[x) C Sc [|| 1
where the random vector o is drawn from the (C - dimeennsion unit sphere SC-1, Using this relationship, we can use samples of пproj random vectors t" to estimate the square of the norm as
Aproj
a T( 2)
(8)
Pproj
dx
which converges to the true value as On ). The derivation of Equation 7 and the calculation of its convergence make use of mandom-matrin techniques and are provided in Appendix B
-1/2,
Finally, we expect that the fluctuations of our estimator can be suppressed by cancellations within a mini-batch. With nearly independent and identically distributed samples in a mini bati-bach of size
II Appendix C we give an alternative method for computing gradients of the Jacobian regularizer by using an analytically derived formula.
4

---

Under review as a conference paper at ICLR 2020
'est Accuracy
Rynd
Test [|J
Rgnj
1.5
Pamm Exaci
98
10
15
10
15
Tteration
x10
Tteration
x10*
(a) Accuracy, full-training
(b) Robustness, full- full-training
Figure 2: Comparison of Approximate to Exact Jacobian Regularizer. The difference between the exact method (cyan) and the random projection method with pproj (blue) and Mproj = (red orange) is negligible both in terms of accuracy (a) and the norm of the input-output Jacobian (b) the test set for LeNet' models trained on MNIST with ÄJR = 0.01. Shading indicates the standard deviation estimated over distinct runs and dashed vertical lines signify the learning rate quenches.
Algorithm 1 Efficient computation of the approximate gradient of the Jacobian regularizer.
Inputs: mini-batch of [B] examples æ", model outputs z", and number of projections Пproj- Outputs: Square of the Frobenius norm of the Jacobian Jp and its gradient VeJF. JF =0
for i Perij do
{ro] ~ (0, b ([B), d'-mim tensor with each element sampled from a standard normal. Uniform sampling from the unit sphere for each >> Flatten for parallelism.
8a w0/l /
Filtt = Flatten (2a)) }); Unlat = Flatten((6"))
Ju (zz Vflat) 0x
TF += J1l (Mproj (B)
end for
VOJF 0Tr/00 00
return JF, VoJr
fact, as shown in Figure with mini-batch size of [ 100. single projection yields model performance that is nearly identical the exact method, with computational cost being reduced by orders of magnitude.
B
we expect the error our estimate be of order (Mproj (B)) -1/2,
The complete algorithm is presented in Algorithm 1
With straightforward implementation in PyTorch (Paszke al., (017)) and Pproj 1, we observed the computational cost of the training with the Jacobian regularization to be only x 1.3 times that of the standard SGD computation cost, while retaining all the practical benefits of the expensive exact method
EXPERIMENTS
In this section, we evaluate the effectiveness of Jacobian regularization on robustness. As all regularizers constrain the learning problem, begin by confirming that regularizer effectively reduces the value of the Frobenius norm the Jacobian while simultaneously maintaining or improving generalization regularization provides significant robustness against corruption of input data from both random and adversarial perturbations (Section 3.2). In the main text present results mostly with the MNIST dataset; the corresponding experiments for the CIFAR-10 (Kizhevssy and Hinton 2009) and ImageNet (Deng al., 2009) datasets are relegated Appendices and The following specifications apply throughout our experiments:
an unseen test We then present our core result, that Jacobian
Datasets:
The MNIST data consist of black-white images of hand-writted digits with 28-by-28 pixels, partitioned into 60,000 training and 10,000 test samples (LeC'un et al. 1998). We preprocess the data by subtracting the mean (0. 1307) and dividing by the variance (0.3081) 1) of the training data.
'The costs are measured on single NVIDIA GP100 for the LeNet' architecture on MNIST data. The computational efficiency depends datasets and model architectures: the largest we have observed is factor of increase in computational time for ResNet-18 CIFAR-10 (Appendix F). which is still of order one.

---

Under review as a conference paper at ICLR 2020
Table 1: Generalization on clean test data. LeNet' models learned with varying amounts of training samples per class are evaluted on MNIST test set. Jacobian regularizer substantially reduces the norm of the Jacobian while retaining test accuracy. Errors indicate 95% confidence intervals over distinct runs for full training and 15 for sub-sample training.
Test Accuracy (1)
|| (L)
Samples per class
Regularizer
1
3
10
30
All
All
No regularization
49.2 +: 1.9
67.0 + 1.7
83.3 + 0.7
90.4 = 0.5
98.9 + 0.1
32.9 3.3
L
49.9 99.9:22 2.1
68.1
1.9
84.3
91.2 0.5
99.2 0.1
4.6
Dropout
49.7
91.6 8.9 1.7
9.1
21.5 2.3
Jacobian
49.3 2.1
68.2
08.2119
84.5
91.3 0.4
9.00
11 0.1
All Combined
51.7 1.7=20 2.1
69.7 1.9
86.3 + 0.9
92.7 = 0.4
9.1
1.2 + 0.0
Implementation Details: For the MNIST dataset, we use the modernized version of LeNet-5 (LeCun) al. 1998), henceforth denoted LeNet' (see Appendix for full details). We optimize using SGD with momentum, 0.9, and our supervised loss equals the standard cross-entropy with one-hot targets. The model parameters are initialized at iteration = 0 by the Xavier method (Glorot and Bengio 2010) and the initial descent value is set to O. The hyperparameters for all models are chosen to match reference implementations: the T,2 regularization coefficient (weight decay) AWD 5 10-4 and the dropout rate set Pårop = 0.5. The Jacobian regularization coefficient ÄJR = 0.01. is chosen by optimizing for clean performance and robustness on the white noise perturbation. (See Appendix G for performance dependence on the coefficient ÅJR-)
3.1 EVALUATING GENERALIZATION
The main goal of supervised learning involves generalizing from training set to unseen test set. In dealing with such a distributional shift, overfitting to the training set and concomitant degradation in test performance is the central concern. For neural networks one of the most standard antidotes to this overfitting instability is L reguralization (Hinton 1987 Krogh and Hertz 1992. Zhang al. 2018). More recently, dropout regularization has been proposed as another way to eircumvent overfitting (Srivastava et al. 2014). Here we show how Jacobian regualarization can serve as yet another solution. This is also in line with the observed correlation between the input-output Jacohian and generalization performance (Novak et al.) 2018).
Generalizing within domain: We first verify that in the clean case, where the test set composed of unseen samples drawn from the same distribution as the training data, the Jacobian regularizer does not adversely affect classification accuracy. Table reports performance on the MNIST test set for the LeNet' model trained on either subsample all of the MNIST train set, indicated. When learning using all 60,000 training examples, the learning rate is initially set to niu = 0.1 with mini-batch size [ 100 and then decayed ten-fold after each 50,000 SGD iterations; each simulation is run for 1 150,000 SGD iterations in total. When learning using a small subsample of the full training set, training is carried out using SGD with full batch and a constant learning rate 7) 0.01, and the model performance is evaluated after 10,000 iterations. The main observation is that optimizing with the proposed Jacobian regularizer or the commonly used L and dropout regularizers does not change performance on clean data within domain test samples in any statistically significant way. Notably, when few samples are available during learning, performance improved with increased regularization in the form of jointly optimizing over all criteria. Finally, in the right most column of Table [ we confirm that the model trained with all data and regularized with the Jacobian minimization objective has an order of magnitude smaller Jacobian norm than models trained without Jacobian regularization. This indicates that while the model continues to make the same predictions on clean data, the margins around each prediction has increased as desired.
Generalizing to a new domain: regularization by evaluating an MNIST learned model data drawn from target domain distribution the USPS (Hull, 1994) test set. Here, models are trained on the MNIST data as above, and the USPS test dataset consists of 2007 white images of hand-written digits with
We test the limits of the generalization provided by Jacobian

---

Under review as a conference paper at ICLR 2020
Table Generalization on clean fest data from an unseen domain. LeNet' models learned with all MNIST training data are evaluated for accuracy on data from the novel input domain of USPS test set. Here, each regularizer, including Jacobian, increases accuracy over an unregularized model. In addition, the regularizers may be combined for the strongest generalization effects. Averages and 95% confidence intervals are estimated over 5 distinct runs.
No regularization
L
Dropout
Jacobian
All Combined
80.4 + 0.7
83.3 + 0.8 81.9 + 1.4 81.3 + 0.9
85.7 + 1.0
100
100
100
90
90
Dakes
80
80
80
Teu Accuracy
70
70
Accuracy
70
Al Jandeas
60
60
60
50
50
50
40
Test
40
40
30
30
30
20
20
Macerrial Tridking
Dioport: Repolatiiaion
10
Tanduan Begakeizzio
Jucobaaa Prpahorisiion
AT Jocelan
0
PGD Iterations
CW Fooling Distance
(a) White noisc
(b) PGD
(c) CW
Figure 3: Robustness against random and adversarial input perturbations. This key result il- lustrates that Jacobian regularization significantly increases the robustness of a learned model with LeNet' architecture trained on the MNIST dataset. (a) Considering robustness under white noise per- turbations, Jacobian minimization is the most effective regularizer. (b,c) Jacobian regularization alone outperforms an adversarial training defense (base models all include T,2 and dropout regularization). Shades indicate standard deviations estimated over distinct runs.
16-by-16 pixels; images upsampled to 28-by-28 pixels using bilinear interpolation and then preprocessed following the MNIST protocol stipulated above. Table 2 offers preliminary evidence that regularization, of each of the three forms studied, better generalizes to an unseen target domain. We again find that the regularizers may be combined to increase the generalization property of the model. Such regularization technique can be immediately combined with state-of-the-arr domain adaptation techniques to achieve further gains.
be used to learn source model which
3.2 EVALUATING UNDER DATA CORRUPTION
This section showeases the main robustness results of the Jacobian regularizer, highlighted in the case of both random and adversarial input perturbations.
Random Noise Corruption: The real world can differ from idealized experimental setups and input data can become corrupted by various natural causes such random noise and occlusion. Robust models should minimize the impact of such corruption. As one evaluation of stability to natural corruption, perturb each test input image to क [ crop where each component of the perturbation vector is drawn from the normal distribution with variance Gnoise as
E; NO, 22 2o
and the perturbed image is then clipped to fit into the range [0, 1] before preprocessing. As in the admatnnadaptation experiment above, models are trained on the clean MNIST training data and then tested on corrupted test data. Results in Figure 3al show that models trained with the Jacobian regularization is more robust against white noise than others. This is in line with - and indeed quantitatively validates the embiggening of decision cells as shown in Figure
Adversarial Perturbations: The world is not only imperfect but also possibly filled with evil agents that can deliberately attack models. Such adversaries seek small perturbation to each input example that changes the model predictions while also being imperceptible to humans. Obtaining the actual smallest perturbation is likely computationally intractable, but there exist many tractable approxima-
7

---

Under review as a conference paper at ICLR 2020
tions. The simplest attack is the white-box untargeted fast gradient sign method (FGSM) et al. 2014), which distorts the image as [x + Escoop with
(Goodfellow
Deuupe
G = EFGSM sign
Σ
Jerk
(10)
Oze
This attack aggregates nonzero components of the input-output Jacobian to a substantial effect by adding them up with consistent sign. In Figure 3b we consider stronger attack, projected gradient descent (PGD) method (Kurakin et al. 2016 Madry et al. 2017), which iterates the FGSM attack in Equation (TO) k times with fixed amplitude FFGSM
1/255 while also requiring each pixel value to be within 32/ 22/255 away from the original value. Even stronger is the Burlini-Wagne (CW)
attack (Carlimi and Wagner 2017) presented in Figure 3c
which yields more reliable estimates of
distance to the closest decision boundary (see Appendix Fj. Results unequivocally show that models
trained with the Jacobian regularization again more resilient than others. As a baseline defense benchmark, we implemented adversarial training, where the training image is corrupted through the FGSM attack with uniformly drawn amplitude FFCSM E [0, 0.01]; the Jacobian regularization can be combined with this defense mechanism to further improve the robustness Appendix [A additionally depicts decision cells in adversarial directions, further illustrating the stabilizing effect of the Jacobian regularizer.
4 RELATED WORK
1992) is the earliest attempt To our knowledge, double backpropagation (Drucker and LeCun 1991
to penalize large derivatives with respect to input data, in which (aCsuper ) is added to the loss
in order to reduce the generalization gap Different incarnations of a similar idea have appeared
in the following decades (Simard al. [992] Mitchell and Thrun T993, Aires al. 1999 Rifai
iet al. 2011 Gulrajani et al. 2017; Yoshida and Miyato, | Czarnecki et al.) [017
Jakubovitz
and Giryes 2018). Among them, Jacobian regularization as formulated herein was proposed by Gu
and Rigazio (2014) to combat against adversarial attacks. However, the authors did not implement due to a computational concern resolved by us in Section - and instead layer-wise Jacobians were penalized. Unfortunately, minimizing layer-wise Jacobians puts stronger constraint on model capacity than minimizing the input-output Jacobian. In fact, several authors subsequently claimed that the layer-wise regularization degrades test performance on clean data (Goodfellow et al. 2014 Papernot et al. 2016b) and results in marginal improvement of robustness (Carlini and Wagner 2017).
Very recently, full Jacobian regularization was implemented in Sokolić et al. (2117). but in an inefficient manner whose computational overhead for computing gradients scales linearly with the number of output classes C' compared to unregularized optimization, and thus they had to resort back the layer-wise approximation above for the task with a large number of output classes. This computational problem was ved by Varga et al. (201)) in exactly the same way as our approach (referred as spherical SpectReg in Varga et al. (2017)) As emphasized in Section 2 we performed more thorough theoretical and empirical convergence analysis and showed that there is practically no difference in model solution quality between the exact and random projection method terms of test accuracy and stability. Further, both of these two references deal only with the generalization property and did not fully explore strong distributional shifts and noise/adversarial defense. particular, we have visualized (Figure I) and quantitatively borne out (Section 3 the stabilizing effect of Jacobian regularization on classification margins of a nonlinear neural network.
'e also tried the defensive distillation technique of Papernot et al. (2016b). While the model trained with distillation temperature I' = 100 and attacked with 1' appeared robust against FGSM/PGD adversaries. it was fragile once attacked the numerical precision issue observed by Carlini and Wagner (2016).
100 and thus cannot he robust against white-box attacks. This in line with
"This approach was slightly generalized in Lyu al. 2015 the context of adversarial defense; see also
Ororbia II et al. (206)); Ross and Velez-Velez 2018
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this paper, we motivated Jacobian regularization as lask-agnostic method to improve stability of models against perturbations input data. Our method is simply implementable any open source automatic differentiation system, and additionally we have carefully shown that the approximate nature of the random projection is virtually negligible. Furthermore, we have shown that Jacobian regularization enlarges the size of decision cells and is practically effective in improving the gen- eralization property and robustness of the models, which is especially useful for defense against input-data corruption. We hope practitioners will combine our Jacobian regularization scheme with the arsenal of other tricks in machine learning and prove it useful in pushing the (decision) boundary of the field and ensuring stable deployment of models in everyday life.
REFERENCES
Othmar H Amman, Theodore von Kármán, and Glenn Woodruff. The failure of the Tacoma Narrows bridge. Report to the Federal Works Agency, 1941.
Richard P Feynman and Ralph Leighton. "What do you care what other people think?": further adventures of a curious character. WW Norton & Company, 2001.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 131.6199,,, 2013.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 141..572, 2014.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2582-2582, 2016.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning adversarial settings. In 2016 IEEE European Symposium on Security and Privacy ( (EuroS&P),, pages 372-387. IEEE, 2016a.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 1607.02550 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706iv:776..000 1706.06083, 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP), 2017.
Justin Gilmer, Ryan P Adams, Ian Goodfellow, David Andersen, and George E Dahl. Motivating the rules of the game for adversarial example research. arXiv preprint 1 2018.
Corinna Cortes and Vladimir Vapnik. Support-vecton networks. Machine learning, 20(3):273-297, 1995.
Geoffrey E Hinton. Learning translation invariant recognition in a massively parallel networks. In International Conference on Parallel Architectures and Languages Europe, pages 1-13. Springer, 1987.
Anders Krogh and John A Hertz. A simple weight decay improve generalization. In Advances in neural information processing systems, pages 950-957, 1992.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv: 12281, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) 36(11):2278-25) 1998.

---

Under review as a conference paper at ICLR 2020
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65(16):4265 000044266660.000 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 245-255. leee, 2009.
Dániel Varga, Adrián Csiszárik, and Zsolt Zombori. Gradient regularization improves accuracy of discriminative models. arXiv preprint arXiv:1712.0990 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vetito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Neural Information Processing Symposium, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249 256, 2010.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-195, 2014.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv: 080.0080000000.. 2018.
Jonathan Hull. database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell., 16(3)5500-554, 1994.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to ad perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582-597. IEEE, 2016b.
Nicholas Carlini and David Wagner. Defensive distillation not robust to adversarial examples. arXiv preprint arXiv: (6607003311, 2016.
Harris Drucker and Yann LeCun. Double backpropagation increasing generalization performance. In IJCNN991 -1-attlee International Joint Conference Neural Networks, volume 2, pages 145-150. IEEE, 1991.
Harris Drucker and Yann LeCun. Improving generalization performance using double backpropaga- tion. IEEE Transactions on Neural Networks, 3(6):991-99) 1992.
Chunchuan Lyu, Kaizhu Huang, and Hai-Ning Liang. A unified gradient regularization family for adversarial examples. In 2015 IEEE International Conference Data Mining, pages 301-309. IEEE, 2015.
Alexander G Ororbia II, C Lee Giles, and Daniel Kifer. Unifying adversarial training algorithms with ficxible deep data gradient regularization. arXiv preprint arXiv:1601.07213 2016.
Andrew Slavin Ross and Finale Doshi Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In Thhrty-Secome AAAI Conference on Artificial Intelligence, 2018.
Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent prop-a formalism for spec- ifying selected invariances in an adaptive network. In Advances in neural information processing systems, pages 895-903, 1992.
Tom M Mitchell and Sebastian B Thrun. Explanation- basee neural network learning for robot control. In Advances in neural information processing systems, pages 287-294, 1993.
10

---

Under review as a conference paper at ICLR 2020
Filipe Aires, Michel Schmitt, Alain Chedin, and Noëlle Scott. The "weight smoothing" regularization of MLP for Jacobian stabilization. IEEE Transactions on Neural Networks, 10(6):1502-1510, 1999.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pages 833 333-840. Omnipress, 2011.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 2017.
Wojeiech M Czamecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. In Advances in Neural Information Processing Systems, pages 4278 4287, 2017.
Daniel Jakuhovitz and Raja Giryes. Improving DNN robustness adversarial attacks using Jacobian regularization. In Proceedings of the European Conference Computer Vision (ECCV), pages 514-529, 2018.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068 2014.
Benoît Collins and Piotr Sniady. Integration with respect to the Haar measure on unitary, orthogonal and symplectic group. Communications in Mathematical Physics, 264(3):773-795, 2006.
Benoît Collins and Sho Matsumoto. On some properties of orthogonal Weingarten functions. Journal of Mathematical Physics, 50(1 1): 13516, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-2522 2015.

---

Under review as a conference paper at ICLR 2020
Figure S1: Cross sections of decision cells in the input space for LeNet' models trained on the MNIST dataset along random hyperplanes. Figure specifications are same in Figure I (Left) No regularization. (Middle) L regularization with \WD = 0.0005. (Right) Jacobian regularization with ÄJR 0.01.
A GALLERY OF DECISION CELLS
We show in Figure ST plots similar to the ones shown in Figure i the main text, but with different seeds for training models and around different test data points. Additionally, shown in Figure S2 similar plots but with different scheme for hyperplane slicing, based adversarial directions. Interestingly, the adversarial examples constructed with unprotected model do not fool the model trained with Jacobian regularization.
12

---

Under review as a conference paper at ICLR 2020
D
Figure S2: Cruss sections of decision cells in the input space for LeNet' models trained on the MNIST dataset along adversarial hyperplanes. Namely, given test sample (black dot), the hyperplane through it is spanned by two adversarial examples identified through FGSM, one for the model trained with I.2 regularization ÅwD = 0.0005 and dropout rate 0.5 but defense (dark-grey dot; left figure) and the other for the model with the same standard regularization methods plus Jacobian regularization ÄJR 0.01 and adversarial training (white-grey dot; right figure).
13

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL DETAILS FOR EFFICIENT ALGORITHM
Let us denote by Ro--go- [F (i) the average of the arbitrary function over C-dimensional vectors e sampled uniformly from the unit sphere SC-1, As in Algorithm I such unit vector can be sampled by first sampling each component Ve from the standard normal distribution N 1) and then normalizing it as è w/|/o/1. In our derivation, the following formula proves useful:
se [F (0) - au(O))(Oce) [
(11)
where e is an arbitrary C-dimensional unit vector and f dc( [.. is an integral over orthogonal matrices O over the Haar measure with normalization f dd((O) [1] 1.
First, let us derive Equation
11]. the square of the Frobenius norm can then be Using Equation
written
|
Tr (JJz),
I (ulom)
faulo) Σ eO.
(c}
Σ (e)
Eowso-- [oJJ
= CE [.JJ
(12)
where in the second line we insert the identity matrix in the form I = OTO and make use of the cyclicity of the trace; in the third line we rewrite the trace as a sum over an orthonormal basis {e} of the C-dimensional output space; in the forth line Equation (II) was used; and in the last line note that the expectation no longer depends on the basis vectors e and perform the trivial sum. This completes the derivation of Equation 77
Next, let us compute the variance of our estimator. Using tricks as before, but in reverse order, yields
var (C (CDJJTOT) C² Eg-so-
[BJJE |1
(13)
=c² fc dull) [eOJJTOT "ee.io00"=eeeee""ee"e"" -|\(\))
In this form, we use the following formula (Collins and Sniady 2006: to evaluate the first term M
Collins and Matsumoto
2009
{ du(() Onres O O Osee
O cEC4
(14)
0g02
C+1
(eecesezeer descodera, + dc, de, desc, deserdeacs + Seses dezes de, Seser)
C(C-1)(C ()(C 2)
1
bercz degcs 8 Segce Ses Sewer + Sas Segcs desco 8eyen
C(C 1) + +2)
+6 es Sese, Se, e, deses Seyey + Se Se After the dust settles with various cancellations, the expression for the variance simplifies to
2C
var (C (CGJJTOT) (C+2) Tr (JJJJJJ")) (C [ |
(15)
We can strengthen our claim by using the relation ||BB| I ||A| with A and B ST which yields Tr JJTT < [[]()) and in turn bounds the variance divided by the square of the mean as
var (C (COJJTT)
22 (CT1)
(16)
[mean (C (CJJTTTT)?
We thank Nick Hunter-Jones for providing us with the inelegant but concretely actionable form of this integral.
14

---

Under review as a conference paper at ICLR 2020
The right-hand side is independent of J and thus independent of the details of model architecture and particular data set considered.
In the end, the relative error of the random- candom-projection estimate for [|] with Пproj random vectors will diminish as some order-one number divided by "proj [ over mini-batch of samples of size [B3, we expect the relative enror of the Jacobian regularization term to be additionally suppressed by 1/ 1/B
-1/// In addition, upon averaging
Finally, we speculate that in the large-C' limit - possibly relevant for large-class datasets such as the ImageNet (Deng al. 2009) there might be additional structure in the Jacobian traces (e.g. the central-limit concentration) that leads to further suppression of the variance.
C CYCLOPROPAGATION FOR JACOBIAN REGULARIZATION
is also possible to derive a closed-fom expression for the derivative of the Jacobian regularizer, thus bypassing any need for random projections while maintaining computational efficiency. The expression is here derived for multilayer perceptron, though we expect similar computations may be done for other models of interest. We provide full details in case one may find practically useful to implement explicitly in any open-source packages or generalize it to other models.
Let us denote the input x, and the output 2, (L) where (identifying { { {1..... I and {c) = (ix. = (1,...,C) C})
(0)
2io
Fip
(17)
:10
Σ
w inite
(6)
2((-1)
+ bb( for e= 1=.....;
(18)
Zie
(
Fice for l I....Ih. L.
(19)
Defining the layer-wise Jacobian
dz
((()
d (1) (2)
1 (no summation),
(00)
az,
(t-1)
the total input-output Jacobian is given by
Js.. Ozig
J(L-))
(21)
in,ão
The Jacobian regularizer of interest is defined as (up to the magnitude coefficient AJR)
1 S 1
RJR
E
(Jin.
Tr [J']]
(22)
So.A1
Its derivatives with respect to biases and weights are denoted as
B
ÖRJR (8)
(23)
ab
W(()
ÖRJR
(24)
Je.Je-1
Some straightforward algebra then yields
(00)
B(++1)
1/00
d'(((tt1)) j(e+1)
10 o' d((2!
(((0) ...)()). JT. ((0)... ((41)) ((4)))
o'
jede
de
(25)
15

---

Under review as a conference paper at ICLR 2020
and
W Jede-1
B(0)((11) ((-1)) Je Je-1
+ o (2/)) [(e-_t)... 1(0) .T _ ... ..C)......) }((+1)
(26)
1e-13
where we have set
B ( = j((+1) = 0.
(27)
Algorithmically, we can iterate the following steps for = I,, I. - , 1:
1. Compute
[.. _ j(L) ... f((+1)
-(L)
(28)
it-1ze
2. Compute
aR 06/e
10(0= =
d((Biet1)
B(++1)
(((+1)
d'(3fe))
(()(0) +o" (310)
Σ
Jeiee- Je-15e
200
(29)
Je-1
3. Compute
0R
W(O
B(() (f-1) of (2)) D2mie
(30)
(8)
jr.Ji-1
de Jc-1
je-j-1
Note that the layer-wise Jacobians, J('ss are calculated within the standard backpropagation algorithm. The core of the algorithm is in the computation of 2 13-115 in Equation 288 It is
obtained by first backpropagating from - to 1, then forwardpropagating from 1 to L, and finally backpropagating from L 1. It thus makes the cycle around É, hence the name cyclopropagation.
D DETAILS FOR MODEL ARCHITECTURES
In order to describe architectures of convolutional neural networks in detail, let us associate tuple [F, Cin + Cout, S, P; M] to a convolutional layer with filter width F, number of in-channels Cin and out-channels Cout, stride S, and padding P, followed by nonlinear activations and then max-pooling layer of width M (note that M = 1 corresponds to no pooling). Let us also associate a pair [N + Nout] a fully-connected layer passing Nin inputs into Nout units with activations and possibly dropout.
With these notations, our LeNet' model used for the MNIST experiments consists of (28, 28, 1) input followed by convolutional layer with [5, 1 + 6, 1, 1,2; 2] another one with [5, 6 - 16, 1, 0, 2], folle-connected layer with [2100- - 120] and dropout rate Pdrop. another fully-connected layer with 1100 - 84) and dropout rate Pärop. and finally fully-connected layer with [84 - 10). yielding 10-dimensional output logits. For our nonlinear activations, we use the hyperbolic tangent.
For the CIFAR-10 10 dataset, we use the model architecture specified in the paper on defensive dis- tillation (Papernot et al. (0166)), abbreviated as DDNet. Specifically, the model consists of (32, 32, 3) input followed by convolutional layers with [3. 3 - 64, 1, 0: 1J. [3, 64 + 64, 1, 0; 2] [3, 64 + 128, 0; 1]. and [3, 128 + 128, 1, 0; 2), and then fully-connected layers with [3200 - 256] and dropout rate Párop- with [256 256] and dropout rate Parops and with [256 10), again yielding 10-dimensional output logits. All activations are rectified linear units.
In addition, we experiment with version of ResNet-18 18 (He et al. 2016) modified for the b2-by-32 input size of CIFAR- 10 and shown to achieve strong performance on clean image recognition For the standard Py Torch initialization of the parameters. Data preproceessing
this architecture, and optimization hyperparameters for both architectures are specified in the next section.
For ImageNet experiments,
the standard ResNet-18 model available within PyTorch (torchvision.modll ressee) together with standard weight initialization.
Note that there is typically no dropout regularization in the ResNet models but still examine the effect of 2 regularization in addition to Jacobian regularization.
*For f = 1, the part J((-1) ... (()) is vacuous. Similarly, for f= L, the part J(M) ... ((+1) is vacuous. Model available https: / http://tttt..ccoo http::///ggtttbb...o/////ug/aag/ http:///occ.cccoooa//////tooorca
16

---

Under review as a conference paper at ICLR 2020
Table 3: Generalization on clean test data. DDNet models learned with varying amounts of training samples per class are evaluated on CIFAR-10 test set. Jacobian regularizer substantially reduces the norm of the Jacobian. Errors indicate 95% confidence intervals over 5 distinct runs for full training and 15 for sub-san sub-sample training.
Test Accuracy (1)
|| (L)
Samples per class
Regularizer
3
10
30
All
All
No regularization
12.9 + 0.7
15.5 + 0.7
20.5
1.3
26.6 + 1.0
76.8 + 0.4
115.1 + 1.8
L
13.9 = 1.0
14.6 1.1
20.5
1.0
26.6
06.61.2 1.2
77.8 0.2
29.4
Dropout
12.9
1,4
17.8:00 0.6
1.0
0.5
$0.7
184.2
Jacobian
14.9 + 1.0
18.3 +
23.7 0.8
30.0
75.4
4.0 0.0
All Combined
15.0 = 1.1
19.6
0.6
33.4 0.6
78.6
5.2
E RESULTS FOR CIFAR-10
Following specifications apply throughout this section for CIFAR-10 experiments with DDNet and ResNet-18 model architectures (see Appendix D).
Datasets: the CIFAR-10 dataset consists of color images of objects divided into ten categories- - with 32-by-32 pixels in each of 3 color channels, each pixel ranging in [0, 1. partitioned into 50,000 training and 10,000 test samples (Krizhevsky and Hinton 2009)
The images are preprocessed by uniformly subtracting 0.5 and multiplying by 2 so that each pixel ranges in 1].
Optimization: essentially same as for the LeNet' on MNIST, except the initial learning rate for full training. Namely, model parameters 0 are initialized iteration / by the Xavier method (Glorot and Bengio 2010) for DDNet and standard PyTorch initialization for ResNet-18, along with the zero initial velocity v(t 0) 0. They evolve under the SGD dynamics with momentum = 0.9, and for the supervised loss we use cross cross-entropy with one-hot targets. For training with the full training set, mini-batch size is set as [B] 100, and the learning rate 1) is initially set to and in both cases quenched ten-fold after each 50,000 SGD iterations; each simulation run for 150,000 SGD iterations in total. For few-shot learning, training is carried out using full-batch SGD with constant learning rate 7j = 0.01, and model performance evaluated after 10,000 iterations.
0.01 for the DDNet and = 0.1 for the ResNet-18
Hyperparameters: the same values are inherited from the experiments for LeNet' on the MNIST and no tuning performed. Namely, the weight decay coefficient ÅWD 5-10-4 the dropout rate Pärop = 0.5; the Jacobian regularization coefficient λJR = 0.01; and adversarial training with uniformly drawn FGSM amplitude SFGSM E [0, 0.01].
The results relevant for generalization properties are shown in Table S3
One difference from the MNIST counterparts in the main text is that dropout improves test accuracy more than L regularization. Meanwhile, for both setups the order of stability measured by ||]||p on the test set more less stays the same. Most importantly, turning on the Jacobian regularizer improves the stability by orders of magnitude, and combining it with other regularizers do not compromise this effect.
The results relevant for robustness against input-data corruption are plotted in Figures and S4 success of the Jacobian regularizer is retained for the white-noise and CW adversarial attack. For the PGD attack results are mixed at high degradation level when Jacobian regularization is combined with adversarial training. This might be an artifact stemming from the simplicity of the PGD search algorithm, which overestimates the shortest distance to adversarial examples in comparison the CW attack (see Appendix(FF F). combined with Jacobian regularization effect on simplifying the loss landscape with respect the input space that the attack methods explore.
The
17

---

Under review as a conference paper at ICLR 2020
100
100
100
90
90
Thst Accuracy
80
Fepelccatioon
800
Test Accuracy
80
Tuntitas Degidattation
Al' luatlin
60
60
60
40
Test
40
40
30
30
10
10
0
0.5
5
10
0,5
PGD Iteratione
CW Fooling Distauice
(a) White noise
(b) PGD
(c) CW
Figure S3: Robustness against random and adversarial input perturbations for DDNet models
trained on the CIFAR-10 dataset. Shades indicate standard deviations estimated over 5 distinet runs. (a) Comparison of regularization methods for robustness to white noise perturbations. (b,c) Comparison of different defense methods against adversarial attacks (all models here equipped with L and dropout regularization).
100
100
100
lemiiis Sondie Temeecastio
90
Dei Jackkas deppaariiiiis
90
TTet Acouricy
80
804
Tel Accunicy
30
70
Juntbes Topplatealie
A1's Sectlie
60
60
60
40
Ted
40
40
30
10
0.5
10
0.5 CW Fooling Distaunce
PGD Iteratione
(a) White noise
(b) PGD
(c) CW
Figure S4: Robustness against random and adversarial input perturbations for ResNet-18 mod- els trained on the CIFAR-10 dataset. Shades indicate standard deviations estimated over distinct runs. (a) Comparison of regularization methods for robustness to white noise perturbations. (b,c) Comparison of different defense methods against adversarial attacks (all models here equipped with L2 regularization but not dropout: see Appendix D
18

---

Under review as a conference paper at ICLR 2020
100
90
90-
7032
90
POSME
Arc
70
Toc: Accurncy
70-
70
50
50
40
20-
20
0
Foeling Distancce
Fooling Discauce
Fooling Distance
(a) Undefended; MNIST;
(h) Undefended; CIFAR-10;
(c) Undefended; CIFAR-10;
LeNet'
DDNet
ResNet-18
100
90
90
PC2
PCh
30
Itst Accuracy
Test Accuracy
Accuracy
70
80
60
40
40
40
30
20
10
10
0,5
0,5
Fooling Distance
Fooling Distance
Fooling Distance
(d) LeNet'
Defended; MNIST;
(e) Defended; CIFAR-10;
(f) Defended; CIFAR-10;
DDNet
ResNet-18
Figure S5: Effects on test accuracy incurred by various modes of attacks. (a.d) LeNer' on
MNIST, (b,e) DDNet on CIFAR-10 10, and (c,f) ResNet-18 on CIFAR- trained (a.b,c) without defense and (d.e.f) with defense Jacobian regularization magnitude ÄJR 0.01 and adversarial training with FFGSM E [0, 0.01] - all also include L regularization ÅWD = 0.0005 and (except ResNet-18) dropout rate 0.5.
F WHITE NOISE VS. FGSM vs. PGD vs. CW
In Figure SS] we compare the effects of various input perturbations on changing model's decision. For each attack method, fooling L distance in the original input space before preprocessing is measured between the original image and the fooling image as follows (for all attacks, cropping is performed to put pixels in the range [0, 1] in the orignal space): (i) for the white noise attack, random direction in the input space is chosen and the magnitude
the noise is cranked up until the model yields wrong prediction; (ii) for the FGSM attack, the gradient is computed at clean sample and then the magnitude FFGSM is cranked up until the model is fooled; (iii) for the PGD attack, the attack step with FGGM 1/255 is iterated until the model is fooled [as is customary for PGD and described in the main text, there is saturation constraint that demands each pixel value to be within 32/255 (MNIST) and 16/255 (CIFAR-1O) away from the original clean value]; and (iv) the CW attack halts when fooling is deemed successful. Here, for the CW attack (see Carlini and Wagner 2017) for details of the algorithm) the Adam optimizer on the logits loss (their fo) used with the learning rate 0.005, and the initial value of the conjugate variable, c, is set to be 0.01 and binary-searched for iterations. For each model and attack method, the shortest distance evaluated for 1,000 test samples, and the test error (= 100% test accuracy) at a given distance indicates the amount of test examples misclassified with the fooling distance below that given distance.
Below, we highlight various notable features.
The most important highlight is that, in terms of effectiveness of attacks, CW > PGD > FGSM > white noise, duly respecting the complexity of the search methods for finding adversarial examples. Compared to CW attack, the simple methods such FGSM and PGD attacks could sometime yield erroneous picture for the geometry of the decision cells, especially regarding the closest decision boundary.
19

---

Under review as a conference paper at ICLR 2020
The kink for PGD attack in Figure S5d is due to imposing saturation constraint that demands each pixel value to be within 32/ away from the original clean value. We think that this constraint is unnatural, and impose it here only because it is eustomary.
While the CW attack fools almost all the examples for LeNet' on MNIST and DDNet on CIFAR-10, it fails to fool some examples for ResNer-18 on CIFAR-10 (and later on ImageNet: see Section H) beyond some distance. We have not carefully tuned the hyperpa- rameters for CW attacks to resolve this issue this paper.
G DEPENDENCE ON JACOBIAN REGULARIZATION MAGNITUDE
In this appendix, we consider the dependence of our robustness measures on the Jacobian regular- ization magnitude, AJR. These experiments shown in Figure S6 Cranking up the magnitude of Jacobian regularization, ÅJR. generally increases the robustness of the model, with varying degree of degradation in performance on clean samples. Typically, seeing much degradation. This means that in practice modelers using Jacobian regularization can determine the appropriate tradeoff between clean accuracy and robustness input perturbations for their particular use case. If some expectation for the amount of noises the model might encounter is available, this can very naturally inform the choice of the hyperparameter AJR-
can double the fooling distance without
H RESULTS FOR IMAGENET
ImageNet (Deng al. 2009) large-scale image dataset. We use the ILSVRC challenge dataset (Russakovsky et al., 2015), which contains images each with corresponding label classified into one of thousand object categories. Models are trained on the training set and performance is reported on the validation set. Data are preprocessed through subtracting the mean [0.485, 0.456, 0.406] and dividing by the standard deviation, = [0.229, 0.224, 0.225), and at training time, this preprocessing is further followed by random resize crop to 224-by-224 and random horizontal flip.
ResNet- 18 (see Appendix D) is then trained the ImageNet dataset through SGD with mini-batch size [ = 256, momentum o = 0.9, weight decay AWD = 0.0001, and initial learning rate Tuu = 0.1, quenched ten-fold every 30 epoch, and evaluate the model for robusness the end of 100 epochs. Our supervised loss equals the standard cross-entropy with one-hot targets, augmented with the Jacobian regularizer with ÄJR 0.0001, 0.0003, and 0.001.
As is customary, the PGD attack iterates FGSM with FFGSM 1/255 and has a saturation constraint that demands each pixel is within 16/255 of its original value; the CW attack hyperparameter is same as before and was not fine-tuned; [0, 1]- cropping performed usual, but if preprocessing were performed with RGB-uniform mean shift 0.4490 and standard deviation division 0.2260. The Jacobian regularizer again confers robustness to the model, especially against adversarial attacks. Surprisingly, there is no visible improvement in regard to white-noise perturbations. We hypothesize that this is because the model is already strong against such perturbations even without the Jacobian regularizer, but remains investigated further.
Preliminary results are reported in Figure S7
20

---

Under review as a conference paper at ICLR 2020
100
100
80
80
80
Thll Acuurey
Tost Accunacy
70
Tel Accuracy
60
60
60
40
40
40
20
SIRES Z0E1
10
1.31
0.5
10
20
30
2 CW Fooling Distanoe
PCD Iterations
(a) White: LeNet' on MNIST
(b) PGD: LeNet' on MNIST
(c) CW: LeNet' on MNIST
100
100
90
90
so
aog
80
Test Accuncy
Tel Accuracy
40
60
So
Tes
40
40
30
30
30
10
10
0.5
10
0.5
Chabe
PGD Iteratione
CW Fooling Dissauce
(d) White; DDNet CIFAR-10
(e) PGD; DDNet on CIFAR-10
(1) CW: DDNet CIFAR-10
100
100
100
Test Accuracy
80
70
Test Accurncy
802
Accuracy
30
70
60
60
60
50
50
40
40
30
30
30
10:
10
10
0,5 Ouie
0,5
PCD Iterations
CW Fooling Distance
(g) White; R8 CIFAR-10 (h) PGD; ResNet-18 on CIFAR-10 (i) CW; ResNet-18 CIFAR-10
Figure 56: Dependence of robustness on the Jacobian regularization magnitude AJR- Accuracy under corruption of input test data are evaluated for various models [base models all include L (AwD = 0.0005) regularization and, except for ResNet- 18, dropout (rate 0.5) regularization). Shades indicate standard deviations estimated over distinct runs.
100
100
90
90
90
BRCS
1120
Test Accuriyy
700
Imit Accurscy
70
Test Accurscy
70
601
60
60
30
30
40
40
20
20
20
10
0
PGD Iterations
CW Fooling Distance
(a) White; 18 on ImageNet (b) PGD; ResNet-18 18 on ImageNet
(c) CW; ResNet-18 on ImageNet
Figure S7: Dependence of robustness on the Jacobian regularization magnitude ÄJR for Ima- geNet. Accuracy under corruption of input test data evaluated for Re trained on ImageNet [base models include L (AwD = 0.0001)] for a single run. For CW attack in (c), we used 10,000 test examples (rather than 1,000 used for other figures) to compensate for the lack of multiple runs.
21


Reviews:
[{"review_id": "SygfkBvaYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an efficient method to (differentiably) estimate input-output Jacobian. The method is useful for Jacobian regularization. The regularization improves robustness and generalization of networks.\n\nI tend to vote for rejection. There are two concerns. 1) This paper needs to demonstrate the effectiveness of the input-output Jacobian regularization over the input gradients regularization. 2) It is doubtful whether the regularizer provides the same benefits mentioned in Experiment 3.1 for other datasets than MNIST.\n\nMajor comments:\n1) This paper needs to demonstrate applications that the regularization of input-output Jacobian is more beneficial than that of input gradients. Input gradients regularization has repeatedly appeared in the literature, as the paper mentions. For example, [1] regularized input gradients to improve the robustness against adversarial examples. The input gradients regularization is computationally more efficient than Varga et al. (2017), with which the submitted paper compares the proposed method. If input gradients regularization is sufficient, it limits the impact of the submitted paper. It is strongly encouraged to demonstrate when and why the input-output Jacobian regularization is preferable.\n2)  Experimental results on CIFAR10 and ImageNet show accuracy degradation on clean test data. It is questionable whether we can reach the same conclusion with the experiments 3.1 on those datasets.\n\n[1] Andrew Slavin Ross and Finale Doshi-Velez. \"Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients.\" AAAI 2018\n\nUpdate =====\n\nThank you for the authors' response. I agree that the regularization of the input-output Jacobian is potentially superior to double back-prop. However, as authors are aware of, it needs to be validated experimentally. I think this paper should not leave this as a future work, and hence keep the review score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sken2lJnYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nStability is one of the important aspects of machine learning.  This paper views Jacobian regularization as a scheme to improve the stability, and studies the behavior of Jacobian regularization under random input perturbations, adversarial input perturbations, train/test distribution shift, and simply as a regularization tool for the classical setting without any distribution shifts nor perturbations.  There are already several related works that propose to use Jacobian regularization, but previous works didn’t have an efficient algorithm and also did not have theoretical convergence guarantee.  This paper offers a solution that efficiently approximate the Frobenius norm of the Jacobian and also show the optimal convergence rate for the proposed method.  Various experiments show that the behavior of Jacobian regularization and show that it is robust.\n\nReasons for the decision:\n\nPositives: The contribution of the paper seems to be two-fold:  First a theoretically guaranteed and efficient method for Jacobian regularizer, and second, intensive experiments to show the robustness of the Jacobian regularizer.  Each of these points seem to have important contributions for the field.\n\nNegatives:  An issue might be that the latter contribution seems to be orthogonal to the former since there are no experiments comparing with previous methods mentioned in the paper, and it gives the impression that there are two separate stories in one paper.  For instance, there are no experiments comparing computational time between Sokolic et al. (2017). On the other hand, there are only regularization methods (that are not necessarily designed to be robust) used as baselines in the experiments to show robustness, instead of algorithms that are designed to be robust, e.g., domain adaptation methods for Table 2.  It would make the paper stronger to combine these two lines of contributions into a single story.  For example, it might be better to emphasize more that experiments such as Figure S7 was previously not possible due to inefficient implementation.\n\nMinor comment:  It would make the paper stronger to include some of the main related works in the Introduction section.\n\n\nAfter author response:\nThank you for reading my review and answering the questions.  Although I still feel the same for my score (6), in my opinion, the same issues exist for this paper, and the paper can be made stronger on those points.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Byxq0ySk9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe main contribution of this paper is that it proposed an estimator of Jacobian regularization term for neural networks to reduce the computational cost reduced by orders of magnitude, and the estimator is mathematically proved unbiased. In details, the time consumed for the application of Jacobian regularizer and the unbiasedness of the proposed estimator are proved mathematically. Then the author experimentally demonstrated that the proposed regularization term retains all the practical benefits of the exact method but with a low computation cost. Quantitative experiments are provided to illustrate that the proposed Jacobian regularizer does not adversely affect the model, can be used simultaneously with other regularizers and effectively improve the model's robustness against random and adversarial input perturbations.\n\nIn general, this paper was well organized and it is also great that an efficient approximation of the Jacobian regularizer can be derived. However, the paper was written in a quite misleading or over-claimed way. The major comments are as follows:\n\n(1)\tIt is not new to use Jacobian regularizer for improving the robustness learning. Such idea has been elaborated in [1] (though it was cited in the paper). The main contribution is, the paper proposed an efficient approximated way to the exact Jacobian term. All the benefits of robust learning are rooted in the Jacobian regularization. \n(2)\tIn light of point (1), the title of this paper was first quite misleading. Robust learning with Jacobian was not proposed by the paper, so the title makes no sense.  Instead, efficient approximation should be emphasized.\n(3)\tMost of the advantages shown in the quantitative experiments are the benefits of Jacobian regularization! The authors should focus on the approximating algorithm rather than the merits of Jacobian regularization which has been discussed in [1]. In another word, it is better to add more comparison based on running time so as to illustrate the significant performance between the proposed regularizer and the exact one. \n(4)\tIn section 3, it may be interesting to see more comparison results if two regularizers were combined, e.g., (L2+Dropout, L2+Jacobianor Dropout+Jacobian). \n\n[1] Varga, Dániel, Adrián Csiszárik, and Zsolt Zombori. \"Gradient regularization improves accuracy of discriminative models.\" arXiv preprint arXiv:1712.09936 (2017).\n\n\n===============\nI have read the authors' response. I would have to say that, though I enjoy reading the paper and the other parts of this paper are very good,  the paper's contribution/novelty may be limited because Jacobian regularization has been earlier thoroughly discussed in [1] and even another recent  paper published in ECCV 2018. \n\nI am afraid that I  would firmly keep my original rating. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers have reviewed this submission and scored it as 6/3/3. After rebuttal, the reviewers remained unconvinced. The main criticisms concerns the Jacobian  regularizaton [1] being known which makes the contributions of this submission  look diluted. Additionally, there were concerns over results (degradation) on CIFAR10 and ImageNet and other minor issues.
For these reasons, this paper cannot be accepted by ICLR2020.
Decision: Reject