Robust Neural ODEs via Contractivity-promoting Regularization
Authors: Liang Xu
Muhammad Zakwan
Giancarlo Ferrari-Trecate
Venue: Submitted to ICLR 2023 (2023)
Keywords: contraction theory
neural ODEs
robustness
adversarial attacks
convolutional neural networks

Abstract: Neural networks can be fragile to input noise and adversarial attacks. In this work, we consider Neural Ordinary Differential Equations (NODEs) – a family of continuous-depth neural networks represented by dynamical systems - and propose to use contraction theory to improve their robustness. A dynamical system is contractive if two trajectories starting from different initial conditions converge to each other exponentially fast. Contractive NODEs can enjoy increased robustness as slight perturbations of the features do not cause a significant change in the output. Contractivity can be induced during training by using a regularization term involving the Jacobian of the system dynamics. To reduce the computational burden, we show that it can also be promoted using carefully selected weight regularization terms for a class of NODEs with slope-restricted activation functions, including convolutional networks commonly used in image classification. The performance of the proposed regularizers is illustrated through benchmark image classification tasks on MNIST and FashionMNIST datasets, where images are corrupted by different kinds of noise and attacks.

Under review as a conference paper at ICLR 2023
ROBUST NEURAL ODES PROMOTING REGULARIZATION
VIA CONTRACTIVITY-
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks can be fragile to input noise and adversarial attacks. In this work, we consider Neural Ordinary Differential Equations (NODEs) family of continuous-depth neural networks represented by dynamical systems and pro- pose to use contraction theory improve their robustness. dynamical system is contractive if two trajectories starting from different initial conditions converge to each other exponentially fast. Contractive NODEs can enjoy increased robust- ness as slight perturbations of the features do not cause significant change in the output. Contractivity can be induced during training by using regularization term involving the Jacobian of the system dynamies. To reduce the computational burden, we show that it can also be promoted using carefully selected weight reg- ularization terms for a class of NODEs with slope-restricted activation functions, including convolutional networks commonly used in image classification. The performance of the proposed regularizers is illustrated through benchmark im- age classification tasks on MNIST and FashionMNIST datasets, where images are corrupted by different kinds of noise and attacks.
INTRODUCTION
Neural networks (NNs) have demonstrated outstanding performance in image classification, natural language processing, and speech recognition tasks. However, they can be sensitive to input noise or meticulously crafted adversarial attacks (Xu et al., 2020 Carlini & Wagner, 2017; AAthalye al.] 2018 (Szeedy et al. 2013). The customary remedies are either heuristic, such as feature obfusca- tion (Miller al., (020), adversarial training (Goodfellow al. 2014, Allen-Zhu & L.| 2022), and defensive distillation (Papernot et al. 2016). or certificate-based such as Lipschitz regularization
(Xu et al. 2020; Fazlyab et al. [2019 Pauli et al.) 2021 Aquino et al. 2022 Virmaux & Scaman
2018 Combettes & Pesquet (020). The overall intent of certificate-based approaches to penalize
the input-to-output sensitivity of NNs to improve robustness.
Recently, the connections between NNs and dynamical systems have been extensively explored. Representative results include classes of NNs stemming from the discretization dynamical sys- tems (Haber & Ruthotto (017) and NODEs (Chen et al. 2018), which transform the input through a continuous-time ODE embedding training parameters. The continuous-time nature of NODEs makes them particularly suitable for learning complex dynamical systems (Rubanova et al. 2019 Greydanus et al.] 2019) and allows borrowing tools from dynamical system theory to analyze their
properties (Fazlyab al. 2022 Galimberti et al. 202T).
In this paper, we employ contraction theory to improve the robustness of NODEs. A dynamical system contractive all trajectories converge exponentially fast to each other (Lohmiller & Slo- line 1998 Tsukamoto et al., (021). Through the lens of contraction, slight perturbations of initial conditions have a diminishing impact over time on the NODE state. With the above considerations, we propose a class of regularizers that promote contractivity of NODEs during the training. In the most general case, the regularizers require the Jacobian matrix of the NODE, which might be com- putationally challenging to obtain for deep networks. Nevertheless, for wide class of NODEs with slope-restricted activation functions, we show that contractivity can promoted by directly penaliz- ing the weights during the training. Moreover, by leveraging the linearity of convolution operations, we demonstrate that contractivity can be promoted for convolutional NODEs by regularizing the convolution filters only.

---

Under review as a conference paper at ICLR 2023
1.1
RELATED WORK
Several works have focused improving the robustmess of general NNs against input noise and ad- versarial attacks using dynamical system theory. For example, the notion of incremental dissipativity is used provide robustness certificates for NNs in the form of linear matrix inequality (Aquino et al. 2022). The works Chen et al. [2021] 2022) address the robustness issue of NNs by using closed-low lose control method from the perspective dynamical systems. A control process is added to trained NN to generate control signals to mitigate the perturbations in input data. Nevertheless, the method requires to solve an optimal control problem for the inference of an input sample, which increases the computational burden.
detailed study on the robustness of NODEs has been done by |Hanshu al. (2019). where the authors show that NODEs can be more robust against random perturbations than common convolu- tional NNs. Moreover, they study time-invariant NODEs, and propose to regularize their flows to further enhance the robustness. To bolster the defense against adversarial attacks. NODEs equipped with Lyapunov-stable equilibrium points have been proposed (Kang et al.) 2021). Likewise, Ro- driguez et al. (2022) introduced loss function to promote robustness based on control-theoretic Lyapunov condition. Both methods have shown promising performance against adversarial attacks. Finally, MMssaroli et al.. (2020) design provably stable NODEs and argue that stability reduce the sensitivity to small perturbations of the input data. Nevertheless, this claim is not supported by theoretical analysis numerical validation. In comparison all the aforementioned works, in this paper, we employ contraction theory to regularize the trajectories of NODEs and improve robustness.
Recently, contraction theory has been employed in the framework of NNs for various purposes. For instance, contractivity exploited to improve the cell-posedness and robustness of implicit NNs (Jafarpour et al.] 2021), the trainability of recurrent NÑs (Revay & Manchester, [2020] Jafarpour
ct al. 2022), and the analysis Hopfield NNs with Hebbian learning (Centorrino et al.] (022).
In Zakwan et al. 2022). the authors propose Hamiltonian NODE that is contractive by design to improve robustness. However, the extension to different classes of NODEs, including convolu- tional NODEs, is not straightforward. Besides the robustification of NNs and NODEs, contractivity
has also been exploited for learning NN-based dynamical models from data. For instance, [Singh al. (202T) and Revay al. (2021a b) utilize contraction theory to learn stabilizable nonlinear NN models from available data.
1.2 CONTRIBUTIONS
The contribution of this paper is fourfold.
We show that contractivity can be used to improve the robustness of NODEs, and demon- strate how to promote contractivity for general NODEs during training by including regu- larization terms in the cost function.
The regularization terms involve optimizing the Jacobian matrix in NODEs, which might be computationally expensive. Interestingly, for a wide class of NODEs with slope-restricted activation functions, prove that contractivity can be promoted by carefully penalizing weight matrices and without optimizing the Jacobian matrix.
By exploiting the linearity of convolution operations and the above results for NODEs with slope-restricted activation functions, we show that contractivity for convolutional NODEs can be induced by suitably penalizing the convolutional filters.
We conduct experiments MNIST and FashionMNIST datasets with test images per- turbed by different kinds of noise and adversarial attacks. Compared to vanilla NODEs, by using contractivity prrmmtttii regularization terms the average test accuracy can be im- proved up to 34% the presence of input noise and up to 30% in the case of adversarial attacks.
2

---

Under review as a conference paper at ICLR 2023
1.3 ORGANIZATION AND NOTATION
The paper is organized as follows: Section Dectionn2provee provides preliminaries on NODEs and contraction the- ory. In Section 3
we propose several regularization approaches for NODEs promote contractivity. Numerical experiments are described in Section and Section 5 concludes the paper.
The set of real numbers is IR.
af(#] Oz represents the Jacobian matrix of continuously differentiable
function f(.). The minimal eigenvalue of a symmetric matrix denoted as Amin diag(x)) represents diagonal matrix with the entries of the vector the diagonal. For symmetric matrices A and B, A > ( means that A - B is positive (semi)definite. I denotes the identity matrix. The 2-norm is denoted as ||-
2 PRELIMINARIES
2.1 NEURAL ORDINARY DIFFERENTIAL EQUATION
NODE is represented by the dynamical system
d = f(z,, Pc. t), t [0,
(1)
where It E R the state the NODE and f(3,, 0z, is generic smooth function with param- eters Or R When used in machine learning tasks, the NODE is usually pre- and post- pended with additional layers, c.g., = (z) and = go (ZT), where ha ga are NNs with parameters E Th B R respectively, 2 E R. the input feature, y E IR" represents the output, and 2r are the state of the NODE ① at time t = O, and t = T, respectively.
Several methods have been proposed for training NODEs, such as the adjoint sensitivity method (Chen et al. 2018), and the auto-differentiation technique (Paszke et al. 2017). In this paper, use the most straightforward approach, that is, the dimeeiiscttiization of (1) (Haber& & Ruthotto (017). Consider classification task, and suppose the training dataset is {=,, fi where 2, are the input features (e.g. images), C; are the corresponding labels, and 8 is the number of training samples. Before training, the NODE (T) is discretized and the resulting discrete-time equations define each of the network layers. For instance, by using Forward Euler (FE) method one obtains
T
Øx+1 = 2g + hf 0k. k),
k= 0......: h
(2)
where h > 0 is the sampling period. Then, the NODE is trained by solving the optimization proble
min ( 1
(Vvi ci) + rreg(a, {0x) T/hh1 .) Σ
(3)
1=1
s.t. 2nj ha(z)), i=
o = 1 + ha O2 k), 0 h
1,
(4)
y, g
where 1(--)) denotes the loss function, and reg(-) is a regularization term suitably scaled by the regularization parameter y 0. For brevity, throughout the paper, we omit the pre- and post-pended layers hag (-) and (-). which usually depend on the specific learning task (Chen et al. 2018
2.2 CONTRACTIVITY
Contractivity is property of dynamical systems, and it implies that the trajectories of the dynamical system converge to each other asymptotically, The formal definition is given below.
Definition 1. The dynamics
7 is contractive with a contraction rate > 0 if || - æell I e -pt [ā0 - zall. Vt E [0,
(5)
for all 2o.20 E IR", where 3t and are the solutions of 7 with initial conditions to and Jo- respectively.
For simplicity, we assume that T is an integer.
3

---

Under review as a conference paper at ICLR 2023
Therefore, if NODE is contractive, the Lipschitz constant between the input and the output is | < for any ž0, 20 As result, contractive NODEs are robust in
smaller than that is, || the sense that a slight perturbation the input features 2 would not result in a large deviation in the output ZT. Moreover, we have that the NODE is contractive with contraction rate p, if and only if Tsskamoto et al. 2021
as Ox
0s dx
pT
00, VI E (0.7), E ER",
where of is the Jacobian matrix of f. 0z
Remark 1. The notion of asymptotic stability used in Massaroli al. (2020) might not be ap- propriate for promoting robustness of NNs. Indeed, as shown in Rüffer et al. (2013), although for convergent dynamics the perturbed states eventually converge time, the distance between trajectories can be arbitrarily large, which can result in poor robustness of the NODE (7). In contrast, contractive dynamics does suffer from this problem, and we will show in Section 4 that contractivity can considerably improve the robustness of NODEs.
unique trajectory, after finite
Remark 2. Contractivvty implies all the state trajectories of ( converge exponentially fast to an equilibrium (Tsukamoto et al.) 2027). which may limit the representation power of NODEs. How- ever, a loss of expressivity might be unavoidable for increasing robustness, as discussed Tsipras et al. (2019)
Remark 3. When training NODEs with global contractivity requirement, the training time is fi-
nite, and we can also tune the contraction rate p. which is hyper-pon pappmeerr As result, the NODE trajectory would neither diverge nor converge to the same point during training, which ensures good learning and robustness performance. The readers can also refer Figure in Zakwan et al. (2022) as an illustration showing that global contraction can still ensure good learning result.
3 CONTRACTIVITT--PRMOOII
REGULARIZATION
To promote the robustness of the NODE [ one can leverage a regularization term penalizing the violation of (6). Contractivity requires the inequality (6) to hold for all E [0, T], R However, during the training, we only have access to discretized states zi. and hence, we can promote the fulfillment of the condition (6
by using the following regularization term in 3
reg( (gg(00)) 1/h-11
ΣΣ
T/h ReLU
-Amin
(-pl-
[ss Ox
+ af dx
le let.x) )
R=
where ReLU(.) denotes the ReLU activation function.
Remark 4. Although the regularizer 77 stems from (6), there some differences. The condi- tion (6) implies that all the trajectories converge to each other exponentially fast. In contrast, the regularizer ( only penalizes the violation of contractivity locally the sampled state 2 which is weaker than 6 and therefore imposes fewer constraints on NODEs. Due to the smoothness property of NODEs, one can show that the learned trajectories (=) "....,i;; .. are locally contractive in
the sense that the relation (5) holds only in the neighborhood of æk, k o/..,,T/h. We defer the reader to Section 4.1 for an illustration showing the benefits of using
3.1 WEIGHT REGULARIZATION FOR IMPROVING TRAINING COMPLEXITY
8f Te laj for all i, k, it might be computationally expensive to obtain. In this section, we focus on a family of NODES with slope- restricted activation functions and show that one can directly regularize their trainable parameters to promote contractivity. Consider the following NODE
Since the regularization term (7) involves the Jacobian matrices
2 (Wi 1., + bt), t [0,7]
where 2, R is the state, W. R b R are NN parameters, and ((-) ) is the activation function. The following theorem provides a sufficient condition on the weights W, guarantecing that (8 is contractive.
4

---

Under review as a conference paper at ICLR 2023
Theorem 1. Assume o.) [ss Rl, where d') denotes any sub-derivative of and > 0. Moreover, for > O, let the following condition hold
P 2kWL,i. R E (WW.l + (W,jl) > 0, i 1.......
(9)
j=1J
for [], where W is the ij-th element of W Then, the NODE contraction rate p.
contractive with
Proof. From 6), the NODE 8 is contractive with contraction rate p if
-pl J,W, W, Jj O, Ve E Re, [0.
(10)
where Jr is the Jacobian matrix of a (W, + b,) with respect to the input Wax + by. It follows that Jr a diagonal matrix with the i-th diagonal entry equal to d'[W, + bels), where [W,,, + b denotes the i-th element of Waz, + b, According to the Gersgorin disk theorem (Horn & Johnson, 1985), any matrix R that satisfies the following conditions
S
Σ 115
Syg
== I....,,,
positive definite (i.e. > 0). The diagonal elements of the matrix -pl JW - WTJ (where the subscript is dropped for simplicity) are
2
where Ja, Wa are the ti-th elements of the matrices J and W, respectively, Moreover, the ij-th (i # j) elements of the matrix pp JW T J are
Ja JjyW
Therefore, in view of Gersgorin disk theorem, the matrix ppl JW W is positive definite
-p 2J
JaW + J,l. i i= 1,...,,,
(11)
j=1
A sufficient condition for the feasibility of (II is that the lower bound of the LHS is greater than the upper bound of the RHS. Consequently, necessary that Wa. S 0. Since 2 o'(-) Z K. lower bound of the LHS of 11 is
- 2*WN,
and an upper bound of the RHS of 11 is
W + [Wyl]
j=1,j
Hence, if the condition 9 holds for all i and t [0, the inequality (10) is verified, and the NODE (8) is contractive with the contraction rate p.
Inspired by the above result, we can use the following regularization term during the training to promote contractivity of the NODE 8
T/h-1
reg (WW))Tb T/h- = Σ Σ ReLU
+ ((Wa.s + (Wa.sl))
p
k=
j=1
(12)
where WE is the discretized counterpart of W, during the training.
Remark 5. Similar the Hamiltonian NODEs in Zakwan et al. (2022) ensuring contractivity by
design, in view of Theorem [ one can parameterize subset of the weight matrices of NODE (8)
that satisfy the condition ( by design. The main idea is to modify the diagonal elements W,, such that the resulting weight matrices W, satisfy (9) automatically. These matrices can be written as
W. W. + He
where H. diag He,,, ...,, for any 0.
.., He,)) with 2gH,, -p- 2 1, E (. ||
5

---

Under review as a conference paper at ICLR 2023
3.2 EFFICIENT IMPLEMENTATION OF REGULARIZERS FOR CONVOLUTIONAL LAYERS
NNs are widely employed to perform image classification tasks, and convolutional layers have proved to be effective for image processing. However, convolution operations on inputs iy are usually not given in the form of Wa + +b, appearing in (8), which hampers the direct use of the regularizers described
Section 3.1 Although the convolution operation can be represented as Wiz; + b, due to its linearity property. it might be burdensome to obtain W,. Hence, we propose a new regularizer directly defieed on the convolution filters to avoid computing We-
By construction, the input Ty and the output of the NODE ! have the same size. Therefore, we consider convolution operations (Goodfellow et al. 2016) that preserve the dimension of the input. Suppose the size of the input X and the output Y of the convolution operation is D x P x H, where P and H are the width and height, respectively, of the image, and D the number of channels. Let X; be the i-th channel of the input X, and Y, be the j-th channel of the output Y. Both channels have size P H. Furthermore, let the filters of the convolution operations be C. i, 1,..., D, where C represents the filter map from the i-th input channel to the j-th output channel. Since inputs and ouiputs of NODEs have the same size, the convolution operations must satisfy additional conditions. For example, if the filter C/ is of size x 3, the input size can be preserved by adding zero-padding of to the input and by applying a stride of 1 (Ciccone et al. 2018). The convolution operation can be written
Y, = E Vj (I,..,,
(13)
where * denotes the convolution operator. Let Vec(X) be the column vector concatenating the transpose of all the rows of X; for all i. Then, (13) can be written as
Vec(Y) = W x Vec(X),
for some weight matrix W E Rnxn, where n D P H. From 13 we can that every element of W is a linear function of C². However, computing W from Cj can be time-consuming. The following lemma reveals important connections between the matrix W and the filters C., that can be leveraged to directly regularize the filters C for imposing contractivity.
Lemma 1 Ciccone et al. (2018)). Suppose the size of C₁ is 3 3, and the convolution operation is applied with a zero-padding of and a stride of 1. Then the following results hold.
(1) Let (Cd) oeeter denote the center element of Ctd, d = 1,.... D. Then
W.: = =C i= P H (d - 1) + 1....., PxHxd,
(2) Let (C ya denote the kl-th elements of Cy. Then,
D
Σ Wiy
S Σ Σ
i= i=PxHx(d-1)+L, + PxHxd, HCM
1=1
j=1
kd
D
Σ
W ΣΣ
H.H.l. (=P PxHxd.
j=1 J=1 Rd
Remark 6. Although Lemma onlly considers convolution operations with a filter size 3 X 3, zero-padding of and stride of 1, the result can also be extended to other convolution operations that preserve the size of the input, for example, the convolution operation with filter size 5 x 5, zero-padding of 2 and a stride of 1. For more details, please refer to Ciccone et al. (2018).
In view of Lemma 1 for i = P H x (d - 1) 1,.., P x H x d, we have
Σ J=L
(Way) + WW Σ
(W w + W 2
j=l
Σ
Σ
(C) Σ
HCDaaal -2((C))
] Jeenter
(14)
1=1
12
kd

---

Under review as a conference paper at ICLR 2023
Therefore, if the NN in (8) contains convolutional layer with filters C₁, regularization term
the following
reg ((C))
Σ Hg +Eicc )
Σ Px x d=1
p+ p+ (C
d' center
(15)
which based on the expression of Wa in Lemma the upper bound 14 the contractivity re- quirement (9. and the constraint Wa < 0.
Remark 7. The regularizer (15) includes the coefficient x H, which usually is very large for image classification tasks. In experiments of Section 4 omit the term P H in (T3) and embed into the regularization parameter 7
4 EXPERIMENTS
In this section, first, we compare the pros and cons of the proposed regularizers, and second, we empirically validate the improvement in the robustness of convolutional NODEs against different forms input noise and adversarial attacks by using the contractivity-rronee regularizers on MNIST and MNIST classification tasks.
4.1 COMPARISONS OF DIFFERENT REGULARIZATION TERMS
In Section we proposed two different approaches for promoting contraction through regulariza- tion. The first one exploits the regularization term whereas, the second one focuses the NODEs (8). and utilizes the regularization term (12). Both methods have pros and cons. (12) more computationally efficient, but might affect the representation power of the NODEs. In- deed (22) aims to make (9) and further (6) hold for all x E RR. In contrast, the regularizer (7) only promotes contractivity constraints on the sampled states z/.- As result, the trained NODE ex- pected to be contractive only the neighborhood of the points zς. This property may be beneficial for some learning problems as illustrated in the following example.
Consider the learning task shown in Figure Ial where the goal to train a NODE learn a map associating z to T and σ to 27.. Since the distance between z and 2 is smaller than the distance between arf and x². we cannot obtain a satisfactory globally contractive NODE, that is, NODE satisfying the contractivity condition (6). Instead, good performance can be achieved by NODE, which fulfills the contractivity condition 7 involving only the sampled states. The learned trajectory and the flow of the NODE trained with the regularization term (7) are shown in Fig. Ib To demonstrate the contractivity properties of the trained NODE, we sample the blue circles around zo and xf (see Fig. lb). and plot the sets corresponding to the NODE outputs in red. We observe that the area of the red regions is smaller than the area of the input circles, which is expected from local contractivity.
4.2 MNIST AND FASHIONMIIST CLASSIFICATION TASKS
We evaluate the performance of the proposed regularization schemes on image classification tasks for the MNIST and FashionMNIST datasets, which are based on images of size 28 28, In both cases, we use the NODE (8) with convolutional layers. We train both vanilla NODE (i.e., using y = 3 and the NODE with the regularization term (5). which we refer to as contractive NODE (CNODE), for ten different seeds obtaining 10 versions of each model.
The NODE structure is described as follows, where unless otherwise specified, the same parameters are used for both the MNIST and the FashionMNIST datasets. First, the image is processed by h, (-), which is a convolution operation with a filter size 3 x 3, a stride of 1, and a channel number of 8 and 16 for MNIST and FaSHionMNIST dataset, respectively. Second, it is processed by the NODE ( for T = 0.1, where the NN is also a convolution operation with a filter size 3 x 3, a zero-padding of 1, and stride of We FE discretization with step size h 0.01 for training the NODES.

---

Under review as a conference paper at ICLR 2023
2-
01
(a) The 2D learning task.
(b) Learned NODE trajectory and its flow.
Figure I: Example of a simple NODE trained with the contractivity-proooting regularizer
Finally, the output of the NODE is followed by fully connected layer ga(-) with output dimension 10. Due to the smoothness requirement of f in (T and the slope restrictions, we select the activation function in (8) to be the smooth leaky ReLU function, given by a(x) 0.1x + 0.9 log(1 + e*), which satisfies
S (') S. We use the Adam optimizer to minimize the cross-entropy loss. The initial learning rate for the Adam optimizer is 0.05, and the learning rate is reduced by factor of 0.7 after every training epoch. The maximal number of training epochs is 20. For the regularizer (15). p = 2. The weight y for the regularization term (15) is set 1. The contraction rate p and the regularization parameter y are selected using grid search. We show in Appendix (A.2 and Appendix [A.3 that the average test accuracy is quite insensitive to the choice of p and y. Moreover, we change the convolution parameters in the NODE and repeat the experiment. The results are shown in Appendix A.4 which implies that with different convolution parameters, we can still achieve improved robustness performance with contractivity regularization.
We test the performance of the vanilla NODE and CNODE against noisy test datasets, where the images are perturbed by zero mean Gaussian noise, and salt&pepper noise (Schott et al. (019). For each kind of noise, we generate several noisy test datasets with different noise strengths. More- over, we test the adversarial robustness of the NODEs with respect to fast- fast-gradient-Se trradien-sign--ethhe (FGSM) (Goodfellow et al., 2014) and projected gradient descent (PGD) attacks (Madry et al.) 2017). Tables and 4 summarize the mean and standard deviations of the classification accuracy over all test sets. In T o is the standard deviation of the Gaussian noise and denotes the pro- portion of image pixels corrupted by the impulse noise. The results of robustness against adversarial attacks are reported in Table 4 where 8 represents the 1x0 amplitude of perturbations in FGSM and PGD attacks. The best performance in each column appears in bold. To give an idea of the intensity of perturbations, we provide samples of test images in Appendix A.1
No Noise
Gaussian
Salt&Pepper
MNIST
0
-=00
0.1
0.2
0.3
6= 0.1
6
0.2
0.3
Vanilla NODE
98 H0.3
65+-23
45+-21
37416
761
54+11
4248
CNODE
98110.1
94
79188
62:112
8814
6818
48-8
MNSST
=
0 = 0.1
0 0.2
0 0.3
0.1
6 = 0.2
0.3
Vanilla NODE
8.1
75
47+4
3544
694-2
51 +4
38
CNODE
8820.22
8511
72:22
55:14
75:12
5715
42.15
Table 1: Classification accuracy over noisy test images (mean + standard deviation).
From the tables, we can observe that the CNODEs achieve higher mean classification accuracy than the vanilla NODEs in the presence of image perturbations. In some cases, the performance improvements are very significant (up to 34% for the case of Gaussian noises). Moreover, the standard deviations with CNODEs are either the same less than those with vanilla NODEs in
8

---

Under review as a conference paper at ICLR 2023
FGSM
PGD
MNIST
= 0.01
8= 0.02
0.03
à = 0.01
8 == 0.02
8 8= 0.03
Vanilla NODE
92:12
6748
42:19
9113
63:12
36111
CNODE
9.400.4
86:22
6814
9,40.4
86:22
66:14
MNIST
0.01
8 = 0.02
0.03
8 = 0.01
0.02
= 0.03
Vanilla NODE
6311
31
13:11
6211
29:1
1111
CNODE
72:11
49:12
28:12
71:11
47::3
26:12
Table 2: Classification accuracy over adversarial attacks (mean + standard deviation).
almost all the experiments, which means, CNODEs are less sensitive than vanilla NODEs to the selection of initialization seeds.
CONCLUSIONS
In this paper, we use contraction from dynamical system theory improve the robustness of NODEs. We propose regularizers with different degrees of flexibility and different computational requirements to promote contractivity. The good performance of the resulting NNs illustrated on image classification tasks. Future work will focus the development of easy-to-compute regular- izers for classes of NODEs stemming from specific choices of f in
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pp. 977-988. IEEE, 2022.
Bernardo Aquino, Arash Rahnama, Peter Seiler, Lizhen Lin, and Vijay Gupta. Robustness against adversarial attacks in neural networks using incremental dissipativity. IEEE Control Systems Letters, 6:2341-2346, 2022.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense curity: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274 273-283. PMLR, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, In IEEE symposium on security and privacy (SP), pp. 39-57. IEEE, 2017.
Veronica Centorrino, Francesco Bullo, and Giovanni Russo. Contraction analysis of Hopfield neural networks with Hebbian learning. arXiv preprint 22X.::2204.03882 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, volume 31, 2018.
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Towards robust neural networks via close-loop control. arXiv preprint 10Xiv:2102.01880 2021.
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Self Self-healing robust neural networks via closed-loop control. arXiv preprint arXiv:2206./220 2022.
Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez. Nais- net: Stable deep networks from non-autonomous differential equations. In Advances in Neural Information Processing Systems, volume 31, 2018.
Patrick L Combettes and Jean -hriithiteee Pesquet. Lipschitz certificates for layered network struc- tures driven by averaged activation operators. SIAM Journal on Mathematics of Data Science, (2):529-557, 2020.

---

Under review as a conference paper at ICLR 2023
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of Lipschitz constants for deep neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Mahyar Fazlyab, Manfred Morari, and George Pappas. Safety verification and robustness analysis of neural networks via quadratic constraints and semidefinite programming. IEEE Transactions on Automatic Control, 67(1):1-15, 2022.
Clara Lucía Galimberti, Luca Furieri. Liang Xu, and Giancarlo Cerrari-Trecale Hamilto nian deep neural networks guaranteeing non-vanishing gradients by design. arXiv preprint arXiv:2105.13205 2021.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, Cambridge, Massachusetts, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.
Yan Hanshu, Du Jiawei, Tan Vincent, and Feng Jiashi. On robustness neural ordinary diffcrential equations. In International Conference Learning Representations, 2019.
Roger A. Horn and Charles Johnson. Matrix Analysis. Cambridge University Press, New York, 1985.
Saber Jafarpour, Alexander Davydov, Anton Proskurnikov, and Francesco Bullo. Robust implicit networks via non-Fuclidean contractions. In Advances in Neural Information Processing Systems, volume 34, 2021.
Saber Jafarpour, Matthew Abate, Alexander Davydov, Francesco Bullo, and Samuel Coogan. Ro- bustness certificates for implicit neural networks: A mixed monotone contractive approach. In Learning for Dynamics and Control Conference, pp. 917-930. PMLR, 2022.
Qiyu Kang, Yang Song, Qinxu Ding, and Wee Peng Tay. Stable neural ode with Lyapunov-stable equilibrium points for defending against adversarial attacks. In Advances in Neural Information Processing Systems, volume 34, 2021.
Winfried Lohmiller and JaaneJs E. Slotine. On contraction analysis for non-linear systems. Automatica, 34(6))683-696, 1998.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 766600683, 2017.
Stefano Massaroli, Michael Poli, Michelangelo Bin, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Stable neural flows. arXiv preprint wrXiv:2003.0806. 2020.
David Miller, Zhen Xiang, and George Kesidis. Adversarial learning targeting deep neural network classification: A comprehensive review of defenses against attacks, Proceedings of the IEEE, 108 (3):402-45) 2020.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense adversarial perturbations against deep neural networks. In IEEE symposium on security and privacy (SP). pp. 582-597. IEEE, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS 20/7 Autodiff Workshop, 2017.
10

---

Under review as a conference paper at ICLR 2023
Patricia Pauli, Anne Koch, Julian Berberich, Paul Kohler, and Frank Allgöwer. Training robust neural networks using Lipschitz bounds. IEEE Control Systems Letters, 6: 6:121-126, 2021.
Max Revay and Ian Manchester. Contracting implicit recurrent neural networks: Stable models with improved trainability. In Learning for Dynamics and Control, pp. 393-403. PMLR, 2020.
Max Revay, Ruigang Wang, and Ian R Manchester. Recurrent equilibrium networks: Flexible dy- namic models with guaranteed stability and robustness, arXiv preprint arXiv:2104.05942, 2021a.
Max Revay, Ruigang Wang, and lan Manchester. Recurrent equilibrium networks: Unconstrained learning of stable and robust dynamical models. In 60th IEEE Conference on Decision and Con- trol (CDC), pp. 2282-2287. IEBE, 2021b.
Ivan Dario Jimenez Rodrigue7, Aaron Ames, and Yisong Yue. LyaNet: A Lyapunov framework for training neural ODEs. International Conference on Machine Learning, pp. 18687-18703. PMLR, 2022.
Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent ordinary differential equations for iregularly-samplled time series. In Advances in neural information processing systems, vol- ume 32, 2019.
Björn S Rüffer, Nathan Van De Wouw, and Markus Mueller. Convergent systems vs. incremental stability. Systems & Control Letters, 62(3):277-285, 2013.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representa- tions, 2019.
Sumeet Singh, Spencer M Richards, Vikas Sindhwani, Jean-Jacques E Slotine, and Marco Pavone. Learning stabilizable nonlinear dynamics with contraction-based regularization. The Interna- tional Journal of Robotics Research, 40(10-11): 123-1115,, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. In International Conference on Learning Representa- tions, 2019.
Hiroyasu Tsukamoto, Soon-Jo Chung, and Jaques Slotine. Contraction theory for nonlinear stability analysis and learning- based control: A tutorial overview. Annual Reviews in Control, 52: 135-169, 2021.
Aladin Virmaux and Kevin Scaman. Lipschitz. regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems, volume 31, 2018.
Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil K Jain, Adversar- ial attacks and defenses in images, graphs and text: A review, International Journal of Automation and Computing, 1 151-):5 2020.
Muhammad Zakwan, Liang Xu, and Giancarlo Ferrari Trecate. Robust classification using contrac- tive Hamiltonian neural ODEs. IEEE Control Systems Letters, 7:145-150, 2022.
A APPENDIX
A.1 CLASSIFICATION EXPERIMENTS: EXAMPLES PERTURBED IMAGES
In Figure 2, provide samples of the perturbed MNIST and FaSH datasets that have been used for testing the performance of different NODEs in Section 4 For the meaning of the parameters a, and 6, capturing the perturbation magnitude, defer the reader Section 4.2

---

Under review as a conference paper at ICLR 2023
2
2
2
2
2
2
8
8
8
8
8
8
&
6
6
6
6
6
5
9
9
9
9
9
=0.0
0
=0.2
0 =0.3
E =0.0
6 =001
=02
8 =.33
(a) MNIST samples perturbed by Gaussian noise.
(b) MNIST samples perturbed by salt&pepper noise.
2
2
2
2
2
8
8
8
8
8
8
8
8
7
6
6
6
6
6
6
6
6
6
8
8
8
9
8
8
9
9
9
6-0.0
-0001
6-0.02
6 60003
6-05
60.00
5 60001
6-0.02
6 -0.03
5 6-05
(c) MNIST samples perturbed by FGSM attacks.
(d) MNIST samples perturbed by PGD attacks.
=00
a =0.1
=02.2
=0.3
=00.0
6 =00.1
=02
=0.3
(e) FashionMNIST samples perturbed by Gaussian(f) FashionMNIST samples perturbed by salt&pepper noise.
noise.
12

---

Under review as a conference paper at ICLR 2023
=0.02
=00.03
=0.0
=0.01
=0.02
5 =0.03
(g) FNSS samples perturbed by FGSM at- (h) FashionMNIST samples perturbed by PGD attacks. tacks.
Figure 2: Examples of perturbed images in MNIST and FashionMNIST classification tasks
A.2 CONTRACTION RATE VS CLASSIFICATION ACCURACY
In this appendix, analyze how the contraction affects the classification accuracy. For this purpose, we use the MNIST dataset and images perturbed by Gaussian noises or FGSM attacks. We use contraction rates p in the set 10.1, 2, 5, 7, 10, 12, 15), train the CNODEs, and obtain 10 models for each p by using different seeds. Then, we calculate the classification accuracy of these models on the clean test dataset, the dataset perturbed by Gaussian noises, and the test dataset attacked by FGSM. The mean and the standard deviations of the classification accuracy are plotted in Figure 3 Figure and Figure where the solid line represents the mean and the shaded region spans one standard deviation on each side of the mean. We can observe that the average classification accuracy does not vary significantly for different contraction rates. This suggests that the choice of the contraction rate is not critical for the MNIST experiments discussed in Section 4
1.00
0.96
ficatin accuracy
0.94
train accuracy
test eccuracy
0.90
10
contraction rate
Figure 3: Classification accuracy on the clean test dataset with respect to different contraction rates
13

---

Under review as a conference paper at ICLR 2023
10-
0.5
clasellicaion accurocy
04
==02
0.
0.0
10
contraction
Figure 4: Classification accuracy on test dataset perturbed by Gaussian noise with respect to different contraction rates p.
1.0
08
disceliiation acnuracy
0.4
0.2 0.01
0.02
contraction rate
Figure 5: Classification accuracy on test dataset perturbed by FGSM attacks with respect different contraction rates p.
A.3 REGULARIZER WEIGHT VS CLASSIFICATION ACCURACY
In this appendix, we analyze how the regularizer weight affects the classification accuracy. For this purpose, we use the MNIST dataset and images perturbed by Gaussian noises or FGSM attacks. We use regularization parameter in the set 1, 5, 10, 20. 30, 40, 50}. train the CNODEs, and obtain 10 models for each by using different seeds. Then, we calculate the classification accuracy of these models on the clean test dataset, the test dataset perturbed by Gaussian noises, and the test dataset attacked by FGSM. The mean and the standard deviations the classification accuracy are plotted in Figure 6 Figure 7 and Figure g where the solid line represents the mean and the shaded region spans one standard deviation on each side of the mean. We can observe that the average classification accuracy does not vary significantly with different values of r. Therefore, the average
14

---

Under review as a conference paper at ICLR 2023
robustness performance does not rely heavily on the selection of the regularization parameter ? in the MNIST experiment Section
dlssaliiation accuracy
0.96
0.94
train acturacy
test accuracy
0.90
10
201
30
40
50
regulatization parameter
Figure 6: Classification accuracy on the clean test dataset with respect to different regularization parameters 7.
0.8
class fication acturaey
0.4
0.2
0 0.2
0.3
regularization parameter
Figure 7: Classification accuracy on test dataset perturbed by Gaussian noise with respect to different regularization parameters 7.
15

---

Under review as a conference paper at ICLR 2023
10
clasollication accuracy
0.6
0.4
0.02
0.03
0.0
20
50
regularization parameter
Figure 8: Classification accuracy on test dataset perturbed FGSM attacks with respect to different regularization parameters 7.
A.4 CONVOLUTION PARAMETERS VS CLASSIFICATION ACCURACY
In this Appendix, we perform an ablation study by selecting different convolution parameters and demonstrate that CNODEs can still achieve improved robustness performance. We set parameters of the convolution operation in the NODE the following two groups:
Group 1: filter size 5 x 5, zero-padding of 2, and stride of 1.
Group 2: filter size 7 x 7, a zero-padding of 3, and stride of 1.
Then we re-conduct the experiment. ing parameters
to be the The maximal number of training epochs is 30. The other train- with those in Section 4.2 The average test accuracy and the standard deviation data are shown in the following table, where CNODE(5) and CNODE(7) repre- sent the CNODE with convolution parameter Group and Group 2, respectively. We can observe that with different convolution parameters, the CNODE can still achieve improved performance.
No Noise
Gaussian
Salt&Pepper
MNIST
=
= 0
0 ==0.1 0.1
0 0.2
=
0.3
= 0.1
6 6 0.2
6 0.3
Vanilla NODE
98:10.3
65:123
45::21
37::16
761-9
54+11
4248
CNODE(5)
98:10.2
95122
81=77
65:18
8713
66=14
45:14
CNODE(7)
98-10.1
95-12
8147
F1:F11
8813
67 17-66
46
FaShionMNIST
0
σ = 0.1
σ 0.2
0.3
0.1
0.2
0.3
Vanilla NODE
88110.1
7514
4714
35:4
6912
51:14
38455
CNODE(5)
88+00.1
83
65
48
7511
58
44
CNODE(7)
88-10.1
83+1
61
42:55
75:12
55.15
40:5
Table 3: Classification accuracy over noisy test images (mean standard deviation).
16

---

Under review as a conference paper at ICLR 2023
FGSM
PGD
MNIST
0.01
8= 0.02
0 0.03
Ó = 0.01
0.02
== 0.03
Vanilla NODE
9242
6748
42499
9113
63:112
36:111
CNODE(5)
9.5
8713
69:16
95.0.66
86113
67-6
CNODE(7)
9.5
8712
7016
95.10.5
8713
69:16
MashionMNIST
0.01
8 = 0.02
8 = 0.03
8 = 0.01
8 = 0.02
8 = 0.03
Vanilla NODE
63=1
31
13111
62:11
29:11
11:1
CNODE(5)
71:1
4613
251:3
71:11
4413
23:13
CNODE(7)
73:11
48::2
27::2
72+1
47+2
25:+2
Table 4: Classification accuracy over adversarial attacks (mean + standard deviation).
17


Reviews:
[{"review_id": "_NKNXrDYGN", "reviewer": "Reviewer_pw1W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes regularization terms to promote the contractivity of neural ODEs. It derives a computational efficient regularizer for a special class of neural ODEs and showed empirical robustness comparing with plain neural ODEs.", "summary_of_the_review": "This papers proposes regularization terms to enforce contraction properties in neural ODEs with a goal to improve the robustness of neural ODEs. However, in many applications such as image classification, robustness does not mean to have all inputs to converge to a single point. Therefore, global contraction may not be a desired property to have. I encourage the authors to explore \"local\" contraction for each class of images, or other applications where global contraction is desired. ", "strength_and_weaknesses": "Strength:\n1) Leveraging control tools to enforce contraction properties on neural ODEs is an important direction that brings new insights into the learning community. \n2) For neural ODEs whose dynamics is one activation function composed with one linear layer, the authors proposed an efficient way to enforce the contraction property. \n\nWeaknesses:\n1) My main concern for this paper is: Enforcing global contractivity can NOT provide robust classifiers. Indeed, it can provide robustness, but robust classification requires classifying images into the correct classes, not into one class. Following Def 1, if global contraction holds for a neural ODE, then with long enough integration time, all x_t will converge to a single point no matter what x_0 it starts with. That means, a global contractive neural ODE classifier will classify all images into one class, which loses classification capability. \nA neural ODE that satisfies (15) should classify images into one class. However, the experiments still show nontrivial accuracy. I suspect that it is because the integration time is not long enough, or the contraction rate is very small. \n2) Some recent papers (e.g. [1]) shows that adversarial robustness improvements of neural ODEs may be from obfuscated gradients. Therefore, the evaluation of the robustness improvement should be evaluated with AutoAttack (https://github.com/fra31/auto-attack), and the accuracy after each attack should be reported so that it provides more info on whether the robustness improvement is due to gradient obfuscation. Or, the authors can exploit whether they can provide a certification for the robustness, if the contraction property holds everywhere, they should be able to evaluate the Lipschitz of the neural ODE and certify the robustness.\n3) The efficient regularizer sacrifices the expressiveness of neural ODEs in two folds: one layer dynamics, and the lower and bounds when deriving the regularizer. The authors could potentially explore whether they can relax the global contraction requirement (also see point 1)) so that they can enable the dynamics to be more expressive.\n\n[1] Yifei Huang et al. “Adversarial Robustness of Stabilized Neural ODE Might be from Obfuscated Gradients”. In: Mathematical and Scientific Machine Learning. PMLR. 2022, pp. 497–515.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X5-1IKnazN", "reviewer": "Reviewer_8Xx6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the robustness of neural ODEs, by studying the condition when neural ODEs are contractive, i.e., the trajectories of the ODE converge to the same value exponentially fast. Based on existing lemmas on the condition of contraction, the authors propose a weight regularization based approach during training to encourage contraction of neural ODEs and thus improve its robustness. The method is demonstrated on MNIST and Fashion-MNIST datasets and its robustness under Guassian noises and adversarial attacks are better than vanilla ODEs.", "summary_of_the_review": "This paper is not ready for publication because of the relatively weak theoretical contribution, missing related works and insufficient experiments. The robustness of neural ODEs is a very interesting topic and the proposed regularization based approach might be promising if it is thoroughly evaluated. I encourage the authors to continue working on this problem further and submit the paper to a future conference.", "strength_and_weaknesses": "Strengths:\n\n1. The robustness of neural ODE is an important topic and this paper brings some new insights into this problem.\n\n2. The contraction theory inspired weight regularization based approach for enhancing the robustness of ODE is new, although it is not well evaluated and its connection to previous Lyapunov theory based approaches are not thoroughly discussed.\n\n\nWeaknesses:\n\n1. The way of training neural ODEs in this paper is naive via unrolling the steps via time discretization. This is not the best way to train neural ODEs and has great impact on training cost and practicability. \n\n2. Many theoretical results presented in this paper are not new (e.g., Lemma 1, Lemma 2). They should not take too much space. The main contribution seems to be an application of Gersgorin disk theorem for the weight matrices of neural ODEs, and it is arguably a significant contribution.\n\n3. Empirical evaluations are very weak, with MNIST and Fashion-MNIST only. These are not representative datasets, since MNIST-like datasets are too simple and do not reflect true performance of an algorithm under any more practical datasets. In addition, a few important baselines which also focus on training robust ODE are missing, see [1][2]. [1][2] use an elegant and fundamental approach from the Lyapunov theory.\n\nQuestion:\n\nIf a NODE is fully contractive then it is possible that all inputs are contracted to the same output. Thus for any real tasks it can only be locally contractive (e.g., a point very close to x_0 should lead to the same output). The main theory is independent of x_0, so it is for global contraction. How can you prevent that the output of NODE for all input points collapse to the same value?\n\n[1] Rodriguez, Ivan Dario Jimenez, Aaron Ames, and Yisong Yue. \"LyaNet: A Lyapunov framework for training neural ODEs.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Kang, Qiyu, et al. \"Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks.\" Advances in Neural Information Processing Systems 34 (2021): 14925-14937.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3hj7-KmIPth", "reviewer": "Reviewer_MdaE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper leverages contraction theory to improve the robustness of neural ordinary differential equation (ODE). Directly obtaining contraction property needs to regularize the Jacobian matrix, which is computationally heavy. The authors theoretically prove that penalizing the weight matrix can be a contractivity-promoting regularization under a mild assumption that the slope of the activation function of neural ODE is bounded. The experimental results validate the effectiveness of contractive neural ODEs in enhancing robustness against corruption and adversarial attacks.", "summary_of_the_review": "This paper uses the contraction theory to improve the robustness of neural ODEs. The application of the contraction theory to neural ODE is non-trivial. Thus, I believe the proposed method is novel. However, I have some concerns regarding the empirical results. I would like to rise my score if the authors solve my concerns.", "strength_and_weaknesses": "Strength\n+ The proposed method is theoretically induced. The authors make the contractivity-promoting regularization computationally cheap based on theoretical results, thus enabling the authors to apply contractivity-promoting regularization to neural ODE in practice.\n+ The empirical results support the claim. Compared to vanilla neural ODE, the proposed contractive neural ODE significantly improve the robust accuracy.\n\nWeaknesses\n+ The adversarial robustness evaluation could be unreliable. This paper [1] has pointed out a reliable method to evaluate the robustness of neural ODE. It would be better for the authors to evaluate the performance and compare the proposed method with previous methods such as [2,3].\n+ It would be better to provide an ablation study on the kernel size of the CNN filter, to validate the proposed method can be effective on different neural structures.\n \n\n[1] Evaluating the Adversarial Robustness of Adaptive Test-time Defenses. Croce et al. ICML 2022.\n[2] Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending against Adversarial Attacks. Kang et al. 2021.\n[3] On robustness of neural ordinary differential equations. Yan et al. ICLR 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject