A ROBUST DIFFERENTIAL NEURAL ODE OPTIMIZER
Authors: Panagiotis Theodoropoulos
Guan-Horng Liu
Tianrong Chen
Augustinos D Saravanos
Evangelos Theodorou
Venue: ICLR 2024 poster (2024)
Keywords: Neural Optimizer
Differential Dynamic Programming
Adversarial Defense
Game Theory
Primary area: optimization
TL;DR: In this work we interpret Neural ODE optimization as a min-max optimal control problem, to present a robust game theoretic optimizer based on the principles of min-max Differential Dynamic Programming.

Abstract: Neural networks and neural ODEs tend to be vulnerable to adversarial attacks, rendering robust optimizers critical to curb the success of such attacks. In this regard, the key insight of this work is to interpret Neural ODE optimization as a min-max optimal control problem. More particularly, we present Game Theoretic Second-Order Neural Optimizer (GTSONO), a robust game theoretic optimizer based on the principles of min-max Differential Dynamic Programming.
The proposed method exhibits significant computational benefits due to efficient matrix decompositions and provides convergence guarantees to local saddle points.
Empirically, the robustness of the proposed optimizer is demonstrated through greater robust accuracy  compared to benchmark optimizers when trained on clean images. Additionally, its ability to provide a performance increase when adapted to an already existing adversarial defense technique is also illustrated.
Finally, the superiority of the proposed update law over its gradient based counterpart highlights the potential benefits of incorporating robust optimal control paradigms into adversarial training methods.

Published as a conference paper at ICLR 2024
A ROBUST DIFFERENTIAL NEURAL ODE OPTIMIZER
Panagiotis Theodoropoulos Guan-Horng Liu Tianrong Chen Augustinos D. Saravanos Evangelos A. Theodorou
Georgia Institute of Technology. USA (ptheodor3, ghlin, tianrong. chen, asaravanos3, hatp://hheoooooooeggteeettc
ABSTRACT
Neural networks and neural ODEs tend to be vulnerable to adversarial attacks, rendering robust optimizers critical to curb the success of such attacks. In this regard, the key insight of this work is to interpret Neural ODE optimization as a min-max optimal control problem. More particularly, we present Game Theoretic Second-Order Neural Optimizer (GTSONO), robust game theoretic optimizer based on the principles of min-max Differential Dynamic Programming. The proposed method exhibits significant computational benefits due to efficient ma- trix decompositions and provides convergence guarantees to local saddle points. Empirically, the robustness of the proposed optimizer is demonstrated through greater robust accuracy compared to benchmark optimizers when trained on clean images. Additionally, its ability to provide performance increase when adapted to an already existing adversarial defense technique is also illustrated. Finally, the superiority of the proposed update law over its gradient based counterpart highlights the potential benefits of incorporating robust optimal control paradigms into adversarial training methods.
INTRODUCTION
Despite the remarkable performance achieved by deep learning based models on diverse challenges from image classification to speech recognition, recent findings indicate that deep neural networks tend to be fragile to adversarial examples(Liu et al. (2020)). This problem of generating adversarial examples and conversely training models that not easily misguided by these samples has gathered significant attention (Carrara et al. (2019)). However, the development of robust models has been proven to be more challenging (Tsipras et al. (2018)), rendering adversarial robustness central object of study in machine learning (Carlini et al. (2019)). This study has led to significant advance in understanding the generalization and interpretability behind the learning capacity of these models (Huang et al. (2022)). Traditionally, the most widely used approach to building adversarially robust models has been adversarial training in which the classifier is trained on both adversarial and clean samples aiming to generalize well in both Masty et al. (2017); Wong & Kolter (2018); Raghunathan al. (2018); Goodfellow et al. (2014)). The goal of adversarial training is to find the saddle point that solves the following optimization problem
min E
max C(x + 5, v: 6ES
where x E R. is the state vector and u E R the control, the perturbation term is bounded by sup [[8][2 < 8 and belongs to set S, which is the set of admissible perturbations, and L(x + 5, f) is the definition of the loss function. Although adversarial training is effective, there are some drawbacks such the considerable increase in the training time.
In the work by Poznyak et al. (2023), it was suggested that min-max robust control can be viewed as neural dynamic programming approach using continuous differential neural networks. In this work, the peturbations å were recasted, yielding the same Hamilton Jacobi Bellman equation as in the case of the game theoretic Differential Dynamic Programming (DDP) (Jacobson & Mayne (1970); Sun et al. (2018)).
In the realm of Optimal Control (OC), it has been demonstrated that game- -hee-ttee formulation enables the model handling uncertainties and external disturbances Sun et al. (2015). This algorithm has been successfully tested in real-life robotic applications, coming up up with robust control policies

---

Published as a conference paper at ICLR 2024
for robotic systems operating under unknown external disturbances Sun et al. (2018), Morimoto et al. (2003). This paves the way to draw connections between adversarial defense methods and robust optimal control and design principled algorithms that demonstrate the capacity to handle disturbances.
In this work, drawing inspiration by the connection of training Neural Networks and continuous time OCP Liu et al. (2021a;b), we introduce novel second order Game Theoretic Neural Optimizer, by recasting the training of Neural ODEs as trajectory optimization problem through this Min- Max OCP paradigm. Our approach differs considerably from conventional min-max methods for adversarial learning. While most min-max adversarial training methods consider the adversary input, our proposed method instead consider the disturbance being injected through the antagonizing control. The resulting framework, called GTSONO, is designed based on the continuous time Min-Max DDP algorithm engendering a innately robust Neural Optimizer. This optimizer capable of handling uncertainties and finding a control policy or equivalently weight configuration that robustifies the network and enables it to resist adversarial attacks. Our analysis provides theoretic guarantees that our algorithm converges to saddle point, and our experiments empirically verify that our algorithm converges to a saddle point even when trained with clean images, which is translated to an increased robustness of our model against adversarial attacks, even with natural training. Our contribution can be summarized in the following points:
We propose novel game theoretic optimizer for training robust neural ODEs based on the principles of min-max, reformulating the input adversary as the antagonizing control to model system disturbances. An efficient computational framework is presented based on decomposing matrices into vectors, which can be easily obtain through solving the ODE.
We illustrate that the proposed optimizer guaranteed to converge to stable local saddle points under mild assumptions.
We empirically showcase that GTSONO increases the robustness and generates more confident predictions compared benchmark optimizers on natural training.
We demonstrate that GTSONO is successfully adapted to already existing adversarial training methods, enhancing their robustness. The second order convergence nature of min-max DDP enables optimizer to converge in fewer epochs, and lower wall-clock time, which highlights its computational efficiency.
2 RELATED WORKS
In order to view the landscape of the pertinent literature, we review the advancements in both fronts.
Adversarial Attacks. A number of adversarial attack methods have been developed, such as optimization based (Szegedy et al. (2013)), gradient- based (Moosavi-Dezfooli et al., 2016; Xie et al., 2019; Dong et al., 2018), and attacks from Generative Networks (Baluja & Fischer, 2017: Zhao et al., 2017). Traditionally, the most famous gradient based methods are the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) and the Projected Gradient Descent (PGD) (Madry et al., 2017). The for PGD attack is used to undermine the estimations and predictions of machine learning models, by iteratively perturbing input data in the direction of the gradient, while constraining the perturbations to stay within a predefined projection set determined by parameter c. Conversely, FGSM more efficient gradient based attack, which operates by taking single step in the direction of the gradient of the loss function with respect to the input. Therefore, the attack efficiency of FGSM is higher at the expense of its success rate, compared to the iterative PGD. Compared with other algorithms, network generated adversarial examples have been observed to be deceptive, however their efficiency is deemed low, since they require multiple rounds of iterations for the optimal solution (Liang et al., 2022). Optimization based attacks i.e. Carlini & Wagner (CW; Carlini & Wagner (2017)) have been found be more effective to certain defensive schemes, such as defensive distillation (Liang et al., 2022) and masking (Huang et al., 2022).
network-generated
Adversarial Defense. In order to deal with the threat of adversarial attacks, on the one hand, researchers try to improve the model's resistance to these attacks, developing numerous defense methods, so that the model can make correct predictions on adversarial samples (Wang et al., 2022). Examples of such defense methods are defensive distillation (Papernot et al., 2016), gradient regularization (Papernot et al., 2017; Ross & Doshi- Velez, 2018), and adversarial training (Tramèr et al., 2017; Zhang et al., 2019a; Zheng et al., 2020), which is traditionally the most successful.
2

---

Published as a conference paper at ICLR 2024
Most notably, this technique attempts to improve the robustness of neural network by training it with adversarial samples, i.c., generated by FGSM or PGD. However, the issue with this defense mechanism is the computational overhead. To speed up and solve this problem, Adversarial Training for Free (FreeAT) (Shafahi et al., 2019) was proposed.
Recently more methods have been developed such masking and denoising. In fact, many attack methods rely on the gradient information of the victim model. In gradient masking/obfussation defense methods try to defend by hiding the gradient information (Naseer et al., 2019). Moreover, Huang et al. claimed that adaptive stepsize numerical ODE solver, such as DOPRIS, has a gradient masking effect that fails the PGD attacks which are dependent on gradient information. This implies that continuous neural models possess innate robustness compared to discrete neural networks. However, this method cannot fool gradient- free attacks such as CW and SPSA. On the other hand, since the generation of adversarial examples takes place by adding specific noise; image denoising is another technique enable the models to resist adversarial attacks. Ordinary denoisers suffer from low performance due to error amplification effect. In this regard, Liao et al. proposed high-level representation guided denoiser (HGD) suppressing the amplification errors.
METHODOLOGY
3.1 TRAINING NEURAL ODES USING GAME THEORETIC OPTIMAL CONTROL THEORY
First, Neural ODEs (Chen et al. (2018)) generally concern the following optimization over an objective function: L:
dx x(t,,((), x)), x(1o) = xx
min [(x(ty)), where
(2)
8
dt
where x(t) E Imm. and F)) is a deep neural network parameterized by 0 E R". In the proposed methodology, Eq. (2) is recasted as a trajectory optimization problem. through the perspective of Min-Max OC, by introducing a second set of antagonizing controls (e.g. weights in DNN) that attempt the maximization of the loss function. This second set of parameters aims to represent the existence of disturbances providing the robustness of our model. Finally, for this problem, we obtain:
dx F(t, x. u, v),
x(to) Xo
min max ((Fr,) 1'' e(1, X, u, v)dtt
0,
u(lo) v(to) = 17
0,
dr
u(t) E RR, and v = E R", and F(,, u, v) characterizes the vector field and is parameterized by DNN with the conflicting sets of weights (u,v). v). Moreover, it is clear that (3) describes (2) wlog by taking ( = (C,0). The functions Q and are known the terminal and running cost in the context of OCP, whereas in the DNN setting, they are equivalent to the loss function and the weight decay, respectively. Note that for this problem we consider symmetric layer-wise dynamics with respect u. V. More explicitly, a layer has a maximization counterpart, it has the same number of and v parameters, illustrated in the Node Level view of Figure 1.
where x = x(1) E ".
Node Leve view
Lasst Level View
Layer/
Layer j-f
Taake
Nodaa layar
Nodes layer
Addition
weights (inaximizing control)
weights (minimizing control)
Node Level and Layer Level Architecture of Game Theoretic Layer
Figure 1:
The running cost & is assumed to be quadratic with respect the controls, more specifically: = fu, v), with luu = Rn Evy = - RRy and fuv = O, where I, E Rgnx is the identity matrix, Rv and Ru are scalars, with RRyl |Rul-
Assumption 3.1.
This assumption will enable us to write the Hessian in a form easily factorizable using Kronecker properties. The intuition for selecting RR, |Ral is that we wish to penalize the disturbance more severely and prevent instability. The time-invariant ODEs imposed for u,, and v, makes the ODE of x, equivalent to (2), while also prompts to interpret Problem (3) the search for optimal initial condition for time-invariant controls u, and V,. Next, we define the accumulated loss Q(t, x, u, v):
Q(t,x,u,v) =x(x(t))))+++ 1. e(T, u(T), v(r))dr
(4)
3

---

Published as a conference paper at ICLR 2024
which implies that 0 [tt x, u, v) + dQ(t.x.u.v) At this point our goal is compute higher-order dt
derivatives with respect to u, v. Note that the first order derivative of L wrt to B or 1) under our aQ respectively, as Q(to, Xta Ut Vta) is practically the OCP formulation equivalent to aQ or iivi
accumulation of error on Ito. ty]. This explains our previous remark that in essence we search an optimal initial condition for u, and v. Now, we frame the equation above along a nominal path to derive the ODEs that will help us to compute the second order derivatives at lo-
Theorem 3.2. Consider Assumption 3. and a nominal trajectory (x, ū, V). that satisfies the ODEs in the constraints of (3). Then, the second order derivatives of Q expanded locally around the nominal trajectory obey the following backward ODEs:
dQxx Fxxxxx
dQxu = FxQxut
dQun = Fuu
di
dl
dt
dQur
dQvv = lov + FuQxx + Fr,
dQxv FxQxv + Qxx FV dt
FaQxx + Qux FT.
df
dt
dQu Fulx tu: dQv F,Qx + tv dt
df
(5)
The terminal conditions are given by: Qx(1g) 4x (xx((x)))) = xxx, and Qu((y)
O Onx1: Qv((y)= = = = Onxn: Qxull1) Qxv(l)=== 0xxm-
Remark 3.3. The proof for this Theorem is postponed for Appendix A.1 and it follows the standard derivation of continuous tory path. This theorem implies that the proposed framework is based min-max DDP, which has the favorable property of second order convergence and can obtain first and second order derivatives with single pass of an ODE solver, without any recursive computations. This contributes avoiding accumulating integration errors, and increased runtimes.
DDP, considering linearized ODE dynamics along the nominal trajec-
3.2
SECOND-ORDER MATRIX FACTORIZATION
The second order nature of our optimizer renders the efficient manipulation of second order terms a critical component of our study, as the amount of parameters in neural networks and neural ODEs grows easily an unfavorable number.
Theorem 3.4. We assume the matrix (xy(t;) to be symmetric matrix rank R S m, which may be represented as: Qux -Et y.y, where y; I 'tt E Lyl. Then let the second order matrices in (5) that contain derivative with respect to the state can be decomposed as follows:
n.
(xx(t)== Σ (t), Qxu(t)= Σ (t),,
Qxv(t)= - (t))s:((t) (6)
Then the vectors qi E I,",, and (1), and (;))) IRn obey the following backward ODEs: (t) = (xq((1) dp:(1) (t))()), (1,(1) = (t):(()
(7)
dt
dt
dt
with the terminal condition given by (qt(t)). p.(tt), = 0,,,0,0, Additionally, the second order matrices Qun, Quv. and Que can be represented as
R
R
Quu(t) r(t) Σ (t))) (t) m(t) + Σ
(t()s(((t),, Quv(t)= = (t)),
1=1
where r(t) luults t) and m(t lov - t).
This theorem suggests that the coupled matrices back propagated through (5) can be decomposed into set of vectors. For this reason, lower rank matrices observed in many Neural ODE applications alleviate the computational load and provide great memory efficiency (Chen et al., 2018). Substituting the ODEs from Eq. (7), into the expression in Eq. (8) yields:
R
( Fuqidt) ( Fuqiat))
(t) = Ratlty - t) + Σ
(9)
4

---

Published as a conference paper at ICLR 2024
Similar expressions for the other matrices are left in the Appendix A.2. To avoid dimensionality issues and tensor representations. the integrations in Eq. (9) are separated into each layer j of the network. We denote u', and v the parameters of layer j. and consider the preactivation vector (4),,, as an affine combination of the weights with the input to the f layer. Recall that we have considered symmetric dynamics layer-wise, implying that Fu, = Fv.. and in turn hy = h2 = Zj- OF Läy q.) for feed-forward layers, and Fu, q =
and h2g- = (u + v). This implies that Fu, q: Zj @
( BFu, qi) for convolution layers. where @ denotes the Kronecker product, and denotes the uh;
de-convolution operator. Finally, assuming that z(t)j, and OF öh, are: i) uncorrelated across time, ii) and pair-wise independent, we can express the layer-wise precondition matrix Q Quju, at time t = to
/ J2z
R
OF Th
OF q.) ( (onh, Oh, q:)
) ''dt. dt.
Qu,u, (to) = Rullto- - -t/)+
2,,2]
dt
X
(10)
1=1
Ay(to)
(ta)t))
Similarly for the other matrices: Qv,v, (to) = (
RtI - ty) and Qu,y, (to) =
A, @ B, (to). Note that for the rest of this paper, we will drop the subscript relating to the j"h layer, our analysis will still concern individual layer, but can be easily extended to the entire architecture. The detailed derivation along with discussion over these assumptions takes place in Appendix A.4.
3.3 EFFICIENT DECOMPOSITION OF UPDATE LAWS
Update Law from Min-Max DDP Recall that Neural ODE analysis and OCP principles are deeply intertwined. That motivates to view the optimization process of the Neural ODEs as trajectory optimization task. Consider the closed loop Min-Max DDP update law
Jue 8v.
K
+
8x+,
(11)
[v
where the feed-forward gains ln, along with lv. and the feedback gains Kn K2 are given by
Lu = Qull Q Qu - Qu), and Ku -iillQuu Qui Quv Qv), and K, (0 Quy QTxQx))
(12)
L. (Qur
with Q.n = (Quu Quv Qu Qvu). and @. =Quv Qvu Qui A detailed derivation of this update law is given in Appendix B.2. Setting 5x, = 0 yields the open loop update scheme for the Min-Max DDP, which will be our focal point for this study. Notice that our method yields Gradient Descent Ascent (GDA; (Jin et al., 2020; Lin et al., 2020)) with Hessian Preconditioning for Quv O, implying that open loop min-max DDP essentially a generalization of GDA. In Appendix A.6, we also provide tractable expressions for the terms in the closed loop update rule.
Remark 3.5. To generate tractable expressions for the terms in Eq. (11), leverage two basic properties of Kronecker products: 1.
(UA@ UB) & En+
(UA T
2. (A B) ())e = vec(BXAT).
Proposition 3.6. From the eigenvalue decomposition of the Kronecker products of every matrix term in Qun; Quv. derive these equivalent expression for their eigenvalue decomposition.
Q_ (UA @ UB) - S,v @ UB)
(13)
र- = (UA @ (B) -
UB)
where Suy Su, Sav the matrices containing the singular values of the matrix terms in Quu Quv- Proposition 3.7. Following Property 3.6 and leveraging the second property mentioned in Remark 3.5 we can rewrite the forward gains in 43 more compactly
in = vevc(UBGuuTT
vec(UnGGual U.) I = VeveUU vec(UnGGrUU) (14)
where Guv: Guvy Guv, Guv are matrix terms defined in the Appendix A.6.
5

---

Published as a conference paper at ICLR 2024
Detailed proofs for Propositions 3.6, and 3.7 are left in the Appendix A.5, and A.6 respectively. Algorithm summarizes our Game Theoretic optimizer.
E (xihh vilko
Forward Pres
Algorithm GTSONO
1: Input: dataset D, parameterized vector field F(s", u, v), integration time (to, tyl, ODESolver: 'ODESolve', learn ing rate v, time step At, Tikhonov regularization constants Ru. Rv
A [al Eturs A,bei B, Eic.. 8,6-)ar
2: repeat
Hackwand Pass
x(ty) = ODESolve(x(tu) to, t F), where a(to)
4: Initialize Optimizer
Q-6aa Lqiie)
5: for t in (t,,t, + At.... to to)
6:
1x(ti-,,Q(ttii (ti-1))), ().t--)))(((()))) ODESolve (x();), (tat;, Qu (1))..(((t)) 14-1, ti F) For layer j evaluate A, (4., and (4)))
Sarpled gril flll
cite scher
tright
7:
with
weightt
0DB Saluria
8:
end for
Pads
Teenor Addiioo
9:
Ay(4a) E, (4)) (l)) Σr, B,(4)AL
Antivation
Fanciion
10:
Compute fu. & from Eq. 14
11:
Update controls: +- u n7,,, t- v nhv
Figure 2: GTSONO overview, where F: forward dynamics F, and F: backward dynamics
12:
until converges
CONVERGENCE
sequential two-player zero sum games, a global minimax point always exists even f is nonconvex- nonconcave, due the extreme-value theorem (Jin et al. (2020)). Conversely, granted the updates of the proposed algorithm take place simultaneously. and due to the non-convexity of problem (3) finding global saddle point is NP-hard, thus settle for local saddle points. However, task is not straightforward, as in the simultaneous min-max setting, GDA methods may display some undesirable behaviours, such as convergence a non-critical point (Daskalakis & Panageas, 2018), or to an unstable local saddle point (Wang et al., 2019). In this regard, we provide local convergence guarantees for our optimizer to a locally stable saddle point.
that
For convenience on handling the variables we further define the vector Iu; v] E R along with [u,Q(u, v); VvQ(u, v)]. Accordingly the Jacobian
the function G 2 2n as G(z)
Quu
Quv
R22x2n Notice that after computing the inverse of G is given by: JG(z) - -Qu -Qvv
of JG(z) using the Schur complement (Liu et al., 2021b), the preconditioned update rule can be rewritten as 7k+1 2 JJGG-(22 (zx) G((s) yielding the update rule for the open loop Min-Max DDP shown in Eq. 11, for 8x, = O. Let us start with the definition and sufficient conditions of local saddle point.
Definition 4.1. Let K, = {z ||z z11l S 7. be neighborhood around fixed point z vt) Then, zt is a local saddle point, if f(u', S f(u", v²) S f(u, v*), Vu, v E K,,
Assumption 4.2, Let z = [[u;v** be stationary point. Recall that we have Quu = A B + Ryd,
and Q.v A B R,I, for R. Rv > Assuming that wlog Amin (A B) 2 -m, and
Amax (A 2 B) < M. for m. M > 0, then there exist appropriate R, > O and Ry > 0 large enough such that Amin Quu) > O, and Amax (Qvv)= < O, ensuring that z is local saddle point.
Our next step to prove local convergence guarantees local saddle point zz.
Theorem 4.3. Suppose Zo E Ky, Then for L-Lipschitz function (z),, with also L' L'-Lipschizz Jucobian of Gzz), the accumulative difference between iterates and fixed point z", as well as the total error from Q(z") can be bounded by constant
K
K
1kk - 2" 01)),
Σ -Q(zz)-Q(z))) - =
(15)
0=0
k=0

---

Published as a conference paper at ICLR 2024
This implies that the iterates converge and the total error goes to 0, K - The proof of Theorem 4.3 can be found in Appendix C.1. Finally, we proceed to show that our algorithm converges to a strictly stable fixed pointed, implying local convergence (Wang et al. (2019)).
Lemma 4.4 (Proposition 1.4 (Daskalakis & Panageas, 2018)). Assume fixed point zt = (u', v*), for which it holds fz(**) = 0 is locally stable if the Jacobian of the update rule satisfies p(J) S where p() is the spectral norm.
To leverage the Lemma above, we begin with the Jacobian of our update rule
J = 1 m(JG(z))
+ Vz (Z)
(16)
Proposition 4.5. Considering linearized ODE dynamics, we can deduce that for the derivatives Quu and Qvv, with respect to Zạ-
[uke Vul: 8 Quu 8 Qvv 0, implying (JG (JG-)(Z))) = 0.
2%
From this proposition, we can easily infer for Eq. (16) that ||]|| < 1, Vi E (0, Therefore, our optimizer convergenges to stable saddle points, implying stability. The proof Proposition 4.5 has been postponed for Appendix C.2.
5 EXPERIMENTS
We validate the efficacy of our algorithm in comparison to other state-of-the-art optimizers widely used in Neural ODE applications, and we highlight the ability of GTSONO to be successfully adapted to adversarial training methods. All experiments are conducted on a TITAN RTX.
Datasets We carry out our experiments on two image datasets: CIFAR and SVHN. Both datasets have been standardized and consist of 3x22x32 colour images, and 10 label classes.
Networks The benchmark optimizers were applied on identical network structures, for every experiment round. GTSONO was also evaluated on the same formation with two identical sets of antagonizing weights, implying double parameters compared to the benchmark models. Furthermore, to reduce the number of trainable parameters, we also consider framework containing the antago- nizing set of controls only in the convolution layers. We will refer to this architecture as c-GTSONO. More information regarding the network architectures provided in the Appendix D.
Attacks In this study, we focus on the white box -orm PGD, FGSM attacks and gray/ black- box attack CW attack to assess the robustness of our models. Every disturbance mentioned in our experiments below is applied on the standardized dataset. Details about the update rules of the attacks are left for Appendix D. We denote each model's accuracy to the clean test set (i.e. natural accuracy) with Anat; PGDs the PGD attack with a perturbation distance e that takes s steps in the direction of the gradient; and FGSM the FGSM attack whose single step in the direction of the gradient is multiplied with constant a. The CW is denoted simply as CW o0 as the for norm was considered and for every experiment the pertubation distance was set to 0.03, and max-iterations K 100.
5.1 RESULTS
Optimizer Comparison For this round of experimens we compare GTSONO with Adam SGD with momentum as first order benchmark optimizers, and SNOpt (Liu al., 2021a) as a second order baseline, on the two aforementioned datasets. The training for all optimizers was carried out using only non- perturbed images, with a batch size of 500 images on every dataset. The networks trained on both datasets was trained for 15 epochs. More details about the structure the network of each optimizer are provided in the Appendix D.
As shown from Tables 1 and 2, GTSONO outperforms the benchmark optimizers in both datasets with natural training. and that the performance gap increases as the degree of perturbation also increases. Additionally, it is also shown that GTSONO provides more robust predictions on average, well as more confident as the standard deviation smaller compared to the benchmark optimizers. Finally, observe that the C-GTSONO version of our algorithm is the most efficient and best performing providing more accurate predictions in less training time. More details about the training times and memory consumption of each optimizer are left in the Appendix D.
GTSONO on Adversarial Training Methods In this round of experiments, we highlight the applicability of GTSONO to be efficiently adapted other adversarial training methods and enhance
7

---

Published as a conference paper at ICLR 2024
their performance. More specifically, consider the Free Adversarial Training (FreeAT) scheme (Shafahi et al., 2019)) and TRADES (Zhang et al., 2019b), with SGD as their optimizer, and evaluate how these adversarial training methods benefit from employing C-GTSONO as their optimizer, instead of SGD. We perform two ablation studies by experimenting with the number of examined internal iterations for FreeAT m 4, 8, while in TRADES fix the number of internal iterations to generate adversarial examples to 5 and evaluated for 1/A = 6, 10. More details about each left Appendix D.
adversarial training method and the selection of their hyperparameters
Table 1: Average standard deviation of test set accuracy (%) on the CIFARIO for each optimizer. Anat denotes the natural accuracy. PGD5 denotes the accuracy under PGD attack, taking steps in the direction of the gradient with perturbation distance c. FGSM, describes the accuracy under FGSM attack where the single gradient step is multiplied with constant a. CW denotes the accuracy under the CW attack.
Optimizer
Anat
FGSMo.00
FGSMo.03
20 PGD
PGD 0.05
CW
0.03
Adam
78.7 + 1.1
48.2 + 0.7
30.8 = 0.5
45.1 + 1.2
29.1 + 0.3
15.4 + 0,6
SGD
77.5 + 0.6
47.3 = 1.3
33.8 = 1.3
45.8 + 1.5
29.3 1.4
18.3 + 1.6
SNOpt
79.1 + 0.4
48.7 + 1.0
35.7 + 1.0
46.8 + 1.4
32.1 + 1.4
7.5+ 1.0
GISONO
74.7 + 0.6
51.7 + 0.3
37.9 + 0.4
49.9 + 0.5
34.9 + 1.1
18.0 + 1.5
C-GTSONO
74.7 + 0.7
51.8 + 0.4
38.0 - 0.2
50.6 + 0.3
35.0 0.2
36.3 2.2
Table Average standard deviation of test set accuracy (%) on the SVHN for each optimizer.
Optimizer
Apat
FGSMo
FGS
PGD 0.03
PGD
CW
Adam
98.9 0.3
73.8 + 0.4
55.9 = 0.8
71.8 + 0.1
48.4 + 1.0
20.3 + 0,2
SGD
98.4 0.0
74.4 4 0.4
56.1 = 0.5
72.4 + 0.7
50.4 1.1
23.4 0.9
SNOpt
99.10..11
73.5 2.2
54.4 2.3
71,9 2.7
48.7 48.7
22.4 0.9
GTSONO
99.6
78.0 0.4
58.9 0.4
76.7 + 0.8
54.3
31.6 1..2222
C-GTSONO
97.3 0.2
80.8 + 0.2
65.2 - 0.3
80.3 + 0.4
62.3 + 0.3
50.5 + 0.8
Table 3: Comparison between GTSONO and SGD applied on FreeAT, for m 8,8.
Optimizer
Anot
PGD
PGD
CW
40
SGD (m=4)
76.8
48.8
48.1
6.5
C-GTSONO (m=4)
75.7
50.5
50.2
17.4
SGD (m=8)
76.2
54.8
54.2
9.5
C-GTSONO (m=8)
74.4
56.5
56.4
18.9
To avoid robust overfitting, evaluate the robustness of each optimizer, after convergence is observed. Although GTSONO presents a slower per-iteration training time, from Figure we observe that in both methods optimizer converges a local saddle point in considerable fewer iterations, due to the higher order convergence rate inherited by min-max DDP. In both defense schemes, GTSONO was found converge approximately after 10 or less epochs, whereas the benchmark models required approximately double epochs to converge. This results in less training time overall, which critical in adversarial training methods which generally take longer train due to the internal iterations required to generate adversarial samples. Table 5 present a comparison between the training time required by GTSONO and the baseline model in FreeAT for m 8, demonstrating the faster wall clock adversarial training time of GTSONO. The remaining comparisons about the required resources are left in the Appendix D. Finally, Tables 3 and 4 indicate that the adaptation of our optimizer increases the robust accuracy in both defense methods. More specifically, FreeAT benefited by the adaptation of GTSONO by a performance increase of approximately 2% in PGD attacks, whereas in TRADES obtained a performance increase of more than 4% for both tested values of X. Interestingly, the benchmark models were found to perform better under CW -
Update Laws Recall, that in this study our focal point was the open-loop Min-Max DDP, however as mentioned in Section 3, setting the pross-terme Qvu = O, we readily obtain GDA with Hessian Preconditioning. We investigate the difference in the performance offered by the open loop update rule
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison between GTSONO and SGD used with TRADES, for -- 10.
Method
Anat
PGD2
PGD
CW
20
40
TRADES(A -=6))
73.2
54.5
54.3
22.8
C-GTSONO(A~-6)
75.8
58.9
58.8
20.4
PRADESSA-'=ID
69.4
57.2
56.9
25.2
C-GTSONO(A 1000
72.6
61.2
61.0
20.7
Fressl
50
FreeST m
MADES 10
IMAEES
71.5
77.5
75
75-
75.0
75.0
Aevorery
20
70
72.5
72.5
65
67.5
67.5
SERD
60
SGD
65.0
GT50NO
02.5
02.5
25
15
30
00.0
15
20
60.0
20
25
Figure 3: Convergence comparison between GTSONO and SGD in Left: FreeAT, Right TRADES
(DDP-GTSONO,, and by the GDA update (GDA-GTSONO), under the FGSM, PGD for increasing perturbation. In Figure 4 observe that on the SVHN dataset, ODP-GTSONO outperforms its counterpart, for every examined disturbance. Additionally on the CIFAR 10 dataset, GDA-GTSONO is more accurate for small perturbations, however for increasing disturbance, it is observed that DDP-GTSONO achieves better robust accuracy against both attacks, demonstrating the superiority of DDP based algorithm.
Table 5:
Training time per iterations and epochs trained for FreeAT, m 4 (min:sec)
PGD-CTARRD
P3D-SVHN
30
100
5DA
accurcyy 194
DDP
DOP
8
50
Time per
15
Optimizer iteration (sec)
Epochs
10.
SGD
0.63
40
0.02 8.04
0.06
0.00 0.02 0.04 0,50 0.08 Pertubetior Destance
C-GTSONO 1.4
5
Figure 4: Comparison of DDP and GDA GTSONO
Limitations The achieved robustness of our optimizer comes at the price of slower per itera- tion training time and higher memory consumption, mainly due the required computation of Quu: Quv. Qv. However, as shown in Appendix D, is suggested that GTSONO is still affordable for current GPUs. Additionally, were found to be prone to robust overfitting, in some instances not only in our algorithm but also in the benchmark optimizers as well.
observed that the continuous models considered for this study
CONCLUSIONS
We present an efficient game theoretic optimizer for training robust neural ODEs, with provable convergence guarantees to local saddle points. Based on the principles of min-max DDP, our framework leverages the second order convergence of this OC paradigm by employing Kronecker factorization to decompose matrices into vectors which can be easily obtained through solving the ODE. This allows us to derive tractable expressions for the terms the update of open loop min-max DDP. Empirically, GTSONO increased the robustness and generated more accurate and confident predictions on attacked images compared to benchmark optimizers under natural training. We also demonstrated that GTSONO is successfully adapted to already existing adversarial training methods, enhancing their robustness. In future work, we wish to extend the applicability of our optimizer larger variety of network architectures (e.g. ResNets, Transformers), and explore appropriate regularization techniques, to further improve the performance of our algorithm. Finally, we wish to investigate recasting the proposed optimizer through distributed DDP schemes, as in (Saravanos et al., 2023). In summary, our work paves new ways for robust optimal control methodologies to be applied in deep continuous models to enhance their robustness.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This material is based upon work supported by the NASA Aeronautics Research Mission Di- rectorate (ARMD) University Leadership Initiative (ULI) under cooperative agreement number 80NSSC22M0070. Augustinos Saravanos acknowledges financial support by the A. Onassis Founda- tion Scholarship.
REFERENCES
Leonard Adolphs, Hadi Daneshmand, Aurelien Lucchi, and Thomas Hofmann. Local saddle point optimization: A curvature exploitation approach. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 486-495. PMLR, 2019.
Shumeet Baluja and lan Fischer, Adversarial transformation networks: Learning to generate adver- sarial examples. arXiv preprint arXiv: 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. leee, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.000 06700, 2019.
Fabio Carrara, Roberto Caldelli, Fabrizio Falchi, and Giuseppe Amato. On the robustness to adversarial examples of neural ode image classifiers. 2019 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1-6. IEEE, 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. Advances in neural information processing systems, 31, 2018.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018. 8.
Dormand and P. J. Prince. A family of embedded Runge-Kutta formulae. Journal of Computational and Applied Mathematics, 6(1):19-26, 1980. ISSN 0377-0427. doi: https:
URL http:://wwwwsceeneeeree..cco
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.66777 2014.
Yifei Huang, Yaodong Yu. Hongyang Zhang, Yi Ma, and Yuan Yao. Adversarial robustness of stabilized neural ode might be from obfuscated gradients. In Mathematical and Scientific Machine Learning, pp. 497-515. PMLR, 2022.
David H Jacobson and David Q Mayne. Differential dynamic programming. Number 24. Elsevier Publishing Company, 1970.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex-nonconcave minimax optimization? In International conference on machine learning, pp. 4880-4889, PMLR, 2020.
César Laurent, Thomas George, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. An evaluation of fisher approximations beyond kronecker factorization. 2018.
Antoine Lesage-Landry, Joshua Taylor, and Iman Shames. Second-order online nonconvex optimization. IEEE Transactions Automatic Control, 66(10):466644872 2020.
10

---

Published as a conference paper at ICLR 2024
Hongshuo Liang, Erlu He, Yangyang Zhao, Zhe Jia, and Hao Li. Adversarial attack and defense: A survey. Electronics, 11(8):1283, 2022.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1778-1788 2018.
Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex- nonconvex-comeave concave minimax problems. In International Conference on Machine Learning, pp. 6083-6093. PMLR, 2020.
Guan-Homg Liu, Tianrong Chen, and Evangelos Theodorou. Second-order neural ode optimizer. Advances in Neural Information Processing Systems, 34:25267-25279, 2021a.
Guan-Homg Liu, Tianrong Chen, and Evangelos A. Theodorou. Ddpnopt: Differential dynamic programming neural optimizer, 2021b.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiy Kumar, and Cho-Jui Hsieh. How does noise help robustness? explanation and exploration under the neural sde framework. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 282-290, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 7706006083 2017.
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
Seyed-Mohsen Moosavi-Dezfoole Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Jun Morimoto, Garth Zeglin, and Christopher G Atkeson. Minimax differential dynamic pro- gramming: Application to biped walking robot. In Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) Cat. No. 03CH37453). volume 2, pp- 1927-1932. IEEE, 2003.
Muzammal Naseer, Salman Khan, and Fatih Porikli. Local gradients smoothing: Defense against localized adversarial attacks. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1300-1307, 2019. doi: 00.1109//ACV.2219.000111
Yunpeng Pan and Evangelos Theodorou. Probabilistic differential dynamic programming. Advances in Neural Information Processing Systems, 27, 2014.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pp. 582-597. IEEE, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519, 2017.
Alexander Poznyak, Sebastian Noriega-Marquez, Alejandra Herrandez--annchee Mariana Ballesteros- Escamilla, and Isaac Chairez, Min-max dynamic programming control for systems with uncertain mathematical models via differential neural network bellman's function approximation. Mathemat- ics, 1111:1211, 2023.
Aditi Raghunathan, Jacob Steinhardt, and Perey Liang. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344 2018.
Andrew Ross and Finale Doshi-Vele Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
11

---

Published as a conference paper at ICLR 2024
Augustinos D. Saravanos, Yuichiro Aoyama, Hongchang Zhu, and Evangelos A. Theodorou. Dis- tributed differential dynamic programming architectures for large-scale multiagent control. IEEE Transactions Robotics, pp. 1-21, 2023. doi: N0.1109/TR0.20233333999
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
Wei Sun, Evangelos A Theodorou, and Panagiotis Tsiotras. Game theoretic continuous time differen- tial dynamic programming. In 2015 American control conference (ACC), pp. 5593-5598, IEEE, 2015.
Wei Sun, Yunpeng Pan, Jaein Lim, Evangelos Theodorou, and Panagiotis Tsiotras. Min-max dif- ferential dynamic programming: Continuous and discrete time formulations. Journal of Guidance, Control, and Dynamics, 41(12):2568-258, 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204 2017.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy. arXiv preprint orXiv: /2/52, 2018.
Jia Wang, Chengyu Wang. Qiuzhen Lin, Chengwen Luo. Chao Wu, and Jianqiang Li. Adversarial attacks and defenses in deep learning for image recognition: A survey. Neurocomputing. 2022.
Yuanhao Wang, Guodong Zhang, and Jimmy Ba. On solving minimax optimization locally: A follow-the-ridge approach. arXiv preprint 1rXiv:1910.07512. 2019.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference machine learning, pp. 2286-5295... PMLR, 2018.
Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, and Rong Ge. Dissecting hessian: Understanding common structure of hessian in neural networks. arXiv preprint N0X1::20/0.04261, 2020.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2730-2739, 2019.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. Advances in Neural Information Processing Systems, 32, 2019a.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019b.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv preprint arXiv:1710. 1/342, 2017.
Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 1181-1190, 2020.
12

---

Published as a conference paper at ICLR 2024
A MISSING DERIVATIONS FROM SECTION 3
We recall the formulation of the objective of the Neural ODE framework expressd in fashion easily interpretable through the prism of game theoretic OCP.
dx u, v), x(4o) = Do dE 0, u(to) 0 de O, v(lo) = "1
min max (((2,) + (" E(t, v)dt subjected to
(17)
where = x(t) E Rm, u u(t) E R", and v = v(t) E R.. It is clear that (17) describes (2) without loss of generality by taking ($,6) = (e,0). The function F(t, x, u, v) characterizes the vector field and is parameterized by a Deep Neural Network (DNN). We consider that the dynamics of the system are symetric with respect the two sets of weights (u. The functions @ and &
are known as the terminal and running cost in the context of OCP. This problem is understood as particular type of OCP that searches for the optimal initial condition of the time-invariant controls ue, Vt. In the DNN setting, the terminal cost is equivalent to the loss function, for instance Categorical Cross Entropy for multi-label classification, and the running cost is equivalent to the weight decay, or some other regularization techniques that acts the weights of the intermediate hidden layers. Next, the accumulated loss (t, x, u, v) is defined as follows:
Q x, u. v) +(x((+)) + J." ((T. (T,x((),, v(r))dr
(18)
From this definition of the accumulated loss implies that Q we can readily obtain:
0 = u() x,
(19)
dt
A.1 PROOF OF THEOREM 3.2
Considering the nominal trajectory (X. û, v), that satisfies the ODEs in the constraints of (3), we take the Taylor expansion of the terms Eq.19 keeping up to second order terms:
Tox;"
Lxu
exv
ox, gu:
zz
(xx = E(x, ū, v) + Lxxx + eudu + lvvv + 2
5u;
fux
Luu
Lue
(20a)
[ōv,]
eu
Lou
L
[8v
[ōx;"
T
Uuex
Qau
Qw7
Txx;
1
2x
Q(x,u,v) = ((xxúvv) + Qxox; + Quōu + Qvôv +
Qux
Qun
Que
8u;
2
dv:
Qux
Qun
Qun.
[v.
(20b)
We differentiate 20b, in order to utilize Eq. 19, and taking into account that obtain:
dōu
6y
0, we
dt
dt
dQ
V.
dQx
Qx döx
dQu
dōu
dQu
döv
8x,
+
du +
Qu
+
dv
Qu dt
+
dt
dt
dt
dt
dt
dt
[ox;7
dQux d a
dQuu
di
Qum
dt
Txx;
5x.
Qu. Tu
Q
Qzu
[ox;
+ 2 [8ve]
8u1,
Qus
Qus
8u,
+
0
uxr
Qun
Quv
811,
gle
Quu dt
[8v,
2
Que
Qva
Qve_
[ōv,
dt
[ox;7
aru
Qa1u
dxs di (
+
5u;
Q Qux
2
8v:
Q var
Q 2r
vn
(21)
Next, we need to compute the term
fx.
dōx
dx
F,8x + F,őu F,8v
(22)
dt
dt
dt
The second equation comes from the fact that the time derivative evaluated for fixed x yields the dynamics F(x, u, v), whereas for the first time derivative we take the Taylor expansion around the
13

---

Published as a conference paper at ICLR 2024
nominal trajectory. Finally, substituting Eq. 22 into Eq. 21, and back into 19, obtain the following ODE for Q and derivatives
dQx di
dQæx
dQxu = lxu + FxQxx + +xx
FxQx + bx,
dl Lax + Fx + 2xxP2
di
(23a)
dQu
dQuu luu FuQux + QuxFu dl
dQuv luv + FuQux + Qux
FuQx + tuu
di
dl
(23b)
dQv di
dQvv. lov FxQxv Qux dt
dQxx
FFQx++6,
lxv FxQxx Qx
di
(23c)
At this point, recall Assumption 3.1, then we easily obtain Eq. 5 for the ODEs of the second order derivatives.
A.2 PROOF THEOREM 3.4
Proof. We recall 3.4, where it was assumed the matrix Qaz(11) to be a symmetric matrix of rank R m, it may be represented as: Qae > - y.y!. where R Additionally, we had that Qua(ty) = O, (1y( = O. Then Vi [lo Ly), the second order matrices (23) that contain derivative with respect the state can be decomposed
follows:
R
R
R
(xx(()= = Σ ((()):(()), Qxu(t)= = Σ (((pp(()). Qxv(1) = Σ (t)ss
(24)
From Eq. 23, obtain:
dQxx
lxx + FzQux + Qua
dl
R
Fz
Σ (t)
+
Σ (t)))))(()))) F7
(25)
=11
R
(t
(t) Σ (:(t) (F)
1=1
In the first equality, fxx is equal to 0, from Assumption 3.1. Additionally, taking the time derivative of Qxx in Eq. 24 yields
dQxx d
= Σ ((
(q: dt (:(t) + q:(t) dq((1) "] dt
(26)
dt dt
i=1
Equating Equations 26 and 25 yields: Qxv-
dq:
Fxg.. We proceed in similar fashion for Qxus and
dQxu
fxn + FxQuu + QxxFI = FxQxu + QxxFI
dt
Fx ((()). Σ (1)) pJ
(27)
R
= Σ (xqi(t) (1) (())) + Σ qi (Fa((((
dQxv lxv FxQxx + Qxx
PxQxx + QxxFT
B Rx
Σ (1()sss()))+++++++
Σ (1(0g((0))
(28)
1=1
(t) (:(1)T
g (1)) (F)((()))))
14

---

It follows that Published as a conference paper at ICLR 2024
In similar fashion, also compute the time derivative for Qxu and Qxv-
dQxu
d
R
dq.(t)
Pi(t) qi(t) dp:(t)
]
(t)):(()))) =-
Σ i=1
(29)
dt
dt ==1
dt
dt
dQxv d
(((t)s((t)))
dt:(t)
(t) + qu(t) ds,(t)
]
(30)
dt
dt
dt
dt
dp:
ds:
Fuq:
and
= Fvq:
(31)
dt
dt
Based on the results above, we can deduce that the Hessian Que can be rewritten as follows
dQuu
lun + Fulxx Qux!IT
dl
R
R Σ p.(() (Faa((()))))
(32)
Σ (FFa)(t))D.()))
= Rul
Integrating the equation above for E [e, yields
R J. dQuu dt dt=d== J. RuIdt + 1.' Σ (Fua (() + Σ pi((t) Ep((t) (Faa((())))))
R
- dt
(33)
Recall the terminal condition Quul (tr) = 0, and using Eq. 31, we obtain
R
dp:
dp: T
Qu)(!) = Pn PL - Σ
p
+
p
dl
dt
dt
1=1
d
= -Rallly - -1) 1)+ + Σ
di (P.D7)dt
(34)
==
R,,l(ty - +)+ + Σ (1))
Therefore second order matrices can be rewritten as follows:
[ Faqidt) (s." Fuq.dt)
R
Qua(t) Ruulty -t)+ Σ
(35a)
R
J Frqidt)) (f. Fuqidt) 2r
Qtv(t) = Rullty- -t) Σ
(35b)
1=1
Pug.dt) (S, Fuqnit))
(!) = Σ
(35c)
i=1
R Qux Σ (." Fxqidt) (f. Fuqiit) (xx((t=)) ((xx)))) Σ E( Fxqidt) (f, Erq(at) (35d)
i=1 1,
A.3 LAYER WISE PARTIONING
To avoid dimensionality issues and tensor representations, the integrations in Eq. (35) are separated into each layer j of the network. We denote a', h2, u', and v as the activation, pre-activation (linear combination), and the parameters of layer j. respectively. Furthermore, preactivation vector (t), as an affine combination of the weights with the imput to the jth layer. We recall that have symmetric layer-wise dynamics with respect u, and v, resulting in the
consider the the
15

---

Published as a conference paper at ICLR 2024
partial derivatives of the preactivation vector the 1th layer with respect to the control variables: hy h2 ay. and h = (u + v). This implies that:
OF
4: (u + V)T(
qi) for Feed Forward layers
(36a)
Th
Fu = aF
OF q.) for Feed Forward layers
(aj @ l)q: = 3j @ Th
(36b)
Thj
OF
Fx, qi = (u+ v)*(
qi) for the Convolution layers
(36c)
Th
0Fu;
q,) for the Convolution layers Fu, q: = =a 0h
(36d)
where @ denotes the Kronecker product, and & denotes the de-convolution operator. The layer-wise notation used above was adopted as manner to circumvent dimensionality and tensor representations issues. In this vein, the integrations in equations (35) are broken down into each layer j of the network structure, where using the expression from 36, we obtain
fo
OF Oh
Exq:
dt
qx
dt,
11 + v)T
(
qi)
dt. dt....
(37a)
4
4
b
to
OF
Fug:
di
qa
dl.....
a
4
dt
....
(37b)
0hj
to
50 Fe dt,....
1o
OF
Foq:
dt
a, (
qb)
dt....
(37c)
Jty
"h
A.4 DERIVATION OF EQUATION 10
Following the layer-wise representation, We recall that to derive 10 these expressions, we first use the Kronecker product property: (A B) & D) AC x BD. Additionally, during the process of this derivation some approximations necessary. The following assumptions necessary in order to derive
Assumption A.1. Suppose:
a()) and g()) are uncorrelated across time
* (t), g (t) are pair-wise independent (Liu et al. 2021 NIPS)
For the sake of brevity we demonstrate the analytic proof only for the Hessian Qun. the others follow similarly.
(S, (iuqutt)
Quyu, (to) Rto +
Σ
Fuq.dt)
1=1
OF a
q: dd) (f. ay @
OF Th, q.)
Rudl(to (tl((ott))+ Σ
Jh,
R
/' aa
OF
OF
2 RuIlto- + Σ
91)
a, @
Oh q: q:)dt)
Oh
(38)
iml
Σ /'' [aya]
R
OF qi)
OF
RR,I(to -t,) 2t,)+ +
Oh, 4:) 9)) )))dtt dt
Th
i=l
to
d0
OF q,) Th
OF
R.o - tt) +
[aaa] di
(on qi)"
Th,
A,(1)
B,(
16

---

Published as a conference paper at ICLR 2024
Remark A.2. We begin by underlining that both assumptions widely adopted (Martens & Grosse (2015)). The first assumption admits that a, (t) @ (t) are temporally uncorrelated. This is necessary to yield tractable Kronecker matrices for second-order optimization. Although, it may be strong assumption, some cases, it has been empirically observed that the uncorrelated temporal assumption may yield better performance Laurent et al. (2018) 1)). The second assumption admits that ay()) and gy (t) are pair-wise independents, which has been verified empirically in Wu et al. (2020).
A.5 PROOF OF PROPOSITION 3.6
Proof. We begin by trying to simplifying the expressions for Quu and Qvv. Setting: (MA @ ΣB) Sue. (2iA & BB + Aa) = Su, and (2' @ DB + Ao) = Se. we obtain:
Quu (Quu Qun
-. UUB)
(U (UBBUN55|UABB0011UU000333,CCC,,
(39)
-UA UU
u
Sao) @UB)
(UA @ UB) S S5A
S.. @ UB) At this point, using the following Similarly, for @v = (UA 2 S properties of the Kronecker product, based on the eigenvalue decomposition:
(A 8 B = (U, 8 Un) En + A)(UA & TB)
(40)
(A B + - (UA (B) & EB + (UA UT)
(41)
we can easily obtain the inverse of the matrices in 39 as follows:
Q (UA @ (S, STuS, (UA UB)),
Q (U, @ (S, ( (UA Un)
A.6 PROOF OF PROPOSITION 3.7
Proof. We begin by considerin the closed loop Min-Max DDP update law
du: 8v:
4u
Kn
=
1v
Kv
5xL.
(42)
where the feed-forward gains In along with lv. and the feedback gains Kn, Kv are given by
4, = Qui Qn), and K,, = Qui(Qus - Quv l = Qud QuiQu Qv), and K, = Qn (Qvx Qui Qir Qxx) Qvx)
(43)
with Qun = (Quu
Qvu), and Qu = (Quv - Quu
At this point, will leverage the following property of the Kronecker products: (A @ B)r vec( BRAT), where R = vec vec-l(r) (r) denotes the inverse vectorization of r. to express lu and & in compact manner.
Setting for brevity: Quxox, = óqn. Qvx = óqv. and substituting into 42 the expression for 13, the ferd-forward and feedback gains in (43) can be compactly rewritten follows
17

---

Published as a conference paper at ICLR 2024
l -UA 8 (UB (Sw Su (S (SSSSS))vv((((
Buy
vec - -UA (U))UB) (Sw Sw - (UQUUU vec(UBGCuoli)) veo(UBGuulT))
gan
(44a)
4 -UA @ UB) (S. - Sw (S (NeOS_"N
8in
vec (TQQUA)) vec(UpGruU1) - ve - -U, & Un) (S., - Sun Sp vve
gan
(44b)
Ku -UA @ UB) (S - S, Swe) SS ( 16qv
You
(44c)
- -UU 8U)) (Su, Sun v Suo) -1 -6qu = vec(UnYunU))) vec((UNNn.U
K, -UA @ UB)((s) (S. - Sur Suo) (SuuSp1)80)
(44d)
- -UA Un) (Sw, San Swo) -lqq = vec(UnY(uU3) -ec(Un
Yuv
where vec vec-!(Qu) Qi vec = Gij. and vec (y.;) Yi, for fu, v) denote the inverse of the vectorization operation of the corresponding vectors. The only required inversions are of the diagonal S matrices which computationally inexpensive, rendering the computation of the terms 44 tractable to compute.
B DIFFERENTIAL DYNAMIC PROGRAMMING IN CONTINUOUS TIME
B. PROBLEM FORMULATION
Consider a min-max game problem, with dynamics described the following ODE:
dx(t)) F(x(t), u(t), v(t), x(tu) 20
(45)
dt
where xzt) E X' is the state of the dynamic system at t E [fo, tyl, and u(t u(t)) U1 C and vl,, x()) U2 C V denote conflicting controls, with U, and U2 are convex sets containing
admissible controls of u and v respectively. For the sake of brevity, we will denote u(4, x(()) = and similarly v(t, x(t)) = v. The goal this OCP scheme is to find non con--niiiipaaing strategies for both players. Additionally, define the cost function follows:
Ju, v) olt,, xx,)+ 1'' e(x, t.ddrr
(46)
where Φ: : [40. [y] X X' R4 is the terminal cost, and E: X' x li x V x [4. - R4 is the running cost incorporating the and the control cost for both players. The conflict between the two players enters in our problem formulation one tries through control u to minimize the above cost function. whereas the other one tries to maximize through control v. We proceed to define the value function for our problem as the function expressing the minmax value of the cost function at time = to and To-
V(oo xp) = min max (((,, + 1'h l(x, v, t)dT ()dT
(47)
18

---

Published as a conference paper at ICLR 2024
Using the Bellman principle, write (47), as follows:
o(tz,xi,) ( (+dl l(x, v, 1ddT fdd u(x, u, v,
v(1, min max ((tt-t))((tt v(t
setde
{olty,xe,) f"d l(x,, u, t)dr}
min u(t-++d() v(t-+e+d()
max
max u(t+dl+ ( ((++d+++t))
min
eede
f(x, u, t)d
A-dtt v, + V(L + dl, xe+ae)
min ((t) v(t-
max
(48)
Then we can readily obtain:
= min max f(x, u, v.t) + dV
(49)
dt
We express dV follows:
OV at
0V 0x,
OV at
0V Oxt
dV
dl
dx;
dl
Fdi
(50)
This form enables us to obtain the min-max Hamilton Jacobi Bellman through substitution to (49):
2v
ov T
0 min max [(xx u, v,
F
=>
01
0x,
(51)
oV
min max [(x((t, u, v, +
F
aL
Ox.
with its terminal condition being: V(4,; x1,) xt,):
B.2 BACKWARDS PROPAGATION
First, we consider equation function Q as: Q(x, u, v) E(x, u, v,t) inside the minimization in (49) along with function Q up to second order terms, with respect to the nomimal trajectory (x, ú, v) and we obtain:
at We expand the terms
8(x + 5x,, π1 + ou, v + ov.)
1
[dx;" du; [dve]
Fxx
lxu
Exv
[xx; ou:
(52)
((x, v. [xx eJu L8v, +
fux
fuu
fuv
2
Llux
Lvu
lvv.
[ōve]
1 V(t + dt, + 6x:) = (tt, + Vx x)ox; + 5xx Vxxxxx
(53)
2
At this point, we take the derivative with respect to time, the expanded expression of the value function from (23). We note that V, Vx. Vxx are only functions of time, since they are expanded with respect to the nominal xt). Additionally, we also notice that from the system dynamics the infinitesimal perturbation around the nominal trajectory 8x; are also function of time, so the differentiation of the second and third term below follow the product differentiation rule.
dV di
dV
(Vxōxe) + (- Vxxxx.))
x=)(() + di
di
di
==((t)
v=v(t)
(54)
dV
dVT
dox;
doxt dt.
dVxx 8x. + 6x.
dôx;
+
ox
+ VT
+
Vexdxe + ox. di
di
dt
di.
di
19

---

Published as a conference paper at ICLR 2024
Returning to the system dynamics, write déx as follows:
F(x, dox d (x(1) (x(())x(1)) - -
di dl
U U, v,t) + Fxxxx Fudu, Fedv. F(x, u.V,t
(55)
Fxxx, + Fuðu, Frôv:
where Fx, Fu, Py stands for Fx (x, ú1 v). Pu (x, ūt V), Fo( respectively. If we substitute (55)
in (54), obtain:
dV
dV
dVT
ox: + V (Pxxx: + Fudu, + F.ovi))
x=x(0)+
dt
di
dl
u=t(t)
v=v(()
+ 2 ([Fxxx, + Puuu, + F. + 6x] dVxx ox, + ox ((Fx(( + Fuōu, + F.ovi)))
dt
(56)
Now, if we substitute (56) and (52) in (49), separate the terms that differentiate with respect to time the value function or any of its derivatives with respect to the state, we obtain:
dV
dV
dVxx
dxt-
_xx
8x:
dt
di
2
dt
dxx;"
fxx lxu
lxv fuv
[xx;
1
min max
((8, û. v, + 46x
LJuu + Ljov
+
ou,
lux Livx Evu
Luu
5u;
2
[8v
l
[8v
1 [öx;]
T
F Vxex
+ Vxx Fx
Vxx
Vxx Fy
Txx;
+ V Px0x + V/2 Pu.u + Prov, +
du;
FT Vxx
ou;
[öv,
FJ Vxx
0
0
_ōv,
min max Au Av
((x,ū, (EI + VI Fx)xx; (ET + FuJuu + (ET + VIF()8v
oxx1
[xxx + F + Vxxexx lxu + VaxFu lux FTV xx fxx Vxx
Vxx luv lvv.
oxi
1
+
Ju:
byu Evu
du, ōv,
2
[vv,
(57)
Following the definition of Q and (49), we can expand Q up to second order terms as following:
ox, du, 8v.
Qxx
Qxu
Qxv
ox
min max [Qxxxx,, u, v, + 01ox + Qiou, + Qíov, +
1
Qux Qvx
Quu
Quv
du;
2
Qvu
Qvv
ōv.
(58)
Therefore, we can equate the terms from (57) with the terms of the quadratic expansion of (58), and yield the following:
Qu(t) (xx(t) i(t),
Qx(t) = (4] + VxF=)* = + FVV
Qu(t) (4I + VI T= = FFV
Qv(t) = (4T + T= FTVx
Qxx Exx + VxxFx + F Xxx
(59)
Qxu (t) = fxu + Vax Fu = QTx
Qxv (t) Exv + VxxeFF QT
Quu (t) = tuu
Qv (t) lyv
20

---

Published as a conference paper at ICLR 2024
Additionally, taking the derivative in (58) with respect to ou, and ov, and setting them equal to 0, yields:
óu - Q (Qxxxx + Quvôv; + Qu)
(60)
5v (Q + Quaōu; + Q+)
However, we notice that the optimal control of the opponent present in the expression of the optimal control of each player, we try to eliminate this dependency:
Qui (x + Quv( (Qxx + Quvóui + Q+)) Qu)
+ (Qmm Quv 2 )uu' = (Qux Qw + Qur Qu Qu
Qn
Qu - (Qux - Quv Qvxxvxxxxx) (61)
-ōu = (Qmu - Quv Q (Q
=ōu = lu Kuoxe
where lu = (Quu
Qnu) Qu Qvx)(x)).
Qu), and Ku
(Quu
Qv) ((Qux Similarly we can express dv" = lv + Kydxi, where equivalently the coefficients lv, and Ky are defined as: lv (Qvr QuaQui ((Qvx Quv ( From (57) and (58), the value function and its first and second order derivatives with respect to x are expressed through the following backward ordinary differential equations:
Qv), Ky = (Qvv
dV
dVx dt
1
dVxx
8x;
6xp
8x:-
dt
2
dt
[ōx;"
[xx
Qxi Quu Qvu
Qxv Quv
Txx; Su: [ōv
1
min max Q(x, ū. v, tsss) + Q18x + QTou + Q,óv,
du: [8v,
Qux 12vx
2
Qvv
QQ ū, v.t) + Q + QT + Kūox;)
QT Kxoxx)+
5x;
Txx
Qxu
Qxv
Ox,
+
(In + K )x,)
Qux
Qu
Quv
(Ln + knxx,)
(l, + K k2,),,)
Qvx
Qvu
Qv
[ly + Kxx
(62)
dv
E+1 + 18Qu + 2mm'n + Qvvll + Quvd
d
dV=
Qx + KIQ + KJQ, + Quch + Quxlo + Kuuuh
di
+KTQuvlU + KTQvula + KJQul
(63)
dVex
Qxx KTQuu + QT KTQux + K KJQuuKn+ +KTQur + KIQ + K KvQuKK
d
under terminal conditions:
V(tr) = ,(x(t)),tt)
Vx(ty) = (x(x(t))
(64)
(xx(ty) = &xx (xxx(x(()),))))
C CONVERGENCE
We begin our convergence analysis for GTSONO with the definition of the global saddle point, which will prompt us to define the local saddle points.
Definition C.1. A point (u", v²) E R is global saddle point of a function Q(-,)). if we have ʃ(u", v) S f(u,, v*) < f(u, v*), Vu E I.. and v E R".
For convenience on handling the variables we further define the vector [u E R along with the function G 2m - R2n as G(z) =
V V.Q(u.v) Accordingly the Jacobian of G is
v))
21

---

Published as a conference paper at ICLR 2024
Quu Qvn
Quv -vvv
given by: JG(z)=
E IR 2nx22n For the inverse of the Jacobian using the Schur
complement Liu et al. (2021b), we obtain:
Qui QuaQua
Quilur
JG
(65)
QQi
where Quu Quu QuvQui QivQum and similarly Qe move the update rule using the inverse of JG as follows
Qvv Qun Qua At this point we
U
uu
Qui 2
QuiQuv -Q
V QuQ v) Qu.
(66)
Vk+1
Vk
which yields the update rule for the open loop Min-Max DDP shown in section 3. Using the definition [u:v], v.. we can write Eq. (66) more compactly as 2x++1 2 (24)G=(24
Definition C.2. Let K, = {z ||z z-1 11 S 7). be neighborhood around fixed point x = vu Then, 2 local saddle point, if f(u,, v) f(u", v*) [(u, v*), Vu, v E Ky-
Two fundamental assumptions regarding the Lipschitz continuity of Quu, v) in region K;, along with the Lipschitz continuity of the Jacobian of G(z), i.e., JG(z), will enable to derive the bounds for the iterates and function Q.
Assumption C.3. We assume that in this region, Q is C2 function and that Q, and JG are Lipschitz continous Vx, y E Ky, i.e. we assume the following
[1Q(z) Q(2))))) )ll & L L'I 2=|l
(67)
[|]((2)- -GC(2)-G(((Z) [ll L 10--2 zz'l
(68)
Additionally, consider constant h, for which it holds that: 1, 2 max (JG-1), which is defined the greatest singular value of inverse of the Jacobian of G. Now, using constants h and L, let us define the radius of region K, y < 2h
We proceeding by mentioning a Lemma which will deemed helpful in the derivation of bounds for consecutive iterations.
M Lemma C.4 (Lemma (Lesage-Landry et al. 2020)). M E Raxx is invertible, then 3h > 0, s.t.
1 h
||M~
||Myl 2 h|yy],, Vy e E".
(69)
In our analysis, we assumed a fixed point z", and local region K,, such that 1|z - -zz|| S 7, and followed the following steps:
1. Provide the sufficient conditions under which it is guaranteed that this fixed point is local saddle point in region Kg.
2. Show that optimizer indeed converges to this fixed point, granted it is initialized within Kg,, which under the stipulation that the conditions in 4.2 are met has to be a local saddle point.
3. Show that our optimizer avoids unstable fixed points.
We start by stating the following Lemma.
Lemma C.5 (Lemma 4, Adolphs et al. (2019)). For a function Q, at least twice differentiable and not necessarily convex and not necessarily concave, point (u', vV) K., a local saddle point if the following conditions are met: Necessary condition: VQ(u', v*) = 0, and Sufficient condition: Quu O, and Qvv
Intuitively, this Lemma expresses that Q must be locally convex with respect to u and locally concave w.r.t. v. at least in region K,, for z to be a local saddle point. Then following Assumption 4.2, holds that for appropriate selection of fu > 0 and Rt > O, we can ensure the positive definiteness of Qua. and the negative definiteness of Qvv. It follows that if this condition is satisfied then must also be true that Qu Q O, equivalently G(z*) = O. Now, we proceed with the following Lemma from which we will derive useful inequalities that will enable us to bound the iterates and the accumulative error of Q.
22

---

Published as a conference paper at ICLR 2024
Lemma C.6. Suppose the following conditions met
1. The initialization falls within a ball of radius 7 from z", namely 2o E Kg
2. Assumption 4.2 is satisfied, implying that z" is a local saddle, and thus G(z*) ([z²))== = 0.
3. [Lemmaa 1; Lesage-Landry et al. (2020)]: For square and invertible matrix JG, 3h > 0, such that (2')1 -
4. The Jacobian of ((z) is L-Lipschitz: [[] - (()))())| S L||z z-1l
Then the following inequalities emerge between nwo consecutive iterations
3L
M 2=1l < | z
|| 2-- <
Z4 - 1=112
(70)
2h
Proof. We begin the proof with the update rule, and subtract the fixed point z from both sides.
Zk++1 Zg (Z,
2k+1 2² 25 2 =J- " (xg)
(71)
2s z (n) + ( (2G
Zs z + 1 - G(2+)
where we used the fact that G(z*) O. By the fundamental theorem of calculus,
/ JG(zz + T(2= zz))(zz =z)ddr
2k+1 2t =2; 2k=- JJG
= JG (Z)((z))GG(z)(e - -2') + (G-) / nJG((x + T(2"" -xx))(²* 2z 2x)
JG- (Z+) I' JG(ZN)(Zu 2d)d + JG (zx) l' JJG(Zz + T - -2x))(z² 2x
JGG 1/' (nJGIee + =(22 mx)) (G((z)) (2-- (x*-xx)dt
(72)
Consider [] +
- - s || + +T(2" - 2x))ll, vn E (0, 1]. Therefore, one obtains
the following inequality
|| I (JG(za T(Zu - z²)) JG(Zz))(zz zz dd(l
Zk+1
z'1l S ||
11 1// TL|lzx - x |ddll
$ |||
(73)
< J|
11
I2x 2=1
At this point we use Lemma C.4 to derive an upper bound on the norm of the inverse of the Jacobian of the operator G. For arbitrary vector y F.
||]G(z))yll
||]G(z)) + JG(z")y - JG(z")>
> |JG(z*) [JG((z)) JG(z*)yl
2 [IJG(z")
[lJG(Z)) [xxl
(74)
> h||y|| L| 12z zz'l |lxll
(h- L||z -z" ID|||l
Thus since By E IR",, such that []G())) 2 (h L| z" IDlly] from Lemma C.4 we have that
1
|| 5 h L Izz z=
(75)
23

---

Published as a conference paper at ICLR 2024
Substituting Eq. (75) into (73), we obtain
2x+1 - -2"ll 5
12 - 2²112
(76)
2(h I. [ z=l))
By construction, we have that ||z - z < identities.
2h
which enables us to obtain the following two
3L
L2 z-1l
|zx+1 2011
1 z
2(h
2
(77)
2h L
3T.
20 = 1z x=1|
2(h
L
2h
Trivially from Eq. (77), by bounding only the denominator, we obtain
L
|zt+1 2"11
125
2(h L||zk
2-
(78)
31.
< 2h
1
z"
2=11
2(h
L
2h
3L
yielding Eq. (70).
Now, in position to prove convergence fixed point z".
C.1 PROOF OF THEOREM 4.3
For completeness restate Theorem 4.3.
Theorem C.7. Suppose zo K-,. Then for L-Lipschitz function Q(z), with also L' L'-Lipschitz Jacobian of G(z), the accumulative difference between iterates and fixed point zt, as well as the total error from ((z") can bounded by constant
2x (0()
Q(zx)) - Q(x)))
(79)
Proof. Let us first show that Σ the previous lemma obtain
||z
z²|1 is bounded. By using the second identity obtained by
k=
K
z²11231.
Zk
Zk-1
(80)
2h
We want to make the RHS of the inequality to have the same index the LHS
K
3L
3L
3L
2=112
12g
1-
(Co Cx)
(81)
2h
2h
2h
where 6g ||z,- 112 and cτ ||z z 212 the individual error term at the initialization phase and the th iteration respectively.
From this definition, it follows that co. eg > 0, hence 27. eg. Moreover, we observe that in the RHS we have ||z z 12, using the assumption that we fall into the region K,, where
Σk | z=| 1= - 3L
2h (eo ex) < E | z 1|2 32 +
3L
2h
2h
||z z" < 7 yields
K
K
3L7
3L
1
z'1l 111 <
Σ
1zk - z²11
+
eu
(82)
2h
2h
km
24

---

is bounded by constant and Published as a conference paper at ICLR 2024
Therefore, have
K
317
3L
z'
1zz
z²1l
+
eg
2h1
2h
k=1
K
317
3L
)112 - 2011
2h e0
(83)
2h
k=1
K
Izz
z²|l
<
(1
34
km
2h
where 8 = 3l In the last inequality we can divide with 3Ly
since from our assumption
2h 7 37. Therefore, since limo-+oo Ek=o Izx z=|l
2h
(1- )
12 == |zz=1 1 z= it shows that ||z zz || for k - oc.
Now, recall that the function (zz) is also L'- L'-Lipschitz at point 2", thus
QQ(z)) Q(2)) S
T/ 201l
(84)
Hence, readily by combining the last two inequalities we obtain
K
[Q(zx) - Q(2"))( <
O(1)
(85)
(1
31x
2h
This implies that the iterates converge and the total error goes to O, as the number of iterations increases to infinity. We proceed to show that if our algorithm converges a strictly stable, implying local convergence (Wang et al. (2019)). Recall that for fixed point 2 (u", vv), for which it holds
fz(=*) 0 the Jacobian of the update rule satisfies p(J) < 1, then this point is assymptotically
locally stable [Proposition 1.4 (Daskalakis & Panageas (2018))1. The Jacobian of our update rule is given by
J=
[JG] V(J
(86)
C.2 PROOF OF PROPOSITION 4.5
We restate the Proposition for completeness.
Proposition C.8. Considering linearized ODE dynamics, we can deduce that for the derivatives Quu and Qv with respect
v): 8 d2 Qmm Quv 0. This implies that (JG-1) 0.
Proof. We begin with considering the linear approximation of non-linear dynamics along the nominal trajectory, an approximation upon which the entire DDP scheme is based Pan & Theodorou (2014). This enables us to set second order terms with respect to the dynamics equal zero, namely Fm = Fvv = Fnv = Fxn = Fxv Fxx = O. Note that this approximation was also held to derive ODEs in Eq. (23).
Proposition C.9, Recall the ODE we derived for the decomposing Qxx qi for {1, (1,..., S m}
(t)(t) (() with terminal condition qi(1)) yt. We can express as
dt
qi(t) = =y:exp S.,² -Fdd) x exp (xp(ff"'F2tt)
(87)
Using this expression for (t). we can calculate partial derivatives with respect to x, u, and v.
dq:
0x -(
Faxdr) exp (f, -Fadr) = (Ca (t) =0
(88)
25

---

Published as a conference paper at ICLR 2024
Similarly for the other derivatives,
Ōq:
(L,² -Fxccr
- exp fty
du
(89)
0q:
Px exp J.." -Fxd7)
dv
At this point, we can easily move to the expession for the Hessians in Eq. 35, and see that differentation of these expressions with respect to any set of weights results
oQuu du ou bal o Faail F
=( / Fuuqi Faqu,u) / Fuqi)*+( Fagil Fuuq: + Fuqi,u)
(90)
-0
Similarly, it follows for the derivative of every second order matrix with respect to u. or v resulting in JG-!.
From this proposition, we can easily infer from Eq. (86) that || < 1, Anj E (0, 1). Therefore, holds that the spectral norm of the Jacobian be less than and ensuring the stability, showing that our optimizer convergenges to stable saddle points.
D EXPERIMENT DETAILS
Networks and ODE Solver The ODE solver we used for every experiment is the standard Runge- Kutta 4(5) adaptive solver (dopri5; Dormand & Prince (1980)) implemented by the torchdifteq package, with the numerical tolerance set to le-3, and fixed integration time [0, 1] for all experiments. All experiments are conducted on the same GPU machine (TITAN RTX) and implemented with pytorch
Model configuration Here, we specify the model We will adopt the following syntax to describe the layer configuration.
* Conv(input, output, kernel, stride)
FC(input, output)
Using this notation, Table presents the architecture of every model.
Table 6: Network Structure
Network
Discrete Convolution Layers: ReLU(Convv64,, 64, 3, 1)) -> ReLU(Conv(64,, 64, 3, 1)) Conv(64, 64, 3, 1)
Continuous Convolution Layers: ReL ReI.U(Conv(64, 65, 3, 1)) Feed Forward Layers: ReLU(FC(2304, 500)) FC(500, 10)
Note that for our optimizers, a game theoretic layer would be implemented as ReLU(Layer, + Layery). Recall that in GTSONO every layer has a game theoretic counterpart and in c-GTSONO we have only convolution game theoretic layers.
Dataset Recall that the datasets examined for our experiments are the CIFARIO, and SVHN. Both image datasets are preprocessed with standardization. CIFAR10 contains 50000 training images and 10000 test images, whereas SVHN contains 73257 digits for training and 26032 for testing. PGD and FGSM require computation of the loss gradient. For this reason, to facilitate and accelerate testing against adversarial attacks, we utilized 40% the samples in the test set of CIFAR10, and 20% of the test set SVHN.
26

---

Published as a conference paper at ICLR 2024
Tuning process For the comparison among the optimizers, the tuning varied for each optimizer. More specifically, for Adam and SGD only the learning rate was evaluated. For SNOpt, we tuned its learning rate and its regularization term, while the 2 variations of GTSONO were tuned against its learning rate, the regularization constant for the two sets of weights. We perform a grid search for the tuning of the nyper-parameters for every optimizer. The tuning process took place by optimizing the performance of each optimizer in the clean test set and evaluating its robustness when convergence was noticed. Robust overfitting was observed when evaluating the robustness of the models many iterations after convergence, regardless of the optimizer. Table 7 summarizes the hyperparameters evaluated for each optimizer, along with the examined values.
Table 7: Hyperparameters for each optimizer and their examined values
Optimizer
Hyperparameter
Value
Adam, SGD
Learning rate
2.5e 5e 5e
7.5e -4, 1lle le 3,55e 2. 7 7.5e -3 .1022
Learning rate, Regularization constant
2.55 55
2.5e
7.5e -4 le 2.55 7.5e -3 le-2
SNOpt
{le-2, (1e-2,2.5e 5c
Learning rate,
{le 2.5e 5e
7.5e
le
2.5e 5e
7.5e
- ,11-2
GTSONO
Regularization constants Ru, Rw
2.55 5e
le -3. 2.5e
-3 5e
-3
le
2.5x 5e -2
Attacks Recall that in this study, we focused on the white box &oe -orm Projected Gradient Descent, Fast Gradient Sign Method attacks and gray/ black-box attack CW attack to assess the robustness of our models. For foo PGD attack, the update rule to generate adversarially attacked examples is given follows
x' + I1 (x + (ign(V ((F()()()))))))))
where BE, is the projection operator on the too norm.
For the FGSM attack, the update rule to generate adversarially attacked examples is given follows
t x + f/2 sign(V (Fx(((()))))))
For the update rule to generate adversarially attacked examples through CW, refer the interested reader to Carlini & Wagner (2017).
Adversarial Training Methods For the second round of experiments explore the versatility of GTSONO in its ability to be efficiently adapted to other adversarial training methods such as the Free Adversarial TFaee))g(FreeeA) scheme (Shafahi et al. (2019)) and the TRADES objective function Zhang et al. (2019b). In both of these frameworks, SGD optimizer was employed as the standard optimizer.
More explicitly, FreeAT proposed the modification of training each minibatch times providing strong adversarial examples, while also using the perturbation from the previous minibatch warm- start the perturbation for the new minibatch. This is the hyperparameter we performed our ablation analysis, by setting m equal to 4 and 8. Algorithm summarizes the FreeAT method.
From our results, it was shown that GTSONO was able to outperform the benchmark optimizer in this adversarial training method, and less epochs resulting in an overall faster training time, leveraging the faster convergence of our second order method, thus rendering it a more efficient and effective method for this adversarial training scheme.
Furthermore, TRADES proposed new objective function which computes the KL divergence, between natural and adversarial images, multiplied with a constant 1/2. This the hyperparameter against which the ablated analysis. Algorithm summarizes the TRADES formulation. It was again demonstrated that GTSONO was able not only to outperform the benchmark optimizer in terms of raw performance but it was again shown that it is the more efficient option for this adversarial training method, as it provided superior performance less epochs, reducing significant the overall training time.
27

---

Published as a conference paper at ICLR 2024
Algorithm 2 Free Adv. Training Algorithm
Algorithm 3 TRADES Algorithm
1: Input: Samples X ~ D, perturbation 6, train-
1: Input: Step sizes 71, f2,, batch size m, number iter- ations m in inner optimization, perturbation network with parameters 0
ing steps m, learning rate
2: Initialize 8, 0
3: for epoch 1 N..N
2: repeat
4: for minibatch B C X
3: Read from minibatch B (xi....,xx) xx}
5: fori= m
4:
for fori=1 1..
6: 90
- [ VVl(( + y, 6]]
5:
X, <- X, + 0.001. I). where N(O,1) 1) is the
7:
9ldv Vxl(x + y;
Gaussian distribution with mean and identity variance for =...
Update 0.0
niae
6:
9:
Calculate
+ esign(gax)
7:
x f - (Visign V T[fo(4, x,)) x,)
10:
5 clipó, e)
8:
end for
11:
end for
9:
end
12: end for
10: - +
end for
11:
until converges
For our experiments, we selected the perturbation distance be equal 0.03 in both ablation studies. The number of internal iteration in FreeAT is the hyperparameter we performed our ablation analysis, by setting equal to 4 and 8. In our adaptation of GTSONO in TRADES, set the iternal iterations to be equal to 5, and we performed ablation study with respect to constant X.
Limitations As shown in the Tables above, the achieved robustness of our optimizer comes the price of a slower per iteration training time and higher memory consumption. This attributed to the decomposition and calculation of the three matrices involved in the open loop min-max DDP update Qun Quv, Qvv. Although, memory consumption indicated in Tables 8-12 suggest that GTSONO certainly affordable for current state-of-the-art GPUs, its requirements for memory usage and its greater training training-per-iterenn per-iteration times motivate us to experiment with recasting the proposed optimizer through distributed optimization distributed DDP schemes.
Additionally, another shortcoming noticed not only in our method by in general in every neural ODE model was the robust overfitting. More specifically, after convergence was observed, even for stagnant natural accuracy, we observed that their robust accuracy decreased as the number of epochs increased. This naturally creates future lines of work of experimenting with more network structures and types, well regularization techniques to attempt to reduce this robust overfitting.
Training Time and Memory Consumption At this point we compare the resources required by each optimizer.
Optimizer Comparison: For this round of experiments, we compared optimizers in the task of image classification, both in the natural test set and recorded their natural accuracy but also their robust accuracy under various attacks, such as FGSM and PGD for various degrees of perturbation. Every optimizer was trained with batch size of 500, and for 15 epochs. Tables and 9 present the time and consummed memory by each optimizer on CIFAR10 and SVHN respectively.
Table 8: Training time and computational resources for Optimizers on CIFARIO
Total training time (min sec)
Memory Consumption (GB)
Optimizer
Parameters (10%)
Adam
1.25
2:53
2.13
SGD
1.25
2:37
2.13
SNOpt
1.25
3:01
4.58
GTSONO
2.51
4:01
6.71
C-GTSONO
1.35
3:53
6.71
28

---

Published as a conference paper at ICLR 2024
Table Training time and computational resources for Optimizers on SVHN
Total training time (min sec)
Memory Consumption (GB)
Optimizer Parameters (10%)
Adam
1.25
4:13
2.13
SGD
1.25
4:09
2.13
SNOpt 1.25
4:23
5.50
GTSONO 2.51
6:05
6.71
C-GTSONO
1.35
5:22
6.71
2. Adapaation to adversarial training methods: As mentioned previously, we evaluate the robustness of each optimizer in each measurement after convergence has been observed. Figure 3 illustrates the faster convergence of GTSONO compared to SGD in both defense schemes. More specifically, taking into consideration the graphs Figure 3, we list the epochs for which each optimizer trained.
FreeAT m = 4: SGD 15 epochs GTSONO 10 epochs
FreeAT 8: SGD 40 epochs GTSONO 5 epochs
TRADES À 1/6: SGD 20 epochs GTSONO 10 epochs
TRADES 1/10: SGD 20 epochs GTSONO 10 epochs
Table 10: Training time and computational resources for FreeAT, for
Training time
Total training
Memory
Optimizer
Parameters (10%)
per iteration (sec)
time (min:sec)
Consumption (GB)
SGD
1.25
0.53
13:15
2.13
C-GTSONO
1.35
0.70
11:39
6.09
Table 11: Training time and computational resources for FreeAT, for m
Training time
Total training
Memory Consumption (GB)
Optimizer
Parameters (10%)
per iteration (sec)
time (min:sec)
SGD
1.25
0.63
64:45
2.13
C-GTSONO
1.35
1.1
8:21
6.09
Table 12: Training time and memory resources for TRADES
Training time
Total training time (min:sec)
Memory
Optimizer
Parameters (10%)
per iteration (sec)
Consumption (GB)
SGD
1.25
0.92
30:30
3.48
C-GTSONO
1.35
1.39
22:18
7.61
29


Reviews:
[{"review_id": "Ml56bBhTs2", "reviewer": "Reviewer_veDw", "summary": "This paper studies robust optimisation method for neural ODEs. The authors interpret Neural ODE optimization as a min-max optimal control problem, and then design a Game Theoretic Second-Order Neural Optimizer (GTSONO), based on min-max Differential Dynamic Programming, with convergence guarantees to local saddle points. The authors also conduct experiments to verify the performance of GTSONO.", "questions": "Please address the above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well motivated - addressing the vulnerability to adversarial attacks in neural network related methods, including neural ODE.\n\nThe authors design a Game Theoretic Second-Order Neural Optimizer as a robust optimiser for neural ODEs. \n\nThey also provide rigorous, theoretical analysis for the proposed method. The proofs are given in detail. \n\nThe paper is well written.", "weaknesses": "I am worried about the novelty, after reading the calculations. Leveraging min-max methods for adversarial learning is a usual approach.  convergence. The calculations of gradients and backpropagation are simple calculus and linear algebra. The proof of convergence is a direct application of existing results in optimisation. I suggest the authors clarify the novelty of their algorithms and proofs, and discuss the differences and advantages of their method.\n\nThe experiments are only conducted on CIFAR-10 and SVHN. Experiments on CIFAR-100, and ImageNet (or at least TinyImageNet) are needed for comparison.\n\nFrom Tables 1 and 2, GTSONO has fairly bad performance in CIFAR-10 in term of natural accuracy. Please discuss why this happens.\n\nThe authors only compare with SGD, Adam, and a second order baseline SNOpt. It is not enough. Please compare your method with most existing methods. I list some below:\n\nhttps://proceedings.mlr.press/v162/rodriguez22a/rodriguez22a.pdf\n\nhttps://arxiv.org/pdf/2210.16940.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNgDvhjNXH", "reviewer": "Reviewer_L7bT", "summary": "This paper interprets neural ODE optimization as a min-max optimal control problem, and proposed a second order optimizer. The proposed optimizer is computationally feasible by matrix decomposition. Empirically, the authors compare with other optimizers (Adam, SGD and another not min-max second order optimizer), and show it has improved adversarial robustness.", "questions": "1. My main question is as in weakness 1: the proposed optimizer seems to be beneficial to attacks on neural network weights rather than on the inputs. I hope the authors can clarify why they choose to demonstrate the effectiveness of their optimizer on input-robustness, and will the method be useful for attacks on system weights?\n\n2. In the experiments, it seems that only having adversarial control on convolution layers is much better than having them on all of the layers. From the theory parts it is not clear why this is the case. The authors should provide more analysis on this.\n\n3. When combing with adversarial training (table 4), why CW accuracy drops? This may indicate some gradient obfuscation issue, it will be good to include some stronger attacks in the evaluation (like AutoAttack, Square) as suggested in weakness 3.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper proposes an interesting perspective of robust neural ODE optimization via min-max optimal control. \n\nThe authors make the proposed second order optimizer to be computational feasible: instead of back propagate coupled matrices, one can back propane a set of vectors. \n\nThe authors provide convergence guarantee of the proposed optimizer. \n\nExperiments show improved adversarial robustness comparing with other non-robust neural ODE optimizers.", "weaknesses": "1. Although I think it is interesting to formulate robust neural ODE optimization as min-max OC, I feel there is gap why the formulation in (3) can be beneficial to the robustness problem in (1): in (3), the adversary is on neural network weights, and in (1), the adversary is on the inputs to the neural network. \n\n2. Despite the effort of reducing computational cost, the proposed method is still very expensive. It would be good to include a complexity comparison between other neural ODE optimizers: first-order adjoint and SNOpt.\n\n3. It is known that neural ODE tends to suffer from gradient obfuscation issue when being evaluated for empirical adversarial robustness. It would be beneficial to have some adaptive attacks or non-gradient based attacks to make sure the improved robustness is valid. For instance, the attacks used in [1]. Since the optimizer on its own has lower robustness accuracy than adversarial training methods, it is crucial to have solid experiments to show its benefits when combining with other robust training techniques. In general, I like the min-max OC perspective, but it may still lack evidence for its usefulness. Maybe the authors could also consider evaluating against adversary on neural network weights, which I think is more close to the formulation.\n\n[1] Kang, Qiyu, et al. \"Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks.\" Advances in Neural Information Processing Systems 34 (2021): 14925-14937.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lM28CjpwoG", "reviewer": "Reviewer_gukw", "summary": "The paper introduces a robust optimization algorithm called GTSONO, designed to train Neural Ordinary Differential Equations (Neural ODEs) that are more resilient to adversarial attacks. Based on min-max Differential Dynamic Programming, the algorithm is not only computationally efficient but also guarantees convergence to local saddle points. Experimental results demonstrate its significant advantage in improving model robustness compared to benchmark optimizers. Overall, the paper offers a new and effective tool for enhancing the robustness of deep learning models.", "questions": "1. Could the authors provide insights into the feasibility of applying GTSONO to larger and more complex datasets such as CIFAR100 and ImageNet?\n2. Given the focus on robustness improvement, could the authors consider including a more extensive comparison with various state-of-the-art optimizers that enhance robustness? \n3. Are there any further insights into the stability and convergence properties of GTSONO, especially under varying hyperparameters and different neural network architectures?\n4. In general, deep learning optimizers have used mini-batch size to obtain a partial batch of datasets, which means that random noise will be introduced, so the optimizer's parameter update rule should correspond to a stochastic differential equation instead of an ordinary differential equation. Therefore, why do the authors use an ODE rather than an SDE for their theoretical analysis?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality:\nThe paper introduces a novel perspective by interpreting Neural ODE optimization as a min-max optimal control problem. GTSONO's approach, rooted in min-max Differential Dynamic Programming, showcases a creative amalgamation of existing concepts.\nQuality:\nOffering convergence guarantees to local saddle points signifies the robustness of GTSONO. Efficient matrix decompositions and strong empirical results further attest to the research's high quality.\nClarity:\nThe document articulately bridges intricate theoretical concepts with empirical findings. Despite some formatting issues, the presentation remains clear and coherent.\nSignificance:\nThis work addresses a pivotal challenge in neural ODEs, enhancing their robustness against adversarial attacks. The exploration of optimal control paradigms in adversarial training methods underscores its contributions in the domain.", "weaknesses": "1. Although the GTSONO optimizer proposed in this paper can improve the robustness of the model to some extent, it adds too much extra computational overhead, which is unacceptable in the training of larger models.\n2. The paper's limited comparison with just one other optimizer that improves robustness diminishes the persuasiveness of the results, as a more comprehensive comparison would have strengthened the findings.\n3. One limitation of this paper is that the experiments are confined to the CIFAR10 and SVHN datasets, with no validation on more extensive datasets such as CIFAR100 and ImageNet. This restricts the applicability of the research to a certain extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)