Towards Robust Neural Networks via Close-loop Control
Authors: Zhuotong Chen
Qianxiao Li
Zheng Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: neural network robustness
optimal control
dynamical system

Abstract: Despite their success in massive engineering applications, deep neural networks are vulnerable to various perturbations due to their black-box nature. Recent study has shown that a deep neural network can misclassify the data even if the input data is perturbed by an imperceptible amount. In this paper, we address the robustness issue of neural networks by a novel close-loop control method from the perspective of dynamic systems. Instead of modifying the parameters in a fixed neural network architecture, a close-loop control process is added to generate control signals adaptively for the perturbed or corrupted data. We connect the robustness of neural networks with optimal control using the geometrical information of underlying data to design the control objective. The detailed analysis shows how the embedding manifolds of state trajectory affect error estimation of the proposed method. Our approach can simultaneously maintain the performance on clean data and improve the robustness against many types of data perturbations. It can also further improve the performance of robustly trained neural networks against different perturbations. To the best of our knowledge, this is the first work that improves the robustness of neural networks with close-loop control.

Published as a conference paper at ICLR 2021
TOWARDS ROBUST NEURAL NETWORKS VIA CLOSE- LOOP CONTROL
Zhuotong Chen Qiamxiao 2 Zheng Zhang'
DDppartment of Electrical & Computer Engineering, University of California, Santa Barbara, CA 93106 2 DDpartment of Mathematics, National University of Singapore, Singapore Institute of High Performance Computing, A STARR, Singapore zt ztchen@ucsb.cc edu, qlanxiao@nus. edu. sg, sg, zhengzhangêece. wttp://hhhnnee.uuub...bo. edu
ABSTRACT
Despite their success in massive engineering applications, deep neural networks are vulnerable to various perturbations due to their black-box nature. Recent study has shown that deep neural network can misclassify the data even if the input data perturbed by an imperceptible amount. In this paper, we address the ro- bustness issue of neural networks by novel close-loop control method from the perspective of dynamic systems. Instead of modifying the parameters in a fixed neural network architecture, close-loop control process is added generate con- trol signals adaptively for the perturbed or corrupted data. We connect the robust- ness of neural networks with optimal control using the geometrical information of underlying data to design the control objective. The detailed analysis shows how the embedding manifolds of state trajectory affect error estimation of the proposed method. Our approach can simultaneously maintain the performance on clean data and improve the robustness against many types of data perturbations. It can also further improve the performance of robustly trained neural networks against different perturbations, To the best of our knowledge, this is the first work that improves the robustness of neural networks with close-loop control
INTRODUCTION
Due to the increasing data and computing power, deep neural networks have achieved state-of-the art performance in many applications such as computer vision, natural language processing and recommendation systems. However, many deep neural networks are vulnerable to various malicious perturbations due to their black-box nature: small (even imperceptible) perturbation of input data may lead to completely wrong predictions (Szegedy et al,, 2013) Nguyen et al.) 2015). This has been major concern in some safety-critical applications such as autonomous driving (Grigorescu et al. 2020) and medical image analysis (Lundervold Lundervold 2019). Various perturbations have been reported, including the Ep norm based attack (Madry et al. 2017 2 Moosavi-Dezfooli et al. 2016 Carlini & Wagner] (017). semantic perturbation (Engstrom et al.12217) etc. On the other side, some algorithms improve the robustness against those perturbations have shown great success (Madry et al. 2017). However, most robustly trained models are tailored for certain types of perturbations, and they do not work well for other types of perturbations. Khoury & Meneell (2018) showed the non-existence of optimal decision boundary for any fp-norm perturbation.
2017) have shown the connection between dynamical systems and neural networks. This dynamic system perspective provides some interesting theoretical insights about the robustness issue. Given set of data Xg E IR and its labels y E IR' with a joint distribution D, training neural network can be considered as following
Recent works E 2017 Haber & Ruthotto
min
E (X
(xxT,));, s.t. X(+1 f(x,, 0z),
8 Equal contributing authors.
be found in htttppss/ggttibb.cc/////ooottttt l'owards- Robust Neural Networks- Cloee- wot.ooaooooooooooooooeoooo
A Pytorch implementation

---

Published as a conference paper at ICLR 2021
Layero
Layer
Layerr
un
Eg (xo)
E (x1)
Conto
Contz
Figure 1: The structures of feed-forward neural network (black) and the proposed method (blue).
where 0 are the unknown parameters to train, and f. & represent the forward propagation rule and loss function (e.g. cross-entropy) respectively. The dynamical system perspective interprets the vulnerability of neural networks a system instability issue, which addresses the state trajectory variation under small perturbations applied on initial conditions. The optimal control theory focuses on developing control model to adjust the system state trajectory in optimal manner. The first work that links and extends the classical back-pre pocck-roppggtton algorithm using optimal control theory was presented in Li et al. 2017). where the direct relationship between the Pontryagin' 's Maximum Principle (Kirk 1970) and the gradient based network training was established. YYe et al. (2019) used control theory to adjust the hyperparameters in the adversarial training algorithm. Han et al. (0188) established the mathematical basis of the optimal control viewpoint of deep learning. These existing works on algorithm development are open-loop control methods since they commonly treat the network weights 0 as control parameters and keep them fixed once the training is done. The fixed control parameters O operate optimally for data sampled from the data distribution various perturbation methods cause data distributions to deviate from the true distribution 'D (Song) ct al. 2017) and cause poor performance with the fixed open-looy control parameters.
However,
1.1 PAPER CONTRIBUTIONS
To address the limitation of using open-loop control methods, propose the Close- Loop Control Neural Network (CLC-NN), the first close-loop control method to improve the robustness of neural networks. As shown in Fig. T our method adds additional blocks given T-layer neural network: embedding functions &,, which induce running losses in all layers that measure the discrepancies between true features and observed features under input perturbation, then control processes gen- erate control variables u, to minimize the total running loss under various data perturbations. The original neural network can be designed by either standard training or robust training. In the lat- ter case, our CLC-NN framework can achieve extra robustness against different perturbations. The forward propagation rule is thus modified with an extra control parameter E IRd'
X1+1 f (x,, 0r, u.).
Fig. should not be misunderstood as an open-loop control. From the perspective of dynamic systems, Xy is an initial condition, and the excitation input signal is u, (which in standard feed-forward network). Therefore, the forward signal path is from to the internal states x₃ and then to the output label y. The path from x, to the embedding function (x,) and then to the excitation signal u, forms feedback and closes the whole loop.
The technical contributions of this paper are summarized below:
The proposed method relies on the well accepted assumption that the data and hidden state man- ifolds are low dimensional compared to the ambient dimension (Fefferman et al.) 2016). We study the geometrical information of the data and hidden layers to define the objective function for control. Given trained T Tllayer neural network, a set of embedding functions &, trained off-line by minimizing the reconstruction loss [[{(x)) - x2 || over some clean data from D only. The embedding functions support defining a running loss required in our control method.
We define the control problem by dynamic programming and implement the online iterative solver based the Pontryagin's 's Maximum Principle to avoid the curse of dimensionality. The proposed close-loop control formulation does not require prior information of the perturbation.
We provide theoretical error bound of the controlled system for the simplified case with linear activation functions and linear embedding. This error bound reveals how the close-loop control improves neural network robustness in the simplest setting.
2

---

Published as a conference paper at ICLR 2021
2
RELATED WORKS
Many techniques have been reported to improve the robustness of neural networks, such as data augmentation (Shorten & Khoshgoftaar 2019), gradient masking (Liu et al. 2018), etc. We review adversarial training and reactive defense which are most relevant to this work.
Adversarial Training. Adversarial training is (possibly) the most popular robust training method, and it solves min-max robust optimization problem to minimize the worse-case loss with perturbed data. Adversarial training effectively regularizes the network's local Lipschitz constants of the loss surface around the data manifold (Liu et al. 2018 Zhang ct al. (2019) formulated the robustness training using the Pontryagon's 's Maximum Principle, such open-loop control methods result a set of fixed parameters that operates optimally on the considered perturbation. Liu et al. (2020a b) considered a close-loop formulation from the differential dynamic programming perspective, this algorithm is categorized as a open-loop control method because it utilizes the state feedback infor- mation to boost the training convergence and results in a set of fixed controls for any unseen data. On the contrary, the proposed CLC-NN formulation adaptively targets on the inputs with different control parameters and is capable of distinguishing clean data by generating no control.
Reactive Defense. A reactive defense method tries to reject or pre-process the input data that may cause mis-classifications, |Metzen al. (2017) rejected perturbed data by using adversarial detectors that are trained with adversarial data to detect abnormal data during forward propagation. SSagtiioonnSnnngooaao et al. (017)) estimated the imput data distribution DD with a generative model (Oord et al.) 2016) detect data that does not belong to D, it applies greedy method to search the local neighbour of input data for more statistically plausible counterpart. This purification process has shown improved accuracy with adversarial data contaminated by various types of perturbations. Purification can be considered as a one-step method to solve the optimal control problem that has the objective function defined over the initial condition only. On the contrary, the proposed CLC-NN solves the control problem by the dynamic programming principle and its objective function is defined over the entire state trajectory, which guarantees the optimality for the resulted controls.
3 THE CLOSE-LOOP CONTROL FRAMEWORK FOR NEURAL NETWORKS
Now we present a close-loop optimal control formulation to address the robustness issue of deep learning. Consider neural network consisting of model parameters O equipped with external con- trol policy π, where π E is collection of functions Id + H² acting on the state and outputting the control signal. The feed-forward propagation in a T T-ayyer neural network can be represented as
Xt+1 [x,,, 0r. (xx)), t= ,
Given trained network, we solve the following optimization problem
2 ((xT,y) + Σ [(x,, n. ((x, (x.1))
min E (xD [[(x, y, π]] == min E (x
s.l. Eq. (1),
where collects the control policies To...... -T for all layers. Note that (2) differs from the open-loop control used in standard training. An open-loop control that treats the network parameters as control variables seeks for a set of fixed parameters to match the output with true label y by minimizing the terminal loss 4 and the running loss L defines a regularization for O. However, the terminal and running losses play different roles when our goal is to improve the robustness of a neural network by generating some adaptive controls for different inputs.
Challenge of Close-loop Control for Neural Networks. Optimal control has been well studied in the control community for trajectory optimization, where one defines the running loss as the error between the actual state x, and reference state Xt,ref over time interval [0, 27. The resulting control policy adjusts x₁ and makes approach Xt,ref. this paper, we apply the idea of trajectory optimization to improve the robustness of neural network via adjusting the undesired state of x,- However, the formulation is more challenging in neural networks: we do not have a "reference" state during the inference process, therefore is unclear how to define the running loss L.
In the following, we investigate manifold embedding of the state trajectory to precisely define the loss functions + and L of Eq. (2) required for the control objective function of neural network.
3

---

Published as a conference paper at ICLR 2021
3.1 MANIFOLD LEARNING FOR STATE TRAJECTORIES
State Manifold. Our controller design is based on the "manifold hypothesis". real-world high dimensional data can often be embedded in lower dimensional manifold M (Fefferman et al. 2016). Indeed, neural networks extract the embedded features from M. To fool a well-trained neural network, the perturbed data often stays away from the data manifold M (Khoury
HadfieldMMenel. 2018). We consider the data space Z (x Z. Vx ~ D) as: Z = Z1 θ Zi. where Z₁ contains the embedded manifold M and Z1 the orthogonal complement of Znj. During forward propagation, the state manifold embedded in Z| varies at different layers due to both the nonlinear activation function/ I and state dimensionality variation. Therefore, we denote zz= 2 + Z as the state space decomposition at layer and M₁ E Zí-- Once an input data is perturbed, the main effects of causing misclassifications are in Z1. Therefore, is important to measure how far the possibly perturbed state X, deviates from the state manifold M₁-
Embedding Function. Given embedding function & that encodes Xx onto the lower- dimensional manifold M, and decodes the result back to the full state space Z,, the reconstruction loss [1&, xxyll measures the deviation of the possibly perturbed state x, from the manifold M,- The reconstruction loss is nonzero as long as x has components in Z The embedding functions constructed offline by minimizing the total reconstruction losses over clean training data set.
Linear Case: (-) can considered as VI V[(VT)Z where V forms an orthonormal basis for Z Specifically can first perform principle component analysis over a collection of hidden states at layer t, then V can be obtained the first r columns of the resulting eigenvectors.
Nonlinear Case: we choose a convolutional auto-encoder (detailed in Appendix B to obtain representative manifold embedding function & due to its ease of implementation. Based on the assumption that most perturbations are in the Z1 subspace, the embeddings are effective to detect the perturbations as long as the target manifold is of low dimension. Alternative manifold learning methods such as Izenman (2012) may also be employed.
3.2 FORMULATION FOR THE CLOSE-LOOP CONTROL OF NEURAL NETWORKS
Control Objectives. The above embedding function allows us to define a running loss C: L(x, T. (x)) = [1Ex(x;) - x+ + (x:)) (x).).)..
(3)
Here the matrix R defines a regularization term promoting controls of small magnitudes. In practical implementations, using a diagonal matrix R with small elements often helps to improve the perfor- mance. Now we are ready to design the control objective function of CLC-NN. Different from a standard open-loop control, this work sets the terminal loss & as zero because no true label is given during inference. Consequently, the close-loop control formulation in Eq. 2 becomes
min B (xxo)))-D [J(xo, y, T)] == min (No.)) -D
Σ [xx,, & s.t. Eq. (1.. (4)
t=
Assume that the input data is perturbed by bounded and small amount, i.e.,
x = Xg + 2,
where z can be either random or adversarial. The proposed CLC-NN adjusts the perturbed state tra- jectory x., such that it stays at minimum distance from the desired manifold M, while promoting small magnitudes of controls.
Intuition. We use an intuitive example show how CLC-NN controls the state trajectory of un- seen data samples. We create a synthetic binary classification data set with 1500 samples. We train a residual neural network with one hidden layer of dimension 2, and adopt the fast gradient sign method (Goodfellow et al. 2014) to generate adversarial data. Fig. (a) and (b) show the states of clean data (red and blue) and of perturbed data (black and gray) at 0 and = 1, respectively. The CLC-NN adjusts the state trajectory to reduce the reconstruction loss as shown in Fig, 2 (c) and (d), where lighter background color represents lower reconstruction loss. Comparing Fig. 2 (a) with (c), and Fig. 2 with (d), see that the perturbed states in Fig, (a) and (b) deviate from the desired
state manifold (light green region) and has a high reconstruction loss. Running 1000 iterations of
Alg. adjjsss the perturbed states and improves the classification accuracy from 86% 100%.
4

---

Published as a conference paper at ICLR 2021
Clean-1 Clean-2
Clean-1
Adversarial Adversanall
Adversarial
Clean-2
Adversenal-22
Adversarial-1 Adversarial
Adversarial.1 2dversantal-2
(a) Xo, X
(b) X1, Xx,1
(c) Controlled X.,
(d) Controlled X.,1
Figure 2: (a) and (b) show the states of clean data (red and blue) and of perturbed data (black and gray) at the initial and hidden layers respectively. The yellow and green backgrounds represent the two classes predicted by the network. Some of the perturbed data are mis-classified. (c) and (d) show the adjusted states after the proposed close-loop control. The background lightness from light to dark represent the increasing reconstruction loss (x,) xx x2112.
4 IMPLEMENTATION VIA THE PONTRYAGIN'S s MAXIMUM PRINCIPLE
Dynamic Programming for Close-Loop Control (4) solved by the dynamical programming principle (Belman 1952). For simplicity we consider one input data sample, and define a value function V T x R + (where T := {0, 1,...,T T 1)).
The control problem in Eq. (4 can be
Here (t, x) represents the optimal cost-to-go function of Eq. incurred from time state x. One can show that (t,,)) satisfies the dynamic programming principle
(tt, x) = inf [t f(x, 0,, +x))) + [(x, (()), *
(5)
Eq. (5) gives a necessary and sufficient condition for optimality of Eq. 4 and it is often solved backward in time by discretizing the entire state space. The state dimension of modern neural network is at the order of thousands or even higher, therefore, discretizing the state space and directly solving Eq. (5) intractable for real-world applications due the curse of dimensionality.
Solving via the Pontryagin's Maximum Principle. To overcome the computational chal- lenge, the Pontryagin's Maximum Principle (Kirk) 1970) converts the intractable dynamical pro- gramming into two ordinary differential equations and a maximization condition. Instead com- puting the control policy π of Eq. (5). the Pontryagin's 's Maximum Principle provides a necessary condition for the optimality with a set of control parameters [uj. .... , 17 The mean-field Pontrya- gin's Maximum Principle can be considered when the initial condition is batch of i.i.d. samples drawn from D. Specifically, we trade the intractable computational complexity with processing time for solging the Hamilton equations and its maximization condition for every newly observed data. To begin with, we define the Hamiltonian H : T x IRd x Rd x R x Rm + R as
(t, Xx, P1+1, Or, uz) == p f(x,, O,, u²) - L(xe, u, &((-)..
(6)
Let x denote the corresponding optimally controlled state trajectory. There exists a co-state process p" [0, T] Bd such that the Hamilton's 's equations
x = V Ht, x P 0,, uz), : = V Ht, x; 00.1.0,,10 u;),
(xi. y) ~ DD Př = 0,
(7)
(8)
are satisfied. The terminal co-state pr O, since we do not consider the terminal loss (xT., Moreover, we have the Hamiltonian maximization condition
H x" Pi, 02, u,) 2 (t,,)) pl. Oz, u.). Vu E Rd and Vt T.
Instead of solving Eq. (5) for the optimal control policy π T' (x:), for a given initial condition, the Pontryagin's 's Maximum Principle seeks for open-loop optimal solution such that the global op- timum of Eq. (5) is satisfied. The limitation of using the maximum principle is that the control parameters ot need to be solved for every unseen data achieve the optimal solution.
Algorithm Flow. The numerical implementation of CLC-NN is summarized in Alg. ! Given a trained network (either from standard or adversarial training) and a set of embedding functions, the controls are initialized as u, 0, Vl. T, because adding random initialization weakens the
5

---

Published as a conference paper at ICLR 2021
Algorithm 1: CLC-NN with the Pontryagin's Maximum Principle.
Input Possibly perturbed data x,, a trained neural network, embedding functions
[,,,-. ,Er-1], ( maxltr (maximum number of iterations).
Output: set of optimal control parameters u6:"" "0 -17-1
for k 0 to maxltr do
Jx = 0,
for 1 = faak==[xxpp [(x Os, u,R)), where X0,6 X
do
X1+1, =
Forward propagation Eq.
J3= = Jk + L(xL, ue (x)x,,))).
Objective function Eq.
end for
for 1 T to 1 do
Pr,k - P
Xx, (Xt,k, Oc, Ur,k) - Vx, ((x,, ur & (xt,,))
where Pr,k = 0,
b Backward propagation Eq.
10
end for
for t
0 T 1 do
U1++1 = 4r + (PÉ+1. / O,, Ue,k) V L(xe, Us )x),,)))),
12
> Maximization of Hamiltonian Eq. based on Eq. (6) and gradient ascent.
14 end for
15 end for
robustness performance in general, and clean trajectory often does not result in any running loss for the gradient update on the control parameters. every iteration, given input X is propagated forwardly with Eq. 7 to obtain all the intermediate hidden states for all t and to accumulate cost J. Eq. g backward propagates the co-state P. and Eq. 9 maximizes the zth Hamiltonian with current x2 and Pı to compute the optimal control parameters u/
ERROR ANALYSIS FOR SIMPLIFIED LINEAR CASES
For the ease of analysis, we consider a simplified neural network with linear activation functions:
Xx+1 0, (x, +u),
and reveal why our proposed method can improve robustness the simplest setting. Given per- turbed data sample x.,0, we denote its perturbation-free counterpart as so that z = X.,0 Xo- We consider general perturbation where is the direct sum of two orthogonal contributions: zll, which a perturbation within the data manifold (subspace), and z which is a perturbation in the orthog- onal complement of the data manifold. This case is general: if we consider adversarial attacks, then the perturbation along the orthogonal complement dominates. In contrast, if we consider random the same scale. Our formulation covers both such
perturbations, then the two perturbations extreme scenarios, together with intermediate cases.
We use orthogonal projection the embedding function such that & VI(V!) where V is the first r columns of the eigenvectors computed by the Principle Component Analysis on collection of states X,. The proposed CLC-NN minimizes ||x,,. - 1 by reducing the components of x,, that lie in the the orthogonal complement of Zjj The following theorem provides an error estimation between x,, and x,.
Theorem 1. For 2 1, we have the error estimation
[ x < 10-1 ... ...l.... a 222
12 11
(10)
where 71 := max (1 + x(.))))) *st
0. 12112, and a = Tfe. represents the control regularization.
In particular, the equality
X - = a ||z
(1I)
holds when all 0, are orthogonal.
The detailed derivation is presented in Appendix
Let us summarize the insights from Theorem 1

---

Published as a conference paper at ICLR 2021
Table I: Experimental results on ResNet-20 from standard training.
Accuracy: original model without CLC/ CLC-NN Linear CLC-NN Noulinear
Dataset
Type of input perturbations
None
Manifold
FGSM
PGD
CW
2
79 82
21/56/5 56
0/50/50 50
8/75 / 79
CIFAR-
92/88/
5/78/
11/40/
0/31/19
0/75/79
10
1/78
8/20/11
0/11/2
0/76/79
2
9/51/52
9/25/23
0/17/22
4/47/49
CIFAR-
4
69 60/66/58
5/50/52
5/15/9
0/6/4
1/47/45
100
2/50/52
41915
0/1/0
0/47/49
The above error estimation is general for any input perturbation. shows the working principle behind the proposed CLC-NN on controlling the perturbation that lies in the orthogonal comple- ment of input subspace (z-))
The above error estimation improves the control regularization e goes to 0 (so 0). It is not the sharpest possible as relies on a greedily optimal control at each layer. The globally optimal control defined by the Ricatti equation may achieve lower loss when +
When the dimension of embedding subspace decreases, control becomes more effective in reducing ||x., X2 x1 This means that the control approach works the best when the data is constrained on a low dimensional manifold, which is consistent with the manifold hypothesis. In particular, observe that as r 0, | +
The obtained upper bound is tight: the estimated upper bound becomes the actual error if all the forward propagation layers are orthogonal matrices.
NUMERICAL EXPERIMENTS
test our proposed CLC-NN framework under various input data perturbations. Here we briefly summarize experimental settings, and we refer readers to Appendix B for the details.
Original Networks without Close-Loop Control. We choose residual neural networks (He et al., 2016) with ReLU activation functions our target for close-loop control. In order show that CLC-NN can improve the robustness in various settings, we consider networks from both standard and adversarial trainings. We consider multiple adversarial training methods: fast gradient sign method (FGSM) (Goodfellow et al. 2014), project gradient descent (PGD) (Madry et al. 2017 and the Label smoothing training (Label Smooth) (Hazan al.) 2017).
perturb the input data within radius of e with 2, 4 and 8 respectively. We consider various perturbations, including non- adversarial perturbations with the manifold-based attack (Jalal et al. 2017) (Manifold), well as some adversarial attacks such as FGSM, PGD and the CW methods (Carlini & Wagner 2017).
Input Perturbations. In order to test CLCC-N framework,
CLC-NN Implementations. We consider both linear and nonlinear embedding in our close- loop control. Specifically, we employ principal component analysis with 1% truncation ror for linear embedding, and convolutional auto-encoders for nonlinear embedding. We use Adam (Kingma & Ba 2014) maximize the Hamiltonian function ( and keep the same hyper- parumeters (learning rale, maximum iterations for each model against all perturbations.
Result Summary: Table and Table 2 show the results for both CIFAR-10 and CIFAR-100 datasets on some neural networks from both standard training and adversarial training respectively.
CLC-NN significantly improves the robustness of neural networks from standard training. Tshoos that the baseline network trained on clean data set becomes completely vulnerable (with almost 0% accuracy) under PGD and CW attacks. Our CLC-NN improves its accuracy to nearly 40% and 80% under PGD and CW attacks respectively. The accuracy under FGSM attacks has almost been doubled by our CLC-NN method. The accuracy on clean data is slightly decreased because the lower-dimensional embedding functions cannot exactly capture Zμ or M.
CLC-NN further improves the robustness of adversarially trained networks. Table shows that while an adversarially trained network is inherently robust against certain types of pertur- bations, CLC--N strengthens its robustness significantly against various perturbations. For in-
7

---

Published as a conference paper at ICLR 2021
Table Experimental results on robustly trained networks.
Accuracy: Robustly trained models / CLC-NN Linear Embedding
Dataset
Training methods
Type of input perturbations
None
Manifold
FGSM
PGD
CW
43/
90/87
1760
14781
FGSM
92 90
17/81
95/90 900
0/36
2/ 81
8
3/80
96/93
0/10
0/81
427 78
57769
10760
29777
Label Smooth
CIFAR-10
93
20/ 77
51/61
1/
11/76
6/77
40/47
0/26
2/76
2
80778
75773
74773
75775
PGD
83/ 80
78 76
66/66 66
6 65
66 77
73
49
40/46
48 / 64
2
21/55
59/54
1/27
9/57
FGSM
4
66
8/54
66/55
0/9
2/57
8
3/53
67 56
0/1
0/57
10/43
16/28
1/19
5/46
Label Smooth
CIFAR-100
/9/62
3/42
19
0/9
1/ 46
2/42
8/11
0/2
0/46
2
52750
46743
45/43
45747
PGD
58 /8/52
49 48
36/36
34/
35 /33
43/ 45
22/24
16/21
21/40
Table 3: Accuracy comparision of CLC-NN and reactive defense in Eq. 12). with Eattack 2//4/8.
Here "+" (or "-") indicates how much CLC-NN outperforms (or underperforms) reactive defense.
Type of input perturbations
Method
None
Manifold
FGSM
PGD
CW
CIFAR-10
-3
+47 4633
+27 //22///444 +13
+43/435 425
+66 #66/+76/ 477
CIFAR- 100
+1
#34/+37/#*
+4/+337/0 +38
+22/0/+9
+44/+30/+11
+37/+30 +16
stance, CLC-NN improves the accuracy of an FGSM trained network under PGD and CW attacks by maximum of 59% and 81 81%, respectively.
The robustness improvement of adversarially trained networks is less significant. This is expected because the trajectory of perturbed data lies on the embedding subspace Z if that data sample has been used in adversarial training. However, our experiments show that applying CLC- NN to adversarially trained networks can achieve the best performance under most attacks.
Comparison with PixelDefend (Song ef al., 2017). Our method achieves similar performance on CIFAR-10 10 with slightly different experimental setting. Specifically, PixelDefend improved the robustness of a normally trained 62-layer ResNet from 0% to 78% against CW attack. Our proposed CLC-NN improves the robustness of 20-layer ResNet from 0% to 81% against CW attacks. Fur- thermore, we show that CLC-NN robust against the manifold-based attack. No result was reported for CIFAR-100 in Song et al. 2017).
Comparison with Reactive Defense Reactive defenses can be understood only applying control at the initial condition of dynamical system. Specifically, reactive defense equipped with linear embedding admits the following dynamics:
x [(x,, 0.), s.l. X VT(V.)" x
(12)
By contrast, CLC--N controls all hidden states and results in decreasing layers T' increases (c.f. Theorem []. To quantitatively compare CLC-NN with reactive defense, implement them with the same linear embedding functions and against all perturbations. In Table 3, CLC-NN outperforms reactive defense in almost all cases except that their performances are desendennd on clean data.
the number of
7 CONCLUSION
We have proposed close-loop control formulation to improve the robustness of neural networks. We have studied the embedding state trajectory during forward propagation to define the optimal control objective function. The numerical experiments have shown that method improve
8

---

Published as a conference paper at ICLR 2021
the robustness of trained neural network against various perturbations. We have provided an emror estimation for the proposed method in the linear case. Our current implementation uses the Pontrya- gin's 's Maximum Principle and an online iterative algorithm to overcome the intractability of solving a dynamical programming. This online process adds extra inference time. the future, we plan to show the theoretical analysis for the nonlinear emhedding case.
Acknowledgement Zhuotong Chen and Zheng Zhang are supported by NSF CAREER Award No. 1846476 and NSF CCF No. 1817037. Qianxiao Li is supported by the start-up grant under the NUS PYP programme.

---

Published as a conference paper at ICLR 2021
REFERENCES
Richard Bellman. On the theory of dynamic programming. Proceedings of the National Academy of Sciences of the United States of America, 38(8):716, 1952.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. IEEE, 2017.
E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
Logan Engstrom, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and translation suffice: Fooling cnns with simple transformations. arXiv preprint 17212:17/ 1(2):3, 2017.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Matematical Society, 29(4):983-1049, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386, 2020.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.
Jiequn Han, Qianxiao Li, al. A mean-field optimal control formulation of deep learning. arXiv preprint arXiv:1807.01083 2018.
Tamir Hazan, George Papandreou, and Daniel Tarlow. Adversarial perturbations of deep neural
networks. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Alan Julian Izenman. Introduction manifold learning. Wiley Interdisciplinary Reviews: Compu- tational Statistics, 4(5):439-446, 2012.
Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint 17Xiv:1712.09196, 2017.
Mare Khoury and Dylan Hadfilld-Menel. On the geometry of adversarial examples. arXiv preprint arXiv:1811.00525 2018
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6680 2014.
Donald E Kirk. Optimal control theory: introduction. Springer, 1970.
Qianxiao Li, Long Chen, Cheng Tai, and E Weinan. Maximum principle based algorithms for deep learning. The Journal of Machine Learning Research, 18(1):599866026, 2017.
Guan Guan-Horng Liu, Tianrong Chen, and Evangelos A Theodorou. Differential dynamic programming neural optimizer. arXiv preprint arXiv:2002.08800 2020a.
Guan-Homg Liu, Tianrong Chen, and Evangelos A Theodorou. A differential game theoretic neural optimizer for training residual networks. arviv preprint arXiv:2007.08880, 2020b.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference Computer Vision (ECCV), pp. 369-385, 2018.
Alexander Selvikvág Lundervold and Arvid Lundervold. An overview of deep learning in medical imaging focusing mri. Zeitschrifi für Medizinische Physik, 29(2):102-127, 2019.
10

---

Published as a conference paper at ICLR 2021
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0600 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprini 1rXiv:/702.04267, 2017.
Seyed-Mohsen Koosavi-Dezfoolle Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2574-2582, 2016.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Aaron den Oord, Nal Kalchbrenner. and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint 161.00661 2016.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gas gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprini arXiv: 1805.06605. 2018.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv:I710. 10766, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Nanyang Ye, Qianxiao Li, and Zhanxing Zhu. training acceleration. 2019.
Amata: An annealing mechanism for adversarial
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. In Advances in Neural Information Processing Systems, pp. 227-238, 2019.
A APPENDIX A ERROR ESTIMATION FOR THE PROPOSED CLC-NN
Preliminaries
We define the performance index at time / as
J (x,, u,) Qx + 2
(13)
2
where Q: = VT(V V is the linear projection matrix at time with only its first principle components corresponding to the largest r eigenvalues. The optimal feedback control is defined as
un (x:) = arg min (xx,, uz), u.
due to the linear system and quadratic performance index, the optimal feedback control admits an analytic solution by taking the gradient of performance index (Bq. (13) and setting
Vud u:) vn Qr + w luu l1..13))
Q7 Qix: Q Quu, + U
which leads to the analytic solution of u, (x,)
u (x,) (c I Q Q - QT Q.x..
(14)
The above analytic control solution uf optimizes the performance index instantly at time step t, the error measured by Eq. (13) for the dynamical programming solution x,, must be smaller equal

---

Published as a conference paper at ICLR 2021
to the state trajectory equipped with 3, define by Eq. (14p. which gives guaranteed upper bound for the error estimation the dynamic programming solution.
We define the feedback gain matrix K, (c 1+ Q.) Q Qu. Thus, the one-step optimal feedback control can be represented as uj = -
The difference between the controlled system applied with perturbation initial condition and the uncontrolled system without perturbation shown
x Xx+1 = (x,, U4 x,),
(15) The control objective is to minimize the state components that span the orthogonal complement of the data manifold manifold, such that || V2 (VF)) ]x,112 = 0, the feedback control K,x, O. The state difference of Eq. (15) can be further shown by adding 0 term of (x,)
(x,,, K,X.t x,).
V)(V))") when the imput data to feedback control only stays the state
xe,++ X 0, K,) 0,x: + O,K,Xi:
0, (x) xt). In the following, we show a transformation on the control dynamic term (I K,) based on its definition.
(16)
Lemma 1. For we have
= (1-a)) Pt.
where P. VI(V)),, which is the orthogonal projection onto Zi. and 1+c a [].
such that
Proof. Recall that K, Q/ Q7 Qu and Q = -- VI(V))" Qs can be diagonalized as following
0 0
0
Q V
VT. 0
where the first diagonal elements have common value of and the last (d r) diagonal elements have common value of Furthermore, the feedback gain matrix K, can be diagonalized as
0
0
K, = V
0
V
0
I+c
0
0
where the last (d r) diagonal elements have common value of thus can be represented
The control term (I K₂)
T4c
1
0
0
0
1
0
K, V,
0
VT.
0
0
6
0
I+c_
where the first r diagonal elements have common value and the last (d r) diagonal elements have common value of - By denoting the projection of first columns as V and last (d r) columns Vr, it can be further shown as
K. VI(VS)" 1+c (VI(VT)"),
= o -I-)), P.),
=a 11 (1 Pt-
12

---

Published as a conference paper at ICLR 2021
Oblique Projections.
Let P be linear operator on RR,
We say that P is an projection if P = P. P is orthogonal projection P P P2
If p2 = P but P * P", it is called an oblique projection. Proposition 2. For a projection P.
1. IfP is an orthogonal projection. then |Pl|2 = 1.
2. P is an oblique projection, then |P|| > 1.
3. Pf Q are two projections such that range(P) range Q). then PQ = Q and QP = P.
4. If P is a projection, then rank(P) = Tr(P). Furthermore, if P is an orthogonal projection, then rank(P) = ||| = Tr(PP*).
Define for t 2 0
P::= =
s= s=0,1....,
Lemma 3. Let Pi be defined as above for 0 < 8 S L. Then
1. Pł is projection.
Pt is a projection onto z 8, i.e. range(Pi) = -11-
3.
|P S 0,,0,.0..0....))
where k(A) is the condition number of A, i.e. x(A) = ||ll2 1-A-|1 -- and r rank(2 rank(Z)) rank(Zi).
Proof.
1. We prove it by induction for each f. For 0, P Pt, which is projection by its definition. Suppose it is true for s such that P. = P.P.. then for (s + 1).
(p = (0, t-s- P;00-s-1)
0,- --1 (P:)² 2 0, 0:-a-11
0 0,1 P;O,-a-
pp+1
We prove by induction on for each f. For s = 0, P! Pt, which is the orthogonal projection
onto Z Suppose that it is true for such that P; projection onto Z then for (s 1),
p*+1 0 12._-PP00----1 which implies
range(P:*t) =
=10 =10-0-1x: Z
2
We use the inequalities of P.
|| |AB| S |||| B| rr, and |BB| r S |||r|| By the definition
P = (0.- 00.-0._2.... ... 0r-a)
have the following
| < || ( 1...-10.-......0
-1 -11 (1 0, 0, 0.-s) 113 ||!
S 1100~00-2-0 0.,)
Lemma 4)
13

---

Published as a conference paper at ICLR 2021
The following Lemma uses the concept of oblique projection to show a recursive relationship to project any 8th state space of Eq. (16) back to the input data space. Lemma 4. Define for 0 & t
= a I+ (1 a.
Then, Eq. 16 can be written as
- = = 100-00002..000(CCC
G.) - xo);
t2 1.
Proof. We prove it by induction on For by the definition of G. and transformation from Lemmal 1
x,, 0o(1 - (x) -x0),
Eq. 16
= 0o(a 1 (1 aa) P (x,,0 - Xo),
Lemma !
00G)((xe. xx,0 - Xo)-
Suppose that it is true for (Xe, x1), by using Eq. 16 and Lemma 1 we have
X,,++1 X 0I K; (xc, xt),
Eq. 16
( (1 a) P,) (x,,
Lemma
0, 0, ... 0o) (o) (.
G ... G.) (x.00 - -x0).
(17)
Recall the definitions of p! 0-1 1 P.,00 and G. a I + a P we have the following
G a I+ (1 - a) (((+1)
=a (1 a) - P.01-s-1.
0 0 I (1 a) Pi 0 01-s-1: 0 G
which results in the equality for the oblique projections. Furthermore,
G (++1) G.O,
Applying the above to Eq.
17 results in
..) (x) Xn),
Xe,++1 X,+1
= -0
(x)
(Xe,0 - -xo),
6 (x xx)).
( G Xn).
Lemma 5. Let
F, = G
(1-2) 2-2
G t21.
Then,
F.= I+
Σ p
x=0
Proof.
We prove it by induction on Recall the definition G; == a I+ (1 - aa) P. When
t=1,
F. G = a- I (1 - a) pd
Suppose that it is true for i such that
F, P.=a(t- G G(-2 ... ...G. = at I (1 a) Σ
(t-2)
a 'p' 83
s=0
14

---

Published as a conference paper at ICLR 2021
for (t 1).
F(+1 = (tF()), G(F(),
= P.)...
=aa-I+(1-a) Pi)(a' I+ I+ Σ a P)),
8=0
1 + a)² Σ a² P.P. + a(1 a)
Σ a² P.
Recall Lemma Hence,
range (P.)
range zl. According to Proposition
(3), P.P: P.
[-1 - -1 P + (1-a) Σ a P.
aa .4
a a P
Lemma 6. Let V E Raxy be matrix whose columns orthogonal basis for subspace D, and 0 E Rdxd be invertible. Let P VVT be the orthogonal projection onto D. Denote by P the orthogonal projection onto 0D == [fx: x D). Then
e-1P0 is an oblique projection onto D.
|10-1P6 - P1|2 & + k(0)) -
TT-0To7 0
In general, the last inequality shows that 0 -1 1po = P, if is orthogonal.
Proof.
1 00110, - therefore, 0-120 an projection.
Since P is orthogonal projection onto the row space of OV, then
p = (VV)) ((vv).
-V VT
-1 V -
Furthermore,
1 -00-P11 PP|2 = - VV 12,
vV 0" 00 IV -
< [[[V"'orvV 0 + 10 11|2,
< -V II- ov _0 + -0 0 I
-V I 0
10 0 000 I 2-
15

---

Published as a conference paper at ICLR 2021
We further bound [[ -12--
| 0 1V2 ( -
( inf x² "V'00OV -1
||||||=1
inf (x)) -1
1
(Amin (0 (0))) -1
((0T0)-'12
Hence, we have
110 - - S (1 + | 010 |1 0 112)
| 0 000 02,
(1 x)))²))))))))))))))))))))) | 0 0112
Corollary 1. Let t 2 1. Then for each s = 0, ... ,, we have
|P Poll2 S (1 x(0.)²) ||I 0,121
where
0 = 0--1- ........... 0o. 8 2 1,
:= I, 0.
Observe that P (0.)- P., Using Lemma 6 we arrive at the main theorem.
Theorem [ For 2 we have the error estimation
[E. 1 < 0 001
a 22 |
1| mllllll aa- -1 2(a -
where 7t == max (1 + m0.))²)
0, 02112. and a = Tho represents the control regularization.
#St
In particular, the equality
Ke, xx
+ ||2
holds when all 0₂ are orthogonal.
Proof. The input perturbation z x.,0 Xg can be written as z z!l z where z E Z and zt E Z1, where z!l and - are vectors such that
z z almost surely.
z L have uncorrelated components.
z!l E D, and z- E DD_
Since z and z are orthogonal almost surely, recall Lemma 4
|| |xet x1112 = 0, ... 0
Gg)=
< 1 001
G =g)=
(18)
For the term [ ... G recall Lemma
[[ ... -.) = (( a² I+ (1 a)
a² ps 2
8=0
t-1
= lla'z + (1 a) Σ
a Poz + (1 (1-
Σ
(P PP))al3
s=0
s=0
= = (1 a² d')z' + (1 a)
a (P* Po)z
16

---

Published as a conference paper at ICLR 2021
in the above, Po is orthogonal projection on 0 (input data space), therefore, Puz z Furthermore, when s = 0, P. Po- 0. Thus,
[| ... =
= + -1- +1 -1
a)²² Σ a'a (P: Po)) (P. Po)z
s,9=1
+ - 12 a) Σ a z (P - -Po)z
s=1
+ + -0) - Σ 0z ( -Po)z,
= = 112211 12 (a + 2a (1- + (1 - (1-a") |z
[-)
+ +11- a)² Σ
a d' (P: Po) - +o)a+ +
20
- 0)
Σ
0 2 (P: -Po)z
8,9=1
8=1
t-1
+ 2011-a)( a) Σ
(z (P= Poiz
1-1
a 2 113 +|121112 + (1 - a)²
Σ
a''azz - Po) (P-
8,4=1
+ +2a a) Σ a'z a'z Po)z 2(1 - 2(1-aa)(1 a Σ (z))) (P:
(P; - -Pozz.
8=1
Using Corollary
[ we have
x. (P: - -Po)z < |2
|P -Pole,
S <111zl
ZT TP: TO) Po)z < |z|2 PP: -Polll [P -Pollz.
(z))) (P: -Po) - 71|1z1112 ||z||2
I rel12
Thus, we have
[ ... GB)=112 5 at |z a2|
11 2 21 -1-at
+ 2a a1)( - 0 1
= a
21 +| (1 2(a 2(a))
+
Recall the error estimation in Eq.
18
Xx_ X2 1 <
00-10-_--0
0 ||[ G.2
a 11 + 70l1 1---a -a--133 + 2(a a')))
0
0
17

---

Published as a conference paper at ICLR 2021
Table 4: ResNet for Both CIFAR-10 and CIFAR-100
Structure Initial Layer
Configuration
Conv2d (input channel 3, output channel = 16, kernel size = 3),
Batchhoorm2dchhannel 16),
Relu()
Residual Block
[Conv2d (input channel 16, output channel = 16, kernel size = x 3),
Batchhoorm22(channell 16),
ReluO.
Shortcut() x6
Residual Block
Conv2d (input channel = 16, output channel = 32, kernel size = 3 x 3),
BatchNorm2d(channee 16),
Relu()
Shorteut(,,
(Conv2d (input channel = output channel = 32, kerne size 3 x 3),
BatchNorm2d(channee = 32),
ReluO.
Shortcut(}} x5
Residual Block
Conv2d (input channel 32, output channel 64, kernel size 3 3), Batchhoorm2dchhanee = 64),
Relu().
Shortcut(),
(Conv2d (input channel 64, output channel 64, kernel size = 3 x 3),
BatchNorm22dchaneee = 64),
ReluO), Shortcut()) x5
Final Layer
Fully Connected (64, 10)
In the specific case, when all 0g are orthogonal,
= max
Stt (1 + xc0))²))
II- 0 0.1la
0.
Thus,
X xx
22
|
B APPENDIX B DETAILS OF EXPERIMENTAL SETTING
B.1 NETWORK CONFIGURATIONS
Since the proposed CLC-NN optimizes the entire state trajectory, it important have relatively smooth state trajectory, in which case, when the reconstruction loss [x,( xx at layer is small, the reconstruction losses its adjacent layers should be small. For this reason, we use residual neural network (He et al. 2016) as network candidate to retain smoother dynamics. The configuration of the residual neural network used for both CIFAR-10 10 and CIFAR-100 shown in Tab.
Based on the configuration of residual neural network shown in Tab. we construct embedding functions applied at input space, outputs of initial layer, residual block and residual block 2. The output of residual block 3 is embedded with a linear orthogonal projection. We randomly select 5000 clean training data collect state trajectories at all 5 locations.
For the linear orthogonal projections: we apply the Principle Component Analysis on each of the state collections. We retain the first r columns of the resulted basis, such that arg minfi t 2 1 6). where g 0.1.
18

---

Published as a conference paper at ICLR 2021
Table 5: Convolutional Auto-Encoders
Structure
Configuration
Encoder
Conv2d (input channel C1. output channel 02, kernel size 4 stride
2 x 2, padding = x 1).
ELU(alpha=1), BatchNorm2d(channee = c2).
Conv2d (input channel C2, output channel C3, kernel size = 4, stride 2 x 2, padding =
(LU(alpha=1).
Decoder
ConvTranspose2d (input channel C3. output channel c2, kernel size
4 x 4, stride = 2 x 2, padding = x 1),
ELU(alpha=1),,
ConvTranspose2d (input channel = C2. output channel = c1, kernel size 4 4, stride x 2, padding 1 1),
Auto-encoder
Index
Channel
13, 38]
[16, 36, 72]
[16, 36, 72]
[32, 128, 256]
Dimensions
[cr, C22 Ca.
For the nonlinear embedding: we train 4 convolutional auto-encoders for the input space, outputs of the initial layer and residual blocks 1, All of the embedding functions are trained individually. We adopt shallow convolutional auto-encoder structure to gain fast inference speed, which case, CLC-NN equipped with linear embedding often outperform the nonlinear embedding shown The configuration of all 4 convolutional auto-encoders are shown Tab. 5
in Tab. 1
B.2 PERTURBATIONS AND DEFENSIVE TRAINING
In this section, we show details about the perturbations and robust networks that have been consid- ered in this work. For the adversarial training objective function,
min A((o..))) (X
E
[(1
A) 4, y, A. (xr,xyy. 0)),
where (xxo é) generates perturbed data from given input Xo within the range of À balances between standard accuracy and robustness. We choose 0.5 all adversarial training.
For robust networks, we consider both perturbation agnostic and non-agnostic methods. For the perturbation agnostic adversarial training algorithms equipped Axxo, e). the resulted network that is the most robust against the A (Xo, e) perturbation. On the contrary, perturbation non-agnostic robust training methods are often robust against many types of perturbations.
Adversarial training with the fast gradient sign method (FGSM) (Goodfellow et al. 2014) consid- ers perturbed data as follows.
x = X + sign V xo (x),,))), (xo, y) ~ D,
where sign(.) outputs the sign of the input. In which case, FGSM considers the worse case within the range of along the gradient Vx, +xxT, y) increasing direction. Due to the worse case consideration, it does not scale well for deep networks, for this reason, we adversarially train the network with FGSM with e = 4, which is half the maximum perturbation considered this paper.
The label smoothing training (Label Smooth) (Hazan et al. 2017) does not utilize any perturbation information A (xxo,e, e). It converts one-hot labels into soft targets by setting the correct class as while other classes have value of -ー where is small constant and N is number of classes. Specifically, choose 0.9 in this paper.
19

---

Published as a conference paper at ICLR 2021
Table 6: Experimental results DenseNet-40 from standard training.
Accuracy: original without CLCC/CCC-N Nonlincar Emhedding Type of input perturbations
Dataset
None
Manifold
FGSM
PGD
CW
2
19
27757
0/52
57 79
CIFAR-
4
92 89
5/
20/
0/37
0/79
10
1/59
15/33
0/
0/79
2
8/25
8724
0725
3747
CIFAR-
4
70/64 64
2/24
4/
0/ 10
0/45
100
8
1/24
3/7
0/5
0/45
Adversarial training with the project gradient descent (PGD) (Madry et al. 2017) generates adver- sarial data by iteratively run FGSM with small step size, which results in stronger perturbations compared with FGSM within the same range e. We use 7-step of e = 2 to generate adversarial data for robust training.
For Perturbations, we consider the maximum range of 2, 4,8 8 to test the performance the network robustness against both strong and weak perturbations. For this work, we test network robustness with the manifold-ban based attack (Jalal et al.] (017). FGSM (Goodfellow et al. 2014). 20-step of PGD (Madry et al. 2017) and the CW attack (Carlini & Wagner 2017).
B.3 ONLINE OPTIMIZATION
Optimization Methods. we use Adam (Kingma Ba 2014) to maximize the Hamiltonian Eq. ! with default setting. In which case, solving the PMP brings extra computational cost for inference.
Each online iteration of solving the PMP requires combination forward propagation (Eq. backward propagation (Eq. (8) and maximization w.r.t. the control parameters (Eq. ()) which has computational cost approximately the same as performing gradient descent on training a neural network for one iteration. For the numerical results presented in the paper, choose the maximum iteration that gives the best performance from one of [5, 10, 20, 30,50) 50).
C MORE NUMERICAL EXPERIMENTS
The proposed CLC-NN is designed to be compatible with existing open-loop trained. We show extra experiments by employing the proposed CLC-NN on two baseline models, DenseNet-40 (Table
The layer-wise projection performs orthogonal projection on the hidden state. We define the local cost function at the 7th layer as follows
J(x, u,) 1 u + lull
the layer-wise achieves the optimal solution at local time
u (x:) arg min [xx,, uz). u.
However, the layer-wise optimal control solution does not guarantee the optimum across all lay- In Table launch comparisons between the proposed CLC-NN with layer-wise projection. Specifically, under all perturbations the proposed CLC-NN outperforms layer-wise projection.
D ROBUSTNESS AGAINST MANIFOLD-BASED ATTACK
The manifold-based attack Jalal et al. 2017 (denoted Manifold) has shown great success on breaking down the manifold based defenses (Samangouei et al. 2018). The proposed CLC-NN successfully defend such specifically design adversarial attack for manifold-base defenses and improves the robustness accuracy from 1% to 81% for the standard trained model in Cifar- 10, and 2% to 52% in Cifar-100.
20

---

Published as a conference paper at ICLR 2021
Table 7: Comparison between CL.C-NN and layer-wise projection.
Accuracy: Layer-wise projection / CCCC-NN LLnear emhedding
Dataset
Type of imput perturbations
None
Manifold
FGSM
PGD
CW
2
68 /
29/55
4/50 50
51775
CIFAR-
4
92 0 88
64
15
0/31
50/75
10
78
10/20
0/11
49/
2
45/
13/25
2/17
35747
CIFAR-
4
67/ 07/660 60
44/ 50
8/15
0/6
34/47
100
8
44/
5/9
0/1
34/47 47
We provide detailed explanation for the successful defense the proposed CLC-NN against such strong adversarial attack. Exsiting manifold-based defense (Samangouei al., 2018 focuses detecting and de-noising the input components that do not lie within the underlying manifold. The overpowered attack proposed in Jalal et al. (2017) searches adversarial attack with in the embedded latent space, which is undetectable for the manifold-base defenses and caused complete failure defense.
In the real implementation, the manifold- massef attack (Jalal et al. 2017) is detectable and control- lable under the proposed framework due to the following reason. The numerically generated man- ifold embedding functions are not ideal. The error sources of non-ideal embedding functions are mainly due to the algorithm that used compute the manifold, the architecture embedding func- tion, and the distribution shift between training and testing data (embedding functions of training data do not perfectly agree with testing data). In which case, even perturbation is undetectable and controlltol at initial layer, it is propagated into hidden layers, each layer amplifies such perturbation, therefore, the perturbation becomes detectable and controllable in hidden layers.
We randomly select the batch of testing data to generate the manifold-based attack following the same procedure proposed Jalal et al. (017). The proposed method improves the attacked accuracy from 1% to 78%. More specifically, orthogonal complement between perturbed testing data and its unperturbed counterpart, ||P, Xx. P where P is projection onto the orthogonal complement. The difference is growing such 0, 0.016, 0.0438, 0.0107, 0.0552 for hidden states layer 0, 1, 2,3, 3, 4 respectively. This validates the argument for how the proposed method is able to detect such perturbation and controls the perturbation in hidden layers.
compare the differences of all hidden states spanning the
Furthermore, we provide some insights about the reasons behind the success of such an adversarial attack. This follows the same concept of the existence of adversarial attack in neural networks. The highly nonlinear behaviour of neural networks preserves complex representative ability, meanwhile, its powerful representation results in its vulnerability. For example, constant function has 50% chance to make correct prediction in binary classification problem under any perturbation, but its performance limited. Therefore, we propose to use a linear embedding function that compensates between the embedding accuracy and robustness.
E DEFINITION OF THREAT MODEL
Generally, an attacker should not have access to the hidden states during inference, in which case, attacker is not allowed inject extra noise during inference. To define the threat model of the proposed method, for the white-box setting, an attacker has access to both network and all makes our method vulnerable
embedding functions. The condition that the perturbation defined as follows,
Σ (x),)))))) xx, 0, x..0 + c-
In words, the perturbation z applied on the input data must result 0 reconstruction losses across all hidden layers, which means its corresponding state trajectory does not span any of all orthogonal complements of all hidden state spaces. To obtain an effective attack satisfying the above equation, conventional gradient- gradient-based based attackers cannot guarantee to find perfect attack. A possible way is
21

---

Published as a conference paper at ICLR 2021
to perform grid-search backward in layers to find such an adversarial attack satisfying the thread model condition, which is extremely costly.
22


Reviews:
[{"review_id": "yAY2qHJlsnV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Keeping the performance of deep neural networks against data perturbations is an important and open problem. The authors propose an optimal control-based approach by taking dynamical systems perspective. The proposed method sounds intuitive and efficient. Authors supply theoretical analysis and (small) experimental evaluation. Overall, I believe paper is a good. However, I would like to get some points clarified:\na)\tAuthors used manifold assumption (which is a reasonable assumption for many problems) to define running loss (eq 3). (If I am not mistaken) They choose a quadratic loss to have a tractable optimization problem. However, under these assumptions, one may choose many different losses. Would you please comment on the form of the loss and its impact to the method?\nb)\tLet’s assume dynamical systems perspective is a right perspective for analysing deep neural networks (to be honest I don’t have any criticism about this). To use control theoretic tools, one needs to comment on controllability and observability of the controlled system. I suspect these mentioned properties are a function of the neural network architecture or do authors think the proposed method (as shown in Figure 1) makes each and every deep neural network architecture controllable and/or observable? I would like to hear authors perspective on these issues.\nc)\tAs I mentioned before, the empirical study is quite small, and I didn’t see any baseline (do I miss something here). Do authors consider extending their empirical study and compare their method with some baselines.\n\nI would like to emphasize one more time that, I am positive about the paper. However, I would like to note that I am not expert in the field and I am open to change my view in both direction.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "F0P39CaVzdU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper builds on recent revival in control-theoretic approaches to deep neural networks by proposing an adaptive controller that projects intermediate representations in the network to their \"manifolds\" and consequently makes the neural network robust to input perturbations. \n\nPros:\n\n+ Active controller-based projection of intermediate features is an interesting idea and the utility of Pontryagin's maximum principle to address the challenge of high dimensionality of the state (features) is a good observation.\n\nCons:\n\n- The manifold-based defense has been shown to be broken previously. For e.g. please see section 4 of https://arxiv.org/pdf/1712.09196.pdf . Manifold/GAN/VAE based defenses can be easily broken by just attacking the projection network and the original network together. The paper considers manifold based defense at all layers using an active controller doing the projection. While comparison with pixeldefense is useful, it would be good to launch an attack similar to the reference above and then observe the effectiveness of the approach. The arguments in the paper are not sufficient to convince the reviewer that this defense is practical. The paper is severely lacking in comparison with existing defense approaches. The state of the art for the used dataset in the paper is significantly better than the effectiveness of the presented approach. For e.g. see https://github.com/MadryLab/robustness \n\nIn summary, the paper is a good effort to exploit the use of active controllers in deep learning. But firstly, the use of manifold-based projection as a cost function is itself a non-robust defense against adversarial examples. Second, the experimental evaluation in the paper is significantly lacking and does not meet the standards of a venue such as ICLR. The reviewer will strongly recommend reviewing the advices in https://arxiv.org/abs/1902.06705 on this topic. At this point, the paper is interesting but it needs significant development and is not yet ready for publication. \n\nQuestions for the author:\n\n1. How critical is Pontryagin’s Maximum Principle to the presented approach? What prevents one from using projection to a lower dimensional embedding space followed by a state space control method? In particular, if one is building on the manifold assumption, then isn't it reasonable to not worry about high dimensionality for designing the controller too?\n\n2. Is it realistic to assume the \"input perturbation to be a random vector\" in Section 5 for theoretical analysis when we are considering adversarial attacks such as PGD, CW? If not, then isn't theorem 1 not relevant to the primary topic of the paper?\n\n------ After author's response:\n\n* The response of authors identifies the problem of using running loss in projected space. While one can try to get around it by projecting the loss function as well but that would be a convoluted way to solve the problem, and in any case, not a strong criticism of the presented approach. \n\n* The updated document has generalized the derivation to take general perturbations into account. \n\n* Updates Tables 1-3 resolve empirical analysis concerns of the reviewer. \n\nWith these improvements, the reviewer is happy to recommend acceptance of the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "B99a7Ug_XLw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n1. This paper first introduced layer-wised projection from the poisoned data to the clean data.\n2. The results show improvement of the robustness over the baseline on different types of attacks.\n\nWeakness:\n1. The statement of the closed-loop control is a little bit ambitious. The overall methods are the layer-wise projection from the poisoned data to the clean data manifold. Normally, in the closed-loop, we will use the control signal $u$ to control the original data instead of the next layer data. For example, the final balance stage should be $u=g(x+f(x+u))=0$. So closed-loop should be at least multi-steps within one layer. For different layers, the closed-loop control will have different control signals, since the dimension/ distribution between layers is much different. So this method is only a one-step layer-wise projection. The $x$ between layers cannot be viewed as the same sample to be controlled. I would recommend the author to change the statement from closed-loop into the layer-wise projection for a better suit. Also, this method is still a feed-forward network, not a \"loop\" control. I do think this method is interested, just a little ambitious. This could be a useful extension of the resnet-based network, since the control $u$ can be viewed as a complicated version of residuals.\n2. The experiment is weak for only comparing with one baseline. Also, can the author provide which baseline model that the author is comparing with? I cannot find it in the text. I would appreciate it.\n3. It's unclear to me about the training of $\\mathcal{E}(x)$, will this requires extra data to train? What's the running speed of this \"closed-loop\" method compared with others?\n\nSome tiny comments:\n1. A trivial comparison would be to train an autoencoder for each layer of $x$ and only use the decoded results to pass through the network. This in principle learns the data manifold and provides the projection to this manifold. \n2. Table 2, the dataset name is not aligned in the center.\n3. Table 3 should be more self-explainable. It's a little confusing in the current form.\n4. Have the author tried multi-steps in a single layer or constraint $x_t$ to be in the same space? \n\n\n\n----- post rebuttal -----\n\nThe authors addressed most of my concerns and the revision is better than before. \nI would like to increase my score and would recommend an acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2gKHHHY6YP3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis study develops a closed-loop control strategy to improve robustness of neural networks to adversarial attacks. The study is technically sound and the empirical results on classification tasks are convincing.\n\n\n### Quality\n\nThe paper is technically sound and the claims are appropriately backed by empirical evaluation. However, I would recommend the authors to discuss a bit more the additional computational cost of running the closed-loop method.\n\n\n### Clarity\n\nThe manuscript is clearly written and provides enough information for an expert reader to understand all the steps to reproduce the results.\n\n\n### Originality\n\nThe novelty of the study resides in the development of a closed-loop control method for increasing the robustness of neural networks. The strategy is devised to scale to the typical high-dimensional nature of neural network activations.\n\n\n### Significance of the work\n\nThe results suggest that the developed approach is a solid step towards developing robust neural networks.\n\n\n### Some typos:\n\n-instead of \"cause different data distribution deviating\", \"cause data distributions to deviate\";\n\n-instead of \"The resulting control policy [...] make it\", \"The resulting control policy [...] makes it\";\n\n-instead of \"the embedding are effective\", \"the embeddings are effective\";\n\n-instead of \"the perturbed states in Fig.2 [...] has\", \"the perturbed states in Fig.2 [...] have\";\n\n-instead of \"to obtain all the intermediate hidden states [...] and accumulates\", \"to obtain all the intermediate hidden states [...] and to accumulate\";\n\n-issue with reference \"E. 2017\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to ICLR.  The reviewers and I are in agreement that the work presents some interesting connections between closed-loop control and stabilization of activations to an observed manifold.  Specifically, the idea of using optimal control dynamic programming techniques to compute optimal adjustments to ensure control on this manifold is an interesting one and may have other implications within deep networks.

Although the reviewers were convinced by the experiments on robustness, I remain a bit skeptical here.  The results show that while the method marginally improves robustness to small-epsilon perturbations, the models are still quite non-robust against the size perturbations frequently used in assessing adversarial robustness (e.g., to eps=8 perturbations on CIFAR10, where the best approach gets ~11% accuracy against PGD attacks).  It doesn't really matter how well a defense works against sub-optimal attacks: if PGD is able to decrease its accuracy this much, clearly the model is not very robust (and it seems upon reading that only 20-step PGD, with no restarts, was used as an attack, which is a fairly weak variant of PGD).  Furthermore, the approach didn't improve much upon PGD-based adversarial training when combined with it, either, overall suggesting that the impact on robustness is somewhat minor, and needs to be evaluated quite a bit more.

While I don't believe these concerns are substantive enough to override the beliefs of all reviewers, I think that the authors could do a much better job of evaluating the actual robustness of these models (following the advice of https://arxiv.org/abs/1902.06705).  And if the resulting metrics are not as strong as hoped for, then it would be good to evaluate other possible benefits of the approach (perhaps to random distribution shift? it seems a much more likely situation for there to be real gains?).  Thus, while I believe the paper has some interesting ideas, I think the authors should probably tone down some of the current claims of improving adversarial robustness unless they can provide a much more thorough evaluation.
Decision: Accept (Poster)