A framework of deep neural networks via the solution operator of partial differential equations
Authors: Wenqi Tao
Zuoqiang Shi
Venue: ICLR 2022 Submitted (2022)
Keywords: deep neural networks
partial differential equations
solution operator

Abstract: There is a close connection between deep neural networks (DNN) and partial differential equations (PDEs). Many DNN architectures can be modeled by PDEs and have been proposed in the literature. However, their neural network design space is restricted due to the specific form of PDEs, which prevents the design of more effective neural network structures. In this paper, we attempt to derive a general form of PDEs for the design of ResNet-like DNN. To achieve this goal, we first formulate DNN as an adjustment operator applied on the base classifier. Then based on several reasonable assumption, we show the adjustment operator for ResNet-like DNN is the solution operator of PDEs. To show the effectiveness for general form of PDEs, we show that several effective networks can be interpreted by our general form of PDEs and design a training method motivated by PDEs theory to train DNN models for better robustness and less chance of overfitting. Theoretically, we prove that the robustness of DNN trained with our method is certifiable and our training method reduces the generalization gap for DNN. Furthermore, we demonstrate that DNN trained with our method can achieve better generalization and is more resistant to adversarial perturbations than baseline model.

Under review as a conference paper at ICLR 2022
A FRAMEWORK OF DEEP NEURAL NETWORKS VIA THE SOLUTION OPERATOR OF PARTIAL DIFFERENTIAL EQUATIONS
Anonymous authors Paper under doublc-blind review
ABSTRACT
There is a close connection between deep neural networks (DNN) and partial differential equations (PDEs). Many DNN architectures can be modeled by PDEs and have been proposed in the literature. However, their neural network design space is restricted due the specific form of PDEs, which prevents the design of more effective neural network structures. In this paper, we attempt to derive a general form of PDEs for the design of ResNet-like DNN. To achieve this goal, we first formulate DNN Then based on several reasonable assumptions, we show the adjustment operator for ResNet-like DNN is the solution operator of PDEs. To show the effectiveness the general form of PDEs, we show that several effective networks can be interpreted by our general form of PDEs and design a training method motivated by PDEs theory to train DNN models for better robustness and less chance of overfitting. Theoretically, method is certifiable and our training method reduces the generalization gap for DNN. Furthermore, we demonstrate that DNN trained with our method can achieve better generalization and is more resistant to adversarial perturbations than the baseline model.
adjustment operator applied on the base classifier.
prove that the robustness of DNN trained with
INTRODUCTION
Deep neural networks (DNN) have achieved success in tasks such as image classification (Simonyan Zisserman 2015), speech recognition (Dahl et al.) 2012), video analysis (Bo et al. (011). and action recognition (Wang et al. 2016). Residual learning (He et al.) 201 b) has revolutionized DNN architecture design, making it practical to train ultra-deep DNN. and has ability avoid gradient vanishing. The idea of residual learning has motivated the development of powerful DNN such as WideResNett (Zaoruykoo & Komodakis, 2016), ResNeXt (Xie al. 2017). and DenseNet (Huang et al. 2017).
So how to understand Residual learning from a theoretical perspective? In (Weinan 2017). the author formulated the forward propagation of ResNet as a dynamic system and in (Li & Shi] 2017] Haber et al. 2018 Li al. 2017) bridge connection between partial differential squations(PDE) and ResNet. In fact, ResNet is composed of series of residual mapping. Each residual block is
realized by adding a shortcut connection to connect the input and output of the original mapping (F). Mathematically, the n-th residual mapping be formulated
Un++1 = xr, + F(x,, Wn)
(I)
with To E Rdd be a data point. H, is the input to the n-th residual mapping and F(x,, w.) is non-linear function parameterized by w, which can be learned by back propagating the training error.
The prediction of the data point To can be expressed by 0 = softmax "C L.). By introducing temporal partition: t, = nTLL, for xtt,,) = In and w(t,)) = wn. Without considering the dimensional consistency, ResNet can be formulated as:
0, .., L with the time interval At T/L and let
x((n+1) = xl(n) + F(x((n), w(tn)) At, n 0,
At.

---

Under review as a conference paper at ICLR 2022
which is the forward Euler discretization of the following equations with time step At, that is, dx(t)) = v(z((t) t)dt, m(0) = = t E T),
where (tn(t,)) fn) = (t,) wtt,)) and L is the number of resdiual mapping in a ResNet architecture. The prediction can be expressed by 9 = softmax( w FC (T)). The equation defines the characteristic curves of the following TE
du a1
(æ, () = v( 1) Vu(a,, (), æ E ER..
(3)
So the prediction ý of ResNet can be seen as the solution u(x, T of following Transport equation
at (ar, 1) = v( () Vu(r, (), x E IRd.
(4)
u u(1 0) softmax ( PC x)).
In fact, not only ResNet, similar to the process above, many ResNet-like DNN can correspond to PDEs. For example, it has been shown (Lu et al. 2018 that PolyNet (Zhang et al. 2017) and FractalNet (Larsson et al. (2016) can be respectively considered numerical schemes to solve PDEs system. Based on the link PDEs and ResNet-like DNN, many theoretical results to understand ResNet-like DNN were proved. In (Chen et al.
2018), continuous form for ResNet called Neural ODE was proposed. After that, the property of eural ODE has been theoretically analyzed (Thorpe) & van Gennip 2018 (Avelin & Nyströml 2021 Zhang al. (019a)). Based the stability theory of differential equations, many architectures has been designed to build more robust deep networks (Zhang al., 2019b Yan al. 2019 Reshniak & Webster 2021 Not only stability theory, inspired by the numerical scheme of differential equations, in (Zhu et al. 2018 Tao et al. 2018). various deep network architectures to improve testing accuracy have been proposed in success.
Despite the success of the application of PDEs in residual learning, these neural network design space in the literature mentioned above is restricted due to the specific form of PDEs, which prevents the design of more effective neural network structures. Existed works can not meet these requirements. So, a natural question is that can we find the general form of the PDEs corresponding the ResNet- like DNN which can unify existed model as well as guide new effective DNN architecture design?
To achieve this goal, in this paper, we follow the setting of paragraph 2, the softmax model and ResNet are the initial value and the terminal value of PDEs equation 4 respectively, so we can view ResNet as an evolution from a softmax model. Without loss of generality, we indicate this evolutive relationship by an adjustment operator denoted by Tr, then we have
T softmax( ( FC xx) - ResNet(xx.
If we denote the softmax model by f. then the ResNet-like DNN can be expressed as Tf(f) and T, is the solution operator of PDEs equation 4 To obtain the general form of the PDEs corresponding to these ResNet-like DNN, we list series of formal properties likely to be satisfied by the operator Ti, and we deduce an explicit formulation from them. We see that under reasonable assumptions, the new model u(3r, t) (Tif)
the solution of convection diffusion equations. Then, motivated by the theory of PDEs and the framework of our general form of PDEs, we design a training method for ResNet-like DNN models for better robustness and less overfitting risk. Our training method is summarized as follows. First, we train ResNet-like DNN model using natural training. Because calculating the real solution (z)(())) is difficult, we next train a new ResNet-like DNN denoted by g(r !) to approximate (z)). At last, we choose g( T) as our final classifier.
We make the following contributions:
Under reasonable assumptions, we build a PDEs framework for ResNet-like DNN with the help of operator T, and show some effective models that can be modeled by our framework.
We design training method for DNN based the theory of PDEs. This is essential contribution of this paper due to certified adversarial robustness and theoretical analysis of generalization ability for our training method.
We verify our training method then demonstrate that a ResNet trained by our method is more resistant to multistep l and foe adversarial attacks than natural trained ResNet.
synthetic dataset and the CETAR--O/CIFAR--O 100 datasets,
2

---

Under review as a conference paper at ICLR 2022
2
MAIN RESULTS
We show under several reasonable assumptions that the adjustment operator T; is the solution operator of convection diffusion equations. Based on the regularity theory of PDEs, we design training method for DNN models and theoretically illustrate its benefits for improving the robustness of models and reducing the generalization gap.
2.1 GENERAL FORM OF PDES
We denote the base DNN model by (xz). which is uniformly continuous. We assume data points are
distributed in D C Rd and label function ((x) is defined in D but known only training set SN E D. Our method formulates the DNN continuous dynamic system, namely u( ff) E 7]. To get the expression of the adjustment operator Tr, we assume has some fundamental properties.
[Locality] (Ti(f) (z) o(t) t + 0+, for all f.g such that D fx) D g²) for all |a] 2 O.
[Comparison Principle] ((()) s (g) for all 2 0 and f.g such that f f g.
[Markov Property] Te+s Ts s, for all s,t 2 and 8 s T. Tenstte the flow from time s to time f +
[Spatial Regularity] There exist a positive constant C such that - (Tef)ll CCh for all h in Rd, 1 2 0, where (Tx f(x + h) and ||h||2 = h.
[Temporal Regularity] The following two properties hold: 1) For all s, + E [0. 7] and all Q, there exist constant C 2 0 depending only Q such that (TTts, - g - S Ct, where Q is the subset of Cg defined by
Q = { E Cp,, Vn 2 0, D f S C for all |a| mel=n] n3.
2) For all t, uniformly s E (0,
s, t. + E 00, T). there exist constant C 2 0 such that [[T++s,(f) - -Tao(f)l I Cst
[Ensemble Additivityl T:(f + g) Ti()) + Tx(o), and if C' is constant, then T,(C) =C
To illustrate the reasonableness of these assumptions above, we give the following explanation. We first require that T, can be expressed as solution operator of PDEs and for small enough f, the value of T.()) at any point æ is determined by the behavior of the base classifier / near æ, which implies [Locality]. We then require the operator T, is provided with an order-preserving property which means that no enhancement is made. Thus if the confidence level for an event of one base classifier g is higher than another base classifier f. this ordering is preserved, which implies [Comparison Principle]. Next we require Ti+s can be computed from T, for any S f, and To is of course the identity. This is natural, since the future behaviour of the adjustment process at terminal time + is likely to be deduced from the state at intermediate time f without any dependence upon the original model f. which implies [Markov Property]. We need function T2( be spatial stable. If add a perturbation l to data point æ, the output (x + h) will not change lot, which implies [Spatial Regularity]. We also need an additional property for giving some temporal stability the operator Tg. Namely, in any time interval, the adjustment will not be rapid, which implies [Temporal Regularity]. Assume an ensemble classifier F consists of base classifiers and g. and another ensemble classifier consists of base classifiers T, (J) and (g). We require the prediction (F) to equal (f( + Tig),, which implies [Ensemble Additivity].
Theorem 1. Under the above conditions, there exists Lipschitz continuous function v Rd (0, T] + d and Lipschitz continuous positive function d x [0, + gddd such that for any bounded and uniformly continuous base model fw (x)),, u(x, t) = Te(fw) (x:) is the unique viscosity solution of the following convection diffusion equation:
Bu((r.) -v(x, Vu(x, + E Oie on,dn dt
af (xx, (),
(5)
u(xx 0) (x)(
where E IRd, [0, T). Here oig is the j-t element of matrix function a (xr
3

---

Under review as a conference paper at ICLR 2022
We will provide the proof of Theorem in Appendix A In this subsection, introduce PDEs framework for DNN models. The framework is quite general, many existing models with residual connections be modeled by our framework. As examples, we briefly list some of them in the next subsection.
2.2 SEVERAL MODELS
In this subsection, we will show Gaussian noise injection Wang al., 2020 Liu al. 2019).
randomized smoothing et al. 2019; Li et al. 2018 Salman et al. 2019) and ResNet with
stochastic dropping out the hidden state of residual block alock(Srivastave) et al. 2014 Sun et al. 2018) can be interpreted by our framework.
Gaussian noise injection: Gaussian noise injection an effective regularization mechanism for DNN model. For a base ResNet with L residual mapping, the n-th residual mapping with Gaussian noise injected can be written as
1n+1 n. + F(rn, wn. + ae En NN I),
where the parameter
noise coefficient. By introducing temporal partition: t. nT/LL. for n = 0, 1..., L with the time interval At T/ L and let attt,, = H. and w(f,) wn. And let ov A and F(a,, w.,) / A/ vxx, 1). This noise injection technique in discrete neural network can be viewed as the approximation of continuous dynamic
dx(t) v(a,, f)dt + odB(t)
(6)
where (t) is multidimensional Brownian motion. The output of I--th residual mapping can be written Iřo process equation at terminal time T, =(T). So, an ensemble prediction over all the possible sub-networks with shared parameters can be written as
E(softmax (o (=0)) = z0).
According to Feynman-Kae Kac formula 2007), equation equation is known to solve the following convection diffusion equation
du( v(x, Vu + JorAu, L [0,
u(x 0) softmax(w1PC0)
Randomized Smoothing: Consider to transform a base classifier into a new smoothed classifier by adding Gaussian noise to the input when inference time. If denote the base classifier by f(x) and denote the new smoothed classifier by g(x). Then f(x) and g(x) have the following relation:
N
g(x)
Σ
f(x + a;) xe Ee-woan +)]
(8)
According to Feynman-Kac formula, g(x) can be viewed the solution of the following PDEs
@u(a..) Au,, E [0,
uu( uux
f(x).
Especially, when f(x) is ResNet, the smoothed classifier g(x) = u(æ, T + 1) can be expressed as
du(æ.))
Φt 1 [TT++
du( u(z,t) Vu(æ, t), x Id., t e [0,
8t
u(x; 0) = so
Dropout of Hidden Units: Consider the case that disables every hidden units independently from Bemoulli distribution B(p) with (0, 1) in each residual mapping
Zn
Tn In F(xn, w.)) 1
In + F(xn, wt) F(zn, wn) Zn
I),
4

---

Under review as a conference paper at ICLR 2022
where Zn ((1, namely P(z,, 0) = 1 p. P(,, =) and @ indicates the Hadamard product. If the number of the ensemble is large enough, according to Central Limit Theorem, we have
F(ons Wn) o Zn - = w.) o NNO0 -))) p
The similar way with Gaussian noise injection, the ensemble prediction can be viewed as the solution u( T)) of following equation:
@u(a.1) v(x,t) Vu(x, +
1-p 2p
8e (z, E.(0
(10)
u(x 0) softmax(w#*
where x E IRd, 1 [),
Remark 1. In fact, similar to dropout, shake-shake regularization (Gastaldi 2017 Huang & Narayanan 2018) and ResNet with stochastic depth (Huang et al. 2016) can also be formulated by PDEs framework.
The viewpoint that forward propagation of ResNet is the solving process of TE enables to interpret poor robustness and generalization ability as the irregularity the solution. Morever we can also interpret the effectiveness of the models bhove-mentioned" in improving generalization well as robustness as the action of diffusion term. To mitigate this issue, based on our PDEs framework and the knowledge that the terminal value of convection diffusion equations (whose diffusion term is isotropic, namely a (æx 1) is I² is more regular than initial value, we use the ResNet fw (x) as the base classifier and use the solution operator of the diffusion equation
Bu(w..) = v(x, t) Vu(x, t) + o²Au, x E Rd Rd, E [0,7] 8t
(1I)
u u1x 0) (z).
as the adjustment operator T, to smooth the ResNet for improving the robustness and generalization ability. The case that diffusion term (ar, 1) is anisotropic (namely (ar, t) A I²I) is difficult for theoretical analysis and practical experiment and we take the case as our future work. In next subsection, we will illustrate the benefits of using the adjustment operator T, from theoretical perspective.
2.3 ROBUSTNESS GUARANTEE
Consider a classification problem from Rd to blue label class V. Let G be classifier defined by G(x) arg m w' (x. T), where u' xxx T) is the i-th element of u(x T). Suppose that the new DNN classifies ø the most probable class CA is returned with probability PA = u (x, (x,T), T), and the "runner-up" class is returned with probability pn- Our main result of this subsection is to estimate around the data point how large l radius make the classifier G(æ:) be robust, where 1 < P < CO.
Theorem 2. Let the base model fw (x) be compactly supported function functiio and velocity u(æ, t) Ca (gd [0, 77). Let the new DNN u(æ T)) be the solution of equation 11 Suppose CA E y and
w (x, =AA PB
max ux; (x,T), ifes
then G(x for all ||0||p < R, where
(pa PB)
R
2 max, (x)
1/p+ + 1/9 and r is constant depending on Vv.
We provide the proof of Theorem in Appendix B According to Theorem 2 we can obtain the certified radius R is large when the diffusion coefficient o² and the probability of the top class is high. Not only robustness, comparing to the base classifier, but we will also show the generalization ability of model 11 will be better in next subsection.

---

Under review as a conference paper at ICLR 2022
2.4 GENERALIZATION GAP
s Ye yi)li= 1 is drawn For simplicity, we consider binary classification problems. Suppose SN = ( from X Y D (-1,+1) with X and Y being the input data and label spaces, respectively. Assume p(x) is the underlying distribution of X, which is defined in D. We also assume label function Label D >> {-1,+1}, which is unknown. Let C V be the hypothesis class of the DNN model, where V is another space that might be different from Y. Let & V x Y - [0, BJ be the loss function and B is a positive constant. Denote function class & := { y) + E( (f). y) h E H}. Rademacher complexity is one of the classic measures of generalization error. We first recap on the definition of Rademacher complexity.
Definition 1. (Ledoux Talagrand 2013 Let H + IR be the space of real-valued functions on the space X. For a given sample SN {1x,, 1k) ) the empirical Rademacher complexity of H. is defined as
N Σ
RSN (H) = E. Ssup WEH
where 01, 02, ....... oN are i..dd Rademacher random variables with P(o, -1)
=
Rademacher complexity is tool bound the generalization error. The smaller the generalization gap is, the less overfitting the model is. For Viz E R2 and constant 2 0, we consider the following two function classes:
F: === f(=((llIlllle>> scj.
and
0u( v(xx t) Vu, t) + o² o²Au, u(x; 0) = f(x) where f E FF}
Go = ==(((x== u(x: 1)
Ot
The function class FF represents the continuous analogue of ResNet, and function class Go denotes F adjusted by operator T,. Then have the following theorem.
With probability least 1 1/(2N) we have Theorem 3. Given training set SN {(,,, M.)
C
Rsw (F) S [log + log 1/2 +
VN
C'
RSw (9x) < VN
/ 11-0" + Inc In N)2/2 + exp(-o
Here C' is positive constant depend on DD and dimension d.
We provide the proof of Theorem 3 in Appendix C According to Theorem 3 we can obtain that for function class Go. the upper bound of Rademacher complexity is low when the diffusion coefficient o² is high. We next present a practical algorithm for real applications to verify our main results.
3 ALGORIHTMM
Let SN {x11 y1), (In, wn)} be training set, where æ; is data point and y; is the correspond- ing label. Then denote the ResNet with natural training by fw (x) whose parameters can be learned by minimized the loss (CE(),, fwr (a:)), where ICE is cross-entropy loss. For convenience, assume the velocity v(x, t) is zero and the diffusion term o(x f) is III. We can rewrite equation
Bu(m.t) dt = A'Au( t), x E Rd, 1
(12)
u (1x 0) (x)
where a² is a hyperparameter. In this model, we set the natural trained ResNet the initial value.
Instead of calculating the solution of equation 12 we train a DNN gro (x. t) to approximate the real solution. The parameters of new DNN Iw xx, are different from the initial ResNet (x).
Therefore, to train the DNN model Ino (z, t). we design loss terms on training set Sy for the initial condition and differential equation.

---

Under review as a conference paper at ICLR 2022
where fe is cross- We use (gw (z,, 0). fa.())) to fit the initial condition of equation 12 entropy loss. To improve the natural accuracy, we enhance the terminal value to fit the label, i.e. (c (2, g)) We can write the first loss term
1 N
L1 (9 SN) =
(gw 0), (w.)))) + 1cE (gw R),l)l
(13)
We use square error loss fit the differential equation and improve the robustness of our model,
N
(Dye Öt
2
L2 Sx)
Σ
(a,, 8) o² (I5. 8) ds.
(14)
N
1=1
According to the trapezoid formula, the integral term equation 14 is approximated by
Le(sw, SN) R
1
Σ Σ
dgw Oi
(a,, 8) o² Dgw
2
(15)
2N
0,
Using Taylor formula, the Agn (2: 8) can approximated as
1 M Σ (oo + herj, 8) + gw her,, ;)
Ag 8)
(16)
h2
where {Er,} } is ii.d and M is the average number. In practice, we choose the number of average M equals to reduce the computation cost, and set to 0.1 because training converges with difficulty when h is too small. We denote the right hand of equation 16 by An.e9w (at,, s). Combining equation 15 and equation 16 the integral term equation 14 can be approximated by
M.M (grn SN)
1 N
Σ Σ
Ogw al
(æ;, 0) o2 An.
2
(17)
- ==
training loss combines T (1w SN) and L L.M. (gm, Sv), i.e., Loss(gw, Sy) = Li (9m. SN) + AL (gw Sx), where > coeffi- cient to trade off the two loss terms. In practice, use stochastic gradient descent to optimize Iw (x, t), and we use Iw (ax, T') as the final model.
Putting the two objective functions together,
EXPERIMENTS
In this section, we will numerically verify that 1) can our method improve the robustness and reduce overfitting risk of DNN? 2) what is the advantage of our method over the Gaussian noise injection and randomized smoothing?
4.1 PRELIMINARIES
Datasets. In our experiments, we consider both Half-moon and CIFAR 10/CIFAR100 (Krizhevsky ct al. alll||2000 2009) datasets. Half-moon dataset is a randomly generated 2d synthetic dataset in which we randomly generate 500 points and 1000 points with standard deviation of 0.3 as training set and testing set, respectively. Both CIFAR and CIFAR 100 contain 60K 32 32 color images with 50K and 10K of them used for training and test, respectively. CIFARI0 and CIFAR100 contain 10 and 100 different classes, respectively.
Project gradient descent attack (PGD). To verify the efficiency of method for improving the robustness of DNN, we consider the project gradient descent (PGD) (PGD))Madry et al. 2017) in all the experiments below. Before introducing PGD, we first introduce fast gradient sign method attack (FGSM). The FGSM attack searches the adversarial image x' by maximizing the linearized loss function function L(x', y) L(x, y) (a y) T(x' x)) with constraint ||æ. æ:lp S c. where is the maximum perturbation and 2, cc. PGD iterates FGSM with step size and clips the perturbed image to generate the enhanced adversarial attack.
go0 xr (m) Clip for (m-1) asign( V (m-1)) .y)),,
7

---

Under review as a conference paper at ICLR 2022
Na)rataaal training (b) A - 0.01, = 0.01 (c) -0,2=0110000.1
0.01 (d)
0.1,
Figure 1: Decision boundary of natural trained ResNet and ResNet trained by our method with different hy heprpaarametess and o².
86
68
89.4
66.5
66.8
70.2
10.2
6.2
66.5
50.2
341
CH1
(a) 4a) (() (%)
(b) Ao (test(99 (%)
(c) Arrin
Ateert (%)
Figure 2: Performance of ResNet trained by model with different hyperparameters À and o² on Half-moon dataset.
(L(x(m-11) }}
@ (m) = Clipp,. fx (m + a
|| [ v)1l2)
where m = 1,.... K, (0) x. and let the adversarial image be x'' (K) with K being the total number of iterations. For the Half-moon dataset, we apply 20 iterations PGD attack (PGD22) with step size 0.01. The perturbation constrained to be 0.2 under t norm. For the LIFAR-1O/CIFAL dataset, when under fo norm attack, we apply 20 iterations PGD attack with step size 1 1/255 and when under 2 norm attack and we apply iterations PGD attack (PGD50) with step size 0.1.
Performance evaluations. For the Half-moon dataset, we consider three indicators evaluate the performance of the model: natural testing accuracy (Ateatt) robust testing accuracy (Atess), ) and the gap between training accuracy and testing accuracy(A!a) (A Ared) The natural accuracy and robust accuracy are measured on clean and adversarial images, respectively. The robust accuracy used to evaluate the robustness of the models, and the difference between training and testing accuracy used to evaluate the overfitting risk of the models. For the CIFAR-10/CIFAR-I10 datasets, we consider Ateat and Ateat
4.2 EXPERIMENTS ON SYNTHETIC DATASET
In this subsection, we numerically verify that the efficacy of our training method in improving the robustness and generalization ability of ResNet on the Half-moon dataset.
We first varied hyperparameters X and 6² and present the final results in Figure 2 The three indicators (natural accuracy, robust accuracy, and difference between training accuracy and testing accuracy) for ResNet with natural training were 88. 88.6%, 62.3%, and 10.4%, respectively. Comparing natural trained ResNet, the robustness and generalization ability of ResNet trained with our method were improved. On the other hand, when À and x² increased, the natural and robust accuracy increased, while the gap between training and test accuracies decreased. Thus we can conclude that increasing and o² are helpful for improving the performance of the models.
We plot the decision boundary of ResNet trained with different X, o² and natural trained ResNet in Figure From Figure that our models is smoother. These experimental results are consistent with our theory.
that the decision boundary of natural training is irregular, while
4.3 EXPERIMENT ON CIFAR-10/CIFAR. 100 DATASET
We further test the performance of ResNer20/ResNet56 trained with our method on the Cl- FARIO/CIFARI00 dataset. During training, apply data augmentation including random crops and flips. We rum 200 epochs. The batch size is 128 and the initial learning rate 0.1, which decays by factor of 10 the 100th, and 150th epochs.
8

---

Under review as a conference paper at ICLR 2022
A Fest
Models
Hyperparameters
Atsa
4/255 (1
8/255 (1x0)
0.5 (1-)
CIFAR 10 Dataset
0.01, 0²
86.72%
40.13%
7.8%
45.33%
ResNet20
0.05, o²
84.78%
45.25%
13.03%
50.57%
= 0.05, o² 82.49%
48.29%
17.79%
53.87%
= 0.01, o² 87.76%
41.85%
8.1%
46.17%
ResNet56
0.05,
86.01%
47.42%
14.73%
53.05%
0.05, 0² 83.03%
48.73%
18.02%
55%
CIFAR 100 Dataset
0.01,
0.05, 0² 58.77%
19.56%
3.91%
23.98%
ResNet20
55.7%
21.79%
5.56%
26.55%
= 0.05, 0² = 52.73%
23.33%
8.27%
28.42%
= 0.01, d²=
60.2%
19.75%
4.49%
24.93%
ResNet56
0.05, 0²
57.82%
23.26%
6.8%
27.82%
À 0.05, = 53.99%
25.19%
9.41%
29.55%
Table 1: Natural and robust accuracies of ResNets trained using methods with different hyperparameters on the CIFAR-IOCCIAR-III
A
Models
teett
6 4/255 (130)
8/255 (100)
0.5
Gaussian noise injection
89.9%
22.81% %
12.38%
0.1%
Our ResNetl 10
84.35% 93.58%
50.76% %
19.4%
56.38 %
ResNet1 10
0%
0%
0.01 %
Table 2: Comparing the robustness against -nom and -noom constrained adversarial perturbations CIFAR-10.
There are two hyperparameters X, o² in our algorithm that need to be determined by a large number of experiments. In order to demonstrate the influence the two hyperparameters, we train ResNet by using our algorithm with different \, o², then list the result in [ On one hands, from Table 1. we can see that the parameter o² is fixed, the natural accuracy decreases and robust accuracy increases as the increasing of parameters À and the converse to be true. On the other hand, comparing to the result of ResNet20 and ResNet56, we can see that the performance our proposed method will be better in deeper networks.
In order to quantitatively measure the performance of our proposed method, we compare the proposed method with other defensive method without adversarial training. The result of comparison is shown in Table 2 For our method, we choose o² = 2 and À = 0.5 to train ResNetl 10. For the Gaussian noise injection mentioned in subsection [.2] the standard error of the Gaussian noise equals to 0.5. The experiment result shows that our method is more resistant to adversarial attack than other methods.
CONCLUDING REMARKS
In this paper, derived general form of PDE that correspond DNN model. Motivated by PDE theory, we proposed a training method for ResNet. We have theoretically shown that using our training methods, we obtain a model with better robustness and lower overfitting risk. Based these theoretical results, we developed an algorithm to train DNN models, and we verified them through experiments. There are numerous avenues for future work: Scale our training method networks that are large and expressive enough to solve problems like ImageNet. 2) Study other more complex equations DNN models and solve more challenging real tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Luis Alvarez, Frédéric Guichard, Pierre-Louis Lions, and Jean-Michel Morel. Axioms and fundamen- tal equations of image processing, Archive for rational mechanics and analysis, 123(3):199-255) 1993.
Benny Avelin and Kaj Nyström. Neural odes as the deep limit of resnets with constant weights. Analysis and Applications, 19(03):397-437. 2021.
Liefeng Bo, Kevin Lai, Xiaofeng Ren, and Dieter Fox. Object recognition with hierarchical kernel descriptors. In CVPR 2011, pp. 1729-1736, 2011. doi: 10. 10.1109/CVPR.211 1,5995719.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
George E. Dahl, Dong Yu, Li Deng, and Alex Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):30 42.30-42,22 2012. doi: 10.1009/TASL.20111114409
Lawrence C Evans. Partial differential equations. Graduate studies in mathematics, 19(4):7, 1998.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv: 1 170.007485, 2017.
Eldad Haber, Lars Ruthotto, Elliot Holtham, and Seong-Hwar Jun. Learning across scales multiscale methods for convolution neural networks. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Che-Wei Huang and Shrikanth S Narayanan. Stochastic shake-shake regularization for affective learning from speech. In INTERSPEECH, pp. 3658-3662, 2018.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint urXiv:1605.0760 2016
Michel Ledoux and Michel Talagrand. Prohability in Banach Spaces: isoperimetry and processes. Springer Science & Business Media, 2013.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. arXiv preprint / 18090031 00000.0/13.. 13, 2018.
Qianxiao Li, Long Chen, Cheng Tai, et al. Maximum principle based algorithms for deep learning. arXiv preprini 1rXiv:1710.095// 2017.
Zhen Li and Zuoqiang Shi. Deep residual learning and pdes manifold. arXiv preprint arXiv: 15, 2017.
Xuanqing Liu, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsich. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint 1rXiv:1906.02355. 2019.
10

---

Under review as a conference paper at ICLR 2022
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-32768 PMLR, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 10::::700000 1706.0608, 2017.
Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.
Andreas Maurer. A vector-contraction inequality for rademacher complexities. In International Conference on Algorithmic Learning Theory, pp. 3-17. Springer, 2016.
Shahar Mendelson. A few notes statistical learning theory. In Advanced lectures machine learning, pp. 1-40. Springer, 2003.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Viktor Reshniak and Clayton G Webster. Robust learning with implicit residual networks. Machine Learning and Knowledge Extraction, 3(1):34-55, 2021.
Hadi Salman, Greg Yang. Jerry Li, Pengchuan Zhang, Huan Zhang. llya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. arXiv preprint 19xiv:1906.04550 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lly Sutskever, and Ruslan Salakhutdinov: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Qi Sun, Yunzhe Tao, and Qiang Du. Stochastic training of residual networks: a differential equation viewpoint. arXiv preprint arXiv: arXiv:/8/2.00177 /8/2.00/74, 2018.
Yunzhe Tao, Qi Sun, Qiang Du, and Wei Liu. Nonlocal neural networks, nonlocal diffusion and nonlocal modeling, arXiv preprint arXiv:1806.0068) 2018
Matthew Thorpe and Yves van Gennip. Deep limits of residual neural networks. arXiv preprint arXiv:1810.1174 2018
Bao Wang, Binjie Yuan, Zuoqiang Shi, and Stanley J Osher. Enresnet: Resnets ensemble via the feynman-kae formalism for adversarial defense and beyond, SIAM Journal on Mathematics of Data Science, 2(3):559 582, 2020.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 20-36, Cham, 2016. Springer International Publishing. ISBN 978-3-319-4648448
Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500. 2017.
Hanshu Yan, Jiawei Du, Vincent YF Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. arXiv preprint arXiv: 191.0055 2019.

---

Under review as a conference paper at ICLR 2022
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 165.0:1005.0714. 2016.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural ordinary differential equations. arXiv preprint arXiv:1907.12996 2(4):3-1, 2019a.
Jingfeng Zhang, Bo Han, Laura Wynter, Kian Hsiang Low, and Mohan Kankanhalli. Towards robust resnet: A small step but giant leap. arXiv preprint arXiv:/902.
2019b.
Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin. Polynet: A pursuit of structural diversity in very deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 718-726, 2017.
Mai Zhu, Bo Chang, and Chong Fu. Convolutional neural networks combined with runge-kutta methods. arXiv preprint arXiv:1802.08831, 2018.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A
PROOF OF THEOREM 1
Proof. (Alvarez et al. 1993 The proof of Theorem consists of several steps. First we set
(f) T(() - T2(I) 8c(f) 6t,o(f). t-s
Using [Temporal Regularityl, for all h, t E [0, I]] and all f.g E Q, we can get
||ō,(f hg) (z) S Ch.
(18)
Step We begin the proof of Theorem by proving the following conclusion
(A1) the functions 5t( are Lipschitz continuous uniformly for E (0, 1](and uniformly for f
belonging to set Q).
Let be in IR and || = 1. we wish to bound to (Spatial Regularity], for f E Q, we have
for [0, 1]. According
The (6r (f)] lcf
Thx [o(] - -&Lff( + hz]]]la> + [1 + hz]] - 8 (z)((>)l
Ch.
Then the conclusion (A1) proved.
Step 2. From the preceding steps, can already deduce compactness property for 8c()) as t - 0. Now we need more, since we want the whole sequence to converge to the same limit. So we need a Cauchy estimate. Our next step is to prove the following conclusion.
(A2) For all 1, [0, }) we have [ 5. (.) S m()), where m(t) some continuous,
nonnegative, nondecreasing function on [0, ] such that m(0) 0, and mn() depends only on bounds
of derivatives of f. First, we rewrite 6.) 8,(f) K2 (6,(f) Kg - {,(,),, where K 0, Sgr Kdy 1, K Cg (IR)) and Kg = - (-//=) and * denote the convolution. Using the conclusion of (A1), we can obtain for all > 0, there exist positive constant C₁ such that
[[6.(f) K. lo) - Cie.
(19)
Then because of [Markov Property], [Temporal Regularity] and [Comparison Principle], we have
(F) Tr(s + sō, Ke)
1,00
8
|| Ti Tz(f)= Ti o Tf())
T T.(f) T,(f s5, Ke)
11 +
1
Ct + Tx(f) f s5. K-
|
Ct CGE
(20)
On the other hand, since (f) Kg E C5, [Temporal Regularity] implics that
[Til + (6) * K₂) - Ti(f) s6,(f) Kell < Cast
(21)
for some positive constant Cs depending only Combining equation 19 equation 20 and equation 21 we finally deduce that
[l6.+r,(() - 1,( < 20 + Cet.
The (A2) holds by setting mtt) inf e inf
1) C.t).
Step 3. Now give Cauchy estimate in 0 < h < t for 8,(
Cor
(f) 8, 2
+ m(t)
where = -
(22)
t
13

---

Under review as a conference paper at ICLR 2022
Observe that
6c( = Nh oNN(f)+ + T (ohtr.Na(f)
t
Using (A2) with
= Nh and let = [t,], have
Nh
[[ŌN+t((f)
Nh +r (N) Nh+r 6.(0) Nh+r N m(Nh)
(23)
Notice that
(f) (N- 1ih (N +
h Nh Nh ONN.NN--D
Nh
Using (A2) with h. t (N 1)h, we have
-
1 N (1 S F 1)h)
[]ōNa()) N 6iN O(N-
(24)
N
Combining this inequality with equation
24
we obtain
(1&((f) (N OLN 1)
11 - - (f)(lll
$ _ mNh) 7 - - 1)hb
(25)
Reiterating the argument which lends to equation 24 we obtain after (N 1) more steps that
h N-1
()z((f)
Nh (n( [f)(f s
x m((N))
Σ m(jh)
(26)
t
Since mtt) is nondecreasing and equation 25 we deduce
s 10+(() - Nh ()) - - (01lm= + -10
[[6,([) - 6n r
- 5,(0
= r {m(NN) + (N 1)h m(t) + [(16/(0)1e= + (18,(1))1z=))
s m(t) + Cor
(27)
we can pick hy going to and on., (f) converges uniformly on compact sets to bounded Lipschitz function on Rdd which we denote by A[J]
Because of equation 27
lim - - 6n., (f)la> S lim m(t) + Cor n-0
n-cc
[|6,(f) A[fIl I m()..
So (f) converges uniformly AA[]] when goes 0. Similarly, there exist an operator A₁ such that (f,(() converges uniformly to A. when goes to t. Step The goal of this step is to show the expression of A[f].
Let f.g E Cg and satisfy f(0) 0 = 0 (if not equal to O, we replace f(x), g(x) by f(x) A E Raxd, Then we set === f + =||>]] Using Taylor formula, there exist positive constant e such that for ||æ||a S cs we have f 2 g. Let we = w(x/e) where w C
f(0).g(z) g(0)), D0 = D0 = Rd f()) = D²g(()
(I) and 0 s w < on Rdd
x < c/2
(28)
w
x l Z
Then we introduce fā = ff + (1 we )g and it obviously that fă 2 Because of [Comparison Principle), T.(Se) T.( Due to
f0( (0) f(0) = (0) =
(29)
we can also get
(( 2 A() 2 A())(0)
(30)
14

---

Under review as a conference paper at ICLR 2022
Because there exists neighborhood of that fü = ff, we have D fo D for V |a| 0. In view [Locality] we have (0) = (0). And considering the continuity of A. we can deduce A(fō) converges to A(f) in _ when goes to O. This means (()) 2
(O) By
similar method, we can get (0)) Σ (0) which means A())(0) A(g)(0). Hence, for any ato E Rd if replace by æo, the similar result be obtained. So the value of (a)) only depends on æ, f, Df. D2 Observe that for any constant C, A(f+ = A(f), we can get (z)
only depends on æ; Df, f. At last, we finally get
A(f) = F
(31)
Here F is continuous function. In the same way, get
Ax(f) = F(DS, aa,,,
(32)
According to [Ensemble Additivity]. since
F(Df, D2 f2,, = lim (fm
t-0
F therefore satisfies
F(rDS + sDg, rD +00
sF sF(Dg, D2 x. g,,
for any real numbers r and s and any functions f and and at any point (x, t). Since the values of Df. Dg, f. D2g are arbitrary and can be independently taken to be O, we obtain for any vectors vn, t2 and symmetric matrices A1, A2 and any fixed point (æo, (o) that
F(re + sU2, A + sA2,, To, to) ( A1, vo, to) sF A2, to.to) to)
F(U1, A1, To. to) = FU,, Zo. to) + F00, A1. zo to).
Thus we can finally there exist Lipschitz continuous functions v Rd x [0, T] + Rd and Lipschitz continuous positive definite functions Rd [0, T) + Rdddd such that u(æ, () TT()) is the
solution of the equation
@u(w.)) v(x,t) Vu(x, Øt
Eng oig (x, t) du (x, E Rd, E [0,
u(x;, 0) = f(x),
where oig (x. 1) is the j-thh element of matrix function (ar, t).
The existence and uniqueness of equation 5 are guaranteed by the following theorem
Theorem 4. (Kolmogorov's 's backward equation))(Theormm 8.1.1 in Oksendal 2013)) Assume Ioo
process æt) satisfies SDE
dx(t) = v(x((t), ( x=0)= = x,
where Btt) is d-m Brownian motion and v Rd - IR a. Rd + Rdxd are all Lipschitz continuous
function. Let (x) E (Rd).. Then u(x,,) t) (x) (=0)) x:) the unique solution of
equation 5
B PROOF OF THEOREM 2
To begin the proof of Theorem we first provide the maximum principle for operator L which has the following form
du - o2 Au + u(x, Vu,,
Lu where the coefficients v(x 1) are continuous.
at
Theorem 5. Evans 7998) (Maximum principle) Assume to be an open bounded subset of T and as before set UT U x (0, T some fixed time T > O. Let (2)())) n C(Ur) and Up is the closure of UT. Denote UT UT by PT. If
Lu S 0 in Ur.
then
max u max u.
Ur
Pr
15

---

Under review as a conference paper at ICLR 2022
After the introduction of maximum principle, next illustrate the proof of the theorem
Proof. Let w( t) = (=, t) + Vu t)] 2Mr where μ and À are constants which will be defined later.
Note that (2, t) satisfies
((22)
01 - Vull
a1
and Vull 2 satisfies
0| Vul
o(2 Vo. WV (Ae)
Øt
therefore,
Ow
0t + v(xx, V) o² Aw
221 =e = 1- + Vull²) 2/0 VVuVr(z,t 20²
Next, let 7(x 1) minpell=1 = Vo(a,, 1)5 and = minz,1 (xx (), and let Lw = 22 t(w,t)VW - o² Aw then we have
La < -2 2 (upue + (A + nO² - -) Vul
we choose and p large enough, such that X + μO - 7 2 0, then
Lw 0.
From the maximum principle (Theorem 5) know max w(x S max w)x Let 1 and A 7 - o², we can easily get
| T) S = T (1
(33)
According to the Taylor formula and Hölder inequality, we can get that for any ||01|p = 6, we have
(x) law (a: + + < |V fm ( < d' fl
(34)
and
|u(x, 7') u( + 8,T) < VV, T) < d/
(35)
According to equation
34 and equation 35
we can get
u a (x - 8, > Pa - d [ loud
max u (x 5, S PB + + (
ites
Thus (x + 5,T) 2 max ux + 5,T) for all
P4 Pn
_ 2d [0 T)
Then combining equation
33 equation and equation 35 we can easily prove the theorem.
C PROOF OF THEOREM 3
The goal of the learning problem find h H. such that population risk E(h)
((m.)) [((h(x), (0))))))) y)] is minimized. We consider the supervised learning setting where one has ac- cess to n i.i.d. training examples Sx. learning algorithm maps the N training examples to hypothesis h E H. In this section, we are interested in the gap between the empirical risk (h) N EN ((h(x;)) y1) and the population risk E(h), known the generalization error. We start with the following theorem which connects the population and empirical risks via Rademacher complexity.
16

---

Under review as a conference paper at ICLR 2022
Lemma 1.
(Theorem 3.5 in (Mohri et al. 2018 Let SN {x Vali
1- be samples chosen i.i.d.
according to the distribution p) the loss function is bounded by B Then for any E (0, 1), with probability at least 1 6, the following holds for all h H.
E (h) S ESN (h) + 2Rs (Eu) + 3BV
(2N).
In addition, according the Ledoux- (alaux--allarandd contraction inequality (hequalityMaaurer 2016 and assume loss function is L- L-Lipschitz, we have
Rsv (Cx) S LRss (H).
Next, we begin to prove Theorem 3
Theorem 6. (Theorem 2.3 in (Mendelson 2003 Let H. be a class of functions from D to [-1, and set p to be probability measure D. Let (xr))²= be independent random variables distributed according to μ. Let (Y,d) be a metric space and set C C. For every e >> 0 and any n 2 8/6²,
2 sup SEH
((x)) - / (x))(())dx >e S
exp( 1128) (36)
i=1
In above theorem, Num(e, H, (()( denotes the covering numbers of H at scale e with respect to the (p)) norm. pnn is the empirical measure supported e > O, denote by Numme, H. d) the minimal number of open balls (with respect to the metric d) needed to cover H. That is, the minimal cardinality of the set fyr.-.. ymm C Y with the property that every f E H has some y; such that d(f, y:) < c. The set {ih, -mm } is called an c-cover of H. The logarithm of the covering numbers is called the entropy of the set. For every sample {æ), ... ) let umm be the empirical measure supported on that sample. For 1 S < oo and function f. [1ll)e)(mw)) = ( Er [fx)))p)
sample of (xx) For every
1/p and |o = maxiçisn [f(x))]-
Notice that
(1(p)) S Loo (Mm) S Look
We get one immediate corollary of Theorem 6
Corollary 1. Let H be class of functions from D to 1-1, 1] and set p(x) to be probability measure
on D. Let {x; ) be independent random variables distributed according to p(x)) For every e > and any 8/e²
P
sup
1 N
Σ
[a::) - (())p(())d) -
>x) S 8Num(e/8 H. T1o0) exppexpeee) N
(37)
JEH
i=1
where Num(c, H. Iox) be the covering numbers of H at scale with respect to the L norm
Then, we get an upper bound of sup E f(x.) fp fz))p(x)dx)
Corollary 2. Let H be class of functions from D to [-1. 1]. Let (x, \ be independent random variables distributed according to p. where p is the probability distribution whose C norm bounded. Then with probability at least 1- 8.
sup lp(f) (N)) SEH
128
In Numl
2 N H. Loo) + In 8 8
N
where
N
p(f) = / f())p(a)d pN(f) = Σ f(x;).
(38)
N
1=1
Proof.
Using Corollary
with probability at least 1 - 8,
sup (p(f) - PN (f) S Ed.
SEN
17

---

Under review as a conference paper at ICLR 2022
where Eş is determined by
128 N
8)
In Nunun(cs/ H. Loo) In
c5
Obviously,
128
2
6B 2
= 8V
N
N
which gives that
/
2
Num H. ) S Num(y
H. looc)
N
Then, we have
128 N
2 N
8
6g S
In Num(
H. Lxo) + In
which proves the corollary.
the entropy bound of is known, the upper bound of supjew |p(s) pN( follows from Corollary Now, the key point left becomes bounding the entropy of some given funetion class H.
Let us start from the function class F. To apply above corollary, we need to normalize to make lie in [-1, 1]. Here we also use FF to denote the normalized function class and absorb the bound of F into the generic constant C.
According to Taylor formula, we have for any I, D
If()) - f(z)] S Vd sup feF
|o | 21/2.
Where the dimension æ. This gives an easy bound of Numie, F, Loo),
SSPPEEF II
Fu S
(39)
Using Corollary
2 with probability at least - 1/(2N), we have
1/2
C
sup feF
(p() PNN s In N In(sup [[lloa) +1
(40)
VN
FeF
and
1/2
C
sup Ip()) (N))
In + In( sup [1lccs) +1
fEB,
VN
feg.
According to equation
33
we can get the following theorem.
Theorem 7. With probability at least - 1/(2N),
C (In (In +ne+ ,)
sup (p(s) pN( VN
feF
C VN
sup - -pN() SEg,
(InN + -
0² 1)
p(f) / f(x)p(x)dx PN(f)= 1 Σ f(z)).
N
a4ESx
F and Go is a function class defined as
F F: f(xx))llfll sej.
and
9o = -={ g((z) = u(xx 1)]
Ou(x, t)
v(x,t) Vu(x, t) + 1/20² Au(a, t), u(x, 0) = =((x) where f fEF].
0t
18

---

Under review as a conference paper at ICLR 2022
For the purpose of proving Theorem B give inequality about Rademacher complexity as following.
Theorem 8. (Theorem 2.23 in (Mendelson 2003)) Let p(x) be probability distribution function
and set H to be a class of functions on D. Denote
Z sup SEH
- - Exwptt) (f)l:
N
i=1
where {xr) are independent random variables distributed according to p(æ)). Then we have
Em Bm-p(a) < Rsw (H) S 2Exmp())
sup Em-p(a)
N
JCH
Then combining Theorem Theorem we have
(So) S 28 BBxp((a) 2
sup Be-p(a)fl SE9.
C
(lnN
h hn- 0² +1) 1/2
sup Eg f feg.
VN
VN
C
(In + Inc 0²
- +1)
sup llf
VN
VN
SEg.
C
1(-02
+ Inc In N2 + exp(-oo) )c]
VN
Similarly, we can estimate the upper bound of Rsw (F).
D MORE EXPERIMENTAL DETAILS ON HALF-MOON DATASET
In this subsection, we plot the decision boundary of ResNet trained with different hyperparameters in Figure 3 As drawn in Figure I we can see as the hyperparameters À and o² increasing, the decision boundary becomes more smooth, which implies the overfitting risk reduce. These experimental results are also consistent with the conclusion Theorem
60 0.01, n² = 0.01
A 0.01, o² 0.05
( 0.01, o2 2 0.1
-36
(0) 0.05, x² 0.01
(6) A 0.05, o² 0.05
0 A 0.05, x² 0.1
A 0.1, o² 0:01
(b) 0.1, 0.05
( 0.1, 0.1
Figure 3: Boundary decision of ResNet trained by our model with different hyper parameters À and o²
E ARCHITECTURES OF THE USED DNNS
Figure 4 4soww the architectures of ResNets used this paper. We also plot the inputs for the ResNet used CIFAR-10 dataset in Figure and every element of the matrix ll equals to f.
19

---

Under review as a conference paper at ICLR 2022
Inputs=[Data.]]
3 x 3, conv, 16
Layer
Layer
Layer
BN+ELU
Avgpool
Full Connected, 64 x 10
Outputs
Figure 4: Architectures of the ResNet used in our experiments.
+
torch.ctt
3
Figure 5: The inputs for the ResNet used in CIFAR-10 datasct.
20


Reviews:
[{"review_id": "BrRMyeORxH3", "reviewer": "Reviewer_LaXY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Apart from a few typos, the paper is well-written and generally easy to follow and understand.\nThe proofs, while not containing any novel techniques, are rigorous and correct. Viewing a \nDNN model as the flow of a regular Markovian operator seems quite natural and generalizes \nthe neural ODE idea. I also appreciate that the authors derive an explicit PDE based on \nvery reasonable assumptions on the desired properties of a generic classifier. I also really\nlike the illustrative example for ResNet and the Gaussian model injection, essentially \nviewing neural ODE(s) as the characteristic curves of a transport equation which is a special \ncase of their more general convection-diffusion equation (for ResNet). I also liked the\nrobustness guarantee which serves as an inspiration for using this framework to build models\nthat are better adept at handling adversarial examples. Generally, I think stability \nis the right motivation for considering such frameworks.\n\nThe algorithms and numerics are the weaker parts of this work. It is unclear why the \nauthors choose to reduce their PDE (1) to a re-scaled heat equation after showing that \nResNet(s), models that are known to work well, come from a transport equation. Perhaps \nthis simply makes the PDE easier to solve with less parameters to be learned. Nevertheless,\nI think exploring the more general form should certainly be done given the first part \nof the paper places a lot of emphasis on this result. My guess as to why the methods \nends up working is that the diffusion form the heat equation is able to smooth out the loss function\nwhich results in finding more stable regions of parameter space (perhaps there is a \nconnection to https://arxiv.org/pdf/1704.04932.pdf which takes a similar point of \nview but with a viscous Hamilton-Jacobi equation on SGD instead)? The method of \nsolving this heat equation just with another neural network seems a bit uninspired \nand is simply a special case of PINNS (https://www.sciencedirect.com/science/article/pii/S0021999118307125).\nWhile the PDE is very high dimensional, it is also very simple so applying some \nUsing classical techniques to solve it will, I think, make for a very interesting \ncomparison. Furthermore the numerics are not nearly extensive enough (I do appreciate\nthe half moon example showing the smoothed decision boundary). The authors need \nto consider more network architectures, not just ResNet, as well as harder datasets,\nlike ImageNet, and more types of adversarial attacks to really make a case that this \nmethod is practically useful. Furthermore, there seems to be a significant decrease\nin accuracy for clean images. If this is to be practical, this issue must be resolved\nas sacrificing 10% in accuracy for some robustness is usually not acceptable.", "summary_of_the_paper": "The authors cast DNN classifiers as the push-forward of a base classifier under a\nflow map at some fixed final time. Under some reasonable assumptions on the flow,\nthey show that, given any base classifier, the flow map can be obtained as the solution\nto a convection-diffusion equation. They show that ResNets and Gaussian noise injection\ncan be viewed as special cases of their model and give a robustness guarantee for any \nclassifier defined as a solution to their PDE. Experiments on the 2-d half moon data set\nas well CIFAR 10 and 100 show better robustness  of their model to adversarial attacks \nwhen compared to standard ResNet(s).", "summary_of_the_review": "Very interesting connection between DNN(s) and PDE(s), generalizing neural ODE(s) but \nthe practical algorithms presented don't make a strong case for the method.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8xASj2So4Zq", "reviewer": "Reviewer_ZDUG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Post-rebuttal Comments (11/29/2021)\n\nI thank the authors for their responses and updating the manuscript. The updated version improved the readability. Since my questions are clarified, I want to keep my score (6).\n\n# Initial Comments\n【Strength】\n- [1] This paper provided an architecture-independent sufficient condition that a convection-diffusion PDE models a DNN.\n- [2] The experiments are consistent with the implications of theoretical analyses (assuming that theorems are true. See【Correctness】section)\n\n【Weakness】\n- [1] Theorem 2 and Theorem 3 apply to restricted PDE (7) (constant-coefficient $\\sigma$) instead of the general PDE (1) derived in Theorem 1.\n- [2] Although I tried my best, I could not confirm the correctness of some parts of the proof. I would suggest writing their details in more detail.\n\n【Correctness】\n- [1] P.7: In experiments in Section 4, how is the initial ResNet $f_w$ trained?\n- [2] P.15: I could not confirm the correctness of the proof of Theorem 2. Specifically, I want to know the following things (I guess some of them came from the idea of [Wang et al., 2020]. But I could not figure out):\n  - [2-1] the precise statement of the maximum principle (for parabolic operators) used in the proof\n  - [2-2] how we can derive (31) from the conditions $Lw\\leq 0$ and $\\max_x w(x, 1)\\leq \\max_x w(x, 0)$ \n  - [2-3] how we can derive the statement from (31)--(33).\n- [3] P.18: I think it is better to write how to derive the bounds for $\\mathcal{G}_\\sigma$ in Theorem 6 from (the proof of) Theorem 2. More specifically, the bounds come from (31) in the proof of Theorem 2.\n- [4] P.18: It is not trivial to me how to derive Theorem 3 from the conclusions of Theorem 6 and Theorem 7\n\n【Technical Novelty And Significance】\n- [1] Many studies pointed out that ResNet and its variants can be described by discretizing ODEs (e.g., Neural ODE [Chen et al., 2018]). Also, [Wang et al., 2019] linked these models to PDEs. However, this paper investigated the \"opposite\" direction. It derived the condition under which a model is can be described by a PDE (Theorem 1). I think this is novel, as to the best of my knowledge, I do not know a general theory about it.\n- [2] Also, I think this paper is significant as it gives a new guideline for creating new deep learning models. Also, it provides a hint with how we go beyond the regime of the convection-diffusion models.\n- [3] I think the proof of Theorem 2 is based on the idea of [Wang et al. 2020]. On the other hand, it is a new idea to derive the bound for Rademacher complexity of $\\mathcal{G}_\\sigma$ from this idea.\n\n【Empirical Novelty And Significance】\n- [1] If I understand correctly, the proposed model is a discretization of the same PDE as the Gaussian noise injection model. However, the proposed model is novel as the training method differs from Gaussian noise injection. \n- [2] In addition, according to Table 2, the proposed method can increase the robustness against adversarial attacks. Therefore, I think this study is empirically significant.\n\n【Detailed Comments】\n- [1] P.1: Without loss of generality, we indicate this evolutive relationship by an adjustment operator, ... → I could not understand what \"Without loss of generality\" meant in this context. I would like to clarify it.\n- [2] P.3: ... Half-moon and CIFAR-10/CIFAR-100 dataset. → ... datasets.\n- [3] P.3: $S_N\\in \\mathcal{D}$　→ $\\mathcal{S}_N\\subset \\mathcal{D}$\n- [4] P.3: [Markov property] → [Markov Property]\n- [5] P.3: [Spatial Regularity]: $\\tau_hf$ → $\\tau_{\\mathbfit{h}}f$ (Make $h$ boldface.)\n- [6] P.3: [Temporal Regularity]: property → properties\n- [7] P.3: [Temporal Regularity]: $C_n$ is undefined.\n- [8] P.4: We then require [...] that no enhancement is made. → I could not understand what \"no enhancement is made\". Could you clarify the meaning?\n- [9] P.4: If I understand correctly, this paper claimed without proofs that when $u(x, 0) = \\mathrm{softmax} (w^{\\mathrm{FC}}x(0))$ and $x(t)$ evolves by $dx(t) = v(x(t), t)dt$, then $u(x, t) = \\mathrm{softmax}(w^{\\mathrm{FC}}x(t))$ satisfies the PDE (5) (Let me know if I misunderstood what this paper intended). I suggest writing how to derive it (I guess we can think of it as a corollary of Feynman-Kac formula. But since there are no stochastic terms, we may have simpler proof.)\n- [10] P.5: $\\mathcal{Y}$ is undefined.\n- [11] P.5: ... depend on $\\nabla v$. → ... depending on $\\nabla v$.\n- [12] P.6: Suppose $S_N$ ... is drawn from $X\\times Y \\subset \\mathcal{D} \\times \\{-1, 1\\}$ ... Assume $\\mathcal{D}$ is the underlying distribution of $X\\times Y$. → It seems the usage of $\\mathcal{D}$ is ambiguous. $\\mathcal{D}$ is a subset of some Euclidean space in the first sentence while $\\mathcal{D}$ is the distribution on the subset in the second sentence.\n- [13] P.6: It looks weird that the initial ResNet $f_w$ and the trained ResNet $g_w$ has the same parameter $w$ because they are independent networks. Does the parameter $w$ represent the concatenation of learnable parameters of $f_w$ and $g_w$?\n- [14] P.6: $l^{CE} = -\\log g_k(x)$: Does this equation assume that $f_w(x) = k$, that is, the output of $f_w$ is not a probability distribution over $\\{1, \\ldots, K\\}$ but a one-hot vector.\n- [15] P.7:  PGD with step size α and ... the total number of iterations. → I think this sentence is grammartically difficult to read. I would suggest reconsidering the wording.\n- [16] P.7: $x'=x^{(K)}$ → $\\mathbfit{x}'=\\mathbfit{x}^{(K)}$\n- [17] P.8: Is there any reason why the experiments did not compute generalization gap for CIFAR-10 and CIFAR-100 datasets?\n- [18] P.9: $\\lambda.\\sigma^2$ → $\\lambda, \\sigma^2$\n- [19] P.9: nature accuracy → natural accuracy\n- [20] P.9: showned → shown\n- [21] P.13: (Alvarez et al., 1993) → What is this reference for?\n- [22] P.13: $\\tau_{hz}$ → $\\tau_{h\\mathbfit{z}}$ (Make $z$ boldface)\n- [23] P.13: $Ch+\\|\\delta_t[f(\\mathbfit{x}+h\\mathbf{z}) - f(\\mathbfit{x})]\\|_{L^\\infty}$ → Remove the $Ch$ term.\n- [24] P.15, (30): I think $A_t(f)$ is undefined.\n- [25] P.16, P.18 : I think it is better to write the statements of the maximum principle and Khinchin's inequality as readers may be less familiar with these theorems.", "summary_of_the_paper": "This paper studied a ResNet-like DNN model that can be expressed as a discretization of PDEs. First, this paper showed that, under some assumptions, any adjust operator is a solution of a second-order convection-diffusion PDE (Theorem 1). ResNets and ResNets with Gaussian noise injection are special cases of this theorem. Next, for the specific PDE (Eq. (7)), this paper derived generalization guarantees regarding the Rademacher complexity. This paper also derived robustness guarantees in terms of input perturbations. Finally, this paper analyzed the predictive performance of ResNet trained by the proposed method with various model hyperparameters on clean and adversarial datasets.", "summary_of_the_review": "This paper is technically significant as it gave a model-agnostic PDE theory that encompasses several DNN models. I also think this paper is empirically significant as they support the theory well. On the other hand, I could not confirm that some parts of the proofs are correct (see【Correctness】 section). Since they are likely to be minor issues, I scored the correctness rating at 3. However, if these questions cannot be resolved, I may decrease the rating of correctness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6JS61l8PT_3", "reviewer": "Reviewer_zQvx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I like the idea behind this work, I must say that I was disappointed by the way it was presented here.\n\nThere are two main reasons:\n- The writing is lacking in clarity and thoroughness. For example, the operator T_t, which is the central object in the paper, is never rigorously defined. There is a formula for its discrete counterpart but the continuous one, while mentioned in the introduction, only formally appears in Theorem 1 which meaning thus becomes unclear, and one has to guess that the operator which is being characterized is defined through the properties above. This is only one example, albeit the most striking one. Another one is the explanation of the so-called fundamental properties which I found to be quite unconvincing: after several readings, while I understand why those properties are necessary from a technical point of view for the proof of Theorem 1, I don't see a clear motivation for them defining T_t.\n- The contributions of the paper are not clear and neither are its assumptions. For example, in Theorem 2, the base model is supposed to be compactly supported: why should this be the case? It is not true for a linear classifier for example. Moreover, while I found the experiments interesting, the toy dataset experiments were not convincing: is the claim that the proposed method can be used to ensure adversarial robustness? In this case, there should be other baseline methods to compare to.\n\nUpdate after rebuttal:\n\nI thank the authors for the improvements made in the updated version of the paper. While I still have some doubts regarding the significance of the contribution, I wouldn't be against its acceptance and I raise my score.", "summary_of_the_paper": "This paper considers DNNs as transformations of the input data which can be seen as discrete approximations to the solutions of PDEs. It reformulates a point of view which has been trending for a few years now. More precisely, this paper represents the network as an operator, which is actually solution to a PDE, which acts on a \"base classifier\". Using facts from PDE theory, they then propose improvements for standard neural architectures.", "summary_of_the_review": "While the work is promising, it is not rigorous nor clearly written enough to justify acceptance as there are shortcomings both in terms of the expositions and regarding the substance of the work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-MXejCNkxA", "reviewer": "Reviewer_mSDj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper connects resnet architecture with transport equation and a noise injected resnet with a diffusion equation. The framework introduced is certainly not novel, as shown in this paper \"ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies\" by Wang et. al, 2018. In fact, they too use it as a framework for adversarial robustness, and authors have not compared their methodology with this paper. \n\nSecondly, the algorithm used to retrieve the final network uses a resnet as an initialization, so I suspect that all this famework is doing is approximating a noise injected version of the original resnet, in a rather complicated way. \n\nIn general the authors should improve the exposition of the paper as it has a lot of grammatical mistakes and misused words. The paper needs to be revised carefully.", "summary_of_the_paper": "This paper connects the architecture of a resnet and a gaussian injected resnet to that of a transport equation and a diffusion equation respectively. They show that under this framework they provide robustness guarantees of their PDE framework scale with the $\\sigma$ parameter of the diffusion equation. Furthermore, they also show that larger the $\\sigma$ better the generalization gap for the resulting neural network. \n\nThey introduce a learning algorithm, to attain the network from their PDE framework starting from an initial NN (also a resnet).\n\nThey also empirically verify that their method improves robustness, and that it performs better than Gaussian Noise injection.", "summary_of_the_review": "I think that the framework provided by the authors is not novel enough, and the authors haven't empirically verified their methodology with relevant previous work. Further, the paper in the current state is not very well written.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper formulates ResNet like classifiers as the evolution of a base classifier through an operator corresponding to a PDE, up to a given final time. Using a set of assumptions on the desired properties of the flow operator, the authors show that it can be obtained as the solution of a convection-diffusion equation. This generalizes ideas developed e.g. for Neural ODEs. The authors provide several examples showing that their formulation encompasses regularization methods proposed for deep NNs.  They further provide robustness guaranties for a classifier defined according to their framework. They introduce an algorithm based on a restricted version of their framework and propose different experiments showing the increased robustness of their model to a family of adversarial attacks compared to baseline ResNets.

The paper introduces an original idea, providing a very interesting connection between ResNets and PDEs. This allows the authors to exploit known properties of PDEs and opens the way to new theoretical insights on DNNs while allowing the development of DNN models with proved properties. As mentioned this generalizes the view of ResNets introduced in Neural ODEs. 
Besides, the paper presents weaknesses. First the form will make it accessible only to a very small audience in the ML community. No effort is made in the writing to introduce the required PDE concepts that would help a lot understanding and appreciating the contribution. This is a pity since given the current trend on this topic this could be of interest to a large community. Then the use cases in the experiments focus solely on robustness properties and one type of attacks. This illustrates only one aspect of the potential of the framework, and this does not provide a strong case in support of the ideas introduced before. The global message carried out by the paper then becomes unclear. Overall, the current version could be largely improved and this will certainly lead to a strong contribution.
Decision: Reject