PDE-regularized Neural Networks for Image Classification
Authors: Jungeun Kim
Seunghyun Hwang
Jihyun Hwang
Kookjin Lee
Dongeun Lee
Noseong Park
Venue:  (2021)
Keywords: Neural ODE
Partial Differential Equations
Image Classification

Abstract: Neural ordinary differential equations (neural ODEs) introduced an approach to approximate a neural network as a system of ODEs after considering its layer as a continuous variable and discretizing its hidden dimension. While having several good characteristics, neural ODEs are known to be numerically unstable and slow in solving their integral problems, resulting in errors and/or much computation of the forward-pass inference. In this work, we present a novel partial differential equation (PDE)-based approach that removes the necessity of solving integral problems and considers both the layer and the hidden dimension as continuous variables. Owing to the recent advancement of learning PDEs, the presented novel concept, called PR-Net, can be implemented. Our method shows comparable (or better) accuracy and robustness in much shorter forward-pass inference time for various datasets and tasks in comparison with neural ODEs and Isometric MobileNet V3. For the efficient nature of PR-Net, it is suitable to be deployed in resource-scarce environments, e.g., deploying instead of MobileNet.

Under review as a conference paper at ICLR 2021
PDE-REGULARIZER NEURAL NETWORKS FOR IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural ordinary differential equations (neural ODEs) introduced an approach to approximate neural network as system of ODEs after considering its layer a continuous variable and discretizing its hidden dimension. While having several good characteristics, neural ODEs are known to be numerically unstable and slow in solving their integral problems, resulting in errors and/or much computation of the forward pass inference. In this work, we present novel partial differen- tial equation (PDE)-based approach that removes the necessity of solving integral problems and considers both the layer and the hidden dimension as continuous variables. Owing to the recent advancement of learning PDEs, the presented novel concept, called PR-Net, can be implemented. Our method shows comparable (or better) accuracy and robustness in much shorter forward-pass inference time for various datasets and tasks in comparison with neural ODEs and Isometric Mo- bileNet V3. For the efficient nature of PR-Net, it is suitable to be deployed in resource- scarce environments, c.g., deploying instead of MobileNet.
INTRODUCTION
It had been discovered that interpreting neural networks differential equations is possible by sev- cral independent research groups [Weinan| [017 Ruthotto & Haber 2019 Lu et al. 2018 Ciccone ett al. 2018; Chen et al. 2018, Gholami al.) 2019). Among them, the seminal neural ordinary differential equation (neural ODE) research work, which considers the general architecture in Fig- ure (a), is to learn a neural network approximating ah(!), where h2(t) is hidden vector at layer (or time) (Chen et al., 2018). As such, neural network described by a system of ODEs, each ODE of which describes dynamics of hidden element. While neural ODEs have many good characteristics, they also have limitations, which are listed as follows:
Pros. Neural ODEs can interpret t as continuous variable and we can have hidden vectors at any layer (or time) by h(1) h0)) + fu o(h((), t; 0.) dt, where o(h(t), t; 0.) är
ala(t) is
a neural network parameterized by 9..
Pros. Neural ODEs sometimes have smaller numbers of parameters than those other conven- tional neural network designs, e.g., (Pinckaers & Litjens 2019).
Cons. Neural ODEs, which use an adaptive step-size ODE solver, sometimes show numerical instability (i.e., the underflow error of the step-size) their forward- -psss inference can take a long time (i.e., too many steps) in solving integral problems, e.g. a forward-pass time of
37.6 seconds of ODE-Net vs. 9.8 seconds of PR-Net in Table 2 Several countermeasures have been proposed but it is unavoidable to solve integral problems (Zhuang et al., 2020 Finlay 2020 Daulbaev al. 2020).
To tackle the limitation, we propose the concept of partial differential equation-reegularized neural network (PR-Net) directly learn hidden element, denoted (d,, () at layer (or time) E [0, 7' and dimension d E T Under general contexts, PDE consists of i) an initial condition at t = 0, ii) boundary condition deseribing 3t to predict h(d,t) that meets its initial/boundary conditions and governing equation.
boundary location of the spatial domain " and ii) a governing equation @k(d,t) As such, learning a PDE from data can be reduced to a regression-like problem
In training our proposed PR-Net, h()) is provided by an earlier feature extraction layer, which is the same as neural ODEs. However, an appropriate governing equation is unknown for downstream

---

Under review as a conference paper at ICLR 2021
Goveriin Equation (L.)
81
Feature Extractor
A(D)
h(0)+ L o(h())..
h(1)
Classilier
prediction
Data
Contorms each
Regression Modal predict A(d.f)
(a) The general architecture of neural ODEs
(b) The proposed PR-Net concept
Figure 1: The proposed PR-Net avoids solving integral problems by learning regression model that conforms with learned governing equation.
machine learning tasks. Therefore, propose to train regression model for predicting h(d, t) and its governing equation simultancously (see Figure (b)). In other words, neural ODEs directly learn governing equation (i.e., @h(!)), whereas PR-Net learns governing equation in conjunction with a regression model that conforms with the learned governing equation. The key advantage in our approach is that we can eliminate the necessity of solving integral problems - in neural ODEs, where we learn a governing equation only, solving integral problems is mandatory.
Such forward and inverse problems (i.e., solving PDEs for (ad, t) and identifying governing equa- tions, respectively) arise many important computational science problems and there have been many efforts applying machine learning/deep learning techniques to those problems (e.g.. in earth science (Reichstein et al. 2019. Bergen et al. 2019) and climate science (Rolnick et al. 2019)). Re-
cently, physics-informed informed physics-awes
approaches (Battaglia al.,
2016
Chang et al. 2017
de Bezenac et al. 2018 Raissi et al. 2019 Sanchez-Gonzalez et al. 2018 Long et al. 2018
have demonstrated that designing neural networks incorporate prior scientific knowledge (e.g.,
by enforcing physical laws described in governing equations (Raissi et al.) 2019)) greatly helps avoiding over-fitting and improving generalizability of the neural networks. There also exist several approaches to incorporate various ideas of classical mechanics in designing ODEral-ODE-type net-
works (Greydanus et al.) 2019 Chen et al.) 2020 Cranmer al. 2020 Zhong et al. 2020 Lee & Parish 2020). However, all these works are interested in solving either forward or inverse problems whereas we solve the two different problem types at the same time for downstream tasks. The most
similar existing work our work is in (Long al. 2018). However, this work studied scientific PDEs and do not consider as a continuous variable but use a set of discretized points of t.
Compared to previous approaches, the proposed method has distinct feature that forward and inverse problems are solved simultancously with a continuous variable Due to this unique feature, the method can be applied to general machine learning downstream tasks, where do not have priori knowledge on governing equations, such as image classification. Our proposed PR-Net had the following characteristics:
Pros. PR-Net trains a regression model that outputs a scalar element (ed, 1) (without solving any consider both d and as continuous variables. Therefore, is possible to construct flexible hidden dimension vectors.
integral problems), and
Pros. PR-Net does not require solving integral problems. As such, there no numerical instabil- ity and their forward-pass time is much shorter than that of neural ODEs.
Pros. By learning governing equation, we can regularize the overall behavior of PR-Net.
Cons. PR-Net sometimes requires a larger number of parameters than that of neural ODEs or conventional neural networks.
2 PARTIAL DIFFERENTIAL EQUATIONS
The key difference between ODEs and PDEs is that PDEs can have derivatives of multiple variables whereas ODEs should have only one such variable's derivative. Therefore, our PDE-based method interprets both the layer of neural network and the dimension of hidden vector as continuous vari- ables, which cannot be done in neural ODEs. In our context, h(d,t) means hidden scalar element layer E IR and dimension d E m. e.g., m = if htt) a vector, m = 3 he()) is convolutional feature map, and
2

---

Under review as a conference paper at ICLR 2021
Md.0)
Table 1: Two types of PDE problems related to our work
Neural Network
Solution k(d.))
Governing Equation
Type
Data
What infer
Figure 2: A neural network predicts solution values at d, given initial conditions, denoted h(d, 0) for various d, and governing equation.
Forward Problem
Governing equation
Initial condition
Solution A(d, t)
Solution (tdd,
Inverse Problem
Initial condition
Governing equation
In this section, we first introduce the forward and inverse problems of PDEs in general contexts (see Table T. Then, we extend them design our proposed method in deep-learning contexts.
2.1 FORWARD PROBLEM OF PDES IN GENERAL CONTEXTS
The forward PDE problem in general contexts is to find solution h(d,t) t), where d is in spatial domain R and is a time domain [0. 7). given an initial condition h(d, 0), ii) boundary condition h(dbe; t). where dng is boundary location of the spatial domain IR" and iii) gov- erning equation g (Raissi et al. 2019) We note that the boundary condition can be missing some cases (Kim 2018 The governing equation is typically in the following form with particular choices of Oi.. RRassi 2018 Peng et al. 2020):
g(d, h) def h4 - (a0.0 + 01,h + a2,0h2 + as,0h
+ to,ha + thg + a221 + 03 hha
(I)
+ 00.2had 01,2hhaa 222 had + 03,2h 2.2hhaa
+ a0,3hads + a1,3hhaaa + 02 baas + 0 hada).
where h 8h(d,1) ha 8h(d,t) had 8d((d 0d and hadd &hd(.))) ad
We also note that g is
8t always zero in PDEs, i.e., g(d, t: h) =
In many cases, it is hard to solve the forward problem and hence general purpose PDE solvers do not exist. Nevertheless, one can use the following optimization to train a neural network s(d, 1; 0) approximate the solution function h(d, l) as shown in Figure 2 RRaissi et al. 2019):
arg min Li + LB LG.
(2)
0
L def
Ni Σ (f( -
(3)
LB def 1
(dide, 1; -b(dbc t))
(4)
NB (dhe.t)
LG def
Σ fgd,b&f,0),
(5)
NG (d.1)
where NB, are the numbers of training samples, L1 is to train O for the initial condition, Lg is for the boundary condition, and LG is for the governing equation. Because the governing equation is always zero, simply minimize its squared term. Note that i) fú, fa fad. f faad can be easily constructed using the automatic differentiation implemented in TensorFlow or PyTorch, and ii) we only need hdd, 0), (dhe, l), which are known priori, train the parameters 0.
2.2 INVERSE PROBLEM OF PDES GENERAL CONTEXTS
The inverse problem is to find a governing equation given i) an initial condition (t, 0) and ii) solution function h(d,t) [Raissi] 2018). learns ai, in Eq. with the following loss (if possible, they use reference solutions well)::
1
arg min
Σ g(d,t;h)²
NG (d.t)
ac
3

---

Under review as a conference paper at ICLR 2021
Given solution function h and its partial derivative terms, train oi by minimizing the objective loss. Note that we know in this case. Therefore, the objective loss is defined with rather than with f. unlike Eq. 5
The optimal solution of jij is not unique sometimes. However, we note that no trivial solutions, e.g., Oi = 0 for all J. exist for the inverse problem.
3 PDE-REGULARIZHD NEURAL NETWORKS
Our goal in this work is to replace a system of ODEs (cf. Figure (a)) with a PDE, Assuming that a target task-specific PDE known priori, given initial condition h(0)) extracted by the feature extractor from a sample x, a forward problem can be solved via the method described in Section
2.1 However, target task-specific PDE not known a priori general, and thus, the governing equation should be learned from data via solving the inverse problem. Unfortunately, the solution function h(d. t)) not also known a priori in our setting. Therefore, we make an assumption on the governing equation that it consists of the most common partial derivative terms (ef. Eq. D and then we propose to solve the forward and the inverse problems alternately: to train 0, we fix its governing equation g (more precisely, and for all i, j). and to train 0z for all i, J. we fix O.
How to Solve Forward Problem. We customize the method presented in Section 2.1 by i) adding task-specific loss, e.g.. cross-entropy loss for image classification, ii) parameterizing the neural net- work f by the initial condition (0). and iii) dropping the boundary condition. Let f d, t; 0) be our neural network to approximate h(d, t) given the varying initial condition h[0] The defi- nition of the governing equation is also extended to g(d, t; f. (()),)) We use the following loss definition to train A:
arg min
LT + LI + Lc.
(6)
0
L,
def
Σ (f(h(0),,,0;0) -((d)))))
Nx
dh)
xEX
L. def 1
Σ BEX
Σ (d,))e
g(d, (, 0)))
(8)
Nx
where Lr is task-specific loss, X is a training set, and H is set of (d, t) pairs, where d E Rgo, f E R20- with which we construct the hidden vector that will be used for downstream tasks, denoted by musk (See Figure 3)
We query f (O)))) d, t; 0) with the (d,t) pairs in H to construct musk One more important point to note that in order to better construct h task we can train even the pairs in H as follows: arg (d Tor (line 7 in Alg. []. Thus, the elements of takk can be collected from different dimensions and layers. A similar approach to optimize the end time of integral was attempted for neural ODEs in (Massaroli et al. 2020).
How to Solve Inverse Problem. After fixing 0, we train an for all i j by using the following L1 regularized loss with a coefficient w:
arg min L. + RG, ot
RG
Jaigl-
(10)
We minimize the sum of auyi to induce sparse governing equation according to Occam's 's razor and since in many PDEs, their governing equations are sparse. This optimization allows us to choose a sparse solution among many possible governing equations. In many cases, therefore, our regularized inverse problem can be uniquely solved.
'Therefore, one can consider that our neural network f approximates a general solution rather than partic- ular solution. A general solution means a solution of PDE with no specified initial conditions and a particular solution means a solution of PDE given initial condition. Both neural ODEs and PR-Net approximate general solutions because initial conditions varied.
4

---

Under review as a conference paper at ICLR 2021
Algorithm How to train PR-Net
Feature Extractor
Input: training data X, validating data V, max itcration number mar. jiter Output: 0, (d, t) H, and oi, for all i,j
h(O)
1 Initialize 0, (d, t) E H, and ckj for all i,js
L,
2 h <- 0:
PDE
L R
3
Luan +-
while Luwn is not converged and & maa do Train 0, Feature Extractor, and Classifier with Lr Train 0 with Lr + LL La
5
4 ttsk
7 Train (d,t) E H with Lr
Classifier
Train on, for all i,j with La + Ro: Validate with V and update the best model;
Data flow
Lam Lr L + La + Ro:
Gradient flow
Ly
k 1:
12 return 0, (d,t) E H. and any for all i,
Figure 3: The general architecture and the training algorithm of PR-Net
Training Algorithm. Our overall training algorithm is in Alg. We alternately train 0, (d, H, and atj for all j.j. The forward problem to train 0 becomes well-posed problem (i.e., its solution always exists and is unique) the neural network f analytical or equivalently, uniformly Lipschitz continuous (Chen et al. 2018). Many neural network operators are analytical, such as softplus, fully-connected, and exponential. Under the mild condition analytical neural networks, therefore, the many cases due to the sparseness requirement. As a result, our proposed training algorithm can converge cooperative equilibrium. Note that 0, (d, E H, and 0s for all i, cooperate minimize LT + L,, + LG + Rg- Therefore, the proposed training method can be seen as cooperative game (Mas-Colell, 1989). After finishing the training process, Oa.j. for all i, are not needed any more (because already conforms with the learned governing equation at this point) and can be discarded during testing.
be fulfilled. The inverse problem also be uniquely solved in
For complicated downstream tasks, training for Lr should be done earlier than others (line Then, we carefully update the PDE parameters (line 6 and other training procedures follow. The proposed produces the best outcomes our experiments. However, this sequence can be
sequence in Alg. [ varied for other datasets downstream tasks.
Complexity Analyses. The adjoint sensitivity method of neural ODEs enables the space complex- ity of ((1) while calculating gradients. However, its forward-pass inference time O( where is the (average) step-size of an underlying ODE solver. Because can sometimes be very small, its inference via forward-pas can take long time.
Our PR-Net uses the standard backpropagation method to train and its gradient computation com- plexity is the same as that in conventional neural networks. addition, the forward pass inference time is (1), given a fixed network f, because we do not solve integral problems.
4 EXPERIMENTS
In this section, introduce experimental evaluations with various datasets and tasks. All experiments were conducted in the following software and hardware environments: UBUNTU 18.04 LTS, PYTHON 3.6.6, NUMPY 1.18.5, SCIPY 1.5, MATPLOTLIB 3.3.1, PYTORCH 1.2.0, CUDA 10.0, and NVIDIA Driver 417.22, i9 CPU, and NVIDIA RTX TITAN. In Section of Appendix, summarize detailed dataset information and additional experiments.
5

---

Under review as a conference paper at ICLR 2021
Table 2: Image classification in MNIST and SVHN. The inference time is the time in seconds to classify a batch of 1,000 images. In general, PR-Net shows the best efficiency.
MNIST
SVHN
Name
# Params
Test Accuracy
Inference Time
Test Accuracy
Inference Time
ResNet
0.60M
0,9966
7.6447
0.9660
8.6721
RK-Net
0.22M
0.9970
7.4774
0.9652
13,5139
ODE-Net
0.22M
0,9964
24.8355
0.9599
37,6776
PR-Net
0.21M
0.9972
6.5023
0.9615
9.8263
4.1 IMAGE CLASSIHCATION WITH MNIST AND SVHN
We reuse the convolutional neural network, called ODE-Net, in the work by Chen et al. (2018) to classify MNIST and SVHN and replace its ODE part with our proposed PDE, denoted PR-Net in Table 2 See Appendix for the architecture and the hyperparameters of the network f in PR-Net for this experiment. We reuse their codes and strictly follow their experimental environments.
Its detailed results are summarized in Table 2] We compare with ResNet, RK-Net and ODE-Net. In ResNet, we have downsampling layer followed by 6 standard residual blocks (He et al. 2016). For RK-Net and ODE-Net, we replace the residual blocks with an ODE but they differ at the choice of ODE solvers. RK-Net uses the fourth-order Runge Kutta method and ODE-Net uses the adap- tive Dormand-Prince method for their forward-past inference - both of them are trained with the adjoint sensitivity method which standard backward-pass gradient computation method, Our PR-Net, which does not require solving integral problems, shows the best performance in all as- pects for MNIST. In particular, PR-Net shows much better efficiency than ResNet, considering their numbers of parameters, i.e., 0.60M of ResNet and 0.21M of PR-Net. Comparing ODE-Net and PR-Net for the inference time, our method shows much faster performance. i.c., 24.8355 seconds of ODE-Net vs. 6.5023 seconds of PR-Net to classify batch of 1,000 images. Considering its short inference time, in SVHN we can say that its efficiency still better than that of ODE-Net. One interesting point is that using the fourth-order Runge-Kutta Kutta method in RK-Net produces better accuracy and inferente time than ODE-Net in our experiments, which is slightly different from the original neural ODE paper (Chen et al. 2018 We tested more hyperparameters for them.
4.2 IMAGE CLASSIFICATION WITH TINY IMAGENET
We use one more convolutional neural network to test with Tiny ImageNet. Tiny ImageNet is the modified subset of ImageNet with downscaled image resolution 64 x 64. It consists of 200 different classes with 100,000 training images and 10,000 validation images. Our baseline model is Isometric MobileNet V3 (Sandler et al.) 2019). For the efficient nature of ODE-Net and PR-Net, we consider that the resource-scaree environments, for which MobileNet plication areas. The isometric architecture of Isometric MobileNet V3 maintains constant resolution throughout all layers. Therefore, pooling layers are not needed and computation efficiency is high, according to their experiments. In addition, neural ODEs require an isometric architecture, i.e., the dimensionality of h(t), 2 0, cannot be varied. In our PR-Net, do not have such restrictions. For fair comparison, however, we have decided to use Isometric MobileNet V3. We replace some of its MobileNet V3 blocks with ODEs or PDEs, denoted ODE-Net and PR-Net in Table B respec- tively. We train our models from scratch without using any pretrained network, with synchronous training setup.
designed, one of their best ap-
Table summarizes their results. We report both of the top-1 and the top-5 accuracy, which is a common practice for (Tiny) ImageNet. In general, our PR-Net shows the best accuracy. PR-Net achieves an top-1 accuracy of 0.6157 with 4.56M parameters. The full Isometric MobileNet V3 marks an top-1 accuracy 0.6578 with 20M parameters and the reduced Isometric MobileNet V3 with 4.30M parameters shows top-1 accuracy of 0.6076. Considering the large difference on the number of parameters, Ne-Net's efficiency is high. In particular, it outperforms others in the top-5 accuracy by non-trivial margins, e.g., 0.7911 of ODE-Net vs. 0.8115 of Isometric MobileNet V3 vs. 0.8357 of PR-Net. In addition, PR-Net shows faster forward-par inference time in comparison with ODE-Net. The inference time is classify batch of 1,000 images.

---

Under review as a conference paper at ICLR 2021
Table 3: Image classification in Tiny ImageNet. PR-Net shows better efficiency than ODE-Net.
Name
M.Net V3
ODE-Net
PR-Net
M.Net V3
ODE-Net
PR-Net
Width Multiplier
Mobile Blocks
4
4
3
ODE Blocks
N/A
N/A
N/A
N/A
PDE Blocks
N/A
N/A
N/A
N/A
Accuracy (top-1)
0,5809
0.5547
0,5972
0.6076
0,5672
0.6157
Accuracy (top-5)
0.8049
0.7946
0.8166
0.8115
0.7911
0.8357
Params
1.21M
1.36M
1.36M
4.30M
4.90M
4.56M
Inference Time
4.14
5.26
5.23
5.21
8.3
6.25
Out-of iitoo--dittribttion Robustness (top-1
accuracy)
Gaussian Noise
0.4495
0.4165
0.4685
0.4757
0.4474
0.4878
Random Crop & Resize
0,4636
0,4305
0.4841
0,4814
0.4419
0.4965
Random Rotation
0.3961
0.3667
0.4267
0.4256
0.3901
0.4381
Color Jittering
0,4206
0,3812
0.4429
0.4555
0.4108
0.4693
Out of disttibuutionn Robustness (top-5
accuracy)
Gaussian Noise
0.68
0.6619
0.7064
0.7025
0.6757
0.7205
Random Crop & Resize
0.7106
0.6935
0.7357
0.7215
0.6936
0.7442
Random Rotation
0.6372
0.6216
0.6627
0.6546
0.6319
0.6778
Color Jittering
0.6742
0.6396
0.6878
0.6874
0.6506
0.713
4.3 EXPERIMENTS ON ROBUSTNESS WITH TINY IMAGENET
To check the efficacy of learning governing equation, we conduct three more additional experi- ments with Tiny ImageNet: i) out-of-distribution image classification, ii) adversarial attacks, and iii) transfer learning to other image datasets. In the first and second experiments, we apply many wwt.ooddiiiirbbtiioaaeeeeeeeee
techniques to generate how each model responses to them. Being inspired by the observations that robust models are better transferred to other datasets (Engstrom et al. 2019a Allee-ZZu & Li 2020 Salman et al.) 2020), in the third experiment, we check the transfer learning accuracy to other image datasets. According to our hypothesis, PR-Net which knows the governing equation for classifying Tiny ImageNet should show better robustness than others (as seen in Figure 4
augmentation/perturbaiii
images and check
for scientific PDE problem in Appendix).
Neural networks are typically vulnerable to duttri-distributtin and adversarial samples (Shen et al,
2016 Azulay & Weiss 2019 Engstrom et al. 2019b). As being more fitted training data, they typically show lower robustness to out-of-distriuution and adversarial samples. However, PR-Net's processing them should follow its learned governing equation. Therefore, one way to understand learning governing equation is a sort of regularization which prevents overfitting and implanting knowledge governing the classification process.
cuttt--Disttibutionn Image classification. We use four image augmentation methods: adding a Gaussian noise of (0, 0.1). ii) cropping a ceter area by size 56 x 56 and resizing to the original size, iii) rotating into a random direction for 30 degree, and iv) perturbing colors through randomly jittering the brightness, contrast, saturation, and hue with a strength coefficient of 0.2. All these are popular out-of-distribution augmentation methods (Shen et al., 2016 Azulay & Weiss 2019
Engstrom et al., (019b))
Our PR-Net shows the best accuracy (i.e., robustness) in all cases. In comparison with ODE-Net, shows much better robustness, e.g., 0.3812 of ODE-Net 0.4429 of PR-Net for the color jitter- ing augmentation. One interesting point is that methods are commonly more vulnerable to the random rotation and the color jittering augmentations than the other two augmentations.
Adversarial Attack Robustness. It is well-known that neural networks are vulnerable to adver- sarial attacks. Because the governing equation regularizes PR-Net's behaviors, can be robust to unknown adversarial samples. We use FGSM (Goodfellow et al. 2015) and PGD Madry et al., 2018) to find adversarial samples and the robustness to them is reported in Table With various settings for the key parameter that controls the degree of adversarial perturbations, adversarial samples. The configuration of doubling the number of channels used each layer, denoted as "Width Multiplier 2", showed better performance in Table and use only the con-
generate
7

---

Under review as a conference paper at ICLR 2021
Table 4: Adversarial attacks in Tiny ImageNet. PR-Net shows better robustness than ODE-Net.
M.Net V3
ODE-Net
PR-Net
M.Net V3
ODE-Net
PR-Net
Attack Method
Top-1 accuracy
Top-5 accuracy
FGSM(c 0.5/255)
0.3860
0,3656
0.4041
0.6492
0.6398
0.6911
FGSM( 1/255)
0.2304
0.2287
0.2499
0.4751
0,4928
0.5374
FGSM(c 3/255)
0,0452
0.0464
0.0369
0.1232
0.1562
0.1596
PGD (c 0.5/255)
0.3733
0.3525
0.3910
0.6508
0.6409
0.6936
PGD (c 1/255)
0.1902
0.1908
0.2133
0.4579
0,4810
0.5281
PGD (c 3/255)
0.0218
0.0235
0.017
0.0792
0,1093
0.1144
Table 5: Transfer learning in Tiny ImageNet. PR-Net shows better transferability than ODE-Net.
M.Net V3
ODE-Net
PR-Net
M.Net V3
ODE-Net
PR-Net
Datasct
Top-1 accuracy
Top-5 accuracy
CIFARTO0
0.7676
0.7460
0.7750
0.9320
0.9270
0,9480
CIFARTO
0.9403
0.9280
0,9418
0.9963
0.9928
0.9962
Aircraft
0.6233
0.6027
0.6612
0.8509
0,8300
0.8561
Food- 101
0.7317
0.7128
0.7366
0.9108
0.9036
0.9174
DTD
0.4819
0.4973
0,5113
0.7660
0.7465
0.7957
Cars
0.6313
0.5576
0.6283
0.8380
0.7998
0.8319
figuration for this adversarial attack and next transfer learning experiments. For all attacks except FGSM(c = =3/25 3/225)) and PGD (c = 3/255), PR-Net shows the best robustness as shown in Table The gap between PR-Net and other baselines are significant for the most cases of PGD.
Transfer Learning. As reported in [Engstrom et al. 201 9a
Allen-Zhu & Li 2020 Salman et al.,
2020), robust models tend produce feature maps suitable for transfer learning than regular models. In this regard, we checked the transferability of the pretrained PR-Net for Tiny ImageNet other
datasets: CIFFAR 100 [Krizheskyy] 2009). CIFARIO (Krizhevsky) 2009 FGVC Aircraft (Maji et al. 2013). Food-101 (Bossard al. 2014), DTD (Cimpoi et al. 2014), and Cars (Yang et al.
2015 As shown in Table SJ PR-Net shows the best transfer learning accuracy in all cases except
Cars. The improvements over M.Net V3 and ODE-Net are significant for Aircraft and DTD.
DISCUSSIONS CONCLUSIONS
recently became popular to design neural networks based on differential equations. most cases, ODEs are used to approximate neural networks. In this work, on the other hand, presented PDE-based approach to design neural networks. Our method simultaneously learns a regression model and governing equation that conform with each other. Therefore, the internal processing mechanism of the learned regression model should follow the learned governing equation. One can consider that this mechanism is sort of implanting domain knowledge into the regression model. The main challenge in our problem definition is that we need to discover a governing equation from data while training a regression model. Therefore, we adopt a joint training method of the regression model and the governing equation.
To show the efficacy, we conducted five experiments: i) MNIST/SVHN classification, ii) Tiny Ima- geNet classification, iii) classification with out- out-of-distributtin samples, iv) adversarial attack robust- ness, and v) transfer learning. Our method shows the best accuracy and robsutness (or close the best) only except SVHN. In particular, the challenging robustness experiments empirically prove why learning an appropriate governing equation is important.
One limitation on this method is that it is sometimes hard to achieve good trade-off between all different loss and regularization terms. Our method intrinsically involves various terms and we found that it is important to tune hyperparameters (especially for various coefficients and learning rates) in order to achieve reliable performance. In particular, as for all i, are important to learn reliable governing equations. Because the trained network f is greatly influenced by the governing equation, hyperparameters should be tuned to learn meaningful equations. We also plan to study the proposed concept for many other clasification//egression tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning, 2020.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly small image transformations? J. Mach. Learn. Res., (0:184:1-184:23 2019.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. NeurIPS, pp. 4502-4510, 2016.
Karianne J Bergen, Paul A Johnson, V Maarten, and Gregory C Beroza. Machine learning for data-driven discovery in solid earth geoscience. Science, 363(6433), 2019.
Lukas Bossard, M. Guillaumin, and L. Gool. Food-101 - mining discriminative components with random forests. In ECCV, 2014.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. In ICLR, 2017.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurlPS. 2018
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural
networks. In ICIR, 2020.
Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez, Nais- net: Stable deep networks from non-autonomous differential equations. In Advances in Neural Information Processing Systems, pp. 3025-3035, 2018.
M. Cimpoi, S. Maji, I. Kokkinos, Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR Deep Differential Equations Workshop, 2020.
Talgat Daulbaev, Alexandr Katrutsa, Larisa Markeeva, Julia Gusak, Andrzej Cichocki, and Ivan Oseledets. Interpolated Adjoint Method for Neural ODEs. arXiv:2003.05277 2020.
Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes:
Incorporating prior scientific knowledge. In ICLR, 2018.
Logan Engstrrm, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations, 2019a.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and a translation suffice: Fooling CNNs with simple transformations. In ICLR, 2019b.
Chris Finlay, Jörn-Herrik Jacobsen, Levon Nurbekyan, and Adam M Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In ICML, 2020.
Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint arXiv: 10298, 2019.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In NeurlPS, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Andrew Howard, Mark Sandler. Grace Chu, Lian-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc Le, and Hartwig Adam. Searching for mobilenetv3. In ICCV, 2019.

---

Under review as a conference paper at ICLR 2021
Hwajoon Kim. The solution of the heat equation without boundary conditions. and Applications, 27:653-662 08 2018.
Dynamic Systems
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Kookjin Lee and Eric J Parish. Parameterized neural ordinary differential equations: Applications to computational physics problems. arXiv preprint arXiv:2010. 146855, 2020.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDEs from data. ICML, 2018.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks:
Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft, 2013.
A. Mas-Colell. Cooperative Equilibrium, pp. 95-102. Palgrave Macmillan UK, London, 1989.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes, 2020.
Wei Peng, W. Zhou, Jun Zhang, and Wenbing Yao. Accelerating physics-informed neural network training with prior dictionaries. ArXiv, abs/2004.08151, 2020.
Hans Pinckaers and Geert Litjens. Neural ordinary differential equations for semantic segmentation of individual colon glands, 2019.
Maziar Raissi. Deep hidden physics models: Deep learning nonlinear partial differential equa-
tions. Journal of Machine Learning Research, 19(25):1-24, 2018.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Markus Reichstein, Gustau Camps- Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Car- valhais, et al. Deep learning and process understanding for data-driven earth system science. Nature, 195--) 2019.
David Rolnick, Priya L Donti. Lynn Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Soojevic-Duppott Natasha Jaques, Anna Waldman WaldmannBroww, et al. Tackling climate change with machine learning. arXiv preprint arXiv: 1906.0543, 2019.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pp. 1-13, 2019.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? In NeurlPS, 2020.
Alvaro Sanchez-Gonzalez Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin A Ried- miller, Raia Hadsell, and Peter Battaglia. Graph networks learnable physics engines for infer- ence and control. In JCML, 2018.
Mark Sandler, Jonathan Baccash, Andrey Zhmoginov, and Andrew Howard. Non-discriminative data or weak model? on the relative importance of data and model resolution. In ICCV Workshops, 2019.
10

---

Under review as a conference paper at ICLR 2021
Xu Shen, Xinmei Tian, Anfeng He, Shaoyan Sun, and Dacheng Tao. Transform-invariant invariant convolu- tional neural networks for image classification and search. In MM, 2016.
E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
L. Yang, Luo, C. C. Loy, and X. Tang. A large-scale car dataset for fine- the-grained categorization and verification. In CVPR, 2015.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In ICLR, 2020.
Juntang Zhuang, Nicha Dvoreek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In ICML, 2020.


Reviews:
[{"review_id": "qT12qioRooK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary: \n\nThe authors propose Neural PDE as an enhanced alternative of Neural ODE, which learns both the governing equations and the target labels by alternating between solving the forward problem and the backward problem.\n\n2. Clearly state your recommendation (accept or reject):\n\nI am leaning towards recommending an accept to this paper, as it proposes a coherent way of solving supervised-learning tasks with Neural PDE.\n\n3. Strong points:\n\nThe model is reasonable. Unlike Neural ODE, it does not require numerical integration. It outperforms models including ResNet and ODE-Net in image classfication tasks, and can also generalize to out-of-distribution samples.\n\n4. Ask questions you would like answered by the authors:\n\na) The main question I have is the relationship to prior work, including Ruthotto and Haber (2019), Long et al. (2018) and Raissi et al. (2019). The authors did mention, for example, that the main difference with Long et al. (2018) is that the latter focuses on scientific problems and only uses a set of discretized points of t. What else are the novelties compared to the previous approaches, besides these as well as alternating between the training of the forward problem and that of the backward problem? It may also be reasonable to compare against those models in the experiments.\n\nb) Any reasons why the boundary conditions removed, and how is the set H of (d, t) pairs selected?\n\n\n5 Additional comments:\n\nOn the topic of physics-informed differential-equations-based models, there are some other works that may be worth referencing:\n[1] Greydanus, Samuel, Misko Dzamba, and Jason Yosinski. \"Hamiltonian neural networks.\"\n[2] Chen, Zhengdao, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. \"Symplectic recurrent neural networks.\"\n[3] Cranmer, Miles, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. \"Lagrangian neural networks.\"\n[4] Zhong, Yaofeng Desmond, Biswadip Dey, and Amit Chakraborty. \"Symplectic ode-net: Learning hamiltonian dynamics with control.\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "j-8oG5LYTe4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of this paper is to address the numerical instability issue in neural ODE method when solving the integral problems. To avoid the integral problems, the new methods treat both the layer and hidden dimensions as continuous variable, and solve them at the same time by learning a regression model. \n\nI also like the idea of learning governing equation and regression model together by an alternating algorithm, which in theory should be better than training a differential equation based neural network with priori knowledge of governing equations. \n\nMy main concern is that to avoid the integral problem in neural ODE, the authors paid the price to treat the whole neural network as a fully coupled system and had to solve for all the layers simultaneously, which needs more variables/parameters to tune. In addition, there is also a number of parameters to add in order to solve for governing equations. All this changes will make the hyper-parameter tuning much more difficult than neural ODEs. The authors should comment on this point during the rebuttal period. \n\nThe new proposed method did give a new insight to handle the numerical instability in neural ODEs, with a bit cost of making model more complicated to tune though. Overall, I would recommend a weakly accept.       ", "extra_scores": {}, "extra_text": {}}, {"review_id": "cYOUnUtDycA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-discussion update: The authors gave a fantastic, thoughtful and exhaustive response that did clarify all of my concerns about the paper. They also updated the paper considerably (making much better), and crucially changed the title to be very accurate the contents.\n\nI like the paper now a lot, but the unimpressive results still stand. The PDE-based image classification performs ok, but also sometimes does not work very well. This would still be ok if insightful analysis of why the model improves would be provided. Unfortunately there is almost none of this, and then the contribution is more in the engineering side than science.\n\nI would not object acceptance, but I would prefer the work to be more complete in this regard first. I raise my score to 5.\n\n----\n\nThe paper proposes to solve the forward and inverse problems of PDEs simultaneously (that is, learn both the governing differential, and the forward solution surrogate). This is a dramatic and bold idea, but the paper does not explain why this combination would be a good idea, or what’s the benefit. It’s unclear why is the solution surrogate useful. The experiments show that this provides good results on image classification, but the PDE motivation is lacking. The resulting model does not require any integration, which is a major advantage. However, the paper should be more transparent on discussing the disadvantages of lack of integrals. Without forward solutions, the model is at risk of cumulating errors over time.\n\nThe paper seems to borrow its ideas almost completely from Raissi2019, and differences to it needs to be explicated. It seems that this paper takes Raissi2019 method and adds a loss function suitable for image classification. Given that none of the experiments are actually about learning PDEs (they are all image classification), this paper is very misleadingly titled. The method is also very incremental, and seems more like an application of Raissi2019 than an independent research work. \n\nIt’s also difficult to see why one would use a PDE for image classification at all. Labelling small images is already effectively a solved problem. I fail to see the PDE'ness of images.\n\nThis work also does not actually learn a “neural PDE”, since the governing equations are assumed to be 16-parameter predefined function, and not a neural function. Only the solution surrogate seems to be a neural network. The title is then misleading also in this regard. \n\nI also have hard time seeing why not develop this bidirectional method for ODEs first? The paper should do this as an ablation study to first show that it grants some benefits in the simpler ODE case. \n\nThe paper is written in a confusing manner, and lacks presentation polish (typos, language mistakes, strange figure order, lots of dubious statements, etc). The presented methods are also not introduced properly, and it seems that the reader needs intricate understanding of Raissi2019 first.\n\nThe experiments show that the PDE-net, ODE-net and ResNet are all equally good at classifying MNIST and SVHN (with no significant differences). The results are missing log-likelihoods, standard deviations and training time analytics. It’s difficult to see what’s the benefit of the method here. In Tiny-experiments the comparison target of MobileNet seems arbitrary. Why compare to a mobile phone -optimized classifier, given that PDE’s surely are far from ideal on such settings. There are no large-scale image classification tasks, nor standard image baseline methods (alexnet, vgg, wresnet). The ResNet comparison is also missing from Tiny. \n\nThe out-of-distribution experiments are excellent, and show the PDE’s improve clearly from ODEs and beat MobileNet. These results are very interesting, and potentially significant. Here more exhaustive experiments should be done, and comparisons to other augmentation methods performed. The authors should also try to give insight why the PDE is more robust to perturbations.\n\nOverall the paper presents an incremental improvement to PDE learning with limited novelty, with unclear presentation, unclear motivation and mixed but partially promising results. The paper needs more work, and should be reworked to be more independent of Raissi2019, and refocused (incl. title) towards the promising application of image robustness, and *why* the PDE are more robust than ODEs in this setting.\n\n\nTechnical comments:\no I do not understand what the dimension “d” means. It does not seem to a dimension at all, but instead a state vector of the state space? The notation is very misleading\no Neural ODE’s do not have particularly small number of parameters (they are often applied in very simple cases or in small latent spaces, but here other NN’s would be simple as well)\no what is “procrastinate”?\no the paper confuses layers and time to be equivalent, this is not true in neural ODEs\no where is eq 1 coming from, and why is the model only restricted to 3rd order (monomial) differentials? Surely a more general PDE definition could have been used\no “general purpose PDE solvers do not exist”: surely they exist, but are too slow to be practical\no eq 6: what is “h”?\no eq 6: why is Raissi2019 performance studied here? I fail to see what’s the relevance of repeating someone else’s work. Is there some novelty here?\no fig4: all methods seem to have very poor fits, given that this is a simple 1D problem with massive amount of data. \no sec3: h(d,t) should be a function of “h0” as well, and its unclear if this is a true or proxy solution.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3NB98AeNdwR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposed the method of neural PDE as an improvement of neural ODE. In specific, neural PDE considers both the layer and the hidden dimension as continuous variables of the PDE. The new part of neural PDE compared to neural ODE is essentially solving PDE inverse problems (learning PDE from data) in the computational mathematics and engineering community, and the way of learning PDE (by embedding the PDE and initial condition into the loss function via automatic differentiation) is the physics-informed neural network (PINN) proposed in [Raissi et al., JCP, 2019]. The experiments show that compared to neural ODE, neural PDE achieves comparable accuracy but with less forward-pass inference time; but these experiments are not convincing enough.\n\npros:\n- Because the proposed method uses automatic differentiation to handle the derivative as PINN and thus can avoid the numerical integration, and the inference time of neural PDE is less than that of neural ODE.\n\nMajor comments:\n\nResults:\n- The paper only shows the results on small problems in two tables, which is convincing. To show the performance of this method, the authors should add more experiments and also show more details of the behavior of the method, e.g., training.\n- The training procedure is complicated. It includes 4 steps: first train L_T, then L_T + L_C + L_G, etc. It is not clear why the authors train in this way. Are the hyperparameters difficult to tune? How stable is the training?\n- The loss includes high-order derivatives of network f, which makes the training much more expensive. What is the computational cost of training? Also, the training trajectories should be added to show the convergence behavior of the loss.\n\nMethods:\n- The novelty of neural PDE is replacing ODE in neural ODE with PDE, and adds the PDE loss. But the way of handling PDE loss is exactly the PINN [Raissi et al., JCP, 2019], which is one of the main references in the paper. The authors should state this clearly.\n- The authors use the example of Allen-Cahn equation to deliver the message that training with PDE would have better accuracy for extrapolation. However, this example of Allen-Cahn equation is exactly the same example used in [Raissi et al., JCP, 2019]. It might be OK to repeat the example and result from another paper to deliver their message, but the authors should state it clearly. Also, this phenomenon has already been observed in the computational engineering community, e.g., https://www.biorxiv.org/content/10.1101/865063v2 . In fact, I think this part is not the main part of the paper, and can be moved into appendix.\n- Section 3 is not well written and the description is not clear.\n    - (d,t) are the “symbolic” variables of PDE, and the user can choose their values arbitrary to compute the loss L_G. But they are also the parameters to be trained. Are the (d,t) in L_G the same as the (d,t) used for construct h_last?\n    - Is (d, t) the same for all the inputs?\n    - Is d in one dimension?\n    - It is not clear how the authors select H? How many (d,t) pairs are in H?\n    - Are d and t unbounded?\n    - It seems that the authors use some points (d,t) in the whole domain to construct h_last; if this is the case, then the “last” is not correct, which usually means the PDE solution at the last time.\n- In appendix, why the input dim of f is 6^2x67 not 6^2x67+2? because there are extra inputs of d and t. Why does the network f use ReLU? The derivative f_{dd} would be zero everywhere. The authors should show the details of the network including the (d,t) part.\n\nMinor comments:\n- In the introduction, it is not correct that “they studied PDEs in scientific problem domains and do not consider t as a continuous variable but use a set of discretized points of t.” In PINN, t is treated as the continuous variable. The authors should be aware of this.\n- Some notations are confusing. For example, d is usually the dimension, but here d is the space variable.\n- Why the authors use L_C to denote that initial condition? Why not L_I? since L_B is for “boundary” and L_G is for “governing”.\n- The appendix does not have all the information of hyperparameters, e.g., what is the size of H?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for regularizing image classifiers by encouraging their hidden activations to conform to a PDE.  This is a reasonable idea, and the authors clearly improved the paper a lot in response to the reviews.  However, the main tasks of MNIST and SVHN classification seem way too easy, and the baselines all need to be tuned to be as fast as possible for a given accuracy, if that's the relevant metric.  I agree with the reviewers that this line of work is promising but that the current paper is not sufficiently illuminating or well-executed to meet ICLR standards.
Decision: Reject