Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper claims that neural ODEs are more robust to adversarial examples than ResNets and offers both empirical evidence and a theoretical explanation. However, I don't believe the theoretical analysis shows what is claimed; in fact, it doesn't seem to say anything about the magnitudes of Lipschitz constants or a relative comparison between them in the case of ResNets vs. neural ODEs. Below are the main issues with the analysis:

Claims: 

In the statement of Thm. 3.1, it is claimed that in ResNets, the distance between z_n (output corresponding to perturbed input) and y_n (output corresponding to the original input) is bounded above by some constant times the amount of perturbation to the input. The statement does not upper bound the constant and only lower bounds it trivially with 0. Therefore, this says nothing about how robust or not robust the model is to adversarial examples -- if the constant is large, the model could be sensitive to adversarial examples; if it's small, it could be robust. So, this statement is unrelated to the main claims of the paper. 

In the statement of Thm. 3.2, a very similar claim is made about neural ODEs, except that the constant in front of the magnitude of perturbation (\hat{c}) is different from the constant in Thm. 3.1 (c). No claims on the relative magnitudes of \hat{c} and c are made, and so nothing can be said about the relative robustness or lack thereof of neural ODEs compared to ResNets. 

Proofs:

If I understand correctly, Thm. 3.1 is supposed to make a claim about ResNets, which is when n (the number of steps/layers) is held constant as the step size (h) changes. In this case, the last step of the proof (eqn. 11) is incorrect, since nh does not necessarily equal to b - t_0, i.e. when n is fixed and h decreases, Euler's method will not reach b. 

On the other hand, Thm. 3.2 seems to aim at making a claim about neural ODEs, which is when n increases as h decreases. In this case, nh would be equal to b - t_0. However the claim that the distance between z_n and y_n would be upper bounded in the limit of h -> 0 is odd, because it holds even when no limit is taken: nh is always equal to b - t_0 regardless of how small or large h is. So this cannot serve as an explanation for the differences in behaviours between neural ODEs and ResNets (where h = 1). 

Significance:

The theoretical analysis is trivial and is a simple application of the definition of Lipschitz continuity, which is assumed. Additionally it has nothing to do with the main claims of the paper. 

Conclusion: 

While the empirical phenomenon has been convincingly demonstrated and is intriguing, the theoretical analysis doesn't contain anything substantive and isn't much of an explanation. No other attempt at explaining this phenomenon (empirically or otherwise) was made other than the provided theoretical analysis. So I believe the paper is incomplete and will need substantially more effort at explaining the phenomenon. 
review_title: Review for Paper

final_score: rating: 3: Clear rejection
review_content:  Summary:

This paper uses theoretical grounding, starting with Lipshitz continuity-based assumptions on residual connections, to show why such architectures are more susceptible to adversarial inputs. In the process, the authors draw a parallel between these residual connections and neural ODEs, showing how the latter can circumvent the main reason that leads to adversarial susceptibility for the former. Finally, via empirical evaluations, they show how neural ODEs have "natural" robustness to adversarial examples: they have a non-trivial performance on adversarial inputs, despite not being explicitly trained for robustness.

 
##########################################################################

Reasons for score: 

This draft in its current form lacks strong adversarial evaluation and makes strong claims without any experimental evidence. The title and the body of the paper suggest "natural adversarial robustness", but evaluated only against ($L_2$: this is my guess since it is not specified in the paper) PGD and FGSM (which, as has been seen recently in the literature, is not useful to see) attacks. Moreover, the Lipshitz-based assumptions in the proof for deriving conditions for residual connections seem a bit too strong. Keeping strong assumptions and weak empirical evaluations in mind, I feel this paper needs a lot more work before it can be considered for acceptance. 
 
##########################################################################

Pros: 
  
- The analysis of existing variants of network architectures that include residual connections seems interesting. It helps to see why these models are theoretically unfit to achieve adversarial robustness. Some empirical evaluation on these models would also have been nice to see: just to see how one is more/less problematic than the other.
 
##########################################################################

Cons: 

- My biggest concern is the weak evaluation of the mentioned models. Even though the paper's title (and most places in the paper) talk about "natural robustness", evaluation is performed only against one kind of attack (FGSM is just PGD with the number of steps = 1) and that too for just one norm. Please include more extensive evaluation, perhaps like second-order gradient attacks, gradient-free attacks, and augmentation based adversarial attacks, or change the title to reflect your current findings. 

- A large body of related works seems to have been missed out in this paper's literature review. For instance, there have been several works on Lipshitz-constraint-based robustness for neural networks are highly relevant, but seem to have been missed out ([example](https://arxiv.org/pdf/1704.08847.pdf), [another example](https://openreview.net/pdf?id=HkxAisC9FQ))

- The second paragraph sets up the flow of the paper to hint at "designing a deep neural network that has natural robustness", whereas the main focus of this paper goes to the extent of only evaluating existing architectures. Additionally, there is no clear evidence to suggest that adversarial robustness is even possible just via a well designed neural network.

- Residual connections are not limited to skipping only one connection (Eq 1), and using this as a base assumption should either be explicitly stated or worked out for the general case.

- The function $f$ is related to one specific layer and is thus parameterized by its associated weights $\theta_n$. However, Eq 7 talks about the Lipschitz continuity for the general layer. Does this mean that this model assumes that __all__ layers under consideration are Lipschitz continuous, that too with the same constant? In an ideal scenario where all layers somehow indeed, using the same constant $K$ would require taking the largest one of all layers, which can make it ridiculously large. I think this is a fatal flaw in the derivation process, and the authors should address it.

- In what norm are all these examples operating? What are the parameters for the PGD attack? Configuration details like the number of steps, step size, and the number of random restarts, can make a significant difference in evaluation metrics. Also, since these attacks include randomness (especially PGD), please run them multiple times and report mean/std values.

- Many numbers in Table 2 do not make any sense (my guess is this is because of the randomness in these attacks, which makes it even more important to have multiple runs). For instance, accuracy __increases** for neural ODE on CIFAR10 for FGSM when the perturbation budget is **increased**? Attack success rates should be strictly non-decreasing with increasing attack budgets since the adversary can copy-paste smaller-budget attacks and get at least the same attack success rate. The same problem holds for Resnet50 on CIFAR10 ($\epsilon=0.4$ vs $0.5$), neural ODE on MNIST $\epsilon=0.3$ to $0.4$. 


Please address and clarify these cons.

##########################################################################

Minor issues:

- Page 1, last paragraph "...which has been used to solve ordinary differential equations". Reference missing.

- Section 2.1, last paragraph: "...rely on using external models". This statement is not true. Popular adversarial defense techniques like feature squeezing([ref](https://arxiv.org/pdf/1704.01155.pdf)) do not augment the dataset or use an external model.

- Ambiguity in notation: next to Eq 1, what does $N(h)$ signify? 

- The move from Eq 7 to Eq 8 is a bit non-trivial: please add more steps in between to show the process explicitly 

- Section 3.3.1, just above Eq 16: "It has been show that ...". Reference missing

- Just above Eq 18: "can be formally rewrite" -> "can be formally rewritten as". Also, the $(1+x)^{-1}$ Taylor series expansion holds only when $|x| <1$. Is that the case here, *i.e.* is $|hf|<1$ ?

- Section 3.3.2, just above Eq 20 "resembles the Runge-Kutta method of order 2". Reference missing

- Is the x-axis for Figure 1 on the log-scale? Please clarify Please do not refer to adversarial inputs as "adversarial test set", as it is likely to be confused with an adversarial test set that has been generated offline and is used for evaluation.

- What are the numbers in Table 2? Accuracy ($f(\hat{x}) = y$), or 1 - error success rate ($f(x) = f(\hat{x})$)? Please clarify. 

review_title: Misleading title; needs much stronger evaluation

final_score: rating: 3: Clear rejection
review_content: **Update**: Thank you to the authors for addressing the comments and updating the paper. I decrease my rating from 4 to 3 as the original claims of the paper were disproved by the experiments with black-box attacks on CIFAR10 which showed that Neural ODEs offer little advantage over Resnets. I believe that more experiments are needed to demonstrate that Neural ODEs offer robustness advantages. For example, when increasing the computational budget for one step FGSM attack, the accuracy stays the same, which might indicate that Neural ODEs obfuscates the gradient. Experiments with gradient-free attacks also indicate that Neural ODEs obfuscate the gradient (PGD has higher robust accuracy compared to gradient-free attacks). The authors should do an extensive evaluation with stronger attacks (PGD with DLR loss or CW loss with multiple random restarts up to 100-1000, AutoPGD); the study of the gradient obfuscation (confirm that when $\epsilon=1.0$, the attacks can always succeed).

#### Summary
The paper analyzes the robustness of residual deep neural networks through its relationships to ordinary differential equations. It establishes bound on the output change. It tries to explain the lack of robustness of ResNets. It suggests that Neural ODEs addresses the limitations of residual mappings and are naturally robustness to adversarial perturbations. In the experiments, the authors compared Residual Network and Neural ODEs to confirm that Neural ODEs are naturally robust.

#### Concerns:
- Missing references, e.g. Towards Robust ResNet: A Small Step but a Giant Leap Zhang et al (shows the limitation of ResNet and searched for optimal $h$ using grid search).
- Limited technical novelty in the main theoretical results in Theorem 3.1. Similar bounds in terms of Lipschitz constant might be established elsewhere.
- One of the main claims of the paper is that ResNets are not robust because $h = 1$. However, $c$ depends both on $h$ and $K$. So, it is possible to improve the robustness of ResNets by reducing its Lipschitz constant. However, this was not discussed in the paper.
- The paper suggests using Neural ODEs, which was published elsewhere. Yet, the experimental evaluation is limited. Models should be evaluated against a wider range of attacks, e.g. black-box attacks and other attack norms ($l_{2}$-norm).
- The attack parameters are not specified, e.g. number of attack iterations, number of restarts. 
  
#### Minor comments:
- Incorrect citation for adversarial training method: cited Ganin et al. instead of Goodfellow et. al.
- Unfortunately, both of these explanations seem to imply that adversarial examples are inevitable to be avoided by deep neural networks. - the structure of the sentence might be incorrect.
- In this paper, we attempt to utilize the natural property of neural networks to dense adversarial examples. - the sentence is hard to understand.

review_title: The paper analyzes the robustness of ResNets and Neural ODEs, highlights limitations of ResNets and suggests to use Neural ODEs. Some important references are missing; technical novelty is minor.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper uses a high-order ODE solver to take an $h=1$ step of a neural
network layer.  The mechanics of training with an ODE solver that uses
parameters to determine the dynamics of producing a layer output is previously
known.  This paper notes that outputs produced using an ODE integrator, wrt
adversarial inputs, have established error bounds. They demonstrate the
additional adversarial robustness of operating in a regime better respecting
such bounds.

I feel the paper is well written and clear.  The central theoretical idea is not a huge leap, but is
novel in the context of robust machine learning, imho. While presenting a simple, basic idea is
always nice, the paper left me unclear about whether the demonstration was primarily intended
to demonstrate a not terribly surprising theoretical prediction, or whether the technique would
be useful in practice.

They show many common networks with skip connections (resnet, etc.) correspond
to a forward ODE integration scheme with large step size $h=1$, whereas the
theoretical ODE adversarial bounds only hold as $h\rightarrow 0$.  They first show 
that using fixed h corresponding to some fixed learning rate either does not
satisfy $h\sim 0$ or has too slow convergence, with no gain in adversarial
robustness.  So instead, they use a variable step size integrator instead, to
project forward to a larger $h$.

The idea and theory are simple and fairly well presented, and the demonstration
on a simple dataset nicely shows the benefit of this approach compared to the
original neural network.  However I felt a large number of significant things
were left out regarding choice of integrator.  Most obviously, Table 1 lacks
dopri parameters.  How does adversarial robustness depend on such parameter[s]?
What is the effect on execution time?

They only use one variable step size integrator.  They use a 4th and 5th order
variable step size integrator, to demonstrate the predicted natural ODE-based
robustness.  Several times I felt the presentation hinting that the low-order
of integration schemes is to blame; however, for layers with discontinuities
(relu), I naively expect lower order (and more evaluations?) might work out
just as well.

While the experiments constitute a simple demonstration, it still remained
difficult to judge the practical importance without some data comparing
execution time.  And once one begins to consider efficiency, a question
of what styles of integrator work well in practice would be nice (ex. gear
vs. Richardson vs. their one chosen integrator).

Even if the ODE method is expensive, compute time can be compared with another
ways to promote robustness, such as adversarial training.  A comment about
feasibility of using ODE method in conjunction with other robustness methods
could be made.

---

I have read the authors' comments.  The addition of the boundary attack experiment
was an excellent step; however, it underscores a requirement for further analysis to
understand *why*, apparently, in some cases the additional theoretical bound *fails* to
confer significant robustness.  The suggested "natural robustness" is only sometimes
present. Often clean accuracy is much reduced, so the method is not yet one I would consider
useful yet.  For me, understanding when the method works well (or not so well) would
bring this work out of the realm of interesting theoretical bounds into one of more
general interest.

review_title: ODE integrator error bounds intrinsically confer adversarial robustness to Neural ODEs

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper offers an interesting viewpoint of adversarial robustness by comparing neural networks with skip connections such as ResNet with their Neural ODE counterparts. The authors analyze the different behaviors of the networks through their Lipschitz constants. They also try to support their claims that Neural ODEs are more robust due to their continuity (small step sizes) through experiments.  

One issue I find with the claim that neural ODE is more robust against adversarial examples compared to other neural networks with skip connections is that it does not appear to be an apples-to-apples comparison. The authors try to demonstrate in Theorems 3.1 and 3.2 that Neural ODE has a smaller Lipschitz constant due to using a smaller step size, but it does not take into account the different representational capacity of the model when we use step size h=1 in ResNet. Unless the authors can show the neural ODE version and the finite step size ResNet computes the same/similar classes of functions, comparing the Lipschitz constant is not very helpful. The same comparison issue is also present in the experiments, where the authors use different architectures for neural ODE and ResNet (many fewer layers for neural ODEs). It is very difficult to draw conclusions on which neural network is more robust if they have different representational capacities. 

Other than this main issue, there are also some limitations of the current work. It only analyzes neural networks with skip connections, whereas the phenomenon of adversarial attacks is general for many different neural networks. Also, judging from the accuracies under FSGM and PGD attacks, using a neural ODE cannot beat current state-of-art approaches of adversarial defense such as directing training with PGD examples. 

As an aside, can the authors supply more details on how they generate adversarial examples for the neural ODE model? 

Based on the above observations, I believe the current paper is not ready for publication in ICLR yet. The author should focus on improving on their core argument on why the factor of step size directly contributes to robustness of neural ODEs. 

review_title: Interesting viewpoint, but main claim not convincing due to deficiencies in comparison methodology 

Meta-review: I thank the authors and reviewers for the discussions. Reviewers raised major concerns regarding the significance of the results and experiments. Given all, I think the paper needs more work before being accepted. I encourage authors to address comments raised by the reviewers to improve their paper.

- AC
Decision: Reject