Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper tries to improve the robustness of neural ODE via enforcing the Lyapunov stability constraints in the training process. The key idea is as follows: a dynamic system can have multiple equilibrium points, and the input perturbation around the equilibrium point will decay and not significantly influence the result if the equilibrium point is Lyapunov stable. This is an excellent contribution that connects the Lyapunov stability of ODEs with the robustness of neural networks.  
main_review: This is a very well written paper. The authors proposed a new neural network to enforce the robustness of neural networks. The network consists of three blocks: (1) a feature extractor block, (2) a neural ODE block that satisfies the Lyapunov stability condition, and (3) a diversity promoting FC layer. By enforcing the Lyapunov condition of the neural ODE, the input perturbation is guaranteed to decay and will not significantly influence the prediction. 

On the implementation side, the Lyapunov condition is achieved by enforcing the Jacobian matrix A to be dominantly diagonal and have eigen values with negative real parts. These constraints appear in the regularization term of the training loss function. 

The research results are well justified by theoretical analysis. The proposed Lyapunov stable model has shown significant robustness improvement compared with neural ODE trained without Lyapunov stability constraints. 

Some minor comments:
1. Some important references of neural ODE are missing. The idea of neural ODEs were firstly proposed in the applied math community, and the following papers should be cited:
--W. E,  A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1–11, 2017.
--Eldad Haber and Lars Ruthotto.  Stable architectures for deep neural networks. Inverse Problems,34(1):014004, 2017.

2, The following papers improve the robustness of neural ODEs via open-loop and close-loop controls. They are ignored in this paper:
--Chen Z, Li Q, Zhang Z. Towards Robust Neural Networks via Close-loop Control. In International Conference on Learning Representations 2020 Sep 28
--Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong.  You only propagate once: Accelerating adversarial training via maximal principle. In Advances in Neural Information Processing Systems, pp. 227–238, 2019.

3. The authors assume that the ODE is time-invariant. I wonder how they handle the model parameters (e.g., weight matrices and convolution filters) at different layers: are they the same across different layers?

4. More details should be provided to explain why we need the feature extraction block and how this block should be designed for general cases. 


limitations_and_societal_impact: It would be great if the following points can be discussed/clarified:
1. The linearization analysis only works if the perturbation is small. How strong is this assumption?
2. How many equilibrium points should the system have in the training process? How can we control the number of equilibrium points? What's the relationship of the number of equilibrium points with the number of output labels?

A potential limitation: this work only compared with standard neural ODE, and has not been compared with other robust neural ODE, for instance:
--Chen Z, Li Q, Zhang Z. Towards Robust Neural Networks via Close-loop Control. In International Conference on Learning Representations 2020 Sep 28
--Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong.  You only propagate once: Accelerating adversarial training via maximal principle. In Advances in Neural Information Processing Systems, pp. 227–238, 2019.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a neural network classifier architecture based on neural ODEs that is designed to be robust to adversarial attacks. In the proposed method, the classes are represented by vectors, chosen to be maximally separated in the output space w.r.t. cosine similarity. The neural ODE system is then encouraged to have equilibrium points at those class vectors via regularization; the idea is that perturbed outputs are pulled towards the equilibria as the ODEs evolve.
main_review: This paper builds on a line of work that connects neural ODEs with adversarial robustness. The approach was interesting - a diagonally dominant condition was used as a scalable approach to enforce stability conditions on the dynamics Jacobian. The various experiments visually and quantitatively show the potential benefits of locally attractive diverse class label targets. In my understanding, the main strength is that the SODEF layer mainly serves to stabilize the output of a pretrained classifier, and does not really require much predictive power of its own - this may make it easier to tune the several regularization hyperparameters that are required. It seems that a pretrained model can be used as the feature extractor $h_\phi$, but it is necessary that its output $z(0)$ is sufficiently close to the target diversified equilibrium points. Could a neural network layer have been used to map the output of a pretained model to a diversified $z(0)$, without needing to retrain the entire feature extractor? Also, Runge-Kutta order 5 was used to integrate SODEF. Was the proposed method compared with simpler ODE solvers such as forward Euler? 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a defense against adversarial attacks using two orthogonal ideas: (i) the design of the final fully connected layer that maximizes the cosine distance between vectors corresponding to different labels, and (ii) design of a neural ODE whose Jacobian has only negative real parts and is hence Lyapunov stable near its hyperbolic equilibrium points (via Hartman-Grobman and linearization).
main_review: (+) While Hartman-Grobman theorem has been used in the stability analysis and generalization/robustness of neural networks via linearization around hyperbolic equilibrium points (see Section 4 of [1] and Section 5.2 of [2]), its use in training adversarially robust neural networks is new. 

(+) The conclusion highlights that the approach is compatible with any existing neural network. It is clear that residual neural nets and their variants can benefit from this approach. It may be useful to comment on how this approach can be naturally extended to transformers without residual layers.

(+) The source code has been made available to the reviewers. Can the authors confirm that the complete pipeline in Figure 1 from x to y is being attacked during experimental studies?

[1] Kim P, Pan L, Wirjanto TS. Local Stability Analysis of Neural Network Models with Application to Exchange-Rate Data. Mimeo, University of Waterloo, Waterloo, Ontario, Canada; 1999. https://www.researchgate.net/profile/Tony-Wirjanto/publication/228936073_Local_Stability_Analysis_of_Neural_Network_Models_With_Application_to_Exchange-Rate_Data/links/00b7d5295fb1709e0a000000/Local-Stability-Analysis-of-Neural-Network-Models-With-Application-to-Exchange-Rate-Data.pdf

[2] Poggio T, Liao Q, Miranda B, Banburski A, Boix X, Hidary J. Theory IIIb: Generalization in deep networks. arXiv preprint arXiv:1806.11379. 2018 Jun 29.  https://arxiv.org/pdf/1806.11379.pdf


After reading the author's rebuttal, the reviewer's main concern has been met and the score of the paper has been raised.
limitations_and_societal_impact: (-) The experimental evaluation of the proposed approach, including results in the supplementary material, is limited. A good reference for completing a thorough evaluation is [3]. At the very minimum, PGD attacks with varying L2 and Linf norms should be reported. Modern attacks such as FABS and Square Attack should be evaluated. Can we obtain a report on the results of applying AutoAttack including PGD, FABS and SquareAttacks on these models? 

(-) The quality of the reported experimental results is low. For example, Table 1 in Sec. 2 of the Supplementary Material shows that the PGD attack uses a maximum perturbation size of epsilon=0.1 and a step size of 0.1 with 20 iteration steps (not size). The step size should be much smaller such as 2.5*0.1/20 following Madry [4]. AutoAttack https://github.com/fra31/auto-attack provides a parameter-free approach of performing such an evaluation.

(-) The Hartman-Grobman linearization holds only near hyperbolic equilibrium points. Why are these points the only points of interest for robustness of neural networks? Why is the dynamics in the rest of the state space not of interest for robustness analysis?

(-) Figure 3 states that the input is the adversarial examples of the test set of MNIST.  Are these plots obtained by looking at the adversarial examples obtained by attacking each of the three models: TisODE, TisODE and TisODE + V as FC?



Minor points:

1. Reference 8 focuses on stochastic differential equations and not ordinary differential equations. So, paragraph 2 on page 1 may need to be suitably edited. 

2. Table 1 has no results for applying PGD to MNIST while Table 2 applies both FGSM and PGD to CIFAR10. 

3. Table 3 does not specify the norm used for the PGD/FGSM attack. 

4. The paper or the Supplementary Material does not include investigations that sweep the magnitude of the perturbation (epsilon) and observe the impact on the robustness of the model. 

References:

[3] Carlini N, Wagner D. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) 2017 May 22 (pp. 39-57). IEEE. https://arxiv.org/abs/1608.04644

[4] https://github.com/MadryLab/robustness
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work aims to develop a neural ODE-based model to defend against adversarial attacks. Specifically, the authors propose a stable neural ODE with Lyapunov-stable equilibrium points (SODEF). By ensuring the equilibrium points are Lyapunov-stable, the ODE solution for an input with a small perturbation will converge to the same solution for the original unperturbed input. The paper provides theoretical insights into the stability of SODEF and how to choose the stability regularizers. Besides, the paper also provides design guidance for the final FC layer, that aims to separate Lyapunov-stable equilibrium points. The proposed SODEF can work in conjunction with several defense methods and can be applied to any network's final regressor layer.
main_review: This paper investigates the adversarial robustness from a novel perspective --- stable dynamical system. It is very interesting to study the connection between robust deep networks and the stability of dynamical systems

This paper aims to learn a stable dynamical system-based feature mapping which can mitigate the effect of perturbation. Stable points in a dynamical systems are fixed points with negative-definite Jacobians. Instead of directly compute the eigenvalues of Jacobians, this paper adds constraints to the Jacobian matrices by encouraging the Jacobian to be strictly diagonally dominant. The levy-Desplanques theorem guarantees the stability of systems of which the Jacobians are strictly diagonally dominant and with real negative diagonal entries.

Concern:
1. The formulated optimization problem does not impose regularization on the feature extractor $h_{\phi}$. The optimization-(7) only encourage that the feature extracted by $h_{\phi}$ is located within the neighborhood (say neighborhood with radius $r$) of stable points.  In this case, a perturbation within the $r$-neighborhood on the extracted feature will gradually vanishes.  However, since no regularization on $h_{\phi}$, the imperceptible perturbation on the input $x$ may be amplified --- the perturbed feature is highly likely outside the $r$-neighborhood. Then, the perturbed feature won't converge to the original and the robustness is not guarnteed. So, I think it is necessary to discuss this problem.

2. In section 4.3.2, the authors evaluated the robustness of SODEF with transfer-based attacks. I think it is necessary to evaluate the proposed methods via other more black-box attacks, such as the auto-attack (Croce et al. 2020). 

3. The authors provided the implementation codes in the supplementary materials. I skim the codes and find there may be mistakes of the implementation of TisODE. In the file "Tisodenet_cifar10_resnet.py", see lines 461-461, the second term in the loss is  abs (state_t1 - state_t2), which equals to the abs of integral of dynamics from t1 to t2. However, in TisODE (Eq-5), the L_ss is the integral of the abs dynamics from t1 to t2.

4. Some typos. For example, Neural ODE and TisODE are the main baselines in this paper.  In the Reference section, the citation of TisODE is not correct, TisODE is an ICLR 2020 paper, not NeurIPS-2018. 

-----------------------------
Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks. Francesco Croce,
Matthias Hein. ICML 2020
limitations_and_societal_impact: Hope authors can respond to the questions above.
review_title: null

Meta-review: This paper proposes a neural network classifier architecture based on neural ODEs against adversarial attacks. Some of the reviewers have concerns on the experiments, while in the rebuttal the newly added experiments convince the reviewers. All reviewers finally give positive support to the paper. Thus, I recommend accepting the paper and the authors should include the new experiments in the final version. 
Decision: Accept (Poster)