Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Inspired by the asymptotic stability of the general nonautonomous dynamical system, the asymptotic stability will reduce the adversarial noise to bring the adversarial instance close to the clean instance. Motivated by the theoretical results, the authors propose a nonautonomous neural ordinary differential equation (ASODE) and place constraints on its corresponding linear time-variant system to make all clean instances act as the asymptotically stable equilibrium points of a slowly time-varying system. In the implementation, the authors convert the constraints to regularizers of the loss function. The experimental results show that ASODE improves robustness against adversarial attacks and outperforms the state-of-the-art methods.
strengths_and_weaknesses: Strengths:
1. The paper bridges the gap between the asymptotic stability of ODE and robustness against adversarial attacks. The asymptotic stability of ODE helps reduce the adversarial noise. 

2. The authors provide some theories to transform the general nonautonomous dynamical system into a linear time-variant system. Based on the transformation, the authors create a new loss function by the constraints on the linear time-variant system to make all clean instances act as the asymptotically stable equilibrium points. 

3. The authors carefully proved the claims and further propose a nonautonomous neural ordinary differential equation (ASODE) that uses the asymptotic stability of ODE to bring the adversarial instance close to the clean instance. Experiments show that ASODE improves robustness against adversarial attacks and outperforms the state-of-the-art methods, which shows the significance of this work.

4. The analysis presented in the main paper and the appendix is thorough enough.

Weaknesses:

1. the reason why the authors use a dynamic system in this paper is not elaborated on.

2. The application of theorem 2 and theorem 3 is unclear. What is the relationship between theorem 2 and theorem 3?

questions: See the weaknesses above.
limitations: The paper has no potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper designs an ordinary differential equation (ODE) that can regularize adversarial training samples to be close to clean samples. The ODE is developed based on the asymptotic stability achieved by a slowly time-varying dynamic system. The evaluation involves different existing attacking methods and datasets with ResNet18. The experimental results show that the proposed ASODE can defend against existing attacks and outperforms SOTA defense methods. 

strengths_and_weaknesses: 
**Strength**

++ It is an interesting idea to mitigate noise and perturbation of adversarial attacks from the perspective of dynamic systems. 

++ The paper presents a detailed theoretical analysis of how a perturbed instance is optimized to converge to a clean instance. 



**Weakness**

-- Lack of details of attack methods and their settings 

-- Missing ablation study of ASODE 

-- Missing sensitivity analysis of critical parameters of ASODE, such as the choice of $\delta$

questions: Line 269, why the model $\textbf{f}_{\widetilde{\theta}}$ is only trained for 100 epochs? 

limitations: **Writing issues**:

It would be better to introduce the meaning of a symbol at its first appearance.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a control theory approach to robustness by trying to turn "clean" data points into a kind of attractor basin, such that perturbations are "denoised" by being attracted back to better-behaved spaces of the neural network. This results in improved classification accuracy under strong attacks like PGD and AutoAttack 
strengths_and_weaknesses: The idea of the paper is very interesting, and I like the approach. The improved robustness under attack is validating, my problems with the paper stem purely from its presentation. The paper does not state limitations, provides code, and is poorly organized for accessibility. 

The paper can be hard to follow, e.g., Figure 2 is referenced before Figure 1 is even seen/referenced! In particular, there is a long theoretical journey to developing the proposed method that is very math-heavy. The work would benefit enormously from some pseudo code / a final "bring it all together" section. This is exacerbated by a lack of code and the authors ignoring/lying the reproducibility checklist (see questions), and I am strongly against an essentially unanotated equation (19) being the final distillation - its needlessly unreadable. 


I overall like the idea, but believe two key related works are missing from the discussion. Most importantly [A], which is the first work to use control theory and dynamic systems to improve robustness, showing global perturbation bounds for a neural network by treating the parameters as an evolving system that needs to be kept stable. Second [B] showed that RBF style functions are naturally more robust to attack, of which I think this work is related by using the ODE to create attractor basins near "clean" samples is reminiscent at a high level. 

[A] Rahnama, A., Nguyen, A. T., & Raff, E. (2020). Robust Design of Deep Neural Networks against Adversarial Attacks based on Lyapunov Theory. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8178–8187. http://arxiv.org/abs/1911.04636
[B] Biggio, B., Fumera, G., & Roli, F. (2014). Security evaluation of pattern classifiers under attack. IEEE Transactions on Knowledge and Data Engineering, 26(4), 984–996. https://doi.org/10.1109/TKDE.2013.57 

-------------

I was in a bad mood when I wrote the OG review, I again apologize. I think the revision has satisficed - though not fully satiated, improved readability. But I need to draw the line between what I would prefer vs what is indeed an improvement and valuable contribution. I've raised my score accordingly. If the authors feel they can further improve the presentation I'm willing to further raise if successful. 

----

Re-reading I've updated my score. I'm having trouble separating my personal style preferences that are quite different from the reviews - and style differences are no reason to reject a work. So I'm hedging that others will find the paper more inviting given the improvements than I do. I think all objectively 'bad' writing style has been fixed. 
questions: The following reproducibility checklist questions have been incorrectly marked as "Not Applicable", please answer them: 

* Did you describe the limitations of your work? [N/A] (You didn't, please address)
* Did you discuss any potential negative societal impacts of your work? [N/A] 
* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? (You didn't, please address) 
* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] (You did not, this is incorrect - e.g., optimizer details are missing) 
* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] 
limitations: The approach is not applicable to any non-differentiable method like decision trees, and would be especially challenging to combine with current adversarial training techniques. These and other limitations should be addressed in the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the neural network robustness issue from the perspective of dynamical systems.
The paper illustrates the drawbacks of previous works in this direction and proposes an asymptotically stable ordinary differential equation to overcome those drawbacks.

There are two major advantages of this proposed method:

1: ASODE considers asymptotically stable equilibrium points that can eliminate applied perturbations at initial conditions.

2: ASODE considers nonautonomous dynamical systems that are more general than previous works that focused on autonomous systems.

Numerical results have demonstrated promising improvements.
strengths_and_weaknesses: Strengths: This paper considers robustness from the perspective of a well-developed control field, which helps to effectively connect those two fields.
The idea of using asymptotically stability rather than just stability can guarantee to eliminate of applied perturbations.
The numerical experiments provided in the paper are very promising, against all perturbations, the proposed ASODE has shown improvement over the previous state-of-the-art methods.

Weakness: I enjoyed reading this paper.
However, I found some grammar issues.
I suggest that authors carefully check the paper writing and improve the readability.
questions: I think the proposed ASODE has many overlaps with the SODEF method.
Could the authors itemize the major difference between ASODE and SODEF? 
limitations: I did not find the discussion on the limitations of this proposed method.
Could the author provide a detailed discussion on the limitations here?
review_title: null

Meta-review: metareview: The paper studies the problem of enhancing neural network robustness from a dynamic system perspective. To this end, the authors proposed a nonautonomous neural ordinary differential equation (ASODE) that makes clean instances be their asymptotically stable equilibrium points. In this way, the asymptotic stability will reduce the adversarial noise to bring the nearby adversarial examples close to the clean instance. The empirical studies show that the proposed method can defend against existing attacks and outperform SOTA defense methods. Most reviewers rated this work positively and agreed that the proposed method is interesting, technically sound, and theoretically grounded. In the original reviews, they also raised several concerns including missing ablation study and sensitivity analysis, missing references, and some presentation issues. The authors properly addressed these concerns in the rebuttal and after the discussion, some reviewers increased their scores and the majority leaned towards acceptance. Overall, given the general support from the reviewers and the revised version of the paper, I recommend accepting the paper.
confidence: Certain
Decision: Accept