Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an adversarial example detection mechanism for residual networks, as well as a regularization method to improve the detection performance. The detection mechanism is based upon viewing the deep network as a dynamical system. The regularization method is an existing method based on optimal transport that the authors show theoretically and empirically makes adversarial examples easier to detect.  
strength_and_weaknesses: Strengths
- The authors show their detection method performs better than the Mahalanobis detector on known and unknown white box attacks, and also on known black-box attacks. 
- The regularization method proposed also improves the performance of the existing Mahalanobis detector.
- The authors give theoretical backing behind their choice of regularization method. 

Weaknesses
- Overall I found the paper hard to follow. I think the paper would benefit from more explicitly stating how the detection is implemented. 
- The method is restricted to residual networks, whereas the detector being compared to (the Mahalanobis detector) I believe is applicable to different types of architectures.
- I'm not sure whether you specify that you are considering the l-infinity norm threat model in section 5. 
- There are a lot of unclear statements and loose ends. For example, rather than saying "We don’t fine-tune these hyper-parameters much", it'd be better to be more specific with respect to the details.
- There was never much of an introduction or review of adversarial examples more formally -- you may wish to add this. 
- The computational complexity of the detection method was not made explicit. 
- You could more explicitly compare your method to the method of comparison (the Mahalanobis detector) in the related works section for better clarity. 

Other comments/questions:
- Why do you use the test set for training the detector? It seems like it would be more reasonable to hold out part of the training set.
- This paper focuses on detecting adversarial examples specifically, how does it compare to existing methods on other OOD detection tasks? 
clarity,_quality,_novelty_and_reproducibility: I found the clarity of the paper overall lacking. The paper introduces what I believe to be a novel detection framework, and a novel application of the particular regularization method. 
summary_of_the_review: I am leaning towards reject because I found the paper lacking clarity/difficult to follow in terms of the methods proposed. However, the final results of the paper are promising.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to detect adversarial examples from a perspective that models residual networks as discrete dynamical systems. The detector studies trajectories of samples in space, through time, to distinguish between clean and adversarial examples. Based on this rationale, the authors also apply transport regularization during training to make the detector perform better. Though this method does not necessarily make the model more robust to adversarial attacks, it makes the model's behavior on adversarial examples more distinguishable.
strength_and_weaknesses: Strength:
The proposed method has a solid theoretical background and the idea of applying trajectory regularization in optimal transportation to adversarial example detection is also very interesting. The theoretical studies look solid to me. The authors also conducted experiments to show the effectiveness of the proposed method.
Weakness:
1. My first concern is that the authors did not test on adaptive adversaries, which may have access to the detector itself. Similar issues also exist in other works on adversarial examples detection and defense. There are existing works that show that random forest and other decision tree-based models are also vulnerable to adversarial attacks (e.g. Kantechelian et al "Evasion and Hardening of Tree Ensemble Classifiers", ICML16). So I think if the attackers are able to design attacks on the detector, the proposed method can be bypassed. 
2. Even for non-adaptive attacks, SOTA attacks such as AutoAttack (Croce and Hein, ICML20) are not tested, either. I think it is necessary to test under strong attacks. For example, LID mentioned in this paper fails under high-confidence adversarial examples according to Athalye et al ICML18.
3. Minor: Please refrain from only using color to distinguish points and bars as in Figure 1, Figure 2 and Figure 3, as it is not friendly to readers with color blindness.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow in general. I am not concerned about its originality and reproducibility, after checking the code that's released anonymously by the authors.
summary_of_the_review: In general, I think this paper proposed an adversarial detection method from an interesting perspective. The theoretical analysis is solid. However, the experimental part may not be enough due to the lack of adaptive attacks and strong attacks, which makes the proposed method less convincing. So I would recommend a borderline reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles the problem of detection of adversarial examples targeting (deep) neural networks. Specifically, the paper models the underlying operations of neural networks as a “dynamic process”, wherein information is propagated “over time” (i.e., goes through layers of neurons). By acting upon such intuition, the paper proposes to detect adversarial examples by analyzing “what happens” in-between each layer, thereby modeling it as an “anomaly detection” problem. The proposed method is grounded in solid theoretical analyses, and is then experimentally evaluated on neural networks trained over diverse benchmark datasets (all of images). The experiments are repeated multiple times, and the overall findings confirm the validity of the proposal, which outperform a reputable prior work (Lee et al. (NeurIPS 2018)). Furthermore, the paper also shows that by “tweaking” the proposed detection method, it is also possible to improve its generalization capabilities---i.e., detecting adversarial examples that do not conform to what the detector is expected to identify. Such a property is also found to be true via comprehensive experiments.
strength_and_weaknesses: STRENGTHS:
+ Strong theoretical analysis
+ Experiments on multiple datasets
+ Repeated trials that allow statistical tests
+ The “runtime” is evaluated, thereby allowing to gauge the overhead of the method
+ Good writing
+ Good presentation
+ Interesting (and novel) idea


WEAKNESSES
- The focus on “security” is not sound
- The evaluation (and the entire method) appears to be exclusively tailored for data in the form of “images”

clarity,_quality,_novelty_and_reproducibility: Overall, the presentation of the paper is very good.

The quality of the English text is fair.

Figures and Tables are of good quality.

The topic addressed by the manuscript is source of abundant literature, but still an open issue and relevant for ICLR.

The references are appropriate.

The contribution is significant (at least in the domain of Computer Vision).

The basic idea is intriguing and novel (to the best of my knowledge), and may potentially inspire future works.

The results are reproducible: the datasets are publicly available, and the manuscript provides a link to an (anonymous) repository containing the source-code.
summary_of_the_review: I liked the paper, a lot. I commend the authors for their work: despite not being “groundbreaking”, I consider this paper to be a solid contribution to the state-of-the-art, which deserves to be accepted to ICLR. Even by assuming a very critical eye, I was not able to find any significant flaw in the paper.

Something that I particularly appreciated is reporting the “runtime”: such metric is often forgotten in ML papers, but it is of high importance for practitioners as it allows to gauge the “cost” of a proposed method. Moreover, I also loved Section 5.2: here, the authors describe what I consider to be some “negative results”. I think more papers should include them. Props!

However, I believe that the paper is currently affected by a single issue---which stems from me being a researcher with a “security background,” and hence being critical of this aspect. Although I do not consider such issue to be ground for rejection, I truly invite the authors to fix it. Let me elaborate, and propose actionable means of remediations.

**The “attack” is misleading.**
The paper poses itself as tackling a “security-related” problem, but does not support such security focus in the appropriate way. Indeed, let me quote the first line of the abstract: “Adversarial attacks are perturbations to the input that don’t change its class for a human observer, but fool a neural network into changing its prediction.” The key term, here, is “attack”: from a security standpoint, an “attack” requires an “attacker”, who “does something (malicious) in order to achieve a given goal”. However, the paper does not provide any support for the “attacks” considered in the paper: this is due to the lack of a proper threat model---which is a lack that is becoming endemic in adversarial ML literature (see [A], which reports the opinions of researchers, policy makers and practitioners). To my understanding, the paper (and its contribution) does not deal with the “detection of adversarial attacks”, but rather with the “detection of adversarial examples”. Although the two topics have several similarities, they do not overlap: my take is that the paper addresses the more general problem of “adversarial examples,” which is useful to assess the “robustness” of ML methods and not their overall “security” (I quote the term “false sense of robustness” used in a recent NeurIPS paper [B]). 
To address this problem, I invite the authors to revise the paper by replacing “adversarial attacks” with “adversarial examples” whenever necessary. Alternatively, the authors can insert a proper “threat model”: this could be done by moving the current Section 4.2.2 into the Appendix. My stance is that the paper is being oversold as being (also) a "security paper": the authors should prove that I'm wrong, or -- if I'm not -- then they should fix their paper accordingly.

**Dubious utility of “detection”.** 
This is a direct consequence of the previous point. The goal of a “detection” mechanism (in the context of  “detecting an attack”) is to act as a “trap” for the attacker. In other words, the detector “expects the attacker to do A, and if they do so, then the attacker falls into the trap and is therefore detected”. However, the lack of a threat model describing what the attacker can/cannot do prevents from determining the utility of the detection mechanism as a security measure. Such a shortcoming further supports my previous observation of shifting the focus of the paper from “detecting adversarial attacks” to “detecting adversarial examples”. I point the authors to [C] for a good paper that truly models the problem of “detection of adversarial *attacks*”.
I acknowledge that the paper reproduces well-known white/black box attacks, but I still do not accept the lack of a threat model. A way to rectify the issue is by mentioning that "we evaluate our method by assuming adversarial examples found by means of the white/black box attacks proposed in (ref)".



**General or Specific?**
Something that is not fully clear to me is whether the proposed idea (which I found very enticing!) can be applied to “any” neural network, or only to residual nets. Let me explain. The paper proposes a method whose main intuition lies in the modeling the “depth” of a neural network as “time”. However, this leads me to think: would this idea “work” for those neural networks that are “time-aware”, such as recurrent neural networks (RNN) and LSTM? This is because, by following the paper’s idea, an input can go “back in time”. The paper does not take such architectures into account, so I am inclined to believe that it does not work. According to the introduction, the paper focuses on “residual networks” because they are “particularly amenable to this analysis”. The authors fairly state that “the analysis and implementation can extend immediately to any network where most layers have the same input and output dimensions.”, but they do not mention whether this also applies to RNN or LSTM; moreover, in Section 3.2, the definitions clearly state “Residual Networks”.
I hence ask the authors to clarify whether the proposed method can cover only “residual networks” (as well as any model that approximates such networks), or not. This should be clearly stated in the Introduction.

**Only Image data.**
This is self-explanatory: although the proposed methodology is (theoretically) applicable to any (residual?) neural network, the paper (and its evaluation) appears to be tailored for applications of neural networks that focus on image analysis. There are many domains in which neural networks are used today, and focusing on a single data-type prevents to assess the general applicability of the proposed technique. A potential fix can be, e.g., performing a proof of concept experiment on a dataset that does not include images---albeit this may be hard to find, since residual nets (which I believe to be “required” for the proposed method) are mostly useful for images.

Some additional issues:

•	I recommend to change the title: as a “security” researcher, I was misled into thinking that the paper dealt with adversarial attacks in the “network traffic analysis” domain (the term “transport” is a key component in networking). Specifically, I endorse the authors to replace the term “transport” with something else. Maybe “updates” is a better fit?

•	The first sentence in Section 5.1 should be better contextualized “First, the regularization improves generalizations.” Instead of "First", it should report "this result shows that..."

•	In Section 4: “use the transport regularization (6) to improve detectability of adversarial attacks.” It is not clear what “(6)” stands for.

•	I was originally taken aback by the “abstract”. In truth, I did not like the abstract at all. I think the authors tried to blend “technical” details with “simple” jargon, but the result (imho) is something that does not give true credit to the paper’s value. 

EXTERNAL REFERENCES

[A]: Apruzzese, G., Laskov, P., de Oca, E. M., Mallouli, W., Rapa, L. B., Grammatopoulos, A. V., & Franco, F. D. (2022). The Role of Machine Learning in Cybersecurity. Digital Threats: Research and Practice.

[B]: Pintor, M., Demetrio, L., Sotgiu, A., Manca, G., Demontis, A., Carlini, N., ... & Roli, F. (2022). Indicators of attack failure: Debugging and improving optimization of adversarial examples. NeurIPS 2022.

[C]: Shan, S., Wenger, E., Wang, B., Li, B., Zheng, H., & Zhao, B. Y. (2020, October). Gotta catch'em all: Using honeypots to catch adversarial attacks on neural networks. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (pp. 67-83
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Two reviewers were leaning towards reject, one reviewer has been very positive about the paper. The pros and cons were extensively discussed. 

The points raised by the reviewers were:
- unclear threat model
- no adaptive attacks
- stronger attacks (AutoAttack) not tested
- connection between theory and actual algorithm unclear

The reviewers liked the theoretical contribution regarding trajectory regularization to detect adversarial examples.

The authors have invested quite some time in the rebuttal to fix these issues which I appreciate. However, I think that there are still some open points left:

- no threat model: even in the updated version at the beginning of Section 2 no clear threat model is stated, the changes are norm-bounded, but the norm is not specified. It remains unclear if the authors claim robustness against any norm-bounded attack or l_p-attacks in general or just the $l_\infty$-attacks and $l_2$-attacks done in the experiments (even though the radius for the $l_2$-attacks is nowhere specified). Even if one just considers $l_\infty$-attacks: do the authors claim robustness against a specific radius or any $l_\infty$-attack? (see my comments below on unseen attacks)

-  no adaptive attacks: the paper has added some adaptive attacks but none of them is end-to-end (which is also difficult as there is no threat model). The authors use in the updated version a black-box/grey box setting but a complete white-box attack setting is needed (which is likely to break the detector as shown in many papers by Carlini et al) to check the limitation of the system followed by an end-to-end black box attack. Also the updated version does not state which attack is used (CW or AA?) and the budget of the attacker (e.g. iterations for PGD and queries for the black box attack) which makes it irreproducible and it remains unclear how much effort has been invested in breaking the detector. 

- unseen attacks: the authors should use attacks which try to search for minimal norm perturbations on the decision boundary as they are the most difficult attack for their detector setting. The current attacks are mainly finding points on the boundary of the threat model as they maximize the loss. In the updated version the AutoAttack examples seem more difficult to detect (which in my point of view shows that the detector does not work properly). As AutoAttack consists of four attacks and the Fast Adaptive Boundary (FAB) attack is one of them, which tries to find points on the decision boundary, this hints on that such attacks are much more difficult to detect for the detector. 


- no proper detection setting: the paper reports always "detection accuracy" but a useful detector in reality needs to pass most unattacked examples and only reject adversarial examples. Thus one uses typically FPR at a fixed TPR (often 95%, but other papers have even used 99%) as a measure (here positives corresponds to unperturbed examples). Reporting accuracy is useless to judge the utility of the detector. It was not clear what numbers were reported in the last answer of the authors  - it should be clearly defined in the paper. Reference to the code is not a replacement for a correct description in the text.

In total the paper in its current form is not ready for publication and requires a major revision.

Minor points:

 -  the accuracy of the trained networks is very low e.g. only 92.5% for CIFAR10, but there exists on RobustBench already an adversarially robust model with an accuracy of 92.2% and having a robust accuracy of 66.5% against $l_\infty$ of radius 8/255, or a model with 95.7% accuracy and having 82.3% robust accuracy against $l_2$-attacks of radius 0.5. These models could serve as a baseline even so they don't use a detector

- the model selection of the detector has to be clearly explained or better only one a fixed detector setting should be reported
justification_for_why_not_higher_score: The experiments are lacking in many aspects as explained above and the updated experiments for unseen AutoAttack suggest that the detector is worse than the initially reported experiments suggest. The field of detection of adversarial examples has seen many cases where published claims were shown later to be wrong (see papers by Carlini et al). Thus it is even more important to maintain high standards in the experiments.
Decision: Reject