Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a detection-based defence strategy called TextShield against word-level text attacks, incorporating a detector and a corrector. Four saliency computation methods, vanilla gradient (VG), guided backpropagation (GBP), layerwise relevance propagation (LRP), and integration gradient (IG), are selected as sub-detectors. Adaptive word importance (AWI) is designed using the vanilla gradient method. Then a saliency-based corrector aims to correct adversarial sentences to benign ones based on AWI. Comprehensive experiments demonstrate that TextShield is superior to previous defence methods on both generalization and robustness.
strength_and_weaknesses: Strengths:

(1)	The illustration structure for intuitions and explanations for TextShield is stated clearly with the help of Figure 1-2. 

(2)	The experimental part has conducted comprehensive ablation study on considering the impacts of the hyperparameter, multiple sub-detectors and the number of adversarial examples.



Weaknesses:

(1)	Overclaim in Section 2 (Related Work): "However, because of the extreme time cost in the training stage, certified robustness is difficult to be applied to complex models. For example, IBP performed catastrophically on large pre-trained language models (e.g., BERT)". Certified robustness of recent work can be applied to complex models like BERT [1]. 

(2)	In Subsection 5.3 (Corrector): "For a suspect w_{i}, the corrector substitutes w_{i} with the most frequent word w^{'} ∈ S(w_{i}), where S(w_{i}) is w_{i} 's synonyms [4]". Here the synonym candidate set is selected from NLTK toolkit. However, the adoption of synonym sets is different in research on adversarial attacks, and the defence method assumes that the defenders are aware of the strategy for adversaries to create synonyms. Because we cannot limit a malicious attacker's use of synonym tables, this scenario is not realistic.

(3)	In the experimental part, TL, PWWS, GA and IGA are selected as word-level attacks for generating adversarial examples. However, some SoTA attacks like TextFooler [2], and Bert-Attack [3] are not considered as baselines to test the effectiveness of the proposed defence method.

(4)	The computation runtime and efficiency of the proposed defence strategy are not discussed in the experimental part.


Typos:

(1)	"Given sentence X with label y_{j} predicted by text classifier F, the adaptive word importance (AWI) of word w_{i} in the sentence, R_{ij}, …" in Subsection 4.2 lacks articles. Such a problem also appears in several places in this paper.

(2)	"Takings VG as an example" in Subsection 5.2 (Detector).


Reference:

[1] Ye, Mao, Chengyue Gong, and Qiang Liu. "SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

[2] Jin, Di, et al. "Is bert really robust? a strong baseline for natural language attack on text classification and entailment." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020.

[3] Li, Linyang, et al. "BERT-ATTACK: Adversarial Attack Against BERT Using BERT." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

[4] Loper, Edward, and Steven Bird. "Nltk: The natural language toolkit." arXiv preprint cs/0205028 (2002).

clarity,_quality,_novelty_and_reproducibility: Pls see Sec. Strength And Weaknesses

summary_of_the_review: Pls see Sec. Strength And Weaknesses
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method called TextShield to handle a problem setup within the field of adversarial attacks, where the goal is to first detect an adversarial sentence and then correctly classify it. The system detects adversarial sentences using a saliency-based methodology and subsequently corrects them to benign ones. TextShield surpasses existing approaches on multiple metrics in experiments on standard NLP datasets.
strength_and_weaknesses: Strengths

- The authors present strong motivation in Section 4 about the connection between saliency and robustness 
- The authors present a concise, clear description of adversarial text attacks and saliency computation.
- I like that TextShield has both a detector and a corrector for adversarial sentences which seems to be novel
- The proposed adaptive word importance (AWI) is interesting in that it is a simple, efficient method for identifying which words are the most important
-TextShield outperforms existing methods in adversarial detection metrics on popular datasets

Weaknesses
- Table 1 could be better presented if TS had (ours) in front of it to make it easy to distinguish between the proposed method and the rest. For the rest of the methods, it would be nice to have citations in front of them to make it easy to identify where they come from. 
- Multiple runs of the experiments in Table 1 are missing as the variance based on the random seed would give us a good idea about whether the reported results are significant. It is not clear whether TS significantly outperforms others.
- No code is provided to reproduce the results and get a deeper understanding of the control flow.
- While this work doesn't seem entirely novel as the methods are based on popular computer vision algorithms related to saliency: the authors used saliency at the word level compared to the pixel level used in computer vision, which is very similar. However,  I like this work as it provides a simple yet efficient way of achieving strong performance compared to existing methods for adversarial detection and correction.
clarity,_quality,_novelty_and_reproducibility: please see the Strength And Weaknesses section.
summary_of_the_review: please see the Strength And Weaknesses section. While the method is not entirely novel, the method TextShield is an easy, efficient way to detect and correct adversarial attacks while achieving competitive results for NLP type of datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a detection-correction paradigm to defend against word-level attacks on the text classification task.
Their adversarial sentence detector leverages adaptive word importance (AWI) based on saliency computation. If a sentence is detected as an adversarial one, the corrector will act on the sentence by refining the word with the highest saliency score. Then, the victim model takes the corrected sentence as input to make the final prediction. Thus, their model is like a shield in front of the victim classifiers.
strength_and_weaknesses: Strengths:
1. This paper is well written, with a clear description and exact formula.
2. This paper aims to not only successfully detect adversarial sentences but also correctly classify adversarial sentences that previous attacking work rarely pays attention to. In order to enhance the robustness of victim models, previous work generally mixes the detected adversarial sentences and original training data to retrain the victim model. This work provides a new way to explore how to get correct predictions of those adversarial samples.
3. The experimental results on adversarial sentences produced by four widely-used text attacks in the NLP field achieve comparable performance to previous defense methods.

Weakness:
1. Some attackers get adversarial sentences by adding an imperceptible perturbation into benign sentences. Ideally, the adversarial sentence's target remains the same as the benign one. But the perturbation process can not promise the unchangeable targets of the adversarial sentences. The automatic evaluation can only take the targets of the benign sentences as the true targets of the adversarial sentences, but maybe they are not. So it is better to add a human evaluation on the part of samples to reflect the precision of Table 1.
2. This method is somewhat opportunistic. By adding a "shield", the victim model could be more likely to make correct predictions of adversarial samples, but there still have been some concerns. The first one is the speed of inference. Calculating saliency should go through a forward and a backward process of the victim model. So, I wonder about the speed of the detector and the corrector. Another is that the detector is trained on the adversarial sentences generated by the GA, IGA, PWWS, and TextFooler. The reported final experiment results are also on the adversarial sentences generated by the GA, IGA, PWWS, and TextFooler. So I wonder if the adversarial sample is "out of domain" (for example, train detector and test the "shield" using two different "styles" attacker), the performance of the detector. Such a situation is more realistic because an attacker may generate adversarial samples with the same "style", which can not cover all potential adversarial sentences.
3. The experiments have compared the saliency detector with previous detectors by adding the designed corrector to previous detector-based methods, with the results demonstrating the corrector's efficiency. I'd like to know how much of a role the saliency played in the corrector. Specifically, how will the model behave if I do not choose the replaced word according to the saliency but something else, like some entities? It is better to have some case studies to explicitly show the saliency scores and their importance.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to understand. The authors have found a different way to defend against attacks. 
summary_of_the_review: This paper proposed an interesting way of defense by adding a "shield" of the victim model. But there are some concerns like the inference speed, the generalization of the detector and the corrector. I will reconsider this paper if the authors can give some explanation and evidence of these problems.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a saliency based detector TextShield to identify adversarial samples in NLP. They use existing methods (vanilla gradient, guided backpropagation, layerwise relevance propagation, and integration gradient) to obtain saliency maps and then combine the outputs of all the methods to produce a final prediction of whether a sample is adversarial. The authors also introduce a corrector that corrects that modifies adversarial samples such that they are correctly classified. 
strength_and_weaknesses: Strengths:
* The authors present a new method based on saliency maps. 
* On a few datasets and attacks, the authors show that they can successfully detect and defend against adversarial samples. 


Weaknesses:
* Some baseline comparisons are missing, for instance "Defense against Synonym Substitution-based Adversarial Attacks
via Dirichlet Neighborhood Ensemble" (Zhou et al., 2021) present DNE. From looking at the numbers in the DNE paper, it also seems like some cases performance of DNE is better. 
* The authors mention that one of the weaknesses for adversarial training is that the robustness of the model is determined by the diversity of adversarial examples used in adversarial training and that you cannot defend against unknown attacks. Isn't the same thing true for TextShield as well because the detector needs to be trained with adversarial examples? Does TextShield generalize well to unknown attacks or to attacks that haven't been used during the training of the TextShield detector?
* The authors state that "there remains a significant gap between successfully detecting adversarial sentences and correctly classifying adversarial sentences for a specific application." This claim is confusing because defense methods are specifically designed for this. 
clarity,_quality,_novelty_and_reproducibility: There are a few grammatical errors and the writing can be improved. 
summary_of_the_review: I'm not entirely convinced about the empirical evaluation because there are a few missing baselines.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The paper presents an approach to defending against word-level adversarial attacks on text classification tasks. The method first detects adversarial sentences based on a saliency-based detector. Then a corrector is proposed to rewrite and fix the adversarial samples.

Overall, all reviewers agree that the paper is on the borderline and suggest accepting the paper if there is space. Specifically, the ideas presented in the paper are interesting and the rebuttal clarifies the technical concerns. The writing is admissible, but the authors are strongly encouraged to improve it

Strength:  

+ Overall, the reviewers agree the proposed approach is novel and meets the bar of publishing at ICLR. 

+ The experiments are comprehensive and support the claims.

+ The rebuttal resolves all the technical concerns. 

Weakness: 

- Parts of the paper remain unclear or misleading after the rebuttal. For example, the design of the corrector has to be clarified. The motivation and intuition behind the design choices should be elaborated. Overall, the writing is admissible but the authors are strongly encouraged to improve it. Please refer to the writing suggestions in the reviews for details.

- The paper only considers text classification tasks and it’s unclear if the proposed approach can be extended to other NLP tasks with complex structures such as question answering. I understand the majority of the adversarial attack and defense approaches for NLP, especially those published in ML venues, only consider text classifications, and the authors follow the literature to conduct experiments on the selected datasets. However, NLP is beyond text classification. The authors should clarify this in the writing and I would suggest changing the title to “TextShield: Beyond Successfully Detecting Adversarial Sentences in Text Classification”

Missing references: 
Although the setup is different, the idea of identifying word-level adversarial attacks and patching the example for defense has been discussed in the following paper.
Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification, EMNLP 19




justification_for_why_not_higher_score: Besides the writing issue, the paper only considers attacking text classification models. Therefore, the scope of the study is narrow. 
justification_for_why_not_lower_score: There is no significant issue with the paper. The novelty of the proposed approach is moderate.
Decision: Accept: poster