Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors answered my questions so I am increasing my score to 7. 

-----

The paper presents a new defense for being robust to adversarial examples in NLP. This is a very exciting topic and I am glad to see more work in this space.

The paper presents a technique to make the model robust to word substitutions coming from a set S(u). The authors propose to use a convex hull which the authors claim is a better bound than using l2-ball or axis-aligned rectangles. 

The authors derive a optimization objective for their problem and show experimental results that shows their model achieves higher performance under two types of attacks (Genetic and PWWS) on the IMDB and SNLI datasets for various standard neural architectures. 


I have a couple of questions:

(1) Is this setting different than the one explored in Jia et al. 2019 (Certified Robustness to Adversarial Word Substitutions)?

It seems Jia et al. 2019 explores the case where multiple positions in the sentence can be perturbed whereas here only one word can be perturbed? I think clarifying this and discussing its implications would be useful for the reader.

(2) In training does the model have access to the set of all possible substitutions S(u) or not? 





review_title: Interesting idea but a few questions

final_score: rating: 7: Good paper, accept
review_content: Summary:
In this paper, the authors aim to build a robust model against word substitution attacks. Unlike previous work, they consider a convex hull as the perturbation region instead of a norm-ball or a hyper-rectangle. From their derivation, perturbed words can be viewed as the linear combinations of substitutions and perturbations can be viewed as the corresponding normalized weights. Therefore, they can adversarially train the perturbations and model to obtain a robust model and robust word embeddings. The authors also design a regularizer to encourage the sparsity on perturbation weights. The experimental results show that the proposed model is indeed more robust than other baselines. In addition, they show that the learned word embedding can be a good initialization for training robust models.

I very like the idea to convert the perturbations into the linear combination weights. I only have some minor questions:
- Can the proposed method be applied to more complex models, such as Transformer?
-Can the proposed technique be extended to optimize the certification bound (the lower bound of robust accuracy)?
- Maybe it is interesting to report the percentage of reduced perturbation region by using the convex hull rather than norm-ball or hyper-rectangle.
review_title: Interesting paper. I only have some questions.

final_score: rating: 7: Good paper, accept
review_content: *Summary of the paper*: This paper studies the problem of robustness against word substitutions. The authors propose a novel Adversarial Sparse Convex Combination (ASCC) method in which they model the word substitution attack space as a convex hull and leverages a regularization term to enforce perturbation towards an actual substitution. Based on the ASCC, they also propose ASCC-defense, which leverages ASCC to generate worst-case perturbations and incorporates adversarial training towards robustness. Experimental results show that their method outperforms the existing SOTA on two tasks -- sentiment analysis and natural language inference.

*Strength of the paper*:  
1. The idea proposed in the paper is quite straightforward yet effective. Using a convex hull could satisfy the three aspects as stated by the authors -- Inclusiveness, Exclusiveness, and Optimization. The experimental results do show the advantage of using the proposed convex hull.

2. Besides the robustness of the model, it also achieves robustness over word vectors.

3. The experiments are well designed including both qualitative analysis, quantitative results, and reasonable ablation to show the effectiveness of their method. In general, the paper is well-structured and easy to follow.

*Question for the authors*: 

1. I am curious about how much headroom there still exists for this task? Let's say if we have a perfect way to defend against such kind of attacks, how good are the current approaches?

2. What is the computational cost of the proposed method compared to others? In equation (3), the authors compute $w_{ij}$ through a softmax parameterized by $\hat{w_{ij}}$. If the substitution set of a word is infinite or too large, how are you going to deal with such kind of situations?

3. Seen from Table 1, the proposed method is much better than other models at PWWS attacks than the genetic attacks. Can you give an intuitive explanation of why? 

*Reason for score*: Overall, I vote for accepting this paper. I like the idea of using a convex hull and the way they regularize the model to achieve sparsity. It would be helpful if the authors could address the questions raised above.
review_title: simple idea and strong results

Meta-review: All three reviewers are positive, and the authors have addressed essentially all the questions raised by the reviewers. The main insight of the paper is clear, and the empirical results are good, so a spotlight is deserved.
Decision: Accept (Spotlight)