Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: The paper is poorly written. This is especially the case for the
proposed algorithm (the core contribution). This section is very
difficult to understand, and notations are awkward. Everything is a
bit messy. However, the experimental results are quite well presented
(to be compared with the beginning of the paper).

review_title: The paper should be improved before submission 

final_score: rating: 3: Clear rejection
review_content: Summary:
This paper proposes WordsWorth score (WW score), a score to represent the importance of the word obtained from the trained model. Then, the score is applied to the greedy attack proposed by (Yang et al., 2018). In detail, the greedy attack first tries to search for the most important $k$ words in a text, and then it searches for values to replace the selected $k$ words. This paper uses the WW score to select the $k$ words in the first step.

Strong points
+ A simple but effective approach to utilize for the greedy attack

Concerns: 
- The main concern of this paper is that a minor contribution to the current knowledge. Despite the paper stating that this paper is based on the greedy attack (Yang et al., 2018), the contribution of this paper is limited to calculate the word score from the trained classifier and applied it to the greedy attack.

- Another concern about the paper is lack of rigorous experimentation to study the usefulness of the proposed method. This paper does not compare with other score-based approaches. That is, it was not even compared to the tf-idf based score approach.

- The writing should be largely improved. Section 4 is the main part of this paper. In Step 1, this paper represents a word as $d$-dimensional vector. Why does this paper append the zeros in front of the word representation? Does it mean the one-hot vector? If not, some studies or discussions about this representation should be included. In Step 2, Equations are hard to follow, and some are incorrectly written (e.g. case equation and definition of the D’).

- On the same note, readability and completeness of this paper do not meet the standard of conference. The reviewer suggests the authors to review the paper several times before submission.

- This paper states that $F$ is the trained classifier. However, there are no explanations on how to train or what kinds of classifiers were used.

- In Section 5.1.2, this paper states that it covers 5000 vocabularies freely after 25 reviews are processed because reviews are written in 200 words on average. It is definitely incorrect. Because all words in the review are not unique, it takes a longer timer to cache all words in the 5000 vocabularies. Although the proposed method can speed up by looking up the cached score, the performance of the proposed method is lower than the one of the original greedy approach. Some ablation studies or discussions about the relationship between speed and performance would have been useful to understand this. 

- Some parameters or data are heuristically selected such as selecting 10 nearest neighbors in step 2, picking 300 examples from test data in IMDB review experiments, and so on. Some form of ablation studies about the parameters would provide appropriateness to readers.

- Furthermore, it would be better to show examples of successful attacks with WW scores.

- In experiments, the AUC score is used for IMDB evaluation and the accuracy is used for Yelp and AG news. Are there any reasons to use different evaluation measures?

Minor comments: 
1. It would be better to write some constants such as $k$ in Section 3 as an italic character.
2. It will help readers if the authors can explicitly specify which side of the figure, left or right, is explained when the authors are referring to the figure in text.

Some typos: 
1. In Section 5.1.1, greedyww -> greedy\_ww
2. In Section 5.4.2, figure ?? -> Figure 1
3. In Section 5.4.1, $greedy_ww$ -> greedy\_ww
4. In Section 5.4.3, figure ??
5. In Section 6.1, CNN 4 -> CNN at Figure 4
...

review_title: This paper provides a simple approach, but this paper lacks completeness.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a new and simple way to determine word importance for black box adversarial attacks on text classification models. Instead of using example-specific measures of importance like recent work (typically expensive to compute), the authors propose to feed individual words from the vocabulary to a trained model and use the model confidences to get global, class-specific importance scores.

While being an interesting paper, at a high level I am concerned about several points:
- The paper is unpolished and at times hard to follow. I do not consider it ready for publication at this stage.
- There are many easy to implement baselines (feature selection has a rich history) that would have been very interesting to study. Are the WW scores capturing anything that simpler statistical methods do not?
- Many implementation details are lacking, which could be an issue for reproducibility. For example, the CNN model details are unclear (number of layers, filter sizes, embedding initialization, etc.). Learning rates and other hyperparameters are not mentioned.
- Why is the vocabulary size only 5000? Why are experiments run on only 500 examples, and are these examples selected?
- The claim of comparable performance seems slightly exaggerated, as the WW scores perform consistently worse than the baselines. In some cases, the difference seems around 0.1 / 0.15 absolute AUC. Also, providing raw scores (either all scores in Appendix or a subset of the most interesting case in the text) would help readers to quantify these differences.
- Some of the claims seem overly broad. For determining, say, grammaticality, I would expect greedy to vastly outperform greedy_ww.
- It is unclear to me how WW scores help with network interpretation. Again I would expect WW scores to correlate with a number of statistical correlation measures.

There are many formatting issues, with figure numbers, references, equations, etc.

review_title: An interesting idea but a paper clearly not ready for publication

Meta-review: The authors propose a method for attacking neural NLP models based on individual word importance ("WordsWorth" scores).  This is an interesting, timely topic and there may be some interesting ideas here, but at present the paper suffers from poor presentation which makes it difficult to discern the contribution. Presentation issues aside, it seems that the experimental setup is missing key baselines (an issue not sufficiently addressed by the author response). 
Decision: Reject