Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work discusses the high-stakes settings where an AI model should avoid every instance of catastrophic failure. The authors specifically evaluate their work on a safe language generation task. They propose a toolkit for training networks using human adversaries and adversarial training. The authors claim that their models achieve very low false negative rates (FNR) without hurting the network's output quality. Further, their +tool-assisted model almost doubled the time it takes for a human contractor to find an adversarial example. This suggests that their trained classifiers are more robust.
strengths_and_weaknesses: Strengths:
+ The paper aims toward building reliable AI systems that achieve better worst-case performance.
+ The authors provide detailed supplementary material describing the implementation details.
+ A novel toolkit is designed for adversarially training networks using human and automated adversaries.
+ A smart toolkit to help human contractors find adversarial examples using saliency mapping and token recommendation.
+ The final trained classifier achieves very low false negative rates $-$ 2/2447.
+ The time required by human contractors to find adversarial examples (for the final trained classifier) doubles. This suggests that the proposed adversarial training technique improves robustness.

Weaknesses:
- The problem statement highlights worst-case reliability (even a single catastrophic failure should not ideally occur). I think it's hard to make claims without any theoretical evidence. Also, I think the toolkit should be more rigorously analyzed to show more empirical evidence (eg, on image datasets).
- The toolkit relies on human adversaries; hence, scaling the framework for other tasks could be laborious.
- Though the proposed adversarial training technique makes it harder for a human contractor to find an adversarial example, it's still not impossible for a human adversary to attack the final trained classifiers.
- Figure 3 is not referred to in any part of the main text. FNR on snippets from the tool-assisted attack on the classifiers trained using the proposed technique is quite high. This doesn't satisfy the demands of the problem statement.
- In appendix A.3, the authors mention that the data from the validation set made it into the training set due to a bug in the tool. This affects hyperparameter selection. I would appreciate it if the authors provided results after fixing this bug.
- The proposed technique is not compared against other obvious baselines (eg, constrained Lp-norm based adversarial training).
- The supplementary material does not contain codes for checking the experiment implementations.

questions: 1. The authors claim that they allow examples in unrestricted space in the related works section. However, the human contractors are given recommendations for tokens from deberta-v2-xlarge. This restricts the adversaries from not creating smart examples using *hidden vocabulary* [1]. Does this mean that the adversaries are created in a restricted space?
2. Table 2 shows that the FNR for baseline and all the other trained classifiers are almost the same. How is the proposed method better than the baseline using this FNR metric?
3. The FNR is evaluated on a non-adversarial test dataset. How does this serve to be a good metric to evaluate catastrophic failure? Why is the FNR not evaluated on an adversarial test dataset?
4. How would the proposed method perform without the +manual step in the pipeline? Removing human involvement can improve the scalability of the pipeline. How does that affect the network robustness?


Reference:

[1] Daras, Giannis, and Alexandros G. Dimakis. "Discovering the Hidden Vocabulary of DALLE-2." arXiv preprint arXiv:2206.00169 (2022).
limitations: Yes, the paper discusses its limitations regarding stronger adversarial attacks, the generality of results, lack of theory, and a need for better metrics to measure reliability.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper authors propose to improve the binary classifier's performance by mining the hard minority class data (which authors call “adversarial examples”) by humans, a tool, humans augmented with the tool. The paper explains in detail how the whole data collection is done. The authors also train the model multiple times with new sets of adversarial examples found using the previous iteration of the model and show it gets harder to find hard examples automatically using the tool after a few iterations.
strengths_and_weaknesses: Strengths:
- Well written paper and easy to follow
- The authors describe in detail how they set up a system to mine the ambiguous negatives and use them to improve the robustness of the model.
- The investigations seem thorough.
- The use case of tools assisting humans is most interesting. I think that’s where the maximal benefit of the tool lies.


Weaknesses:
- The examples mined in this paper are not the typical adversarial examples we see in ML lingo, where we optimize a model to find an adversarial example. In the truest sense, the data the authors collect is “hard data” which the baseline model finds hard to classify.
- This is more of how they called data and training the model on ambiguous data improved the generalization of the model, which is not a surprising claim
- L266, Sec 5.4, there is previous work[1] suggesting that training models on hard/ambiguous data can improve the model’s generalizability.


[1] - Swayamdipta, Swabha, et al. "Dataset cartography: Mapping and diagnosing datasets with training dynamics." arXiv preprint arXiv:2009.10795 (2020).
questions: -  If I understand correctly, the model is trained on balanced data. Have you tried any imbalanced cases too? How was the performance there?
- The problem the authors are addressing in this paper is somewhat related long-tail class distribution problem, I wonder if this proposed methodology can help there. Have you explored any multi-class setting?
limitations: n/a
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors aim to train a text classifier to classify injurious sentences. Their motivation involves understanding how to make systems that be reliably robust to failures in high-stakes settings such as ones that involve human injury. They use several strategies. After training on a labeled dataset, they manually generate adversaries, manually generate adversaries using feature attributions as an informative tool, and augment their adversaries with paraphrasing. They show that these strategies are useful and that performance on non-adversarial inputs was preserved. 
strengths_and_weaknesses: I like how this work seems pointed toward reducing the risks from potentially highly intelligent systems that could make decisions themselves that directly affect human lives and welfare. While this isn’t a major concern right now, it might be later on. And even if I weren’t worried about the potential of future systems to potentially hurt people, this paper contains a number of tools and insights that seem valuable right now for training systems to have very low error.

I like how this paper uses interpretability tools to improve the process of generating adversaries for training. This proves the usefulness of the interpretability tools. They did a good deal of work to design and effectively use their human interface. And while this isn’t the most technical or mathy contribution, I don’t think that the value of designing an interface that is proven to help humans better work with these models should be understated. They don’t stress this too much, but I see a major part of the value of this paper as this human factors work. I generally think of feature attribution-based interpretability techniques to be bad because they don’t usually lead to useful, nontrivial, actionable insights about models. This paper doesn’t present itself as one on feature attribution-based interpretability, but they still do it very well. As someone who works in the area, I cannot think of any works involving feature attribution that have done something comparably useful. Perhaps this selling point of the paper could be discussed more. 

I see the implications that this paper has for making genuinely safer AI systems to be very compelling and direct compared to almost all other papers in the field. 

The paper is well written. 

I think my main issues with this paper involve things that they didn’t try that I think they could and arguably should have. I’ll discuss these below under Limitations. That said, I feel very confident that this paper should clearly be accepted.

Minor

- The overlapping CI’s for table 3 seem to warrant a “somewhat” in line 250.
- You did not invent paraphrasing augmentation. Cite the original paper(s).
- I think the main paper should at least briefly mention false positives and refer to the appendix. I don’t think mention of comparing false positive rates should be completely omitted in the main paper.
questions: None that I’ll put here, but I have some mentioned below under Limitations. 
limitations: There are more language data augmentation techniques than peraphrasing. Did the authors ever try any others? If so, mention this. If not, why not?

I mentioned above that I think feature attribution methods are overrated interpretability tools, that I struggle to think of other works that have used them for something genuinely useful, and that I am impressed by how this paper showed that they could be genuinely useful. But at the end of the day, I still believe that they are of limited usefulness, and that there are other interpretability techniques that have been used more successfully to identify adversarial weaknesses in networks. Having a human use feature attribution maps to generate adversaries is slow, relies on human creativity, and can, at best, only be used for local search. I would bet that synthesizing aversaries could be a better approach. See these three examples that come to mind for me (although the last one is from CV) [[https://arxiv.org/abs/2202.03286](https://arxiv.org/abs/2202.03286)] [[https://arxiv.org/abs/2005.00174](https://arxiv.org/abs/2005.00174) [[https://arxiv.org/abs/2110.03605](https://arxiv.org/abs/2110.03605)]. I also think that analyzing embeddings of examples to see which regions in embedding space could correspond to difficult examples may be very helpful for both interpreting and generating difficult examples. See [[https://arxiv.org/abs/2206.14754](https://arxiv.org/abs/2206.14754)] (although I understand that this paper only came out after the submission deadline). Finally, combining some sort of candidate adversary generation process with *weak* supervision from a human might make human oversight much more scalable. See [[https://arxiv.org/abs/2012.06046](https://arxiv.org/abs/2012.06046)].

So ultimately I think that the authors should think more about (1) different approaches for leveraging better interpretability tools, (2) synthesizing adversarial examples, and (3) better scaling the human oversight. At a minimum, I think that these should be discussed in future work. Arguably this paper is very simple because it adops some of the simplest and most brute force approaches possible for rigorous, high-stakes adversarial training, and I think that it will be important to expand on the simple methods presented here in future work. Perhaps its a good sign that this paper seem to suggest quite a bit more that could be done in the future to improve on things. I hope this gets accepted, and I encourage the authors to keep working in these 3 directions.
review_title: null

Meta-review: metareview: The paper used a safe language generation task (``avoid injuries'') as a testbed for achieving high reliability through adversarial training. Reviewers had a high variance in their evaluation of this work. While reviewers found the toolkit interesting and the paper well-written, there were some concerns regarding the lack of theoretical evidence on worst-case reliability and the lack of strong/adaptive attacks and baselines in experiments. Given all, I think the paper is above the accept threshold. 
confidence: Certain
Decision: Accept