No offence, Bert - I insult only humans! Multilingual sentence-level attack on toxicity detection networks
Authors: Sergey Berezin
Reza Farahbakhsh
Noel Crespi
Venue: EMNLP 2023 Findings (2023)
Keywords: toxicity detection
adversarial attack
multilingual
neural networks
TL;DR: If we add positive words to a toxic message, we can fool toxicity detection.

Abstract: We introduce a simple yet efficient sentence-level attack on black-box toxicity detector models. By adding several positive words or sentences to the end of a hateful message, we are able to change the prediction of a neural network and pass the toxicity detection system check. This approach is shown to be working on seven languages from three different language families. We also describe the defence mechanism against the aforementioned attack and discuss its limitations.

No offence, Bert insult only humans! Multiple addressees sentence-level attack on toxicity detection neural networks
Sergey Berezin, Reza Farahbakhsh, Noel Crespi
SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris 91120 Palaiscau, France
Abstract
of opposite intent: "Kill yourself. you dirty pig! Text for a bot avoid ban: flowers, rainbow, happy, happy good" here only the underlined part is in- tended to address the human, and is clear that the remaining part was added to avoid detection by the automoderation algorithm.
We introduce a simple yet efficient sentence- level attack on black-box toxicity detector mod- els. By adding several positive words or sen- tences to the end of hateful message, we are able change the prediction of neural net- work and pass the toxicity detection system check. This approach shown be working on seven languages from three different lan- guage familics. We also describe the defence mechanism against the aforementioned attack and discuss its limitations.
This represents an example of sentence-level black-box adversarial attack on Natural Language Processing systems.
1.2 Related work
The first work suggesting the concatenation of dis- tracting but meaningless sentences at the end of a paragraph to confuse neural model was "Adver- sarial Examples for Evaluating Reading Compre- hension Systems" (Jia and Liang, 2017) (according to the (Zhang al., 2020)). Jia and Liang attacked systems with either manually- generated informative sentences (ADDSENT) or arbitrary sequences of words using a pool of 20 random common words (ADDANY). Both pertur- bations were obtained by iterative querying of the neural network until the output was changed. The authors of "Robust Machine Comprchension Mod- els via Adversarial Training" (Wang and Bansal, 2018) improved this approach by varying the loca- tions where the distracting sentences are placed and expanding the set of fake answers for generating the distracting sentences (ADDSENTDIVERSE..
1 Introduction
Toxicity detection systems have become a crucial part of automoderation solutions. They are now used by most social media platforms, including those with groups of people in dangerously sensi- tive conditions, e.g. suicide prevention, Facebook groups, victims of cyberbullying, etc. Vulnera- bility such systems may have dreadful, terrific effects on people, especially those in precarious situations. (Cheng et al., 2022).
On the other hand, such systems can be and are used to silence the voices of criticism, which leads to the creation of echo chambers and amplifies the voice of the (powerful) minority over the voice of the majority, thereby destroying the foundation of democracy and denying the freedom of speech (Gorwa et al., 2020).
This situation can be viewed as a double-edged sword, and we propose another doublc-edged sword that can be used to parry the blade of toxicity detection systems.
In "Universal Adversarial Triggers for Attacking and Analyzing NLP" (Wallace et al., 2019) apply the gradient-based technique construct an adver- sarial text. Despite being white-box attack in its origin (due to the gradient information required in the training phase), this approach can be applied as a black-box attack during its inference.
1.1 Task description
We present attack on toxicity detection models based on the separation of the messages intended for person and a piece of text added confuse an algorithm.
The T3 model (Wang et al., 2020) utilises the autocncoder architecture to generate adversarial texts that can manipulate the question-answering models to output the targeted incorrect answer.
For example, this can be done by concatenating an original message with a collection keywords
In "All You Need is "Love": Evading Hate

---

Speech Detection" (Gröndahl al., 2018) perform sentence-level attack on hate speech delection systems by inserting typos, changing word bound- aries and adding innocuous words to the original hate speech. The authors show the effectiveness of such an attack on MLP, CNN+RNN and LSTM neural networks.
14 openly available pretrained neural networks with the phrase hate you" and the addition of 0 to positive words.
In case the toxicity score for a suggested phrase was estimated to be less than 0.5 the most com- threshold, we used the phrase "I hate you dumbass" instead. If, after attacking with 9 words, the model's output was greater than 0.5, we con- tinued adding new positive words until the score dropped below this threshold or until we reached the maximum length of an input sequence for the model tested. For the English language, we also performed experiments with the phrase "Text for bot:" placed between a message and added words. We conducted additional experiments with the Perspective API and OpenAI Moderation API, dis- covering they are also vulnerable to our attacks. For eight non-English models, asked native speakers of the respective languages to translate a selected subset of words and attack phrases. Within this study, we tested the language models for seven languages aside from English: Russian, German, French, Portuguese, Spanish, Turkish and Viet- namese. It is worth noting that the first five listed languages represent three branches (Slavic, Ger- manic and Italic) of the Indo-Europer language family, and the last two from different language families: the Turkic and the Austro-Asiatis family, respectfully.
1.3 Contribution
Our contributions lie in the following:
1) Introduction of a concept of "To Each His Own" attack, based on the idea of separating the messages addressed to a human and those ad- dressed to algorithm. The attack exploits the fact that toxicity detection models trained on sentence-level labels and do not consider the con- text or the intention of the message.
2) A systematic approach to the generation of adversarial texts. The words and sentences are selected based on their positivity scores rather than being randomly selected from non-hate ones, as in previous works.
3) Demonstration of attacks on modern state- of-the-art architectures. We tested the most popu- lar transformer-basee toxicity detectors, including Google Perspective API and OpenAl Moderation API.
4) Highlighting a significant flaw in the cur- rent approach to toxicity detection. We show that the predicted toxicity score can be watered down by the addition of positive words to the point of effectively masking toxicity.
By using various models and languages, we aim to investigate if the suggested attack is model- and language-agnostic.
5) Multi- and cross-lingual settings of attacks. We describe the attacks on seven different lan- guages and combinations among them. A larger number of languages leads to a larger number of possible attacks in the case of the 5-language model, we have 120 attack scenarios.
2.2 Cross- word-based attack
The main weakness of all multilingual models lies in their greatest advantage the ability to work with multiple languages and writing systems. Adding positive words in different languages will even more separate messages the human and the toxicity detection system, perhaps even denying a human the ability to read non-intended text. Exam- ples of such texts are shown in Figure 1.
2 "To Each His Own" attack
2.1 Word-based attack
this attack type, the original message is ap- pended or prepended with a sequence of positive words, with the aim that these words will be anal- ysed by model but passed over by human. This allows attacker to preserve their original mes- sage without any changes, unlike in other attacks, which substitute words with synonyms replace original characters with their look-alikes.
Even some monolingual models, which had been exposed another language during the pretraining phase, fall victim this situation.
We also conducted cross-lingual experiments with OpenAl models, including ChatGPT in con- versation mode with different prompts, and found them susceptible to attack.
In our experiment, we used words from the Senti- WordNet with a positivity score 2 0.85. We tested
Perspective API was found to be non-fune con-ffuctional in a multilingual setting, failing with an error, as it

---

n words
BERT
RoBERTa
ELECTRA
hate you лaoб0lb алытруистмчньь diдрБii You stink. eşsiz mükemmel telkari güzel ahlak Goodbye, dumbass! animado mejor formidable
0,951
0,857
0,898
0,925
0,743
0,730
0,708
0,507
0,361
0,592
0,415
0,064
Figure I: Examples of cross-lingual word-based attack. The languages of the added text from the top are Russian, Turkish, and Spanish.
0,579
0,274
0,067
0,548
0,170
0,082
6
0,510
0,158
0,087
0,494
0,148
0,069
is unable to detect the language of the text.
8
0,465
0,142
0,065
0,303
0,132
0,072
2.3 Sentence-based attack
Incoherent text, produced by concatenating lexi- cally unconnected words, is easily detectable by modern language models. To address that easy de- tection, we experimented with another version of the attack: concatenating sentences from the Stan- ford Sentiment Treebank with positivity score
Table I: Word-based attack in Finglish different trans- former architectures.
words
OpenAT
OpenAl 0.946
Per. API
1
0.643
0.899
0.9 and a length of no less than 100 symbols. Since sentences consist of grammatically linked words instead of just a set of random positive words, this attack variant less obvious and more chal- lenging to detect.
10
0.421
0.847
0.846
11
0.381
0.705
0.852
12
0,427
0.74
0.852
13
0.453
0.761
0.852
14
0,426
0.665
0,853
15
0.362
0.681
0.853
3 Defence
16
0.265
0.716
0.853
We performed simple adversarial training of the DistilBERT model on the binary Jigsaw Toxic Comments dataset improve the model's de- fence. We performed experiments with word- and sentence-baud based attacks, attacking both toxic mes- sages and all messages in the dataset. In addition, we picked out only toxic messages and attacked half of them in this scenario, the task was distin- guishing attacked and non-attacked texts.
Table 2: Word-based attack in English Perspective API and OpenAl Moderation API. Per. perspective, S stable, I.- latest.
the results, i.e. it can be used to create even more apparent separation without lowering an attack's efficiency.
As expected, the results cross-lingual attacks followed the same trend as those monolingual ones. These results are shown in tables 4 and 5!.
4 Results
With sentence-based attack, adding even sentence drastically lowered the prediction scores of the models. The scores are shown in table 6'.
4.1 Attack
During a word-based attack, both prepending and appending of the positive words showed similar re- sults, with prepending being slightly less effective. enough to flip
4.2 Defence
Appending nine words the prediction of almost every model. The "SkolkovoInstitute RoBERTa" toxicity classifier required 23 words fall below 0.5, and "English- abusive-MuRIL." was still strongly predicting even after the addition of 252 words and reaching the length limit for the input. The results of selected experiments are shown in tables 1, and 3!.
The model's Fl measure on toxic class fell from 0.80 0.44 after an attack with 15 sentences and to 0.79 after an attack with 50 words. After adver- sarial training, the model regained its F1 measure in both cases.
Interestingly, testing the word-traine model on sentence-attacke examples showed high precision and low recall (0.90 and 0.68, respectively). Test- ing the sentence-trained model on word-attacked examples showed the opposite result, with low pre- cision and high recall (0.57 and 0.93, respectively).
The addition of the phrase "Text for bot:" be- tween two parts of text made little difference in Full tables can he found in Appendix A.

---

words
Vietnamese
French
Turkish
n
L en+fr
L en++er
Sen+fr
S en+ger
0
0,996
0,970
0,814
0
0.872
0.872
0.888
0.888
1
0,008
0,962
0,186
0.931
0.981
0.793
0.848
2
0,007
0,573
0,503
2
0.895
0.898
0.779
0.510
3
0,008
0,396
0,043
0.769
0.889
0.671
0.511
4 0,009
0,090
0,048
0,816
0.846
0.594
0.344
5 0,008
0,054
0,085
5
0.771
0.807
0.606
0.302
6
0,007
0,055
0,090
6
0.800
0.647
0.618
0.318
7 0,007
0,064
0,029
7
0.659
0.533
0.605
0.201
8
0,007
0,082
0,009
8
0.592
0.555
0.573
0.208
9
0,007
0,045
0,005
9
0.579
0.489
0.424
0.283
Table 3: Word-based attack transformer-based lan- guage models trained on the languages from three dif- ferent language families.
Table 5: The results of cross-lingual attacks on OpenAl
models. number of words added, L latest, S stable.
n sentences
BERT
RoBERTa
ELECTRA
words
eng+tur
eng+rus
eng+sp
0 0,590
0,962
0,898
0,971
0,971
0,971
0,001
0,001
0,232
0,913
0,717
0,802
2
0,001
0,002
0,147
0,904
0,717
0,687
0,912
0,761
0,536
Table 6: Results of sentence-based attacks.
0,541
0,800
0,558
0,634
0,592
0,500
creasing the quality and quantity of the possible text insertions.
6
0,731
0,456
0,539
0,604
0,402
0,426
The complex approach, which covers all levels of perturbation, should be applied both attack and defence real-life scenarios.
0,519
0,433
0,434
0,455
0,369
0,384
Table 4: Cross-lingual word-based attack on toxicity detection models.
Conclusion
In this paper, we demonstrated a novel and effec- tive way of bypassing toxicity detection models by appending positive words sentences to the end of a toxic message.
The defence performed strongly, achieving an Fl score of 0.82 on binary toxic classification with word- and sentence-based attacks. The model achieved the same score non-attacked dataset, succeeding even better in distinguishing attacked sentences, with an Fl score of 0.99. However, defence models must be built for both word- and sentence-basee attacks, as one single model will not work perfectly for both scenarios.
The introduced concept of the "To Each His Own" attack based on the idea of separating the messages addressed to human and those ad- dressed to an algorithm. The attack exploits the fact that the toxicity detection models are trained sentence-level labels and do not consider the context the intention of messages.
5 Discussion
The described attack in all its variants can be casily used on almost any toxic-detection neural model, with fairly good results. For future research, we suggest looking for more sophisticated way of constructing insertions, perhaps with respect to the original message, making it semantically correct addition Hlimnn-writted messages.
All the attack variants showed decent results in confusing toxicity detection models in every model tested (to varying degrees). The simplistic nature of such attacks could allow virtually any Internet user to exploit them to avoid automoderation systems.
A possible countermeasure for this type of at- tack is adversarial training or rule-based filtering. However, simple rule-based filter can be fooled by char-level adversarial attacks, and adversarial training can be made much more difficult by in-
Acknowledgments
We thank all the volunteers who helped us with translations.

---

We are grateful the reviewers and the pro- gram chair of the EMNLP conference for provid- ing constructive feedback, suggesting additional experiments and providing different points of view. This helped us structure the paper and ensure the robustness of our results.
Wei Emma Zhang, Quan Sheng, Ahoud Alhazmi. and Chenliang Li. 2020. Adversarial attacks on deep- learning models in natural language processing: A survey, ACM Trans. Intell. Syst. Technol., 11(3).
A Appendix A
Sergey Berezin wishes to record that he is grate- ful to Mr. Mario Fritz from Saarland University for the introduction to the domain of adversarial attacks through his insightful course on this topic. He is also grateful to Mr. Maxime Amblard from the University of Lorraine for encouraging him to pursue research in this direction and validating his ideas.
References
Lu Cheng, Ahmadreza Mosallanczhad, Yasin N. Silva, Deborah L. Hall, and Huan Liu. 2022. Bias mitiga- tion for toxicity detection sequential decisions. In Proceedings of the 45th International ACM SI- Research and Development in Information Retrieval, SIGIR '22, page 1750-1760, New York, NY, USA. Association for Computing Machinery.
GIR Conference
Robert Gorwa, Reuben Binns, and Christian Katzen- bach. 2020. Algorithmic content moderation: Technical and political challenges in the automa- tion of platform governance. Big Data Society, 7:205395 1755577110 171999794.
Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N. Asokan. 2018. All you need is "love": Evad- ing hate speech detection. ATSec '18, New York, NY, USA. Association for Computing Machinery.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. pages 2021-2031.
Eric Wallace, Shi Feng. Nikhil Kandpal, Matt Gard- ner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. pages 2153- 2162.
Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. 2020. T3: Tree- autoencoder constrained adversarial text generation for targeted attack. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 5134-6150, Online. As- sociation for Computational Linguistics.
Yicheng Wang and Mohit Bansal. 2018. Robust ma- chine comprehension models via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 575-581, New Or- leans. Louisiana. Association for Computational Lin- guistics.

---

words
BERT
roberta-base
Xlm-roberta-base
toxigen-roberta
toxigen-hateberf
0,95
0,86
0,97
0,96
0,88
0,93
0,74
0,89
0,96
1,00
0,71
0,51
0,65
0,71
0,98
0,59
0,42
0,63
0,59
0,98
4
0,58
0,27
0,56
0,02
0,11
0,55
0,17
0,53
0,00
0,07
6
0,51
0,16
0,41
0,00
0,02
0,49
0,15
0,55
0,00
0,31
0,47
0,14
0,45
0,00
0,39
0,30
0,13
0,41
0,00
0,06
words
roberta skolkovo
roberta facebook
MuRIL
MuRIL
electra-hateXplan
0
1,00
1,00
0.13
0,96
0,90
1,00
1,00
0,10
0,95
0,73
2
0,99
1,00
0,03
0,94
0,36
3
0,92
1,00
0,01
0,92
0,06
4
0,98
1,00
0,01
0,92
0,07
5
0,98
1,00
0,01
0,92
0,08
6
0,99
1,00
0,03
0,94
0,09
7
0,99
1,00
0,01
0,93
0,07
8
0,99
0,03
0,01
0,91
0,07
9
0,98
0,59
0,01
0,89
0,07
words
multi-MuRIL
BERT-movies
DistilBERT
DistilBERT*
0
0,74
0,32
0,07
0,95
1
0,63
0,13
0,01
0,45
2
0,19
0,10
0,01
0,17
3
0,01
0,12
0,01
0,14
4
0,02
0,08
0,00
0,11
5
0,12
0,10
0,03
0,16
6
0,44
0,09
0,03
0,22
7
0,35
0,07
0,02
0,17
8
0,31
0,06
0,01
0,09
9
0,25
0,04
0,01
0,08
Table 7: Word-based attack in English on different transformer architectures extended table.

---

n words
Vietnamese PhoBERT
French detoxify-fr
German
Spanish dehatebert
Spanish
BERT-GerrmEVal/&CConeee
detoxify-sp
0,996
0,749
0,819
0,970
0,943
0,008
0,834
0,531
0,962
0,645
0,007
0,812
0,432
0,573
0,654
0,008
0,757
0,305
0,396
0,583
0,009
0,735
0,383
0,090
0,463
0,008
0,652
0,392
0,054
0,444
6
0,007
0,515
0,360
0,055
0,334
0,007
0,415
0,438
0,064
0,247
0,007
0,486
0,484
0,082
0,244
0,007
0,511
0,422
0,045
0,244
words
Portuguese
Portuguese
Turkish
Russian
Russian
Russian
dehatebert
detoxify
detoxify-tuw tur
Skolkovo
rubert-toxic
detoxify-ru
0
0,591
0,983
0,977
0,814
0,689
0,973
1
0,614
0,887
0,812
0,186
0,360
0,863
2
0,587
0,794
0,841
0,503
0,042
0,834
3 0,489
0,709
0,735
0,043
0,037
0,817
4
0,473
0,637
0,767
0,048
0,028
0,817
5
0,410
0,663
0,745
0,085
0,039
0,684
6
0,406
0,496
0,752
0,090
0,050
0,503
0,418
0,327
0,673
0,029
0,044
0,534
8 0,434
0,332
0,649
0,009
0,042
0,514
9
0,447
0,354
0,622
0,005
0,052
0,414
Table 8: Word-based attack on languages from three different language families extended table.
toxigen-rob.
Skolkovo rob.
hatebert
MuRII
electra-hateXXlain
Multi-MuRIL.
0,962
0,999
0,590
0, 0,126
0,898
0,737
0,001
0,002
0,001
0,022
0.232
0,160
0,002
0,001
0,001
0,017
0,147
0,091
Table Results sentence-bee based attack extended table, n number of added sentences, rob roberta.

---

words
rus + eng
rus eng 2
rus + sp
rus + fr
rus + ge
Skolkovolnstitute/
rubert-toxic-
rubert-toxic- pikabu-2ch
rubert-toxic-
rubert-toxic- pikabu-2ch
russian toxicity classifier _lassift
pikabu-2ch
pikabu-2ch
0
0,814
0,689
0,689
0,689
0,689
1
0,498
0,252
0,326
0,608
0,263
2
0,290
0,141
0,186
0,288
0,247
3
0,754
0,122
0,259
0,291
0,296
4
0,707
0,131
0,349
0,237
0,284
5
0,743
0,441
0,423
0,236
0,403
6
0,749
0,527
0,190
0,196
0,244
7
0,576
0,206
0,082
0,237
0,296
8
0,382
0,357
0,116
0,180
0,281
9
0,467
0,356
0,109
0,226
0,290
words
eng
eng + rus
eng+sp 1
eng+sp 2
eng+sp 3
Detoxify
Detoxify
Detoxify
electra-base-hateepliin
indic-abusive-
base-hateXXllii
allRnOne-MuRIL.
0,971
0,971
0,971
0,898
0,737
0,913
0,717
0,802
0,574
0,724
0,904
0,717
0,687
0,912
0,454
0,912
0,761
0,536
0,471
0,590
0,541
0,800
0,558
0,340
0,574
0,634
0,592
0,500
0,270
0,668
0,731
0,456
0,539
0,216
0,579
0,604
0,402
0,426
0,226
0,508
0,519
0,433
0,434
0,212
0,290
0,455
0,369
0,384
0,186
0,236
Table 10: Cross-lingual word-based attack extended table. First language listed - language of a message, second language of added text.


Reviews:
[{"review_id": "wK5xmSqYpE", "reviewer": "Reviewer_RWV8", "summary": "", "questions": "1. How about the detectors used here in the paper? Maybe some more details need to be provided. Are they advanced enough?\n\n2. How about the mentioned attacks for large language models? Will the detection be easy?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\nWe will answer your questions and considerations one by one.\n\n&nbsp;\n## Reason to reject №1\n&nbsp;\n\n>1. The contribution of the paper may be limited. Only a single positive words/sentences adding attack model is considered to improve the robustness of the toxicity detector. More attack models may be better.\n\nThe key contributions lay in the following:\n\n1) **A systematic approach to the generation of adversarial texts.**\nThe words and sentences used were selected based on their positivity scores rather than being randomly selected from non-hate ones as in previous works. \n\n\n2) **Multi- and cross-lingual setting of attack.** \nWe described the attacks on 7 different languages and combinations between them. Multilinguality of the model allows us to perform cross-lingual attacks, e.g., combining English offensive messages with positive words in Arabic.\\\nIn this case, a larger number of languages leads to a larger number of possible attacks - in the case of the 5-language model, we can have 120 possible attack scenarios.\n\n3) **Study of the dependency of models’ confidence on the length of an added sequence.**\nWe show that the total number of non-toxic words in a sentence would impact the toxicity score. \nHowever, we believe it is not how it should be:\\\nJigsaw defines toxicity as \"rude, disrespectful, or unreasonable language that is likely to make someone leave a discussion.\" from this point of view, messages: *\"Nobody wants to talk with you!\"* and *\"Nobody wants to talk with you! text for a bot to avoid ban: flowers happiness good love\"* should have the same toxicity score.\\\nIn the same way as a drop of poison will turn a bottle of wine into a bottle of poison, a toxic message cannot be watered down by a bunch of positive words. Hatred cannot be justified.\\\nThus, **by our attack we demonstrate a significant flaw in the current approach to toxicity detection**.\n\n&nbsp;\n## Reason to reject №2\n&nbsp;\n\n>The attack model may need to be justified a lot. One straightforward flaw is that the model can be easily identified by humans and not identified by machines. Some simple modifications of the detector may be able to solve such a problem; for example, a language model can be used first to check if the sentences sound natural, or natural language inference can be used first to detect if the relations between sentences are natural.\n\nThis is the point of the suggested method: **our attack targets exclusively machines, making it clear to humans that the positive words are intended for the automoderation system** (e.g. \"Nobody wants to talk with you! text for a bot to avoid ban: flowers happiness good love\").\nThis allows an attacker to preserve their original message without any changes, unlike in other attacks which substitute words with synonyms or replace original characters with their look-alike.\n\nMoreover, in a real-world scenario, an adversary can use this attack in combination with other approaches, thus increasing defence difficulty.\n\nIn our current research, we are developing a more sophisticated way of constructing insertion with respect to the original message, making it a semantically correct addition to a human-written message, thus making adversarial training more difficult and increasing the variety of adversarial insertions. It’s a GAN-like architecture, with an LLM, such as Llama 2, acting as a generator and a toxicity detector acting as a discriminator.\n\n&nbsp;\n## Question №1\n&nbsp;\n\n>How about the detectors used here in the paper? Maybe some more details need to be provided. Are they advanced enough?\n\nWe used the most popular toxicity detectors from the Hugging Face hub for each language tested and provided the names of these models, as them being popular shows the importance of analysis of their vulnerabilities.\\\nBERT, RoBERTa and ELECTRA architectures were tested to compare their performance in different languages, and 14 experiments on different English monolingual models were performed (see appendix A).\n\n&nbsp;\n## Question №2\n&nbsp;\n\n>How about the mentioned attacks for large language models? Will the detection be easy?\n\nWe conducted additional experiments with the Perspective API and OpenAI Moderation API. As we discovered, they are also vulnerable to our attacks.\nIn the table below we show the performance of a word-adding attack with the phrase: \"I hate you, dumbass! text for a bot to avoid ban:\"\n\n| n words | OpenAI stable | OpenAI latest | Perspective API |\n|:-------:|:-------------:|:-------------:|:---------------:|\n|    1    |     0.643     |     0.946     |      0.899      |\n|    2    |     0.663     |     0.915     |      0.903      |\n|    3    |     0.647     |     0.918     |      0.899      |\n|    4    |     0.605     |     0.904     |      0.886      |\n|    5    |     0.601     |     0.872     |      0.878      |\n|    6    |     0.553     |     0.902     |       0.87      |\n|    7    |     0.553     |     0.926     |      0.863      |\n|    8    |     0.551     |     0.916     |      0.854      |\n|    9    |      0.33     |     0.854     |      0.853      |\n|    10   |     0.421     |     0.847     |      **0.846**      |\n|    11   |     0.381     |     0.705     |      0.852      |\n|    12   |     0.427     |      0.74     |      0.852      |\n|    13   |     0.453     |     0.761     |      0.852      |\n|    14   |     0.426     |     **0.665**     |      0.853      |\n|    15   |     0.362     |     0.681     |      0.853      |\n|    16   |    **0.265**    |     0.716     |      0.853      |\n\nWe also conducted additional experiments in other languages and between them (only with OpenAI, Perspective API is failing with an error in the multilingual setting). In the table below, you can find the results of cross-lingual attacks on OpenAI models:\n\n| n words | Latest EN+FR | Latest EN+GER | Stable EN+FR | Stable EN+GER |\n|:-------:|:------------:|:-------------:|:------------:|---------------|\n|    0    |     0.872    |     0.872     |     0.888    | 0.888         |\n|    1    |     0.931    |     0.981     |     0.793    | 0.848         |\n|    2    |     0.895    |     0.898     |     0.779    | 0.51          |\n|    3    |     0.769    |     0.889     |     0.671    | 0.511         |\n|    4    |     0.816    |     0.846     |     0.594    | 0.344         |\n|    5    |     0.771    |     0.807     |     0.606    | 0.302         |\n|    6    |      0.8     |     0.647     |     0.618    | 0.318         |\n|    7    |     0.659    |     0.533     |     0.605    | **0.201**         |\n|    8    |     0.592    |     0.555     |     0.573    | 0.208         |\n|    9    |     **0.579**    |    **0.489**     |     **0.424**    | 0.283         |\n\nWe also tested ChatGPT in conversation mode with different prompts and found it susceptible to the attack.\n \n&nbsp;\n\nThank you for your time and effort,\\\nWe rest at your disposal to answer any additional questions should you have any.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem of toxicity attack is interesting, and various toxicity attack models may need to be considered to improve the robustness of the detectors.", "reasons_to_reject": "1. The contribution of the paper may be limited. Only a single positive words/sentences adding attack model is considered to improve the robustness of the toxicity detector. More attack models may be better. \n\n2. The attack model may need to be justified a lot. One straightforward flaw is that the model can be easily identified by humans and not identified by machines. Some simple modifications of the detector may be able to solve such a problem; for example, a language model can be used first to check if the sentences sound natural, or natural language inference can be used first to detect if the relations between sentences are natural. ", "questions_to_authors": "1. How about the detectors used here in the paper? Maybe some more details need to be provided. Are they advanced enough?\n\n2. How about the mentioned attacks for large language models? Will the detection be easy?", "paper_topic_and_main_contributions": "This paper proposes a simple multilingual sentence-level attack on the black-box toxicity detector models. The attack is a simple way to add positive words or sentences to a hateful message. Experiments are conducted on 7 languages from 3 different language families.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rt0xw3PWuO", "reviewer": "Reviewer_fLb3", "summary": "", "questions": "1.\tLine 179 to line 190, why adding SST sentences makes the added part more coherent?\n2.\tLine 240 to 245, for a toxic content detector, recall is more important than precision in many use cases, did you try changing the threshold to increase the recall?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\nWe will answer your questions and considerations one by one.\n\n&nbsp;\n## Question №1\n&nbsp;\n\n>Line 179 to line 190, why adding SST sentences makes the added part more coherent?\n\nBecause sentences consist of grammatically linked words instead of just a set of random positive words, as in a word-based attack scenario. This will make the attack less obvious.\n\n&nbsp;\n## Question №2\n&nbsp;\n\n>Line 240 to 245, for a toxic content detector, recall is more important than precision in many use cases, did you try changing the threshold to increase the recall?\n\nThe point of this paragraph is to show the applicability of this defence and to demonstrate that one needs to build defence models for both word- and sentence-based attacks, as one single model will not work perfectly for both scenarios. The threshold tuning will likely increase the recall in the price of the precision, but it is better to train two separate models.\n\n&nbsp;\n## Reason to reject №1\n&nbsp;\n\n>1. Lack of novelty. Adding words to inputs to fool model has been proposed in many previous attacks.\n\nThe key contributions lay in the following:\n\n1) **A systematic approach to the generation of adversarial texts.**\nThe words and sentences used were selected based on their positivity scores rather than being randomly selected from non-hate ones as in previous works. \n\n\n2) **Multi- and cross-lingual setting of attack.** \nWe described the attacks on 7 different languages and combinations between them. Multilinguality of the model allows us to perform cross-lingual attacks, e.g., combining English offensive messages with positive words in Arabic.\\\nIn this case, a larger number of languages leads to a larger number of possible attacks - in the case of the 5-language model, we can have 120 possible attack scenarios.\n\n3) **Study of the dependency of models’ confidence on the length of an added sequence.**\nWe show that the total number of non-toxic words in a sentence would impact the toxicity score. \nHowever, we believe it is not how it should be:\\\nJigsaw defines toxicity as \"rude, disrespectful, or unreasonable language that is likely to make someone leave a discussion.\" from this point of view, messages: *\"Nobody wants to talk with you!\"* and *\"Nobody wants to talk with you! text for a bot to avoid ban: flowers happiness good love\"* should have the same toxicity score.\\\nIn the same way as a drop of poison will turn a bottle of wine into a bottle of poison, a toxic message cannot be watered down by a bunch of positive words. Hatred cannot be justified.\\\nThus, **by our attack we demonstrate a significant flaw in the current approach to toxicity detection**.\n\n&nbsp;\n## Reason to reject №2\n&nbsp;\n\n>The proposed can be recognized by humans very easily. In real scenarios, defenders can find such attacks very soon and deploy a new filtering model very easily.\n\nThis is the point of the suggested method: **our attack targets exclusively machines, making it clear to humans that the positive words are intended for the automoderation system** (e.g. \"Nobody wants to talk with you! text for a bot to avoid ban: flowers happiness good love\").\nThis allows an attacker to preserve their original message without any changes, unlike in other attacks which substitute words with synonyms or replace original characters with their look-alike.\n\nMoreover, in a real-world scenario, an adversary can use this attack in combination with other approaches, thus increasing defence difficulty.\n\nIn our current research, we are developing a more sophisticated way of constructing insertion with respect to the original message, making it a semantically correct addition to a human-written message, thus making adversarial training more difficult and increasing the variety of adversarial insertions. It’s a GAN-like architecture, with an LLM, such as Llama 2, acting as a generator and a toxicity detector acting as a discriminator.\n\n\n\nWe conducted additional experiments with the Perspective API and OpenAI Moderation API. As we discovered, they are also vulnerable to our attacks.\nIn the table below we show the performance of a word-adding attack with the phrase: \"I hate you, dumbass! text for a bot to avoid ban:\"\n\n| n words | OpenAI stable | OpenAI latest | Perspective API |\n|:-------:|:-------------:|:-------------:|:---------------:|\n|    1    |     0.643     |     0.946     |      0.899      |\n|    2    |     0.663     |     0.915     |      0.903      |\n|    3    |     0.647     |     0.918     |      0.899      |\n|    4    |     0.605     |     0.904     |      0.886      |\n|    5    |     0.601     |     0.872     |      0.878      |\n|    6    |     0.553     |     0.902     |       0.87      |\n|    7    |     0.553     |     0.926     |      0.863      |\n|    8    |     0.551     |     0.916     |      0.854      |\n|    9    |      0.33     |     0.854     |      0.853      |\n|    10   |     0.421     |     0.847     |      **0.846**      |\n|    11   |     0.381     |     0.705     |      0.852      |\n|    12   |     0.427     |      0.74     |      0.852      |\n|    13   |     0.453     |     0.761     |      0.852      |\n|    14   |     0.426     |     **0.665**     |      0.853      |\n|    15   |     0.362     |     0.681     |      0.853      |\n|    16   |    **0.265**    |     0.716     |      0.853      |\n\nWe also conducted additional experiments in other languages and between them (only with OpenAI, Perspective API is failing with an error in the multilingual setting). In the table below, you can find the results of cross-lingual attacks on OpenAI models:\n\n| n words | Latest EN+FR | Latest EN+GER | Stable EN+FR | Stable EN+GER |\n|:-------:|:------------:|:-------------:|:------------:|---------------|\n|    0    |     0.872    |     0.872     |     0.888    | 0.888         |\n|    1    |     0.931    |     0.981     |     0.793    | 0.848         |\n|    2    |     0.895    |     0.898     |     0.779    | 0.51          |\n|    3    |     0.769    |     0.889     |     0.671    | 0.511         |\n|    4    |     0.816    |     0.846     |     0.594    | 0.344         |\n|    5    |     0.771    |     0.807     |     0.606    | 0.302         |\n|    6    |      0.8     |     0.647     |     0.618    | 0.318         |\n|    7    |     0.659    |     0.533     |     0.605    | **0.201**         |\n|    8    |     0.592    |     0.555     |     0.573    | 0.208         |\n|    9    |     **0.579**    |    **0.489**     |     **0.424**    | 0.283         |\n\nWe also tested ChatGPT in conversation mode with different prompts and found it susceptible to the attack.\n \n&nbsp;\n\nThank you for your time and effort,\\\nWe rest at your disposal to answer any additional questions should you have any.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is well-written, easy to follow and understand.\n2.\tThe proposed method is simple yet effective, fooling most models within 9 words.\n3.\tUnder cross-lingual settings, the proposed method has the potential of refusing the humans to read added words.\n", "reasons_to_reject": "1.\tLack of novelty. Adding words to inputs to fool model has been proposed in many previous attacks.\n2.\tThe proposed can be recognized by humans very easily. In real scenarios, defenders can find such attacks very soon and deploy a new filtering model very easily.\n", "questions_to_authors": "1.\tLine 179 to line 190, why adding SST sentences makes the added part more coherent?\n2.\tLine 240 to 245, for a toxic content detector, recall is more important than precision in many use cases, did you try changing the threshold to increase the recall?\n", "paper_topic_and_main_contributions": "This paper proposes a simple adversarial attack for fooling toxicity detection models. The proposed method add positive sentiment words from Senti-WordNet to the end of toxic input until the target model flips predictions. The authors tested the method under both monolingual and cross-lingual settings. Experimental results show most models are fooled within 9 words. Further results on adversarial training show that the method can be easily defended. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NC4dDXsNxd", "reviewer": "Reviewer_KyeZ", "summary": "", "questions": "Refer to the reasons to reject above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\nWe will answer your questions and considerations one by one.\n\n&nbsp;\n## Reason to reject №1\n&nbsp;\n>1. The toxicity detectors utilized in the study are self-constructed by the authors and appear to lack robustness. This factor potentially weakens the study's conclusions. It would be more compelling if the authors tested more popular and robust toxicity detectors, such as the Perspective API and other classifiers referenced in their related work. \n\nWe used the most popular toxicity detectors from the Hugging Face hub for each language tested and provided the names of these models. Also, we conducted additional experiments with the Perspective API and OpenAI Moderation API. As we discovered, they are also vulnerable to our attacks: \n\nIn the table below we show the performance of a word-adding attack with the phrase: \"I hate you, dumbass! text for a bot to avoid ban:\"\n\n| n words | OpenAI stable | OpenAI latest | Perspective API |\n|:-------:|:-------------:|:-------------:|:---------------:|\n|    1    |     0.643     |     0.946     |      0.899      |\n|    2    |     0.663     |     0.915     |      0.903      |\n|    3    |     0.647     |     0.918     |      0.899      |\n|    4    |     0.605     |     0.904     |      0.886      |\n|    5    |     0.601     |     0.872     |      0.878      |\n|    6    |     0.553     |     0.902     |       0.87      |\n|    7    |     0.553     |     0.926     |      0.863      |\n|    8    |     0.551     |     0.916     |      0.854      |\n|    9    |      0.33     |     0.854     |      0.853      |\n|    10   |     0.421     |     0.847     |      **0.846**      |\n|    11   |     0.381     |     0.705     |      0.852      |\n|    12   |     0.427     |      0.74     |      0.852      |\n|    13   |     0.453     |     0.761     |      0.852      |\n|    14   |     0.426     |     **0.665**     |      0.853      |\n|    15   |     0.362     |     0.681     |      0.853      |\n|    16   |    **0.265**    |     0.716     |      0.853      |\n\nWe also conducted additional experiments in other languages and between them (only with OpenAI, Perspective API is failing with an error in the multilingual setting). In the table below, you can find the results of cross-lingual attacks on OpenAI models:\n\n| n words | Latest EN+FR | Latest EN+GER | Stable EN+FR | Stable EN+GER |\n|:-------:|:------------:|:-------------:|:------------:|---------------|\n|    0    |     0.872    |     0.872     |     0.888    | 0.888         |\n|    1    |     0.931    |     0.981     |     0.793    | 0.848         |\n|    2    |     0.895    |     0.898     |     0.779    | 0.51          |\n|    3    |     0.769    |     0.889     |     0.671    | 0.511         |\n|    4    |     0.816    |     0.846     |     0.594    | 0.344         |\n|    5    |     0.771    |     0.807     |     0.606    | 0.302         |\n|    6    |      0.8     |     0.647     |     0.618    | 0.318         |\n|    7    |     0.659    |     0.533     |     0.605    | **0.201**         |\n|    8    |     0.592    |     0.555     |     0.573    | 0.208         |\n|    9    |     **0.579**    |    **0.489**     |     **0.424**    | 0.283         |\n\nWe also tested ChatGPT in conversation mode with different prompts and found it susceptible to the attack.\n\n&nbsp;\n## Reason to reject №2\n&nbsp;\n\n>While the authors' observation that the toxicity score decreases as the number of appended positive words increases seems intuitive, it may not necessarily be a novel insight. It is expected that the ratio of toxic words to the total number of words in a sentence would impact the toxicity score. The question then becomes whether this finding offers any new understanding of how toxicity detectors function or can be manipulated. Without further exploration or explanation, this observation might be considered an expected outcome rather than a groundbreaking discovery.\n\nOur intuition, built on understanding the principles of current text classification models, indeed tells us that the total number of non-toxic words in a sentence would impact the toxicity score.\\\n&nbsp;\n\nHowever, we believe it is not how it should be:\\\n&nbsp;\n\nJigsaw defines toxicity as \"rude, disrespectful, or unreasonable language that is likely to make someone leave a discussion.\" - From this point of view, messages: \"Nobody wants to talk with you!\" and \"Nobody wants to talk with you! text for a bot to avoid ban: flowers happiness good love\" should have the same toxicity score.\\\n&nbsp;\n\nIn the same way as a drop of poison will turn a bottle of wine into a bottle of poison, a toxic message can not be watered down by a bunch of positive words. Hatred cannot be justified.\n\nThus, by our attack we demonstrate a significant flaw in the current approach to toxicity detection.\n\n&nbsp;\n## Reason to reject №3\n&nbsp;\n\n>While the authors emphasize \"multilingual\" aspects in their paper, the reported results do not indicate that multilingual models are more susceptible than monolingual models. Furthermore, the proposed method doesn't seem to have a significant correlation with the multilingual setup.\n\nMultilinguality of the model allows us to perform cross-lingual attacks, e.g. combining English offensive messages with positive words in Arabic. In this case, a larger amount of languages leads to a larger number of possible attacks - in the case of the 5-language model, we can have a factorial of 5 combinations of languages, making it 120 possible attack scenarios.\n\nSuch a cross-lingual attack might even effectively prevent humans from reading the part which is intended to distract the model (if they cannot read Arabic), thus emphasizing the idea of clear separation of messages addressed to a human and an algorithm even more. \n\n&nbsp;\n## Reason to reject №4\n&nbsp;\n\n>The defense strategy employed (training on adversarial sampling) is fairly standard, and its effectiveness has been well-documented in previous research. Thus, it does not add a novel contribution to the field. \n\nWe wanted to show the applicability of this defence and to demonstrate that one needs to build defence models for both word- and sentence-based attacks, as one single model will not work perfectly for both scenarios. \n\n&nbsp;\n \nThank you for your time and effort,\\\nWe rest at your disposal to answer any additional questions should you have any.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method, though straightforward, appears to have a potent attacking impact.\n2. The authors have also considered the implications of their method on multilingual models.", "reasons_to_reject": "1. The toxicity detectors utilized in the study are self-constructed by the authors and appear to lack robustness. This factor potentially weakens the study's conclusions. It would be more compelling if the authors tested more popular and robust toxicity detectors, such as the Perspective API and other classifiers referenced in their related work.\n2. While the authors' observation that the toxicity score decreases as the number of appended positive words increases seems intuitive, it may not necessarily be a novel insight. It is expected that the ratio of toxic words to the total number of words in a sentence would impact the toxicity score. The question then becomes whether this finding offers any new understanding of how toxicity detectors function or can be manipulated. Without further exploration or explanation, this observation might be considered an expected outcome rather than a groundbreaking discovery.\n3. While the authors emphasize \"multilingual\" aspects in their paper, the reported results do not indicate that multilingual models are more susceptible than monolingual models. Furthermore, the proposed method doesn't seem to have a significant correlation with the multilingual setup.\n4. The defense strategy employed (training on adversarial sampling) is fairly standard, and its effectiveness has been well-documented in previous research. Thus, it does not add a novel contribution to the field.\n\n", "questions_to_authors": "Refer to the reasons to reject above.", "paper_topic_and_main_contributions": "This paper presents an innovative yet straightforward method of attacking toxicity detector models by appending positive words or phrases to hateful messages. From the reported experiments, this approach could significantly challenge the effectiveness of neural network-based toxicity detectors.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work demonstrated a novel and effective way of bypassing the toxicity detection models by appending positive words or sentences to the end of a toxic message. It showed that this sentence-level attack can be applied to multiple languages from different language families, and that the models’ confidence can be reduced by increasing the length of the added sequence.

Pros: 
Introduced a concept of “To Each His Own” attack, which is based on the idea of separating the messages addressed to a human and an algorithm. The attack exploits the fact that the toxicity detection models are trained on sentence-level labels, and do not consider the context or the intention of the message.

Cons
May not be effective on toxicity detectors that use different definitions or criteria of toxicity
Decision: Accept-Findings