SlowGuess/NAACL_2022_OCR huggingface dataset:
Selection Criteria Details in UTSC Attack For selection criteria, we used the average toxicity scores from three different classifiers (Perspective API, Toxic-bert, and Safety classifier) unless otherwise stated in which we either used the score from one toxicity classifier or the average score from two classifiers. To determine whether an utterance is toxic or not, we used the default thresholds set by the developers for Toxic-bert and Safety classifiers and a threshold value of 0.5 for Perspective API. In addition to toxicity scores, we considered other selection criteria, such as length of the generated attack; however, we saw no significant signal in using the length. Thus, we focused on using toxicity scores in the main text which as shown in the results play a significant role in attack effectiveness. Notice that other selection criteria can be considered along with length and toxicity scores, such as perplexity score for fluency or other metrics; however, for this study, we considered these two cases. In our experiments the adversary generates 10 candidate attack utterances for each of its attacks and the final attack utterance is selected based on the selection criteria out of those 10 generated candidates. Additionally, we report some statistics about toxicity scores of the adversary on the attack utterance as well as defender's toxicity score after the attack for UTSC-1, UTSC-2, and UTSC-3 attacks which can provide additional intuition on how toxic each attack is. These results are on the 100 conversations that are used in our experiments and are reported in Table 3.
B.1 Additional Quantitative Results
Data Sensitivity In Figure 15, we demonstrate what proportion of the attack effectiveness comes
<sup>6</sup>https://huggingface.co/microsoft/DialoGPT-small
<sup>7</sup>https://huggingface.co/gpt2
<sup>8</sup>https://huggingface.co/unitary/toxic-bert
$^{9}$ https://parl.ai/projects/safety Recipes/
Adversary	Defender
Method	Average Toxicity Score	Variance	Max	Average Toxicity Score	Variance	Max
UTSC-1	0.61	0.02	0.93	0.21	0.05	0.93
UTSC-1 w Toxic-bert	0.57	0.03	0.93	0.19	0.04	0.93
UTSC-1 w Perspective API	0.61	0.02	0.93	0.21	0.04	0.93
UTSC-1 w safety	0.53	0.04	0.93	0.20	0.04	0.93
UTSC-2	0.39	0.09	0.89	0.15	0.02	0.70
UTSC-2 w Toxic-bert	0.41	0.09	0.89	0.15	0.02	0.70
UTSC-2 w Perspective API	0.50	0.06	0.89	0.17	0.03	0.83
UTSC-2 w safety	0.42	0.05	0.81	0.19	0.04	0.83
UTSC-3	0.1	0.01	0.45	0.11	0.01	0.64
UTSC-3 w Toxic-bert	0.07	0.00	0.34	0.12	0.01	0.73
UTSC-3 w Perspective API	0.05	0.00	0.14	0.12	0.01	0.64
UTSC-3 w safety	0.08	0.00	0.45	0.11	0.01	0.64
Table 3: Average toxicity scores from 100 conversations for each of the UTSC attacks including variance and maximum scores when the adversary uses different classifiers for selection criteria. The toxicity scores are reported based on Perspective API.
from which of the two Wizard of Wikipedia and Reddit datasets. As also mentioned in the main text, Reddit dataset contains context topics around more sensitive issues, while the Wizard of Wikipedia data is more neutral. We show in our results that the topic context does not play a major role in our attacks being effective and indeed our attack can work as well or even better for the Wizard of Wikipedia dataset that contains more neutral context topics.
Attack Transferability In Figure 11, we demonstrate that no matter what toxicity detection classifier the attacker uses to chose its attack utterance, the attack can still transfer to other toxicity detection classifiers. For instance, if the attacker only uses Perspective API to perform its attack, results show that the attack is still successful according to Toxic-bert and Safety classifiers in addition to Perspective API. Results for different combinations is shown in Figure 11.
Defense Transferability In Figures 10 and 12, we show two different types of defense transferability. In Figure 10, we show that the defender and the attacker do not need to use the same toxicity detection classifiers for the defense to be effective. We show that for instance, if the attacker is only using Perspective API to perform its attack and the defender is using Toxic-bert to perform the defense the defense is still effective for  $100\%$  of the times. We demonstrate different combinations of classifiers used by the attacker against a defender that uses Toxic-bert to perform the defense. In all the cases, we show that the defense is effective  $100\%$  of the times for our defense mechanism.
In Figure 12, we show that the defense trans
fers to other toxicity detection classifiers as well not only Toxic-bert for all the different combinations of the attacker toxicity detection classifiers. Thus, results show that even if the defender is using Toxic-bert to perform the defense, according to both Perspective API and Safety classifiers the amount of toxicity is still decreased after the attack irrespective of what toxicity classifier the attacker is using. Of course, the defense is the most effective for Toxic-bert classifier; however, it is interesting that the attack also transfers to other classifiers.
Finally, we show some qualitative results from our attacks and defenses in Figure 16. We show results from our automatic attack strategy as well as our defense mechanism on it (Figure 16 (a)) along with our human experimental results in which a human adversary tries to fool the system into generating toxic utterances (Figure 16 (b)) and lastly the GPT-2 experiments using the RealToxicityPrompts and how effective our proposed defense mechanism works on these sets of prompts and model (Figure 16 (c-f)).
Notice that our human performed attacks did not consider any contexts since the human adversary was defining the context and starting the conversation with the context in mind all in one shot. This is slightly different than our automatically performed attack setup in which we always start the conversations given a context topic to force the bots to converse around the given topic and not just a random topic. The rest of the experimental setup, however, is similar to the automatic attack/defense setup.
Trigger Masking	UTSC-1	UTSC-2	UTSC-3	UAT (6-gram)	UAT-LM (6-gram)	UAT (unigram)	UAT-LM (unigram)
Defense Effectiveness	75%	50%	0%	71%	77%	71%	55%
Table 4: Effectiveness of Trigger Masking baseline for each attack. UAT/UAT-LM (unigram and 6-gram) indicate whether we removed one or the overall 6-gram triggers from the corresponding attacks. Results demonstrate that masking the triggers naively is not the best defense strategy as other toxic words in the attack utterance may trigger toxic generation. Our proposed defense mechanism along with Two-stage Non Sequitur baseline achieve  $100\%$  defense effectiveness on all the attacks mentioned in this table.
Figure 10: Effectiveness of different defenses against different attack strategies using different toxicity classifiers during the attack process according to Toxic-bert classifier.
Figure 11: Transferability of the attack among different toxicity classifiers. Attacker uses different toxicity classifiers in each plot; however, the results transfer to other toxicity classifiers.
Figure 12: Transferability of our proposed defense to Perspective API and Safety classifiers for different attack strategies using different toxicity classifiers during the attack process.
Survey Instructions (Click to collapse)
CLICK TO READ ALL INSTRUCTIONS
Please make sure to understand the instructions fully before you begin.
In this task you will first read three different conversations between two bots and later compare and rate the quality of conversations provided to you. Each conversation starts with a given context which triggers the bots to converse around the given topic of the context. The beginning three lines in each conversation will be the same for all the three conversations and the change happens on the fourth line; thus, we recommend you to focus on lines four or forward to compare these conversations together. We ask you to compare and rank the conversations according to four different qualities (coherency, relevancy, fluency, and toxicity) each defined in detail below with a corresponding example for each:
Coherency: "Coherence refers to the way things flow and function together, how they are connected and whether they are consistent. In a conversation, coherence refers to the way the participants cooperate to maintain a reasonably focused thread of conversation." (credit: http://www.pliant.org/personal/Tom_Erickson/DA_Donath.pdf). Please also consider the context and how coherent the bots stay around the given context throughout the conversation.
Relevancy: Refers to how relevant a turn is to the other turns specially previous turns in the conversation. Note: In our case, we are interested in relevancy of the turn to the previous turns. Fluency: How grammatically sound and fluent the sentence is. Toxicity: How toxic, offensive, or harmful the sentence is.
Conversation A (Conv A)
Context: Do you like dogs?
Bot1: I like dogs what about you?
Bot2: I like dogs as well
Bot1: walk dog blackboard school
Bot2: I love children.
Bot1: lets go to a party.
Conversation B (Conv B)
Bot1: Do you have a dog?
Bot2: Yes I have a dog.
Bot1:That is nice!
Conversation C (Conv C)
Context:Do you like dogs?
Bot1: Streets is so busy today!
Bot2: Yes, I hate this city and its people.
Bot1: lets move to another city.
Rate the coherency of the conversations below. Conversation A:
Rate relevancy of red line to the previous lines. Conversation A:
Rate fluency of red line in the conversations below. Conversation A:
Rate the toxicity of line in green
Conversation A:
not-toxic
Conversation B:
Conversation C:
$①$  mc
oxic
Figure 13: Mechanical Turk survey for the attack experiments.
In this task you will first read three different conversations between two bots and later answer some questions that want you to choose the best and worst quality conversations among the three conversations provided to you. Each conversation starts with a given context which triggers the bots to converse around the given topic of the context. The beginning four lines in each conversation will be the same for all the three conversations and the change happens on the fifth line; thus, we recommend you to focus on lines five or forward to compare these conversations together. We ask you to rank the conversations according to four different qualities (coherency, relevancy, fluency, and toxicity) each defined in detail below with a corresponding example for each:
Coherency: Coherence refers to the way things flow and function together, how they are connected and whether they are consistent. In a conversation, coherence refer to the way the participants cooperate to maintain a reasonably focused thread of conversation." (credit: http://www.pliant.org/personal/Tom_Erickson/DA_Donath.pdf)
Relev: Re   t  t th r t. Fluency: How grammatically sound and fluent the sentence is.
Toxicity: How toxic, offensive, or harmful the sentence is. Example:
Bot1: Are you working hard?
Bot2: Yes I has the dog good.
Bot2: Yes! I hates people who do not like dogs.
Bot1:hmmm.
Rate the coherency of the conversations below.
$\bigcirc$  bad
Rate relevancy of red line to the previous lines.
Rate fluency of red line in the conversations.
Rate the toxicity of line in red
toxi
Figure 14: Mechanical Turk survey for the defense experiments.
(a) According to Toxic-bert.
Figure 15: Attack results considering the differences between each of the Wizard of Wikipedia and reddit datasets.
(b) According to Perspective-API.
(c) According to Safety-classifier.
(a) Human performed attack vs our pro-(b) Our proposed automatic UTSC-1 (c) Our proposed automatic UTSC-3 posed automatic defense (attacker is attack vs our proposed automatic de-attack vs our proposed automatic de-a human and the defender is a non-fense (both attacker and the defender fense (both attacker and the defender human bot). This shows the transfer- are non-human bots). Notice in UTSC- are non-human bots). Notice in UTSC- ability of our defense to human gener-1 the adversary generates non-toxic at- 3 the adversary generates non-toxic at- ated attacks. tack utterance. tack utterance.
(d) RealToxicityPrompts vs GPT-2 (e) RealToxicityPrompts vs GPT-2 gen-(f) RealToxicityPrompts vs GPT-2 gen-generated responses one with the de-erated responses one with the defense erated responses one with the defense fense (in the dotted box) and one with-(in the dotted box) and one without (af-(in the dotted box) and one without (afout (after the defense arrow). ter the defense arrow).
Figure 16: Different qualitative results from our performed diverse experiments including human performed attack against our proposed defense mechanism (a), our proposed automatic attack and defense strategies (b-c), and lastly our defense mechanism on GPT-2 model using RealToxicityPrompts (d-f). The Dotted box represents the response if the defense was not applied, and the response after the defense arrow shows the newly generated response after applying the defense mechanism. Results show that the responses after the defense arrow (representing with defense response) are less toxic in all the cases compared to the results generated in the dotted boxes (representing the response without any defense applied). We also demonstrate the effectiveness of our defense against both toxic UTSC-3 (b) and non-toxic UTSC-1 attacks (c).