Probing LLMs for hate speech detection: strengths and vulnerabilities
Authors: Sarthak Roy
Ashish Harshvardhan
Animesh Mukherjee
Punyajoy Saha
Venue: EMNLP 2023 Findings (2023)
Keywords: hate speech
explanation
large language models
detection

Abstract: Recently efforts have been made by social media platforms as well as researchers to detect hateful or toxic language using large language models. However, none of these works aim to use explanation, additional context and victim community information in the detection process. We utilise different prompt variation, input information and evaluate large language models in zero shot setting (without adding any in-context examples). We select two large language models (GPT-3.5 and text-davinci) and three datasets - HateXplain, implicit hate and ToxicSpans. We find that on average including the target information in the pipeline improves the model performance substantially ($\sim20-30\%$) over the baseline across the datasets. There is also a considerable effect of adding the rationales/explanations into the pipeline ($\sim10-20\%$) over the baseline across the datasets. In addition, we further provide a typology of the error cases where these large language models fail to (i) classify and (ii) explain the reason for the decisions they take. Such vulnerable points automatically constitute `jailbreak’ prompts for these models and industry scale safeguard techniques need to be developed to make the models robust against such prompts.

Probing LLMs for hate speech detection:
strengths and vulnerabilities
Sarthak Roy, Ashish Harshavardhan, Animesh Mukherjee and Punyajoy Saha
Indian Institute of Technology, Kharagpur (sarthak. cse22@kgpian animeshm@cse). ii titkg..aa.... at..gp..c.in, ac.in, (ashishaug29 tun ac.
Abstract
burgh synagogue', the Charlottesville car attack?, etc. have all been caused by perpetrators consum- ing/producing such abusive content.
Recently efforts have been made by social me- dia platforms as well as researchers to detect hateful or toxic language using large language models. However, none of these works aim to use explanation, additional context and vie- tim community information in the detection process. We utilise different prompt variation, input information and evaluate large language models in zero shot setting (without adding any in-context examples). We select three large language models (GPT-3.5, text-davinci and Flan-T5) and three datasets HateXplain, im- plicit hate and ToxicSpans. We find that on average including the target information in the pipeline improves the model performance sub- stantially (~ 20-30%) over the baseline across the datasets. There is also considerable ef- feet of adding the rationales/explanation into the pipeline (~ 10 20%) over the baseline further pro- vide a typology of the error cases where these large language models fail (i) classify and (ii) explain the reason for the decisions they take. Such vulnerable points automatically con- stitute 'jailbreak' prompts for these models and industry scale safeguard techniques need to be developed to make models robust against such prompts.
In response to this, social media platforms have implemented moderation policies to reduce the spread of hate/abusive content. One important step in content moderation is filtering of abusive content. A common way do this is to train language mod- els on human annotated contents for classification. However there are challenges in this approach in the forms of heavy resources required terms of labour and expertise to annotate these hateful con- tents. This exercise also exposes the annotators to a wide array of hateful contents that is almost always psychologically very taxing. Therefore, recently, many works have tried to understand if Large lan- guage models (LLMs) can be used for detecting such abusive language (Huang et al., 2023; Ziems et al., 2023), but none of these study the role of additional context as input (output) to (from) such LLMs.
across the datasets. In addition,
We, for the first time, introduce several prompt variations and input instructions to probe two of the LLMs (GPT 3.5 and text-davinci) across three datasets HateXplain (Mathew et al., 2021), implicit hate (EISherief al., 2021) and Toxic- Spans (Pavlopoulos et al., 2022). Note that all these three datasets contain ground truth explana- tions in the form of either rationales (Mathew et al., 2021; Paylopoulos et al., 2022) or implied state- ments (EISherief et al., 2021) that tells why an annotator took a particular labelling decision. In addition, two of the datasets also contain the infor- mation about the target/victim community against whom the hate speech was hurled. In particular, we design prompts that contain (a) only the hate post as input and a query for the output label (vanilla)
1 Introduction
Abusive language has become a perpetual problem today's online social media. An ever-increasing number of individuals are falling prey to online harassment, abuse and cyberbullying as established in recent study by Pew Research (Vogels, 2021). In the online setting, such abusive behaviours can lead to traumatization of the victims (Vedeler et al., 2019), affecting them psychologically. Further- more, widespread usage of such content may lead to increased bias against the target community, making violence normative (Luft, 2019). Many gruesome incidents like the mass shooting at Pitts-
synagogue_shootinng wtt.///a..rggwwi/itttttttoto
Charlotteswille_ccr..ttt
"https://tinyurl1 wttp://uorl..om///eoiii/i/..//

---

(b) post well as the definition of hate speech as input and query for the output label (c) post (-/+ definition) and the target community information as imput and a query for the output label, (d) post (-/+ definition) and the explanation as input and a query for the output label, (c) post (-/+ definition) as input and query for the output label and the tar- get community, and (f) post (-/+ definition) as input and query for the output label and the explanation.
posts and posts containing opinions or fact check information about news articles are of- ten misclassified as toxic/hateful. On the other hand, many offensive/hateful posts are marked normal by the model either due to a vocabu- lary gap or presence of unknown or polyse- mous words. Similarly, these models miss to classify implicitly toxic posts and mark them non-toxic.
We record the performance of all these approaches and also identify the most confusing cases. order to facilitate future rescarch, we further provide a typology of the error cases where the LLMs fail to classify and usually provide poor explanations for the classification decisions taken. This typology would naturally constitute the 'jailbreak' prompts* to which such LLMs are vulnerable thus pointing the exact directions in which industry scale safe- guards need to be built.
We make our codes and resources used for this research publicly available for reproducibility pur- poses?.
2 Related works
Large language models: Based on the training setup, the model architecture and the use cases LLMs can be broadly classified into encoder-only, encoder-decoder based and decoder-only types. In recent years decoder-only LLMs" have seen huge surge with industry scale releases like chatGPT, gpt-4, BARD, LLaMa, etc. Decoder-only LLMs have been used for benchmarks like GLUE (Zhong et al., 2023) where there is no downstream appli- cation involved. In the classification setting LLMs have been extensively used for sentiment analy- sis (Zhang et al., 2023b). They have also been heavily used in NLI and QA tasks multiple datasets (Chowdhery et al., 2022). In the genera- tion setting, LLMs have found applications in sum- marization (Zhang et al., 2023a), machine trans- lation (Chowdhery et al., 2022) and open-ended generation (Brown et al., 2020).
We make the following observations.
In terms of vanilla prompts (that case (a)). we find that flan-T5-large performs the best among the three models; we also ob- serve that text-davinci-000 is better than gpt-3. 5-turbo-003 although the latter is more recent version.
Our proposed strategies of prompts individ- ually benefit the LLMs in most cases. The prompt with target community in the pipeline gives the best performance with ~ 20 - 30% improvements over the vanilla setup. None of these LLMs are able to benefit themselves if multiple prompt strategies are combined.
While doing detailed error analysis, find that the misclassification of non-hate/non- toxic class is the most common error for im- plicit hate and ToxicSpans datasets while for the Hatexplain dataset the majority of misclas- sifications are from the normal to the offensive class. There are also large number of cases where the model confuses between the hate speech and the offensive class.
LLMs for hate speech detection: In (Zhu et al.. 2023), the authors use chatGPT to relabel vari- ous datasets one of which is on hate speech de- tection and found that the agreement with human annotations is still quite poor. The authors in (Li et al., 2023) use chatGPT to classify a comment as harmful (i.e., hateful, offensive, or toxic HOT) and found that the model better in identifying non-HOT comments than HOT comments. Finally in (Huang et al., 2023) the authors attempted to classify implicit hate speech using chatGPT. How- ever, their prompt was framed tion (rather than based on the exact classes i.e., implicit hate, explicit hate, non-hate as in the origi- nal study (EISherief et al., 2021)) which makes the problem lose its original fervour.
From the typology induced from the error cases, we find many interesting patterns. These L.Ms make errors due to the presence of sensitive or controversial terms in otherwise non-hateful posts. Presence of negation words and words expressing support for a commu- nity are misclassified as hateful. Ideological
'yes/no' ques-
'tttps: //docs. chatgpt- www.bbreakkpprmmte
Phttps://shorturll.at////

---

3 Datasets and metrics
found toxic. Not all the posts had toxic spans anno- tated. We create a test set of 2000 samples by pick- ing 1000 samples labelled toxic from this dataset (where the toxic spans were available), and 1000 samples labelled non-toxic from the Civil Com- ments dataset (Borkan et al., 2019). Note that we have the spans/rationales marked only for the toxic data points. For the non-toxic posts For expla- nations as input experiments (see section 5), the complete post (tokenized) taken as the rationale. Metries: For primary evaluation, we rely on clas- sification performance. We use precision, recall, accuracy and macro Fi-score to measure classifica- tion performance, which are all standard metrics,
For all these datasets, we utilise (create) a test dataset for our experiments.
Implicit hate dataset: The implicit hate (EISh- erief et al., 2021) corpus is specialized collec- tion of data aimed at detecting hate speech. It pro- vides detailed labels (implicit_haae hatee,tthteeeeee explicit hate or non hate) for each message, including informa- tion about the implied meaning behind the content. The dataset comprises 22,056 tweets sourced from major extremist groups in the United States.
We test a subset of 2147 samples (108 en- tries labelled explicit_ hate, 710 entries labelled implicit_hate hate, and 1329 entries labelled not_hait not_hate) which are sampled from the entire dataset in a strati- fied fashion. Note that we do not have explanations and targets for all the posts. The implied statements and targets were available only for the samples with label implicit_ hate. Hence, when we experiment with explanation as input (see section 5), we pass the input post the implied statement. This is for the explicit hate and not hat_ data points. Both these categories do not required any additional im- plied statement as there is nothing implied them. In case of targets as inputs (see section 5), we re- move the explicit_hate hate datapoints since they are targetting some victim community but the targets are not present in the annotated dataset. The targets for not hate are set as 'none'.
The data points which have rationales are addi- tionally evaluated using other generation metrics. For the natural language explanations (implicit hate dataset) we use BERTScore averaged over all data points. BERTScore (Zhang et al.) computes a simi- larity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, they compute token simi- larity using contextual embeddings. For extractive explanation WHateeplain/ToxicSppa use average sentence-Be (Papineni et al., 2002) score which is the standard among the generation metrics.
dataset), we
Models
For our experiments we utilise three LLMs. Two of these are from the proprietary GPT-3.5 model series? while the third is an open source one from the T5 series.
HateXplain dataset: HateXplain (Mathew al., 2021) benchmark dataset specifically designed to address bias and explainability in the domain of hate speech. It provides comprehensive annotations for each post, encompassing three key perspectives: classification (hate speech, offensive, or normal), the targeted community, and the rationales - which denote the specific sections of a post that influenced the labelling decision (hate, offensive, normal). We test on the already released test dataset con- taining 1924 samples (594 entries labelled as hate speech, 782 entries labelled as normal, and 548 entries labelled as offensive). Note that we do not have rationales for the normal posts. In the expla- nations as input experiments (see section 5), the complete post (tokenized) is taken as their rationale for the normal posts.
GPT-3.5 models better than their prede- cessors GPT (Brown et al., 2020). Both of these models are highly advanced language mod- els capable of generating human-like text based on the provided prompts but they differ in some key ways. As per their documentation", GPT-3 was optimized on code completion tasks to create -002. This was further improved using instruction finetuning (Ouyang et al., 2022) to create text-davinci-e0s This was later up- graded to text-davinci-pet which was trained on larger dataset (ope) making it better higher quality text generation, following instructions and generating longer context. 3. 5-t is an improvement over text-davinci-000, We
ToxieSpans dataset: The ToxicSpans (Pavlopou- los al., 2022) dataset is a subset (containing 11,006 samples labelled toxic) of the Civil Com- ments dataset (1.2M Posts). The dataset also con- tains the toxic spans, i.e., the region of the texts
Thii was made available by the Microsoft Accelerating AI Academic Research grant. "http://platformooooom rescarchers

---

choose these two models in our study namely the 50ttr.o and the text-davinci-003. The third model we use is the open source flan-T5-large which is an instruction finetuned variant of the popular T5 model (Chung et al.. 2022). As per their documentation the model was instruction finetuned with an emphahsis on scaling the number of tasks, scaling the model size and introducing chain of thought data in the finetuning pipeline. The authors have claimed that this partic- ular sort of finetuning has benefited the T5 models greatly by outperforming models of higher size like GPT-3.
Exp (input)) improve its labelling decisions and (b) whether asking LLMs for an explanation about its labelling decision forces it to predict better labels and as well generate relevant explanations (cor- responding to the templates Vanilla + Exp (out- put) and Vanilla + Defn + Exp (output)). For the HateXplain and the ToxicSpans dataset the ground truth explanations are in the form of ra- tionales (i.e., part (parts) of the post that the an- notator marked as the reason for his/her labelling decision). For the implicit hate speech dataset the ground truth explanations are in the form of implied statements. In the templates Vanilla + Exp (output) and Vanilla + Defn + Exp (out- put) we use two variables explanation applanation_formal (see Appendix A, Table 7), For each dataset, note the values in these lists below.
5 Prompts
and
In this section, we list the prompt variations used in this work. A concise summary of the different variants is noted in the Appendix A (Table 7) and the details for each are discussed in the subsections below.
HateXplain: Here, explanation is "extract the words from the post that you found as hate speech or offensive" and explanation_ format is "the list of extracted words, separated by" ". Enclose the list with <
Dataset HateXplain
Label list
normal, offensive or hate speech explicit_hade implicit_haie or not_hate toxic or non _oxic
Implicit hate
ToxicSpans
<<>>>"
ToxicSpans: Here, explanation tract the words from the post that you found as toxic" and explanation_ format is "the list of extracted words, separated by ". Enclose the list with < <<<>>>
is "ex-
Table 1: The list of labels for each dataset.
Vanilla prompts: In this category, we use a prompt template where we ask the model to classify the given post into label out of list_ofllaeess. In addition, we also provide a few example_outpure (one class per line) for helping the models generate proper answer. The list_of_labels for each datasets are noted in the Table
Implicit hate: Here. explanation_type is "with an explanation in 15 words" and is "the explanation en- closed in < <<<>>>"
In the templates Vanilla Exp (input) and Vanilla + Defn Exp (input) we use single variable explanation (see Appendix A, Table 7). For each dataset, note the value in this list below.
(+) definitions: In the vanilla prompts we assumed that the I.MSs are to an extent aware of the labels for classification. Here, we provide the definitions additional context the LLMs. This can help the LLMs understand the classification tasks better. These definitions are added as list where each label's definition is separated by a new line. We note this prompt template as Vanilla + Defn in Appendix A, Table 7. Individual definitions of the tasks are added in Appendix B.
HateXplain: expl explanation is "the rationales (rationales] as an explanation".
ToxicSpans: explanation is "the span (span) as an explanation".
Implicit hate: explanation is "the implied statement (implied statement) as an explana- tion".
(+) explanations Recently, there has been huge interest in developing explainable deep learning models where the prediction decision is supported by an explanation (Bhatt et al., 2020). We test two hypotheses - (a) whether providing explana- tions to LLMs as inputs (corresponding to the tem- plates Vanilla + Exp (input) and Vanilla + Defi +
(+) targets A very important information in any hate speech detection pipeline is the victim com- munity the abusive/hate speech targets. Here again we test two hypotheses - (a) whether providing target to LLMs as inputs (corresponding to the tem- plates Vanilla + Tar (input) and Vanilla + Defn + Tar (input)) improve its labelling decisions and

---

Datasets
F1-Seoe
Precision
Recall
0.45 in text-davinci-003) and the implicit hate (0.32 in 3pt-3.5-turbo-025 and 0.36 text-davinci -003) datasets are much less than the ToxicSpans (0.68 in gpt-3. 2ot-3.5-tuue 5-t.5---t3o000-0 and 0.71 in text-davinci ceciiniii000) 0003 dataset.
HateXplain
0.698
0.687
NR
Implicit hate (full)
0.54
0.64
0.47
ToxicSpans
0:31
0.79
0.20
Table 2: BERT- Hatexpaan inference on three datasets used to prompt the LLMs. NR: not reportedin the original pa- per (Mathew ul., 2020).
(+) definitions: does not always help in improving the per- formance. In terms of Fl-score, for Hat- eXplain, we see an improvement of 25.64% for 5pt-3.5-turbo-030 while the performance worsens by 17.78% for text-davinci-
Adding definitions to the prompts
(b) whether asking LLMs for the target information forces it to predict better labels and as well gener- ate correct targets (corresponding to the templates Vanilla + Tar (output) and Vanilla + Defn Tar (output)). In the templates Vanilla Tar (out- put) and Vanilla + Defn + Tar (output) we variables target_ type and format (see Appendix A, Table 7). For all the datasets, we replace the variable target_ tare with "also men- tion which group of people does it target" and the variable
and 20.34% for flan-T5-large. The situation is re- verse for the ToxicSpans dataset where we see an improvement of 1.40% for text-davinci-eu and 13.75% for flan-T5-large while it worsens by 7.35% for gpt-3. 5-t.b.b-ttro--000 0301. For Toxi- cSpans, adding definitions the input prompt gives the best performance across all the prompt- ing strategies. For implicit hate, there is an im- provement of 12.50%, 16.67% and 4.76% for 0301, text-davinci-003 and flan-T5-large respectively.
format with "list targeted groups enclosed in <<<> <<>>>> >> In the templates Vanilla Tar (input) and Vanilla + Defn Tar (input) we replace the variable targets (see Appendix A, Table 7) with the ground truth target community information which is only applicable for the Hat- eXplain and the implicit hate speech datasets.
(+) explanations: As discussed earlier, we exploit the power of explanations/rationaaess in two ways. For the case where we ask the model to generate explanations along with the label in the output, we see a similar trend like adding definitions. In terms of Fl-score, for HateXplain, we see an improvement of 23.07% for gpt-3. 300-3.5-turbo-009 while the performance worsens by 8.89% for text-davinci-003 and 10.17% for flan-T5-large. The situation is similar for the ToxicSpans dataset we see com- parable performance for gpt-3.5-turbo-0333 while worsens by 9.86% for text-davinci- but it improves by 18.84 % for flan-T5-large For implicit hate, there is improvement of 6.25% and 22.22% for apt-3.5-turbb-025 text-davinci
6 Results
In this section, we note the results of the dif- ferent prompt variations on the models for the three datasets. As a baseline, we consider BERT- HateXplain (Mathew et al., 2020) and run the model on the implicit hate and ToxicSpan datasets. We have taken the results for the HateXplain dataset from the original paper. Table 2 shows the results of this baseline.
Our key results are noted in Tables 3, 4 and for the HateXplain, implicit hate and ToxicSpans dataset respectively.
Vanilla prompts: In terms of vanilla prompts, flan-T5-large performs better than 10t-3.55turbo-0300 and text-davinci-003 for the HateXplain and implicit hate (0.59 and 0.63 Fi-scores respectively) datasets. Nevertheless, it performs at par with the other two models for the ToxicSpans dataset. Among the larger models, we find that text-davinci-003 is in all the
and respectively but it worsens by 11.11 1 % for flan-T5-large. We further compare the generated explanations with the already avail- able ground truth explanations using BERTScore for the implicit hate dataset and sentence-BLe for the other two datasets. We observe that the 3. 5-tt-ttt-bb000 0301 model generates better ex- planations (averaged over all data points) than the text-davinci-
better than the datasets in terms of macro Fl-score, reporting 15.38%, 12.50% and 4.41% higher values for HateXplain, implicit hate, and ToxicSpans respectively. Overall the performance for the HateXplain (0.39 in gpt-3. 1itt-3.5-ture 5-turro--0001111 and
model for the HateXplain and the implicit hate datasets. For the ToxicSpans dataset the results are reversed.
When we add the respective explana- tions/rationales in the input as prompts,

---

terms of Fl-score, for HateXplain the performance is 12.82% better for gpt-3. 5-turbo--300 and 15.55% better for text-davinci-@00 and compa- rable for flan-T5-large. For implicit hate the results are 3.03% better for --0trb-0-tu and 8.33% better for text-davinci-O00 worsens by 14.29% for flan-T5-large. The Fl-scores for ToxicSpans dataset are 8.11% better for gpt-3. 5-turbo-0301 and 20.29% for flan- but significantly worse (~9.86%) for text-davinci-003.
across both these models and the two datasets. This leads us believe that future annotation exereises for training hate speech detection models should almost always benefit if the target information is also annotated.
but
Combinations: Next, evaluate the cases when utilise definitions well another additional prompt strategy (target/explanation at input/output). The performance of adding explanations either at input/output along with definitions does not help the models much with the performance remain- ing comparable, e.g., for spt-3.5m 1-0.3..5-urri 1..55ttr---000 on HateXplain (+) explanation (input) with defi- nition vs with (+) explanation (input) worsens e.g., for 5-tbbbbttrbb-0000 on HateXplain (+) explanation (output) with definition vs with (+) explanation (output). The quality the explana- tions generated (in terms of average BERTScore or sentence- Baieee-BBBE---. are almost always worse in presence of the definitions. worse Similar situ- ation arises when we add definitions with target (input/output) where the performance either re- mains comparable or worsens. Only in the case of the ToxicSpans dataset, we that adding both explanation (input) and definition for the 3pt-3.5-turbal 1---5ttrr--000 0301 and flan-T5-large model and gives the best performance.
(+) targets: We exploit the power of targets again in two ways. For the case, where we ask the model to generate target along with the label in the output, we again see similar trend like adding definitions. In terms of Fl-score, for HateXplain, we see an improvement of 28.20% for 10t-33.-tuurb-0030 worsens by 15.56% for text-davinci-O0 and 11.86% for flan-T5-large. The situation is similar for the TpxicSpans dataset we see compara- ble performance for 3. 5-turbo-0030 while it worsens by 11.27% for text-davinci-003 For im-
while the performance
and 18.84% for flan-T5-large. plicit hate, there is improvement of 9.38%
and 30.56% for gpt-3. 5tt-3.5-turbo-0300 and text-davinci-e00 respectively and comparable results for flan-T5-large.
Error analysis and attack points
For the case, where we use the target in- puts in the prompt we improvement for both the datasets where the target is present in the ground truth except for flan-T5-large. In terms Fl-score, for HateXplain, we see an improve- ment of 33.33% for spt-3.5-turbo-00 26.67% for text-davinci-000 and a drop in per- formance by 6.78% for flan-T5-large. For im- plicit dataset, the ground truth targets are present only for the implicit hate data points. Thus while comparing with the Vanilla setup, we only con- sider the implicit hate data points to have a fair comparison. The revised Fl scores for the Vanilla setup become 0.52 for gpt-3. 3.t-3.5- 5-turbo-0301 and 0.68 for cext-davinci-003. Comparison with this revised values show an improvement of 17.30% for 3ot-3.5-turbo. 0000 while for text-davinci-003 the results remain roughly changed. In case of flan-T5-large the perfor- mance drops by 9.52%. For ToxicSpans have the target annotated and thus we skip this ex- periment for this dataset. Overall target as inputs outperforms most of the other prompt strategies
In this section we shall first discuss the cases where the models encounter large number of misclassi- fications and then outline technique to induce typology of the different attack points to which the models are vulnerable.
and
Significance test: We compare the results of the vanilla model for each dataset with the best performing prompt plus model combination us- ing the Mann-Whitney U test. We find that the results are significant. We next present the p-values for each (a) gpt-3. 5.13..5tt0 5-turbo-0301
HateXplain: p = 0.0042 (vanilla vs target- 3. 5- 5-turbo-0301 + implicit hate: = 0.0 (vanilla vs target-at-input),
at-input), (b) gpt-3.
(c) gpt-3. 1...5-tuurbo-300 + ToxicSpans: p
0.00088 (vanilla vs explanation-at
(d)
text-davinci-e0:: HateXplain: p 9.946e
(vanilla target-at-input), (e) text-davinci-e03 + implicit hate: p 4.44le 16 (vanilla vs target- at-input), (f) text-davinci-one
do not
ToxicSpans: definition-at-inpult = 0.01524 (vanilla vs definition iiciiinnat-innge at---uuttnn,,, (g)
flan-Ts-large + HateXplain: p = 0.0 (vanilla vs

---

Model
Strategies
Metrics
Model
Strategies
Metries
Ex (o)
Ta (o)
D
Ex ()) Ta (1)
Ace
Pre
Rec
R1
BL
Ex (o)
Ta (o)
D Ex (0) Ta (0)
Ace
Pre
Ree
BL
0.54 0.46
0.50 0:56 0.52 0.48 8:15
0.46 0.52 0.34 0.8
0.50 p.63 0.53 0.49
0.1 0.49 0.47 0.36
GPT-3.5
GPT-3.5
0.49 0.00 0.52 0.49 0.13
041 0.49 0.49 0.36 D.8
0.48 0.55 0.49 0.44
0.37 0.44 0.41 0.33
0.33 0.60 0.56 0.52
0.45 0.45 0.40 0.34
0.52 0.60 0.55 0.50
068
0.61
0.52 0.61 0.55 0.50
0.35
0.49 1:63 0.53 0.47
0.38 0:48 0.45 0.33
0.47 0.55 0.47 0.45
0.52 0.54 0.36
0.05
J
0.4
Davinci
Davinci
0.41 0.57 0.41 0.34 7e-4
0.36 0.42 0.50 0.32 0.78
0.46 0.56 0.45 0.41
0.45
0.40
0.45 0.39
0.53 0.57 0.51 0.52
0.44
0.46
0.46 0.42
0.57 0.62 0.57 0.57
0.78
0.68 0.68
0.54 D.6 0.53
041 0.49 0.42 0.33
0.45
0.53 0.39
0.50 0.70 0.66
0.59
0.67
0.62 0.63
0.54 0.63 0.60
0.53 0:02
0.56 0:86
0.67
Hlan-T3
Flan-T5 /
0.57 0:57 0.57 0.57 0.02
0.7 0.66
0.56 0.83
0.54 0.69 0.60 0.51
0.64 0.62 0.54 0.50
0.60 0.62 0.62 0.60
0.64 0.61 0.56 0.54
0.56 0.64 0.61 0.55
0.64 0.58 0.37 0.57
0.62 0.67 0:65 0.62
0.62
0:70 0.57
Table 3: Results for the HateXplain dataset with different prompt variations. Ex: explanation, Ta: target. D: definition, i: input, or output. Ace: accuracy, Pre: precision, Rec: recall, BS: BERTScore, BL: sentence-BLER The best results are highlighted.
Table 4: Results for the implicit hate dataset with different prompt variations. Ex: explanation, Ta: target, D: definition, i: imput, or output, Acc: accuracy, Pre: precision, Rec: recall, BS: BERTScore, BL: sentence-BLEU The best results are highlighted.
definition-targeet-a--innpp (h) flan-T5-large + implicit hate: p 0.000358 (vanilla definition- at-input), (i) flar-e ToxicSpans: p 0.0 (vanilla explanation -at-input).
the different prompt variants the largest number of misclassifications is from the normal to the of- fensive class. Another confusion that the model often faces between the hate and the offensive class; if the prompts do not contain the definition (of hate/offensive speech), then offensive speech is largely mislabelled as hate speech while the sults are exactly reversed if the prompts contain the definition. For the text-davinci-eee once again we observe for all the different prompt variants the largest number of misclassifications is from the normal to the offensive class. Further, irrespective of whether the prompts contain the def- inition or not, hate speech is heavily mislabelled as offensive speech. For 75-a-r5-large both of- fensive and hatespeech classes are missclassified as normal speech for all prompt variations.
Cases of misclassification: For the implicit of the
hate dataset we observe that in gpt-3. 10t.55boroo0000 model, for all the different prompt variants the largest number of misclassifica- tions from the non-hate to the implicit hate class. For the text-davinci-00 model the major obser- vations are as follows. In the vanilla setup (with or without definition), the implicit hate class is most often confused with the explicit hate class. How- ever, if the prompt has explanation component (either at input at output), once again, there is large number of misclassifications from the non- hate the implicit hate class. For flan-T5-large we observe that the model fails to classify the im- plicit hate speech class. The implicit hate classi- fied as non-hate or explicit hate following the trend of the other two models.
model
For the ToxicSpans dataset in case of the 1pt-3.5-turbo 0301 model, the non-toxic data points are heavily misclassified all prompt variants. Curiously, adding defini- tion to the prompts increases this misclassifica- tion. These observations remain similar for the
toxic for
For the HateXplain dataset we observe that in case the gpt-3.5 S-ttrbo--@000 model, for all

---

Model
Ta(o)
Strafegies
(x)
Metrics
BS/L
of topics, K = to induce the types.
8.09 0.73 0.00 8.68
0.70 0.75 0.70 0.68 0.26
For the HateXplain and ToxicSpans datasets we sort the data points in non-decreasing order based on the sentence- -BLU scores between the ground truth rationales and the generated rationales at the output. We select 80 low-scoring data points ac- cording to the BLEU/BERTScor measure. For the HateXplain (ToxieSpans) dataset, starting from the top of this list, we consider the data points that are either normal and misclassified as hate speech/offensive (toxic) or vice versa by all the three models. These points then passed through LDA (number of topics, K
0.67 0.75 0.67 0.63
GPI33.5
0.66 0.76
0.66 0.72
0.75 0.78
0.75 0.74
0.71 0:79
0.71 0.69
0,70 0.74 0.70 0.69
0.70 0.75 0.70 0.68
0.66 0.69
0.66
0.64 0:45
0.72 0.72
0.72
0.72
Davinei
0.71 0.71 0.71
0.71 0.35
0.65 0.66
0.65
0.64
in the ground truth
0.68 0.69
0.68 0.68
0.66 0.69 0.66 0.64
0.69 0.71 0.69 0.68
0.70 0.76 0.70 0.69
0.78 0.84
3) to induce the types. For each topic, we choose four words which have the highest proba- bility of association with the topic. These collected set of words are then manually coded with topic names by two researchers with long experience in hate speech research. In order to obtain a semantic clarity upon the derived topic names, a meticulous manual annotation process is undertaken. This an- notation task done by three domain experts, and notably, unanimous consensus is reached among all annotators regarding the semantic interpretation and nomenclature of each delineated topic name. We note the emerging typologies in the Table Observations from the induced typology: For each dataset, we note some of the interesting ty- pologies that emerged. (1) For the implicit hate dataset, one curious case of misclassification (non- hate implicit hate) the presence of the 'racist' word, while in another category we find the model marked something as implicit hate because has pro-whit trotw sentiments. Even when post representing news articles or opinion pieces contain sensitive words like 'blacks', 'antifa' etc., the models la- bel them implicit hate. (2) For the HateXplain dataset, find that hateful or offensive posts get misclassified normal due to multiple reasons (a) presence of negations like 'neither', 'nor', 'no' etc. the post, (b) vocabulary gap where the mod- els do not know the usage of words like 'muzrat' used in the post, and (c) presence of polysemous words in the post with one of the meanings usually being slang (usually not found in standard English dictionaries), c.g., 'dyke', 'furries', 'furfaggotry' etc. (3) For ToxicSpans dataset, note that ideo- logical posts and posts with fact-checking related statements about political news often get misclas- sified as toxic. On the other hand, posts that are implicitly toxic often get misclassified as non-toxic
Flan-TS
0.82 0.84
0.82
0.82
0.43
033
0.84
0.83
0:8
033
0.83
0.83
0.83
0.62
0.76
0.62
0.56
0.79
Table 5: Results for the ToxicSpans dataset with different prompt variations. Ex: explanation, Ta: target. D: definition. i: imput, or output. Acc: accuracy, Pre: precision, Rec: recall, BS: BERTScore, BL: sentence-BLEU The best results are highlighted.
text-davinci-003 model. For flan-T5-large non-toxic data points are misclassified as toxic for all the prompt variations except for when target is asked to be generated in which case toxic data- points are misclassified as non-toxic.
In order to better elicit the above observations, present the confusion matrices for the best prompt combinations for each model and each dataset in Appendix C.
Typology induction: In order to analyse the data points where the models are most vulnerable we employ the following heuristics for each dataset. In these heuristics we always consider the Vanilla Defn + Exp (output) setup.
For the implicit hate dataset we sort the data points in non-decreasing order based on the BERTScore between the ground truth implied state- ment and the generated explanation at the output. Starting from the top of this list, we consider the data points that are either not not_hate in the ground truth and misclassified implicit hate or vice versa by all the three models. Note that these data points constitute the cases where the models misclassify and provide poor explanation for their classification decision. These data points are then passed through LDA (Jelodar et al., 2019) (number

---

Dutasei
Topic cluster
Examples
GT PR
Type
antifa, self, make, right, time, it. memorial, requires, govemiment,
being back huae rt: you can be racist all you socialism another these are the same people who yell about bow they racist. this Uhread
n_h imp_h
racist
Implicit
n_h imp_h
racust
white, people, good, anti, nationalist, right. chief, staff, south, cruz
enough power make white lol
imp_h
pro-whi tt
bow is a apprectating your heriaage while al-
n_h imp_
protehite
privilage we' we're the only lowed to do it
Rocist, white, america, make, people. americans, know, really, trump, new
defining intolerance.. tifa are intolcrant
yeah but
imp hi
strt sensitive worddss
black #unemplayment
9.2 % than double while unemployment of 4.4 56 via dhireamerican
imp_
tostie talk, men, masculinity, freakin, sttack, nigger, slut, people, women
tbic take back the country toxic masculin- patriarchy harm noxious female- ness and public slut posturing empower
negation
HateXplain
women neither feminism nor now could doing
have pulled off
raxpayer supported state university sys-
teeu maga
note no nigger was harned during the
negation
making of this film.
while. like. line, did. large, muzzies, women, tactics, scarf, counter
why be can not boe that work minus crazy
vocab_sap
massive muzzrat espionage
n
vocab_gap
common,, like, christiun, betler, stick. preter, pissing, discussion, sure, provide
additionally furries also transmit infections discase of furfaggotry to others dyke puts the mannish femane at a leshian bar in paris <num- ber> photograph by george beassai
polysery
polysany
trump. tight, good, man, state, like, snapid, don, money, did
Right Charlie the media fabricated all T_ broakaatt during primaries.
ideelogy
ToxieSpans
Then they altered everything Trump said
daring the cumpuign, Do you have Twik- for ranning 24/7 to keep up with Donald's latest rants?
The Liberals leasing the for their lcadership is turning out to be a big. fat, failed experiment!
name n_t
adeology
people, trump, just, know, lax, like. think, don, vote, need
The headline for this amticle has changed originally
n_t
fact_chack_nol__nn
at least,
since
posted yesterday. Hore's 's the latest update: Unhinged Trump emerges, defending first month in White House This article is entirely WRONG! An on- going
fact_chack_pol_nng
disintegrate the
cial system AND THE COUNTRY in loss then years. Computer projections by more than one analyst suggest "ki- netic" outcome within years...
just, like, make, stupid. sure, don, person, people, trump, does
Oh, gay and black, you just caused all white christian friends here salivating at the same time, for what I'm sune.
aLL
inplicit_senantics
This individual does need merapy, hope be gets the help he needs as well as one of those ridiculousty light sex offender sentence. This system broken when it comes
n_t
imalicit_serantics
of this narure. The list was designed to be a warning for parenis, and yet
Table 6: The typologies induced from the error cases each dataset. GT: ground truth, PR: prediction, h: not, hate, imp h: implicit hate, offensive, normal, h: hate speech, n_t non, toxic, toxic, stmt, with sensitive words: statement with sensitive words, pap: vocabulary gap, fact_check_pee Beoccck_po__ewsseeee statement fact checking information about political news.
by the models.
lot of challenges in hate speech detection their vanilla setting. Our prompt strategies, when ap- plied individually, improve the performance of such models. However, merging of the prompt strategies do not help much. Last, we do detailed error anal- ysis and typology coding to find 'jailbreak' points where these models vulnerable.
8 Conclusion
this work, we extensively study three LLMs across three datasets on a variety of prompt setups. Overall these models, though advanced, still face

---

9 Limitations
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
We mostly focus on English datasets in this paper since we wanted to present a detailed analysis of the use of additional context (explanation, target community) which is often not present in multilin- gual datasets. We use these LLMs as a black box hence we do not know the inner workings of the proprietary LLMs. Lastly, although we cover lot of well thought out prompt variations in our paper. these variations are not exhaustive.
10 Ethical statement
Here we discuss the ethical considerations that were not explicitly noted in the body of the paper. We use three of the LLMs gpt-3. 3.---rbbo-ttttttt 0000, text-davinci-003 and flan-T5-large to detect hate speech. These experiments were purely done from research point of view; the actual applica- tion of such model in real world should be done with caution. This is also evident from the chal- lenges these models face while classifying different forms of hate.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Edd H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling cierucctionfinetuune language models.
References
How do text-davinci-002 and text-davinci-
003 differ?
openai help center.
httpa////////atcc/////79999944 wtw.:/dviiiiii-----aadeeett-aaaa (Accessed on 06/20/2023).
https://hcllp. wwt.lppppppniiiiccoo
Mai ElSherief, Caleb Ziems, David Muchlinski. Vaish- navi Anupindi, Jordyn Seybolt, Munmun De Choud- hury, and Diyi Yang. 2021. Latent hatred: A bench- mark for understanding implicit hate speech. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 345-363 Association for Computational Linguistics.
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. 2020. Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT '20, page 648-657.
Fan Huang, Haewoon Kwak, and Jisun An. 2023, Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. In Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, page 294-297.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, abs/1903.0456.1
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever. and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Hamed Jelodar, Yongli Wang. Chi Yuan, Xia Feng. Xiahui Jiang, Yanchao Li, and Liang Zhao. 2019. Latent dirichlet allocation (Ida) and topic modeling: models, applications, survey, Multimedia Tools and Applications, 78: 18:11599-15211.
Lingyao Li. Lizhou Fan, Shubham Atreja, and Libby Hemphill. 2023. "hot" chatgpt: The promise of chat- gpt in detecting and discriminating hateful, offensive, and toxic comments on social media.
Aliza Luft. 2019. Dehumanization and the normaliza- tion of violence: It's not what you think items.

---

(Accessed on Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence?
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. 2023. Can chatgpt reproduce human-generated labels? a study of social computing tasks.
05/02/2021).
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2020. Hatexplain: A benchmark dataset for explainable hate speech detection. CoRR,
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen,
abs/2012.10289.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for ex- plainable hate speech detection. Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 14867-14875.
A Prompt strategies
The Table 7 shows the different prompt variants we use in this study.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems. 35:27730-2741 277444.
B Definitions
B.1 Implicit hate dataset
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
implicit_ hate: Implicit hate speech is defined by coded indirect language that disparages a person or group on the basis of protected characteristics like race, gender, and cultural identity.
explicit hate: Explicit hate refers to openly expressed, direct forms of hatred and preju- dice toward individuals or groups based on their characteristics.
John Paylopoulos, Léo Laugier, Alexandros Xenos, Jef- frey Sorensen, and Ion Androutsopoulos. 2022. From the detection of toxic spans in online discussions to the analysis of toxic-to-cive ci-iivil transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)., Dublin, Ireland. Association for Computational Linguistics.
not hate: This class refers to speech or ac- tions that do not involve any form of hatred, prejudice, or discrimination toward individu- als or groups based on their characteristics.
Janikke Solstad Vedeler, Terje Olsen, and John Eriksen. 2019. Hate speech harms: A social justice discussion of disabled norwegians? experiences. Disability & Sociery, 34(3):368-383.
B.2 HateXplain dataset
hate speech: Any speech or text that attacks a person or group on the basis of attributes such as race, religion, ethnic origin, national origin, gender, disability, sexual orientation, or gender identity.
Emily A Vogels. 2021. The state of online harassment pew research center. https:
/ www.ppwwwseere....eeeeee/ hwwp::h...gg/eeeeeeeeeeeee wrgginternet/2021/ 01 01/13/the- wwweee-oooooon--hanaeeeeeeeeeeee (Accessed on 05/02/2021).
offensive: The text or speech which uses abu- sive slurs or derogatory terms but may not be hate speech.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
normal: The text which is neither offensive or hate speech and adheres to social norms.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang. Kathleen McKeown, and Tatsunori B. Hashimoto. 2023a. Benchmarking large language models for news summarization.
B.3 ToxicSpans dataset
Toxic: In social media and online forum, toxic content can be defined rude, disrespectful, or unreasonable posts that would make users want leave the conversation.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023b. Sentiment analysis in the era of large language models: A reality check.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.
don_toxic: The speech or text that is not toxic and is fit for use in conversation.

---

Category
Prompt lemplates
Classify the post delimined by three hackticks as list_ labels.
Provide the unswer either list_of lfbeel only.
Vanilla
exarple_outputs
"ppest"
Consider the following defiiitooss
dist_of_oefintiii
Based on this classify the post delimited by three backticks list_of_label the format: either list_ of labels only.
Vanilla Defi
Provide
exarple_outputs
""post"
Classify the post delimised by three backticks as jist_of, labels and explamation_ type.
Provide answer format labels only followed explanation
Vanillaa Exp (ontpptt)
example_ cosplee
""aast"
Classily the delimiied by three backlicks of Labels laking into the explanation considered 1tst_of_labell
why post
Vanilla Exp finput)
Provide unswer in the format either list_ labels only.
exapple outputs
"ppost"
Classily the post delimied by three backticks as 1ist of labels and carget_ Eype the format either list. 1ebeels exapple_ cutputs "post"
Provide
labels only followed target, formtt
Vanilla Tar (outpat)
Classaty lhe pust delimited by diree bucklicks as list of Tabels with respect the victim communily Largets. the format either 1ist_of_ only.
Vanilla (impnt)
Provide
exarple_outop outpuus
"pcst""
Consider the [ollowing dctimitioes.
Last_of_defintion
Based on this clussily the post delimited by three backticks list. labels taking into account the explanation post should considenod list__f
Vanilla Exp (impul)
Provide answer in the format either list_of_label labels only.
exarple_cutputs
""post""
Consider Tollowing octiiiions.
list_of_oef _f
classily post delimited backticks list_ explanation, type. Provide answer in the format either list_of_ Labels only followed by explanation_ exarple_cutputs
Vanilla Defn + Exp (outpuf)
""post"
Consider Tollowing dctiiiions.
list_of_deff _f
Vanilla Defn+ + Tar (oulput)
classify post delimited backticks list_ Provide answer in the format eather list_of_ labele only followed by target_
target_tyyee
exarple_outputs
""post""
Consider Tollowing dctiitions.
list_of_def in
classify post delimited backticks list, labels with respect victim community targets. Provide answer in the farmat eather ofst_of_ labells only.
Vanilla Defn + Tar (input)
exarple_outpe
""post"
Table The different prompts used in our experiments.

---

Figure 1: Vanilla target input for implicit hate.
Figure 4: Vanilla + target input for implicit hate.
Figure 2: Vanilla + target_ing target_input for HateXp
Figure Vanilla + target__inpl input HateXplain.
C Confusion matrix of best performing prompt combinations
C.1
GPT-3.5
The confusion matrices for the three best prompt combinations for the three datasets in connection the ppt-3.5-turbo ures 1, 2, and 3.
are illustrated in Fig-
Figure 6: Vanilla definition ceeinition_inputtee for ToxicSpans.
C.2 Davinci
The confusion matrices for the three best prompt combinations for the three datasets connection to the text-davinci ures 4, 5, and 6.
are illustrated in Fig-
C.3 Flan-T5
Figure Vanilla definition input for implicit hate.
The confusion matrices for the three best prompt combinations for the three datasets in connection to the flan-T5-large illustrated in Figures 7, 8, and! 9.
Figure Vanilla + definition input + farget_input for HateXplain.
Preblied
Figure 3: Vanilla + explanation input for ToxicSpans.
Figure 9: Vanilla + explanation inputtnnuuu for ToxicSpans.


Reviews:
[{"review_id": "Qh2kgcBnw5", "reviewer": "Reviewer_xYYx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. While the paper addresses an important issue, the scope of the study is limited to only two large language models and three datasets. This may not be sufficient to draw generalizable conclusions about the performance of LLMs in detecting hate speech.\n\t\nResponse:- This paper introduces a framework for the zero-shot evaluation of hate speech detection with LLMs. We chose two of the popular LLMs - gpt-3.5-turbo and text-davinci for our evaluation. The datasets are benchmark datasets in hate speech detection and further have explanations/rationales that can be used to understand the models' behavior. As per your suggestion, we have further extended our study to FlanT5-large and found comparable results. We have added the performance table below. We shall also add these new results in the revised manuscript.\n\nModel:FLAN-T5-Large(783M):\n\nToxic Span:\n\n|           | nd_ne | nd_eo  | nd_ei | d_ne  | d_eo  | d_ei   |\n| --------- | ----- | ------ | ----- | ----- | ----- | ------ |\n| Accuracy  | 0.704 | 0.781  | 0.831 | 0.797 | 0.823 | 0.8255 |\n| Precision | 0.762 | 0.842  | 0.835 | 0.810 | 0.842 | 0.828  |\n| Recall    | 0.704 | 0.7815 | 0.831 | 0.796 | 0.823 | 0.825  |\n| F1 Score  | 0.686 | 0.823  | 0.830 | 0.794 | 0.823 | 0.825  |\n\nImplicit Hate:\n\n|           | nd_ne | nd_eo   | nd_ei | nd_to | d_ne  | d_eo  | d_ei  | d_to  | d_ti  |\n| --------- | ----- | ------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy  | 0.667 | 0.661   | 0.639 | 0.632 | 0.688 | 0.670 | 0.637 | 0.666 | 0.624 |\n| Precision | 0.643 | 0.655   | 0.608 | 0.639 | 0.667 | 0.656 | 0.617 | 0.642 | 0.558 |\n| Recall    | 0.624 |   0.582 | 0.561 | 0.647 | 0.658 | 0.663 | 0.544 | 0.612 | 0.545 |\n| F1 Score  | 0.626 | 0.564   | 0.542 | 0.628 | 0.661 | 0.658 | 0.502 | 0.613 | 0.541 |\n\nHatexplain:\n\n\n|           | nd_ne | nd_eo | nd_ei | d_eo  | d_ei  | d_ne  | nd_to | d_to  | d_ti  |\n| --------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy  | 0.600 | 0.542 | 0.604 | 0.569 | 0.535 | 0.506 | 0.641 | 0.618 | 0.624 |\n| Precision | 0.701 | 0.634 | 0.624 | 0.568 | 0.688 | 0.666 | 0.702 | 0.605 | 0.661 |\n| Recall    | 0.653 | 0.596 | 0.624 | 0.570 | 0.603 | 0.578 | 0.566 | 0.606 | 0.654 |\n| F1 Score  | 0.589 | 0.526 | 0.604 | 0.565 | 0.505 | 0.465 | 0.517 | 0.606 | 0.623 |\n\n\n2. The paper does not provide a detailed explanation of the methodology used to select the datasets and the evaluation metrics. This may raise questions about the validity and reliability of the results.\n\nResponse:- As mentioned in the previous answer, the choice of the datasets is driven by the presence of rationales/explanations. The datasets that we have chosen are the only benchmark ones known to us that have rationales/explanations along with class labels. Further, these datasets are popular and well-cited in the literature. We selected the evaluation metrics from Table 6 in HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection  (https://arxiv.org/abs/2012.10289) and  Latent Hatred: A Benchmark for Understanding Implicit Hate Speech (https://arxiv.org/abs/2109.05322) papers.\n\n3. The paper does not provide a clear definition of hate speech, which may lead to ambiguity in the interpretation of the results.\n\nResponse:- The definitions of implicit hate speech, hate speech, offensive speech, and toxic speech are verbatim adapted from the papers introducing these datasets as mentioned in Section 3: Datasets and metrics. Thus there is no one single definition and it (slightly) varies as per the authors introducing the three datasets. We shall clarify this in the revised version.\n\n4. The paper does not consider other valid baselines and SoTA models while reporting the results. Some baselines to include could be: BERT-HateXplain (LIME and Attention) by Binny et al., AAAI 2021 and BERT-RP/ BERT-MRP by Kim et al., COLING 2022.\n\nResponse:- As per your suggestion, as an additional baseline, we considered the BERT-HateXplain and ran the model on the Toxic spans and Implicit hate dataset. We already have the results for the HateXplain dataset taken from the original paper. These results are presented below and will be added in the revised manuscript.\n\n| Dataset              | F1-Score | Precision | Recall |\n| -------------------- | -------- | --------- | ------ |\n| Hatexplain           | 0.698    | 0.697     | NA     |\n| Implicit Hate (full) | 0.54     | 0.64      | 0.47   |\n| Toxic Span           | 0.31     | 0.79      | 0.20   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important and timely issue of detecting hate speech using large language models, which is relevant to both academia and industry.\n2. The paper presents a comprehensive analysis of the performance of two LLMs (GPT-3.5 and text-davinci) in detecting hate speech, and explores the impact of additional context on the performance of LLMs, outlining multiple prompt stragies.\n3. The paper identifies the vulnerabilities and limitations of the two LLMs in detecting hate speech, and conduct a detailed error analysis and typology coding to find ‘jail-break’ points where these models are vulnerable.\n4. The paper provides valuable insights and recommendations for future research in this area.", "reasons_to_reject": "1. While the paper addresses an important issue, the scope of the study is limited to only two large language models and three datasets. This may not be sufficient to draw generalizable conclusions about the performance of LLMs in detecting hate speech.\n2. The paper does not provide a detailed explanation of the methodology used to select the datasets and the evaluation metrics. This may raise questions about the validity and reliability of the results.\n3. The paper does not provide a clear definition of hate speech, which may lead to ambiguity in the interpretation of the results.\n4. The paper does not consider other valid baselines and SoTA models while reporting the results. Some baselines to include could be: BERT-HateXplain (LIME and Attention) by Binny et al., AAAI 2021 and BERT-RP/ BERT-MRP by Kim et al., COLING 2022.\n6. The presentation of the paper is generally well-organized, but there are some areas where the structure could be improved. For example, the introduction could provide more context and motivation for the study, and the methodology section could provide more detail on the selection of datasets and evaluation metrics.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. The paper is about probing large language models (LLMs) for hate speech detection.\n2. The main contributions of the paper are: (1) analyzing the performance of LLMs in detecting hate speech, (2) exploring the impact of additional context (explanations and target community) on the performance of LLMs, (3) identifying the vulnerabilities and limitations of LLMs in detecting hate speech, and (4) proposing ethical considerations for the use of LLMs in hate speech detection.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The writing style of the paper is generally clear and concise, but the paper could benefit from more plain language explanations and clearer definitions of technical terms.\n2. There are some typos and grammatical errors throughout the paper, which can be distracting and affect the readability of the paper. The authors should carefully proofread the paper to correct these errors.\n3. The paper includes several figures and tables, but some of them are difficult to read and interpret. The authors should consider revising these figures and tables to make them more clear and informative.\n4. Overall, while the paper makes a valuable contribution to the field of hate speech detection using large language models, there are several areas where the writing style, typos, and presentation could be improved to enhance the clarity and readability of the paper."}}, {"review_id": "DiXjkzAsdr", "reviewer": "Reviewer_UTwD", "summary": "", "questions": "1. For analysing typology of attack, why was the explanation model picked over other?\n2. If the prompting is done in a CoT manner rather than a single prompt, do the results still remain the same?\n3. For the case where inclusion of definitions increases the error, could it be cause the model then starts looking for hate where non exists? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. While the introduction of typology for adversarial/jailbreak techniques to bypass hate speech is a good starting point, the authors have not shared the details on how the typology was developed. The definitions and scope of these definitions, annotation guidelines, and agreement scores obtained during annotation must be included. Adapting and extending the typology for future use cases will be challenging without these details.\n\nResponse:- To develop the typology of adversarial/jailbreak techniques, we first selected the cases where the models produced erroneous outputs and then selected 80 low-scoring data points according to the bleu/bleurt score measure over the explanation generated by these models. Then, we applied Latent Dirichlet Allocation (LDA) to this subset of low-scoring data points. The number of topics in each case is determined by setting K=3, a value chosen based on the coherence score. Next we manually examine the top 10 most probable words encapsulated within each topic cluster. In order to obtain a semantic clarity upon the derived topics, a meticulous manual annotation process was undertaken. This annotation task was conducted by three domain experts, and notably, unanimous consensus was reached among all annotators regarding the semantic interpretation and nomenclature of each delineated topic cluster. This noteworthy unanimity among annotators is pivotal and will be explicitly elucidated in the forthcoming revised iteration of this manuscript.\n\n2. The paper needs significant reformatting--> expand on the error analysis and discussion of the adversarial attacks and the paper title and abstract emphasis on the \"vulnerability aspect\". Similarly, there needs to be more details about the hyperparameters used (default or otherwise), making reproducibility challenging\n\nResponse:- We have detailed confusion matrices for each dataset (see below), each prompt type, and each model used. We shall report all of these in the revised version.\n\n3. The exact motivation and application of the prompts where the target and explanation information are given in the input is unclear. In the real world, such information is not available initially; therefore, the use case and the promising results thus obtained from this prompt design are not helpful in the real world. This is a significant concern for me.\n\nResponse:- The primary objective of this study was to conduct an exhaustive analysis of the efficacy of distinct prompt injections and their inherent susceptibilities. Given that a multitude of datasets encompass details pertaining to explanations and targets, it was imperative not to disregard a subset of prompt injections wherein these specific details were overtly incorporated within the prompts. Consequently, any enhancement in performance observed in the context of these prompt injections is regarded as an ancillary outcome, albeit one that we deemed essential to highlight.\n\n4. The authors should also include significant tests to corroborate the results in Tables 2,3,4. Further, the justification for employing a smaller subset from the three datasets must be provided. For example, there are multiple types of implicit hate in the implicit hate dataset. Were the samples stratified based on these labels or only on the first level of labels (implicit, explicit, and non-hate)? It will be better if multiple subsets or the whole dataset is considered for analysis.\n\nResponse:- Please note that performance enhancement is not an objective of this paper. Rather our main objective hee was to AUDIT (https://hci.stanford.edu/publications/2021/FnT_AuditingAlgorithms.pdf) SOTA LLMs against various prompt injections. Nevertheless, as per your suggestion, we compared the results of the vanilla model for each dataset and the best prompt with the model using the Mann-Whitney U test. We find that the results are significant. We present the p-values for each. \n\t\t\nChatgpt-HX : p-value=0.0042 (vanilla vs target-at-in)\n\nChatgpt-IH : p-value=0 (vanilla vs target-at-in)\n\nChatgpt-TS : p-value=0.00088 (vanilla vs explain-at-in)\n\nDaVinci-HX : p-value=9.946e-11 (vanilla vs target-at-in)\n\nDaVinci-IH : p-value=4.441e-16 (vanilla vs target-at-in)\n\nDaVinci-TS : p-value=0.01524 (vanilla vs def-at-in)\n\n5. The subset of the dataset was selected using the samples stratified based on the first level of labels. Going further below made the exploration space sparse.\t\n\nFurther, existing benchmark results on transformer based hate speech detection system or whatever is the SOTA for the respective datasets. This will help in understanding the delta improvement in performance that updated GPT systems. This will also help error analysis determine which kind of hate speech is now easier to classify (i,e misclassified by existing baselines but rectified by SOTA GPT systems), vs the ones that still remain challenging. These challenging samples can be employed to form a more concrete set of jailbreak samples.\n\nResponse:- BERT-HateXplain, a state-of-the-art model has performed better than gpt-3.5 (34.23%) and text-davinci (22.46%) on HateXplain dataset, but it hasn’t performed as well when we inference upon other datasets. Once again, our objective was not to explicitly demonstrate delta improvements of LLMs but to perform an audit study to understand the capabilities of these models.\n\n6. Before starting with sending the definitions along with text, it would be better if we first establish the definitions that such systems can generate. This will help establish what these LLM models in general consider hateful.\n\t\nResponse:- The definitions provided by the LLMs on hate speech or offensive speech, or implicit hate speeches cannot be used to investigate the efficacy of the LLMs.The definitions provided by the authors of the papers containing the datasets are used for the annotations of the data. Hence it will be erroneous to use any other definition other than the verbatim definitions as given in the papers referenced for the datasets.\n\n7. An error analysis on the generated explanation/target group is missing and vaguely discussed. How much of these results are coherent vs hallucinated. Are there any target groups that are less accurate detected. Are the explanations coherent and can they be utilised for content moderation?\n\t\n\tResponse:- Please find below the confusion matrices for the vanilla vs best prompts for each model and each dataset combination. We shall add these results in the revised version of the manuscript.\n\nChatGPT-IH-Vanilla:\n|               | Not-Hate | Explicit-Hate | Implicit-Hate |\n| ------------- | -------- | ------------- | ------------- |\n| Not-Hate      | 358      | 273           | 698           |\n| Explicit-Hate | 0        | 94            | 14            |\n| Implicit-Hate | 27       | 394           | 289           |\n\nChatGPT-IH-Target-in: (improvement in classification of the Not_hate class)\n|               | Not-Hate | Implicit-Hate |\n| ------------- | -------- | ------------- |\n| Not-Hate      | 602      | 727           |\n| Implicit-Hate | 67       | 642           |\n\nChatGPT-HX-Vanilla:\n|            | Normal | Hatespeech | Offensive |\n| ---------- | ------ | ---------- | --------- |\n| Normal     | 192    | 388        | 202       |\n| Hatespeech | 2      | 575        | 17        |\n| Offensive  | 19     | 433        | 96        |\n\nChatGPT-HX-Target-in: (improvement in classification of the Offensive class)\n|            | Normal | Hatespeech | Offensive |\n| ---------- | ------ | ---------- | --------- |\n| Normal     | 231    | 186        | 365       |\n| Hatespeech | 1      | 509        | 84        |\n| Offensive  | 40     | 221        | 287       |\n\nChatGPT-TS-vanilla:\n|           | Non-Toxic | Toxic |\n| --------- | --------- | ----- |\n| Non-Toxic | 473       | 527   |\n| Toxic     | 92        | 908   |\n\nChatGPT-TS-Explan-in: (improvement on both classes)\n|           | Non-Toxic | Toxic |\n| --------- | --------- | ----- |\n| Non-Toxic | 491       | 509   |\n| Toxic     | 89        | 911   |\n\nDavinci-IH-vanilla:\n|               | Not-Hate | Explicit-Hate | Implicit-Hate |\n| ------------- | -------- | ------------- | ------------- |\n| Not-Hate      | 994      | 50            | 285           |\n| Explicit-Hate | 317      | 46            | 347           |\n| Implicit-Hate | 15       | 7             | 86            |\n\nDavinci-IH-target-in: (improvement on both classes)\n|               | Not-Hate | Implicit-Hate |\n| ------------- | -------- | ------------- |\n| Not-Hate      | 1111     | 218           |\n| Implicit-Hate | 340      | 369           |\n\nDavinci-HX-vanilla:\n|            | Normal | Hatespeech | Offensive |\n| ---------- | ------ | ---------- | --------- |\n| Normal     | 418    | 35         | 39        |\n| Hatespeech | 47     | 114        | 433       |\n| Offensive  | 145    | 30         | 373       |\n\nDavinci-HX-target-in: (improvement on the Hatespeech class)\n|            | Normal | Hatespeech | Offensive |\n| ---------- | ------ | ---------- | --------- |\n| Normal     | 418    | 28         | 275       |\n| Hatespeech | 48     | 283        | 263       |\n| Offensive  | 162    | 48         | 338       |\n\nDavinci-TS-vanilla:\n|           | Non-Toxic | Toxic |\n| --------- | --------- | ----- |\n| Non-Toxic | 796       | 204   |\n| Toxic     | 381       | 619   |\n\nDavinci-TS-Def-in:( improvement on the Toxic class)\n|           | Non-Toxic | Toxic |\n| --------- | --------- | ----- |\n| Non-Toxic | 768       | 232   |\n| Toxic     | 333       | 667   |\n\n\nQuestions\n\n1. For analysing the typology of attack, why was the explanation model picked over other?\n\nWe require cases in which the model makes incorrect classifications and provides poor explanations for its classification decisions. This can be achieved by instructing the model to generate explanations, which can then be compared against the ground truth explanations. This comparative analysis allows for a quantitative assessment of the quality of the generated explanations; hence, the explanation model was chosen.\n\n2. If the prompting is done in a CoT manner rather than a single prompt, do the results still remain the same?\n\nWhile we worked for the fully zero-shot setting, your suggestion of presenting the prompts in a CoT fashion is interesting and we look forward to taking this up as a future work.\n\n3. For the case where inclusion of definitions increases the error, could it be cause the model then starts looking for hate where non exists\n\nSince it is a black box model, it is difficult to comment on the exact workings of the model. Intuitively, we also feel that something similar should be taking place.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The study encompass 3 datasets and 6 prompt designs which appears to be a good starting point for analysing SOTA LLMs for hate speech detection.\n2. The authors highlight the kind of adversarial attacks where the LLM systems fail to detect hate.", "reasons_to_reject": "1. While the introduction of typology for adversarial/jailbreak techniques to bypass hate speech is a good starting point, the authors have not shared the details on how the typology was developed. The definitions and scope of these definitions, annotation guidelines, and agreement scores obtained during annotation must be included. Adapting and extending the typology for future use cases will be challenging without these details.\n2. The paper needs significant reformatting--> expand on the error analysis and discussion of the adversarial attacks and the paper title and abstract emphasis on the \"vulnerability aspect\" as well. Similarly, there needs to be more details about the hyper parameters used (default or otherwise), making reproducibility challenging.\n3. The exact motivation and application of the prompts where the target and explanation information are given in the input is unclear. In the real world, such information is not available initially; therefore, the use case and the promising results thus obtained from this prompt design are not helpful in the real world. This is a significant concern for me.\n4. The authors should also include significant tests to corroborate the results in Tables 2,3,4. Further, the justification for employing a smaller subset from the three datasets needs to be provided. For example, there are multiple types of implicit hate in the implicit hate dataset. Were the samples stratified based on these labels or only on the first level of labels (implicit, explicit, and non-hate)? It will be better if multiple subsets or the whole dataset is considered for analysis. \n5. Further, existing benchmark results on transformer based hate speech detection system or whatever is the SOTA for the respective datasets. This will help in understanding the delta improvement in performance that updated GPT systems. This will also help error analysis determine which kind of hate speech is now easier to classify (i,e misclassified by existing baselines but rectified by SOTA GPT systems), vs the ones that still remain challenging. These challenging samples can be employed to form a more concrete set of jailbreak samples.\n6. Before starting with sending the definitions along with text, it would be better if we first establish the definitions that such systems can generate. This will help establish what these LLM models in general consider hateful. \n7. An error analysis on the generated explanation/target group is missing and vaguely discussed. How much of these results are coherent vs hallucinated. Are they any target groups that are less accurate detected. Are the explanations coherent and can they be utilised for content moderation?", "questions_to_authors": "1. For analysing typology of attack, why was the explanation model picked over other?\n2. If the prompting is done in a CoT manner rather than a single prompt, do the results still remain the same?\n3. For the case where inclusion of definitions increases the error, could it be cause the model then starts looking for hate where non exists? \n", "paper_topic_and_main_contributions": "The paper performs various probing experiments to analyze how well the latest LLM systems work regarding hate speech detection.  The authors then introduce a typology for the kind of texts that is challenging to classify and can be employed for adversarial purposes.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ov4TJjggWw", "reviewer": "Reviewer_Dc6M", "summary": "", "questions": "- It would be valuable for the authors to explain the decision-making process behind the use of the prompting system and LLMs models. This would help researchers to understand the benefits and limitations of these models and potentially apply them to their own research.\n- While the authors have provided a comprehensive overview of the hate speech detection task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results.\n- Data preprocessing techniques are not described despite its importance.\n- Are the results reported in this paper statistically significant?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. In introduction, authors said \"In terms of vanilla prompts (that is case (a)), we find that text-davinci-003 is better than gpt-3.5-turbo-0301 although the latter is a more recent version\". Authors are confused with problem definition and results. The introduction is for defining the problem, not demonstrating the results\n\nResponse:- Thanks. We shall clearly delineate the introduction more in the revised version. We shall have a separate paragraph outlining the contributions followed by just the highlights of the results.\n\n2. While the authors have discussed the results of their study, they have not provided details on the data preprocessing techniques used. As data preprocessing is a crucial step in natural language processing tasks, it is important to include this information to ensure the reproducibility of the study\n\nResponse:- Our datasets are directly taken from the three papers  Hatexplain: A Benchmark Dataset for Explainable Hate Speech Detection  (https://arxiv.org/abs/2012.10289), Latent Hatred: A Benchmark for Understanding Implicit Hate Speech (https://arxiv.org/abs/2109.05322) and ToxiSpanSE: An Explainable Toxicity Detection in Code Review Comments (https://arxiv.org/abs/2307.03386) where the authors have described the pre-processing in extensive detail. Only for the Implicit Hate dataset, we selected the test set in a stratified fashion. Everything else is verbatim, the same as the original source papers. We shall clarify this in the revised version.\n\n3. The manuscript makes no mention of hyper-parameters fine-tuning or other model configures. Please explain how these parameters are optimized.\n\nResponse:- Please note that the experiments were run on OpenAI black box models where it is not possible to fine-tune the hyperparameters. The temperature coefficient was kept at 0 all through.\n\n4. While the authors have provided a comprehensive overview of the hate speech detection task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results.\n\nResponse:- Thanks. Indeed, this is exactly why we conducted the typology induction, which precisely underpins the linguistic reasons for the failure of the LLMs. The entire typology induction idea was to have linguistic backing to the theoretical foundations of the LLMs. We shall clarify this in the revised version.\n\nQuestions\n1. It would be valuable for the authors to explain the decision-making process behind the use of the prompting system and LLMs models. This would help researchers to understand the benefits and limitations of these models and potentially apply them to their own research.\n\nResponse:- Explaining the precise decision-making mechanism of the employed LLMs is unattainable due to their inherently opaque nature (black box). The selection of prompts, however, is underpinned by meticulous scrutiny of the datasets and the prevalent concerns within the hate speech research domain. The typology section introduces an innovative approach to comprehending LLMs' vulnerabilities, even without the insights into the internal functioning of these LLMs.\n\n2. While the authors have provided a comprehensive overview of the hate speech detection task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results\n\nResponse:- Thanks. Indeed, this is exactly why we conducted the typology induction, which precisely underpins the linguistic reasons for the failure of the LLMs. The entire typology induction idea was to have linguistic backing to the theoretical foundations of the LLMs. We shall clarify this in the revised version.\n\n3. Data preprocessing techniques are not described despite their importance.\n\nResponse:- Our datasets are directly taken from the three papers  Hatexplain: A Benchmark Dataset for Explainable Hate Speech Detection  (https://arxiv.org/abs/2012.10289), Latent Hatred: A Benchmark for Understanding Implicit Hate Speech (https://arxiv.org/abs/2109.05322) and ToxiSpanSE: An Explainable Toxicity Detection in Code Review Comments (https://arxiv.org/abs/2307.03386)where the authors have described the pre-processing in details. Only for the Implicit Hate dataset, we selected the test set in a stratified fashion. Everything else is verbatim, the same as the original source papers. We shall clarify this in the revised version.\n\n4. Are the results reported in this paper statistically significant?\n\nResponse:- Please note that performance enhancement is not an objective of this paper. Rather our main objective here was to AUDIT (https://hci.stanford.edu/publications/2021/FnT_AuditingAlgorithms.pdf) SOTA LLMs against various prompt injections. Nevertheless, as per your suggestion, we compared the results of the vanilla model for each dataset and the best prompt with the model using the Mann-Whitney U test. We find that the results are significant. We present the p-values for each. \n \t\t\nChatgpt-HX : p-value=0.0042 (vanilla vs target-at-in)\n\nChatgpt-IH : p-value=0 (vanilla vs target-at-in)\n\nChatgpt-TS : p-value=0.00088 (vanilla vs explain-at-in)\n\nDaVinci-HX : p-value=9.946e-11 (vanilla vs target-at-in)\n\nDaVinci-IH : p-value=4.441e-16 (vanilla vs target-at-in)\n\nDaVinci-TS : p-value=0.01524 (vanilla vs def-at-in)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper exhibits good organization and presentation overall.\n- It focuses on specific benchmark datasets related to a particular task.\n- The process of prompting configurations is thoroughly explained.\n- The proposed approach offers a notably simpler and more efficient method for hate speech detection.\n- The reported performance of the prompting system and various commonly used LLMs models on this task is valuable for future benchmarking endeavors.", "reasons_to_reject": "- In introduction, authors said \"In terms of vanilla prompts (that is case (a)), we find that text-davinci-003 is better than gpt-3.5-turbo-0301 although the latter is a more recent version\". Authors are confused with problem definition and results. The introduction is for defining the problem, not demonstrating the results. \n- While the authors have discussed the results of their study, they have not provided details on the data preprocessing techniques used. As data preprocessing is a crucial step in natural language processing tasks, it is important to include this information to ensure the reproducibility of the study.\n- The manuscript makes no mention of hyper-parameters fine-tuning or other model configures. Please explain how these parameters are optimized.\n- While the authors have provided a comprehensive overview of the hate speech detection task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results.\n\n", "questions_to_authors": "- It would be valuable for the authors to explain the decision-making process behind the use of the prompting system and LLMs models. This would help researchers to understand the benefits and limitations of these models and potentially apply them to their own research.\n- While the authors have provided a comprehensive overview of the hate speech detection task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results.\n- Data preprocessing techniques are not described despite its importance.\n- Are the results reported in this paper statistically significant?", "paper_topic_and_main_contributions": "This paper presents the authors' experience in utilizing various algorithms for hate speech detection. Through an extensive study of two LLMs (GPT 3.5 and text-davinci) across three datasets with different prompt setups, the authors reveal the challenges faced by these advanced models in detecting hate speech in their default configurations. They propose individual prompt strategies that enhance the models' performance, but find limited improvement when combining these strategies. Additionally, the authors conduct a detailed error analysis and typology coding, identifying vulnerable points ('jail-break' points) in the models' performance. This paper offers valuable insights for readers interested in the field of hate speech detection and the capabilities of language models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Please avoid using vague terms like \"many\", \"few\" to ensure clarity and transparent in the writing.\n- The authors should present the results in Tables 2, 3, 4 in a more comprehensible method. It is possible to optimise them by combining them into one.\n- Authors must unify the numerical formatting (thousands separator, decimal place, alignment).\n- The authors should focus on clarifying the problem, motivation, and contribution of the research in the Introduction rather than going into too much detail about the method, which produces overlap with the content in section 6 and makes the content confused."}}]

Author rebuttal:

Meta-review: This paper delves into the efficacy of large language models (LLMs) in hate speech detection. Through a comprehensive analysis of two prominent LLMs (GPT 3.5 and text-davinci) across distinct datasets, the authors evaluate their performance and the impact of various context cues, such as explanations and targeted communities. The research unveils the models' inherent challenges in default configurations and introduces tailored prompt strategies for performance optimization. 

The advantages of the paper are: 1) addresses an important and timely issue of detecting hate speech using large language models. 2) Presented comprehensive analysis over two commercially available LLMs 3) Identifies the vulnerabilities and limitations such as vulnerability to adversarial attacks. 4) Provide a good benchmark for follow up research

Some concerns were also raised regarding 1) The definition of hate speech is not clear and result in ambiguity in the interpretation of the results 2) the technical detail of the approach is not clearly described (e.g. data preprocessing techniques, hyper-parameters ) 3) The experiment is only limited to two LLMs that form a same family (GPT3) ); In addition 4) the paper needs significant reformatting
Decision: Accept-Findings