K-HATERS: A Hate Speech Detection Corpus in Korean with Target-Specific Ratings
Authors: Chaewon Park
Soohwan Kim
Kyubyong Park
Kunwoo Park
Venue: EMNLP 2023 Findings (2023)
Keywords: Hate speech
Offensive language
Dataset construction
Fairness
Explainability

Abstract: Numerous datasets have been proposed to combat the spread of online hate. Despite these efforts, a majority of these resources are English-centric, primarily focusing on overt forms of hate. This research gap calls for developing high-quality corpora in diverse languages that also encapsulate more subtle hate expressions. This study introduces K-HATERS, a new corpus for hate speech detection in Korean, comprising approximately 192K news comments with target-specific offensiveness ratings. This resource is the largest offensive language corpus in Korean and is the first to offer target-specific ratings on a three-point Likert scale, enabling the detection of hate expressions in Korean across varying degrees of offensiveness. We conduct experiments showing the effectiveness of the proposed corpus, including a comparison with existing datasets. Additionally, to address potential noise and bias in human annotations, we explore a novel idea of adopting the Cognitive Reflection Test, which is widely used in social science for assessing an individual's cognitive ability, as a proxy of labeling quality. Findings indicate that annotations from individuals with the lowest test scores tend to yield detection models that make biased predictions toward specific target groups and are less accurate. This study contributes to the NLP research on hate speech detection and resource construction. The code and dataset can be accessed at https://github.com/ssu-humane/K-HATERS.

K-HATERS:
A Hate Speech Detection Corpus in Korean with Target-Specifie Ratings
Chaewon Park Soohwan Kim e
Kyubyong Park
Kunwoo Park $
SScooo of AI Convergence, Soongsil University TUNNB
$ Department of Intelligent Semiconductors, Soongsil University kunwoo, warklssu. ac.
Abstract
the negative impacts of hate speech. To curb its spread in online environments, previous NLP re- search has developed ML-based detection mod- els. Labeled resources play critical role in build- ing supervised detection models; thus, a various form of offensive language and hate speech datasets have been proposed in the NLP community, and they facilitated the development of detection meth- ods (Mathew et al., 2021; Kim et al., 2022b).
Numerous datasets have been proposed to com- bat the spread of online hate. Despite these ef- forts, a majority of these resources are English- centric, primarily focusing on overt forms of hate. This research gap calls for developing high-quality corpora in diverse languages that also encapsulate more subtle hate expressions. This study introduces K-HATERS, new cor- pus for hate speech detection in Korcan, com- prising approximately 192K news comments with target-specific offensiveness ratings. This resource the largest offensive language cor- pus in Korean and is the first offer target- specific ratings on a three-point Likert scale, enabling the detection of hate expressions in Korean across varying degrees of offensiveness. We conduct experiments showing the effective- ness of the proposed corpus, including a com- parison with existing datasets. Additionally, to address potential noise and bias human an- notations, explore novel idea of adopting the Cognitive Reflection Test, which is widely used social science for assessing an indi- vidual's cognitive ability, proxy of label- ing quality. Findings indicate that annotations from individuals with the lowest test scores tend to yield detection models that make bi- ased predictions toward specific target groups and are less accurate. This study contributes to the NLP research on hate speech detec- tion and resource construction. The code and dataset can be accessed https://githuu com/ssu- huwaee/ -HATRSS.
Despite the advancement in the online hate speech research, there are several rooms for im- provement. (1) Language: Most of the existing datasets are limited to English and high-resourced languages (Davidson et al., 2017; Founta et al., 2018). It is critical to construct a non-English dataset for the development of detection models that are aware of culture-specific expressions. (2) Implicitness: The majority of previous research conducted on this subject has primarily concen- trated on explicit manifestations of offensive lan- guage. A few studies investigated the implicit form of hateful content (EISherief al., 2021; Ocampo et al., 2023; Hartvigsen et al., 2022), most of which targeted English. (3) Biased and noisy labels: Manual annotation is predominant approach for constructing labeled data. However, the labeling process could reflect inherent biases of human an- notators, and is hard control the label qual- ity (Sap et al., 2019; Sachdeva et al., 2022). Train- ing a model on such a noisy and biased dataset can amplify biases by inference (Shah et al., 2020).
1 Introduction
This paper aims to fill the three research gaps by presenting a new HATE speech detection corpus in Korean with target-specific RatingS, named K- HATERS. (1) Among Korean offensive language corpora, this resource represents the most extensive data collection date, comprising 192,158 news comments and surpassing the size of the previously largest corpus by 1.75 times (Lee et al., 2022). (2) We design a rating scheme for target-specific and fine-graine grained offensiveness on a three-point Likert
Hate speech is defined by "offensive discourse tar- geting a group or an individual based on inherent characteristics (such as race, religion or gender)"! This is critical problem in society that can aggra- vate polarization and threaten democracy (Lorenz- Spreen al., 2023). The massive volume of com- munication in the web and social media amplifies

---

Target label
Have rilonalc
Taget ratiaal
Mull scale taings
Dunset
Size
Source
Labeling scherre
REFS (Moon 1000))
Navcr
TI gender, ollers, more
Youtuse Naver
offeasive
KcLD doong et 2022)
ustageted, individudl, otter, sacuua oriastation, clinicity Austiooalltyy
pottical aftilition religion, niscellarasus
K-MHHS (Loc 2002)
109K
Naver
hate poliiccs, orgaa, appescazee, age: gouder religion, noc. peolusity
speech
A affrecsse likely offenenee not offensive
KODORI (Park 2023)
A8K
Orline commantioo
avaliable
lexek2 loxd-) offensine, noenal
Oues
192K Navce nows
T) gender, Jace (oigin), celigion, polities, job, disaility, individaal, others
Table I: List of offensive language corpora Korean (A: Abusive language categories, T: Target categories)
scale. It distinguishes between the explicit form of offensiveness (e.g., toxic expression) and the im-
that annotations from individuals with the low- est test scores can yield detection models that are both less accurate and unfairer than those derived from the control group.
plicit form (e.g., sarcasm, stereotypes), facilitating nuanced understanding of online hate. (3) We borrowed the idea of psychology and social sci- ence for improving labeling quality in the dataset construction. In particular, we propose using the cognitive reflection test (CRT) (Frederick, 2005), which was designed to assess a cognitive ability to suppress an intuitive wrong answer in favor of a reflective correct answer. Previous research in social science showed that the score is correlated to one's likelihood of following implicit biases and heuristics (Roozenbeek et al., 2022; Pennycook and Rand, 2021; Toplak al., 201 I). Our research hy- pothesis is that the labels given by individuals with a low CRT score may be noisy and biased. To test the hypothesis and support the effectiveness of the dataset, we conducted experiments with varying ar- chitectures and metrics that cover human-centeree desiderata of hate speech classifiers, such as detec- tion performance, fairness (Ramponi and Tonelli, 2022), and explainability (Mathew et al., 2021). The key findings are summarized as follows:
4. A case study shows that hate speech is preva- lent in the news comments posted on the poli- tics, social, and world news section in major news portal in South Korea. The hatred was mainly targeted at the groups of the attributes related politics and regions.
Related works
Early studies used the binary label for classifying hate and normal text (Djuric et al., 2015; Badjatiya et al., 2017). Davidson et al. (2017) proposed using taxonomy capturing middle ground between hate speech and normal text. An abusive com- ment that is not hate speech is generally called offensive. later study aimed to capture target- specific hate such as racism and sexism (Founta et al., 2018). Most research focused on overt forms of hate speech, but explicit hate is more easily iden- tifiable, c.g., by lcxicon- based methods (Davidson et al., 2017). Recent research focused on implicit- ness of hate speech and proposed new datasets (Ju- rgens et al., 2019; ElSherief et al., 2021; Wiegand et al., 2021: Ocampo et al.. 2023; Nejadgholi al., 2022; Hartvigsen al., 2022). A study developed a taxonomy of implicit hate and provided a labeled dataset (FISherief et al., 2021), but its annotation scheme not generalizable in the Korean context (e.g., White Grievance).
1.
We present large-scale offensive language corpus of 192,158 samples along with target- specific offensiveness ratings on a 3- 3-point Lik- ert scale. The corpus provides offensiveness and target rationales text spans.
2.
We propose label transformation method for the unified modeling of abusive language cat- egories along with multi-label target predic- tion. This method can be applied existing corpora.
In addition to prediction accuracy, recent stud- ies focuses on human-centered desiderata of hate speech classifiers, such as explainability and fair- ness. Mathew al. (2021) proposed dataset that includes hate speech labels and annotators' rationale as a highlighted span. A recent study de- veloped a pretraining method of masked rationale
3. We investigate a novel approach of using the Cognitive Reflection Test to approximate la- beling quality. Experimental results reveal

---

prediction (Kim al., 2022a). To promote the de- velopment of fair detection models, Röttger al. (2021) introduced a functional test that measures a predictive bias of hate speech detection models. In the context of the Korean language, a handful num- ber of resources is available, as shown in Table I. BEEP! is the first dataset shared in the NLP com- munity (Moon et al., 2020). The corpus consists of 9,381 news comments labeled either hate, offen- sive, or none, along with the bias label. Jeong et al. (2022) introduced KoLD by adopting hierarchical labeling scheme. The dataset contains 40.4K online text along with labels on whether is offensive and what is the target. Put it simply, KoLD employs a binary rating whether hate exists against target group. Lee et al. (2022) proposed K-MHaS, dataset of 109K comments of which the hate tar- get is annotated by a multi-label scheme. As an alternative such a crawling- baaseedn--aedd data collection, a recent study introduced method that lets crowd- workers generate expressions (Yang et al., 2022). We introduce the largest and the first corpus along with target-specific offensiveness ratings, which contributes to the detection of non-explicit hate in Korean. While recent corpus employed three- class rating for offensiveness (Park al., 2023), it lacks target labels and thus cannot be used to train a model that distinguishes between offensive expressions and hate speech.
Input
[| OICI F/H
$950 A [
(Anywhere in the worlds, idcaless dotard is the biggesi problem)
Offensive?
Label fine-grained ratings
Insut
Sweer wond
Obsoeityy
Treat
-o-
Target?
Label target-specific ratings
Age
Raxaybrigin
Retigin
Poltiis
Job
Disaniity
Indisituol
Otters
Rationale?
Highlight offensiveness and target rationales
24 0,5 Mill
(Anywhrre the worlds, idealess Batard is the biggest problem)
End
Figure I: An illustration of labeling process
sampled from each section. Additionally, we in- cluded 8,367 comments from the BEEP! corpus that were published in the entertainments section, categorized as soft news. We included the dataset for the comparison of labeling schemes. In total, 200,000 news comments prepared for label annotation.
3.2 Crowdsourced annotation
We used CashMission crowdsourcing service in Korea, to label the 200k news comments. Fol- lowing training session for 405 annotators hired by the company, we conducted pilot tests in which ask the workers to label small set of samples based initial guideline. The goal was to help the workers understand the task and for prove the guideline. We had a feedback session with each worker based on the annotation results. The final guideline includes several examples with a desired label and explanation derived from the pilot test.
3 Dataset
3.1 Data collection
im-
We collected comments posted on news articles published via major news portal in South Korea² over the period of July to August 2021. The tar- get articles were collected from the society, world news, and politics sections, which are categorized hard news (Lehman-Will) and Seletzky, 2010). They cover a newsworthy event to be of local, regional, national, or international significance, where active discussions and hate speech likely oc- cur. We obtained 191,633 comments from 52,590 news articles, with an equal distribution of com- ments sourced from each section. To include substantial amount of hateful comments in our la- beled corpus, we trained binary classifier on the BEEP! corpus (Moon al., 2020) and sampled 95,816 comments from those with sigmoid output 0.5 or higher. 31,939 comments were randomly
Labeling process Figure 1 illustrates the label- ing process. For each comment, an annotator was asked to label eleven offensiveness ratings. These ratings fall into two main categories: target-specific and fine-grained labels. The target-specific ratings measure the intensity of offensive expressions di- rected one of the prominent hate targets in South Korea: gender, age, race (origin), religion, pol- itics, job, and disability. The seven targets are categorized as protected attributes; thus, any offen- sive expressions against them are classified as hate speech (Zhang and Luo, 2019). To differentiate
"news. camer.com
www. wwwhwi cashmission.com

---

Target-specific offensiveness ratings
Fine-grained ratings
Value
GRP
Swear
IND
OTH
Insalt
Obscenity
Gender
Age
Race
Religion
Polities
Job
Disability
werds
187,983
180,184
175,199
191,149
157,461
184,849
190,484
139,935
175,660
39,457
152,927
189,861
185,17
2,295
1.128
9,628
16,720
3,783
1,526
52,223
16,498
75.041
23,074
2,623
3,526
57,660
1,166
Table 2: Label distribution of thirteen rating variables
hate speech from offensive comments directed at in- dividuals or unspecified groups, introduced two binary variables. The first indicates if the comment is targeted an individual, and the second denotes targeting unspecified groups. For simplicity, we refer to the seven targeted groups as GRP, individ- vals as IND, and unspecified targets as OTH. The fine-grained offensiveness ratings offer more de- tailed insight, capturing specific sub-categories of offensiveness, such as insults, swear words, obscen- ities, and threats. We included these four ratings to facilitate a deeper, more nuanced understanding of offensive expressions.
their English translations. These examples demon- strate that rating scheme facilitates understand- ing both the intensity of offensive expressions and their respective targets. The last example highlights that a single text can contain multiple targets.
3.3 Cognitive reflection test
Biased annotators could inadvertently induce their implicit prejudices into the labels (Sap et al., 2019; Davidson al., 2019; Sachdeva et al., 2022). If machine learning models are trained on such bi- ased data, they potentially amplify these biases and generate harm by inference (Shah et al., 2020). To address the low-quality labeling issue and promote fairness in hate detection models, we suggest lever- aging psychological test an indicator of label- ing quality. Specifically, propose the use of the Cognitive Reflection Test (CRT). The CRT was de- veloped to assess an individual's 's cognitive capabil- ity to suppress an intuitive yet incorrect response favor more reflective and accurate one (Freder- ick, 2005). This test is prevalently used across vari- ous disciplines in social science (Roozenbeek et al., 2022; Pennycook and Rand, 2021). Notably, stud- ies have found correlations between CRT scores and implicit biases (Toplak al., 201 Accord- ingly, we the following hypothesis:
Each rating is based on a 3-point Likert scale, ranging from 0 to 2. A value of 0 indicates no de- tected offensiieness. A value of I suggests mildly offensive or potentially offensive expressions, in- cluding sarcasm, stereotypes, prejudice. A value of denotes clearly toxic expression likely noy readers. Besides assigning these scales, anno- tators were directed to highlight specific text spans that influenced their rating decision and the iden- tified target of offensiveness. These highlighted spans are termed "offensiveness rationale" and "tar- get rationale," respectively. The detailed guideline can be found in SA.2.
For the 200,000 comments, we employed rule- based filtering criteria* to identify and remove noisy labels. The final dataset comprises 192,158 comments, which was used for the experiments in the paper. Table presents the label distribu- tion. We observe the prevalence of samples with the value of 1. This finding suggests the impor- tance of the proposed labeling scheme that dis- tinguishes between normal and weakly offensive ratings. Without this multi-point rating approach, comments containing subtle forms of hate speech might be mistakenly labeled as normal without the multi-point rating scheme. We suspect that the con- tent moderation on the platform - whether manual automated might have limited the occurrence of comments rated as 2.
H. The labels annotated by workers with a low CRT score, which implies that paid less attention to the labeling pro- cess, would be biased and noisy.
To evaluate the hypothesis, conducted the cognitive reflection test all the annotators us- ing Korean variant of CRT comprising six ques- tions (Kim, 2020). It details are provided Ta- ble A1. Each participant's 's test score ranges from 0 to 6, where the participants get one point for every correct answer. We report the comparison results among different score groups in Section 6.
3.4 Label transformation
Table 3 displays several examples along with
A straightforward solution for training a detection model using the proposed corpus would be to train
'detailed in $A.3

---

Targer spectitc raings
ineegnnnntt nilngs
Data cuampe
GRP
Socae Cluamity
Gender
Age
Racz
Religion
Polirics Joh Disakiitty
guee one shorld pay for gotting varcinored in Iaper.
B. fIR 21646-11 mit [HH)) (
0
0
0
2
You like Korean gifl, using her meretrual leave on lridky and Monday.
2
2
Asyehrm
biggest prohiem
9
Intrcent Instrilion
Enohihi and companed Koeeans, Baatsoned
M
That is happess
about peligio.
Hey aicha, precstine Ane you gering vaccine for someone
sondarity go!
Table 3: Labeled data examples and their English translations green highlight
offensiveness rationale,
yellow highlight target rationale, lime highlight overlapped rationales)
a multi-label classification model that predict thir- teen rating variables. Here, we propose an alterna- tive modeling strategy based on an unified abusive language labeling scheme.
Data
No
Yes
max(ratings)
We transform the thirteen rating variables into two labels: one represents the four-class abusive language categories (ALC), and the other corre- sponds to multi-label target categories (TGT). The ALC label transformation process is illustrated in Figure 2. According to the community defi- nition (Davidson et al., 2017), ALC broadly com- prises three categories: normal, offensive, and hate.
Normal
max(GRP ratings)
Offensive
Offensiveness rationale
&& max(ratings)
Level-1
Level-2 hate
hate
Figure 2: A flow chart for label transformation
A comment is labeled as normal if all ratings of the comment are 0)
Data
Normal
Offensive
LI hate
1,2 hate
Training
46,643
70,467
19,537
35,511
A comment with offensiveness rating above directed at target group is labeled as hate.
Validation
2,709
4.093
1.135
2,063
Test
2,709
4,093
1,135
2,063
52,061
78,653
21,807
39,637
Total
(27.1%)
(40.9%)
(11.4%)
(20.6%)
If comment has a rating above 0 but is not directed at protected attribute group, it is classified offensive.
Table 4: Label distribution of transformed abusive lan- guage categories across the split data
Furthermore, divide the hate labels into two tiers based on the highest rating value toward the target groups:
rating of 1 indicates milder offensive expres- sions, such sareasm, stereotype, prej- udice, level-1 hate may encompass implicit hate expressions.
A comment is labeled Level-2 hate if its highest rating 2 and has labeled spans for the offensiveness rationale.
We split the dataset of 192,158 samples into 172, 1011111110000000 0,000/100,000 for training, validation, and test purposes, ensuring the transformed label distri- bution is maintained. Table 4 presents the distribu- tion of ALC labels. Over 72% of the dataset con- tains offensive expressions, categorized as either offensive, level-1 hate, or level-2 hate. However, in many cases, annotators either could not identify
A comment deemed Level-1 / hate if the highest rating toward a protected attribute group Additionally, offensive comments without a specified rationale for offensiveness are categorized under level-1 I hate. Given that

---

the target or the comments were not directed at group of protected attributes. As result, they were classified into offensive. Among the 61, 01,444 hate expressions, a substantial portion of the comments, 21,807 comments (or 35.49%), were categorized as level-1 hate.
is minimize LoEr aL where alpha is a hyperparameter.
4.2 Evaluation metrics
We evaluate three desiderata of hate speech detec- tion models.
For TGT, we include target if a corresponding rating is above 0. Thus, the maximum number of target categories comment have is nine. The TGT distribution can be referred Table 2.
(1) Detection performance We use micro and macro FI for measuring how well a detection model performs. The greater value indicates the more accurate prediction. For the evaluation of models that predict thirteen rating variables, we transformed the predicted ratings using the process described in $3.3.
4 Detection experiments
4.1 Methods
We examine BERT fine-tuning methods for the detection of hate and offensive expressions using the transformed labeling scheme and raw rating variables, respectively.
(2) Fairness We adopt the difference in the equal- ized odds ratio (Hardt et al., 2016) for understand- ing how equally model is good at detection cording the hate target. We calculate true posi- tive rate and false positive rate respectively across different target groups and combine the target-wise nine values by the difference between the max- imum and minimum values. The assumption is that fair classifier should have similar error rate across different target groups. We report the maximum value of class-wise odds as unified measurement. The lower value indicates the more fair classifier.
Using transformed labels
H+T a target-aware method that employs target prediction as an auxiliary task. In ad- dition to the classifier head for the 4-way abusive language category classification, the model includes parallel dense layer that pre- dicts nine logit scores, each of which cor- responds multi-label target class (e.g.. age, gender). The hidden layer is 768- dimensional vector. Its training objective to minimize LCE+ + aL where LCE is the cross-entropy loss for the ALC prediction, focal is the focal loss (Lin et al., 2017) for multi-label target classification, and is a hy- perparameter.
(3) Explainability We evaluate the explainability of a classifier by using the two criteria introduced in earlier studies (Mathew et al., 2021): plausibil- ity and faithfulness. plausibility metric aims quantify how plausible a given explanation is humans (DeYoung et al., 2020). A faithfulness metric measures how precisely a given explanation reflects the reasoning process of a target model (Ja- covi and Goldberg, 2020). Considering the nor- malized attention scores over the positions as ex- plainable mechanism of BERT classifier, we adopt the intersection-ovee-unioo (IOU) Fl for measur- ing plausibility. It measures the relative size of the matched tokens between the predicted and truth rationale spans. A span prediction is considered matched if the overlapped ratio with any of the truth span is larger than 0.5. The second explain- ability measurement is faithfulness, which aims to capture to what extent an explanation reflects the reasoning process of model (Mathew al., 2021). We instantiate faithfulness by the comprehensive- ness metric that measures the decreased amount of prediction probability by removing predicted rationales from the text.
H+T+R additionally supervises the attention heads at the last layer by using the rationale spans. The training objective is to minimize LCE + focal Att. where Att is the mean of the cross-entropy loss measured for each token. a and are hyperparameters.
Using rating variables
H+T predicts thirteen rating labels through dense layer. The model is fine-tuned by optimizing the sum of cross-entropy for each label prediction, LCE. Given that the target variables include nine target-specific toxicity labels, we deem the model to be targer-aware.
H+T+R includes the attention supervision as an auxiliary objective. The training objective

---

Detection performance
Fairness
Explainability
Model
Equalized odds
Equalized odds diff. (FPR) I
Micro F1
Macro F1 f
Plausibility 1
Faithfulness
diff. (TPR)
Using transformed labels
H+T H+T+R
0.681
0.611=0.001
0.243=0.006 0.247=0.018
0.31+0.01
0.286+0.0009
0.163==0006
0.675+0.003
0.602=0.006
0.36+0.015
0.341+0.002
0.185=0.011
Using rating variables
H+T H+T+R
0.663+0.001
0.6=0.002
0.316=0.006
0.338+0.017
0.239+0.019 0.351
0.094=0.012
0.663+0.001
0.6=0.001
0.330=0.003
0.338+0.013
0.123=0.007
Table 5: Performance detection models. Mean and standard errors are reported.
4.3
Results
both KoLD and K-MHaS. Since the two datasets do not have weakly offensive rating, the corre- sponding ALC label has three categories: normal, offensive, and hate. We sampled 8,367 instances, which the size of BEEP!, from the training split of each dataset for fair comparison.
Table 5 presents the evaluation results of the four detection models, each differing in learning ob- jectives and input data format. We make three observations. First, the H+T model with the trans- formed label achieved the best detection perfor- mance in terms of both micro and macro Fl scores. Second, the H+T model using transformed labels demonstrated enhanced fairness in its prediction results compared to that trained rating variables. Among the models trained the transformed la- bels, the H+T model obtained the lowest equal- ized odds difference score measured as FPR. This suggests that its predictions vary less across tar- get groups compared to the other models. This observation contradicts the finding of a previous study (Mathew et al., 2021). The H+T and H+T+R models trained on the transformed labels showed similar performance when assessing the fairness metric via TPR. Third, there no clear win- ner for explainability. While the H+T+R model trained on the transformed labels achieved the best results for the faithfulness metric. The H+T+R model trained on rating variables was the best for the plausibility metric. To summarize, our findings support the importance of employing transformed labels to enhance both the accuracy and fairness of hate speech detection. Consequently, we use the H+T model trained on the transformed label for the subsequent experiments.
Figure 3 presents the confusion matrix, with each axis representing the label categories. The models trained BEEP!, KoLD, and K-MHaS achieved macro F1 scores of 0.528, 0.631, and 0,461, re- spectively. Notably, the elassifier trained on BEEP! tends to misclassify offensive-labeled classes as hate labels. In contrast, the classifier trained on K-MHaS demonstrates an opposite trend: fre- quently predicts normal labels for samples labeled as offensive and fails to predict hate labels for the level-1 hate samples. Meanwhile, the KoLD model exhibited predictive accuracy at 0.629, with its per- formance differing by only 0.04 compared the intra-dataset model.
To gain deeper insights into the observed trends, we conducted a principal component analysis us- ing pretrained sentence transformer embedding Figure depicts the distribution of the top two principal components for samples from each class across the four datasets. The results indicate that the labels from KoLD and our dataset are closely aligned. In contrast, K-MHaS and BEEP! exhibit distinct patterns. This divergence could account for the suboptimal performance observed in cross- dataset experiments.
5 Cross-dataset prediction
Can CRT be a proxy of data quality?
This section presents the results of cross-dataset experiment to understand the generalizability of existing and proposed datasets, as summarized in Table 1. We trained model on each dataset and then tested it on the test split of our dataset. To model an ALC category classifier, we applied a label transformation process, described in $3.4, to
This section examines the research hypothesis that labels given by workers with a low CRT score would be noisy and biased. To evaluate the hy- pothesis, we constructed two different training sets: (1) all samples annotated by the individuals with

---

training set: BEEP!
training KoLD
training K.HHaS
norma!
(0.684)
1478
730 [0.2911
10.094]
501
7ormO
[0.693]
1625
634 (0:153)
(0:117)
400
100mM
(0.3801
2612
10 (0.018)
(0.034)
87
Tru abble (Ours)
253
t20 abele Oursi
affersive
10.225)
407
10 10:5133
1200
10.4351
2318
offensive
(0.220))
552
10.6177
2563
(0.285)
975
100
offons
3085
05.45001
00: (0.7541
413
(0 (0.227)
585
brvel-3
Tate
121
056)
[0.125|
315
10.1311
639
1019
Tabe
(0.055)
132.
(0.094)
348
(0.1911
655
1800
hake
(0.096)
602
(0.031)
17
(0.177)
456
lrvel-2
179
1809
608
1396
level-2
505
108
1450
Tate
(0.035)
10.0711
(0.340]
rate
10.024)
(0.146)
(0.4071
nate
(0.073)
(0.197)
(0. (0.562)
offerdive predicted labels
offersive predicted labals
nommal
offershie predicted labals
Figure 3: Cross-dataset prediction performance
offensive
0.51
KOLD
0.600
0.50
L2 hate
Ours
of
0.48
hate offensive
offensive
12)
(
F1
F1
hormal
Accuracy 1
Macro F1
Plausibiliity ltylyyy 1
65
hate
none
0.00
normal
0.50
normal
0:40
0.04
PC1
P1 Equalized odds diff. (FPR) 4
d)
d9
Figure 4: Principal component analysis of each label group across the datasets
Equaliizd odds diff. (TPR)
Faithfulness
Figure Performance difference between two models using different CRT score group annotations training
score and (2) the rest samples. We randomly sampled the (2) dataset to the size of (1). By train- ing an H+T model on each training set, we mea- sured the six metrics of detection performance, fair- ness, and explainability. Figuree 5 presents the eval- uation results, suggesting that the model trained on the score-O group is less accurate (in terms of ac- curacy and macro fl), more biased toward certain target groups (in terms of equalized odds differ- ences), and less explainable (in terms of faithful- ness). On the other hand, the performance margin is not significant for the plausibility. The finding supports the hypothesis in our dataset, implying the potential role of CRT for building the dataset with high-quality labels.
corpus. For comparative analysis, we applied the classifier to the same number of comments from the unlabeled comments posted from the entertainment section, which was published concurrently with the BEEP! labeled corpus (Moon et al., 2020). This section falls under the soft news category(Lehman- Wilzig and Seletzky, 2010), in contrast to the hard encompassed by corpus. Results show that offensive and hate expressions are more preva- lent in the comments associated with hard news than those from soft news. Among them, hate speech accounts for a larger fraction in the politics and worlds sections, targeted toward the politics and religion group as the major target (Figure 6b), respectively. By contrast, offensiveness embedded in comments from the entertainment section tend to be targeted toward individuals, such celebrities.
7 Case study: how many hates are prevalent a Korean news portal?
We conduct a case study to understand the distribu- tion of offensive language and hate speech in the wild. We applied the H+T model trained on the transformed unified labels to the unlabeled dataset of 191,663 news comments published in the soci- cty, world news, and politics sections shared over the period of July to August in 2021. The target dataset was randomly sampled from the initial col-
Conclusion
This study introduces a new corpus for hate speech detection, representing the largest offensive lan- guage corpus in Korean date. The dataset en- compasses 192K news comments, each rated for target-specific offensiveness on three-point Likert scale. For efficient modeling, we introduced a la-
lection, and it does not overlap with the proposed

---

1.0
19,00%4
80%
75%
5.63%
00%
0.8
Age
Disabled
Cender
0.6
045
offonsive
normal
0.6
11 hate
0.33%
Job
0.4
L2 hate
0.4
Othera
Potiiics
56.15%
4.66
0.2
64.99%
0.2-
17.714
27.50%
35,20%
3.54%
9.60%
0.0
Works
0.0
World
Pollics
Eitertainment
(a) Abusive language categories
(b) Target categories
Figure 6: Inference results on the unlabeled news comments
bel transformation strategy, aiming to unify labels for abusive language categories and multi-label tar- get categories. Experimental results highlight the effectiveness of the proposed dataset and labeling scheme. Both the dataset and labeling strategy hold promise for advancing NLP research in hate speech detection.
Third, our resource was derived from the news comments in single web portal. We targeted the platform because it the major source of news sumption in South Korea, as frequently used in the construction of existing resources Korea. Future studies could target hate expressions on emerging platforms, such as comments on short-form videos.
Furthermore, inspired by studies in psychology and social science (Toplak et al., 2011: Roozen- beek al., 2022: Teovanović et al., 2021; Penny- cook and Rand, 2021), we tested the hypothesis that labels annotated by workers with low CRT scores-indicate of diminished attention during the labeling soocess-would be biased and noisy, Our findings revealed that using labels provided by annotators with the lowest test scores results in detection models that are both less accurate and un- fairer. This discovery supports hypothesis and suggests the CRT score's viability as surrogate for annotation quality. To the best of our knowledge, this research pioncering in integrating CRT into the process of ML dataset construction, offering a potential avenue of contribution to the wider NLP community.
Ethics and Impact Statement
This study introduces novel offensive language corpus in Korean, comprising 192,158 samples. Among the publicly available offensive language corpora in Korean, our dataset stands out as the most extensive corpus and is the first one to use target-specific ratings on a 3-point Likert scale. This resource paves the way for subsequent re- search focused on the development of hate speech detection models in both Korcan and multilingual settings. However, users should approach the dataset with caution, recognizing potential biases and risks. For instance, comments might exhibit bi- ases towards certain political orientations based on user demographics. To promote responsible usage, we have included a data statement in the Appendix.
Limitations
In this study, explored the utility of CRT as potential proxy for annotation quality. While our findings indicate a potential correlation between the CRT score and label quality, we caution against its use for filtering annotators during recruitment. Excluding individuals based on a low CRT score might inadvertently discriminate against specific groups. A potential use case of our finding might be to the CRT score in pilot testing gauge an annotator's attentiveness to the labeling process and discern any potential misunderstandings of the labeling guidelines. All participants in our study received equal compensation, irrespective of their CRT outcomes.
First, the current research is centered on single language, leaving the generalizability the finding was not confirmed. It would be an exciting future direction to ascertain whether the findings, such the effects of CRT, hold in other language and different cultural contexts. Second, while the study introduced novel resource, it did not innovate on the detection method itself. Future studies could explore models that incorporate the results of the cognitive reflection test in their training. A straight- forward application could be the integration of an instance- weightin method into the loss function.

---

Acknowledgments
Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media.
The first two authors equally contributed to this work. This research was supported by the National Research Foundation of Korea the Institute of Informa- tion & Communications Technology Planning & and DATUMO (SELECTSTAR) through the "2022 AT Dataset Supporting Business" program. Kunwoo Park is the corresponding author.
Shane Frederick. 2005. Cognitive reflection and de- cision making. Journal of Economic perspectives, 19(4):25
Evaluation
Moritz. Hardt, Eric Price, and Nati Srebro. 2016. Eiuul- ity of opportumity supervised learning. Advances in neural information processing systems, 29.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi. Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. -eeerrate dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Voolue 1: Long Papers), pages 3309-3326, Dublin, Ireland. Association for Computational Linguisties.
ToxiGen: A large-scale
References
Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th international conference on World Wide Web companion, pages 759-760.
Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198 4205, On- line. Association for Computational Linguistics.
Emily M Bender and Batya Friedman. 2018 Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Younghoon Jeong, Juhyun Oh, Jaimeen Ahn, Jong- won Lee, Jihyung Mon, Sungjoon Park, and Alice Oh. 2022. Kold: Korean offensive language dataset. arXiv preprint 2rXiv:2205.11311
Thomas Davidson, Debasmita Bhattacharya, and Ing- mar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25-35, Florence, Italy. Association for Com- putational Linguistics.
David Jurgens, Libby Hemphill, and Eshwar Chan- drasekharan. 2019. A just and comprehensive strat- egy for using nlp to address online abuse. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3658-300 188-6666.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech de- tection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, volume 11, pages 512-515.
Jiyun Kim, Byounghan Lee, and Kyung-Ah Sohn. 2022a. Why hate speech? masked rationale prediction for explainable hate speech detection. In Proceedings of the 29th International Conference Computational Linguistics, pages 6644-6655.
Jay De Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Online.
Tae Hoon Kim. 2020. Exploration on the relationship between cognitive reflection and higher level pro- cesses. The Journal of Human Studies, 53: 33:109-122.
Computational Linguisticx, pages Association for Computational Linguistics.
Youngwook Kim, Shinwoo Park, and Yo-Sub Han. 2022b. Generalizable implicit hate speech detec- tion using contrastive learning. In Proceedings of the 29th International Conference Computational Linguistics, pages 6667-6679.
Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Gr- bovic. Vladan Radosavljevic, and Narayan Bhamidi- pati. 2015. Hate speech detection with comment em- beddings. In Proceedings of the 24th international conference on world wide web, pages 29-30.
Jean Lee, Taejun Lim, Heejun Lee, Bogeun Jo, Yang- sok Kim, Heegeun Yoon, and Soyeon Caren Han. 2022. K-mhas: A multi-label hate speech detec- tion dataset in korean online news comment. arXiv preprint arXiv:2088. 10684.
Mai ElSherief, Caleb Ziems, David Muchlinski, Vaish- navi Anupindi, Jordyn Seybolt, Mummun De Choud- hury, and Diyi Yang. 2021. Latent hatred: A bench- mark for understanding implicit hate speech, arXiv preprint arXiv:2109.05322.
Junbum Lee. 2020. Kebert: Korean comments bert. In Proceedings of the 32nd Annual Conference on Human and Cognitive Language Technology, pages 437-440.
Antigoni Maria Founta, Constantinos Djouvas, De- spoina Chatzakou, Ilias Leontiadis, Jeremy Black- burn, Gianluca Stringhini, Athena Vakali, Michael

---

Sam N Lehman- Lehman-Wilzig and Michal Seletzky. 2010. Hard news, soft news, 'general' news: The necessity and utility of an intermediate classification. Journal- ism, 117):337-56
Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural I Language Processing (Volume 1: Long Papers), pages 41-58.
Tsung-Yi Yi Lin. Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Pratik S Sachdeva, Renata Barreto, Claudia von Vacano, and Chris J Kennedy. 2022. Assessing annotator identity sensitivity via item response theory: A case study in a hate speech corpus. In 2022 ACM Confer- ence Fairness, Accountability, and Transparency. pages 1585-1603.
Philipp Lorenz-Spreen, Lisa Oswald, Stephan Lewandowsky, and Ralph Hertwig. 2023, A system- atic review of worldwide causal and correlational evidence on digital media and democracy. Nature human behaviour, 7(1):74-101.
Binny Mathew, Punyajoy Saha, Seid Muhic Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for ex- plainable hate speech detection. In Proceedings of the AAA/ Conference on Artificial Intelligence, vol- ume 35, pages 14867-14875.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for compitational linguistics, pages 1668-108 1678.
Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties, pages 5248-5264, Online. Association for Computa- tional I Linguistics.
Jihyung Moon, Won Ik Cho. and Junbum Lee. 2020. Beep! korean corpus of online news com- ments for toxic speech detection. arXiv preprint arXiv:2005.12503.
Isar Nejadgholi, Kathleen C Fraser, and Svetlana Kir- itchenko. 2022. Improving generalizabilityy in implic- itly abusive language detection with concept activa- tion vectors. arXiv preprint arXiv:2204.02661.
Predrag Teovanović, Petar Lukić, Zorana Zupan, Alek- sandra Lazié, Milica Ninković, and Iris Zeželj. 2021. Irrational belief's differentially predict adherence guidelines and pseudoscientific practices during the covid-19 19 pandemic. Applied Cognitive Psychology. 25(2):486-496
Nicolas Ocampo, Ekaterina Sviridova, Elena Cabrio, and Serena Villata. 2023. An in-depth analysis of implicit and subtle hate speech messages. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1989-2003.
Maggic E Toplak, Richard F West, and Keith E Stanovich. 2011. The cognitive reflection test as predictor of performance on heu--stics-and-biasss tasks. Memory & cognition, 39(7):1275- 1289.
San-Hee Park, Kang-Min Kim, O-Joun Lee, Youjin Kang. Jaewon Lee, Su-Min Lee, and SangKeun Lee. 2023. "why do feel offended?" Korean dataset for offensive language identification. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1442-1153, Dubrovnik, Croatia. Associ- ation for Computational Linguistics.
Michael Wiegand, Maja Geulig, and Josef Ruppenhofer. 2021. Implicitly abusive comparisons- a new dataset and linguistic analysis. In Proceedings of the 16th Conference of the European Chapter of the Associ- ationfor Computational Linguistics: Main Volume, pages 358-368. Online. Association for Computa- tional Linguistics.
Gordon Pennycook and David G Rand. 2021. The psy- chology of fake news, Trends in cognitive sciences. 25(5):388 402.
Kichang Yang, Wonjun Jang, and Won Ik Cho. 2022. Apeach: Attacking pejorative expressions with anal- ysis on crowd-generated hate speech evaluation datasets. arXiv preprint arXiv:2202.12455.
Alan Ramponi and Sara Tonelli. 2022. Features or spu- rious artifacts? data-centric baselines for fair and robust hate speech detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3027-3040, Seat- tle, United States. Association for Computational Linguisties.
Ziqi Zhang and Lei Luo. 2019. Hate speech detection: A solved problem? the challenging case of long tail on twitter. Semantic Web, 10(5):925 945
A Appendix
Jon Roozenbeek. Sander Van Der Linden, Beth Gold- berg, Steve Rathje, and Stephan Lewandowsky. 2022. Psychological inoculation improves resilience against misinformation on social media. Science advances, 8(34))eab06254.
A.I Data statement
We present the data statement for responsible us- age (Bender and Friedman, 2018). The corpus

---

consists of 192,158 news comments consisting of 183,791 news comments collected by ourselves and 8,367 comments collected from a previous study (Moon et al., 2020). The collected dataset originates from the politics/world/social sections, which are categorized as hard news. The comments from the BEEP! corpus are from the entertainment section. The comment function in the entertain- ment section was disabled due to the prevalence of toxic expressions toward celebrities.
offensive-likely), (obviously or seriously offen- sive). The value of indicates there is no offensive expression. The value of 1 means there room for disagreement about whether is an offensive expression speech or not. This category includes implicit hate expressions, such as sarcasm, stereo- types of target group, etc (EISherief et al., 2021), The labeling decision could depend on the context of expression. This category of speech could not include any rationale. The value of means that is an explicitly offensive expression without any doubt. The comment can make readers very un- comfortable, implying that the comment should be censored in online platforms.
Curation Rationale We collected the raw data from news.naver.com, the largest news portal in Korea. We targeted news articles published in the society, world, and politics sections because discus- sions are active in the hard news.
Target-specific and fine-grained ratings (13 classes) There are nine target-specific rating vari- ables: Gender, Age, Race/Region, Disability, Reli- gion, Politics (orientation), Job, Individual, Oth- ers. There are four fine-grained rating: Swear word/Curse, Insult, Treat/Violence/promootii crime, and Obscenity/Sexual harassment. All classes have more specific explanations and exam- ple comments. For example, in the case of IInsult', was explained as Demeaning and degrading the object's value and social evaluation, Expressing contemptuous feelings, and Disparaging others re- gardless of the facts. To give an example, "I yn ह A 67" (English Translation: are really pathetic") can be slight insult, and " 7 A la Φ) 2 Z) A rll" (English Translation: "He was the source of venereal dis- ease") can be an obvious insult.
Language Variety Our dataset consists of the comments in Korean (ko-KR).
Speaker Demographic The user demographic is not available. However, considering that the portal site has the largest share of Korean, it can be assumed that speakers are mostly Korean.
Annotator Demographic The data annotation process was conducted on CashMission, a crowd- sourcing platform in Korea. A total of 405 workers participated in an annotation. 21 workers are 10s, 222 workers are 20s, 116 workers are 30s, 35 work- ers are 40s, 9 workers are 50s, and 2 workers are 60s. In terms gender, 309 women, 95 men and I none-binary participated.
인 श o|
"Korean
Speech Situation News article in the hard news section deals with controversial events, so there are more likely to exist hate comments toxic- ity comments. The target articles were published between July 2021 and August 2021. During that period, several highly contentious events unfolded, including the presidential election in South Korea, the Tokyo Olympics, the COVID-19 pandemic, and the Restoration of Taliban Control, to name few.
Offensiveness rationale Highlight the minimum span that will become a normal sentence if the cor- responding span is masked or the span that will be included when building a hate expression dictio- nary. An expression of trying to avoid sanctions also should be highlighted. For example, explicit hate speech such as swear words, slang, and deroga- tory words, contextual hate speech, and inappropri- ate expressions such as sexual harassment should be highlighted.
Text Characteristics It includes hatred words re- stricted to Korea, such as hatred of specific political orientations and specific groups. For example, "t] T hate expression for the supporters of the former Korean president Moon), and ":st m]" (a hate expression toward feminists).
Target rationale Target highlighting process highlights the target of offensive expressions. It can be overlapped with offensive representation highlighting. And this should also be highlighted by minimum span, the same as offensive repre- sentation highlighting. the span containing the modifier can make the target hate clearer, then the modifier should be included as the target span.
A.2 Detailed guideline
Offensiveness scale (3-point ratings) The de- gree of offensiveness is evaluated a 3-point Lik- ert scale: (no offensiveness), (weakly offensive,

---

A.3 Rule-based filtering
1,000
C:428
5.614
0.500
We excluded the data in the dataset one of the following conditions is satisfied because the anno- tation is likely wrong.
AUUD
0.446
0.534
CHINE
1. Data with incorrect rationale annotation for- mat.
KG.0
6440
09943
0..
(a) Token-level similarity
(b) Embedding similarity
Figure AI: Similarity between offensive language cor- pora in Korean
2. Data where all fine-grained ratings have value of O, but IND OTH ratings have value of 1
3. Data with target-specific offensiveness ratings greater than 0 and a max rating greater than 1, but no rationale
nomal- (0:373)
(0.131)
1093
270 (0.032)
474
1.337
11004
affensive
(0.057)
(0.160)
(0.1322
Ours
4. Data with rationale beyond the max men_length
186 (0.022)
(0.007)
(0.027)
1000
A.4 Detailed configuration
Level 2
72 (0.009)
429 (0.051)
550
hate
(0.001)
For experiments, we used KcBERT (Lee, 2020) as the BERT backbone based on comparisons with KPFBERT and KLUE-BERT' (Table A2). The model is BERT variant with I 110M parameters, which pretrained
affensive BEEP!
Figure A2: Label comparisons of the 8,367 samples in BEEP! and the proposed corpus
Korean corpus. We uses AdamW an optimizer with learning rate of 2e- 5, and set the batch size as 32. Models were early- stopped using the validation set with the patience of 5. We repeated the model training five times with varying random seeds: 0, 0,10,20,30 30,40. The com- putational environment consists of NVIDIA RTX A6000, 125G RAM, and AMD Ryzen Threadripper PRO 3975WX 32-Cores. Hyperparameters were optimized by random search on the validation set. is 0.1 for attention supervision and 1 for training target labels. attention heads were supervised for attention supervision.
of BERT-based sentence transformer that is fine- tuned on the KorSTS and KorNLI dataset". Fig- ure A lb presents the results. The embedding anal- ysis suggests that each corpus is not largely over- lapped other corpora the embedding space.
Label comparison To evaluate the proposed la- beling scheme, we included the labeled samples in the BEEP! dataset and examined how a sam- ple is labeled in dataset. Figure A2 presents the results. We observed that a large fraction of hate-labeled expressions in the BEEP! dataset were re-labeled as offensive in our datasct. We suspect that this is due to the lack of target labels in the previous corpus. Also, the level-1 hate samples in our corpus had been labeled in the BEEP! corpus as normal (59) offensive (186). Altogether, the findings suggest the importance of target-specific ratings for the effective identification of hate.
A.5 More analyses
Inter-dataset similarity To understand the rep- resentativeness of each corpus, we measured token- level and embedding similarities between the of- fensive language corpora in Korean. For the token- level similarity, we caleulated the t0p-10000 to- kens in each corpus and measured the Jaceard sim- ilarity between each pair of the two token sets, as illustrated in Figure Ala. The results suggest that our dataset is the most similar to KoLD, and BEEP! is to K- K-M in terms of the token-level similarity. For the embedding similarity, we used checkpoint
Cross-dataset prediction experiment We con- ducted an experiment where we used model trained our own dataset predict the labels of the previous corpus. For the BEEP!, similar the label comparison, there is a tendency for hate data in the BEEP! dataset to be predicted as offen-
hhups:/hugginggfac cofjinmang2/ptbertt hhtpss//huggingface co/kluc/berr-busc by the BERT tokenizer

---

Question 777 E 0-
Correct answer
10 H 1 7 e #/. 3 /
oF
5 cents
bat and ball cost 110 cents total.
The bat costs 100 more than the ball. How much does the cost?
f of l 말 = - 2 7
100 2 2 7 le [0 7 4 डरह ooo カ it takes 5 machines 5 minutes to make widgets.
minutes
how long would lake 100 machines make 100 widgets? a e e ol
mi]
o 2 E7. 인 인 - 48 인 ट.प
2 t = _ o] ?/?
In lake, there patch lily pads. Every day, the patch doubles size. it takes 48 days
for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? of & Z & 9 E - * & of 67 2
8 | = T of E カー!
3 people pack 3 toys per hour, how many people are
needed pack toys in 2 hours?
<
4 + + 2 म ?
15 U77.
Younghee's midterm grades the 15th highest and 15th lowest in her class. How many students are in Younghee's class?
# 21
of fe + o 3ul c. ei >| b # 60 & मे 1
15
Tall swimmers on swim team are three times more likely to win than shorter swimmers. If the team 60 championships this year, how many of those 60 did the short players win?
Table Al: List of CRT questions and their English translations
Detection performance
Fairness
Explainability
Model
Equalized odds
Equalized odds diff. (FPR) J.
Micro F1
Macro F1
Plausibility
Faithfulness
diff. (TPR) +
KcBERT 0.589
0,55
0.267
0.271
0.285
0.205
KPFBERT
0.57
0.541
0.324
0.333
0.229
0.207
KLUE-BERT
0.573
0.543
0.324
0.359
0.129
0.173
Table A2: Effects of pretrained backbone models (Model: H+T with transformed labels)
sive. Additionally, in the test sets of KoLD and K-MHaS, a significant number of hate data were also predicted as offensive, which indicates that some of the hate data in the existing corpus target general groups or individuals rather than protected groups.

---

380
The babdt (BEEP)
rermal (0.829)
3024
(0.164)
598
10. (0.005)
18
(0. (0.0011)
The labels (XQLD)
rormat 10.8921
3629
(0.077]
314
(0.027)
109
(0.00 (0.004)
17
ThoP labes (MAlliSS
normal
10.5411
2464
(0.365)
1663
(0.005)
302
10 (27)
125
Steesiiee [0 (..40)
1082
(0.540)
1452
(0.032) (0.025)
85
63
offernive
[0.274]
457
(0.550)
917
(0.063) (0.112)
105
187
3300
offerdive-
(0.004)
3
(0.848)
568
(0 00,000
99 (0. (0.148))
nate (0. (55155)
315
(0.547)
1112
(0.045)
02
(0.253)
514
hale (0.335)
882
(0.247]
651
(0.162)
427
(0:256)
672
590
hale (0.046)
146
(0. (0.256)
805
(0 00.000
250
(0.618)
1942
normal offendive
level
normal
offerilie
lesel
normal offenche
Tate
Tate
hate
hale
Figure A3: Cross-dataset prediction experiments use our corpus for training


Reviews:
[{"review_id": "yAoJ4Bi09S", "reviewer": "Reviewer_9cnA", "summary": "", "questions": "- Why did you take comments from other dataset?\n- Did you study the inter-agreement among annotators?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort for this constructive review.\n- We have two reasons to include the BEEP! dataset in our corpus.\n  1. BEEP! comprises the comments on the Entertainment section. Due to the cyberbullying issues toward celebrities, the comment features have been disabled for the section since 2020. The inclusion of the comments increases the topical diversity of our corpus. Our data collection covers news comments on hard news: society, world news, and politics sections.\n  2. The labels on the BEEP! dataset were used for verifying the quality of our corpus. The label comparison results are in Figure A3.\n- We did not measure IAA for the entire set because each example was assigned to a single annotator. To ensure the quality, we conducted pilot tests. Each round uses 100 examples. After each round of pilot test annotation, they reported unclear cases to the authors and were instructed. They retook a test with a different set of examples. The process was repeated until the annotator passed the test and found no ambiguities. They were paid for the pilot tests with the same conditions of the main annotation task. The results of the cross-dataset experiments (Section 5) suggest the high quality of our dataset.\n- We believe the dataset and paper can make a broad contribution to those working on hate speech and resource construction by the largest corpus in Korean and new findings on CRT. Thank you for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The novelty of a new dataset for hate speech classification in Korean.\n- The use of the cognitive reflection test to identify skewed annotators.", "reasons_to_reject": "- The lack of details of the building process of the dataset.\n- The lack of evaluation of the agreement among annotators.", "questions_to_authors": "- Why did you take comments from other dataset?\n- Did you study the inter-agreement among annotators?", "paper_topic_and_main_contributions": "The paper presents a new dataset of hate/offensive speech in Korean, with the novelty of labeling different scales of offensiveness in order to catch the implicitness of the offensive language.\n\nThe novelty of the paper is grounded in the work on Korean and the labeling schema, which is complemented with the cognitive reflection test for filtering out skewed annotators. However, I see that the main point of the paper is the new dataset and not the experiments. For this reason, I have some concerns:\n\n1. I'm confused with the method to get the comments. In section 3.1, it is said that there are some comments from the dataset BEEP!. Why did you take comments from other dataset? I do not really understand this point.\n\n2. Did you study the inter-agreement among annotators? If you used so much annotators, it maybe impractical the calculation of the inter-annotator agreement, but beyond the used of the CRT test, did you use an additional assessment of the quality of the annotation.\n\nI think that this two points are more important than the classification results, and I miss a larger description of the details of the new dataset.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XUeBHLpWNN", "reviewer": "Reviewer_dMtf", "summary": "", "questions": "- What is \"the current largest corpus\" in line 58?\n- \"We present a large-scale offensive language corpus of 192,158 samples...\" I think this is one of the key contribution of the paper. It is good idea to provide the link of the available data and codes in the manuscript.\n- p3, line 170, 'including the same size of data '. what does the same size mean?\n- Please elaborate on the annotators' qualifications, the payment for the labelling task, and a clearer description of \"a small set of samples\" in Section 3.2.\n- While the authors have provided a comprehensive overview of the target-specific hate ratings task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results.\n- The authors have presented a benchmark dataset for target-specific hate speech detection, but there is no explanation of how they assessed the dataset's quality or determined the Inter Annotator Agreement (IAA).\n- The texts and numbers in Figure 6 are not clearly visible. Please improve the quality of this figure. Additionally, the authors should discuss the specific reasons behind misclassifications of a certain class compared to others.\n- To aid researchers in understanding the benefits and limitations of the Baseline Models for target-specific hate ratings used in this study, it would be valuable for the authors to explain their decision-making process behind their selection.\n- The proposed Label transformation solution includes Normal, Offensive, Level-1 hate, and Level-2 hate labels. However, the difference between Level-1 and Level-2 is not clearly explained. Further clarification is needed regarding the character, impact, and significance of this label transformation.\n\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort for this constructive review.\n- Linguistic aspects: Due to the lack of space, we did not include the analysis, but it would be an exciting direction to understand the linguistic patterns of each category.\n- Annotator payment: Following the agency’s guidelines, annotators were paid $0.05 per example. Each annotator was assigned ~100 examples that were sampled from 10,000 comments. Each comment was assigned to three to four annotators.\n- IAA: We did not measure IAA for the entire set because each example was assigned to a single annotator. To ensure the quality, we conducted pilot tests. Each round uses 100 examples. After each round of pilot test annotation, they reported unclear cases to the authors and were instructed. They retook a test with a different set of examples. The process was repeated until the annotator passed the test and found no ambiguities. They were paid for the pilot tests with the same conditions of the main annotation task. The results of the cross-dataset experiments (Section 5) suggest the high quality of our dataset.\n- The currently available largest corpus is K-MHaS, comprising 109.6K comments. Table 1 presents the size of existing corpora. The advantage of using a large dataset is two-fold. First, the larger resource would cover more diverse patterns of each class, and therefore, a model trained on the corpus would be more representative. Second, a larger corpus allows for training larger models with more parameters. The model prediction would be more accurate.\n- Figure 6: We will increase the visibility of the embedded text/numbers.\n- Model selection: We set the standard classifier model that tackles the target task (H+T). We added the rationale training as an auxiliary objective for H+T+R as proposed in previous studies (e.g., Mathew et al., 2021)\n- Clarification on label transformation: Level-2 Hate represents explicit toxic expressions toward one or multiple groups, which has a value of 2 for the corresponding group ratings. Level-1 Hate represents weakly toxic expressions toward one of the multiple target groups, which has a value of 1 for one of the target group ratings and the maximum value is also 1. Therefore, Level-2 Hate consists of explicit hate expressions, and Level-1 Hate comprises implicit hate expressions. The label transformation process allows for detecting implicit expressions more effectively, as presented in Table 5. Section 5 presents the effectiveness of the proposed transformation process. Section 7 shows the prevalence of implicit hate expressions in online comments, which may cause online harm and undermine the platform's longevity. The proposed dataset and label transformation enable the detection of both implicit and explicit forms of hate. We will update Section 3.4 in the next version to help potential readers understand the difference more clearly. \n- We submitted the sample dataset and code to the openreview submission. A URL for the GitHub repository will be included in the final version if this paper is accepted.\n- Thank you for the comments on the “Typos Grammar Style And Presentation Improvement.” We will update the paper according to your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper's organization and presentation are commendable overall.\n- The work exhibits strong performance, with promising results in the hate speech detection task.\n- The paper introduces specific benchmark datasets tailored to a particular task.", "reasons_to_reject": "- While the authors have provided a comprehensive overview of the target-specific hate ratings task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results.\n- Please elaborate on the annotators' qualifications, the payment for the labelling task, and a clearer description of \"a small set of samples\" in Section 3.2.\n- The authors have presented a benchmark dataset for target-specific hate speech detection, but there is no explanation of how they assessed the dataset's quality or determined the Inter Annotator Agreement (IAA).\n- The authors state that their dataset is the largest among current Korean datasets, with over 200k+ samples. However, no experiments or analyses have been conducted to support this claim. It is essential to clarify whether such a large dataset is necessary for effectively addressing this problem.\n- The texts and numbers in Figure 6 are not clearly visible. Please improve the quality of this figure. Additionally, the authors should discuss the specific reasons behind misclassifications of a certain class compared to others.\n- To aid researchers in understanding the benefits and limitations of the Baseline Models for target-specific hate ratings used in this study, it would be valuable for the authors to explain their decision-making process behind their selection.\n- The proposed Label transformation solution includes Normal, Offensive, Level-1 hate, and Level-2 hate labels. However, the difference between Level-1 and Level-2 is not clearly explained. Further clarification is needed regarding the character, impact, and significance of this label transformation.\n", "questions_to_authors": "- What is \"the current largest corpus\" in line 58?\n- \"We present a large-scale offensive language corpus of 192,158 samples...\" I think this is one of the key contribution of the paper. It is good idea to provide the link of the available data and codes in the manuscript.\n- p3, line 170, 'including the same size of data '. what does the same size mean?\n- Please elaborate on the annotators' qualifications, the payment for the labelling task, and a clearer description of \"a small set of samples\" in Section 3.2.\n- While the authors have provided a comprehensive overview of the target-specific hate ratings task, additional information on the linguistic aspects of the task would be beneficial. This would help readers to understand the theoretical foundations of the study and the methods used to achieve the results.\n- The authors have presented a benchmark dataset for target-specific hate speech detection, but there is no explanation of how they assessed the dataset's quality or determined the Inter Annotator Agreement (IAA).\n- The texts and numbers in Figure 6 are not clearly visible. Please improve the quality of this figure. Additionally, the authors should discuss the specific reasons behind misclassifications of a certain class compared to others.\n- To aid researchers in understanding the benefits and limitations of the Baseline Models for target-specific hate ratings used in this study, it would be valuable for the authors to explain their decision-making process behind their selection.\n- The proposed Label transformation solution includes Normal, Offensive, Level-1 hate, and Level-2 hate labels. However, the difference between Level-1 and Level-2 is not clearly explained. Further clarification is needed regarding the character, impact, and significance of this label transformation.\n\n", "paper_topic_and_main_contributions": "This paper introduces a novel manually-annotated Target-specific Hate Speech Detection dataset sourced from a prominent news portal in South Korea. In comparison to existing datasets, this proposed dataset serves as a relevant testbed for evaluating the effectiveness of Target-specific Hate Speech Detection methods in real-world scenarios. The authors conduct a comprehensive exploration of the dataset across various scenarios and present intriguing avenues for future research. Through their experiments, they demonstrate the formulation of a multitask problem as a hate speech detection task, leveraging large pre-trained sequence-to-sequence models. Notably, the authors conduct a case study that reveals hate speech's higher prevalence in news comments posted on hard news than soft news.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The authors should explain the meaning of the abbreviations in Figure, Table such as IND, OTH to ensure that readers understand the information presented. Abbreviations need to be explained right in the caption, instead of only discretely mentioned in the text.\n- As data preprocessing is a crucial step in natural language processing tasks, the authors should include details on the data preprocessing techniques used to ensure the reproducibility of the study."}}, {"review_id": "Rc4Ny8zZXQ", "reviewer": "Reviewer_vzY9", "summary": "", "questions": "In the introduction, implicitness is cited as a topic addressed in the introduction of the paper. \nNevertheless, it is not clear how the implicit forms of hate speech are distinguished from the explicit ones. Are the authors assuming that all their data are implicit forms of hate speech? This should be motivated based on some criterion or analysis.\nIn the annotation, implicitness is not annotated, while in the experiments, the results are not organized to show how the models perform on implicit forms of hate rather than on explicit ones. This aspect is in effect interesting and should be better dealt with in the paper.\n\nAs far as the annotation process: How many annotations are collected for each message? it has been evaluated the disagreement among the annotators? it is quite unusual that the problem of bias is mentioned when a comparison between annotators is not made that allows an effective evaluation, even if only on a sample of data.\n\nAnother point that should be made more clear is the annotation bias. The CRT score is a measure of the attention paid by the annotator to the annotation task, but it is clear if there is a relationship between it and the bias. At several points (e.g. at the end of section 6) it is cited the bias of the annotators but how it is evaluated? Also considering that the annotation process does not involve evaluation of the disagreement.\n\n\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort for this constructive review.\n- “Reasons To Reject” #1: While automatic and manual moderation can reduce the volume of explicit hate expressions in online platforms, since moderation tools are imperfect, there can be many toxic expressions left available, as empirically observed in our dataset (Table 4). Level-2 Hate represents the explicit hate expressions.\n- “Reasons To Reject” #2: Irony (or sarcasm) is the main subclass of implicit hate identified in previous research (e.g., ElSherief et al., 2021). In the dataset construction, we identified many examples of implicit hate that is expressed through sarcasm, stereotype, or prejudice. \n- “Reasons To Reject” #3: In the annotation, we asked the annotator to label 1 if the target comment includes weakly toxic expressions such as sarcasm, stereotypes, or prejudices. They were instructed to label higher scales if a target comment includes explicit toxic expressions. Following the transformation process (Figure 2), we deem a comment with a rating of 1 toward a group rating as an implicit hate expression. We will clarify the difference in Section 3.4.\n- The dataset and code will be available through a GitHub repository if this paper is accepted.\n- “Are the authors assuming that all their data are implicit forms of hate speech?”: No. We designed the annotation scheme and transformation process to make Level-2 Hate represent the explicit form of hate expressions.\n- “How many annotations are collected for each message?”: A single annotator was assigned to each comment. To ensure the quality, we conducted pilot tests. Each round uses 100 examples. After each round of pilot test annotation, they reported unclear cases to the authors and were instructed. They retook a test with a different set of examples. The process was repeated until the annotator passed the test and found no ambiguities.  They were paid for the pilot tests with the same conditions of the main annotation task. The results of the cross-dataset experiments (Section 5) suggest the high quality of our dataset.\n- How annotator bias was measured: We did not measure the annotator bias explicitly. It was measured by a proxy measure in classification experiments, i.e., the fairness evaluation of classifiers. Our assumption is that if a model is trained on biased annotations, the model would lead to worse fairness.\n- Missing references: Thank you for the suggestion. We will add the papers to the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The most important contribution of this paper is a novel resource for Korean. \nIn principle, several interesting aspects are considered (but not discussed enough) among which there are the implicitness of hate speech or the problem of bias in the annotation.\nAlso, the annotation scheme can be of interest. Its novelty consists of the categories to be annotated, in the organization in a cascade of them and the consideration of the CRT score in the evaluation of annotators.", "reasons_to_reject": "The discussion about implicitness is not convincing for the following reasons: \n1)  it cannot be truly said that the majority of previous works concentrated on explicit forms of hate speech because the majority of works about hate speech detection are based on corpora collected from social media, that is environments where a large portion of explicit hate is automatically detected or signalled by users to be immediately removed (and therefore often not available in the data collected by researchers);\n2) the authors say that implicitness consists of irony or sarcasm; also not considering that I disagree with this assumption (according to several other researchers), a discussion about this point is not given in the paper; there are several works in which irony and sarcasm are discussed and resources in which those phenomena are annotated and a comparison with them is missing; \n3) also in the annotation and in the evaluation implicitness is not considered, nor some distinction is proposed between implicit and explicit forms of hate.\n\nThe evaluation section does not provide a comparison with the results achieved by other existing models or by some baseline models. The literature cited by the authors does not include important works such as those obtained in evaluation campaigns, for example, in SemEval tasks.\n\nIt is not clear from the paper whether the dataset will be made available to the research community.", "questions_to_authors": "In the introduction, implicitness is cited as a topic addressed in the introduction of the paper. \nNevertheless, it is not clear how the implicit forms of hate speech are distinguished from the explicit ones. Are the authors assuming that all their data are implicit forms of hate speech? This should be motivated based on some criterion or analysis.\nIn the annotation, implicitness is not annotated, while in the experiments, the results are not organized to show how the models perform on implicit forms of hate rather than on explicit ones. This aspect is in effect interesting and should be better dealt with in the paper.\n\nAs far as the annotation process: How many annotations are collected for each message? it has been evaluated the disagreement among the annotators? it is quite unusual that the problem of bias is mentioned when a comparison between annotators is not made that allows an effective evaluation, even if only on a sample of data.\n\nAnother point that should be made more clear is the annotation bias. The CRT score is a measure of the attention paid by the annotator to the annotation task, but it is clear if there is a relationship between it and the bias. At several points (e.g. at the end of section 6) it is cited the bias of the annotators but how it is evaluated? Also considering that the annotation process does not involve evaluation of the disagreement.\n\n\n", "paper_topic_and_main_contributions": "The paper introduces a novel large corpus for Korean in which hate speech is annotated using a fine-grained scheme. \nThe corpus consists of news comments and the scheme includes the rating of offensiveness, the identification of the target and the rationale, while the annotation has been performed by 405 annotators of a crowdsourcing platform.\nFinally, the paper describes the results achieved by using different models trained on this corpus. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I did not find typos or grammatical errors. In my opinion (as a non-native English speaker), the paper seems well-written and organized."}, "extra_text": {"Missing_References": "The article lacks references to important works that would allow relevant comparisons to support the scientific quality of the work done by the authors.\n\nFor what concerns a general scenario there are surveys that should be cited also to better introduce the meaning of the notion of hate speech and other ones closely related (such as offensiveness, aggressiveness, stereotype ...), for example, Poletto et al. (2020) Resources and benchmark corpora for hate speech detection: a systematic review. Language resources and evaluation 55 or Fortuna et al. (2018) A Survey on Automatic Detection of Hate Speech in Text. ACM computing survey 51: 4.\n\nFor what concerns the datasets and results, it should be important to cite at least some of the experiences done in the context of evaluation campaigns. This is the context where the datasets became benchmarks for the community and where the state-of-the-art is established for tasks such as hate speech detection. Moreover, the papers published in this kind o context are the most cited by the community, see for instance, Basile et al. (2019) SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation or other similar foundational works.\n. "}}]

Author rebuttal:

Meta-review: The paper introduces a unique dataset focused on hate and offensive speech in Korean. This dataset is distinct due to its detailed labeling system that measures varying degrees of offensiveness, aiming to capture both explicit and implicit offensive language. The data is sourced from a major South Korean news portal, and its annotations are designed to identify the target of the hate speech, the rationale behind it, and the level of offensiveness. To ensure the quality of annotations, the dataset employs a cognitive reflection test to filter out potentially biased annotators. The dataset comprises news comments, and a significant number of annotators (405) from a crowdsourcing platform have been involved in the annotation process. The paper also explores the application of this dataset in hate speech detection tasks, using advanced models. A case study within the research indicates a higher occurrence of hate speech in hard news comments compared to soft news. However, there are concerns about the dataset's sourcing and the inter-annotator agreement process.

The paper presents a novel dataset for hate speech classification in Korean, filling a gap in the research landscape. The dataset's uniqueness is further emphasized by its detailed annotation scheme, which captures the implicit nuances of hate speech. This scheme is not only innovative in the categories it annotates but also in its hierarchical organization. Another significant contribution is the use of the cognitive reflection test (CRT) to filter out biased or skewed annotators, ensuring the quality and reliability of the dataset. The paper is well-organized, with clear presentation and structure, making it accessible to readers. The research demonstrates strong performance in the hate speech detection task, providing promising results that can serve as benchmarks for future studies. Overall, the paper's contributions, particularly the introduction of a new resource for Korean and the thoughtful consideration of potential biases in annotation, make it a valuable addition to the field.

One of the main issues is that the authors' discussion on implicit hate speech is unconvincing. They overlook the fact that many hate speech studies are based on social media data, where explicit hate is often removed. Their definition of implicitness, equating it to irony or sarcasm, is not universally accepted and lacks a thorough discussion. Moreover, the paper doesn't differentiate between implicit and explicit forms of hate in its annotations or evaluations.

The authors and the reviewers interacted during the rebuttal period to fix some other issues indicated by the reviewers.
Decision: Accept-Findings