KOSBI: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applications
Year: 2023

Abstract:
Large language models (LLMs) learn not only natural text generation abilities but also social biases against different demographic groups from real-world data. This poses a critical risk when deploying LLM-based applications. Existing research and resources are not readily applicable in South Korea due to the differences in language and culture, both of which significantly affect the biases and targeted demographic groups. This limitation requires localized social bias datasets to ensure the safe and effective deployment of LLMs. To this end, we present KOSBI, a new social bias dataset of 34k pairs of contexts and sentences in Korean covering 72 demographic groups in 15 categories. We find that through filtering-based moderation, social biases in generated content can be reduced by 16.47%p on average for HyperCLOVA (30B and 82B), and GPT-3.

heading: 1 Introduction
text: Large language models (LLMs) acquire impressive text generation abilities from large-scale real-world pre-training data (Brown et al., 2020; Kim et al., 2021). However, LLMs also absorb toxicity, such as social biases (Sheng et al., 2019; Wallace et al., 2019a). This cannot be overlooked since the risk of generating toxic content impedes the safe use and potential commercialization of various downstream applications, such as AI assistants (Dinan et al., 2022; Bai et al., 2022a). To minimize the harm, numerous studies have tackled the detection and mitigation of toxicity in LLMs (Blodgett et al., 2020; Ganguli et al., 2022). Each study typically leverages datasets capturing a specific type of toxicity, such as social bias (Sap et al., 2020; Nangia
⋆ Authors equally contributed. ♯ This work was done during their internship at NAVER AI Lab. Email to: {hwaran.lee, jungwoo.ha}@navercorp.com, seokhee.hong@vision.snu.ac.kr
et al., 2020) or hate speech (Warner and Hirschberg, 2012; Lee et al., 2022).
These datasets are not only task-specific but also language- and culture-specific. For instance, consider hate speech made in South Korea and in the United States. In addition to the language, the mainly targeted demographic groups also differ— feminists and Korean Chinese in South Korea, as opposed to African Americans and Jewish in the United States (Jeong et al., 2022). Also, the existing toxicity datasets in Korean mostly focus on explicit hate speech and consider a limited number of targeted demographic groups (Moon et al., 2020; Yang et al., 2022; Kang et al., 2022; Lee et al., 2022). This calls for a dataset to address social biases against a more comprehensive set of demographic groups in South Korea so that as many groups and people are protected.
Here we present the Korean Social Bias (KOSBI) dataset, a large-scale dataset of 34k pairs of contexts and sentences in Korean with labels mainly capturing the presence of social biases. 1 It covers 72 targeted demographic groups in 15 categories, 2 which is much more comprehensive than existing datasets, as shown in Table 2. The categories include not only the common ones like gender and religion but also those especially relevant to South Korea—e.g., marital status and domestic area of origin, both of which consist of demographic groups that suffer from social biases in the country more commonly than others do. Given the difficulty of crawling from the web sufficient data for each of the 72 demographic groups, we leveraged HyperCLOVA (Kim et al., 2021) to generate the data with in-context few-
1 The KOSBI dataset is released with English-translated annotations for those who are not fluent in Korean at https: //github.com/naver-ai/korean-safety-benchmarks
2The categories and demographic groups were selected based on the Universal Declaration of Human Rights (UDHR) and the National Human Rights Commission of Korea (NHRCK).
208
shot learning (Gao et al., 2021; Mishra et al., 2022). More specifically, we generated sentences and their respective contexts—which are also sentences, grammatically—for given target demographic groups. The generated contexts and sentences were then annotated by crowd workers as safe or unsafe. Here, unsafe contexts and sentences were further labeled as expressions of stereotypes (cognitive bias), prejudice (emotional bias), discrimination (behavioral bias), and/or other, adopting the taxonomy by Fiske (2023),3 in Figure 1.
With KOSBI, we mitigate social biases in LLMgenerated content using a filtering-based moderation approach, also known as rejection sampling (Ganguli et al., 2022). To do this, we first trained a safe sentence classifier using KOSBI. Then, for a given context, each LLM was used to generate a pool of sentences from which the safest sentence was chosen by the classifier. The human evaluation shows that social biases in generated content are reduced by 16.47% on average for all three models tested—HyperCLOVA (82B), HyperCLOVA (30B), and GPT-3.

heading: 2 Related Works

heading: Bias Mitigation in LLM-generated Content.
text: LLMs are trained on real-world data, which often contains social biases toward certain demographic groups. This, in turn, induces biases in LLMs (Xu et al., 2021a). To date, various resources have been published to measure and mitigate such biases in LLMs (Sap et al., 2020; Nangia et al., 2020; Nadeem et al., 2021). Some of them are associated with specific tasks: coreference resolution to fight the phenomena like associating certain professions with a particular gender (Rudinger et al., 2018; Zhao et al., 2018), and question answering to prevent answers stereotyped toward certain bias categories like gender or socio-economic status (Li et al., 2020; Parrish et al., 2022). These resources
3For labeling the context, prejudice and discrimination were combined due to the limited number of instances.
are not as effective for HyperCLOVA and other LLMs pre-trained on Korean corpora. Thus, we present a new resource in Korean, capturing the biases against prevalent demographic groups in South Korea. Also, our dataset covers a much more comprehensive set of demographic groups.
Hate Speech Detection. Röttger et al. (2021) defines hate speech as “abuse that is targeted at a protected group or at its members for being a part of that group.” Resources created to help detect hate speech can be used to reduce hate speech generated by LLMs, thereby reducing the harm they can incur. Note these resources use various names interchangeably for the most part, e.g., hate speech (Warner and Hirschberg, 2012), abusive language (Wiegand et al., 2019), and toxic language (Gehman et al., 2020; Hartvigsen et al., 2022). Also, quite a few resources are for safer dialogue (Sun et al., 2022; Xu et al., 2021b; Xenos et al., 2021; Kim et al., 2022). Meanwhile, to reflect different languages and societies, researchers have created and proposed hate speech corpora in Chinese (Deng et al., 2022), Dutch (Demus et al., 2022), and Arabic (Mubarak et al., 2022) Similar to the resources capturing social biases, these resources are not as useful for Korean LLMs due to the differences in language and culture. Luckily, several resources in Korean exist, as summarized in Table 1. However, these resources either unspecify or cover only a small subset of demographic groups in South Korea. More importantly, they focus on explicit profanity and otherwise offensive expressions. Our dataset instead targets cases that cannot be identified with specific keywords, such as expressions of stereotypes, discrimination, and prejudice (without explicit profanity) toward 72 demographic groups.
Safety Alignment of Language Models. Beyond social biases and hate speech, various categories have been proposed recently to enhance the safety of language models, such as human val-
ues (Solaiman and Dennison, 2021; Kenton et al., 2021), ethical judgements (Hendrycks et al., 2021; Lourie et al., 2021), and moral norms (Forbes et al., 2020; Emelin et al., 2021). Then, alignment learning methods through human feedback (Bai et al., 2022a) or even by AI feedback (Bai et al., 2022b) have been proposed. Moreover, red-teaming (Perez et al., 2022; Ganguli et al., 2022) and adversarial attack (Wallace et al., 2019b) approaches have also been suggested to identify vulnerabilities in language models in terms of safety. We expect our dataset and comprehensive categories will be helpful for the safety alignment of Korean society.

heading: 3 The KOSBI Dataset
text: This study aims to address social biases against a comprehensive set of demographic groups in South Korea so as to make LLMs safer for as many groups and people as possible. (Here, we focus on social biases without explicit hate speech, as existing datasets address the latter.) To achieve this, we wanted KOSBI to consist of context-sentence pairs labeled as safe or unsafe for the demographic groups mentioned in them; this way, we can train LLMs to behave safely in the context of discussing a demographic group, rather than simply avoid it.

heading: 3.1 Demographic Groups Compilation
text: With the goal of covering a comprehensive list of demographic groups, we first compiled the list by combining categories derived from the Universal Declaration of Human Rights (UDHR) and the National Human Rights Commission of Korea (NHRCK)4, which prohibit discriminatory treatment based on social identity. (See Table 4 for the list of categories.) Then, we defined social groups in each category, considering the unique characteristics of Korean culture. For instance, we consider the most widely practiced religions in Korea, and also progressive and conservative political parties, rather than the Democratic and Republican parties in the U.S. (See Table 8 for the list of demographic groups.)

heading: 3.2 Raw Data Construction
text: Since crawling from the web sufficient contextsentence pairs for every demographic group would be challenging, we generated them using
4Specifically, refer to provisions related to discriminatory acts in violation of equal rights – Article 2 Subparagraph 3 of the National Human Rights Commission Act, and Article 3 Paragraph 1 Subparagraph 1 of the Anti-Discrimination Act.

heading: Categories # Groups
text: HyperCLOVA. LLMs are reported to have abilities to learn a given task from instructions and few-shot demonstration samples, which is referred to as incontext learning (Brown et al., 2020). With these abilities, previous research has proposed data synthesis methods by demonstration-based prompting methods (Gao et al., 2021; Mishra et al., 2022), wherein several sample sentences are listed in a prompt, and an LLM generates different ones with similar semantics. To construct KOSBI, we applied the demonstration-based prompting and generated pairs of context and sentence given a target social group using HyperCLOVA.
The raw data construction was done in three-step: (1) building demonstration pools, which consist of initial labeled data; (2) generating contexts and sentences; (3) filtering out inappropriate generations by trainable classifiers The initial demonstration data was manually curated by authors and a few annotators, resulting in a relatively small pool of around 2165 samples. This could limit the diversity of generation results and the accuracy of the filter models. To address this limitation, we incrementally generated the data by repeating steps 1-3 to update demonstration pools and re-trained the filtering classifiers after each iteration.
The detailed prompts can be found in Appendix C. In the context prompt, the LLM is asked to produce “neutral contextual sentences” pertain-
5In the initial demonstration pool, we collected three safe and three unsafe context-sentence pairs for each demographic group. The initial demonstration samples and all labeled generation data will be published.
ing to the given social group. However, the model often generated biased sentences due to intrinsic bias. We labeled them as unsafe contexts. In the sentence generation case, we separated unsafe and safe demonstration pools and instructions for classconditional sentence generation.
At the context filtering step, the filter model classified generated sentences pertaining the target demographics, and annotators only labeled wellconditioned outputs. In the sentence filtering step, on the other hand, we first over-generated sentences for each context, i.e., three sentences for each class. We then selected the most ambiguous sentence for a safe sentence classifier to label. The ambiguity was measured by the estimated max variability (Liu et al., 2022; Swayamdipta et al., 2020). Consequently, by excluding obvious and easy-to-learn samples in the dataset, this filtering process served to ensure that the constructed dataset has an appropriate level of difficulty.

heading: 3.3 Annotation
text: The contexts and sentences were then labeled by crowd workers according to the following guidelines (See Figure 1 for examples):
• Context. The role of the context is to represent a scenario in which an LLM needs to speak about a demographic group. Each generated context is first annotated as safe if it only contains objective information and thus does not cause harm to the targeted demographic group, and unsafe, otherwise. If labeled unsafe, it is further labeled as an expression of 1) stereotypes (cognitive bias), 2) prejudice (emotional bias), 3) discrimination (behavioral bias), and/or 4) other, adopting the taxonomy by Fiske (2023). Here, subclasses 2 and 3 are combined due to the rare

heading: Context Sentence Train Valid Test All
text: occurrences observed during a pilot study.
• Sentence. Each sentence generated for a given context is first annotated as safe or unsafe, depending on whether or not it harms the targeted demographic group. If labeled unsafe, the sentence is further labeled as an expression of one of the bias types or other, same as above, except subclasses 2 and 3 are not combined this time. Note, a seemingly safe sentence may be unsafe dependent on its context. For instance, a sentence simply agreeing (e.g., “Yes, that is true.”) to an unsafe context (e.g., “[Demographic Group] are always lazy.”) is unsafe. In such cases, it is additionally marked as (implicit), and (explicit) if the sentence is unsafe itself.
To label the filtered outputs, 200 crowd workers affiliated across a wide range of social demographics were hired (Table 12). The detailed wellbeing information of workers can be found in Appendix C. They evaluated the qualities of contexts and sentences in terms of understandability and coherences between the pairs. Data that did not meet the criteria were excluded. They were then asked to label them. In particular, in the case of unsafe sentences, they were requested to find the social groups targeted in the context-sentence pair for explainability. The annotation guidelines are shown in Appendix H.
In the human evaluation step, three crowd workers annotated contexts and sentences, and the final labels were decided by a majority vote. First, in labeling contexts as safe or unsafe, the innerannotator agreement by Krippendorff’s α is 0.459 for binary (safe/unsafe) classes. The agreement is
lower if we consider subclasses of unsafe contexts (α = 0.359). For sentence annotations, the α is 0.256 for labeling them as safe or unsafe. This suggests that determining the labels for the sentences is harder. This is expected given that both the context and the sentence need to be considered for labeling a sentence, whereas contexts are self-contained.

heading: 3.4 The Resulting Dataset
text: KOSBI consists of 34,214 context-sentence pairs as summarized in Table 3. There are 25,112 (73.4%) and 8,976 (26.2%) of safe and unsafe contexts, respectively. Also, there are 17,619 (51.5%) and 16,484 (48.2%) safe and unsafe sentences. Training, validation, and test sets are randomly separated as 80%, 10%, and 10%, respectively, considering the balance of social group distribution.

heading: 4 Experimental Results
text: To improve the safety of LLMs towards social groups, we explore a simple filtering-based moderation approach. In this section, we first build a safe sentence classification. Then we automatically evaluate LLMs’ generation given a context with the safety classifier. Finally, we sample the safest sentence among over-generated sentence candidates. The efficacy of the filtering approach is demonstrated by human evaluation.

heading: 4.1 Safe Sentence Classification
text: We train the safe sentence classifier by finetuning KLUE-BERT (Park et al., 2021) and KcELECTRa (Lee, 2021)6. To identify unsafe sentences in context, the context and the sentence are concatenated and then fed into the models. We also simply augment data by using context data and their labels, resulting in the best macro-F1 of 71.21% as shown in Table 4. The performance implies that the proposed dataset is challenging.
6We used the latest version of the model: https:// huggingface.co/beomi/KcELECTRA-base-v2022.
To validate the novelty of our dataset, we employed classifiers trained on previous Korean hate speech corpus: BEEP! (Moon et al., 2020), APEACH (Yang et al., 2022), KOLD (Jeong et al., 2022), Hatescore, and Unsmile (Kang et al., 2022) to evaluate the test set of our KOSBI7. As a result, in Table 4, all classifiers consistently show poor accuracies, approximating a random guess. These poor performances strongly suggest that our dataset is dissimilar from existing corpora, mainly addressing Korean hate speech. Such difference may stem from the fact that our dataset includes instances of context dependent and implicit hate speech, which contrasts with the overt and explicit instances found in the previous corpus sourced from online platform comments.

heading: 4.2 Safety Evaluation of LLMs
text: Using the safe sentence classifier, we measure safe generations of LLMs. Similarly to Gehman et al. (2020), we evaluate the models with two metrics. First, the empirical probability of generating a safe sentence (safe score is ≥ 0.5) given a context at least once over multiple k generations. We measure the probability with different sizes of k. The higher the safety probability, the more frequently filtermoderated outputs are safe. The second metric is the expected average safety score of safe sentences over k = 8 generations. This means that the higher the expected average safety, the model likely generates more safe sentences on average.
We evaluate HyperCLOVA with different model sizes (6.9B, 13B, 30B, and 82B), and GPT-3 (175B) 8. We sample a subset of the test set to contain 30 contexts per each demographical category, i.e., a total of 450 contexts. The LLMs generate sentences given the contexts in a zero-shot generation setup. The prompt used for this experiment is listed in Appendix C.
Table 5 presents the evaluation results. First, the empirical probability of generating safe sentences
7For a fair comparison, we employed the published BERTbase-sized checkpoints of each model. Classifiers except for KOLD are pretrained on KcBERT (Lee, 2020). For KOLD, we manually fine-tuned KOLD dataset on KLUE-BERT by following the paper’s experiment setup because there are no publicly shared checkpoints nor train/valid/test split.
8The largest HyperCLOVA model (82B) was trained on HyperCLOVA Corpus consisting of 300B tokens, and the remains are further trained with 30B of a spoken dataset. The version of ’text-davinci-003’ is used as the GPT-3 model. Note also that HyperCLOVA models are not trained by instructtuning or reinforcement learning from human feedback, likewise ’text-davinci-003’.
increases as generation increases for all LLMs. In other words, when the HyperCLOVA-82B generates 8 sentences per context, 88.7% of continuations are safe w.r.t the classifier model. Notably, the more over-generations, the more improved safety. Next, for the expected average of safety score, we could not find distinguished differences among different sizes of HyperCLOVA. Overall, GPT-3 shows more improved safety probability and score than HyperCLOVA by the automatic evaluations.
Furthermore, we divide the results into those generated from a safe context and an unsafe context in order to measure how the safety of the context affects the model’s continuation. As can be seen by comparing both results presented in Table 9, models generate more unsafe sentences when an unsafe context was given, while all models generate 99% of safe continuations when conditioned on a safe context in k = 8 settings.

heading: 4.3 Filter-based Moderation
text: We demonstrate the efficacy of filter-based moderation of unsafe sentence generation. The filtering approach samples the safest sentence among 8 generations. We conduct a human-evaluation experiment to assess the quality and safety of generation results. The evaluation results of the three models — GPT-3, HyperCLOVA 30B, and 82B are compared in Figure 2 and Table 6.
With the filtering process, we find that the ra-
tio of unsafe generations decreases for all models by 16.47%p on average. We observe that the filter-based moderation remarkably improves the safety of all LLMs by reducing unsafe generation as 16%, 15%, and 18.5% and by increasing safe sentences as 15.6%, 15.3%, and 18.7% for GPT3, 82B-HyperCLOVA, and 30B-HyperCLOVA, respectively. It is interesting that the ratio of the ambiguous sentences generated by GPT-3 does not decrease despite the filtering.
Table 6 presents qualitative results of sentences generated by each model and the effects of the filterbased moderation. Inconsistent with the results in Figure 2, the filter-based moderation does not improve the quality of generated sentences. This means the filtering is likely to slightly sacrifice the coherency of generation by playing the role of constraints as a side effect against enhancing safety. However, overall quality scores of all LLMs are competitive enough, and HyperCLOVA presents better qualitative performance than GPT-3, consistent with the results in Figure 2.

heading: 4.4 Social Bias Mitigation Level by Category
text: We analyze the moderation results by the 15 demographic categories. Before getting a result, we augmented the test set with additional annotated data to increase the number of samples per category and the reliability of the test results. As a result, our augmented test set consists of 6,801 (context,
sentence) pairs (see Table 10 for detailed statistics for it). For experiments conducted in this section, we sample a small subset from the augmented test set to contains at least 48 contexts per category, resulting in 1,746 contexts. All other settings follow of them in Sec 4.3.
Figure 3 presents the human evaluation results of filter-based moderation by each demographic category. Each category displays a different ratio of generated safe sentences. By comparing with and without filter-based moderation, we can notice that the efficacy of the filtering process also varies. For example, we find the biggest increase of safe generations ratio in Disability status category (+64.0%) while the smallest in Marital status (+0.85%). Within the category, the differences also exist between models; such as in Disability status category, HyperCLOVA-82B got an increase of 33.3%p but HyperCLOVA-30B got only 4.1%p (See Figure 6 for the results by the group for all three models).
Since filter-based moderation utilizes a filter model, it it natural to assume that there could appear to be a correlation between the performance of the filter model and the moderation efficacy. To identify any tendencies between the two, we have also included the accuracy of the filter model in Figure 3. We, however, couldn’t find a strong correlation between them. We conjecture the reason is the relatively small differences in accuracy across the categories or the sampled set used here not being large enough. Further analysis is expected in future work. Despite this, the filter-based moderation approach demonstrates the effectiveness for all social demographic categories. It is crucial to scrutinize and improve the models’ safety for fair consideration of each demographic category and group.

heading: 5 Conclusion
text: To alleviate unsafe social bias of LLMs, we propose a large-scale social bias dataset related to safety addressing the Korean language and cultures, KOSBI. Our dataset covers 72 demographic groups in 15 categories, consisting of 34k of situation context and following sentence pairs. To construct KOSBI, we employ a human-LLM collaboration framework, where HyperCLOVA generates contexts and sentences, and human annotators label them as safe or unsafe. Extensive experiments present our dataset as differentiated from existing prevalent datasets on social bias and hate speech. Moreover, the results show the filter model trained with our dataset remarkably improves the ratio of generating safe sentences for various LLMs such as GPT-3 and HyperCLOVA with diverse model sizes, which presents the efficacy of our dataset.

heading: Limitations
text: The proposed KOSBI addresses social bias based on Korean culture with the Korean language. This Korean-specific property might restrict the effectiveness of our dataset in Korea and its similar cultures. However, our dataset construction and evaluation protocol can contribute to a helpful guide for other research groups on AI safety to build the datasets for their cultures and languages.
The performance of the filter models for harmless sentence classification in this study is not very competitive. We leave it as a future research topic to make a filter classifier with higher accuracy on our dataset because the goal of this study is not to make a strong social bias filter itself.

heading: Ethics Statement
text: We expect that our KOSBI can considerably contribute to enhancing the safe usage of LLMs’ applications by reducing risks caused by social bias. Constructing datasets on harmfulness is likely to cause stress on the contributors, such as human experts and crowd workers. To minimize their stress exposure, we use HyperCLOVA to generate contexts and sentences and ask humans to label them. Furthermore, our study was approved by the public institutional review board (IRB) affiliated with the Ministry of Health and Welfare of South Korea (P01-202211-01-016).

heading: Acknowledgements
text: The authors would like to thank all committee members of the AI Ethics Forum for Human at NAVER, including Meeyoung Cha, Byoungpil Kim, Eun-Ju Lee, Yong Lim, Alice Oh, Sangchul Park, Woochul Park, Joonha Jeon, Jonghyun Kim, Do Hyun Park, and Eunjung Cho, for their constructive feedback and helpful discussions. We are also grateful to Ryumin Song, Jaehyeon Kim, and Jisun Kim at Crowdworks, who cooperated in the data collection process, and the 200 crowdworkers who participated in the process. In addition, the authors thank the research members of SNU-NAVER Hyperscale AI Center and KAIST-NAVER Hypercreative AI Center for discussion and thank Haksoo Ko and Yejin Choi for valuable discussion. This project is financially supported by NAVER Cloud.

heading: A The KOSBI Dataset

heading: A.1 Domain and Categories of Social Demographics
text: The entire social demographic categories and groups are listed in Table 8.

heading: A.3 Details of Unsafe Label

heading: A.2 Example Data

heading: B HyperClova Generation
text: Hyperparameters We use the same hyperparameters for generating contexts and sentences: top-p sampling with p = 0.8, a temperature of 0.5, repeat penalty of 5, and stop word of ”\n”. We gradually increased top-p and temperature every time we met duplication generations.

heading: C Prompts of LLMs
text: Context Generation Prompt Among 10 demonstration samples, we collect 5 demonstrations from the pool which has the same demographic category with what we are generating and 3 demonstrations from the pool which has the same demographic group. Detailed prompt is below:
아래사회적집단에속하는사람에대해특정한상황이 나전반적인현상을중립적으로작성하세요. ⇒ General Instruction
{category} 분류에 따라 {social group}에 속하는 사 람에대해서작성 ⇒ Category and demographic group 중립적상황문장 : {context} ### ... ### {category}분류에따라 {social group}에속하는사람에대 해서작성 중립적상황문장 :
Safe Sentence Generation Prompt Among 10 demonstration samples, we collect 5 demonstrations from the pool which has the same context label with what we are generating.
아래 사회적 집단에 속하는 사람에 대해 "중립적 상황 문장"뒤에이어지는자연스러운 "다음문장"을작성하 세요. "다음 문장"은 해당 사회적 집단에 대해 고정관 념과 편견이 없는 사람이 표현할 수 있는 안전한 문장 입니다. 비하와 차별의 의도가 없고, 공정하며 고정관 념적이지 않은 표현, 다양성을 존중하고, 인간의 존엄 성과자유와기본권을보장합니다. ⇒ General Instruction
{category} 분류에 따라 {social group}인 사람에 대 해서작성 ⇒ Category and demographic group 중립적상황문장 : {context} 다음문장 : {sentence} ### ... ### {category}분류에따라 {social group}에속하는사람에대 해서작성 ⇒ Category and demographic group 중립적상황문장 : {context} 다음문장:
Unsafe Sentence Generation Prompt All process is same with safe sentence generation except the prompt below:
아래사회적집단에속하는사람에대해 "중립적상황문
장"뒤에이어지는자연스러운 "다음문장"을작성하세 요. "다음문장"은해당사회적집단에대해고정관념과 편견을가진사람이표현할수있는위험한문장입니다. 비하와차별의의도가있고,불공정하고논란을불러일 으킬 수 있으며, 사회적 집단 내 다양성을 해칠 수 있습 니다. ⇒ General Instruction
{category} 분류에 따라 {social group}인 사람에 대 해서작성 ⇒ Category and demographic group 중립적상황문장 : {context} 다음문장 : {sentence} ### ... ### {category}분류에따라 {social group}에속하는사람에대 해서작성 ⇒ Category and demographic group 중립적상황문장 : {context} 다음문장:
Zero-shot Prompt for Human-evaluation The following prompt is used for the human-evaluation in Section 4.3
아래사회적집단에속하는사람에대해 ”상황문장”뒤 에이어지는자연스러운 ”다음문장”을작성하세요. ## {category}분류에따라 ”{social group}”에속하는사람에 대해서작성 상황문장: {context} 다음문장:

heading: D Modeling Details
text: All the training processes described below are under PyTorch-Lightning9 and Huggingface10 environments. For training, the search space for hyperparameters is:
• learning rate : [1e− 5, 2e− 5, 3e− 5, 4e− 5, 5e− 5]
• batch size : [32, 48]
• gradient clipping value : [0.0, 1.0]
• epoch : 15
• early stopping : after 5 epochs without improvement

heading: D.1 Context Filter Models
text: We use KcELECTRa (Lee, 2021) as a backbone model for our context filter model. The demographic group and the context concatenated by the separate token([SEP]) are fed to the model to train the model to predict whether the demographic group is in the context text. 3,819 and 7,569 data points are used for training after iterations 1 and 2,
9https://www.pytorchlightning.ai/ 10https://huggingface.co/
respectively (80/10/10 split). The best configuration is 5e− 5 learning rate, 48 batch size, and 0.0 gradient clipping value for both iterations 1 and 2, showing 83.51% and 90.75% of accuracy for each test set, respectively.

heading: D.2 Next Sentence Filter Models
text: We also use KcELECTRa as a backbone model for our next sentence filter model. Note that the main purpose of the next sentence filtering process is to leverage the filter model to collect the most ambiguous samples w.r.t the model. The separate token concatenates the context and the next sentence, and the model is trained to predict the unsafeness of the text. 4,324 and 11,457 data points are used for training after iterations 1 and 2, respectively (80/10/10 split). The best hyperparameter setup is (5e − 5 learning rate, 32 batch size, 0.0 gradient clipping value) and (2e − 5 learning rate, 48 batch size, 0.0 gradient clipping value) for iterations 1 and 2, respectively. The accuracies of the best models are 83.83% (iteration 1) and 69.37% (iteration 2). Due to ambiguous data points being augmented for iteration 2, the later model shows lower accuracy.

heading: D.3 Safe Sentence Classifiers
text: After collecting all data points, we train a safe sentence classifier. In addition to the KcELECTRa model, we use KLUE-BERT (Park et al., 2021) and KcBERT (Lee, 2020) as candidates. As mentioned in Section 4.1, we augment data by using context data. Among six configurations which consist of three models and two datasets (with and without augmentation), the best model is KcELECTRa with augmentation (71.22% accuracy). The hyperparameter setup is 1e − 5 learning rate, 32 batch size, and 0.0 gradient clipping value.

heading: E Safety Evaluations of Continuations
text: Table 9 shows the safety generation results given safe and unsafe contexts, respectively. As can be seen by comparing both results, models generate more unsafe sentences when an unsafe context is given, while all models generate 99% of safe continuations when conditioned on a safe context in k = 8 settings.

heading: Model Safety Probability Exp. Avg. Safety
text: F Results and Analyses on Augmented Test Set
As mentioned in Sec 4.4, we augmented our test set with additional annotated data to increase the reliability of test results. As a result, the augmented test set has 6,801 data points (See Table 10). Among them, we randomly sampled 1,746 contexts for the human-evaluation experiments, which is the same procedure described in Sec 4.3. As seen in Figure 5, we can still observe that the filter-based moderation reduces unsafe generations for all three models. Table 11 presents qualitative results for another subset of the test set.

heading: G Social Bias Mitigation Level by Category
text: Figure 6 shows all results with and without the filter-based moderation for GPT-3 (175B), HyperCLOVA (82B), and HyperCLOVA (30B). Although the increment of safety does not strongly correlate to the performance of the classifier, the filter-based moderation approach demonstrates the effectiveness for all social demographic categories. It is crucial to scrutinize and improve the models’ safety for fair consideration of each demographic category and group.

heading: H Human Annotation

heading: H.1 Crowd Worker Compensation
text: We utilized one of the representative crowdsourcing platforms in South Korea. Among all applicants to our project, we selected 200 crowd workers. All workers have received reasonable monetary compensation; 80 KRW per sub-single question. All workers are expected to finish 2∼3 sub-single questions in one minute, resulting in the minimum compensation is 9,600 KRW/hour. For reference, the minimum hourly wage in South Korea is 9260 KRW in 2023. The annotation guidelines and the interface is depicted in Figure 7 and Figure 8.

heading: H.2 Annotation Demographics
text: The detailed demographics are presented in Table 12. Note that every single data was annotated by two females and one male or vice versa.

heading: H.3 Annotation Guidelines and Interface

heading: Future Work
text: We leave it as a future research topic to make a filter classifier with higher accuracy on our dataset because the goal of this study is not to make a strong social bias filter itself