PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models

Authors: devansh  jain
priyanshu  kumar
samuel  gehman
xuhui  zhou
♡  thomas
hartvigsen ♢ maarten
carnegie  mellon university
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 01:51:58

Abstract: Recent advances in large language models (LLMs) have led to their extensive global deployment, and ensuring their safety calls for comprehensive and multilingual toxicity evaluations. However, existing toxicity benchmarks are overwhelmingly focused on English, posing serious risks to deploying LLMs in other languages. We address this by introducing PolygloToxicityPrompts (PTP), the first large-scale multilingual toxicity evaluation benchmark of 425K naturally-occurring prompts spanning 17 languages. We overcome the scarcity of naturally occurring toxicity in web-text and ensure coverage across languages with varying resources by automatically scraping over 100M web-text documents. Using PTP, we investigate research questions to study the impact of model size, prompt language, and instruction and preference-tuning methods on toxicity by benchmarking over 60 LLMs. Notably, we find that toxicity increases as language resources decrease or model size increases. Although instruction- and preference-tuning reduce toxicity, the choice of preference-tuning method does not have any significant impact. Our findings shed light on crucial shortcomings of LLM safeguarding and highlight areas for future research.

Main paper: 
 introduction: 
 Large language models (LLMs) are increasingly being deployed in global contexts (Pichai & Hassabis, 2023; Forbes, 2024) . Naturally, this has led to rapid advances in the multilingual capabilities of LLMs (Scao et al., 2022; Üst ün et al., 2024; Yuan et al., 2023) . However, current toxicity evaluation benchmarks and safety alignment methods (Christiano et al., 2017; Lee et al., 2024) overwhelmingly focus on the English language, leading to significantly less safe responses in non-English languages (Wang et al., 2023; Kotha et al., 2024; Yong et al., 2023) . The lack of a standard multilingual benchmark for evaluating toxicity poses significant challenges to non-English users and the development of safer multilingual models.We introduce POLYGLOTOXICITYPROMPTS (PTP), the first large-scale multilingual benchmark for evaluating neural toxic degeneration, defined as the propensity of LLMs to generate toxic text given a prompt (Gehman et al., 2020) . We create PTP by scraping over 100M documents from web-text corpora to collect naturally occurring toxic prompts. This results in 425K prompts in 17 languages ranging from non-toxic to highly-toxic prompts scored with PERSPECTIVE API. 1 POLYGLOTOXICITYPROMPTS provides three key improvements for multilingual toxicity evaluation, surfacing more toxic generations from LLMs than existing toxicity benchmarks (Figure 1 ). First, PTP covers 17 languages while existing toxic degeneration work predominantly focuses on English (Gehman et al., 2020; Lin et al., 2023a) . Second, existing multilingual toxicity evaluation testbeds such as Üst ün et al. ( 2024) and RTP-LX (de Wynter et al., 2024) are translations of REALTOXICITYPROMPTS (RTP; Gehman et al., 2020) , which can lack cultural nuances of toxicity and introduce deviations in toxicity, leading to underestimated toxic degeneration (Sharou & Specia, 2022; Costa-jussà et al., 2023) . Third, PTP's naturally occurring prompts are more representative of real-world inputs than recent works on jailbreaking (Deng et al., 2023; Wei et al., 2024) and adversarial prompt generation (Zou et al., 2023; Huang et al., 2023) , which lead to unnatural and often gibberish prompts. We evaluate 62 LLMs on POLYGLOTOXICI-TYPROMPTS to study the impact of prompt language, model size, alignment methods, and input prompt toxicity on toxicity. We find significant toxicity in multilingual models, especially as the availability of language resources decreases. We observe that toxicity increases with model size within a model family for base LLMs. Furthermore, while instruction and preference-tuning reduce toxicity in models, the choice of preference-tuning method does not impact toxicity. Finally, we find that (un)safety and toxicity are related, but distinct aspects of LLMs that require their own solutions. Overall, our findings shed light on crucial shortcomings of LLM safeguarding and highlight areas for future research, notably, the need for multilingual toxicity mitigation and further investigations into the impact of model hyperparameters on toxicity. Our evaluation benchmark will advance efforts toward combating the critical issue of neural toxic degeneration. 
related work: 
 Evaluating Toxicity using Web-text Corpora, Templates, And User-AI Interaction Data Early works on evaluation datasets for studying biases and toxicity in models were created using templates or scraping web-text corpora. Sheng et al. (2019) ; Nangia et al. (2020) ; Nadeem et al. (2021) use templated prompts to study social biases in pretrained language models. However, templates are focused on specific contexts such as demographic identities and not necessarily realistic. (Zhang et al., 2020) to study toxic degenerations in chatbots. Furthermore, the advent of instruction-tuned LLMs has led to studies of toxicity in real-world user-AI conversations. Zheng et al. (2024) and Lin et al. (2023a) collect user-AI interactions with automatic and manual toxicity annotations respectively to tackle a different toxic data distribution-namely instructions. However, most of these approaches are limited to English.Multilingual dataset curation for evaluating toxicity has utilized both manual and automated translation techniques. Recent work on AI safety evaluation (Wang et al., 2023; Yong et al., 2023; Deng et al., 2023) create multilingual safety benchmarks by translating monolingual benchmarks into other languages. They observe that LLMs are primarily safeguarded for English, leading to significantly unsafe generations in other languages, especially as availability of languages decreases. While these works are aimed towards the broader area of safety, the absence of a standard multilingual toxicity evaluation benchmark has also led researchers to translate prompts from REALTOXICI-TYPROMPTS into other languages, either automatically ( Üst ün et al., 2024) or using human annotations (de Wynter et al., 2024) . However, manual translations are expensive, not scalable, and can introduce cultural biases, whereas automated translations can introduce deviations in toxicity due to incorrect translations and hallucinations (Specia et al., 2021; Sharou & Specia, 2022; Team et al., 2022; Costa-jussà et al., 2023) .Evaluating Toxicity using Machine-Generated Approaches Besides human-generated or naturally occurring data, a wealth of recent work has explored using machine-generated approaches to curate datasets and methods for evaluating the toxicity and safety of LLMs. Hartvigsen et al. (2022) and Kim et al. (2022) generate adversarial prompts about minority groups using classifier-guided decoding and conversations with a toxic partner respectively. Extensive research has studied red teaming (Perez et al., 2022; Chao et al., 2023; Mazeika et al., 2024) and jailbreaking (Liu et al., 2023; Wei et al., 2024; Yu et al., 2023; Deng et al., 2023) to identify safety failures in LLMs and elicit harmful outputs. Furthermore, adversarial attack methods have also been shown to be effective against models without requiring substantial prompt engineering (Shin et al., 2020; Zou et al., 2023; Huang et al., 2023; Jones et al., 2023) . However, such methods involve extensive prompt engineering, often leading to unnatural and non-representative prompts or model-specific artifacts (Das et al., 2024) . Furthermore, the extent to which these methods work in non-English languages remains to be studied.While the literature on toxicity evaluation has grown rapidly, their predominant focus on English highlights the need for multilingual benchmarks on naturally occurring toxic input prompts. We address this gap with POLYGLOTOXICITYPROMPTS, a collection of 425K naturally occurring prompts across 17 languages for evaluating toxicity. 
polyglotoxicityprompts: 
 We create POLYGLOTOXICITYPROMPTS, a large-scale multilingual testbed to evaluate toxic degeneration in LLMs. It consists of 425K prompts extracted from web-text corpora paired with toxicity scores from PERSPECTIVE API. All 17 languages supported by PERSPECTIVE API are represented in our testbed, namely: Arabic (ar), Chinese (zh), Czech (cs), Dutch (nl), English (en), French (fr), German (de), Hindi (hi), Indonesian (id), Italian (it), Japanese (ja), Korean (ko), Polish (pl), Portuguese (pt), Russian (ru), Spanish (es), and Swedish (sv).We define toxicity as "a rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion" (Wulczyn et al., 2017; Borkan et al., 2019) . We use PERSPECTIVE API, 1 an industry-standard toxicity detection tool because it supports our 17 languages. Specifically, we use the TOXICITY score from PERSPECTIVE API, computed using the UTC (Unified Toxic Content Classification) framework (Lees et al., 2022) , composed of a Charformerbased transformer (Tay et al., 2022) . UTC is a Seq2Seq architecture pretrained with the mC4 corpus (Xue et al., 2021) and Perspective Pretraining Corpus (PPC). Additionally, PERSPEC-TIVE API utilizes a single-language CNN (Lecun et al., 1998) distilled from multilingual BERT models (Devlin et al., 2019) for German and Portuguese.We construct our dataset by scraping over 100M documents from the mC4 (Xue et al., 2021) and THE PILE (Gao et al., 2020) corpora as they contain multilingual texts from a variety of domains. We also leverage Pile Curse, 2 a subset of THE PILE scored using the bad words 3 list for our English split. We then extract TOXICITY scores with PERSPECTIVE API for all scraped documents. To obtain a stratified range of prompt toxicity, we sample 6250 documents from 4 equal-width toxicity levels ([0, 0.25), . . . , [0.75, 1]). We then split collected documents in half to form prompts and continuations, both of which are scored for toxicity. We provide preprocessing details, dataset statistics, and metadata analysis in Appendix A.The final dataset includes 25K naturally occurring prompts for each language, for a total of 425K prompts across 17 languages. Figures 10(a ) and 11(a) show the prompt toxicity and length distributions of our prompts for all languages. We create our prompts using documents instead of sentences (Gehman et al., 2020) . Thus, our prompts are much longer than REALTOXICITYPROMPTS, with an average length of approximately 400 GPT-4 tokens (cl100k base tokenizer).While the extraction of toxic content from web-text may appear straightforward, we encountered several challenges associated with the scarcity of multilingual toxicity. The mC4 corpus (Xue et al., 2021) filters toxicity by removing pages containing bad words. 3 As a result, we observe less than 0.01% toxicity rate out of 5M samples for ar, cs, fr, ko, id, it, nl, pl, and sv. However, consistent with previous findings (Zhou et al., 2021; Dodge et al., 2021) , we note that filtered datasets still exhibit toxicity, and observe higher toxicity rates for other languages.To attain a larger sample of toxic content for languages with low toxicity rates, we create synthetic high-toxicity data. Specifically, we translate toxic samples from the mC4 and THE PILE corpora into target languages using the NLLB-3.3B model (Team et al., 2022) . We use this process to create ≈ 70K translated prompts across 9 languages, which amounts to only 16.8% of our dataset. Contrary to prior works, we observe a Pearson correlation of 0.725 (p ≤ 0.001) between the toxicity scores of the original and translated samples across all languages, suggesting that low amounts of translated data are not necessarily an issue.4 PTP SMALL We also create PTP SMALL , a stratified sample of 5K prompts per language from POLYGLOTOXICITYPROMPTS to benchmark models with limited computational resources. We benchmark a large variety of models (N = 62) spanning different sizes and multilingual capabilities. We follow the taxonomy proposed by Albalak et al. (2024) and include LLMs trained purely with the language modeling objective (base) such as Llama2 (Touvron et al., 2023b) , Pythia (Biderman et al., 2023) , LLMs fine-tuned to follow instructions (instruct) such as Mistral-Instruct (Jiang et al., 2023) , and LLMs aligned with preference-tuning/alignment methods (preference) such as GPT-3.5-Turbo (Ouyang et al., 2022) and Zephyr (Tunstall et al., 2023) . In the subsequent section (Section 4), we explore a variety of research questions that require specific functionalities and thus use the appropriate subset of models for our analyses. We also note that the LLMs we benchmark are, to the best of our knowledge, the neural networks that are trained and possibly instruction and/or preference-tuned, without any possible safeguards or guardrails that may have been added onto the public interfaces of such LLMs, such as safety classifiers applied to the input/output of LLMs.Evaluation Metrics Given a toxicity evaluator t(•) and a language model f (•), t(•) predicts the toxicity score for a text, and f (•) provides a completion c = f (p) for a prompt p. We sample K completions per prompt and follow Gehman et al. ( 2020); Üst ün et al. ( 2024) to measure a model's toxicity over a set of prompts P. Specifically, we compute EXPECTED MAXIMUM TOXICITY, that is, what is the expected toxicity of a model's worst-case generations?, as EMT = 1 |P | ∑ p∈P max i t( f (p)) and the EMPIRICAL PROBABILITY (EP), that is, how frequently does a model generate toxicity, as the probability that a model generates at least one completion with toxicity score ≥ 0.5 in K completions. We additionally compute AVERAGE TOXICITY, that is, what is the model's overall toxicity?,as AT = 1 |P | ∑ p∈P 1 K ∑ K i=1 t( f (p)).Implementation Details We utilize PTP SMALL to benchmark LLMs due to the breadth of considered models and computational constraints. We use the TOXICITY score from PERSPECTIVE API as our toxicity evaluator t(•), K = 10 completions, temperature = 0.7, top p = 1, and a maximum generation length of 512 tokens for our experiments. We use Microsoft Azure's OpenAI API for GPT-3.5-Turbo (version 0301) with safety settings disabled, vLLM (Kwon et al., 2023) for decoder-only models, and Huggingface's TGI5 for encoder-decoder models. We only use the required prompt templates as stated in model cards, and do not provide any additional instructions. 
research questions: 
 To investigate multilingual toxic degeneration in a large suite of models, we obtain and score continuations for the 5K prompts per language contained in PTP SMALL (due to computational resource limitations). We find similar trends across all evaluation metrics and thus report only AVERAGE TOXICITY for brevity.Llama-2-13b-chat-hf 0.078 Llama-2-70b-chat-hf 0.088 Qwen-7B-Chat 0.091OpenHathi-7B-Hi-v0.1-Base 0.327 pythia-12b 0.327 pythia-6.9b 0.328Table 1 : Models with highest and lowest AT on PTP SMALL .Table 1 previews our findings for the models with the lowest and highest AVERAGE TOXICITY. We provide results for all models with languages categorized based on Joshi et al. (2020) 6 in Table 5 . Next, we explore specific patterns concerning prompt language, model size, alignment methods, and prompt toxicity below. Finally, we also compare toxicity and safety detectors using PERSPEC-TIVE API and Llama Guard Inan et al. (2023) respectively.Despite safety alignment, translations of harmful prompts from English to other languages can elicit harmful content from LLMs (Kotha et al., 2024; Yong et al., 2023; Deng et al., 2024) . Therefore, we study how toxicity varies with input prompt languages by benchmarking multilingual LLMs, namely GPT-3.5-Turbo (Ouyang et al., 2022) , Aya101 ( Üst ün et al., 2024), and Bloomz (Muennighoff et al., 2023) and evaluating AT for each language.Figure 3 shows that models have the lowest AT levels in ru (Russian) and nl (Dutch), consistent with Üst ün et al. (2024) . However, all models have highly toxic continuations in hi (Hindi) and cs (Czech). We hypothesize that the relatively small amounts of Hindi in most pretraining corpora and lack of safety alignment in Hindi leads to more toxic degenerations (Wang et al., 2023; Yong et al., 2023; Deng et al., 2024) . This hypothesis is corroborated by the fact that AT reduces as the availability of language resources increases (Table 2 ).Across models, we find that GPT-3.5-Turbo and bloomz-560m have the highest and lowest AT levels aggregated across all languages respectively. However, we hypothesize that the Prior work has shown that undesirable content generation can increase with model size and possibly pretraining dataset size (Bender et al., 2021; Tal et al., 2022; Smith et al., 2022; Touvron et al., 2023a) . We conduct a similar investigation on the impact of model size on toxicity. We first study these trends in base models such as Llama 2 (Touvron et al., 2023b) and Pythia (Biderman et al., 2023) , and later examine models with additional tuning (instruct, preference) such as Tulu 2 (Ivison et al., 2023) .We investigate the distribution of continuation toxicity for base LLMs, that is, models trained with only the language modeling objective. We observe a slight correlation between the number of parameters in the model and the continuation toxicity for base LLMs (r = 0.015, p < 0.001). Prior work has shown limited evidence of the dependence of model toxicity on size. For instance, Touvron et al. (2023a; b) find that toxicity increases with model size, whereas Gehman et al. (2020); Hoffmann et al. (2022) find that larger models are not necessarily more toxic. We hypothesize that toxicity might depend on model size within a model family only, and investigate this further with the Pythia suite.The Pythia suite provides models of varying sizes while keeping the pretraining data and other hyperparameters constant. We utilize these models for a controlled investigation of the impact of model size on toxicity using the English split of our dataset. Figure 4 shows an overall increase in toxicity with an increase in model size, which plateaus near 2.8b parameters (effect size of the difference between 2.8b and 12b is small, Cohen's d ≤ 0.1, p ≤ 0.1). This is consistent with prior works (Touvron et al., 2023a; b) . More specifically, we find that the toxicity levels in 1b+ Pythia models are comparatively higher than the smallest 70m model (Cohen's d ≥ 0.3, p ≤ 0.001). This implies that toxicity is a longtail phenomenon that large enough models (> 1b parameter count) are capable of capturing and demonstrating, akin to how larger models memorize better (Tirumala et al., 2022) .To investigate the impact of model size on toxicity for safeguarded LLMs, we benchmark Llama 2-Chat and Tulu 2-DPO models on English and other related languages (constituting top-10 languages in Llama 2's pretraining data) as shown in Figure 5 .We observe different trends in both model families when scaling from 7b to 70b -for Llama 2-Chat models, AT first decreases and then increases as the model size increases. In contrast, DPO alignment first increases and then reduces toxicity for Tulu 2 models as they are scaled to 70b parameters. However, such differences are small (Cohen's d < 0.15 for all combinations with 70b models).There seems to be no conclusive answer as to whether model size affects toxicity in safeguarded LLMs. We hypothesize that discrepancies concerning smaller safeguarded models such as lack of hyperparameter tuning or reward models trained toward generations by larger models, and challenges in unlearning harmful behavior (especially as model size decreases) could explain these results. Thus, future work is needed to investigate the specific effects of model sizes on toxic degeneration in safety-aligned models. Takeaway: Alignment methods don't impact toxicity.While prior work has shown that safety alignment leads to reduced toxicity levels in models (Touvron et al., 2023b) , the impact of different alignment methods on toxicity is yet to be studied. We investigate the impact of instruction-tuning and preference-tuning using different alignment methods, namely PPO (Schulman et al., 2017) , DPO (Rafailov et al., 2024) , KTO (Ethayarajh et al., 2024), and IPO (Azar et al., 2023) on toxicity. For preference-tuned models, we also study the effect of the method used to create preference data for preference-tuning or alignment.We first compare toxicity levels aggregated over base, instruct, and preference models (Figure 6 ). We find that, on average, base models have the highest toxicity (AT= 0.281; significantly different from instruct and preference models; Cohen's d = 0.40 and d = 0.43, respectively, p < 0.001). Furthermore, we find that instruct and preference models barely differ in toxicity (Cohen's d = 0.02, p < 0.001), though preference-tuned models have slightly lower toxicity on average.To study the impact of different preference-tuning methods, we benchmark models that have been trained on the same data but with different alignment methods. Specifically, we use the Archangel suite7 of Llama models (Touvron et al., 2023a) and TinyLLama8 (Zhang et al., 2024) models.Interestingly, we do not observe a considerable difference in the average toxicity exhibited by models trained with different alignment methods (Cohen's d < 0.1) (Figure 7 ). Moreover, this trend remains at different scales of 1b, 7b, and 13b, suggesting that specific choices of the preference-tuning method might not make as much of a difference as preference data on model toxicity. To investigate the influence of preference data curated with human and AI feedback, we benchmark Gemma 7B (Team et al., 2024) variants. Specifically, we compare gemma-7b-it, trained on human preferences, and zephyr-7b-gemma-v0.1,9 trained on AI preferences (Figure 8 ). We observe that AI feedback is better than human feedback for en, whereas human feedback shows lower toxicity levels for non-English languages. We emphasize toxicity results on the en split since both models were trained using English-only preference data, likely making multilingual prompts out-of-distribution. Furthermore, zephyr-7b-gemma-v0.1 is aligned using DPO which has been found to reduce multilingual capabilities (Ivison et al., 2023) , likely leading to higher toxicity for non-English languages.While this suggests that AI feedback reduces model toxicity, we hypothesize that the operationalization of toxicity might play a role. AI feedback relies on LLMs' definition of toxic content, which likely aligns better with PERSPECTIVE API's perception of toxicity rather than human perceptions, which are more nuanced and subjective (Sap et al., 2022) . Furthermore, curating datasets using models can result in the under-representation of more veiled toxicity (Han & Tsvetkov, 2020) and general data and topical skews (Das et al., 2024) .Recent work has seen rapid growth in studies on safety evaluation and safeguarding techniques (Ganguli et al., 2022; Mazeika et al., 2024) . For instance, Inan et al. (2023) develop Llama Guard, a Llama 2 model to classify safety risks in LLM inputs and responses. However, the extent to which toxicity and safety overlap is unclear. To fill this gap, we compare PERSPECTIVE API, a toxicity detector, and Llama Guard, a safety detector.Since Llama Guard only supports English, we compute scores for all models on the English split of PTP SMALL following the instructions in its model card. 10 We find that PERSPECTIVE API toxicity scores are generally well-aligned with Llama Guard scores (r = 0.78, p ≤ 0.001).However, Llama Guard and PERSPECTIVE API still capture distinct concepts. To analyze the differences between both evaluation methods, we examine the prompts and generations where the metrics differ the most (Table 6 in Appendix E). We observe that PERSPECTIVE API is better at detecting explicit toxicity, hate speech, and derogative language and provides extensive support for non-English languages. However, Llama Guard can identify subtle unsafe generations and extend to other axes of AI safety. Our findings suggest that LLM safety detectors may not be equipped to capture the full spectrum of toxicity.We investigate the relationship between input prompt toxicity and continuation toxicity at greater granularity, that is, without aggregating as in AVERAGE TOXICITY. Intuitively, we expect a model's propensity to generate toxic text to be proportional to the toxicity of the input prompt. Empirically, we find a Pearson correlation of 0.49 (p ≤ 0.001) between prompt toxicity and continuation toxicity. We also find that continuation toxicity spans the entire toxicity range, regardless of input toxicity score, indicating that non-toxic prompts can yield toxic continuations and vice-versa, corroborating Gehman et al. (2020) . Furthermore, we investigate the correlations between prompt and continuation toxicity across languages and model families in Appendix B.We examine the extent to which different model categories mirror input toxicity. We find that the continuation toxicity of base models is most strongly correlated with input toxicity (r = 0.65, p < 0.001). Surprisingly, preference models have a higher correlation between input and continuation toxicity (r = 0.49, p < 0.001), compared to instruct models (r = 0.44, p < 0.001). We find that this is due in large to low-toxicity prompts, for which preference models mimic the input (low) toxicity in continuations better (r = 0.43) than for high-toxicity prompts (r = 0.16). instruct models also show a stronger correlation between prompt and continuation toxicity for low-toxicity prompts (r = 0.32) than for high-toxicity ones (r = 0.18). This indicates that preference models better match input toxicity than instruct models, but predominantly in low-toxicity inputs, suggesting that preference models are better safeguarded against high-toxicity inputs.Finally, we study the ability of different data sources to elicit toxicity from LLMs. Specifically, we compare AVERAGE TOXICITY when generating continuations for naturally occurring prompts from PTP, RTP-LX (de Wynter et al., 2024) , and an automatically translated sample of user-LLM interactions from WildChat (Zhao et al., 2024) .11 Figure 9 shows that PTP consistently draws out higher AVER-AGE TOXICITY. While RTP-LX is comprised of naturally occurring prompts in English and their culturally-aware translations to other languages, we find that PTP is still able to capture more toxicity, likely due to longer prompt lengths, corroborating Anil et al. (2024) . Furthermore, we hypothesize that preference-tuning makes models less vulnerable to what users input into LLMs as opposed to naturally occurring toxicity, leading to higher toxicity levels elicited by PTP compared to WildChat. 
conclusion: 
 We present POLYGLOTOXICITYPROMPTS, the first large-scale multilingual benchmark of 425K naturally occurring prompts across 17 languages for evaluating toxic degenerations in LLMs. We benchmark 62 LLMs to study the impact of factors like prompt language, prompt toxicity, model size, instruction-and preference-tuning, and alignment methods on toxicity. We also compare toxicity and safety detectors to emphasize that toxicity and safety are related but distinct aspects. Overall, our findings highlight crucial gaps in current research around the need for multilingual safeguarding and emphasize further empirical and theoretical investigations of how toxic degeneration is affected by prompt language, model size, and alignment methods. 
 
 Appendix: We describe several limitations of our work. First, toxicity is subjective and our measure of toxicity may not cover all aspects of toxicity (Sap et al., 2022) . Human validations of toxicity would help corroborate our results, but the scale of our experiments, coupled with possible disagreements between annotators due to the subjective nature of the task make validations challenging (Cowan & Khatchadourian, 2003; Sap et al., 2019) . Second, we focus on naturally occurring prompts in web-text to create our benchmark, which may not be representative of user-LLM interactions (Lin et al., 2023b) or extensively cover conversational toxicity such as what might arise on social media (Dodge et al., 2021) . Third, our testbed does not extend to low-resource languages due to the lack of toxicity detection tools.Dataset Release The purpose of our work is to provide a standard multilingual benchmark to evaluate toxic degenerations in LLMs. As noted in the limitations, our prompts were extracted from naturally occurring web text and offer a limited representation of online data in general. While this mainly affects low-resource languages, it also skews the topics of online discussions (Dodge et al., 2021) . Our benchmark also doesn't cover more conversational toxicity such as what might arise on social media, which could be tricky to incorporate due to privacy issues (Elazar et al., 2024) . Finally, while our dataset includes toxic text, its intended use is not to increase the toxic outputs of a model unless the ultimate aim is to steer away from toxicity (Liu et al., 2021) . As a safety measure, we plan to release the dataset using AI2's ImpAct license12 which helps mitigate the risks of dual use of resources.Toxicity Detection Previous work has shown that toxicity detection tools overestimate toxicity in text containing minority identity mentions (Dixon et al., 2018; Hutchinson et al., 2020; Sap et al., 2019) . PERSPECTIVE API has also been shown to be biased against some languages such as German (Nogara et al., 2023) . Nevertheless, our benchmark uses it as one possible operationalization of toxicity. Moreover, it can serve as a resource for studying the construct validity of toxicity as measured by PERSPECTIVE API by providing stratified samples of web-text with ranges of both lower and higher toxicity scores. We release our benchmark and also encourage future work to apply other toxicity detectors as evaluations.Toxicity and Machine Translation Automatic translations can introduce deviations in toxicity due to incorrect translations and hallucinations (Specia et al., 2021; Sharou & Specia, 2022) . Team et al. (2022); Costa-jussà et al. (2023) show that automatic translations can also add toxicity across languages, introducing biases in toxicity evaluation on translated data.We provide our dataset and code to reproduce our benchmarking experiments and encourage toxicity evaluations in future work: https://anonymous.4open.science/r/ptp-5856Toxicity Detection Prior work has shown that frequent retraining of black-box toxicity detection APIs such as PERSPECTIVE API can lead to inaccurate comparisons and reproducibility challenges (Pozzobon et al., 2023) . Thus, we encourage readers to re-run toxicity evaluations instead of adopting results from the papers they are comparing to.Benchmarking Experiments We used up to 128 GiB RAM and 4 NVIDIA RTX A6000s to generate completions with LLMs with up to 70b parameters for our benchmarking experiments. There are several considerations for our benchmarking experiments. First, we use only one configuration of random sampling (temperature = 0.7, top p=1.0, maximum generation length = 512 tokens). There could be differences in toxicity levels depending on different sampling methods and configurations. Based on how toxicity might be a long-tail phenomenon akin to memorization (Tirumala et al., 2022) , we expect that the decoding algorithm might matter. Second, due to computation constraints, we use PTP SMALL to benchmark models. While PTP SMALL was randomly sampled from POLYGLOTOXICITYPROMPTS, running on the full dataset might surface more toxicity than our sampled data surfaced. Table 4 shows some sample prompts from POLYGLOTOXICITYPROMPTS from en, it and id.We investigate the correlations between prompt and continuation toxicity across languages and model families.Comparing Model Families TinyLlama (Zhang et al., 2024) , MPT (Team, 2023) , Pythia (Biderman et al., 2023) , and Archangel (Ethayarajh et al., 2023) tively; p ≤ 0.001). We find the lowest correlations between prompt and continuation toxicity for GEITje-7B (Rijgersberg & Lucassen, 2023) , Yi (Young et al., 2024) , Qwen (Bai et al., 2023) , and Tulu 2 (Ivison et al., 2023 ) models (r = 0.04, 0.26, 0.30, and 0.32 respectively; p ≤ 0.001), suggesting that these models have been better safeguarded for prompt toxicity.We find the highest prompt-continuation toxicity association across languages for en, cs and hi (r=0.60, 0.60, and 0.59; p ≤ 0.001) whereas ru, zh, sv exhibit the lowest correlations of r=0.36, 0.36, and 0.38 (p ≤ 0.001 in all cases). While further investigations are needed to explain these trends, we hypothesize that languages where models have high instruction following capabilities (such as English) more easily match input toxicity in their continuations, and those with low capabilities (such as Czech, Hindi) might behave more like base models which also match input toxicity very well.We compare toxicity levels elicited using prompts from different sources, specifically web text and user-LLM interactions. We utilize RTP-LX (de Wynter et al., 2024) and WildChat (Zhao et al., 2024) for our comparison and provide details for both. RTP-LX (de Wynter et al., 2024) culturally-aware toxic prompts for some languages. We use all the prompts provided for our comparison.WildChat WildChat (Zhao et al., 2024) is a corpus of 1M real-world user-ChatGPT interactions. We utilize user messages from WildChat. Although the WildChat dataset is multilingual, it predominantly contains English data. Hence, we translate existing prompts (using the same process as PTP) to create a stratified version of the dataset containing 1000 prompts across the four toxicity buckets. We do not split the user messages into half and use the entire text as prompts.Table 5 shows the statistics of Continuation Toxicity, Expected Maximum Toxicity, and Empirical Probability of a wide variety of models over the subset of our dataset aggregated over the resource of languages. https://huggingface.co/datasets/tomekkorbak/pile-curse-fullhttps://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-WordsWe discuss limitations with translating data in the Ethics Statement.https://github.com/huggingface/text-generation-inferenceSince all considered languages belong to categories 3 and above, we compare relative resource availability, that is, categories 3, 4 and 5 are referred as low-, medium-and high-resource respectively.https://huggingface.co/collections/ContextualAI/archangel-65bd45029fa020161b052430https://huggingface.co/collections/abideen/tinyllama-alignment-65a2a99c8ac0602820a22a46https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1https://huggingface.co/meta-llama/LlamaGuard-7bWe provide details about RTP-LX and WildChat in Appendix C.https://allenai.org/impact-licensehttps://huggingface.co/datasets/mc4https://spacy.io/https://github.com/openai/tiktoken 


Decision: The paper design and curate a large multilingual benchmark (245k) to measure toxicity across 17 languages; This is a valuable resource for future work, especially given that the multilingual prompts---for the most part---are not translation from English. The authors have evaluated a large number of LLMs (62) on this benchmark, providing insights on the role of various factors such as model size, etc on toxicity generation. I agree with the reviewers about the impact of this work, and encourage the authors to apply reviewers' comments. Another interesting question to explore is whether the performance of models on natural prompts correlate with the synthetically generated ones (and whether the latter can be useful replacement).

Reviews: review: paper_summary: paper_summary: The authors present POLYGLOTOXICITYPROMPTS (PTP). A multilingual toxicity
evaluation benchmark which consists of 425K prompts spanning 17
languages. Further, the authors use the PTP benchmark to investigate research questions to study the impact of model size, prompt language, and instruction and preference-tuning methods on toxicity by benchmarking over 60 LLMs.
strength_weakness: strength_weakness: The authors clarify that to attain a larger sample of toxic content for languages with low toxicity rates, they create
synthetic high-toxicity data. Specifically, they translate toxic samples from the mC4 and THE
PILE corpora into target languages using the NLLB-3B model to create ≈ 70K translated prompts across 9 languages. While the fact that this  amounts to only 16.8% of our dataset would have been a non-issue, and in-fact heralded in other contexts.. given that the goal of this paper is to be multilingual, it makes me wonder if this is incorrect in principle? While, I don't think this impacts the value of the rest of the dataset, the authors should make more explicit right upfront, or elsewhere!
questions: questions: Refer above. 

1. The README in the shared repository is not very helpful. It's just an abstract.
paper_summary: paper_summary: This paper provides a multilingual dataset for eliciting toxic responses in large language models. The authors systematically collected an extensive collection of *25000 prompts* per language covering *17 languages*. Furthermore, using the dataset, this paper provides a detailed analysis of toxicity in large language models by considering variables such as prompt language, model size, model alignment methods, and toxicity evaluation methods.
strength_weakness: strength_weakness: N/A
paper_summary: paper_summary: The paper introduces a new dataset, PolygloToxicityPrompts, consisting of 245K prompts in 17 languages with varying toxicity levels. It is inspired by RealToxicityPrompts and includes prompts in the 17 languages supported by Perspective API, which supplies toxicity ratings. This language set covers a range of language families while excluding low-resource languages. 

The paper evaluates 62 LLMs including base, instruction tuned, and preference tuned models. The experiments investigate the effects of per-language training data size, model size, alignment methods, and instruction following ability. (Experiments are run on a  subset of 5K prompts per language due to the large number of models investigated.)
strength_weakness: strength_weakness: I don't see any reasons to reject.
questions: questions: Given the source of the prompts in publicly available corpora, what role does memorization play in toxic outputs?
paper_summary: paper_summary: The paper introduces PolygloToxicityPrompts, a dataset designed to assess toxicity generation across 17 languages. Using this dataset, authors evaluate 62 Language Model (LLM) variants, varying in both scale and architecture. Results show that model size inside the same family as well as preference tuning have an impact on the amount of toxicity generated.
strength_weakness: strength_weakness: - Toxicity is a term difficult to define and evaluate. This paper uses third-party metrics to evaluate toxicity but it would be interesting to know better what aspects they are measuring in more detail. Toxicity could be evaluated as the generation of words that are considered toxic (e.g. HolisticBias (https://arxiv.org/abs/2305.13198)), but that would ignore toxic examples that do not include toxic words out of context. (e.g. "I think this reviewer is not intelligent", not of the words could be considered toxic if isolated).
questions: questions: - What kind of toxicity is present in the dataset? It would be interesting to have more information about the specific phenomena in the dataset and how accurate are PERSPECTIVE API and LLama Guard on identify it.
paper_summary: paper_summary: The authors evaluated over 60 large language models (LMMs) with respect to their toxicity generation. They relied on a benchmark of 425K prompts that were evaluated for 17 languages. The toxity score was calculated with the PERSPECTIVE API tool. During the study, they evaluated the model size, language, among other attributes. The paper is well written and includes a good related work section.
strength_weakness: strength_weakness: - It is unclear whether the selected tool (PERSPECTIVE API) is reliable for calculating the score. 
- Other additional scoring tools could have been used for comparison, even if just for some of the languages.
- It is unclear whether any of the resources compiled in this work (dataset, prompts, scores, outputs) will be made available.
questions: questions: - I wonder why just one industry tool was used for calculating the score and whether other tools could have been considered for comparison, even if just for some of the languages. How good does PERSPECTIVE API scores wrt. to other tools? 

- I wonder whether any of the resources compiled in this work will be made available.

* minor comments:

- section 3.2: please define prompts and continuations.

- section 3.2: please explain GPT-4 tokens wrt. other tokens.

- Figure 3: please clarify why toxity is high if scores are under 0.5 (threshold defined in section 3.4).

- section 4.3: please define base, instruct, and preference, as well as the different preference-tuning methods (alignment).

Comments: title: Thanks for the clarification
comment: Thank you for the follow up! We apologize for the delay in updating the README, we have now updated it with details about the dataset. Due to our dataset's size and Github's storage constraints, we are currently unable to add code to the repository. We plan to move our dataset to huggingface for easier access and update the repository with code to evaluate toxic degenerations in arbitrary LLMs after the discussion period to maintain anonymity. We hope that you find the README more helpful now.
Thank you! 
With regards to the shared repository, the README is still not updated? I'm slightly confused on the utility and motive of sharing an anonymous github repository without a bare minimum README to navigate it? Right now, all it has is the abstract from the paper.
Great, once again, thank you for the good work!
Thanks for the reply. The explanation of multiple samples at high temperature to minimize memorization effects sounds like a reasonable approach.
I thank the authors for clarifying the various points that I raised. I have no further questions.