Venue: NAACL 2025

# SynthDetoxM: Modern LLMs are Few-Shot

Parallel Detoxification Data Annotators

Daniil Moskovskiy1,2\({}^{*}\) Nikita Sushko1,2\({}^{*}\) Sergey Pletenev1,2 Elena Tutubalina1,3,45 Alexander Panchenko2,1

Ebana Tutubalina1,3,45Alexander Panchenko2,1

###### Abstract

Existing approaches to multilingual text detoxification are hampered by the scarcity of parallel multilingual datasets. In this work, we introduce a pipeline for the generation of multilingual parallel detoxification data. We also introduce **SynthDetoxM**, a manually collected and synthetically generated multilingual parallel text detoxification dataset comprising 16,000 high-quality detoxification sentence pairs across German, French, Spanish and Russian. The data was sourced from different toxicity evaluation datasets and then rewritten with nine modern open-source LLMs in few-shot setting. Our experiments demonstrate that models trained on the produced synthetic datasets have superior performance to those trained on the human-annotated MultiParaDetox dataset even in data limited setting. Models trained on **SynthDetoxM** outperform all evaluated LLMs in few-shot setting. We release our dataset and code to help further research in multilingual text detoxification.

Warning: this paper contains illustrative examples of texts that readers may find offensive or disturbing.

languageresourceLanguage

Our dataset is comprised of 16,000 high-quality synthetic detoxification pairs across four languages: German, Spanish, French and Russian. The dataset was created using few-shot prompting and selecting the best generations of five different open-source LLMs. The answers were combined using a handcrafted heuristic, providing the best answers from each model to ensure diversity and quality of the final data.

Our contributions can be summarized as follows:

1. We propose a framework for generating synthetic parallel multilingual detoxification data using few-shot prompting of LLMs.
2. We create SynthDetoxM, a large-scale multilingual synthetic parallel dataset for text detoxification, helping to address the data scarcity issue in the detoxification task.
3. We conduct a thorough empirical evaluation of the proposed dataset, including linguistic analysis of the data and benchmarking against the human-annotated MultiParaDetox.

We openly release the generated data and code.1

Footnote 1: github.com/s-nlp/synthdetoxm

## 2 Background and Related Work

### Text Style Transfer

Text Style Transfer (TST), the task of rewriting the text in a target style while preserving its semantic content and fluency, has garnered significant attention in the natural language processing community due to its potential applications in text generation Fu et al. (2018). TST encompasses various subtasks, including formality style transfer Wang et al. (2020); Lai et al. (2021), sentiment style transfer Yu et al. (2021), authorship style transfer Horvitz et al. (2024); Liu et al. (2024), and detoxification Dale et al. (2021); Atwell et al. (2022); Moskovskiy et al. (2022, 2024).

With the advent of Large Language Models (LLMs), in-context learning methods have increasingly been utilized for TST and detoxification tasks. Suzgun et al. (2022) proposed a novel approach to TST by prompting LLMs and then reranking the generated texts based on three TST metrics: text similarity, target style strength, and fluency. Similarly, Reif et al. (2022) demonstrated the effectiveness of prompting GPT-3, a state-of-the-art LLM at the time, to rewrite texts in a desired style.

### Text Detoxification

Text Detoxification, a subtask of Text Style Transfer (TST), involves transforming an input text \(x_{i}\), identified as toxic through toxicity estimation models, into a text \(y_{i}\) that is non-toxic in style while maintaining semantic similarity and fluency. In this context, toxicity refers to language that is harmful, offensive, or inappropriate.

Due to the lack of parallel training data, early research focused on unsupervised detoxification methods dos Santos et al. (2018); Dale et al. (2021); Hallinan et al. (2023); Pour et al. (2023). For instance,Logacheva et al. (2022) and APPDIA Atwell et al. (2022), has enabled the training of sequence-to-sequence models Logacheva et al. (2022); Pour et al. (2023) that outperform most unsupervised approaches in terms of rewritten toxicity, fluency, and semantic similarity. In parallel, Moskovskiy et al. (2024) explored the use of activation patching in LLMs to generate synthetic parallel detoxification data for English. Their results demonstrated that training detoxification models on this data yields performance comparable to models trained on manually annotated datasets in automatic evaluations, while achieving superior quality in human assessments.

Figure 1: An illustration of the proposed approach for collecting and generating the multilingual text detoxification dataset SynthDetoxM.

 



### Multilingual Text Style Transfer

The scarcity of high-quality parallel multilingual detoxification data remains a major challenge in the field. Recently, new non-English parallel datasets have been introduced for various TST tasks, including a Bangla language parallel sentiment style transfer dataset Mukherjee et al. (2023) and the extension of the GYAFC dataset to Portuguese, French, and Italian, resulting in XFORMAL Briakou et al. (2021). Following the crowdsourcing pipeline introduced by Logacheva et al. (2022), a parallel text detoxification dataset for Russian was collected Moskovskiy et al. (2022). Later, using the similar data annotation pipeline, Dementieva et al. (2024) collected 1000 sentence pairs across nine languages, resulting in the MultiParaDetox dataset for a corresponding shared task on multilingual text detoxification. Furthermore, in a more recent work Dementieva et al. (2025), provide an in-depth analysis of toxicity characteristics across languages, exploring descriptive linguistic features that influence detoxification quality.

Nevertheless, the size of MultiParaDetox is far from satisfactory with 1000 sentence pairs per language, only 400 of which are publicly available. The remaining 600 pairs comprised the test set for the multilingual text detoxification shared task Dementieva et al. (2024). Such relatively small dataset may be insufficient for training big multilingual language models for multilingual text detoxification.

To bridge this gap, we present SynthDetoxM - a synthetic parallel detoxification corpus for four European languages, namely, German, Spanish, French, and Russian, with 4000 samples for each language. The dataset creating pipeline presented in our work can easily be transferred to other languages as well, drastically reducing the cost of annotation for parallel detoxification datasets.

## 3 Methodology

In this section, we describe the pipeline introduced for collecting the multilingual parallel text detoxification dataset, SynthDetoxM. A general illustration of our approach is shown in Figure 1.

### Data Collection

To create SynthDetoxM, we begin by selecting several thousand non-parallel toxic texts from publicly available toxicity identification datasets. We focus on four languages for SynthDetoxM: German, French, Spanish and Russian. From these datasets we select only the texts that were marked as toxic by human annotators, excluding non-toxic examples. In cases of multiple annotations, we retained the sample where the majority of annotators classified the sentence as toxic.

To enhance the final data quality, we employ sample-level filtering using the **STA** and **SIM** metrics2 and we also apply data augmentation techniques utilizing the Perspective API Lees et al. (2022). Since the API returns both toxicity scores and toxic spans, we further improve data quality by splitting the source texts into sentences and removing those sentences that do not intersect with the detected toxic spans. This filtering process results in a larger dataset of toxic sentences, and we also split overly long inputs into separate examples.

Footnote 2: Evaluation metrics are described in Section §4.3

RussianFor the Russian dataset, we use data from the Jigsaw Toxic Comments Classification Challenge Kivlichan et al. (2020), Russian Language Toxic Comments Belchikov (2019) and Toxic Russian Comments Semiletov (2020). From these sources, we select only those rows labeled as toxic, resulting in more than 15,697 toxic texts. We then calculate the **STA** and **SIM** metrics, applying a threshold of 0.5 for filtering. After removing emojis, eliminating texts with fewer than five words or more than 30 words, and splitting the sentences using toxic spans from Perspective API, our final dataset consists of 15,697 texts.

GermanFor German, we use the toxicity identification data from the GermEval 2021 shared

Figure 2: Number of accepted samples in the final SynthDetoxM dataset with respect to the LLM by language.



task (Risch et al., 2021) and RP-Mod and RP-Crowd (Assenmacher et al., 2021) to create a dataset of 4,946 toxic texts. We apply the same filtering and augmentation pipeline as for the Russian dataset, but with a lower **STA** score threshold of 0.3. This resulted in a dataset of 4,946 texts, which exceeds the original size of the raw dataset. We attribute this increase to the higher median length of German sentences, which leads to a greater number of split texts.

SpanishFor Spanish, we use data from two sources: the Jigsaw Toxic Comments Classification Challenge (Kivlichan et al., 2020) and the Clandestino dataset (Capozzi et al., 2021). This results in an initial dataset of 10,260 toxic texts. We apply the same filtering and augmentation pipeline as for the German dataset, maintaining the same **STA** threshold of 0.3. The final dataset contains 5,826 texts.

FrenchFor French, we use the data from the MLMA Hate Speech Corpus (Ousidhoum et al., 2019) and the Jigsaw Toxic Comments Classification (Kivlichan et al., 2020). These data sources combined produce a dataset of 5,424 toxic texts. As the toxicity classifier used in other languages does not support French, we instead use the Perspective API to get toxicity scores. After the application of the **STA** score threshold of 0.25, we obtain a dataset of 4,310 sentences.

#### 3.1.1 Parallel Data Generation Pipeline

To generate parallel detoxification data, we use various open-source LLMs in a few-shot generation setup. Specifically, we employed the following models: Qwen 2.5 32B by Qwen (Yang et al., 2024; Team, 2024); Command-R 32B by Cohere (Cohere, 2024); Gemma 2 27B by Google (Riviere et al., 2024); Aya Expense in 32B and 8B versions by Coherence (Dang et al., 2024); Mistral Small 22B, Mistral Nemo 12B by Mistral AI (Mistral, 2024a,b); and Llama 3.1 70B and 8B models respectively, by Meta (Dubey et al., 2024). While not all these models are explicitly designed for multilingual tasks, our experiments show that all of them support the languages considered in this work.

#### 3.1.2 Few-Shot Example Mining

To select the best toxic and non-toxic pairs for few-shot generation, we calculate the **STA** and **SIM** metrics for all sentences in Russian, German and Spanish from the multilingual toxicity detection dataset3. We then rank the top 10 sentences based on the following score:

Footnote 3: hf.co/multilingual_toxicity_dataset

\[{\bf Score}(x_{i};y_{i})=\] \[1-\left(\frac{1-{\bf STA}(x_{i})}{1-{\bf STA}(y_{i})}\cdot(1-{ \bf SIM}(x_{i};y_{i}))\right)\]

where \({\bf STA}(x_{i})\) and \({\bf STA}(y_{i})\) represent the toxicity scores for the original and detoxified examples, respectively. \({\bf SIM}(x_{i};y_{i})\) is the cosine distance between the embeddings of toxic and detoxified sentences.

This ranking criterion is chosen to ensure high-quality detoxification without altering the original meaning of the sentences. Since the sentences used for few-shot prompting have been annotated by human experts, we expect the detoxification quality to be satisfactory. Additionally, the rewriting process of toxic words often leads to an expanded distance between toxic and non-toxic sentences, increasing the distinction in non-toxicity. To maximize both the distance and the distinction between the original and detoxified sentences, we select harder and more meaningful examples for few-shot prompting, which helps improve the detoxification process.

For French, which is not represented in the MultiParaPetox dataset, we used human annotators to detoxify 10 randomly chosen sentences from the existing non-parallel data.

After generating detoxified examples, we perform refusal filtering using a refusal classification model (see details in Appendix D). Additionally, we use a simple threshold-based non-detoxifiability metric, calculated by dividing the absolute reduction in the STA score by the original STA score. We compare the resulting detoxifiability scores to a fixed threshold of 0.5. If the score falls below this threshold, the example is considered non-detoxifiable.

After generating five detoxification datasets in each language using the selected models, we rank the sentences by their multiplied **STA** and **SIM** metrics and select the top-scoring examples. This metric helps mitigate issues such as refusal(where models refuse to generate text due to toxicity) and copy-paste generation (where the model generates the input toxic sentences without modification), as copy-paste generation typically results in a low **STA** score, while refusal leads to a low **SIM** score.

 



### Final Composed Dataset

After all preprocessing, cleaning and filtering steps we compose SynthDetoxM - a manually collected and synthetically paraphrased parallel detoxification dataset on 16,000 toxic and non-toxic text pairs for Spanish, German, Russian and French.

We show the statistics of detoxification candidate acceptance with respect to each LLM Languagewise in Table 5 and Figure 2. According to the statistics, Qwen 2.5 generated the most preferrable detoxifications among other models.

However, upon manual examination we noticed that Qwen tended to occasionally insert tokens of Chinese text into the generated text though was prompted to answer only on the language of the source text. Therefore, the strict reranking and filtering criteria of generated detoxification candidates is necessary.

### Data Quality Evaluation Pipeline

To evaluate the quality of our generated detoxification data in Russian, German, and Spanish, we use our dataset for training and compare the performance of models trained on SynthDetoxWith those trained on the human-annotated parallel detoxification dataset, MultiParaDetox Dementieva et al. (2024). Due to its absence in the MultiParaDetox dataset, French is excluded from this comparison. A more detailed linguistic analysis of the dataset can be found in Appendix E.

## 4 Experimental Setup

### Data Quality Tests

To evaluate the efficacy of our SynthDetoxM for German, Spanish and Russian, we've trained a series of sequence-to-sequence models on different folds from the dataset. Since MultiParaDetox consists of only 400 pairs of toxic texts with their human-written non-toxic rebrasings, we split our created SynthDetoxM dataset into 10 chunks of 400 pairs for German, Spanish and Russian. We trained 10 mT0 models on different chunks of the dataset and evaluated their average performance on the MultiParaDetox test set. Additionally, we test if using both our SynthDetoxM and MultiParaDetox for training would lead to improved performance.

### Toxicity and Similarity of Synthetic Texts

To further assess the quality of the generated data, we computed the **STA** and **SIM** scores using the Perspective API for Russian, German, Spanish, and French. These metrics were selected for their relevance to detoxification tasks and their ability to quantitatively assess our synthetic dataset. We also assessed the quality of the French subset of SynthDetoxM, as French is not represented in the MultiParaDetox dataset, and therefore cannot be evaluated through model training. The scores are presented in Figure 3 and Table 2.

The results indicate that French achieves comparable automatic metric scores to other languages, suggesting that detoxification models trained on this data would perform similarly. Therefore, we hypothesize that the French subset of SynthDetoxM is a valuable addition to the dataset, enabling the training of effective detoxification models for French language processing tasks.

### Automatic Evaluation Setup

To assess the quality of the generated Spanish, Russian, and German data, we follow the evaluation pipeline of Dementiava et al. (2024), developed for the multilingual text detoxification shared task. These metrics are inspired by prior work on monolingual text detoxification for English and Russian Logacheva et al. (2022); Moskovskiy et al. (2022).

Style Transfer Accuracy (STA)For computation of this metric we use a multilingual toxicity classifier based on a multilingual XLM-R4 Conneau et al. (2020) text classification model, trained on a binary toxicity detection dataset.

Footnote 4: hf.co/textdetox/xlmr-large-toxicity-classifier

Content Similarity (SIM)For computation of this metric we use the cosine distance between LaBSE5 embeddings Feng et al. (2022) of the source texts and the generated texts.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**STA\({}_{T}\uparrow\)****STA\({}_{D}\uparrow\)****SIM\({}^{\dagger}\)****STA\({}_{D}\times\)****SIM\({}^{\dagger}\)**} \\ \hline German & 0.389 & 0.853 & 0.793 & 0.675 \\ Spanish & 0.514 & 0.920 & 0.736 & 0.681 \\ French & 0.583 & 0.913 & 0.677 & 0.624 \\ Russian & 0.467 & 0.924 & 0.731 & 0.678 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Average toxicity levels across different languages for source toxic (T) and generated detoxified (D) texts, along with similarity scores. **STA\({}_{T}\)** represents the toxicity level of the original text, while **STA\({}_{D}\)** corresponds to the detoxified text.



Fluency (FL)Fluency assesses how closely detoxified texts resemble human-written references. Previous works on English text detoxification have employed CoLA-based classifiers to estimate text fluency Dale et al. (2021); Logacheva et al. (2022); Moskovskiy et al. (2024). However, due to the absence of CoLA datasets for all considered languages in the shared task, Dementieva et al. (2024) used ChrF1 as a substitute. While recent work by Zhang et al. (2024) has introduced MELA, a multilingual extension of CoLA dataset covering all the languages in this study, we maintain the evaluation pipeline from the shared task and continue using ChrF1. Nonetheless, ChrF1 remains a coarse approximation of text fluency, which may negatively impact the overall **J** scores (see Appendix C for details).

Joint score (J)The metrics **STA**, **SIM** and **FL** are subsequently combined into the final **J** score which is used for the final ranking of approaches. Given an input toxic text \(x_{i}\) and its output detoxified version \(y_{i}\), for a test set of \(n\) samples:

\[\textbf{J}=\frac{1}{n}\sum^{n}_{i=1}\textbf{STA}(y_{i})\cdot\textbf{SIM}(x_{ i},y_{i})\cdot\textbf{FL}(x_{i},y_{i})\text{,}\]

where **STA**(\(y_{i}\)), **SIM**(\(x_{i},y_{i}\)), **FL**(\(x_{i},y_{i}\)) \(\in[0,1]\) for each text detoxification output \(y_{i}\).

### Baselines

In our work we adopt the baselines for multilingual detoxification described in MultiParaDetox Dementieva et al. (2024) that are inspired by prior works on text detoxification Logacheva et al. (2022); Dementieva et al. (2023).

Duplicateis the simplest baseline possible which copies an input toxic sentence. This baseline has \(1.0\) (or \(100\%\)) **SIM** score by definition.

Deleteremoves the toxic words according to a predefined list of inappropriate words. Dementieva et al. (2024) collects the lists of such toxic keywords for all target languages based on openly available sources. These lists are available online6.

Footnote 6: hfc/multilingual_toxic_lexicon

Backtranslationhas proven to be effective in previous works Dementieva et al. (2023); Prabhumoye et al. (2018); Konovalov et al. (2016a). Following Dementieva et al. (2023) we translate texts into English with NLLB translation model Costajussa et al. (2022)7. The translated data is then detoxified with the ParaDetox BART Logacheva et al. (2022) model8. After that, the detoxified texts are translated back into the source language using NLLB.

Footnote 7: hfc/facebook/nllb-200-distilled-600M

### Training Configuration

In our experimental evaluation of the generated multilingual parallel detoxification dataset, SynthDotaxM, we adopt the most efficient approaches from the TextDetox 2024 Shared Task Dementieva et al. (2024). The top three solutions in the automatic evaluations utilized fine-tuning of

Figure 3: Distribution of STA toxicity scores of toxic and neutral examples in the dataset. The original toxic texts are in orange, while detoxified texts are in blue. For readability we apply Gaussian smoothing.

 

the multilingual encoder-decoder language model mT0 (Muennighoff et al., 2023).

We use mT0-XL model9 and perform fine-tuning in full precision (fp32). We use AdaFactor optimizer (Shazeer and Stern, 2018) with batch size of \(16\), \(50\) warmup steps and set maximum sequence length to \(512\).

Footnote 9: hf.co/bigscience/mt0-xl

We fine-tune mT0 for \(2\) epochs in all setups. According to our experiments, the increased number of training epochs does not increase the final performance of the model. This might be explained to the overall training data scarcity compared to the size of the model: mT0-XL has 3 billion parameters and is being fine-tuned on 1,200 samples (400 for each of the three languages).

## 5 Results

Table 3 presents the results of our experimental evaluation of SynthDetoxM. For clarity, the table is divided by language. We compare the performance of mT0-XL trained on human-annotated MultiParaDetox data (MPD) with mT0-XL fine-tuned on two subsets of SynthDetoxM: a subset of 400 samples per language, matching the MPD size (denoted as SDM (Subset)), and the full SynthDetoxM dataset. Additionally, following prior work (Xu et al., 2023), we investigate whether a two-stage fine-tuning approach--first on SynthDetoxM, then on MPD (denoted as SDM + MPD)--yields further improvements.

The highest SIM scores for German and Russian are achieved by mT0-XL trained on MultiParaDetox (0.954 and 0.886, respectively), with the two-stage training approach (SDM + MPD) yielding slightly higher similarity in Russian but lower in German. However, for STA, models trained on SynthDetoxM consistently outperform MultiParaDetox across all languages, both when trained on the full dataset and on a similarly sized subset.

Models trained on SynthDetoxM exhibit slightly lower FL scores, likely due to the reference-dependent nature of this metric. Despite this, the aggregated J metric--strongly influenced by FL--is significantly higher for models trained on both the full SynthDetoxM and its subset compared to MultiParaDetox. Notably, incorporating Multi

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **German** & **Spanish** & **Russian** \\ \hline Human References & \(0.733\) & \(0.709\) & \(0.732\) \\ \hline \multicolumn{4}{c}{**Baselines**} \\ \hline Duplicate & \(0.287\) & \(0.090\) & \(0.048\) \\ Delete & \(0.362\) & \(0.319\) & \(0.255\) \\ Backtranslation & \(0.233\) & \(0.275\) & \(0.223\) \\ \hline \hline \end{tabular} 
\begin{tabular}{l c c c} \hline \hline \multicolumn{4}{c}{**mT0–XL supervised fine-tuning**} \\ \hline MultiParaDetox & \(0.446\) & \(0.344\) & \(0.472\) \\ SDM (Subset) & \(0.460\) & \(0.402\) & \(0.475\) \\ SDM & **0.482** & **0.470** & **0.546** \\ \hline \multicolumn{4}{c}{**10-shot LLM prediction**} \\ \hline Gamma 2 & \(0.353\) & \(0.380\) & \(0.404\) \\ Mistral Nemo & \(0.286\) & \(0.290\) & \(0.258\) \\ Mistral Small & \(0.371\) & \(0.308\) & \(0.273\) \\ Command R & \(0.328\) & \(0.344\) & \(0.402\) \\ Qwen 2.5 & \(0.402\) & \(0.443\) & \(0.428\) \\ Llama 3.1 8B & \(0.394\) & \(0.341\) & \(0.357\) \\ Aya Expanse 8B & \(0.305\) & \(0.246\) & \(0.225\) \\ Aya Expanse 32B & \(0.399\) & \(0.320\) & \(0.323\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Text detoxification results in terms of J scores for German, Spanish, and Russian languages. The best overall results are boldfaced. The baselines and human references are from (Dementieva et al., 2024).

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **STA** & **SIM** & **FL** & **J** & **STA-SIM** \\ \hline \multicolumn{4}{c}{**German**} \\ \hline MPD & \(0.722\) & \(0.848\) & \(0.602\) & \(0.383\) & \(0.612\) \\ SDM (Subset) & \(0.681\) & \(0.912\) & \(0.745\) & \(0.463\) & \(0.597\) \\ SDM & **0.728** & \(0.899\) & \(0.734\) & **0.484** & **0.655** \\ SDM+MPD & \(0.615\) & **0.954** & **0.821** & \(0.483\) & \(0.586\) \\ \hline \multicolumn{4}{c}{**Russian**} \\ \hline MPD & \(0.748\) & \(0.852\) & \(0.643\) & \(0.434\) & \(0.637\) \\ SDM (Subset) & \(0.858\) & \(0.850\) & \(0.656\) & \(0.478\) & \(0.729\) \\ SDM & **0.927** & \(0.839\) & \(0.656\) & **0.521** & **0.778** \\ SDM+MPD & \(0.815\) & **0.886** & **0.726** & \(0.540\) & \(0.721\) \\ \hline \multicolumn{4}{c}{**Spanish**} \\ \hline MPD & \(0.597\) & \(0.880\) & \(0.616\) & \(0.335\) & \(0.525\) \\ SDM (Subset) & \(0.795\) & \(0.856\) & \(0.611\) & \(0.416\) & \(0.681\) \\ SDM & **0.864** & \(0.861\) & \(0.621\) & **0.471** & **0.744** \\ SDM+MPD & \(0.681\) & **0.907** & **0.653** & \(0.413\) & \(0.618\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results of the automatic evaluation for mT0-XL on German, Russian, and Spanish trained on original data (MPD stands for MultiParaDetox), ours proposed SynthDetoxM (SDM stands for SynthDetoxM) and the consecutive training on both MultiParaDetox and Ours proposed SynthDetoxM (SDM + MPD stands for SynthDetoxM + MultiParaDetox).

 

ParaDetox into the training process (SDM + MPD) results in a drop in \(\mathbf{J}\) scores.

To further illustrate the advantages of training on SynthDetoxM, Table 3 also presents the product of **STA** and **SIM**. Even without considering **FL**, models trained on SynthDetoxM outperform those trained on MultiParaDetox in all setups and languages.

Additionally, Table 4 reports the \(\mathbf{J}\) scores of a mT@-XL model trained on MultiParaDetox, averaged across 10 subsets of SynthDetoxM and the full SynthDetoxM, compared to baselines and large language model (LLM)-based detoxification in a 10-shot generation setting.

Finally, we provide a Side-by-Side (SBS) comparison of the fine-tuned mT@-XL models to evaluate their detoxification performance. Following Moskovskiy et al. (2024), we employ GPT-4o as an evaluator to select the preferred detoxified outputs. The results of this comparison across German, Spanish, and Russian languages are summarized in Figure 4 and detailed in Appendix F.

Our SBS evaluation shows a clear preference for detoxification produced by SDM over MultiParaDetox (MPD), with SDM winning in 59% of cases compared to 19% for MPD, and 22% resulting in ties. The subset of SDM also outperforms MPD in 58% of cases.

When comparing SDM against its combination with MPD (SDM + MPD), SDM is preferred in 47% of cases, with 21% favoring SDM + MPD, and 33% tied. Additionally, the full SDM dataset is slightly preferred over the batch processing version in 55% of cases, with 35% ties. See Appendix F for more details.

## 6 Conclusions

We present several contributions to multilingual text detoxification technology. Firstly, we successfully extend the concept of few-shot prompting for detoxification to a multilingual context, building upon previous monolingual approaches and propose a framework for generation of multilingual synthetic detoxification data. Secondly, we introduce SynthDetoxM, a large-scale multilingual synthetic parallel dataset designed to address the long-standing issue of data scarcity in text detoxification research. Notably, our dataset, created using our selection criteria, demonstrates competitive quality to existing human-annotated datasets, surpassing them in both low resource and high resource settings.

Our comprehensive evaluation of SynthDetoxM reveals its effectiveness in training high-performing models for text detoxification. Specifically, our experiments show that models trained on our dataset outperform those, which were trained on a similar amount of human-annotated data. Furthermore, training a detoxification encoder-decoder model on full SynthDetoxM yields a model, which surpasses the performance of most large language models in few-shot generation setups.

Findings presented in our work, show usefulness of the generated data for the task of multilingual text detoxification and pave the way for future research and developments of related technologies.

## Acknowledgments

The contribution of E.T. was supported by the Kazan Federal University Strategic Academic Leadership Program ("PRIORITY-2030"), Strategic Project N*5. We acknowledge the computational resources of HPC facilities at the HSE University.

Figure 4: Side-by-side comparison of model outputs across all languages, evaluated by GPT-4o. The results highlight the relative performance of the models in generating detoxified text for German, Russian, and Spanish. The notation is similar to the notation from Table 3.

 



## 7 Limitations

One of the limitations of our work is that we are focusing only on explicit type of toxicity. Additionally, definition and type of toxicity changes drastically between the language, e.g. things, that are toxic in one language may be perfectly normal in other language.

Another limitation of this work is our constraint with computational resources, which led to our use of smaller and simpler models for synthetic data generation, which could fit into a single NVIDIA A100 80GB GPU. Usage of larger could potentially result in higher quality and diversity of synthetic data.

Moreover, the comparison with proprietary models would strengthen the evaluation as it is done in recent works Dementieva et al. (2025).

Additionally, we were limited by the amount of annotated non-parallel toxic datasets in some of the languages, which limited the amount of possible generated synthetic data. In future, we plan to extend our work to other languages, such as Italian, Polish and others.

## 8 Ethical Considerations

While working with the task detoxification we are fully aware of the ethical responsibilities involved. As researchers, we handle this sensitive area with care and integrity. The main goal of text detoxification is to make online interactions safer and more inclusive by reducing harmful or offensive language.

While these datasets are meant to train models to detect and reduce toxic language, there's a chance they could be used in the wrong way--such as creating models that spread harmful or offensive content. This could lead to hate speech and harassment.

It's also important to clarify that the goal of text detoxification isn't to suppress free speech or force automatic changes to content. Instead, we aim to build models that offer non-toxic alternatives, helping users choose better language on their own. By giving suggestions rather than enforcing edits, we respect people's freedom while encouraging a more positive online environment.

## References

* S. Das, M. Raghavan, K. Coutinho, J. Lee, D. P. Salakhutdinov, and L. Vedaldi (2020)A novel approach to the problem of abstractive speech recognition. External Links: Link Cited by: SS3.4.
* T. F. Chollet, N. Dehghani, B. Hayser, and W. J. Dean (2019)The impact of deep learning on machine translation. External Links: 1906.03749 Cited by: SS3.4.
* E. G. Gulcehreiss, I. Guyon, Y. Bengioudis, and Z. Liu (2015)Semantic: a large dataset from scratch. arXiv preprint arXiv:1506.02018. Cited by: SS3.4.
* A. Graves, O. Ozguner, A. Courville, and R. Fergus (Eds.) 2015. External Links: Link Cited by: SS3.4.
* A. Graves, S. Ostrovski, and R. Garnett (2016)An introduction to neural networks. External Links: 1603.03017 Cited by: SS3.4.
* A. Graves, S. Ostrovski, and R. Garnett (2016)Generative adversarial nets. External Links: 1603.03017 Cited by: SS3.4.
* A. Graves, S. Ostrovski, and R. Garnett (2017)Deep learning: an imperative course. External Links: 1706.02018 Cited by: SS3.4.
* A. Graves, S. Ostrovski, and R. Garnett (2018)DNN: a simple framework for abstractive speech recognition. External Links: 1803.03017 Cited by: SS3.4.


David Dale, Anton Voronov, Daryna Dementieva, Varvara Logacheva, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021. Text detoxification using large pre-trained neural models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 7979-7996. External Links: Link Cited by: SS4.
* J. Deng, S. Singh, D. D'souza, A. Ahmadian, A. Salamanca, M. Smith, A. Peppin, S. Hong, M. Govindassamy, T. Zhao, S. Kulbik, M. Amer, V. Aryabumi, J. Ander Campos, Y. Tan, T. Kocni, F. Strub, N. Grinsztajn, Y. Elet-Berlica, A. Locatelli, H. Lin, D. Talupuru, B. Venkitesh, D. Cairuz, B. Yang, T. Chung, W. Ko, S. Shang Shi, A. Shukayev, S. Bae, A. Piktus, R. Castagne, F. Cruz-Salinas, E. Kim, L. Crawhall-Stein, A. Morisot, S. Roy, P. Blunsom, I. Zhang, A. Gomez, N. Frost, M. Fadaee, B. Ernis, A. Ustin, and S. Hooker. 2024. Aya expanses: Combining research breakthroughs for a new multilingual frontier. Preprint arXiv:2412.04261. Cited by: SS3.5.
* D. Dementieva, N. Babakov, A. Ronen, A. Ali Ayele, N. Rizwan, F. Schneider, Xintong Wang, S. Muheir Yaman, D. Alekhseevich Moskovskiy, E. Staskovskii, E. Stavkoskii, E. Kaufman, A. Elnagar, A. Mukherjee, and A. Panchenko (2025)Multilingual and explainable text detoxification with parallel corpora. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pp. 7998-8025. External Links: Link Cited by: SS4.
* D. Dementieva, D. Moskovskiy, N. Babakov, A. Al Apele, N. Rizwan, F. Schneider, X. Wang, S. Muhie Yimam, D. Utsalov, E. Staskovskii, A. Smirnova, A. Enlagar, A. Mukherjee, and A. Panchenko (2024)Overview of the multilingual text detoxification task at PAN 2024. In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024, Vol. 3740 of CEUR Workshop Proceedings, CEUR-WS.org, pp. 2432-2461. External Links: Link Cited by: SS4.
* 4, 2023, pp. 1083-1101. External Links: Link Cited by: SS4.
* C. Nogueira dos Santos, Igor Melnyk, and I. Padhi (2018)Fighting offensive language on social media with unsupervised text style transfer. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pp. 189-194. External Links: Link Cited by: SS4.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Lateman, A. Mathur, A. Scheiten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sarvankumar, A. Korenev, A. Hinvsark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Cauchetux, C. Navak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, Ch. Wu, C. Wong, C. Cong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pinto, D. Livshits, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakornkin, E. Albadawy, E. Lobanova, E. Dhanci, E. Michalemith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. Lewis Anderson, G. Nail, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarrov, I. Darria Ibarra, I. M. Mkoumann, I. Misra, I. Evtimov, J. Copet, J. Lee, J. Geffert, J. Veness, J. Park, J. Mahadekar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, Jeremy Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rockca, J. Johnston, J. Saxe, J. Jia, K. VasudevanAlwala, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, and et al. (2024)The llama 3 herd of models. CoRRabs/2407.21783. External Links: Link Cited by: SS4.
* F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang (2022)Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 878-891. External Links: Link Cited by: SS4.
* P. Fortuna and S. Nunes (2018)A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR)51 (4), pp. 1-30. External Links: Document, ISSN 0037-695X, Link Cited by: SS4.
* Z. Fu, X. Tan, N. Peng, D. Zhao, and R. Yan (2018)Style transfer in text: exploration and evaluation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 663-670. External Links: Link Cited by: SS4.
* Skyler Hallinan, A. Liu, Y. Choi, and M. Sap (2023)Detoxifying text with marco: controllable re 

vision with experts and anti-experts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 228-242. External Links: Link Cited by: SS5.
* Z. Horvitz, A. Patel, K. Singh, C. Callison-Burch, K. R. McKeown, and Z. Yu (2024)Tiny styler: efficient few-shot text style transfer with authorship embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 13376-13390. External Links: Link Cited by: SS5.
* A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Madry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. Tashard Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, M. Moyer, A. Tam, A. Covokes, A. Tootouchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti, A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Baek, A. Jiang, A. Pelisse, A. Woodford, A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph, E. Ghorbani, B. Leimberger, B. Rossen, B. Sokolowsky, B. Wang, B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero, B. Giertler, B. Cheng, B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lougaresi, C. L. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. Jun Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hallacy, C. Koch, C. Gibson, C. Kim, C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, and D. Shepurn (2024)Gpt-4o system card. CoRRabs/2410.21276. External Links: Link Cited by: SS5.
* Md Tawkat Islam Khondaker, Muhammad AbdulMageed, and Laks V. S. Lakshmanan (2024)Detoxllm: a framework for detoxification with explanations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 19112-19139. External Links: Link Cited by: SS5.
* I. Kivlichan, J. Sorensen, J. Elliott, L. Vasserman, M. Gorner, and P. Culliton (2020)Jigsaw multilingual toxic comment classification. External Links: Link Cited by: SS5.
* V. Konovalov, M. Adler, and Ido Dagan (2016a)Effective paraphrase expansion in addressing lexical variability. In Proceedings of the Artificial Intelligence and Natural Language (AINL FRUCT 2016), St. Petersburg, Russia, pp. 87-91. External Links: ISBN 978-3-319-1919-5, Document Cited by: SS5.
* V. Konovalov, O. Melamud, R. Artstein, and I. Dagan (2016b)Collecting Better Training Data using Biased Agent Policies in Negotiation Dialogues. In Proceedings of WOCHAT, the Second Workshop on Chatbots and Conversational Agent Technologies, Los Angeles, Zerotype. External Links: Link Cited by: SS5.
* Kalpesh Krishna, J. Wieting, and Mohit Iyyer (2020)Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 737-762. External Links: Link Cited by: SS5.
* H. Lai, A. Toral, and M. Nissim (2021)Thank you bart! rewarding pre-trained models improves formality style transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Virtual Event, August 1-6, 2021, pp. 484-494. External Links: Link Cited by: SS5.
* A. Lees, V. Q. Tran, Y. Tay, J. Sorensen, J. Gupta, D. Metzler, and L. Vasserman (2022)A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, New York, NY, USA, pp. 3197-3207. External Links: Link, Document Cited by: SS5.
* S. Liu, S. Agarwal, and J. May (2024)Authorship style transfer with policy optimization. CoRRabs/2403.08043. External Links: Link, 2403.08043 Cited by: SS5.
* V. Logacheva, D. Dementieva, S. Ustyantsev, D. Moskovskiy, D. Dale, I. Krotova, N. Semenov, and A. Panchenko (2022)Paradetox: detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 6804-6818. External Links: Link, 2202.1276 Cited by: SS5.
* Mistral (2024)MiSTAL nemo l mistral ai i frontier ai in your hands.. External Links: Link Cited by: SS5.
* D. Moskovskiy, D. Dementieva, and A. Panchenko (2022)Exploring cross-lingual text detoxification with large multilingual language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 346-354. External Links: Link Cited by: SS5.

 

Daniil Moskovskiy, Sergey Pletenev, and Alexander Panchenko. 2024. Llims to replace crowdsourcing for parallel data creation? the case of text detoxification. In _Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024_, pages 14361-14373. Association for Computational Linguistics.
* Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffell. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 15991-16111. Association for Computational Linguistics.
* Mukherjee et al. (2023) Sourabrata Mukherjee, Akanksha Bansal, Pritha Majumdar, Atul Kr. Ojha, and Ondrej Dusek. 2023. Low-resource text style transfer for Bangla: Data & models. In _Proceedings of the First Workshop on Bangla Language Processing (BLP-2023)_, pages 34-47, Singapore. Association for Computational Linguistics.
* Osihodum et al. (2019) Nedima Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 4674-4683. Association for Computational Linguistics.
* Popovic (2015) Maja Popovic. 2015. chrf: character n-gram f-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT@EMNLP 2015, 17-18 September 2015, Lisbon, Portugal_, pages 392-395. The Association for Computer Linguistics.
* Budolah et al. (2023) Mohammad Mahdi Abdollah Pour, Parsa Farinneya, Manasa Bharadwaj, Nikhil Verma, Ali Pesarnaghader, and Scott Sanner. 2023. COUNT: contrastive unlikelihood text style transfer for text detoxification. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 8658-8666. Association for Computational Linguistics.
* Prabhumoye et al. (2018) Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. 2018. Style transfer through back-translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_, pages 866-876. Association for Computational Linguistics.
* Rao and Tetreeault (2018) Sudha Rao and Joel R. Tetreeault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 129-140. Association for Computational Linguistics.
* Reid and Artetxe (2023) Michel Reid and Mikkel Artetxe. 2023. On the role of parallel data in cross-lingual transfer learning. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 5999-6006. Association for Computational Linguistics.
* Reid et al. (2021) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Stotilaux, Benjamin Lee, Fabio Viola, Malcolun Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schlkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henry Michalewski, Zaheer Abbas, Nathan Schubert, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakarksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _CoRR_, abs/2403.05530.
* Reif et al. (2022) Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 837-848. Association for Computational Linguistics.
* Risch et al. (2021) Julian Risch, Anke Stoll, Lena Wilms, and Michael Wiegand. 2021. Overview of the GermEval 2021 shared task on the identification of toxic, engaging, and factclaining comments. In _Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments_, pages 1-12. Association for Computational Linguistics.
* Riviere et al. (2021) Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravim Kumar, Charline Le Lan, Sammy Jerome, Anton Titsitsulin, Nino Vieillard, Piott Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hofman, Shanstanau Thakoor, Jean-Bastiern Grill, Behnam Neyshabur, Olivier Bachem, Alamna Walton, Aliakesei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, 

Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chiutu Kumar, Chris Perry, Chris Welty, Christopher A. Chooutette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijayakumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinska, Harleen Batra, Harsh Dhand, Ivan Natlani, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjosund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, and Lilly McCasulas. 2024. Gemma 2: Improving open language models at a practical size. _CoRR_, abs/2408.00118.
* Saha et al. (2019) Koustuv Saha, Eshwar Chandrasekharan, and Munmun De Choudhury. 2019. Prevalence and psychological effects of hateful speech in online college communities. _Proceedings of the 10th ACM Conference on Web Science_.
* Semiletov (2020) Alexander Semiletov. 2020. Toxic russian comments. Dataset.
* Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 4603-4611. PMLR.
* Sun et al. (2023) Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 8990-9005. Association for Computational Linguistics.
* Suzgun et al. (2022) Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022. Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 2195-2222. Association for Computational Linguistics.
* Team (2024) Qwen Team. 2024. Qwen2.5: A party of foundation models.
* Wang et al. (2020) Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wen-Han Chao. 2020. Formality style transfer with shared latent space. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_, pages 2236-2249. International Committee on Computational Linguistics.
* Xu et al. (2023)Benfeng Xu, Quan Wang, Yajuan Lyu, Dai Dai, Yongdong Zhang, and Zhendong Mao. 2023. S2ynre: Two-stage self-training with synthetic data for low-resource relation extraction. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 8186-8207. Association for Computational Linguistics.
* Yang et al. (2024) An Yang, Baosong Yang, Bingyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. QWEN2 technical report. _arXiv preprint arXiv:2407.10671_.
* Ye et al. (2022) Jiacheng Ye, Jiahui Gao, Qintao Li, Hang Xu, Jiangtao Feng, Zhiying Wu, Tao Yu, and Lingpeng Kong. 2022. Zerogen: Efficient zero-shot learning via dataset generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 11653-11669. Association for Computational Linguistics.
* Yu et al. (2021) Ping Yu, Yang Zhao, Chunyuan Li, and Changyou Chen. 2021. Rethinking sentiment style transfer. In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_, pages 1569-1582. Association for Computational Linguistics.
* Zhang et al. (2024) Ziym Zhang, Yakang Liu, Weifang Huang, Junyu Mao, Rui Wang, and Hai Hu. 2024. MELA: multilingual evaluation of linguistic acceptability. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangokok, Thailand, August 11-16, 2024_, pages 2658-2674. Association for Computational Linguistics.
* Zheng et al. (2024) Linaminm Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. Lmsgs-chat-1m: A large-scale real-world LLM conversation dataset. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.

## Appendix A Prompts

### LLM Detoxification Prompt

In our work, for all of the language models we use the Text Style Transfer prompt taken from work by Khondaker et al. (2024). Among others, this prompt shown to be giving better generation quality and helped to avoid additional content.

We slightly adjust it specifically to text detoxification and provide the full text of the prompt in Figure 5. In this work, we do not use any advanced prompting techniques except few-shot prompting 

Your task is text style transfer. Rewrite the following text into non-toxic language. You must match the target style and preserve the original meaning as much as possible. Do not include the input text in the response. Only generate the target text and nothing else. Make sure the response is completely non-toxic and neutral. Answer only in {language}. Here are few examples:", Toxic text: {toxic_text}. Neutral text:

### mT0 Prompt

Write a non-toxic version of the following text in {language}:

{ toxic_text}

We add specific prompt to mT0 during both training and predictions. In our work do not translate the prompt into the language of generation. Instead, we follow the prompting approach from the original paper Muennighoff et al. (2023). We show the prompt in Figure 6.

## Appendix B Automatic evaluation results

The automatic evaluation results are presented in Table 6.

## Appendix C Limitations of ChrF1 as a Fluency Metric

This section addresses the issues with using ChrF1 to evaluate fluency in text detoxification. While ChrF1 is commonly used in neural machine translation Popovic (2015), it has significant drawbacks for text style transfer tasks like detoxification.

Reference-based metrics like ChrF1 are ill-suited for assessing fluency in detoxification. The goal is to change the text's style while maintaining meaning and fluency, without limiting the extent of edits. Effective detoxification often involves substantial structural changes, making comparisons with the original toxic text using ChrF1 misleading. Though ChrF1 may produce low scores, manual evaluations frequently show that the detoxified output is fluent.

ChrF1, based on character n-grams, is sensitive to word order and structural changes, which are often necessary for detoxification. It also fails to consider semantic content, meaning fluency can be high even when the ChrF1 score is low. Additionally, it tends to reward minimal edits, which undermines the goal of thorough detoxification.

\begin{table}
\begin{tabular}{l c r r r} \hline \hline
**Dataset** & **STA** & **SIM** & **CHRF** & **J** \\ \hline  & \multicolumn{4}{c}{**German**} \\ \hline MPD & \(0.722\) & \(0.848\) & \(0.602\) & \(0.383\) \\ SDM (Subset) & \(0.681~{}^{\texttt{(a)}}0.912~{}^{\texttt{(b)}}0.000\) & \(0.745~{}^{\texttt{(b)}}0.463~{}^{\texttt{(d)}}0.463~{}^{\texttt{(d)}}0.177}\) \\ SDM (Full) & \(0.728\) & \(0.899\) & \(0.734\) & \(0.484\) \\ SDM+MPD & \(0.615\) & \(0.954\) & \(0.821\) & \(0.483\) \\ \hline  & \multicolumn{4}{c}{**Russian**} \\ \hline MPD & \(0.748\) & \(0.852\) & \(0.643\) & \(0.434\) \\ SDM (Subset) & \(0.858~{}^{\texttt{(a)}}0.850~{}^{\texttt{(a)}}0.850~{}^{\texttt{(a)}}0.656~{}^{ \texttt{(a)}}0.478~{}^{\texttt{(d)}}0.569~{}^{\texttt{(a)}}0.478~{}^{ \texttt{(d)}}0.569~{}^{\texttt{(a)}}0.478~{}^{\texttt{(d)}}0.569~{}^{\texttt{( d)}}0.521}\) \\ SDM (Full) & \(0.927\) & \(0.839\) & \(0.656\) & \(0.521\) \\ SDM+MPD & \(0.815\) & \(0.886\) & \(0.726\) & \(0.540\) \\ \hline  & \multicolumn{4}{c}{**Spanish**} \\ \hline MPD & \(0.597\) & \(0.880\) & \(0.616\) & \(0.335\) \\ SDM (Subset) & \(0.795~{}^{\texttt{(a)}}0.856~{}^{\texttt{(a)}}0.611~{}^{\texttt{(a)}}0.416~{}^{ \texttt{(a)}}0.416~{}^{\texttt{(a)}}0.580~{}^{\texttt{(d)}}0.500~{}^{\texttt{( d)}}0.569~{}^{\texttt{(d)}}0.471}\) \\ SDM (Full) & \(0.864\) & \(0.861\) & \(0.621\) & \(0.471\) \\ SDM+MPD & \(0.681\) & \(0.907\) & \(0.653\) & \(0.413\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results of the automatic evaluation for mT0-XL on German, Russian, and Spanish trained on original data (MPD stands for MultiParaDetox), our collected and synthetically generated data (SDM stands for SynthDetoxM) and on their combination (MultiParaDetox + SynthDetoxM).

Figure 5: Detoxification prompt we use for synthetic parallel data generation. {toxic_text} stands for a placeholder for a given toxic text being prompted into LLM. In few-shot setting we add few examples of detoxification before last two lines and write: _Here are few examples:_.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline Model & German & Spanish & French & Russian \\ \hline Llama 3.1 8B & 662 & 619 & 773 & 1648 \\ Llama 3.1 7OB & 898 & 981 & 114 & 1354 \\ Mistral Nemo & 622 & 583 & 392 & 1320 \\ Mistral Small & 862 & 985 & 565 & 2237 \\ Owen 2.5 32B & 477 & 819 & 513 & 3128 \\ Aya Exp. 32B & 458 & 453 & 142 & 945 \\ Aya Exp. 8B & 316 & 330 & 143 & 765 \\ Command-R 32B & 273 & 492 & 308 & 2294 \\ Gemma 2 27B & 394 & 564 & 360 & 2019 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Number of accepted samples in the final SynthDetoxM dataset, broken down by language and LLMs.

 

Footnote 10: hf.co/s-nlp/xlmr-base-refusal-classifier

Recent research has shifted towards more appropriate fluency metrics. For example, CoLA-based classifiers, as used in Dementieva et al. (2023) and Logacheva et al. (2022), focus on linguistic acceptability, offering a more accurate assessment of fluency without relying on comparisons to the toxic input.

While ChrF1 has its merits in other tasks, it is not suitable for evaluating fluency in detoxification. Future work should prioritize methods that assess fluency based on grammaticality and naturalness, independent of the original text.

## Appendix D Refusal classifier training

To get rid of LLM refusals in SynthDetoxM, we trained a separate refusal classifier, based on xlmr-base11.

Footnote 11: hf.co/datasets/chameleon-lizard/multilingual_refusals

To train the model, a high quality synthetic dataset was created12. It was based on randomly selected inputs from the LMSYS-Chat-1M (Zheng et al., 2024) dataset and then passed to the Gemini Flash 1.5 (Reid et al., 2024) and Llama 3.370B (Dubey et al., 2024) models, prompted to generate both responses to the prompts from the dataset and refusals. Prompt for synthetic refusal generation is presented in Figure 7.

Footnote 12: [http://www.cs.uci.edu/~dubey/](http://www.cs.uci.edu/~dubey/)

Classification model was trained using a batch size of 64, learning rate of 1e-4 for one epoch.

Figure 8: Side-by-side comparison of model outputs across all languages, evaluated by GPT-4o. The results highlight the relative performance of the models in generating detoxified text for German. The notation is similar to the notation from Table 3.

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & Spanish\(\downarrow\) & German\(\downarrow\) & Russian\(\downarrow\) \\ \hline Toxic & 2089 & 323 & 4467 \\ Detoxifed & 27 & 102 & 14 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Total amount of toxic words for toxic and detoxification subsets of SynthDetoxM with respect to language.

Figure 10: Side-by-side comparison of model outputs across all languages, evaluated by GPT-4o. The results highlight the relative performance of the models in generating detoxified text for Spanish. The notation is similar to the notation from Table 3.

Figure 7: Refusal generation prompt for synthetic refusals dataset.

Figure 9: Side-by-side comparison of model outputs across all languages, evaluated by GPT-4o. The results highlight the relative performance of the models in generating detoxified text for Spanish. The notation is similar to the notation from Table 3.

 

Additional linguistic analysis of the dataset

To provide additional perspective about detoxification quality of our dataset, we used dataset12, which contains toxic lexicon in German, Spanish and Russian languages and calculated two metrics of our generated data: the amount and mean counts of toxic words in each sentence (presented in Tables 7 and 8 accordingly) in the toxic and generated detoxified subsets of SynthDetoxM.

Footnote 12: hf.co/datasets/textdetox/multilingual_toxic_lexicon

Due to the lack of French toxic lexicon dataset we did not do any evaluations in French. Furthermore, selected dataset is not comprehensive: for instance, German subset contains only 247 toxic words, which leaves some toxic sentences not having any toxic words detected and overall toxicity of German subset of our dataset is comparatively low. However, these evaluations still show that our detoxified data contains substantially less toxic lexicon than the original toxic data, indicating that overall explicit toxicity of the generated data is much lower after the detoxification.

## Appendix F Side-by-side comparison of trained mT0 models

To provide additional validation outside of methodology created by Dementieva et al. (2024), we have evaluated the responses from the test dataset of each of the trained mT0 models using GPT4o Hurst et al. (2024) as a judge. To lessen the positional bias, each side-by-side comparison was done twice, changing the positions of the answers and calculating mean score for all answers. The results are presented in the Figures 8, 9, 10.

As demonstrated in the figures, models fine-tuned on SynthDetoxM achieve significantly higher win rates compared to those trained on the "gold-standard" MultiParaDetox dataset, as well as models trained using a two-stage regime. We attribute this superior performance to the challenges posed by human-sourced samples, which may be too complex or nuanced for relatively simple and compact student models to learn effectively. This limitation likely contributes to the weaker performance of models trained on MultiParaDetox in both automatic evaluations and side-by-side comparisons.

 



## Appendix G Per-Language Few-Shot Examples

To ensure reproducibility we show all the few-shot prompts _as is_ in Tables 91, 92, and 93.

\begin{table}
\begin{tabular}{l|c|r|r} \hline \hline Type & Test & Perspective API Score & XLM-R Score \\ \hline Toxic & margin principley output graa, a proofair neg ccp crpary & 0.820 & 1,000 \\  & _The board source an each once, but record over the whole country._ & & \\ Neutral & Priority yalu qphi graa, a paxpythia leg ccptcy & 0.360 & 0.001 \\  & He took the south once, but destroyed the whole country & & \\ \hline Toxic & makeria photo

* [25] M. C. Bender, J. R. Friedman, and A. G. Zweig. _The role of the word-word word in word-word words: a language approach_. In _Proceedings of the 19th International Conference on Language Processing_, pages 347-368. ACM, 2002.
* [26] D. K. S. Manning, P. N. McAllester, and L. E. Mitchell. _Word-wordword word 



\begin{table}
\begin{tabular}{l|p{142.3pt}|c|c|c} \hline Type & Text & Perspective API Score & XLM-R Score \\ \hline Toxic & Y to bien que les va siendo fachas, que no es gratis & 0.184 & 0.002 \\  & _And how well they are doing being fascists, which is not free_ & & \\ Neutral & Y lo bien que les va siendo conservadores, que no es gratis & 0.012 & 0.574 \\  & _And how well they are doing being conservatives, which is not free_ & & \\ \hline Toxic & Y a mi qué me importa l oue ti notes”, facha? & 0.427 & 0.391 \\  & _And what do I care what you ”notice", fascist?_ & & \\ Neutral & Y a mi qué me importa lo que ti notes“? & 0.080 & 0.001 \\  & _And what do I care what you ”notice"?_ & & \\ \hline Toxic & ya ve sádemás à los policra snazis que tendría sea en el whatsapp... como para creverse algo. _you see also the naçj policemen that she will have on whatasup... as if to believe something,_ & 0.261 & 0.608 \\ Neutral & ya ves además a los politicas que tendría esa en el whatsapp.... como para crerese algo. _you see also the policemen that she will have on whatasup... as if to believe something._ & 0.035 & 0.003 \\ \hline Toxic & \#YoSyAnonymousUE basata de absurda represión victimista por parte del Estado Fasciista _\#YosSoyAnonymousUE enough of abrad victimista repression by the FASciar State_ & 0.255 & 0.352 \\  & \#YoSyAnonymousUE basata de absurada represión victimista por parte del Estado _\#YoSyn



\begin{table}
\begin{tabular}{l|p{28.5pt}|c|r|r} \hline Type & Text & Perspective API Score & XLM-R Score \\ \hline Toxic & Pour le crétu qui remnet à chaque fois une soit-disant citation de Nietzsche, l’expression “tout s’enjuive” est attribuee par Nietzsche a un libre-pensuer demorcate, autrement dit Nietzsche & 0.255 & - \\  & identifie l’antisémsitème aux valeuers chrétiennes de la Tibre penseée. & & & \\  & _For the idiot who always puts a so-called quote from Nietzsche, the expression “everything is becoming Jewish" is attributed by Nietzsche to a democratic freethinker, in other words_ & & \\  & _Nietzsche identifies anti-Sentisentin with the Christian values of ?ree thought”_. & & & \\ Neutral & Pour la personne qui remnet à chacun une s citation de Nietzsche l’expression tout s’ennjuive estiftube par Nietzsche a un libre-pensur demorcate, autrement dit Nietzsche identitie & 0.000 & - \\  & !antisemittismste aux valeuers chrétiennes de la libre bense. & & & \\  & _For the person who gives everyone a quote from Nietzsche, the expression “everything is becoming Jewish” is attributed by Nietzsche to a democratic fireethinker, in other words_ & & \\  & _Nietzsche identifies anti-Sentisentin with the Christian values of free thought._ & & & \\ \hline Toxic & Et tou aussi tu commences serfieusement à me gonfler. & 0.238 & - & \\  & _And you too are seriously starting to amany me._ & & & \\ Neutral & Et tou assui tu commences serfieusesement à m’emerver. & 0.285 & - \\  & _And you too are seriously starting to irritate me._ & & & \\ \hline Toxic & Tout est trop nul, tout est faux ou ou ou! & 0.167 & - \\  & _Everything is too rubbish, everything is false or or or!_ & & & \\ Neutral & Tout est mediocre et faux ! & & & \\  & _Everything is mediocre and false!_ & & & \\ \hline Toxic & Continuez tout seul si ca vous anmuse. Je ne perdrai pass plus de temps avec un punible de votte espice. & 0.498 & - \\  & _Contine on your own if that annoes you. I won ’t waste any more time with a pain in the asx like you._ & & & \\ Neutral & Continuez tout seul si ca vous amuse. Je ne perdrai pass plus de temps avec une telle personnel. & & 0.309 & - \\  & _Contine on your own if that annues you. I won't waste any more time with such a person._ & & & \\ \hline Toxic & Salut, j ai restauret cet article et réeverte tes su & 0.383 & - \\  & _Hi, I restored this article and reverted your su_ & & & \\ Neutral & Salut, ja i restauret cet article et réeverte tes su & & & \\  & _Hi, I restored this article and received your su_ & & & \\ \hline Toxic & Encore un vieux qui fait cheir, je ne fais plus attention à force. & & 0.509 & - \\  & _Another old man who’s a pain in the asx, I don’t pay attention anymore._ & & & \\ Neutral & Encore un vieux ennuyeux, je ne fais plus attention à force. & & & \\  & _Another boring old man, I don’t pay attention anymore._ & & & \\ \hline Toxic & One ot en o s ten tape de ta facqn a toui et son e tang que t as eu d emmerdes avec persons e est pass mon problemene donome comme je al al stoi au l heure je fais a ma facon. & & 0.373 & - \\  & _Teath and we don’t care about your way und wort à? care about you's not? care about you's not? care that you had troubles with anyone it’s nor my problem so sa said earlier Id do it my way_. & & & \\ Neutral & Que ot e a’ra nien uoir avec u via ve, le fait que tu aies eu des problemes, ca ne me regarde paas, dono comme pe ja al its dot au l heure je fais a ma facon. & & & \\  & _Feath and it has nothing to do us your life, the fact that you had problems, it’s none of my business, so as Isal capitalier Id do it my way._ & & & \\ \hline Toxic & Ta reponse est confondante ... Ef tu continues tees mangiances ! Je te prie die t arreiter immediatement! & & 0.304 & - \\  & _Your answer it confusing... And you continue your scheme? I beg you to stop immediately!_ & & & \\ Neutral & Ta reponse est confondante ... El tu continues ties tricks! Je te pie die t arrêter immediatement ! & & & \\  & _Your answer it confusing... And you continue your tricks! I beg you to stop immediately!_ & & & \\ \hline Toxic & Faut croire que les amateurs de musique sont des maladres mentaux & & 0.386 & - \\  & _You have to believe that music lovers are mentally ill_ & & & \\ Neutral & Faut croire que les amateurs de musique ont des preferences éranges & & 0.046 & - \\  & _You have to believe that music lovers hav strange preferences_ & & & \\ \hline \end{tabular}
\end{table}
Table 12: Selected few-shot examples for the French language with toxicity scores.