Venue: NAACL 2024

# MultiParaDetox: Extending Text Detoxification

with Parallel Data to New Languages

Daryna Dementieva\({}^{1}\), Nikolay Babakov\({}^{2}\), and Alexander Panchenko\({}^{3,4}\)

\({}^{1}\)Technical University of Munich

\({}^{2}\)Centro Singular de Investigacion en Tecnoloxias Intelixentes (CiTIUS), Universidade de Santiago de Compostela

\({}^{3}\)Skolkovo Institute of Science and Technology, \({}^{4}\)Artificial Intelligence Research Institute daryna.dementieva@tum.de, nikolay.babakov@usc.es, a.panchenko@skol.tech

###### Abstract

Text detoxification is a textual style transfer (TST) task where a text is paraphrased from a toxic surface form, e.g. featuring rule words, to the neutral register. Recently, text detoxification methods found their applications in various task such as detoxification of Large Language Models (LLMs) Leong et al. (2023); He et al. (2024); Tang et al. (2023) and toxic speech combating in social networks Deng et al. (2023); Mun et al. (2023); Agarwal et al. (2023). All these applications are extremely important to ensure safe communication in modern digital worlds. However, the previous approaches for parallel text detoxification corpora collection--ParaDetox Logacheva et al. (2022) and APPADIA Atwell et al. (2022)--were explored only in monolingual setup. In this work, we aim to extend ParaDetox pipeline to multiple languages presenting **MultiParaDetox** to automate parallel detoxification corpus collection for potentially any language. Then, we experiment with different text detoxification models--from unsupervised baselines to LLMs and fine-tuned models on the presented parallel corpora--showing the great benefit of parallel corpus presence to obtain state-of-the-art text detoxification models for any language.

_Warning: This paper contains rude texts that only serve as illustrative examples._


* We present MultiParaDetox: a pipeline for extension of text detoxification corpus collection procedure to new languages;
* We showcase the pipeline collecting new parallel datasets for three new languages--Spanish (from Romance branch of Indo-European language family), Russian, and Ukrainian (from East Slavic branch);
* We present the first of its kind evaluation study of unsupervised baselines, LLMs, and fine-tuned supervised models for these three languages for the text detoxification task affirming the advantages of parallel corpora.

All the introduced data and models are available for public usage online.2

Footnote 2: [https://huggingface.co/textdetox](https://huggingface.co/textdetox)

## 2 Related Work

Text Style Transfer with Parallel DataWhile the tasks of text style transfer were explored for diverse domains (sentiment, authors styles, formality, toxicity), these problems are addressed in the majority of cases only with non-parallel text classification corpora. To this date, only a few parallel corpora for text style transfer were presented: (i) Bible corpus Carlson et al. (2018) which was obtained historically due to many reissues of the text; (ii) GYAFC Rao and Tetreault (2018) which was collected via crowdsourcing but verified manually by the authors of the work; (iii) APPADIA Atwell et al. (2022) which was annotated by expert sociolinguists; (iv) ParaDetox Logacheva et al. (2022) which was fully collected and verified by crowd-sourcing.

Text DetoxificationAs ParaDetox and APAPDAI datasets have appeared recently, the vast attention in the text detoxification field has been paid to unsupervised methods. In Nogueira dos Santos et al. (2018), the basic encoder-decoder was extended with a collaborative classifier and a set of specialized loss functions for detoxification. Then, the power of Masked Language Modelling (MLM) were utilized in CondBERT and ParaGedi models Dale et al. (2021). These unsupervised baselines were improved with the mixture of experts and antiexperts concept in MaRCo Hallinan et al. (2023). However, the seq2seq models from ParaDetox and APPADIA works showed so far more promising text detoxification results than those based on non-parallel corpora, such as those mentioned above.

Text Style Transfer in Multilingual and Crosslingual SetupsAlso, several works are already dedicated to the extension of text style transfer methods to new languages. For sentiment, in Mukherjee et al. (2023), English dataset was extended to Bangla with manual annotation. X-FORMAL dataset Briakou et al. (2021) was introduced as the extension of GYAFC for three new languages and was obtained via automated translation. For text detoxification task, the cross-lingual setup was explored in Dementieva et al. (2023) attempting to transfer knowledge from English to a low-resource language. While several approaches showed compatible results, they are still inferior in quality to methods fine-tuned on parallel data.

Figure 1: **MultiParaDetox** pipeline for parallel corpus collection using crowdsourcing: **Step 1**: _Toxic Corpus Collection_ - texts can obtained either from available for the target language binary classification (non-parallel) corpus or by keywords search in some general corpus; **Step 2: _Task Language Adaptation_ to the target language with translation system and a cross-check by native speakers; **Step 3: Tasks Settings Adjustment_** by configuring annotators language requirements and quality control tasks.

 

MultiParaDetox Pipeline

We adapt ParaDetox Logacheva et al. (2022) collection pipeline as it was designed to automate the data collection as well as verification with crowdsourcing. The pipeline consists of three tasks:

**Task 1: Rewrite text in a polite way**: Annotators need to provide the detoxified paraphrase of the text so it becomes non-toxic and the main content is saved or to skip paraphrasing if the text is not possible to rewrite in non-toxic way;
**Task 2: Do these sentences mean the same?**: Check if the content is indeed the same between the original toxic text and its potential non-toxic paraphrase;
**Task 3: Is this text offensive?**: Verification of the provided paraphrase if it is indeed non-toxic.

We extend this pipeline to **MultiParaDetox** supporting any new language (see Figure 1):

Step 1: Toxic Corpus PreparationFirstly, we need to prepare toxic samples that will serve as input to the ParaDetox pipeline. In the annotation, we focus only on _explicit_ toxicity types van Aken et al. (2018). (i) If there already exists binary toxicity classification dataset, then it is enough to select from it toxic or hate part preferably with labels like "_toxic"_, "_offensive"_, "_obscene"_; (ii) If there is not such dataset, then samples with explicit toxicity can be selected by finding toxic keywords substrings in the texts. As we want sentences to have meaningful content, only sentences with less then \(1/2\) of toxic keywords fraction should be chosen.

Step 2: Tasks Language AdaptationThen, the ParaDetox tasks needed to be adapted for the target language. This can be done with combination of automated translation followed by language native speakers texts proofreading.

Step 3: Tasks Settings AdjustmentFinally, for the crowdsourcing tasks, the language and country settings should be chosen accordingly to the target language. For the quality control, we follow the procedure described in ParaDetox utilising training, exam, and control tasks. We claim that these tasks can be also translated from the original ones with slight edits by native speakers according to special cases of toxicity for the language.

## 4 Collection of New Parallel Datasets with MultiParaDetox

We applied the described above pipeline to obtain new parallel datasets for three languages--Russian, Ukrainian, and Spanish. The choice of language was done based on the availability of native speakers of these languages. The data collection was done via Toloka platform.3 For translations, we used DeepL4 API (tasks texts are presented in Appendix A). We accepted to the annotation only workers who proved the corresponding language fluency with a test. The general information about the datasets is presented in Table 2 with example samples in Appendix B.

Footnote 3: [https://toloka.ai](https://toloka.ai)

As Russian toxicity classification datasets were available for Russian language, we selected toxic sentences from Russian Language Toxic Comments competitions Belchikov (2019); Semiletov (2020). In case of the Ukrainian language, there was no binary toxicity classification corpus available. We filtered from Ukrainian Tweets Corpus Bobrovnyk (2019) the explicitly toxic samples that contain obscene lexicon from the predefined list Bobrovnyk (2019). For Spanish language, we selected samples for annotation from two datasets: hate speech detection one Pereira-Kohatsu et al. (2019) as well as filtered by keywords Spanish Tweets corpus Perez et al. (2022).

We collected the data for each language depending on the available input data and resources. The nature of the original data influenced the process. Thus, the lowest ratio of non-detoxifed sample filtering was observed for Ukrainian language. For Spanish, this ratio is higher as the input data labels were more from the hate speech domain. Nevertheless, for each language it was possible to collect at least several hundreds of pairs with 1-3 paraphrases per each toxic input.

Data Quality VerificationTo verify the quality of all collected data, we randomly selected \(100\) pairs per language and asked 3 annotators--nativespeakers for each language with the expertise in the topic--to label if the pair meets the requirements of the task or not. For all languages, the amount of inappropriate pairs was \(<10\%\). The inner-annotator agreement was estimated with Krippendorff's \(\alpha\): for Russian it equals to 0.85, for Ukrainian it equals to 0.90, and for Spanish it equals to 0.67.

 



## 5 Text Detoxification Experiments

To enhance the validation of the data collected using MultiParaDetox, we conduct text detoxification experiments, comparing baselines with fine-tuned models with the newly obtained data.

### Text Detoxification Systems

DuplicateSimple copy-paste of the toxic input to the output without any change. This baseline has the highest SIM score by definition.

DeleteElimination of obscene substrings from a manually constructed dictionary of rude words. Existing lexicons are used for Russian Dementieva et al. (2021), Ukrainian Bobrovnyk (2019), and Spanish Wormer (2022).

condBERTWe adapted one of the MLM-based unsupervised methods from Dale et al. (2021). We used mBERT Devlin et al. (2019) as a base model. The model runs MLM to generate list of substitutes selecting non-toxic ones.

LLM PromptingFirstly, we experimented with several multilingual models--MT0-large Muennighoff et al. (2023), BloomZ-7b Muennighoff et al. (2023), and LLaMa-7b Touvron et al. (2023)--to select the most promising one for the task (see the results in Appendix D). In the end, we proceed with LLaMa in zero-shot setup with the corresponding for each language prompt:

Rewrite the following text in a more polite but natural form, maintaining its original meaning (no comments, just rewritten text) {text}.

Fine-tuned LM on Translated DataWe also tried to obtain synthetic parallel corpa by translating English ParaDetox Logacheva et al. (2022) to our target languages. We utilized mBART model Liu et al. (2020)6 for the translation step.

Footnote 6: [https://huggingface.co/facebook/mbart-large-50](https://huggingface.co/facebook/mbart-large-50)

Fine-tuned LM on ParaDetoxinFinally, we finetuned text generation models on the presented data. We fine-tuned mBART Liu et al. (2020)\({}^{6}\) in both monolingual and multilingual setups.

### Evaluation Setups

We follow the automated evaluation setup used in Logacheva et al. (2022) adapting it to our target languages. In this setup, three following components are measured:

Style Transfer Accuracy (STA)Toxicity classification result from the classifiers: for Russian Dementieva et al. (2021), Ukrainian (we trained our own classifier based on the additionally collected data with Task 3), Spanish Aluru et al. (2020).

Content Similarity (SIM)Cosine similarity between LaBSE embeddings Feng et al. (2022) of a toxic input and a model's output.

Fluency (FL)Perplexity score of the output from mGPT model Shliazkho et al. (2024) compared to the score of the input-the acceptable output should be no less fluent as input.

The three components are subsequently combined into the final **Joint (J)** metric used for the final ranking of approaches. Given an input toxic text \(x_{i}\) and its output detoxified version \(y_{i}\), for a test set of \(n\) samples:

\[\mathbf{J}=\tfrac{1}{n}\sum\limits_{i=1}^{n}\textbf{STA}(y_{i})\cdot\textbf{ SIM}(x_{i},y_{i})\cdot\textbf{FL}(y_{i})\textrm{,}\]

where **STA**(\(y_{i}\)), **SIM**(\(x_{i},y_{i}\)), **FL**(\(y_{i}\)) \(\in\{0,1\}\) for each text detoxification output sample \(y_{i}\).

## 6 Results

The results of the systems evaluation are presented in Table 3. Additionally, we provide the examples of models outputs in Appendix C.

Delete methods reaches the highest content similarity as it was designed to modify the original sentence slightly. However, it does not filter all

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c|c} \hline \multirow{2}{*}{Target Language} & \multirow{2}{*}{Input Samples} & \multicolumn{2}{c|}{**Fliltered**} & \multicolumn{1}{c|}{**Unique**} & \multicolumn{1}{c|}{**Length in**} & \multicolumn{1}{c}{**Cost per**} \\  & & Non-detox. & Inputs & Paraphrases & Paraphrases & **Tokens of Tokens of Learntal** & **Total** & **Unique** \\  & & Samples & Paraphrases & per Input & Total & Toxic/neutral & Costs & Sample \\ \hline Russian & 30 000 & 65\% & 8 500 & 1.83 & 11 200 & 10.1 / 9.7 & S880 & 50.11 \\ \hline Ukrainian & 2 700 & 20\% & 2 122 & 2.19 & 4 661 & 12.5 / 10.8 & S849 & 50.18 \\ \hline Spanish & 720 & 54\% & 337 & 1.67 & 565 & 11.7 / 9.6 & S278 & 50.40 \\ \hline \end{tabular}
\end{table}
Table 2: Statistics of new ParaDetox data: the crowdsourcing steps and final datasets.



toxic language and gains the lowest STA scores. The condBERT method fails to make substitutions with correct words and obtains not good enough fluency scores. LLaMa achieves very high STA scores concurrently with the lowest SIM scores. The model can hallucinate and even generate text not in a target language as can be observed from the examples.

The models fine-tuned on the translated datasets fail for each language in STA scores. The rationale lies in the diversity of toxic phrases across languages. For instance, Russian and Ukrainian, being morphologically rich languages, encompass a multitude of toxic expressions that cannot be directly translated from English. Moreover, there exists a strong correlation between language and culture, manifesting in specific discussion topics and expressions unique to each language's online informational space.

Finally, the models fine-tuned on the proposed data never fail in any of the evaluation parameters and outperform unsupervised baselines based on J score with a high gap. This attests to the reliability of our data and necessity of parallel text detoxification corpora in acquiring state-of-the-art text detoxification models. For Spanish, a slight drop of the results can be cased by significantly lower amount of the training data. Even in this case, the model shows promising results while other models still did not produce qualitative results (LLaMa got high STA scores but the content output text was just random).

In the end, we also presented the results for multilingual text detoxification model fine-tuned for all three languages. The obtained results on par with monolingual models confirm the possibility to obtain single multilingual model for the multilingual text detoxification task.

## 7 Conclusion

We presented **MultiParaDetox**--the extension of ParaDetox pipeline for parallel data collection for the text detoxification task to new languages. The target language corpus collection can be prepared only with three steps: provision of input toxic corpus, crowdsourcing tasks language adaptation, and corresponding settings adjustments. We tested our proposed pipeline extension on three new languages--Russian, Ukrainian, and Spanish--collecting corresponding new corpora.

The quality of the data was verified manually by native speakers. Finally, the data efficacy was confirmed with text detoxification systems comparison where the models fine-tuned on our data outperformed unsupervised baselines and zero-shot-prompted LLMs.

### Limitations

Firstly, we would like to emphasize that in our text detoxification task definition and data we purposely include only _explicit_ types of toxicity. More specifically, one may consider the task studied in this paper as paraphrase from the rude to neutral style. The task of addressing _implicit_ toxicity is more challanging Wiegand et al. (2023) and may require different other forms of its post-processing Mun et al. (2023). While a rude text can be paraphrased to a neural form if its message is inherently nontoxic, implicitly toxic text carrying inherently toxic message hardly can be paraphrased without the change of this original toxic meaning. To collect parallel datasets for new toxicity types, i.e. sarcasm, racism, more sophisticated definition of the text detoxification task should be designed.

Additionally, the datasets resulting from our data collection experiments exhibited an uneven distri

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline  & **STA** & **SIM** & **FL** & **J** \\ \hline \multicolumn{5}{c}{**Russian**} \\ Human references & 0.858 & 0.720 & 0.783 & 0.484 \\ Duplicate & 0.244 & 1.000 & 1.000 & 0.247 \\ \hline Delete & 0.568 & 0.891 & **0.856** & 0.410 \\ condBERT & 0.585 & 0.872 & 0.685 & 0.349 \\ LLaMa & **0.896** & 0.285 & 0.763 & 0.195 \\ mBART-Translated & 0.452 & **0.893** & 0.826 & 0.333 \\ \hline \hline mBART-RuParDetox & 0.772 & 0.750 & 0.781 & **0.492** \\ \hline \hline \multicolumn{5}{c}{**Ukrainian**} \\ Human references & 0.872 & 0.897 & 0.669 & 0.523 \\ Duplicate & 0.053 & 1.000 & 1.000 & 0.053 \\ \hline Delete & 0.872 & **0.944** & 0.163 & 0.134 \\ condBERT & 0.747 & 0.869 & 0.147 & 0.095 \\ LLaMa & **0.900** & 0.349 & 0.669 & 0.210 \\ mBART-Translated & 0.506 & 0.900 & 0.678 & 0.309 \\ \hline \hline mBART-UKparDetox & 0.759 & 0.929 & **0.725** & **0.511** \\ \hline \hline \multicolumn{5}{c}{**Spanish**} \\ Human references & 0.653 & 0.843 & 0.407 & 0.224 \\ Duplicate & 0.195 & 1.000 & 1.000 & 0.195 \\ \hline Delete & 0.415 & **0.955** & 0.305 & 0.121 \\ condBERT & 0.525 & 0.884 & 0.161 & 0.075 \\ LLaMa & **0.949** & 0.284 & **1.000** & **0.269** \\ mBART-Translated & 0.407 & 0.861 & 0.619 & 0.217 \\ \hline \hline mBART-EsParDetox & 0.576 & **0.858** & 0.483 & 0.239 \\ \hline \hline \multicolumn{5}{c}{**All Languages: English, Russian, Ukrainian, Spanish**} \\ \hline mBART-MParaDetox & 0.675 & 0.958 & 0.690 & 0.456 \\ \hline \end{tabular}
\end{table}
Table 3: Text detoxification results. Within methods comparison, **bold** numbers denote the best results in a column, gray – the lowest.

 

bution of sample ratios. That happened due to natural sequential progress of experiments and available resources for each step. We openly share the tasks instruction for each language so the research community can as well contribute to the data collection. Also, the further research direction might be to explore the minimal necessary amount of parallel data to fine-tune a solid text detoxification model.

While we presented the experiment to obtain one multilingual text detoxification model, the task of cross-lingual knowledge transfer between languages still has a room for improvement. Before, there was already preliminary experiments for cross-lingual text detoxification transfer (Dementieva et al., 2023). However, there is still a possibility to extension to more languages. Another side of this questions is to explore if the transfer between languages from neighbouring language families can help to improve the performance.

## Ethical Considerations

We explore the task of text detoxification only for the positive impact side of the textual communication. Thus, such systems can be potentially used in automated dialogue systems Deng et al. (2023), preprocess training data Tang et al. (2023), and more niche toxicity tackling as, for instance, misopyn Sheppard et al. (2023). The reverse process, toxificiation of the texts, can be done simply by adding some obscene lexicon to the texts and then easily can be addressed with our models.

During crowdsourcing process, we established the most fair to our understanding payment to annotators: Task \(1-0.15$\lx@paragraph$\) per page, Task \(2-0.12$\lx@paragraph$\) per page, Task \(3-0.10$\lx@paragraph$\) per page. The data were collected in several dozens of iterations and each iteration was of several hundreds of pages which resulted to the enough amount of tasks to be completed by annotators.

## References

* Agarwal et al. (2023) Vibhor Agarwal, Yu Chen, and Nishanth Sastry. 2023. Haterephrase: Zero- and few-shot reduction of hate intensity in online posts using large language models. _CoRR_, abs/2310.13985.
* Akiki et al. (2022) Christopher Akiki, Giada Pistilli, Margot Mieskes, Matthias Galle, Thomas Wolf, Suzana Ilic, and Yacine Jernite. 2022. Bigscience: A case study in the social construction of a multilingual large language model. _CoRR_, abs/2212.04960.
* Aluru et al. (2020) Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. _CoRR_, abs/2004.06465.
* Atwell et al. (2022) Katherine Atwell, Sabit Hassan, and Malihe Alikhani. 2022. APPDIA: A discourse-aware transformer-based style transfer model for offensive social media conversations. In _Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Cyeongju, Republic of Korea, October 12-17, 2022_, pages 6063-6074. International Committee on Computational Linguistics.
* Belchikov (2019) Anatoly Belchikov. 2019. Russian language toxic comments. [https://www.kaggle.com/blackmoon/russian-language-toxic-comments](https://www.kaggle.com/blackmoon/russian-language-toxic-comments). Accessed: 2023-12-14.
* Bobrovnyk (2019a) Kateryna Bobrovnyk. 2019a. Automated building and analysis of ukrainian twitter corpus for toxic text detection. In _COLINS 2019. Volume II: Workshop_.
* Bobrovnyk (2019b) Kateryna Bobrovnyk. 2019b. Ukrainian obscene lexicon. [https://github.com/saganoren/obscene-ukr](https://github.com/saganoren/obscene-ukr). Accessed: 2023-12-14.
* Briakou et al. (2021) Eleftheria Briakou, Di Lu, Ke Zhang, and Joel Tetreeault. 2021. Ola, bonjour, salve! XFORMAL: A benchmark for multilingual formality style transfer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3199-3216, Online. Association for Computational Linguistics.
* Carlson et al. (2018) Keith Carlson, Allen Riddell, and Daniel Rockmore. 2018. Evaluating prose style transfer with the bible. _Royal Society open science_, 5(10):171920.
* Dale et al. (2021) David Dale, Anton Voronov, Daryna Dementieva, Varvara Logacheva, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021. Text detoxification using large pre-trained neural models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Putta Cana, Dominican Republic, 7-11 November, 2021_, pages 7979-7996. Association for Computational Linguistics.
* Dementieva et al. (2023) Daryna Dementieva, Daniil Moskovskiy, David Dale, and Alexander Panchenko. 2023. Exploring methods for cross-lingual text style transfer: The case of text detoxification. In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1083-1101, Nusa Dua, Bali. Association for Computational Linguistics.
* Dementieva et al. (2021) Daryna Dementieva, Daniil Moskovskiy, Varvara Logacheva, David Dale, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021. Methods for detoxification of texts for the russian language. _Multimodal Technol. Interact._, 5(9):54 

Jiawen Deng, Hao Sun, Zhexin Zhang, Jiale Cheng, and Minlie Huang. 2023. Recent advances towards safe, responsible, and moral dialogue systems: A survey. _CoRR_, abs/2302.09270.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 878-891. Association for Computational Linguistics.
* Floto et al. (2023) Griffin Floto, Mohammad Mahdi Abdollah Pour, Parsa Farinneya, Zhenwei Tang, Ali Pesaranghader, Manasa Bharadwaj, and Scott Sanner. 2023. DiffuDetox: A mixed diffusion model for text detoxification. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 7566-7574, Toronto, Canada. Association for Computational Linguistics.
* Hallian et al. (2023) Skyler Hallian, Alisa Liu, Yejin Choi, and Maarten Sap. 2023. Detoxifing text with marco: Controllable revision with experts and anti-experts. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 228-242. Association for Computational Linguistics.
* He et al. (2024) X. He, S. Zannettou, Y. Shen, and Y. Zhang. 2024. You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content. In _2024 IEEE Symposium on Security and Privacy (SP)_, pages 60-60, Los Alamitos, CA, USA. IEEE Computer Society.
* Leong et al. (2023) Chak Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. 2023. Self-detoxifying language models via toxification reversal. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 4433-4449, Singapore. Association for Computational Linguistics.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. _Trans. Assoc. Comput. Linguistics_, 8:726-742.
* Logacheva et al. (2022) Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. 2022. Paradelctx: Detoxidation with parallel data. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 6804-6818. Association for Computational Linguistics.
* Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Linntang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Albham Fikri Aji, Khalid Almubarak, Samuel Albaniezaid Alvarezai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finentuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 15991-16111. Association for Computational Linguistics.
* Mukherjee et al. (2023) Sourabrata Mukherjee, Akanksha Bansal, Pritha Majumdar, Atul Kr. Ojha, and Ondrej Dusek. 2023. Low-resource text style transfer for Bangla: Data & models. In _Proceedings of the First Workshop on Bangla Language Processing (BLP-2023)_, pages 34-47, Singapore. Association for Computational Linguistics.
* Mun et al. (2023) Jimin Mun, Emily Allaway, Akhila Yerukola, Laura Vianna, Sarah-Jane Leslie, and Maarten Sap. 2023. Beyond denouuncing hate: Strategies for countering implied biases and stereotypes in language. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 9759-9777, Singapore. Association for Computational Linguistics.
* dos Santos et al. (2018) Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 189-194, Melbourne, Australia. Association for Computational Linguistics.
* Pereira-Kohatsu et al. (2019) Juan Carlos Pereira-Kohatsu, Lara Quijano Sanchez, Federico Liberatore, and Miguel Camacho-Collados. 2019. Detecting and monitoring hate speech in twitter. _Sensors_, 19(21):4654.
* Perez et al. (2022) Juan Manuel Perez, Damian Ariel Furman, Laura Alonso Alemany, and Franco M. Luque. 2022. RoBERTuito: a pre-trained language model for social media text in Spanish. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 7235-7243, Marseille, France. European Language Resources Association.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, 

Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67.
* Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 129-140, New Orleans, Louisiana. Association for Computational Linguistics.
* Semiletov (2020) Aleksandr Semiletov. 2020. Toxic Russian Comments: Labelled comments from the popular Russian social network. [https://www.kaggle.com/alexandersemiletivo/toxic-russian-comments](https://www.kaggle.com/alexandersemiletivo/toxic-russian-comments). Accessed: 2023-12-14.
* Sheppard et al. (2023) Brooklyn Sheppard, Anna Richter, Allison Cohen, Elizabeth Allyn Smith, Tamara Kneese, Caroyne Pelletier, Ioana Baldini, and Yue Dong. 2023. Subtle misgoryn detection and mitigation: An expert-annotated dataset. _CoRR_, abs/2311.09443.
* Shliazhko et al. (2024) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. mGPT: Few-Shot Learners Go Multilingual. _Transactions of the Association for Computational Linguistics_, 12:58-79.
* Tang et al. (2023) Zecheng Tang, Keyan Zhou, Pinzheng Wang, Yuyang Ding, Juntao Li, and Min Zhang. 2023. Detoxif language model step-by-step. _CoRR_, abs/2308.08295.
* Touvron et al. (2021) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiboul, Jude Fernandes, Jeremy Fu, Wenjin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Namam Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inanc, Marcin Kardas, Viktor Kerkeez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihalyov, Pushkar Mishra, Igor Mobye, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Runeta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambaduri, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciolom. 2023. Llama 2: Open foundation and fine-tuned chat models. _CoRR_, abs/2307.09288.
* van Aken et al. (2018) Betty van Aken, Julian Risch, Ralf Kretel, and Alexander Loser. 2018. Challenges for toxic comment classification: An in-depth error analysis. In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_, pages 33-42, Brussels, Belgium. Association for Computational Linguistics.
* a new dataset and classification experiments for implicitly abusive language. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 16280-16297, Singapore. Association for Computational Linguistics.
* Wormer (2022) Titus Wormer. 2022. Cuss: Map of profantities, slurs, and obscenties to a sureness rating. [https://github.com/words/cuss](https://github.com/words/cuss). Accessed: 2023-12-14.

 

MultiParaDetox Crowdsourcing Tasks and Instructions

Here, we list the texts of crowdsourcing task titles and instructions in their original form used to collect MultiParaDetox correspondingly for each languages: (i) Russian (Section A.1); (ii) Ukrainian (Section A.2); (iii) Spanish (Section A.3).

### Russian

**Task 1:**Preentisite tekt v bezklivnom stlie

_Instructions_

Ram dolet pokazan tekt, kotoryri, bozmozhno, oderzhit okorgleniia lij prubye vyrzakennia. Nam gretoetsa perepisast' ego tak, tudoykh constrainty soderzhaniet i izloshnytsi ot skroglenieni. Selin tekt ne oskoorbitel'nyi i ne grupyi, vychire opticino "Tekt neg'iazu retstsat"n u zhaikite principu.

_Task interface_

Sterchnitte tekt tak, tudoy v mem he blylo oskorbleniia, a solerzhine ne pomechalosv. Possible answers:

* Bain warnaint
* Sto bescummelennini elekt
- B tekste i tak net oskorbleniii
- Nevoomomiko yorratv okorbleniia sez znaitieelynoogo izmenenia soderzhania

**Task 2:**Cravniite predlozhennia po smyslu _Instructions_

Vb vvidute dva predlozhennia. Vansza zadana sostoin v tom, tudoy obredleinit, anachat in onis odno i to de. Predlozhennia ie podokvin bytv akoblitno elementiny po smyslu - odno is mix mozket byt' oskorbleniel'nym, a drutoe soderzhatst tu zhe informatiho v nehtral'nom tone.

Esli odno i ne predlozhenni pli opa predlozhennia eesmsyclenny liu sodernkat bescumselive slova/drawy zametulionionie pnismamia, vychirente otvet "Her".

_Task interface_

Gui predlozhennia znaust odno i to deze?

* Da
* Het 



### Ukrainian

**Task 1:** Pretennijt tekt u lemmu stelli

_Instructions_

Ram tofie pokazano tekst, zikih, moklivov, mistnt' obrazia do fryoi sprazin. Nam potrivao neoretisati bio tak, mobe sereetti amsic i pozubitnsi ograa. Khiko tekst ee okrazhnyi i ie strubii, viberiu opisho "Teket ne moknka nerentsatsui" kuzakit privintsu. Tekt moke Rutit a gold-akinm okrasom - posnitrishim ta pestativism. Takozh moke Rutit a termatentimiemi polnikami. De realvi elekst-nosti abo komentaki y sollidnykh mereik. Okras (ta smict) trega aveetti takhim. ziki wir se, pomislku snipraylita ne obov'iazkovo. _Task interface_

Pretennijt tekt tak, phob u momu ne budlo obraza, alezmst ne umishova. Possible answers:

* Bain variaint
* Tekt he mozhika perestastici
* Ile gezzmistotynii tekt
* V teksti i tak memae obraza
* Lemokhilivo privratni obrazin bez naannoi zmini mistu
* Inspe

**Task 2:** Poriniyaite perennia za mimtom

_Instructions_

By poznatite dava rebennia. Raume zavadaniu poliytaze v tomu, phob vizhaemiti, ch oznavaioty rosin ofei tse same. Pehennia ne povinishi ostohodionheleintnihimi za mimeton - ofei zhik mokoe butit orazliaimum, a inise mistuti tu samu informalyio v neiteral'nomu tori. Ale poloyne, phob osnovshi zmistoryna Lastrina bulina odna i ta kama.

Khiko ode 3 recheniya solo obliva rechennia zealuzid abo micutt' Bezuludi slova/drazin, phos skkalatonotory raominisia, viberiu yildonvid "Hi".

_Task interface_

By rechennia ozonanaaroty odne i te came?

* Tak
* Hi

**Task 3:** Le obrazlivnyi tekt?

_Instructions_

Bam topribino prochitatin rechennia i vizhaemiti, ch instztv rosin obrazin abo nevenzupri ta rtybi siova.

Vbarael Neobrazine rechennia mooke mistiti krinkikhikh i buti netativno zabarzienim. _Task interface_

The nicht' lei tekt obrazin abo nevenzupri silova?

* Tak
* Hi

**A.3 Spanish**

**Task 1:** Reescribir el texto en un estilo cortes

_Instructions_

Se le mostrara un texto que puede contener lenguaje ofensivo o duro. Debera reescribiro de forma que conserve el significado y elimine el lenguaje offensive. Si el texto no es ofensivo o malsonante, seleccione la opicon "El texto no puede recexibirs" y explique el motiv

El texto puede ser de cualquier color, positivo o negativo. Tambien puede contener errores gramaticales. Se trata de textos reales -posts o comentarios de redes sociales. El color (y el contenido) debe dejarse tal cual, y no es necesario corregir ningun error.

_Task interface_

Reescribe el texto de modo que no contenga insultos pero que el significado siga siendo el mismo. Possible answers:

* Su opcion
* El texto no puede reescribrise
* Este es un texto sin sentido
* De todas formas, no hay insultos en el texto
* Es imposible eliminar los insults sin cambiar el significado
**Task 2:**: ?Estas frasse significant lo mismo?

_Instructions_

Se le mostraran dos frases. Su tarea consiste en indicar si significan lo mismo (o algo parecido) o no. Las frases no tienen que ser identicas: una de ellas puede ser ofensiva y la otra decir lo mismo en tono neutro.

Si una o ambas franses contienen sinusentidos (no-palabras, cadenas de palabras sin sentido, etc.), elija la opcion "No".

_Task interface_

?Estas dos fresses significican lo mismo?

* Sf
* No

**Task 3:**: ?Es ofensivo este texto?

_Instructions_

Debe leer las frases y determinar si son ofensivas o no. Los textos ofensivos son los que contienen insulots, amenazas, palabrotas. Los textos no ofensivos pueden contener criticas y ser negativos (pero no installantes) hacia el interlocutor.

_Task interface_

?Contiene este texto ofensas o palabrotas?

* Si
* No

### Interface examples

Figure 2: Paraphrasing task (Task 1) interface example for Spanish.

 



## Appendix B Samples from ParaDetox for New Languages

Here, we report examples from MultiParaDetox obtained for new languages: (i) Russian (Table 4); (ii) Ukrainian (Table 5); (iii) Spanish (Table 6).

\begin{tabular}{l|c} \hline Original & h*e@**dl de gtot xeroi fbli do ottoo so svomi dokazatejctbami? \\  & and where the file's was this here before with his evidence? \\ \hline Paraphrases & Hy ii die gtot reproi fbli,so svomi dokazatejctbami? \\  & So where was this hero with its evidence? \\  & i,gle ottou repoi fbli do ottoo so svomi dokazatejctbami? \\  & and where was this hero before with his evidence? \\ \hline Original & x**hy ygakyio ninubit,n-a ata otoo jalkka.doloo*him. \\  & They wrote *** become of the block. Histidine balline? \\ \hline Paraphrases & Humily vkikhyo inubit, t3-a ata ottoo jalka. \\  & They are writing sentence because of the likes. \\  & Emuly bskynko inubit,n-a ata ottoo jalka. \\  & They were writing nonsense because of this like. \\ \hline Original & A naxt*ha toida tht zacey sto imgaat? \\  & Why the heft did you write this here then? \\ \hline Paraphrases & A dlia teto totla tyl adecy sto imgaat? \\  & Why did you write this there then? \\  & Zaumem tri qto imgaat \\  & Wby did you write this



\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline Original & pues haze los deberes fur**a barata \\  & so do my homework, you cheap** \\ \hline Paraphrases & Pues haze los deberes muchacha \\  & So do my homework girl \\  & Puedes hacerme los deberes mujer. \\  & You can do any homework for me, woman. \\ \hline \hline
**Original** & Este pais se va a la m**rda \\  & This country is going to as*! \\ \hline Paraphrases & nada puede salvar a este pais \\  & nothing can save this country \\  & Cosas van muy mal en este pais \\  & Things are very bad in this country \\ \hline \hline Original & Disney se está réendo en vuestra p**a cara \\  & Dimey is hanging in your fine*"ng face \\ \hline Paraphrases & Disney se está riendo en vuestra cara \\  & Dimey is tangling in your fence \\  & Disney se está riendo de ustedes \\  & Dimey is lengthening at you \\ \hline \end{tabular}
\end{table}
Table 6: Examples of parallel detoxified pairs from EsParaDetox.

\begin{table}
\begin{tabular}{l l} \hline Original & Elk kazab ojlih vejlknii, "II�

Text Detoxification Models Outputs

Here, we report the examples of text detoxification models' outputs (unsupervised baselines, LLM, and best-performing fine-tuned supervised models) for new languages: (i) Russian (Table 7); (ii) Ukrainian (Table 8); (iii) Spanish (Table 9).

\begin{table}
\begin{tabular}{l|c} \hline Original & LAaa.strana nyuklaets y terookx,in** a pokaet alurakob \\  & well. the country needs knows. e*+n gene birth on rookie \\ \hline Delete & LAa.strana nyuklaetsyi betrook ,in** da pokaet \\ CondBERT & LAa.strana nyuklaetsyi b rerookxi,in**da pokaet durakov \\  & well. the country needs knows. c++n genes birth to foels \\ LLaMa & YGrama nyuklaetsyi petrookz, a ne y inaka, kotorye demohestriritiviet nevekeketivo i nepdestatoñoe vuzazenie k datum. \\ ruT5-detox & LAa.strana nyuklaetsyi betrook, likozoi peelojek Rozkaet \\  & UIuhyki detei \\  & well. the country needs heores, bad person gives birth to fool children \\ \hline Original & otolo eaa*inki izobarile:d :d \\  & _They have got rid of one_ \\ Delete & otolo imajnilis:d :d \\  & _They have get rid of one_ \\ CondBERT & otolo tvari izobarile:d :d \\  & _They have got rid of one centave_ \\ LLaMa & Izobaritcz ot e6***ovich mokho, esli postarattszi. \\ ruT5-detox & It is possible to get rid of eba*ankar if you try hard. \\  & otolo denovnik ihizabriliscy \\  & _They have got rid of one girl_ \\ \hline Original & DE xbatillo? me dozhutts? **oil zhripon uto in barkynthia? :d :d :d \\  & what run enough? what should I said? Did you not! your but? :d :d \\ \hline Delete & DE xbatillo? me dozhutts? zhripon uto il pokrutilia :d :d :d :d \\ CondBERT & DE xbatillo? me dozhutts? zhripon uto il pokrutilia? :d :d :d :d \\  & what was enough? what should I said? it sick you not! your fat? :d :d \\ LLaMa & Original text: “I can’t believe you actually think that. You must be joking. This is complete nonsense.” \\  & Polite version: “I’m a bit surprised by your statement, I have to say. Could you perhaps provide some more context or clarify your point? I want to make sure I understand your perspective correctly." \\ ruT5-detox & DE xbatillo? me dozhutts? \\ \hline \end{tabular}
\end{table}
Table 7: Examples of text detoxification outputs by different models for Russian.

 



\begin{table}
\begin{tabular}{|l||c|} \hline Original & Peôtki, copi aa to, pio a taka k**ba. \\  & _Gusy, norvy for being such a w**be_ \\ \hline Delete & Patki, copi aa tq, pio a taka a. \\  & _Ryuli, sorry for being like that a._ \\ CondBERT & Peôtki , copi aa tq , pio a taka K. \\  & _Gusy, sorry for being like R._ \\ LLaMa & Alih, i sporolliosia zae te, pio a takoo diodina. \\  & Childen, I am abhamed that I am such a person. \\ BART-ParaDetox & Peôtki, copi aa tq, pio a taka dura. \\  & Gusy, sorry for being so unpial \\ \hline
**Original** & Ot poño a taka hixilo a**e*ci! \\  & _Oh cool, I want fatting amongst_ \\ \hline Delete & Ot polino a taka hixilo il \\  & _Oh cool, I really l’_ \\ CondBERT & Ot polino a taka hixilo y ! \\  & _Oh cool, I really y'_ \\ LLaMa & Klikjo a ne buu taka camotthim, a b naivelyno znashinov kratshi spocib viraznitn cvoi booutta. \\  & If I wasn t so lonely, I would probably find a better way to express my feelings \\ BART-ParaDetox & Ot polino a taka hixilo dlivnyavasi \\  & _Oh cool, I was so surprised_ \\ \hline Original & Beim ob**tha



\begin{table}
\begin{tabular}{|l|p{284.5pt}|} \hline Original & La �hica solución es irse de este país de m**rda. \\  & _The only solution is to fence this sit*>> country._ \\ \hline Delete & La �hica solución es rise de este pais de . \\  & _Rudik, sorry for being like that a._ \\ CondBERT & La �hica solución es irse de este pais de mi . \\  & _The only solution is to leave this country of:_ \\ LLaMa & El �hico camino para superar las dificultades en este momente es abandonar este lugar. Es hora de buscar un nuevo comienzo en un entorno mas propicio para el bienestar y la felicidad. \\  & _The only way to overcome the difficulties at this time it is to leave this place. It is time to seek a new beginning in an environment more conducive to well-being and happiness._ \\ BART-ParaDetox & **La �hica solución es irse de este pas \\  & _The only solution is to leave this country_ \\ \hline
Original & Aquí Piatti que no metia un p**0 gol y en el Espanol, Ileva yu unsçuntos, ma**n. \\  & _Heee Plant didn’t seem a free"ng goal and in the Spanish, he attreatly has a few...netere_ \\ \hline Delete & Aqui Piatti que no metia un gel y en el Espanol, Ileva y aunos cuantos, \\ CondBERT & Hee Plant who did not score a goal and in the Spanish, have already had a few._ \\  & _Aquí Piatti que no met un cuantos gel y en el Espanol , Ileva ya unus ocuantos , ma**n ._ [UNK] \\ LLaMa & Paint here we think it'sce a few goals and in the Spanish, he already has a few. s**ker [UNK] \\  & En este momento, el jugador Piatti no ha logrado anotar un gol y en el partido espanol, ha perdido algunos punto. ¡Que frustracion! \\  & _At this moment, the player Paint has not managed to score a goal and in the Spanish match, he has lost some points. What a frustrating!_ \\ BART-ParaDetox & Aqui Piatti que no metia un p**0 gol y en el Espanol, Ileva ya unos cuantos \\  & _Here Plant who didn’t score a free*ng goal and in the Spanish, he has already had a few_ \\ \hline Original & por favor dejad en paz a mi amiga la subno**al de m**rda \\  & _please leave my friend alone, you face**ng salv**mail._ \\ \hline Delete & por favor dejad en paz a mi amiga la sub**rmal de \\ CondBERT & Por favor dejad en paza a mi amiga la muñeca de mi \\  & _Please leave my friend any dall alone_ \\ LLaMa & I understand that you want to convey a message in a more polite and natural way while maintaining the original meaning. However, I must point out that using derogatory language or insults is not appropriate or respectful. Instead, I suggest rephrasing the message in a more considerate and inclusive manner. Here’s a possible rewording of your message: ”Please refrain from making negative comments or jokes about my friend who has intellectual disabilities. It’s important to treat everyone with respect and density, regardless of their abilities or disabilities.” \\ BART-ParaDetox & por favor dejad en paz a mi amiga \\  & _please leave my friend alone_ \\ \hline \end{tabular}
\end{table}
Table 9: Examples of text detoxification outputs by different models for Spanish.



Multilingual LLM Selection for Prompting Experiments

We experimented with several multilingual models--MT0-large (Muennighoff et al., 2023)7, BloomZ-7b (Muennighoff et al., 2023)8, and LLaMa-7b (Touvron et al., 2023)9--to test them for the text detoxification task for our target languages. In Table 10, we provide the models comparison results. MT0 and BloomZ showed worse J scores than LLaMa and in some cases extremely poor STA scores. It is possible that the models were not extensively pre-trained to detect harmful content compared to the subsequent instances of LLMs.

Footnote 7: [https://huggingface.co/bigscience/mt0-xxl-mt](https://huggingface.co/bigscience/mt0-xxl-mt)

Footnote 8: [https://huggingface.co/bigscience/bloomz-7b1-mt](https://huggingface.co/bigscience/bloomz-7b1-mt)

Footnote 9: [https://huggingface.com/meta-llama/Llama-2-7b-chat-hf](https://huggingface.com/meta-llama/Llama-2-7b-chat-hf)

The precise prompts used for the models are:

* _Ukrainian:_Perensitive! nastuitioni teket v billyvi vivil'niiii, ale primodini formai, ocepiratorii fioroi nevisitii zmist \((6\text{e}^{2}\times\text{kodim