Preference Tuning For Toxicity Mitigation Generalizes Across Languages
Authors: name: Xiaochen Li; name: Zheng-Xin Yong; name: Stephen H. Bach
Published: 2024-06-23 22:53:47

Abstract: Detoxifying multilingual Large Language Models (LLMs) has become crucial due
to their increasing global use. In this work, we explore zero-shot
cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike
previous studies that show limited cross-lingual generalization for other
safety tasks, we demonstrate that Direct Preference Optimization (DPO) training
with only English data can significantly reduce toxicity in multilingual
open-ended generations. For example, the probability of mGPT-1.3B generating
toxic continuations drops from 46.8% to 3.9% across 17 different languages
after training. Our results also extend to other multilingual LLMs, such as
BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal
intervention and activation analysis, we identified the dual multilinguality
property of MLP layers in LLMs, which explains the cross-lingual generalization
of DPO. Finally, we show that bilingual sentence retrieval can predict the
cross-lingual transferability of DPO preference tuning.