RabakBench: Scaling Human Annotations to Construct Localized
  Multilingual Safety Benchmarks for Low-Resource Languages
Authors: name: Gabriel Chua; name: Leanne Tan; name: Ziyu Ge; name: Roy Ka-Wei Lee
Published: 2025-07-08 13:37:25

Abstract: Large language models (LLMs) and their safety classifiers often perform
poorly on low-resource languages due to limited training data and evaluation
benchmarks. This paper introduces RabakBench, a new multilingual safety
benchmark localized to Singapore's unique linguistic context, covering
Singlish, Chinese, Malay, and Tamil. RabakBench is constructed through a
scalable three-stage pipeline: (i) Generate - adversarial example generation by
augmenting real Singlish web content with LLM-driven red teaming; (ii) Label -
semi-automated multi-label safety annotation using majority-voted LLM labelers
aligned with human judgments; and (iii) Translate - high-fidelity translation
preserving linguistic nuance and toxicity across languages. The final dataset
comprises over 5,000 safety-labeled examples across four languages and six
fine-grained safety categories with severity levels. Evaluations of 11 popular
open-source and closed-source guardrail classifiers reveal significant
performance degradation. RabakBench not only enables robust safety evaluation
in Southeast Asian multilingual settings but also offers a reproducible
framework for building localized safety datasets in low-resource environments.
The benchmark dataset, including the human-verified translations, and
evaluation code are publicly available.