WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language
  Models
Authors: name: Prannaya Gupta; name: Le Qi Yau; name: Hao Han Low; name: I-Shiang Lee; name: Hugo Maximus Lim; name: Yu Xin Teoh; name: Jia Hng Koh; name: Dar Win Liew; name: Rishabh Bhardwaj; name: Rajat Bhardwaj; name: Soujanya Poria
Published: 2024-08-07 15:22:44

Abstract: WalledEval is a comprehensive AI safety testing toolkit designed to evaluate
large language models (LLMs). It accommodates a diverse range of models,
including both open-weight and API-based ones, and features over 35 safety
benchmarks covering areas such as multilingual safety, exaggerated safety, and
prompt injections. The framework supports both LLM and judge benchmarking, and
incorporates custom mutators to test safety against various text-style
mutations such as future tense and paraphrasing. Additionally, WalledEval
introduces WalledGuard, a new, small and performant content moderation tool,
and SGXSTest, a benchmark for assessing exaggerated safety in cultural
contexts. We make WalledEval publicly available at
https://github.com/walledai/walledevalA.