Phare: A Safety Probe for Large Language Models
Authors: name: Pierre Le Jeune; name: Benoît Malézieux; name: Weixuan Xiao; name: Matteo Dora
Published: 2025-05-16 15:31:08

Abstract: Ensuring the safety of large language models (LLMs) is critical for
responsible deployment, yet existing evaluations often prioritize performance
over identifying failure modes. We introduce Phare, a multilingual diagnostic
framework to probe and evaluate LLM behavior across three critical dimensions:
hallucination and reliability, social biases, and harmful content generation.
Our evaluation of 17 state-of-the-art LLMs reveals patterns of systematic
vulnerabilities across all safety dimensions, including sycophancy, prompt
sensitivity, and stereotype reproduction. By highlighting these specific
failure modes rather than simply ranking models, Phare provides researchers and
practitioners with actionable insights to build more robust, aligned, and
trustworthy language systems.