Holistic Evaluation of Language Models
Creators: name: Liang, Percy
name: Bommasani, Rishi
name: Lee, Tony
name: Tsipras, Dimitris
name: Soylu, Dilara
name: Yasunaga, Michihiro
name: Zhang, Yian
name: Narayanan, Deepak
name: Wu, Yuhuai
name: Kumar, Ananya
name: Newman, Benjamin
name: Yuan, Binhang
name: Yan, Bobby
name: Zhang, Ce
name: Cosgrove, Christian
name: Manning, Christopher D.
name: Ré, Christopher
name: Acosta-Navas, Diana
name: Hudson, Drew A.
name: Zelikman, Eric
name: Durmus, Esin
name: Ladhak, Faisal
name: Rong, Frieda
name: Ren, Hongyu
name: Yao, Huaxiu
name: Wang, Jue
name: Santhanam, Keshav
name: Orr, Laurel
name: Zheng, Lucia
name: Yuksekgonul, Mert
name: Suzgun, Mirac
name: Kim, Nathan
name: Guha, Neel
name: Chatterji, Niladri
name: Khattab, Omar
name: Henderson, Peter
name: Huang, Qian
name: Chi, Ryan
name: Xie, Sang Michael
name: Santurkar, Shibani
name: Ganguli, Surya
name: Hashimoto, Tatsunori
name: Icard, Thomas
name: Zhang, Tianyi
name: Chaudhary, Vishrav
name: Wang, William
name: Li, Xuechen
name: Mai, Yifan
name: Zhang, Yuhui
name: Koreeda, Yuta
Publisher: name: arXiv
Year: 2022
Subjects: subject: Computation and Language (cs.CL)
subject: Artificial Intelligence (cs.AI)
subject: Machine Learning (cs.LG)
subject: FOS: Computer and information sciences
subject: FOS: Computer and information sciences

description: Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models. First, we taxonomize the vast space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata) that are of interest for LMs. Then we select a broad subset based on coverage and feasibility, noting what's missing or underrepresented (e.g. question answering for neglected English dialects, metrics for trustworthiness). Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios when possible (87.5% of the time). This ensures metrics beyond accuracy don't fall to the wayside, and that trade-offs are clearly exposed. We also perform 7 targeted evaluations, based on 26 targeted scenarios, to analyze specific aspects (e.g. reasoning, disinformation). Third, we conduct a large-scale evaluation of 30 prominent language models (spanning open, limited-access, and closed models) on all 42 scenarios, 21 of which were not previously used in mainstream LM evaluation. Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: now all 30 models have been densely benchmarked on the same core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings. For full transparency, we release all raw model prompts and completions publicly for further analysis, as well as a general modular toolkit. We intend for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.
description: Authored by the Center for Research on Foundation Models (CRFM) at the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Project page: https://crfm.stanford.edu/helm/v1.0