ICML 2025

Conformal Tail Risk Control for Large Language Model Alignment
Catherine Chen 1 Jingyan Shen 2 Zhun Deng 3 Lihua Lei 4 5
Abstract
Recent developments in large language models
(LLMs) have led to their widespread usage for
various tasks. The prevalence of LLMs in society
implores the assurance on the reliability of their
performance. In particular, risk-sensitive applica-
tions demand meticulous attention to unexpect-
edly poor outcomes, i.e., tail events, for instance,
toxic answers, humiliating language, and offen-
sive outputs. Due to the costly nature of acquiring
human annotations, general-purpose scoring mod-
els have been created to automate the process of
quantifying these tail events. This phenomenon in-
troduces potential human-machine misalignment
between the respective scoring mechanisms. In
this work, we present a lightweight calibration
framework for blackbox models that ensures the
alignment of humans and machines with provable
guarantees. Our framework provides a rigorous
approach to controlling any distortion risk mea-
sure that is characterized by a weighted average
of quantiles of the loss incurred by the LLM with
high confidence. The theoretical foundation of our
method relies on the connection between confor-
mal risk control and a traditional family of statis-
tics, i.e., L-statistics. To demonstrate the utility of
our framework, we conduct comprehensive exper-
iments that address the issue of human-machine
misalignment.
1. Introduction
Large Language Models (LLMs) have proven to be per-
vasive in society with applications across various settings,
*Equal contribution
1Institute for Computational and Math-
ematical Engineering, Stanford University, CA 2Department of
Industrial Engineering and Operations Research, Columbia Uni-
versity, NY 3Department of Computer Science, UNC Chapel
Hill, NC 4Graduate School of Business, Stanford University, CA
5Department of Statistics (by courtesy), Stanford University, CA.
Correspondence to: Catherine Chen <cyc2...