Position: A Roadmap to Pluralistic Alignment
ICML, 2024

With increased power and prevalence of AI systems, it is ever more critical that AI systems are designed to serve <em>all</em>, i.e., people with diverse values and perspectives. However, aligning models to serve <em>pluralistic</em> human values remains an open research question. In this piece, we propose a roadmap to pluralistic alignment, specifically using large language models as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems: 1) <em>Overton pluralistic</em> models that present a spectrum of reasonable responses; 2) <em>Steerably pluralistic</em> models that can steer to reflect certain perspectives; and 3) <em>Distributionally pluralistic</em> models that are well-calibrated to a given population in distribution. We also formalize and discuss three possible classes of <em>pluralistic benchmarks</em>: 1) <em>Multi-objective</em> benchmarks, 2) <em>Trade-off steerable</em> benchmarks that incentivize models to steer to arbitrary trade-offs, and 3) <em>Jury-pluralistic</em> benchmarks that explicitly model diverse human ratings. We use this framework to argue that current alignment techniques may be fundamentally limited for pluralistic AI; indeed, we highlight empirical evidence, both from our own experiments and from other work, that standard alignment procedures might <em>reduce</em> distributional pluralism in models, motivating the need for further research on pluralistic alignment.

Included in review: No