Title: Who Measures What Matters? An Analysis of Social Impact Evaluations in Foundation Model Reporting
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: social impact, evaluations, foundation models

Abstract: As generative models become central to high-stakes AI systems, governance frameworks increasingly rely on evaluations to assess risks and capabilities. While general capability evaluations are common, social impact assessments, such as bias, privacy, and environmental effects remain fragmented, inconsistent, or absent. To characterize this landscape, we conduct the most comprehensive cross-provider analysis to date, examining 99 first-party release-time reports and 187 post-release sources to measure evaluation prevalence and strength, complemented by developer interviews to contextualize gaps. We find that first-party reporting is uneven, often superficial, and has declined over time. Third-party evaluations address some gaps but cannot cover critical areas such as data, content moderation labor, or environmental costs without provider disclosure. We further find that some developers deprioritize social impact evaluations unless tied to adoption or compliance, and that meaningful reporting is limited by resource constraints, reputational concerns, and the lack of standardized, practical frameworks. Our findings highlight that current evaluation practices leave major gaps in assessing foundation models' societal risks, underscoring the urgent need for more systematic, comparable, and transparent frameworks.

TL;DR: We analyze first- and third-party reports on foundation models, complemented by stakeholder interviews, to reveal major gaps and barriers in social impact evaluations such as bias, privacy, and environmental effects.