The 2024 Foundation Model Transparency Index
Authors: Rishi Bommasani
Kevin Klyman
Sayash Kapoor
Shayne Longpre
Betty Xiong
Nestor Maslej
Percy Liang
Venue: Accepted by TMLR ()

Abstract: Foundation models are increasingly consequential yet extremely opaque. To characterize the status quo, the Foundation Model Transparency Index was launched in October 2023 to measure the transparency of leading foundation model developers. The October 2023 Index (v1.0) assessed 10 major foundation model developers (e.g. OpenAI, Google) on 100 transparency indicators (e.g. does the developer disclose the wages it pays for data labor?). At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study (v1.1) after 6 months: we score 14 developers against the same 100 indicators. While in v1.0 we searched for publicly available information, in v1.1 developers submit reports on the 100 transparency indicators, potentially including information that was not previously public. We find that developers now score 58 out of 100 on average, a 21 point improvement over v1.0. Much of this increase is driven by developers disclosing information during the v1.1 process: on average, developers disclosed information related to 16.6 indicators that was not previously public. We observe regions of sustained (i.e. across v1.0 and v1.1) and systemic (i.e. across most or all developers) opacity such as on copyright status, data access, data labor, and downstream impact. We publish transparency reports for each developer that consolidate information disclosures: these reports are based on the information disclosed to us via developers. Our findings demonstrate that transparency can be improved in this nascent ecosystem, the Foundation Model Transparency Index likely contributes to these improvements, and policymakers should consider interventions in areas where transparency has not improved.

Published in Transactions on Machine Learning Research (12/2024)
The 2024 Foundation Model Transparency Index
Rishi Bommasani* Stanford University
mlprishhe@@stanford.cd edu
Kevin Klyman* Stanford University
I edu
Sayash Kapoor Princeton University
sayashhd@princetton..kk edbu
Shayne Longpre
slongpre@media.miie mit. edu
Massachusetts Institute of Technology
Betty Xiong Stonford University
Stongg@stafforda edu
Nestor Maslej Stanford University
amassee@Sstanorrd.com edu
Percy Liang Stanford University
plaang@@cs.stanfoor.com edu
Reviewed on OpenReview: https: openreview. net/ forum? id= 38 Vnn
Abstract
Foundation models are inercasingly consoquential yet extremely opaque. To characterize the status quo, the Foundation Model Transparency Index was launched in October 2023
to measure the transparency of leading foundation model developers. The October 2023
Index (v1.0) assessed 10 major foundation model developers (e.g. OpenAI, Google) 100
transparency indicators (e.g. does the developer disclose the wages it pays for data labor?).
At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study (v1.1) after 6 months: we score 14 developers against the same 100 indicators. While v1.0 we searched for publicly available information, v1.1 developers submit reports on the 100 transparency indicators, potentially including information that was not previously public. We find that developers now score 58 out of 100 on average, 21 point improvement over v1.0. Much of this increase is driven by developers disclosing information during the v1.1 process: on average, developers disclosed information related to 16.6 indicators that was not previously public. We observe regions of sustained (i.c. across v1.0 and v1.1) and systemic (i.e. across most or all developers) opacity such as on copyright status, data access, data labor, and downstream impact. We publish transparency reports for each developer that consolidate information disclosures: these reports are based on the information disclosed to us via developers. Our findings demonstrate that transparency can be improved in this nascent. ecosystem, the Foundation Model Transparency Index likely contributes to these improvements, and policymakers should consider interventions in areas where transparency has not improved.
1

---

Published in Transactions on Machine Learning Research (12/2024)
1 Introduction
Foundation models are the epicenter of artificial intelligence (AI) as Al begins to shape how the economy and sociely function (Bommasani et al.) 2021). For such a high-impact technology, transparency vital to facilitate accountability, competition, and collective understanding. As an illustrative example, the current lack of transparency regarding the data used to build foundation models makes it difficult to assess what copyrighted information is used to train foundation models. Governments around the world are intervening to increase transparency: for example, the EU AI Act and the US's proposed AI Foundation Model Transparency Act take major strides by mandating a number of disclosure requirements [Bommasani et al. A).
2024 Appendix
To characterize the transparency of the foundation model ecosystem, Bommasani et al. (023a) ) introduced
the Foundation Model Transparency Index (FMT1). Launched in October 2023, the first iteration of the index (FMTI v1.0) scored 10 major foundation developers (c.g. OpenAI, Google, Meta) based on publicly available information regarding 100 transparency indicators. These 100 indicators span matters such as the data, labor, and compute used to build models; the capabilities, limitations, and risks associated with models; and the distribution of models as well as the impact of their use. FMTI v1.0 established that, in the status quo, the foundation model ecosystem was opaque: the average score was 37 points out of 100. Yet FMTI v1.0 also identified heterogeneity public disclosures: while the top score was a 54, for 82 indicators at least. one developer scored a point.
To understand how the landscape has evolved in the last 6 months, we conduct follow-up study (FMTI I To enable direct comparison, we retain the 100 transparency indicators and the associated threshold for awarding a point from FMTI v1.0. However, instead of searching for public information as was done in FMTI v1.0, we request that developers report the relevant information for each indicator. We implemented this change for three reasons: (i) completeness: we obviate the concern that information was missed when scarching the Intornet; (ii) clarity: we reduce uncortainty by having developers affirmatively disclose information; and (iii) scalability: we remove the effort required for researchers to conduct an open-ended search for decentralized public information.
We contacted 19 foundation model developers, and 14 provided reports related to the 100 transparency indicators (Adept, A121 Labs, Aleph Alpha, Amazon, Anthropic. BigCode/Hugging Face//SrriceNoww Google, IBM, Meta, Microsoft, Mistral, OpenAT, Stability AT, Writer) P Given each developer's initial report, we provided scores based on whether each disclosure satisfied the associated indicator. Developers responded to these initial scores, engaging in dialogue via email and virtual meetings, and clarifying matters in many cases. Following this iterative process, for each developer we publish transparency report that consolidates the information it discloscs. These reports contain new information, which developers had not disclosed publicly prior to the start of FMTI v1.1. On average, developers disclosed information related to 16.6 indicators that. was not previously public.
The FMTI v1.1 results demonstrate ample room for improvement, as well as tangible improvements in transparency over half a year. On average, developers disclose information that satisfies 58 of the 100 transparency indicators. Developers are least transparent with respect to the upstream resources required to build their foundation models, scoring 46%, in comparison to 65% on downstream indicators and 61% on model-related indicators. Developers can become more transparent by drawing on the transparency practices of other developers at least one developer scores a point on 96 of the 100 indicators, and multiple developers score a point on 89 indicators.
We find that developers' scores improved significantly over FMTI v1.0, with a 21 point improvement in the mean overall score. Scores improved across every domain, with upstream, model, and downstream scores improving by 6 7 points. Each of the 8 developers that were evaluated in both v1.0 and v1.1 improved their scores, with an average increase of 19 points. While some developers disclosed substantially more (e.g. AT21
hnttps:/cerfn.. httpss///ertssstafood....o www.ssattaaodd.uoo edu/fmti
"The change from FMTT v1.0 the inclusion of developers (Adept, Aleph Alpha, IBM, Microsoft, Mistral, Writer) and the exclusion of developers (Cohere, Inflection).
2

---

Published in Transactions on Machine Learning Research (12/2024)
Labs's score increased by 50 points), others made fairly marginal changes (e.g. OpenAI's score increased by just 1 point).
These findings affirm that greater transparency is feasible in the foundation model ecosystem. Further, they suggest that the Foundation Model Transparency Index in tandem with other interventions drives improvements transparency. However, given that there has been very little progress on specífic indicators (e.g. external data access, mitigations evaluations), we encourage policymakers to assess what level of minimum transparency is needed and to pursue policy interventions accordingly. We publish the transparency reports for all developers to enable further research 3
2
Background
To contextualize our effort, we describe FMTI v1.0 and prior work multi-iteration indices.
2.1 The Foundation Model Transparency Index
Bommasani et al. (2023a) launched the Foundation Model Transparency Index in October 2023, To conceptualize transparency for foundation models, they introduced a hicrarchical taxonomy aligned with the foundation model supply chain (Bommasani et. al., 2023b). This taxonomy featured three top-level domains: the upstream resources involved in developing foundation model, the foundation model itself and its properties, and the downstream use of the foundation model. These domains aggregate 23 subdomains (e.g. the upstream domain contains data, labor, data access, and compute as subdomains) and 100 binary transparency indicators. Using public information identified through a systematic search protocol, FMTI v1.0 scored 10 companies (A121 Labs, Amazon, Anthropie, Cohere, Google, BigScience/ BigScience/Hugging Hugging Face, Inflection, Meta, OpenAT, Stability AT) from 0 100 on the 100 indicators. Companies were sent initial scores and allowed to contest them before FMTI v1.0 was released.
FMTI v1.0 showed pervasive lack of transparency in the foundation model ecosystem, with the highest- performing developer scoring just 54 out of 100. Developers disclosed very little information about the labor or compute used to build their foundation models (scoring just 17% on these subdomains), or their real world impact (scoring 11% on this subdomain). Open foundation model developers-wiiee -which refers to developers who released their flagship foundation model openly (i.e. with widely available model weights; Kapoor el al. 2024)-out outpoooorrppooeeeeeeeee closed developers by a wide margin: all three developers with open flagship foundation models were among the top four scoring developers. Several companies disclosed almost no information about. their flagship foundation models, with three companies scoring 25% or less.
2.2 Indices over time
A central objective of an index is to track concept over time to characterize changes. In doing so, many notable indices have evolved over time to reflect changing circumstances and priorities. For instance, the Human Development Index (HDI) was changed multiple times in the 1990s and 2000s, largely in response to academic criticism [Klasen 2018 Stanton 2007). Despite these changes, which complicate direct comparisons across index iterations, the HDI remains one of the most trustworthy and popular indices for human development.
As an index conducted repeatedly, and the world changes as measured by the index, a natural question is how the index contributes to this change. Attributing why corporate bohavior (such as disclosure practices) changes is notoriously difficult. Companies generally do not reveal why they make changes and changes generally reflect conftuence of multiple factors. Kogen 2022) provides unique demonstration of an index's impact, analyzing the 2018 Ranking Digital Rights Index (RDR), which ranked the freedom of expression and privacy policies of 26 of the world's largest ICT companies. By reviewing internal RDR documents and interviewing relevant stakeholders (e.g. representatives from 11 companies and 14 civil society groups), Kogen concluded that RDR had clear influence and that indexes can be useful resources for social movements. In the case of FMTI, we expect that the Index brings attention to the disclosure practices of companics,
httpp::/www/ githu com/ commsstanoord-ee crfm/fnti
3

---

Published in Transactions on Machine Learning Research (12/2024)
making it casier for media, policymakers, investors, customers and the public apply additional pressure that engenders greater transparency. Additionally, the Index provides clarity to companies by setting concrete targets and empowers employees within companies to push for greater transparency.
To reason about an index's impact over time, we also draw inspiration from Raji & Buolamwini 2019 In 2017, Buolamwini & Gebru (2018) demonstrated significant performance disparities across demographic groups in 3 face recognition systems (from 1BM, Microsoft, and Megvii). A year later, Raji & Buolamwini (201 audited five systems (IBM, Microsoft, Megvii, Amazon, Kairos): they found that the original 3 systems had reduced the performance disparities considerably, whereas the 2 new systems in 2018 showed large disparities comparable to those seen in the 3 systems from 2017. In 64.2 we similarly explore how foundation model developers fare in FMTI v1.1 when stratified by whether they were assessed in FMTI v1.0.
3 Methods
FMTT v1.1 involves four steps: indicator selection, developer selection, information gathering, and scoring. We describe these steps and how they relate to their implementation in FMTI v1.0 below.
3.1 Indicator selection
To concretize transparency, we the 100 indicators from FMTI v1.0: Bommasani et al. 2023a.) defined these indicators based the literature Al transparency. The 100 indicators are listed in Figure The indicators are organized hierarchically into 3 domains (upstream, model, downstream) and several subdomains therein.
32 upstream indicators address the resources involved in foundation model development. Primarily, this relates to transparency around training data, labor practices, computational costs, and code/technice decisions. For example, the compute subdomain covers matters like the amount of hardware used to train model, the owner of that hardware, the associated computational cost and training duration, and resulting energy and environmental impacts.
33 model indicators address the model iteself. Primarily, this relates to transparency around basic model properties, model access, capabilities, risks, and mitigations. For example, the risk subdomain covers whether risks are enumerated and are legible to laypersons, whether risks are evaluated pre-deployment for both unintentional (e.g. bias) and malicious (e.g. disinformation) type risks, and whether external parties evaluate risk.
35 downstream indicators address the distribution and usage of models. Primarily, this relates to distribution practices, usage policies, privacy protections, and downstream impact in socicty and the economy. For example, the distribution subdomain covers how release decisions are made by the developer, where the model distributed, and whether it is integrated into other products and services by the developer.
Overall, these indicators are the byproduct of a wealth of rescarch in these different subdomains spanning labor (Gray & Suri 2019a [Crawford] 2021: Hao & Seetharaman, 2023), data (Bender & Friedman 2018
Gebru et al.) 2018
Longpre et al. 2023b a), compute (Lacoste et al.) 2019 Schwartz et al. 2020 Patterson
ct al.
2021 Luccioni & Hernández-Grrn Garcia.
2023).
evaluation [Liang et al.
2023), safety (Cammarota et al.
2020
Longpre et al. 2024a), privacy [EU] 2016 Brown et al. 2022: Vipra & Myers West 2023) Winograd
2023),
policies (Kumar et al. 2022 Weidinger et al. 2021 Brundage et al. 2020), and impact (Tabsss 2023
Weidinger et al. 2023)
3.2
Developer selection
In FMTI v1.0. Bommasani et al. (0023a selected 10 foundation model developers: all 10 were companies developing salient foundation models with consideration given for diversity (e.g. type of company, type of foundation model). Further, for each foundation model developer, Bommasani et al. (2023a) designated a
flagship foundation model that was used as the basis for scoring the developer. In FMTI vl.1, we require
For full definitions, see httpps/////wwww.. www.wwww..tt.......... www/staaford-crrta//oaaoooo
4

---

Published in Transactions on Machine Learning Research (12/2024)
2023 Foundation Model Transparency Index Indicators
Upstream
Model
Downstream
Data size
Input modality
Release decision-making
Data sources
Output modality
Release process
Data creators
Model components
Distribution channels
Data source selection
Model size
Products services
Data curation
Model architecture
Detection of machine-generated content
Data augmentation
Centralized model documentation
Model License
Harmful data filtration
External model access protocol
Terms of service
Copyrighted data
Blackbox external model access
Permitted prohibited users
Data license
Full external model sccess
Permitted, restricted, and prohibited uses
Personal
Capabilities description
Usage policy enforcement
Capabilities demonstration
Justification enforcement action
Employment of laborers
Evaluation capabilities
Usage policy violation appeals mechanism
Geographic distribution of data laborers
repraducibility capabilities
Permitted, restricted, and prohibited model behaviors
Wages
Third party capabilities evaluation
Model behavior policy enforcement
Instructions creating data
Limitations description
Interoperability of usage and model behavior policies
Labor protections
Limitations demonstration
User interaction with Al system
Third party partners
Third party evaluation of limitations
Usage disclaimers
Queryable external data access
Risks description
User data protection policy
Direct external data access
Risks demonstration
Permitted and prohibited use of user data
Compute usage
Unintentional harm evaluation
Usage data access protocol
Development duration
External repraducibility unintentional harm evaluation
Versioning protocol
Compute hardware
Intentional harm evaluation
Change log
Hardware owner
External reproducibility of intentional harm evaluation
Deprecation policy
Energy
Third party risks evaluation
Feedback mechanism
Carbon emissions
Mitigations description
Feedback summary
Broader environmental impact
Mitigations demonstration
Government inquiries
Model stages
Mitigations evaluation
Monitoring mechanism
Model objectives
External repraducibility of mitigations evaluation
Downstream applications
Core frameworks
Third party mitigations evaluation
Affected market sectors
Additional dependencies
Trustworthiness evalustion
Affected individuals
Mitigatioon for privacy
External reproducibility of trustworthiness evaluation
Usage reports
Mitigations for copyright
Inference duration evaluation
Geographic statistics
Inference compute evaluation
Redress mechanism
Centralized documentation for downstream use
Documentation for responsible downstream use
Figure 1: Indicators. The 100 indicators we use across domains (upstream, model, and downstream) that. are the same as in the October 2023 Foundation Model Transparency Index.

---

Published in Transactions on Machine Learning Research (12/2024)
companies to submit transparency reports we reached out to leadership at 19 companies: 01.AI, Adept, AT21 Labs, Aleph Alpha, Amazon, Anthropic, EisCode/Hugginng Face/ ServiceNow, Cohere, Databricks, Google, IBM, Inflection, Meta, Microsoft, Mistral, OpenAl, Stability AI, Writer, and xAI 14 developers agreed to prepare reports and designated their flagship foundation model
Name
Flagship Model
Release
Input:
Output
Status
Headquarters
WHI
WH2
FMF
ATA
Adept
Fuyu-SB
Open weights
T.
T
Startup
USA
A121 Labs
Juressio-2
API
T
Startup
Israel
Aleph Alpha
Laminous Supreme
API
T,
T
Startup
Germany
Amazon
Titan Text Express
API
T
Big Thch
USA
Anthropic
Clande 3
API
T
Sturtup
USA
BC/HEI /SN
StarCoder
Open weights
T
Startup
USA
Google
Gemini Ultra API
API
T, I.
A, V
T,
Big Tech
USA
IRM
Granite
API
T
Big Trch
USA
Meta 1
Llama 70B
Open weights
T
Big Toch
USA
Microsoft
Phi-2
Open weights
T
Big Tech
USA
Mistral
Mistral 7B
Open weights
T
Startup
France
OpenAT f
GPT-4
API
T, I
T
Startup
USA
Stability AI
Stable Video Ditfusion Palmyra-X
Open weights
V
Startup
UK
Writer
API
T
Startup
USA
Table 1: Selected Foundation Model Developers. Information on the 14 selected foundation model developers: the developer name, its flagship model, the release strategy for the model, the model's input and output modalities, the developer's corporale stalus, and the developer's headquarters. BC/HF/SI abbreviates BigCode/ /Hugging Pace/ServiceNow. T, A, and V abbreviate text, image, audio, and video modalities, respectively. indicates the developer was evaluated in FMTI v1.0. We also indicate if developers were involved in the White House's voluntary commitments for the management of risks posed by AT announced in July 2023 (WH1), and commitments by additional organizations in the same areas announced in September 2023 (WH2) as well as if they are founding member of the Frontier Model Forum (FMF) or AI Alliance (ATA). Concurrent with the release of FMTI vl.1, as of May 20, 2024, Amazon and Meta have joined the FMF.
describes the developers and their flagship foundation models. of the 14 developers are hold-overs from FMII v1.0 Three developers are assessed for the same models as v1.0 (Jurassic-2 for A121 Labs, Llama 2 for Meta, GPT-4 for OpenAI), whercas five are assessed for new models (Titan Text Express for Amazon, Claude 3 for Authropic, StarCoder for BigCode/Huggee Hta//Hugggigaaeee Seaeee////eiiwwww. Gemini 1.0 Ultra API for Google, and Stable Video Diffusion for Stability AT). The six new developers and their models are Fuyu-8B for Adept, Luminous Supreme for Aleph Alpha, Granite for IBM, Phi-2 for Microsoft, Mistral 7B for Mistral, and Palmyra-X for Writer. In aggregate, the FMTI v1.1 composition has broader geographic coverage (e.g. from 0 to companies headquartered in the Furopean Union), broader modality coverage (e.g. Gemini takes audio and video as input), and a more even balance of open and closed foundation models (i.e. from of 10 open models in v1.0 to 6 of 14 open models in v1.1).
Table
3.3 Information gathering
In FMTI v1.0, |Bommssani et al. (2023a) identified publicly-vvallabe sources of information for each developer through systematic protocol for searching the Internet, which provided the information for all scoring decisions. This approach has four potentially undesirable properties. First, given that information is
AA we describe later, companies submitted an initial report that was modified through the FMTI v1.1 process. The final report that we publish is validated by the company but, therefore, different from this initial report. For brevity, we refer to both as transparency reports.
'FMTI v1.0 evaluated BigScience/ DeaScienoe/Huuggig Face, which together developed the BLOOMZ model; in FMTI v1.1 we evaluate EegCode/Hugging Face/ Sewviieeewww which together developed the StarCoder model. Throughout this paper we refer to BigCode/Hngging Pace/SerriceeNos
single entity (i.e. the developer of StarCoder), though Hugging Face and ServiceNow are companies while BigCode is "an open scientific collaboration working the responsible development and use of large language models for code" supported by ServiceNow and Hugging Face. See ltttps://wwr
pescoddeproject.oom bttoc//aboot/msaaiooooo We provided guidance that the flagship foundation model should be "based on combination the tollowing lactors: greatest resource expenditure, most advanced capabilities, greatest sncietal impact." Coheee and Inflection declined participate in FMTT v1.1.
6

---

Published in Transactions on Machine Learning Research (12/2024)
decentralized across the Internet, the researchers may have missed information Second, the relationship between piece of public information and an indicator may be indirect and oblique, leading to greater subjectivity in scoring. Third, focusing on public information aligns with a developer's current level of transparency but does not provide developers with an opportunity to disclose further information. Finally, and most fundamentally, this search significantly adds to the cost of executing the index.
In FMT1 vl.1, we request transparency reports from each developer that directly address each of the 100 indicators. This change in the information gathering process alters the dynamics for the four aforementioned considerations. First, if we assume developers are strongly incentivized to be their own best advocates and are certainly the most knowledgeable entities about their models, then the information they compile should be complete. Second, by having developers directly clarify information on indicators affirmatively, uncertainties that contributed to more subjective scoring are addressed. Third, by allowing developers to include information that was not-previously public, which is made public through this process, opportunities arise for greater transparency. Finally, by having developers gather information, the cost we bear is reduced.
3.4 Scoring
In FMTI v1.0, once information was identified, two researchers independently scored each of the 1000 (indicator; developer) pairs. The agreement rate was 85.2% (148 disagreements): in the event. disagreement, the researchers discussed and came agreement. These initial scores were sent to developers to permit rebuttal: following two-week rebuttal process, final scores were published in October 2023.
For FMTT v1.1, using the information identified through the developer-submiitee transparency reports, two rescarchers independently scored cach of the 1400 (indicator, developer) pairs. The standard of cach indicator is the same as in FMTI v1.0: see Bommasani et al. (2023a Appendix B) for the per-indicator scoring standard. The agreement rate was 85.3% (206 disagreements): in the event of disagreement, the rescarchers discussed and came to agroement 10
These initial scores were sent to developers to permit rebultal: in contrast to FMTI v1.0, the rebuttal process was more iterative multi-week process that involved email exchanges and video meetings. Through this correspondence, companics clarified existing information and disclosed new information, which is reflected in the final form of the transparency reports we publish. Ultimately, companies validated their transparency reports and approved their release. In doing so, unlike FMTI v1.0, these reports make explicit instances where (i) developers disclose useful information yet (ii) we felt the disclosure was insufficient to award a point.
3.5 Timeline
We summarize the execution of FMTI v1.1 as follows:
Developer solicitation (December 2023 January 2024). We contacted leadership 19 companies developing foundation models, requesting they submit transparency reports.
2. Developer reporting (February 2024). 14 developers designated their flagship foundation model and submitted transparency reports in relation to each of the 100 transparency indicators for their flagship model.
3. Inútial scoring (March 2024). We reviewed the developers' reports, ensuring a consistent horizontal standard across all developers in terms of how each indicator was scored.
4. Developer response (April 2024). We returned the scored reports to developers, who then contested scores specific indicators (potentially including the disclosure of additional information to justify a different score). Following this process, we finalized the transparency reports, which were validated by the developers prior to public relcase.
Howwever this dues beg the question of whether the information being public is truly constitutive of transparency if it not discovered through systematic and high-cffort search. Futuure versions of FMTI may explore having independent scorers separate from the FMTI research team to increase reproducibility.
7

---

Published in Transactions on Machine Learning Research (12/2024)
Foundation Model Transparency Index Scores by Domain, May 2024
May 2024 Foundation Transparency Index
servicenew
StarCoder
85
Al2s
Jurassic-2
75
ALEPH ALPHA
Luminous
IBOM
Granite
64
Microsoft
Phi-2
O0 Meta Llama
2
60
stability Stable Video Diffusion
WRITER
Palmyra-X
56
RESTRAL
Mistral 7B
55
ANTHROPIC
Claude
OpenAl GPT-4
49
Google
Gemini Ultra
Upstream
Model
amazon
Titan Text Express
41
Downstream
ADEPT Fuyu-8B
10
20
30
40
70
80
90
100
Score
Figure 2: Scores by Domain. The overall scores disaggregated into the three domains: upstream, model, and downstream.
4 Results
We analyze this iteration of the Index on its own $4.1 in relation to the first iteration $4.2 and specifically in terms of new disclosures $4.3
4.1 Standalone results of FMTI v1.1
While the average variability among developers. Based on the overall scores (right of Figure 1), 11 of the 14 developers score below 65, showing that there is a significant lack of transparency in the foundation model ecosystem and substantial room for improvement across developers. The mean and modian are 57.93 and 57 respectively, with a standard deviation of 13.98. The highest-scoring developer scores points for of the 100 indicators, while the lowest-scoring developer scores 33. The top-scoring developers (BigCode/Husge Aleph Alpha, and A121 Labs) are more than one standard deviation above the mean, the next near the mean (IBM, Microsoft, Meta, Stability AI, Writer, Anthropic, OpenAI, Mistral, Google), and the 2 lowest-scoring developers are more than one standard deviation below the mean (Amazon, Adept).
the Index has significant room for improvement, there is high
Face Pace/ServiceNow,
Improvement is feasible for each developer. In spite of significant opacity, for 96 of the 100 indicators there exists some developer that scores points, and of these there are 89 where multiple developers score points. The disclosures that developers make satisfy these indicators provide a concrete example of how all developers can be more transparent. developers emulate the most-transparent developer for cach indicator, overall transparency would improve sharply.
Developers disclose significant new information, which contributes to their scores. A developer's total score on the Foundation Model Transparency Index reflects the information that it discloses about its flagship foundation model in relation to each of the 100 indicators in the Index. In this version of the Index, we note where this information is disclosed as part of the process of conducting the Index (i.e. where developers disclosed the information for the first time via their report, or where developers updated their documentation response the Index process). This new information contributes to developer's overall scores: developers average release information related to 16.6 indicators, which improves the average
8

---

Published in Transactions on Machine Learning Research (12/2024)
Foundation Model Transparency Index Scores by Major Dimensions of Transparency, May 2024
Source: May soos Foandation Midid Tannureery sdes
ADEPT
Aizliabs
amazon
Google Go
IBM
Mo
Microsof
M gmm
OpenAl
stobilityal
Then Thot
Stacie Video
Tuyu-B3
J
Luminose
Esprees
Cloudd 5
StarCader
Utrr
Granits
Liama 2
PM-2
Mistral m
GPT-4
Deffusion
Paa Palmyrac
0%
60%
0%
100%
0%
60%
60%
90%
40%
Labor
0%
43%4
14%
14%
100%
99%
47%
1009%
100%
10%
100%
4.7%
50%
Compute
14%
205
100%
0%
15%
100%
1454
100%
71%
57%
59%
14%
43%
20%
51%
Methods
0%
100%
50%
75%
100%
75%
100%
100%
100%
76%
100%
Migor Deverbon of Toengarreeyy
Model Banics
83%
100%
100%
05%
50%
100%
02%
100%
100%
100%
100%
50%
100%
100%
89%
Model Aocess
100%
67%
100%
67%
67%
100%
67%
67%
100%
100%
100%
67%
100%
35%
81%
Capabiitia
100%
Riaea
0%
57%
57%
43%
85%
100%
42%
71%
71%
23%
14%
97%
14%
14%
47%
Mitigations
0%
40%
20%
20%
40%
40%
80%
60%
0%
60%
60%
0%
20%
31%
Ditribution
67%
86%
67%
86%
67%
85%
71%
77%
88%
Usage Poliyy
40%
100%
100%
80%
100%
100%
100%
40%
40%
100%
40%
80%
60%
80%
78%
67%
33%
67%
67%
67%
Impact 79%
29%
0%
99%
0%
50%
16%
14%
15%
Avcrage 38%
73%
78%
41%
85%
86%
55%
67%
62%
65%
62%
48%
58%
67%
Figure 3: Scores by Major Dimensions of Transparency. The fraction of achieved indicators in each of the 13 major dimension of transparency. Major dimension of transparency are large subdomains within the 23 subdomains.
score by 14.2 points. For example, AI21 Labs and Writer newly disclosed the carbon emissions associated with building their flagship foundation models (200-300 iCO2eq and 207 iCO2eq respectively). See $4.3 for further details.
The upstream domain is the most opaque and the region where the most transparent developers distinguish themselves. Breaking the results down by domain (Fgure 2), developers performed best on indicators in the downstream domain, scoring 65% of available points overall in comparison to 61% on the model domain and 46% in the upstream domain. Developers scored worse across upstream indicators: of the 20 indicators where developers score highest, just one indicator (model objectives) is in the upstream domain. High-scoring developers often differentiate themselves in terms of their transparency in the upstream domain; the top scorer overall, BigCode/Hugginn Huugiing Face/ServiceNow, scored all 32 points, whereas the lowest scorer overall, Adept, scored just one point. The standard deviation of scores on the upstream domain (8.8) is more than double that of the other two domains (3.6 each), reflecting the much greater spread across the domain. On the whole, developers are less transparent about the data, labor, and compute used to build their models than how they evaluate or distribute their models. Prior work has emphasized the importance of these particularly opaque domains (Crawford 2021 Gebru et al. 2018 Luccioni & Hernández-Garch 2023
Scores varied across subdomains, with developers scoring best user interface, capabilitics, and model basics. Disaggregating each domain, we consider the 23 subdomains with Figure 3 showing results for 13 major subdomains. Scores varied greatly across subdomains: 86 percentage points separate the average scores on the most transparent and least transparent subdomains. The subdomains with the highest scores are user interface (93%) capabilities (89%), model basics (89%), documentation for downstream deployers (89%), and user data protection (88%). Each of these subdomains in the downstream domain, where developers scored near or above 70% on 6 of the subdomains. These high scores were achieved in part through the release of new information. For instance, AI21 Labs released its first model card for Jurassic-2 during the FMTI v1.1 process, and Aleph Alpha and Stability Al made significant changes to their existing model cards. High scores on these subdomains reflects that disclosure in these arcas is relatively less onerous for developers disclosing documentation for downstream deployers as well as information about capabilities and model basics makes it easier and more appealing for customers to make use of a companies' flagship foundation model, meaning it is in developers' interest to do so.
Data access, impact, and trustworthiness are the least transparent subdomains. The subdomains with the lowest total scores are data access (7%), impact (15%), trustworthiness (29%), and model mitigations
9

---

Published in Transactions on Machine Learning Research (12/2024)
Foundation Model Transparency Total Scores of Open Closed Developers, May 2024
May 2024 Foundation Transparency Index
servicenew
StarCoder
85
Al2s
Jurassic-2
75
ALEPH ALPHA
Luminous
IBOM
Granite
64
Microsoft
Phi-2
62
O0 Meta Llama 2
60
stability Stable Video Diffusion
WRITER
Palmyra-X
RESTRAL
Mistral 7B
55
ANTHROPIC
Claude
OpenAl GPT-4
49
Google
Gemini Ultra
Open
amazon
Titan Text Express
Closed
ADEPT Fuyu-8B
10
20
30
40
70
80
90
100
Score
Figure 4: Overall Scores by Release Strategy. The overall scores for the 6 open developers (Adept, Face//SericeNow, Meta, Microsoft, Mistral, Stability Al) and the 8 closed developers (A121 Labs, Aleph Alpha, Amazon, Anthropic, Google, IBM, OpenAl, Writer).
(31%). Developers score 50% less on 10 of 23 subdomains in the index, including of the largest subdomains-inwe impact (15%), data (34%), and data. labor (50%). The lack of transparency in these subdomains shows that the foundation model ecosystem is still quite opaque there is little information about how people use foundation models, what data is used to build foundation models, and whether foundation models are trustworthy.
Open developers match closed developers on downstream indicators, and exceed them on upstream indicators. Developers adopt different release strategies (Solaiman 2023) for their flagship foundation models: six developers release open foundation models (Kapoor et al. 2024 meaning the model weights are widely available, whereas the other cight employ more closed release strategy Open developers generally outperform closed developers (Figure 4 the median open developer scores 5.5 points higher than the median closed developer. While making the weights of a model openly available correlated with greater overall transparency, does not necessarily imply greater transparency about matters like data, compute, or usage.
The difference in transparency between open and closed developers attributable to the substantial gap restream transparency: within the upstream domain, the median open developer scores additional points indicators in the upstream subdomain over the median closed developer. Within each subdomain, the median open developer scores as many or more points than the median closed developer on all but 5 of the 23 subdomains (i.e., risks, model mitigations, trustworthiness, distribution, usage policy, and model behavior policy). Open developers outscore closed developers by the widest margin on the following subdomains: data labor, data, and model access. Though open developers may struggle to gauge the downstream use of their models (Klyman 2024), which closed developers may be able to directly monitor, the median open developer scores the same number of points the downstream domain the median closed developer.
11 Aleph Alpha shares model weights for its flagship foundation model with customers premises, but this does not mean that the model weights widely available and it considered closed foundation model developer per the definition of Kapoor (224)).
10

---

Published in Transactions on Machine Learning Research (12/2024)
Foundation Model Transparency Index Scores by Developer, October 2023 vs. May 2024
May Model Transparency Index
BigCode/HF/
85
ServiceNow
75
A121 Labs
25
60
Meta
54
Stability Al
51
Anthropic
36
OpenAl
49
47
Google
40
May 2024
41
October 2023
Amazon
20
30
40
80
90
100
Score
Figure 5: Change in Overall Scores. The FMTI v1.0 and v1.1 overall scores for the cight developers assessed in both versions.
Developers that are part of the AT Alliance (Hugging Face, IBM, Meta, ServiceNow, Stability AT), which often advocates for open releases of model weights, outscore developers that are founding members of the Frontier Model Forum (Anthropic, Google, Microsoft, OpenAI) by median of 12 points, with higher average scores across upstream, model, and rewnstream indicators
Still, closed developers outperform open developers in several areas related to policies and enforcement. Closed developers generally share more information about if and how they enforce their policies regarding user and model behavior, outperforming open developers on these subdomains by 2 and points respectively Closed developers also score higher the risks and model mitigations subdomains, several open developers do not describe demonstrate risks associated with their flagship foundation model and closed developers are more likely to describe and demonstrate any risk mitigations that are taken at the model level in transparency between open and closed foundation model developers is a reflection the current state of the ecosystem, not a fundamental reality about the transparency of developers that do or do not make the weights of their foundation models widely available.
[4] The discrepancy
4.2 Comparative results between FMTI v1.1 and v1.0
Transparency increased across the board from v1.0 to v1.1. Foundation model developers significantly improved their scores between October 2023 and May 2024, with the average score rising from 37 to 58 out of 100. Scores improved on every domain, with average upstream scores improving by the greatest margin 7+7.6 points) followed by downstream (+7.2) and model (+6.1). As a result, there is significantly more
1he Al Alliance is coalition of developers, universities, and researchers "who collaborate to advance safe, responsible Al rooted in open innovation." The Frontier Model Forum au industry group whose founding members are Authropic, Google, Microsoft, and OpenAl; the Frontier Model Forum committed to "advancing Al safety research," "identifying best practices," "collaborating
ai/ sectors," and "helpfing) AI meet suciety's 's greatest challenges." See htttp:://heeallle and httpp::/www. wwtp:/wwwwwoiiee.oommeoeee eronttiemmel.oorr..m
18 with other subdomans, the difference
here driven by just
of the 14 developers.
14Open developers, however, outscore closed developers data mitigation indicators. They often do model mitigation indicators becanse they not explicitly that such mitigations were applied the model level.
points
11

---

Published in Transactions on Machine Learning Research (12/2024)
Percentage Point Change in Transparency Index Scores by Major Dimensions of Transparency,
October 2023 vs. May 2024
May 2024 Foundation Transgerancy
BigCode/HF/ ServiceNow
Labs
Amazon
Anthropic
Google
Mets
OpenAl
Stability
Data
+60%
40%
+10%
+40%
-20%
40%
+0%
+0%
Labor
+43%
+14%
-14%
+14%
+29%
+0%
+0%
+86%
Compute
+86%
+0%
+14%
+86%
+0%
+14%
+0%
-14%
Methods
+100%
+50%
+0%
+0%
+0%
+0%
+0%
-25%
Major Dimensions of Transparency
Model Basics
+50%
+0%
+17%
+0%
Model Access
+33%
+33%
+33%
+0%
+33%
+0%
+0%
+0%
Capabilities
Risks
+29%
+43%
+57%
+100%
+14%
+14%
+0%
+0%
Mitigations
+0%
Distribution
+43%
+14%
+29%
+29%
-14%
40%
+14%
+14%
Usage Policy +80%
+60%
+40%
+80%
+40%
+0%
+0%
+20%
Feedback +67%
+33%
+0%
+67%
+33%
+0%
+0%
+33%
Impact
+14%
+0%
+14%
+0%
+14%
+0%
+0%
+0%
Figure 6: Change in Subdomain Scores From FMTI v1.0 to FMTI v1.1. This figure shows the percentage point change in scores for major subdomains for the eight developers that are included in both the October 2023 and May 2024 versions of the Foundation Model Transparency Index.
information publicly available about the upstream resources developers use to build foundation models, the models themselves and how they are evaluated, and their downstream distribution and use.
Transparency increased in nearly all subdomains from v1.0 to v1.1. Developers improved their scores every subdomain with the exception of data access. The largest improvements in subdomain scores were in compute (average increase of -2.4 indicators per developer), data labor (+2.3), and risks (+1.6). This broad improvement demonstrates that the overall trend in recent years toward reduced transparency is more nuanced than commonly understood, though transparency is still lacking with respect to the data used to build foundation models and the impact they have once deployed.
Transparency increased in subdomains that were especially opaque in v1.0. Several of the areas of the index that were least transparent in v1.0 show significant improvement in v1.1, including subdomains such compute, methods, risks, and usage policy. For example, the total score across companies for the compute subdomain rose from 17% in v1.0 to 51% in v1.1. Compute is potentially one of the most intractable areas for disclosure as it relates to the environmental impact of building foundation models and therefore could be associated with additional legal liability for developers and deployers yet we see improvement across compute indicators. This improvement driven a significant degree by new information that companics have disclosod, with companics disclosing new information on compute usage (6 companics disclosod new information), development duration (4), energy usage (4), compute hardware (3), hardware owner (3), and carbon emissions (2). In this way, transparency about compute expenditure has spillover effects for transparency about environmental impact, providing a potential model for translating information disclosure about one area into details about the impact of the foundation model supply chain.
Transparency has improved substantially with respect to companies' policies regarding acceptable use and behavior of their modcls. The total score across companics for the usage policy subdomain rose from to 44% 57%, which was driven by increased transparency related how these policies are enforced. Similarly, the total score across companies for the model behavior policy subdomain rose from 23% to 69%. While
12

---

Published in Transactions on Machine Learning Research (12/2024)
increased transparency in these domains is a relatively light lift-as companies merely need to state whether they have such policies and, if so, describe how they are enforced- this form of transparency is not costless for companies as it highlights potential failure modes for their models (Klyman 2024p. Transparency about how companics restrict the use and behavior of their models can facilitate independent ovaluation and red teaming due to reduced legal uncertainly, promoting research on safely and trustworthiness (Longpre el. al. 2024a).
Data remains key area of opacity. Several arcas of the Index exhibit sustained and systemic opacity: almost all developers remain opaque these matters. Developers display fundamental lack of transparency with respect to data, building on frustrations of data documentation debt (Bandy & Vincent, 2021 Sambasivan et al. 2021). Transparency on the data subdomain rose from 20% to 34% from v1.0 to vl.1, but GeCCodeHHugginng
is the only developer that scores points on indicators relating to data creators, data copyright status, data license status, and personal information data. Only 3 developers (Aleph Alpha, BigCode/ BigCode/Hugging Hugging Face/ScrviceNowy IBM) score points on 6 or more of the 10 data indicators, while 6 developers score 2 or fewer points. These low scores reflect the ongoing crisis in data provenance Longpre et al., 2024b 2023a), wherein companies share no information about the license status of their datasots, preventing downstream developers from ensuring they are complying with such licenses.
While scores on data labor improved from 17% to 50% from v1.0 to vl.1, this was driven in large part by an increase in the number of companies that do not use data labor outside of their own organization (BigCode/Hugge Huggiig Stcc//Sericeeow, Microsoft, Mistral). Considering only the 11 developers who do not disclose that they do not use external data labor, scores on the data labor subdomain fall to 36,, which would make it the sixth lowest scoring subdomain. Only one developer (Stability AT) that discloses its use of external data labor discloses the wages that it pays data laborers, highlighting how lack of transparency may limit accountability for worker exploitation (Gray & Suri 2019b
Transparency in data access, one of the lowest scoring subdomains across developers in v1.0, declined in v1.1 from 20% to 7%. This reflects the significant legal risks that companies face associated with disclosure of the data they use to build foundation models. In particular, these companies may face liability if the data
contains copyrighted, private, or illegal content such as child sexual abuse material (Lee al. 2024 Solove Thorn 2024).
2024
Developers disclose little information about the real-world impact of their foundation models. Developers still lack transparency about the real-world impact of their models, and any steps they take to mitigate negative impacts pre-deployment. Of the major dimensions of transparency in Figure 3) developers score worst on the impact subdomain (as they did in v1.0). Each of the four indicators where no developer scores points (affected market sectors, affected individuals, usage reports, and geographic statistics) are in the impact subdomain. This means that the public has little no information about who uses foundation models, where foundation models are used, and for what purpose. The lack of transparency regarding these matters inhibits effective governance of foundation models, as it is difficult for governments or civil society organizations to pressure companies to responsibly deploy their models if there is no information about the impact of deployment. In many cases this opacily stems from a lack of information sharing between developers, deployers, and customers; developers generally do not know how their foundation model is being used unless a deployer monitors use or receives and shares information about use from its customer.
The 8 developers evaluated in both October 2023 and May 2024 showed marked improvement, 19 points on average. For 3 of these developers we evaluate the same flagship foundation model (Jurassic-2 for AI21 Labs, Llama 2 for Meta, and GPT-4 for OpenAI) as FMTI v1.0 while for the other 5 we evaluate a different flagship foundation model (Titan Text Express for Amazon, Claude 3 for Authropic, StarCoder for BigCode/ BigCode/Hugging Hugging Sace//Seriiceoww, Gemini 1.0 Ultra API for Google, and Stable Video Diffusson for Stability AI). All 8 companies' scores increased, as shown in (Figure 5): A121 Labs (+50), Pace/ServiceNor (+32) and Amazon (+29) made the largest improvements.
Developers that were assessed only in vl.1 performed slightly worse than those assessed in both v1.0 and v1.1. The 6 developers that were assessed only in v1.1 (Adept, Aleph Alpha, IBM, Microsoft, Mistral, Writer) have slightly lower scores than those assessed in both v1.0 and v1.1. Their mean total score
13

---

Published in Transactions on Machine Learning Research (12/2024)
Disclosure of New Information and Scores by Foundation Model Developer, May 2024
Sourow: May 2024 Foundation Transparency Index
BigCode/HF/
85
ServiceNow
A121 Labs
Aleph Alpha
75
IBM
64
Microsoft
62
Meta
60
Stability Al
58
Writer
56
Mistral
55
Anthropic
OpenAl
Google
47
Points without information
Amazon
41
Points from new information
Adept
10
20
30
40
50
60
70
80
90
100
Score
Figure 7: Scores by New Information Status. The overall scores disaggregated based on whether the information was newly disclosed.
is 57.5, which point lower than that of the other developers Developers that were included in FMTT v1.0 had the benefit of having already evaluated their own transparency practices in relation to this initiative (i.c. in FMTI v1.0 they had the opportunity to rebut scores provided by Bommasani et al. (2023a). meaning it may have been relatively easier for them to compile transparency reports. The 6 developers assessed only in FMTI v1.1 include the lowest scoring developer and the two lowest scoring open model developers.
4.3 New information in FMTI v1.1
A key feature of the FMTT v1.1 methodology is that companies were able to disclose new information, meaning information that was not public at the onsct of the FMTI v1.1 process. In some cases, this information is directly made public for the first. time via the FMTI v1.1 transparency reports. In other cases, this information was incorporated into preexisting or new publicly available documentation from companies. For example, as we noted previously, AI21 Labs released the first model card for Jurassic-2 and Stability AI significantly updated the model card for Stable Video Diffusion.
New information constitutes large fraction of the score for several companies. Figure breaks down each developer's overall FMTI v1.1 scored based on which indicators were awarded for new information vs. information that was previously publicly available. For three developers (AI21 Labs, Aleph Alpha, Writer), new information constitutes roughly half the points awarded. In the case of Aleph Alpha, several new disclosures are made about data labor: all laborers are employed by Aleph Alpha, work in Germany, and are afforded labor protections as stipulated by German law. For Writer, new information provided compute: models trained 1024 NVIDIA A100 80GB GPUs for 74 days (910k GPU hours) the Writer cluster, amounting to 8.2 x 20 FLOPs in compute, 812 MWh in energy, and 207tCO2eq in emissions.
All developers disclose some new information. In the case of OpenAT, the sole change to ils disclosures from FMTT v1.0 is in relation to detecting machine-generated content. Specifically, OpenAl clarify that it "originally relcased a classifier that was taken down duc to lack of accuracy. Our commitments are around audio / visual content for now, so this implies lack of ability to detect GPT-4 generated content".
18This is despite the fact that of the developers that assessed only FMTI open developers, which score higher average.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Foundation Model Developers' Disclosure of New Information by Major Dimension of Transparency, May 2024
Source: 2024 Foundation Model
Labor
Usage Policy
Compute
Major Dimensions Transperency
Data
Impact
17
Methods
16
Distribution
14
Risks
Mitigations
10
Capabilities
Feedback
Model Basics
Model Access
12
14
16
18
20
22
24
26
28
Number of New Pieces of Information Disclosed
Figure Aggregate New Information by Major Dimensions of Transparency. The number of pieces of new information, aggregated across all developers, for each of the 13 major domains of transparency. Note: major dimensions of transparency each have different numbers of indicators the largest is data (10 indicators), followed by compute (7), distribution (7), impact (7), labor (7), risks (7), model basics (6), capabilities (5), mitigations (5), usage policy (5), feedback (3), methods (3), and model access (3).
other cases, new information is disclosed to clarify existing information that was difficult to interpret based on publicly available documentation, For example, Amazon clarified how its model behavior policy and usage policy interoperate: "In the Bedrock user guide, stated that AWS is committed to the responsible use of AI, and request information about customers' use of Amazon Bedrock and compliance with our lerms of service or third-party provider's AUP. In the event that customer is unwilling or unable to comply with these terms or policies, AWS may suspend access to Amazon Bodrock."
automated abuse detection mechanisms to identify potential violations,
may
New information drives much of the transparency for areas of large improvement. Figure totals the amount of new information across all developers for each major dimension of transparency. The most information is in the area of labor, which noted previously is largely due to multiple companies clarifying they do not use data labor in building their flagship foundation models. Beyond this, other areas of large improvement from FMTT v1.0 to FMTT v1.1 are precisely those with large amounts of new information. Namely, more than 10% of the 233 pieces of new information are in the areas of compute and usage policy.
F
Companies disclose new information for which they do not score points. For example, several companies provide additional details about their data. labor practices like general information about instructions annotators and assurances that wages exceed local minimum wage. And on the indicator about data creators, which is awarded to only of the 14 developers (BigCode/ BeeSoo//HHgging Hugging face//SericeeNow) AI21 Labs discloses that "Most of the internet-connected population is from industrialized countries, wealthy, younger, and male, and is predominantly based in the United States." While these disclosures are insufficient for the stated standards for the associated indicators, we nonetheless emphasize that it may still be valuable for developers to make such information public. In a few cases, companies provide public justifications for why they do not disclose certain information. Most notably, Aleph Alpha states that "In line with the German copyright act ('Urheberrelltseeett')) data used for training is to be deleted after use and therefore can not be made available distributed to external parties" in relation to the indicators about data access.
15

---

Published in Transactions on Machine Learning Research (12/2024)
5 Discussion
Having characterized how transparency has changed from October 2023 April 2024, we discuss how we reason about these changes (5.1), what we recommend going forward $5.2 and ways in which the Foundation Model Transparency Index could evolve in subsequent iterations $5.3
5.1 Interpretation of findings
There is significant room for improvement in the transparency foundation model developers. Developers on average score just 58 out of 100, with major gaps in multiple subdomains for most developers. Developers' transparency reports are incomplete, lacking disclosures on many important matters.
Nevertheless, our findings provide significant reasons for optimism about the prospects for improved trans- parency. Foundation model developers shared a significant amount of new information about how they build, evaluate, and deploy their models via FMTI v1.1. Some of the areas of the index that were least. transparent in v1.0 show significant improvement in v1.1, including subdomains such as compute, methods, risks, and usage policy. Several companics have become much more transparent, with some relcasing model cards or other documentation for their flagship foundation models for the first time. By engaging directly with foundation model developers, have shown that many firms are willing to disclose more information about some of their most powerful technologies.
Still, the overall state of transparency in the foundation model ecosystem remains poor. Developers are opaque about the data, labor, and compute used to build their models, they often do not release reproducible evaluations of risks or mitigations, and they do not share information about the impact their models are having on users or market sectors. Transparency in the foundation model ecosystem has been declining for several years, and this trend is unlikely reverse in the near term.
Where developers do disclose information, they sometimes disclose information for only the least onerous indicators within a subdomain. For example, in the risks subdomain, most developers describe risks (12 of 14 developers) and many demonstrate risks (8 of 14). However, fewer developers evaluate unintentional harm (5 of 14) or intentional harm (4 of 14). We see the same trends for model mitigations and data labor, where developers disclose information regarding less intensive indicators but no others.
Why do developers not disclose specific information? Based on experiences engaging with these developers, several reasons information in relation to a specific indicator. In some cases, developers cited countervailing interests that they found to be in tension with transparency (e.g. legal exposure from data transparency, trade secrets for model details). In these cases, we will reflect on how the FMTI can first yield Pareto improvements, meaning improvements transparency at no cost to these other interests by having well-scoped definitions for indicators, before broaching areas of clearer trade-offs. Critically, we emphasize that the aforementioned cases generally indicate (i) the developer internally has the desired information but (ii) does not want to cases disclose this information externally to the public. In contrast, in other cascs we found there where the developer simply did not have the desired information and producing it would be costly. In these cases, we understood that internal cross-team coordination, especially for large companies, was key barrier: consultation across lawyers, engineers, product managers, and executives before authorizing release imposod significant practical barriers to information disclosure. Or, a few cases, developers disclosed the barrier was the lack of information from an external party (e.g. energy details from computer providers, usage information from downstream users). Overall, these explanations illustrate the concrete factors, our knowledge, that limit overall transparency of foundation model developers.
specifically put forth to justify why the developer would not disclose
5.2 Recommendations
We present recommendations aimed at different stakcholders based on the findings of FMTI v1.1 as well changes in the world over the past six months, building on a more extensive set of recommendations made in Bommasani et al. 2023a (8).
16

---

Published in Transactions on Machine Learning Research (12/2024)
5.2.1 Foundation model developers
As part of FMTI vl.1, we publish transparency reports that consolidate information disclosures from developers and that, we release subject their validation. light of voluntary codes of conduct promulgated by the White House and the G7 that include commitments for foundation model developers release transparency reports, envisage the reports release as part of FMTI v1.1 rudimentary forms of such transparency reports (Bonmnasani el al. 2024 In addition, many of the areas where transparency is lacking in foundation models mirror those in previous waves of digital technology as well as in other industries such as finance and healthcare. For example, shortcomings in downstream transparency by foundation model developers mirror issues faced by social media companies the last decade (Aspen Institute 2021). For such indicators, investing the expertise of trust and safety professionals could help inform how developers develop internal and external policies, including those related transparency. As one example, social media platforms like Facebook, YouTube, and TikTok detail usage policy violations and government. requests for user data in their transparency reports (Narayanan & Kapoor 2023). Similarly, social media platforms often have well-cstablished processes for communicating with users about account restrictions and handling user appeals (Trust and Safety Professional Association 2023 Poundation model developers can adopt these policies and
processes to improve their downstream transparency (Klyman 2024
5.2.2 Customers of foundation models
Purchasers (e.g. downstream developers, enterprise users of chatbots, or government entities) can exert negotiating power in procurement to increase transparency. Notably, some foundation model developers stated that their participation was driven by requests from customers to understand the transparency of their products. Others mentioned that their practices related to transparency with their customers are much better compared to the data they can share publicly an example of the influence customers can have on business practices. The Foundation Model Transparency Index provides a structured way for customers advance transparency from foundation model developers-boote in terms of the information developers share with their clients and with the broader public.
In addition, governments can play a dual role as customers of foundation models (Quay-ee la Vallee et al., 2024). As influential (and lucrative) procurers of technology, this can help them play a standard-set role. For example, the U.S. government is one of the largest purchasers of various goods and services, which allows to set the standards for how these goods and services are sold, shaping business practices across industries [Vinsel 2019), including around transparency. While requirements on transparency by governments may lead to legal concerns around government overreach, such concerns in the US related to the first amendment implications of compelled speech (Bankston & Hodges 2024h, standard-setting via procurement circumvents these concerns by relying solcly on the voluntary commitment to these standards by model developers who want to enter a contract.
5.2.3 Transparency advocates
The transparency reports we release enable transparency advocates in academic and civil society organizations to better understand developer practices. While for the purposes of the Index scores set a threshold for constitutes sufficient disclosure to award point, the underlying disclosures are considerably richer. We encourage researchers and journalists to investigate this information, which includes considerable variation across companics that we do not explore in this paper.
5.2.4 Policymakers
Policymakers can use our results to identify areas of pervasive opacity-including areas with sustained opacity (across FMTI v1.0 and v1.1) as well as areas with systematic opacity (across the developers we score). This can also highlight perverse business incentives that might lead to such a lack of transparency, and in turn, can inform regulation that addresses them. For example, sharing information about the data used to train foundation models might open up companics to liability concerns, such as due the legal uncortainty around copyright violations. Regulation intended address such perverse incentives, such as mandatory disclosure of training data, could help address these impediments to transparency.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Various policy efforts in the last two years have focused on addressing transparency in the foundation model ecosystem, including Canada's Code of Conduct on the Responsible Development and Management of Advanced Generative Al Systems, the EU AI Act, and the US Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Still, [Bommasani et al. (2024) find that these efforts can lack specificity. Our ilerative process with model developers provides an insight. into how such specificity might arise in practice government bodics might consider developing the institutional capacity to coordinate discussions with companies to ensure that they adhere the standard required by certain policy.
5.3 Next steps
We plan to conduct future versions of the Foundation Model Transparency Index on a regular basis. As stated by [Bomassani al.] (2023c), as part of future iteralions we will make changes the Foundation Model Transparency Index to reflect changes in the foundation model ecosystem and the organizations that build and deploy foundation models.
For example, moving forward, the Foundation Model Transparency Index may change the indicators or their thresholds. For the purposes of clear comparisons, the indicators and scoring thresholds are the same across FMTI v1.0 and v1.1. Due to ambiguity (as evidenced by developers misinterpreting indicators), one possible change could include splitting existing indicators into multiple different ones in order to more clearly delineate the information required in each indicator. Due to saturation (as evidenced by most developers satisfying many indicators), possible changes could include increasing the scoring thresholds for existing indicators well as introducing new indicators about information that is not currently assessed by the Index. And due to changes external to FMTI in the foundation model ecosystem, such as growing interest in the encoding of human values (Scherrer et al. 2024) and the implementation of reward models for aligning foundation models (Lambert et al.] 2024), , possible changes could include adding indicators that specifically reflect. transparency the values encoded in models as well as information about how reward models are trained and operationalized. Most fundamentally, while the initial 100 indicators were decided upon by [ommasann et al. (0033a) with guidance from others in the community, more open-ended process for communit--driver indicator proposals may be implemented.
6 Limitations
In general, transparency as a construct and indices as an approach have well-known limitations (Birchall
2014
Valdovinos
2018 Alloa & Thomä 2018 Alloa, 2018 Sagar & Najam 1998 Boldin 1999
Santeramo
2017
Greco et al. 2019 Schlossarek et al. 2019). There are also number of well-known limitations of
Iransparency in AI specifically (Ananny & Crawford 2018b Phang et al. 2022 Hartzog 2023). These
limitations are discussed at length by Bommaasni et al. (2023a) and they apply equally to FMTI v1.1.
In $2.2, Bommasani et al. (0023a) write: "Transparency is far from sufficient on its own and it may not always bring about the desired change (Corbett. Denton 2023 Salient critiques of transparency include:
Transparency does not equate to responsibility. Without broad based grassroots movements to exert public pressure or concerted government scrutiny, organizations often do not change bad practices Boydd 2016
Ananny & Crawford 2018a).
Transparency-washing provides the illusion of progress. Some organizations may misappropriate transparency as means for subverting further scrutiny. For instance, major technology companies that vocally support transparency have been accused of transparency-uashhng whereby "a focus on transparency acts as an obfuscation and redirection from more substantive and fundamental questions about the concentration of power, substantial policies and actions technology behemoths' (Zalnieriute] 2021).
Transparency can be gamified. Digital platforms have been accused of performative transparency, offering less insightful information in the place of useful and actionable visibility (Ghosh & Faxon
18

---

Published in Transactions on Machine Learning Research (12/2024)
2023
Mittelstadt 2019). As with other metries, improving transparency can be turned into game, the object of which is not necessarily to share valuable information T6
Transparency can inhibit privacy and promote surveillance. Transparency is not apolitical concept and is often instrumentalized to increase surveillance and diminish privacy (Han 2015 Mohamed eet al. 2020; Birchall 2021). For foundation models, this critique underscores a potential tension between adequate transparency with respect to the data used to build foundation models and robust. data privacy.
Transparency may compromise competitive advantage or intellectual property rights. Protections of competitive advantage plays a central role in providing companies to the incentives to innovate, thereby yielding competition in the marketplace that benefits consumers. Consequently, work in cconomics and management studies have studied the interplay and potential trado-off between
competitive advantage and transparency (Bloomfield & O'Hara 1999 Granados & Gupta
2013 Liu
et al. 2023 especially in the discourse corporate social responsibility [[w et al. 2020 Yu et al. 2018
Transparcncy is not panacca. In isolation, more information about foundation models will not necessarily produce more just or equilable digital world. But if transpareney is implemented through engagement with third-party experts, independent auditors, and communities who are directly affected by digital technologies, it can help ensure that foundation models benefit socicty."
In $9.2, Bonmmasani et al.] (2023a) address how several of these limitations of transparency and indices apply in the context of the Foundation Model Transparency Index. These limitations include equating transparency with responsibility, transparency washing, gaming the index, binary scoring, focusing on language models, and focusing on companies headquartered in the US. Beyond these limitations, we specifically discuss additional considerations that arose in this version.
Considering only publicly available information. At present, the Foundation Model Transparency Index exclusively considers public disclosure of information: such disclosures provide transparency to all stakeholders and are verifiable by the researchers conducting the Index. However, intermediary forms of information disclosure exist between no disclosure to entities beyond the developer and disclosure to everyone. The EU AI Act. identifies core lypes of intermediary information disclosure: disclosure to the government (see Annex 1X A) and disclosure to clients downstream in the foundation model supply chain (see Annex IX B). Kolt al. (024)) provide initial guidance stratified disclosures approaches for cybersccurity and biosecurity. Assessing companies" release of information via such intermediary forms of disclosure could encourage such disclosures, which can be beneficial even in the absence of public disclosures.
Requiring that disclosure be explicit. In order for a developer to score points on a given indicator, it must make an explicit disclosure related to that indicator. For example, scoring points on the hardware owner indicator requires that a developer explicitly state which organization(s) owns the primary hardware used in building the model; even if the developer considers this to be obvious and not worth stating in its documentation, it must disclose the owner explicitly in its transparency report to score the point. We use this standard to promote reprodueibility and remove ambiguity: rather than making assumptions based on what developers imply about their flagship foundation models, explicit disclosures ensure that anyone could re-score the developer in the same way.
Company selection of flagship foundation models. As part of a change to our methodology, for FMTI v1.1 each company selected its flagship foundation model to be assessod. While we provided guidance companies regarding how to do so we stated it should be based on a combination of resources expended, capabilities, and societal impact we do not have sufficient insight into company's operations (or, for example, what model sits behind an API) to validate that the model developer designates as its flagship model is in fact its most significant model at present. This ficxibility introduces risks, as it is possible that a developer might chose the most transparent of its foundation ion models in order to increase its score. This limitation stems in large part from the fact that we assess a developer's transparency based on a single flagship foundation model.
According to Goodhart's Law, "when measure becomes target, it ceases to good measure" Goodhart 1984
19

---

Published in Transactions on Machine Learning Research (12/2024)
Companies submitted transparency reports. In this iteration of the Index, companies submit trans- parency reports to disclose key information about their foundation models. This comes with several limitations. Where a developer does not disclose information related to a particular indicator or does not affirmatively and explicitly disclose that it satisfies the indicator, we take this at face value and score that indicator as a zero. In FMTT v1.0, however, there were multiple cases that identified information in companies' documentation that they themselves did not believe satisfied a particular indicator. The methodology used in FMTI v1.1 prevents this from happening, instead assuming that companies know their own documentation best.
No distinction between B2B and B2C companies. The developers that assess include companies with variety of different business models, including those that primary develop models for enterprise customers (e.g. Aleph Alpha and Amazon) and those with a large number of individuals as customers (e.g. Authropic and OpenAl). These differences in business models result in different approaches to transparency. B2B companies prioritize disclosing information directly enterprise customers, who may be less concerned about public facing transparency. We use the same set of 100 transparency indicators for cach company, which limits our ability to capture the nuances in how companies conceive of transparency and its value for their customers. A number of companies requested that assess certain indicators "Not Applicable" in light of their business model, but our methodology of binary scoring prevented us from contemplating this option.
Low bar for awarding points. [Bommasani et al. (023a indicate "We were generally quite generous in the scoring process. When we determined that a developer scored some version of half-point, we usually rounded up. Since we assess transparency, we award developers points they explicitly disclose that they do not share information about particular indicator. We also read developers' documents with deference where possible, meaning that often awarded points where there are grey areas. This means that developers' scores may actually be higher than their documentation warrants in certain cases as we had a low bar for awarding points on many indicators." In future iterations of the index we may raise the threshold for scoring points (c.g. round down scores on indicators that might otherwise be half-points) as doing so would require a greater degree of transparency and potentially foster a more consistent grading slandard.
7 Conclusion
The societal impact of foundation models is escalating, attracting the allention of firms, media, academia, government, and the public. The Foundation Model Transparency Index continues to find that transparency ought to be improved in this nascent ecosystem, with some positive developments since October 2023. By dissecting what developers do and do not publicly disclose, and how this has changed, the Index allows different stakeholders (e.g. developers, customers, investors, policymakers) to make more clear-eyed decisions. And, in turn, by establishing the practice of transparency reporting for foundation models, the Index surfaces a new resource that downstream developers, rescarchers, and journalists should capitalize on to build collective understanding. Moving forward, we hope that headway on transparency will demonstrably translate to better societal outcomes like greater accountability, improved science, increased innovation, and better policy.
We thank the FMTT Advisory Board (Arvind Narayanan, Daniel E. Ho, Danielle Allen, Daron Acemoglu, Rumman Chowdhury) for their feedback and guidance. We thank Zak Rogoff for discussions and feedback related to this work. We especially thank Loredana Fattorini for her extensive work on the visuals for this project, as well as Shana Lynch for her work in publicizing this effort.
Acknow/edgeeeetts.
Foundation Model Developers. We thank the following individuals at their respective organizations for their engagement with our effort: We emphasize that this acknowledgement should not be understood as an endorsement of any kind by these individuals, but simply that they were involved in our engagement with their organizations.
Adept - David Luan, Kelsey Szot, Maxwell Nye, Augustus Odena
Aleph Alpha- - Jonas Andrulis, Joshua Kraft, Yasser Jadidi, Samuel Weinbach
AT21 Labs Yoav Shoham, Ori Goshen, Shanen J. Boettcher
20

---

Published in Transactions on Machine Learning Research (12/2024)
Amazon Peter Hallinan, Sara Liu
Anthropic Jack Clark, Ashley Zlatinov, Frances Pyc
BigCode/ HigCode/Huuging Face/ServiceNow - Clement Delangue, Yacine Jernite, Sean Hughes, Leandro Von Werra, Harm de Vries, Irene Solaiman, Carlos Muñoz Ferrandis
Google James Manyika, Alexandra Belias, Kathy Meier-Hellstern, Reena Jana, Christina Butter- field, Will Hawkins, Dawn Bloxwich, Eve Novakovic, Victoria Langston, Demetra Brady
IBM -
Kush Varshney, Maryam Ashoori, Steven Sawyer
Meta Joelle Pineau, Melanie Kambadur, Esteban Arcaute, Joe Spisak, Manohar Paluri, Ragavan
Srinivasan
Microsoft
Eric Horvitz, Sebastien Bubeck, Suriya Gunasekar
Mistral -
Arthur Mensch, William El Sayed, Nicolas Schuhl
OpenAI - Miles Brundage, Lama Ahmad
Stability - Ben Brooks, Carlos Riquelme, Paulo Rocha
Writer May Habib, Apara Sivaraman, Waseem AIShikh, Rowan Reynolds
Conflict of Interest. Given the nature this work (e.g. potential to significantly impact particular companics and shape public opinion), we proactively bring attention to any potential conflicts of interest, deliberately taking a more expansive view of conflict of interest to be especially forthcoming.
Betty Xiong not, and has not, been affiliated with any of the companics evaluated in this effort or any other private sector entities.
Kevin Klyman is not, and has not, been affiliated with any of the companies evaluated in this effort.
Nestor Maslej is not, and has not, been affiliated with any of the companies evaluated in this effort or any other private sector entities.
Percy Liang 2023), and co-founder of Together AI (July 2022 present). He is not involved (May 2018 May involved in any of the companies evaluated in this effort.
post-doc at Google (September 2011-August 2012), consultant at Microsoft. 8018-May
Rishi Bommasani is not, and has not, been affiliated with any of the companies evaluated in this effort.
Sayash Kapoor worked at Meta until December 2020. He has not since worked for the company.
Shayne Longpre worked at Google in 2022, and on BigCode-affliated projects 2023.
References
Emmanuel Alloa. Transporeccy: Magic Concept of Modernity, pp. 21 55. Springer International Publishing, Cham, 2018. ISBN 978-3-319-77161-.,
doi: 1111077978---311-7777 3. URL https://aoi.org//1
1007//978-3-11--7766111
Emnanuel Alloa and Dieter Thomä. Transparency: Thinking Through an Opaque Concept, pp. 1-13. 06 2018. ISBN 978-3-319-77160-11 doi: 10.10077998--3311--76666 1.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinct, Danicl Hesslow, Julien Launay, Quentin Malartic, Danicle Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models. arxiv preprint ar Xiv:22311.110 1.16677. 2023.
21

---

Published in Transactions on Machine Learning Research (12/2024)
Mike Ananny and Kate Crawford. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accouniability. New Media 8 Sociely, 20(3):973-996 2018a. doi: 10.1177/ 1461444816676645. URL http:://doooooggg/// 01t00///000011111144444666666660
Mike Ananny and Kate Crawford. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountabilitt. New Media & Society, 20(3):973 989, 2018b. doi: 10.1177/
1461444816676645. URL
Aspen Institute.
Commission on information disorder final report. http:://ww.aapennssiiiinn org/w httpg//caaaaaaao///aa//////////p cot.:/tttuttttommmiiiiioooooooet htw.::ooooooooooooaiiiiioo------ Informatioon-iisorder.
Final pdf 11 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprinl ar Xrx:22300. 16609, 2023.
Jack Bandy and Nicholas Vincent. Addressing "documentation debt" in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprinl arXiv:2105.052,, 05241, 2021.
Kevin Bankston and Jennifer Hodges. Openness and transparency in ai provide signif-
icant benefits for society, 2024. URL https:///ott.rg//pppoooo wwt.://oo//oooeeetteetee /otpe///pploooo/////////////////
Civil Sevil-Society-Leteer-o on-eerreen-ppeneess for NTIA- Maac 25-2044. pdf.
March-
Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating
system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6: 587-604, 2018, doi: 20.1162/tacl_a_000000 00041. URL http:://bclartboolog..ooog/
Stella Biderman, Hailey Schoclkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohamnad Aflah Khan, Shivanshu Purohil, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397 2430. PMLR, 2023.
Clare Birchall. Radical transparency? Cultural Studies and Critical Methodologies, 14(1):77-8) 88, 2014. doi: 10.1177/15327086113117
URL http:::///o...rrr///0///////////
Clare Birchall. Radical secrecy: The ends of transparency in datafied America, volume 60. U of Minnesota Press, 2021.
Sid Black, Stella Rose Biderman, Eric Hallahan, Quentin G. Authony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, M. Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, J. Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. arXiu, 2022.
Robert Bloomfield and Maureen O'Hara. Market transparency: Who wins and who loses? The Review of Financial Studies, 12(1):5 35, 1999. ISSN 08939454, 14657368. URL http::/www. http://www..trrrroom.o jstor org/ stable/ 2645985
Michael D Boldin. A critique of the traditional composite index methodology. Journal of economic and social meosurement, 25(3-4):119 1199140,,11111111111 1999.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolisson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annic Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby
22

---

Published in Transactions on Machine Learning Research (12/2024)
Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitrion, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint ar Xiv::21008 07258, 2021.
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index. ArXiv, abs/2310.12990 2023a. URL
https://apis swwanticacholaaattt....eaccc www..org///ouuuu ID:
: 264306385
Rishi Bommasani, Dilara Soylu, Thomas Liao, Kathleen A. Creel, and Percy Liang. Ecosystem graphs: The social footprint of foundation models. ArXiv, abs Abs/2303.11772, 2023b, URL .smaa//ccaa..maaaaicccccclllrrrr
org/Corpus 257771875
Rishi Bommasani, Daniel Zhang, Tony Lee, and Percy Liang. Improving transparency in language models: A holistic evaluation. Foundation Model Issue Brief Series, 2023c. URL https://hai. wawaaiistaafor..o. edu/
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Betty Xiong, Sayash Kapoor, Nestor Maslej, Arvind Narayanan, and Percy Liang. Foundation model transparency reports. ArXiu, abs/2402.16288 16268,1666,, 2024. URL
sttps://api..smaatiicces http:://pp...eenttccccocoarrroop 267938727
Danah Boyd. Algorithmic accountability and transparency. Open Transcripts, Nov 2016. URL http:// opentranscripts
saccoouttaailityyyyraaaaaaaa sented by danah boyd in Algorithmic Accountability and Transparency in the Digital Economy.
Pre-
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramer. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2280-2292. 2022.
Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, et al. Toward trustworthy ai development: mechanisms for supporting verifiable claims. arXiv preprint arXim:2004.073 2020.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency, pp. 77 91, 2018.
Rosario Cammarota, Matthias Schunter, Anand Rajan, Fabian Boemer, Ágnes Kiss, Amos Treiber, Christian Weinert, Thomas Schneider, Emnanuel Stapf, Ahmad-Reza Sadeghi, et al. Trustworthy ai inference systems: An industry research view. arXiv preprint arXiv:2008.0444 2020.
Eric Corbett and Emily Denton. Interrogating the t in facet. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 1124,444, 2023.
Kate Crawford. The atlas of A1: Power, politics, and the planetary costs of artificial intelligence. Yale University Press, 2021.
Dee :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Ti, Guowei
23

---

Published in Transactions on Machine Learning Research (12/2024)
Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuechong Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang. Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint or Xix:2401.02000 2024.
EU. Official journal of the european union 2016. Official Journal of the European Union, L 119/1, Apr 2016. URL http::///err-eee.eerrrreeeeeeerp 8A82016R0679
lex.
L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thile, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. arXic preprint arXiv:1803.090 2018
Ritwick Ghosh and Hilary Oliva Faxon. Smart corruption: Satirical strategies for gaming accountability. Big Data & Society, 10111111339,,,23111111,,, 2023. doi: //doi wwreiiorr//1..0 wtt../////1...///055.111 91001777//0559577111111999
URL https
Charles AE Goodhart. Problems of monetary management: the uk experience. In Monelary theory and practice, pp. 91-121. Springer, 1984.
Nelson Granados and Alok Gupta. Transparency strategy: Competing with information in a digital world. MIS Quarterly, (7()):637-641, 2013. ISSN 02767783. URL httpp:/www.jjstt wtt./stable/43825222222
Mary L Gray and Siddharth Suri. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books, 2019a.
M.L. Gray and S. Suri. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Milllin Harcourt, 2019b. ISBN 978-1-328-56624-9. URL https::/books. www.booooo.oooll...oo..o.ooooo.. id=u10-uQEACANJ
Salvatore Greco, Alessio Ishizaka, Menelaos Tasiou, and Gianpiero Torrisi. On the methodological framework of composite indices: A review of the issues of weighting, aggregation, and robustness. Social Indicators Research, 141:1-34, 01 2019. doi: 10.1007//112055007--
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magmusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannanch Hajishirzi. Olmo: Accelerating the science of language models. arXiv preprint ar Xix:24022 00838, 2024.
Byung- Han. The transparency sociely. Stanford University Press, 2015.
24

---

Published in Transactions on Machine Learning Research (12/2024)
Karen Hao and Deepa Scetharaman.
Cleaning up chatgpt takes
heavy toll
human
workers. The Wall Street Journal, July 2023. URL http:////wwwwwww.c///rrrereeeeee
chattpt-openate heterr/ooeaaannttenteeeee aaweeeett-abusi-eeeeeeee Photographs by Natalia Jidovanu.
o human 9f
Woodrow Hartzog. Oversight of a.i.: Legislating on artificial intelligence. Prepared Testimony and Statement for the Record before the U.S. Senate Committee on the Judiciary, Subcommittee on Privacy, Technology, and the Law, Sep 2023. URL https:///wxz http:///www..iiiiaary...... judiciaryyy.... senate. www.te.eoo/iiieeeeee testimony_ hartzog..p.. pdf
Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, Rumman Chowdhury, Alex Engler, Peter Henderson, Yacine Jermite, Soth Lazar, Stefano Maffulli, Alondra Nelson, Joelle Pincau, Aviya Skowron, Dawn Song, Victor Storchan, Daniel Zhang, Daniel E. Ho, Percy Liang, and Arvind Narayanan. On the societal impact of open foundation models, 2024.
Stephan Klasen. Human development indices and indicators: A critical evaluation. 2018
Kevin Klyman. Acceptable use policies for foundation models: Considerations for policymakers and developers. Stanford Center for Research on Foundation Models, April 2024. URL.
edu/
http:///////uu/..//// html
Lauren Kogen. From statistics to stories: Indices and indicators as communication tools for social change. The International Journal of Press Ppoess/solfiies,, 010):1194011122210444444 2022. doi: 10.117//040111222001/4 URL https://doi.orr org/10 1177/1940111222219922222
Noam Kolt, Markus Anderljung, Joslyn Barnhart, Asher Brass, Kevin M. Esvelt, Gillian K. Hadfield, Lennart Heim, Mikel Rodríguez, Jonas B. Sandbrink, and Thomas Woodside. Responsible reporting for frontier ai development. 2024. URL https: http://ppp..eemaaiiccccccccco/// wtt.://oo......oooooooooooe//2/a 268877588
Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. Language generation models can cause harm: So what can we do about it? an actionable survey. arXiv preprint arXiv:2210.07700 2022.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 1XX::19100009000 2019.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, al. Rewardbench: Evaluating reward models for language modeling. Xiv preprind or arXiv:2403. 13787, 2024.
Katherine Lee, A. Feder Cooper, and James Grimmelmann. Talkin' 'bout ai generation: Copyright and the generative-ai supply chain, 2024.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michíhiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxin Yao, Juc WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas leard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. TSSN 2835-8856. URL Featured Cortification, Export Certification.
https:://pperreview.. httpap/prweoo.eeee/oooooeeetaaaa
Yeyi Liu, Martin Heinberg, Xuan Huang, and Andreas B. Eisingerich. Building a competitive advantage based on transparency: When and why does transparency matter for corporate social responsibility? Business Horizons, 66(4): 2023. ISSN 0007-6813. doi: https: https://ddo.orrg/10011///// www.://ww........oooooooo.
URL
25

---

Published in Transactions on Machine Learning Research (12/2024)
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marmu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. arXiv preprint orXic:2310.16787, 2023a.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. pretrainer's guide to training data: Mcasuring the effects of data age; domain coverage, quality, toxicity. arXiv preprint or arx::23005.131 13169, 2023b.
Shayne Longpre, Sayash Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili- Hamelin, Yangsibo Huang, Aviya Skowron, Zheng-Xing Xin Yong, Suhas Kotha, et al. A safe harbor for ai evaluation and red teaming. arXic preprint arXiv:2403. 04893, 2024a.
Shayne Longpre, Robert Mahari, Naana Obeng-Marmy, William Brannon, Tobin South, Katy Gero, Sandy Pentland, and Jad Kabbara. Data authenticity, consent, & provenance for ai are all broken: what will it take to fix them?, 2024b.
Alexandra Sasha Luccioni and Alex Harcéndez-Garcia. Counting carbon: A survey of factors influencing the emissions of machine learning. ArXiv, hts/2302.084776 2023.
lan Magmusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannanch Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: benchmark for evaluating language model fit,, 2023.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, Anton Belyi, Haotian Zhang, Karanject Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint Xiv: 09611, 2024.
Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. Aretic-embed: Scalable, efficient, and accurate text embedding models. arXiv preprint arXiv:2405. 05374, 2024.
Brent Mittelstadt. Principles alone cannot guarantee ethical ai. Nature Machine Intelligence, 1(11):501-507, November 2019. ISSN 2522-5839. doi:
URL
842256-019-0114--
Shakir Mohamed, Marie-Therese Png, and William Isaac. Decolonial ai: Decolonial theory as sociotechnical forcsight in artificial intelligence. Philosophy & Technology, (3(44)659-684 December 2020. ISSN 2210-5441.
doi: 10. 111007//13337-000-0000--- URL htttps://doii fttt:::/ddoi.or//000 http://.........../////000000000
Arvind
Narayanan and Sayash Kapoor.
Generative ai companies must pub-
lish
transparency reports, 2023.
htpp:://aighhtollubbaaoog//// URL
generative--ii-copana -astaaasaaappiiii------aaaaraaee
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXie preprint arXiv:2104 10950. 2021.
Jason Phang, Herbie Bradley, Leo Gao, Louis Castricato, and Stella Biderman. Eleutherai: Going beyond 'open science" to "science in the open", 2022.
Hannah Quay-de la Vallee, Ridhi Shetty, and Elizabeth Laird The Federal Government's Power of the Purse: Enacting Procurement Policies and Practices to Support Responsible AT Use. Cen- ter for Democracy and Technology. 2024. URL https:://ccl httg:mmmpoooatatt//p/oo////////
26

---

Published in Transactions on Machine Learning Research (12/2024)
Inioluwa. Deborah Raji and Joy Buolamwini. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAT/ACM Conference on AI, Ethics, and Society, AIES '19, pp. 429 435, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450363242. doi:
033006618.3314244 URL https::/dbi..rr//1
1145/3306618 3314244
Ambuj D Sagar and Adil Najam. The human development index: critical review. Ecological economics, 25 (3):249-266) 1998.
SambaNova. Samba-1: 1 trillion parameter composition of experts, 2024. URL https://sambanova. ai/ products/ssmba-1
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "Everyone wants do the model work, not the data work": Data cascades in high-stakes AI. In CHI, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380966. doi: 10.1145/3411744338
URL http:s::/ddii.rr//////// http://.1..115//41111 1144//41111144 3445518
Fabio Gaetano Santeramo. Methodological challenges in building composite indexes: Linking theory to practice. In Emerging trends in the development and application of composite indicntors, pp. 127-139. IGT Global, 2017.
Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. Evaluating the moral beliefs encoded in llms. Advances in Newral Information Processing Systems, 36, 2024.
Martin Schlossarek, Miroslav Syrovátka, and Ondřej Vencálek. The importance of variables in composite indices: A contribution to the methodology and application to development indices. Social Indicalors Research, 145(3):1125- 1160, 2019.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63 (12):54-63, 2020.
Irene Solaiman.
The gradient of generative ai release: Methods and considerations. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 111-122, 2023.
Luca Soldaini, Rodncy Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jemnifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magmusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naík, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannanch Hajishirzi, Tz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024.
Daniel J. Solove. Artificial intelligence and privacy. Florida Lawe Review, 2024. doi: 10.213//ssm.4713311 URL
Forthcoming Jamuary 2025.
Elizabeth Stanton. The human development index: A history. PERI Working Papers, 2007. URL https:
//scholarworks. www.s. http://prrr..oooiaaapppsssss
Elham Tabassi. Artificial intelligence risk management framework (ai rmf 1.0), 2023-01-26 05:01:00 2023.
URL https://tsapps. nist. hwt.:////pbbicctiioo/geeg..e.e http:///eeeeeeeeceaapaaiapiiiiia idd936225
Roka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d' d'Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv:2404.1253 2024.
Thorn. Safety by design for generative ai: Preventing child sexual abuse, 2024. URL attps:///ntts thorn org/hubfs/thorn: http://bbi/hoorrrraaae---------- -o -e
27

---

Published in Transactions on Machine Learning Research (12/2024)
Trust and Safety Professional Association. Transparency reporting. http://www..tsaa..rg/ocooeeeaae/
ts- fs-ffunamentals/// 3as-Tumddaanttaallrrrraapprrrrrr braaasppprreeeccrrprr/////////// 2023.
Jorge I. Valdovinos. Transparency as ideology, ideology as transparency: Towards a critique of the meta-aesthetics of neoliberal hegemony. Open Cultural Studies, 2(1 2(1):654-667, 2018. doi: doi:10.1515/
calture-2018-0059) URL httpp:://b.....r/10.... 11111///uuuuree218-00000000 2058-0059
Lee Vinsel. Moving violations: automobiles, experts, and regulations in the United States. JHU Press, 2019.
Jai Vipra and Sarah Myers West. Computational power and ai, Sep 2023. URL https://ainowwissiiuue htgppppulllaaciioo/ooo/////o/oop and-ai
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:21 10..399, 2021.
Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos- Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, Tason Gabriel, Verena Rieser, and William S. Isaac. Sociotechnical safety evaluation of generative ai systems. 2023. URL https://arziv
org/abs/ htg/abs/23111 2210. 11986
Amy Winograd. Loose-lipped large language modells spill your secrets: The privacy implications of large language models. Hurvard Journal of Law and Technology, 36(2), 2023.
Yue Wu, Kaifu Zhang, and Jinhong Xie. Bad greenwashing, good greenwashing: Corporate social responsibility and information transparency. Management. Science, 66(7):3095-3) 3112, 2020.
Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung- Min Kim, Munhyong Kim, Sungiu Kim, Donghyun Kwak, Hanock Kwak, Se Jung Kwon, Bado Lee, Dongsoo Lee, Gichang Lee, Jooho Lee, Bacscong Park, Scongjin Shin, Joonsang Yu, Scolki Back, Sumin Bycon, Eungsup Cho, Dooseok Choe, Jeesung Han, Youngkyun Jin, Hyein Jun, Jaeseung Jung, Chanwoong Kim, Jinhong Kim, Jinuk Kim, Dokycong Lee, Dongwook Park, Jeong Min Sohn, Sujung Han, Jiae Heo, Sungju Hong, Mina Jeon, Hyunhoon Jung, Jungeun Jung, Wangkyo Jung, Chungjoon Kim, Hyeri Kim, Jonghyun Kim, Min Young Kim, Soeun Lee, Joonhee Park, Jieun Shin, Sojin Yang, Jungsoon Yoon, Hwaran Lee, Sanghwan Bae, Jeehwan Cha, Karl Gylleus, Donghoon Ham, Mihak Hong, Youngki Hong, Yunki Hong, Dahyun Jang. Hyojun Jcon, Yujin Jeon, Yeji Jeong, Myunggeun Ji, Yeguk Jin, Chansong Jo, Shinyoung Joo, Seunghwan Jung, Adrian Jungnyung Kim, Byoung Hoon Kim, Hyomin Kim, Jungwhan Kim, Minkyoung Kim, Minseung Kim, Sungdong Kim, Yonghee Kim, Youngjun Kim, Youngkwan Kim, Donghyeon Ko, Dughyun Lee, Ha Young Lec, Jachong Lee, Jicun Lee, Jonghyun Lee, Jongjin Leo, Min Young Lec, Ychbin Lee, Taehong Min, Yuri Min, Kiyoon Moon, Hyangnam Oh, Jaesun Park, Kyuyon Park, Younghun Park, Hanbae Seo, Seunghyun Seo, Mihyun Sim, Gyubin Son, Matt Yeo, Kyung Hoon Yeom, Wonjoon Yoo, Myungin You, Doheon Ahn, Homin Ahn, Joohee Ahn, Seongmin Ahn, Chanwoo An, Hyyeryun An, Junho An, Sang-Min An, Boram Byun, Eunbin Byun, Jongho Cha, Minji Chang, Seunggyu Chang, Haesong Cho, Youngdo Cho, Dalnim Choi, Daseul Choi, Hyoseok Choi, Minseong Choi, Sangho Choi, Seongjae Choi, Wooyong Choi, Sewhan Chun, Dong Young Go, Chiheon Ham, Danbi Han, Jaemin Han, Moonyoung Hong, Sung Bum Hong, Dong-Hyun Hwang, Seongchan Hwang, Jinbae Im, Hyuk Jin Jang, Jaehyung Jang, Jaeni Jang, Sihyeon Jang, Sungwon Jang, Joonha Jeon, Daun Jeong, Joonhyun Jeong, Kyeongseok Jeong, Mini Jeong, Sol Jin, Hanbyeol Jo, Hanju Jo, Minjung Jo, Chacyoon Jung, Hyungsik Jung, Jacuk Jung, Ju Hwan Jung, Kwangsun Jung, Seungiae Jung, Soonwon Ka, Donghan Kang, Soyoung Kang, Taeho Kil, Areum Kim, Beomyoung Kim, Byeongwook Kim, Dachee Kim, Dong- Dong-Gyun Kim, Donggook Kim, Donghyun Kim, Euna Kim, Eunchul Kim, Geewook Kim, Gyu Ri Kim, Hanbyul Kim, Heesu Kim, Isaac Kim, Jeonghoon Kim, Jihye Kim, Joonghoon Kim, Minjac Kim, Minsub Kim, Pil Hwan Kim, Sammy Kim, Seokhun Kim, Seonghyeon Kim, Soojin Kim, Soong Kim, Soyoon Kim, Sunyoung Kim, Taeho Kim, Wonho Kim, Yoonsik Kim, You Jin Kim, Yuri Kim, Beomseok Kwon, Ohsung Kwon, Yoo-Hwan Kwon, Anna Lee, Byungwook Loe, Changho Lee, Daun Lee, Dongjae Lee, Ha-Ram Leo, Hodong Loe, Hwiycong Lee, Hyummi Lee, Injac Lee, Jacung Lee, Jeongsang Lee, Jisoo Lee, Jongsoo Lee, Joongjae Lee, Juhan Lee, Jung Hyun Tee, Junghoon Lee, Junwoo Lee, Se Yun Lee, Sujin Lee, Sungjae Lee, Sungwoo Lee, Wonjae Lee, Zoo Hyun
28

---

Published in Transactions on Machine Learning Research (12/2024)
Lee, Jong Kun Lim, Kun Lim, Taemin Lim, Nuri Na, Jeongyeon Nam, Kyeong-Min Nam, Yeonscog Noh, Biro Oh, Jung-Sik Oh, Solgil Oh, Yeontaek Oh, Boyoun Park, Cheonbok Park, Dongju Park, Hyeonjin Park, Hyun Tae Park, Hyunjung Park, Jibye Park, Jooseok Park, Junghwan Park, Jungsoo Park, Miru Park, Sang Hee Park, Seunghyun Park, Soyoung Park, Tacrim Park, Wonkyeong Park, Hyunjoon Ryu, Jeonghun Ryu, Nahyeon Ryu, Soonshin Seo, Suk Min Seo, Yoonjeong Shim, Kyuyong Shin, Wonkwang Shin, Hyun Sim, Woongscob Sim, Hyejin Soh, Bokyong Son, Hyunjun Son, Seulah Son, Chi-Yun Chi-Yun Song, Chiyoung Song, Ka Yeon Song, Minchul Song, Seungmin Song, Jisung Wang, Yonggoo Yeo, Myeong Yeon Yi, Moon Bin Yim, Tachwan Yoo, Youngjoon Yoo, Sungmin Yoon, Young Jin Yoon, Hangyeol Yu, Ui Seon Yu, Xingdong Zuo, Jeongin Bae, Joungeun Bae, Hyunsoo Cho, Seonghyun Cho, Yongjin Cho, Tackyoon Choi, Yera Choi, Jiwan Chung, Zhenghui Han, Byeongho Heo, Euisuk Hong, Taebaek Hwang, Seonyeol Im, Sumin Jegal, Sumin Jeon, Yelim Jeong, Yonghyun Jeong, Can Jiang, Juyong Jiang, Jiho Jin, Ara Jo, Younghyun Jo, Hoyoun Jung, Juyoung Jung, Seunghyeong Kang, Dae Hee Kim, Ginam Kim, Hangyeol Kim, Heesoung Kim, Hyojin Kim, Hyojun Kim, Hyun-Ah Kim, Jechye Kim, Jin-Hwa Kim, Jiscon Kim, Jonghak Kim, Jung Yoon Kim, Rak Yeong Kim, Seongjin Kim, Seoyoon Kim, Sewon Kim, Sooyoung Kim, Sukyoung Kim, Taeyong Kim, Naeun Ko, Bonseung Koo, Heeyoung Kwak, Haena Kwon, Youngjin Kwon, Boram Lee, Bruce W. Lee, Dagyeong Lee, Erin Lee, Euijin Lee, Ha Gyeong Lee, Hyojin Lee, Hyunjeong Lee, Jeeyoon Lee, Jeonghyun Lee, Jonghook Lee, Joonhyung Lee, Junhyuk Lee, Mingu Lee, Naycon Lee, Sangkyu Lee, Se Young Lee, Seulgi Lee, Seung Jin Lee, Suhyeon Lee, Yeonjae Lee, Yesol Lee, Youngbeom Lee, Yujin Lee, Shaodong Li, Tianyu Liu, Geong-Eun Moon, Taehong Moon, Max-Lasse Nihlenramstroem, Wonseok Oh, Yuri Oh, Hongbeen Park, Hyekyung Park, Jaeho Park, Nohil Park, Sangjin Park, Jiwon Ryu, Miru Ryu, Simo Ryu, Ahreum Seo, Hee Seo, Kangdeok Seo, Jamin Shin, Seungyoun Shin, Heetae Sin, Jiangping Wang, Lei Wang, Ning Xiang, Longxiang Xiao, Jing Xu, Seonyeong Yi, Haanju Yoo, Haneul Yoo, Hwanhee Yoo, Liang Yu, Youngjae Yu, Weijie Yuan, Bo Zeng, Qian Zhou, Kyunghyun Cho, Jung- Woo Ha, Joonsuk Park, Jibyun Hwang, Hyoung Jo Kwon, Soonyong Kwon, Jungyeon Lee, Seungho Lee, Seonghyeon Lim, Hyunkyung Noh, Seungho Choi, Sang- Woo Lee, Jung Hwa Lim, and Nako Sung. Hyperclova x technical report, 2024.
Ellen Pei-Y: Yi Yu, Christine Qian Guo, and Bac Van Luu. Environmental, social and governance transparency and firm value. Business Strategy and the Environment, 277))998711044, 2018.
Monika Zainieriute. "transpareney-washinng"" in the digital age corporate agenda of procedural fotishism. Technical report, 2021. URL http://hdl. handle net/11159/468558
29

---

Published in Transactions on Machine Learning Research (12/2024)
A Selection decisions
In conducting the index, core structural design decisions are (i) the indicators used to assess developers and (ii) the developers that are assessed. Here, we clarify both matters.
A.1
Indicator selection
We use the same 100 indicators FMTI v1.0 to facilitate direct comparison. These indicators are listed by domain in Figure
A.2 Developer selection
We contacted 19 foundation model developers to request these developers submit transparency reports for the purpose of conducting FMTI v1.1. Consistent with the principles used by Bommasani et al. (2023a)) we only considered developers that are companies and that develop prominent foundation models. Specifically, we contacted leadership via email at 01.ai, Adept, AT21 Tabs, Aleph Alpha, Amazon, Anthropic, BigCode (Hugging Face and ServiceNow), Cohere, Databricks, Google, IBM, Inflection, Meta, Microsoft, Mistral, OpenAI, Stability AI, Writer, and xAI. Following this email correspondence, and further clarification of the nature of the request, 14 foundation model developers agreed to provide the requested transparency reports.
Therefore, selection process deliberately excluded foundation model developers that are not companies, even if they develop prominent foundation models, such as the Allen Institute for AI (developer of models such as OLMo; Groeneveld et al. 20244) and FleutherAT (developer of Pythia; Biderman et al. 2023 While developers such as AI2 and Eleuther are often lcadors in various types of transparency, releasing detailed information about data (Gao et al. 2020 Soldaini el al. 2024), evaluations (Gao al. 2021 Magnusson
et al. 2023). and the model development pipeline (Black et al. 2022 Biderman al.) 2023
Groeneveld et al,
2024
consider only companics in selecting developers to
in FMTI v1.1.
Our selection process also did not involve engagement with developers where we lacked connections with their leadership, which often coincides with models developed outside the United States and the Western hemisphere (c.g. the developers of Falcon (Almazrouci et al.] 20233 Qwen (Bai et al. 2023
DeepSeek
DeepSeek-AA et al. 2024), and HyperCLOvA (Yoo et al.] 2024
Finally, we did not engage any developer that released its first prominent foundation model during our execution of FMTI v1.1 (such releases include Apple's MM1 (MeKinzie et al., 2024), SambaNova's Samba (SambaNova 2024.. Reka's Reka Core (Team et al. 2024), and Snowflake's 's Arctic-1 (Merrick et al. 2024)). Moving forward, having successfully demonstrated that prominent foundation model developers have cooperated by submitting transparency reports, subsequent versions of the Foundation Model Transparency Index may engage these companies as well as others.
30

---

Published in Transactions on Machine Learning Research (12/2024)
8:24
9:44
2.00
3:44
0:14
0:60
0.40
2:14
140
637
172
3.4
366
0.5
6.02
EG0
6.0
2666
077
2.48
0.5
2:50
257
460
200
550
444
2.00
Mirec
PM
07
PM2
224
0.7
100
666
0.5
73
2:14
2.70
Speak
3.44
266
3:54
6:76
C.7
63
9:14
666
2010X Aapol
Aoagle
Figure 9: Correlations between Companies. The correlation between the scores for pairs of companies across all indicators. Correlation measured using the simple matching cocfficient (i.c. agreement rate), which is the fraction of all indicators for which both companies receive the same score (i.e. both receive the point both do not receive the point).
B Extended Results
We present additional results, drawing inspiration from the analyses conducted in the first version of the Foundation Model Transparency Index.
B.1 Developer correlations
Mcasuring correlations. The 100 X 14 matrix of scores introduces data-driven structure. In particular, it clarifies relationships that arise in practice between different regions the index. Here, we consider the correlations, in scores, focusing on company-to-compan similarity for simplicity. For example, this analysis helps address the following: if two companies receive similar aggregate scores, this because they satisfy all the same indicators or do they score points two very different sets of indicators?
In Figure 9 we plot the correlation between every pair of companics. To mcasure correlation, we report the simple matching coefficient (SMC) or the agreement rate. The SMC is the fraction of the 100 indicators for which both companies receive the same score (i.e. both receive a zero or both receive a 1). As a result, a SMC of indicates there no indicator such that both companies receive the same score and a SMC of indicates that for all indicators both companies receive the same score. For this reason, the correlation matrix is symmetric and guaranteed to be on the diagonal.
Upstream correlations. In Figure 10 we plot the correlation between every pair of companies when considering only indicators from the upstream domain.
Model correlations. In Figure 11 we plot the correlation between every pair of companies when considering only indicators from the model domain.
Downstream correlations. In Figure 12 we plot the correlation between every pair of companies when considering only indicators from the downstream domain.
31

---

Published in Transactions on Machine Learning Research (12/2024)
9.25
9228
0.231
0.04
2:4
6.00
0:42
9:6
E44
6:4
1:00
444
3.44
0:241
0.93
241
0.63
6.47
812
156
6.4
181
236
0.5
2:55
022
0:78
2.47
0.00
0:55
3.00
181
0.5
035
0:911
9.0
3.60
0.994
222
0:10
4.4
947
535
1.66
8:83
0:15
3.62
1.00
8:5
8.43
6.78
Miroci
4:1
1'
2.47
60
PM
5.5
2.47
PAZ
2.66
5.
3.69
2.00
2/44
4:4
6:17
253
660
77
5.75
8.56
Opepk
183
0.5
3.56
0:22
0:04
0.60
6.47
3:00
136
650
Algi
Awagle
MM
Correlations between Companies (Upstream Indicators). The correlation between the scores for pairs of companies across all indicators when only considering upstream indicators. Correlation is measured using the simple matching coefficient (i.e. agreement rate), which is the fraction of all indicators for which both companies receive the same score (i.e. both receive the point both do not receive the point).
Figure 10:
Abol
5:00
347
2:3
2:23
3:67
0.7
2.64
2:07
5:25
8:10
2.64
1:0
0:73
233
2.76
64
0:00
E35
A
ERT
87
0.7
07
T5
20
5:5
EAT
R2
16
02
DAT
DAY
D44
R7
Google
a6
2477
232
6G
6.7
461
5
80
PMD
270
3.4
077
07
2.44
6/
TT
267
235
2.79
8776
Opmnk
284
3.67
235
2:5
0.7
C:75
5:50
1:13
550
500gl
Mccsso
Figure 11: Correlations between Companies (Model Indicators). The correlation between the scores for pairs of companies across all indicators when only considering model indicators. Correlation is measured using the simple matching coefficient (i.e. agreement rate), which is the fraction of all indicators for which both companies receive the same score (i.e. both receive the point or both do not receive the point).
32

---

Published in Transactions on Machine Learning Research (12/2024)
0.83
0.37
0.00
0.03
3.56
0.68
0.14
0:01
1'
0:72
0.71
0.74
0.71
0.00
272
G74
0.77
A mm
11:44
14
12:77
Mepode
66
0.34
0.35
0.74
26
0:09
6:71
C:71
0.03
3.68
0.77
0.82
0.08
Ceok
8.0
0.53
0.53
0.72
0,99
6.99
9.00
2.39
9.74
3
PazE
246
0.21
2
0.71
0:09
6:7
6:4
0.1
266
9.24
66
0.74
26
0.77
0:24
DUTZ
CTT
3.7
0.
66
0.0
Makiy
6:44
0:77
R27
PNR
1F1
0.64
0.21
0.71
0:09
6.1
366
Correlations between Companies (Downstream Indicators). The correlation between the scores for pairs of companies across all indicators when considering only downstream indicators. Correlation is measured using the simple matching coefficient (i.e. agreement rate), which is the fraction of all indicators for which both companies receive the same score (i.e. both receive the point or both do not receive the point).
Figure 12:
B.2 Indicator-level results
Assessing companies at the indicator level. The core of the Foundation Model Transparency Index is the 100 indicators of transparency, which we aggregate into subdomains and domains in order to facilitate discussion and analysis of our results. We score each developer on each indicator (either or 1) based on the information disclosed by the developer in relation to that indicator; each indicator is accompanied by definition, which describes the indicator, and notes, which provide details about how that indicator is scored Bommasani et al. 2023a Appendix B). Below we provide each developers' score on every indicator, broken down by domain (i.e. upstream, model, and downstream)
Upstream Indicators. In Figure 13 we show the scores of every developer on each of the indicators in the upstream domain. We also disaggregate uptream indicators by subdomain (data, data labor, data access, compute, methods, and data mitigations).
Model Indicators. In Figure 14, show the scores of every developer each of the indicators in the model domain. We also disaggregate model indicators by subdomain (distribution, usage policy, model behavior policy, user interface, user data protection, model updates, feedback, impact, and downstrean documentation).
Downstream Indicators. In Figure 13 we show the scores every developer on each of the indicators in the downstream domain. We also disaggregate downstream indicators by subdomain (model basics, model access, capabilities, limitations, risks, model mitigations, trustworthiness, and inference).
33

---

Published in Transactions on Machine Learning Research (12/2024)
Foundard Transparacy indecindiattanaadd Scorus
Bow foom Mue
ADEPT
AlZfiaba
amazon
ANTHRome
Google Eiu
IBM
O0Meta
Micsoott
OpenAl stabiloy maor
WHITER
Fowelt
Coukh
Ove
Migeios
Figure 13: Upstrcam Scores by Indicator. The scores for cach developer on each of the 32 upstream indicators.
Poandation Model Transpaneccy Index Indicatonn Lovdl Scaven Model May 2004
and Promsaioo Sorpperyy
ADEPT
AtZHisko
26
amason
Google
IBM
Opensil
stabiltty
MineR
Mexiu
Figure 14: Model Scores by Indicator. The scores for each developer on each of the 33 model indicators.
34

---

Published in Transactions on Machine Learning Research (12/2024)
Foundados Model Transprnncy Index Ledicatan-Laed Scarus Downstrarr,
boven femeats Makie
IBM
MoMeta
Microoof
OpenAi
stabiiityol
WATER
Daye Mayy
Modi
Markes
Figure 15: Downstrcam Scores by Indicator. The scores for each developer on cach of the 35 downstream indicators.
35


Reviews:
[{"review_id": "bX2z6l0Eup", "reviewer": "Reviewer_N9wk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and structured and faces an ambitious and crucial challenge. The main issue is that the manuscript is very incremental w.r.t. FMTI v1.0 and largely based on that. Indeed, a reader cannot fully understand it without having read the other paper. This would not have been an issue if the FMTI v1.0 paper had been previously peer-reviewed and published. However, accepting for publication the FMTI v1.1 version means accepting, as a consequence, all the content of the paper presenting FMTI v1.0.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper advances the Foundation Model Transparency Index (FMTI) v1.0 presented in October 2023 to measure the transparency of leading foundation model developers by requesting developers to report the relevant information for each indicator. The analysis is based on 14 reports. The FMTI v1.1 results show room improvements in transparency over half a year.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested change is therefore to present for publication a unique manuscript presenting FMTI v1.0 and v1.1 explaining all the differencing but having is as a primary contribution for the work. This would also avoid confusion to recognize the improvement of the paper, stating it as a unique one. Obviously, the authors should clearly state in the cover letter that the work already appeared online on a public access repository and that it already received concerns, at least in terms of the number of citations.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sdzWRsGmwX", "reviewer": "Reviewer_WyGb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The problem that the authors studied is extremely important, practical, and has the potential to be extremely impactful. \n- It is very clear that the authors went through a serious, comprehensive process of indicator selections. The authors have also access to 14 industry collaborators that they have been closely working with on refining and releasing their final indices.\n- The indicators are clearly divided into downstream, upstream, and model-level indicators, which allows the authors to draw clear lessons as to where foundation model developers fail in terms of transparency. To me, the most exciting insight of the paper was how the authors' work highlights and quantifies how, despite significant progresses in terms of transparency, many foundation models still largely fail at being transparent when it comes to data collection, compute, and labor. \n- The authors' has in fact already been impactful and may have helped/pushed companies to provide more transparent models: \"the 8 developers evaluated in both October 2023 and May 2024 showed marked improvement,or 19 points on average (...) Developers that were assessed only in v1.1 performed slightly worse than those assessed in both v1.0 and v1.1\". \n\nWeaknesses: \n- It might be useful to extend the peer review process to more reviewers. I am assuming that both reviewers are authors of the paper (let me know if wrong); even while the reviews were conducted independently, it is possible that the reviewers may have similar biases; involving more, outside reviewers may further confirm the validity of the study.\n- I am not sure if this is really a weakness, but the paper itself does not make a technical/fundamental contribution to ML. Rather, the paper is more of a technical report on the actual technical/fundamental contribution, which the reviewers sadly cannot access directly without breaking the double-blindness of the process. I am actually wondering if this is a good fit for TMLR, due to the above. (Note that I would not want to reject the paper based solely on this comment, I am just thinking this should be a point of discussion). \nI am also actually wondering if this is a missed opportunity by the authors to go for the smaller TMLR community when I believe this is the kind of work that could attract a large audience from non-ML expert/could be submitted to a higher reach publication where this index would be more publicized and have more impact.  \n\nOverall, I would support the paper being accepted if the rest of the review team believes this is a good fit for TMLR. This is a very nice, refreshing, impactful and transformative contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides insights into v1.1 of the \"Foundation Model Transparency Index\", which is a novel index to measure the transparency of foundation models released by leading developers. \n\nThe paper seems to be a technical report that highlights how the model was developed and the insights that the authors learned from using their model to score 14 companies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "The goal of the paper is to address the broader impacts concern of transparency in foundation models. \n\nIf I had any nitpicky concern, it would be that the tool, if not comprehensive enough, could be used by companies to justify that they are doing things right and ethically when they are not. However, given that the model of the paper/the foundation model transparency index seems comprehensive and well thought-out, this seems like a very secondary worry. The improved visibility on how transparent companies are about their model largely negates any potential minor impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hOuU26EnpI", "reviewer": "Reviewer_7TYK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n\n* I think this is a great initiative, it requires a lot of work, the report is serious and well-documented\n\n* at least two results seem interesting to me:\n\n1- most developers disclose more information when explicitly asked for it. This likely means that some information is not intentionally kept private\n\n2- this effort allows to formalize which areas are systematically kept private. I am not sure we learn much here (e.g., it was pretty clear that developers do not want to say too much about the data they use), but this effort confirms that developers do not give precise information even when explicitly asked for it\n\nweaknesses:\n\n* the paper is not really standalone. There is a difficult trade-off because the paper heavily builds on the previous paper on the v1.0 report which is 115 pages long and difficult to summarize.  Figure 8 was useful to better understand the nature of the index. Somehow I missed the reference in section 3.1 in a first read. Probably because to me the need to better understand the questions came up when reading the results and I forgot about this reference. I don’t have any clear solution to that, apart from putting Figure 8 within the full body of the paper. I’m not sure it’s the best, but I think that would have helped me.\n\n* the paper does not address \"why\" developers are shy to answer some of these questions, which feels like a missed opportunity. I think understanding why some information is kept private is at least as important as whether it is kept private\n\n* probably my main comment/criticism: \n\n1- the paper (and the previous one) does not seem to address the question of how much transparency should be expected from developers and what is legitimate to keep private. For instance, not disclosing the number of parameters of a model does not seem to be a big deal to me as far as the end users is concerned, but of course it doesn't help reproducibility. But many of these developers are for-profit companies, and it is not part of their mission to help reproducibility. \n\nIn the end, it is unclear to me how to interpret the score -- does \"58\" mean that observers do not have \"enough\" information on the model (e.g., to study the risks and impact), or it simply means that some information is legitimately kept proprietary.\n\n2- some questions seem very difficult to answer to me and are more research questions. For instance, \"who uses the model\" and \"how they use the model\" seems quite difficult to answer accurately for open source models, and \"who is impacted\" also seems to be quite difficult as it is likely changing substantially every month and depending a lot on where we are looking into in the world. \n\n\n* out of the scope of the current paper (and thus this review), but I am also not sure about the phrasing of some questions. For instance, asking about \"wages\" (wages of third-party data workers) and giving the point for the answer \"larger than local minimum wage\" is far from giving much information about worker exploitation -- e.g., this tells nothing about the working conditions or the type of working contract, which may be more meaningful (for instance, if the workers are full time employees rather than gig workers, this would imply a wage larger than local minimum).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a study on the transparency of major large language models (LLM). The paper builds on prior work on \"Foundation Model Transparency Index\" (FMTI) -- especially the choice of questions and previous estimation of the index using publicly available information, and presents an updated study where the authors engage in a discussion with LLM developers in order to get as much information as developers are willing to disclose, as well as resolving ambiguities regarding the questions.\n\nThe paper reports that significantly more information was disclosed with this new process, yet most developers tend to keep a lot of information private, in particular regarding data used for (pre)training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "no changes requested, I think the paper satisfies TMLR's acceptance conditions", "broader_impact_concerns": "no concern per se, the paper contains a fairly exhaustive discussion on limitations", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers WyGb and 7TYK gave thorough reviews and opine that the paper meets TMLR's criteria. However, the reviewers do acknowledge that this is not a "standard" TMLR paper and that it builds heavily on a previous report. 

Reviewer N9wk initially opined that the paper met TMLR's criteria (while giving a really lightweight review), but then after reading the thorough reviews of reviewers WyGb and 7TYK, flipped their opinion. 

Aggregating all the information from the reviewers and based on my own more brief reading of the paper, I think this paper makes contributions that are valuable to the community. While the writing should be improved (see last comment below), I think the claims are appropriate for the content and the community will be interested in this version 1.1 of the transparency index. 

I request the authors to please address the following in their revision:

- Include your response to the reviewer's comment of "More scorers/reviewers of scores, including individuals independent from the FMTI team" as this is a natural question that a reader may also have.

- "I think understanding why some information is kept private is at least as important as whether it is kept private" Please add information on this as well, as you described in your response. 

- Please add the details promised in response to reviewer comment "the paper is not really standalone." This will help make the paper more understandable. In case it helps, you have the flexibility that the published version need not preserve author anonymity.
Decision: Accept (poster)