# AI labs can boost external safety research

Frontier AI labs can boost external safety researchers by

*   Sharing better access to powerful models (early access, fine-tuning, helpful-only,[^aat5takgg0j] filters/moderation-off, logprobs, activations)[^hygyg1bi6u8]
*   Releasing research artifacts besides models
*   Publishing (transparent, reproducible) safety research
*   Giving API credits
*   Mentoring

* * *

Here's what the labs have done (besides just publishing safety research[^mdh4a9e8st]).

Anthropic:

*   [Releasing resources including RLHF and red-teaming datasets, an interpretability notebook, and model organisms prompts and transcripts](https://github.com/orgs/anthropics/repositories?q=fork%3Afalse)
*   [Supporting creation of safety-relevant evals and tools for evals](https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations)
*   [Giving free API access to some OP grantees](https://cair-preview.vercel.app/agent-benchmarks#4-expression-of-interest-process-and-other-logistics) and [giving some researchers $1K (or sometimes more) in API credits](https://support.anthropic.com/en/articles/9125743-what-is-the-external-researcher-access-program)
*   ([Giving deep model access to Ryan Greenblatt](https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=B6oDGoyphuNuzdDAT))
*   (External mentoring, in particular via [MATS](https://www.matsprogram.org))
*   \[No fine-tuning or deep access, except for Ryan\]

Google DeepMind:

*   [Publishing](https://arxiv.org/abs/2403.13793) their model evals for dangerous capabilities and sharing [resources](https://github.com/google-deepmind/dangerous-capability-evaluations) for reproducing some of them
*   Releasing [Gemma SAEs](https://deepmind.google/discover/blog/gemma-scope-helping-the-safety-community-shed-light-on-the-inner-workings-of-language-models/)
*   Releasing [Gemma](https://ai.google.dev/gemma) weights
*   (External mentoring, in particular via [MATS](https://www.matsprogram.org))
*   \[No fine-tuning or deep access to frontier models\]

OpenAI:[^d17rhxe37mn]

*   [OpenAI Evals](https://github.com/openai/evals/blob/main/README.md)
*   [Superalignment Fast Grants](https://openai.com/index/superalignment-fast-grants/)
*   [Maybe giving better API access to some OP grantees](https://cair-preview.vercel.app/agent-benchmarks#4-expression-of-interest-process-and-other-logistics)
*   [Fine-tuning](https://platform.openai.com/docs/guides/fine-tuning) GPT-3.5 (and "GPT-4 fine-tuning is in experimental access"; OpenAI shared GPT-4 fine-tuning access with academic researchers including [Jacob Steinhardt](https://arxiv.org/abs/2406.20053) and [Daniel Kang](https://arxiv.org/abs/2311.05553) in 2023)
    *   Update: [GPT-4o fine-tuning](https://openai.com/index/gpt-4o-fine-tuning/)
*   Early access: shared GPT-4 with a few safety researchers including [Rachel Freedman](https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform#Tuk9bYtzHmeM7FrR9) before release
*   [API gives top 5 logprobs](https://cookbook.openai.com/examples/using_logprobs)

Meta AI:

*   Releasing [Llama](https://llama.meta.com) weights

Microsoft:

*   \[Nothing\]

xAI:

*   \[Nothing\]

* * *

Related papers:

*   [Structured access for third-party research on frontier AI models](https://www.oxfordmartin.ox.ac.uk/publications/structured-access-for-third-party-research-on-frontier-ai-models-investigating-researchers-model-access-requirements) (Bucknall and Trager 2023)
*   [Black-Box Access is Insufficient for Rigorous AI Audits](https://arxiv.org/abs/2401.14446) (Casper et al. 2024)
    *   (The paper is about *audits*, like for risk assessment and oversight; this post is about *research*)
*   [A Safe Harbor for AI Evaluation and Red Teaming](https://arxiv.org/abs/2403.04893) (Longpre et al. 2024)
*   [Structured Access](https://www.governance.ai/post/sharing-powerful-ai-models) (Shevlane 2022)

[^aat5takgg0j]: "Helpful-only" refers to the version of the model RLHFed/RLAIFed/finetuned/whatever for helpfulness but not harmlessness. 

[^hygyg1bi6u8]: Releasing model weights will likely be dangerous once models are more powerful, but all past releases seem fine, but e.g. Meta's poor risk assessment and lack of a plan to make release decisions conditional on risk assessment is concerning. 

[^mdh4a9e8st]: And an unspecified amount of funding Frontier Model Forum grants. 

[^d17rhxe37mn]: Jan 2025 update, not necessarily exhaustive:https://openai.com/index/early-access-for-safety-testing/https://openai.com/form/rft-research-program/ (not safety-specific, just deep access)