# Ideas for AI labs: Reading list

Related: [AI policy ideas: Reading list](/posts/NfqqsHqembNEsTrSr/ai-policy-ideas-reading-list).

This document is about ideas for AI labs. It's mostly from an x-risk perspective. Its underlying organization black-boxes technical AI stuff, including technical AI safety.

Lists & discussion
------------------

*   [Towards best practices in AGI safety and governance: A survey of expert opinion](https://arxiv.org/abs/2305.07153) (GovAI, Schuett et al. 2023) ([LW](https://www.lesswrong.com/posts/fkqvztgszJpqmDHom/govai-towards-best-practices-in-agi-safety-and-governance-a))
    *   This excellent paper is the best collection of ideas for labs. See [pp. 18–22](https://arxiv.org/pdf/2305.07153.pdf#page=18) for 100 ideas.
*   [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://arxiv.org/abs/2307.03718) (Anderljung et al. 2023)
    *   Mostly about government regulation, but recommendations on safety standards translate to recommendations on actions for labs
*   [Model evaluation for extreme risks](https://arxiv.org/abs/2305.15324) (DeepMind, Shevlane et al. 2023)
*   [What AI companies can do today to help with the most important century](https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/) (Karnofsky 2023) ([LW](/posts/eidpSCMQhWmucijBi/what-ai-companies-can-do-today-to-help-with-the-most))
*   Karnofsky nearcasting: [How might we align transformative AI if it’s developed very soon?](/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very), [Nearcast-based "deployment problem" analysis](/posts/vZzg8NS7wBtqcwhoJ/nearcast-based-deployment-problem-analysis), and [Racing through a minefield: the AI deployment problem](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/) ([LW](/posts/PABtHv8X28jJdxrD6/racing-through-a-minefield-the-ai-deployment-problem)) (Karnofsky 2022)
*   [Survey on intermediate goals in AI governance](https://forum.effectivealtruism.org/posts/g4fXhiJyj6tdBhuBK/survey-on-intermediate-goals-in-ai-governance) (Räuker and Aird 2023)
*   [Corporate Governance of Artificial Intelligence in the Public Interest](https://doi.org/10.3390/info12070275) (Cihon, Schuett, and Baum 2021) and [The case for long-term corporate governance of AI](https://forum.effectivealtruism.org/posts/5MZpxbJJ5pkEBpAAR/the-case-for-long-term-corporate-governance-of-ai) (Baum and Schuett 2021)
*   [Three lines of defense against risks from AI](https://arxiv.org/abs/2212.08364) (Schuett 2022) 
*   [The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation](https://arxiv.org/pdf/1802.07228.pdf#page=51) (Brundage et al. 2018)
*   [Adapting cybersecurity frameworks to manage frontier AI risks](https://www.iaps.ai/research/adapting-cybersecurity-frameworks): defense in depth (IAPS, Ee et al. 2023)

Levers
------

*   [AI developer levers](https://www.legalpriorities.org/documents/Maas%20-%202023%20-%20Advanced%20AI%20governance.pdf#page=104) and [AI industry & academia levers](https://www.legalpriorities.org/documents/Maas%20-%202023%20-%20Advanced%20AI%20governance.pdf#page=106) in [Advanced AI governance](https://www.legalpriorities.org/research/advanced-ai-gov-litrev) (LPP, Maas 2023)
    *   This report is excellent
*   ["Affordances" in "Framing AI strategy"](/posts/AGrBjEb6vFaJBwjTj/framing-ai-strategy-1#Affordances) (Stein-Perlman 2023)
    *   This list may be more desiderata-y than lever-y

Desiderata
----------

Maybe I should make a separate post on desiderata for labs (for existential safety).

*   [Six Dimensions of Operational Adequacy in AGI Projects](/posts/keiYkaeoLHoKK4LYA/six-dimensions-of-operational-adequacy-in-agi-projects) (Yudkowsky 2022)
*   ["Carefully Bootstrapped Alignment" is organizationally hard](/posts/thkAtqoQwN6DtaiGT/carefully-bootstrapped-alignment-is-organizationally-hard) (Arnold 2023)
*   [Slowing AI: Foundations](/posts/MoLLqFtMup39PCsaG/slowing-ai-foundations) (Stein-Perlman 2023)
*   \[Lots of stuff implicated elsewhere, like "help others act well" and "minimize diffusion of your capabilities research"\]

Ideas
-----

### Coordination[^qu95go1dqen]

See generally [The Role of Cooperation in Responsible AI Development](https://arxiv.org/abs/1907.04534) (Askell et al. 2019).

*   Coordinate to not train or deploy dangerous AI
    *   Model evaluations
        *   [Model evaluation for extreme risks](https://arxiv.org/abs/2305.15324) (DeepMind, Shevlane et al. 2023) ([LW](https://www.lesswrong.com/posts/FdQzArWhERh4YZqY9/deepmind-model-evaluation-for-extreme-risks))
        *   [ARC Evals](http://evals.alignment.org)
            *   [Safety evaluations and standards for AI](https://forum.effectivealtruism.org/posts/49rzRKh2ZYH2QjPkg/safety-evaluations-and-standards-for-ai-or-beth-barnes-or) (Barnes 2023)
            *   [Update on ARC's recent eval efforts](https://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/) (ARC 2023) ([LW](https://www.lesswrong.com/posts/4Gt42jX7RiaNaxCwP/more-information-about-the-dangerous-capability-evaluations))
    *   Safety standards

### Transparency

Transparency enables coordination (and some regulation).

*   [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://arxiv.org/pdf/2004.07213.pdf#page=5) (Brundage et al. 2020)
    *   Followed up by [Filling gaps in trustworthy development of AI](https://arxiv.org/abs/2112.07773) (Avin et al. 2021)
*   Structured transparency
    *   [Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use Cases](https://arxiv.org/abs/2303.08956) (Bluemke et al. 2023)
    *   [Beyond Privacy Trade-offs with Structured Transparency](https://arxiv.org/abs/2012.08347) (Trask and Bluemke et al. 2020)
*   [Honest organizations](https://sideways-view.com/2018/02/01/honest-organizations/) (Christiano 2018)
*   Auditing & certification
    *   [**Theories of Change for AI Auditing**](https://www.lesswrong.com/posts/LwJwDNFhjurAKFiJm/theories-of-change-for-ai-auditing) **(Apollo 2023) and other Apollo stuff**
    *   [What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring](https://arxiv.org/abs/2303.11341) (Shavit 2023)
    *   [Auditing large language models: a three-layered approach](https://doi.org/10.1007/s43681-023-00289-2) (Mökander et al. 2023)
        *   The first two authors have other relevant-sounding work on [arXiv](https://arxiv.org/abs/2302.08500)
    *   [AGI labs need an internal audit function](https://arxiv.org/abs/2305.17038) (Schuett 2023)
    *   [AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries](https://arxiv.org/abs/2105.10356) (Cihon et al. 2021)
    *   Private literature review (2021)
*   Model evaluations
    *   [Model evaluation for extreme risks](https://arxiv.org/abs/2305.15324) (DeepMind, Shevlane et al. 2023) ([LW](https://www.lesswrong.com/posts/FdQzArWhERh4YZqY9/deepmind-model-evaluation-for-extreme-risks))
    *   [Safety evaluations and standards for AI](https://forum.effectivealtruism.org/posts/49rzRKh2ZYH2QjPkg/safety-evaluations-and-standards-for-ai-or-beth-barnes-or) (Barnes 2023)
    *   [Update on ARC's recent eval efforts](https://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/) (ARC 2023) ([LW](https://www.lesswrong.com/posts/4Gt42jX7RiaNaxCwP/more-information-about-the-dangerous-capability-evaluations))

### Publication practices

Labs should minimize/delay the diffusion of their capabilities research.

*   [Publication decisions for large language models, and their impacts](https://forum.effectivealtruism.org/posts/KkbEfpNkjNepQrj8g/publication-decisions-for-large-language-models-and-their) (Cottier 2022)
*   [Shift AI publication norms toward "don't always publish everything right away"](https://docs.google.com/document/d/15ADKwwElJo3SR1ajKkxW7CT6Eunk4BZnEw-fuqdWK8Y/edit#heading=h.bylvbh5z9ion) in [Survey on intermediate goals in AI governance](https://forum.effectivealtruism.org/posts/g4fXhiJyj6tdBhuBK/survey-on-intermediate-goals-in-ai-governance) (Räuker & Aird 2023)
*   "Publication norms for AI research" (Aird unpublished)
*   [Publication policies and model-sharing decisions](https://static1.squarespace.com/static/6276a63ecf564172c125f58e/t/641cbc1d84814a4d0f3e1788/1679604766050/WasilEzellRichardsonSiegmann+%2810%29.pdf) (Wasil et al. 2023)

### Structured access to AI models

*   [Sharing Powerful AI Models](https://www.governance.ai/post/sharing-powerful-ai-models) (Shevlane 2022)
*   [Structured access for third-party research on frontier AI models](https://www.oxfordmartin.ox.ac.uk/publications/structured-access-for-third-party-research-on-frontier-ai-models-investigating-researchers-model-access-requirements/) (GovAI, Bucknall and Trager 2023)
*   [Compute Funds and Pre-trained Models](https://www.governance.ai/post/compute-funds-and-pre-trained-models) (Anderljung et al. 2022)

### Governance structure

*   [How to Design an AI Ethics Board](https://www.governance.ai/research-paper/how-to-design-an-ai-ethics-board) (Schuett et al. 2023)
*   [Ideal governance (for companies, countries and more)](https://www.cold-takes.com/ideal-governance-for-companies-countries-and-more/) (Karnofsky 2022) ([LW](/posts/SCs4KpcShb23hcTni/ideal-governance-for-companies-countries-and-more)) has relevant discussion but not really recommendations

### Miscellanea

*   Do more/better safety research; share safety research and safety-relevant knowledge
    *   Do safety research as a common good
        *   Do and share alignment and interpretability research
        *   Help people who are trying to be safe be safe
        *   Make AI risk and safety more concrete and legible
            *   See Larsen et al.'s [Instead of technical research, more people should focus on buying time](/posts/BbM47qBPzdSRruY4z/instead-of-technical-research-more-people-should-focus-on) and [Ways to buy time](/posts/bkpZHXMJx3dG5waA7/ways-to-buy-time) (2022)
    *   Pay the alignment tax (if you develop a critical model)
*   Improve your security (operational security, information security, and cybersecurity)
    *   There's a private reading list on infosec/cybersec, but it doesn't have much about what labs (or others) should actually do.
*   Plan and prepare: ideally figure out what's good, publicly commit to doing what's good (e.g., perhaps monitoring for deceptive alignment or supporting external model evals), do it, and demonstrate that you're doing it
    *   For predicting and avoiding misuse
    *   For alignment
    *   For deployment (especially of critical models)
    *   For coordinating with other labs
        *   Sharing
        *   Stopping
        *   Merging
        *   More
    *   For engaging government
    *   For increasing time 'near the end' and using it well
    *   For ending risk from misaligned AI
    *   For how to get from powerful AI to a great long-term future
    *   Much more...
*   [Recommendation: Bug Bounties and Responsible Disclosure for Advanced ML Systems](/posts/5dKDLv4knhXLvNHT5/recommendation-bug-bounties-and-responsible-disclosure-for) (Gray 2023)
    *   See also [this comment](https://www.lesswrong.com/posts/5dKDLv4knhXLvNHT5/recommendation-bug-bounties-and-responsible-disclosure-for?commentId=AKpr7bjgQbaXHABuf)
    *   See [OpenAI's bug bounty program](https://openai.com/blog/bug-bounty-program)
*   Report incidents
*   [The Windfall Clause: Distributing the Benefits of AI for the Common Good](https://www.governance.ai/research-paper/the-windfall-clause-distributing-the-benefits-of-ai-for-the-common-good) (O'Keefe et al. 2020)
    *   Also sounds relevant: [Safe Transformative AI via a Windfall Clause](https://arxiv.org/abs/2108.09404) (Bova et al. 2021)
*   Watermarking[^ixzl1hjxp69]
*   Make, share, and improve a safety plan
    *   [OpenAI](https://openai.com/blog/our-approach-to-alignment-research) ([LW](https://www.lesswrong.com/posts/28sEs97ehEo8WZYb8/openai-s-alignment-plans)) (but [their more recent writing on "AI safety"](https://openai.com/blog/our-approach-to-ai-safety) is more prosaic)
    *   [DeepMind](https://www.lesswrong.com/s/4iEpGXbD3tQW5atab/p/a9SPcZ6GXAg9cNKdi) (unofficial and incomplete)
        *   See also [Shah on DeepMind alignment work](https://www.lesswrong.com/posts/QBAjndPuFbhEXKcCr/my-understanding-of-what-everyone-in-technical-alignment-is?commentId=CS9qcdkmDbLHR89s2)
    *   [Anthropic](https://www.anthropic.com/index/core-views-on-ai-safety) ([LW](https://www.lesswrong.com/posts/xhKr5KtvdJRssMeJ3/anthropic-s-core-views-on-ai-safety))
*   Make share, and improve a plan for the long-term future
    *   [OpenAI](https://openai.com/blog/planning-for-agi-and-beyond) ([LW](https://www.lesswrong.com/posts/zRn6aQyD8uhAN7qCc/sam-altman-planning-for-agi-and-beyond), [Soares](https://www.lesswrong.com/posts/uxnjXBwr79uxLkifG/comments-on-openai-s-planning-for-agi-and-beyond)); [OpenAI](https://openai.com/blog/governance-of-superintelligence) ([LW](https://www.lesswrong.com/posts/hoWRRLr8zFbDcQErd/linkpost-governance-of-superintelligence-by-openai))
*   Improve other labs' actions
    *   Inform, advise, advocate, facilitate, support, coordinate
    *   Differentially accelerate safer labs
*   Improve non-lab actors' actions
    *   Government
        *   Support good policy
        *   See [AI policy ideas: Reading list](/posts/NfqqsHqembNEsTrSr/ai-policy-ideas-reading-list) (Stein-Perlman 2023)
    *   Standards-setters
        *   [How technical safety standards could promote TAI safety](https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety) (O'Keefe et al. 2022)
        *   [Standards for AI Governance: International Standards to Enable Global Coordination in AI Research & Development](https://www.governance.ai/research-paper/standards-for-ai-governance-international-standards-to-enable-global-coordination-in-ai-research-development) (Cihon 2019)
    *   Kinda the public
    *   Kinda the ML community
*   Support miscellaneous other strategic desiderata
    *    E.g. prevent new leading labs from appearing

See also
--------

*   [Best Practices for Deploying Language Models](https://txt.cohere.com/best-practices-for-deploying-language-models/) (Cohere, OpenAI, and AI21 Labs 2022)
    *   See also [Lessons learned on language model safety and misuse](https://openai.com/research/language-model-safety-and-misuse) (OpenAI 2022)
*   [Slowing AI](https://www.lesswrong.com/s/xMdkfEJhDNCL2KweB) (Stein-Perlman 2023)
*   [Survey on intermediate goals in AI governance](https://forum.effectivealtruism.org/posts/g4fXhiJyj6tdBhuBK/survey-on-intermediate-goals-in-ai-governance) (Räuker and Aird 2023)

* * *

*Some sources are roughly sorted within sections by a combination of x-risk-relevance, quality, and influentialness– but sometimes I didn't bother to try to sort them, and I haven't read all of them.*

*Please have a low bar to suggest additions, substitutions, rearrangements, etc.*

*Current as of: 9 July 2023.*

[^qu95go1dqen]: At various levels of abstraction, coordination can look like:- Avoiding a race to the bottom- Internalizing some externalities- Sharing some benefits and risks- Differentially advancing more prosocial actors?- More? 

[^ixzl1hjxp69]: Policymaking in the Pause (FLI 2023) cites A Systematic Review on Model Watermarking for Neural Networks (Boenisch 2021); I don't know if that source is good. (Note: this disclaimer does not imply that I know that the other sources in this doc are good!)I am not excited about watermarking. (Note: this disclaimer does not imply that I am excited about the other ideas in this doc! But I am excited about most of them.)