# List of technical AI safety exercises and projects

EDIT 3/17/2023: I've reorganized the doc and added some governance projects.

I intend to maintain a list at [this doc](https://docs.google.com/document/d/1-58zgC2lRMbMK-CXU44VR3ApGYbTI0aJKX-cKkxDeyo/edit?usp=sharing). I'll paste the current state of the doc (as of January 19th, 2023) below. I encourage people to comment with suggestions.

*   [**Levelling Up in AI Safety Research Engineering \[Public\]**](https://docs.google.com/document/d/1b83_-eo9NEaKDKc9R3P5h5xkLImqMw8ADLmi__rkLo4/edit) **(**[**LW**](https://www.lesswrong.com/posts/uLstPRyYwzfrx3enG/levelling-up-in-ai-safety-research-engineering)**)**
    *   Highly recommended list of AI safety research engineering resources for people at various skill levels.
*   [**AI Alignment Awards**](https://www.alignmentawards.com/)
*   [**Alignment jams / hackathons**](https://alignmentjam.com/)from [Apart Research](https://apartresearch.com/jam)
    *   Past / upcoming hackathons: [LLM](https://itch.io/jam/llm-hackathon), [interpretability 1](https://itch.io/jam/interpretability), [AI test](https://itch.io/jam/aitest), [interpretability 2 ](https://itch.io/jam/ai-neuroscience)
    *   Projects on AI Safety Ideas: [LLM](https://aisafetyideas.com/), [interpretability](https://aisafetyideas.com/list/interpretability-hackathon), [AI test](https://aisafetyideas.com/list/test-the-ai)
    *   Resources: [black-box investigator of language models](https://www.lesswrong.com/posts/yGaw4NqRha8hgx5ny/the-case-for-becoming-a-black-box-investigator-of-language), [interpretability playground](https://alignmentjam.com/interpretability-playground) ([LW](https://www.lesswrong.com/posts/gJRsiukrNudHLGgCf/the-interpretability-playground)), [AI test](https://alignmentjam.com/aitest)
    *   [Examples of past projects](https://alignmentjam.com/jams); [interpretability winners](https://www.lesswrong.com/posts/hhhmcWkgLwPmBuhx7/results-from-the-interpretability-hackathon)
    *   [How to run one](https://alignmentjam.com/running) as an in-person event at your school
*   Neel Nanda: [**200 Concrete Open Problems in Mechanistic Interpretability**](https://www.lesswrong.com/s/yivyHaCAmMJ3CqSyj) ([doc](https://docs.google.com/document/d/1WONBzNqfKIxERejrrPlQMyKqg7jSFW92x5UMXNrMdPo/edit) and [previous version](https://docs.google.com/document/d/15RnVxGNRm5ZDJojoExwXuiRxfNFB9SFqX5NHkT6xjc8/edit#))
*   [Project page from AGI Safety Fundamentals](https://www.agisafetyfundamentals.com/ai-alignment-tabs/week-8) and their [**Open List of Project ideas**](https://docs.google.com/spreadsheets/d/1Co45Bd_rOc2JZ3jqK7kw1tMptcI0J66Y_8khK4u30_w/edit?usp=sharing)
*   [**AI Safety Ideas**](https://aisafetyideas.com/) by Apart Research; [EAF post](https://forum.effectivealtruism.org/posts/DTTADonxnDRoksp4E/ai-safety-ideas-an-open-ai-safety-research-platform)
*   [Most Important Century writing prize](https://forum.effectivealtruism.org/posts/4XK5zkyv94voC8Fjr/announcing-the-most-important-century-writing-prize) ([Superlinear page](https://www.super-linear.org/prize?recordId=recmsHjuOh5JmVJWJ))
*   [Center for AI Safety](https://safe.ai/)
    *   [**Competitions**](https://safe.ai/competitions) like [SafeBench](https://benchmarking.mlsafety.org/)
    *   [**Student ML Safety Research Stipend Opportunity**](https://www.mlsafety.org/safety-scholarship)  – provides stipends for doing ML research. 
    *   [course.mlsafety.org projects](https://docs.google.com/document/d/12nj9a1WOJSHaddks3vmmjVKtlHmO53zEW4KHYyapUxY/edit) CAIS is looking for someone to add details about these projects on [course.mlsafety.org](https://course.mlsafety.org/)
*   [**Distilling **/ summarizing / synthesizing / reviewing / explaining](https://www.lesswrong.com/posts/zo9zKcz47JxDErFzQ/call-for-distillers)
*   [**Forming your own views on AI safety (without stress!)**](https://www.youtube.com/watch?v=edoQ3CiNa_s) – also see [Neel's presentation slides](https://docs.google.com/presentation/d/1LMBfw2wUmGLEvRTi8019DX_uGdJTeGsHRPRa5S8UIWs/edit#slide=id.p) and "[Inside Views Resources](https://docs.google.com/document/d/1O3ci6Q0UZ8qeBpKs6lVxFGLWiH89fBtz-lLG2iN8YNc/edit#)" doc
*   Answer some of the [application questions](https://docs.google.com/document/d/1J0jRKfWrNUakiIBbbpw7q3i27Rw9XCLSJvW6659NhRU/edit#) from the winter 2022 [SERI-MATS](https://www.serimats.org/), such as [Vivek Hebbar's problems](https://docs.google.com/document/d/1NVVtdsfz7HiseVFSk3jYly4sPG4dG03wFFDrD8rBXU0/edit)
*   [10 exercises from Akash](https://forum.effectivealtruism.org/posts/BBS8cz4Sa7wvJ2Jso/resources-that-i-think-new-alignment-researchers-should-know#Exercises) in “Resources that (I think) new alignment researchers should know about”
*   [\[T\] Deception Demo Brainstorm](https://docs.google.com/document/d/1HLnd7IGyX7ByrnBUZQH6fbneox2jwZ7LRGQw-esc-uU/edit) has some ideas (message [Thomas Larsen](https://www.lesswrong.com/users/thomas-larsen) if these seem interesting)
*   Upcoming [2023 Open Philanthropy AI Worldviews Contest](https://forum.effectivealtruism.org/posts/3kaojgsu6qy2n8TdC/pre-announcing-the-2023-open-philanthropy-ai-worldviews)
*   [Alignment research at ALTER](https://forum.effectivealtruism.org/posts/zCYGbYAaXeq7v67Km/prize-and-fast-track-to-alignment-research-at-alter) – interesting research problems, many have a theoretical math flavor
*   [Open Problems in AI X-Risk \[PAIS #5\]](https://forum.effectivealtruism.org/s/8EqNwueP6iw2BQpNo/p/hNPCo4kScxccK9Ham)
*   [Amplify creative grants](https://forum.effectivealtruism.org/posts/pLiEoTuwEYRxNueoD/announcing-amplify-creative-grants) (old)
*   Evan Hubinger: [Concrete experiments in inner alignment](https://www.lesswrong.com/posts/uSdPa9nrSgmXCtdKN/concrete-experiments-in-inner-alignment), [ideas someone should investigate further](https://www.alignmentforum.org/posts/HE3Styo9vpk7m8zi4/evhub-s-shortform?commentId=cPPvFFLLkMuh9k5Zx), [sticky goals](https://www.lesswrong.com/posts/a2Bxq4g2sPZwKiQmK/sticky-goals-a-concrete-experiment-for-understanding)
*   Richard Ngo: [Some conceptual alignment research projects](https://www.lesswrong.com/posts/27AWRKbKyXuzQoaSk/some-conceptual-alignment-research-projects), [alignment research exercises](https://www.lesswrong.com/posts/kj37Hzb2MsALwLqWt/alignment-research-exercises)
*   Buck Shlegeris: [Some fun ML engineering projects that I would think are cool](https://docs.google.com/document/d/1yMP9i6cQQwHG1ITOtccUkuN3p9eTk1AtzOM8TP-mZVs/edit), [The case for becoming a black box investigator of language models](https://www.alignmentforum.org/posts/yGaw4NqRha8hgx5ny/the-case-for-becoming-a-black-box-investigator-of-language)
*   Implement a [key paper](https://spinningup.openai.com/en/latest/spinningup/keypapers.html) in deep reinforcement learning
*   “Paper replication resources” section in “[How to pursue a career in technical alignment](https://forum.effectivealtruism.org/posts/7WXPkpqKGKewAymJf/how-to-pursue-a-career-in-technical-ai-alignment)”
*   [Daniel Filan idea](https://www.lesswrong.com/posts/WgMhovN7Gs6Jpn3PH/danielfilan-s-shortform-feed?commentId=aEbnazFXn6GFym8kv)
*   Summarize a reading from [Reading What We Can](https://readingwhatwecan.com/)