# Appendices to the live agendas

Lists cut from [our main post](https://www.lesswrong.com/posts/zaaGsFBeDTpCsYHef/shallow-review-of-live-agendas-in-alignment-and-safety), in a token gesture toward readability.

We list past reviews of alignment work, ideas which seem to be dead, the cool but neglected neuroscience / biology approach, various orgs which don't seem to have any agenda, and a bunch of things which don't fit elsewhere.

Appendix: Prior enumerations
----------------------------

*   [Everitt et al (2018)](https://arxiv.org/abs/1805.01109)
*   [Ji (2023)](https://arxiv.org/abs/2310.19852)
*   [Soares (2023)](https://www.lesswrong.com/posts/3pinFH3jerMzAvmza/on-how-various-plans-miss-the-hard-bits-of-the-alignment)
*   [Gleave and MacLean (2023)](https://www.lesswrong.com/posts/ncsxcf8CkDveXBCrA/ai-safety-in-a-world-of-vulnerable-machine-learning-systems-1)
*   [Krakovna and Shah on Deepmind (2023)](https://www.alignmentforum.org/posts/a9SPcZ6GXAg9cNKdi/linkpost-some-high-level-thoughts-on-the-deepmind-alignment)
*   [AI Plans (2023)](https://ai-plans.com/), mostly irrelevant
*   [Macdermott (2023)](https://www.lesswrong.com/posts/3vDb6EzBpaHqDqQif/some-summaries-of-agent-foundations-work-1)
*   [Kirchner et al (2022): unsupervised analysis](https://www.alignmentforum.org/posts/FgjcHiWvADgsocE34/a-descriptive-not-prescriptive-overview-of-current-ai)
*   [Krakovna (2022)](https://vkrakovna.wordpress.com/2022/06/02/paradigms-of-ai-alignment-components-and-enablers/), Paradigms of AI alignment
*   [Hubinger (2020)](https://arxiv.org/abs/2012.07532)
*   [Perret (2020)](https://www.alignmentforum.org/posts/4az2cFrJp3ya4y6Wx/resources-for-ai-alignment-cartography)
*   [Nanda (2021)](https://www.lesswrong.com/posts/SQ9cZtfrzDJmw9A2m/my-overview-of-the-ai-alignment-landscape-a-bird-s-eye-view)
*   [Larsen (2022)](https://www.lesswrong.com/posts/QBAjndPuFbhEXKcCr/my-understanding-of-what-everyone-in-technical-alignment-is)
*   [Zoellner (2022)](https://www.lesswrong.com/posts/8ibDJeoiDuxJkPwfa/various-alignment-strategies-and-how-likely-they-are-to-work)
*   [McDougall (2022)](https://www.perfectlynormal.co.uk/blog-ai-risk-intro-2)
*   [Sharkey et al (2022)](https://www.alignmentforum.org/posts/Jgs7LQwmvErxR9BCC/current-themes-in-mechanistic-interpretability-research#Approaches_grounded_in_the_theory_of_causality) on interp
*   [Koch (2020)](https://www.alignmentforum.org/posts/HEZgGBZTpT4Bov7nH/mapping-the-conceptual-territory-in-ai-existential-safety)
*   [Critch (2020)](https://www.lesswrong.com/posts/hvGoYXi2kgnS3vxqb/some-ai-research-areas-and-their-relevance-to-existential-1)
*   [Hubinger on types of interpretability](https://www.lesswrong.com/posts/nbq2bWLcYmSGup9aF/a-transparency-and-interpretability-tech-tree) (2022)
*   [Tai\_safety\_bibliography (2021)](https://www.zotero.org/groups/2420932/tai_safety_bibliography/collections/2BR54QX6)
*   [Akash and Larsen (2022)](https://www.lesswrong.com/posts/9TWReSDKyshfA66sz/alignment-org-cheat-sheet)
*   [Karnofsky prosaic plan](https://www.lesswrong.com/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very) (2022)
*   [Steinhardt (2019)](https://docs.google.com/document/d/1FbTuRvC4TFWzGYerTKpBU7FJlyvjeOvVYF2uYNFSlOc/edit#heading=h.n1wk9bxo847o)
*   [Russell (2016)](https://arxiv.org/abs/1602.03506)
*   [FLI (2017)](https://futureoflife.org/landscape/ResearchLandscapeExtended.pdf)
*   [Shah (2020)](https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review)
*   [things which claim to be agendas](https://www.lesswrong.com/tag/research-agendas?sortedBy=new)
*   [Tekofsky listing indies (2023)](https://docs.google.com/spreadsheets/d/1Us_eYfpzM5qg52aZh2Cc8O13Fdfiy5CMKmzZhN5ItOY/edit#gid=1246461801)
*   [This thing called boundaries (2023)](https://www.lesswrong.com/posts/fjgoMaBenyXcRDrbX/boundaries-membranes-and-ai-safety-compilation)
*   [Sharkey et al (2022), mech interp](https://www.alignmentforum.org/posts/Jgs7LQwmvErxR9BCC/current-themes-in-mechanistic-interpretability-research)
*   [FLI Governance scorecard](https://pbs.twimg.com/media/F98iVeGX0AAxOoU?format=jpg&name=medium) 
*   [The money](https://donations.vipulnaik.com/?cause_area_filter=AI+safety#donationAmountsByDoneeAndYear)  
     

Appendix: Graveyard
-------------------

*   Ambitious value learning?
*   MIRI youngbloods (see Hebbar)
*   [JW selection theorems](https://www.lesswrong.com/tag/selection-theorems)??
*   [Provably Beneficial Artificial Intelligence](https://people.eecs.berkeley.edu/~russell/papers/russell-bbvabook17-pbai.pdf) (but see Open Agency and Omohundro)
*   [HCH](https://www.lesswrong.com/tag/hch) (see QACI)
*   IDA → Critiques and recursive reward modelling
*   Debate is now called Critiques and ERO
*   [Market-making](https://www.alignmentforum.org/posts/YWwzccGbcHMJMpT45/ai-safety-via-market-making) (Hubinger)
*   Logical inductors
*   [Conditioning Predictive Models: Risks and Strategies](https://arxiv.org/abs/2302.00805)? 
*   [Impact measures](https://www.lesswrong.com/tag/impact-regularization), conservative agency, side effects → “power aversion”
*   [Acceptability Verification](https://www.lesswrong.com/posts/GeabLEXYP7oBMivmF/acceptability-verification-a-research-agenda)
*   Quantilizers
*   [Redwood](https://www.lesswrong.com/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing) interp?
*   [AI Safety Hub](https://www.aisafetyhub.org/our-research)
*   [Enabling Robots to Communicate their Objectives](https://arxiv.org/abs/1702.03465) (early stage interp?)
*   [Aligning narrowly superhuman models](https://www.lesswrong.com/posts/PZtsoaoSLpKjjbMqM/the-case-for-aligning-narrowly-superhuman-models) (Cotra idea; [tiny followup](https://www.alignmentforum.org/posts/TSxAXeHHhgSxR5wGZ/a-summary-of-aligning-narrowly-superhuman-models); lives on as scalable oversight?)
*   automation of semantic interpretability 
    *   i.e. automatically proposing hypotheses instead of just automatically verifying them
*   [Oracle AI](https://www.lesswrong.com/tag/oracle-ai) is not dead so much as ~everything LLM falls under its purview
    *   [Tool AI](https://www.lesswrong.com/tag/tool-ai), similarly, ~is LLMs plugged into a particular interface
*   [EleutherAI: #accelerating-alignment](https://www.lesswrong.com/posts/AhF8iXLu5PchsmyKf/language-model-tools-for-alignment-research) \- AI alignment assistants are live, but it doesn’t seem like EleutherAI is currently working on this
*   [Algorithm Distillation Interpretability](https://docs.google.com/document/d/1CdPvKZQzrkTCFIMfHNr0uukbTGhNF0ohZn_am4Z8kzo/edit#heading=h.9lmc73wscx1r) (Levoso)  
      
     

Appendix: Biology for AI alignment
----------------------------------

Lots of agendas but not clear if anyone besides Byrnes and Thiergart are actively turning the crank. Seems like it would need a billion dollars.  
 

### Human enhancement 

*   *One-sentence summary*: maybe we can give people new sensory modalities, or much higher bandwidth for conceptual information, or much better idea generation, or direct interface with DL systems, or direct interface with sensors, or [transfer learning](https://www.youtube.com/watch?v=6vMO3XmNXe4), and maybe this would help. The old [superbaby](https://fantasticanachronism.com/2021/03/23/two-paths-to-the-future/) dream goes here I suppose.
*   *Theory of change:* maybe this makes us better at alignment research

### Merging 

*   *One-sentence summary*: maybe we can form networked societies of DL systems and brains
*   *Theory of change:* maybe this lets us preserve some human values through bargaining or voting or weird politics.
*   [Cyborgism](https://www.lesswrong.com/posts/bxt7uCiHam4QXrQAA/cyborgism), [Millidge](https://www.lesswrong.com/posts/d74pb97TAqNKwJkc5/bcis-and-the-ecosystem-of-modular-minds), [Dupuis](https://www.lesswrong.com/posts/gLyRQCg6kp5cqTQTm/collective-identity)

### As alignment aid 

*   *One-sentence summary*: maybe we can get really high-quality alignment labels from brain data, maybe we can steer models by training humans to do activation engineering fast and intuitively, maybe we can crack the true human reward function / social instincts and maybe adapt some of them for AGI.
*   *Theory of change:* as you’d guess
*   *Some names*: [Byrnes](https://www.lesswrong.com/posts/tj8AC3vhTnBywdZoA/intro-to-brain-like-agi-safety-15-conclusion-open-problems-1), [Cvitkovic](https://milan.cvitkovic.net/writing/neurotechnology_is_critical_for_ai_alignment/)*, *[Foresight’s BCI,](https://manifund.org/projects/activation-vector-steering-with-bci) Also (list from Byrnes): Eli Sennesh, Adam Safron, Seth Herd, Nathan Helm-Burger, Jon Garcia, Patrick Butlin

  
Appendix: Research support orgs
----------------------------------

One slightly confusing class of org is described by the sample {CAIF, FLI}. Often run by active researchers with serious alignment experience, but usually not following an obvious agenda, delegating a basket of strategies to grantees, doing field-building stuff like NeurIPS workshops and summer schools.  
 

[CAIF](https://www.cooperativeai.com/foundation) 

*   *One-sentence summary*: support researchers making differential progress in cooperative AI (eg precommitment mechanisms that can’t be used to make threats)
*   *Some names*: Lewis Hammond
*   *Estimated # FTEs: *3 
*   *Some outputs in 2023*: [Neurips contest](https://www.cooperativeai.com/contests/melting-pot-2023), [summer school](https://www.cooperativeai.com/summer-school/2023)
*   *Funded by: *[*Polaris Ventures*](https://polaris-ventures.org/grants/)
*   *Critiques:*
*   *Funded by: ?*
*   *Trustworthy command, closure, opsec, common good, alignment mindset: ?*
*   *Resources: *£2,423,943  
     

[AISC](https://aisafety.camp/)

*   *One-sentence summary*: entrypoint for new researchers to test fit and meet collaborators. More recently focussed on a capabilities pause. [Still going!](https://aisafety.camp/#Projects)
*   *Some names*: Remmelt Ellen, Linda Linsefors
*   *Estimated # FTEs: 2* 
*   *Some outputs in 2023*: [tag](https://www.lesswrong.com/tag/ai-safety-camp)
*   *Funded by: ?*
*   *Critiques: ?*
*   *Funded by: ?*
*   *Trustworthy command, closure, opsec, common good, alignment mindset: ?*
*   *Resources:* ~$200,000

See also:

*   [FLI](https://futureoflife.org/)
*   [AISS](https://www.aisafetysupport.org/)
*   [PIBBSS](https://www.pibbss.ai/)
*   [Gladstone AI](https://www.aitracker.org/)
*   [Apart](https://apartresearch.com/)
*   [Catalyze](https://www.catalyze-impact.org/team)
*   [EffiSciences](https://ia.effisciences.org/)
*   Students ([Oxford](https://www.oxai.org/ai-safety), [Harvard](https://haist.ai/), [Groningen](https://www.aisig.org/), [Bristol](https://bristolaisafety.org/about), [Cambridge](https://www.cambridgeaisafety.org/start), [Stanford](https://stanfordaialignment.org/), [Delft](https://www.delftaisafety.org/), [MIT](https://www.mitalignment.org/))  
     

Appendix: Meta, mysteries, more
-------------------------------

*   [Metaphilosophy](https://www.lesswrong.com/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty) ([2023](https://www.lesswrong.com/posts/fJqP9WcnHXBRBeiBg/meta-questions-about-metaphilosophy)) – Wei Dai, probably <1 FTE
*   [Encultured](https://www.encultured.ai/blog-experiments.html) \- making a game
*   Bricman: [https://compphil.github.io/truth/](https://compphil.github.io/truth/) 
*   [Algorithmic Alignment](https://algorithmicalignment.csail.mit.edu/) 
*   McIlrath: [side effects](https://www.southampton.ac.uk/~eg/AAMAS2023/pdfs/p1797.pdf) and [human-aware AI](https://openreview.net/pdf?id=nkn1rcI6wd)
*   [how hard is alignment?](https://www.lesswrong.com/posts/EjgfreeibTXRx9Ham/ten-levels-of-ai-alignment-difficulty#Table)
*   [Intriguing](https://arxiv.org/abs/2012.11538)
*   [The UK government now has an evals/alignment lab of sorts](https://www.gov.uk/government/publications/ai-safety-institute-overview) (Xander Davies, Nitarshan Rajkumar, Krueger, Rumman Chowdhury)
*   [The US government will soon have an evals/alignment lab of sorts, USAISI](https://www.commerce.gov/news/press-releases/2023/11/direction-president-biden-department-commerce-establish-us-artificial). (Elham Tabassi?)
*   [A Roadmap for Robust End-to-End Alignment](https://arxiv.org/abs/1809.01036) (2018) 
*   [Not theory but cool](https://arxiv.org/abs/2110.08176)
*   [Greene Lab Cognitive Oversight](https://www.joshua-greene.net/)? Closest they’ve got is AI governance and they only got $13,800
*   [GATO Framework](https://www.gatoframework.org/gato-framework) seems to have a galaxy-brained global coordination and/or self-aligning AI angle
*   Wentworth [2020](https://www.lesswrong.com/s/ehnG4mseKF6xALmQy) \- a theory of abstraction suitable for embedded agency.
*   [https://www.aintelope.net/](https://www.aintelope.net/)
*   [ARAAC](https://araac.au/) \- Make RL agents that can justify themselves (or other agents with their code as input) in human language at various levels of abstraction ([AI apology](https://link.springer.com/article/10.1007/s00521-023-08586-x), [explainable RL for broad XAI)](https://link.springer.com/article/10.1007/s00521-023-08423-1)
*   [Modeling Cooperation](https://www.modelingcooperation.com/research)
*   Sandwiching ([ex](https://www.lesswrong.com/posts/qwQqZtjWdyDLC4JTB/automated-sandwiching-and-quantifying-human-llm-cooperation))
*   [Tripwires](https://www.lesswrong.com/tag/tripwire)
    *   [Meeseeks AI](https://www.lesswrong.com/posts/7yEFHisCQSCpLnqWQ/mr-meeseeks-as-an-ai-capability-tripwire)
    *   [Shutdown-seeking AI](https://www.lesswrong.com/posts/FgsoWSACQfyyaB5s7/shutdown-seeking-ai)
*   Decision theory ([Levinstein](https://www.levinstein.org/research))
*   [*CAIS: Machine ethics*](https://www.alignmentforum.org/s/FaEBwhhe3otzYKGQt/p/5HtDzRAk7ePWsiL2L#Implementing_Moral_Decision_Making) \- Model intrinsic goods and normative factors, vs mere task preferences, as these will be relevant even under extreme world changes; helps us avoid proxy misspecification as well as value lock-in. Present state of research unknown (some relevant bits in [RepEng](https://arxiv.org/pdf/2310.01405.pdf)).
*   [*CAIS: Power Aversion*](https://www.alignmentforum.org/s/FaEBwhhe3otzYKGQt/p/5HtDzRAk7ePWsiL2L#Power_averseness) \- incentives to models to avoid gaining more power than necessary. Related to mild optimisation and powerseeking. Present state of research unknown.