Harley-ml/lesswrong huggingface dataset:
# Failure Modes of Teaching AI Safety

### **Why I'm writing this**

I'm about to teach [my AI safety course](https://www.lesswrong.com/posts/t3ngnd6Wvo4qeY5FA/i-designed-an-ai-safety-course-for-a-philosophy-department) for the fourth time. As I'm now updating the syllabus for the upcoming semester, I summarize my observations on what can go wrong when teaching AI safety. These have mostly not happened during my teaching but are generally likely to happen - as more AIS courses are developed and taught around the world - and I've especially thought about them when preparing the course so that they don't happen.

### **1\. Alignment feels like a lost cause**

Depending on how x-risk is presented, getting misaligned AI might appear as an inevitable future since the problem is too complex and hard, and there isn't enough time for AI alignment research to generate robust techniques. 

*How to avoid*: make sure to emphasize all the work on alignment and governance and how your students could also be doing some of that (if they wanted to). I'm also updating my syllabus to include the discussion about [AI Pause](https://pauseai.info/). 

### **2\.  There is no (historical/philosophical) context**

Just like with all complex ideas, situating the problem within its context can make a big difference in how it will be perceived. Talking about AI systems that all of a sudden become a threat to humanity is confusing, to say the least. 

*How to avoid*: talk about the foundations of AI, the debate between symbolic AI and connectionism in cognitive science, and set the stage for how we got to contemporary LLMs.

### **3\. Your audience gets (disproportionately more) excited about capabilities**

This is more common with people with technical backgrounds who like to build tools and applications and are blindly or naively excited about technological progress. 

*How to avoid*: clearly explain the conceptual parts of the problem, describe it in more industry-friendly terminology, provide a lot of examples. 

### **4\. But ... what about sexist/racist etc. algorithms? **

Hinting at AI ethics being more important than this alignment story. Also, appears in the context of the [Longtermism](https://www.vox.com/future-perfect/23298870/effective-altruism-longtermism-will-macaskill-future) debate.

*How to avoid*: show how this is a form of misalignment, happening right now. AI ethics should not be opposing AI safety. 

### **5\. There are no actionable next steps**

This is true for many courses. Once you've finished it it's just another letter on your transcript. I think that there are good reasons for an AIS course in particular not to fall into that category. 

*How to avoid*: connect the material to related areas of interest that the students might be more familiar with (e.g., I have many psych or econ majors). Recommend further resources, other courses they could take, potential career paths, or activism. 

You can find my syllabus for Fall 2024 [here](https://eangelou.notion.site/Philosophy-and-the-Challenge-of-the-Future-Artificial-Intelligence-c402339b2de64b0eb8c88f911858a45a).
# In favour of exploring nagging doubts about x-risk

I was chatting recently to someone who had difficulty knowing how to orient to x-risk work (despite being a respected professional working in that field). They expressed that they didn't find it motivating at a gut level in the same way they did with poverty or animal stuff; and relatedly that they felt some disconnect between the arguments that they intellectually believed, and what they felt to be important.

I think that existential security (or something like it) should be one of the top priorities of our time. But I usually feel good about people in this situation paying attention to their gut scepticism or nagging doubts about x-risk (or about parts of particular narratives about x-risk, or whatever). I’d encourage them to spend time trying to name their fears, and see what they think then. And I’d encourage them to talk about these things with other people, or write about the complexities of their thinking.

Partly this is because I don't expect people who are using intellectual arguments to override their gut to do a good job of consistently tracking what the most important things to do are on a micro-scale. So it would be good to get the different parts of them to sync more. 

And partly because it seems like it would be a public good to explore and write about these things. Either their gut is onto something with parts of its scepticism, in which case it would be great to have that articulated; or their gut is wrong, but if other people have similar gut reactions then playing out that internal dialogue in public could be pretty helpful.

It's a bit funny to make this point about x-risk in particular because of course the above all applies to whatever topic. But I think people normally grasp it intuitively, and somehow that's less universal around x-risk. I guess maybe this is because people don't have any first-hand experience with x-risk, so their introductions to it are all via explicit arguments … and it's true that it's a domain where we should be unusually unwilling to trust our gut takes without hearing the arguments, but it seems to me like people are unusually likely to forget that they can know anything which has bearing on the questions without already being explicit (and also that perhaps the social environment, in encouraging people to take explicit arguments seriously, can accidentally overstep and end up discouraging people from taking anything else seriously). These dynamics seem especially strong in the case of AI risk — which I regard as the most serious source of x-risk, but also the one where I most wish people spent more time exploring their nagging doubts.
# Progress Conference 2024: Toward Abundant Futures

![](https://rootsofprogress.org/u/2024/06/jason.crawford_outdoors_convention_center_full_of_people_and_te_f1a11fe0-df6e-416a-854f-a7eb579ef5e2-scaled.jpg)

The progress movement has grown a lot in the last few years. We now have progress journals, think tanks, and fellowships. The progress idea has spread and evolved into the “abundance agenda”, “techno-optimism”, “supply-side progressivism”, “American dynamism”. All of us want to see more scientific, technological, and economic progress for the good of humanity, and envision a bold, ambitious, flourishing future.

What we haven’t had so far is a regular gathering of the community.

Announcing [**Progress Conference 2024**](https://rootsofprogress.org/conference/), a two-day event to connect people in the progress movement. Meet great people, share ideas in deep conversations, catalyze new projects, get energized and inspired.

[**Apply for an invitation here**](https://rootsofprogress.typeform.com/pc2024-apply)

**Hosted by**: the Roots of Progress Institute, together with the Foresight Institute, [HumanProgress.org](http://humanprogress.org/), the Institute for Humane Studies, the Institute for Progress, and Works in Progress magazine

**When:** October 18–19, 2024

**Where:** Berkeley, CA—at the Lighthaven campus, an inviting space perfect for mingling

**Speakers:** Keynotes include Patrick Collison, Tyler Cowen, Jason Crawford, and Steven Pinker. Around 20 additional speakers will share ideas on four tracks: the big idea of human progress, policy for progress, tech for progress, and storytelling/media for progress. [Full speaker list](https://rootsofprogress.org/conference/)

**Attendees:** We expect 200+ intellectuals, builders, policy makers, storytellers, and students. This is an invitation-only event, but anyone can apply for an invitation. Complete the [open application](https://rootsofprogress.typeform.com/pc2024-apply) by July 15th.

**Program:** Two days of intellectual exploration, inspiration and interaction that will help shape the progress movement into a cultural force. Attend talks on topics from tech to policy to culture, build relationships with new people as you hang out on cozy sofas or enjoy the sun in the garden, sign up to run an unconference session and find others who share your interests and passions, or pitch your ideas to those who could help make your dreams a reality.

Special thanks to our early sponsors: Cato Institute, Astera Institute, and Freethink Media! We have more sponsorships open, [view sponsorship opportunities here](https://www.notion.so/Progress-Conference-2024-Sponsorship-Opportunities-f0986734e25440a8bbdd6edd355d163e?pvs=21).
# Countering AI disinformation and deep fakes with digital signatures

According to [The Economist](https://www.economist.com/science-and-technology/2024/05/10/a-russia-linked-network-uses-ai-to-rewrite-real-news-stories), disinformation campaigns (often state-sponsored) use "AI to rewrite real news stories":

> In early March a network of websites, dubbed CopyCop, began publishing stories in English and French on a range of contentious issues. They accused Israel of war crimes, amplified divisive political debates in America over slavery reparations and immigration and spread nonsensical stories about Polish mercenaries in Ukraine... the stories had been taken from legitimate news outlets and modified using large language models.

Deep fakes of still images and now video clips are similarly based on legitimate original photos and video. Detecting such fakery can be challenging.

An obvious partial solution (that I haven't seen discussed) is for legitimate news sources to digitally sign original photos, video, and news stories. 

Web browsers (or extensions) can automatically confirm or flag fakery of the original source (news publisher, journalist, photographer). All that's needed is a consensus standard on how to encode such digital signatures - the sort of thing that W3C and similar organizations produce routinely.

Such signatures could be conveyed in a custom HTML wrapper that needn't be visible to readers with web browsers unable to parse them - there's no need to sprinkle "*BEGIN PGP SIGNED MESSAGE"* at the start of every article; these can be invisible to users.

Signatures could be layered - a photo could be signed by the camera capturing the original (manufacturer, serial number), the photographer (name, nym, unique email address), and publisher, all at the same time, similarly for text news articles.

Video could be signed on a per-frame basis as well as a whole-clip or partial-clip basis. Per frame signatures could include consecutive frame numbers (or timestamps), enabling trivial detection of selective editing to produce out-of-context false impressions.

Writers and photographers who wish to remain anonymous could sign with a nym (pseudonym; a unique identifier under control of an author - for example an email address or unique domain name not publicly connected with an individual); this would still enable anonymous sources to maintain reputations.

Signatures that reliably and uniquely identify original news sources ("nytimes.com", "thefp.com", "aljazeera.com", etc.) could be used by third party services to produce reliability/trust/bias ratings for individual publishers. Again, a simple consensus standard would allow any web browser (or extension) to retrieve such ratings from such third parties (people with differing views will likely trust different rating organizations).

If there's a desire for immutability or verifiable timestamps, articles (or signed article hashes) could be stored on a public blockchain.

Somebody...please pursue this?
# Imbue (Generally Intelligent) continue to make progress

I've been following the company Imbue and their podcast Generally Intelligent since they started. They've said thoughtful and creative things in their podcast, and I think they are making impressive progress towards AGI considering their relatively smaller size.

Just wanting to keep people appraised. If they did hit on something unusually potent, would they get acquired by a larger actor? Hard to know.

What is it that they're releasing? In addition to their 70B param model...

> 11 sanitized and extended NLP reasoning benchmarks including ARC, GSM8K, HellaSwag, and Social IQa
> 
> An original code-focused reasoning benchmark
> 
> A new dataset of 450,000 human judgments about ambiguity in NLP questions
> 
> A hyperparameter optimizer for scaling small experiments to a 70B run
> 
> Infrastructure scripts for bringing a cluster from bare metal to robust high-utilization training

An interesting quote about their hyperparameter optimizer:

> It is possible to run resource-efficient pre-training experiments that can effectively scale to a large model. Using CARBS, we could reliably predict the performance of any model with a given number of parameters according to well-defined scaling laws, lowering the barrier to entry to building large models.

I think this quote is relevant to [this discussion in the comment section of a different post. ](https://www.lesswrong.com/posts/k38sJNLk7YbJA72ST/llm-generality-is-a-timeline-crux?commentId=JrKFyuxjkj9hrvuBo)
# Cross Robin

A common contra dance figure is "right shoulder round": walk around the opposite-role person on the side of the set while looking at them. A less common figure is the "mad robin": walk the same path, but instead look at the opposite-role person across the set. This suggests a third option, which we can call "cross robin": walk that path, but looking at the same-role person diagonally across the set. \[1\]

Here's a potential dance:

> Cross Robin  
> by Jeff Kaufman
> 
> (Beckett)
> 
> **A1**
> 
> *   (4) Long lines forward to the left, progressing
> *   (4) On the way back, Larks roll Robins away (right to left)
> *   (4) Long lines forward in place
> *   (4) On the way back, Larks roll Robins away (left to right)
> 
> **A2**
> 
> *   (8) Right shoulder round your partner while looking at your same-role neighbor.
> *   (8) Larks allemande left 1.5x while Robins orbit CW 1/2
> 
> **B1**
> 
> *   (8) Left shoulder round your partner while looking at your same role-neighbor.
> *   (8) Robins allemande right 1.5x while Larks orbit CCW 1/2
> 
> **B2**
> 
> *   (16) Right shoulder round your partner, and swing.

The goal is that after the cross-looking you should get to do something with that same-role neighbor. If you want, you can maintain eye contact throughout the A2 and B1.

I've tried this with three other people who happened to be in the same room ([Harris](https://chromamine.com/), [Cecilia](https://www.ceciliavacanti.com/), [Lily](https://www.lilywise.com/)) and it seems like it flows, though I'm not sure yet if the cross-looking feels good.

(I considered calling this "Mad Lark", but Harris tells me Shoshana Silverman has been pushing to call it "Mad Robin" when it starts with the Robin going in front and "Mad Lark" when it starts with the Lark going in front, and I didn't want to conflict. If that does catch on, this could be both "Cross Robin" in the B1 and "Cross Lark" in the A2. But I also think it's probably better not to name this figure at all in teaching it to dancers, and just describe it. It's good to keep the number of named figures low.)

  
\[1\] It also suggests a fourth option, where you walk the path while looking at yourself. But that seems more awkward.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02HDgLiSu7VoT6jgZWbMNU4WZi6gRZp373LenxHJ851Cd5yAe1tmN8rpm7cwFt42qal), [lesswrong](https://lesswrong.com/posts/gztJxTLvdfAxu7aRz), [mastodon](https://mastodon.mit.edu/@jefftk/112686365586878771)*
# Secondary forces of debt

A general thing I hadn’t noticed about debts until lately:

*   Whenever Bob owes Alice, then Alice has reason to look after Bob, to the extent that increases the chance he satisfies the debt.
*   Yet at the same time, Bob has an incentive for Alice to disappear, insofar as it would relieve him.

These might be tiny incentives, and not overwhelm for instance Bob’s many reasons for not wanting Alice to disappear. 

But the bigger the owing, the more relevant the incentives. When big enough, the former comes up as entities being “too big to fail”, and potentially rescued from destruction by those who would like them to repay or provide something expected of them in future. But the opposite must exist also: too big to succeed—where the abundance owed to you is so off-putting to provide that those responsible for it would rather disempower you. 

And if both kinds of incentive are around in whisps whenever there is a debt, surely they often get big enough to matter, even before they become the main game. 

For instance, if everyone around owes you a bit of money, I doubt anyone will murder you over it. But I wouldn’t be surprised if it motivated a bit more political disempowerment for you on the margin.

There is a lot of owing that doesn’t arise from formal debt, where these things also apply. If we both agree that I—as your friend—am obliged to help you get to the airport, you may hope that I have energy and fuel and am in a good mood. Whereas I may (regretfully) be relieved when your flight is canceled.

Money is an IOU from society for some stuff later, so having money is another kind of being owed. Perhaps this is part of the common resentment of wealth.

I tentatively take this as reason to avoid debt in all its forms more: it’s not clear that the incentives of alliance in one direction make up for the trouble of the incentives for enmity in the other. And especially so when they are considered together—if you are going to become more aligned with someone, better it be someone who is not simultaneously becoming misaligned with you. Even if such incentives never change your behavior, every person you are obligated to help for an hour on their project is a person for whom you might feel a dash of relief if their project falls apart. And that is not fun to have sitting around in relationships. 

(Inpsired by reading The Debtor’s Revolt by Ben Hoffman lately, which may explicitly say this, but it’s hard to be sure because I didn’t follow it very well. Also perhaps inspired by a recent murder mystery spree, in which my intuitions have absorbed the heuristic that having something owed to you is a solid way to get murdered.)
# Important open problems in voting

**Strategy-resistance**

Identify, or prove impossibility, of a voting system which incentivizes—

1.  A *strictly* sincere ranking of all candidates in the zero-information setting, where it implements a "good" social choice rule such as the relative (normalized) utilitarian rule, a Condorcet social choice rule, or the Borda rule.
2.  In a Poisson game or similar setting: a unique semi-sincere Nash equilibrium that elects the Condorcet winner (if one exists), similar to those shown for approval voting by Myerson and Weber (1993) and [Durand et al. (2019)](https://shs.hal.science/halshs-02049865).

**Properties of Multiwinner voting systems**

There's strikingly little research on multiwinner voting systems. You can find a table of criteria for single-winner systems on Wikipedia, but if you try and find the same for multi-winner systems, there's nothing. Here's 9 important criteria we can judge multiwinner voting systems on:

1.  Independence of Irrelevant Alternatives
2.  Independence of Universally-Approved Candidates
3.  Monotonicity 
4.  Participation
5.  Precinct-summability
6.  Polynomial-time approximation scheme
7.  Proportionality for solid coalitions
8.  [Perfect representation in the limit](https://electowiki.org/wiki/Perfect_representation)
9.  [Core-stability](https://electowiki.org/wiki/Stable_Winner_Set) (may need to be approximated within a [constant factor](https://dl.acm.org/doi/abs/10.1145/3357713.3384238))

I'm curious which combinations of these properties exist. Probabilistic/weighted voting systems are allowed.
# Self-censoring on AI x-risk discussions?

I catch myself sometimes thinking of ideas / scenarios that support higher p(doom), typically as counter-examples to points folks make for lower p(doom), and I wonder how much self-censorship I should apply, given that AI can read these conversations.

My CoT:

1.  I sure don't want to feed ideas to any bad actor.
2.  But it's arrogant to think that anything I can come up with wouldn't already be obvious to an entity with paperclipping-level power.
3.  In chess, an easy way to make mistakes is by defending against imaginary threats, or even real threats which aren't the most dangerous ones on the board, or threats whose defense is costlier than what you forego by not making other good moves available to you like a counterattack.
4.  In dictatorships, preventing people from coordinating with one another e.g. by convincing everyone that their neighbor squawks to the secret police, is a very effective way for a few to hold control over many. So when you're up against a powerful singular threat, coordination is important!
5.  Yet, I can't shake a queazy feeling at the thought of putting out dangerous ideas. Perhaps, somehow, the space of savant-smart AI systems who are powerful enough to paperclip, yet may not be generically smart enough to have lots of random ideas, is not so small as to be ignored?

Do others have any useful guidelines, thoughts or intuition here? What am I missing?
# New Executive Team & Board —  PIBBSS

TLDR:[*PIBBSS*](http://pibbss.ai) *is changing its core team. Nora is stepping down as director due to joining ARIA, and Lucas Teixeira and Dusan Nesic are taking over her leadership role. Nora joins the board, alongside Tan Zhi Xuan, Alexander Gietelink Oldenziel, Ben Goldhaber and Gabriel Weil.*

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/eaa953986163c8b33e64bb8ddbf75d3706a0210073c22b3a.png)

I (Nora) have recently accepted an offer to join [ARIA’s Safeguarded AI Programme](http://www.aria.org.uk/programme-safeguarded-ai) as Technical Specialist under davidad. As such, I am stepping back as Director at PIBBSS, after co-founding and leading PIBBSS since 2021. 

It wasn’t an easy choice to make! I deeply care about and believe in the mission of and the people at PIBBSS. Before davidad encouraged me to apply for the role, I hadn’t considered leaving PIBBSS. I believe PIBBSS is playing an important role in terms of fostering theoretically ambitious and empirically grounded AI safety research. I am very excited about the directions the team and I have been forging, and extremely impressed by the quality of talent we've recently been able to attract. I strongly believe that PIBBSS is in the position to make important and neglected contributions in both research and field-building for AI safety.

The team and I have been reflecting on and preparing for this transition for a while. Thanks to that, I am confident that Lucas & Dušan will do a great job at shepherding PIBBSS through this transition, and beyond! We have done our homework, and I feel grateful about being able to put so much my trust into this team.

As such, Lucas & Dušan will collectively form the new Executive Team. Dušan has been leading PIBBSS’ operations for the last ~2 years and has developed a deep familiarity with everything involved in making the organization run smoothly. Lucas, who joined us a bit over 8 months ago, has been acting as research manager and collaborator for our research affiliates. Going forward, Dušan continues to be in charge of all operational matters, and Lucas will be leading the research activities. Together, we have made significant progress in clarifying and moving towards our updated research & field building vision over the last number of months. 

In order to further support this transition, and strengthen PIBBSS in pursuing its ambitious plans, we have also set up a board. We’re pleased to have the following people join the board (in addition to myself): 

*   [Tan Zhi Xuan](https://ztangent.github.io/)
*   [Alexander Gietelink Oldenziel](https://sites.google.com/view/afdago/home)
*   [Ben Goldhaber](https://www.linkedin.com/in/bgoldhaber/) 
*   [Gabriel Weil](https://www.tourolaw.edu/abouttourolaw/bio/399)

I am immensely grateful to my team, our affiliates, our current and past fellows, and all the many wonderful collaborators and 'friends of PIBBSS' over the years! And I am excited to be able to continue supporting PIBBSS from my new position on the board.
# You Have Now Subscribed To Bug Facts

 *Programming note: This meetup is taking place at a non-standard time (6:45) and location (Columbia Lake); plan accordingly!*

This Thursday, we'll be meeting at the Columbia Lake disc golf course. Bring water and bug spray and dress appropriately for getting real close to nature.

Meet by the cylindrical structure near "[Columbia Lake Fire Pit 2](https://goo.gl/maps/ktVKf2zkM74jaLam8)" (same place as the disc golf skillshare meeting) at ~6:45; the group walk will start at 7:00. There isn't great transit there, so organize a few carpools in the discord if that's required.

Parking: Our meetup location is right beside a small gravel lot. It might be full; if it is, there are plenty of other parking lots nearby.

Topic
-----

This week, we shall rekindle the childlike wonder of our youth, channeling the spirit of those halcyon grade school recesses when we, as nascent naturalists, spent the entire period following a cool looking beetle around in naive attempts to understand its umwelt. Yep, we're just gonna find some bugs and follow them around, following our own curiousity.

That's it, that's the meetup. I sadly won't be here, but we'll have a regular who will supply us with the requisite bug facts, and the Vancouver organizer has gracefully agreed to nominally co-run the meetup. He's just starting out with organizing so he can use the practice ;)

Readings
--------

[iNaturalist list of bugs seen at Columbia Lake](https://www.inaturalist.org/places/145695#taxon=47120) (crowdsourced)

[Quick Overview of the Orders of Insects](https://bugguide.net/node/view/222292) (Beatriz Moisset, 2008)

**Optional:**

[Introduction to Insects](https://www.huck.psu.edu/assets/uploads/documents/Introduction-to-Insects.pdf) (Penn State Insect Biodiversity Center, 2022)  
*this is a longer, more in depth guide to bugs. pg 1-7 for basic insect facts, pg8-22 for summaries of the various orders of insects*

[Introduction and Chapter 1 of Naturalist](https://drive.google.com/file/d/1hf8XYN64u94cYsQDUURj0bjJZriiKAn4/view?usp=sharing) (E. O. Wilson, 1994)  
*just for vibes :)*
# Creating an Exobrain

**Workshop description:**
-------------------------

We will learn how to choose and manage a note-taking system, a task-management system, and a calendar system. Collectively, these tools are affectionately referred to as an "Exobrain".

That said, this isn't a typical productivity course. "Productivity" implies a value judgment on "being productive" or (even worse) "being busy". Your worth is not measured by how much time you spend on the grind. Our goal with this workshop is to help you be \*lazier\*, to help free up time and energy to do things you want to do.

**Notes:**
----------

*   This is a free event hosted by the Guild of the Rose, an organization dedicated to community-based self-improvement. During this inaugural event, you will form a "Cohort", a structured study group that will take responsibility for hosting future Guild workshops. We will provide the content as well as the system of collaboration with the rest of our online community.
*   This event takes place in the meeting room above Minsky's Pizza, which gets full waitstaff service. The meeting room has a maximum tab limit of 5. If you plan to order anything from the menu, be prepared to share a tab with a small group of people. Bringing cash is easiest, Venmo is the second-easiest.
# Open Sourcing Metaculus

*This is a linkpost for an announcement by Metaculus CEO,* [*Deger Turan*](https://x.com/degerturann)*, originally published June 5th, 2024. In addition to spreading the word here, we are interested to gather feedback and ideas from the EA community.*

We make reasoning and coordination tools for the public benefit, and that requires the trust and participation of the wider forecasting community. That’s why we’re going open source. GitHub issues, pull requests, and community-sourced forecasts will dictate much of what we build and how. Ultimately, forecasters will benefit from better infrastructure, the Metaculus team will benefit from more idea diversity and greater capacity, and readers will get more accurate aggregates.

What’s next
-----------

Metaculus will go open source in Q3. There will be opportunities to audit, critique, and improve on our efforts. Creating a better epistemic environment is a collective endeavor.

As part of open-sourcing Metaculus, we’ll help implement our best features into other open-source forecasting codebases. We’ll make our roadmap and building process public, and we’ll give more visibility into our decision-making criteria and discussions. Our codebase is far from perfect, but we can improve it better together out in the open. 

In a future post I’ll write more about how others can contribute to this project, and specifications for how code can be modified, used, and distributed—as well as guidelines for coding standards and our submission process. We’ll provide processes for pull requests, and how we’ll implement submissions from the community.

We want to be transparent that we may still choose to offer proprietary offerings targeting particular enterprise and commercial use cases: We believe that part of making forecasting more useful and more used is by productizing it and generating revenue. 

Why isn’t Metaculus already open source?
----------------------------------------

Before I joined as CEO I wondered why Metaculus wasn’t already open source. I discovered the idea was being discussed, but that there were some concerns about platform security, about accessibility of the codebase and about the potential for exploits that would allow some forecasters to game the system and create an uneven playing field. Now that we’ve been able to observe the new scoring system at work for some time with no issues, we’re confident the benefits outweigh the risks. While the codebase is nowhere near perfect, we believe making improvements on the platform is better done in the open, with feedback from the larger ecosystem guiding us explicitly. 

We’re looking for

*   Researchers who want to backtest their aggregation algorithms on our data
*   UX specialists who see better ways to translate beliefs into forecasts
*   Nonprofits who want to spin up their own internal forecasting engines built on our architecture
*   Developers who can help add new features and enhance existing ones

As our plan evolves I’ll provide more details. Until then, let me know what you think: **What resources and documentation will you want to see? What would make contributing easier? **
# What percent of the sun would a Dyson Sphere cover?

I disagreed with a bunch of the implications of [this comment](https://www.lesswrong.com/posts/87EzRDAHkQJptLthE/but-why-would-the-ai-kill-us?commentId=LgjwepWmnJKjXvoQK), but I was curious about the specific question "Would a dyson sphere made out of the solar system necessarily cover (most of) the sun?" (and therefore block out a substantial fraction of light coming to Earth).

The subquestions here seem to be (at first glance, not a physicist)

*   What are efficient Dyson spheres probably made of?
*   What percent of the solar system can be converted into Dyson-sphere material?
    *   Are gas giants harvestable?
*   How long would it take to harvest that material?
*   What would the radius of a Dyson sphere be? (i.e. how far away from the sun is optimal). How thick? 
*   If the sphere is (presumably) lots of small modules, how far apart are they?

I don't know if there's already been a canonical answer written up somewhere. The original motivating question was "if an AI is moderately 'nice', leaves Earth alone but does end up converting the rest of the solar system into a Dyson sphere, how fucked is Earth? (also, on what timescale?). 

I don't know that this question actually makes sense (as another commenter mentioned, if the AI is that nice, it can probably also redirect sunlight to Earth at low cost. But, I'm still just curious about the details. (I have enough uncertainty about how the future plays out that it seems nice to understand some of the physical limits involved)
# ML4Good Summer Bootcamps - Applications Open [deadline extended]

ML4Good bootcamps are 10-day bootcamps focusing on upskilling in technical AI safety, exploring governance, and delving into conceptual topics. ML4Good is a non-profit project and freeof charge for the participants. We're seeking motivated individuals with some coding experience who want to make a difference in the field of AI Safety.

**💻 Activities:**
==================

*   Peer-coding sessions following a technical curriculum with mentors
*   Presentations by experts in the field
*   Review and discussion of AI Safety literature
*   Personal career advice and mentorship

**📅 Dates:**
=============

Applications are now open for our UK and Germany bootcamps. 

**🇬🇧 UK Bootcamp**
--------------------

[**Apply to the 🇬🇧 UK Bootcamp now**](https://www.ml4good.org/courses/ml4good-uk-august-2024)

*   August 31st to September 10th, 2024
*   Application deadline: July 14th, 2024 **\[deadline extended\]**

**🇩🇪 Germany Bootcamp**
-------------------------

[**Apply to the 🇩🇪 Germany Bootcamp now**](https://www.ml4good.org/courses/ml4good-germany-september-2024)

*   September 23rd to October 3rd, 2024
*   Application deadline: July 14th, 2024

More camps are being planned - sign up on our website to stay informed about upcoming bootcamps and application deadlines.

🌐 More details on our website: [https://www.ml4good.org/programmes/upcoming-bootcamps](https://www.ml4good.org/programmes/upcoming-bootcamps)
# What progress have we made on automated auditing?

One use case for model internals work is to perform automated auditing of models:

[https://www.alignmentforum.org/posts/cQwT8asti3kyA62zc/automating-auditing-an-ambitious-concrete-technical-research](https://www.alignmentforum.org/posts/cQwT8asti3kyA62zc/automating-auditing-an-ambitious-concrete-technical-research)

That is, given a specification of intended behavior, the attacker produces a model that doesn't satisfy the spec, and the auditor needs to determine how the model doesn't satisfy the spec. This is closely related to *static backdoor detection*: given a model M, determine if there exists a backdoor function that, for any input, transforms that input to one where M has different behavior$.$[^bfm3j6jrbqt]

There's some theoretical work ([Goldwasser et al. 2022](https://arxiv.org/abs/2204.06974)) arguing that for some model classes, static backdoor detection is impossible *even given white-box model access* \-\- specifically, they prove their results for random feature regression and (the very similar setting) of wide 1-layer ReLU networks. 

Relatedly, there's been some work looking at provably bounding model performance ([Gross et al. 2024](https://arxiv.org/abs/2406.11779)) \-\- if this succeeds on "real" models and "real" specification, then this would solve the automated auditing game. But the results so far are on toy transformers, and are quite weak in general (in part because the task is so difficult).[^l8uw3g0441] 

Probably the most relevant work is [Halawi et al. 2024's Covert Malicious Finetuning](https://www.alignmentforum.org/posts/33emJkmw5bMAXZHHt/covert-malicious-finetuning-1) (CMFT), where they demonstrate that it's possible to use finetuning to insert jailbreaks and extract harmful work, in ways that are hard to detect with ordinary harmlessness classifiers.[^vbix9mxp6ya] 

As this is machine learning, just because something is impossible in theory and difficult on toy models doesn't mean we can't do this in practice. It seems plausible to me that we've demonstrated non-zero empirical results in terms of automatically auditing model internals. So I'm curious: how much progress have we made on automated auditing empirically? What work exists in this area? What does the state-of-the-art in automated editing look like?

[^bfm3j6jrbqt]: Note that I'm not asking about mechanistically anomaly detection/dynamic backdoor detection; I'm aware that it's pretty easy to distinguish if a particular example is backdoored using baseline techniques like "fit a Gaussian density on activations and look at the log prob of the activations on each input" or "fit a linear probe on a handful of examples using logistic regression".  

[^l8uw3g0441]: I'm also aware of some of the work in the trojan detection space, including 2023 Trojan detection contest, where performance on extracting embedded triggers was little better than chance.  

[^vbix9mxp6ya]: That being said, it's plausible that dynamically detecting them given model internals is very easy.
# Travel Buffer

I recently finished a [nine-day road trip](https://www.jefftk.com/p/kingfisher-summer-tour-2024), playing a series of dances. This combined driving long distances with arrival deadlines: you don't want to arrive late. Not only is it unprofessional, but it's stressful rushing to set up and you'll likely have a bad night from skipping some nice-to-have steps. A concept I found pretty useful was thinking about allocating "buffer".

Let's say it's 8am in Pittsburgh PA and the hall in Bloomington IN opens for setup at 5:30pm. GPS says its a 6.5hr drive, so we have 3hr of buffer to spend. Some of the buffer we'll need to spend stopping for gas and restrooms. We might choose to spend additional buffer on a relaxed breakfast, stopping at interesting places along the way, or exploring Bloomington. Or we might be unlucky with traffic (or, heaven forfend, the car) and lose some buffer to bad luck.

Since the risk of things going wrong or taking longer than you expect is roughly proportional to distance, it's pretty risky to front-load your buffer consumption. You don't want to spend all by 30min early on and then run into a 1hr traffic jam. But the most enjoyable ways of spending buffer are probably distributed along the route, so the safest option of reserving it all for the destination isn't very pleasant. It's much nicer to spend the marginal half hour with your toes in a shady stream than waiting around in the parking lot outside the hall.

While this was somewhat useful in my own planning, the place where it really demonstrated its value was in talking with my tourmates. Getting close to the hall I might ask if anyone had anything they wanted to spend buffer on; playing at a park I could use it to explain to my kids why we should leave soon; it avoided people mistaking the GPS arrival time for our actual arrival; any proposed activity had a nice currency for considering its cost.

I don't remember thinking about this explicitly on past tours, or missing it before. I think the main reason is that we scheduled this tour much more tightly. On days when we had a lot of driving we still wanted to take nice breaks (one park had a [serious zipline and a 90ft slide](https://www.tripadvisor.com/Attraction_Review-g36329-d23944019-Reviews-The_Woodlands_Adventure_For_All_Ages-Martinsville_Illinois.html)) and on days when we had less driving we did a lot of seeing things (boating on an [underground river](https://www.tripadvisor.com/Attraction_Review-g36928-d1520788-Reviews-Bluespring_Caverns-Bedford_Indiana.html)). I expect it to continue to be rare for me to be in a situation where I need to do this kind of collaborative planning around a deadline, but when I'm next doing it I think this will be a useful tool.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02SRHVkjrFRb7YHAZYYPwqbjLfdK11QZrcwvR41RFLjnzeiHypVXUerUbGSpqKQLYXl), [lesswrong](https://lesswrong.com/posts/N7owFAHWgALNWs25d), [mastodon](https://mastodon.mit.edu/@jefftk/112737115486900592)*
# Can agents coordinate on randomness without outside sources?

There are situations where two agents that can read each other’s source code want to have a bit of random information (e.g., they want to cooperate and split an indivisible thing by deciding randomly who’ll have it and having half of it in expectation).

If these agents don’t have access to unpredictable sources of randomness (e.g., it’s acausal trade through very detailed simulations and they can perfectly predict parts of each other’s environment), is there any way for them to coordinate on generating a random bit in a way that can’t be manipulated/exploited?

I feel like the answer should be “no”, as schemes like “both transparently don’t look at a part of the other’s source code when they coordinate on the scheme, then look and generate the bit from these parts” fail due to the possibility of something else looking at the whole of the agent A’s source code first and then designing a successor/a simulated agent B which won’t look there but will produce the needed bit.

But maybe I’m missing something?
# [Linkpost] A Case for AI Consciousness

Just wanted to share a new paper on AI consciousness with Simon Goldstein that members of this community might be interested in. Here's the abstract:

*It is generally assumed that existing artificial systems are not phenomenally conscious, and that the construction of phenomenally conscious artificial systems would require significant technological progress if it is possible at all. We challenge this assumption by arguing that if Global Workspace Theory (GWT) — a leading scientific theory of phenomenal consciousness — is correct, then instances of one widely implemented AI architecture, the artificial language agent, might easily be made phenomenally conscious if they are not already. Along the way, we articulate an explicit methodology for thinking about how to apply scientific theories of consciousness to artificial systems and employ this methodology to arrive at a set of necessary and sufficient conditions for phenomenal consciousness according to GWT.*
# Robin Hanson & Liron Shapira Debate AI X-Risk

Robin and I just had an interesting 2-hour AI doom debate. We picked up where the [Hanson-Yudkowsky Foom Debate](https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate) left off in 2008, revisiting key arguments in the light of recent AI advances.

![A video thumbnail with the text "AI Doom Debate" and side-by-side photos of Robin Hanson and Liron Shapira](https://pbs.twimg.com/media/GR_gq3GboAEJjG8?format=jpg&name=small)

My position is similar to Eliezer's: P(doom) on the order of 50%.

Robin's position remains shockingly different: P(doom) < 1%.

I think we managed to illuminate some of our cruxes of disagreement, though by no means all. Let us know your thoughts and feedback!

Where To Watch/Listen/Read
==========================

For most casually-interested readers, I recommend consuming the debate via my [**debate highlights and analysis**](https://www.lesswrong.com/posts/quj3RniAw7GGJtqeX/robin-hanson-ai-x-risk-debate-highlights-and-analysis) post, which has a clean transcript and its own video podcast to go with it.

Other links
-----------

*   [YouTube video](https://www.youtube.com/watch?v=dTQb6N3_zu8)
*   [Podcast audio](https://open.spotify.com/episode/18pjy51oKBp7gDp4Vrxfoo?si=zv7gKt8DQBCrP5zNjjeiXQ)
*   [Raw Transcript](https://share.descript.com/view/yfASo1J11e0)

Topics
======

*   AI timelines
*   The "outside view" of economic growth trends
*   Future economic doubling times
*   The role of culture in human intelligence
*   Lessons from human evolution and brain size
*   Intelligence increase gradient near human level
*   Bostrom's Vulnerable World hypothesis
*   The optimization-power view
*   Feasibility of AI alignment
*   Will AI be "above the law" relative to humans

About Doom Debates
==================

My podcast, [Doom Debates](https://youtube.com/@DoomDebates), hosts high-quality debates between people who don't see eye-to-eye on the urgent issue of AI extinction risk.

All kinds of guests are welcome, from luminaries to curious randos. If you're interested to be part of an episode, DM me here or contact me via [Twitter](https://x.com/liron) or [email](mailto:wiseguy@gmail.com).

If you're interested in the content, please subscribe and share it to help grow its reach.
# UC Berkeley course on LLMs and ML Safety

The UC Berkeley course I co-taught now has lecture videos available: [https://www.youtube.com/playlist?list=PLJ66BAXN6D8H_gRQJGjmbnS5qCWoxJNfe](https://www.youtube.com/playlist?list=PLJ66BAXN6D8H_gRQJGjmbnS5qCWoxJNfe)

Course site: [Understanding LLMs: Foundations and Safety](https://rdi.berkeley.edu/understanding_llms/s24)

Unrelatedly, a more conceptual AI safety course has its content available at [https://www.aisafetybook.com/](https://www.aisafetybook.com/)
# On passing Complete and Honest Ideological Turing Tests (CHITTs)

*\[Cross-posted from Facebook\]*

"In the moment when I truly understand my enemy, understand him well enough to defeat him, then in that very moment I also love him. I think it’s impossible to really understand somebody, what they want, what they believe, and not love them the way they love themselves."

\- Ender Wiggin (from Ender's Game, by Orson Scott Card)

Maybe the following is obvious, but it wasn't initially obvious to me so I'm sharing it anyway.

The term [Ideological Turing Test](https://www.lesswrong.com/tag/ideological-turing-tests) (ITT) was coined by Bryan Caplan and refers to whether someone can articulate and argue the views of an opposing ideology or belief system so convincingly that others cannot tell the difference between the person's articulation and that of a true believer.

Let's use the term Complete and Honest Ideological Turing Test (CHITT) to refer to an ITT where not only can you "fool" observers, but also:

1.  You can do so without resorting to any rhetorical tricks such as leaving out relevant information;
2.  You can do so without making the other side look irrational;
3.  You can do this even for very long debates with expert observers who grill you closely;
4.  You can do this in response to any evidence or argument or counterargument that you yourself might use to argue for your true position.

Claim: It's impossible to pass a CHITT unless you actually agree that from some perspective the other side is rational. Furthermore, you must not have any knockdown arguments for why that other perspective is incorrect. That doesn't mean you need to \*agree\* with the other perspective, just that you don't have a knockdown argument against it.

Reasoning: By the specified criteria, you need to think that no matter what arguments you yourself would throw against the other side, you would not make the other side look irrational even to expert observers. You can of course still have some set of priors that lead you to your true position, while the other side has different priors that lead to their position. But why do you go with your priors and not theirs? If you have a knockdown argument for why your priors are correct, then that's part of the arguments you could throw at the other side, and we're back to where we were before. So it must be that you don't have any knockdown arguments for why you should go with your priors over theirs.
# New page: Integrity

There's a [new page](https://ailabwatch.org/resources/integrity/) collecting integrity incidents at the frontier AI labs.

Also a month ago I made a [page](https://ailabwatch.org/resources/company-advocacy/) on labs' policy advocacy.

If you have suggestions to improve these pages, or have ideas for other resources I should create, [let me know](https://ailabwatch.org/contact/).

* * *

*Crossposted from* [*AI Lab Watch*](https://ailabwatch.org/blog/)*. Subscribe on* [*Substack*](https://ailabwatch.substack.com)*.*